Search the Omeda Knowledge Base
Data Loader – Loading Data
Initial Layout and Uploading
When the User arrives to the Data Loader section, if no file has been uploaded previously for the Selected Profile (Figure-1.C), they will be met with a screen similar to Figure-1. Otherwise, the only difference will be a list of previously uploaded files instead of the splash screen suggesting to drag a file onto the window.
Data Loader currently only accepts Delimited Files. They can be either comma, pipe, semicolon, or tab delimited.
To upload a file, the User can either use the Upload File Button(s) (Figure-1.A) or drag-and-drop the file into the Dropzone (Figure-1.B). The Upload File Button will open up a standard file-browser, allowing the user to search their system for the desired file. The Dropzone is signified by the dotted outline around the content of the page. Either will bring up the intermediary dialog for supplying additional information prior to submitting the file(s) for upload.
Upload supports multiple files, shown by Figure-2.A. The User may use the file-named tabs to switch between options they want to setup individually for each file. Each file should be reviewed using these tabs. A Delimiter must be specified for each file (Figure-2.C). Data Loader will try to predict which delimiter it is by the file’s extension, otherwise it will default to comma and may be changed to a more appropriate one if needed. The user may also specify a Template (Figure-2.B) to be used as a baseline of the file, but it is not required. Templates are explained in more depth in their own article.
When the User is satisfied with the selected options for each file, they may click the Upload Button (Figure-2.D). This will send the files to Omeda which will prepare the system for the next steps.
After the dialog closes, Data Loader should change to show the files that have been uploaded (see screenshot above). The main information displayed for each file is as follows:
- File Name
- Upload Date/Time
- User Note indicator
- This explained in the next section.
- If a User has added a Note to the file, a little sticky-note image will appear on this row.
- Template used (if any)
- Who uploaded the file (Owner)
- Status of the File
Upon uploading, the file’s initial state is Pending. Once the system picks up the file will change to Ready to Map.
Depending on the size of the file, this could take some time. While the file is being loaded into the system, it will make note of various aspects of the file to attempt to make the rest of the process proceed smoothly. It will grab the header row and attempt to predict which incoming field will map to one of Omeda’s fields. This works best with the customer and contact related fields as they are to be the most universal. Doing this will potentially speed up the Field Mapping step later, so it is recommended for file headers for such fields to have common naming. An example of this is using similar variations of “First Name”, “first_name”, or “fname” to represent “first name”.
The other main thing this process does is to note each row and column. Some minor validation exist at this step, but is mostly related to the structure of the file’s rows and not the actual content of the file. An example of this is if a row is found to have a number of cells less or more than the number of cells. In such a case, the row will be marked and skipped.
Details and Actions
Once the file has been uploaded, it will likely end up in the Ready to Map state. At this point the User can interact with the file in various ways. The most basic of which is to click on the file’s row. This will open a panel from the right (Figure-4.A) which is used to give a bunch of basic information pertaining to the file.
- Last Update – This marks the last time something about the file was changed. Most interactions with the file that causes information about it to change will affect this time.
- Last Updated By – This is the user who touched something about the file recently. This user is tied to the “Last Update” timestamp.
- File Type – This is just stating what type the file was identified as.
- Rows – This is typically the number of rows in the file excluding the header row. This represents the possible rows that could be processed. If a row was found in error as part of the upload process, they will still be counted here.
- The “Record” and “Record Errors” counts should total this count.
- Columns – This is the number of headers found in the file.
- Records – This is the number of rows that were successfully processed to the database.
- Record Errors – This is the number of rows that either partially or completely failed to process. This includes any rows that may have been skipped as part of the upload.
Tracking Code & Sample Records:
- Batch Tracking Code – This area will be filled after the file has processed at least once. This will be used to identify records created or changed by this file. This can be used in Audience Builder to look up which customers were involved with this file. This will include existing customers, if they were found in the file.
For additional steps on using Audience Builder to lookup Data Loader files, see Querying Data Loader in Audience Builder (OnQ)
- The Sample Record Preview Card offers one record at a glance. Initially, only the first six headers and their values are shown, but clicking the Show All will reveal any others.
You can add various keywords (separated by commas) to the file being loaded. Much like Keywords for saved queries, this will allow you to search for all associated files with the same keywords within Audience Builder (OnQ).
An additional feature of this pane is being able to add a Note to the file. This is a purely optional space used to communicate something important or clerical about the file and doesn’t need to be used for every file. Only one note can be stored at time. This means the User can edit an existing one or completely rewrite it. Saving a new note will cause the previous note to no longer exist. The user can tell which file has a Note by looking for a sticky-note image among the file list.
The button at the end of the file’s row will display a menu of further actions the User may take with that particular file
Choosing this will open the Field Mapping dialog where the User can dictate how the system interprets the file. The process of this is explained in the next section.
This option is how to take a successfully mapped file – it will be in the state of Ready to Process – and attempt to put the data in the database. This will go over in more detail in a later section.
Upon selecting it, a warning will appear to inform the User that processing the file is something that can potentially ruin their database and cannot simply be undone.
View File State:
If there are any rows that have errors, this section will list them and the errors associated with those errors. Currently, those rows will need to be fixed and re-submitted via a new Upload. A future goal is to allow the user to change data related to certain errors and re-process instead of uploading a new file.
Note: if there are excessive errors (60% or more of the file) found within the data, the file will immediately stop processing and return an ‘excessive error’ status. You are able to delete the file without any impact on your database. At this point, the records have not yet been processed.
This option is for removing the file from the system. This could be because it was uploaded accidentally or the file didn’t upload correctly. This option only works after the file has been uploaded and before it has been processed.
If the Field Mapping tied to the File has been set to be used as a Template, the Template will still exist even if the File is removed. This way, if the file needs to be re-uploaded, the changes to the Field Mapping can be saved and used instead of having to re-map the files.
General Aspects of the Dialog
There are many elements of the dialog that appear in every step, keeping certain information and actions present whenever the User decides it to be useful. Each step has a small description (Figure-5.A) to explain the goal either the step is presenting or the user is to accomplish.
In the top right corner, the number of Records in the file is displayed (Figure-5.F). This is basically the number of non-header rows in the file. The “Jump to” drop down (Figure-5.E) can be used to jump to different mapping tasks, in any order.
Along the bottom the User will find information about the Field Mapping and the navigational buttons.
Field Mapping Validation/Feedback
Warnings, errors, and general information is tracked along the bottom (Figure-5.H). Clicking either the Warnings or Errors will bring up a dialog that will display a brief explanation about things that the User might need to take action upon.
Warnings do not prevent the file from processing and is more to communicate that something about the Field Mapping doesn’t seem correct. An example is mapping both “Postal Address Id” and “Customer Id”. These serve the same purpose – looking up an existing customer – so mapping them at the same time is redundant. Another example is mapping a full name, such as “First Name Last Name”, while mapping separately “First Name” and/or “Last Name” in the same file will warn that the name could potentially not appear as desired.
Errors prevent the file from processing and need to be acted upon before the file can be processed. The most common one of these is the requirement of a known Data Source. Some other option besides the “Unidentified” should be chosen instead.
Unmapped Fields (introduced with the Template feature) will have it’s own little alert in this area. Having Unmapped Fields is not an issue and there is no requirement to have all fields mapped. This just keeps track of how many fields are currently unmapped, just in case the User would like to know this information. In the General Mapping step, these unmapped fields are highlighted as such so that they can be found at a glance.
To go from step-to-step, the User will use the “Back” and “Next” Buttons (Figure-5.I). The “Finish” Button that appears, marks the last step. However, it is possible that changes to the Field Mapping could add or remove steps, causing the last step to change. More about this is discussed in the General Mapping step, as this case is typically triggered from there.
This is the initial step when viewing any uploaded file. In the content section of the dialog (Figure-5.G), a snippet of the file (first of up to 100 rows) is displayed. The rows and columns are order as they appear in the file. This step is used to get a feel of how the file uploaded and how the system will process the information. The best way to use this step is to see at a glance if the data is being placed in the proper columns. With delimited files, if the file is malformed or the wrong delimiter is chosen, data could end up being mapped to the wrong fields. If data doesn’t look proper, the User should make sure the file has columns formatted properly.
For example, if the File is delimited by comma and one of the fields is supposed to be “Last Name, First Name” in the same column, then each cell should be wrapped in quotes to prevent something like “Doe, Jane” from being viewed as 2 columns instead of 1. Something like this could cause the row to be skipped during upload or data being misplaced in other columns.
General Mapping Step
The overall goal of this step is to map incoming headers to their equivalent Omeda fields. Near the top of the page (Figure-6.A), information reminding what file is being mapped to which brand is displayed. Below that is a place to declare the source of the data in the file (Figure-6.B). “Undefined” is not an allowed Data Source, so the user is forced to change it. The remainder is mapping each field (Figure-6.C). The headers in the left column can be adjusted and edited if you would like a different naming convention than what was uploaded. NOTE: The headers cannot be left blank – they will prevent a user from saving the work until a header is input.
During the upload process, some of fields may have been identified and already mapped. These are to be viewed as suggestions and can be changed by the User as they see fit. Any field that was not suggested will be set to –Not Mapped–.
By default, most contact fields are considered of the Contact Type “Business” (Figure-6.D). This can be changed by the User by choosing another available option. It is important that the user makes sure all relevant fields have a matching Contact Type. If it is setup that all of the fields are tied to the Contact Type “Business” except, for example, City is set to “Home”, then it will result in two different incomplete addresses. If the file contains a way to identify an existing customer, then mapping contact information will cause a new step to appear.
In addition to contact information, mapping can also be applied to demographic data such as job title or company size (Figure-6.5). Details on uploading your demographic data can be found on the Data Loader Demographics page.
Lastly, you can also map to any number of deployment types (or all) and decide if the value is Opt-in or Opt-out.
When you have a column in the original file that you don’t intend to map to the database you can choose to delete/inactivate the header, rather than removing the column from the original spreadsheet.
- To remove/inactivate a header, click the garbage can icon to the left of the header name:
This will move the header to an ‘Inactive Headers’ section below the rest of the data.
You can reactivate a header by clicking the ‘reverse’ icon to the left of the column header. This will reinstate the header and allow you to map it to the database.
For any step that can change aspects about the mapping, if the User tries to leave the step – such as clicking Finish (Figure-6.E) – they will be prompted with a confirmation screen (shown below).
Cancel will close the dialog, useful if it was accidentally opened. Continue without Saving will perform the action that was used to trigger this dialog, but no changes will be saved. Save and Continue will save the changes and proceed with whatever action caused the dialog to appear.
If the changes were saved, there is a chance that the number of steps may have changed. If this happened, then a dialog will open noting the changes (Figure-8). If “Finish” was clicked prior to saving, if more steps were added, the next step will be displayed. If “Next” was clicked and steps were removed, then the dialog will close stating that no more steps existed.
Contact Type Step
This step only appears if the User has mapped Postal and/or Email related fields and has mapped a way to detect an existing customer (such as mapping Customer Id). The main goal of this step is to figure out how incoming contact information should be applied to the existing subscriptions of an existing customer. This step will not create new subscriptions. All this will do is swap out the relevant contact information; for example, it will change the Shipping Address to use the incoming address.
Each Contact Type combination can have it’s own rules (Figure-7.A). The default is “None” (Figure-7.D), but can be changed by selecting an option in the dropdown (similar to Figure-7.B). Selection of Products is special in that it provides a granular way of dictating which products get updated. These are divided by product type. The products displayed are based on those available to the Selected Profile. Checking the box to the left of the product type – such as Magazines – will select all of the products of that (Figure-7.C). For example, the User could want a single magazine and all of the newsletters to change, so they would select one of the available Magazines and then the checkbox next to “Newsletters”.
Contact Type Validations
Because “None” is the default value and is an acceptable value, it makes this step completely optional. The user isn’t required to map incoming contact information to products. However, if they do make some selections, Data Loader will attempt to find potential issues with certain combinations (or lack there of). It is considered an error to use “Selection of Products” and not select any products. This will make the file unable to be processed.
The most likely warning would be the overlapping of products for a particular contact object across multiple types. For example, if the User maps two Email fields, one for Business and one for Home, they will have the opportunity to select which products should receive those. If during the selection of the products and overlap is found (for example, they both map in explicitly or implicitly to Product-A), then a warning will appear stating that the user should probably change the data or face potentially incorrect data. A subscription can only have one email address or shipping address assigned to it, so having multiple incoming emails apply to the same subscription will cause one of the emails to win over the other.
Settings and Validation Setup
This step is used to dictate how the data in will be processed and interpreted by Data Loader’s Processor.
A bulk of these options are validation checks to be performed on that data. If a piece of data fails a validation, then that row will be not be processed. Any row that passes validation will continue on to be processed. Depending on the selected rules, a single field can cause the whole row to fail. An example would be the Postal Address Required validation rule. It requires that most of the fields exist for a complete address. Therefore if a field such as city is missing, that row will fail. At the moment, the only recourse is to fix the problem and re-upload the file. Descriptions of which rows and cells had errors can be viewed in the View Row Errors option, found in the action menu for each file.
It should be noted that some validation rules overlap. For example, Reject New Customer would overlap with Customer Id Required, Customer Id Not Found, and/or External Id Not Found. This is because any of the three rules are looking for existing customers and if they don’t find any, the row will fail anyways. There are many rules that work similarly, but different just enough to be separate. While there shouldn’t be an issue running all of the rules together, keep in mind that an overlap in what’s being checked may exist and that the reason for failing might be ambiguous if both specific and general rules are applied.
Some rules inform the user what fields they should have mapped in order to best utilize the rules (Figure-9.5A). It is best to use the rules when the correct fields are mapped. If the fields aren’t mapped, then more things are likely to fail validation than would be desired. The Postal Address Required rule example above is a good example. If city wasn’t mapped at all, then every row would fail.
If you’re loading in data that will match with existing customers in your database, you can also select 3 different customer matching lookup options including:
- Customer Matching-Lookup – Name + Address
- Customer Matching-Lookup – Name + Email
- Customer Matching-Lookup – Name + Phone Number
Finishing Field Mapping
After going through all of the steps or closing the dialog, if the Field Mapping was saved, there is a chance that the file will be evaluated to either of two statuses.
Mapping Incomplete appears if the Field Mapping has Errors in it. These are things that must be addressed before Data Loader can move forward with that file. The common error is not re-mapping Data Source during General Mapping.
Ready to Process (Figure-11.A) means that the Field Mapping is in a state that it can be processed. Warnings may still exist in the Field Mapping, but as stated before, warnings do not prevent processing. At this point, the file may be processed by using the button and clicking the Process option (Figure-11.B).
Upon clicking the Process option, the User is presented with a dialog warning about the potential danger of uploading a file (Figure-12). This will be presented each time the user attempts to process the file, reiterating that the process has a chance of making changes that are not easily reversed. Therefore, they should make sure that what they are uploading is in a sound state. Accepting these terms (Figure-12.A) will enable them to press the Commit to Database Button (Figure-12.B). Pressing this button starts the processing cycle, which cannot be interrupted.
Processing Steps Overview
Data Loader’s processing component is made up of three stages. With the Beta, a lot of these areas are currently under utilized, but it is possible to glimpse some of these stages from the User Interface. When a file is being processed, it will go through each stage with each stage having it’s own queue. It starts with the Pre-Processor; this stage responsible for validating the data, looking for anything that may need to be held up. After that completes it gets passed to the Processor; this stage is responsible for creating new data and modifying existing based on the Field Mapping. The third stage, Post-Processor, currently isn’t being used but is designed to perform any data manipulation or validation after the data has been processed to the database.
The User may see the status Waiting to Pre-Process or Waiting to Process. This means that file is scheduled for that stage and is waiting for a server to pick it up.
When everything is a done, the user will receive an email stating everything is complete. It will contain some basic info and a small report of the overall results.
This will be the first stop for the data and is used to check and prep the data prior to processing. Most user-chosen settings and validations occur at this time to prevent data being partially processed.
One of the major jobs of the Pre-Processor is to determine if the customer is an existing one. It only does so if either Customer Id, Encrypted Customer Id, Postal Address Id, or External Customer Id is mapped; otherwise the customer is assumed to be a new one. If the customer is not found, then a new one is made instead. The exception to this is if any validation rules have been chosen, then some of them will prevent the new customer from being made.
Validation rules applied at this point will determine if the data in the row is acceptable for processing. If no rules are chosen, then the data will just continue onto processing. If a row fails, even if it’s related to a single column, then the whole row will fail.
During this stage the program is taking all of the rows that are cleared for processing and pushes the data into the database. Each row is processed fully before moving on to the next. Processing the data in the row is done in sequential steps, ensuring data gets added in a proper order if certain things are required by other steps.
- Customer – Processing a row starts by identifying the customer and whether or not they are to be treated as a new one or not. If Postal Address Id or Customer Id has been mapped, then it will attempt to look for an existing customer with that value. If they don’t exist, then a new customer will be created. Otherwise any customer related field will be applied to the customer.
- If the customer is an existing customer, mapped fields will overwrite existing values. For example, if First Name is mapped, then the customer’s name will change to new value. The old value is preserved in a Memo placed on the customer. It can be viewed in Audience Search for that customer. A Memo is also made for Title changes.
- External Customer Ids – This is the second step and is dedicated to figuring out how to interpret the incoming External Customer Id. Mostly it’s deciding if it’s a new one, in which case it will add it to the customer. Data Loader does not update existing External Customer Ids. If it sees that the external namespace exists for that customer, the External Customer Id will be skipped.
- Postal Addresses – This step starts off by creating a temporary version of the address. It fills in all the fields and once it has everything that it has been supplied, it will compare this address to any existing. This comparison is close enough to be considered an exact match comparison. If it is found to match an existing address, then the verification date and batch will be updated on the existing address. Otherwise a new address will be added to the customer.
- Email Addresses – This step is similar to the Postal Addresses step where the program creates a temporary email address with the given mapped fields. If the email address exists, then no new address will be created and the verification date and batch tracking is updated on the existing address. Otherwise a new email is created.
- Phones – Anything in the phone category (Phone, Mobile, Pager, Fax) ends up being processed here. Incoming phone numbers are checked against any existing phones. If they are the same, verification date and batch are updated on the existing. Otherwise a new phone is created for the customer.
- Apply New Contact Data to Existing Subscriptions – If the row is tied to an existing customer and the user set rules for how new contact information is applied, this step will fry to find any relevant subscriptions and swap out the old contact data with the new one. This applies for the Shipping Address and/or Email Address of the subscription. If no rules have been setup during the Contact Type Mapping, then no changes to the existing subscription will occur. This step does not create new subscriptions.
- Housekeeping – This is a type of finalizing step, wrapping up any loose ends that may have been created or attaching additional meta-data to the customer. One thing that occurs is that the customer is marked as changed and given a timestamp for it. This is relevant for other applications and reports. If the customer created for this row was a new customer, if any contact information exists, one of each type of contact is set as Primary. Many things care about Primary status, so Data Loader makes sure that the new customers have one. A nightly process will reassign the Primary status based on the rules setup for the brand as part of a nightly data cleanup.
That’s the bulk of how the processor works. At the moment, Audience Builder is not rebuilt as part of this. Before the customers can be used in Audience Builder, it will need be rebuilt.
After a File has Processed
When everything is done processing, the Status column (Figure-13.A) will update depending on how well the Processing stages went.
Status messages include:
- Completed – All of the rows in the file have processed successfully.
- Processed with Errors – The file has been partially processed. It is likely that some rows were completely processed, while others were not. The errors can be viewed in View Row Errors option from the file’s menu.
- Excessive Errors – If 60% or more of the file contains errors, the file will not continue processing. If this occurs, the file can be deleted and re-uploaded once corrections have been made.
- Pre-Processor, Processor, Post-Processor Failure – These three all mean mostly the same thing but occurred during the respective stage. Essentially, some internal server error that wasn’t accounted for occurred and the file was not processed completely.
- In either of these cases, a support ticket should be opened so that the problem can be fixed for similar scenarios. The information in that ticket should include:
- Name of the File that was Processing
- Name of the Client
- Name of the Profile (if possible)
- Around when the Failure occurred.
- In either of these cases, a support ticket should be opened so that the problem can be fixed for similar scenarios. The information in that ticket should include:
You also have the ability to poll the status of your files by hovering over the ‘Status’ header. A message regarding what the tool is currently working on will appear in a grey box:
Reviewing the File Details
The counts of rows which processed versus those that received an error can be viewed in the Details pane from clicking on the file’s row (Figure-13.B). As a reminder, Records represents the number of successfully processed rows, while Record Errors is the number found with an error.
You can also access the Details pane by selecting “Toggle Details Pane” from the dropdown.
Each file has it’s own Batch Tracking Code (Figure-13.C) which can be used to track customers and their data related to file. Clicking on it brings up a menu (Figure-14 below).
Currently, only one option exists and that is to copy it to your computer’s clipboard so that it can be pasted in other places. There is a way to query the customers used in a file via the Data Loader Tracking Code ‘skittle’. Future options will use the batch of customers for other Omeda tools.
Rows with Errors
At the moment, the only solution to rows with errors is to correct them, create a new file, and upload the new file. You can also delete the uploaded file without disrupting your database as the file has not yet been processed.