Search the Omeda Knowledge Base
Data Loader – Loading Data
Step 1: Uploading a new file
Note: Data Loader only accepts Delimited Files. They can be either comma, pipe, semicolon, or tab delimited.
To upload a file, the User can either use the Upload File Button from the home page or drag-and-drop the file into the Dropzone. The Upload File Button will open up a standard file-browser, allowing the user to search their system for the desired file. The Dropzone is signified by the dotted outline around the content of the page.
Once you’ve selected the file(s) to upload, you’ll need to set a template (optional), a delimiter (defaults to comma), and Keywords (optional), before selecting ‘Upload’.
To save you time, you can select multiple files to upload at once. When multiple files are selected, you can switch between files using the < > arrows at the top of the dialog box:
The blue line beneath the file name designates the ‘selected file’. You can choose separate templates, delimiters, and keywords for each file if you so choose.
Once you’re satisfied with the selected options for each file you can click ‘Upload’. This will send the files to Omeda which will prepare the system for the next steps.
File Status: Pending
Immediately after hitting ‘upload’ you’ll be taken back to the home screen where the file(s) you selected to upload will be listed. Initially they will have a Pending status while it processes on the back end.
Depending on the size of the file, it may stay Pending for some time. You can check how many rows have processed and which file(s) are next in the Queue section as well.
File Status: Ready to Map
So long as the upload has been successful, it will be given a Ready to Map status.
To begin mapping, select “Edit File’s Mapping” from the actions drop down.
Step 2: Mapping
A standard file has three parts of mapping you’ll need to review before the file can be processed:
- General Mapping
Depending on how things are mapped, the standard steps may change and new steps can be added.
The following are additional steps that may be added depending on the incoming data and how it has been mapped:
- Demographic Mapping
- Product Class Mapping
- Contact Mapping
- Deployment Type Mapping
- Contact Type Step
The first part of the mapping is an overview page that shows all of the incoming data from your file. You’ll also have the option to ‘Apply a Template’ if you didn’t already apply one on upload.
Field Mapping Validation/Feedback
Warnings, errors, and general information is tracked along the bottom on the left. Clicking either the Warnings or Errors will bring up a dialog that will display a brief explanation about things that the User might need to take action upon.
Warnings do not prevent the file from processing and is more to communicate that something about the Field Mapping doesn’t seem correct. An example is mapping both “Postal Address Id” and “Customer Id”. These serve the same purpose – looking up an existing customer – so mapping them at the same time is redundant. Another example is mapping a full name, such as “First Name Last Name”, while mapping separately “First Name” and/or “Last Name” in the same file will warn that the name could potentially not appear as desired.
Errors prevent the file from processing and need to be acted upon before the file can be processed. The most common one of these is the requirement of a known Data Source. Some other option besides the “Unidentified” should be chosen instead.
Unmapped Fields is not an issue and there is no requirement to have all fields mapped. This just keeps track of how many fields are currently unmapped in case the User would like to know this information. In the General Mapping step, these unmapped fields are highlighted as such so that they can be found at a glance.
For any step that can change aspects about the mapping you will be prompted to either ‘Continue without Saving’ or ‘Save and Continue’
- Cancel will close the dialog, useful if it was accidentally opened
- Continue without Saving will perform the action that was used to trigger this dialog, but no changes will be saved
- Save and Continue will save the changes and proceed with whatever action caused the dialog to appear
The overall goal of this step is to map incoming headers (bold text on the left) to their equivalent Omeda fields (white drop-downs on the right).
- The first thing you’ll need to do here is to declare the source of the data in the file. “Undefined” is not an allowed Data Source, and is the default field. You will need to select an option from the drop-down in order to process the file.
- During the upload process, some of fields may have been identified and already mapped. These are to be viewed as suggestions and can be changed by the User as they see fit. Any field that was not suggested will be set to –Not Mapped–. You can leave fields un-mapped and they will be passed over during processing and not committed to the database.
- By default, most contact fields are considered of the Contact Type “Business”. This can be changed by the User by choosing another available option. It is important that the user makes sure all relevant fields have a matching Contact Type. If the file contains a way to identify an existing customer, then mapping contact information will cause a new step to appear.
Note: If it is setup that all of the fields are tied to the Contact Type “Business” except, for example, City is set to “Home”, then it will result in two different incomplete addresses.
Additional details on the fields, demographics and products available for mapping can be found here:
When you have a column in the original file that you don’t intend to map to the database you can choose to delete/inactivate the header, rather than removing the column from the original spreadsheet.
To remove/inactivate a header, click the garbage can icon to the left of the header name:
This will move the header to an ‘Inactive Headers’ section below the rest of the data.
You can reactivate a header by clicking the ‘reverse’ icon to the left of the column header. This will reinstate the header and allow you to map it to the database.
Adding File Constants
If there are headers that you would like to add to the file – especially if this is a template file you intend to use multiple times – you can click ‘Add Constant’ at the bottom of the screen.
A pop-up will appear prompting you to name the constant before adding it to the mapping section
Once you’ve named and created the constant, you can map it to an incoming field (e.g. promo code) and provide a value that will append to all of the rows in your file.
Example: Your new constant field maps to the Promo Code field in the database – your constant in this case could be telemarket1219 (telemarketing file for Dec 2019).
Example 2: You want to map all contacts on the file to a specific behaviors such as attended an event. In this case, your constant may be a numeric value that you can map to a specific field in the behavior mapping section. (e.g. #1 which you can map to ‘Attended’ )
One of the most critical parts of file mapping is the validation rule selection within Settings
This list of validation and pre-processing rules will help match your incoming data to existing customers in your database. It will also allow you to choose what happens with data that may not match.
- A bulk of these options are validation checks to be performed on that data. If a piece of data fails a validation, then that row will be not be processed. Any row that passes validation will continue on to be processed. Depending on the selected rules, a single field can cause the whole row to fail. An example would be the Postal Address Required validation rule. It requires that most of the fields exist for a complete address. Therefore if a field such as city is missing, that row will fail.
- It should be noted that some validation rules overlap. For example, Reject New Customer would overlap with Customer Id Required, Customer Id Not Found, and/or External Id Not Found. This is because any of the three rules are looking for existing customers and if they don’t find any, the row will fail anyways. There are many rules that work similarly, but different just enough to be separate. While there shouldn’t be an issue running all of the rules together, keep in mind that an overlap in what’s being checked may exist and that the reason for failing might be ambiguous if both specific and general rules are applied.
- Validation Rules Show/Hide based on the Fields the User has mapped in General Mapping Step. The user must map at least one of the linked fields for a Validation Rule to be selectable in the Settings Step.
- When a record processes through Data Loader without state information, but does have a valid US zip code, the state code will be populated based on the state that zip code resides in.
A full list of the rules and additional explanations can be found here.
If you have any questions, please contact your Audience Services team and they will walk you through which rules are best applied to your file.
Finishing Field Mapping
Once you have initially mapped the file, there are one of two statuses that may be applied:
- Mapping Incomplete
- Ready to Process
Mapping Incomplete appears if the Field Mapping has Errors in it. These are things that must be addressed before Data Loader can move forward with that file. The common error is not re-mapping Data Source during General Mapping.
To correct this, re-visit the mapping sections to make sure required fields are in place.
Ready to Process means that the Field Mapping is in a state that it can be processed. Warnings may still exist in the Field Mapping, but as stated before, warnings do not prevent processing. At this point, you can now move to the third step of loading your data: Processing the file
Step 3: Processing
One your file has a status of ‘Ready to Process’ you can commit the data to your database.
To begin, choose “Process” from the actions drop down:
The following window will appear – – – –
This will be presented each time the you go to process a file, reiterating that the process has a chance of making changes that are not easily reversed.
Accepting these terms (Figure-12.A) will enable you to press the Commit to Database Button (Figure-12.B). Pressing this button starts the processing cycle, which cannot be interrupted.
Processing Steps Overview
Data Loader’s processing component is made up of two stages:
- Pre-Processor: This stage is responsible for validating the data and
- Processor: This stage generates any new data and modifies existing data based on the Field Mappings you’ve set up
Once you’ve started to process the file the status will change to ‘Processing’. You will be able to track the file as it moves through these two stages in the Queue Section:
When everything is a done, the user will receive an email stating everything is either complete or it will notify you if something came up during processing that caused the file to fail.
It will contain some basic info and a small report of the overall results.
Note: Audience Builder is not rebuilt as part of this. Before the customers can be used in Audience Builder, it will need be rebuilt.
After a File has Processed
When everything is done processing, the file status will update depending on how well the Processing stages went.
Status messages include:
- Completed – All of the rows in the file have processed successfully.
- Completed w/Errors – The file has been partially processed; some rows were completely processed, while others were not. The errors can be viewed in View Row Errors option from the file’s menu.
- Excessive Errors – If 60% or more of the file contains errors, the file will not continue processing. If this occurs, the file can be deleted and re-uploaded once corrections have been made.
- Processing Failure – This status means that an internal server error occurred and the file was not processed completely. When a file has this status, we are notified internally, but you will need to file a JIRA ticket and notify your Audience Service team so the issue can be addressed.
- The information in that ticket should include:
- Name of the File that was Processing
- Name of the Client
- Name of the Profile (if applicable)
- Around when the Failure occurred
- The information in that ticket should include:
A confirmation will also be sent to the user when processing completes. The confirmation email will include a summary of the number of input rows, number of rows processed and number of rows with errors. The confirmation email will also include the Data Loader Tracking Code.
Rows with Errors
If your file is marked ‘Completed w/Errors’ you can view a full listing of each error by clicking ‘View File State’ from the actions drop down:
The popup window that appears will list each row # that triggered an error, along with a note of why it failed.
If you would like to correct any of these errors you will need to isolate the rows in the original file and re-upload them.
A future enhancement will allow you to correct and commit the errors from within the tool.
Reviewing File Details
The counts of rows which processed versus those that received an error can be viewed in the Details pane. Records represents the number of successfully processed rows, while Record Errors is the number found with an error.
Once it’s been processed, each file gets it’s own Batch Tracking Code which can be used to track customers and their data related to file. You can also use this code to lookup the audience within Audience Builder. Clicking on the code will automatically copy it to your clipboard and allow you to easily paste it into the Data Loader Skittle in Audience Builder.
Archiving Processed Files
Once a file has been successfully processed and is marked ‘Completed’ you can remove it from the list of current files by choosing to ‘Archive’ it.
Note: Completed files cannot be deleted, since they have already been converted to the database.
To archive a file, click the actions dropdown and select ‘Archive‘.
A popup will appear asking you to confirm the archive.
Clicking ‘Archive File’ will move it to a separate list.
Viewing / Restoring Archived Files
To view files that you have previously archived, select ‘Archived’ from the drop down at the top and then the blue ‘Search’ button.
Your archived files will pull up on a separate list.
To restore a file that was previously archived, click ‘Restore’ from the actions dropdown:
A popup will appear asking you to confirm.
Clicking ‘Restore File’ will move it back to the main list.