Search the Omeda Knowledge Base

Data Loader – Loading Data

Step 1: Uploading a new file

Note: Data Loader only accepts Delimited Files. They can be either comma, pipe, semicolon, or tab delimited.

Single Files:

To upload a file, the User can either use the Upload File Button from the home page or drag-and-drop the file into the Dropzone. The Upload File Button will open up a standard file-browser, allowing the user to search their system for the desired file. The Dropzone is signified by the dotted outline around the content of the page.

Once you’ve selected the file(s) to upload, you’ll need to set a template (optional) and a delimiter before selecting ‘Upload’.

Multiple Files:

To save you time, you can select multiple files to upload at once. When multiple files are selected, you can switch between files using the < > arrows at the top of the dialog box:

The blue line beneath the file name designates the ‘selected file’. You can choose separate templates and delimiters for each file if you so choose.

Once you’re satisfied with the selected options for each file you can click ‘Upload’. This will send the files to Omeda which will prepare the system for the next steps.

Post Upload Status: Pending

Immediately after hitting ‘upload’ you’ll be taken back to the home screen where the file(s) you selected to upload will be listed. Initially they will have a Pending status while it processes on the backend.

Depending on the size of the file, it may stay Pending for some time.

While the file is being loaded into the system, it will make note of various aspects of the file to attempt to make the rest of the process proceed smoothly. It will grab the header row and attempt to predict which incoming field will map to one of Omeda’s fields. This works best with the customer and contact related fields as they are to be the most universal. Doing this will potentially speed up the Field Mapping step later, so it is recommended for file headers for such fields to have common naming. An example of this is using similar variations of “First Name”, “first_name”, or “fname” to represent “first name”.

The other main thing this process does is to note each row and column. Some minor validation exists at this step, but is mostly related to the structure of the file’s rows and not the actual content of the file. An example of this is if a row is found to have a number of cells less or more than the number of cells. In such a case, the row will be marked and skipped.

Post Upload Status: Ready to Map

Once the file has been uploaded, it will likely end up in the Ready to Map state. At this point the you will be able to to interact with the file in various ways. The most basic of which is to click on the file’s row. This will open a panel from the right which is used to give a bunch of basic information pertaining to the file. Additionally, you’ll be ready to map the columns in your spreadsheet to the appropriate fields within your database

Step 2: Mapping

Once your file has been uploaded and is no longer pending, it will have a “Ready to Map” status.*  From the Actions Menu, you can select “Edit Mapping” to get started.

Alternatively, you can also pre-map files before you upload them using the ‘Mappings’ section found on the home page. For instructions on how to use this, check out the Mapping Section below.

Overview Page

The first thing you’ll see is a short overview page that lets you scroll through all of the rows and columns you also have the option at the top of the screen to choose to use this as a template for future files.

Field Mapping Validation/Feedback

Warnings, errors, and general information is tracked along the bottom on the left. Clicking either the Warnings or Errors will bring up a dialog that will display a brief explanation about things that the User might need to take action upon.

Warnings do not prevent the file from processing and is more to communicate that something about the Field Mapping doesn’t seem correct. An example is mapping both “Postal Address Id” and “Customer Id”. These serve the same purpose – looking up an existing customer – so mapping them at the same time is redundant. Another example is mapping a full name, such as “First Name Last Name”, while mapping separately “First Name” and/or “Last Name” in the same file will warn that the name could potentially not appear as desired.

Errors prevent the file from processing and need to be acted upon before the file can be processed. The most common one of these is the requirement of a known Data Source. Some other option besides the “Unidentified” should be chosen instead.

Unmapped Fields (introduced with the Template feature) will have it’s own little alert in this area. Having Unmapped Fields is not an issue and there is no requirement to have all fields mapped. This just keeps track of how many fields are currently unmapped, just in case the User would like to know this information. In the General Mapping step, these unmapped fields are highlighted as such so that they can be found at a glance.

General Mapping

The overall goal of this step is to map incoming headers to their equivalent Omeda fields. Near the top of the page , information reminding what file is being mapped to which brand is displayed.

Below that is a place to declare the source of the data in the file. “Undefined” is not an allowed Data Source, and is the default field. You will need to select an option from the dropdown in order to process the file.

The remainder is mapping each field. The headers in the left column can be adjusted and edited if you would like a different naming convention than what was uploaded. NOTE: The headers cannot be left blank – they will prevent a user from saving the work until a header is input.

During the upload process, some of fields may have been identified and already mapped. These are to be viewed as suggestions and can be changed by the User as they see fit. Any field that was not suggested will be set to –Not Mapped–.

By default, most contact fields are considered of the Contact Type “Business”. This can be changed by the User by choosing another available option. It is important that the user makes sure all relevant fields have a matching Contact Type. If the file contains a way to identify an existing customer, then mapping contact information will cause a new step to appear.

Note: If it is setup that all of the fields are tied to the Contact Type “Business” except, for example, City is set to “Home”, then it will result in two different incomplete addresses. 

Additional details on the fields, demographics and products available for mapping can be found here:

Adding/Removing Headers

When you have a column in the original file that you don’t intend to map to the database you can choose to delete/inactivate the header, rather than removing the column from the original spreadsheet.

To remove/inactivate a header, click the garbage can icon to the left of the header name:

This will move the header to an ‘Inactive Headers’ section below the rest of the data.

You can reactivate a header by clicking the ‘reverse’ icon to the left of the column header. This will reinstate the header and allow you to map it to the database.

If there are headers that you would like to add to the file – especially if this is a template file you intend to use multiple times – you can click ‘Add header’ at the bottom of the screen.

This will add a new, empty column into your file. You will need to name and map this line just like the original ones with the file.

Adding a Constant

To add a column to your file where all of the incoming contacts are assigned the same value, you can use the ‘Add Constant’ option.

Similar to a header, you’ll need to name the constant and then map it to an incoming field (e.g. promo code)

To finalize this, you’ll also need to provide the value that will attach to all of the rows in your file.

Example: Your new constant field maps to the Promo Code field in the database – your constant in this case could be telemarket1219 (telemarketing file for Dec 2019).

Example 2: You want to map all contacts on the file to a specific behaviors such as attended an event. In this case, your constant may be a numeric value that you can map to a specific field in the behavior mapping section. (e.g. #1 which you can map to ‘Attended’ )

Navigating

For any step that can change aspects about the mapping you will be prompted to either ‘Continue without Saving’ or ‘Save and Continue’

Cancel will close the dialog, useful if it was accidentally opened. Continue without Saving will perform the action that was used to trigger this dialog, but no changes will be saved. Save and Continue will save the changes and proceed with whatever action caused the dialog to appear.

If the changes were saved, there is a chance that the number of steps may have changed. If this happened, then a dialog will open noting the changes (see below). If “Finish” was clicked prior to saving, if more steps were added, the next step will be displayed. If “Next” was clicked and steps were removed, then the dialog will close stating that no more steps existed.

The following are additional steps that may be added depending on the incoming data and how it has been mapped:

  • Demographic Mapping
  • Product Class Mapping
  • Contact Mapping
  • Deployment Type Mapping
  • Contact Type Step
  • Settings (Validation Setup)

Figure-9

This step only appears if the User has mapped Postal and/or Email related fields and has mapped a way to detect an existing customer (such as mapping Customer Id). The main goal of this step is to figure out how incoming contact information should be applied to the existing subscriptions of an existing customer. This step will not create new subscriptions. All this will do is swap out the relevant contact information; for example, it will change the Shipping Address to use the incoming address.

Each Contact Type combination can have it’s own rules (Figure-7.A). The default is “None” (Figure-7.D), but can be changed by selecting an option in the dropdown (similar to Figure-7.B). Selection of Products is special in that it provides a granular way of dictating which products get updated. These are divided by product type. The products displayed are based on those available to the Selected Profile. Checking the box to the left of the product type – such as Magazines – will select all of the products of that (Figure-7.C). For example, the User could want a single magazine and all of the newsletters to change, so they would select one of the available Magazines and then the checkbox next to “Newsletters”.

Contact Type Validations

Because “None” is the default value and is an acceptable value, it makes this step completely optional. The user isn’t required to map incoming contact information to products. However, if they do make some selections, Data Loader will attempt to find potential issues with certain combinations (or lack there of). It is considered an error to use “Selection of Products” and not select any products. This will make the file unable to be processed.

The most likely warning would be the overlapping of products for a particular contact object across multiple types. For example, if the User maps two Email fields, one for Business and one for Home, they will have the opportunity to select which products should receive those. If during the selection of the products and overlap is found (for example, they both map in explicitly or implicitly to Product-A), then a warning will appear stating that the user should probably change the data or face potentially incorrect data. A subscription can only have one email address or shipping address assigned to it, so having multiple incoming emails apply to the same subscription will cause one of the emails to win over the other.

Settings and Validation Setup

Figure-9.5

This step is used to dictate how the data in will be processed and interpreted by Data Loader’s Processor.

A bulk of these options are validation checks to be performed on that data. If a piece of data fails a validation, then that row will be not be processed. Any row that passes validation will continue on to be processed. Depending on the selected rules, a single field can cause the whole row to fail. An example would be the Postal Address Required validation rule. It requires that most of the fields exist for a complete address. Therefore if a field such as city is missing, that row will fail. At the moment, the only recourse is to fix the problem and re-upload the file. Descriptions of which rows and cells had errors can be viewed in the View Row Errors option, found in the action menu for each file.

It should be noted that some validation rules overlap. For example, Reject New Customer would overlap with Customer Id Required, Customer Id Not Found, and/or External Id Not Found. This is because any of the three rules are looking for existing customers and if they don’t find any, the row will fail anyways. There are many rules that work similarly, but different just enough to be separate. While there shouldn’t be an issue running all of the rules together, keep in mind that an overlap in what’s being checked may exist and that the reason for failing might be ambiguous if both specific and general rules are applied.

Some rules inform the user what fields they should have mapped in order to best utilize the rules (Figure-9.5A). It is best to use the rules when the correct fields are mapped. If the fields aren’t mapped, then more things are likely to fail validation than would be desired. The Postal Address Required rule example above is a good example. If city wasn’t mapped at all, then every row would fail.

If you’re loading in data that will match with existing customers in your database, you can also select 3 different customer matching lookup options including:

  • Customer Matching-Lookup – Name + Address
  • Customer Matching-Lookup – Name + Email
  • Customer Matching-Lookup – Name + Phone Number

Finishing Field Mapping

After going through all of the steps or closing the dialog, if the Field Mapping was saved, there is a chance that the file will be evaluated to either of two statuses.

 

Figure-10

Mapping Incomplete appears if the Field Mapping has Errors in it. These are things that must be addressed before Data Loader can move forward with that file. The common error is not re-mapping Data Source during General Mapping.

 

Figure-11

Ready to Process (Figure-11.A) means that the Field Mapping is in a state that it can be processed. Warnings may still exist in the Field Mapping, but as stated before, warnings do not prevent processing. At this point, the file may be processed by using the button and clicking the Process option (Figure-11.B).

 

Processing

 

Figure-12

Upon clicking the Process option, the User is presented with a dialog warning about the potential danger of uploading a file (Figure-12). This will be presented each time the user attempts to process the file, reiterating that the process has a chance of making changes that are not easily reversed. Therefore, they should make sure that what they are uploading is in a sound state. Accepting these terms (Figure-12.A) will enable them to press the Commit to Database Button (Figure-12.B). Pressing this button starts the processing cycle, which cannot be interrupted.

Processing Steps Overview

Data Loader’s processing component is made up of three stages. With the Beta, a lot of these areas are currently under utilized, but it is possible to glimpse some of these stages from the User Interface. When a file is being processed, it will go through each stage with each stage having it’s own queue. It starts with the Pre-Processor; this stage responsible for validating the data, looking for anything that may need to be held up. After that completes it gets passed to the Processor; this stage is responsible for creating new data and modifying existing based on the Field Mapping. The third stage, Post-Processor, currently isn’t being used but is designed to perform any data manipulation or validation after the data has been processed to the database.

The User may see the status Waiting to Pre-Process or Waiting to Process. This means that file is scheduled for that stage and is waiting for a server to pick it up.

When everything is a done, the user will receive an email stating everything is complete. It will contain some basic info and a small report of the overall results.

A glossary of fields can be found here.

Pre-Processor

This will be the first stop for the data and is used to check and prep the data prior to processing. Most user-chosen settings and validations occur at this time to prevent data being partially processed.

One of the major jobs of the Pre-Processor is to determine if the customer is an existing one. It only does so if either Customer IdEncrypted Customer IdPostal Address Id, or External Customer Id is mapped; otherwise the customer is assumed to be a new one. If the customer is not found, then a new one is made instead. The exception to this is if any validation rules have been chosen, then some of them will prevent the new customer from being made.

Validation rules applied at this point will determine if the data in the row is acceptable for processing. If no rules are chosen, then the data will just continue onto processing. If a row fails, even if it’s related to a single column, then the whole row will fail.

A page dedicated to the various Settings and Validations can be found here.

Processor

During this stage the program is taking all of the rows that are cleared for processing and pushes the data into the database. Each row is processed fully before moving on to the next. Processing the data in the row is done in sequential steps, ensuring data gets added in a proper order if certain things are required by other steps.

  1. Customer – Processing a row starts by identifying the customer and whether or not they are to be treated as a new one or not. If Postal Address Id or Customer Id has been mapped, then it will attempt to look for an existing customer with that value. If they don’t exist, then a new customer will be created. Otherwise any customer related field will be applied to the customer.
    1. If the customer is an existing customer, mapped fields will overwrite existing values. For example, if First Name is mapped, then the customer’s name will change to new value. The old value is preserved in a Memo placed on the customer. It can be viewed in Audience Search for that customer. A Memo is also made for Title changes.
  2. External Customer Ids – This is the second step and is dedicated to figuring out how to interpret the incoming External Customer Id. Mostly it’s deciding if it’s a new one, in which case it will add it to the customer. Data Loader does not update existing External Customer Ids. If it sees that the external namespace exists for that customer, the External Customer Id will be skipped.
  3. Postal Addresses – This step starts off by creating a temporary version of the address. It fills in all the fields and once it has everything that it has been supplied, it will compare this address to any existing. This comparison is close enough to be considered an exact match comparison. If it is found to match an existing address, then the verification date and batch will be updated on the existing address. Otherwise a new address will be added to the customer.
  4. Email Addresses – This step is similar to the Postal Addresses step where the program creates a temporary email address with the given mapped fields. If the email address exists, then no new address will be created and the verification date and batch tracking is updated on the existing address. Otherwise a new email is created.
  5. Phones – Anything in the phone category (Phone, Mobile, Pager, Fax) ends up being processed here. Incoming phone numbers are checked against any existing phones. If they are the same, verification date and batch are updated on the existing. Otherwise a new phone is created for the customer.
  6. Apply New Contact Data to Existing Subscriptions – If the row is tied to an existing customer and the user set rules for how new contact information is applied, this step will fry to find any relevant subscriptions and swap out the old contact data with the new one. This applies for the Shipping Address and/or Email Address of the subscription. If no rules have been setup during the Contact Type Mapping, then no changes to the existing subscription will occur. This step does not create new subscriptions.
  7. Housekeeping – This is a type of finalizing step, wrapping up any loose ends that may have been created or attaching additional meta-data to the customer. One thing that occurs is that the customer is marked as changed and given a timestamp for it. This is relevant for other applications and reports. If the customer created for this row was a new customer, if any contact information exists, one of each type of contact is set as Primary. Many things care about Primary status, so Data Loader makes sure that the new customers have one. A nightly process will reassign the Primary status based on the rules setup for the brand as part of a nightly data cleanup.

That’s the bulk of how the processor works. At the moment, Audience Builder is not rebuilt as part of this. Before the customers can be used in Audience Builder, it will need be rebuilt.

After a File has Processed

 

Figure-13

When everything is done processing, the Status column (Figure-13.A) will update depending on how well the Processing stages went.

Status messages include:

  • Completed – All of the rows in the file have processed successfully.
  • Processed with Errors – The file has been partially processed. It is likely that some rows were completely processed, while others were not. The errors can be viewed in View Row Errors option from the file’s menu.
  • Excessive Errors – If 60% or more of the file contains errors, the file will not continue processing. If this occurs, the file can be deleted and re-uploaded once corrections have been made.
  • Pre-Processor, Processor, Post-Processor Failure – These three all mean mostly the same thing but occurred during the respective stage. Essentially, some internal server error that wasn’t accounted for occurred and the file was not processed completely.
    • In either of these cases, a support ticket should be opened so that the problem can be fixed for similar scenarios. The information in that ticket should include:
      • Name of the File that was Processing
      • Name of the Client
      • Name of the Profile (if possible)
      • Around when the Failure occurred.

You also have the ability to poll the status of your files by hovering over the ‘Status’ header. A message regarding what the tool is currently working on will appear in a grey box:

Archiving Processed Files

Once a file has been successfully processed and is marked ‘Completed’ you can remove it from the list of current files by choosing to ‘Archive’ it.

Note: Completed files cannot be deleted, since they have already been converted to the database.

To archive a file, click the actions dropdown and select ‘Archive‘.

A popup will appear asking you to confirm the archive.

Clicking ‘Archive File’ will move it to a separate list.

Viewing / Restoring Archived Files

To view files that you have previously archived, check the ‘Show Archived’ box above the blue search button and then click ‘Search’

Your archived files will pull up on a separate list.

To restore a file that was previously archived, click ‘Restore’ from the actions dropdown:

A popup will appear asking you to confirm.

Clicking ‘Restore File’ will move it back to the main list.

Reviewing the File Details

The counts of rows which processed versus those that received an error can be viewed in the Details pane from clicking on the file’s row (Figure-13.B). As a reminder, Records represents the number of successfully processed rows, while Record Errors is the number found with an error.

You can also access the Details pane by selecting “Toggle Details Pane” from the dropdown.

Each file has it’s own Batch Tracking Code (Figure-13.C) which can be used to track customers and their data related to file. Clicking on it brings up a menu (Figure-14 below).

 

Figure-14

Currently, only one option exists and that is to copy it to your computer’s clipboard so that it can be pasted in other places. There is a way to query the customers used in a file via the Data Loader Tracking Code ‘skittle’. Future options will use the batch of customers for other Omeda tools.

Rows with Errors

At the moment, the only solution to rows with errors is to correct them, create a new file, and upload the new file. You can also delete the uploaded file without disrupting your database as the file has not yet been processed.

Advanced Features – Mapping Section

 

Last Updated On January 15, 2020
Knowledge Base Feedback