Guide to Importing Data Into your Saas Application

What is data importing?

Data importing is the process of transporting data into a software product. The data itself is collected from other software vendors or simply created during a company's daily operations and stored in formats such as .xlsx, .csv, etc.

Usually, a software requires a specific format to work with that data. Right now, the easiest option for businesses is using spreadsheet templates, forcing their customers to go through the arduous reformatting process while still leaving the chance that an import error might pop up.

Importing data has several phases, each of which takes time and offers various potential sources of error. Following consultation with customers and different error sources and workflow analyses, we identified six primary data-importing phases. Each of these phases involves touchpoints with highly sensitive customer data and should run as smoothly as possible to avoid the negative impact of a poor data import process on the customer and the customer relationship.

Convert data & file to a usable intermediate format

The most commonly used formats are .csv and .xlsx files, but data can be stored in a wide variety of file formats. Even though the most commonly used formats are Excel and CSV files, this does not mean that companies can exchange their data without further preparation. If the required data format is not communicated in advance, this small detail can lead to longer import cycles and the need for more communication, costing companies more time and money

Select file data

After converting the data into the correct file format, the exchange data points must be selected. This can be done manually via drag and drop, by utilizing an import button, or by automatically retrieving the data source through prior scheduling.
Experience has shown that trying to keep track of all the relevant points to select them efficiently causes problems for many companies and costs them unnecessary time and effort. This step has the least potential for errors, yet it can quickly become tedious if the files are screened manually and are too extensive for all rows and columns to be visible at one glance. Endless scrolling through data sheets is, therefore, usually unavoidable.

Table and field mapping

In addition to the target format, the target data model must also match. The existing data points must be mapped from one schema to another. As well as the headings of the individual columns, the units and details of the unique data points must also correspond to the target data model. All these details and possible differences for each data point have a high potential for errors that lead to lengthy communication cycles and slow down the entire import process.

Since the manual effort exceeds the time frame of almost every company, those who can afford it try to develop an in-house importer. However, developing such an importer also costs time, capacity, and money. For this reason, many companies continue to format the data points themselves and sometimes even charge a fee or have them formatted by their customers, despite the high effort involved. This results in many pain points at different levels in this step, especially on the side of the customer support team and the customers.

Data Cleaning

Data cleaning is the most complex and challenging step of a data importing process as we now have to look at data on a single entry-level and correct that data based on several rules. After all, and this is the fourth step, the data that has been selected and entered into the target data model despite the incorrect format must now usually be cleaned up manually. As simple and tedious as this process is, it has already become an integral part of the import process for many companies and is an indispensable part of the conventional procedure for guaranteeing the quality of the imported data. Companies employ staff to manually clean files by hand or develop scripts for specific scenarios to minimize the overall cleaning efforts. Expensive alternatives to this are data hooks and highly-automated processes for data cleansing.

Actual Import

Real data importing is only possible after the first four time-consuming, costly, and anxiety-inducing steps. The data is transferred to the new target data model as a compressed package (e.g., JSON)

For more information, please reach out to nuvo