Data Import 101: Business Analyst Guide

As an IT business analyst, in most of the projects that you will encounter, there will be a need to interface with a preexisting platform(s). For instance, an interface will have to be established in order to retrieve data or to send data. In my line of work in e-commerce projects, this had to be done in every project. In big structures, the e-commerce platform is not usually the complete primary source of data, the ERP is. Here, when we say data, we talk mainly about the products, prices, discounts, and/or clients/users.

Data import interfaces can have two types. It can be synchronous or asynchronous. Here, the following questions will need to be thought of:

• How frequently does the data change?
• What is the tolerance to have inaccurate data?

Synchronous

Synchronous interfaces means that whenever a data is needed, a request will have to be made to retrieve the data. There will always be an alignment between the source and the target, and they will always be synchronized.

This means that there is no need to store the data in the database. However, in order to avoid affecting the performance and increasing the response time, there is a possibility to cache the retrieved data. The possibility to do so and the time of caching the data depends on the tolerance of the business. Consequently, you will have to ask questions like:

• What is the time that can be tolerated to refresh the data? (give examples: 0 secondes, 10 minutes, 1 hour, etc.)

How?

For synchronous interfaces, the used technology is web services. It is a request that will be sent from one end (data receiver) that will wait for a response from the other end (data sender). Here, it is important to handle the different cases, and therefore, as a Business Analyst, the following points need to be discussed with the client or its representative:

• Identify the different mandatory fields and the behaviour if they were missing.
• Identify the different errors: 4xx and 5xx responses in addition to timeouts.
• Identify the extreme cases and the behaviour to be followed: empty responses and special situations, such as no results.

Files

In this case, the data is usually received in the form of a Json file format. Json stands for JavaScript Object Notation, and it provides the representation of the received data in the form of data structures and objects. In this stage, you have to ask for the following:

• The swagger is a Json file that defines the contract between the sender and the receiver. It describes the data structure with which the data will be received, and it will describe the nature/type of received data. In addition, it should describe the different responses that will be received whether in the case of success or the case of error.
• If it is possible, a sample of the response would be a good addition. This way you will be able to have an example of the data to be received, and thus, you will have a foundation of a stub, which is always a good idea to have in case the other side is not ready yet to implement a response.

Asynchronous

Asynchronous interfaces means that, at a certain moment, the source and the target might not be aligned/synchronized in terms of data. In this case, you will have to make sure that the business can tolerate this difference in data.

If the asynchronous interface coincides with the business needs, then the following question needs to be asked:

• When is the best time to execute the interface?

Here, you will need to guide the discussion to know when is the peak of the traffic, and thus avoid it in order not to affect the customer experience. In addition, it is important to know the approximate/average amount of data (database lines) that will be transmitted in order to predict the time of processing.

How?

Asynchronous interfaces can be implemented in two ways:

Periodic Job

A job can be implemented to run periodically (ex: daily). A trigger can be set for this job or it can be executed manually. This means that an import file needs to be placed in a defined folder, for the job to consume it.

Hot Folder

Another way to consume the import files is to use Hot Folders. In this method, the trigger to consume the import file is to put it in a folder (Hot Folder). Once this is done, a job consumes the file instantly.

Files

In order to prepare for this type of interfaces, there are two files that needs to be provided:

• An example of the import file would be a very helpful file to understand the structure of the received data. In addition, it gives a good example of data that will be received.
• The XSD file is a vital input that needs to be requested because it defines the agreement and the contract between the sender and the receiver. It will list all the parameters that will be sent, and this includes their names and types. In addition, it will specify which parameters are mandatory.

Here, it is important to agree with the client on the behaviour that needs to be adopted in case a mandatory parameter is missing. Each mandatory field should be treated separately because each one might have a different business weight.

(!) An important point that needs to be raised with the client is not only the behaviour when a parameter is missing, but what is the behaviour that needs to be adopted when a tag is missing. The tag is the name of the parameter that can be found in the import file that identifies when the value of the parameter is going to be specified.

Conclusion

I hope this article has given you an introduction of how to deal with data import as a Business Analyst. I meant it to be guide to facilitate your job; this is why I have added some questions that would guide your discussion with the client.