Data Migration: Challenges & Opportunities

Introduction:

Data migration is the biggest work and challenge in platform migrations and/or platform switching projects. Data migration can be triggered as part of many technology projects. Most prominent ones are:

• Switching from on-prem to cloud or vice-versa
• Switching from one cloud vendor to another
• Expanding from one cloud vendor to multi-cloud or hybrid models
• Switching data platforms (both OLTP, OLAP platforms)
• Consolidating databases (Even if source & target data technologies are same)
• Splitting data into multiple databases/data platforms
• ILM (Information Lifecycle Management) projects such as backup & restore, Archiving, data audit tracking projects
• Switching data onboarding and/or storage formats
• Switching a software system that uses different underlying data structure, data platform or data format(s)
• And many more...

No matter what triggers data migration, a successful data move need to ensure business continuity, data quality and data integrity are in tact.

Here are 5 aspects such a project should consider for successful data migration.

???????? ????????????????????:

Understanding what data extraction mechanisms exist in the current source system is the first step. Asking following questions and addressing them can not only helps successful data migration but also provides clear visibility into effort estimate and project timelines.

• Are there any pre-built data pull options?
• Does the vendor need to build custom extractions?
• Does your team need to build custom extracts?
• Does your team has the know-how and capacity to do it?
• Are there any additional fees/costs involved for outbound data

???????? ????????????????????????????:

In a perfect world one may not need to do any data transformation but there is no such thing as perfect world. Clear definition of business rules, data transformation conditions ahead of the project helps in better user experience. Following conditions should be thought through and understood.

• Reference values such as states, zip codes
• Lookup values such as gender codes, boolean data values
• Calculated values such as price, amount, age
• Data formats between systems such as date formats
• Data mapping at file and data element level across the source and target systems

???????? ????????????:

Generally ignored aspect in data migration projects is understanding and planning for volume of data to be moved. Clarity in this aspects helps in proper architecture and capacity planning for both historical data (typically one time work) as well as ongoing (if applicable) data move.

Answering below questions would help this planning.

• What volume of data would you transfer?
• How many years of transactional data would you like to go back to?
• Would you like to transfer inactive members/accounts into the new system?
• What should be the sequence of data move (such as reference data followed by master data and then transactional data) so data integrity is maintained in the target system.

At times when historical data is too large, one can plan to consider moving 3-6 months worth of data into new system for go-live needs and move rest of the data in parallel after system goes live for better project time line management.

At times this can also be an opportunity to archive the older data into another cheaper storage instead of loading into the target system.

???????? ??????????????:

Consider data quality challenges in the existing system. Over time systems may have changed their data validations and data quality requirements. Many times new systems can have new data checks, validations, and quality restrictions to be considered too.

Data migration projects can also be a new opportunity to improve data quality as you extract and transform the data before loading into target system.

Doing pilot data migration and running through complete data quality checks including enduser testing can help the go-live go smooth without hiccups.

???????? ??????????????????:

Knowing your target system well is an essential part before successfully ingesting your data. Few critical questions to answer in this phase are:

• How does the new system support data ingestion?
• Will there be a standard interface or do you need custom processes to load?
• Does the target system allow different methods such as UI, API, and File ingestions?
• Do you need to use all ingestion methods or some of them ?
• Do you need different ingestion methods for bulk/historical data loads vs ongoing loads?

Conclusion:

Each data migration project and use case can be unique depending on your business context, internal skills, previous experience, tools & platforms involved, project budget, project timeline, number of end users impacted, criticality of the data for your end users, sensitivity (such as HIPAA) of data and so on.

Along with above considerations, you may also want to evaluate different tools that could help you in this process as well as any external consulting help that can bring speed, quality and better budget controls.

Why else would you consider and what unforeseen challenges have you faced in your data migration projects?

#data #datamigration #datamapping #datamanagement #DataDiaries #ETL #ELT #datawarehouse