10 Tips for the Data Migration Process

Here are 10 tips and tricks for executing a successful Data Migration Process, following the lessons learned and previous articles.?

1) Typically, there are multiple surprises throughout the process; thus, we recommend starting early and running the data migration process often. No surprises; dig deeper and look harder!?

• We suggest using the Auditor layer from the Data Migration Framework to aid in anomaly detection by analyzing logs and metrics collected from the running system.?

2) Run the data migration process daily or weekly to ensure consistency, repeatability, and accuracy. Record the results per execution to compare and contrast as often information hides in plain sight.?

3) Consider keeping sensitive data within the customer’s environment to reduce risk.?

4) Communicate with external and internal stakeholders managing their expectations while leveraging their help in identifying opportunities for improvement.?

5) Establish an environment where the data migration process can be executed frequently and used by all stakeholders to increase testing and reduce future risk. Use Infrastructure as Code (IaC) to automate provisioning and testing of this environment to increase repeatability and reduce time.?

• Bonus: the migrated data in the test environment is naturally tested by the other auxiliary tests, like smoke testing, regression testing, and other system-level testing outside of results verification. Two for the price of one!?

6) Hold pre-migration meetings to talk through the current process and iterate on better ways to automate.?

7) Document a backup plan just in case things go awry while testing the data migration process. Answer the following:?

• How can you re-establish the old system if it fails??

• How can you re-load the entirety of the data into the old system if it fails??

8) Automate any verification steps required to check results so that you can continue to iterate quickly over small improvements while consistently verifying the results.?

9) Ensure that all code, scripts, IaC, etc. used for the data migration process are maintained in Source Control Management systems.?

10) Ask tough questions:?

• Is there a Data Retention Policy (should there be)???

• Should any of this data NOT be migrated??

• Who are the stakeholders (internal and external)??

We hope you find these tips helpful and we look forward to sharing more of our knowledge in the future. Comment below with any questions and share this article and others in this series with your colleagues. Thank you for reading!