Lessons learned from migrating complex systems at KIS

Lessons learned from migrating complex systems at KIS Solutions

At KIS we have been very lucky to tackle several cool and complex enterprise system migrations. We share in this article some notes from a recent migration completed for one of our enterprise clients.

The scope

• Reverse-engineering a 10+ year old legacy system;
• Designing a new database structure and API calls for all core customer use cases;
• Interfacing with existing consumers of data and APIs to ensure new system met requirements;
• Interfacing with existing consumers for integration testing and cutover activities;
• Migration of Legacy data (300M+ account and contact records, 2B total records);
• Monitoring, data validation and cutover;

General lessons learned

1. There is no substitute for reverse engineering. There is rarely a way to get around “the hard work”.

2. If you are expecting to get a full and complete accounting of all the business rules and logic from your partners and subject matter experts, you might be in trouble

3. Systems that run for 10+ years and grow organically over time are complex and hard to understand. Rarely is there anyone in the organization who “really knows how it works”

4. Use interfaces to hide migration complexity from dependent systems. Often, incredibly old legacy systems are used by many other systems. Whenever possible, try to leverage the existing interfaces so that dependent systems are “shielded” from your legacy update.

5. If the legacy system has APIs with a given message format, keep those message formats so that API consumers can migrate, and things should remain the same.

6. You may not get to fix everything you want to fix out of the gate, but that’s ok. Fix it in phases. Removing dependencies has tremendous value and decreases complexity.

7. Multi-billion record legacy databases are particularly challenging and require complex coordination for massive data migration.

Dry Runs

We typically plan for at least 3 types of dry run activities.?

Several are “burnout” activities. Run the migration to understand if your process runs in reasonable time and try to find what may arise from moving around massive datasets.

Keep running dry runs until you are sure that your migration process will work, that it will run in reasonable time and that it will move all the data.

Plan for “SoftProd”

Stand up as much of the new infrastructure in “production” as early as possible.

Run it in parallel with the current system. Do this in soft prod. Meaning the new system is running against actual data but no one is consuming that data.

Validate your new infrastructure using the newly created soft production environment prior to migrating over any production data. Run data comparison reports between the legacy and the new system for some period until you are confident both platforms agree.

Finally, plan for a phased cutover. Phased cutovers?break the track into small pieces?and each piece is cutover one section at a time. Small manageable pieces allow the solution to mature over time.

Migrating Old Systems

When moving very old systems (10+ years), there is just too much risk in turning it off on a given day and time.?

Rather, stand up the new system and run it in parallel with the legacy system.

Roll things out in phases, pick some of your lower priority processes or data consumers. The ones with less risk. Prioritize migrating these things first. Monitor how the low priority processes behave on the new system for a while.

Slowly and gradually move more processes and dependencies from the legacy system to the new system and eventually, everything will be moved.

This approach can allow for a fallback. If you move over a certain process and something goes wrong, you can move it back to the legacy system until any issues are resolved on the new platform.

Make a data validation plan and post cutover monitoring plan. Make sure you understand how you will confirm that the new system aligns with the legacy system. Plan to spend considerable time writing data validation queries and reports and engage with business partners on key metrics.

Run data validation/comparison reports and share the results with business partners. Do this until everyone can agree that the two systems align on critical data metrics.

Conclusion

Your first runs of production on your system might have many flaws.?

You will probably make a few mistakes.

Plan for these mistakes. Build a post-production monitoring system so you can find them.

We hope this information can help in your projects. Share with us other data migration tips that you have used!

#keepitsimple #data #legacy #tech