Smooth transitions: best practices for data migration

The easiest way to understand data migration is to think of it as moving documents from one filing cabinet to another in your office. Sometimes it’s just moving files around, but more often it’s a matter of finding lost folders, rearranging books, restoring damaged papers or creating a new filing system. Rarely does everything go as planned, but when you're done, the documents are easier to access and the shelves are in the nicer part of the office. The same thing happens when migrating data to applications or online databases, except that files are virtual.

The definition of data migration is simple. It’s the process of preparing, extracting and finally transferring data from one computer system or data storage to another. The first scenario usually takes place when a company moves to a new application. Typically it involves things like changing vendors and modifying data models. In insurance, this usually means moving policy, claim or payment history from a legacy system to a new one. A second scenario can occur when a company switches its data platform or upgrades its current one.

Data migration in the insurance industry is challenging for a number of reasons:

• Ensuring that data transferred between legacy and target systems is complete and compatible with all relevant software.
• The high level of customised products in insurance can mean that target applications are not prepared for some types of data from legacy systems.
• Integration issues arise because insurers use separate policy, billing or claims management tools.
• This is compounded by the complexity of CRM tools, price comparison engines and other external software used by insurers, producers and policyholders.
• Insurers often rely on multiple external data sources (e.g. telemetric) or databases (e.g. vehicle registration databases), adding to the complexity.

There is also matter of security and privacy. Data is often vulnerable during the migration process, and insurance companies handle both sensitive and confidential information. Tools such as encryptors, virtual desktops and authenticators can help with this part, but nothing beats good security practices and ongoing staff training.

There are quite a lot of challenges when it comes to migration. Most of them are data-related: quality and integrity, complexity, compatibility or volume. In my experience of migration-focused projects, most of the problems have been around integration. When it comes to larger and older legacy systems, they’re often running software that looks like it was created in the days of Windows 3.11 or even MS DOS. Even when we’re migrating the core system, many customers prefer to keep some parts of their internal ecosystem. Our job in these scenarios is to make sure that everything works with both new and old applications, and that can be quite challenging for the data model.

However, one of the most difficult things to overcome was the unpredictable human factor. I remember one situation where the legacy system was frozen for migration and yet a customer complaint was somehow being handled by two different employees in two systems. One of them managed to restore the legacy account and the corrected invoice was released from both systems independently, resulting in two refunds.

More typical issues experienced are related to the compatibility and quality of older data. Some policyholders may have a long history with the insurer and preserving it for future policy calculations may be necessary. In this case, at least some representation of the history is required.

Let’s go back to our filing cabinets comparison. If you want to move the contents of your shelves, you can’t just throw them into the new space without a plan or order. What if your co-workers can’t access them? Or if the books are too big for the shelves, will you just cut them in half? There are a few things you can do to make sure the whole process of migration goes smoothly.?

What can be done to mitigate risks and delays in the migration process? Answer is very simple: hire specialists. They’ll plan everything, choose the right tools and manage the whole process. But what if, for some reason, you need to do significant parts of the project yourself? First, you need a good migration strategy. Planning can make or break the whole process. Choose the right tools, decide on the method and phases of migration, make sure that there is sufficient knowledge of the data structure of all the systems involved. If you can bring together the business side SME, administrators of the current data source and good specialists who will be responsible for the data transfer, you can create a good plan that will help you finish on time, with very little disruption and downtime. A separate migration team doesn’t sound so bad. You can’t overestimate knowledge, communication and preparation. I’m going to risk sounding redundant, but this needs to be emphasized: get people talking to each other. Even the best specialists won’t get far, if the stakeholders don’t tell them what they need. And vendors, testers and IT teams need to ask relevant and specific questions and provide feedback when needed.

In any migration, it is important to decide whether it is better to migrate the entire base at once, or to do it in stages, or individually. In the insurance industry, the last option works well for short-term contracts, that can only be moved when they are processed, such as motor policy that is renewed every year. The first one would be better for long-term contracts. This aspect is important, because one of the crucial parts of any migration is to make it as little disruptive to regular work as possible.

Experts and plans are one thing, but you’ll also need software. The choice of migration tools should be made in advance, as there are many things that will be needed. Not just migration, transfer and integration solutions, but also backup tools, encryption software and reconciliation instruments.

Another tool that can improve the migration process is good documentation. Data mapping for both source and target databases, an integration map or a responsibility matrix can help to overcome a lot of issues.

Some practical tips from my experience would be to avoid making the whole process bloated or unnecessarily complicated. Cleanse the data before you start the process, you don’t really need 250,000 drafts created when employees were learning to use the software. Talk to your legal team, maybe you can only migrate 5 years of billing history instead of 20.

Finally, a solid testing team should be involved throughout the process. Testers and QA should work together at every stage of the migration. It’s good practice to test migrated data in stages, and a dedicated team can do this. They should verify the source data, compare migrated files between old and new databases, and verify that the target software handles the migrated information correctly. Work shouldn’t stop once the migration is complete. Tests and audits should continue until the desired level of confidence is reached. And you don’t want testers who just check the shelves for misplaced books – you want them to climb up the cabinet, try to move it, or try to fit a chair in there.

It doesn’t matter whether your task is to migrate databases, software or handle cloud migration. It’s always a challenge, and more often than not, it takes more time and resources than initially anticipated. But if you are prepared to spend enough time planning, talking to people and giving them what they need to do their jobs, you should be fine. Soon enough your new shelves will be filled to the brim with elegantly displayed files, and you’ll be able to enjoy a cup of coffee in a tidy office.

?ukasz ?uczak , Senior Tester at Sollers Consulting