How can you make data migration transparent to end-users?

Some of less typical questions, that might get forgotten during early planning are: What separates your happy customer from your sad customer? Do they want to pay more, or have longer outages? What is your budget for the migration? What about "error budget", do you have SLA or any other binding agreements? Do you have people that have clearance to actually work hands-on and debug actual data, or you will need to work with indirect access to hased data/ anonymized errors when developing process? What will be process to de-provision and erase test servers, if any?

When assessing the scope and requirements in a migration it's best to always consider: get the complete entity structure (file, table, source) when migrating. Second, be sure you listen to the customer key problem and don't get stuck on the solution you will receive from them, sometimes in makes better sense to think outside the box instead of doing the obvious thing the customer wants. Usually errors get automated when doing the obvious thing that has always been done

In my experience, EMR to EMR data migration scope heavily relies on the Legacy vendor's ability to export the data. Discussions based on different types, quantity and quality of data can help define the proper scope of the project and can help define tentative timeline and identify any risks or roadblocks that might happen.

Automation tools can help simplify the migration process and reduce the risk of errors. For example, you can use tools that automatically map the data from the old system to the new system, reducing the need for manual intervention.

The right tools may be different when considering data volume, current storage technology, intended storage technology, operativity and data criticality. When migrating an operational database always set up a strategy to prevent issues with stale data, a replica is one that has worked for me well. If the operational database has too much data, change to a current_table/history_table structure, then replicate then migrate. In regards to reporting databases, warehouses and data lakes, this pressure is normally less pronounced so the migrations tend to be simpler.

Whatever data migration pipeline or methodology a person chooses to implement, one has to have a layover time period where old system and new system run parallel. This is a must clause. Only thing that person has to decide is what cost to run both and it's layover time period. Some less risky migration may have layover time period of less than 6 hours whereas high risk migration layover may be over 6 months.

This is the biggest pain for the users and reducing downtime is major win for the migration. Use ZDM (zero downtime migration) tools to address the issues with downtime. these tools helps to reduce the downtime and help improve the availability of the systems.

There are multiple strategies when minimizing disruption, The Blue Green strategy tends to be one of the most common and efficient, have the current(old) "Production" environment continue working then start the new production environment start, stabilize it and validate it, once it is working consistently then rename or point towards the new one as the "current" production and slowly remove access to the old one as it is shown to be no longer needed, if it is not critical or has been validated with multiple methods like row counts, field distincts count and cheksums you can even do the switch, turn the old one and only restart it if needed before finally decomissioning it.

Proper monitoring for a migration can be done with some common tools, just beware that the effort to have realtime or near realtime monitoring may be bigger than the migration effort itself, so in a low complexity migration a manual non real time monitoring may make more sense. Some of these tools are comparing a row count on the source and a row count on the target. Having the # of distinct values for each field you can see if they are different in source and target, though it requires more work. A checksum can be a fast way of detecting differences even if not as useful in identifying them. Also aggregations of the measures (numeric values) can be a good alternative for making sure data is the same on both sides of the migration.

Is possible with a good and deep transition planning with people that knows the business, but require origin and destination compatibility. A simple example is indication of some confirmation, one database may consider "Yes/No", other "Y/N" and other "0/1", will be necessary some data conversion, will not be a simple move, and you may have some user impact or resistence changing the legacy way or work and presentation of data. Other example is time register of some action or process, one database may work with begin/end date and periods with no data, "not justified time", other database may work just with begin time, in this case the end time will be the begin time of next action and do not allow unjustified periors.

In case of any issues during the migration, provide a fallback mechanism that allows users to access the old system until the issue is resolved.

Do a lot of test migration runs outside the production environment, gradualy increasing size of sampled data. If you can replicate 100% of existing data to a temporary setup, and also mirror/shadow 100% of the incomig production writes in real time to the test setup, you can be somewhat confident that new system works. If you have luxury of also mirroring reads to the new system (so thet it also gets locks, and you get a way to compare amswers from old and mew system) you can have high confidence. However, be sure you understand your customers needs and preferences. In many cases they may choose lower fees, and tolerate longer outage, and not pay for such complex scenaros at all.

Early engagement. You have to talk to your subject matter experts and your key business users early. They need to be brought along the journey, to help guide migration rules, rather than be dumped with the outcome. Having business on your side will make the journey less painful in the end as they will be more understanding of any unforeseen problems and will be more likely to own the resultant solution, allowing it to bed in nicely. ??

Before all the things will be done in the Live Environment, a proper simulation or testing should be done using the prepared/approached method to see what are impacts (if they occur) and do the rollback scenario for unexpected situations. A similar environment is expected so all the behavior will be considered the same as in the Live Environment. Once all is done, then the Live Migration can be done in a safe way to go.