How do you manage data warehouse migration risks and challenges?

You should also assess data ownership and data lineage. Do systems have shared databases, or a database per Microservice. Do systems use immutable events.

Absolutely, run a data quality assessment before understanding the differences between your existing environment and the new. An example; Oracle EDW required a copy of a table to exist, the logical model(if one exist) and physical model (required to execute the batch) could have countless DDL's which differ from the logical. Pay attention or ensure you do not assume the logical model is the actual processing requirement. In fact, the duplicate tables/indices caused by former environment processing will cause poor performance in a dual processing environment such as Teradata appliances. Look for duplicate indices Look for date format errors Look for decimal format errors Look for null values The effort is critical to success.

Are you migrating only the data warehouse or modernizing the entire system? If only the data warehouse, look at features of target warehouse technology to help with migration. If modernizing the entire system, it's generally best to go domain by domain so that value can be delivered incrementally.

As mentioned, with a caveat being doing the "lift and shift" migration from one environment (traditional) to a new parallel processing could force the poor performance. Your vendor and the investment may be severely degraded. You may find that trying to execute the required performance projects as a future project, may cause the supplier of the new investment to rightfully exclude your environment from warranty or any fault on not meeting the performance expectations. Be cautious in any situation where Duplicate Indices, Null Values, Decimal format errors, or Data format errors are showing up. Too much Power usage or processing seems to run for too long all have direct sources with the issues stated. Corrective action could fail.

Part of preparation should be listing data producers, data consumers, and all downstream impacts of data. You'll need to communicate and coordinate with downstream teams so they can adjust to data source changes without interruption.

Transforming and enriching your data "before" you migrate is the safest migration approach. Data type changes, from traditional to a different architecture/technology appliance could cause the new appliance to fail, as the fix seems simple to most data warehouse leaders and data architects. Beware, correcting one performance fix at a time is the only option. More than one at a time, can take down the entire environment. Avoidance and Prevention would be the ONLY option. As Eric Smeby contributed data producers, data consumers and effort may seem impossible to identify the downstream teams. In this scenario, it is imperative to be transparent, investigate who is responsible and accountable for the data. Include SOX process owners.

To give downstream teams time, a common pattern is to run legacy and target systems in parallel. This way data checks can compare results, and downstream teams can have time to transition to the target system. This is also a good time to build control reports and other tests that can continually validate the target system.

People change must include the new technology operating requirements, leaders must do the heavy lifting required to ensure practices used in former environments are not replicated. The experts in one environment may force what they know into a new technology, if the leaders fail to understand and support the new Architecture (resource(s)).

Transition first and optimize second. Plan to need people post transition, as new systems often bring both new challenges, and new features to utilize.

Understanding who in the IT needs to be consulted or informed (data architects) and who is Accountable and Responsible will be an eye opening effort. This goes to the heart of validity and accuracy, authorization and access are managed by the source system and the process owners who are ineffective in the process when data goes to a storage behind the application and changes or duplicates during processing. Financial transactions require the source to remain as the users of the source system are unable to save their work in the source, when a zip code field is left blank. A source system (ERP) does not allow null values, nor does the source system save when date or decimal format errors are attempted. Tax and duty fees, pricing impacted