You're migrating data for a major project. What tools can help you do it quickly and efficiently?

Enterprises grapple with data silos, sparse metadata, and integration challenges. Existing tools lack visibility and error handling. Flodata.io bridges these gaps, offering no-code/low-code solutions for seamless integration across systems. Our accelerators drive efficiency and cost savings, empowering enterprises to optimize data management. Join us in transforming data conversations and integrations. Experience the future of enterprise data management with Flodata.io

Na minha experiência a forma mais efetiva de migrar dados seja on-premise ou cloud é usando uma solu??o de CDC (Change Data Capture), impondo uma carga de trabalho mínima ao banco de dados, pois ao invés de fazer select para extra??o dos dados, é lido o arquivo de log de transa??es. Na AWS temos o DMS que faz isso e on-premise pode se usar o Debezium.

In my experience first thing that should be addressed in planning is - inventory of existing data sources and etl code. One must try to create full inventory and their respective owners and dependencies. It is also important to identify the code redundancies. Once you have full view of existing data estate it gives solid foundation to identify right technologies to address the migration.

Automation is key across different areas of migration, which is often overlooked. Based on experience, there isn't a single product/tool that does all effectively and efficiently. You will need to look at products/tools which does automated crawling across the ecosystem to understand end-to-end lineage, ETL code conversion for code refactoring, DDL code conversion from one database to another (covering SPs, macros, UDFs etc.), tool to support one-off cost effective data migration which can be repeated for subsequent delta catchups as and when required, automated testing tool covering data validation and reconciliation etc.

Tech comes next, platforms come even later...what needs to come first are 2 major building blocks: 1) What are your KRAs on the business side and what pieces of migration with minimal disruption will help you achieve that. Profile your data landscape, have your master data catalogs, CDEs and SDEs sorted and have data governance team in place. Without that it is like riding a rudderless ship. 2) Is your human resource pool skilled enough to support the migration technology platforms on your IT roadmap and the processes therein. If not, how do you plan to upskill them and how do they get involved during the whole migration process. Because if they don't start owning the migration process, even the best in the world won't work for you.

My favorite has been Precisely Connect, previously known as Syncsort. Sources, targets, and compute can be mainframe, on-prem, and cloud. Developers find it easy to use. Some developers fear a tool takes away from their coding craft. I have, however, pointed out that their coding is still needed in other areas. Let the tool do the heavy lifting. Developers can then focus on more innovative coding.

The tools choice primarily depends on budget. Provided this hurdle is handled, selection of tool for ETL should consider the rollout method to be adapted and the source, target platforms involved. Choice of tool has a great influence on the overall success of the migration activity.

This may be time for a potentially unpopular opinion, but after doing this multiple times in a variety of ways, the most successful solution was a platform agnostic custom-built .NET application. Yes, the database does matter due to SQL language differences, however the rest does not. Find yourself a good .NET developer, and build your own ETL tool if you have an ETL operation that requires repeatability, performance, and reliability.

Multiple things decide which tool should be finalized, which could be a mix of business requirements, Budget, key features required, end use case depending on how real-time you want to get tool selection will also differ. Some of the best of breed tools that you can use are - Qlik Replicate + Talend - Informatica CDC and PowerCenter - Power Platform with Dax - Azure/AWS cloud components

Hybrid cloud tool will be perfect choice for data ETL as it will not be cloud locked in solution like AWs glue or Azure data factory

Lo peor que puede pasar en un proyecto de migración de datos es que los datos no cuadren. Hay mil motivos y problemas al cambiar de plataforma, conversiones de tipos que implican ligeras modificaciones en los datos, estructuras diferentes entre origen y destino y un sin fin .. En los proyectos en los que trabajo, intento que al final haya una parte de testing, generalmente en varios niveles: 1.- Conteo de elementos en origen y destino (hash completo a veces) 2.- Si conteo es <> o hash <>, comparacion fila a fila por clave primaria, 3.- Si hay demasiados datos para comprobar por pk, comparación por grupos, ej, suma de hash por a?os, para mirar solo a?os que no cuadran La clave es que esto se automatice, garantizando la calidad de los datos

May be a Data migration pipeline configured in a tool that can contain dependencies along with scheduling (as found in Jenkins or Azure DevOps). This pipeline line can orchestrate and automate not only the scope based migration activity but also the ST, UAT, and data deployment to higher environments and act as an end to end Audit trial platform for every migration schedule. This can be configured as a light weight pipeline in an ETL tool layer or in a tool like Jenkins that has migration schedule linked to requirements, builds, tests and environments (based on the viability). As always, everything depends on the requirements, risks, business value, the type of migration (one time, sequential, incremental, parallel), budget and timeline.

There are multiple tools and technologies available for migration of data across platforms/OSs but the key factors are 1. How the migrated data would be used by either application or end users. Post migration usability of the migrated data is one of the sucess criteria. 2. The migration tools should also consider impact on downstream applications or users to provide seamless experience pre and post migration.

Ein wichtiger Aspekt ist die Datenbereinigung. Es ist ein wesentlicher Schritt, der darauf abzielt, Daten von Ungenauigkeiten, Duplikaten und Inkonsistenzen zu befreien. Das Ziel ist es, Daten zu bereinigen und zu standardisieren, um ihre Qualit?t und Zuverl?ssigkeit zu verbessern. Eine der gr??ten Herausforderungen ist die Vielfalt und Komplexit?t der Datenquellen. Unternehmen sammeln Daten aus verschiedenen Systemen, Plattformen und Quellen, was zu einer heterogenen Datenlandschaft führt. Daten k?nnen in verschiedenen Formaten vorliegen, unterschiedliche Strukturen haben und in verschiedenen Datenbanken gespeichert sein. Dies erschwert die Datenbereinigung erheblich und erfordert fortschrittliche Tools und Techniken.

When we load the data, that is not end for the job. The loaded data has to maintain quality and accuracy with clean structure. We need validate and test it thouroughly

Firstly there are many tools in the market that provides data migration. In my experience I could suggest the best tools like informatica power center developer idmc talend etc. How ever cloud platforms like aws azure gcp provides different services too. In my experience I could say aws glue crawler serves best .

Tools depends on the existing technology and migration to system. For example if you are migrating data from an old erp to new erp like s4 hana there are multiple tools which synonymously works with sap and provide pre built packages for standard objects. A good analysis of the requirements for migration and cleansing keeping in mind the existing tech stack and staff capabilities is required

There are many tools in the market which can help in migrating data sets and in most cases the technology stack you are migrating to or from will have a service for this specific task. But important things to consider: - plan out the data migration so that both source and target systems have minimal impact - testing and validation of data in target system - sometimes your code may not work in the target environment which may put a kink to your future operations so testing all the possible transformations is a must in target system. - complete testing and validation in the QA environment which involves all the modules for all possible types of data transformations.

In my experience, yes, most bulk batch process tools can accomplish the task easily but bottle necks such as how fast your source query runs and how fast target database accept the data can be limitations as well. So its important to evaluate your end to end flow and understand the bottle necks. In addition, I think cleaning and curating of the data would be critical for the project success more than the technology.

Analysis and discovery tools. If you start migration before considering all the application landscape you will definitely bump into a wall. after spending so much time and effort it’s not going to be easy to change the direction for an alternative solution. So before planning how to migrate you must know what to migrate.