登录查看更多内容

How do you handle data integration and migration in real-time and batch scenarios?

由人工智能和领英社区提供技术支持

Data integration and migration are essential processes for any business intelligence (BI) project that involves combining and transferring data from different sources and systems. However, depending on the data volume, complexity, and frequency, you may need to choose between real-time and batch scenarios. In this article, you will learn how to handle data integration and migration in both scenarios, and what are the advantages and challenges of each approach.

此文章中的业界达人

由社区从 6 条内容中精选。了解更多

Amandeep S.
Alex Scattergood

Making data make sense. - Data Migration,reporting, warehousing, cleansing and management expert.
Phil Rowlands

Looking for new opportunity. Open to contract or direct employment.

1 Real-time data integration and migration

Real-time data integration and migration means that data is moved and transformed as soon as it is generated or changed, without any delay or schedule. This allows you to have the most up-to-date and accurate data for your BI analysis and reporting. However, real-time data integration and migration also requires more resources, bandwidth, and performance from your source and target systems, as well as your integration and migration tools. You also need to ensure data quality, security, and consistency across multiple streams and transactions. Some examples of real-time data integration and migration tools are Apache Kafka, StreamSets, and Oracle GoldenGate.

添加您的观点

Amandeep S.
举报内容
Depends on type and use of data. For example in real time discrete manufacturing, some data like machine errors, production quantity etc requires real time analysis for counter-measures. A machine that is generating scrap above acceptable limit would need to be stopped and investigated. Other data that requires pulling new material from warehouse can be batch processed at defined interval (1-2 hours) and finally data that requires deep analysis, cleaning or is required to train ML models might only be processed once a day.

已翻译

赞
Phil Rowlands

Looking for new opportunity. Open to contract or direct employment.
举报内容
Wholly dependent on the source data. For data for which there is a constant flow; such as cash values going through the tills of a retail business, or flight data from a control tower, real-time integration and migration are a necessity, and merit the extra cost. For clarification data on sales is needed to track trends, the faster patterns are identified the quicker a business can react, a competitive advantage; for flight data cost is justified through increased safety. The remainder, the vast majority of source data, including all that is manually input, is seldom real-time, hence no need for real-time integration and migration.

已翻译

赞
Alex Scattergood

Making data make sense. - Data Migration,reporting, warehousing, cleansing and management expert.
举报内容
First as always you need to identify the wants from the needs. "I want all the data in real-time" is normally reconsidered when you explain it would cost 10 or 100 times our budget. Eventually you get to the company actually needs 90% overnight 5% three times a day and 5% near real time because of XYZ and you have a contract from which you can derive SLAs. You need to define the data and reason carefully to ensure success. Legacy systems frequently surprise you with speed limitations so you need to test for performance. It is simpler to implement nowadays because many RDBMS support native CDC or replication or there are third party tools as mentioned above. Of course once you deliver 5% real time they want more.

已翻译

赞

2 Batch data integration and migration

Batch data integration and migration means that data is moved and transformed in batches, usually based on a predefined schedule or trigger. This allows you to reduce the load and complexity of your data integration and migration process, as well as optimize the use of your resources and bandwidth. However, batch data integration and migration also means that your data may not be as current and relevant as real-time data, and you may miss some opportunities or insights that depend on timely data. You also need to plan and manage your batch schedules and workflows carefully, and handle any errors or exceptions that may occur. Some examples of batch data integration and migration tools are SSIS, Talend, and AWS Glue.

添加您的观点

Alex Scattergood

Making data make sense. - Data Migration,reporting, warehousing, cleansing and management expert.
举报内容
Again you need to agree with your audience what they expect and when. Once this is done you need to document the limitations, any extracts that happen multiple times a day (not just overnight with the other data) should ideally be self supporting, try to include a date /time extracted in the data they will be asking you if you don't. Again once you start other uses will be found so expect it to evolve.

已翻译

赞
Alexander Adams

Sr. Manager - Financial Systems and Analytics at McKesson
举报内容
This is a great example and use case for Alteryx. It’s pretty rare when real-time data is actually needed and typically data migrations and refresh rates of around a day is often more than fine for most business scenarios. With Alteryx and Alteryx server you can have just about anyone create robust ETL workflows and schedule them to run on whatever frequency makes sense for you. Alteryx is a great tool when you need data transformations and migrations to be built and stood up quickly.

已翻译

赞
Phil Rowlands

Looking for new opportunity. Open to contract or direct employment.
举报内容
If a predefined schedule this is a repeat of the last question. If a trigger then there is a dependency, whether on a person or a process. Presumably, a person will run the batch data integration and migration when they need to. If a process, then dependent on the purpose for which it is designed to deliver.

已翻译

赞

3 Hybrid data integration and migration

Hybrid data integration and migration means that you use a combination of real-time and batch scenarios, depending on the data type, source, and destination, as well as your BI objectives and requirements. This allows you to balance the benefits and drawbacks of both scenarios, and leverage the best of both worlds. However, hybrid data integration and migration also means that you need to design and implement a more complex and flexible data integration and migration architecture, and coordinate and synchronize your real-time and batch processes. You also need to select and integrate the right tools and platforms that can support both scenarios. Some examples of hybrid data integration and migration platforms are Azure Data Factory, Informatica, and MuleSoft.

添加您的观点

4 Data integration and migration best practices

No matter the scenario you choose, there are certain best practices to ensure a successful data integration and migration project. To begin, you should define your strategy, scope, and goals in accordance with your BI vision and business needs. Additionally, you should assess your data sources, systems, and formats to identify any gaps, issues, or dependencies that may affect the process. Furthermore, you must select the correct tools and platforms that can handle your data volume, variety, velocity, and veracity; as well as integrate with your existing infrastructure and BI tools. Furthermore, you must design and test your data integration and migration workflows, mappings, transformations, and validations; while also documenting your data models, standards, and rules. Lastly, it is important to monitor and troubleshoot performance, quality, and reliability; while also implementing backup and recovery mechanisms.

添加您的观点

5 Data integration and migration challenges

Data integration and migration can be difficult in today's dynamic and diverse data landscape. Common challenges include data silos and fragmentation, large and complex data volumes, issues with data quality and consistency, security and compliance requirements, and the need to keep up with changes in data. Data silos make it hard to access, integrate, and migrate; large or complex data can cause delays, errors, or failures; data quality and consistency affect the validity of BI analysis; sensitive or regulated data require proper encryption; and changes in data require constant updates.

添加您的观点

6 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?

添加您的观点

Business Intelligence

+ 关注

给文章评分

我们借助人工智能创建了此文章。您认为这篇文章怎么样？

很棒不太好

举报此文章

查看全部

How do you handle data integration and migration in real-time and batch scenarios?

1

2

3

4

5

6

1 Real-time data integration and migration

2 Batch data integration and migration

3 Hybrid data integration and migration

4 Data integration and migration best practices

5 Data integration and migration challenges

6 Here’s what else to consider

Business Intelligence

给文章评分

感谢您的反馈

更多Business Intelligence相关文章

更多相关阅读内容

How do you handle data integration and migration in real-time and batch scenarios?

1

2

3

4

5

6

1 Real-time data integration and migration

2 Batch data integration and migration

3 Hybrid data integration and migration

4 Data integration and migration best practices

5 Data integration and migration challenges

6 Here’s what else to consider

Business Intelligence

给文章评分

感谢您的反馈

查看其他技能