登录查看更多内容

How can you scale MDM to handle big data in Data Engineering?

由人工智能和领英社区提供技术支持

Master data management (MDM) is the process of creating and maintaining a single, consistent, and accurate source of truth for key business data, such as customers, products, suppliers, and locations. MDM can help data engineers improve data quality, reduce data duplication, enhance data governance, and support business intelligence and analytics. However, MDM can also face challenges when dealing with big data, such as high volume, variety, velocity, and veracity. How can you scale MDM to handle big data in data engineering? Here are some tips and best practices to consider.

此文章中的业界达人

由社区从 3 条内容中精选。了解更多

Tyrell Perera

Enterprise Solutions Architect | Cloud and Distributed Systems | EAI
James Hocking

Technical Co-Founder @ Hocking Digital ? AI, Data, Information Technology, & Web3 Expertise in Strategy, Architecture…

1 Choose the right architecture

Depending on your data sources, domains, and use cases, you may need to adopt a different MDM architecture to handle big data. For example, you can use a centralized MDM approach, where all the master data is stored and managed in a single repository, or a distributed MDM approach, where the master data is distributed across multiple systems and synchronized through a hub. You can also use a hybrid MDM approach, where some master data is centralized and some is distributed, or a multi-domain MDM approach, where different domains of master data are managed by different systems and integrated through a common platform. Each architecture has its own advantages and disadvantages, so you need to evaluate your requirements and trade-offs carefully.

添加您的观点

Haythem Saad

Solutions Architect - Principal Engineer
举报内容
Scaling MDM for big data in Data Engineering involves using distributed storage, parallel processing with frameworks like Apache Spark, indexing, and partitioning for optimization. Hybrid batch and real-time processing, cloud-based MDM solutions, data quality checks, governance, and ongoing monitoring are essential for scalability and accuracy in handling large datasets.

已翻译

赞

2 Leverage cloud and big data technologies

Another way to scale MDM to handle big data is to leverage cloud and big data technologies that can provide scalability, flexibility, performance, and cost-efficiency. For example, you can use cloud-based MDM solutions that can scale up or down as needed, without requiring large upfront investments or maintenance costs. You can also use big data technologies, such as Hadoop, Spark, NoSQL, or Kafka, that can process and store large volumes and varieties of data in parallel, distributed, and real-time manner. These technologies can also enable you to apply advanced analytics and machine learning techniques to your master data, such as data profiling, matching, cleansing, enrichment, and classification.

添加您的观点

3 Implement data governance and quality standards

Scaling MDM to handle big data does not mean compromising on data governance and quality standards. In fact, it means that you need to implement them more rigorously and consistently across your data sources, domains, and systems. Data governance and quality standards can help you define and enforce the policies, roles, responsibilities, and rules for managing your master data, such as who can access, modify, or delete it, how it should be validated, updated, or archived, and how it should be measured, monitored, or reported. Data governance and quality standards can also help you ensure compliance with regulatory and ethical requirements, such as data privacy, security, or ethics.

添加您的观点

Tyrell Perera

Enterprise Solutions Architect | Cloud and Distributed Systems | EAI
举报内容
I would break down the governance and quality standardisation process into 4 areas; Standardized data models: Implement consistent and standardized data models across all systems to ensure data integrity and facilitate efficient integration. Data quality management: Implement data quality checks and cleansing processes to remove duplicate or inaccurate data before it enters the MDM system. Data lineage tracking: Track the origin and transformation of data throughout its lifecycle to ensure auditability and compliance. Data governance framework: Establish a robust data governance framework with clear ownership, access controls, and data quality standards.

已翻译

赞

4 Adopt an agile and iterative approach

Finally, scaling MDM to handle big data requires adopting an agile and iterative approach that can adapt to the changing needs and expectations of your business and stakeholders. Instead of trying to implement a perfect and comprehensive MDM solution from the start, you should focus on delivering incremental and measurable value to your business and stakeholders, by prioritizing the most critical and impactful data domains, sources, and use cases, and by testing and validating your MDM solution with real data and feedback. You should also continuously monitor and improve your MDM solution, by identifying and resolving any issues, gaps, or opportunities, and by incorporating new data sources, domains, or use cases as they emerge.

添加您的观点

5 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?

添加您的观点

James Hocking

Technical Co-Founder @ Hocking Digital ? AI, Data, Information Technology, & Web3 Expertise in Strategy, Architecture, Implementation, and Adoption for Business Leaders, & Executives
举报内容
Having worked on MDM and Big Data projects, my advice is Event Driven Architecture. MDM has been around for years but it started life pre-Social Media and its architecture doesn't naturally fit "Big" data. This means that you cannot throw super large datasets at it and expect it to work. Like any aspect of Data Engineering, MDM as a source and destination requires the data to be small. This means Event Driven. I deliberately avoid the "Streaming" moniker, and focus instead on message queue, or AWS Event Bridge tech instead. Keep in mind that for MDM lookup use-cases, caching of mastered data for use in Big Data is your best bet. Event Driven is best for write use-cases

已翻译

赞

Data Engineering

+ 关注

给文章评分

我们借助人工智能创建了此文章。您认为这篇文章怎么样？

很棒不太好

举报此文章

查看全部

How can you scale MDM to handle big data in Data Engineering?

1

2

3

4

5

1 Choose the right architecture

2 Leverage cloud and big data technologies

3 Implement data governance and quality standards

4 Adopt an agile and iterative approach

5 Here’s what else to consider

Data Engineering

给文章评分

感谢您的反馈

更多Data Engineering相关文章

更多相关阅读内容

How can you scale MDM to handle big data in Data Engineering?

1

2

3

4

5

1 Choose the right architecture

2 Leverage cloud and big data technologies

3 Implement data governance and quality standards

4 Adopt an agile and iterative approach

5 Here’s what else to consider

Data Engineering

给文章评分

感谢您的反馈

查看其他技能