登录查看更多内容

How can data lineage and provenance improve data architecture?

由人工智能和领英社区提供技术支持

Data is the lifeblood of any organization, but it can also be a source of confusion, errors, and inefficiencies if not managed properly. Data architecture is the discipline of designing, implementing, and governing the data assets and flows that support the business goals and processes. One of the key aspects of data architecture is ensuring the quality, accuracy, and consistency of the data, which can be achieved by using data lineage and provenance. In this article, you will learn what data lineage and provenance are, why they are important, and how they can improve your data architecture.

此文章中的业界达人

由社区从 6 条内容中精选。了解更多

Sejas Patel

Data & ML Platforms, Marketing Technology and Customer Experience Platforms in Online & Hybrid Retail Businesses
Agha Furrukh Zahid

Integration Architect | Building scalable cloud integrations | MIT Fellow | Thoughtful Leader | Azure Certified * 3 |…
Siddharth Rajagopal

Chief Architect at Informatica | Executive MBA at Quantic | Author |

1 What is data lineage?

Data lineage is the traceability of the data from its origin to its destination, including all the transformations, manipulations, and uses that it undergoes along the way. Data lineage helps you understand where your data comes from, how it changes, and where it goes. Data lineage can be visualized as a map or a graph that shows the data sources, data flows, data transformations, data dependencies, and data destinations.

添加您的观点

Agha Furrukh Zahid

Integration Architect | Building scalable cloud integrations | MIT Fellow | Thoughtful Leader | Azure Certified * 3 | Power Apps | Biztalk | .Net | Sql | Philanthropist | Community Builder
举报内容
Improving data architecture requires careful consideration of data provenance and lineage. Comprehending data lineage, which documents the source and evolution of data, guarantees traceability and transparency and facilitates compliance, debugging, and data quality evaluation. Provenance, detailing the history of data changes, fosters trust and reliability. By offering a clear picture of data origins, transformations, and quality, integrating them into data architecture improves data governance, permits efficient data management, lowers errors, helps regulatory compliance, and promotes more informed decision-making.

已翻译

赞
Sagar Navroop

Data Architect | AI | MLOps | AWS | SIEM | Observability | Technologist
(已编辑)
举报内容
1) Data lineage provides a view of the origin and transformation of data, while Provenance ensures data authenticity, contributing to higher overall data quality 2) Data lineage brings visibility on how data moves through systems while provenance ensures traceability, helps with the compliance/regulatory needs 3) Data lineage helps stakeholders make informed decisions while provenance ensures data reliability, brings trust in process 4) Data lineage aids in quickly identifying and resolving issues while provenance helps, streamline the troubleshooting processes 5) Data lineage helps integrate diverse data sources while provenance supports the efforts, ensures data consistency Bottomline both compliment each other.

已翻译

赞

加载更多内容

2 What is data provenance?

Data provenance is the documentation of the history, context, and ownership of the data, including the metadata, annotations, and citations that describe the data. Data provenance helps you understand who created the data, when, why, and how. Data provenance can be stored as a record or a log that contains the details and evidence of the data creation, modification, and usage.

添加您的观点

加载更多内容

3 Why are data lineage and provenance important?

Data lineage and provenance are important for multiple reasons, such as data quality, data governance, and data analysis. Data quality is improved by tracing the data back to its source to verify accuracy, validity, and completeness. Additionally, tracking the data changes can detect and correct anomalies, outliers, or biases. Data governance is established by monitoring the data flows and recording the data ownership to assign roles, responsibilities, and rights. Furthermore, data analysis is enhanced by understanding the data transformations and capturing the data metadata. Finally, citing the data references can validate the reliability, credibility, and authority of the data.

添加您的观点

Siddharth Rajagopal

Chief Architect at Informatica | Executive MBA at Quantic | Author |
举报内容
One specific usecase of Data Lineage and provenance can be during Application Consolidation/Migration and Modernization Initiatives. By providing automated lineage Enterprises can easily perform impact analysis and identify what other applications, datasets, reports and stakeholders are impacted. Provenance ensures the right change management processes to be setup to handle and mitigate during such initiatives.

已翻译

赞

4 How can data lineage and provenance improve data architecture?

Data lineage and provenance can improve your data architecture by enabling you to design your data architecture with clarity, coherence, and consistency. By mapping your data lineage, you can identify and define your data sources, data flows, data transformations, data dependencies, and data destinations. Documenting your data provenance will help you specify and describe your data history, context, and ownership. Automating your data lineage can capture and update your data flows in real-time. Storing your data provenance will allow you to access and retrieve your data history on demand. Integrating your data lineage and provenance with tools and platforms can enable the architecture to adapt and evolve with changes in the data volume, variety, and velocity. Auditing your data lineage can monitor and measure its quality, accuracy, and consistency. Securing your data provenance can protect the privacy, confidentiality, and integrity of the data. Reporting the lineage and provenance can demonstrate and verify its compliance with regulations. Data lineage and provenance are essential components of a successful data architecture that can help you improve the quality of your data governance and analysis for better business performance and value.

添加您的观点

Sejas Patel

Data & ML Platforms, Marketing Technology and Customer Experience Platforms in Online & Hybrid Retail Businesses
(已编辑)
举报内容
The story is very simple. Lineage establishes strong foundation for integrity of data (as an asset) in your entire architecture. Provenance is tightly associated with custodianship and ownership of that said asset - somebody’s concern and tools to manage, improve and timely remediation of the risks associated with the value of that asset. Together these two aspects of data management “framework” cultivate a robust data architecture that is defined, scalable and agile to act upon in its nature. It also enables custodians of data to create an architecture with purpose that not only scales technically but also organisationally.

已翻译

赞

5 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?

添加您的观点

Data Architecture

+ 关注

给文章评分

我们借助人工智能创建了此文章。您认为这篇文章怎么样？

很棒不太好

举报此文章

查看全部

How can data lineage and provenance improve data architecture?

1

2

3

4

5

1 What is data lineage?

2 What is data provenance?

3 Why are data lineage and provenance important?

4 How can data lineage and provenance improve data architecture?

5 Here’s what else to consider

Data Architecture

给文章评分

感谢您的反馈

更多Data Architecture相关文章

更多相关阅读内容

How can data lineage and provenance improve data architecture?

1

2

3

4

5

1 What is data lineage?

2 What is data provenance?

3 Why are data lineage and provenance important?

4 How can data lineage and provenance improve data architecture?

5 Here’s what else to consider

Data Architecture

给文章评分

感谢您的反馈

查看其他技能