登录查看更多内容

How can you define data lineage and track it across multiple systems?

由人工智能和领英社区提供技术支持

Data lineage is the process of tracing the origin, transformation, and usage of data across multiple systems and applications. It helps you understand where your data comes from, how it changes, and how it affects your business outcomes. Data lineage is essential for data quality, data governance, data security, and data compliance. In this article, you will learn how to define data lineage and track it across multiple systems using some best practices and tools.

此文章中的业界达人

由社区从 2 条内容中精选。了解更多

Nadeem Malik

Enterprise Architect - providing enterprise-wide insights, promoting standardization, reducing costs, mitigating risk…

1 Why is data lineage important?

Data lineage provides a clear and comprehensive view of your data flows and dependencies, enabling you to improve data quality, enhance data governance, strengthen data security, and ensure data compliance. It also helps to optimize your data architecture, streamline your data integration, and support your data analysis and reporting. With data lineage, you can identify and resolve data errors, inconsistencies, and anomalies; establish data ownership, accountability, and standards; detect and prevent data breaches, leaks, and unauthorized access; and adhere to data regulations, policies, and audits.

添加您的观点

Nadeem Malik

Enterprise Architect - providing enterprise-wide insights, promoting standardization, reducing costs, mitigating risk, increasing agility, realizing strategies, and driving innovation.
举报内容
I would advise implementing an end-to-end data lineage solution with tools to automatically discover, map, and visualize data flows across systems. This provides transparency into data provenance and impacts on governance.

已翻译

赞

2 How to define data lineage?

Data lineage can be defined at different levels of granularity, depending on your needs and objectives. This includes tracking the origin, transformation, and usage of each individual data element, such as a column or field; a group of related data elements, such as a table or file; or a collection of data sets involved in a specific business process, such as a workflow or pipeline. To define data lineage, you need to identify and document the source, destination, transformation, and usage of your data. The source can be an internal or external system that generates or provides your data. The destination can be an internal or external system that consumes or stores your data. The transformation can be any operation that modifies, filters, joins, or enriches your data along the way. And the usage can be any metric, indicator, or insight derived from your data to support decision making or action taking.

添加您的观点

3 How to track data lineage across multiple systems?

Tracking data lineage across multiple systems can be a difficult task, particularly in complex, heterogeneous, and dynamic data environments. To ensure consistency and reliability when capturing, storing, and accessing your data lineage information, you should employ best practices and tools. Metadata is a great way to record and store your data lineage information in a centralized and standardized way. You can then query and retrieve this data when necessary. A data catalog can also be used to discover, organize, and document your data assets. This tool scans, extracts, and indexes metadata from various sources and destinations, while providing a graphical or natural language interface to visualize, search, and explore your data lineage information. Additionally, a data lineage tool can be used to create, manage, and analyze this information by integrating with the data catalog and the sources/destinations. It also helps generate, validate, update the information via code analysis or machine learning. Finally, it assists in monitoring, reporting, and auditing the data lineage information via dashboards or notifications.

添加您的观点

4 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?

添加您的观点

José Lopez Colina

Head of Engineering, Data & AI @ RiverBank | Transforming Financial Services with Data and AI
举报内容
The best way to track the data lineage is with a data catalog tool. I've recently been working with my team with DataHub. It is a data catalog that can store and display datasets metadata, tags, business terms, table lineage and column lineage and more. Datasets and their lineage can be detected automatically for some source integrations but there is also an API to add links if needed. We currently use a combination of both methods.

已翻译

赞

Data Architecture

+ 关注

给文章评分

我们借助人工智能创建了此文章。您认为这篇文章怎么样？

很棒不太好

举报此文章

查看全部

How can you define data lineage and track it across multiple systems?

1

2

3

4

1 Why is data lineage important?

2 How to define data lineage?

3 How to track data lineage across multiple systems?

4 Here’s what else to consider

Data Architecture

给文章评分

感谢您的反馈

更多Data Architecture相关文章

更多相关阅读内容

How can you define data lineage and track it across multiple systems?

1

2

3

4

1 Why is data lineage important?

2 How to define data lineage?

3 How to track data lineage across multiple systems?

4 Here’s what else to consider

Data Architecture

给文章评分

感谢您的反馈

查看其他技能