登录查看更多内容

What are the key features of a data catalog for data lineage and impact analysis?

由人工智能和领英社区提供技术支持

Data governance is the practice of managing the quality, security, and usability of data assets in an organization. A data catalog is a tool that helps data governance by providing a centralized repository of metadata, such as data definitions, sources, owners, and usage. One of the main benefits of a data catalog is that it enables data lineage and impact analysis, which are essential for understanding the origin, transformation, and destination of data, as well as the potential consequences of data changes. In this article, we will explore what are the key features of a data catalog for data lineage and impact analysis, and how they can help you achieve your data governance goals.

此文章中的业界达人

由社区从 5 条内容中精选。了解更多

Jade Jones

Metadata & Data Quality Manager at Sainsbury’s | Top Twenty Rising Stars in Data & Analytics 2024

1 Data lineage visualization

Data lineage is the traceability of data from its source to its destination, across different systems, processes, and transformations. Data lineage visualization is a feature that allows you to see the data flow in a graphical and interactive way, using nodes, edges, and attributes. This helps you to identify the data sources, dependencies, transformations, and quality issues, as well as to verify the accuracy and completeness of the data. Data lineage visualization can also help you to comply with data regulations, such as GDPR or CCPA, by showing you where the sensitive or personal data is stored, accessed, and used.

添加您的观点

Jade Jones

Metadata & Data Quality Manager at Sainsbury’s | Top Twenty Rising Stars in Data & Analytics 2024
(已编辑)
举报内容
It is imperative that the lineage that is surfaced to consumers is 'true' lineage. Many data cataloguing tools utilise query log ingestion as a basis for lineage, but it is often far too broad, ingesting test queries and those carried out by human users. To be valuable and clear, lineage ingestion should be restricted to those implemented by ETL processes, and ideally, only persisting objects. As with all artefacts in a data catalogue, lineage as a whole, but also on a transformation basis, should be owned; who can users query about the transformation logic that moves data from point A to point B? If possible, the logic or syntax should be captured in the lineage documentation.

已翻译

赞

2 Data impact analysis

Data impact analysis is the assessment of the potential effects of data changes on other data elements, processes, or systems. Data impact analysis is a feature that allows you to evaluate the impact of data changes before they are implemented, such as adding, deleting, or modifying a data field, table, or schema. This helps you to avoid data errors, inconsistencies, or failures, as well as to optimize the data performance and quality. Data impact analysis can also help you to manage data risks, such as data breaches, by showing you the exposure and vulnerability of the data.

添加您的观点

Jade Jones

Metadata & Data Quality Manager at Sainsbury’s | Top Twenty Rising Stars in Data & Analytics 2024
举报内容
Cross-system lineage plays a fundamental part in upstream impact analysis. At its simplest, data catalogues can visualise the affected upstream objects when a data breach or data quality issue is identified - this can be done on a small scale, within a singular database with multiple layers, or in a mature context, can be done cross system.

已翻译

赞

3 Data catalog search

Data catalog search is a feature that allows you to find and access the relevant data assets in the data catalog, using keywords, filters, or natural language queries. Data catalog search helps you to discover and understand the data that is available, such as the data name, description, type, format, location, owner, and usage. Data catalog search can also help you to compare and contrast different data assets, such as the data lineage, impact, or quality, by showing you the similarities and differences among them.

添加您的观点

Jade Jones

Metadata & Data Quality Manager at Sainsbury’s | Top Twenty Rising Stars in Data & Analytics 2024
举报内容
To make this useful, it requires initial investment and an understanding of how users should be able to discover objects, and groupings of objects. Contributors should be empowered to link together different kinds of artefacts; a business process about demand forecasting, the resultant data objects, and the associated visualisations should all be discoverable together in an easy search.

已翻译

赞

4 Data catalog collaboration

Data catalog collaboration is a feature that allows you to share and communicate with other data users, such as data analysts, data scientists, or data stewards, in the data catalog. Data catalog collaboration helps you to exchange and enrich the data knowledge, such as the data definitions, standards, policies, or best practices. Data catalog collaboration can also help you to solicit and provide feedback, such as the data ratings, reviews, or comments, by enabling you to rate and review the data assets, or to comment and reply on the data lineage or impact analysis.

添加您的观点

Jade Jones

Metadata & Data Quality Manager at Sainsbury’s | Top Twenty Rising Stars in Data & Analytics 2024
举报内容
Communicating the expectations, accountabilities and responsibilities of stewards and other prevalent roles in your data catalogue, early on is imperative. It should be clear from the initial implementation that these SMEs are accountable for reviewing content, contributing expertise and being a point of contact for collaboration and clarification - without this, these roles are baseless and are simply signposts.

已翻译

赞

5 Data catalog integration

Data catalog integration is a feature that allows you to connect and synchronize the data catalog with other data sources, platforms, or tools, such as databases, data warehouses, data lakes, or BI tools. Data catalog integration helps you to automate and update the data metadata, such as the data schema, structure, or quality, by scanning and extracting the metadata from the data sources. Data catalog integration can also help you to leverage and enhance the data value, such as the data analysis, visualization, or reporting, by enabling you to access and use the data assets from the data catalog in other tools.

添加您的观点

Jade Jones

Metadata & Data Quality Manager at Sainsbury’s | Top Twenty Rising Stars in Data & Analytics 2024
举报内容
A cost-benefit analysis should be carried out when considering connecting to a new source, or type of source. The infrastructure and configuration costs, both financially and in time and resource, can be significant particularly when looking to connect to early pipeline technologies or unstructured data. The correct questions should be asked; who will benefit from being able to discover this data? Is this the missing gap in a cross-system lineage flow?

已翻译

赞

6 Data catalog governance

Data catalog governance is a feature that allows you to control and monitor the data catalog activities, such as the data creation, modification, or deletion, in the data catalog. Data catalog governance helps you to enforce and audit the data rules, such as the data access, security, or quality, by applying and tracking the data policies, roles, or permissions. Data catalog governance can also help you to measure and improve the data outcomes, such as the data usage, trust, or value, by generating and analyzing the data metrics, reports, or alerts.

添加您的观点

7 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?

添加您的观点

Data Governance

+ 关注

给文章评分

我们借助人工智能创建了此文章。您认为这篇文章怎么样？

很棒不太好

举报此文章

查看全部

What are the key features of a data catalog for data lineage and impact analysis?

1

2

3

4

5

6

7

1 Data lineage visualization

2 Data impact analysis

3 Data catalog search

4 Data catalog collaboration

5 Data catalog integration

6 Data catalog governance

7 Here’s what else to consider

Data Governance

给文章评分

感谢您的反馈

更多Data Governance相关文章

更多相关阅读内容

What are the key features of a data catalog for data lineage and impact analysis?

1

2

3

4

5

6

7

1 Data lineage visualization

2 Data impact analysis

3 Data catalog search

4 Data catalog collaboration

5 Data catalog integration

6 Data catalog governance

7 Here’s what else to consider

Data Governance

给文章评分

感谢您的反馈

查看其他技能