登录查看更多内容

How can you ensure data quality across different sources and domains?

由人工智能和领英社区提供技术支持

Data quality is a key aspect of data governance, as it ensures that the data you use for analysis, reporting, and decision making is accurate, consistent, and reliable. However, data quality can be challenging to maintain when you have to deal with different sources and domains of data, such as databases, files, APIs, web pages, or external providers. How can you ensure data quality across different sources and domains? Here are some tips to help you.

此文章中的业界达人

由社区从 5 条内容中精选。了解更多

Tiankai Feng

Data & AI Strategy Director @ Thoughtworks | Author of "Humanizing Data Strategy" | TEDx Speaker | Data Musician
Stanley Guinn MS Data Analytics

Data Analyst @ Missile Defense Agency | Data Engineering, Data Analysis
Peter Ndiforchu

I help entry-level, non-technical professionals get their first data analyst role | Senior Data Analyst @RBC |Top rated…

1 Define data quality rules and metrics

The first step to ensure data quality is to define what data quality means for your context and goals. You need to establish data quality rules and metrics that specify the criteria and standards for data validity, completeness, accuracy, timeliness, consistency, and relevance. For example, you may have rules for data formats, values, ranges, patterns, dependencies, or business logic. You may also have metrics for data quality dimensions, such as error rate, completeness ratio, accuracy score, or timeliness gap. You can use data quality tools or frameworks to help you define and document your data quality rules and metrics.

添加您的观点

Peter Ndiforchu

I help entry-level, non-technical professionals get their first data analyst role | Senior Data Analyst @RBC |Top rated Data Career Coach on topmate.io
举报内容
Data Quality rules are a great way to monitor data quality issues. In order to design efficient data quality rules a few things need to be considered: 1. What are the measuring points? Are you measuring on a System of Record(SOR) or an Authorized Provisioning Point (APP). Knowing your measuring helps with how to best design the DQ rules. 2. What is the hierachy of the DQ dimensions and which one covers a multitude? You don't just want to design rules, you want to design efficient rules. An accuracy rule might be best suited to measure against a source of truth than a validity rule 3. What is your threshold for raising an alarm? Set a meaning threshold based on materiality and resources. You can always change the threshold later on.

已翻译

赞

2 Assess data quality across sources and domains

The next step is to assess the current state of your data quality across different sources and domains. You need to apply your data quality rules and metrics to your data sets and evaluate how well they meet your expectations and requirements. You can use data quality tools or frameworks to help you perform data quality assessment, such as data profiling, data cleansing, data validation, or data monitoring. You can also use data quality dashboards or reports to visualize and communicate your data quality results and issues.

添加您的观点

Tiankai Feng

Data & AI Strategy Director @ Thoughtworks | Author of "Humanizing Data Strategy" | TEDx Speaker | Data Musician
举报内容
Assessing data quality is not only about finding out problems, but also about investigating WHY there are problems. Any data quality issue can be roughly traced back to three root cause types: technical issues, business process related issues and human behavior related issues. Different problem types require different solutions, but having a pre-defined solution approaches can turn the assessment into improvement actions more efficiently.

已翻译

赞

3 Implement data quality improvement actions

The final step is to implement data quality improvement actions based on your data quality assessment results and issues. You need to identify and prioritize the root causes and impacts of your data quality problems and plan how to resolve them. You can use data quality tools or frameworks to help you implement data quality improvement actions, such as data correction, data enrichment, data standardization, data integration, or data governance policies. You can also use data quality feedback loops or reviews to track and measure your data quality improvement progress and outcomes.

Data quality is not a one-time activity, but a continuous process that requires ongoing attention and collaboration. By following these tips, you can ensure data quality across different sources and domains and enhance your data governance capabilities and performance.

添加您的观点

Tiankai Feng

Data & AI Strategy Director @ Thoughtworks | Author of "Humanizing Data Strategy" | TEDx Speaker | Data Musician
举报内容
Avoiding data quality issues should always be preferred over fixing them reactively - unless it’s technically not feasible or it’s too high efforts to do so. Setting up data quality rules and standards for any new data objects upfront, doing continuous trainings with data producers and embedding automated validations and harmonizations in data systems can help with that. Any prevention can never fully replace data quality monitoring and remediation, but they can be a powerful combination when used together.

已翻译

赞
Peter Ndiforchu

I help entry-level, non-technical professionals get their first data analyst role | Senior Data Analyst @RBC |Top rated Data Career Coach on topmate.io
举报内容
Data quality issue remediation is not a one time done and dusted effort. It requires joint effort from all stakeholders in remediation the issue from the source and not just fixing it, but ensuring that there's education done to ensure to issue is not repeated. Sometimes it could be just a simple fix required from IT whereby, data integrity constraint is implemented at the database level. For example, ensuring that valid dates are entered 'YYYYMMDD' and not free form for customer orders. Other times the change requires the opinion of multiple stakeholders including the business and final consumers of the data, finance, HR, etc around how the business logic should be enforced.

已翻译

赞

4 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?

添加您的观点

Stanley Guinn MS Data Analytics

Data Analyst @ Missile Defense Agency | Data Engineering, Data Analysis
举报内容
Obviously this is why we should have a well defined Data Governance/Management program and policies in place. Another key to this issue is also having a Master Data Architecture in place with maturity of the program. I don't have the space here to explain all of the in's and out's of a Data Management program but I work with and prescribe to the DAMA-DMBOK Data Management Body of Knowledge, I would suggest that you pull up a copy of this architecture and it will help answer most if not all of the questions you may have.

已翻译

赞

Data Governance

+ 关注

给文章评分

我们借助人工智能创建了此文章。您认为这篇文章怎么样？

很棒不太好

举报此文章

查看全部

How can you ensure data quality across different sources and domains?

1

2

3

4

1 Define data quality rules and metrics

2 Assess data quality across sources and domains

3 Implement data quality improvement actions

4 Here’s what else to consider

Data Governance

给文章评分

感谢您的反馈

更多Data Governance相关文章

更多相关阅读内容

How can you ensure data quality across different sources and domains?

1

2

3

4

1 Define data quality rules and metrics

2 Assess data quality across sources and domains

3 Implement data quality improvement actions

4 Here’s what else to consider

Data Governance

给文章评分

感谢您的反馈

查看其他技能