登录查看更多内容

What are the key metrics to measure the quality and completeness of your data profiling results?

由人工智能和领英社区提供技术支持

Data profiling is the process of analyzing the structure, content, and quality of your data sources, such as databases, files, or APIs. It helps you understand the characteristics, patterns, anomalies, and relationships of your data, and identify any issues or gaps that need to be addressed before using it for your data architecture projects. But how do you measure the quality and completeness of your data profiling results? Here are some key metrics that you can use to evaluate your data profiling outcomes and ensure that they meet your data architecture goals and standards.

此文章中的业界达人

由社区从 6 条内容中精选。了解更多

Dan Hare
Sanyog Sharma

Data Engineering| Data Products | Search and Recommendation | Product Analytics | Reporting | Compliance

1 Data completeness

Data completeness measures how much of the expected or required data is available in your data sources. It can be calculated as the percentage of non-null or non-missing values in a column, table, or dataset. Data completeness can also be assessed at the record level, by checking if all the mandatory or key attributes are populated for each record. A low data completeness score indicates that your data sources have gaps or inconsistencies that may affect your data architecture design and performance.

添加您的观点

Sanyog Sharma

Data Engineering| Data Products | Search and Recommendation | Product Analytics | Reporting | Compliance
(已编辑)
举报内容
Data completeness in terms of ETL can be a bit different. It tracks from source data are you able to copy complete data to target or not. Suppose in my source DB daily transactions are ~70K but while copying data to warehouse we miss ~1K transaction due to any failure. Completeness % makes sures target data matches with source data.

已翻译

赞

2 Data accuracy

Data accuracy measures how well the data values match the reality or the source of truth. It can be evaluated by comparing the data values with external references, such as authoritative sources, business rules, or domain knowledge. Data accuracy can also be verified by performing data validation checks, such as data type, format, range, or logic. A high data accuracy score indicates that your data sources have reliable and consistent information that can support your data architecture decisions and outcomes.

添加您的观点

Sanyog Sharma

Data Engineering| Data Products | Search and Recommendation | Product Analytics | Reporting | Compliance
举报内容
Data accuracy ensures that data going in downstream systems is usable, is in correct data type and matches business logic. For eg. You are an ecommerce company and use clickstream data for training of recommendation models and one of the metric you use is CTR. For calculation of CTR we use clicks and impressions. You can define for a particular category your CTR cannot be greater than 100% or infact it should be between 1-10% range. This ensures that data that is going to recommendation model is correct so that your model also gives accurate predictions.

已翻译

赞

3 Data consistency

Data consistency measures how well the data values are aligned and coherent across different data sources, columns, or records. It can be measured by identifying and quantifying any data conflicts, discrepancies, or duplicates that may exist in your data sources. Data consistency can also be improved by applying data standardization, normalization, or deduplication techniques. A high data consistency score indicates that your data sources have harmonized and integrated data that can enable your data architecture integration and interoperability.

添加您的观点

Sanyog Sharma

Data Engineering| Data Products | Search and Recommendation | Product Analytics | Reporting | Compliance
举报内容
In various cases while copying data from source DB to target DB it can happen due to any error there can be duplicates, discrepancies in your target data. To avoid this we can use data deduplication and data standardization techniques. eg. For generating invoicing data we are using transactions data and one tranasaction_id repeated multiple times this might cause errors while calculating invoice amounts. To avoid this we can use deduplication techniques

已翻译

赞

4 Data timeliness

Data timeliness measures how current and relevant the data values are for your data architecture purposes. It can be determined by examining the data freshness, frequency, and latency of your data sources. Data freshness refers to how often the data values are updated or changed. Data frequency refers to how often the data sources are accessed or refreshed. Data latency refers to how long it takes for the data values to be available or delivered from the source to the destination. A high data timeliness score indicates that your data sources have up-to-date and responsive data that can support your data architecture agility and scalability.

添加您的观点

5 Data uniqueness

Data uniqueness measures how well the data values are distinct and identifiable in your data sources. It can be assessed by counting the number of distinct values in a column, table, or dataset, and comparing it with the expected or desired number of unique values. Data uniqueness can also be ensured by defining and enforcing data identifiers, such as primary keys, foreign keys, or surrogate keys. A high data uniqueness score indicates that your data sources have clear and unambiguous data that can facilitate your data architecture traceability and accountability.

添加您的观点

6 Data diversity

Data diversity measures how well the data values represent the variety and richness of your data sources. It can be analyzed by exploring the data distribution, frequency, or outliers of your data sources. Data distribution refers to how the data values are spread or clustered in a column, table, or dataset. Data frequency refers to how often the data values occur or repeat in a column, table, or dataset. Data outliers refer to the data values that are significantly different or deviate from the rest of the data values. A high data diversity score indicates that your data sources have comprehensive and diverse data that can enhance your data architecture innovation and insight.

添加您的观点

7 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?

添加您的观点

Dan Hare
举报内容
Data quality should be the first priority of every organisation to improve their confidence level and simplify downstream processes; start capturing code-free, time-based metrics with simple burndown dashboards now ! Data Quality, cataloguing, lineage and mastering are also the perfect fields to be disrupted with AI; so much issue identification and remediation is formulaic but currently incomplete, disconnected, manual and therefore expensive.

已翻译

赞
Monika Kovacs

///
举报内容
I was considering this from Data Stewards / Data Sponsors perspective, or from the perspective of those who work with them. The metrics I would choose to use for analysis would be those that are within their power to control. For example, measuring data completeness is not that useful if you are never able to improve it. Too many metrics can lead to confusion, I think I would seek not to use more than 3

已翻译

赞

加载更多内容

Data Architecture

+ 关注

给文章评分

我们借助人工智能创建了此文章。您认为这篇文章怎么样？

很棒不太好

举报此文章

查看全部

What are the key metrics to measure the quality and completeness of your data profiling results?

1

2

3

4

5

6

7

1 Data completeness

2 Data accuracy

3 Data consistency

4 Data timeliness

5 Data uniqueness

6 Data diversity

7 Here’s what else to consider

Data Architecture

给文章评分

感谢您的反馈

更多Data Architecture相关文章

更多相关阅读内容

What are the key metrics to measure the quality and completeness of your data profiling results?

1

2

3

4

5

6

7

1 Data completeness

2 Data accuracy

3 Data consistency

4 Data timeliness

5 Data uniqueness

6 Data diversity

7 Here’s what else to consider

Data Architecture

给文章评分

感谢您的反馈

查看其他技能