How can you establish and enforce data quality standards?

One thing I found helpful is to look at all the data quality dimensions and address each dimension . Here are the dimensions - completeness , accuracy, conformity, consistency, coverage , timeliness and uniqueness. It is vital to address all these DQ dimensions across the data supply chain.

By establishing and enforcing data quality standards is essential for organizations. It involves defining clear standards, creating a data governance framework, and using tools for profiling and assessment. Data quality metrics, rules, and documentation aid in monitoring and maintaining data quality. Training and fostering a data-driven culture are crucial. Continuous monitoring, remediation, auditing, and reporting ensure adherence to standards. Feedback-driven improvements and accountability are key aspects. Ultimately, a strong data quality culture should prevail, making data accuracy, completeness, consistency, timeliness, and relevance a shared responsibility and core value within the organization.

One important aspect of Data Quality is to fix issues at the root. So, I think, it is crucial to invest in the implementation of data quality metrics at the first data source that is feeding others in order to deliver proper data. Fixing data issues later in the data lineage is more expensive as it potentially needs to be done multiple times and the issues might be harder to detect due to data manipulation on the way.

With data quality dimensions providing the overall framework, it is imperative to identify specific data quality indicators within each dimension. Quality indicators should be customized to the purpose of each dataset, must have clear, standardized definitions, and will ideally be weighted with regard to degree of concern or risk of adverse impact. This approach will help drive a sensible process for identifying data issues and determining appropriate corrective actions, and will ultimately decrease the risk of subsequent data quality problems.

Accuracy and Timelines acts vital for every role from End user till delivery management group. Any Delays or accuracy caused by any user between this massive working group can affect the outcome. Hence, having a track of these two magnificient words from start up and especially in every group, can help us to reach the goal smoothly and efficiently. Sincere contribution from each user becomes very helpful in achieving the driven aspect.

1. Data Accuracy: If the collected data is not accurate to the standards set around it, the quality is compromised. 2. Validity: Ensure we have valid and complete data is an integral part of data quality. 3. Integrity: The data collected must be unique and consistent. Inconsistent across different source systems compromises the integrity. 4. Relevance: Unwanted and irrelevant data collected overtime compromises the quality and in turn adds more complexity. A clean data helps. 5. Data Definition: Proper data definition ensures data quality. 6. Continuous Testing: While following CICD, continuous testing is an important step that should not be compromised. This is one of the most integral part to ensure data quality.

In my experience, assessing data quality levels should be a periodic activity. The frequency of which can be determined based on the criticality of the data in question. In addition, to what the article describes here it is also important to decide the sample size that you would assess for determining data quality levels. Indeed, assessing data quality requires time and efforts hence it is important to expend these judiciously. Assessing data quality levels is similar to taking routine health checks. Data profiling is an effective way to do this but with the help of AI & ML organizations can deploy these capabilities to perform automated data anomaly detection.

Here are some additional ways to get started with measuring data quality levels: - Define Data Quality Metrics that are relevant to your organization - Data Quality Scorecards: Create data quality scorecards or dashboards to track key metrics over time. - Data Quality Assessment Frameworks: like DAMA DMBOK or ISO 8000. - Data Quality Surveys: Collect feedback from data users and stakeholders through surveys or interviews to understand their perception of data quality. - Data Proficiency Testing: Test the proficiency of data users in using the data. - Data Quality Tools: Invest in data quality tools and software that can automate the measurement and monitoring.

Assessing data quality is paramount in my role as a Data Engineer. We utilize a robust framework of data quality dimensions, including accuracy, consistency, completeness, and more. By employing various tools and methods such as data profiling, auditing, cleansing, and validation, we're able to gain deep insights into our data sources, processes, and systems. Data profiling helps us understand the data's structure and content, while data auditing ensures we continually monitor and improve our data quality performance. It's a dynamic process that keeps our data reliable and valuable for informed decision-making.

Most often data quality arises at the entry level. If we fail to capture and address it, we are giving it a free run for it to penetrate deep into all our systems and process. So to alleviate this problem, avoid manual data entry and when possible capture data collection through a well defined system. As a next step, check for the completeness of the data and report the metrics for all your data across systems. Constant monitoring will ensure whether your data quality is meeting the prescribed standards set by your organisation.

Establish clear data quality standards and metrics that align with your data quality objectives. Define acceptable values and tolerance levels for each data attribute. Then create data quality rules and validation checks based on the defined standards and metrics. These rules will be used to enforce data quality at various stages, including data entry, data integration, and data migration

Besides implementing data quality rules within our data pipeline, we should also consider training our staff on data quality and implementing data validation at the application level to minimize human errors as much as possible and to develop a data quality culture in our organization.

The data quality rules should be defined at various severity levels. Not all data quality issues are the same. Anomalies are just deviations from the regular range and should result only in warnings that are reviewed as part of the data quality process. Other data quality issues such as duplicate values in a key column should result in stopping the data pipeline to avoid spreading the issue across other downstream systems.

Clearly define the the roles and responsibilities for (data owners, data stewards, IT department , Information Security Team, Data Quality Team) as data quality is a collaboration work between all the different teams, not only data quallity team. Provide the awareness sessions to whole the employees can support the implementation of data quality policy.

Apart from above mentioned points, Data Quality Culture and Training are the key to success- Provide training to employees on data quality policies, rules, and best practices. Ensure that everyone understands their role in maintaining data quality. Foster a data quality culture within the organization, where everyone understands the importance of data quality and takes responsibility for it.

In my experience, an important aspect of improving data quality continuously is also creating data awareness. Fixing data quality issues is as trivial as applying bandage to never healing wound. Data drives decision-making, and for better decision-making it is not just the data quality that needs to be improved but also the data awareness of the people involved across the data lifecycle. Training goes a long way in creating a culture of data-driven.

Defining the Data working group in each department can facilitate the data quality practises, intiating Dashboards for each department to check their quality level may ecourage them tp do more

Once the initial data quality score is calculated as part of the profiling process, the same data quality checks should be repeated at regular intervals. The data quality score measured as a KPI can be compared to the baseline KPI calculated during profiling. It can be also compared to the KPI in the previous month. Comparing the KPI change across months will allow us to measure progress and plan data cleansing tasks in an agile way.

I personally think that the best way to get good data is actually to use it a lot. In my own experience, data normally flows from bottom to top, when the use of that data normally flows top to bottom. If you manage to create dashboards, KPIs and metrics that are actually useful for the day to day use of the feeders of that data, or at least the department more closely related to where it has been created. The usefulness of it will be a daily reminder of the importance of the data quality for their own processes and KPIs. Managers and employees alike will ensure that the numbers are correct. And you will fill feel that data quality also at the other end of the pipeline. Our main enemy here is distance and rustiness. Use your data!

Absolutely agree, continuous monitoring and improvement are vital for maintaining high data quality standards. Using dashboards, scorecards, and alerts for real-time tracking and communication is an effective way to stay ahead of potential issues. Root cause analysis, along with corrective and preventive actions, are essential for addressing not just the symptoms but the underlying causes of data quality issues. Adopting a data quality maturity model like TDWI or Gartner provides a structured path for assessing and enhancing data quality practices over time. It's a journey of constant evolution and adaptation to new challenges and technologies in the data landscape.

Data Quality checks can be divided into 2 parts : 1) Observability Checks: These deal with metadata and DQ dimensions like timeliness, volume, freshness 2) Data Quality Checks: These deal with dimensions like uniqueness, consistency, completeness, and business logic checks. Having a healthy mix of both these categories in all the data pipelines will enforce data quality standards as once you have these checks all different DQ dimensions will be covered as well as the business rules will be checked.

Data quality is something you should govern. If you establish data governance first, then you have the people identified who you need for data quality. This is a cross-functional effort because different departments can define data quality differently for the same data elements. You need to collect all those perspectives to define data quality for the company as a whole. You need people defined who are going to be monitoring data quality and bringing data quality issues to data governance for resolution. It becomes something that gets integrated into normal operations.

In addition to Observability and DQ Checks, it's equally important to manage the PERCEPTION of Data Quality. Perception = Expectation - Observation. The same observation might be OK for one person, and NOK for someone else, and both may be legit within their domains. The devil is always in the definitions: "Lead Performance" has a different meaning for Sales and Marketing, "AVG Revenue" might not look the same for Accounting as for the Pricing team. What helps to manage the expectation is providing context: * Having Data Product Owners * Easily findable and accessible documentation that contains calculation logic * Having metadata that indicates if dataset/metric is used frequently downstream (lineage, popularity), or "verified" badge

If you want to establish and enforce data quality, you should be working in a governed environment. If you set up your data governance organization first, you've got the people identified who will be working with data quality. Using representation throughout the company, you have different perspectives based on the different ways departments interact with the data. High quality for one might be sub-standard for another. Once defined, the rules need to be implemented and followed. If some data fails data quality checks, it needs to be resolved, again within your governed environment.

Establish a data governance framework that outlines roles and responsibilities for data quality management. This includes appointing data stewards who are accountable for data quality within their respective domains. Data governance also helps in setting up processes for data quality monitoring and enforcement.