How do you face common data quality risks?

Data Quality plays a critical role in ensuring that the data collected and processed is accurate, complete, and reliable, as poor data quality can lead to erroneous insights and decisions. Here are some essential strategies to address these risks: Data Validation: Implement rigorous data validation checks to catch missing values, outliers, and anomalies as early as possible in the data pipeline. Deduplication: Identify and remove duplicate records or entries from your datasets to prevent over-representing specific data points. Data Profiling: Conduct thorough data profiling to understand the characteristics of your data, such as data distributions and patterns.

Each use case is unique. After typical libraries and code coverage are checked, I then use my engineer brain to logically step through each step in the pipelines to see if there are edge cases bleeding through, then algorithmically code for edge cases that are hard to predict before putting a product out in the wild. The best practice is to build POC with wild data to catch as many of these pre-production. That skill is something I honed via my background in both computer science and data science. The ability to predict “yet unseen” cases is, as described to me by a very smart man, part of my skill set in also having a background in cryptography and post quantum cryptography. It saves time & $ because less “fine tuning” of models.

Start by understanding the key dimensions of data quality, such as accuracy, completeness, consistency, timeliness, and reliability. Recognizing these dimensions helps in pinpointing what aspects of your data need attention and forms the basis for a comprehensive data quality strategy.

Data Transformation: Standardize and clean data during the transformation process, ensuring consistency in formats, units, and terminology. Data Lineage Tracking: Implement data lineage tracking to trace data's journey from source to destination, facilitating quick error identification and correction.

One way to assess the level of your data quality is to measure how much your data consumers trust the data they are using. If they trust it, and are confident in using it, then the data is of good enough quality for their use case. To measure this you can run regular surveys with the data consumers, asking questions like: - How much trust do you have in the data? Can you use it directly, or do you often need to validate it? - How often do you use data for your daily tasks? What’s preventing you from making use of more data? - Where have you recently used data successfully to generate business value? What data was it, and how did you use it? We now have some qualitative data we can use to measure the quality of the data.

Assessing data quality levels for the first time should be done as part of profiling. The initial data quality level should be measured for every table. A reliable measure is a percentage of passed data quality checks out of all checks that were verified on the table. One point should be noted here: not all data quality issues are equal. Some data quality issues are severe and should be given a higher score. We can also identify other issues that are just anomalies, which are just warnings. The true data quality level should be easy to communicate to the stakeholders and the data producer. Calculating a weighted KPI works the best. KPI is a percentage of passed data quality checks for issues with severity above a warning.

As per the Data Quality Framework, start by identifying key data quality metrics such as accuracy, completeness, consistency, timeliness, and reliability focusing on the CDE's that is very critical for Business. As part of this, try to understand key business drivers and associated data initiatives for each business domain. Use data profiling tools to examine your data and identify issues like missing values, duplicates, and outliers. Consult with business stakeholders to understand their expectations and requirements regarding data quality as they drive these data on daily business decision making.

Conduct assessments to gauge the current quality levels of your data across different dimensions. Utilizing tools and metrics for this evaluation provides a clear picture of where your data stands, helping identify areas that require immediate improvement.

Most of us call RCA - Identify the root causes of data quality issues. This may involve examining data sources, data collection processes, and data transformation procedures. As part of this journey, we might keep track of the following: Identifying Key Data Sets, Metadata, data silos Develop Data Dictionary Building data relationship and data lineage Identify Data Sources, owners & relationships, Conduct Data Profiling. Create scorecards or dashboards to track data quality metrics over time. This helps in monitoring and identifying trends.

Use DMAIC Six Sigma data driven approach helps to categorize, define, classify, analyze, and measure the data quality issues. Post this, the classification of issues based on impact and occurrence leads to assigning priority the right way. Following these helps to establish the proper implementation of improvement and control measures.

Dive deep into the root causes of identified data quality issues. Whether it's discrepancies in data entry, outdated information, or system integration problems, understanding the underlying causes is crucial for developing effective solutions.

Data quality issues are detected by running data quality checks. However, the results of the checks are data quality issues. They are raised by data quality libraries used inside data pipelines or by dedicated data quality tools. Every detected issue results in some extra work for the data teams that must be planned between other activities. The data quality issues must be assessed and planned as regular tasks, using the same work planning system that is used by teams who can fix the issue. How to achieve that integration? Each data quality issue should trigger a task. Integrate the notification system with your task management platform (Jira, TFS, ServiceNow) and pass the issues to the right teams. Otherwise, nobody will fix them.

on your analysis, implement targeted actions to address data quality issues. This might involve cleansing data, reconciling discrepancies, and updating outdated information. Employing automated tools can streamline these processes, ensuring data integrity and boosting overall data quality

Ideally, you stop data quality problems from coming in in the first place. That means controls on data entry and controls on data loading. However, if you choose not to implement those controls or you choose to bring the data in with problems, then you assume the risk. When you have the risk, you need to clearly communicate to people what the data quality issues are, or communicate that you have not checked data quality. People can then choose to accept the risk and use the data or pause their work until the data quality issue is resolved.

Implement validation rules and checks to prevent the entry of low-quality data at the source and maintain documentation on data quality standards, processes, and improvements. Ensure that data quality status and improvements are communicated to relevant stakeholders. As a DG practitioner, I need to foster a culture of data quality within the organization by emphasizing its importance and making data quality a shared responsibility. Based on the maturity of the organization, invest in data quality management tools and technologies to streamline processes and automate data quality checks.

To prevent future data quality issues, set up ongoing controls and checks. These could include validation rules, regular audits, and quality thresholds that data must meet before being used. Establishing these controls helps maintain high data quality standards consistently

The whole data quality control is about monitoring, measuring, reacting, and improving. Monitor the data quality using data quality checks, you need a set of agreed data quality checks used across the organization. Measure the data quality by applying acceptable thresholds for all data quality measures. 5% of null values is still accepted in a column, that is fine. There are 6% - raise an issue. Now, you can measure the issues, count them. React to the issues by setting up a whole process around reacting to issues, prioritizing them, and assigning them to the right teams. Finally improve. Fix the issues that are identified. Look at the numbers - the most common issues, tables affected most frequently, focus on them and reprioritize...

While refining Data Quality Activities or introducing new processes doing Root Cause Analysis of existing Data Quality Issues can help a lot. We can apply a technique like 5 Why's? to get to the bottom of Data Quality issue and then work with appropriate stakeholders to define processes which will help in mitigating these issues in future.

Data quality management is an ongoing process. Regularly review and refine your data quality strategies, processes, and controls based on new insights, changes in data usage, or evolving business needs. Continuous improvement ensures your data quality initiatives remain effective over time.

Data Quality risks can emanate from a number of reasons including poor data quality checks, outdated data policies, inadequate data maturity scale and misaligned data strategy. Poor data quality checks would be in the nature of redundant checks which do not arrest data abberation. Outdated data policies would not be in line with the ever-changing market and organisational change. Inadequate data maturity would be when data governance has not scaled with the way data is consumed by the organisation. Data Strategy would be misaligned when it is not in vogue with the Business Strategy. All these shortcomings would elevate data quality risks if not considered at a timely manner.

Beyond the immediate steps for managing data quality risks, consider the broader organizational culture around data. Promoting data literacy, fostering a culture of quality, and ensuring executive support for data quality initiatives are critical for long-term success. Additionally, staying abreast of emerging technologies and methodologies can offer new ways to enhance data quality.