How do you fix data quality problems?

I just like to think of them as ‘bugs’ like you do in any software. For me there is one true type of data quality problem - i.e. the source data on entry is wrong. It may be missing, it may be the wrong hierarchy, it may be inconsistent, Data quality bugs that are introduced in data pipelines are not data quality issues. They are production system issues. So they need to be rooted out and fixed in the code, or in the source, just like any other bug.

The first step in fixing data quality issues is to accurately identify them. This involves conducting a thorough review of your data to spot inconsistencies, inaccuracies, and any other issues that compromise its quality. Utilize tools and techniques like data profiling and auditing to uncover these problems. Remember, correctly identifying the issues is the foundation for all subsequent steps in improving data quality.

It’s important to scope the data quality approach first. For instance, in one narrow definition of data quality is bad if the values do not correspond to the underlying (physical) world. However, data quality is often seen much broader using the “fit for purpose” definition, including aspects such as context (discoverability) and data structure. Identifying data quality issues accurately is not always as trivial trivial as it sounds. For instance, detecting that a sensor has gone stale and is no longer updating can be difficult if flatlines naturally occur for the sensor at hand. In such cases, more advanced model based or AI based approaches may be needed to implement a scalable approach to the identification of data quality issues.

First of all, we should identify our project objectives and scope. Next, we should specify what we are expecting from the data. This will help us identify what is bad data in our scenario.

Once data quality issues are identified, the next step is a deep dive into their root causes. This analysis helps in understanding why these problems are occurring, whether they are due to process gaps, system errors, or other factors. By analyzing the underlying causes, you can develop more targeted and effective solutions to address these issues. Remember, thorough analysis is key to not just fixing problems temporarily, but to implementing long-lasting data quality improvements

Once you are aware of a specific issue, I like to paint a process diagram. And just follow up the data up to the place where it stop working and start asking questions. Once you understand what is happening, is easy enough to just find the variables that affect it and create alerts and controls around them. The problem comes when the fix is not obvious and needs some human interaction. Ok so the date is obviously incorrect, but then which is the correct one? Finding the right person, providing them with the question and asking yo answers is another topic. And again i think data has to be just part of the day to day life of everyone, and our job is to make it useful to all the parts involve. Not just a mandatory quality control.

While data quality problems need to be resolved, this AI-article evolved into a random list of somewhat related terms. If you have a data quality problem, you have to fix it at the source. Anything less means you have just operationalized a regular fixing process. That might be something you have to do temporarily while you're investigating and testing the solution, but it shouldn't be your long-term solution. Once fixed, this is also a good opportunity to implement a regular data quality check in case something similar happens again.

Eliminating unused data is a crucial step in enhancing data quality and reducing operational costs. This dormant data not only fails to contribute to business value but also burdens IT resources, increases storage demands, and potentially leads to misguided decisions. In our experience, organizations can effectively archive or discard upwards of 20% to 60% of their data without compromising operations or decision-making. By purging unnecessary data, businesses can streamline their data management processes, optimize resource allocation, and ultimately enhance data quality.

Eliminating unused data is crucial in enhancing data quality and reducing operational costs. It's very easy to flag and tell people they have a data quality problem, but not many platforms can solve it scalable. In the case of IoT/OT data, time series Gen-AI is needed to augment and repair data. There is no one-size-fits-all regarding cleansing. The use case decides what augmentation strategy is necessary.

Defining what is bad data for a specific project or situation and the possible causes of having such data is the key to defining how to resolve the quality issues present in our data. Resolving data quality problems may require implementing some complex or intelligent techniques such as machine learning guided cleaning, knowledge bases, and record linkage...

After identifying and analyzing the issues, the focus shifts to resolving them. This step involves implementing specific corrective actions, which could range from data cleansing and validation procedures to process reforms and system enhancements. The solutions should be tailored to address the root causes identified in the analysis phase. Effective resolution requires meticulous planning and execution to ensure that data quality is not only restored but also sustained over time.

Prepare a list of data quality rules that you want to run on every data load, or run daily. These data quality rules must be passed to consider whether the data quality is acceptable. Of course, not all issues are the same. Prioritization of data quality issue severity level is important. Less severe issues are just warnings to be reviewed. Highly severe data quality issues, such as a key column being empty, should be used to pause data pipelines. You can monitor these issues directly from the data pipeline before trying to load corrupted data.

I believe data quality can be approached by looking at it in three pillars: Accuracy, Completeness, and Value. Data that has Value has true worth to the business. Ownership of that data requires a business to make an internal contract with themselves with integrity, checks, and balances. It ensures that spending the time to ensure it is complete at the start and accurate when it's input or maintained. Once data loses it's value those contracts are often abandoned, and that is okay - but they should still exist should the data become valuable again.

Creating a data quality live dashboard that continuously reads the output and summaries of the data quality assessment operations is one of the most efficient approachs to keep an eye on the data quality metrics of our data.

Post-resolution, continuous monitoring is vital to ensure the data quality issues do not recur. This involves setting up ongoing checks and controls, like regular data audits and quality reports, to keep a close watch on data integrity. Monitoring helps in quickly identifying any new issues or lapses, allowing for prompt corrective actions. Remember, consistent monitoring is key to maintaining high data quality standards and preventing future problems

Begin with profiling data to identify inconsistencies, & inaccuracies. Establish clear data quality standards & define key metrics for assessment. Implement data cleansing processes to rectify errors, using tools like Talend or Informatica. Enforce data validation rules during input to prevent future issues. Collaborate with data owners to ensure accountability, also establishing data stewardship programs. Regularly monitor data quality for data flow management. Foster a culture of data awareness and education among teams to promote proactive problem identification & resolution. Implement data quality reporting mechanisms to provide transparency & accountability. Refine processes to uphold data integrity as an ongoing commitment.

Before talking about assessing and improving data quality, we should make sure the train our staff on data quality, the cost of bad data, how to maintain good data quality, and the universal data quality standards. Also, we should make sure that data validation is implemented on the application level to minimize as many as possible human errors at the data entry-level... In brief, we should develop a data quality culture in our organizations.

People and culture are most impactful aspects in data governance too. Getting a commitment (not just buy-in) from senior leadership in tech and business for your data governance teams is crucial than setting up processes and DQ monitoring and tracking systems. If you get committed tech and business folks, you can proceed even despite sophisticated tools, but it fails if it’s the case exactly opposite. No tool can help you get everyone onboard to fix DQs!

Beyond identifying, analyzing, resolving, and monitoring issues, it's important to foster a data-quality-focused culture within the organization. This includes training staff on the importance of data quality, updating policies and procedures to prevent future issues, and continuously adapting to changing data landscapes. Also, consider leveraging advanced tools and technologies for better data management. Remember, a holistic approach is crucial for long-term data quality success.

Data Quality issues can be resolved by tracing through the data lineage to identify where the quality of the data is getting detoriated. Thereafter, solution feasibility analysis should be performed, solution finalised, prioritised and ultimately implemented with timelines. However, it should not end there. The team should implement DQ checks in the root cause lineage step to avoid the issue again. All Data Quality issues should be documented throughout it's lifecycle to aid in learning.