How do you manage data errors?

Any tasks, processes, or standards without the final goals in mind would only serve as business waste. Thus, the first question you should ask is what is your goal with your data? if you are only using a small batch of data that is driven from other data to create a useful data visualization dashboard or BI. Then, creating a cross-platform data quality standard make no sense. The sensible action would be to make sure the output of the ETL process complies with data visualization requirements. Not all companies can afford a dynamic big data system and not companies change their data usage goal all the time.

Data quality is a crucial factor for any organization that wants to leverage its data for better decision making, innovation, and performance. By understanding the causes and solutions of data quality issues, you can take proactive steps to ensure that your data is accurate, complete, consistent, relevant, and timely.

In the humanitarian and motivational posts I share on LinkedIn, data accuracy remains non-negotiable. For humanitarian causes, accurate data is vital for building trust and securing support. In motivational posts, I back up my advice with reliable facts and figures to inspire confidence in the youth. If data errors arise, I rectify them promptly to maintain credibility in both realms

Establishing clear data quality standards is fundamental for effective data error management. These standards serve as a blueprint for evaluating data integrity, ensuring it meets the expectations for informed decision-making. Define criteria like completeness, consistency, validity, timeliness, and accuracy to ensure data is complete, uniform, accurate, up-to-date, and error-free. Set data quality thresholds and targets to establish acceptable levels of errors, creating a benchmark for assessment. By defining these standards, you create a common understanding among stakeholders and users regarding data quality requirements, ensuring everyone is on the same page for data integrity.

It's important to regularly clean your data by developing procedures that remove duplicate records, standardize formats, and validate data against trusted sources. Make sure to train your team members involved in data management on the significance of accurate data and how to avoid common errors. Additionally, it's crucial to raise awareness among employees about the potential impact of data errors on business operations.

Validation is one of the most important steps in data analytics, especially in a highly regulated industry. Visualizations and reports should be validated with the raw data used to generate them to ensure accuracy. Documentation of validation and validation strategies are cor to data analytics and data management.

Exploratory Data Analytics still stands as one of the best ways for validating the data for errors- skewness check, catching the outliers, and locating missing values. Validating for errors can be leveraged with exploration via visualizations, but the bias in the data can be caught with democratizing the data. Make complete data transparency of data/source for the models being built, so the end-user can validate if there were any bias involved.

An example to understand this better: Let's say a company has an online registration form for new customers. This form requires users to enter their email addresses. Without proper data validation, users might accidentally or intentionally enter incorrect or fake email addresses, which could lead to inaccurate Insights.By implementing data validation, the company can ensure that only valid email addresses are accepted during the registration process. Thus, data validation helps maintain the overall quality and integrity of the company's data, which is essential for making informed decisions, providing quality services, and building trust with customers.

Data validation rules should be simple as much as possible. It is better to have many simple rules which are easy to understand, than few with complex calculations.

The data valuation rules for the drilling/Well Engineering database at my ex company were implemented two-fold firstly on the standalone front-end drilling reporting system running on offshore drilling rigs and secondly on the parent drilling/Well Engineering database running on back-end servers in the office environment onshore. The front-end offshore validation was implemented via the programming scripts inherent to the offshore standalone apps which validated the drilling data live. The on-shore office drilling data validation was implemented via a data tech tool which runs batch on the drilling data synchronized from offshore. The 2-fold data validation set-up improved data quality and was duplicated at many drilling sites globally

The current techniques of data management are as good as the data validation rules that are being defined. However, data science is a great and yet to be fully discovered in terms of attaining a good quality. Utilising data science to identify the erroneous data can also result in predicting risks that are unforeseen even when utilising risk based approach. Additionally, real time data error identification and early prediction of risks can mitigate issues that the conventional data management is not able to provide.

In my experience, while we are defining the standards and validating them through various frameworks, there is an essential part which is the audit and review that ensures the standards are well defined and enforced correctly. In addition, the audit can reveals any gap at the defined process and ensure it’s compliance.

While wells were being drilled offshore and operational drilling reporting on-going on the offshore rig PCs, IQM reports were used to monitor data quality of the drilling reports at the data center onshore synchronized from offshore. The offshore and onshore database set-up was warranted by poor offshore network At a higher level of data management, we used data visualization tool such as Power BI to generate charts periodically from the central onshore database to check for data completeness of all global wells drilled grouped under separate global regions (Asia Pacific, Europe, America etc….).

I agree that setting data quality standards is vital toward data quality, but in my experience as a data analyst, DQ standards is a part of a whole picture called data quality framework. So to handle data errors, means, establishing a comprehensive, detailed and procedural data quality plan, in which it will the base stones to build a company DQ Framework.

Data quality monitoring dashboards must be accessible not only for data consumers, but for those who can have impact on the business processes (even if they don't have Data Owner or Data Steward position and responsibilities)

How do you manage data errors? Is the wrong question. How do you manage data quality? is the right question. Most cost effective way to manage errors is to prevent them from entering the clinical database and / or containing error sources. How do you do that? See the paper I published with the DIA titled: "Dear CRF, Please Meet the Trial Protocol; Intelligent Electronic Data Capture (iEDC) with Expert System Redefines Clinical Data Collection". Other references, my articles "EMR-iEDC without ETL"; 'Hierarchical Integration - DICE"; "The Era of EDC is Coming to an End" To advance clinical data quality you need to employ this new science of ERROR RESEARCH handled by Clinical Data Scientists. QUALITY is a science based in Stats, OR and DM.

The first step in quality data is cleaning data and making everything standard so that errors are more easily picked up and can easily be spotted. Reading through your data helps make sure your data is correct. Having an eye for detail is vital while working with large data dumps. Always work with a copy of your data dump, so that way, your original data is always available in case you make a major error. Data Visuals are also helpful in displaying discrepancies in some cases.

Data visualization charts were used to monitor the global drilling data population ratios to ensure completeness of drilling data capture. Where data completeness fell short due to errors, the data team used IQM reports to pin point them for correction by the appropriate drilling team. At the region where I worked, a final data QA/QC was conducted between the data team and drillers to fix any remnant of error at the end of a drilling campaign for each well Data error prevention was handled by the scripts (mentioned in earlier section) developed for data validation rules. And for data prevention assurance at the onset, initial training was conducted to onboard new drilling recruits to report using data fields

In my experience it is a good practice to perform a thorough exploratory data analysis to prevent data issues that might arise in later phase. Examining the data for its accuracy and correctness in the initial phase makes our job easier to attain the goals( it can be either building a good model or having accuracy in data interpretation and BI reports). Noisy, faulty data degrades the quality of product that we aim to build and so data cleaning and preprocessing is of utmost importance.

It is hard or even impossible to achieve 100% data quality. So, you also need to measure the cost of changes in the business processes in order to prevent data errors. Sometimes, data errors are OK for data consumers.

An example I have seen of this is AI algorithms being used to generate reports. It is important to not only to present the suggestions but also the confidence of said suggestions. Also keeping track of the success and failure rates to ensure the algorithm's performance is improving over time

Our global central drilling data management (DM) team communicated the generic aspect of data errors via the scheduled CIN conference call to the regional data management teams. At the regional level, the DM team would use its own IQM reports and dashboard to communicate the data errors with the drillers who made the drilling reports. The data errors can be specific to each region due to different well type at different global regions. Mode of communication at the regions can be virtual or face-to-face in the same office, and in a bigger group such as refresher user training. Last but not least, the Well Engineering data population ratios for all wells were communicated to the Wells management (stakeholders) as well

It is very important to be transparent for data consumers. Data team must be the first tribe in the company who signals about data errors. This is the key point to make data be trusted

Effective data management that is grounded in transparency and integrity is destined to succeed. Communication is key; analysts and executives must be informed and coordinated on data quality to create problem-solving strategies. By using tools like reports, dashboards, and alerts, we have a real-time response system to act swiftly. Meticulous error documentation is critical both for fostering a culture that values data integrity and for complying with audits and regulations. This comprehensive approach not only addresses current issues but also minimizes future risks, serving as the backbone of any data management strategy.

Communication is a crucial point to reduce as much as one can the errors. I believe that the proper communication with colleagues is the best way to make progress. And of course, everyone should keep tracks of what the others have learned. This happens only with the correct documentation on the matter of interest.

For the users (different Well Engineers in this case), they can learn and become proficient at using the company’s reporting tool to capture Well Engineering (WE) data which can be used for generating the well schematic diagrams and as documentation for maintaining existing wells and drilling future wells. For the data teams, they can learn and improve on their DM skills, work with the Well Engineers to enhance the WE data quality which is important as the cost per footage of pipes and mechanical jewelry are usually high, and gain insights of any shortcoming of the data tools that would allow them potentially to provide information for the systems developers to improve on existing tools or construct future better ones

A robust data managent system is one that is continually improving and is also self learning. Improvement of data quality is as important as the data itself. Effort has to be invested into ensuring data errors are continually identified, corrected and the outcome used to update the data quality documentation. Confidence and conformity should always be increased. A benchmark is also required in order to be able to ascertain how well data is performing. A good system of benchmarking is key to a good data management system.

You need to have visibility into your data quality first. There are plenty of tools that will provide insights into your data quality. Use them.

To ensure high data quality, it is important to have a clear understanding of the end product and analytics to be developed. Starting from there, we can work backwards to determine the necessary data sourcing and transformation processes. The top priority should be error prevention, and therefore, audits should be implemented throughout the sourcing and transformation stages.

Data Quality initiative must be supported by C-level staff, otherwise all generated recommendations will be ignored by specialist in the business units

Understanding the data and it’s purpose People need training and further explanation to understand what data we are looking for exactly, the purpose for the data capture and why it’s necessary taken them. They should be able to explain to the clients they are seeking the information from. Data gathering may look simple but one the people don’t know why we need such data, we get wrong data for decision making.

The source of data and the nature of the sources itself can give fair idea about the domain of values in the data. But no sources are 100% clean or predictable. So data controls framework with extensive Data quality rules implemented close to source would help check data errors.