How do you fix data cleaning errors in your code?

One thing I would like to add is: Prior to data cleaning, the important consideration is to identify the problem we are trying to solve. The next step is to read the technical documentation for the datasets and select the appropriate procedure. If our objective is to develop predictive modeling, the approach mentioned should be fine. However, cleaning up the training and validation sets would be a more appropriate approach. On the other hand, if our objective is different, such as undertaking structural equation modeling, we would need to develop a theoretical framework at first.

-One thing I would like adding here is getting the domain knowledge before performing EDA and finding anomalies in data. Getting domain knowledge will help what features we should focus on.

In addition to identifying the errors, the information must be categorized into cardinal, ordinal, or numerical (scaled) data. There is many techniques, but if something such as zero imputation is used, it will throw off of your statistics. There is a MICE (Multiple Imputation by Chained Equation) method which is used for many types of periodic or numerical data sets. Removing all feature duplicate records may work, and I would likely use it, to balance the dataset.

Once you’ve identified data cleaning errors, you can take several steps to fix them: 1. Handle missing values: Determine the reason behind missing values and decide on the appropriate approach to handle them. You can either remove the rows or columns with missing values, impute them using statistical methods (mean, median, mode), or use more advanced techniques like interpolation or machine learning algorithms. 2. Remove duplicates: Check for duplicate records in your dataset and remove them to avoid duplication bias. Deduplication can be based on specific columns or the entire dataset, depending on your requirements.

I could also keep the raw data ALWAYS, do not ever delete your raw tabular data. Sometimes you will want to compare raw data outputs with cleaned data outputs as well.

Fixing data cleaning errors involves applying appropriate fixes like imputing/deleting missing values, removing/replacing outliers, dropping/merging duplicates, standardizing/transforming data types/values, and resolving/reporting inconsistencies. Libraries like pandas, numpy, scipy, sklearn (Python) or dplyr, tidyr, stringr, lubridate (R) can help. Document code changes and use functions like dropna(), drop_duplicates(), and str_replace(). Validate fixes and update code regularly for data quality.

I realise that data validation often comes from understanding the business domain first. While checking missing data (or other data issues) is important for data validation, it's through the understanding of business domain that we can see why the data looks the way it does and what's wrong with that.

Validating data cleaning results is essential. Test code logic with unit tests, assertions, and edge cases. Compare cleaned data with original/expected results for consistency. Use visualizations and statistics to verify data quality. Analyze distributions and assess alignment with objectives. Iterate if issues arise. Ensure reliable and accurate cleaned data for further analysis.

Some data pipelines such as geospatial data, are very difficult to automate, and may require more time consumption to automate the curation than using a UI. Also a keen human eye needs to be used to make sure blanket data procedures is not a one shoe fits all sizes problem.

Automating data cleaning is vital. Convert steps into scripts/functions, saving time and ensuring consistency. Build pipelines with tools like airflow, luigi, or prefect (Python) or drake or targets (R) for orchestration and monitoring. Implement version control for collaboration and reproducibility. Schedule and monitor pipelines, handling data updates and validating quality. Document workflows for knowledge sharing and efficiency. Automating data cleaning frees up time for analysis and ensures accurate and reproducible results.

One of the best practices to improve the process of data cleaning is to have a good logging system. Without a good logging system to keep track of error traces, it's often hard to know what went wrong when we have errors in our code. In short, traceability leads to code improvement.

Consistent Logging and Documentation: Ensure you have consistent and clear logs of the data cleaning that has taken place. This will help you understand the exact steps that have been taken, enabling you to identify where a problem may have occurred if something does go wrong. Back-Up Your Data: Before starting any cleaning process, it's essential to make a backup of your raw data. In the event something goes wrong during cleaning, you have the original data to fall back on.