How do you test the accuracy of your clean data?

Data Quality criteria are like rules that help us to know if data is good or not. Good data is accurate, complete, doesn’t change too much, is up to date, and is useful for what we need.

Exploring data is like unwrapping a mystery - and I love the thrill of exploratory Data Analysis to get deeper insight into the data. I'm personally a fan of using Python's trusty trio: pandas, NumPy, and matplotlib. and for visualization Seaborn is my go-to. These tools are like loyal companions.

A good rule of thumb is, - Check the number of missing values - Check whether dropping those values would degrade the data quality - Check the statistical properties and plot the distribution of each feature/predictor to select a method for imputation for the cases where dropping the missing values is impractical. - Plot the correlation heatmap to check the correlated features.

While standard data cleaning techniques address the technical issues such as presence of special characters, spaces, missing data, human introduced artifacts such as using non-standardized vocabularies, wrong codes such as for institutions, procedures and diagnoses, and other domain specific artifacts and errors require expertise in specific fields to address those. Working with domain experts are possibly the only way to address them.

Imputation or removal decision generally depends on the impact of the removal and the characteristics of the feature. For example, time series data related to the stock market might not work well with an imputation of missing data. Similarly, the outliers may have a considerable impact on the model. The outliers caused by errors should be eliminated. Whereas removing other (genuine) outliers depends on the purpose of the study and the possible effect of the outliers on the study. Regarding feature engineering, sometimes humans might better grasp the data, which may help the models (e.g., adding a wealth category next to the wage data).

I think data validation for adherence with predefined business rules is as important as data accuracy testing. Feature engineering is especially relevant because feature abstraction, interaction, and selection decisions can vary depending on the business context.

Depending on the level of modifications applied to the individual columns, various analyses may be employed. For example, a basic histogram or distribution plotting may be enough to observe the effect of data imputation (especially with constant data). Additionally, a two-tailed student's-t test may be applicable to test whether the original and modified data have the same mean or variance. For cases that require a more in-depth analysis, one might employ statistical tests such as Kolmogorov-Smirnov to check whether the original and modified datasets belong to the same distribution.

Before diving into data cleaning, it's smart to validate your data first. Tools can help you spot errors or inconsistencies right away—consider this your first quality checkpoint. After cleaning, validate again to confirm you've met your quality criteria. Don't overlook specialized tools like Ray Dataset, Great Expectations and Pandera either; they can make data management easier with built-in validation features. Why does all this matter? - In the world of machine learning, missing these validation steps can lead to issues like model drift. By validating before and after cleaning, you're setting up safeguards for more reliable machine learning models.