What metrics do you use to evaluate data cleaning?

Data completeness, a crucial aspect in any analytical endeavor, gauges the presence of missing values within your dataset and the adeptness with which they are managed. These voids hold the potential to skew analysis outcomes, diminish statistical robustness, and even propagate inaccuracies. It is imperative to meticulously assess the proportion of absent data entries and make judicious choices in their treatment. Approaches encompass elimination, imputation, or marking, contingent upon the data's nature and the contextual framework.

Indeed data completeness would help you have richer data for modelling. However, the end objective is not merely having a model with good performance, rather the one that mimics the real-world challenge and improvises on it. While data completeness could be one aspect of measuring data quality, care must be taken to consider the statistical representation of the data that is being captured

Completeness would also refer to the business definition and data quality terms. For example, some data is expected to have missing data because it is supposed to be missing. For example, there is a feature called "Car Ownership", then the next feature is "Car Owned Brand". In the row where no car ownership, the car owner brand feature would be missing as there are no cars to answer here. Of course, we can fill it with something such as "No Car", but remember that missing data itself is a part of the dataset.

There are two ways of looking at data consistency: -Business POV -Technical POV Standard values are usually already established from the business POV so that we can adhere to them. Research and establish this upfront so we don't waste too much time during the analysis. From Technical POV, it depends on the analysis we want to perform. For example, many clustering analysis need the numerical data to be in the same unit.

Data accuracy should also refer to the data definition, as statistical analysis might not be able to catch that. Outlier wasn't necessarily mean that it did not represent the data, In fact, it's expected to be there sometimes. For example, you have an income feature from your company. It's expected there would be outlier data because the higher position would have higher income. So, working on correcting something representing the data need to be done carefully. If you remove the outlier from the dataset, it might help the model but might not represent the actual population.

Knowing the data relevancy to the problem you want to solve is very important. Only some data are necessary for the analysis and the model. Often, we need to follow the business needs or feature selection technique to select the important features.

Timeliness isn't only about data recency but aligning the data with the actual business cycle or phenomena under study. Sometimes, using 'old' data is necessary to capture seasonality or long-term trends. Continuous evaluation of data freshness and relevance to the current problem is imperative.

Ensuring data integrity isn't just a technical challenge but a governance one as well. Clear data ownership, stringent access controls, and robust audit trails aren't only best practices but often legal requirements. Collaborating with legal, security, and IT teams is often as crucial as the data itself.

Data integrity is all about making sure your data is correct and trustworthy from start to finish. Think of it like building a house: you'd want solid bricks (data) from the start and ensure they stay in place (remain accurate) until the end. If there's a problem with the bricks or they shift out of place, the house (your analysis or model) might fall apart. So, it's crucial to keep an eye on your data, just like you'd watch over a house you're building. Some ways to ensure your data stays reliable include using trusted tools and software, setting rules about who can change the data, and regularly checking the data to catch any mistakes.

The intersection of business understanding, technological capability, and domain expertise is where effective data cleaning happens. It's often an iterative, collaborative process rather than a linear one. Embedding the process within the broader organisational context ensures that data cleaning adds value, not just correctness.