How do you evaluate the impact of data cleaning on your data analysis and modeling outcomes?

It is key not to introduce additional bias when performing data cleaning tasks. Often, data scientists can take arbitrary/subjective decisions to make ML modeling easier, but without consideration of domain-related issues. For instance, removing categorical variables often helps data scientists come up with an "MVP" (minimum viable product) to establish whether a task is feasible or not. However, removing such categorical variables can introduce bias, hiding key information to the model. This applies to many other applications, e.g. outlier removal, subsetting data, etc. and should be kept in mind when performing data cleaning.

Sensitivity Analysis is used to evaluate the robustness of your results to different data cleaning approaches or parameter settings. Apply alternative cleaning methods or adjust cleaning thresholds to observe if the conclusions drawn from the analysis remain consistent or vary significantly. Sensitivity indices, such as the Sobol index or Morris method, are often used in sensitivity analysis to quantify the relative importance of input variables in contributing to the output variance. These indices help identify the most influential factors in your analysis or model and determine how changes in those variables impact the output.

In finance, a team embarked on a data odyssey to create a precise credit scoring model, aiming to minimize defaults. Setting metrics of accuracy and F1-score, they cleansed the dataset with precision—addressing outliers and inconsistencies. The result? A refined dataset sculpted into a financial masterpiece. The credit scoring model, now calibrated with pristine data, achieved remarkable accuracy and a balanced F1-score. The outcome: a finely tuned lending process, minimizing defaults and fostering financial stability.

Business, Business, Business. It's a word I keep repeating for myself and the others. We were hired to solve business problems, so our goals and metrics should align with the situation. The defining should already happen since the data collection activity, so when we select the features for data cleaning, we already have the necessary dataset. We also need to select a suitable metric based on the business goal. For example, we want to have a model for predicting potential leads. The business goal is to have as many customer conversions with minimum misprediction as it would cost the company. So, Precision might be the best metric as we want to minimize the False Positive.

If the data comes from multiple sources, it would be good practice to have subgroup analysis of the data. This will help to see if the data quality fairs well for a specific group then as a whole. This might give us the picture on the bias. Generally, the goal is to do the best to reduce bias in the data so that any prediction model developed from the data will be generalizable.

Evaluate the relevance of each variable or attribute in your dataset for your specific analysis or modeling goals. Remove any unnecessary or redundant variables to streamline your analysis and improve the interpretability of results.

For me; visualizing pieces of features of the data (even if it’s just aggregates and counts for different buckets or segmentations), many different ways is a great first step to ensuring you’re not removing important features from the data you’re “cleaning” for analysis.

It is crucial to understand the data you're working with. There might not be a standard approach in terms of handling and manipulating data, butut analyzing the data and obtaining visualizations, would help in coming up with approaches to handle inconsistencies and tackle issues for processing quality information for desired business purposes.

It is crucial to document and communicate the data cleaning process, including the techniques applied, decisions made, and the rationale behind them. This documentation ensures transparency, reproducibility, and allows others to understand the data cleaning steps taken when interpreting the analysis or replicating the results.

Evaluating the impact of data cleaning on analysis and modeling is crucial for deriving reliable insights. Real-world examples showcase the transformative effects of effective data cleaning practices, such as addressing missing values for accurate sentiment analysis, managing outliers for robust risk assessments, resolving duplicates for precise marketing attribution, standardizing variables for meaningful comparisons, and handling errors and anomalies for reliable revenue forecasts. These stories highlight how meticulous data cleaning enhances model performance, uncovers hidden patterns, and contributes to more informed decision-making in various industries.

Sort of a no brained, but often a step I find myself just spot-checking rather than fully testing. Double and triple check that the operations your performing on the raw data don’t eliminate your ability to perform what your initial intention was (eg; if you’re cleaning timestamps into dates, did you take into account the timezone of the time stamp, and convert correctly).

When you have clean data, you can make decisions using the highest-quality information and eventually boost productivity. You can use different techniques and tools to assess the model's performance and iteratively tune the models until the desired performance is achieved.

If you have a separate validation dataset, use it to compare the model performance before and after data cleaning. This will provide an independent evaluation of the impact of data cleaning on your models.

Ideally you always work with as close to raw data as possible, and have a layer of explainable processes ontop of the raw data to get it to answer the questions you need. You don’t want to accidentally use a faulty dataset only to realize you could have just cleaned the raw (source of truth) to your needs!

Perform a cost-benefit analysis to determine if the effort and cost associated with data cleaning were worthwhile. Compare the potential gains in data quality, model performance, and decision-making capabilities against the costs incurred in terms of time, resources, and potential delays in analysis and modeling timelines.