How do you test your data preprocessing assumptions?

1 Understand your data

The first step is to understand your data, its characteristics, and its distribution. You can use descriptive statistics, such as mean, median, standard deviation, and quartiles, to summarize your numerical data. You can also use visualizations, such as histograms, boxplots, and scatterplots, to explore your data and identify any outliers, skewness, or patterns. For categorical data, you can use frequency tables, bar charts, and pie charts to show the distribution of your classes and labels. You can also check for any missing values, duplicates, or errors in your data.

2 Define your assumptions

The next step is to define your assumptions about your data and how you want to preprocess it. For example, you may assume that your data is normally distributed, linearly related, or independent. You may also assume that you need to scale, normalize, encode, or transform your data in some way. You should write down your assumptions and justify them based on your data understanding and your machine learning goal.

3 Test your assumptions

The third step is to test your assumptions using statistical tests, metrics, or experiments. For example, you can use tests such as Shapiro-Wilk, Kolmogorov-Smirnov, or Chi-square to check if your data follows a certain distribution or has a certain relationship. You can also use metrics such as correlation, variance inflation factor, or mutual information to measure the strength and direction of the association between your variables. You can also use experiments such as cross-validation, grid search, or feature selection to compare the performance of different preprocessing methods on your models.

4 Evaluate your results

The fourth step is to evaluate your results and see if they support or reject your assumptions. You should interpret the output of your tests, metrics, or experiments and check if they are statistically significant, meaningful, or relevant. You should also look for any discrepancies, anomalies, or surprises in your results and try to explain them. You should also consider the limitations, assumptions, and trade-offs of your tests, metrics, or experiments and how they may affect your conclusions.

5 Refine your assumptions

The fifth step is to refine your assumptions based on your evaluation and feedback. You may need to modify, revise, or discard some of your assumptions if they are not supported by your results or if they are not suitable for your data or problem. You may also need to try different preprocessing methods, parameters, or combinations to improve your results or address any issues. You should document your changes and reasons and repeat the testing and evaluation process until you are satisfied with your preprocessing assumptions.

6 Apply your preprocessing

The final step is to apply your preprocessing to your data and prepare it for your machine learning models. You should use the same preprocessing methods and parameters that you tested and evaluated on your training data and apply them to your validation and test data. You should also check that your preprocessing does not introduce any errors, biases, or leaks in your data. You should also monitor and evaluate your models' performance and see if they meet your expectations and objectives.

7 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?