How can you make your machine learning models more interpretable with data cleaning?

Imagine losing $15 million dollars due to poor quality of data; that’s exactly how much organizations lose yearly because of bad data according to a #Gartner research. I was working on a particular data, and then I realized that I was spending so much time on manual cleaning, chasing different people to fill missing gaps. I felt like crying ??, it can be really draining, I am sure you can relate. Many people want to build fantastic machine learning models or AI, but forget to pay attention to getting good data first, when in fact, the Insights, AI or Model they want to get is only as good as the quality of data it is built on. Data quality shapes the destiny of insights, Machine Learning Models and AI - it’s Garbage in, garbage out

Data cleaning plays an important role when it comes to visualising correct data. Imagine big tech giants losing millions of dollars because of having non standard data. Data cleaning means understanding the pattern of data, standardising the data and then visualising the clean data

To enhance ML model interpretability via data cleaning, perform feature engineering, handle outliers and missing data, normalize/scale, encode categorical variables, select relevant features, visualize data, preprocess, and leverage domain knowledge for better understanding. To enhance the interpretability of machine learning models through data cleaning: Engineer features thoughtfully. Detect and handle outliers. Address missing data. Normalize and scale features. Encode categorical variables. Select relevant features. Use data visualization for insights. Apply data preprocessing techniques. Incorporate domain knowledge. By performing these actions, you can improve the clarity and transparency of your machine learning models.

Data cleaning, also known as data cleansing or data scrubbing, is a critical step in the data preparation process. It involves identifying and correcting errors, inconsistencies, and inaccuracies in datasets. Data cleaning matters for several reasons: Data Quality: High-quality data is essential for making informed decisions, conducting accurate analyses, and producing reliable results. Dirty data, which contains errors and inconsistencies, can lead to incorrect conclusions and unreliable insights. Data Accuracy: Data cleaning helps ensure the accuracy of your data. Errors in data, such as typos, duplicate records, or missing values, can lead to inaccurate reporting and analysis.

Data cleaning can be compared to oil refining - just like you cannot simply put oil in your car tank, you cannot simply give raw data to a ML model and hope for the best. It's trivial, but true: garbage in = garbage out.

Data cleaning is the process of ensuring your dataset is accurate and reliable for analysis or machine learning. It involves tasks like handling missing values, removing duplicates, addressing outliers, converting data types, processing categorical data, scaling features, and more. Effective data cleaning enhances data quality, leading to more trustworthy results. You do not want to ruin a business by making a business decision based on wrong data.

Cleaning your data is a crucial step in the data preparation process to ensure its accuracy and reliability. Data cleaning involves identifying and rectifying errors, inconsistencies, and inaccuracies in your dataset. Below are few examples of data cleaning approaches: 1) Understand the data 2) Data Profiling 3) Handle missing data 4) Remove duplicate records 5) Standardize data types

Step 1. Understand your product and the goal of the ML model. Step 2. Understand the data sources that you have available Step 3. Select the best metrics to feed your model with, base on the knowledge you got on steps 1. and 2. Step 4. Analyse data completeness and quality of the metrics Step 5. Remove metrics that are highly incomplete (more than ~20% missing data) or unreliable (strange values of the variable, e.g. gender = 493.23) Step 6. Measure correlations between the variables, remove those that correlate extremely highly (more than ~0.9) Step 7. Run a manual analysis to prepare basic segments and new metrics that can be derived from the existing ones. Step 8. Try to use the data to create a simple version of the final model. Ready!

Definitely SHAP and LIME are state of the art and model agnostic ways for estimating interpretability of the your model. One additional easier and use case dependent approach can be using tree-based models, preferably ensemble models that also give feature importance and are itself quite interpretable

There are different ways model can be interpreted using recent statistical process such as SHAP , LIME etc I believe that using confidence interval for evaluating each variable dependence or impact in your model speaks the main effect of your variable contribution in your model. The confidence interval effects could portraits the changes and provide a deep- in explanation of each variable concept contributing into the model.

The digging into the fact and figure of the data reveals the potential possibility of improving your model. There are many statistical methods such as R2 , AIC, BIC etc. they explain the possibility of a great model. If its fit or not. But the main focus is making sure the data reveals the problem until then the model can solve the problem.

It's *extremely* easy to overspend on ML. Always keep in mind that an AI algorithm is a cherry on top of a great quality data and great analysis. Often, it's better to use a manually created model (they are 100% interpretable) than trying to run a complex AI project that will lead to a lot of confusion. Do not underestimate power of human mind when it comes to interpretability.