What are the most common mistakes in model selection for an applied ML project?

In the perilous terrain of model selection for applied ML, overfitting and underfitting are treacherous foes. Overfitting, the mirage of perfection, molds a model too intricately to training data, stumbling in the real world. Underfitting, a hasty shortcut, oversimplifies, failing to grasp nuances. Tread wisely; the Goldilocks balance is where true insights lie.

?? Prioritize precision in model selection! I'd underscore that mastering model selection is pivotal for effective machine learning outcomes. Falling prey to overfitting or underfitting can distort your model's applicability, so it's imperative to meticulously harness regularization techniques and fine-tune hyperparameters. Beware of data leakage; ensuring rigorous data segregation and implementing robust cross-validation techniques can be instrumental. Recognizing inherent assumptions and constraints of algorithms is equally vital, ensuring model appropriateness. Lastly, resist a tunnel-vision approach. Experiment with diverse algorithms, perpetually challenging your choices to find the optimal solution.

One cause of underfitting is when we specify a model that does not resemble the 'true' data - generating process. A simple example of this is trying to fit a linear regression to data where in fact the response does not have a relationship of such kind with the predictors (or with transformations of these). While we often do not know the data generating process, it's always good to first explore the type of associations among our variables when choosing a model.

Every model error consists of reducible (bias & variance) and irreducible error. Balancing bias with variance is the key trade-off, related to reducible error of your Machine Learning model Overfitting: - when the model is too flexible, follows training data, even its noise - Great on training data but poor on test data Underfitting: - Occurs with a rigid model, misses out on underlying data patterns - Mediocre performance on all data Ways to solve: - Cross-Validation: Test the model on various data subsets (K-fold CV, LOOCV..) - Regularization: Techniques that prevent model from becoming too flexible (Ridge L2, Lasso L1, Drop-Out) - Feature Selection: select most relevant features (Forward or Backward Selection) - Pruning: For trees

you're an AI system responsible for identifying potential terrorists at an airport. Without any prior data to characterize terrorist behavior, what's your approach? One option is to label every incoming passenger as "non-terrorist." This method may yield over 99.9% accuracy or even higher. However, while the model is highly accurate, it fails to serve its intended purpose. predicting the weight of a new student from a class of 100. You randomly sample 80 students, calculate their avg weight, and repeat this process 100 times (epochs). This is an example of overfitting your model. In both cases, it's essential to strike a balance between the model's performance and the underlying business logic.

To avoid the data leakage problem, you should follow some suggestions. First of all, you need to split your data into two or even three parts (training data, validation data, and test data). These parts should be independent and consistent. Next, you must thoroughly understand your data and context to avoid any inappropriate correlations between your inputs and your target. You should also consider issues related to future inputs. For example, if you want to predict the opening price of a stock, you cannot include the closing price of the same day in your features, as the closing price is influenced by the opening price, which is your target.

One important thing to consider is the problem of confounders. The leakage might cine from the fact that an unseen variable is affecting both the target and some of the features of the models (or viceversa). We need to understand our data and perhaps understand the context where our data lives.

Data leakage could also occur through improper feature engineering! Creating features which rely on information from the target variable or future knowledge can lead to leakage; with an understanding of what information is available at each stage of the data, is crucial to prevent this. In context of time-series data, understanding the temporal order & ensuring that future data isn't used to predict past data is critical; failing to account for temporal dependencies could introduce data leakage. Also, in data preprocessing, such as imputation, scaling, or transformation, should be conducted separately for training and test datasets to avoid leakage.

Following the hype!! Some algorithms become trending at a certain point of time but that does not mean they fit all the different applications. A researcher/ data scientist is responsible for experimentation and making informed decision about which model to use and why in the case study they are working on.

First of all I would say to know the process of data collection and to know where the data comes from, if it is first hand and quality data. Then understand this data and know what you want to apply it for, without understanding the data you will not know what model to apply.

Every ML model carries inherent assumptions. For instance: - Linear Regression requires linearity between variables. Ignoring this can yield spurious results. - Logistic Regression expects linear log odds with predictors. Overlooking this risks misclassification. - K-means Clustering assumes spherical, similarly sized clusters. Neglecting this may misrepresent actual data groupings. - DBSCAN focuses on data density; non-adherence can lead to fragmented clusters. Violating these assumptions can cause models to overfit (too specific) or underfit (too general). Ensuring these principles align with data is crucial for reliable predictions, especially in critical sectors like healthcare or finance.

In the realm of model selection, a tale unfolds at my workplace. It's a story of the pitfalls of ignoring assumptions and limitations. Each algorithm and model possesses its unique strengths and weaknesses, demanding specific data conditions. Some insist on linearity, normal distribution, or independence, while others are vigilant against outliers, noise, or multicollinearity. They each have their purpose - be it classification, regression, or clustering. To navigate this, we embraced the theory and logic behind each model, ensuring our data met the prerequisites or underwent the necessary transformations and preprocessing.

Knowing when to stop exploration takes a lot of experience - I agree that examining only one model or metric is too narrow, but when can you be certain you’ve explored enough? One useful path is testing on extensively benchmarked data. If you have a well known problem set (such as image classification), there exists several well researched datasets that have benchmark accuracies that you can measure your methods against (such as MNIST). If your accuracies are sufficiently close to top scores, you can feel confident that your methodology is sound - now all that is left is to tune your hyperparameters through grid search (but don’t forget to leave a holdout for final evaluation)!

??To select the optimal algorithm for an applied ML project, consider more than just performance. ??Training Time: XGBoost, while high-performing, often requires longer training and tuning. ?Serving Time: RandomForest can be slower during inference because it aggregates predictions from many trees. ??Interpretability: Linear models and decision trees offer higher interpretability for transparent insights. Deep learning models are typically less interpretable. ??Resource Requirements: Deep learning models demand substantial GPU resources during training, while simpler models like linear regression are more resource-efficient. ????♀?In summary, a comprehensive algorithm choice balances these factors for successful real-world application

Failing to explore different model options can limit your project's success. Avoid the trap of sticking with the familiar or first satisfactory model. Conduct research and experiments to evaluate various algorithms, comparing them using appropriate metrics for your specific problem.

In my experience, it is easy to forget the big picture when the entire model development process is not domain oriented. At every stage of the model development process, constant checks must be put in place to evaluate the steps and their relevance to solving the specific problem in the domain. This is why a data science manager or in his/her absence the team lead must always guide the engineers at every phase of the project. Engineers can get lost in the technical details of the project but it is the job of the manager to make sure the project continues to be relevant to the specific business problem it set out to solve

This is a dangerous trap to fall into! Something I like to refer to as "Machine learning ADHD" feels real. Given the beefy hardware and capable models available, it is easy to become lost in the exploration of different options and drown in technical details. It is not difficult to waste weeks of valuable engineering time if you are not cautious. Another example is over-future-proofing a solution before it can solve the current problem. When the model is finally future-proofed, the problem that you were going to solve has either been solved by other solutions or has changed in such a way that the solution is no longer useful. A quick minimal working ML solution is far superior to wasting time on things that may be useful later!

Losing sight of your project's main goal is detrimental. Focusing solely on technical aspects can lead to models that don't address stakeholders' needs or provide actionable insights. Continuously align your work with the project's context and goals, and ensure you communicate effectively with relevant parties.

The critical question you should ask yourself is "What now?". Nobody trains and picks ML models to keep them on a hard drive. They are built to be deployed, used, posing additional constraints. The 'bad' reflex of ML engineers is to focus on getting the best performance possible, but that often requires using complex models and non-conventional implementations. If your model needs to be deployed on a platform that doesn't support these specificities, it's a waste of time even if they help increase the performance. Remember Ockham's razor and what really matters: a model that has 100% accuracy but cannot be deployed/integrated will always be useless against a model that has 90% accuracy but is easy to deploy.

Model Interpretability : Choosing a model that is too complex to provide interpretable insights, which can be problematic in certain applications. In some cases, Interpretability is more important than raw predictive power. Consider using simpler models when understanding model decisions is essential.

Keep in mind, when creating a new version of a model that its labels will be taken from the previous version and this introduces bias into the new model when the model's decisions affect labeling; you may be training the new model on mistakes made by the older one. I'd probably draw a random sample with a human in the loop for this.