Approaches for Selecting Statistical Hypothesis Tests in Model Selection for Machine Learning

Introduction:

Selecting the best model from multiple machine learning methods is a critical step in applied machine learning. However, comparing models solely based on mean skill scores obtained through resampling methods such as k-fold cross-validation can be misleading. It is challenging to determine whether the observed difference in skill scores is statistically significant or simply a result of chance.

To address this issue, statistical hypothesis tests can be employed to quantify the likelihood of observing the skill scores under the assumption that they are drawn from the same distribution. By rejecting the null hypothesis, we can infer that the difference in skill scores is statistically significant, enhancing our confidence in model selection.

The Importance of Statistical Hypothesis Tests in Model Selection:

Model selection aims to identify the model with the best performance on unseen data. However, evaluating model performance requires assessing the reliability of estimated skill scores. Statistical hypothesis tests provide a robust framework to determine whether the observed differences in skill scores are real or due to chance.

Understanding Statistical Hypothesis Tests:

Statistical hypothesis tests compare two samples and assess the likelihood of observing them under the assumption of the same distribution. By accepting or rejecting the null hypothesis, we can determine if the observed differences in model skill are statistically significant or a result of chance.

Two Possible Outcomes:

1. Insufficient evidence to reject the null hypothesis: If the statistical test indicates that there is insufficient evidence to reject the null hypothesis, it suggests that the difference in skill scores is likely due to chance.
2. Sufficient evidence to reject the null hypothesis: If the statistical test indicates sufficient evidence to reject the null hypothesis, it implies that the difference in skill scores is likely due to a genuine difference between the models.

Challenges in Choosing the Right Hypothesis Test:

Selecting an appropriate statistical hypothesis test for model selection can be challenging. It requires considering various factors, such as the chosen measure of model skill, the repeated estimation of skill scores, the distribution of estimates, and the summary statistic used to compare model skill.

Previous Findings and Recommendations:

Research in this field has identified potential issues with naive approaches and proposed alternative methods. Some key findings and recommendations include:

1. McNemar's test or 5×2 Cross-Validation: McNemar's test is recommended when limited data is available, and each algorithm can only be evaluated once. Additionally, 5×2 cross-validation, incorporating a modified paired Student's t-test, is suggested for situations where the algorithms are efficient enough to be run multiple times.
2. Refinements on 5×2 Cross-Validation: Researchers have proposed further refinements to the paired Student's t-test to account for the violation of the independence assumption in repeated k-fold cross-validation. These refinements aim to improve replicability and provide better correction for the dependence between estimated skill scores.

Recommendations for Model Selection:

While there is no one-size-fits-all approach for selecting a statistical hypothesis test for model selection, several options can be considered based on the specific requirements of the problem at hand:

1. Independent Data Samples: When sufficient data is available, gathering separate train and test datasets can provide truly independent skill scores for each model, allowing for the correct application of the paired Student's t-test.
2. Accept the Problems of 10-fold CV: Naive 10-fold cross-validation with an unmodified paired Student's t-test can be used when other options are not feasible. However, it is important to acknowledge the high type I error associated with this approach.
3. Use McNemar's Test or 5×2 CV: McNemar's test is suitable when algorithms can only be evaluated once, while 5×2 cross-validation with a modified paired Student's t-test is recommended when efficiency allows