登录查看更多内容

What are the best practices for randomization and replication in ML experiments?

由人工智能和领英社区提供技术支持

If you are working on a machine learning project, you might wonder how to design and evaluate your experiments in a rigorous and reliable way. Randomization and replication are two key principles that can help you achieve this goal. In this article, you will learn what they are, why they are important, and how to apply them in your ML experiments.

此文章中的业界达人

由社区从 8 条内容中精选。了解更多

Iain Brown Ph.D.

Head of Data Science | Adjunct Professor | Author
Shahrzad Bahari

CTO ??AI venture builder ??Social Impact Hacker ?? Hands-on Startup CTO ????
Murtuza Dahodwala

Data Scientist II at Aramex International | Data Science Trainer | Mentor | SME | Consultant | Upgrad | BITS Pilani

1 What is randomization?

Randomization is the process of assigning your experimental units, such as data samples, models, or parameters, to different treatment groups or conditions, based on chance. This helps to reduce the bias and confounding factors that might affect your results, such as the order of data, the initial state of models, or the selection of hyperparameters. Randomization also enables you to estimate the variability and uncertainty of your outcomes, and to perform statistical tests and inference.

添加您的观点

Iain Brown Ph.D.

Head of Data Science | Adjunct Professor | Author
举报内容
Having worked with diverse data sets and ML models, I've seen the risks of neglecting randomization. Often, real-world datasets have biases. Consider customer data from an e-commerce site sorted by sign-up date. Without randomization, splitting this data sequentially for training and testing could make our model excel with older customers but fail with newer ones. This order introduces bias. Additionally, in hyperparameter tuning, random searches often surpass grid searches due to diverse parameter exploration. Thus, randomization isn't merely theoretical but a practical tool ensuring robust, generalizable models.

已翻译

赞
Shahrzad Bahari

CTO ??AI venture builder ??Social Impact Hacker ?? Hands-on Startup CTO ????
举报内容
As a CTO, I've learned firsthand the critical importance of proper randomization and replication. Randomly sampling training data is crucial to avoid sampling bias and overfitting. I always set random seeds for reproducibility across experiments. Running multiple trials with different seeds accounts for variability. Cross-validation techniques like k-fold are essential. The key is removing bias and noise to ensure robust models that generalize. Documenting everything enables sharing knowledge with the community. After a decade developing ML systems, these best practices are ingrained in my workflow. They lead to trustworthy models that stand the test of time.

已翻译

赞
Murtuza Dahodwala

Data Scientist II at Aramex International | Data Science Trainer | Mentor | SME | Consultant | Upgrad | BITS Pilani
举报内容
Data shuffling in ML is like carefully mixing ingredients in cooking. Just as you blend various elements in a recipe to create a harmonious dish, shuffling data mixes diverse samples in a dataset, preventing any specific order or bias from affecting your model's learning. It's the ML chef's way of ensuring a fair and unbiased training and testing process, just like a chef ensuring every bite of their dish tastes consistently delicious by mixing and layering flavors.

已翻译

赞
Moiz Batliwala

Vice President - Bank Analytics and Solutions | Credit Risk Underwriting | Customer Management | Collections Management | Cross-sell | PFM | Scorecard Development | Income Estimation
举报内容
Random Forest model will ensure randomization with each of its model at feature as well as sample level. Thus is evident from the fact that everytime we run the model the outcome will slightly change given the random change in underlying data.

已翻译

赞
Dr Saurabh Sawhney

Curious citizen
举报内容
Good randomization is perhaps the most under-rated way to achieve good generalization of models. An ensemble of models using the same algorithm but randomized in different ways and at different levels is likely to be far more robust than a single, well-tuned model.

已翻译

赞

2 What is replication?

Replication is the process of repeating your experiment multiple times, under the same or similar conditions, to verify and generalize your results. This helps to increase the precision and accuracy of your estimates, and to reduce the noise and variability that might arise from random factors, such as data sampling, model initialization, or optimization algorithms. Replication also allows you to compare and contrast your results across different settings, such as data splits, model architectures, or evaluation metrics.

添加您的观点

Murtuza Dahodwala

Data Scientist II at Aramex International | Data Science Trainer | Mentor | SME | Consultant | Upgrad | BITS Pilani
举报内容
Replication in ML experiments is akin to rehearsing a recipe. Just as a chef tests a dish multiple times to ensure it consistently turns out well, replicating experiments in ML involves running the same tests or models multiple times to confirm the reliability of your results. It's like practicing a culinary masterpiece several times to guarantee it's not a one-time wonder, but a reliable and consistent outcome.

已翻译

赞

3 How to randomize your data?

One of the most common ways to randomize your data is to shuffle or permute it before splitting it into training, validation, and test sets. This ensures that your data sets are representative and balanced, and that your models are not overfitting or underfitting to specific patterns or features. Another way to randomize your data is to use cross-validation, which is a technique that divides your data into k folds, and uses each fold as a test set once, while using the remaining folds as a training set. This helps to reduce the variance and bias of your model performance, and to evaluate it on different subsets of data.

添加您的观点

4 How to randomize your models?

One of the ways to randomize your models is to use different initialization methods or seeds for your model parameters, such as weights and biases. This ensures that your models are not influenced by the initial state of the parameters, and that they can converge to different local optima or solutions. Another way to randomize your models is to use different hyperparameters or configurations for your model architecture, such as learning rate, activation function, or regularization. This helps to explore the trade-offs and effects of different choices on your model performance, and to find the optimal or best settings for your problem.

添加您的观点

Murtuza Dahodwala

Data Scientist II at Aramex International | Data Science Trainer | Mentor | SME | Consultant | Upgrad | BITS Pilani
举报内容
As per my experience, if you're performing k-fold cross-validation, it's essential to shuffle the data before each fold to guarantee that every subset used for training and testing is a well-mixed representation of the entire dataset, similar to how a deck of cards is shuffled before each game to ensure fairness.

已翻译

赞
Mario Filho

Self-taught machine learning geek that somehow became a Kaggle Competitions Grandmaster
举报内容
The traditional way is setting a random seed (or random state for scikit-learn users), but this can fail in unexpected ways. For example, try setting the random state in a Random Forest and then shuffle your DataFrame column order... it will train a different model! The most reliable way I know is training a model with different seeds and averaging their predictions (works great for GBDT and Neural Nets) For RF, you can increase the number of trees as much as possible or even try some resampling (like training on 90% samples w/o replacement and averaging the results)

已翻译

赞

5 How to replicate your experiments?

One of the ways to replicate your experiments is to use different random seeds or numbers for your data shuffling, model initialization, or hyperparameter tuning. This ensures that your experiments are not dependent on a specific or fixed random state, and that they can capture the variability and uncertainty of your results. Another way to replicate your experiments is to use different data sources or modalities, such as images, text, or audio. This helps to test the robustness and generalization of your models, and to evaluate them on different domains or tasks.

添加您的观点

6 How to report your results?

One of the ways to report your results is to use descriptive statistics, such as mean, standard deviation, or confidence interval, to summarize and compare your results across different treatment groups or conditions. This helps to communicate the central tendency and dispersion of your results, and to indicate the significance and reliability of your differences. Another way to report your results is to use visualizations, such as plots, graphs, or tables, to illustrate and highlight your results in a clear and concise way. This helps to convey the patterns and trends of your results, and to support your conclusions and recommendations.

添加您的观点

7 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?

添加您的观点

Machine Learning

+ 关注

给文章评分

我们借助人工智能创建了此文章。您认为这篇文章怎么样？

很棒不太好

举报此文章

查看全部

What are the best practices for randomization and replication in ML experiments?

1

2

3

4

5

6

7

1 What is randomization?

2 What is replication?

3 How to randomize your data?

4 How to randomize your models?

5 How to replicate your experiments?

6 How to report your results?

7 Here’s what else to consider

Machine Learning

给文章评分

感谢您的反馈

更多Machine Learning相关文章

更多相关阅读内容

What are the best practices for randomization and replication in ML experiments?

1

2

3

4

5

6

7

1 What is randomization?

2 What is replication?

3 How to randomize your data?

4 How to randomize your models?

5 How to replicate your experiments?

6 How to report your results?

7 Here’s what else to consider

Machine Learning

给文章评分

感谢您的反馈

查看其他技能