What are the best practices for ensuring reproducibility in distributed training experiments?

1 Define a random seed

A random seed is a number that determines the sequence of random values generated by your code. Random values are often used for initializing weights, shuffling data, splitting data, and sampling data. If you use different random seeds for different runs of your experiment, you will get different results, even if everything else is the same. Therefore, you should define a fixed random seed for your experiment and make sure that it is the same across all the devices or nodes that participate in the distributed training. You can set the random seed for different frameworks or libraries using the seed() function or a similar method.

2 Use deterministic algorithms

Another source of variability in your results is the use of non-deterministic algorithms, which are algorithms that can produce different outputs for the same inputs, depending on factors such as the order of execution, the hardware, or the environment. For example, some optimization algorithms, such as Adam or SGD, are non-deterministic because they rely on asynchronous updates or floating-point operations that can vary across devices or nodes. To avoid this, you should use deterministic algorithms, which are algorithms that always produce the same outputs for the same inputs, regardless of other factors. For example, you can use gradient descent with a fixed learning rate and a fixed batch size, or you can use a library that provides deterministic implementations of common algorithms, such as TensorFlow Determinism or PyTorch Reproducibility.

5 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?