Why Experiment Tracking is Crucial to OpenAI

Carey did a great interview with Peter at OpenAI about his work and his team uses Weights and Biases to track their machine learning experiments. Here's a great excerpt:

Before we started using Weights & Biases, everyone had their own little setup of how they would get results. Some people were using Tensorboard, some people would be using their own homebrew version of some visualization tool. Everything was very fragile. If I wanted to share results, the best I could usually hope for was a screenshot of my graph pasted in an email. Whenever they would need to get something more, more often than not I’d have to rerun the experiment. So now we have a central place to have all of that information, and it's very easy for anybody to access that transparently and compare against each other's results. 

My colleague Lillian, for example, can take whatever she has trained and compare that with what I trained, create a quick report, and I can download the model she trained. I don't have to ask her where it is— I can go in and look at other metrics very easily since I have all the raw data. It's reduced a lot of the overhead in communication to make us focused on what matters.

Comparing results in general it's much faster when you have all the data in one place. So we do this a lot in our workflows comparing against old baselines and so on. So we can keep on having old runs available and compare against us over and over and over again.

More at https://www.wandb.com/blog/why-experiment-tracking-is-crucial-to-openai