Data, Parameters and Compute: The Delicate Balance in Model Training

In the quest to unlock the full potential of Large Language Models (LLMs), the industry has ventured into a labyrinth of parameters, where the numbers attached to LLMs—be it 1 billion, 70 billion, or even 175 billion—aren't merely a quantitative leap but a qualitative one.

The goal in pre-training large language models is minimizing loss when predicting tokens. Two ways to improve performance are increasing training data and model parameters. Though scaling both could help in theory, compute budget constraints feasibility. Compute budget refers to available GPUs and training time. Petaflop-days quantify compute needs. 1 petaflop-day runs 8 NVIDIA V100 GPUs fully for a day. More powerful GPUs like 2 A100s match 8 V100s. Compute budget sets hard limits on feasible data size and model size for training. While more data and larger models can improve performance, compute budget is the key constraint. Petaflop-days help quantify compute resources required for different model configurations. The compute budget determines the max feasible limits for training data size and model size.

In practice, available compute resources are a hard constraint, set by hardware, training time, and budget. With a fixed compute budget, the levers to improve model performance are training data size and parameters.

OpenAI researchers found increasing training data improves performance. Also, larger models reduce test loss. This raises the question - what is the ideal balance of model size, and training data for a given compute budget?

In a paper published in 2022, a group of researchers led by Jordan Hoffmann, Sebastian Borgeaud, and Arthur Mensch carried out a detailed study of the performance of language models of various sizes and quantities of training data.

Key takeaways from their Chinchilla paper:

Optimal Model Size

• For a compute budget, there is a size that maximizes performance
• The optimal parameters are ~20x the training data tokens
• Many models are over-parameterized with excess parameters. Meaning they have more parameters than needed.

Optimal Training Data

• More training data is better - model performance continues to improve as dataset size increases.
• The ideal data size is ~20x the number of parameters
• Models like GPT-3 (trained on 0.3 trillion tokens) were likely undertrained

Compute-Optimal Models Excel

• This indicates bigger is not always better - smaller models can match or exceed the performance of larger ones if trained optimally.
• Models trained in a compute-optimal manner outperformed larger models

In Summary, compute budget is the key constraint when developing large language models. The goal is minimizing loss when predicting tokens, and more data and larger models can improve performance. However, available compute resources like GPUs and training time impose hard limits.

Chinchilla paper provides a framework for training compute-optimal models - choosing the ideal model size and training data to maximize performance for a given compute budget. The takeaways are to use a training set around 20x the number of parameters, and choose a model size fitting your compute resources.

Properly optimizing for compute can allow smaller models like 50B parameter Bloomberg GPT to match or exceed larger over-parameterized models. Compute optimization is vital alongside scaling up data and model size. Following these insights allows practitioners to develop high-performing large language models while making the most of available compute budgets.

Disclaimer: The intent of this blog is to explain complex machine learning concepts clearly to a non-expert audience. The perspectives and opinions expressed are my own interpretations based on cited research papers. This post is meant for educational purposes to summarize key points from established studies and does not represent official guidance from any research group. The goal is to synthesize insights accessibly, not make definitive claims. All viewpoints are my own for the purpose of this explanatory piece. This post does not aim to undermine any model or research.