Scaling Laws and Compute Optimal Training
Scaling laws are empirical rules that relate model loss to compute, parameter count, and data size. They matter because frontier labs spend enormous budgets, and even small allocation mistakes can waste millions of dollars.
Primary sources:
The Basic Idea
Language model performance tends to improve predictably as:
- Parameters increase.
- Training tokens increase.
- Compute increases.
- Data quality improves.
But these resources trade off. A model can be:
- Too small for the compute budget.
- Too large but undertrained.
- Trained on too many low-quality tokens.
- Architecture-limited or optimizer-limited.
Kaplan-Style Scaling
The earlier scaling-law picture emphasized smooth power-law improvements with larger models, larger datasets, and more compute.
Important mental model:
loss ~= irreducible_loss + scale_terms
The exact fitted equations matter less than the discipline:
- Train many smaller models.
- Fit predictable curves.
- Extrapolate before spending frontier compute.
Chinchilla Correction
Chinchilla argued that many large language models were undertrained. For a fixed compute budget, the compute-optimal model should use fewer parameters and more training tokens than earlier practice suggested.
The practical lesson:
- Do not worship parameter count.
- Token budget and data quality are first-class.
- A smaller model trained on more tokens can beat a larger undertrained model.
Compute Allocation
Rough training compute for dense transformers is often estimated as proportional to:
parameters * training_tokens
That hides many details, but it is useful for first-pass reasoning.
When thinking like a researcher, ask:
- Is this architecture improving loss at fixed compute?
- Is it improving downstream ability at fixed compute?
- Is it improving inference cost at fixed quality?
- Is it changing the compute-optimal allocation?
Scaling Beyond Loss
Cross-entropy loss is not the whole story.
Frontier labs care about:
- Reasoning.
- Tool use.
- Code ability.
- Long-context reliability.
- Instruction following.
- Safety behavior.
- Multimodal ability.
- Serving cost.
Some capabilities appear gradually; some appear sharply on certain evaluations. Be careful with "emergence" claims because metric choice and scaling axis can create artifacts.
Data Quality And Repetition
Token count is not enough. Tokens differ in value.
Important dimensions:
- Deduplication.
- Domain mixture.
- Code/math density.
- Synthetic data quality.
- Contamination control.
- Curriculum.
- Multilingual coverage.
- Long-context documents.
This links to Data Tokenization and Pretraining Objective.
Research Questions
- How do scaling laws change with synthetic data?
- How do sparse MoE models scale compared with dense models at fixed active parameters?
- What scaling law predicts reasoning after post-training?
- How should compute be split between pretraining, midtraining, long-context adaptation, and post-training?
- Can small-scale experiments reliably predict frontier-scale behavior for a new architecture?
What To Practice
Run tiny scaling experiments:
- Train model sizes
S, M, L. - Train on token budgets
N, 2N, 4N. - Plot validation loss.
- Fit simple power laws.
- Compare loss to downstream task performance.
The habit matters more than the toy result.