Frontier Model Case Studies
This note connects architecture ideas to public model reports. Treat these as case studies in research tradeoffs, not as a complete history.
Primary sources:
- GPT-3
- PaLM
- LLaMA
- GPT-4 Technical Report
- Llama 3 Herd of Models
- Mixtral of Experts
- DeepSeek-V3
- DeepSeek-R1
GPT-3
Why it mattered:
- Made few-shot prompting and in-context learning central.
- Showed broad capability scaling with model size and data.
- Shifted attention toward large decoder-only transformers.
What to study:
- Scaling behavior across model sizes.
- Prompting as an interface to base models.
- Limits of pretraining-only behavior.
Connection:
PaLM
Why it mattered:
- Large-scale dense model training with Pathways infrastructure.
- Strong results across language, reasoning, and code tasks.
- Useful systems case study.
What to study:
- Training infrastructure.
- Evaluation breadth.
- Emergent-ability framing and its controversies.
Connection:
LLaMA
Why it mattered:
- Showed strong open foundation models trained efficiently.
- Helped establish modern open LLM recipes.
- Reinforced the value of high-quality data and more tokens.
What to study:
- Efficient training.
- Token budget choices.
- Modern transformer ingredients.
Connection:
GPT-4 Technical Report
Why it mattered:
- Public report from a frontier lab, but limited architectural disclosure.
- Highlights evaluation, safety, and predictable scaling more than implementation detail.
What to study:
- What frontier labs disclose and omit.
- Evaluation and safety methodology.
- Scaling predictability claims.
Connection:
Llama 3 Herd
Why it mattered:
- Detailed modern open model report.
- Covers pretraining, post-training, safety, multimodal direction, and evaluation.
- Useful template for how to read a full model-family report.
What to study:
- Data and training recipe.
- Post-training pipeline.
- Evaluation suite.
- Deployment-minded model family choices.
Connection:
Mixtral
Why it mattered:
- Public sparse MoE model with strong performance for active parameter count.
- Shows practical top-k expert routing in an open setting.
What to study:
- Sparse versus dense comparison.
- Active parameters versus total parameters.
- Serving implications.
Connection:
DeepSeek-V3
Why it mattered:
- Modern high-performance MoE report.
- Useful for studying advanced sparse training, cost, and engineering choices.
What to study:
- Routing and expert design.
- Training stability.
- Cost reporting.
- Systems choices.
Connection:
DeepSeek-R1
Why it mattered:
- Public reasoning-focused training report.
- Shows how reinforcement learning can shape reasoning behavior after pretraining.
What to study:
- Reasoning RL.
- Distillation.
- Evaluation on math/code/reasoning.
- Difference between base capability and elicited behavior.
Connection:
Pattern Summary
| Model/report | Main lesson |
|---|---|
| GPT-3 | Scale unlocks broad few-shot behavior. |
| PaLM | Infrastructure and broad evaluation matter. |
| LLaMA | Efficient data/compute recipes can beat naive size. |
| GPT-4 report | Frontier disclosure emphasizes eval/safety over architecture details. |
| Llama 3 | Modern open model families require integrated pretraining/post-training/eval. |
| Mixtral | Sparse MoE can improve active-parameter efficiency. |
| DeepSeek-V3 | MoE plus systems engineering can produce strong cost/performance. |
| DeepSeek-R1 | Post-training RL can strongly shape reasoning behavior. |