Frontier Lab Notes

Paper Reading Ladder Frontier LLMs

This is the reading sequence for the frontier LLM architecture sphere. Prefer reading papers actively: reconstruct the mechanism, write down the bottleneck, and identify what experiment would falsify the claim.

Level 0: Orientation

  1. Attention Is All You Need

    • Why: establishes attention, multi-head attention, residual blocks, and sequence modeling without recurrence.
    • Pair with: Transformer Block Anatomy.

  2. Language Models are Few-Shot Learners

Level 1: Scaling

  1. Scaling Laws for Neural Language Models

    • Why: introduces the empirical scaling-law mindset.

  2. Training Compute-Optimal Large Language Models

    • Why: Chinchilla reframed compute-optimal allocation between parameters and tokens.

  3. PaLM: Scaling Language Modeling with Pathways

    • Why: useful example of large-scale training, infrastructure, and evaluation.

Level 2: Modern Transformer Ingredients

  1. RoFormer: Enhanced Transformer with Rotary Position Embedding

    • Why: RoPE is a standard positional method in many modern LLMs.

  2. Root Mean Square Layer Normalization

    • Why: RMSNorm is common in modern LLM blocks.

  3. GLU Variants Improve Transformer

    • Why: SwiGLU-style gated MLPs are a standard modern ingredient.

  4. Fast Transformer Decoding: One Write-Head is All You Need

    • Why: multi-query attention explains a key inference-efficiency tradeoff.

  5. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints

  1. FlashAttention

Level 3: Optimization And Systems

  1. AdamW: Decoupled Weight Decay Regularization

    • Why: optimizer details matter for stable large-scale training.

  2. Megatron-LM

    • Why: model parallelism made multi-billion-parameter transformer training practical.

  3. ZeRO

    • Why: optimizer/gradient/parameter partitioning is central to memory-efficient large-scale training.

  4. Tensor Programs V / muP

    • Why: hyperparameter transfer is crucial if small-scale experiments are supposed to predict large-scale behavior.

Level 4: Open And Frontier-Style Model Reports

  1. LLaMA: Open and Efficient Foundation Language Models

    • Why: key open model lineage and efficient training recipe.

  2. The Llama 3 Herd of Models

    • Why: modern open frontier-style report with pretraining and post-training details.

  3. GPT-4 Technical Report

    • Why: illustrates what a restricted frontier model report looks like and what it omits.

Level 5: MoE

  1. GShard

    • Why: large-scale conditional computation.

  2. Switch Transformers

    • Why: simple top-1 MoE routing at scale.

  3. Mixtral of Experts

    • Why: open sparse MoE language model example.

  4. DeepSeek-V3 Technical Report

    • Why: modern high-performance MoE training report.

Level 6: Long Context And Alternatives

  1. ALiBi

    • Why: simple positional bias for length extrapolation.

  2. Mamba

    • Why: selective state-space sequence model challenging attention in some regimes.

  3. RWKV

    • Why: recurrent-style alternative inspired by transformer behavior.

  4. Retentive Network

    • Why: retention mechanism as transformer successor proposal.

Level 7: Post-Training

  1. InstructGPT

    • Why: canonical instruction following with human feedback.

  2. Training a Helpful and Harmless Assistant with RLHF

    • Why: helpfulness and harmlessness training setup.

  3. Constitutional AI

    • Why: AI feedback and principle-guided alignment.

  4. Direct Preference Optimization

    • Why: simpler preference optimization objective.

  5. DeepSeek-R1

    • Why: public reasoning-focused RL training report.

Level 8: Evaluation

  1. MMLU

    • Why: broad multitask benchmark and a common model-report staple.

  2. BIG-bench

    • Why: broad benchmark suite for difficult language model behaviors.

  3. HELM

    • Why: evaluation as a holistic measurement discipline.

  4. GSM8K

    • Why: grade-school math reasoning and verifier framing.

  5. HumanEval / Codex

    • Why: code-generation evaluation and pass@k style metrics.

Paper Note Template

For each paper, write:

# Paper: Title

## Claim

## Bottleneck Attacked

## Mechanism

## Experiment Setup

## Main Evidence

## Missing Baselines

## Scaling Guess

## Failure Modes

## What I Would Try Next

Related