Frontier Lab Notes

Visual Guide

Diagrams that illustrate key concepts across the notes.

Suggested Study Path

Suggested Study Path A four-phase roadmap to research-capable understanding 1 Foundations Build the core transformer + attention mental model Math and ML Foundations for Frontier LLMs Transformer Block Anatomy Transformer Math and Implementation Deep Dive Attention Mechanics and KV Cache 2 Scaling and Training How compute, data, and stability set the frontier Scaling Laws and Compute Optimal Training Training Optimization and Stability Deep Dive Data Tokenization and Pretraining Objective 3 Frontier Pressure Points Where real systems bend: sparsity, context, serving Mixture of Experts Architectures Long Context and Efficient Sequence Models Frontier Model Systems and Inference Post Training Alignment and Reasoning 4 Research Taste Judge claims, ask good questions, write it up Evaluation Benchmarks and Scientific Method Open Questions, Memo Template, Case Studies

A vertical four-phase roadmap (Foundations, Scaling and Training, Frontier Pressure Points, Research Taste) listing the notes to study in each phase.

Related note →

Transformer Block Anatomy

Pre-Norm Transformer Block h = h + Attn(Norm(h)); h = h + MLP(Norm(h)) residual stream [batch, seq, d_model] Token stream in RMSNorm Multi-Head Attention Q heads K heads V heads softmax(QKᵀ/√d)·V, causal mask, RoPE + residual add RMSNorm MLP · SwiGLU SiLU(W_gate·x) W_up·x W_out( gate * up ) — expands to d_ff, per-token + residual add To next block (×N) Sublayers write updates into the residual stream; they never overwrite it. Pre-norm keeps deep stacks stable to optimize.

One pre-norm decoder block: the residual stream flows through RMSNorm, multi-head attention (QKV heads), a residual add, another RMSNorm, a SwiGLU MLP, and a second residual add.

Related note →

Attention and the KV Cache

Attention & the KV Cache Why autoregressive decode caches keys and values 1 · Prefill Process the whole prompt at once, fill the cache prompt: t₁ t₂ t₃ t₄ Q K V compute Q,K,V for all prompt positions KV cache stores K,V for t₁..t₄ cost ≈ O(T²·d) attention 2 · Decode Generate one token at a time; reuse the cache Qₜ only the new token cached K , V (t₁..t₄, tₜ) new tₜ₊₁ softmax(Qₜ·Kᵀ)·V over cached history → append new K,V 3 · Cache grows with sequence length step → KV bytes grows with batch · layers · KV heads · precision Saves recompute but costs memory — decode becomes memory-bandwidth-bound. MQA / GQA shrink the cache.

Why autoregressive decoding caches keys and values: prompt prefill fills the cache, each decode step runs one new query against cached K/V, and the cache grows with sequence length.

Related note →

Mixture-of-Experts Routing

Mixture-of-Experts Layer Router picks top-k of N experts per token token Router · gating softmax scores over N experts E1idle E2active E3idle E4active …N top-k = 2 selected (e.g. Top-2 routing) Weighted sum gᵢ · Eᵢ(token) → residual stream Load balancing If the router overloads one expert, it bottlenecks while others sit idle. Training adds auxiliary balancing losses. More total params, similar active params/token → better quality per inference FLOP. Costs: routing instability, capacity limits, cross-device communication.

A sparse MoE layer where a router picks the top-k of N expert MLPs per token and weights their outputs, with a note on load balancing and the capacity-versus-compute tradeoff.

Related note →

Scaling Laws

Scaling Laws Loss vs compute — the compute-optimal frontier (log–log) loss (log) training compute C = 6·N·D (log) small N large N, undertrained compute-optimal frontier lower envelope of all model sizes ● best size for each budget Chinchilla lesson For a fixed compute budget, scale parameters N and tokens D together. Many big models were undertrained. A smaller model on more tokens can beat a larger one. Fit curves on small runs, extrapolate before you spend.

Stylized log-log loss-versus-compute curves for different model sizes with the compute-optimal frontier as their lower envelope, illustrating the Chinchilla parameter-versus-data tradeoff.

Related note →

Inference Serving Stack

Inference Serving Stack From request to token — vLLM-style paged KV cache Incoming requests Continuous batching scheduler packs many requests to fill the GPU Prefill whole prompt in one pass; fills KV cache compute-bound Decode one token per step, reads weights + cache memory-bandwidth-bound Paged KV cache fixed-size blocks, allocated on demand (PagedAttention) near-zero fragmentation → higher batch → more throughput Sampling temperature · top-p · (speculative decoding) Output tokens LATENCY dominated by decode: time-to-first-token = prefill, then per-token step time. Speculative decoding cuts it. THROUGHPUT ↔ bigger batches, but batch size fights latency.

The serving path from request batching through the prefill/decode split, a vLLM-style paged KV cache, and sampling, annotated with latency versus throughput tradeoffs.

Related note →

Post-Training Pipeline

Post-Training Pipeline Broad capability → shaped, aligned, reasoning behavior Base model pretrained on broad data — capable but not steerable Supervised Fine-Tuning (SFT) imitation on instruction–response demonstrations changes: chat format, instruction following, style Preference optimization RLHF: reward model + policy, KL-anchored to SFT DPO: optimize directly from preferred / rejected pairs changes: helpfulness, harmlessness, preference tradeoffs Reasoning RL RL on checkable tasks: math, code, verifier / outcome rewards process supervision + test-time compute (e.g. DeepSeek-R1) changes: chain-of-thought depth, problem-solving reliability Aligned reasoning model then: safety tuning, red-teaming, deployment eval Watch the failure modes Reward hacking, over-refusal, sycophancy, style collapse, plausible-but-wrong reasoning traces. Not every behavior is architecture — much is data and post-training.

The pipeline from base model through SFT, preference optimization (RLHF/DPO), and reasoning RL, noting what each stage changes and the common failure modes.

Related note →