Math and ML Foundations for Frontier LLMs

This note lists the background you need to read frontier LLM papers without feeling like every paragraph is a trapdoor.

You do not need to master all of math before touching models. Learn in loops: read, implement, get confused, study the missing math, return.

Linear Algebra

You need to be comfortable with:

• Vectors and matrices.
• Matrix multiplication shapes.
• Dot products and cosine similarity.
• Norms.
• Orthogonality.
• Projections.
• Low-rank approximation.
• Eigenvalues/eigenvectors at an intuitive level.
• Tensor reshaping and broadcasting.

Why it matters:

• Attention is matrix multiplication plus softmax.
• Embeddings are vector representations.
• LoRA is low-rank adaptation.
• MoE routing is vector scoring plus sparse dispatch.
• GPU performance is mostly large matrix operations.

Practice:

• Write QKV attention with explicit shapes.
• Implement low-rank matrix factorization.
• Compute cosine similarities between token embeddings.

Connected notes:

• Transformer Math and Implementation Deep Dive
• Attention Mechanics and KV Cache

Probability

You need:

• Random variables.
• Conditional probability.
• Expected value.
• Entropy.
• Cross-entropy.
• KL divergence.
• Sampling from categorical distributions.
• Calibration.

Why it matters:

• Language models output probability distributions over tokens.
• Pretraining minimizes cross-entropy.
• RLHF/DPO involve distribution shifts and preference probabilities.
• Sampling parameters change output distributions.

Important intuition:

The model is not directly choosing words. It is producing a distribution over possible next tokens.

Practice:

• Compute cross-entropy by hand for a tiny vocabulary.
• Implement temperature, top-k, and top-p sampling.
• Compare calibrated versus overconfident predictions.

Connected notes:

• Data Tokenization and Pretraining Objective
• Post Training Alignment and Reasoning

Calculus And Backpropagation

You need:

• Derivatives.
• Chain rule.
• Gradients.
• Jacobian intuition.
• Gradient descent.
• Vanishing/exploding gradients.

Why it matters:

• Training is gradient-based optimization.
• Residual connections and normalization help gradients flow.
• Loss spikes and NaNs are gradient/activation pathologies.

Practice:

• Implement a tiny MLP and backprop manually.
• Derive softmax plus cross-entropy gradient.
• Inspect gradient norms during transformer training.

Connected notes:

• Training Optimization and Stability Deep Dive
• Transformer Block Anatomy

Optimization

You need:

• SGD.
• Momentum.
• Adam/AdamW.
• Weight decay.
• Learning-rate schedules.
• Warmup.
• Gradient clipping.
• Batch size effects.

Why it matters:

• Frontier model training is expensive enough that optimizer mistakes are catastrophic.
• Small-model hyperparameters may not transfer without care.
• Training stability is a research skill, not plumbing.

Practice:

• Train the same tiny transformer with SGD, Adam, and AdamW.
• Sweep learning rates.
• Observe divergence and recovery.

Connected notes:

• Training Optimization and Stability Deep Dive
• Scaling Laws and Compute Optimal Training

Information Theory

You need:

• Entropy as uncertainty.
• Cross-entropy as coding cost.
• KL divergence as distribution mismatch.
• Perplexity as exponentiated cross-entropy.

Why it matters:

• Language modeling loss is an information-theoretic quantity.
• Scaling laws are often fit on loss.
• Post-training can improve user preference without always improving base-model perplexity.

Practice:

• Convert cross-entropy to perplexity.
• Compare two token distributions with KL divergence.

Connected notes:

• Scaling Laws and Compute Optimal Training
• Evaluation Benchmarks and Scientific Method

Deep Learning Concepts

You need:

• Embeddings.
• Residual networks.
• Normalization.
• Dropout.
• Activation functions.
• Overfitting.
• Train/validation/test splits.
• Fine-tuning.
• Transfer learning.

Why it matters:

• Transformer blocks are deep residual networks.
• Fine-tuning changes behavior in ways that can hide base capabilities.
• Evaluation depends on clean splits and held-out data.

Practice:

• Train a classifier with embeddings.
• Add residual connections to an MLP.
• Compare LayerNorm and RMSNorm.

Connected notes:

• Transformer Block Anatomy
• Post Training Alignment and Reasoning

Systems Basics

You need:

• GPU memory hierarchy.
• FLOPs versus memory bandwidth.
• Batch size and throughput.
• Latency versus throughput.
• Distributed training basics.
• Profiling.

Why it matters:

• FlashAttention is about memory IO.
• KV cache is an inference memory problem.
• MoE is a communication and load-balancing problem.
• Long context is a serving problem.

Practice:

• Profile attention at different sequence lengths.
• Estimate KV-cache memory.
• Measure tokens/sec for prefill and decode separately.

Connected notes:

• Frontier Model Systems and Inference
• Long Context and Efficient Sequence Models

Scientific Thinking

You need:

• Hypothesis formation.
• Baseline selection.
• Controlled experiments.
• Ablation.
• Error analysis.
• Reproducibility.
• Falsification.

Why it matters:

• Frontier research is noisy and expensive.
• Benchmarks can lie.
• Architecture claims are often confounded by data, compute, and implementation.

Practice:

• For every paper, write one missing baseline.
• For every experiment, write one result that would change your mind.

Connected notes:

• Research Memo Template for LLM Papers
• Evaluation Benchmarks and Scientific Method
• Researcher Skill Stack for Frontier Labs

Minimum Viable Foundation

If you want the shortest useful path:

1. Matrix multiplication and tensor shapes.
2. Softmax and cross-entropy.
3. Gradient descent and AdamW.
4. Transformer forward pass.
5. Validation loss and contamination.
6. GPU memory versus compute.
7. Controlled experiments and baselines.

That is enough to start reading papers seriously while filling gaps.