Architecture Concept Graph Frontier LLMs

This note explains how the whole frontier LLM stack connects. The main mistake beginners make is studying topics as isolated facts: attention, scaling laws, MoE, RLHF, evals, inference. In real research, every decision pushes on every other layer.

The Whole System

flowchart TD
  Data["Data mixture, tokenizer, objective"] --> Pretrain["Pretraining loss"]
  Pretrain --> Block["Transformer block"]
  Block --> Attention["Attention + positional encoding"]
  Block --> MLP["MLP / expert layers"]
  Attention --> Context["Context length + KV cache"]
  MLP --> MoE["Dense vs MoE capacity"]
  Pretrain --> Scale["Scaling laws + compute allocation"]
  Scale --> Systems["Training systems + hardware"]
  Context --> Inference["Inference serving cost"]
  MoE --> Inference
  Systems --> Inference
  Pretrain --> PostTrain["SFT / RLHF / DPO / reasoning RL"]
  PostTrain --> Behavior["Helpful, safe, tool-using behavior"]
  Behavior --> Eval["Evaluation + red-teaming"]
  Eval --> Data
  Eval --> PostTrain
  Eval --> Scale

The Core Loop

Frontier labs repeatedly run this loop:

1. Choose a capability target.
2. Estimate the bottleneck: data, architecture, optimization, scale, inference, or post-training.
3. Design a small experiment.
4. Train and evaluate.
5. Compare against strong baselines.
6. Scale if the result survives.
7. Re-check cost and safety at deployment.

Every topic in this sphere maps into that loop.

Data Controls What Can Be Learned

Data Tokenization and Pretraining Objective determines the model's training distribution. If the data lacks high-quality math, code, multilingual text, or long documents, architecture cannot fully compensate.

Connections:

• Data mixture changes Scaling Laws and Compute Optimal Training because token quality changes the value of each token.
• Long documents are required for Long Context and Efficient Sequence Models to be more than a positional trick.
• Instruction and preference data shape Post Training Alignment and Reasoning.
• Contamination affects Evaluation Benchmarks and Scientific Method.

Research instinct:

Before blaming architecture, ask whether the model saw the right data in the right format.

Architecture Controls The Computation The Model Can Easily Express

Transformer Block Anatomy defines the repeated compute unit. The two main sublayers have different jobs:

• Attention moves information across positions.
• MLPs transform features at each position.

Connections:

• Attention design drives Attention Mechanics and KV Cache.
• MLP design drives Mixture of Experts Architectures.
• Positional encoding affects Long Context and Efficient Sequence Models.
• Normalization and residual structure affect Training Optimization and Stability Deep Dive.

Research instinct:

Architecture is not "what can the model theoretically represent?" It is "what can the model learn efficiently at scale?"

Scaling Decides What Is Worth Doing

Scaling Laws and Compute Optimal Training turns architecture into economics.

A trick matters if it improves one of these:

• Loss at fixed compute.
• Capability at fixed compute.
• Capability at fixed data.
• Latency at fixed quality.
• Cost at fixed user experience.
• Safety/reliability at fixed capability.

Connections:

• MoE changes total parameters versus active compute.
• Long context changes token budget and serving cost.
• Post-training changes observed behavior without necessarily changing base loss.
• Systems decide whether theoretical FLOP savings survive real hardware.

Research instinct:

Always ask "fixed what?" A paper can win at fixed parameters and lose at fixed dollars.

Systems Turn Architecture Into Reality

Frontier Model Systems and Inference is not a side issue. It decides what models can be trained and served.

Connections:

• KV cache makes attention an inference memory problem.
• MoE makes routing a communication problem.
• Long context makes prefill and cache management core product bottlenecks.
• Quantization can change model behavior.
• Parallelism constraints shape architecture choices.

Research instinct:

A model architecture is also a distributed systems design.

Post-Training Turns Capability Into Behavior

Post Training Alignment and Reasoning changes how the base model behaves:

• SFT teaches interface and format.
• RLHF/DPO teaches preference-shaped behavior.
• Reasoning RL can elicit longer solution search.
• Safety tuning shapes refusal and boundary behavior.

Connections:

• Post-training depends on the base model's pretrained capabilities.
• Preference data can hide or reveal architecture weaknesses.
• Evaluation must separate "can do" from "will do."
• Reasoning methods increase test-time compute demands.

Research instinct:

If a model fails a task, distinguish capability failure from elicitation failure.

Evaluation Is The Feedback Signal For Research

Evaluation Benchmarks and Scientific Method controls what labs optimize.

Connections:

• If evals are contaminated, scaling laws look better than reality.
• If evals reward style, post-training can overfit to vibes.
• If evals are too narrow, architecture improvements may not generalize.
• If evals ignore latency/cost, systems regressions hide.

Research instinct:

A benchmark is a microscope, not the territory. Know what it cannot see.

The Stack As Research Questions

What To Memorize

The frontier LLM stack is a set of coupled bottlenecks:

Data -> Objective -> Architecture -> Optimization -> Scale -> Systems -> Post-training -> Evaluation -> Data

Do not become a person who only knows one node. Labs need people who can reason across edges.

Related

• Frontier LLM Architectures MOC
• Transformer Math and Implementation Deep Dive
• Training Optimization and Stability Deep Dive
• Evaluation Benchmarks and Scientific Method
• Frontier Model Case Studies