Open Research Questions Frontier LLM Architectures
This note turns the learning sphere into research taste. A frontier researcher is always asking which bottleneck matters, what evidence would be convincing, and what experiment would change their mind.
Architecture
- Are transformer blocks near a local optimum, or just the best-supported hardware/software path?
- Which block changes survive scale-up?
- Can recurrent or state-space models match transformer reasoning at frontier scale?
- How much depth versus width is ideal under modern data and inference constraints?
- How should architecture change for agentic tool use?
Attention And Memory
- Which tasks truly require full attention over the entire context?
- Can KV-cache compression preserve retrieval and reasoning?
- What is the right memory architecture for long-running agents?
- Can models learn durable working memory beyond prompt context?
- How should retrieval interact with long-context transformers?
MoE
- Do experts specialize in human-interpretable ways?
- What routing objectives create useful specialization without collapse?
- How should post-training update sparse experts?
- What is the best serving layout for low-latency MoE?
- Should reasoning tokens route differently from factual recall tokens?
Scaling
- How should compute be split between pretraining, midtraining, long-context adaptation, and post-training?
- Do scaling laws for loss predict reasoning capability?
- How do synthetic data and verifier data change scaling laws?
- Can small experiments predict frontier-scale architectural wins?
- What is the correct unit: FLOPs, dollars, joules, latency, or researcher time?
Data
- What data teaches robust reasoning rather than imitation?
- How do we avoid benchmark contamination at web scale?
- When does synthetic data improve capability versus induce collapse?
- Can data mixture design replace some architectural complexity?
- How should multilingual and code data be balanced?
Post-Training
- How do we train models to reason faithfully instead of narrating plausible reasoning?
- How much reasoning is learned in pretraining versus elicited in post-training?
- Can preference training preserve calibration and uncertainty?
- What is the best objective for long-horizon agent reliability?
- How can safety tuning avoid suppressing useful capability?
Systems
- What architecture gives the best quality per dollar at serving time?
- Can speculative decoding be made robust across diverse tasks?
- When does quantization damage subtle reasoning?
- How do batching and scheduling constraints shape model design?
- What model designs are best for consumer hardware versus datacenter inference?
Personal Research Practice
For each question, maintain:
- Best paper so far.
- Current personal guess.
- Why that guess might be wrong.
- Small experiment.
- Scaled experiment.
- What result would change your mind.
How To Turn A Question Into A Project
Use this pattern:
- Pick one bottleneck.
- Write the current best guess.
- Find the strongest baseline.
- Build the smallest toy version.
- Measure loss, capability, and systems cost.
- Inspect failures manually.
- Decide whether the idea deserves scale.
Example:
Question: Does GQA hurt retrieval at long context?
Toy project: train or load small MHA/GQA models, run controlled retrieval tasks, sweep context length, measure KV memory and accuracy by answer position.
Connected notes: [Attention Mechanics and KV Cache](Attention%20Mechanics%20and%20KV%20Cache.md), [Long Context and Efficient Sequence Models](Long%20Context%20and%20Efficient%20Sequence%20Models.md), [Evaluation Benchmarks and Scientific Method](Evaluation%20Benchmarks%20and%20Scientific%20Method.md).