Long Context and Efficient Sequence Models
Long-context modeling asks how LLMs can read, retain, retrieve, and reason over very long sequences without attention cost exploding.
Primary sources:
- ALiBi: Train Short, Test Long
- FlashAttention
- Mamba: Linear-Time Sequence Modeling with Selective State Spaces
- RWKV: Reinventing RNNs for the Transformer Era
- Retentive Network
The Long-Context Problem
Full attention scales quadratically in sequence length during prefill/training:
cost ~= O(T^2)
For generation, the KV cache grows linearly with context length and layers:
memory ~= O(T * layers * kv_heads * head_dim)
Long context is therefore both an algorithm problem and a serving problem.
Three Kinds Of Long Context
Do not treat all long context as the same.
1. Retrieval Context
The answer is somewhere in the prompt. The model must find and use it.
Failure mode:
- Needle-in-a-haystack works, but multi-hop synthesis fails.
2. Working-Memory Context
The model must maintain evolving state across a long interaction or document.
Failure mode:
- It sees the text but loses the thread.
3. Reasoning Context
The model must use long chains of intermediate computation.
Failure mode:
- The context window is large, but reasoning degrades with distractors.
Positional Generalization
Position schemes matter because a model trained on shorter lengths may fail at longer lengths.
Approaches include:
- RoPE scaling and interpolation.
- ALiBi linear attention bias.
- Long-context continued pretraining.
- Sliding-window or local-global attention.
The key lesson: "supports 1M tokens" is not the same as "reliably reasons over 1M tokens."
Efficient Attention
Approaches:
- IO-aware exact attention, as in FlashAttention.
- Sparse attention.
- Sliding-window attention.
- Block-sparse attention.
- Memory tokens or retrieval tokens.
- External retrieval instead of putting everything in context.
Each approach trades expressivity, implementation complexity, and hardware efficiency.
State-Space And Recurrent Alternatives
Models such as Mamba, RWKV, and RetNet explore alternatives to full attention.
Why they are interesting:
- Linear or near-linear sequence processing.
- Better scaling to long sequences in some regimes.
- Potentially lower KV-cache pressure.
Why transformers remain dominant:
- Full attention is extremely flexible.
- Training/serving ecosystems are optimized for transformers.
- Replacement architectures must win on quality, speed, and scaling reliability.
Evaluation Traps
Long-context benchmarks can be misleading.
Ask:
- Is the answer copied from the prompt?
- Does the task require synthesis or just retrieval?
- Are there distractors?
- Does performance degrade with answer location?
- Does the benchmark test actual long-range dependency?
Research Questions
- What tasks actually need full attention across all tokens?
- Can state-space models match transformer reasoning at frontier scale?
- How should memory be structured for agents and long-running tasks?
- Is retrieval-augmented generation better than giant context for most real tasks?
- How do we evaluate long-context reasoning without benchmark theater?