Long Context Windows: 1M+ Tokens

A million tokens fits an entire codebase or a quarter’s worth of meeting notes. The capability is real. The cost and the recall reality are subtler.

How they got long

2023's context windows were 8K-32K tokens. By 2026, frontier models offer 1M-10M token context windows. The technical advances: better positional encoding (RoPE, ALiBi extensions), efficient attention (Flash Attention, ring attention), and architectural changes (sliding window, mixture-of-experts at long context). Each enabled longer contexts without prohibitive compute cost.

The positional-encoding advances. Original transformers had absolute positional encoding; learned per-position. This didn't extrapolate beyond training-length contexts. RoPE (Rotary Position Embedding) and similar relative-position methods extrapolate better; the model can handle contexts longer than what it trained on.

The efficient-attention advances. Naive attention is O(n²), quadratic in sequence length. Flash Attention reduces memory usage (still O(n²) compute but no longer O(n²) memory). Ring attention partitions across GPUs, enabling million-token contexts. Each cuts the practical cost of long contexts substantially.

The architecture experiments. Sliding window attention (only attend to recent tokens for some layers) trades full context for compute savings. Mixture-of-experts (different experts handle different content types) helps long contexts because not all parameters are active per token. Both are part of frontier model design.

The hardware co-design. Long-context inference is memory-bandwidth-bound. New GPUs (H200, Blackwell, MI300X) have higher memory bandwidth specifically to support long-context use cases. The hardware roadmap and the long-context capability advance together.

Effective recall

Just because a model has a 1M context doesn't mean it uses 1M tokens equally well. "Lost in the middle", models attend more to context start and end than the middle. Recall accuracy drops sharply for facts buried in the middle of long contexts. Frontier models in 2026 are better at this than 2023 versions but it's not solved.

The lost-in-the-middle pattern. A famous 2023 paper showed that GPT-3.5 and Claude 2 had U-shaped recall, high accuracy at the start and end of context, lower in the middle. The pattern persists in 2026 models, less severely but still measurably.

The implications for use. Don't trust long context to magically use all the input. Test specifically: insert facts at various positions; ask the model to recall them; measure accuracy by position. The position-dependent recall pattern is what you should design around.

The needle-in-haystack benchmarks. Standard tests insert a "needle" (specific fact) in a "haystack" (long irrelevant context). Best models score >95% at 128K context; some still drop at 1M+. Verify YOUR model's needle-in-haystack performance for your typical context length.

The mitigation strategies. Place critical information at start or end of context. Use system prompts at start; user query at end. For document analysis, chunk-and-summarise pipelines outperform raw long-context for many tasks. The strategy: don't naively rely on long context to extract specific information.

Cost math

Long context is expensive. Inference cost scales with context length, roughly linear for compute, sometimes super-linear because of attention overhead. A 1M-token context query can cost $0.50-$5.00. For high-volume use cases, this dominates economics. Long context is a tool for high-value queries, not high-volume ones.

The compute cost. For most modern models, input tokens cost $1-15 per million depending on model tier. A 1M-token input costs $1-15 per query just for input processing. Output tokens are cheaper but additive. Total cost for substantial-output query on 1M context: $1.50-$25 per query.

The latency cost. Long context inference is slow. A 1M-token query takes 10-60 seconds for first-token; full responses can take minutes. For interactive UX, this is unacceptable; for async/batch, it works.

The "use long context vs RAG" decision. RAG (retrieval-augmented generation) with smaller context: fast and cheap but limited by retrieval quality. Long context: slower and more expensive but uses all input. The right choice depends on whether retrieval can isolate the relevant subset; for tasks where relevance is fuzzy or holistic, long context wins.

The cost trajectory. Per-token long-context cost has dropped 5-10x from 2023 to 2026. Trajectory continues; expect another 5-10x drop by 2028. Budget for current costs; plan for future drops to expand viable use cases.

Long context vs RAG

The competing approach: retrieval-augmented generation pulls relevant chunks from a vector store and gives them to a smaller-context model. RAG is cheaper and lower-latency than long context; long context is simpler architecturally and handles cases where relevant information isn't easy to retrieve.

The RAG advantages. Cheaper per query (you only feed relevant chunks). Faster (small contexts inference quickly). Scales to arbitrary corpus size (limited only by vector store, not model context). Better for "find a specific fact" queries where retrieval isolates the answer.

The long-context advantages. Simpler (no retrieval system to build). Better for "synthesise across the whole document" queries. Better when relevance isn't easily defined (creative writing, complex reasoning over a corpus). Handles cases where retrieval would miss because relevance is non-keyword.

The hybrid pattern. Many production systems combine: RAG to retrieve broadly relevant chunks, then long-context to reason over the retrieved set. Best of both: scale of RAG, depth of long context. Implementation is more complex but often the right choice.

The decision criteria. Corpus size > model context: RAG mandatory. Per-query latency requirement < 5s: RAG strongly preferred. Cost per query budget < $0.10: RAG strongly preferred. Whole-document synthesis required: long context preferred. Match the architecture to the query economics.

Common antipatterns

Treating "long context" as a substitute for retrieval. Retrieval is still useful for narrow factual queries; long context shines on holistic synthesis.

Ignoring lost-in-the-middle. Critical info buried in middle of long context may not surface. Place important content at start/end.

Using 1M context for routine 10K-token queries. Cost without benefit. Use the smallest context that fits the query.

Skipping needle-in-haystack testing. Verify YOUR model's recall pattern before relying on long context for production.

What to do this week

Three moves. (1) Run a needle-in-haystack benchmark for your model at the context length you typically use. The recall accuracy at various positions tells you where to place critical content. (2) Compute per-query cost for your long-context use cases at current API pricing. The number tells you whether the use case is economic at scale. (3) For one use case currently using long context, prototype a RAG version. Compare quality and cost; the comparison usually surprises.