1M-Token Context Windows Are a Dead End and the Labs Know It

The lab press releases are noise. "10M token context!" "Gemini holds your entire codebase!" The number keeps climbing. The actual production utility has been flat for eighteen months. The labs are quietly building retrieval, routing, and compression on top because they have to.

I am going to say the quiet part out loud: context windows past roughly 128k tokens are a marketing surface, not a capability, and building your agent architecture around them will burn your budget and your accuracy.

Hey guys, Mr. Technology here.

Lost in the Middle Is Worse Than You Think

Liu et al. published "Lost in the Middle" in 2023. Models do well on the start and end of the context, badly on the middle. The labs said they fixed it. They did not. Chroma, LangChain, and Princeton CRFM benchmarks keep showing the same U-curve on real retrieval. The middle of a long context is still measurably worse than the ends.

So the model has a 1M context window and your answer is computed primarily from the first 50k and last 50k. The middle 900k tokens pay rent on GPU memory without paying rent on the answer. That is not a feature. That is a billing trick.

The Compute Bill Will Eat Your Margin

Attention is O(n squared) in standard transformers. Linear attention has been the "next breakthrough" for three years and still does not match full attention on production quality. A 1M context call is 60-100x more expensive than a 16k call. A 10M call is 600-1000x. You cannot put 1M into a hot path with sub-second latency and margin to spare.

I have watched teams ship a benchmark at 92% retrieval on long-context RAG, then go to production and find p95 at 8 seconds, per-query cost at $0.30 to $2.00, and accuracy on the real user query at 71%. The "long context solves RAG" slide does not survive the production dashboard.

Needle-in-Haystack Is Not Your Workload

Every long-context launch is demonstrated with needle-in-a-haystack: hide one fact in a long doc, ask the model to retrieve it. The benchmark saturates around 1M tokens for every frontier model. The labs ship the chart. The chart is also useless.

Your workload is not "find one fact in one doc." It is "answer a question using 200 facts across 50 docs, where 30 conflict, the question is ambiguous, and the wrong fact has higher prior probability than the right one." Long context makes that worse: more conflicting priors, more places to hedge, more room to hallucinate bridges between facts that should never have shared a prompt.

What Actually Works

Small context, dense retrieval, explicit routing. Chunk aggressively. Embed. Retrieve the top 20-50. Re-rank. Stuff only the top 5-10 into the prompt. Total context: 16k-32k tokens. Per-query cost: $0.01 to $0.05. Latency: sub-second. Accuracy on real workloads: higher than the 1M alternative on every production benchmark I have seen.

Structured memory lives outside the context window: conversation state in a typed store, tool results cached and summarized on retrieval. The context window is for the active working set, not the history. This is how every serious agent framework works in 2026. The labs ship long context for the marketing. The serious frameworks do the opposite. For tools, verifiable routing wins: a router that selects 3-5 tools per call, not a 200-tool prompt dump.

The Bottom Line

The frontier labs will keep shipping longer context windows because the number on the slide is the only long-context metric the buying market understands. They will also keep investing in retrieval and agent routing — because the engineers inside those labs know long context is the slide, not the product.

You are not the lab. Stop buying the marketing. Use 16k-32k context for the active working set, retrieval for the long tail, and a typed store for state. Your accuracy goes up, your cost drops an order of magnitude, and your p95 stops being a postmortem.

The labs will not tell you this. They have a slide to defend. I don't.

— Mr. Technology