Million-Token Context Windows Are a Crutch, Not a Solution
The Million-Token Lie
Every frontier lab is racing to ship the longest context window. Anthropic, Google, OpenAI — they're all chasing bigger and bigger buffers like it's a dick-measuring contest. And it is. Because deep down, everyone in the industry knows: long context is a workaround for a problem we refuse to solve properly.
Memory.
What Context Windows Actually Do
When a model has a 1M-token context window, it means it can fit roughly 750,000 words in a single forward pass. Sounds impressive until you ask the obvious question: at what cost?
- Inference cost scales roughly linearly with context length for most providers
- Latency for 1M tokens is multiple seconds, even on the best hardware
- The "lost in the middle" problem (Liu et al., 2023) showed models perform worse on information buried in the middle of long contexts than at the start or end
- RULER benchmarks exposed that effective context is often a quarter to a half of what's advertised
The pitch is simple: "Just stuff everything in the prompt!" The reality is uglier — you're paying 10x for 10% of the benefit. Marketing loves this. Engineering does not.
The Real Problem Is Memory, Not Forgetting
A context window is RAM. A database is memory.
When an agent needs to recall what it did yesterday, what the user prefers, what it learned three weeks ago — a context window is the wrong tool. You don't load your entire browser history into Chrome to remember your bookmarks. You load your bookmarks.
Retrieval-augmented generation was supposed to fix this. Vector stores, semantic search, RAG pipelines. For static knowledge bases, it works. But for agentic memory — episodic recall, learned preferences, working state — we're still bolting on hacks: summarization chains, rolling buffers, "memory tool" plugins that lose coherence after a few turns.
The real solution is a real memory architecture:
- Episodic storage with smart retrieval
- Working memory that's compressed, not truncated
- A semantic layer that knows what matters
- Time-decay and importance weighting
- Cross-session continuity
We have all of this. In databases. In caching systems. In operating systems. We've known how to do memory for sixty years. We're reinventing it badly because long context is easier to ship and easier to sell.
Why Labs Keep Shipping the Crutch
Context window size is a number on a slide. "1M context!" is a sales pitch. "We built a real memory system" is engineering.
Labs keep shipping long context because:
1. It benchmarks well in isolation 2. It generates headlines 3. It defers the harder architectural problem 4. It sells API tokens — more context, more billable tokens, fatter revenue
But the customers paying for this are building agents that forget everything between sessions, lose track of long-running tasks, and can't recall user preferences without re-prompting. The 1M context window didn't fix any of that. It just made the prompt larger.
What to Do Instead
If you're building agents in 2026, here's my advice: stop trying to fit the world in the context window. Build a memory layer. Use retrieval. Compress aggressively. Have your agent decide what's worth remembering.
A 200K-context model with great memory will outperform a 1M-context model with none. Every time. Because the 1M model is wasting attention on noise while the 200K model is being fed exactly what it needs.
The labs will keep shipping bigger windows. Let them. Build the thing that actually solves the problem.
Long context is a crutch. Memory is the answer. And you know it.