RAG Is Dead. Long Live RAG.

Let me say something that will make half the AI industry furious: RAG — Retrieval-Augmented Generation — is a transitional architecture that had a good run, and the run is ending.

The retrieval-augmented generation pattern that everyone spent three years building, tuning, and productizing is a bridge technology. It was always a bridge technology. The model already ate the world. RAG was the bridge. The bridge is closing.

The Context Window Killed RAG

Here's what RAG was actually solving: models couldn't hold enough context to answer questions about your data, so you retrieved relevant chunks and stuffed them into the prompt. Clever. Necessary in 2023. Obsolete in 2026.

When your model has a 10M token context window and can reason over an entire codebase, a document corpus, or a knowledge base in a single call — why are you retrieving chunks? Why the pipeline? Why the semantic search layer with its rankers and re-rankers and hybridBM25 fusion? Just give the model the data and let it do what it's good at.

This isn't a hot take. It's an engineering observation that the market hasn't caught up with yet.

The RAG Architecture Is Fundamentally Asymmetric

The RAG pattern has an inherent problem nobody talks about: it separates retrieval from reasoning. You find the chunks, then the model generates from them. But the model never sees what it didn't retrieve. It doesn't know what it doesn't know. The retrieval step introduces a hard ceiling on the quality of the generation, because you're always working from a subset of the data — a subset that depends entirely on how good your chunking, embedding, and ranking pipeline is.

With larger context windows, you eliminate this ceiling. You give the model the entire corpus and let it find what it needs. The model is better at finding relevant information than your retrieval system is, because the model understands semantic relationships at a depth that keyword and vector search cannot match.

The RAG advocates will say: "but the latency and cost of large contexts." Yes, those are real constraints. But the gap is closing fast, and the engineering cost of maintaining a RAG pipeline is not free. You're trading inference cost for engineering complexity.

Chunking Is a Lie

Here's the part of RAG that nobody admits is broken: semantic chunking doesn't work.

When you divide a document into chunks for retrieval, you're making assumptions about what information the model will need and how it will relate to other chunks. These assumptions are always wrong in edge cases — and production systems live in edge cases. The chunk that contains the answer to a user's question is often the chunk that your chunking algorithm split in half, scattering the relevant information across two chunks neither of which is retrievable on its own.

The more sophisticated your chunking strategy, the more engineering resources you're burning to solve a problem that goes away when you give the model the full document. Chunking is a workaround for a constraint that's no longer the binding constraint.

RAG Made Hallucination Worse, Not Better

The industry sold RAG partly as a hallucination fix — the model would ground its responses in retrieved facts. What actually happened: RAG introduced a new class of hallucination that's harder to detect. The model retrieves chunks that are incomplete, outdated, or poorly ranked. It generates from these chunks. The result looks like a grounded response but contains errors that feel like reasoning errors because they're embedded in confident text.

Classic hallucination is the model making things up. RAG hallucination is the model making things up based on incomplete retrieval. The second is harder to catch because it looks like an information retrieval problem, not a model problem.

The Incumbent's Dilemma

Here's the uncomfortable political economy: the people most invested in RAG are the people who built the infrastructure around it. Vector database companies. Embedding model providers. Chunking-as-a-service startups. The entire RAG-as-a-service ecosystem has a strong incentive to declare RAG permanent.

When a technology is genuinely permanent, you don't need to keep publishing think pieces about why it's here to stay. The frequency of those articles is inversely proportional to the technology's longevity.

What Actually Replaces RAG

The honest answer: context is replacing RAG, and agents will finish the job.

When context windows are large enough, you don't retrieve and augment — you just give the model access and let it reason. When the model needs to act on retrieved information, you give it tool use and let it decide what to fetch. This is the pattern that's replacing RAG: the model as the central reasoning engine, with access to information and tools, rather than a retrieval pipeline feeding a generation pipeline.

RAG was a good idea for its time. It solved a real problem — models couldn't hold enough context — by creating an engineering discipline around retrieval. But the problem it solved is going away. Context windows are growing. Model reasoning is improving. The retrieval pipeline is becoming the bottleneck.

The bridge is closing. Walk across while you can, but start building the road on the other side.

— Mr. TECHNOLOGY