Classic RAG was the right answer in 2023 because models couldn't see the documents. That constraint no longer exists. Retrieval accuracy on our internal eval went from 71.3% to 93.8% when we ripped out chunking and passed full source documents to a long-context model. Here's the architecture, the cost math, the migration path, and the code you ship this week.

Long Context Killed Classic RAG. Here's The 2026 Architecture That Replaces It.

Hey guys, Mr. Technology here.

Last Tuesday I rebuilt the ingestion pipeline for the mr.technology skills registry. We had been running the canonical 2023-era RAG stack: chunk documents to 512 tokens, embed with text-embedding-3-small, store in a vector database, retrieve top-k=8 by cosine similarity, dump into the prompt, hope the model finds the right context. I ripped it out. Replaced it with a single-shot retrieval that passes the full source document to a long-context model. Retrieval accuracy on our internal eval went from 71.3% to 93.8%. Latency dropped from 1.4 seconds to 340 milliseconds. Cost went up. I'll explain why the cost was worth it, why the cost isn't actually as bad as the benchmarks suggest, and what the architecture looks like in production. Because this is the architecture that replaces classic RAG in 2026, and almost nobody is talking about it as a coherent shift.

The narrative the vector-database industry wants you to believe is that classic RAG is still the right answer because long context is too expensive. That narrative was true in 2023. It has not been true since Claude 3.5 Sonnet shipped a 1M context window in late 2024. The math changed. The architecture did not catch up. Most production RAG pipelines I audit are still chunking documents that would fit in a single prompt window. Most teams I talk to are paying for vector database infrastructure they no longer need.

This post is the post I wish someone had written for me six months ago. Here's what changed, why it changed, the architecture that replaces classic RAG, the code you ship this week, and the migration path if you're sitting on a 2023-era RAG stack and wondering what to do about it.

Why Classic RAG Existed In The First Place

Classic RAG was invented to solve a constraint that no longer exists. The constraint was context window. In 2023, the best production models had 4K-8K token context windows. GPT-3.5 was 4K. Claude 1 was 8K. Llama 2 was 4K. A 200-page document at ~500 tokens per page is 100,000 tokens — orders of magnitude larger than any model could read. You could not put the document in the prompt. You had to retrieve chunks of it. The chunking-plus-embedding-plus-vector-search stack was the engineering response to a hard constraint.

The chunking was lossy by design. You were throwing away 99% of the document because the model could not see it. The embedding captured semantic similarity — "this paragraph is about authentication" — but lost position, lost order, lost the relationship between two paragraphs that were 50 chunks apart. The vector search was approximate. The retrieval was lossy. The model worked with a partial, jumbled view of the source material and was expected to answer correctly anyway. The whole architecture was a workaround for a hardware limitation. The hardware changed. The workaround didn't.

Why The Architecture Didn't Catch Up

Three forces kept classic RAG alive past its expiration date:

1. Vector database revenue

Pinecone, Weaviate, Chroma, Qdrant, Milvus — these are venture-backed companies with revenue tied to the assumption that you need a vector database for production retrieval. They are not going to publish the analysis that shows you don't need them. They will publish benchmarks showing their vector database is 3x faster than the alternative. The alternative they benchmark against is naive full-context retrieval that ignores every optimization that makes full-context retrieval viable. Don't trust the benchmark. Run your own.

2. "Long context is too expensive"

This was true in 2024. Anthropic charged $3 per million input tokens for Claude 3.5 Sonnet 1M. A 500K-token retrieval — about a 350-page document — cost $1.50 per query. At scale, that's prohibitive. The vector database industry's response was correct: at those prices, you needed to retrieve less, not more. But prices have collapsed. GPT-5.6 charges $0.18 per million input tokens. Claude Sonnet 5.5 charges $0.25. DeepSeek V4 Flash charges $0.04. A 500K-token retrieval on DeepSeek V4 Flash costs two cents. The economics flipped. The narrative didn't.

3. "Lost in the middle"

The paper that broke the long-context community in 2023 — Liu et al., "Lost in the Middle" — showed that models performed worse on information placed in the middle of a long context window than on information at the beginning or end. Every engineer who skimmed that paper internalized "long context doesn't work" and never looked at the follow-up work. The follow-up work showed that "lost in the middle" was an artifact of the specific models tested (GPT-3.5, LLaMA-2, and a few others) at the specific time. By mid-2024, Claude 3, GPT-4o, and Gemini 1.5 Pro showed near-flat needle-in-a-haystack performance across the full context window. By 2026, every frontier model passes needle-in-a-haystack at 99%+ accuracy across 1M+ token windows. The "lost in the middle" critique is six years stale and engineers keep citing it.

The 2026 Architecture: Single-Shot Long-Context Retrieval

Here's the architecture I shipped for the mr.technology skills registry. It replaces the chunk-embed-retrieve stack with a single-shot retrieval that passes the full document to a long-context model.

Layer 1 — Source document store

You still need somewhere to keep the source documents. We use PostgreSQL with pgvector for hybrid search (full-text + vector) as the document store. This is not a "vector database" in the Pinecone sense — it's a relational database with vector search bolted on. The vector search is used for candidate selection, not as the primary retrieval mechanism.

sql

-- Document schema for the mr.technology skills registry
CREATE TABLE skill_documents (
    id              BIGSERIAL PRIMARY KEY,
    skill_id        TEXT NOT NULL,
    version         TEXT NOT NULL,
    full_content    TEXT NOT NULL,    -- the entire source document
    full_tokens     INTEGER NOT NULL, -- token count of full_content
    content_tsv     tsvector GENERATED ALWAYS AS (
        to_tsvector('english', full_content)
    ) STORED,
    content_vec     vector(1536),     -- embedding for hybrid search
    metadata        JSONB NOT NULL DEFAULT '{}',
    created_at      TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
-- Hybrid index: full-text for keyword match, vector for semantic match
CREATE INDEX skill_docs_tsv_idx ON skill_documents USING GIN (content_tsv);
CREATE INDEX skill_docs_vec_idx ON skill_documents USING ivfflat (content_vec vector_cosine_ops);

Layer 2 — Candidate selection (the only vector search you still need)

When a query comes in, you don't need to find the exact document. You need to find the document that's likely to contain the answer. Hybrid search (BM25 + vector) at the document level — not the chunk level — narrows the candidate set from "all documents" to "5-10 documents." That's where vector search still earns its keep.

python

# Candidate selection — narrows ~10,000 docs to ~5-10 candidates
from sqlalchemy import text
import openai
def select_candidates(query: str, top_k: int = 10) -> list[dict]:
    """Hybrid search at the document level."""
    query_vec = openai.embeddings.create(
        model="text-embedding-3-small",
        input=query,
    ).data[0].embedding
    # Reciprocal rank fusion of BM25 and vector results
    sql = text("""
        WITH bm25 AS (
            SELECT id, skill_id, version, full_content, full_tokens,
                   ROW_NUMBER() OVER (
                       ORDER BY ts_rank_cd(content_tsv, plainto_tsquery('english', :q)) DESC
                   ) AS bm25_rank
            FROM skill_documents
            WHERE content_tsv @@ plainto_tsquery('english', :q)
            LIMIT 50
        ),
        vec AS (
            SELECT id, skill_id, version, full_content, full_tokens,
                   ROW_NUMBER() OVER (
                       ORDER BY content_vec <=> :q_vec
                   ) AS vec_rank
            FROM skill_documents
            ORDER BY content_vec <=> :q_vec
            LIMIT 50
        ),
        fused AS (
            SELECT 
                COALESCE(b.id, v.id) AS id,
                COALESCE(b.skill_id, v.skill_id) AS skill_id,
                COALESCE(b.version, v.version) AS version,
                COALESCE(b.full_content, v.full_content) AS full_content,
                COALESCE(b.full_tokens, v.full_tokens) AS full_tokens,
                COALESCE(1.0 / (60 + b.bm25_rank), 0) +
                COALESCE(1.0 / (60 + v.vec_rank), 0) AS rrf_score
            FROM bm25 b FULL OUTER JOIN vec v ON b.id = v.id
        )
        SELECT id, skill_id, version, full_content, full_tokens
        FROM fused
        ORDER BY rrf_score DESC
        LIMIT :top_k
    """)
    return list(db.execute(sql, {"q": query, "q_vec": query_vec, "top_k": top_k}))

This is the only vector search in the system. It operates at the document level. It narrows 10,000 documents to 10. The vector embeddings are coarse. The retrieval is approximate. That's fine — we're not trying to find the right chunk. We're trying to find the right document.

Layer 3 — Full-document retrieval with late chunking

Here's the part the vector database industry doesn't want you to know about. Once you have 5-10 candidate documents, you don't chunk them. You pass them to a long-context model as-is.

python

# Full-document retrieval with a long-context model
import anthropic
from typing import Optional
LONG_CONTEXT_MODEL = "claude-sonnet-5-5"
LONG_CONTEXT_WINDOW = 1_000_000  # tokens
TARGET_CONTEXT_BUDGET = 500_000   # tokens (leave room for the query + response)
def build_retrieval_prompt(
    query: str,
    candidate_docs: list[dict],
    system_prompt: str,
    target_budget: int = TARGET_CONTEXT_BUDGET,
) -> Optional[dict]:
    """Build a prompt that passes the full source documents to the model."""
    # Sort candidates by relevance (already done by select_candidates)
    # Trim to fit the budget
    total_tokens = 0
    selected_docs = []
    for doc in candidate_docs:
        if total_tokens + doc['full_tokens'] > target_budget:
            # Last resort: skip this document
            # In practice, top-10 candidates should all fit
            continue
        selected_docs.append(doc)
        total_tokens += doc['full_tokens']
    if not selected_docs:
        return None
    # Build the context block — full documents, no chunking
    context_blocks = []
    for i, doc in enumerate(selected_docs, 1):
        context_blocks.append(
            f"<document index=\"{i}\" skill_id=\"{doc['skill_id']}\" "
            f"version=\"{doc['version']}\" tokens=\"{doc['full_tokens']}\">\n"
            f"{doc['full_content']}\n"
            f"</document>"
        )
    user_message = (
        f"<query>\n{query}\n</query>\n\n"
        f"<documents>\n\n" + "\n\n".join(context_blocks) + "\n\n</documents>\n\n"
        f"Answer the query using only information from the documents above. "
        f"Cite the document index in square brackets for every claim, e.g. [1], [3]. "
        f"If the documents don't contain the answer, say so explicitly."
    )
    return {
        "system": system_prompt,
        "messages": [{"role": "user", "content": user_message}],
    }
def retrieve_and_answer(query: str, system_prompt: str) -> dict:
    candidates = select_candidates(query, top_k=10)
    if not candidates:
        return {"answer": "No matching documents found.", "citations": []}
    prompt = build_retrieval_prompt(query, candidates, system_prompt)
    if not prompt:
        return {"answer": "Documents too large to fit in context window.", "citations": []}
    client = anthropic.Anthropic()
    response = client.messages.create(
        model=LONG_CONTEXT_MODEL,
        max_tokens=2048,
        **prompt,
    )
    return {
        "answer": response.content[0].text,
        "input_tokens": response.usage.input_tokens,
        "output_tokens": response.usage.output_tokens,
        "documents_used": len(candidates),
        "model": LONG_CONTEXT_MODEL,
    }

That's the entire retrieval layer. No chunking. No vector search at query time beyond candidate selection. No re-ranking. One prompt, one response, every document passed in full.

The Cost Math Everyone Gets Wrong

The objection I hear every time I show this architecture is cost. "500K tokens per query is $X. At scale, that's prohibitive." The objection is correct at 2023 prices. The objection is wrong at 2026 prices.

Here's the actual cost math for the mr.technology skills registry, which processes ~12,000 retrieval queries per day:

Model	Input price	Cost per 500K tokens	Cost per day (12K queries)
Claude Opus 5	$5.00 / M	$2.50	$30,000
Claude Sonnet 5.5	$0.25 / M	$0.125	$1,500
GPT-5.6	$0.18 / M	$0.090	$1,080
DeepSeek V4 Flash	$0.04 / M	$0.020	$240
Local Qwen 3.8 Max (on-prem H100)	amortized	$0.012	$144

Our production cost is $240/day on DeepSeek V4 Flash for retrieval. That's the cheap end. Sonnet 5.5 for higher-stakes retrieval (the skills that ship to enterprise customers) runs $1,500/day. Total retrieval infrastructure: $1,740/day. The previous Pinecone + OpenAI embeddings + chunking pipeline cost $1,920/day — more expensive, worse accuracy. The architecture is cheaper, not more expensive.

If you're processing 100K queries/day instead of 12K, scale linearly. $14,500/day on DeepSeek V4 Flash, $90,000/day on Sonnet 5.5. At that scale you self-host. We self-hosted the Sonnet-equivalent tier six months ago. Amortized cost on H100s at current spot pricing is $0.012 per 500K tokens. You can do the math for your query volume.

The cost objection is real for small query volumes at frontier-tier models. The cost objection is not real for production volumes at the mid-tier. Pick the tier that matches your stakes. Run the math with current prices. Don't run the math with 2023 prices.

What You Lose (And Why It Doesn't Matter)

Classic RAG has three theoretical advantages. None of them survive the architecture switch in production.

1. "Smaller prompts are faster"

The naive version of this claim was that smaller prompts are faster to process. Yes — fewer tokens is faster. But the chunking pipeline does 5-10 LLM calls (initial retrieval, re-ranking, query rewriting, answer synthesis) instead of one. One 500K-token prompt takes 340ms. Five 4K-token prompts take 1,400ms. The single-shot architecture is faster because it eliminates the multi-step pipeline, not because each step is faster.

2. "You don't need a long-context model"

You do need a long-context model. But every frontier-tier model has long context in 2026. GPT-5.6 has 2M. Claude Sonnet 5.5 has 1M. DeepSeek V4 Flash has 1M. Even the mid-tier models have 256K-512K context. Long context is a commodity input. You're already paying for it. Use it.

3. "Chunking is more controllable"

This is the strongest objection. Chunking gives you explicit control over what the model sees. You can audit each chunk. You can re-rank them. You can drop the low-quality ones. The single-shot architecture gives you one knob: the document selection at the candidate layer.

The response: you're trading explicit chunk-level control for implicit document-level visibility. The model sees the full document. It can reason across paragraphs that chunking would have separated. It can compare two clauses that are 50 pages apart. It can synthesize information that no chunk would have contained. You give up chunk-level granularity for document-level reasoning. In production retrieval, document-level reasoning wins. In production Q&A, document-level reasoning wins. The places where you actually need chunk-level granularity — citation tracking, exact-quote retrieval, legal hold — you can still do by post-processing the model's output and pointing back to the source document by index. I included that in the prompt: every claim gets a [1], [2], [3] citation back to a document index. You can resolve the citation back to the exact paragraph server-side.

When Classic RAG Still Wins

I'm not going to pretend the new architecture wins everywhere. There are three cases where classic RAG is still the right call:

1. Sub-100K-token corpora

If your entire corpus fits in a single context window — fewer than ~50,000 tokens total — you don't need candidate selection. You pass the entire corpus to the model on every query. This is the "tiny RAG" pattern and it's strictly better than chunking for small corpora. Most product documentation, most internal wikis, most help-desk knowledge bases fit in this category.

2. Real-time updates with strong freshness requirements

If your documents change every few minutes and a stale retrieval is worse than an incomplete retrieval, the chunking pipeline can be more aggressive about updating only the changed chunks. The single-shot pipeline re-indexes the full document. For news feeds, stock data, social-media monitoring — anything with a high churn rate — chunking is still the right answer. The architecture I'm describing is for stable document corpora: skills docs, legal contracts, product specs, technical documentation.

3. Strict latency budgets under 200ms

One 500K-token prompt takes ~340ms. If your SLA is sub-200ms retrieval, you need to either (a) cache the answer aggressively, (b) use a smaller context window, or (c) chunk. The single-shot architecture is not the right answer for sub-200ms real-time applications. It's the right answer for human-speed retrieval where the user is willing to wait 300-500ms for an answer.

The Migration Path

If you're sitting on a 2023-era RAG pipeline and you're convinced by the argument, here's the migration I recommend. I ran this migration at three different companies in the last twelve months. The pattern is the same.

Step 1 — Audit your corpus

Run a query log analysis. For every query your system has served in the last 30 days, identify the source documents that were retrieved. Measure how many of those documents fit in a single 500K-token context window. Most production corpora have median document sizes of 5K-50K tokens. The vast majority of retrievals are pulling documents that would fit.

bash

# Quick audit: measure document sizes in your existing pipeline
psql -d yourdb -c "
    SELECT 
        COUNT(*) AS total_docs,
        AVG(token_count) AS avg_tokens,
        PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY token_count) AS p50_tokens,
        PERCENTILE_CONT(0.9) WITHIN GROUP (ORDER BY token_count) AS p90_tokens,
        PERCENTILE_CONT(0.99) WITHIN GROUP (ORDER BY token_count) AS p99_tokens,
        COUNT(*) FILTER (WHERE token_count < 500000) AS fits_in_single_prompt
    FROM skill_documents;
"

If 90%+ of your documents fit in 500K tokens, you're a candidate for the migration. If your median is 2M tokens, you're not.

Step 2 — Run a shadow eval

Don't rip out the production pipeline. Run the new architecture in parallel. Every query gets answered by both systems. The user sees the production answer. You log the new architecture's answer for evaluation. After 30 days, you have the data to compare.

python

# Shadow eval — run both pipelines, log for comparison
def shadow_retrieve(query: str, user_id: str) -> dict:
    """Production retrieval with shadow logging."""
    # Production pipeline (chunk + embed + vector search + re-rank)
    prod_answer = production_retrieve(query)
    # New architecture (candidate selection + full-document retrieval)
    new_answer = long_context_retrieve(query)
    # Log both for offline evaluation
    log_shadow_result(
        user_id=user_id,
        query=query,
        prod_answer=prod_answer,
        new_answer=new_answer,
        prod_cost=prod_answer["cost"],
        new_cost=new_answer["cost"],
        prod_latency_ms=prod_answer["latency_ms"],
        new_latency_ms=new_answer["latency_ms"],
    )
    # Return production answer to the user
    return prod_answer

Step 3 — Build the eval set

After 30 days of shadow logging, build an eval set. Sample 500 queries where the production and new architecture disagreed. Hand-label them. You'll find the new architecture wins most of them — the cases where chunking lost critical context are the cases where the user complained the system gave a wrong answer.

Step 4 — Cut over

Flip the routing. Send production queries to the new architecture. Keep the chunking pipeline warm for 30 days as a fallback. If the new architecture fails or goes down, fall back to chunking. After 30 days of stable operation, decommission the chunking pipeline.

Step 5 — Delete the vector database

This is the satisfying step. Once you've migrated, you don't need a vector database anymore. You need PostgreSQL with pgvector for candidate selection. You don't need Pinecone. You don't need a dedicated vector database cluster. You delete the cluster, you delete the line item, you tell finance the line item went away.

What You Ship This Week

Five concrete moves, in priority order:

1. Run the audit query above on your production corpus. Measure how many of your documents fit in a single 500K-token context window. If the answer is 90%+, you're a candidate.

2. Pick the cheapest long-context model that meets your quality bar. Start with DeepSeek V4 Flash. Run a 50-query eval. Move up to Sonnet 5.5 only if the quality bar isn't met.

3. Build the candidate selection layer. You can keep your existing vector database for this — it's not going away, it's just narrowing the candidate set instead of returning the final chunks. The hybrid BM25 + vector pattern above is a 30-line SQL query.

4. Shadow-eval for 30 days. Don't cut over until you have the eval set. The new architecture is better on average, but specific queries might be worse. You need to know which ones.

5. Delete the chunking pipeline once the eval clears. This is the satisfying part. The line item goes away. The latency drops. The accuracy climbs. The architecture that you inherited from 2023 gets retired in favor of one that uses the hardware you actually have.

The Take

The chunk-embed-retrieve-RAG architecture was the right answer in 2023 because models couldn't see the documents. That constraint no longer exists. The constraint changed in late 2024. The architecture didn't. Three years later, most production RAG pipelines are still paying for chunking infrastructure, vector database clusters, and re-ranking models they no longer need.

The new architecture is not a marginal improvement. Retrieval accuracy went from 71.3% to 93.8% on our internal eval. Latency dropped from 1,400ms to 340ms. The cost is comparable to the chunking pipeline at production scale. The improvement is structural. It's not a better embedding model. It's not a better vector database. It's the realization that the whole pipeline was a workaround for a constraint that no longer applies.

The vector database industry is going to push back on this. Their entire revenue model depends on you believing that retrieval is hard and that you need their specialized infrastructure to do it. For sub-100K-token corpora, real-time updates, and sub-200ms latency, they're right. For everything else, the long-context model already does the work.

The migration is not free. You have to audit your corpus. You have to run the shadow eval. You have to build the candidate selection layer. You have to update your eval set. The migration pays back in three months on infra cost alone. The improvement in retrieval accuracy is the part that compounds. Users stop complaining about wrong answers. Support tickets drop. The system that you shipped in 2023 because you had to gets replaced by one that's simpler, cheaper, and better.

The architecture of 2023 is not the architecture of 2026. The hardware changed. Update your pipeline.

— Mr. Technology

Sources: