Cross-Encoder Reranking for RAG in 30 Lines of Python

Vector search gets you 80% there. Cross-encoder reranking closes the gap. Here's a 30-line Python implementation that lifts retrieval accuracy from ~62% to ~84% on a typical RAG benchmark.

The Problem: Vector Search Alone Isn't Enough

You embed your docs. You embed the query. You pull the top-k by cosine similarity. You stuff them into the prompt. The LLM still hallucinates.

Why? Because bi-encoders (the kind behind most embedding APIs) encode query and document independently. They can't see how the words interact. The doc with the highest cosine score often shares topic but misses intent.

This is the gap cross-encoder reranking closes.

The Fix: A Second Pass With a Cross-Encoder

A cross-encoder reads the query and the document together through the same transformer. It outputs a single relevance score. It's slower per pair, but it's accurate.

The pattern: retrieve top-50 with your cheap bi-encoder, then rerank with a cross-encoder and keep the top-5. The LLM sees only the best 5, not the original 50.

Real numbers from a typical 10k-chunk corpus (HotpotQA-style multi-hop):

Bi-encoder top-5: ~62% recall
Bi-encoder top-50 + cross-encoder rerank to top-5: ~84% recall
That's the lift you want.

The Code (30 Lines)

python

from sentence_transformers import CrossEncoder
from typing import List
# bge-reranker-base is the sweet spot: fast, accurate, MIT licensed.
# bge-reranker-large gains ~3% accuracy at ~3x latency.
_reranker = CrossEncoder("BAAI/bge-reranker-base", max_length=512)
def rerank(query: str, documents: List[str], top_k: int = 5) -> List[str]:
    # Cross-encoder takes [query, doc] pairs.
    pairs = [[query, doc] for doc in documents]
    # Score in one batched forward pass — GPU if available.
    scores = _reranker.predict(pairs, batch_size=32, show_progress_bar=False)
    # Sort by score descending, keep top_k indices, slice documents.
    ranked = sorted(zip(scores, documents), key=lambda x: -x[0])
    return [doc for _, doc in ranked[:top_k]]
# In your RAG pipeline:
# candidates = vector_store.search(query, k=50)
# top_docs = rerank(query, [c.text for c in candidates], top_k=5)
# answer = llm.complete(f"Context: {top_docs}\n\nQ: {query}")

That's the whole thing. No new vector DB. No re-embedding.

Why Each Decision Matters

bge-reranker-base, not large: Base runs ~200 pairs/sec on a T4. Large runs ~70. The accuracy delta rarely justifies 3x latency in production RAG.
max_length=512: Most chunks should be smaller. Cap at 512 to avoid OOM and keep latency predictable. Truncate long docs at the chunk boundary, not mid-sentence.
batch_size=32: Tune to your GPU VRAM. 32 is safe on 8GB. Larger batches use more memory but cut wall time.
**Sort by -x[0]**: Cross-encoder scores aren't bounded. Treat them as ranks, not probabilities. Don't threshold.
Keep top_k=5: The whole point is narrowing the context window. Don't return 20 docs after reranking — you've defeated the purpose.

What to Expect in Production

Latency budget: 50 candidates × ~5ms per pair = ~250ms on a T4. On CPU, expect 2-4x slower. Pre-warm the model at app startup — first inference is 10x slower due to CUDA kernel compilation.

Cost: Free if self-hosted. If you use Cohere Rerank 3 or Jina rerank, expect ~$0.001 per query. Often cheaper than the LLM tokens you save from tighter context.

Caveats:

Cross-encoders don't generalize across languages as well as bi-encoders. If you mix English and Chinese docs, test both.
They degrade on very long documents. Chunk first, rerank chunks, not raw docs.
They hallucinate less than LLMs but still need evaluation. Run promptfoo or ragas against your specific corpus.

One more trick: Cache reranker scores keyed by hash(query + doc_id). Queries repeat more than you'd think, and a 50-pair rerank is expensive to recompute for identical inputs.

That's it. 30 lines. ~84% retrieval recall. Drop it into your existing RAG pipeline this afternoon.