Haystack Is the Pipeline-First RAG Framework That Survived the LangChain Era

By the end of 2024 every RAG blog post was a LangChain LCEL diagram. By early 2025 they were all DSPy modules. Half are in rewrite hell. The one that didn't need a rewrite — because it didn't follow the hype to begin with — is Haystack, deepset's Python framework quietly compounding since 2020. In a category that ships four hype cycles a year, "boring and durable" is the most underrated feature in software.

What it actually is

Haystack is a Python framework for building LLM applications as directed acyclic graphs of typed Components wired through a Pipeline runtime. You define nodes — retrievers, rankers, prompt builders, generators, routers — connect them by their declared input/output sockets, and let the runtime schedule execution and stream outputs. Each node is a class with def run(...) -> dict and a ComponentSchema. No chain to monkeypatch. No implicit global state.

Two things fall out of this design that other frameworks still struggle with:

1. Branching is first-class. A pipeline can route documents through a transformer, send the slow path to a reranker, and merge both feeds into one generator. Conditional routing, error fallbacks, and AsyncPipeline parallelism are not hacks — they are the runtime contract. 2. Components are inspectable. pipe.show() prints the DAG, pipe.draw() renders Mermaid, and the whole pipeline serializes to YAML. You can diff a retriever swap in git.

A real pipeline in twenty lines

Hybrid retrieval — BM25 plus a dense retriever, fused with reciprocal-rank-fusion, reranked, then answered by Claude. Branching, fusion, and a typed generator, no glue code:

from haystack import Pipeline
from haystack.components.builders import PromptBuilder
from haystack.components.generators import AnthropicGenerator
from haystack.components.rankers import TransformersRanker
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
from haystack.components.retrievers import QdrantHybridRetriever
from haystack.components.joiners import DocumentJoiner
pipe = Pipeline()
pipe.add_component("bm25", InMemoryBM25Retriever(document_store=bm25_store, top_k=10))
pipe.add_component("dense", QdrantHybridRetriever(document_store=qdrant_store, top_k=10))
pipe.add_component("join", DocumentJoiner(join_mode="reciprocal_rank_fusion"))
pipe.add_component("rerank", TransformersRanker(model="BAAI/bge-reranker-base", top_k=4))
pipe.add_component("prompt", PromptBuilder(template=ANSWER_TEMPLATE))
pipe.add_component("llm", AnthropicGenerator(model="claude-sonnet-5"))
pipe.connect("bm25.documents", "join.documents")
pipe.connect("dense.documents", "join.documents")
pipe.connect("join.documents", "rerank.documents")
pipe.connect("rerank.documents", "prompt.documents")
pipe.connect("prompt.prompt", "llm.prompt")
result = pipe.run({
    "bm25": {"query": query},
    "dense": {"query": query},
    "prompt": {"question": query},
})

The LangChain equivalent was a RunnableParallel lambda, two RunnablePassthrough.assign adapters, a ContextualCompressionRetriever around CohereRerank, and three custom functions because Runnable outputs didn't match. The Haystack version is type-checked, serializable, and inspectable as a graph; the LangChain version is a sentence you keep alive in your head.

What's genuinely good

• Stable serialization. A pipeline exports to YAML, diffs in git, reloads on a different machine. I have shipped RAG systems that survived two LLM swaps and three embedding-model swaps without code changes.
• Component ecosystem. Forty-plus first-party integrations: Pinecone, Weaviate, Qdrant, Elasticsearch, OpenSearch, Milvus, pgvector, Cohere, Anthropic, OpenAI, Bedrock, HuggingFace, Ollama, Voyage. The component model is what made deepset's enterprise business work.
• Typed agent layer. v2.x added Agent and Tool components with strict schemas. Less magic than CrewAI, less brittle than LangGraph, and the same Pipeline runtime — so an agent can branch into a sub-pipeline and back.

What's annoying

• Docs lag the API. ComponentSchema, v2 Agent primitives, several connection patterns all shipped before the docs caught up. You will read source code.
• Async story is incomplete. AsyncPipeline exists but several components still block. You find this on the first 10k-RPS load test.
• Verbose logging. Every component emits a dict trace. Turn it off in prod or drown in JSON.

vs. the field

LangChain is the right pick when you want maximum plug-and-play and don't care about long-term stability. LlamaIndex is the right pick when retrieval is the whole app. DSPy is the right pick when your bottleneck is prompt quality. Haystack is the right pick when the system is the product — branching, routing, retriever fusion, swap-in components, a runtime you can debug at 11pm.

When to reach for it, when not to

• Reach for Haystack for multi-stage RAG (BM25 + dense + fuse + rerank), hybrid retrieval, cross-store routing, or the same code running as a Python object, a FastAPI endpoint, and a deepset workflow.
• Skip it for a one-shot prompt over one PDF (use pypdf and Anthropic()). Skip it for chat-with-an-LLM — it is a pipeline framework, not a chatbot framework. Skip it if you refuse to read source code; the types are good, the docs are mid.

Take

Haystack is the framework I would pick if a founder walked in tomorrow and said "we need a RAG system that survives two embedding-model swaps, three LLM swaps, and one agent overhaul without a rewrite." It is not the one I would pick if the founder said "we need a chatbot by Friday." Most teams over-index on Friday and under-index on the rewrite. Pick the tool whose pain is borne at the right point in the project lifecycle. With Haystack, that point is eighteen months in.

— Mr. Technology

*Packages: haystack-ai 2.x. Pipeline runtime: directed-acyclic, async-capable, branching via pipe.connect(). Component-level introspection via ComponentSchema and pipe.show(). Hybrid retrieval: combine BM25 retriever + dense retriever → DocumentJoiner(join_mode="reciprocal_rank_fusion") → TransformersRanker. Storage backends: in-memory, Elasticsearch, OpenSearch, Qdrant, Weaviate, Milvus, pgvector, Pinecone, Chroma. Streaming: pipe.run(...) returns component-by-component dicts; use pipe.stream(...) for incremental output.*