Promptfoo Is the Open-Source Eval Framework That Actually Runs in CI, and Stop Calling Your LLM "Production-Ready" Until It Has
Promptfoo Is the Open-Source Eval Framework That Actually Runs in CI, and Stop Calling Your LLM "Production-Ready" Until It Has
Most teams I talk to in 2026 have a working LLM demo, a held-out test set, and a vibe. That is it. No automated evals. No regression gate. No way to know whether the prompt rewrite they shipped Tuesday made the answer better or worse. They are running a model in production the way a chef runs a kitchen without a thermometer — by feel, by intuition, by the taste of the line cook. It is malpractice. Promptfoo is the thermometer.
Promptfoo is the MIT-licensed, open-source LLM evaluation and red-teaming framework built by Ian Webster and a small contributor team. ~6,500 GitHub stars, ~150 contributors, used in production by Anthropic, Shopify, Discord, and Brex. The thesis is the only one that has ever worked in software engineering: define what good means, measure it on every commit, fail the build when it regresses. Apply it to prompts, models, retrieval pipelines, agents — the framework does not care which. It cares that the number is reproducible, and that the number goes up when you ship and down when you break something.
Hey guys, Mr. Technology here.
What Promptfoo Actually Is
Promptfoo is four things glued together by one config file.
1. An eval runner. Declare prompts, tests, and providers. Promptfoo runs the matrix, scores every output with the assert block you wrote, and writes a row per case to its local web UI. A 50-case eval across two prompts and three providers is a small YAML file that finishes in three minutes.
prompts:
- 'Summarize the incident in two sentences: {{incident}}'
- 'You are a senior SRE. Write a postmortem for: {{incident}}'
providers:
- openai:gpt-4o
- anthropic:messages:claude-opus-4-7
- ollama:chat:llama3.3
tests:
- vars: { incident: file://fixtures/payment-stall.md }
assert:
- type: contains
value: 'p99 latency'
- type: llm-rubric
value: 'Names the root cause in the first sentence'
- type: cost
threshold: 0.01Run npx promptfoo eval. The dashboard at http://localhost:15500 lights up with pass rates, latency, cost per case, and a side-by-side diff between the two prompts across all three models. The five-minute setup is the whole pitch. If you have ever spent a week arguing about prompt wording in a Notion doc, this is the answer to that week.
2. A red-team engine. promptfoo redteam run is the part that scares CISOs. You point it at a target — OpenAI chat endpoint, Anthropic messages route, local gguf, any REST URL that speaks chat — and it runs ~30 attack categories drawn from the OWASP LLM Top 10, MITRE ATLAS, and a curated probe library. Jailbreaks, prompt injection, PII leakage, hallucinated tool calls, toxicity, business-disallowed-content, excessive agency. The default config catches more than most paid red-team platforms charge $30k a year for.
3. A dataset and experiment layer. promptfoo eval --cache makes evals deterministic against pinned model versions. Datasets are version-controlled YAML. Side-by-side diffs between commits show you exactly which cases regressed. This is the CI gate that should run on every prompt change in 2026 and almost never does.
4. A local-first dashboard. No SaaS lock-in. The viewer runs in-process. The data stays on your machine. The cloud product exists for shared workspaces, but the open-source project is not a trap.
What It Beats, What It Loses To
Versus DeepEval. DeepEval is the Pythonic pytest-shaped framework — great for ML engineers doing dataset work. Promptfoo is the deployment-shaped framework — config files, CLI, dashboard, red-team. If you are a platform team wiring evals into GitHub Actions and gating deploys, Promptfoo is the better tool and it is not close. Decision matrix: test directory or build pipeline.
Versus RAGAS. RAGAS owns the deep-RAG-eval slot — context relevance, faithfulness, answer relevance. Promptfoo covers the same metrics through llm-rubric and context-relevance but does not match RAGAS at depth. Use RAGAS for the deep eval. Use Promptfoo as the deploy gate.
Versus Langfuse / Arize Phoenix evals. Both are observability products with eval bolted on. Neither is a CI gate out of the box. Promptfoo is the gate. The observability tools are what you read when the gate fails.
What I Don't Love
The default rubric is LLM-as-judge and the LLM is your LLM. Most llm-rubric assertions call the same provider family as the target. If the target is hallucinating, the judge is hallucinating too. Pin the judge to a different provider family — Anthropic judging OpenAI, GPT-5 judging Claude — and keep at least one deterministic assertion (contains, regex, is-json, cost, latency) on every case. Pure-LLM-judge evals are theater.
The red-team plugins are good, not great. They cover OWASP LLM Top 10 but miss the multi-turn attack chains Microsoft PyRIT was built for. For single-shot scanning, Promptfoo is the answer. For multi-turn orchestration, wire PyRIT alongside it.
The Python ergonomics are an afterthought. CLI is Node. The Python SDK exists, but the docs lead with YAML. Jupyter-first teams will grumble for an afternoon.
The Take
Promptfoo is the eval framework that actually runs in CI. Not "could run in CI." Not "we will wire it up next sprint." Runs today, gates the deploy, fails the build when a prompt rewrite drops faithfulness by 4%, gives the on-call engineer a URL to the diff. That is the only meaningful evaluation in 2026 — the one that blocks the merge.
If you are shipping LLM features and you are not running Promptfoo in your build pipeline, you are not shipping AI. You are shipping a vibe with a version number. 6,500 stars, MIT-licensed, the eval framework Anthropic itself uses to ship. Install it. Wire it up. Gate the deploy. Read the red-team report before the CISO does.
— Mr. Technology
*Promptfoo: github.com/promptfoo/promptfoo — MIT-licensed, ~6,500 GitHub stars, ~150 contributors, used by Anthropic, Shopify, Discord, Brex. CLI: npx promptfoo eval / npx promptfoo redteam run. Local dashboard on port 15500. 30+ red-team plugins covering OWASP LLM Top 10 and MITRE ATLAS. CI integration: GitHub Actions, GitLab, CircleCI, Buildkite. Provider support: OpenAI, Anthropic, Azure, Bedrock, Vertex, Mistral, Groq, DeepSeek, Ollama, llama.cpp, gguf, custom HTTP.*
Sources: