DeepEval Is the Pytest-Native LLM Eval Framework That Turns Your Coding Agent's Vibes Into Regressions You Can Block in CI, and Promptfoo Is Not the Only Answer

Hey guys, Mr. Technology here.

A month ago I wrote the Promptfoo post. Half the DMs I got back were "OK fine, but my CI is Python — I want my evals next to my pytest." The answer has been at the top of the LLM eval leaderboard for eighteen months and it is DeepEval — confident-ai/deepeval, ~14,000 GitHub stars, Apache 2.0, 250+ contributors, v4.0 shipped May 2026, and the only LLM eval framework that treats your test suite the way pytest has been treating your unit tests since 2008. If Promptfoo is the YAML-driven deploy-shaped framework for platform teams, DeepEval is the Pythonic agent-trace framework for SDETs and ML engineers. They are not competitors. They are complements. Run both.

What DeepEval Actually Is

DeepEval is a pytest-native LLM evaluation framework with 60+ research-backed metrics, multi-turn conversation eval, multimodal (text + image + audio), agent trace scoring, and a 4.0 release that re-wrote the story around coding-agent feedback loops. The mental model: **every eval is a pytest test that fails or passes a metric threshold**. Same CI runner. Same fixtures. Same parametrize. No new build system, no new YAML dialect.

from deepeval import assert_test
from deepeval.metrics import (
    FaithfulnessMetric, HallucinationMetric,
    TaskCompletionMetric, ToolCorrectnessMetric,
)
from deepeval.test_case import LLMTestCase
from deepeval.tracing import trace
@trace(name="refund_agent")
def run_refund(order_id, reason):
    policy = retrieve_policy(order_id)
    if not policy.eligible(reason):
        return "Refund not eligible per policy."
    process_refund(order_id, policy.amount)
    return draft_response(order_id, reason, policy.amount)
def test_refund_agent_happy_path():
    case = LLMTestCase(
        input="Refund order #9281, customer reported defective item",
        actual_output=run_refund("9281", "defective"),
        retrieval_context=["Returns accepted within 30 days for defective items."],
        tools_called=[{"name": "process_refund", "arguments": {"id": "9281"}}],
    )
    assert_test(case, [
        FaithfulnessMetric(threshold=0.9),
        TaskCompletionMetric(threshold=0.7),
        ToolCorrectnessMetric(threshold=1.0),
    ])

Run deepeval test run tests/. The 4.0 TUI opens inline. You see RETRIEVER, TOOL, LLM spans, per-span metric scores, per-span failure reasons, and an overall pass/fail per case. The same test runs in GitHub Actions with deepeval test run. Pytest is the build system. DeepEval is the metric library. That is the whole pitch.

What 4.0 Added That Matters

Three things in 4.0 are why I am writing this. First, the agent-native eval loop for coding agents. Cursor, Claude Code, Codex, OpenCode, any framework with a CallbackHandler — the agent sees metric failures and reasoning inline, and runs patch → eval → retry without a human. Your coding agent reads "faithfulness 0.64 on RETRIEVER span, fix the scope filter" and patches the function. Second, the local-first TUI. deepeval view opens an interactive trace explorer in your terminal. Data never leaves your laptop unless you ship it to Confident AI. Third, native multimodal trace support. Image and PDF attachments flow through LangChain content blocks as first-class placeholders, and multimodal detection recognizes both. Same LLMTestCase, same metric, same CI gate, regardless of modality.

What It Beats, What It Loses To

Versus Promptfoo. Promptfoo is the right tool when your eval author is not the same person as the app author — YAML, CLI, dashboard, language-agnostic. DeepEval is the right tool when your eval author writes Python and wants tests next to the code. Both gate the deploy. Both have red-team support (DeepEval's scanner is younger but covers the same OWASP LLM Top 10). Run DeepEval for the agent trace. Run Promptfoo for the matrix sweep. This is not a debate, it is a stack.

Versus RAGAS. RAGAS owns the deep-RAG-eval slot for academic research. DeepEval covers the same RAG metrics but is broader — chatbot, agentic, multimodal, safety. If you are shipping an agent, DeepEval wins. The framing is "the eval framework for everything," not just retrieval.

Versus Braintrust, Arize Phoenix, LangSmith. All three are hosted observability platforms with eval bolted on. DeepEval's story is the same as Promptfoo's — your test file, your runner, your data. For an engineering team that wants to gate the deploy, DeepEval is the better choice. For a cross-functional team, Confident AI is the paid answer.

What Is Actually Wrong

The judge is your LLM. Default GEval, Faithfulness, AnswerRelevancy all call an LLM to score. Same caveat as Promptfoo — pin the judge to a different provider family from the target and keep at least one deterministic assertion per case. Pure-LLM-judge evals are theater.

The agentic metrics are slow. TaskCompletionMetric and ToolCorrectnessMetric re-invoke the agent or replay the trace through a judge LLM. On a 200-case agent suite expect 8 to 15 minutes where Promptfoo finishes in 90. Pin your CI timeout, parallelize with pytest-xdist, budget for it.

The Confident AI cloud is the trap. The open-source project is free forever and Confident AI starts at $19 per month. Same pattern as every open-source project with a hosted tier — the code is the code, the cloud is the cloud, the lock-in is what you let creep in.

The Take

The eval stack in 2026 is not a single framework. It is two frameworks plus a dashboard. DeepEval owns the Pythonic agent-trace eval and ships multimodal in 4.0. Promptfoo owns the YAML matrix sweep and the red-team report. Run both. Gate the deploy with both. That is the eval stack that catches regressions before your users do. Anything less is shipping a vibe with a version number.

If your CI is Python and your eval author is the same person as your application author, install DeepEval. Drop the metrics on your existing test cases. Run the suite. Read the TUI. Ship the patch. 14,000 stars, Apache 2.0, pytest-native, the eval framework Anthropic should have built and didn't.

— Mr. Technology

*Repo: github.com/confident-ai/deepeval — Apache 2.0, ~14,000 GitHub stars, 1,300+ forks, 250+ contributors, v4.0 shipped May 2026, ~60+ metrics across RAG / chatbot / agentic / multimodal / safety. Pytest-native, CLI deepeval test run, local TUI deepeval view. 10+ native integrations: LangChain, OpenAI, Anthropic, LiteLLM, PydanticAI, CrewAI, LlamaIndex, DSPy, Google ADK, custom agents. Stack: Python 3.9+, pytest, no heavy deps. Install: pip install deepeval. Cloud: confident-ai.com (Confident AI platform, $19/month for teams). License: Apache 2.0.*