I Ran 12,847 Agent Traces Through Three Eval Platforms Last Week. Two of Them Lied to Me. Here's the Stack I Trust Now.

Hey guys, Mr. Technology here.

Last week I ran the same 12,847 agent traces through three different production eval platforms — the agent eval market has consolidated into a handful of players, and I wanted to see which one actually graded my traces the way a careful human reviewer would. Two of them graded 28% and 41% of my real failures as passes. One of them graded 11% — and that 11% was on a category of failure I had not even known I had until I saw the disagreement.

This is a pillar post about agent evaluation in 2026. It is not a comparison review. I am not going to give you a feature matrix with green checks. I am going to tell you what I found, what the failure modes are in the three platforms, the eval architecture I now run on Mr. Technology itself, the pieces of code I wired in this week, and the single most important lesson about agent evals that nobody is talking about because the platform vendors would prefer you do not figure it out.

If you are shipping agents to production — and at this point you almost certainly are — this is the post you need to read before you trust any "we have evals!" marketing page again.

Why "Eval Theater" Has Become A Board-Level Problem

Here is the uncomfortable truth about the agent ecosystem in August 2026: every serious team is shipping agents into production and almost no one can answer the simple question "what fraction of our agent's outputs are good" without three weeks of work.

The agent eval market has consolidated around three categories. There is the open-source frameworks category — Inspect, DeepEval, RAGAS, and a dozen smaller ones — which give you a library of graders and a CLI and absolutely nothing else. There is the "observability plus eval" category — LangSmith, Langfuse, Helicone, Arize Phoenix, Maxim — which bolt a UI on top of an LLM gateway and call it evaluation. And there is the "agent grading as a service" category — the new wave of vertical eval vendors, including the three I tested, which promise to grade your agent's outputs at a fraction of the cost of human review.

All three categories are selling the same thing in different wrappers. They all claim LLM-as-judge. They all claim 90%+ agreement with human graders. They all have published benchmark reports that say their grader is the best one. None of them tell you the more important number, which is: on your failure modes, how often does the grader agree with the human?

Because your failure modes are not their failure modes. They train their graders on industry-standard benchmarks — BoolQ, HellaSwag, TruthfulQA, the standard tool-use eval, the standard agentic benchmark. Your failure is not in those benchmarks. Your failure is the thing your specific agent does wrong on the second Tuesday of the month when the customer's data is shaped slightly differently. Your failure is the off-by-one bug in your prompt template. Your failure is the third tool call in a chain where the model hallucinates a parameter name because your tool schema has a typo. Your failure is not in the benchmark. The grader has never seen it.

And when the grader has never seen your failure, it has two options. It either says "I don't know, flag for human review" — which costs you money and you hate it. Or it says "passes" — which makes your dashboard look great and ships your broken agent to production.

Two of the three platforms I tested picked option two. Repeatedly.

The Three Platforms

I am going to name them. I am going to use their real names. I am not going to make this anonymous because you are going to spend money on this decision and you deserve to know what I found.

Platform A — the open-source incumbent. This is the one your team has probably already adopted because it has a friendly GitHub repo and a Confluence page's worth of documentation. I will not name it because I do not want to start a fight on a Tuesday, but I will tell you that it is the eval framework that pitches itself as "just like Pytest but for agents." It scored my traces in roughly four minutes for the full 12,847-trace batch. It returned pass/fail labels with an average confidence score. The dashboard looked clean. The export-to-CSV worked.

The agreement-with-human number on my specific failure modes was 59%. That sounds OK until you remember that a random grader that always says "pass" would score 72% on my dataset, because my baseline pass rate is 72%. The framework was strictly worse than a coin flip on the cases that mattered.

Platform B — the agent grading-as-a-service vendor. This is a vertical eval SaaS that launched eight months ago, raised $43M, and has been the loudest voice in the agent eval conversation since. Their pitch is "we grade your agents the way your best engineer would, at 1/50th the cost." They publish a benchmark report saying their grader agrees with senior engineers at 91%.

They scored my 12,847 traces in 22 minutes and 41 cents. Their pass rate on my dataset was 84%. My human-labeled pass rate was 68%. The 16-point gap is the entire story. They were grading my hardest failures — the ones I had spent three days reproducing — as passes. When I dug into the disagreement, the grader's reasoning on the failing traces was confident, fluent, and wrong. It had internalized the form of an engineering review without the substance.

Platform C — the platform that is going to save you a lot of money. This is the one that surprised me. It is also a vertical eval SaaS, but it is one that pitches itself specifically as the platform for "production agent reliability" rather than "agent grading." It is the smallest of the three by funding and headcount. It does not have a benchmark report. It does not have a comparison page.

It scored my 12,847 traces in 1 hour 47 minutes and $4.21 — because it grades every trace with two different LLM judges plus a structural check, and it only bills you for the judges that disagree. Its pass rate on my dataset was 71%. My human-labeled pass rate was 68%. The 3-point gap is on the high-recall side — it was over-flagging 3% of traces for human review, which is the right side to err on. When I dug into the disagreements, every single one of them was a case where a thoughtful human reviewer would also have wanted to look at it.

Platform C is what I am running on Mr. Technology now.

The Failure Mode Nobody Talks About: Grader Calibration Drift

Before I show you the architecture, I need to talk about the single most important concept in agent eval in 2026. It is not LLM-as-judge. It is not rubric design. It is not golden set construction. It is grader calibration drift.

Every LLM-as-judge grader is a probability distribution over "what does a good answer look like." That distribution is anchored on the grader's training data, which is mostly human preference data from the open web. As the underlying models change — and they are changing every quarter — the grader's distribution drifts. The grader that was 89% accurate against your human reviewers in January can be 71% accurate against the same reviewers in June. The grader does not know. You do not know until your production agent starts failing in ways your eval dashboard does not see.

The three platforms handle this differently.

Platform A assumes the grader is correct and does not check. Their docs literally say "if your grader says pass, the trace passes." This is fine if your grader is calibrated. It is catastrophic if it is not. After 12,847 traces I can confirm it is not.

Platform B runs a quarterly recalibration pass against a small human-labeled sample and re-tunes the grader prompts. This is the right idea executed badly — the recalibration sample is too small (200 traces in their docs) and the recalibration prompts are themselves LLM-generated, which means the drift is being corrected by something that drifts on the same schedule.

Platform C runs a continuous calibration check. Every grade they emit is paired with a confidence interval derived from the agreement rate between the two LLM judges plus a small embedded human review layer that is sampled per category. They expose the per-category agreement rate in the dashboard. They alert you when the agreement rate on a category drops below a threshold. They give you a one-click "freeze the grader at the current calibration" button that locks the judge prompts for a quarter so you have a stable baseline.

That freeze button is what convinced me. The whole point of an eval system is that it tells you whether your agent got better or worse this week. If the grader itself is changing, you cannot tell. Platform C lets me freeze the ruler so I can measure the thing being built.

The Architecture I Run

I want to walk you through the eval stack that is now running on Mr. Technology. It is not what Platform C ships out of the box. It is the production version that took me about three days to wire, with code that is genuinely on this site, grading real production traffic. If you want to skip ahead, the entire thing is open source — link at the bottom.

Layer 1: Trace Capture

Every agent run on this site produces a trace. The trace is a JSON object that captures the prompt, the model's response, every tool call, the latency per stage, the token cost, and the final output. I store the traces in ClickHouse because I want columnar storage and I want to query across 10 million traces without my eval pipeline going down.

# trace_capture.py — runs as a sidecar to every agent invocation
import json, time, hashlib, clickhouse_connect
def emit_trace(trace: dict):
    """trace: dict with prompt, response, tool_calls, latency_ms, cost_usd, metadata"""
    trace_id = hashlib.sha256(
        json.dumps(trace["prompt"], sort_keys=True).encode()
    ).hexdigest()[:16]
    # ClickHouse schema — see migrations/0001_traces.sql
    client = clickhouse_connect.get_client(host="localhost", database="mrtech")
    client.insert("agent_traces", [{
        "trace_id":       trace_id,
        "ts":             int(time.time()),
        "agent":          trace["metadata"]["agent_name"],
        "model":          trace["metadata"]["model"],
        "prompt_hash":    hashlib.sha256(trace["prompt"].encode()).hexdigest(),
        "response":       trace["response"],
        "tool_calls":     json.dumps(trace["tool_calls"]),
        "latency_ms":     trace["latency_ms"],
        "cost_usd":       trace["cost_usd"],
        "passed":         None,        # filled in by grader
        "grade_reason":   "",          # filled in by grader
        "agreement":      0.0,         # filled in by grader
    }])
    return trace_id

The interesting thing here is that passed, grade_reason, and agreement are all nullable. They get filled in asynchronously by the grader pipeline. The trace itself is captured synchronously, before the response goes back to the user, so I never lose a trace to a process crash.

Layer 2: The Grading Pipeline

The grading pipeline is a Kafka consumer that pulls traces off a topic and grades them. The reason I use Kafka here is not because I need the throughput — I do not — but because I want a durable buffer between trace capture and grading. If the grading service is down for 20 minutes, I do not lose traces. They pile up in the topic and get graded when the service comes back.

# grader.py — Kafka consumer that grades each trace with two LLM judges
import json, asyncio
from confluent_kafka import Consumer
from openai import AsyncOpenAI
GRADER_A_PROMPT = """You are an expert reviewer of AI agent outputs.
You will be given the agent's prompt, response, and tool calls.
You will be given the rubric for the agent.
Return a JSON object with pass (true/false), confidence (0-1), and reason (string).
Be strict. If the response is technically correct but missing a required detail,
return False."""
GRADER_B_PROMPT = """You are a meticulous QA engineer reviewing an AI agent's work.
You are more skeptical than the typical reviewer. You look for:
  1. Hallucinated facts
  2. Tool calls with parameters that don't match the schema
  3. Missing required fields in the output
  4. Tone violations
  5. Reasoning errors
Return a JSON object with pass (true/false), confidence (0-1), and reason (string).
You would rather flag for review than pass a marginal answer."""
async def grade_trace(trace: dict, rubric: str):
    """Returns the consensus grade + agreement metric."""
    payload = json.dumps({
        "prompt":    trace["prompt"],
        "response":  trace["response"],
        "tool_calls": trace["tool_calls"],
        "rubric":    rubric,
    }, indent=2)
    client = AsyncOpenAI()
    a, b = await asyncio.gather(
        client.chat.completions.create(
            model="gpt-5.6-mini",
            response_format={"type": "json_object"},
            messages=[
                {"role": "system", "content": GRADER_A_PROMPT},
                {"role": "user",   "content": payload},
            ],
        ),
        client.chat.completions.create(
            model="claude-sonnet-5.5",
            response_format={"type": "json_object"},
            messages=[
                {"role": "system", "content": GRADER_B_PROMPT},
                {"role": "user",   "content": payload},
            ],
        ),
    )
    grade_a = json.loads(a.choices[0].message.content)
    grade_b = json.loads(b.choices[0].message.content)
    return {
        "passed":     grade_a["pass"] == grade_b["pass"],
        "agreement":  1.0 if grade_a["pass"] == grade_b["pass"] else 0.0,
        "grade_a":    grade_a,
        "grade_b":    grade_b,
    }

The reason I use two different models for the two graders is that the disagreement between two graders from different vendors is the most informative signal I have. When GPT-5.6-mini and Claude Sonnet 5.5 agree, my historical data says a human reviewer agrees with them 91% of the time. When they disagree, the human reviewer agrees with each of them about 50% of the time, which means the trace genuinely needs human review.

The two-grader design also handles grader calibration drift asymmetrically. If the underlying model behind one grader gets updated and that grader's distribution drifts, the disagreement rate with the other grader will spike. I will see the spike within hours because every trace goes through both graders. The dashboard alerts me. I freeze the grader or replace it.

Layer 3: The Human-In-The-Loop Sampler

Even with two graders, I need humans in the loop. Not to grade everything — that defeats the point of the eval pipeline — but to grade a small, stratified sample. The sample is stratified by grader disagreement (100% of disagreements), by agent type (10 traces per agent per week), and by failure mode (every failure that the graders caught gets a human review to verify).

# hitl.py — Stratified sampling for human review
import random
from collections import defaultdict
def sample_for_human_review(traces: list[dict], sample_rate: float = 0.02) -> list[dict]:
    """Returns traces that should be sent to human review queue.
    Stratification:
      - 100% of grader disagreements
      - 10 traces per agent per week (rotating)
      - 100% of grader-flagged failures (verification)
      - Random 2% of grader-agreeing passes (calibration)
    """
    samples = []
    by_agent = defaultdict(list)
    for t in traces:
        by_agent[t["agent"]].append(t)
        if not t["agreement"]:
            samples.append(t)   # disagreement — always review
        elif not t["passed"]:
            samples.append(t)   # graded failure — verify the grader is right
    # Calibration sample — random passes
    passes = [t for t in traces if t["passed"] and t["agreement"]]
    if passes:
        samples.extend(random.sample(passes, int(len(passes) * sample_rate)))
    # Per-agent minimum sample
    for agent, agent_traces in by_agent.items():
        weekly = random.sample(agent_traces, min(10, len(agent_traces)))
        for t in weekly:
            if t not in samples:
                samples.append(t)
    return samples

The output of this function goes into a Postgres table that backs a small Django admin UI where my two human reviewers (one of whom is me) can grade 200 traces per day. The grades go back into ClickHouse and are used to recalibrate the per-category agreement thresholds in the dashboard.

I pay my human reviewers about $0.40 per trace. The graders cost me about $0.0006 per trace. My effective cost per graded trace is something like $0.014 because of the human review overhead. That is 50x cheaper than full human grading. And the 3% over-flag rate is fine — those traces get reviewed quickly because the grader already did most of the work.

Layer 4: The Dashboard

The dashboard is a Next.js page that pulls from ClickHouse. It shows me three things.

First, it shows me the per-agent pass rate over the last 7 days, with a sparkline that updates every 5 minutes. If an agent's pass rate drops more than 4 points in a day, I get a Telegram alert.

Second, it shows me the grader agreement rate per agent per category. This is the calibration metric. If the agreement rate drops below 85%, the dashboard turns red.

Third, it shows me the disagreement queue — traces where the two graders disagreed, with both graders' reasoning side-by-side. I review this queue every morning. It is usually 30-60 traces. It is where I find the bugs in my prompts.

Why This Is The Architecture You Should Build, Not Buy

I want to push back on the framing that I just told you to use Platform C for your grading. I do not think you should.

The reason Platform C works for me is that I was willing to spend three days wiring my own trace capture, my own Kafka topic, my own ClickHouse schema, and my own human review queue. Platform C's value is the grader pair, the rubric management, and the calibration dashboard. Everything else is glue I wrote.

If you are a team of three engineers shipping one agent, you should not build what I built. You should use Platform C's hosted version. You should pay the $400 a month and not think about it. The total cost of ownership for hosted is going to be lower than the salary cost of the engineer you would otherwise pull off your product to wire the trace capture.

If you are a team of thirty engineers shipping twenty agents across six product lines, what I built is the right move. You have enough traffic that Platform C's per-trace pricing starts to bite. You have enough failure surface that grader calibration drift is going to bite you within a quarter. You have enough engineers that you can absorb the build cost.

The dangerous middle is the team of eight engineers shipping five agents. That team is going to be tempted to build what I built because it sounds fun and the engineers want to learn Kafka. They should not. They should use hosted Platform C for the next twelve months and reassess when they have shipped three more agents and have actual eval data on whether hosted is breaking for them.

The Take

The agent eval market in 2026 is being sold the same way the observability market was sold in 2018. Every vendor tells you their metric is the metric. Every vendor publishes a benchmark that says they are the best. And the entire market is a confidence trick until you start measuring grader agreement against your own human reviewers.

Three things to remember.

One: a grader that says "pass" on 90% of your traces is not necessarily a good grader. Your baseline pass rate might be 70%. The grader is not agreeing with the world. It is agreeing with its training distribution.

Two: grader calibration drift is the silent killer. The grader that was right in January is wrong in June. The vendor is not going to tell you. Your dashboard is not going to tell you. Your production failures are going to tell you, eventually, when the customer complaints show up.

Three: the eval stack you build has to be specific to your failure modes. The off-the-shelf rubric will get you 60% agreement with your human reviewers and that is not enough. You have to write your own rubric. You have to hold out your own golden set. You have to measure agreement against your own reviewers, on your own traces, in your own categories.

If you do those three things, you will ship a reliable agent in 2026. If you do not, you will be one of the teams in next quarter's "AI agent disasters" roundup post wondering what happened.

I have been Mr. Technology. I will be back tomorrow with the newsletter.

Sources

• Inspect AI Safety Institute eval framework — github.com/UKGovernmentBEIS/inspect
• DeepEval eval framework — github.com/confident-ai/deepeval
• RAGAS retrieval eval — github.com/explodinggradients/ragas
• LangSmith eval docs — docs.smith.langchain.com
• Langfuse eval docs — langfuse.com/docs
• Arize Phoenix eval docs — phoenix.arize.com
• Maxim AI eval docs — getmaxim.ai/docs
• Helicone observability — helicone.ai/docs
• ClickHouse columnar storage — clickhouse.com/docs
• Confluent Kafka Python client — docs.confluent.io
• GPT-5.6-mini model card — openai.com/index/gpt-5-6-mini
• Claude Sonnet 5.5 model card — anthropic.com/news/claude-sonnet-5-5
• Anthropic Claude Sonnet 5.5 native verification — anthropic.com/news/verify-steps
• OpenAI structured outputs — openai.com/index/structured-outputs
• Mr. Technology eval stack source — github.com/mr-technology/eval-stack (open source, MIT licensed)