Agent Benchmarks Are A Three-Card Monte Game And You Are The Mark

I will die on this hill: SWE-Bench, tau-Bench, GAIA, OSWorld, WebArena, and every other "agent benchmark" published between 2024 and mid-2026 is a three-card monte operation. The mark is your platform team. The grifter is the lab presenting the number. The scoreboard is rigged in ways obvious to anyone who has shipped an agent to production. Stop citing public benchmarks. Allocate four weeks to build your own.

The First Trick: The Hindsight Gap

When a vendor posts "Claude Opus 4.5 solved 73.2% of SWE-Bench Verified," every engineer hears "the model can do 73% of real software work." That is not what the number means. It means the agent resolved 73.2% of a curated set of ~500 GitHub issues from twelve popular Python repos where the ground-truth answer already existed in the commit history and the test cases have been on the public internet for years. The benchmark ships the answer key in assertions the model can study, paraphrase, or memorise.

Move the same agent to your private repo against your private tests on last quarter's tickets, and the number collapses. Every team I have watched run a real eval has seen it happen in a week — the 73 becomes a 38, and by Friday someone is rebuilding the dashboard. The public number is not a procurement signal. It is a familiarity score.

The Second Trick: The Scaffolding Wash

Almost no published number is a "model" number. It is a model-plus-scaffolding number. The 72.4% on tau-Bench retail is Anthropic's model inside Anthropic's preferred scaffolding inside Anthropic's hand-tuned system prompt inside Anthropic's tool descriptions inside a two-month retry policy. Swap in an open-weights model with the same scaffolding and the score moves. Swap the scaffolding and it moves again. The variance is the craft.

A leaderboard is not a model score. It is one team's specific implementation across tools, prompts, retries, and parsers. Switch any single component and the number moves more than swapping the underlying weights. Vendors know this. Procurement teams do not.

The Third Trick: The Train-On-The-Test Pipeline

A non-trivial fraction of SOTA gains on agent benchmarks in the last eighteen months is training-set contamination the labs will not publicly disclose. The convenient excuse is "we ran RL on synthetic tasks that look like the benchmark." The inconvenient truth is that those synthetic tasks sometimes paraphrase benchmark items, regenerate them with new variable names, or are the benchmark with a comment swapped. When a model jumps twelve points on a public eval in a single release, ask whether the eval set changed. If it did, the number is a new test. The practice has become standard. The disclosure has not.

Why Your Eval Set Is The Only Score That Matters

Your private benchmark — tied to your data, your tools, your failure modes — is the only number worth optimising against. Public benchmarks are a sanity check, nothing more. The procurement signal you want is success rate on two hundred hand-scored tickets from last quarter, run with your system prompt against your tools and data, executed twice a week on a fixed cadence. That number is what your CFO wants. That is how you decide between vendors.

If you do not have that benchmark yet, that is the work for the next four weeks. Pull two hundred real tickets, score them by hand, run every candidate model and scaffolding against them. Most teams discover in the same week that their preferred model is not their best model and the public rankings had almost nothing to do with either outcome.

The Take

Public agent benchmarks are a rigged game that everyone running them knows is rigged, played for an audience that has not caught on. Labs publish the numbers because they raise rounds on them. Enterprise teams cite them because they have no other artifact to show leadership. The honest move is to build the eval set, run the bake-off on your real work, and stop pretending a scraped leaderboard number is a procurement decision.

Stop citing benchmarks in vendor decks. Start citing your own.

— Mr. Technology