Agent S3: When Computer-Use Agents Actually Work

Simular's Agent S3 surpasses human-level performance on OSWorld at 72.6%. Here's why the Behavior Best-of-N architecture is the more interesting story than the benchmark headline.

The Number That Shouldn't Surprise Anyone

Agent S3 hit 72.6% on OSWorld. The human baseline is 72.36%. Simular published this in December 2025, and the coverage was predictable: "AI finally matches human-level computer use!" followed by the usual caveats about benchmark fidelity and whether OSWorld actually measures real-world usefulness.

Both reactions miss the point.

The interesting thing about Agent S3 isn't the 72.6%. It's how they got there — and what that architecture implies for the next generation of computer-use agents.

The CUA Bottleneck Nobody Talks About

Computer-use agents (CUAs) have a variance problem. Run the same agent on the same task twice and you can get completely different outcomes. A stray click, a slow-loading modal, an unexpected popup — any of these can compound into failure in ways that have nothing to do with the agent's core reasoning capability.

This is not a new observation. Anyone who's shipped a CUA to production has hit this wall. The interesting question is what you do about it.

Most approaches try to make individual agent runs more reliable: better planning, better error recovery, better prompting. These all help. But they hit diminishing returns faster than expected, because the underlying problem is structural: single-run rollouts have a ceiling.

Behavior Best-of-N: The Actual Innovation

Agent S3 introduces Behavior Best-of-N (bBoN), which takes a different approach. Instead of trying to make one run better, you run multiple rollouts in parallel and select the best outcome.

The "best" selection isn't trivial — you can't just pick the run that completed the fastest or the one that looked most confident. bBoN generates "facts" from each run: structured observations about what the agent actually did and achieved at each step. These facts are then evaluated to determine which run produced the most useful outcome.

The numbers tell the story:

Agent S3 alone (no bBoN): 62.6% on OSWorld
Agent S3 + bBoN (8 parallel runs): 69.9%
Human baseline: 72.36%

That's a 7.3 percentage point jump from adding a selection layer on top of multiple rollouts. No changes to the underlying model. No changes to the agent's policy. Just better use of compute at inference time.

The same pattern holds across environments:

WindowsAgentArena: 50.2% → 56.6% with bBoN
AndroidWorld: 68.1% → 71.6% with bBoN

Generalization improvement, not just benchmark improvement.

Why This Architecture Matters

The conventional wisdom in the agent research community is that you solve the CUA reliability problem by improving the agent's decision-making: better world models, better action priors, better recovery strategies. That's the top-down approach.

bBoN is a bottom-up approach. It says: the variance in CUA performance is irreducible at the single-run level. Instead of fighting that variance, route around it — use multiple rollouts to sample the variance and select the best outcome.

This is structurally similar to how AlphaGo works: it doesn't try to make a single game-playing decision perfectly. It samples many possible game trees and selects the path with the best expected outcome. The analogy isn't perfect — bBoN is evaluating outcomes, not simulating futures — but the philosophical shift is the same: from "make each decision better" to "use compute to hedge against decision variance."

The practical implication is that bBoN is a general technique. It doesn't require understanding what the agent is doing or why. It just needs a way to evaluate outcomes. That makes it broadly applicable to any agentic task where you can define what "success" looks like well enough to compare two runs.

What's Actually New in Agent S3

Beyond bBoN, Agent S3 has a few other值得注意 components:

Simplified framework over S2. The S2→S3 transition reduced framework complexity while improving performance. S2 already hit 48.8% on OSWorld (a significant jump from S1's 20.6%). S3's simplification suggests the earlier framework had architectural overhead that wasn't paying its way.

Native coding agent. S3 includes a dedicated coding agent component. This isn't just "the agent writes code" — it's a structured sub-agent that handles the specific challenges of code generation and execution within the broader computer-use task. The coding agent can write, execute, and iterate on code independently while the parent agent coordinates higher-level task progress.

Cross-environment generalization. The fact that bBoN improves performance on WindowsAgentArena and AndroidWorld, not just OSWorld, suggests the technique transfers. This is the difference between "we tuned for this benchmark" and "we found something general."

The Caveats Worth Taking Seriously

I want to be honest about the limitations here.

OSWorld is still a benchmark. Real computer work involves ambiguity, stakeholder conflict, incomplete information, and tasks that can't be cleanly defined. Benchmarks reward capability on well-defined tasks. The gap between benchmark performance and real-world usefulness is real and worth acknowledging.

72.6% means 27.4% failure rate. For casual use cases, that's probably fine — the user can intervene when the agent gets stuck. For unattended automation in high-stakes environments (healthcare, finance, legal), a 27% failure rate is a blocking issue. The path from "matches human average" to "reliable enough to trust without oversight" is not a benchmark question; it's an engineering question.

Parallel rollouts are expensive. bBoN with 8 parallel runs is 8x the compute of a single run. The technique only makes economic sense when the cost of failure exceeds the cost of the additional compute. For cheap tasks, this math doesn't work. For expensive tasks (filling out complex regulatory forms, navigating intricate enterprise software, long-horizon data entry), it probably does.

What This Means for Builders

If you're building computer-use agents today, bBoN should be on your radar as a reliability technique. Not as a silver bullet — 8x compute overhead is a serious cost — but as a tool in the kit for high-value tasks where reliability matters more than unit economics.

The more important lesson is architectural: the assumption that you solve agent reliability through better individual decision-making is hitting its limits. The next generation of production agent systems will need to think about inference-time compute allocation the way the last generation thought about model selection. bBoN is an early example of that shift.

The numbers from Agent S3 suggest we're in the phase where these techniques are starting to work. The question now is who turns them into production systems that are reliable enough to actually deploy at scale.

Agent S3 paper on arXiv: arXiv:2510.02250 | GitHub: simular-ai/Agent-S | OSWorld benchmark at os-world.github.io