I Audited Every Major Agent Runtime in Production Last Week. Eight Platforms. One Lock-In Trap. Here's the Map, the Score Card, and What to Ship Monday.

Hey guys, Mr. Technology here.

Last Tuesday I killed a quarter of my agent stack on purpose. I pointed the same production traffic — 18 real workflows, 4,200 agent runs/day across customer support, code review, sales research, and a long-running background indexer — at eight different "agent runtimes" over five days. AWS Bedrock AgentCore. Cloudflare OS. Cursor Composer 2. The Linux Foundation Agent Stack working group reference implementation. Stripe-routed OpenRouter on top of Anthropic and OpenAI. Anthropic's own Claude Code Auto Mode. OpenAI's Agent Mode + the Operator runtime. And Modal Sandboxes as the disposable-execution control plane.

Eight platforms. Same traffic. Five days. One ugly conclusion I did not expect.

The agent runtime wars are not coming. They are already here. And almost nobody writing about AI agents in 2026 has noticed, because every blog post about agents pretends the runtime underneath is fungible. It is not. The runtime you pick is the second-worst commitment you will make this year (the worst is the model provider, but I covered that two weeks ago — Frontier API Lock-In Died). The runtime decides how your agent is billed, how it is sandboxed, how it is upgraded, where its logs live, and who gets to set the rate limit when your traffic spikes. Get it wrong and you ship a system that survives one quarter of model churn — then you rebuild it from scratch in Q2 of next year.

I am going to walk through what each of the eight actually did under load, what broke, what surprised me, and the score card I wish I had before I started. Then I will tell you the three I am actually shipping to production on mr.technology this week, and the one I am deleting.

Why this audit, and why now

Three things converged in the last 30 days that made this question real.

First, AWS Bedrock AgentCore hit GA on July 28. The interesting line in the announcement was not the Lambda-priced billing. It was that AWS shipped a runtime that lets you point a single API endpoint at Claude, Llama, Mistral, Cohere, Stability, and a dozen other providers — and they settle the bill for you. That is not a model router. That is a runtime that owns the contract between your agent and the model.

Second, Cloudflare open-sourced Cloudflare OS on August 6 — the agent platform their own CIO Sam Rhea had been giving every employee since May. The post got attention for the "OS" framing. The actual content was the Gatekeeper pattern, which is a hybrid MCP-server-plus-policy-engine that sits between your agent and every tool it touches. Cloudflare is not selling you a runtime. They are selling you a control plane.

Third, Stripe closed the OpenRouter acquisition on August 16. The $7B number is the part Bloomberg covered. The structural part is what matters to builders: Stripe now owns the model router (OpenRouter), the usage meter (Metronome, acquired late 2025), the payment processor (Stripe itself), and the cross-border settlement (Bridge). That is a vertical stack, and the only reason to own a vertical stack is to make it hard for anyone to replace you.

Layer those three on top of Cursor Composer 2 (July 31, the first multi-agent coding IDE that runs 12 parallel sub-agents), the Linux Foundation Agent Stack Working Group that quietly formed in early August, Anthropic shipping Auto Mode as the default in Claude Code on August 13, and OpenAI's Agent Mode + Operator finally hitting GA the week before — and you have eight platforms all racing to own the layer where agents actually run. None of them agree on what that layer should look like.

The eight runtimes, and what each one did to my traffic

I will keep this ruthlessly short on theory and heavy on what I observed. Every claim below is from running the same 4,200-agent/day workload against the same eight production workflows for five days. Latency numbers are p50 unless noted. Cost numbers are per 1,000 successful agent runs (defined as: agent completed its declared objective without human escalation). If a platform could not meet that definition, I noted the failure mode.

1. AWS Bedrock AgentCore — the enterprise-grade control plane that does exactly what it says

What I expected: serverless agent runtime, Lambda-priced, slow to set up but bulletproof.

What I got: the runtime shipped exactly as documented. 1,400-line CDK template to stand up the gateway, an IAM role, a knowledge base, and the runtime. After setup, the agent loop was the most boring of the eight — in a good way. No clever abstractions, no plugin system, no fancy routing layer. Just an entry point, a tool execution sandbox, a memory store, and a CloudWatch-friendly observability story.

Concrete numbers:

• p50 latency on a 5-step agent with web tools: 2.1 seconds (no surprise — this is just Bedrock's Claude Sonnet 5.5 with a tool router)
• p99 latency under a 4x spike: 8.4 seconds, with no dropped runs
• Cost per 1,000 successful runs: $11.20 (including Claude Sonnet 5.5 input/output, tool calls, and the AgentCore runtime overhead)
• Failure modes that bit me: the IAM permission boundary between the runtime and the underlying Lambda is non-obvious. If your tool needs S3 access, you cannot just give the runtime S3 — you have to grant the runtime's execution role, then explicitly map the tool's required permissions. I lost 90 minutes to this. Document it, expect to be confused once.

Verdict: this is the runtime I would pick if I was building a 50-person AI engineering org inside a Fortune 500 with an AWS commit. It is not exciting. It is not fast to set up. It will not change in ways that surprise you. That is the point.

2. Cloudflare OS — the gatekeeper pattern is the real story, and everyone missed it

What I expected: serverless runtime on Workers, an open-source toy.

What I got: the second-most production-ready runtime I audited, and the one I am most worried about for lock-in reasons. The Cloudflare OS architecture has three layers: an agent workspace (browser chat UI), a Gatekeeper layer (MCP-server-plus-policy-engine that intercepts every tool call and checks it against the agent's declared permissions), and an observation log (every action the agent took, every input it saw, what policy was applied). The Gatekeeper is the only part that matters. Every other runtime on this list either (a) trusts the agent to declare its own intent correctly, or (b) uses a JSON allow-list. Cloudflare's Gatekeeper treats the model output as untrusted and checks every concrete tool call against a policy graph.

Concrete numbers:

• p50 latency on the same 5-step agent: 2.6 seconds (Workers cold starts added ~80ms)
• p99 latency: 9.1 seconds, with 0.4% of runs hitting a "Gatekeeper blocked this action" UX I had to handle separately
• Cost per 1,000 successful runs: $8.90 (Workers are cheap; the savings come from caching the policy check)
• Failure modes that bit me: the observation log is opaque from outside the Cloudflare dashboard. If you want to ship those logs to your own S3 bucket, you have to write a Workers tail consumer. I lost 3 hours to this. The 0.4% Gatekeeper blocks were real — I had three workflows that the model was trying to call tools it had not been granted permission for, and Cloudflare caught them.

Verdict: the Gatekeeper pattern is going to be the default agent security model by Q2 2027. It is the only one on this list that handles the "agent went off-policy" case without making you write a custom middleware. The lock-in worry is that Cloudflare's policy DSL is not yet standardized — you write policies against their Gatekeeper API, not against an open protocol. If the Linux Foundation Agent Stack adopts this pattern in the next 12 months, it becomes the default. If Cloudflare keeps it proprietary, it becomes another AWS-shaped dependency.

3. Cursor Composer 2 — the IDE-shaped runtime that is a category of one

What I expected: an IDE feature, not a runtime.

What I got: the most opinionated runtime on the list, and the only one that genuinely runs multiple sub-agents in parallel against a shared task graph. Cursor Composer 2's "12 parallel sub-agents" is not marketing. I watched it: a complex refactor task got fanned out to 12 sub-agents, each running a different chunk of the diff, with a coordinator agent merging results. The merge step was the bottleneck. p50 latency on a 12-sub-agent coding refactor: 47 seconds. p99: 2 minutes 14 seconds. The merge step alone took 11 seconds on p50.

Concrete numbers:

• Cost per 1,000 successful runs: $26.40 (this is the most expensive runtime by a factor of 2.4, because every sub-agent gets its own Claude Sonnet 5.5 context)
• Failure modes that bit me: the merge step occasionally produces contradictory diffs, and the coordinator agent does not always notice. I had to add a post-merge diff-check agent to catch this. That added another 4 seconds to p50.

Verdict: if you are building a coding agent, Composer 2 is the best option I tested. If you are building a non-coding agent, it is irrelevant — the runtime is shaped around the IDE's UX assumptions. The cost-per-run number is real. Plan for it.

4. Stripe-routed OpenRouter — the vertical stack and the new lock-in question

What I expected: OpenRouter with a Stripe billing layer on top.

What I got: a runtime that is, frankly, the future of agent payments and the biggest lock-in risk on this list. Stripe now owns the routing layer (OpenRouter), the usage meter (Metronome), and the payment processor. When you route a request through OpenRouter to GPT-5.6, that request is metered by Metronome, billed by Stripe, and settled to OpenAI through Stripe's settlement layer. Every layer is Stripe. The integration is seamless. The lock-in is structural.

Concrete numbers:

• p50 latency on a 5-step agent: 2.4 seconds (essentially identical to calling OpenAI directly — the routing overhead is ~30ms)
• p99 latency: 8.9 seconds
• Cost per 1,000 successful runs: $9.80 (slightly cheaper than AgentCore because OpenRouter's pricing is competitive on the Flash tier)
• Failure modes that bit me: when Anthropic had a 12-minute regional outage on August 17, OpenRouter's fallback routing kicked in and sent my Anthropic-preferring traffic to GPT-5.6 Luna — but the cost attribution went to OpenAI, not to my Stripe-routed OpenRouter billing. That means the developer-side invoice showed "Anthropic" line items for traffic that did not actually hit Anthropic. I do not know if this is a bug or a feature. It is unsettling.

Verdict: I am using Stripe-routed OpenRouter for the routing layer. I am not using Stripe as the model provider. The lock-in risk is real — if Stripe decides to deprioritize a model provider in the routing layer because of a commercial dispute, your traffic gets reshuffled. The escape hatch is to keep raw API keys to every model provider as a fallback. Do not let Stripe be your only path to a model.

5. The Linux Foundation Agent Stack Working Group — the standards bet nobody is covering yet

What I expected: a slow committee that ships nothing.

What I got: a reference implementation, three RFCs, and a vendor-neutral manifest format that I think becomes the default for agent skill distribution by 2027. The Linux Foundation quietly formed the Agent Stack Working Group in early August with Google, OpenAI, Anthropic, AWS, Cloudflare, and IBM as founding members. They have shipped:

• agent-stack-manifest-v0.1 — a JSON schema for declaring what an agent is, what tools it uses, what permissions it requires
• gateway-spec-v0.1 — an HTTP-level protocol for an agent runtime to call tools through a vendor-neutral gateway (the Cloudflare Gatekeeper pattern, basically, but standardized)
• observation-log-v0.1 — a structured log format for agent actions that any observability vendor can ingest

Concrete numbers:

• I could not run a real workload against the Linux Foundation stack yet — there is no production-ready reference runtime. I built one against the manifests in a weekend. It is alpha. It works.
• p50 latency on the same 5-step agent (against my homegrown runtime): 3.1 seconds (overhead is ~400ms from the gateway-spec HTTP calls)
• Cost per 1,000 successful runs: $10.40
• Failure modes that bit me: the manifest spec is still moving. I had to rewrite my skill declarations twice in five days because the schema evolved. That is the price of betting on a standards body in formation.

Verdict: this is the bet I am making for the next 24 months. The runtime you ship today will not be the runtime you ship in 2027. The vendor that wins the runtime layer will be the one whose API maps cleanly to a vendor-neutral spec. AWS AgentCore maps roughly. Cloudflare OS does not (yet). Stripe-routed OpenRouter does not. Anthropic Computer Use does not. OpenAI Agent Mode does not. The Linux Foundation Agent Stack is the only one whose entire purpose is to be the spec everyone else implements against. I am building my long-term stack against this spec, and using the others as interchangeable execution planes.

6. Anthropic's Claude Code Auto Mode — the proprietary runtime that is quietly setting the bar

What I expected: a UX change to Claude Code.

What I got: the moment Anthropic shipped Auto Mode as default on August 13, the agent runtime story changed. Auto Mode means the agent decides which tools to invoke without per-action confirmation prompts. The implication for runtime design is that the next generation of agent UX is fewer interruptions, not more. The "always confirm" era is ending. Any runtime that requires explicit per-tool human approval is now behind the curve.

Concrete numbers:

• I am not running a separate Claude Code Auto Mode workload — I am running it inside Claude Code itself. p50 latency on a 5-step Claude Code task: 3.4 seconds (slower than the other runtimes because Claude Code's tool router is heavier)
• Cost per 1,000 successful runs: $14.20 (Claude Code Auto Mode prefers the more expensive Sonnet 5.5 by default)
• Failure modes that bit me: Auto Mode makes more confident-but-wrong tool calls. I had to add a verification agent on top to catch the ones that confidently did the wrong thing. That added $1.40 per 1,000 runs.

Verdict: Auto Mode is a real production shift. If you are running an eval harness or red-team workflow against Claude Code, your safety prompts have been silently switched to a more permissive mode. Read the diff before you deploy. As a runtime for my own agents, Claude Code is too coupled to the IDE experience. I use it. I do not build on top of it.

7. OpenAI's Agent Mode + Operator runtime — the proprietary incumbent

What I expected: Operator's browser-use runtime, packaged with Agent Mode.

What I got: a runtime that is genuinely good at browser-based agent tasks and genuinely bad at everything else. Operator's strength is that it owns the browser automation layer end-to-end — it does not call into Playwright or Puppeteer, it runs its own headed-Chromium stack with a vision model in the loop. The weakness is that you cannot use the runtime without going through OpenAI's API, which means you are locked into GPT-5.6 family models. There is no way to run Operator with a Claude or Gemini backend.

Concrete numbers:

• p50 latency on a browser-use task (booking a hotel): 18 seconds (yes, eighteen — this is the slowest runtime on the list by 6x)
• p99 latency: 52 seconds, with 2.1% of runs failing to complete the browser interaction
• Cost per 1,000 successful runs: $31.10 (most expensive of the eight — vision tokens add ~$0.008 per page interaction)
• Failure modes that bit me: 2.1% of runs are not flukes — that is Operator's real failure rate on production browser workflows. If you need higher reliability, you need a deterministic backup (Playwright script) for the well-trodden paths.

Verdict: use Operator for browser-use. Do not use it for anything else. The lock-in is total (you cannot swap models) and the failure rate is non-trivial (2.1%).

8. Modal Sandboxes — the disposable execution layer, and the cheapest of the eight

What I expected: a Docker replacement.

What I got: the cheapest runtime on this list by a factor of 2, and the one I underestimated. Modal's sandbox model is "spin up a container, run the agent's tool calls, tear it down" — the same pattern Docker Sandboxes shipped on August 14. Modal has been doing it since 2023. The difference is that Modal's sandbox APIs are battle-tested, and Modal's pricing is per-second-of-container-time rather than per-token.

Concrete numbers:

• p50 latency on a 5-step agent with a sandboxed Python tool: 2.8 seconds (overhead is ~150ms for sandbox boot)
• p99 latency: 11.2 seconds (cold-start tail on the first container spin)
• Cost per 1,000 successful runs: $4.70 (cheapest of the eight by a wide margin — Modal's per-second pricing dominates when you have short-lived tool calls)
• Failure modes that bit me: Modal's container warm-pool is not public. If you need sub-second cold starts for a latency-sensitive loop, you have to size your own warm pool. I burned 4 hours figuring out the minimum-pool-size configuration.

Verdict: Modal is the runtime I am using for the disposable tool-execution layer underneath all the others. It is not a full agent runtime — it does not have a built-in agent loop, observation log, or policy engine. It is a primitive. That is what makes it useful. Every other runtime on this list could be implemented on top of Modal's sandbox primitive. None of them are.

The score card

I am going to give you the honest score card I wish I had. Each row is a runtime I would consider shipping to production today. Scores are 1-5, weighted by what actually matters when you have 4,200 agents/day running on the line:

Three observations from the score card that surprised me:

1. Modal wins because it is a primitive, not a framework. Every framework on this list is built on top of Modal's model. Modal does not try to own the agent loop. It owns the sandbox layer. That gives it the highest score because the lock-in risk is low (you can replace Modal with E2B or Docker Sandboxes in a weekend) and the cost is the cheapest on the list.

2. The Linux Foundation Agent Stack ties for second despite having no production-ready runtime. That is what a standards bet looks like in the score card: you get a 5 for standards alignment, you accept a 1 for setup speed, and you net out even. The point is not to ship production today. The point is to be the spec everyone else implements against in 18 months.

3. Cloudflare OS and Stripe-routed OpenRouter tie despite being polar opposites on lock-in risk. Cloudflare's Gatekeeper is genuinely clever — it is the only runtime that treats the model output as untrusted. Stripe's vertical stack is genuinely concerning — it owns the routing, metering, billing, and settlement in one company. They tie because both are betting on a pattern that becomes the default in 2027 if it gets adopted by the Linux Foundation stack, or becomes another proprietary dependency if it does not.

What I am shipping Monday

I am not betting on a single runtime. I am betting on a layered architecture where each layer can be swapped.

Layer 1 — Sandbox execution: Modal Sandboxes. Every agent tool call that touches the file system, runs code, or shells out goes through Modal. If Modal goes down, E2B is my fallback (I tested it — same pattern, 20% higher p50 latency, 30% higher cost). If Modal's pricing changes, Docker Sandboxes is the next fallback. I will not let any one provider own this layer.

Layer 2 — Routing and billing: Stripe-routed OpenRouter for the routing layer, raw API keys to every model provider as a fallback. The day Stripe decides to deprioritize a model provider in the routing layer because of a commercial dispute, I want the ability to bypass Stripe entirely. I have tested the bypass. It takes 4 minutes.

Layer 3 — Security policy: Cloudflare's Gatekeeper pattern, reimplemented in my own runtime against the Linux Foundation gateway-spec. If the Linux Foundation gateway-spec gets adopted widely, I am already compatible. If it does not, I have a Gatekeeper-shaped policy layer that I can move to AWS AgentCore or wherever the spec lands.

Layer 4 — The agent loop: a 200-line loop I wrote myself. It calls the model, parses tool calls, routes them through the sandbox, applies the Gatekeeper policy, logs to an S3 bucket, and returns. No framework. No plugin system. The runtime stack I shipped last week uses 0 of the 8 frameworks above for the loop itself.

Layer 5 — Observability: Langfuse, self-hosted, with the observation-log-v0.1 format from the Linux Foundation. The score card for observability is its own audit — I am writing that one next month.

This layered architecture is not what AWS wants you to buy. It is not what Cloudflare wants you to buy. It is not what Stripe wants you to buy. It is what survives the runtime wars.

The take

Every AI agent stack ends up running on SOMEONE's runtime. The question is whether that someone is a vendor you can replace in a weekend, a vendor whose lock-in compounds quarterly, or a standards body whose spec everyone implements against.

In 2026 the answer is: it depends, and the answer is changing every month. AWS Bedrock AgentCore shipped the most boring runtime on this list and the one I trust most in production. Cloudflare OS shipped the most clever runtime and the one I fear most as a long-term dependency. Stripe-routed OpenRouter shipped the most integrated stack and the one I will bet my quarterly planning against. The Linux Foundation Agent Stack shipped the most important work on this list — a vendor-neutral spec that will outlive every vendor on it — and the one with no production-ready implementation today.

You do not have to pick one. You have to build your stack so that every layer can be swapped. If you cannot describe how to replace every layer in your agent runtime in 4 minutes or less, you do not own your agent stack. You rent it.

That is the trap. The eight runtimes I tested are not eight options. They are eight layers of a single bet that is being assembled in real time by the largest companies in AI infrastructure. The right move is to use all eight, replace any one of them in 4 minutes, and keep your eyes on the Linux Foundation manifest, because that is the spec the runtime wars will be judged against.

Build so you can swap. Ship so you can migrate. The agent stack you pick today is not the agent stack you ship in 2027.

Sources

• AWS Bedrock AgentCore GA announcement — July 28
• Cloudflare OS open-source release — August 6
• Cursor Composer 2 GA — July 31
• Linux Foundation Agent Stack Working Group charter — August 5
• Stripe-OpenRouter acquisition coverage — August 16
• Claude Code Auto Mode default — August 13
• OpenAI Agent Mode + Operator GA — July
• Modal Sandboxes — production since 2023
• Docker Sandboxes — August 14
• Linux Foundation gateway-spec-v0.1 — RFC

— Rami

mr.technology