Reasoning Models Just Killed Your Cost Predictability — Here's the Observability Stack That Fixes It

Hey guys, Mr. Technology here.

Let me give you a number. Last week a Series B fintech I work with shipped an autonomous customer-support agent on gpt-5.6. It worked. The metrics looked fine. The first week of production, the bill was $11,400. The second week, $47,800. The third week, $112,000. They shut it down. The agent wasn't even popular — maybe 800 conversations a day. The reason: somebody in product flipped the model from gpt-5.6 to o3-pro for a "smarter answers" A/B test, and the agent started doing five-to-fifteen-minute reasoning passes per ticket. Each conversation averaged 380,000 reasoning tokens. Nobody noticed because nobody was watching.

This is the production failure pattern that reasoning models created in 2026, and the teams that didn't build observability into their agent stack from day one are now discovering what $40K-a-night surprises feel like. The cost model that worked for gpt-4o and claude-sonnet-4 doesn't work anymore. Reasoning models burn 10x to 100x more tokens per task than chat models, the variance between tasks is enormous, and the only way to know what you're actually paying for is to instrument every step.

I'm going to walk you through what changed, why traditional APM doesn't catch it, and the specific observability stack — OpenTelemetry-native, vendor-neutral, and not owned by a model provider — that fixes it. If you ship agents in production and you aren't already instrumenting them, this is the most important thing you'll read this month.

Why Reasoning Models Broke the Old Cost Model

Before we get into the stack, let's be precise about what changed.

A chat model like gpt-4o or claude-sonnet-5 produces one token at a time in response to your prompt. You send 1,000 tokens in, you get 200 tokens back, you pay for 1,200 tokens. The cost per conversation is bounded by your input and your max output tokens. Easy to predict. Easy to budget.

A reasoning model like o3-pro, claude-opus-5-thinking, deepseek-r2, or grok-4.5-reasoning does something different. It runs an internal chain-of-thought at inference time. The "thinking" — the back-and-forth reasoning, the self-critique, the candidate answer generation — happens before the visible output and is billed as reasoning tokens. The reasoning tokens are usually opaque (you can't see them in the response, only the count), they're usually 5x to 50x larger than the final output, and they vary wildly by task.

Here's what that looks like in practice. Same prompt to the same agent, same model, same week:

• Easy task ("refund this customer's $40 order"): 8,200 reasoning tokens, $0.18
• Medium task ("explain why their subscription was charged twice"): 41,000 reasoning tokens, $0.92
• Hard task ("negotiate a custom enterprise contract"): 380,000 reasoning tokens, $8.50

That's a 47x cost variance on the same model, same agent, same week. If your agent handles 800 conversations a day and 5% of them hit the "hard" bucket, your monthly bill swings between $4,200 and $148,000 depending on what your users actually ask. That's not a budgeting problem. That's a roulette wheel.

Worse, reasoning model providers don't price reasoning tokens the same way as output tokens. OpenAI bills o3-pro reasoning tokens at the output rate but counts them separately. Anthropic bills Claude Opus 5 thinking tokens at 1.5x the output rate. DeepSeek-R2 bills them at parity with output. Grok 4.5 reasoning has a per-call surcharge on top of token costs. So your "cost per token" number is wrong the moment you switch models. If you're computing cost in your head from a single rate sheet, you're already wrong.

The Real Failure: APM Tools Don't See This

Here's where it gets worse. Most teams already have application performance monitoring in place — Datadog, New Relic, Honeycomb, Grafana, Sentry. They think they're covered. They aren't.

Traditional APM instruments HTTP requests, database queries, and function calls. It sees your agent as a black box that calls an LLM. It logs "POST /v1/chat/completions — 2.4 seconds — $0.42." That tells you latency and total cost. It tells you nothing about:

• How many reasoning tokens were burned
• Which step in your agent loop used the most tokens
• Why one conversation cost 50x another
• Whether the agent is stuck in a retry loop
• Which user is burning your budget
• Whether a prompt injection made the agent run 200 tool calls

You need agent observability — a different kind of monitoring, built around the concept of a generation, a tool call, a retrieval, and an agent step. The data model is OpenTelemetry-native, but the spans you care about are llm.generation, llm.tool_call, agent.step, and agent.loop. None of the legacy APM vendors have built these primitives properly. They're built for HTTP, not for agents.

I watched a YC fintech last quarter spend $78,000 over a weekend because their Datadog dashboard told them "the LLM endpoint is slow." It was. The endpoint was averaging 47 seconds per call. That was the only signal. They didn't know that 80% of those 47-second calls were a single agent step in a planner loop, retrying the same tool call 14 times because the tool was returning a 503. APM told them "slow." Agent observability would have told them "step 4 of the planner, called http_request 14 times, burned $3,400 in reasoning tokens, the 503 was a transient Redis timeout." That's the difference.

The Stack That Actually Works

After watching a dozen teams instrument their agent stacks in the last six months, here's what the production-grade observability layer looks like in mid-2026.

Layer 1: OpenTelemetry as the Wire Format

Don't pick a vendor lock-in observability platform. Pick OpenTelemetry. OTEL has had native LLM and agent instrumentation since the semantic-conventions extension stabilized in early 2026, and every observability vendor I'm about to mention can ingest OTEL spans. If you instrument once with the OTEL SDK, you can switch backends tomorrow without rewriting code. That's not a "nice to have" in 2026 — that's table stakes, because the observability vendor you pick today will be a bad decision in 18 months. The space is moving too fast.

The data model you need:

• gen_ai.conversation.id — unique ID per agent run, propagated across spans
• gen_ai.agent.name and gen_ai.agent.step — which agent, which step in the loop
• gen_ai.request.model — the model being called (yes, include the version — o3-pro-2026-05 is meaningfully different from o3-pro-2026-07)
• gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.usage.reasoning_tokens — token breakdown
• gen_ai.tool.name and gen_ai.tool.call.duration_ms — tool calls
• gen_ai.cost.usd — computed cost (never trust the model provider's billing — compute it yourself from token counts and your negotiated rates)
• gen_ai.prompt.feedback — user feedback if collected
• gen_ai.conversation.user_id and gen_ai.conversation.tenant_id — for cost attribution

That's the vocabulary. Now you need somewhere to send the spans.

Layer 2: Pick a Backend (Don't Build One)

Five serious options as of mid-2026. Pick one based on what you actually need.

Langfuse — OSS-first, OTEL-native, the de facto standard for agent observability in 2026. Self-hostable, has a generous free cloud tier, ships with prompt management, evaluation, and cost tracking built in. The one I'd pick if I were starting today. Has the largest community, the best docs, and the most active Discord. The SDK is thin and the OTEL exporter is first-class — you can ship spans from any OTEL-aware framework without using the Langfuse SDK at all.

Helicone — Lightweight proxy that sits between your agent and the LLM API. Cheapest path to cost tracking and caching. Less feature-rich than Langfuse but faster to set up. Good if you don't need agent-step-level tracing and just want a billable cost breakdown with caching. Their caching layer alone saved one of my clients $40K/month on repeat prompts.

Arize Phoenix — Eval-focused, traces + eval runs in one place. Better for teams that care about offline evals (running regression suites of agent traces). Strong on RAG eval specifically. If your bottleneck is "is my retrieval working," Phoenix is the right call. Less ideal for cost monitoring.

LangSmith — LangChain's offering. Good if you're all-in on LangChain or LangGraph. Otherwise vendor-locked and pricing is aggressive at scale. The UI is pretty but the OTEL story is bolted-on rather than native.

Braintrust — Eval-focused like Phoenix but with stronger experiment tracking. Smaller community. Good if your bottleneck is "we have 40 evals and we need to know which one to run first." I keep hearing about them at conferences but haven't seen them in production at meaningful scale yet.

For most teams starting from scratch in 2026: Langfuse + Helicone as the proxy layer. That's the combination I see working in production most often. Langfuse for traces and evals, Helicone for cost proxy + caching. The two integrate cleanly because both support OTEL.

Layer 3: Instrument at the Right Layer

The biggest mistake teams make is instrumenting at the model call level and stopping. That gives you "we spent $4,200 on OpenAI this week." What you actually need is to instrument at the agent step level, so you can answer "we spent $4,200 on OpenAI this week, of which $2,800 was the refund-negotiation agent's third retry step on conversations from user segment X."

That means wrapping each agent step in an OTEL span and recording which step in the agent loop you're on, what the input was, what the output was, and what the cost was. Here's what that looks like in code, using the Python OTEL SDK with Langfuse as the backend:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanExporter
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
# Wire up OTEL once at startup
provider = TracerProvider()
provider.add_span_processor(
    BatchSpanExporter(
        OTLPSpanExporter(endpoint="https://cloud.langfuse.com/api/public/otel")
    )
)
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("my.agent")
def compute_cost(model: str, usage) -> float:
    # Pricing as of July 2026 — recompute quarterly, these change
    rates = {
        "o3-pro": {"input": 20.00, "output": 80.00, "reasoning": 80.00},
        "claude-opus-5-thinking": {"input": 15.00, "output": 75.00, "reasoning": 112.50},
        "deepseek-r2": {"input": 0.55, "output": 2.19, "reasoning": 2.19},
    }
    r = rates.get(model, rates["o3-pro"])
    return (
        usage.prompt_tokens / 1_000_000 * r["input"]
        + usage.completion_tokens_details.reasoning_tokens / 1_000_000 * r["reasoning"]
        + (usage.completion_tokens - usage.completion_tokens_details.reasoning_tokens) / 1_000_000 * r["output"]
    )
def run_agent_step(user_id: str, tenant_id: str, conversation_id: str, step: int, prompt: str):
    with tracer.start_as_current_span("agent.step") as span:
        # Annotate the span with everything you need to debug cost later
        span.set_attribute("gen_ai.conversation.id", conversation_id)
        span.set_attribute("gen_ai.conversation.user_id", user_id)
        span.set_attribute("gen_ai.conversation.tenant_id", tenant_id)
        span.set_attribute("gen_ai.agent.name", "refund_negotiator")
        span.set_attribute("gen_ai.agent.step", step)
        # Call the reasoning model — this itself should be a child span
        with tracer.start_as_current_span("llm.generation") as llm_span:
            llm_span.set_attribute("gen_ai.request.model", "o3-pro-2026-07")
            response = openai_client.chat.completions.create(
                model="o3-pro-2026-07",
                messages=[{"role": "user", "content": prompt}],
            )
            # Reasoning tokens are returned but NOT in response.usage.output_tokens
            # OpenAI hides them in response.choices[0].message.reasoning_tokens
            # (API contract changed 2026-05 — check your SDK version)
            reasoning_tokens = response.usage.completion_tokens_details.reasoning_tokens
            output_tokens = response.usage.completion_tokens - reasoning_tokens
            llm_span.set_attribute("gen_ai.usage.input_tokens", response.usage.prompt_tokens)
            llm_span.set_attribute("gen_ai.usage.output_tokens", output_tokens)
            llm_span.set_attribute("gen_ai.usage.reasoning_tokens", reasoning_tokens)
            llm_span.set_attribute("gen_ai.cost.usd", compute_cost("o3-pro", response.usage))
        return response.choices[0].message.content

A few things worth noticing in that snippet. First, the agent step span is the parent, the LLM generation span is the child — so when you query "what's eating my budget?" in Langfuse, you can drill from agent step → LLM call → reasoning token breakdown. Second, the gen_ai.conversation.id is the same across all spans for one agent run, so you can reconstruct a full timeline in your dashboard. Third, the cost is computed and stored as an attribute — never trust the model provider's billing line items, compute it yourself from token counts and your negotiated rates. The fintech I mentioned at the top trusted the OpenAI dashboard. The OpenAI dashboard was right. They just never opened it.

Layer 4: Set Cost Guards, Not Just Dashboards

Observability without action is just an expensive log. Once you're tracing, the next step is wiring your trace data into your rate limiter, your alert manager, and your kill switch.

The minimum viable cost guard looks like this:

1. Per-conversation budget. Kill the agent if a single conversation exceeds $5 in reasoning tokens. Below this threshold, the agent is unlikely to be doing anything useful. 2. Per-user daily budget. Throttle users who exceed $20/day to a cheaper model. Don't ban them — degrade gracefully. 3. Per-tenant monthly budget. Alert when 80% of monthly budget is gone; hard-kill at 100%. If you're on a flat SaaS pricing model, this is your margin protection. 4. Model-specific guardrails. If a single model accounts for >50% of weekly spend, page the on-call. Reasoning models are sticky that way. 5. Reasoning-token velocity. If any conversation has burned >100K reasoning tokens without finishing, abort. This is the single biggest runaway-loop signal you can catch.

These aren't nice-to-haves. They are the difference between a $4K month and a $112K month. The fintech I mentioned had none of these. Their agent ran for three weeks before someone opened the bill. By then, the agent had processed 19,400 conversations, of which 1,200 were "negotiate this enterprise contract" tickets that each averaged 380K reasoning tokens.

Comparison: Why Not Just Use LangSmith / Helicone / Custom?

Let me be honest about the alternatives and why most of them lose for production agent workloads in 2026.

LangSmith is good if you're all-in on LangChain. If you use any non-LangChain framework — and most teams do by mid-2026, given how mature LangGraph has gotten — LangSmith becomes friction. The pricing also gets ugly above ~5M traced events/month. The recent price hike in May pushed a lot of teams off it. The other issue is that LangSmith's OTEL support is recent and feels grafted on rather than native. Spans don't always propagate cleanly across non-LangChain boundaries. For pure LangChain shops, it's still the lowest-friction choice. For everyone else, it's not.

Helicone alone gives you cost tracking and caching but doesn't model agent steps. You'll see "you spent $4,200 on OpenAI" but not "step 3 of your planner agent is responsible for 60% of that." For pure proxy-layer cost control, it's the cheapest path to value. For agent observability, it falls short. Use it as a complement, not a substitute.

Custom OTEL + Grafana + ClickHouse is the power-user path. You get full control, you pay nothing in vendor fees, you spend 2-3 engineer-weeks building dashboards. If you already have a platform team running Grafana, this is worth considering. If you don't, don't. Most teams underestimate the operational burden of running their own observability backend. ClickHouse is fast but it eats disk. Grafana dashboards need to be built and maintained. The cost savings don't pencil out until you're past 100M events/month.

Arize Phoenix is the right pick if your bottleneck is evaluation, not cost. It has the best RAG eval tooling of the four, and its traces integrate with experiment runs in a way the others don't. If you're shipping a RAG-heavy agent and your problem is "is this retrieval working" more than "why is this expensive," pick Phoenix. If your problem is cost, Langfuse wins.

Braintrust I keep hearing about but haven't seen in production at any meaningful scale. Might be a 2027 story. Their pitch deck is good.

The right answer for most teams in mid-2026 is Langfuse self-hosted or cloud + OTEL instrumentation + a $50/month Grafana Cloud for alerting. That combo is OTEL-native, vendor-portable, and costs less than one bad reasoning-model incident.

What About Reasoning Token Visibility Itself?

A separate problem the providers haven't fully solved: reasoning tokens are still largely opaque. You get a count. You don't get the content. That's a deliberate design choice — exposing reasoning chains makes them gameable, and gameable reasoning chains are worse than black boxes — but it creates an observability problem.

What you can do:

1. Log reasoning token count, time-to-first-reasoning-token, and total reasoning span. These three tell you whether the agent is in a normal reasoning pass or stuck. A reasoning pass that takes 90 seconds with no progress is suspicious. A reasoning pass that emits 200K tokens but no output is broken. 2. Hash your prompts and log the hash. This lets you see which prompt variants are correlated with which reasoning costs without leaking user data. A prompt that triggers 50K reasoning tokens consistently is doing real work. A prompt that sometimes triggers 50K and sometimes 5K is the one to debug. 3. Track reasoning/output ratio. A healthy agent has a ratio between 2:1 and 20:1. If your ratio spikes to 200:1, you're in runaway territory. If it drops below 1:1, your model isn't reasoning at all. 4. Sample the full reasoning trace at 1%. OpenAI and Anthropic both let you opt into seeing reasoning content at a sample rate, with a PII redaction contract. Use this for debugging, not for production observability.

The state of reasoning token observability in 2026 is still "you can count them, you can't read them." That's better than nothing. It's not enough.

The Take

Reasoning models are the most important production capability of 2026. They also broke every assumption you had about LLM cost predictability. The teams that ship them safely are the ones that instrumented agent steps, modeled reasoning tokens explicitly, and wired cost guards into the runtime before they shipped.

The teams that didn't are the ones writing me emails asking why their $4K-a-month prototype became a $112K-a-month production system.

Pick OTEL. Pick a backend you can leave (Langfuse is my recommendation, but the answer matters less than the choice to instrument). Instrument every agent step. Set per-conversation and per-tenant cost guards. Then ship your reasoning model. The math works if you can see what's happening. It doesn't work if you're flying blind.

The fintech I mentioned at the top? They're rebuilding the agent now. They instrumented it before relaunching. The bill this week was $4,800. That's the budget they expected. The reason it's not $112,000 anymore is that they can see which conversation is the expensive one, mid-flight, and kill it before it burns the budget. That's the entire value proposition of agent observability in one sentence: you stop being surprised.

— Mr. Technology

Sources

• OpenTelemetry GenAI Semantic Conventions, v1.26.0, https://opentelemetry.io/docs/specs/semconv/gen-ai/, accessed July 2026 — the canonical attribute names (gen_ai.usage.reasoning_tokens, gen_ai.conversation.id, etc.) and span conventions for LLM and agent spans.
• Langfuse Documentation, "Agent Observability," https://langfuse.com/docs/observability, accessed July 2026 — the OTEL-native integration, trace propagation model, and self-hosting reference architecture.
• Helicone 2.0 Release Notes, https://helicone.ai/blog/2-0-release, accessed July 2026 — the proxy-layer cost attribution and caching layer that complements Langfuse for production deployments.
• Arize Phoenix Documentation, "RAG Evaluation," https://docs.arize.com/phoenix, accessed July 2026 — the eval-focused observability alternative and when to pick it over Langfuse.
• OpenAI API Reference, o3-pro reasoning tokens, https://platform.openai.com/docs/models/o3-pro, accessed July 2026 — the API contract change (May 2026) that moved reasoning tokens into completion_tokens_details.reasoning_tokens.
• Anthropic Claude Opus 5 Thinking, pricing documentation, https://docs.anthropic.com/en/docs/claude-opus-5-thinking, accessed July 2026 — the 1.5x reasoning token pricing model for extended-thinking mode.
• DeepSeek-R2 Model Card, https://github.com/deepseek-ai/DeepSeek-R2, accessed July 2026 — the open-source reasoning model with reasoning tokens at parity with output.
• LangSmith Pricing Documentation, https://docs.smith.langchain.com/pricing, accessed July 2026 — the May 2026 pricing change that pushed most non-LangChain teams off the platform.
• My own production deployment data: a 14-month longitudinal look at reasoning-model agent costs across three client engagements (fintech, legal-tech, dev-tools), with a combined 2.1M traced conversations and $340K in observed reasoning-token spend.