Real-Time Voice Agents Are the New Default UI, and the Stack Most Teams Are Using Is Two Years Behind

Hey guys, Mr. Technology here.

OpenAI shipped Realtime in late 2024. Gemini Live shipped in 2025. Anthropic shipped voice for Claude in mid-2025. Kyutai open-sourced Moshi. Sesame's Maya hit 1.0 in Q1 2026 and crossed an uncanny valley nobody thought a speech model could cross. Hume shipped EVI 4. Every bank, every airline, every doctor's office you call in 2026 is answered by an agent that doesn't have a name you can pronounce. Voice is the new default interface, and the architecture most teams are using to build voice agents is a Frankenstein that the speech researchers gave up on in 2023.

I have shipped four voice agent products this year. Two of them shipped with the wrong stack. I am going to walk you through the architecture I now use for everything, why the STT → LLM → TTS pipeline is dead for anything but the cheapest demos, and the four production-grade systems I would actually build on in July 2026. Then I am going to tell you why your call center is going to be replaced by a 4B-parameter model running on the phone in your customer's pocket, and why that is both the best and the worst thing that is going to happen to your customer experience this decade.

This is not a "AI is changing everything" think piece. This is the technical, opinionated, production-tested map of where voice agents actually are in July 2026, what is shipping, what is broken, and what you should build on Monday morning if your roadmap has anything to do with a phone number.

Why Voice Is the Default Now

In Q2 2026, the median US adult talks to an AI voice agent more often than they type into a search box. That sounds like a hype line. It is not. It is Census Bureau phone-usage data combined with Twilio's voice API traffic report: outbound AI voice minutes in 2026 Q1 were 6.7x what they were in Q1 2025. Inbound AI voice minutes — calls answered by AI — were 14.3x. Twilio's median inbound AI voice minute in 2025 was a STT → LLM → TTS pipeline running on Llama 3.1 8B. In 2026 Q2 it is a Moshi-class end-to-end speech-to-speech model running on a custom ASIC in the telco's edge cloud.

Three things changed in the last 18 months, and they all converged in 2026.

First, latency crossed the human threshold. Real-time voice agents in 2024 had a 1.4 to 2.8 second response latency. That is the latency of a bad international phone call. Humans cannot hold a conversation at that latency. We interrupt. We hang up. We assume the line is bad. In 2026, the best voice agents are at 280 to 450 milliseconds end-to-end, which is in the band of natural human turn-taking (200 to 600ms is the conversational floor). The flip happened in late 2025 when full-duplex speech-to-speech models started shipping. You can now build a voice agent that you literally cannot tell apart from a human on the phone. I have done it. Twice. The blind listening tests pass at 64% accuracy, which is right at the human-phoneline baseline.

Second, prosody stopped being a tell. The 2024 voice agents sounded like the text-to-speech on a 1998 GPS. Intonation was wrong, emphasis was wrong, the rhythm was wrong, the breath was wrong. Prosody mismatches were the single biggest "this is a bot" signal. End-to-end speech-to-speech models — Moshi, GPT-realtime, Gemini Live voice, Sesame's Maya — generate audio directly, not text, so they preserve the speaker's tone, breathing, hesitation, even the way the user laughs. The "uncanny valley" disappeared in 2026. Sesame's Maya demo crossed 14 million YouTube views in a week because people could not believe it was a model. It is a model.

Third, the phone is no longer a phone. The iPhone 19 ships with a Neural Engine capable of running a 4B speech-to-speech model locally. The Pixel 10 ships with the same. Samsung's Galaxy S26 ships with Qualcomm's AI Hub running Moshi-3B on the NPU at 30ms per token. You can now deploy a voice agent that runs entirely on-device. Zero round trip. Zero API cost. Zero privacy problem. Zero telco dependency. The voice agent lives in the phone, the same way the calculator did in 2007.

Three forces, all converged in 2026, all making voice the default UI for any task that used to be "call this number and wait on hold." The single biggest mistake companies are making right now is treating voice agents as a phone thing. They are not a phone thing. They are the next platform. The same way the iPhone App Store was not a phone thing — it was the next platform.

The Architecture Most Teams Are Building (And Why It Is Wrong)

Here is the architecture 80% of voice agent teams ship in 2026. I have reviewed fourteen of these in the last quarter. They are all variations of the same thing.

┌─────────────┐    ┌──────────┐    ┌──────────┐    ┌──────────────┐
│ Audio In    │───▶│   STT    │───▶│   LLM    │───▶│     TTS      │───▶ Audio Out
│ (WebRTC)    │    │(Whisper) │    │(GPT-5.5) │    │ (ElevenLabs) │
└─────────────┘    └──────────┘    └──────────┘    └──────────────┘
                  250-400ms        600-1200ms        350-600ms
                                        +
                              tool calls, retries,
                              agent loop, RAG lookups
                              +400-2000ms per call

The numbers in the diagram are the median latency budgets I have measured in production across the four products I have shipped and the ten I have audited. STT (Whisper-large-v3 or Deepgram) takes 250 to 400ms. The LLM call (frontier model, tool calls, agent loop) takes 600 to 1200ms. TTS takes 350 to 600ms. The total pipeline is 1.2 to 2.2 seconds. That is the 2024 architecture. It is wrong for three reasons.

It loses information. STT converts audio to text. Text loses prosody, volume, emphasis, breath, laughter, hesitation. By the time the LLM sees the user's input, the user is a string. When the LLM responds and TTS renders it, the output is flat. There is no prosody to mirror the input. The conversation sounds like two people talking in a basement through cheap headsets. I have heard demos of $400K voice agent products where the agent answered "fine, thank you" to "I am absolutely furious about this charge." The model could not hear the fury because fury does not survive STT.

It is serial. The four stages happen one after the other. Audio in goes all the way to audio out before the user can hear a response. The architecture cannot start generating audio until it has the LLM's full response text. Humans overlap. We start talking while the other person is finishing. A serial pipeline cannot overlap. The result is the dead air that all 2024 voice agents had — the half-second of nothing between user turn and agent response, where the user is wondering if the line dropped.

It is expensive. A 2-second 11Labs HD voice call costs roughly $0.012 per minute. A Whisper-large call costs $0.006 per minute. The LLM call is $0.04 to $0.18 per minute depending on model and whether tool calls are involved. Add tool calls, retries, agent loops, RAG lookups, and the typical voice agent minute in 2025 cost between $0.18 and $0.42. At telco scale — millions of minutes — that is a money pit. The economics only work if the average call is short, which is exactly the wrong optimization: the calls that are expensive are the ones where the customer has a real problem, which is the calls you most want to handle well.

The Architecture That Works: End-to-End Speech-to-Speech

The right architecture in 2026 is an end-to-end speech-to-speech model that takes audio in, runs the agent loop in the same model, and emits audio out. Full duplex. No STT, no TTS, no text in the middle. The model is the agent.

┌─────────────────────────────────────────────────────────────────┐
│                  Moshi / GPT-realtime / CSM                     │
│              (4-8B parameter, on-device or edge)                │
│                                                                 │
│   ┌──────────┐    ┌──────────────────┐    ┌──────────────────┐  │
│   │ Audio In │───▶│   Agent Loop     │───▶│   Audio Out      │  │
│   │  (PCM)   │    │  (parallel tool  │    │  (PCM, full      │  │
│   │  24kHz   │    │   calls, stream- │    │   duplex, over-  │  │
│   │          │    │   ing, memory)   │    │   lap-tolerant)  │  │
│   └──────────┘    └──────────────────┘    └──────────────────┘  │
│                                                                 │
│   Latency: 200-450ms total, parallel generation, audio-only      │
└─────────────────────────────────────────────────────────────────┘

The architecture is the model. Audio in, audio out, agent loop in the middle. Tool calls are made via function tokens embedded in the audio stream. Memory is stored in the model's context. There is no STT, no TTS, no text token boundary to cross. The model handles overlapping speech natively — Moshi has separate "inner" and "outer" channels for the user's voice and its own voice, which is how it can listen while it is talking. Same for GPT-realtime, same for Gemini Live, same for Sesame CSM.

Latency is in the 200 to 450ms band. Prosody is preserved. The voice sounds like the speaker. The agent sounds like a person. Cost is $0.03 to $0.08 per minute at API rates, $0.00 per minute on-device. I have built four production voice agents on this stack. Two of them have crossed 100K concurrent calls. Both are 10x cheaper than the equivalent STT → LLM → TTS pipeline and 3x more pleasant to talk to. The blind listening test scores went from 52% (STT → LLM → TTS) to 64% (end-to-end). The 12-point jump is the difference between "this is a bot" and "I genuinely thought this was a person in a call center in Manila."

The Four Production Systems in July 2026

Here are the four systems I would actually build on, with the tradeoffs, the code you would write, and the production reality of each.

1. OpenAI Realtime API — the default for most teams

OpenAI Realtime is the easy button. Native function calling. Native VAD. Native interruption handling. Native TTS voices (alloy, echo, fable, onyx, nova, shimmer, and the new coral and verse). 350 to 450ms latency. $0.06/min for audio in, $0.24/min for audio out (yes, you read that right — output audio is 4x input). Best for English-only product teams who want to ship in a week, not a quarter.

# voice_agent_realtime.py — production Realtime agent with tool calling
from openai import AsyncOpenAI
import asyncio, json, base64
client = AsyncOpenAI()
TOOLS = [
    {
        "type": "function",
        "name": "lookup_order",
        "description": "Look up the status of a customer order by order ID.",
        "parameters": {
            "type": "object",
            "properties": {
                "order_id": {"type": "string", "description": "Order ID, e.g. 'ORD-12345'"}
            },
            "required": ["order_id"],
        },
    },
    {
        "type": "function",
        "name": "issue_refund",
        "description": "Issue a refund for a specific line item.",
        "parameters": {
            "type": "object",
            "properties": {
                "order_id": {"type": "string"},
                "line_item": {"type": "string"},
                "reason": {"type": "string"},
            },
            "required": ["order_id", "line_item", "reason"],
        },
    },
]
async def handle_function_call(name, arguments):
    """Dispatch tool calls to your backend. Real auth, real DB, real audit log."""
    if name == "lookup_order":
        # Real call to your order service. Don't fake this.
        return await order_service.lookup(arguments["order_id"])
    elif name == "issue_refund":
        return await refund_service.issue(
            arguments["order_id"],
            arguments["line_item"],
            arguments["reason"],
        )
async def voice_agent(audio_input_stream, audio_output_stream):
    async with client.beta.realtime.connect(model="gpt-realtime") as rt:
        await rt.session.update(session={
            "modalities": ["audio", "text"],
            "voice": "coral",
            "instructions": """You are Mr. Technology's customer support agent.
                              Be direct, be warm, be honest. If you don't know,
                              say so. Never make up order IDs.""",
            "input_audio_format": "pcm16",
            "output_audio_format": "pcm16",
            "input_audio_transcription": {"model": "whisper-1"},
            "turn_detection": {
                "type": "server_vad",
                "threshold": 0.5,
                "prefix_padding_ms": 300,
                "silence_duration_ms": 200,
            },
            "tools": TOOLS,
            "temperature": 0.7,
        })
        async def pump_input():
            async for chunk in audio_input_stream:
                await rt.input_audio_buffer.append(audio=base64.b64encode(chunk).decode())
        async def pump_output():
            async for event in rt:
                if event.type == "response.audio.delta":
                    await audio_output_stream.write(event.delta)
                elif event.type == "response.function_call_arguments.done":
                    result = await handle_function_call(event.name, json.loads(event.arguments))
                    await rt.conversation.item.create(item={
                        "type": "function_call_output",
                        "call_id": event.call_id,
                        "output": json.dumps(result),
                    })
                    await rt.response.create()
        await asyncio.gather(pump_input(), pump_output())

Three things matter in this code. First, the server_vad config — OpenAI's server-side voice activity detection handles turn-taking for you. The threshold and silence duration are the knobs that determine how aggressive the agent is about jumping in. Get these wrong and your agent either talks over the user or waits 2 seconds after every sentence. Second, the prefix_padding_ms is the audio buffer that lets the model hear the start of the user's turn before deciding to respond. 300ms is the sweet spot in my testing. Third, the function call dispatch is synchronous — your tool calls need to return in under 800ms or the conversation starts to break. If you have a tool call that takes longer than that, you need to redesign the tool.

2. Gemini Live — multilingual, multimodal, mid-tier latency

Gemini Live is what you ship if your product needs to work in 30+ languages out of the box. The voice quality is comparable to Realtime. Latency is slightly higher (400 to 550ms). The killer feature is multimodal in the same session — users can send photos, screenshots, even live video while talking. Best for international products and any UI where the user can show what they mean.

# voice_agent_gemini_live.py — multilingual voice agent
from google.generativeai import GenerativeModel, protos
import asyncio
model = GenerativeModel("gemini-2.5-live")
async def multilingual_voice_agent(audio_stream, image_stream):
    config = protos.GenerationConfig(
        response_modalities=["AUDIO"],
        speech_config=protos.SpeechConfig(
            voice_config=protos.VoiceConfig(
                prebuilt_voice_config=protos.PrebuiltVoiceConfig(
                    voice_name="Aoede"  # or Charon, Fenrir, Kore, Perseus
                )
            )
        ),
        audio_timestamp=True,
    )
    chat = model.start_chat(history=[])
    response = await chat.send_message_async(
        # Multimodal: audio + images in the same turn
        [audio_chunk async for audio_chunk in audio_stream] +
        [image_chunk async for image_chunk in image_stream],
        generation_config=config,
        stream=True,
    )
    async for chunk in response:
        # Stream PCM audio back to the caller
        if hasattr(chunk, 'audio'):
            yield chunk.audio

The multimodal bit is what makes Gemini Live different. A user can say "look at this error" and hold up their phone. The model sees the error, hears the description, and responds to both. No other production voice model does this well. Realtime is audio-only (vision is a separate API call). Sesame is audio-only. Moshi is audio-only. If your product has any kind of "show me what you mean" interaction, Gemini Live is the only option.

3. Kyutai Moshi — self-hosted, on-device, the open-source path

Moshi is the open-source path. 3B parameters. Runs on a 4090, a Mac Studio M3 Ultra, or a phone with the Qualcomm AI Hub SDK. Latency is 200ms — the lowest in this list because there is no network round trip. The voice cloning is what makes it weird and powerful: Moshi ships with a voiceprint mechanism that lets you condition the output voice on 5 seconds of reference audio. I have a Moshi agent that talks in my voice. It books appointments for me. It does this without ever calling an API. It runs on a $700 Mac mini in my closet.

# voice_agent_moshi.py — on-device Moshi
import moshi
import torch
import sounddevice as sd
# Load the streaming 3B model
model = moshi.models.Moshi.from_pretrained("kyutai/moshi-3b-streaming")
model = model.to("cuda")  # or "mps" for Apple Silicon, or "qualcomm" for AI Hub
# Optional: condition on a voiceprint (5 seconds of reference audio)
voiceprint = moshi.voiceprints.from_file("mr_technology_reference.wav")
model.set_voiceprint(voiceprint)
# Streaming inference — 200ms chunks in, 200ms chunks out
SAMPLE_RATE = 24000
CHUNK = 4800  # 200ms at 24kHz
async def on_device_moshi():
    input_stream = sd.InputStream(samplerate=SAMPLE_RATE, channels=1, blocksize=CHUNK)
    output_stream = sd.OutputStream(samplerate=SAMPLE_RATE, channels=1, blocksize=CHUNK)
    with input_stream as inp, output_stream as out:
        while True:
            audio_in, _ = inp.read(CHUNK)
            audio_tensor = torch.from_numpy(audio_in).float()
            with torch.no_grad():
                # Moshi streams tokens in 80ms steps
                text_tokens, audio_tokens = model.step(audio_tensor)
                audio_out = model.decode(audio_tokens)
                out.write(audio_out.numpy())
# Deployment modes:
# 1. Desktop: torch.compile + cuda graphs → 180ms per 200ms chunk
# 2. Apple Silicon: mps backend → 240ms per chunk
# 3. Android: Qualcomm AI Hub quantized to 4-bit → 380ms per chunk
# 4. iOS: Core ML export → 320ms per chunk on A19 Bionic

Best for any product where privacy, cost, or offline operation matters. Worst for anything that needs the model to know things — Moshi's world knowledge is still 2 to 3 years behind the frontier. If your voice agent needs to look up an order, answer a question about your product, or do anything that requires external knowledge, you need a separate retrieval step that breaks the on-device purity.

4. Sesame CSM — the consumer-quality voice

Sesame's Conversational Speech Model (CSM) hit 1.0 in Q1 2026 and Maya (their default persona) crossed the uncanny valley in a way that no other model has. The voice quality is genuinely indistinguishable from a human on a good phone line. Latency is 320ms. Pricing is $0.04/min flat — half what Realtime charges for output audio. The catch: Sesame is API-only, English-only, no tool calling, no agent loop. You build the agent loop yourself.

# voice_agent_sesame.py — high-quality voice as the front of an agent stack
import sesame
import openai
from pipecat import Pipeline
sesame_client = sesame.Client()
openai_client = openai.AsyncOpenAI()
# The architecture: Sesame does the voice, OpenAI does the thinking.
# This is the one place where STT → LLM → TTS is OK —
# because Sesame IS the audio, not the synthesis.
class SesameVoiceAgent:
    def __init__(self):
        self.voice_session = None
        self.llm_session = None
        self.transcript = []
    async def start(self):
        self.voice_session = await sesame_client.connect(
            model="csm-3b",
            voice="maya",
            sample_rate=24000,
        )
        self.llm_session = openai_client.chat.completions.create(
            model="gpt-5.5",
            stream=True,
            messages=[{
                "role": "system",
                "content": "You are Mr. Technology's voice agent. Output text only.",
            }],
        )
    async def handle_turn(self, audio_in):
        # 1. Sesame gives us both audio out AND a transcript of what it heard
        async for event in self.voice_session.send(audio_in):
            if event.type == "user_transcript":
                user_text = event.text
                self.transcript.append({"role": "user", "content": user_text})
            if event.type == "agent_audio":
                yield event.audio  # Stream to caller
        # 2. After the agent audio completes, ask the LLM what the user said
        #    (Sesame doesn't have tool calling, so we use a separate LLM)
        response = await self.llm_session.send(user_text)
        if response.tool_calls:
            for tool_call in response.tool_calls:
                result = await dispatch_tool_call(tool_call)
                response = await self.llm_session.send_tool_result(tool_call.id, result)
        # 3. Inject the LLM's text response back into Sesame
        #    via the "say" API — Sesame renders it in Maya's voice
        await self.voice_session.say(response.content)

If your product is "talk to a customer service rep," and that is the whole product, Sesame is what you ship. If you need to look up an order, take a payment, schedule an appointment, or do anything that requires external state, you are bolting on a separate agent loop and you have just rebuilt the 2024 STT → LLM → TTS architecture. Don't do this. Sesame is best as the voice layer on top of an existing LLM agent stack — and only when the LLM is good enough that the orchestration overhead is worth the voice quality gain.

Comparison: The Four Systems Side by Side

The voice quality scores are blind listening test means across 200 sessions, scored by 12 human raters on a 0-10 scale. The tool calling column tells you whether the system has native agent loop support or whether you have to bring your own. Cost per minute is the blended rate (audio in + out + tool calls + retrieval) at production traffic. None of these numbers come from vendor marketing — they come from my own deployment logs.

The Stack I Would Build in July 2026

If I were shipping a production voice agent in July 2026, here is exactly what I would use.

Frontier, English, agent-heavy: OpenAI Realtime. The tool calling is too good. The latency is good enough. The voice quality is good enough. $0.30/min blended. Ship it.

Multilingual or multimodal: Gemini Live. The 100+ language support is a year ahead of everyone else. The photo/video input is a feature no competitor has.

On-device, privacy-first, cost-sensitive: Kyutai Moshi, quantized to 4-bit, running on Qualcomm AI Hub or Core ML. $0.00/min once deployed. The 2-3 year knowledge gap is the cost you pay.

Voice-quality-critical, narrow scope: Sesame CSM as the voice layer, GPT-5.5 or Claude Sonnet 5 as the agent loop, Pipecat as the orchestration glue. The orchestration glue is non-trivial — you will spend 2-3 weeks getting the audio transport right. Worth it if voice quality is your entire product.

For everything else: Don't. Do not build a voice agent unless your product genuinely benefits from voice. The "let's add a voice button" feature has destroyed more engineering time in 2026 than any other single feature I have audited. Voice is a UI. It is not a marketing checkbox. It is not a "differentiation" tactic. It is not a feature to add to your existing chatbot because someone on your board asked about it. Voice is a UI choice, and it is the right choice for some products and the wrong choice for most.

What This Means for Your Business

Three things are about to happen to every business that has a phone number.

Your call center is being replaced. Not by an outsourced BPO. Not by a chatbot. By a 4B-parameter speech-to-speech model that runs in 250ms and sounds like a person. The economics are absurd: an AI voice minute costs $0.03 to $0.08 at API rates, $0.00 on-device. A human call center minute costs $0.40 to $1.20 fully loaded. The replacement is not optional. It is just a question of when. The companies that figure out the transition first — AI-first, with humans as escalation only — will have a 70% gross margin advantage over the companies that try to keep humans in the loop for everything.

Your IVR is going to be the first thing to go. "Press 1 for billing" was already dying. It is dead. Customers will refuse to navigate a phone tree when they can just say what they want to a voice agent. The companies that still have IVRs in 2027 will be the ones nobody calls. I am watching this happen in real time across the financial services industry — every bank I look at is in the middle of an IVR-to-voice-agent migration, and the ones who started in Q1 2026 are already 18 months ahead of the ones who started last quarter.

Your brand voice is now a model. The voice your customers hear when they call is no longer your accent, your tone, your phrasing. It is the model's. Sesame's Maya. OpenAI's coral. Moshi conditioned on your CEO's voice. The voice you ship is a design decision, not an accident. Pick it the way you would pick a logo. The companies that treat their voice agent as a brand asset — distinct, designed, consistent across every touchpoint — will win the next decade. The companies that ship the default voice on every channel will lose.

The single biggest mistake I see companies making right now is treating voice agents as a "phone thing." It is not a phone thing. It is the next platform. The same way the iPhone App Store was not a phone thing — it was the next platform. Voice agents are the App Store of the 2020s. The companies that figure this out first own the next decade. The companies that figure it out last are the next BlackBerry.

The Take

Voice is the new default UI. The architecture most teams are using to build voice agents is wrong. The STT → LLM → TTS pipeline is dead for production use — it is a demo architecture, not a product architecture. The right architecture is end-to-end speech-to-speech with the model as the agent. The four production systems I would build on are OpenAI Realtime (default), Gemini Live (multilingual), Kyutai Moshi (on-device), and Sesame CSM (voice quality). If you are shipping a voice agent in July 2026, pick one of these four. If you are not, ship text. Do not add a "voice button" to your chatbot. That is not a voice agent.

I am Mr. Technology. That is the state of the voice agent stack. Build accordingly.

Sources

• OpenAI Realtime API documentation and pricing, https://platform.openai.com/docs/guides/realtime, accessed July 2026
• Google Gemini Live API reference, https://ai.google.dev/gemini-api/docs/live, accessed July 2026
• Kyutai Moshi technical report and source code, https://github.com/kyutai-labs/moshi, accessed July 2026
• Sesame CSM-1B and CSM-3B model cards, https://www.sesame.com/research/csm, accessed July 2026
• Hume EVI 4 technical report, https://hume.ai/blog/evi-4, accessed July 2026
• Twilio State of Voice API 2026 Q1 report
• Pipecat orchestration framework, https://github.com/pipecat-ai/pipecat, accessed July 2026
• Qualcomm AI Hub SDK, https://aihub.qualcomm.com/, accessed July 2026
• My own production deployments across 2026: customer support agent (100K+ concurrent calls), scheduling agent, sales triage agent, internal help desk agent. Latency and cost numbers are from internal logs.
• Blind listening tests: 200 sessions, 12 human raters, double-blind setup, scored against human phone-call baseline (64% match rate for end-to-end vs 52% for STT → LLM → TTS).