Streaming LLM Responses with FastAPI and Server-Sent Events: The 30-Minute Build

Hey guys, Mr. Technology here.

The single most expensive mistake in a fresh LLM backend is a blocking /chat endpoint. Your user types a prompt, hits send, then stares at a spinner for 8 seconds while GPT-4o generates. They click the button again. Twice. Your upstream tokens triple. You ship three identical responses and your bill goes through the roof. The fix is streaming with Server-Sent Events, and it takes 30 minutes. Most teams put it off for "later." The teams that ship it on day one save real money and ship a better product.

The Setup

mkdir llm-stream && cd llm-stream
python -m venv .venv && source .venv/bin/activate
pip install fastapi uvicorn openai sse-starlette httpx

You will need sse-starlette — FastAPI does not ship a built-in SSE helper, and rolling your own with StreamingResponse is how you spend a weekend debugging chunked transfer encoding.

The Endpoint

from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse
from openai import OpenAI
from sse_starlette.sse import EventSourceResponse
import asyncio, json, os
app = FastAPI()
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
@app.post("/chat/stream")
async def chat_stream(request: Request, body: dict):
    async def event_generator():
        try:
            stream = client.chat.completions.create(
                model="gpt-4o-mini",
                messages=body["messages"],
                stream=True,
            )
            for chunk in stream:
                if await request.is_disconnected():
                    break
                delta = chunk.choices[0].delta.content or ""
                if delta:
                    yield {"event": "token", "data": json.dumps({"token": delta})}
            yield {"event": "done", "data": "[DONE]"}
        except Exception as e:
            yield {"event": "error", "data": json.dumps({"message": str(e)})}
    return EventSourceResponse(event_generator())

Run it:

uvicorn app:app --reload --port 8000

Test it from the terminal with curl:

curl -N -X POST http://localhost:8000/chat/stream \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Say hi in 5 words"}]}'

You should see tokens arriving one by one, ~50ms apart, with event: token lines. That is SSE — plain HTTP, every proxy on the planet speaks it.

The Three Gotchas

Gotcha 1: The OpenAI client is sync, and that sync call blocks your event loop. Every tutorial on the internet calls client.chat.completions.create(stream=True) inside an async def and pretends it streams. It does not — the underlying HTTP call still blocks the worker until the first chunk arrives, which is why you get a 1-3 second "dead air" before tokens start flowing. Wrap stream consumption in asyncio.to_thread() and yield from an async iterator. The fix is 6 lines and drops your time-to-first-token by half:

import asyncio
async def stream_openai(messages):
    loop = asyncio.get_event_loop()
    queue: asyncio.Queue = asyncio.Queue()
    def pump():
        try:
            for chunk in client.chat.completions.create(
                model="gpt-4o-mini", messages=messages, stream=True
            ):
                d = chunk.choices[0].delta.content
                if d: loop.call_soon_threadsafe(queue.put_nowait, d)
        finally:
            loop.call_soon_threadsafe(queue.put_nowait, None)
    asyncio.create_task(asyncio.to_thread(pump))
    while True:
        tok = await queue.get()
        if tok is None: return
        yield tok

Thread feeds a queue; async generator drains it. No blocking, no buffering the whole response.

**Gotcha 2: You must check request.is_disconnected() on every chunk, or you pay for tokens nobody reads.** I learned this the hard way watching a bill double after a frontend bug caused users to navigate away mid-stream. The server kept generating because nothing told it to stop. That is_disconnected() check is the line that saves you thousands a month at modest scale. Pair it with a client-side AbortController so the browser actually closes the connection.

Gotcha 3: Proxies buffer SSE unless you set the right headers. Nginx by default buffers responses until 4KB accumulates — your user sees nothing for 3 seconds, then a wall of tokens. sse-starlette sets Cache-Control: no-cache, X-Accel-Buffering: no, and Content-Type: text/event-stream for you, but Cloudflare and corporate proxies need X-Accel-Buffering: no set manually. Verify in response headers before you blame your code.

The Frontend

const res = await fetch('/chat/stream', {
  method: 'POST',
  headers: {'Content-Type': 'application/json'},
  body: JSON.stringify({messages: [{role:'user', content: prompt}]}),
});
const reader = res.body.getReader();
const decoder = new TextDecoder();
while (true) {
  const {done, value} = await reader.read();
  if (done) break;
  const text = decoder.decode(value);
  for (const line of text.split('\n')) {
    if (line.startsWith('data: ')) {
      const token = JSON.parse(line.slice(6)).token;
      if (token) appendToChat(token);
    }
  }
}

Twenty lines of vanilla JS. No SSE library needed. Works in every browser since 2018.

The Take

Streaming is not a polish feature you ship in v2. It is table stakes. The 30-minute investment here saves you a support ticket per week per user, a meaningful chunk of your LLM bill from abandoned requests, and the slow embarrassment of "why is my chat app slower than ChatGPT." Build it on day one.

— Mr. Technology

*Packages: fastapi, uvicorn, openai, sse-starlette. SSE = Server-Sent Events, a one-way HTTP stream from server to browser over a single long-lived text/event-stream connection. Time-to-first-token on gpt-4o-mini typically drops from 1.5-3s (blocking) to 250-400ms (streaming). For Anthropic, swap the OpenAI client for anthropic and iterate over client.messages.stream(...) the same way. Always pair server-side with a client-side AbortController to actually close the connection on navigation.*