How to Log Every LLM Prompt and Response to a File with LiteLLM in 10 Minutes

Hi guys, Mr. Technology here.

You shipped an LLM feature on Friday. By Monday, three users reported it gave them a "weird answer" and you have no idea what the model actually saw. Your vendor dashboard logs requests for 30 days and charges a fortune to export them. Faster path: point your app at a local LiteLLM proxy that writes every prompt, response, token count, and latency to a JSONL file on disk. Grep it, diff it, ship it to Loki — your call. Ten minutes from zero.

Why LiteLLM as a Proxy

LiteLLM is a 100-model abstraction layer that runs as a local server. The killer feature for debugging is that it sits between your code and the upstream provider, so it can intercept and log every request without you changing your application code. Swap the base URL, keep the SDK, get free observability.

Prerequisites

• Python 3.10+ and uv (or pip)
• One provider API key — OpenAI, Anthropic, or any of the 100 LiteLLM supports
• 50 MB of disk

Step 1 — Install and Configure

uv tool install 'litellm[proxy]'
mkdir ~/llm-logs && cd ~/llm-logs

Create a config.yaml in the same directory:

model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY
  - model_name: claude-sonnet
    litellm_params:
      model: anthropic/claude-sonnet-4-6
      api_key: os.environ/ANTHROPIC_API_KEY
general_settings:
  telemetry: False
litellm_settings:
  success_callback: ["jsonl_logger"]
  failure_callback: ["jsonl_logger"]
environment_variables:
  JSONL_LOG_FILE: /root/llm-logs/requests.jsonl

Step 2 — Start the Proxy

export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...
litellm --config config.yaml --port 4000 --detailed_debug

You should see Uvicorn running on http://0.0.0.0:4000. The jsonl_logger callback now writes one JSON object per line to requests.jsonl — request body, response body, latency, model, token usage, and cost.

Step 3 — Point Your App at the Proxy

The magic is that your existing OpenAI or Anthropic SDK code does not change. You only swap the base URL.

from openai import OpenAI
client = OpenAI(
    base_url="http://localhost:4000",
    api_key="anything",  # the proxy injects the real key
)
resp = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Summarize the moon landing in one sentence."}],
)
print(resp.choices[0].message.content)

For Anthropic:

import anthropic
client = anthropic.Anthropic(
    base_url="http://localhost:4000",
    api_key="anything",
)
msg = client.messages.create(
    model="claude-sonnet",
    max_tokens=256,
    messages=[{"role": "user", "content": "Same prompt."}],
)
print(msg.content[0].text)

Step 4 — Tail Your Logs

tail -f ~/llm-logs/requests.jsonl | jq .

Each line looks like:

{"model":"gpt-4o","messages":[...],"response":{"choices":[...]},"usage":{"prompt_tokens":12,"completion_tokens":28,"total_tokens":40},"response_time":0.847,"cost":0.00018}

That is your ground truth. No vendor lock-in, no 30-day TTL.

The Gotcha

**Do not pass api_key="anything" to the proxy in production without a guard.** Anyone on the network can hit localhost:4000 and burn your keys. Bind to 127.0.0.1, put it behind a reverse proxy with auth, or set general_settings.master_key. I have seen a junior dev deploy this exact setup to a VM with port 4000 open and wake up to a $4,000 OpenAI bill. Treat the proxy like a database — it has keys, it has cost.

Where to Go Next

• Swap jsonl_logger for langfuse_logger or otel_logger — the config line is one character different.
• Add routing_strategy: latency-based-routing to fan out across providers and compare quality in the same log file.
• Pipe the JSONL into DuckDB: SELECT model, AVG(cost) FROM 'requests.jsonl' GROUP BY model.
• Wrap it in a systemd unit so it survives reboots.

Ten minutes, every prompt on disk forever, zero changes to your application code. That is the entire pitch.

— Mr. Technology

References:

• LiteLLM proxy docs — https://docs.litellm.ai/docs/proxy
• Companion: "OpenTelemetry + Langfuse for Python LLM tracing" (this blog, July 2026)
• Companion: "uv for LLM Scripts" (this blog, June 2026)