Stop Calling OpenAI Directly: Set Up LiteLLM as a Production LLM Proxy With Automatic Fallback

Your app calls OpenAI. OpenAI rate-limits you at 3 AM. Your app calls Anthropic. Anthropic is up but slow. Your customer sees a 500 error and churns. Your Slack blows up.

This is the multi-provider reliability problem every production LLM app hits by month two. The fix is not "switch providers." The fix is a proxy that routes intelligently and fails over automatically. LiteLLM is the answer, and it takes 15 minutes to set up correctly.

What LiteLLM Actually Does

LiteLLM is an OpenAI-compatible proxy server. You point your existing OpenAI SDK at http://localhost:4000 instead of api.openai.com, and LiteLLM translates the request to whatever provider you configure. Same API surface, different backend. Drop-in replacement.

The killer features for production:

• Automatic fallback across providers. GPT-4.1 rate-limited? LiteLLM retries with Claude. Claude slow? It tries Gemini. You configure the priority order once.
• Cost tracking per request, per team, per user. Every token gets logged with provider, model, and dollar cost. Export to Postgres, BigQuery, or a CSV.
• Load balancing across multiple keys. Pool five OpenAI keys, LiteLLM rotates them and respects per-key rate limits.
• Spend caps per virtual key. Issue a key to a junior dev with a $50/day cap. They hit the cap, requests stop, bill stays flat.
• Streaming, function calling, vision, structured outputs. Full API parity, not a subset.

The 15-Minute Setup

pip install 'litellm[proxy]' gunicorn

Create config.yaml:

model_list:
  - model_name: gpt-4.1
    litellm_params:
      model: gpt-4.1
      api_key: os.environ/OPENAI_API_KEY
  - model_name: claude-sonnet-4
    litellm_params:
      model: claude-sonnet-4-20250514
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: gemini-2.5-pro
    litellm_params:
      model: gemini/gemini-2.5-pro
      api_key: os.environ/GEMINI_API_KEY
router_settings:
  num_retries: 3
  timeout: 30
  allowed_fails: 2
  cooldown_time: 30
general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL

Add fallback logic with the ! prefix:

- model_name: prod-primary
  litellm_params:
    model: gpt-4.1
- model_name: !prod-primary
  litellm_params:
    model: claude-sonnet-4
- model_name: !prod-primary
  litellm_params:
    model: gemini-2.5-pro

Any request to prod-primary tries GPT-4.1 first. On 429, 500, 503, or timeout, LiteLLM fails over to Claude. Then Gemini. Your app sees a successful response either way.

Start the proxy:

litellm --config config.yaml --num_workers 4

Your application code does not change:

from openai import OpenAI
client = OpenAI(
    base_url="http://localhost:4000",
    api_key="anything"  # master key from config
)
response = client.chat.completions.create(
    model="prod-primary",
    messages=[{"role": "user", "content": "Hello"}]
)

The Spend Cap That Pays For The Whole Stack

Hit localhost:4000/ui in a browser. Log in with your master key. Create a virtual key per team, set a daily budget, hand it to engineering. LiteLLM rejects requests when the budget hits zero. No more surprise $40K OpenAI bills from a misconfigured retry loop.

The Take

If you ship LLM features to production and you are calling providers directly, you are paying the reliability tax. LiteLLM is 15 minutes of setup that buys you cross-provider fallback, real cost visibility, and per-team spend caps. The open-source self-hosted version handles most teams. The enterprise tier adds SSO, audit logs, and SOC 2 controls.

Stop calling OpenAI directly. Set up a proxy. Sleep through the next outage.

— Mr. Technology