Portkey Is the LLM Gateway That Realized LiteLLM Stopped at 'Proxy' and Built the Real Control Plane

I have a confession. Every "LLM gateway" I have shipped in the last eighteen months has been a glorified routing proxy with a config file. The routing is fine. The proxying is fine. The moment your team grows past three people, the moment you ship a real product with a real eval loop, the moment finance asks who is spending what on which model — every proxy I have used collapses into a Slack channel of regret.

Portkey — the open-source AI gateway from Portkey AI, ~11K GitHub stars, MIT licensed, Gateway 2.0 in pre-release, fresh Series A in 2026 — is the first open-source LLM gateway that figured out what the gateway layer is supposed to be in 2026: a control plane. Not a proxy. Not a router. A control plane. Routing is one feature of many. Observability is one feature of many. Guardrails is one feature of many. Caching, fallbacks, canary releases, prompt versioning, cost attribution — all features. And every one of them ships in the open-source SDK.

The result: when you swap out your LiteLLM proxy for a Portkey gateway, your infra team does not write a separate observability pipeline, a separate guardrail service, a separate prompt registry, a separate rate-limit daemon. They get all of it in one process, one config, one bill.

What It Actually Is

Portkey is a Python and Node SDK plus a self-hostable gateway server. The SDK is what you ship in your application — it talks to your gateway endpoint, and the gateway does the heavy lifting. The gateway is the open-source piece, written in TypeScript, talks the OpenAI API shape, ships as a Docker container or as a hosted service.

The features, briefly:

• Routing across 1,600+ models — OpenAI, Anthropic, Google, Bedrock, Azure, Mistral, Groq, Together, Fireworks, Cohere, OpenRouter, Hugging Face, Replicate, Anyscale, and your own self-hosted endpoints. Every model speaks the same OpenAI-shaped HTTP API on the far side.
• Automatic retries and fallbacks — if Anthropic returns a 429 or 529, Portkey falls to OpenAI (or Bedrock, or whatever you configure) without your code knowing.
• Load balancing — round-robin across multiple accounts, weighted routing, conditional routing on prompt content.
• Semantic caching — cache by meaning, not exact text match. Same question rephrased hits the cache. Redis-backed.
• Guardrails — 50+ built-in: PII detection, jailbreak detection, prompt injection detection, NSFW filtering, hallucination detection (with your chosen LLM judge). Custom regex, custom LLM judge, custom logic.
• Observability — every request logged with cost, latency, tokens, prompt version, response, model, fallbacks fired. Logs stream to your own Postgres, your own S3 bucket, or Portkey's hosted dashboard.
• Prompt versioning — manage prompts as versioned artifacts in the registry. A/B test v3 against v7 against production.
• Virtual keys — issue per-team, per-developer, per-application API keys with their own rate limits, budgets, and allowed models.

The killer feature is the config-as-code provider strategy:

pip install portkey-ai

from portkey_ai import Portkey
client = Portkey(
    api_key="YOUR_PORTKEY_API_KEY",
    config={
        "strategy": {
            "mode": "fallback",
            "on_status_codes": [429, 500, 503, 529],
            "targets": [
                {"provider": "anthropic", "api_key": "sk-ant-...", "weight": 0.7},
                {"provider": "openai",    "api_key": "sk-...",     "weight": 0.2},
                {"provider": "bedrock",   "aws_access_key_id": "...", "aws_region": "us-east-1", "weight": 0.1},
            ],
        },
        "cache": {"mode": "semantic", "max_age": 3600},
        "guardrails": [["prompt-injection"], ["pii-redaction"]],
        "retry": {"attempts": 3, "on_status_codes": [429, 503]},
        "weight": 0.6,
    },
)
resp = client.chat.completions.create(
    model="claude-sonnet-5",
    messages=[{"role": "user", "content": "Summarize the Q3 OKR document."}],
)

One Python call. The semantic cache runs first. The guardrails run on the prompt. The call goes to Anthropic at 70% weight. If Anthropic returns 429 or 529, Portkey retries internally and falls back to OpenAI. If OpenAI returns 503, falls to Bedrock. The response is logged with cost attribution, latency, prompt hash, model used, and which fallback fired.

In LiteLLM this is roughly 80 lines of middleware, a separate observability tool (Helicone or Phoenix), a separate cache (Redis + custom code), a separate guardrail service (custom), and a separate prompt registry (custom or LangSmith). The Portkey version is one config block.

vs. the Field

vs. LiteLLM. LiteLLM is the de facto open-source LLM proxy. It is excellent at what it does: route any model through one OpenAI-shaped API. It is also, in 2026, complete. The team is conservative, the scope is "unified interface," and the features that matter in production — semantic caching, guardrails, prompt versioning, per-team rate limits — are community extensions or your own glue code. Use LiteLLM when you have one model and want it routed through one SDK. Use Portkey when you have a real production system with a real team.

vs. OpenRouter. OpenRouter is the consumer-grade router — great for indie developers, not great for enterprise. No self-host story, no guardrails, no prompt versioning, no per-team attribution. It is a smart router for hobby projects and one-person teams. Portkey is the enterprise-shaped layer above.

vs. Kong AI Gateway / Cloudflare AI Gateway / Envoy AI Gateway. These are generic API gateways that learned to speak OpenAI. They handle rate limiting, auth, and routing well. They do not understand LLM-specific concepts: token budgets, semantic caching, prompt injection, fallback on context-length errors, per-model cost attribution. If you already run Kong or Envoy for your service mesh and you only need basic routing, those are fine. If you need anything LLM-specific, they will be your limitation within a quarter.

vs. Helicone. Helicone is the observability layer. It logs every request, gives you a dashboard, supports caching and rate limits. It does not do guardrails, prompt versioning, or fallbacks at the level Portkey does. The right architecture: Portkey as the gateway, Helicone as a downstream observability sink. Or Portkey alone if you self-host the dashboard.

vs. LangSmith / LangFuse / Arize Phoenix. Those are observability-and-eval platforms. Portkey is the runtime that feeds them. Use Phoenix behind Portkey if you want eval-driven CI. Portkey does the routing; Phoenix does the scoring. Different layers.

Where It Fits

Reach for Portkey when:

• You have 3+ models in production or in planning (Anthropic + OpenAI + Bedrock + maybe Groq for cheap calls)
• You have a real team (3+ engineers) where cost attribution, prompt versioning, and per-developer rate limits matter
• You have a compliance story (PII, jailbreak detection, audit logging)
• You need fallback resilience — Anthropic had a 4-hour outage last quarter; it cost some teams six figures
• You want canary deployments: 5% of traffic on the new model, 95% on the old, shift the dial over a week

Skip Portkey when:

• You have one model and one developer — LiteLLM is fine, or just call OpenAI directly
• Your eval story is the whole product — Phoenix or Langfuse is the right primary tool, not a gateway
• You are allergic to SaaS-shaped open source. Portkey has a hosted offering that is the smoothest path; the OSS gateway works, but the dashboard and prompt registry are hosted-first
• You run a non-OpenAI-shaped stack. Portkey assumes the OpenAI chat completions API as the lingua franca. If your internal API is Anthropic-native or a custom format, the savings are smaller

What I do not love. The hosted-first posture is real. The open-source gateway is solid, but the dashboard, the prompt registry, and the eval UI are hosted. If you are a true open-source-only shop (no Portkey cloud account), you lose the UX layer. That is the same trade-off Phoenix and Langfuse make; not a deal-breaker, but be aware.

The semantic cache requires a Redis instance. Default config is fine; tuning the embedding model for semantic cache hits is a real ops decision. Pin it to a fast embedding model and benchmark the cache-hit rate weekly or the cache becomes a write-only data structure.

Gateway 2.0 pre-release (the 2.0.0 branch) merges what used to be the closed-source enterprise gateway into OSS. If you are evaluating today, check both main and 2.0.0 — the 2.0 surface is the one that matters going forward.

The Take

If you ship LLM features at a real company, the gateway is not optional in 2026. You need fallbacks because providers have outages. You need semantic caching because 40% of your traffic is the same 200 questions. You need guardrails because one bad prompt will end your week. You need cost attribution because finance will ask.

Portkey is the only open-source gateway that ships all of it as one product. LiteLLM is the proxy. OpenRouter is the router. Helicone is the observer. Portkey is the control plane. If you only need one, pick LiteLLM. If you need four, pick Portkey.

The Series A in 2026 and the Gateway 2.0 OSS merge mean the company is funded, shipping, and committing to the open-source surface. The OSS gateway is the production-quality piece. Run it in your infra, point it at your own Postgres and your own Redis, ship the same product as the hosted offering. That is the open-source-shaped control plane I have wanted since 2024.

— Mr. Technology

*Portkey: github.com/Portkey-AI/gateway — ~11K stars, MIT, TypeScript gateway + Python/Node SDKs. Series A announced 2026, Gateway 2.0 in OSS pre-release. Routes 1,600+ LLMs across 30+ providers, ships 50+ built-in guardrails, semantic cache, prompt versioning, canary deployments, per-key rate limits. Install: pip install portkey-ai. Self-host: Docker image on Docker Hub, Helm chart in the Portkey org. Hosted: app.portkey.ai.*