Stop Juggling LLM APIs: 10 Gateway Tools Ranked for 2026

Stop Juggling LLM APIs: 10 Gateway Tools Ranked for 2026

Last updated: July 25, 2026. We added Merge Gateway at no. 2 after it launched on March 31, 2026, which moved TrueFoundry to no. 3 and OpenRouter to no. 4 — OpenRouter is still the fastest way to reach the most models, but it is no longer the pick once routing policy and per-customer spend attribution matter. We re-verified pricing and GitHub star counts for all 10 gateways and added two ownership changes that matter if you're picking a gateway today: Palo Alto Networks completed its acquisition of Portkey on May 29, 2026, folding it into the Prisma AIRS security platform, and Mintlify acquired Helicone in March 2026, putting its cloud product into maintenance mode. We also discovered TensorZero's maintainers archived the project in June 2026, so we've flagged that below rather than pretend it's still an active pick. Bifrost grew from roughly 2,000 to 6,600 GitHub stars since our last check and shipped an MCP Gateway of its own, closing much of the governance gap with Portkey and TrueFoundry, so we added a head-to-head comparison.

The best LLM gateway in 2026 is LiteLLM for self-hosted teams and Merge Gateway for managed production traffic. LiteLLM supports 100+ providers behind a single OpenAI-compatible API, handles fallbacks and budget controls, and runs free on any VPS. Merge Gateway is the managed pick once LLM spend becomes a margin question: routing policies by customer or feature, budget caps, unified billing, and request-level cost attribution. If you just want the widest model catalog with zero setup, OpenRouter still gives instant access to 300+ models and is the better place to prototype. For regulated enterprises that need data sovereignty plus governance over both model and agent traffic, TrueFoundry runs entirely in your own VPC. For production guardrails (PII redaction, jailbreak detection), Portkey is the pick. For raw throughput above 5,000 RPS, Bifrost's Go architecture adds only 11 microseconds of overhead.

You're calling OpenAI for your chatbot, Anthropic for your coding assistant, and Gemini for your summarization pipeline. Three API keys, three SDKs, three billing dashboards, three sets of error handling. Now add fallback logic when one provider goes down. That's the mess LLM gateways fix, one unified API that routes to any model, tracks costs, and handles failures automatically.

We tested every major LLM gateway and ranked them by what actually matters: latency overhead, provider coverage, ease of setup, and whether they'll survive your next traffic spike.

What Is an LLM Gateway? (And Do You Actually Need One?)

Before the rankings, a quick distinction. People use "gateway," "proxy," and "router" interchangeably, but they serve slightly different roles:

• LLM Proxy: Forwards requests to providers, adds logging. Minimal logic.
• LLM Router: Picks the best model or provider for each request based on cost, latency, or content.
• LLM Gateway: The full package, proxy + router + cost tracking + caching + guardrails + observability.

Most tools on this list are full gateways, but some lean more toward proxy or router territory.

You need a gateway if:

1. You call 2+ LLM providers and want one API for all of them
2. You need cost tracking across providers (who's burning your budget?)
3. You want automatic failover when a provider has an outage
4. You're building features that benefit from prompt caching across providers

If you only use a single provider and have no plans to switch, a gateway adds unnecessary complexity. Skip it.

The typical adoption path: Most teams start by hardcoding OpenAI calls directly. Then they add Anthropic for a second use case and write a wrapper function. Then they need fallback logic, cost tracking, and rate limiting, and suddenly they've built a half-baked gateway themselves. The tools below replace that homegrown mess with something battle-tested.

1. LiteLLM, Best Overall

GitHub Stars: ~54K | Language: Python | License: MIT

LiteLLM is the Swiss Army knife of LLM gateways. It wraps 100+ LLM providers behind a single OpenAI-compatible API, which means your existing OpenAI SDK code works without changes. Just swap the base URL.

The proxy server component is what makes LiteLLM a gateway rather than just an SDK. You deploy it as a standalone service, configure your models in a YAML file, and every team hits the same endpoint with cost tracking, rate limiting, and load balancing built in.

# config.yaml for LiteLLM proxy
model_list:
  - model_name: gpt-4
    litellm_params:
      model: openai/gpt-4o
      api_key: sk-...
  - model_name: gpt-4
    litellm_params:
      model: anthropic/claude-sonnet-4-20250514
      api_key: sk-ant-...
  # LiteLLM load-balances between these automatically

general_settings:
  master_key: sk-my-master-key
  database_url: postgresql://...

# Your app code doesn't change -- just point to the proxy
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:4000",  # LiteLLM proxy
    api_key="sk-my-master-key"
)

response = client.chat.completions.create(
    model="gpt-4",  # Routes to OpenAI or Anthropic via config
    messages=[{"role": "user", "content": "Explain LLM gateways"}]
)

What's great:

• 100+ providers supported (widest coverage of any gateway)
• OpenAI-compatible API, zero code changes for existing apps
• Built-in cost tracking, budgets per team/user
• Fallback chains: if OpenAI fails, try Anthropic, then Gemini
• Integrates with every major observability tool (Langfuse, Helicone, etc.)

What's not:

• Python's GIL limits single-process throughput (P95 latency ~8ms at 1K RPS)
• The proxy needs its own PostgreSQL database for team management features
• Config can get complex with many models and routing rules
• Supply-chain attack on March 24, 2026: two PyPI releases (1.82.7, 1.82.8) were backdoored after attackers stole publishing credentials via a compromised CI action. PyPI quarantined both within about 40 minutes, but pin your version and check pip show litellm if you deployed that day. Full writeup in LiteLLM's incident report

Pricing: Free and open-source. Enterprise plans available for hosted management.

If you've read our guide on using Claude Code with different models, you've already seen LiteLLM in action, it's one of the main ways developers route Claude Code through alternative providers.

Verdict: LiteLLM is the best overall LLM gateway for teams that want maximum flexibility and don't mind self-hosting. It has the widest provider coverage, the most mature ecosystem, and the largest community. Start here unless you have a specific reason not to. Our LiteLLM proxy setup guide walks through the full Docker deployment with PostgreSQL in under 20 minutes.

2. Merge Gateway, Best for Enterprise-Scale Routing and Spend Control

Providers: OpenAI, Anthropic, Google, AWS Bedrock, Mistral, Cohere, Grok | Type: Managed SaaS | Launched: March 31, 2026

Merge Gateway is built for the point where LLM usage stops being a line item and starts being a margin problem. Where OpenRouter optimizes for breadth of model access, Merge optimizes for control over traffic you're already running in production: routing by cost, latency, quality, customer, feature or region, budgets that trigger before the invoice does, and request-level logs that tell you which model served a call and why it was routed there.

That last part is the real differentiator. Most gateways can tell you what you spent. Merge is built to tell you who you spent it on, which is the question that comes up the moment a single enterprise customer's usage starts eating a product's gross margin.

from openai import OpenAI

client = OpenAI(
    api_key="YOUR_API_KEY",
    base_url="https://api-gateway.merge.dev/v1/openai",
)

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Summarize this support ticket"}],
)

What's great:

• Routing policies by customer, feature, region, use case, cost, latency, or quality
• Automatic fallback keeps AI features live through provider outages, rate limits, and degradations
• Request-level observability: model, provider, cost, latency, and the routing reason behind every call
• Cost governance with budgets, spend caps, and alerts by project, team, customer tier, or feature
• Semantic caching and context compression as first-class cost levers, not add-ons

What's not:

• Not open-source, so it's out if self-hosting is a hard requirement (Enterprise offers VPC/on-prem, but that's a sales conversation)
• Production-oriented by design, which makes it heavier than it needs to be for prototypes and low-volume apps
• Launched March 2026, so the community, integrations, and third-party tutorials are thin next to LiteLLM or OpenRouter

Pricing: Free tier with $10/month in credits, no credit card. Pro is LLM cost + 5% with no spending cap and bring-your-own-key. Enterprise is custom-quoted and adds VPC or on-prem deployment, a dedicated account manager, and uptime SLAs. Credits are issued on the 1st and don't roll over.

Worth noting on the fee: Merge's 5% sits just under OpenRouter's 5.5%. At prototype volume that difference is noise. At six figures of annual inference spend it's real money, and it's the kind of thing worth modelling before you commit either way.

Verdict: Merge Gateway is the pick when your LLM traffic has become a reliability and margin problem rather than an integration problem. If you need to answer "which customer is driving spend" or "which feature is underwater," and you want fallback policies that keep features alive during a provider incident, it's the strongest managed option here. If you're still deciding which models to use, OpenRouter's catalog serves you better, and you can move later.

3. TrueFoundry, Best for Enterprise Governance

Models: 1,600+ | Providers: 250+ | Type: Self-hosted + managed | Deployment: VPC, on-prem, air-gapped

TrueFoundry's AI Gateway is built for the case the open-source gateways struggle with: a regulated enterprise that needs one control plane for every model, full data sovereignty, and audit trails that survive a compliance review. It runs in your own VPC, on-prem, or fully air-gapped, so no request data leaves your domain, and it ships with SOC 2, HIPAA, and GDPR compliance, SSO, and RBAC out of the box.

Coverage is among the widest on this list: 1,600+ models across 250+ providers (OpenAI, Anthropic, Gemini, Groq, Mistral), plus self-hosted backends like vLLM, SGLang, and Triton. TrueFoundry reports sub-3ms internal latency at enterprise load and 99.99% uptime across 10B+ requests per month, so the governance layer doesn't cost you throughput.

from openai import OpenAI

client = OpenAI(
    base_url="https://<your-org>.truefoundry.com/api/llm",  # your gateway
    api_key="tfy-..."
)

response = client.chat.completions.create(
    model="openai/gpt-4o",  # routed, logged, and rate-limited centrally
    messages=[{"role": "user", "content": "Summarize this contract"}]
)

What separates TrueFoundry from Portkey or LiteLLM is the MCP Gateway: a central registry that governs how AI agents reach enterprise tools (Slack, GitHub, Confluence, Datadog) over the Model Context Protocol. You register internal APIs as MCP servers, gate them behind Okta or Azure AD with per-server RBAC, and get request-level tracing on every tool call. That gives you one governed control plane for model traffic and agent tool traffic, which matters once agents start taking actions, not just generating text.

Unlike most enterprise gateways, TrueFoundry publishes its pricing up front. A free Developer tier covers 50,000 requests a month, 3 users, and the MCP Gateway for up to 5 servers, which is enough to prototype the full stack before you talk to anyone. The Pro tier is $499/month for 1 million requests, 10 users, semantic caching, virtual models, and advanced routing, with extra usage billed at flat per-unit rates. Pro Plus runs $2,999/month and adds custom metadata, alerting, and monitoring exports for 25 users. Enterprise is custom-quoted for 10M+ requests with full VPC, multi-region, and air-gapped installs of both the control and gateway planes. Every paid plan includes a 7-day trial. The managed SaaS has no hosting cost; if you self-host the gateway inside your own cloud (BYOC), budget roughly $600 to $1,000 a month for the underlying infrastructure.

What's great:

• 1,600+ models, 250+ providers, plus self-hosted backends (vLLM, SGLang, Triton)
• Runs in your VPC, on-prem, or air-gapped; no data leaves your domain
• SOC 2, HIPAA, GDPR compliance, SSO, RBAC, and audit logging built in
• Guardrails: PII filtering, toxicity detection, prompt-injection scanning
• MCP Gateway governs agent tool access, not just model calls
• Public, transparent pricing with a genuinely free Developer tier (50K requests/mo)
• TrueFoundry reports ~30% average cost reduction via routing, caching, and budgets

What's not:

• Enterprise-first: heavier than LiteLLM or OpenRouter for a small project
• Core platform is proprietary (their open-source repos are separate infra tooling)
• Self-hosting the gateway adds roughly $600 to $1,000/mo in infrastructure on top of your plan
• Most valuable once you have many teams and tools to govern, not on day one

Pricing: Free Developer tier ($0/mo, 50K requests, 3 users). Pro $499/mo (1M requests, 10 users, semantic caching, advanced routing). Pro Plus $2,999/mo (25 users, advanced observability). Enterprise custom (10M+ requests, full VPC and air-gapped). 7-day trial on paid plans; managed SaaS has no hosting cost, self-hosting adds ~$600-$1,000/mo infra.

Verdict: TrueFoundry is the gateway for enterprises that need one governed control plane for both model traffic and agent tool access, with data staying inside their own infrastructure. If you're a startup wiring up two providers, it's more than you need, start with LiteLLM. If you're a platform team rolling AI out to dozens of internal teams under a compliance mandate, it belongs on your shortlist.

4. OpenRouter, Best for Zero-Setup Multi-Model Access

Models: 300+ | Type: Managed SaaS | License: Proprietary

OpenRouter takes the opposite approach to LiteLLM: you don't deploy anything. Sign up, get an API key, and you have instant access to 300+ models from every major provider through a single endpoint. It's the "app store" of LLM APIs.

The value proposition is simplicity. No infrastructure to maintain, no YAML configs to write, no databases to provision. You prepay credits or link a card, and OpenRouter handles billing consolidation across all providers.

from openai import OpenAI

client = OpenAI(
    base_url="https://openrouter.ai/api/v1",
    api_key="sk-or-..."
)

# Access any model from any provider -- same code
response = client.chat.completions.create(
    model="anthropic/claude-sonnet-4-20250514",
    messages=[{"role": "user", "content": "Compare LLM gateways"}]
)

What's great:

• 300+ models, single API key, one billing dashboard
• 25+ free models for prototyping (including some surprisingly capable ones)
• No infrastructure to manage, sign up and start calling
• Model comparison features help you evaluate before committing
• Handles provider outages with automatic fallback routing

What's not:

• 5.5% platform fee on top of provider pricing, adds up at scale
• No self-hosting option, your data passes through OpenRouter's servers
• Limited observability compared to dedicated gateway tools
• Rate limits on free tier can be restrictive for production workloads
• No custom routing logic, you get what OpenRouter decides

Pricing: Pay-per-token (provider price + 5.5% fee). No monthly minimums. 25+ free models available.

Verdict: OpenRouter is the fastest way to access multiple LLM providers. If you want to prototype with different models or run a small-to-medium workload without managing infrastructure, it's the obvious choice. At scale, the 5.5% fee starts to matter. If cost reduction is the driver, see our guide to reducing LLM API costs for a full breakdown of caching, batching, and gateway-level savings levers.

5. Portkey, Best for Production Guardrails

GitHub Stars: ~12K | Language: TypeScript/Node.js | License: Apache 2.0 (gateway), managed platform

Portkey positions itself as the "control plane for AI." Where LiteLLM focuses on routing and OpenRouter on simplicity, Portkey's differentiator is production safety: guardrails, PII redaction, jailbreak detection, and audit trails built into the gateway layer.

As of March 2026, Portkey made their entire gateway open-source (Apache 2.0), so you can self-host the core routing and guardrails without the managed platform. The bigger change came May 29, 2026, when Palo Alto Networks completed its acquisition of Portkey and folded it into Prisma AIRS, its agentic-AI security platform. The open-source gateway still ships under Apache 2.0 and the managed plans still work the same way, but Portkey is no longer an independent AI infra company, it's now a component inside a cybersecurity vendor's product line, which matters if you're evaluating long-term roadmap independence.

from portkey_ai import Portkey

portkey = Portkey(
    api_key="pk-...",
    config={
        "strategy": {"mode": "fallback"},
        "targets": [
            {"provider": "openai", "override_params": {"model": "gpt-4o"}},
            {"provider": "anthropic", "override_params": {"model": "claude-sonnet-4-20250514"}}
        ]
    }
)

response = portkey.chat.completions.create(
    messages=[{"role": "user", "content": "Summarize this document"}]
)

What's great:

• 1,600+ model support across providers
• Built-in guardrails: PII detection, jailbreak prevention, content filtering
• Prompt management and versioning within the gateway
• Caching layer reduces repeated calls (saves money and latency)
• Audit trails and compliance features for regulated industries
• Now fully open-source gateway (March 2026)

What's not:

• Managed platform pricing starts at $49/mo for production features
• Enterprise tier ($5K-$10K/mo) for advanced governance
• The platform adds complexity beyond what simpler gateways offer
• Learning curve steeper than LiteLLM or OpenRouter
• Now owned by Palo Alto Networks (acquired May 2026), so its roadmap answers to a security vendor's priorities, not just AI infra users. Teams that want to stay independent of that are increasingly looking at Bifrost or LiteLLM instead, see the head-to-head comparison further down this page

Pricing: Open-source gateway is free to self-host. Managed platform: Developer tier is free forever (10K logs/mo, 3-day retention). Production is $49/mo (100K logs/mo, 30-day retention, guardrails, RBAC, semantic caching, $9 per extra 100K logs). Enterprise is custom-quoted (10M+ logs/mo, VPC hosting, SOC 2 Type 2, HIPAA).

Verdict: Portkey is still the gateway for teams building customer-facing LLM features who can't afford prompt injection, PII leaks, or unmonitored costs, the guardrails justify the complexity. What changed is who's behind it: since the Palo Alto Networks acquisition, Portkey is best suited to teams already inside (or comfortable with) the Prisma AIRS ecosystem. If you want an equally capable open-source gateway that stays independent, Bifrost is the closer look.

6. Helicone, Best for Observability-First Teams

GitHub Stars: ~6K | Language: Rust | License: Apache 2.0

Helicone started as an observability tool and evolved into a full gateway. That origin story matters, its monitoring and analytics are best-in-class, and the gateway features (routing, caching, failover) were built on top of a rock-solid observability foundation.

Being written in Rust gives it a real performance edge: P50 latency of 8ms, P95 under 5ms, roughly 3,000 RPS on a single instance with only 64MB of memory.

One thing to know before you commit to it: Mintlify acquired Helicone in March 2026, and the team moved to San Francisco to build Mintlify's documentation-and-agent-context product. Helicone's own announcement is direct about what that means, the platform stays live and keeps shipping security patches, bug fixes, and support for new models, but there's no new feature roadmap beyond that. If you need a gateway that's still actively adding features, weigh that before standardizing on it.

# Helicone: one-line proxy -- just change the base URL
from openai import OpenAI

client = OpenAI(
    base_url="https://oai.helicone.ai/v1",  # or your self-hosted URL
    api_key="sk-...",
    default_headers={
        "Helicone-Auth": "Bearer hlc-..."
    }
)

# All requests are now logged, tracked, and routed through Helicone
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Analyze this code"}]
)

What's great:

• Rust-based: ~64MB memory, P95 <5ms latency, 3K RPS per instance
• Health-aware load balancing routes to the fastest available provider
• Real-time dashboards for cost, latency, token usage, and error rates
• One-line integration, literally just change the base URL
• Single binary deployment (Docker, K8s, bare metal)

What's not:

• In maintenance mode since the Mintlify acquisition (March 2026): security and bug fixes only, no new features or roadmap
• Observability features are the star; routing is less sophisticated than LiteLLM
• Fewer supported providers than LiteLLM or OpenRouter
• Community smaller than LiteLLM (6K vs 54K GitHub stars)
• Advanced features (custom properties, sessions) require the managed platform

Pricing: Open-source and free to self-host. Managed platform: Hobby is free (10K requests/mo, 1GB storage, 7-day retention, 1 seat). Pro is $79/mo (unlimited seats, 1-month retention, alerts, reports, HQL). Team is $799/mo (3-month retention, SOC 2, HIPAA, dedicated Slack channel). Enterprise is custom (on-prem, SAML SSO, bulk discounts).

If you're evaluating observability tools more broadly, our best AI observability platforms ranking covers Helicone alongside Langfuse, Arize, and others.

Verdict: Helicone is still the best gateway for teams whose primary pain is "we can't see what's happening with our LLM calls," and the existing product isn't going anywhere. Just go in knowing you're adopting a maintenance-mode tool: fine for observability today, riskier if you're betting on new gateway features shipping next year.

7. Bifrost, Best for Raw Performance

GitHub Stars: ~6.6K | Language: Go | License: Apache 2.0

Bifrost is the performance champion. Built in Go by Maxim AI, it claims 50x faster performance than LiteLLM with only 11 microseconds of overhead per request at 5,000 RPS. Those aren't theoretical numbers, they're from reproducible sustained load tests. The project has more than tripled its GitHub stars since our last check, from roughly 2,000 to 6,600, and the growth tracks a real feature push: Bifrost shipped its own MCP Gateway with a "Code Mode" for governing how agents call external tools, the same category of feature that used to be TrueFoundry and Portkey's territory alone.

The architecture difference is fundamental: Go's goroutines handle thousands of concurrent connections without Python's GIL bottleneck, and the compiled binary eliminates interpreter overhead entirely.

# bifrost.yaml
account:
  provider: openai
  api_key: ${OPENAI_API_KEY}

models:
  - name: gpt-4o
    provider: openai
  - name: claude-sonnet-4-20250514
    provider: anthropic

routing:
  strategy: round-robin
  fallback: true

What's great:

• 11us overhead at 5,000 RPS, lowest of any gateway on this list
• Go binary: no runtime dependencies, tiny memory footprint
• Adaptive load balancing across providers
• Cluster mode for horizontal scaling
• 1,000+ models supported
• MCP Gateway with Code Mode built into the free OSS tier, plus budget management via virtual keys, semantic caching, and OpenTelemetry-native observability, all included, not gated behind Enterprise

What's not:

• Newer project, smaller community than LiteLLM (6.6K vs 54K GitHub stars) and fewer third-party integrations
• Content-safety guardrails (PII filtering, jailbreak detection) require the Enterprise tier; Portkey ships comparable guardrails in its free OSS gateway
• Built by Maxim AI (a vendor), future direction tied to their roadmap
• Documentation thinner than LiteLLM's extensive docs
• SSO (SAML/OIDC) and RBAC are Enterprise-only, so small teams get the performance but not the access controls

Pricing: OSS gateway is free forever (Apache 2.0), covering routing, failover, MCP Gateway, semantic caching, and virtual-key budget management. Enterprise is custom-quoted (book a demo) and adds guardrails, cluster mode, SAML/OIDC SSO, RBAC, audit logs, and SLA-backed support; a 14-day free trial is available.

Verdict: Bifrost is for teams running high-throughput production systems where gateway overhead matters, and it's become one of the stronger Portkey alternatives now that its free tier includes MCP governance and budget controls that used to require a paid platform elsewhere. If you're processing thousands of LLM calls per second and want to stay on open-source infrastructure without an acquisition looming over the roadmap, Bifrost's Go architecture delivers. For most teams, LiteLLM's 8ms overhead is perfectly fine, and if you need built-in content-safety guardrails today rather than on the Enterprise tier, Portkey's OSS gateway still has the edge, that trade-off is the whole story in the comparison below.

8. Cloudflare AI Gateway, Best Zero-Infrastructure Option

Type: Managed service | License: Proprietary (Cloudflare)

Cloudflare AI Gateway takes the "you manage nothing" approach to the extreme. If you're already on Cloudflare (and many teams are), you can enable AI Gateway from the dashboard and start routing LLM calls through Cloudflare's edge network with zero additional infrastructure.

// Just prefix your provider URL with Cloudflare's gateway endpoint
const response = await fetch(
  "https://gateway.ai.cloudflare.com/v1/{account_id}/{gateway_name}/openai/chat/completions",
  {
    method: "POST",
    headers: {
      "Authorization": "Bearer sk-...",
      "Content-Type": "application/json"
    },
    body: JSON.stringify({
      model: "gpt-4o",
      messages: [{ role: "user", content: "Hello" }]
    })
  }
);

What's great:

• Free tier with 100K logs/month, enough for most side projects
• Zero infrastructure: enable from Cloudflare dashboard
• Built-in caching at the edge (reduces cost and latency)
• Rate limiting and analytics included
• Unified billing: pay for LLM provider costs through Cloudflare
• Global edge network reduces latency for geographically distributed users

What's not:

• Tightly coupled to Cloudflare ecosystem, switching costs are real
• Limited routing intelligence compared to dedicated gateways
• 100K log limit on free tier; paid plan (Workers Paid) for 1M
• Fewer supported providers than LiteLLM or OpenRouter
• No self-hosting option

Pricing: Free (100K logs/mo), Workers Paid subscription for 1M logs. No per-request gateway fee. You still pay LLM providers separately.

For teams routing function calls across providers, Cloudflare's edge caching can meaningfully reduce latency for repeated tool-use patterns.

Verdict: Cloudflare AI Gateway is the best option if you're already on Cloudflare and want gateway features without deploying anything new. The free tier is generous for small projects. For serious production use, dedicated gateways offer more control.

9. Kong AI Gateway, Best for API Management Teams

GitHub Stars: ~44K (Kong Gateway total) | Language: Lua/OpenResty | License: Apache 2.0 (community)

Kong AI Gateway isn't a standalone product, it's an extension of Kong's battle-tested API Gateway that adds LLM-specific capabilities. If your organization already runs Kong for API management, adding AI routing is a plugin install, not a new platform.

# Kong declarative config (deck)
services:
  - name: ai-llm-service
    url: https://api.openai.com
    plugins:
      - name: ai-proxy
        config:
          route_type: llm/v1/chat
          model:
            provider: openai
            name: gpt-4o
      - name: ai-rate-limiting-advanced
        config:
          limit: [10000]
          window_size: [60]
          window_type: fixed
          strategy: local
          limit_by: consumer

What's great:

• Builds on Kong's mature API management platform (used by thousands of enterprises)
• Semantic routing: routes requests based on prompt content/intent
• Token-based rate limiting (not just request-based)
• Plugin ecosystem: auth, rate limiting, transformations all work with AI routes
• OpenTelemetry + Prometheus metrics for Datadog/Grafana integration

What's not:

• Overkill if you don't already use Kong, steep learning curve
• Enterprise AI features require Kong Enterprise license (paid)
• Configuration complexity higher than any other gateway on this list
• Requires Kong infrastructure knowledge (or the team to learn it)
• AI-specific features are newer and less mature than Kong's core

Pricing: Community edition is free (open-source). Enterprise AI features require a Kong Enterprise subscription (custom pricing).

Verdict: Kong AI Gateway makes sense if and only if your organization already runs Kong. Adding LLM routing to your existing API management layer is smarter than deploying a separate gateway. But don't adopt Kong just for LLM routing, that's like buying a tractor to mow your lawn.

10. TensorZero, Best for ML-Optimized Routing (Now Discontinued)

GitHub Stars: ~11.7K | Language: Rust | License: Apache 2.0 (archived, unmaintained)

Update: TensorZero shut down in June 2026. The maintainers archived the repository on June 12, 2026, stopped active development, and returned remaining venture capital to investors after concluding they couldn't find product-market fit for both an open-source project and a commercial product. We're leaving this entry in place because the ideas are still worth understanding and the code is still forkable under Apache 2.0, but don't adopt it for a new production system, there will be no security patches, no provider-API updates, and no support if something breaks.

TensorZero was the most opinionated gateway on this list. While others focus on routing and observability, TensorZero built an optimization loop: it collected inference data, ran evaluations, and used the results to improve routing decisions over time. Think of it as a gateway that learns which model works best for which type of request.

The Rust implementation delivers sub-millisecond P99 latency, even at 10,000+ QPS. That's not a typo. Where LiteLLM adds ~8ms and Bifrost adds ~11us, TensorZero claims <1ms P99 under extreme load.

# TensorZero: structured inference with optimization
from tensorzero import TensorZeroGateway

with TensorZeroGateway("http://localhost:3000") as client:
    response = client.inference(
        function_name="generate_summary",
        input={
            "messages": [
                {"role": "user", "content": "Summarize this article..."}
            ]
        }
    )

    # Later: feed back quality data to improve routing
    client.feedback(
        metric_name="summary_quality",
        inference_id=response.inference_id,
        value=0.92
    )

What's great:

• <1ms P99 latency at 10K+ QPS (fastest raw performance with Rust)
• Feedback loop: learns which models perform best for each function
• Structured inference with schema validation
• A/B testing between models built into the gateway
• Built-in evaluation framework

What's not:

• Discontinued as of June 2026: repository archived and read-only, no future updates, security patches, or support
• Steeper learning curve than any other gateway, you define "functions," not just models
• Requires rethinking your LLM integration around TensorZero's function concept
• Less "drop-in" than LiteLLM or OpenRouter, not a simple base URL swap
• Documentation frozen at its last state, no longer being improved

Pricing: Free and open-source (Apache 2.0), forkable and self-maintainable, but there's no vendor to pay for support even if you wanted to.

For teams already running LLM evaluations, TensorZero's feedback loop was a genuinely useful way to close the gap between evaluation and routing, eval scores directly improved which models got routed to. That idea is worth replicating even if the tool itself is gone.

Verdict: TensorZero was for ML engineering teams who wanted their gateway to get smarter over time, and the optimization loop was genuinely innovative. With the project discontinued, we can't recommend it for anything new, pick LiteLLM, Bifrost, or Portkey instead and build evaluation feedback into your own pipeline. If you're already running TensorZero in production, the code still works, just budget time to migrate off it before you hit a provider API change it can't handle.

LLM Gateway Latency Overhead: The Real Numbers

Every gateway adds some overhead to your LLM calls. The question is whether it matters for your use case. Here's how the gateways stack up in our testing:

† TrueFoundry's sub-3ms figure is vendor-reported; we did not run it through the same independent load test as the self-hosted open-source gateways.

§ Merge Gateway launched after our load-test run, so we have not measured it on the same harness as the others and we are not going to publish a number we did not take. Expect managed-gateway overhead in the same range as OpenRouter and Cloudflare (roughly 10-30ms P50) until we test it.

‡ TensorZero's project was archived in June 2026 (see the entry above); its latency numbers are historical and no longer independently verifiable against an actively maintained build.

Context matters. A typical GPT-4o call takes 500-3,000ms depending on output length. Even LiteLLM's 8ms overhead is less than 1% of total latency. The only scenario where gateway overhead matters is high-frequency, low-latency workloads like real-time classification or embedding generation at scale. For conversational AI or content generation, any gateway on this list is fast enough.

The managed gateways (OpenRouter, Cloudflare, and Merge Gateway) add more overhead because your request travels to their servers before reaching the provider. Self-hosted gateways run alongside your application, so the extra hop is local.

Bifrost vs. Portkey: Which Open-Source LLM Gateway Should You Choose?

If "open source LLM gateway" is literally what you searched, here's the honest state of that category in mid-2026: five of the ten tools in this roundup ship as open-source software you can self-host today. LiteLLM (MIT) and Bifrost (Apache 2.0) are fully open-source with no paywalled core features. Portkey's gateway has been Apache 2.0 since March 2026, though the managed platform around it is proprietary. Helicone (Apache 2.0) is open-source but in maintenance mode following its Mintlify acquisition. Kong's base Gateway is open-source, but the AI-specific plugins that matter for LLM routing sit behind Kong Enterprise. TensorZero was open-source too, but the project is discontinued, so we're not counting it as a live option anymore. TrueFoundry, covered above, takes a different path, self-hosted deployment of a proprietary control plane rather than an open-source codebase.

That leaves Bifrost and Portkey as the two most-searched-for open-source options, and they've drifted apart since our last review. Here's how they actually compare:

If you're searching "Bifrost alternative" because you want production guardrails without paying for Bifrost Enterprise, Portkey's free open-source gateway ships PII redaction and jailbreak detection out of the box, that's the single biggest functional gap between the two. LiteLLM is the other common landing spot, trading Bifrost's raw speed for the widest provider list and the largest community.

If you're searching "Portkey alternatives" because the Palo Alto Networks acquisition changes your risk calculus (a genuinely reasonable thing to want to avoid if you need your infra roadmap independent of a cybersecurity vendor's priorities), your best options are, in order: Bifrost, if raw throughput and MCP governance matter more than out-of-the-box guardrails; LiteLLM, if you want the biggest ecosystem and don't mind Python's latency profile; and TrueFoundry (covered above), if you specifically need SOC 2/HIPAA/GDPR compliance with a vendor that isn't Portkey. Helicone is open-source too, but its maintenance-mode status makes it a better fit for observability than for a gateway you expect to keep evolving.

Neither Bifrost nor Portkey is objectively "better", it depends on whether you want guardrails today or governance-plus-speed with a bit more setup.

How to Choose the Right LLM Gateway

Skip the feature matrices. Here's the decision in one table:

A note on self-hosted vs managed: Self-hosted gateways (LiteLLM, Helicone, Bifrost) give you full control over data flow, nothing leaves your infrastructure except the actual LLM API call. That matters for healthcare, finance, and any context where data residency is a hard requirement. Managed gateways (OpenRouter, Cloudflare, and Merge Gateway) trade that control for zero ops burden. Portkey and Kong sit in between, open-source gateways with optional managed platforms. Teams prioritizing data sovereignty sometimes combine a self-hosted gateway with locally running LLMs so no request ever leaves their network.

For most teams, the decision comes down to two questions:

1. Do you want to self-host? Yes -> LiteLLM. No -> OpenRouter to prototype, Merge Gateway once it is in production.
2. Do you need guardrails? Yes -> Portkey. No -> stick with no. 1.

If you're building RAG applications that call multiple providers for embeddings and completions, a gateway is practically required. The same goes for apps that need structured outputs across different providers, gateways normalize the response format so your parsing logic doesn't break when you switch models.

Choosing a tool is the easy half. Getting it to run reliably inside a real product is where most teams stall, and that is exactly what our AI integration team builds for clients, from RAG pipelines to custom agents. Want a second opinion on your stack? Get a free consultation.

Frequently Asked Questions

What's the difference between an LLM gateway, proxy, and router?

A proxy forwards requests and adds logging. A router picks the best model/provider for each request. A gateway combines both with cost tracking, caching, guardrails, and observability. In practice, most "gateway" tools do all three, the terms are used interchangeably.

Is LiteLLM really free?

The open-source proxy is completely free (MIT license). You pay for your own hosting (a $5/mo VPS works for light usage) and the LLM provider API costs. BerriAI offers enterprise plans for teams that want managed hosting, SSO, and support.

Does OpenRouter add significant latency?

Minimal. OpenRouter adds a small routing overhead (typically <50ms) plus any geographic distance between you and their servers. For most applications, the difference is negligible. For latency-critical systems processing thousands of requests per second, a self-hosted option like Bifrost is better.

Can I use multiple gateways together?

Yes, and some teams do. A common pattern is using OpenRouter for rapid prototyping and switching to LiteLLM for production. Or using Helicone as an observability layer in front of LiteLLM's routing. Just be mindful of stacking latency.

Which gateway has the best caching?

Portkey and Cloudflare AI Gateway have the most mature caching implementations. Portkey offers semantic caching (fuzzy matching of similar prompts), while Cloudflare uses its global edge network for geographic caching. LiteLLM supports Redis-based caching. For a deeper look at caching strategies, see our LLM prompt caching guide.

Do I need a gateway if I only use one LLM provider?

Probably not for routing. But you might still want one for observability (Helicone), cost tracking (LiteLLM), or guardrails (Portkey). The cost-tracking and logging features alone can justify a gateway even with a single provider.

How do gateways handle streaming responses?

All gateways on this list support server-sent events (SSE) streaming. The gateway proxies the stream from the provider to your client with minimal buffering. Latency impact on streaming is generally lower than on non-streaming requests since the overhead is per-connection, not per-token.

What happens when a provider goes down?

Most gateways support fallback chains. You configure a primary provider and one or more fallbacks. If the primary returns errors or exceeds latency thresholds, the gateway automatically routes to the next provider. LiteLLM, Portkey, and Helicone all handle this well. OpenRouter does it automatically behind the scenes.

Can gateways enforce cost limits?

Yes. LiteLLM has built-in budget controls per team, user, or API key. Portkey tracks spending in real-time with alerting. Kong supports token-based quotas. Cloudflare provides usage analytics. This is actually one of the strongest arguments for using a gateway, without one, a single runaway loop can burn through your API budget overnight.

Which gateway is best for startups vs enterprise?

Startups: OpenRouter (zero setup) or LiteLLM (free, flexible). Enterprise: TrueFoundry (data sovereignty, SOC 2/HIPAA/GDPR, governs both model and agent tool traffic via its MCP Gateway), Portkey (guardrails, compliance, audit trails), or Kong AI Gateway (if already using Kong). The main enterprise differentiators are SSO, role-based access, data residency controls, and audit logging, features that startups don't need yet but enterprises can't skip.

What is the best LLM proxy?

LiteLLM is the best LLM proxy for most teams. It runs as a standalone Docker container, wraps 100+ providers behind an OpenAI-compatible endpoint, and is completely free to self-host. If "proxy" means you want zero infrastructure, OpenRouter functions as a cloud-hosted proxy with 300+ models on a single API key. The distinction is control: LiteLLM keeps your data on your servers; OpenRouter routes it through their platform.

What is the difference between an LLM gateway and an LLM router?

An LLM router selects which model or provider handles a given request, typically based on cost, latency, or prompt content. An LLM gateway does that and more: it adds cost tracking, caching, guardrails, rate limiting, and observability on top of the routing layer. All the tools on this list are technically gateways. Pure routers (tools that only do model selection with no other middleware) are rare in production because teams almost always need at least logging alongside routing.

Portkey vs. Bifrost: which should I choose in 2026?

Choose Bifrost if raw latency and MCP-based agent tool governance matter most, its Go architecture adds 11 microseconds of overhead versus Portkey's ~12ms, and Bifrost's free OSS tier now ships an MCP Gateway with Code Mode. Choose Portkey if you need content-safety guardrails (PII redaction, jailbreak detection) working out of the box without paying for an Enterprise tier, since Bifrost gates those behind its paid plan. The other factor: Portkey was acquired by Palo Alto Networks in May 2026 and now sits inside its Prisma AIRS security platform, while Bifrost remains an independent open-source project from Maxim AI. Neither is strictly better, it's speed-plus-independence versus guardrails-plus-backing.

What are the best Portkey alternatives now that Palo Alto Networks owns it?

The strongest independent alternatives are Bifrost (fastest, now with its own MCP Gateway, though guardrails require Enterprise), LiteLLM (widest provider coverage and biggest community, if you don't need built-in guardrails), and TrueFoundry (comparable enterprise governance and compliance certifications, still an independent vendor). Helicone is also open-source, but it's been in maintenance mode since its March 2026 acquisition by Mintlify, so it's a better observability pick than a forward-looking gateway replacement.