1 Β· What an AI gateway is β and is not definition
An AI gateway is a reverse proxy that understands model traffic. Ordinary proxies see bytes and URLs; an AI gateway sees models, prompts, tokens and cost β which is what lets it do rate limiting by tokens, caching by meaning, and budgets in dollars.
| It is often confused with | The actual difference | Do you need both? |
|---|---|---|
| An API gateway (Kong, APIM, Apigee) | An API gateway counts requests; an AI gateway counts tokens, understands streaming responses, and knows a model alias from a deployment | Usually the same product with AI plugins enabled β one proxy, not two |
| An SDK wrapper (a shared internal library) | A library only governs code that imports it, in one language, and needs a redeploy to change policy | The library is fine for one team; it fails the moment a second language or a vendor tool appears |
| A service mesh (Istio, Linkerd) | The mesh secures service-to-service traffic; it has no idea what a token is | Complementary β the gateway usually runs as a service inside the mesh |
| A model router (OpenRouter, Not Diamond) | Routers pick the best/cheapest model per request; that is one feature of a gateway, sold separately and usually hosted by a third party | A router can sit behind your gateway, but it should not replace it |
| An inference server (vLLM, TGI, Ollama) | Inference servers run the model; gateways route to them | Both, if you self-host models β the gateway makes vLLM look like every other provider |
2 Β· Where it sits in the request path diagram
(code you don't own)"] --> GW subgraph GW["AI GATEWAY β one OpenAI-shaped API"] direction TB K["Virtual key
authn Β· team Β· budget"] --> RL["Rate limit
RPM Β· TPM"] RL --> GI["Guardrails in
PII Β· injection"] GI --> C{"Cache hit?"} C -->|"yes"| RET["return cached"] C -->|"no"| R["Route Β· load balance"] R --> FB["Retry Β· fallback"] FB --> GO["Guardrails out"] end GW --> P1["Anthropic API"] GW --> P2["Bedrock Β· Vertex Β· Foundry"] GW --> P3["vLLM / self-hosted"] GW -.->|"spend Β· payloads Β· spans"| OBS[("Observability
+ billing")] classDef a fill:#eef6ff,stroke:#3b7bd0,color:#1f5fa8; classDef b fill:#f3f0ff,stroke:#764ba2,color:#4a3b7a; class A1,A2,A3,P1,P2,P3 a; class K,RL,GI,C,R,FB,GO,RET b;
Every arrow into the box is an application that needed no code to gain keys, quotas, budgets, caching, guardrails and logging. The third one is the argument: you cannot add a library to software you do not own.
3 Β· The 14 capabilities checklist
Use this as the RFP checklist. The core items are why the gateway exists; the rest are why you eventually pick one over another.
4 Β· Open-source gateways β what each offers matrix
| Gateway | Runtime | Providers | Virtual keys | Budgets | Token rate limit | Semantic cache | Guardrails | OTel | MCP | Admin UI | The honest summary |
|---|---|---|---|---|---|---|---|---|---|---|---|
| LiteLLM ProxyOSS + SaaS | Python | 100+ | β | β | β | β Redis | β Presidio, Lakera, Bedrock | β | β | β | The richest OSS feature set, by a distance. Also the heaviest hop β it is a Python service, so size it and run more than one. |
| Portkey GatewayOSS + SaaS | TypeScript | 200+ | β platform | β | β | β | β mostly paid | β | β | β paid | Very low added latency, edge-deployable, tiny footprint. The OSS core routes; the governance lives in the paid platform. |
| Bifrost (Maxim)OSS | Go | β | β | β | β | β | β | β | β native | β | Lowest overhead of the full-featured options, with first-class MCP support. Younger and smaller community. |
| Envoy AI GatewayOSS | Envoy / Go | β | β via k8s | β | β token-aware | β external | β external | β | β | β CRDs | Kubernetes-native, configured as CRDs, built on the proxy that already runs the internet. Scales hardest, operates heaviest. |
| Kong AI GatewayOSS + SaaS | Lua / Nginx | β
ai-proxy | β consumers | β | β advanced is enterprise | β enterprise | β
ai-prompt-guard | β | β | β enterprise | The right answer if Kong is already your ingress β AI becomes another plugin on infrastructure you already operate. |
| Apache APISIXApache 2.0 | Lua / Nginx | β | β | β | β | β | β | β | β | β | Same logic as Kong, fully open β less gated behind an enterprise tier. |
| HigressApache 2.0 | Go / Envoy | β | β | β | β | β | β | β | β | β | Unusually complete AI plugin set for an ingress; strong in Kubernetes estates, documentation is improving from a China-first base. |
| Gloo AI Gateway (Solo.io)OSS + SaaS | Envoy | β | β | β | β | β | β | β | β | β | Policy-rich and mesh-adjacent. Enterprise-shaped in both the good and the expensive sense. |
| HeliconeOSS + SaaS | TypeScript | β | β | β | β | β | β | β | β | β | Observability-first proxy that grew gateway features. One hop for both jobs β simple, and a shared blast radius. |
| Arch (Katanemo), TrueFoundry, llm-gatewaymixed | Rust / Go | β | β | β | β | β | β intent models | β | β | β | Newer entrants. Arch is interesting for running small models in the proxy for routing and guardrail decisions. |
5 Β· Cloud-provider gateways β the real comparison key table
Each cloud solves this differently, and the differences are structural rather than cosmetic. Databricks and Azure ship a product; AWS ships a pattern; GCP routes you through its API-management platform.
| Capability | Databricks Mosaic AI Gateway |
Azure API Management + AI Foundry |
AWS Bedrock + API Gateway pattern |
GCP Apigee + Vertex AI |
Cloudflare AI Gateway |
|---|---|---|---|---|---|
| What it actually is | A managed feature of Model Serving β enable it on an endpoint | Policies you write on an existing APIM instance, plus a model router in Foundry | A reference architecture you assemble from API Gateway, Lambda and Bedrock | LLM policy templates on Apigee, an established API-management product | A hosted edge proxy β change your base URL and it is on |
| External models | β OpenAI, Anthropic, Cohere and others as βexternal modelsβ | β any HTTP backend; Foundry hosts many model families | β Bedrock models natively; external providers need your own Lambda | β any backend, Apigee is provider-agnostic | β broad provider list |
| Token rate limiting | β per endpoint / user | β
llm-token-limit policy β the most mature implementation of this |
β request-based by default; token limits are yours to build | β via LLM policy templates | β |
| Budgets / spend caps | β usage tracking + alerts; hard caps via limits | β quotas by product/subscription, not dollars | β Budgets alert; enforcement is yours | β quota policies | β |
| Cost attribution | β usage tracking β system tables, per endpoint and user | β
llm-emit-token-metric with custom dimensions β Azure Monitor |
β application inference profiles β tag Bedrock invocations for per-team cost | β Apigee analytics + billing labels | β analytics per app |
| Payload logging | β inference tables β requests and responses land as Delta tables in Unity Catalog | β APIM diagnostics β App Insights / Event Hubs | β model invocation logging β S3 / CloudWatch | β Apigee message logging β Cloud Logging | β logs + log push |
| Caching | β not a gateway feature | β
llm-semantic-cache backed by Azure Redis with vector search |
β Bedrock prompt caching at the model; proxy caching is yours | β semantic caching policy | β exact + configurable TTL |
| Guardrails | β built-in safety, PII detection, topic restriction | β Content Safety / Prompt Shields callable as a policy | β Bedrock Guardrails β the strongest standalone guardrail product of the five | β Model Armor integrated as an Apigee policy | β guardrails + DLP, newer |
| Fallback / load balance | β fallbacks across served entities | β backend pools with circuit breaker; balances PTU then spills to pay-go | β cross-region inference profiles; multi-provider is yours | β target servers and failover | β universal endpoint with fallbacks |
| Identity & governance | β Unity Catalog β permissions, lineage and audit on the same objects as your data | β Entra ID, subscriptions, products, full audit | β IAM everywhere β the strongest identity story | β Cloud IAM + Apigee developer apps | β Cloudflare account model |
| You operate | Nothing | An APIM instance | Gateway + Lambda + the code in it | An Apigee org | Nothing |
| Choose it when | The workload and the data already live on Databricks β this is the obvious answer and needs no new system | Azure-standardised, and especially with a PTU + pay-as-you-go capacity mix to balance | AWS-only estate that wants IAM to be the single authorisation model, and is willing to build | Apigee is already in place, or Model Armor is the deciding feature | Public-facing apps already behind Cloudflare and you want this working this afternoon |
What an Azure APIM AI policy actually looks like
Worth seeing, because βAPIM has AI gateway featuresβ means writing this β the policy language is a real skill and a real dependency. (The LiteLLM equivalent is on the Tooling Landscape page.)
<!-- inbound: token quota per team, then check the semantic cache --> <inbound> <llm-token-limit counter-key="@(context.Subscription.Id)" tokens-per-minute="50000" estimate-prompt-tokens="true" remaining-tokens-header-name="x-tokens-remaining" /> <azure-openai-semantic-cache-lookup score-threshold="0.05" embeddings-backend-id="embeddings-pool" embeddings-backend-auth="system-assigned" /> <!-- backend pool = PTU deployment first, pay-go as overflow --> <set-backend-service backend-id="aoai-pool" /> </inbound> <outbound> <!-- this is your cost attribution: dimensions become Azure Monitor metrics --> <llm-emit-token-metric namespace="genai"> <dimension name="Team" value="@(context.Subscription.Name)" /> <dimension name="UseCase" value="@(context.Request.Headers.GetValueOrDefault("x-use-case","unknown"))" /> <dimension name="ModelId" /> </llm-emit-token-metric> <azure-openai-semantic-cache-store duration="3600" /> </outbound>
estimate-prompt-tokens. Token limits must be enforced before the call, but the
true token count is only known after it β so every gateway either estimates up front or
reconciles afterwards. Ask which, because it decides whether your limit is a guarantee or an
approximation.
6 Β· Routing strategies decoded trade-off
βRoutingβ covers five genuinely different behaviours. Teams routinely configure one and expect another.
| Strategy | How it picks | Use it for | Failure mode |
|---|---|---|---|
| Simple / round-robin | Next deployment in the list | Identical deployments across regions | Sends traffic to a degraded endpoint until health checks notice |
| Weighted | Fixed percentages | Canarying a new model version; draining a region | Weights become stale and nobody revisits them |
| Latency-based | Tracks recent p50/p95 per deployment and prefers the fastest | The sensible default for multi-region or multi-provider | Can stampede onto one endpoint and then oscillate; check the smoothing window |
| Usage / capacity-based | Routes by remaining TPM headroom | Mixing reserved capacity (PTU) with pay-as-you-go overflow | Needs accurate token accounting or it overflows too early and costs more |
| Semantic / quality routing | A classifier or small model reads the request and picks a tier β cheap model for easy prompts, frontier for hard ones | Real cost reduction at high volume | Non-deterministic behaviour introduced in the proxy. Same prompt, different model, different answer β and evals that never saw the cheap model |
7 Β· Semantic caching β what it is and when it is wrong trade-off
Exact caching returns a stored response when the prompt is byte-identical. Semantic caching embeds the incoming prompt, finds the nearest cached prompt by vector similarity, and returns its answer if the distance is under a threshold.
β Where it genuinely pays
- High-volume FAQ and support surfaces where thousands of users ask the same thing in different words
- Documentation and product Q&A with a stable corpus
- Bursty read-heavy traffic where latency matters more than freshness
- Cost profiles dominated by repeated short questions
β Where it quietly breaks things
- Anything personalised. βWhat is my balance?β and βwhat is my balance?β from two users are semantically identical β cache key must include identity
- Negations and small numeric differences sit close in embedding space: βunder 50mgβ vs βover 50mgβ
- Time-sensitive answers β yesterday's response returned confidently today
- Agent steps, where the same tool call in a different context needs a different result
- Anything you must reproduce for audit: the model never ran, so the trace shows a call that did not happen
8 Β· Virtual keys & cost attribution the killer feature
If the gateway only ever did this, it would still be worth running. A virtual key is a credential you issue that carries its own model allow-list, quota, budget and cost tags β while the real provider key never leaves the gateway.
| Without a gateway | With virtual keys |
|---|---|
One provider key shared over Slack, in four .env files | One key per team, issued and revoked centrally, never in an app repo |
| Rotating the key means coordinating four deploys | Rotate the provider key in one place; no application notices |
| βWho spent $40k last month?β β nobody knows | Spend broken down by team, project and use case, from tags applied at issue time |
| A runaway agent loop is discovered on the invoice | The budget cap returns an error at $400 and alerts at $320 |
| Any app can call any model, including external ones | Model allow-lists per key β βthis team may not call a non-EU endpointβ |
| Onboarding a team = a security conversation about secrets | Onboarding a team = one API call |
9 Β· Guardrails: at the gateway or in the app? trade-off
| Control | Best placed | Why |
|---|---|---|
| PII redaction on input | Gateway | Must apply to every app including ones you do not own; needs no context to do its job |
| Prompt-injection screening | Gateway | Same detector everywhere, updated once when a new technique appears |
| Toxicity / safety classification | Gateway | Uniform policy, and central evidence for an auditor |
| Model allow-lists, region pinning | Gateway | It is an authorisation decision, and authorisation belongs at the boundary |
| Output schema validation | Application | Only the app knows what shape it asked for |
| Citation / grounding checks | Application | Requires the retrieved context, which the gateway never sees |
| Authorisation on tool actions | Application | Needs the end user's identity and the business rule, not the calling service's |
| Loop and step budgets | Application | The gateway sees individual calls, not that they form one runaway agent trajectory |
10 Β· MCP gateways β the new category emerging
Once agents call tools over MCP, the tool servers need exactly the governance the model calls already have β and it is a different proxy, because the traffic is tool invocations rather than completions.
| Option | What it provides | Maturity |
|---|---|---|
| Docker MCP GatewayOSS | Runs MCP servers as containers behind one endpoint, with secret injection and per-server isolation | The easiest starting point locally |
| IBM ContextForge (MCP Gateway)OSS | Registry, federation of many servers behind one, virtual servers, auth and admin UI | The most complete OSS option today |
| AWS AgentCore Gatewaycloud-native | Turns existing APIs and Lambdas into MCP tools with IAM-backed access control | Managed, AWS-only |
| Azure API Management + API Centercloud-native | Exposes existing APIs as MCP servers and applies the same APIM policies and Entra auth | Managed, fits existing APIM estates |
| LiteLLM, Bifrost, Higressmixed | MCP proxying added to the model gateway you already run | One system instead of two β usually the pragmatic answer |
| MintMCP, Lasso, Obotcommercial | Enterprise MCP governance β audit, DLP, per-tool permissions | Young segment, moving weekly |
11 Β· Failure modes & high availability decide first
The gateway is on the critical path of every AI request in the organisation. Its failure modes are the most consequential design decision on this page, and the one most often deferred.
| Failure | What users see | The mitigation to decide now |
|---|---|---|
| Gateway is down | Every AI feature in the company is down at once | Run β₯2 instances behind a load balancer and decide explicitly: direct-to-provider fallback in the client SDK, or an accepted outage with a stated SLO. There is no third option |
| Its datastore is down (Redis/Postgres) | Depends entirely on config β either degraded, or total failure | Confirm it fails open: no cache and no spend tracking is survivable; refusing all traffic is not |
| Added latency | Every call is slower by the hop | Measure it. A Go/Rust proxy adds low single-digit milliseconds; an under-provisioned Python one can add hundreds under load |
| Streaming breaks | Responses arrive as one block after a long pause | Test SSE passthrough explicitly, with buffering disabled at every layer. This is the most common integration bug |
| Budget cap fires | A production feature starts returning errors mid-afternoon | Alert well before the cap, and make the app degrade gracefully β a hard 429 with no handling is a self-inflicted incident |
| A provider throttles | 429s, or silent slowdown | This is what the gateway is for β verify fallback and circuit-breaker config actually fire, by testing them, not by reading the YAML |
| The gateway is the trace gap | Traces stop at the app and resume at the model with nothing between | Propagate trace context through the proxy and have it emit its own spans, or you have created a blind spot in the middle of every request |
12 Β· An adoption path that works practice
- Do not start with one. A single application calling a provider SDK directly needs no gateway. Adding one on day one buys operational burden before it buys leverage.
- Introduce it at the second application, or the first time someone asks what AI cost by team. Start in passthrough mode: routing and logging only, no policies. Prove the hop is invisible.
- Issue virtual keys and delete the shared provider key. This alone justifies the project, and it is a one-afternoon change once the proxy is up.
- Turn on cost tags and budgets with generous ceilings and alerts. Let a month of real data arrive before anyone argues about limits.
- Add fallbacks and a circuit breaker, then test them by taking a provider away in a non-production environment. Untested failover is a belief, not a control.
- Add guardrails last, once you can see in the traces what actually needs guarding. Guardrails configured from imagination block legitimate traffic and miss the real attacks.
- Semantic caching and semantic routing are step seven at the earliest β both change what the model returns, so neither should land before you have evals that would notice.
13 Β· Anti-patterns avoid
| Anti-pattern | Why it hurts | Do instead |
|---|---|---|
| A shared internal SDK instead of a gateway | Governs only code that imports it, in one language, and every policy change is a coordinated redeploy across every team | A proxy β then the SDK is optional ergonomics, not the control |
| Single gateway instance | One process is now a company-wide single point of failure | Two or more, health-checked, with an explicit down-path |
| Gateway and observability in the same hop | Convenient until the shared blast radius takes both out together | Acceptable early; separate them before it is load-bearing |
| Logging full payloads with no redaction or retention policy | You have built a searchable archive of customer data with an unbounded bill attached | Redact at the gateway, sample successes, keep errors, set retention on day one |
| Semantic cache scoped across tenants | One customer's answer served to another. This is a data-leak class of bug, not a caching bug | Tenant and identity in the cache key, always |
| Adopting a third-party hosted router in a regulated setting | Prompts containing customer data now traverse a party nobody reviewed | Self-host, or complete the review before the pilot, not after |
| βWe'll add HA laterβ | Later is during the incident | Two instances is not a project; it is a config change on day one |
| Treating the gateway as a cost-optimisation product | The savings are real but second-order; the durable value is control, attribution and reversibility | Justify it on governance. The cost reduction is a bonus you will also get |