Gen AI Fundamentals β€Ί AI Gateway

πŸšͺ AI Gateway β€” The Control Point

What it is, what every gateway offers, and how open source compares to each cloud
Why this got its own page. The gateway is the only component that applies to every AI application in the organisation, in any language, with zero application code β€” which makes it the highest-leverage thing on the diagram and the one most often bolted on a year late. Companion pages: Observability & Eval tools for the other two boxes, and AI Tooling Landscape for the short choose-one-tool version and a LiteLLM config example.
OSS open source, self-host OSS + SaaS open core with a hosted option cloud-native tied to one platform βœ… yes   ◐ partial / paid tier   β€” no

1 Β· What an AI gateway is β€” and is not definition

An AI gateway is a reverse proxy that understands model traffic. Ordinary proxies see bytes and URLs; an AI gateway sees models, prompts, tokens and cost β€” which is what lets it do rate limiting by tokens, caching by meaning, and budgets in dollars.

It is often confused with The actual difference Do you need both?
An API gateway (Kong, APIM, Apigee) An API gateway counts requests; an AI gateway counts tokens, understands streaming responses, and knows a model alias from a deployment Usually the same product with AI plugins enabled β€” one proxy, not two
An SDK wrapper (a shared internal library) A library only governs code that imports it, in one language, and needs a redeploy to change policy The library is fine for one team; it fails the moment a second language or a vendor tool appears
A service mesh (Istio, Linkerd) The mesh secures service-to-service traffic; it has no idea what a token is Complementary β€” the gateway usually runs as a service inside the mesh
A model router (OpenRouter, Not Diamond) Routers pick the best/cheapest model per request; that is one feature of a gateway, sold separately and usually hosted by a third party A router can sit behind your gateway, but it should not replace it
An inference server (vLLM, TGI, Ollama) Inference servers run the model; gateways route to them Both, if you self-host models β€” the gateway makes vLLM look like every other provider
The one-line test for whether you need one: if more than one application, or more than one team, calls a model β€” and someone will eventually ask β€œwhat did we spend on AI last month, by team?” β€” you need a gateway. Both halves are usually true by month three.

2 Β· Where it sits in the request path diagram

flowchart LR A1["App A Β· Python"] --> GW A2["App B Β· TypeScript"] --> GW A3["Vendor tool
(code you don't own)"] --> GW subgraph GW["AI GATEWAY β€” one OpenAI-shaped API"] direction TB K["Virtual key
authn Β· team Β· budget"] --> RL["Rate limit
RPM Β· TPM"] RL --> GI["Guardrails in
PII Β· injection"] GI --> C{"Cache hit?"} C -->|"yes"| RET["return cached"] C -->|"no"| R["Route Β· load balance"] R --> FB["Retry Β· fallback"] FB --> GO["Guardrails out"] end GW --> P1["Anthropic API"] GW --> P2["Bedrock Β· Vertex Β· Foundry"] GW --> P3["vLLM / self-hosted"] GW -.->|"spend Β· payloads Β· spans"| OBS[("Observability
+ billing")] classDef a fill:#eef6ff,stroke:#3b7bd0,color:#1f5fa8; classDef b fill:#f3f0ff,stroke:#764ba2,color:#4a3b7a; class A1,A2,A3,P1,P2,P3 a; class K,RL,GI,C,R,FB,GO,RET b;

Every arrow into the box is an application that needed no code to gain keys, quotas, budgets, caching, guardrails and logging. The third one is the argument: you cannot add a library to software you do not own.

3 Β· The 14 capabilities checklist

Use this as the RFP checklist. The core items are why the gateway exists; the rest are why you eventually pick one over another.

1 Β· Unified API coreOne request shape β€” almost always OpenAI-compatible β€” for every provider. Swapping Claude for Gemini becomes a config line, not a refactor.
2 Β· Key management coreProvider keys live only in the gateway. Applications get virtual keys you can scope, rotate and revoke without touching any app.
3 Β· Routing & load balancing coreOne alias, many deployments β€” across regions, across providers, across PTU and pay-as-you-go capacity.
4 Β· Retries & fallback coreAutomatic retry on 429/5xx, failover to a second provider, and a circuit breaker so a dead deployment stops receiving traffic.
5 Β· Rate limiting corePer key, user and team β€” and crucially by tokens per minute, not just requests, because request counts do not predict provider throttling.
6 Β· Budgets & spend caps coreHard dollar ceilings per team or project, with alerts before the stop. The control that turns a runaway loop into a 429 instead of an invoice.
7 Β· Cost attribution coreEvery request tagged to team, project, cost centre and use case, so the monthly bill can be broken down without anyone instrumenting anything.
8 Β· Payload loggingRequest and response capture, forwarded to your observability backend. Verify it can redact before it stores.
9 Β· CachingExact-match on identical prompts, and optionally semantic β€” shared across all callers rather than reimplemented per app.
10 Β· GuardrailsPII redaction and injection screening applied uniformly, pre-call and post-call, including to applications you do not control.
11 Β· Observability exportOTel spans, or native hooks into Langfuse/Datadog. Without this the gateway becomes a blind spot in the middle of every trace.
12 Β· Governance & RBACSSO, audit logs, model allow-lists per team β€” β€œthis team may not call an external provider” enforced centrally.
13 Β· Non-chat endpoints checkEmbeddings, reranking, batch, image, audio, and file uploads. Coverage varies far more than the chat endpoint, and it is where migrations break.
14 Β· MCP / tool proxying emergingThe same governance applied to tool servers, not just model calls. New, moving fast, and covered in Β§10.

4 Β· Open-source gateways β€” what each offers matrix

Gateway Runtime Providers Virtual keys Budgets Token rate limit Semantic cache Guardrails OTel MCP Admin UI The honest summary
LiteLLM ProxyOSS + SaaS Python100+βœ…βœ…βœ… βœ… Redisβœ… Presidio, Lakera, Bedrockβœ…βœ…βœ… The richest OSS feature set, by a distance. Also the heaviest hop β€” it is a Python service, so size it and run more than one.
Portkey GatewayOSS + SaaS TypeScript200+◐ platformβ—βœ… βœ…β— mostly paidβœ…βœ…β— paid Very low added latency, edge-deployable, tiny footprint. The OSS core routes; the governance lives in the paid platform.
Bifrost (Maxim)OSS Goβœ…βœ…βœ…βœ… βœ…β—βœ…βœ… nativeβœ… Lowest overhead of the full-featured options, with first-class MCP support. Younger and smaller community.
Envoy AI GatewayOSS Envoy / Goβœ…β— via k8sβ—βœ… token-aware ◐ external◐ externalβœ…β—β€” CRDs Kubernetes-native, configured as CRDs, built on the proxy that already runs the internet. Scales hardest, operates heaviest.
Kong AI GatewayOSS + SaaS Lua / Nginxβœ… ai-proxyβœ… consumers◐◐ advanced is enterprise ◐ enterpriseβœ… ai-prompt-guardβœ…β—β— enterprise The right answer if Kong is already your ingress β€” AI becomes another plugin on infrastructure you already operate.
Apache APISIXApache 2.0 Lua / Nginxβœ…βœ…β—βœ… β—βœ…βœ…β—βœ… Same logic as Kong, fully open β€” less gated behind an enterprise tier.
HigressApache 2.0 Go / Envoyβœ…βœ…β—βœ… βœ…βœ…βœ…βœ…βœ… Unusually complete AI plugin set for an ingress; strong in Kubernetes estates, documentation is improving from a China-first base.
Gloo AI Gateway (Solo.io)OSS + SaaS Envoyβœ…βœ…β—βœ… β—βœ…βœ…βœ…β— Policy-rich and mesh-adjacent. Enterprise-shaped in both the good and the expensive sense.
HeliconeOSS + SaaS TypeScriptβœ…βœ…βœ…βœ… βœ…β—β—β€”βœ… Observability-first proxy that grew gateway features. One hop for both jobs β€” simple, and a shared blast radius.
Arch (Katanemo), TrueFoundry, llm-gatewaymixed Rust / Goβœ…β—β—β— β—βœ… intent modelsβœ…β—β— Newer entrants. Arch is interesting for running small models in the proxy for routing and guardrail decisions.
Tiers move. Semantic caching, advanced rate limiting and guardrails have all migrated between free and paid tiers in several of these projects. Verify the current tier of any feature before it becomes load-bearing β€” and check it again at renewal.

5 Β· Cloud-provider gateways β€” the real comparison key table

Each cloud solves this differently, and the differences are structural rather than cosmetic. Databricks and Azure ship a product; AWS ships a pattern; GCP routes you through its API-management platform.

Capability Databricks
Mosaic AI Gateway
Azure
API Management + AI Foundry
AWS
Bedrock + API Gateway pattern
GCP
Apigee + Vertex AI
Cloudflare
AI Gateway
What it actually is A managed feature of Model Serving β€” enable it on an endpoint Policies you write on an existing APIM instance, plus a model router in Foundry A reference architecture you assemble from API Gateway, Lambda and Bedrock LLM policy templates on Apigee, an established API-management product A hosted edge proxy β€” change your base URL and it is on
External models βœ… OpenAI, Anthropic, Cohere and others as β€œexternal models” βœ… any HTTP backend; Foundry hosts many model families ◐ Bedrock models natively; external providers need your own Lambda βœ… any backend, Apigee is provider-agnostic βœ… broad provider list
Token rate limiting βœ… per endpoint / user βœ… llm-token-limit policy β€” the most mature implementation of this ◐ request-based by default; token limits are yours to build βœ… via LLM policy templates βœ…
Budgets / spend caps ◐ usage tracking + alerts; hard caps via limits ◐ quotas by product/subscription, not dollars ◐ Budgets alert; enforcement is yours ◐ quota policies ◐
Cost attribution βœ… usage tracking β†’ system tables, per endpoint and user βœ… llm-emit-token-metric with custom dimensions β†’ Azure Monitor βœ… application inference profiles β€” tag Bedrock invocations for per-team cost βœ… Apigee analytics + billing labels βœ… analytics per app
Payload logging βœ… inference tables β€” requests and responses land as Delta tables in Unity Catalog βœ… APIM diagnostics β†’ App Insights / Event Hubs βœ… model invocation logging β†’ S3 / CloudWatch βœ… Apigee message logging β†’ Cloud Logging βœ… logs + log push
Caching β€” not a gateway feature βœ… llm-semantic-cache backed by Azure Redis with vector search ◐ Bedrock prompt caching at the model; proxy caching is yours βœ… semantic caching policy βœ… exact + configurable TTL
Guardrails βœ… built-in safety, PII detection, topic restriction βœ… Content Safety / Prompt Shields callable as a policy βœ… Bedrock Guardrails β€” the strongest standalone guardrail product of the five βœ… Model Armor integrated as an Apigee policy ◐ guardrails + DLP, newer
Fallback / load balance βœ… fallbacks across served entities βœ… backend pools with circuit breaker; balances PTU then spills to pay-go ◐ cross-region inference profiles; multi-provider is yours βœ… target servers and failover βœ… universal endpoint with fallbacks
Identity & governance βœ… Unity Catalog β€” permissions, lineage and audit on the same objects as your data βœ… Entra ID, subscriptions, products, full audit βœ… IAM everywhere β€” the strongest identity story βœ… Cloud IAM + Apigee developer apps ◐ Cloudflare account model
You operate NothingAn APIM instanceGateway + Lambda + the code in itAn Apigee orgNothing
Choose it when The workload and the data already live on Databricks β€” this is the obvious answer and needs no new system Azure-standardised, and especially with a PTU + pay-as-you-go capacity mix to balance AWS-only estate that wants IAM to be the single authorisation model, and is willing to build Apigee is already in place, or Model Armor is the deciding feature Public-facing apps already behind Cloudflare and you want this working this afternoon
The structural gap on AWS. There is no first-party product called β€œAWS AI Gateway” β€” the published generative-AI gateway guidance is an architecture you deploy and then own. That is fine, and it is IAM-native, but budget for it as a system your team maintains, not a checkbox. Many AWS-centric teams run LiteLLM on ECS in front of Bedrock for exactly this reason and treat Bedrock Guardrails as the enforcement layer behind it.

What an Azure APIM AI policy actually looks like

Worth seeing, because β€œAPIM has AI gateway features” means writing this β€” the policy language is a real skill and a real dependency. (The LiteLLM equivalent is on the Tooling Landscape page.)

<!-- inbound: token quota per team, then check the semantic cache -->
<inbound>
  <llm-token-limit counter-key="@(context.Subscription.Id)"
                   tokens-per-minute="50000"
                   estimate-prompt-tokens="true"
                   remaining-tokens-header-name="x-tokens-remaining" />

  <azure-openai-semantic-cache-lookup score-threshold="0.05"
                   embeddings-backend-id="embeddings-pool"
                   embeddings-backend-auth="system-assigned" />

  <!-- backend pool = PTU deployment first, pay-go as overflow -->
  <set-backend-service backend-id="aoai-pool" />
</inbound>

<outbound>
  <!-- this is your cost attribution: dimensions become Azure Monitor metrics -->
  <llm-emit-token-metric namespace="genai">
    <dimension name="Team"     value="@(context.Subscription.Name)" />
    <dimension name="UseCase"  value="@(context.Request.Headers.GetValueOrDefault(&quot;x-use-case&quot;,&quot;unknown&quot;))" />
    <dimension name="ModelId" />
  </llm-emit-token-metric>
  <azure-openai-semantic-cache-store duration="3600" />
</outbound>
Note estimate-prompt-tokens. Token limits must be enforced before the call, but the true token count is only known after it β€” so every gateway either estimates up front or reconciles afterwards. Ask which, because it decides whether your limit is a guarantee or an approximation.

6 Β· Routing strategies decoded trade-off

β€œRouting” covers five genuinely different behaviours. Teams routinely configure one and expect another.

Strategy How it picks Use it for Failure mode
Simple / round-robinNext deployment in the list Identical deployments across regionsSends traffic to a degraded endpoint until health checks notice
WeightedFixed percentages Canarying a new model version; draining a regionWeights become stale and nobody revisits them
Latency-basedTracks recent p50/p95 per deployment and prefers the fastest The sensible default for multi-region or multi-providerCan stampede onto one endpoint and then oscillate; check the smoothing window
Usage / capacity-basedRoutes by remaining TPM headroom Mixing reserved capacity (PTU) with pay-as-you-go overflowNeeds accurate token accounting or it overflows too early and costs more
Semantic / quality routingA classifier or small model reads the request and picks a tier β€” cheap model for easy prompts, frontier for hard ones Real cost reduction at high volumeNon-deterministic behaviour introduced in the proxy. Same prompt, different model, different answer β€” and evals that never saw the cheap model
On semantic routing. It is the most-hyped gateway feature and the one to adopt last. It changes which model answers, which means your evaluation results no longer describe production. If you do use it, pin the routing decision in the trace and evaluate each tier separately β€” otherwise a quality regression looks like random noise.

7 Β· Semantic caching β€” what it is and when it is wrong trade-off

Exact caching returns a stored response when the prompt is byte-identical. Semantic caching embeds the incoming prompt, finds the nearest cached prompt by vector similarity, and returns its answer if the distance is under a threshold.

βœ… Where it genuinely pays

  • High-volume FAQ and support surfaces where thousands of users ask the same thing in different words
  • Documentation and product Q&A with a stable corpus
  • Bursty read-heavy traffic where latency matters more than freshness
  • Cost profiles dominated by repeated short questions

β›” Where it quietly breaks things

  • Anything personalised. β€œWhat is my balance?” and β€œwhat is my balance?” from two users are semantically identical β€” cache key must include identity
  • Negations and small numeric differences sit close in embedding space: β€œunder 50mg” vs β€œover 50mg”
  • Time-sensitive answers β€” yesterday's response returned confidently today
  • Agent steps, where the same tool call in a different context needs a different result
  • Anything you must reproduce for audit: the model never ran, so the trace shows a call that did not happen
The threshold is a safety control, not a tuning knob. Loosen it to raise hit rate and you are increasing how often one user's answer is served to another user's question. Start strict, measure the hit rate, and never scope a semantic cache more broadly than a single tenant. Prefer the model provider's own prompt caching (which caches the processed prefix and still runs the model) when what you actually want is cheaper long system prompts β€” it has none of these failure modes.

8 Β· Virtual keys & cost attribution the killer feature

If the gateway only ever did this, it would still be worth running. A virtual key is a credential you issue that carries its own model allow-list, quota, budget and cost tags β€” while the real provider key never leaves the gateway.

Without a gateway With virtual keys
One provider key shared over Slack, in four .env filesOne key per team, issued and revoked centrally, never in an app repo
Rotating the key means coordinating four deploysRotate the provider key in one place; no application notices
β€œWho spent $40k last month?” β€” nobody knowsSpend broken down by team, project and use case, from tags applied at issue time
A runaway agent loop is discovered on the invoiceThe budget cap returns an error at $400 and alerts at $320
Any app can call any model, including external onesModel allow-lists per key β€” β€œthis team may not call a non-EU endpoint”
Onboarding a team = a security conversation about secretsOnboarding a team = one API call
Design the tag schema before you issue the first key. Team, cost centre, environment, use case, and a data-classification tag if you are regulated. Tags are almost impossible to backfill β€” the spend they would have explained has already happened. Ten minutes now, or a reconciliation project later.

9 Β· Guardrails: at the gateway or in the app? trade-off

Control Best placed Why
PII redaction on inputGateway Must apply to every app including ones you do not own; needs no context to do its job
Prompt-injection screeningGateway Same detector everywhere, updated once when a new technique appears
Toxicity / safety classificationGateway Uniform policy, and central evidence for an auditor
Model allow-lists, region pinningGateway It is an authorisation decision, and authorisation belongs at the boundary
Output schema validationApplication Only the app knows what shape it asked for
Citation / grounding checksApplication Requires the retrieved context, which the gateway never sees
Authorisation on tool actionsApplication Needs the end user's identity and the business rule, not the calling service's
Loop and step budgetsApplication The gateway sees individual calls, not that they form one runaway agent trajectory
The dividing line is simple: if the control needs application context, it cannot live in the gateway. Everything else should, because that is the only way it reaches software you do not control. Implementing Guardrails has working code for the application half.

10 Β· MCP gateways β€” the new category emerging

Once agents call tools over MCP, the tool servers need exactly the governance the model calls already have β€” and it is a different proxy, because the traffic is tool invocations rather than completions.

Option What it provides Maturity
Docker MCP GatewayOSS Runs MCP servers as containers behind one endpoint, with secret injection and per-server isolation The easiest starting point locally
IBM ContextForge (MCP Gateway)OSS Registry, federation of many servers behind one, virtual servers, auth and admin UI The most complete OSS option today
AWS AgentCore Gatewaycloud-native Turns existing APIs and Lambdas into MCP tools with IAM-backed access control Managed, AWS-only
Azure API Management + API Centercloud-native Exposes existing APIs as MCP servers and applies the same APIM policies and Entra auth Managed, fits existing APIM estates
LiteLLM, Bifrost, Higressmixed MCP proxying added to the model gateway you already run One system instead of two β€” usually the pragmatic answer
MintMCP, Lasso, Obot Enterprise MCP governance β€” audit, DLP, per-tool permissions Young segment, moving weekly
Why this matters more than it looks. An MCP tool call is an action, not a completion β€” it writes to systems. The blast radius of an ungoverned tool server is larger than that of an ungoverned model call, and the questions are the same ones you already answered for models: who may call it, with whose identity, logged where, and what happens when a prompt injection tells the agent to call it.

11 Β· Failure modes & high availability decide first

The gateway is on the critical path of every AI request in the organisation. Its failure modes are the most consequential design decision on this page, and the one most often deferred.

Failure What users see The mitigation to decide now
Gateway is down Every AI feature in the company is down at once Run β‰₯2 instances behind a load balancer and decide explicitly: direct-to-provider fallback in the client SDK, or an accepted outage with a stated SLO. There is no third option
Its datastore is down (Redis/Postgres) Depends entirely on config β€” either degraded, or total failure Confirm it fails open: no cache and no spend tracking is survivable; refusing all traffic is not
Added latency Every call is slower by the hop Measure it. A Go/Rust proxy adds low single-digit milliseconds; an under-provisioned Python one can add hundreds under load
Streaming breaks Responses arrive as one block after a long pause Test SSE passthrough explicitly, with buffering disabled at every layer. This is the most common integration bug
Budget cap fires A production feature starts returning errors mid-afternoon Alert well before the cap, and make the app degrade gracefully β€” a hard 429 with no handling is a self-inflicted incident
A provider throttles 429s, or silent slowdown This is what the gateway is for β€” verify fallback and circuit-breaker config actually fire, by testing them, not by reading the YAML
The gateway is the trace gap Traces stop at the app and resume at the model with nothing between Propagate trace context through the proxy and have it emit its own spans, or you have created a blind spot in the middle of every request
Say the answer out loud before you adopt one: β€œwhen the gateway is down, X happens.” If nobody in the room can finish that sentence, the gateway is currently a single point of failure for every AI application simultaneously β€” and that fact will be discovered during the first incident, by the person least equipped to fix it.

12 Β· An adoption path that works practice

  1. Do not start with one. A single application calling a provider SDK directly needs no gateway. Adding one on day one buys operational burden before it buys leverage.
  2. Introduce it at the second application, or the first time someone asks what AI cost by team. Start in passthrough mode: routing and logging only, no policies. Prove the hop is invisible.
  3. Issue virtual keys and delete the shared provider key. This alone justifies the project, and it is a one-afternoon change once the proxy is up.
  4. Turn on cost tags and budgets with generous ceilings and alerts. Let a month of real data arrive before anyone argues about limits.
  5. Add fallbacks and a circuit breaker, then test them by taking a provider away in a non-production environment. Untested failover is a belief, not a control.
  6. Add guardrails last, once you can see in the traces what actually needs guarding. Guardrails configured from imagination block legitimate traffic and miss the real attacks.
  7. Semantic caching and semantic routing are step seven at the earliest β€” both change what the model returns, so neither should land before you have evals that would notice.

13 Β· Anti-patterns avoid

Anti-patternWhy it hurtsDo instead
A shared internal SDK instead of a gateway Governs only code that imports it, in one language, and every policy change is a coordinated redeploy across every team A proxy β€” then the SDK is optional ergonomics, not the control
Single gateway instance One process is now a company-wide single point of failure Two or more, health-checked, with an explicit down-path
Gateway and observability in the same hop Convenient until the shared blast radius takes both out together Acceptable early; separate them before it is load-bearing
Logging full payloads with no redaction or retention policy You have built a searchable archive of customer data with an unbounded bill attached Redact at the gateway, sample successes, keep errors, set retention on day one
Semantic cache scoped across tenants One customer's answer served to another. This is a data-leak class of bug, not a caching bug Tenant and identity in the cache key, always
Adopting a third-party hosted router in a regulated setting Prompts containing customer data now traverse a party nobody reviewed Self-host, or complete the review before the pilot, not after
β€œWe'll add HA later” Later is during the incident Two instances is not a project; it is a config change on day one
Treating the gateway as a cost-optimisation product The savings are real but second-order; the durable value is control, attribution and reversibility Justify it on governance. The cost reduction is a bonus you will also get
If you take one thing: the gateway is where governance stops being a document and becomes a runtime control. Keys, quotas, budgets, attribution, allow-lists and uniform guardrails are all enforced there β€” for every application, in every language, including the ones you will not write and cannot modify. That is why it is worth running a proxy you would otherwise rather not operate.
Related: Observability & Eval tools β€” where the gateway's logs and spend data should land Β· AI Tooling Landscape β€” the short choose-one version, with a LiteLLM config example Β· Stack Decisions β€” which agent frameworks work through a gateway to which model endpoints Β· AI Engineering Blueprint β€” the nine of ~37 requirements this layer satisfies on its own.