Going deeper on two of them: Observability & Eval β What Each Tool Offers has the feature-by-feature matrices and the cloud-provider equivalents, and AI Gateway β The Control Point covers that layer on its own page.
1 Β· Observability & tracing landscape
LLM observability is ordinary distributed tracing plus three things ordinary APM does not capture: the prompt and response, token usage and cost, and a quality score. That last one is why generic APM alone is rarely enough β it can tell you the call took 900 ms, not that the answer was wrong.
| Tool | What it is best at | Watch out for | Reach for it when |
|---|---|---|---|
| LLM-native, open source | |||
| LangfuseOSS + SaaS | Tracing, prompt management, evals, cost β the most complete open-source single answer | Self-hosting means running Postgres + ClickHouse | You want one OSS tool and are willing to run it |
| Arize PhoenixOSS + SaaS | Trace inspection and evaluation notebooks; strong on retrieval debugging | Leans experimentation more than long-run production | You are debugging RAG relevance |
| MLflow TracingOSS | Traces, prompt registry and evaluation alongside model lifecycle | Best experience inside Databricks | You already run MLflow β do not add a second tool |
| Opik (Comet)OSS + SaaS | Tracing plus an opinionated evaluation library | Younger ecosystem | Eval-first teams |
| TruLensOSS | Feedback functions β programmatic quality scoring | Not a full observability backend | You want quality scoring, not infrastructure |
| HeliconeOSS + SaaS | Proxy-based capture β one line to adopt | Proxy sits on the critical path | You want logging without touching app code |
| Laminar, Literal AIOSS + SaaS | Traces plus dataset curation from production | Smaller communities | You want production traces to become eval sets |
| LLM-native, commercial | |||
| LangSmithcommercial | Deep LangChain/LangGraph integration, datasets, annotation queues | Pulls you toward one framework's worldview | You are already on LangGraph |
| Braintrustcommercial | Eval-centric workflow, side-by-side experiment comparison | Eval-first framing suits some teams and not others | Prompt iteration is your bottleneck |
| W&B Weavecommercial | Traces plus the ML experiment lineage teams already know | Heavier than needed for pure GenAI apps | You already run Weights & Biases |
| Galileo, Humanloop, PromptLayercommercial | Guardrail metrics, prompt collaboration for non-engineers | Overlaps tools you may already own | Product/PM colleagues need to edit prompts |
| General APM that now speaks LLM | |||
| Datadog LLM Observabilitycommercial | One pane with the rest of your estate; mature alerting | Cost at volume; prompt payloads inflate ingest | Datadog is already the company standard |
| New Relic, Dynatrace, Elastic, Splunk, Coralogixcommercial | AI monitoring bolted onto existing APM | Quality evaluation is usually thinner than LLM-native tools | Consolidation matters more than depth |
| SigNoz, HyperDX, Uptrace, Grafana + Tempo/LokiOSS + SaaS | OTel-native backends you can self-host | You assemble the LLM semantics yourself | You want OTel end to end with no vendor |
| Cloud-native | |||
| Databrickscloud-native | MLflow Tracing, Agent Evaluation, Lakehouse Monitoring, system tables for cost | Only for workloads on Databricks | The workload already lives there |
| Azurecloud-native | Application Insights + AI Foundry tracing and evaluation | Split across several services | Azure-standardised estates |
| AWScloud-native | CloudWatch + X-Ray, Bedrock invocation logging and model evaluation | Assemble-it-yourself for LLM specifics | Bedrock-centric builds |
| GCPcloud-native | Cloud Trace/Logging plus Vertex AI evaluation and monitoring | As above | Vertex-centric builds |
2 Β· OpenTelemetry and its alternatives trade-off
Worth being precise, because the question is usually asked wrongly: OpenTelemetry is a standard, not a product. Its true alternatives are proprietary agents. The LLM-specific libraries people name as alternatives are mostly conventions built on top of it.
| Option | Relationship to OTel | Trade-off | Choose when |
|---|---|---|---|
| OpenTelemetry + GenAI semantic conventionsOSS | The standard itself | Vendor-neutral, huge ecosystem; GenAI conventions are still settling, so expect some churn | Default. Especially if the backend may change. |
| OpenLLMetry (Traceloop)OSS | On top of OTel | Auto-instruments LLM SDKs and emits OTel spans β not a replacement | You want LLM spans without writing them |
| OpenInference (Arize)OSS | On top of OTel | Alternative span conventions for LLM/RAG; pairs with Phoenix | You use Phoenix or want richer RAG spans |
| Vendor SDKs β Datadog, New Relic, Dynatrace OneAgent, Elastic APMcommercial | True alternatives | Deeper out-of-the-box features and support; re-instrumenting later is real work | The vendor is mandated and unlikely to change |
| SentryOSS + SaaS | Alternative, with OTel support | Error-first rather than trace-first | Exceptions matter more than latency |
| Prometheus + StatsDOSS | Metrics only | No traces β cannot answer "why did this request behave that way" | You need aggregates, not per-request forensics |
| Zipkin / Jaeger clients, OpenTracing / OpenCensusOSS | Predecessors, merged into OTel | Legacy β OpenTracing and OpenCensus are archived | Only existing systems; do not start here |
| SDK-native tracing β Langfuse SDK, LangSmith SDK, MLflowmixed | Alternative, most now OTel-compatible | Fastest to adopt; check there is an OTel export path before committing | Speed matters and the tool is already chosen |
3 Β· AI gateways landscape
An AI gateway is a reverse proxy that understands model traffic. It exists so that key management, routing, fallback, rate limiting, caching, cost attribution, guardrails and logging live in one place instead of being reimplemented in every application. If you have more than two applications calling models, you already need one.
| Gateway | Strengths | Trade-off | Choose when |
|---|---|---|---|
| Open source | |||
| LiteLLM ProxyOSS + SaaS | 100+ providers behind one OpenAI-shaped API; virtual keys, budgets, fallbacks, cost tracking | Python service to operate; a hop on the critical path | Default for most teams β quickest route to multi-provider |
| Portkey GatewayOSS + SaaS | Very fast edge gateway, routing, retries, caching, guardrails | Richer features sit in the paid tier | Latency-sensitive, want OSS with a hosted upgrade |
| Kong AI GatewayOSS + SaaS | AI plugins on a mature API gateway β auth, quotas, transformations | Kong concepts to learn if you are not already on it | Kong is already your API layer |
| Envoy AI GatewayOSS | Kubernetes-native, built on Envoy Gateway; strong traffic control | Operationally heavier; k8s assumed | Platform teams standardised on Envoy |
| Apache APISIX, Higress, Traefik AI pluginsOSS | AI routing on an existing ingress | AI features are newer than the core proxy | You want one proxy for all traffic |
| Gloo AI Gateway (Solo.io)OSS + SaaS | Envoy-based, service-mesh adjacent, policy-rich | Enterprise-shaped | You run a mesh already |
| Bifrost, GPT Router, llm-gatewayOSS | Lightweight, low-overhead routers | Smaller feature sets and communities | You want the thinnest possible hop |
| Managed & cloud-native | |||
| Databricks Mosaic AI Gatewaycloud-native | Rate limits, payload logging, guardrails, PII detection, usage tracking, all governed by Unity Catalog | Databricks-resident workloads | You are on Databricks β this is the obvious answer |
| Azure API Management (AI gateway features)cloud-native | Token-based rate limiting, semantic caching, load balancing across Azure OpenAI deployments | APIM policy language is its own skill | Azure estates with PTU + pay-go mixes |
| Apigee (GCP)cloud-native | Mature API management with LLM policy templates | Enterprise pricing | Apigee already in place |
| AWS β Bedrock + API Gateway / generative-AI gateway patternscloud-native | IAM-native, no new vendor | More reference architecture than product β you assemble it | AWS-only estates that want IAM everywhere |
| Cloudflare AI Gatewaycommercial | Edge caching, analytics, rate limiting; trivial to switch on | Traffic leaves your perimeter | Public-facing apps already on Cloudflare |
| OpenRouter, Unify, Martian, Not Diamondcommercial | Aggregation and quality/price-aware routing across providers | A third party in the data path β a real review item in regulated settings | Experimentation and model comparison |
What the gateway takes off every application
The reason a gateway is the highest-leverage shared component: it applies to every application, in any language, with zero application code. Nine of the thirty-seven requirements in the blueprint can be satisfied here and nowhere else as cheaply.
| Concern | What you stop writing in every app |
|---|---|
| Model access & routing | Provider SDKs, endpoint config, one API shape for every model |
| Fallback & retries | "Primary down β secondary" logic, backoff, load balancing across deployments |
| Rate limits & quota | Per-key, per-user, per-team limits β including token-based, not only request-based |
| Budgets & cost caps | Spend ceilings per team or project, alerts, hard stops |
| Cost attribution | Spend tagged to team, project and use case without touching app code |
| Payload logging | Request and response capture into your observability backend |
| Caching | Exact and semantic caching across all callers, not once per app |
| Guardrails | PII redaction and injection screening applied uniformly |
| Key management | Virtual keys per team β rotate centrally, never distribute provider keys |
Open-source gateways β capability matrix
| Gateway | Lang | Routing & fallback | Rate limit / budget | Caching | Guardrails | Logging | Notes |
|---|---|---|---|---|---|---|---|
| LiteLLM Proxy | Python | β 100+ providers | β per key / user / team, token-based | β Redis + semantic | β Presidio, Lakera, Bedrock Guardrails | β Langfuse, OTel, S3, Datadog | Richest OSS feature set. Admin UI, virtual keys, spend tracking |
| Portkey Gateway | TypeScript | β | β | β simple + semantic | β οΈ mostly paid platform | β hooks | Very low latency, edge-deployable |
| Bifrost | Go | β | β | β | β οΈ limited | β | Lowest overhead; MCP support |
| Envoy AI Gateway | C++/Go | β | β token-based | β οΈ via Envoy | β οΈ external | β OTel | Kubernetes/CRD-native; scales hardest |
| Kong AI Gateway | Lua | β
ai-proxy | β οΈ advanced limiting is enterprise | β οΈ semantic cache enterprise | β
ai-prompt-guard | β | Right answer if Kong is already your ingress |
| Apache APISIX | Lua | β | β | β οΈ | β
ai-prompt-guard | β | Same logic if APISIX is your ingress |
| Higress | Go | β | β token limit | β | β security guard | β | Strong AI plugin set |
What "gateway config" actually looks like
LiteLLM, because it is the most likely default. Note that the applications never learn any of
this exists β they keep calling claude-sonnet at the gateway URL.
# config.yaml model_list: - model_name: claude-sonnet # the alias apps ask for litellm_params: model: anthropic/claude-sonnet-5 api_key: os.environ/ANTHROPIC_API_KEY - model_name: claude-sonnet # same alias again = load balanced litellm_params: model: bedrock/anthropic.claude-sonnet-5 aws_region_name: eu-west-1 router_settings: routing_strategy: latency-based-routing fallbacks: [{ claude-sonnet: [claude-haiku] }] num_retries: 2 allowed_fails: 3 # circuit breaker cooldown_time: 30 litellm_settings: cache: true cache_params: { type: redis, ttl: 3600, supported_call_types: [acompletion] } callbacks: [langfuse, otel] # tracing + payload logging guardrails: - guardrail_name: pii litellm_params: { guardrail: presidio, mode: pre_call } - guardrail_name: injection litellm_params: { guardrail: lakera, mode: pre_call } general_settings: master_key: os.environ/LITELLM_MASTER_KEY max_budget: 5000 # org ceiling, USD per period budget_duration: 30d
Per-team virtual keys then carry their own limits and cost tags:
curl -X POST $GATEWAY/key/generate -H "Authorization: Bearer $MASTER_KEY" \ -d '{"team_id":"ops-eai","models":["claude-sonnet"], "max_budget":400,"budget_duration":"30d", "tpm_limit":200000,"rpm_limit":500, "metadata":{"cost_centre":"EAI-1042","use_case":"incident-search"}}'
Which one to pick
| Situation | Choose | Why |
|---|---|---|
| On Databricks | Mosaic AI Gateway | Governed by Unity Catalog; no extra system to run |
| On Azure | API Management | Token-based limiting and balancing across PTU and pay-go deployments |
| Anywhere else, starting out | LiteLLM Proxy | Best OSS coverage, fastest to value |
| Kubernetes, high scale | Envoy AI Gateway | CRD-native, built on proven traffic infrastructure |
| Latency-critical / edge | Portkey or Bifrost | Lowest added hop |
| Kong or APISIX already in place | Their AI plugins | One proxy, not two |
4 Β· Orchestration & agent frameworks landscape
The alternatives to LangGraph fall into three families, and the third is the one most teams overlook.
| Framework | Family | Strengths | Choose when |
|---|---|---|---|
| Graph & state-machine frameworks | |||
| LangGraphOSS | Graph | Explicit nodes and edges, checkpointing, human-in-the-loop, mature ecosystem | Complex branching you want to see as a graph |
| LlamaIndex WorkflowsOSS | Event-driven | Event/step model rather than a graph DSL; strong retrieval heritage | RAG-heavy systems |
| Burr (DAGWorks)OSS | State machine | Explicit state machine with good introspection | You think in states, not graphs |
| Pydantic AIOSS | Typed agents | Type-safe, dependency-injected, familiar to FastAPI teams | Python-first teams who value types |
| Agent-first frameworks | |||
| Claude Agent SDKOSS | Agent | The agent loop, tools, skills, subagents and permissions as first-class concepts | Building on Claude β the loop is handled for you |
| OpenAI Agents SDKOSS | Agent | Lightweight handoffs and guardrails | OpenAI-centric builds |
| Microsoft Agent Framework / Semantic Kernel / AutoGenOSS | Agent | Enterprise .NET and Python story; multi-agent conversation patterns | Microsoft-standardised organisations |
| Google ADKOSS | Agent | Pairs with Vertex Agent Engine for managed deployment | GCP-centric builds |
| Strands Agents (AWS)OSS | Agent | Model-driven loop, deploys neatly to AWS | Bedrock-centric builds |
| CrewAI, Agno, smolagents, Griptape, Atomic AgentsOSS | Agent | Role-based crews and fast prototyping | Getting to a demo quickly |
| DSPyOSS | Compiler | Optimises prompts against a metric instead of hand-tuning them | You have an eval set and want it to do the tuning |
| Durable execution β the overlooked family | |||
| TemporalOSS + SaaS | Durable | Battle-tested durability, retries, timers, versioning, resumption β in plain code, no DSL | Long-running or must-not-lose-progress work |
| Restate, Inngest, DBOSOSS + SaaS | Durable | Lighter durable execution with modern developer experience | You want durability without operating Temporal |
| Airflow, Prefect, Dagster, Databricks Workflowsmixed | Data | Scheduling, backfills, lineage, data-aware retries | The work is a pipeline that happens to call a model |
| Step Functions, Durable Functions, Cloud Workflowscloud-native | Durable | Managed, IAM-integrated, nothing to run | Cloud-standardised estates |
5 Β· Graph & memory frameworks landscape
Two layers people conflate: the framework that builds and queries a knowledge graph for an agent, and the database underneath it. Choose them separately.
Frameworks β graph-shaped memory and retrieval
| Framework | What it does | Trade-off | Choose when |
|---|---|---|---|
| Graphiti (Zep)OSS | Temporal knowledge graph β facts carry validity intervals, so it models what was true when | Ingest cost; needs a graph database | Agent memory where facts change over time |
| Microsoft GraphRAGOSS | Builds entity/community graphs over a corpus for global "what are the themes" questions | Expensive indexing pass; batch-oriented | Corpus-wide sense-making that vector RAG cannot do |
| LightRAG, nano-graphragOSS | Cheaper, simpler GraphRAG implementations | Fewer features than the original | You want GraphRAG without the indexing bill |
| CogneeOSS | Pipelines that build graph + vector memory together | Younger project | You want both substrates from one pipeline |
| Mem0OSS + SaaS | Memory layer with extraction and consolidation; hybrid vector + graph | Opinionated about what is worth remembering | Per-user long-term memory in an assistant |
| ZepOSS + SaaS | Managed memory service built on Graphiti | A service to depend on | You want memory without building it |
| Letta (MemGPT)OSS | Agents with self-managed tiered memory | A whole agent runtime, not a library | Memory management is the core problem |
| LlamaIndex Property Graph IndexOSS | Graph construction and querying inside a familiar RAG framework | Tied to LlamaIndex abstractions | Already on LlamaIndex |
| neo4j-graphragOSS | First-party GraphRAG patterns for Neo4j | Neo4j-specific | Neo4j is already chosen |
Databases β where the graph actually lives
| Database | Strengths | Trade-off | Choose when |
|---|---|---|---|
| Neo4jOSS + SaaS | Most mature; Cypher, tooling, vector index built in | Enterprise features are licensed | Default for a real knowledge graph |
| FalkorDBOSS | Very low latency, Redis-based, built for agent memory access patterns | Smaller ecosystem | Graph reads on the request path |
| KΓΉzuOSS | Embedded β no server, like SQLite for graphs | Not for large multi-writer workloads | Per-agent or per-tenant local graphs |
| Apache AGE (Postgres)OSS | Graph queries inside the Postgres you already run | Slower than native graph engines at depth | Modest graphs; one fewer system |
| Memgraph, ArangoDB, NebulaGraph, Dgraph, TigerGraphmixed | In-memory speed, multi-model, or very large scale | Each has a distinct query language to learn | Specific scale or latency demands |
| Neptune (AWS), Spanner Graph (GCP), Cosmos DB Gremlin (Azure)cloud-native | Managed, IAM-integrated, no operations | Less portable; ecosystem tooling thinner | Cloud-standardised estates |
6 Β· Code structure graphs landscape
A knowledge graph of a codebase β files, symbols, calls, imports, inheritance, data flow. Distinct from document RAG, and increasingly the substrate that makes coding agents useful: it answers "what calls this?" and "what breaks if I change it?", which similarity search cannot.
| Tool | What it produces | Coverage | Best for |
|---|---|---|---|
| Built for agents & LLM consumption | |||
| GraphifyApache 2.0 | On-device graph via tree-sitter; ships as an MCP server, slash command, pre-tool hook and memory-file directive | 36 languages, plus SQL, Terraform, docs | Dropping a code graph straight into an AI coding assistant with no glue code |
| BlarifyOSS | Parses a repo into a Neo4j/FalkorDB graph of files, classes, functions and their relationships | Multi-language via tree-sitter + LSP | Giving an agent multi-hop code context β the closest thing to "graphify my codebase" |
| PotpieOSS | Repo knowledge graph plus prebuilt agents over it | Multi-language | Q&A and onboarding on an unfamiliar codebase |
| tree-sitter + tree-sitter-graphOSS | Parse trees, and a DSL for turning them into graphs on your own schema | 40+ grammars | Rolling your own β most custom tools start here |
| LlamaIndex CodeHierarchyNodeParserOSS | Hierarchical code chunks preserving structure for retrieval | Multi-language | Code RAG without standing up a graph database |
| Repo maps β Aider, ContinueOSS | Ranked symbol map of a repo, sized to a context window | Multi-language | Cheap context for a coding agent, no graph store at all |
| Precise code intelligence | |||
| SCIP / LSIF (Sourcegraph)OSS | Definitions, references and implementations as an index | Per-language indexers | Accurate cross-repo navigation at scale |
| Kythe (Google)OSS | Language-agnostic graph schema for cross-references | Several, via extractors | Very large monorepos |
| Stack Graphs (GitHub)OSS | Incremental name-binding graphs β resolves symbols without a full build | Growing set | Fast, incremental indexing |
| Glean (Meta)OSS | A queryable database of facts about source code | Multi-language via indexers | Building your own code-intelligence backend |
| Semantic & data-flow analysis | |||
| JoernOSS | Code Property Graph β AST + control flow + data flow in one queryable graph | C/C++, Java, Python, JS, Go and more | The richest structural graph available; security and impact analysis |
| CodeQL (GitHub)free for OSS | Code as a relational database you query | Major languages | Vulnerability hunting and deep semantic queries |
| jQAssistantOSS | Scans Java/Maven into Neo4j β architecture rules as Cypher | Java/JVM | Java estates; enforcing architecture constraints in CI |
| Moose / PharoOSS | Meta-model of a system for custom analyses | Many, via importers | Research-grade or bespoke analysis |
| Dependency & architecture views | |||
| dependency-cruiser, madge, pydeps, jdeps, go-callvisOSS | Module and call dependency graphs | One language each | Quick answers with no infrastructure |
| CodeScene, NDepend, Structure101, Understand, CAST Imagingcommercial | Architecture maps, coupling metrics, hotspots; CAST targets whole application portfolios | Broad | Legacy modernisation programmes and estate-wide assessment |
7 Β· Token efficiency on large codebases practice
Working an AI coding assistant against a large repository is mostly a context problem, not a model problem. The levers below are ordered by impact, and the first four cost nothing.
| # | Lever | Why it works | Cost |
|---|---|---|---|
| 1 | Delegate exploration to subagents | A subagent burns its own context reading twenty files and returns a paragraph. Your main window receives the conclusion, not the dumps. "Find where retry logic lives across the services" should almost never run in the main session. | Free |
| 2 | Clear between unrelated tasks | Compaction is lossy and costs tokens to perform; clearing is clean. Check where the window is actually going before optimising anything else β the answer is usually surprising. | Free |
| 3 | Keep the project memory file lean | CLAUDE.md is prepended to every request. Point at files rather than inlining them. A 500-line memory file is a tax on every turn of every session. |
Free |
| 4 | Prune MCP servers | Every connected server's tool schemas sit in context on every request. Three chatty servers can cost more than the code you are reading. The most common silent bloat. | Free |
| 5 | Reference files explicitly | @src/auth/session.py skips an entire search-and-read cycle versus "find the session handling code". |
Free |
| 6 | Skills instead of a large memory file | Skills load on demand; the memory file is always resident. | Free |
| 7 | Plan before executing | Design first, then act β avoids exploratory thrash filling the window with dead ends. | Free |
| 8 | Structural search over text search | ast-grep matches code shapes; far fewer false positives than grep, so fewer files opened. |
Install |
| 9 | Symbol-level retrieval | A language server returns one function, not one file. | Setup |
| 10 | A code knowledge graph | Structural questions answered as nodes and edges instead of file text. Worth it once the repo is large, unfamiliar and the questions are genuinely structural. | Setup + upkeep |
Free and open-source options
| Tool | Licence | What it gives you | Best for |
|---|---|---|---|
| Built in β nothing to install | |||
| Subagents | included | Isolated context for exploration; only the conclusion returns | Start here. Biggest single win |
| Skills | included | Procedures loaded on demand rather than always resident | Conventions, checklists, workflows |
| Hooks | included | Intercept tool calls β e.g. consult an index before a broad search | Enforcing retrieval discipline |
| Search & retrieval | |||
| ripgrepMIT | MIT | Fast literal and regex search, already used under the hood | Everything. Narrow before you read |
| ast-grepMIT | MIT | Structural search on the syntax tree β match code shapes, not strings | Refactors and pattern audits with far fewer false hits |
| SerenaMIT | MIT | MCP server over language servers β symbol-level reads and edits | Large repos where whole-file reads are the cost |
| Language-server MCPsOSS | Varies | Go-to-definition, find-references, hover as callable tools | Typed languages with good LSP support |
| Zoekt / SourcebotOSS | Apache 2.0 | Indexed cross-repo code search | Many repositories rather than one large one |
| Packing & context shaping | |||
| RepomixMIT | MIT | Packs a repo into one structured, compressible file with token counts | Handing a whole small repo over once |
| Aider repo mapApache 2.0 | Apache 2.0 | Ranked tree-sitter symbol map sized to a context window | Cheapest useful whole-repo overview β no database |
| Context7MIT | MIT | Current library documentation on demand | Stopping guessed API signatures and the retries they cause |
| Code knowledge graphs | |||
| GraphifyApache 2.0 | Apache 2.0 | On-device graph over 36 languages via tree-sitter, no embeddings. Ships as an MCP server, a slash command, a pre-tool hook and a memory-file directive | Structural questions on a large, unfamiliar codebase |
| BlarifyMIT | MIT | Repo into Neo4j or FalkorDB; you own the schema | Building your own graph-backed tooling |
| JoernApache 2.0 | Apache 2.0 | Code Property Graph β AST plus control and data flow | Impact analysis and security, not day-to-day coding |
| PotpieApache 2.0 | Apache 2.0 | Repo knowledge graph with agents over it | Onboarding to an unfamiliar codebase |