1 Β· Three concerns, one blurred market framing
Observability, evaluation and the gateway are three different questions, and almost every vendor in this space now claims all three. That is why the market looks confusing: the categories are clean, the products are not.
| Concern | The question it answers | When it runs | What it produces |
|---|---|---|---|
| Observability | βWhy did this particular run behave that way?β | Always, in production | Traces, spans, token counts, cost, latency, errors |
| Evaluation | βIs version B better than version A β and on what evidence?β | In CI before release, and sampled in production | Scores against a dataset, pass/fail gates, regressions |
| Gateway | βWho is allowed to call what, at what cost, through which route?β | On every request, in the path | Routing, keys, quotas, budgets, caching, logs |
inputs + expectations"] --> EX["Experiment run"] EX --> SC["Scores / graders"] SC --> GATE{"Gate:
regression?"} end subgraph RUN["Run time"] APP["Application"] --> GW["AI Gateway
keys Β· routing Β· budget"] GW --> LLM["Model"] APP -.spans.-> OBS[("Observability
backend")] GW -.payload logs.-> OBS end OBS -->|"curate failing traces"| DS OBS -->|"sampled online eval"| SC GATE -->|"pass"| APP classDef a fill:#eef6ff,stroke:#3b7bd0,color:#1f5fa8; classDef b fill:#f3f0ff,stroke:#764ba2,color:#4a3b7a; class APP,GW,LLM a; class DS,EX,SC,OBS b;
2 Β· The capability vocabulary definitions
The matrices below use these columns. Vendors use the same words for slightly different things, so pin the meaning down before comparing.
3 Β· Observability β open source matrix
| Tool | Tracing | Prompt mgmt | Datasets | LLM judge | Human annot. | Online eval | Cost | Alerts | OTel in | Self-host | What it is really for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| LangfuseOSS + SaaS | β | β | β | β | β | β | β | β | β | β full | The most complete single open-source answer β tracing, prompts, datasets and evals in one product. Self-host needs Postgres + ClickHouse. |
| Arize PhoenixOSS + SaaS | β | β | β | β | β | β | β | β | β native | β | Debugging retrieval. Best-in-class span inspection for RAG, embedding drift views, notebook-first workflow. |
| MLflow TracingOSS | β | β | β | β | β | β | β | β | β | β | Keeping GenAI in the same lifecycle tool as your models. Strongest if you already run MLflow β and the default on Databricks. |
| Opik (Comet)OSS + SaaS | β | β | β | β | β | β | β | β | β | β | Eval-first teams who also want tracing. Ships a strong opinionated metric library out of the box. |
| HeliconeOSS + SaaS | β | β | β | β | β | β | β | β | β | β | Zero-code adoption. It is a proxy β change the base URL and you have logs, cost and caching. Also part gateway. |
| LaminarOSS + SaaS | β | β | β | β | β | β | β | β | β | β | Turning production traces into labelled datasets with the least friction. Rust backend, fast ingest. |
| LangtraceOSS | β | β | β | β | β | β | β | β | β native | β | OTel-purist tracing β emits standard GenAI spans, so you can point it at any OTel backend later. |
| OpenLITOSS | β | β | β | β | β | β | β | β | β native | β | OTel-native GenAI instrumentation including GPU metrics; pairs with Grafana or SigNoz rather than replacing them. |
| LunaryOSS + SaaS | β | β | β | β | β | β | β | β | β | β | Chat-product analytics β per-user and per-conversation views, feedback capture, lightweight to run. |
| TruLensOSS | β | β | β | β | β | β | β | β | β | β | Feedback functions β programmatic quality scoring you embed in the app. A scoring library, not a backend. |
| OTel + SigNoz / Grafana Tempo / HyperDXOSS + SaaS | β | β | β | β | β | β | β | β | β native | β | Full control, no vendor, one backend for AI and everything else. You supply all the LLM semantics yourself. |
4 Β· Observability β commercial matrix
| Tool | Tracing | Prompt mgmt | Datasets | LLM judge | Human annot. | Online eval | Cost | Alerts | OTel in | Self-host | What it is really for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| LLM-native platforms | |||||||||||
| LangSmithcommercial | β | β | β | β | β | β | β | β | β | β enterprise | LangChain/LangGraph teams. Annotation queues and dataset workflow are genuinely mature. |
| Braintrustcommercial | β | β | β | β | β | β | β | β | β | β hybrid | Prompt iteration as the core loop β side-by-side experiment diffs are the product. Hybrid deploy keeps data in your VPC. |
| Arize AXcommercial | β | β | β | β | β | β | β | β | β | β | The production/enterprise tier above Phoenix β drift, cohorts, guardrails, scale. |
| W&B Weavecommercial | β | β | β | β | β | β | β | β | β | β | Teams already living in Weights & Biases who want GenAI runs beside their training lineage. |
| Galileocommercial | β | β | β | β | β | β | β | β | β | β | Guardrail-style metrics computed continuously β hallucination and adherence scoring as a first-class product. |
| HoneyHive, Humanloop, PromptLayer, Vellumcommercial | β | β | β | β | β | β | β | β | β | β | Non-engineers editing and testing prompts. Buy these for the collaboration, not the tracing. |
| General APM that now speaks LLM | |||||||||||
| Datadog LLM Observabilitycommercial | β | β | β | β | β | β | β | β mature | β | β | One pane with the rest of the estate. Alerting and on-call integration are far ahead of LLM-native tools; prompt payloads inflate the ingest bill. |
| New Relic, Dynatrace, Elastic, Splunk, Coralogixcommercial | β | β | β | β | β | β | β | β | β | β | Consolidation. AI monitoring bolted onto existing APM β thin on quality evaluation, strong on everything operational. |
| Sentry AI MonitoringOSS + SaaS | β | β | β | β | β | β | β | β | β | β | Error-first rather than trace-first. Good if your real question is βwhat threwβ, not βwhat was saidβ. |
5 Β· What βevalβ actually means framing
βWe need evalsβ is four different projects wearing one word. Tools specialise in different rows, and buying the wrong row is the most common mistake in this category.
| Kind | What it checks | Runs where | Tools built for it |
|---|---|---|---|
| Unit / assertion evals | Deterministic checks on one output: JSON valid, contains the citation, under N tokens, no PII, latency budget | CI, on every PR | promptfoo, DeepEval, pytest + any judge library |
| Regression evals | A curated dataset scored before and after a change β did the change make it worse anywhere? | CI, pre-release gate | Langfuse, Braintrust, LangSmith, Opik, MLflow Evaluate |
| RAG-specific evals | Faithfulness, answer relevance, context precision/recall β separating retrieval failure from generation failure | CI and offline analysis | Ragas, Phoenix, TruLens, continuous-eval |
| Agent / trajectory evals | Did it pick the right tools, in a sensible order, without looping β and did the task complete? | CI, and sampled in production | Inspect AI, DeepEval, AgentEvals, Opik, Agent Evaluation (Databricks) |
| Safety / red team | Jailbreaks, prompt injection, toxic and unsafe output, data leakage | Pre-release, and periodically | garak, PyRIT, promptfoo redteam, Giskard |
| Benchmarks | Public academic scores β MMLU, GSM8K, HumanEval. Useful for picking a model, near-useless for your app | Model selection only | lm-evaluation-harness, HELM, Inspect AI |
| Online / production eval | Scoring a sample of real traffic; user feedback; drift in inputs | Continuously, in production | Langfuse, Galileo, Arize, Datadog, Lakehouse Monitoring |
6 Β· Eval β open source matrix
| Tool | Unit / assert |
Regression | RAG | Agent / traj. |
Red team | LLM judge | CI-first | UI | What you actually get |
|---|---|---|---|---|---|---|---|---|---|
| promptfooMIT | β | β | β | β | β strong | β | β yes | β local | A YAML test file, a CLI, and a matrix view comparing prompts Γ models Γ providers. Also ships a serious red-team generator. The fastest thing to put in CI on day one. |
| RagasApache 2.0 | β | β | β purpose-built | β | β | β | β | β library | The reference metric set for RAG: faithfulness, answer relevancy, context precision and recall. A library, not a platform β you bring the runner and the storage. |
| DeepEval (Confident AI)Apache 2.0 | β | β | β | β | β DeepTeam | β | β pytest | β SaaS | βPytest for LLMsβ β 40+ prebuilt metrics as assertions in a normal test suite. The most natural fit if your team already lives in pytest. |
| Inspect AI (UK AISI)MIT | β | β | β | β strong | β | β | β | β viewer | A rigorous evaluation framework β solvers, scorers, sandboxed tool use, agent trajectories. Built by a national safety institute, and it shows in the rigour. |
| GiskardApache 2.0 | β | β | β | β | β scan | β | β | β | An automatic vulnerability scan β point it at a model and it generates its own probes for hallucination, bias, injection and disclosure. |
| EvidentlyApache 2.0 | β | β | β | β | β | β | β | β reports | Drift and data-quality heritage extended to LLMs β the strongest of these on monitoring over time rather than one-off scoring. |
| MLflow EvaluateOSS | β | β | β | β | β | β | β | β | Evaluation results stored as first-class MLflow runs, versioned alongside prompts and models. The zero-new-tools answer if you run MLflow. |
| TruLensMIT | β | β | β | β | β | β | β | β | Feedback functions that run inside the app, so scoring happens on live calls rather than only in a batch job. |
| lm-evaluation-harness (EleutherAI), HELM (Stanford)OSS | β | β | β | β | β | β | β | β | Academic benchmark harnesses. Use to choose a base model; they say nothing about your application. |
| Opik, Langfuse, Phoenix eval modulesOSS + SaaS | β | β | β | β | β | β | β | β | Evals attached to the same store as the traces β so a failing production trace becomes a dataset row in one click. See the observability matrix above. |
7 Β· Eval β commercial platforms matrix
These overlap heavily with the observability table, because the serious eval products are observability products β the dataset comes from the traces. What you are buying beyond the OSS libraries is the workflow around them: annotation queues, permissions, experiment history and reporting people outside the team will actually read.
| Platform | What it adds over the OSS libraries | Watch out for | Buy it when |
|---|---|---|---|
| Braintrustcommercial | Experiment diffs a human can read, prompt playground with real datasets behind it, hybrid deployment that keeps payloads in your VPC | Priced around eval volume β costs scale with how seriously you use it | Prompt iteration is the bottleneck and several people do it |
| LangSmithcommercial | Annotation queues, dataset versioning, tight LangGraph trajectory views | Pulls you toward one framework's worldview | Already on LangChain/LangGraph |
| Galileocommercial | Proprietary hallucination/adherence metrics that run continuously without a labelled set | You are trusting metrics you cannot fully inspect | You need production quality signal before you have a dataset |
| Patronus AI, Okareo, Freeplay, Maxim AIcommercial | Managed judges, scenario generation, and non-engineer review workflows | Crowded segment with fast-moving feature parity | You want the workflow bought rather than assembled |
| Confident AIcommercial | The hosted layer over DeepEval β dataset management, regression history, sharing | Tied to DeepEval's metric model | You started with DeepEval and outgrew local runs |
8 Β· Safety & red-team tooling often missed
A separate category, and the one most often discovered late β usually during a security review, three weeks before launch. These generate adversarial inputs rather than scoring your own.
| Tool | Origin | What it does | Use it for |
|---|---|---|---|
| garakApache 2.0 | NVIDIA | A vulnerability scanner for LLMs β dozens of probe families (jailbreak, encoding attacks, prompt leakage, toxicity, package hallucination) run as a sweep | βScan this endpoint and tell me what breaksβ |
| PyRITMIT | Microsoft | A risk-identification framework with multi-turn attack orchestration and converters β automates the escalating conversation, not just single prompts | Structured red-team programmes; underpins Azure's red-teaming agent |
| promptfoo red teamMIT | promptfoo | Generates application-specific attacks from a description of your app, mapped to OWASP LLM Top 10 and NIST AI RMF | Red teaming that produces an auditable report |
| DeepTeamApache 2.0 | Confident AI | Red-teaming built on DeepEval's metric model | Already standardised on DeepEval |
| Giskard scanApache 2.0 | Giskard | Automatic probe generation across hallucination, bias, injection and disclosure | A first pass with almost no setup |
| Llama Guard / Purple LlamaOSS | Meta | Classifier models for input/output safety, plus CyberSecEval benchmarks | A self-hosted safety classifier in the request path |
9 Β· The cloud-provider Rosetta stone key table
The same capability, in each platform's vocabulary. This is the table to bring to an architecture review, because most disagreements in that room are people naming the same thing differently.
| Capability | Databricks | Azure | AWS | GCP | Open-source equivalent |
|---|---|---|---|---|---|
| Observability | |||||
| Trace capture | MLflow Tracing (auto-instruments major frameworks) | AI Foundry tracing β Application Insights / Azure Monitor | CloudWatch GenAI Observability, X-Ray; AgentCore Observability for agents | Cloud Trace + Vertex AI Agent Engine tracing | OpenTelemetry + Langfuse / Phoenix |
| Payload logging | AI Gateway inference tables (requests + responses land in Unity Catalog) | APIM diagnostics; AI Foundry model logging | Bedrock model invocation logging β S3 / CloudWatch | Vertex AI request-response logging β BigQuery | Gateway callbacks (LiteLLM β Langfuse) |
| Cost attribution | System tables (billing usage) + AI Gateway usage tracking | Cost Management + APIM token metrics per subscription/product | Cost Explorer + Bedrock application inference profiles for per-tag cost | Billing export to BigQuery, labels on Vertex resources | LiteLLM spend tracking, Langfuse cost per trace |
| Quality monitoring | Lakehouse Monitoring on inference tables | AI Foundry continuous evaluation on live traffic | CloudWatch metrics + scheduled Bedrock evaluation jobs | Vertex AI Model Monitoring | Langfuse / Evidently online scores |
| Evaluation | |||||
| Offline eval runner | MLflow Evaluate; Mosaic AI Agent Evaluation | Azure AI Evaluation SDK (azure-ai-evaluation) |
Bedrock Evaluations β automatic, human, and LLM-as-judge | Vertex AI Gen AI Evaluation Service | promptfoo, DeepEval, Inspect AI |
| Built-in judges | Groundedness, relevance, safety, chunk relevance, plus custom judges | Groundedness, relevance, coherence, fluency, similarity, safety, plus agent evaluators | Correctness, completeness, faithfulness, helpfulness, harmfulness | Pointwise and pairwise autoraters with customisable rubrics | Ragas, DeepEval metrics |
| RAG-specific eval | Retrieval judges over the retrieved chunks | Document retrieval + groundedness evaluators | Bedrock Knowledge Bases RAG evaluation | Vertex RAG Engine evaluation | Ragas, TruLens |
| Human review | Review App β domain experts label agent outputs | Human evaluation flows in AI Foundry | Bedrock human-based evaluation (SageMaker Ground Truth workforce) | Human-in-the-loop via Vertex pipelines | Langfuse / Argilla / Label Studio |
| Red teaming | Bring your own (garak, promptfoo) | AI Red Teaming Agent (PyRIT-based) in AI Foundry | Bring your own; Guardrails for enforcement | Bring your own; Model Armor for enforcement | garak, PyRIT, promptfoo redteam |
| Gateway β see the AI Gateway page for the full comparison | |||||
| AI gateway | Mosaic AI Gateway | API Management AI gateway policies; AI Foundry model router | Bedrock + API Gateway pattern; AgentCore Gateway for tools | Apigee with LLM policies; Vertex endpoints | LiteLLM, Portkey, Envoy AI Gateway |
| Guardrails in path | AI Gateway guardrails + PII detection | Azure AI Content Safety / Prompt Shields | Bedrock Guardrails | Model Armor | Presidio, Llama Guard, Lakera |
10 Β· Which tools cover which boxes overlap
The practical question is rarely βwhich observability toolβ β it is βhow few tools can I get away withβ. Here is the honest overlap.
| Tool / platform | Observability | Eval | Gateway | Guardrails | Realistic verdict |
|---|---|---|---|---|---|
| Langfuse + LiteLLM | β | β | β | β | The default open-source pairing. Two systems to run, covers everything except deep red teaming. Add promptfoo in CI and you are done. |
| Databricks (Mosaic AI) | β | β | β | β | Genuinely covers all four, governed by Unity Catalog β if the workload lives on Databricks. Nothing extra to operate. |
| Azure AI Foundry + APIM | β | β | β | β | Complete, but split across four services with four learning curves. APIM policy language is its own skill. |
| AWS Bedrock stack | β | β | β | β | Evaluation and Guardrails are strong products; the gateway and LLM-native observability are reference architectures you build. |
| GCP Vertex AI + Apigee | β | β | β | β | Good evaluation service and Model Armor; trace tooling is general-purpose rather than LLM-shaped. |
| LangSmith | β | β | β | β | One tool for the build-time loop. You still need a gateway and guardrails elsewhere. |
| Braintrust | β | β | β proxy | β | Its proxy covers key management and caching, but it is not a substitute for an enterprise gateway. |
| Datadog | β | β | β | β | Best operational story, thinnest quality story. Common pattern: Datadog for ops alerting, an LLM-native tool for quality. |
| Helicone / Portkey | β | β | β | β | Gateway-and-observability in one hop β attractively simple, and the same box is now a single point of failure for both. |
11 Β· Five stacks that actually work practice
| Situation | Stack | Why this shape |
|---|---|---|
| One team, first production app | OTel instrumentation β Langfuse (hosted) Β· promptfoo in CI Β· direct provider SDK, no gateway yet | A gateway before the second application is premature. Everything here is a weekend of work. |
| Several teams, no platform mandate | LiteLLM Proxy β Langfuse (self-hosted) Β· promptfoo + Ragas in CI Β· Presidio guardrails at the gateway | The gateway becomes the control point the moment more than two apps exist. Cost attribution stops being a spreadsheet. |
| Regulated enterprise, Azure-standardised | APIM AI policies β AI Foundry evaluation + Content Safety Β· App Insights for traces Β· AI Red Teaming Agent pre-release | Data residency and identity integration usually decide this before anyone compares features. |
| Data already on Databricks | Mosaic AI Gateway β MLflow Tracing β Agent Evaluation + Review App Β· Lakehouse Monitoring on inference tables | The governance boundary is already Unity Catalog. Adding an external tool means exporting the very data you were trying to keep governed. |
| Prompt quality is the bottleneck | Braintrust or LangSmith for the build loop Β· keep your existing APM for ops Β· gateway unchanged | You are buying a workflow for humans, not infrastructure. Do not let it also become your alerting system. |
12 Β· How these tools bill you economics
The pricing model matters more than the sticker price, because LLM payloads are unusually large and the bill scales with something you do not control.
| Billing model | Who uses it | What surprises people |
|---|---|---|
| Per trace / per span | Langfuse Cloud, LangSmith, Braintrust | An agent run is dozens of spans, not one. Agentic workloads can cost 20β50Γ a simple chat app at identical user volume. |
| Per GB ingested | Datadog, Splunk, Elastic | Prompts and responses are large text blobs. Full payload capture on a RAG app can dwarf the rest of your APM bill β sample it, or truncate. |
| Per evaluation run | Braintrust, hosted eval platforms | Plus the judge model's own token cost, which is billed by your model provider and easy to forget entirely. |
| Per seat | Humanloop, PromptLayer, Vellum | Fine while it is the AI team; re-price it before you invite the whole product org in. |
| Self-hosted infrastructure | Langfuse OSS, Phoenix, SigNoz | Free licence, real cost: Postgres + ClickHouse + object storage, plus someone to own upgrades and retention. |
| Bundled into the platform | Databricks, Azure, AWS, GCP | βIncludedβ usually means the compute and storage are billed elsewhere on the same invoice β cheap to start, hard to attribute later. |
13 Β· Decide these before you buy checklist
- Can it export raw OTel spans elsewhere? If not, you are choosing instrumentation and backend together, permanently.
- Where do payloads live? Prompts and responses routinely contain customer data. βSaaS with a DPAβ and βself-hostedβ are different answers to a question Legal will ask β better before the pilot than after.
- What is the path from a bad production trace to a test case? If it is manual copy-paste, evals will quietly stop being updated within a month.
- Who calibrates the judges? An LLM-as-judge that nobody has checked against human labels produces confident numbers that measure nothing.
- What is sampled, and what is retained for how long? Decide before the first bill, not after.
- Does it understand agents? Many tools show a flat list of LLM calls. If you are building loops with tools and sub-agents, a flat list is close to useless.
- What happens when it is down? Tracing should fail open and asynchronously. A blocking observability call in the request path is an outage waiting for a bad afternoon β and a proxy-based tool is in that path by design.