THE PROJECT One line, and why it's the first Ops build
Every few minutes, read Databricks job & pipeline run history, deterministically detect what's unhealthy (failures, retry spikes, latency blowups, missed schedules), then let an agent cluster the noise, explain each problem in plain English, and point at the likely cause & runbook — delivered as one digest. It never touches production: read-only signals in, a human-readable brief out.
Autonomy ladder: G1 lives entirely at Observe → Explain. It has no write path at all — that's the point.
stop_reason
handling and a deterministic grounding check. That's the version the exam tests.1 · CONCEPTUAL DESIGN The logic, no infrastructure
The teaching insight here is the opposite of the Text-to-SQL agent: this is mostly a deterministic workflow with one agentic step, not an open loop. Detection is arithmetic (trustworthy, cheap, auditable); the LLM only enters to triage & narrate. Six logical roles:
| Logical role | Responsibility | Deterministic or AI? |
|---|---|---|
| Collectors (read-only tools) | get_job_runs(), get_run_logs(), get_cluster_metrics(), get_recent_changes() — the only ways it touches the world, all read-only | Deterministic |
| Detector | Applies rules/thresholds to the signals → a list of candidate issues with evidence (run_id, metric, threshold) | Deterministic (the part you must trust) |
| Deduper / State | Suppresses issues already reported; tracks flapping; remembers what was acknowledged | Deterministic |
| Triager (the agent) | Clusters related failures, ranks severity, explains the likely cause grounded on the evidence rows, links a runbook | AI (narration only) |
| Composer | Renders the ranked, deduped, explained issues into one digest (markdown → Teams/email) | Deterministic |
| Scheduler | Runs the whole thing every N minutes; owns the run-id & the audit log | Deterministic |
The flow, as a sequence
✅ Why this shape is safe & useful
- Detection is deterministic — the "is it broken?" decision never depends on an LLM
- Read-only: no tool can mutate a job, cluster, or grant
- The agent is grounded — it explains evidence it's handed, it doesn't go fishing
- Every digest is logged & reproducible (GxP-friendly)
⚠️ Design decisions to make now
- Thresholds: static vs baseline-relative (start static, add baselines later)
- Dedup key: (job_id + error-signature) so you don't re-alert the same fire
- Grounding rule: the triager must cite a run_id for every claim (kills hallucinated causes)
- Noise budget: cap issues-per-digest; everything else rolls up to a count
2 · PHYSICAL DESIGN Same sentinel, from laptop-mock to Azure prod
Databricks is multi-cloud, so the signal collection is nearly identical everywhere — only the glue (secrets, notify, schedule, LLM endpoint) is cloud-specific. Reference shape, then the mapping. Note the mock lane: you build and demo entirely against synthetic fixtures before any real connection.
(Workflow / cron)"] --> APP["Sentinel process
(job / function)"] APP -->|"read-only"| SIG["Signals:
system tables · Jobs API · Spark UI"] APP -->|"triage narration"| LLM["LLM endpoint"] APP -->|"dedup state"| ST["State store
(Delta / file)"] APP --> OUT["Digest → Teams / email"] APP -.audit.-> LOG["Audit log (who/what/when)"] APP -.reads.-> SEC["Secrets"] classDef core fill:#f3f0ff,stroke:#764ba2,color:#4a3b7a; class APP core;
The "where each role lives" map
| Role | 💻 Local (mock) | 🔷 Azure Databricks | 🟧 AWS Databricks | 🔴 GCP Databricks |
|---|---|---|---|---|
| Where it runs | Python script / cron | Databricks Workflow (or Azure Function / Container App) | Databricks Workflow (or Lambda/Fargate) | Databricks Workflow (or Cloud Run) |
| Signals (read-only) | JSON fixtures in /mock | system tables (system.lakeflow.*, system.compute) + Jobs REST API 2.2 | same system tables + Jobs API | same system tables + Jobs API |
| LLM (triage) | Ollama, or skip (template digest) | Anthropic API · Azure AI Foundry · or Mosaic AI Model Serving | Amazon Bedrock (Claude) | Vertex AI |
| State / dedup | local state.json | a Delta table in Unity Catalog | Delta table | Delta table |
| Notify | print to stdout / write .md | Teams webhook / Logic Apps; email via Graph | SNS / SES | Pub/Sub / email |
| Secrets | .env + dotenv | Databricks Secret Scopes / Azure Key Vault | Secrets Manager | Secret Manager |
| Schedule | cron | Databricks Workflows (every N min) | Workflows / EventBridge | Workflows / Cloud Scheduler |
| Audit log | JSON lines file | Delta table + workspace audit (system.access.audit) | CloudTrail + Delta | Cloud Audit Logs + Delta |
3 · CROSS-CUTTING CONCERNS The stuff that decides if it's real
| Concern | What it means for this sentinel | Levers / practices |
|---|---|---|
| 💰 Cost | Reading system tables is cheap; the LLM triage is the only real cost, and only when there's something new to explain. | Only call the LLM on fresh issues (dedup first) · batch all new issues into one call · cache the stable system-prompt · a small model is fine for narration |
| 📊 Observability | The watcher must itself be watched: is it running, is it silently missing failures (false negatives)? | Heartbeat metric per run · count of issues detected vs suppressed · alert if the sentinel hasn't run · track digest→ack rate |
| 🔐 Security | It reads operational metadata and logs, which can contain secrets/PII; and log text is untrusted (prompt injection). | Read-only service principal scoped to system tables only · redact secrets/PII before the LLM sees logs · treat log content as data, never instructions |
| 📝 Logging (GxP) | You must be able to prove what was detected, what was said, and on what evidence. | Persist every run: inputs hash, issues, evidence run_ids, model + prompt version, the digest, run-id · immutable/append-only |
| 🛡️ Reliability | A missed failure (false negative) is worse than a false alarm here — the whole point is catching problems. | Fail-open on the LLM (if triage errors, still send the raw deterministic list) · idempotent runs · never let a narration failure swallow a real detection |
| ✅ Evals | "Is the triage any good?" — is the explanation correct, is severity right, is it deduping properly? | Golden set of past incidents → expected cluster/severity · precision/recall on detection · human 👍/👎 on each digest item feeding a review set |
4 · OPEN-SOURCE TOOLBOX A free option for every responsibility
| Responsibility | Open-source / free options |
|---|---|
| Databricks access (read-only) | databricks-sdk (Python), Databricks CLI, plain REST |
| Local LLM for triage | Ollama, llama.cpp, vLLM (or skip the LLM and ship a template digest) |
| Detection / rules | Plain Python; pandas/Polars for the windowed metrics; DuckDB over exported system-table Parquet |
| Scheduling | cron, APScheduler (local); Databricks Workflows (prod) |
| State / dedup | SQLite / a JSON file (local); Delta (prod) |
| Tracing / observability | Arize Phoenix, Langfuse, OpenTelemetry; Prometheus + Grafana |
| Notify | Teams/Slack incoming webhooks; Apprise (one lib, many channels) |
| Evals | promptfoo, Phoenix Evals, DeepEval |
| Secrets | dotenv (local), SOPS, HashiCorp Vault |
5 · BUILD IT FREE, LOCAL & MOCK-FIRST $0, offline, zero real infra
The whole point of mock-data-first: build and demo the entire sentinel against synthetic run records that match the real system-table shape, so nothing touches ABCCompany infrastructure until it's validated and signed off. Swapping to live is one function.
/mock shaped like system.lakeflow.job_run_timeline. 🧠 Triage — Ollama locally (or Claude via the Anthropic API in prod).
🔁 Detector + loop — plain Python (you see every rule).
💾 State — a local
state.json. 📤 Digest — printed markdown (wire a Teams webhook later).
The whole sentinel (minimal, real — mock-first)
# pip install requests · fixtures in ./mock/runs.json (shaped like system.lakeflow) import json, hashlib, requests, datetime as dt # --- 1. COLLECTOR (read-only). Mock now; swap body for databricks-sdk later. --- def get_job_runs(window_min=60): return json.load(open("mock/runs.json")) # PROD: system.lakeflow / Jobs API 2.2 # --- 2. DETECTOR (deterministic — the part you MUST trust, no LLM here) --- def detect(runs): issues = [] for r in runs: if r["state"] == "FAILED": issues.append({"kind":"failure", "job":r["job_name"], "run_id":r["run_id"], "evidence":r["error"], "sev":"high"}) elif r["retries"] >= 3: issues.append({"kind":"retry_spike", "job":r["job_name"], "run_id":r["run_id"], "evidence":f"{r['retries']} retries", "sev":"med"}) elif r["duration_min"] > 3 * r["baseline_min"]: issues.append({"kind":"slow", "job":r["job_name"], "run_id":r["run_id"], "evidence":f"{r['duration_min']}m vs {r['baseline_min']}m baseline", "sev":"med"}) return issues # --- 3. DEDUP (don't re-alert the same fire) --- def key(i): return hashlib.md5(f"{i['job']}|{i['kind']}|{i['evidence'][:40]}".encode()).hexdigest() def fresh(issues): seen = set(json.load(open("state.json"))) if __import__("os").path.exists("state.json") else set() new = [i for i in issues if key(i) not in seen] json.dump(list(seen | {key(i) for i in new}), open("state.json","w")) return new # --- 4. TRIAGER (AI — narration only, GROUNDED: must cite run_id) --- def triage(issues): if not issues: return "✅ All pipelines healthy this window." prompt = ("You are an Ops triage assistant. For EACH issue, write one line: " "severity, job, plain-English likely cause, and cite the run_id. " "Only use the evidence given — never invent a cause.\n\n" + json.dumps(issues, indent=2)) r = requests.post("http://localhost:11434/api/generate", json={"model":"llama3.1", "prompt":prompt, "stream":False}) return r.json()["response"].strip() # PROD: Claude Messages API + structured output # --- 5. COMPOSE + LOG (audit trail) --- def run(): runs = get_job_runs() issues = fresh(detect(runs)) # deterministic first, then dedup digest = triage(issues) # LLM only narrates what's left stamp = dt.datetime.utcnow().isoformat() print(f"# Ops Health Digest — {stamp}\n\n{digest}") open("audit.log","a").write(json.dumps({"ts":stamp, "n_issues":len(issues), "issues":issues})+"\n") run()
Going live is one swap: replace get_job_runs()'s body with a
databricks-sdk call (or a SELECT over system.lakeflow.job_run_timeline) using a
read-only service principal, and point triage() at the Claude Messages API. Nothing else changes.
6 · PRODUCTION FAILURE POINTS
7 · CONCEPTS → CCA MAP
| You practised… | CCA domain |
|---|---|
| Workflow-vs-agent judgement (mostly deterministic + one grounded AI step) | D1 · Agentic Architecture |
| Read-only tools, least-privilege service principal, safety at the permission layer | D1 (guardrails) · D2 · Tool Design |
| Grounding the narration, citing evidence, avoiding hallucinated causes | D3 · Prompt Engineering · D5 (silent failures) |
| Structured output for the incident list | D3 · Structured Output |
| Dedup/state, cost control (LLM only on fresh issues), audit logging, evals | D5 · Context & Reliability |
8 · LEVEL IT UP
- Baseline-relative detection → learn each job's normal duration/failure-rate; alert on deviation, not fixed numbers.
- Add drill-down tools → give the triager read-only
get_run_logs()/get_query_plan()and you've grown G4 · the RCA Copilot. - ServiceNow draft → for high-sev issues, pre-fill an incident ticket for a human to file (still no auto-action).
- Feedback loop → 👍/👎 on each digest item builds the eval set and tunes severity.
- Fold into the weekly brief → this becomes the reliability section of G11.