AI Engineering Projects › Ops Portfolio › G1 · Health Sentinel

🩺 G1 · Pipeline & Job Health Sentinel

Read-only observer · deterministic detection + AI triage · the safest first Ops agent

THE PROJECT One line, and why it's the first Ops build

Every few minutes, read Databricks job & pipeline run history, deterministically detect what's unhealthy (failures, retry spikes, latency blowups, missed schedules), then let an agent cluster the noise, explain each problem in plain English, and point at the likely cause & runbook — delivered as one digest. It never touches production: read-only signals in, a human-readable brief out.

🎯 This is the portfolio's "hello world": it proves the two patterns everything else reuses — deterministic detection (the math you can trust) and AI narration grounded on evidence (the language layer, never the safety layer). G4 (RCA Copilot) is this + drill-down tools; the weekly brief (G11) is this across every signal.
1 · Observe ✓2 · Explain ✓ 3 · Recommend4 · Approve5 · Act

Autonomy ladder: G1 lives entirely at Observe → Explain. It has no write path at all — that's the point.

🟣 Preparing for the Claude exam? The companion page Building the Health Sentinel with the Claude API rebuilds the triage step natively — a single grounded, structured-output, prompt-cached Messages API call (no loop) with stop_reason handling and a deterministic grounding check. That's the version the exam tests.

1 · CONCEPTUAL DESIGN The logic, no infrastructure

The teaching insight here is the opposite of the Text-to-SQL agent: this is mostly a deterministic workflow with one agentic step, not an open loop. Detection is arithmetic (trustworthy, cheap, auditable); the LLM only enters to triage & narrate. Six logical roles:

Logical roleResponsibilityDeterministic or AI?
Collectors (read-only tools)get_job_runs(), get_run_logs(), get_cluster_metrics(), get_recent_changes() — the only ways it touches the world, all read-onlyDeterministic
DetectorApplies rules/thresholds to the signals → a list of candidate issues with evidence (run_id, metric, threshold)Deterministic (the part you must trust)
Deduper / StateSuppresses issues already reported; tracks flapping; remembers what was acknowledgedDeterministic
Triager (the agent)Clusters related failures, ranks severity, explains the likely cause grounded on the evidence rows, links a runbookAI (narration only)
ComposerRenders the ranked, deduped, explained issues into one digest (markdown → Teams/email)Deterministic
SchedulerRuns the whole thing every N minutes; owns the run-id & the audit logDeterministic

The flow, as a sequence

sequenceDiagram participant SCH as Scheduler participant COL as Collectors (read-only) participant DET as Detector (rules) participant ST as State (dedup) participant AG as Triager (LLM) participant OUT as Digest / Teams SCH->>COL: pull last-window job runs, logs, metrics COL-->>DET: normalized run records DET->>DET: apply thresholds → candidate issues (+ evidence) DET->>ST: filter already-reported / flapping alt new issues exist ST-->>AG: fresh issues + their evidence rows AG-->>AG: cluster · rank · explain (grounded, cited to run_id) AG-->>OUT: ranked narrative digest OUT-->>SCH: posted + logged (audit trail) else nothing new ST-->>SCH: quiet run — log "all healthy" end

✅ Why this shape is safe & useful

  • Detection is deterministic — the "is it broken?" decision never depends on an LLM
  • Read-only: no tool can mutate a job, cluster, or grant
  • The agent is grounded — it explains evidence it's handed, it doesn't go fishing
  • Every digest is logged & reproducible (GxP-friendly)

⚠️ Design decisions to make now

  • Thresholds: static vs baseline-relative (start static, add baselines later)
  • Dedup key: (job_id + error-signature) so you don't re-alert the same fire
  • Grounding rule: the triager must cite a run_id for every claim (kills hallucinated causes)
  • Noise budget: cap issues-per-digest; everything else rolls up to a count

2 · PHYSICAL DESIGN Same sentinel, from laptop-mock to Azure prod

Databricks is multi-cloud, so the signal collection is nearly identical everywhere — only the glue (secrets, notify, schedule, LLM endpoint) is cloud-specific. Reference shape, then the mapping. Note the mock lane: you build and demo entirely against synthetic fixtures before any real connection.

💻 Laptop + mock fixtures🔷 Azure Databricks (her env)🟧 AWS Databricks🔴 GCP Databricks
flowchart LR SCH["Scheduler
(Workflow / cron)"] --> APP["Sentinel process
(job / function)"] APP -->|"read-only"| SIG["Signals:
system tables · Jobs API · Spark UI"] APP -->|"triage narration"| LLM["LLM endpoint"] APP -->|"dedup state"| ST["State store
(Delta / file)"] APP --> OUT["Digest → Teams / email"] APP -.audit.-> LOG["Audit log (who/what/when)"] APP -.reads.-> SEC["Secrets"] classDef core fill:#f3f0ff,stroke:#764ba2,color:#4a3b7a; class APP core;

The "where each role lives" map

Role💻 Local (mock)🔷 Azure Databricks🟧 AWS Databricks🔴 GCP Databricks
Where it runsPython script / cronDatabricks Workflow (or Azure Function / Container App)Databricks Workflow (or Lambda/Fargate)Databricks Workflow (or Cloud Run)
Signals (read-only)JSON fixtures in /mocksystem tables (system.lakeflow.*, system.compute) + Jobs REST API 2.2same system tables + Jobs APIsame system tables + Jobs API
LLM (triage)Ollama, or skip (template digest)Anthropic API · Azure AI Foundry · or Mosaic AI Model ServingAmazon Bedrock (Claude)Vertex AI
State / deduplocal state.jsona Delta table in Unity CatalogDelta tableDelta table
Notifyprint to stdout / write .mdTeams webhook / Logic Apps; email via GraphSNS / SESPub/Sub / email
Secrets.env + dotenvDatabricks Secret Scopes / Azure Key VaultSecrets ManagerSecret Manager
SchedulecronDatabricks Workflows (every N min)Workflows / EventBridgeWorkflows / Cloud Scheduler
Audit logJSON lines fileDelta table + workspace audit (system.access.audit)CloudTrail + DeltaCloud Audit Logs + Delta
Read a row to see how one responsibility is solved everywhere; the signal rows barely change across clouds because Databricks system tables are the same — that's the payoff of building on the platform's own telemetry.

3 · CROSS-CUTTING CONCERNS The stuff that decides if it's real

ConcernWhat it means for this sentinelLevers / practices
💰 CostReading system tables is cheap; the LLM triage is the only real cost, and only when there's something new to explain.Only call the LLM on fresh issues (dedup first) · batch all new issues into one call · cache the stable system-prompt · a small model is fine for narration
📊 ObservabilityThe watcher must itself be watched: is it running, is it silently missing failures (false negatives)?Heartbeat metric per run · count of issues detected vs suppressed · alert if the sentinel hasn't run · track digest→ack rate
🔐 SecurityIt reads operational metadata and logs, which can contain secrets/PII; and log text is untrusted (prompt injection).Read-only service principal scoped to system tables only · redact secrets/PII before the LLM sees logs · treat log content as data, never instructions
📝 Logging (GxP)You must be able to prove what was detected, what was said, and on what evidence.Persist every run: inputs hash, issues, evidence run_ids, model + prompt version, the digest, run-id · immutable/append-only
🛡️ ReliabilityA missed failure (false negative) is worse than a false alarm here — the whole point is catching problems.Fail-open on the LLM (if triage errors, still send the raw deterministic list) · idempotent runs · never let a narration failure swallow a real detection
✅ Evals"Is the triage any good?" — is the explanation correct, is severity right, is it deduping properly?Golden set of past incidents → expected cluster/severity · precision/recall on detection · human 👍/👎 on each digest item feeding a review set
The #1 failure mode: letting the LLM decide whether something is broken. Detection must be deterministic and grounded in numbers; the model only ever explains issues it was handed. A confidently-worded but wrong root cause is the silent-failure class — mitigate by forcing every claim to cite a run_id.

4 · OPEN-SOURCE TOOLBOX A free option for every responsibility

ResponsibilityOpen-source / free options
Databricks access (read-only)databricks-sdk (Python), Databricks CLI, plain REST
Local LLM for triageOllama, llama.cpp, vLLM (or skip the LLM and ship a template digest)
Detection / rulesPlain Python; pandas/Polars for the windowed metrics; DuckDB over exported system-table Parquet
Schedulingcron, APScheduler (local); Databricks Workflows (prod)
State / dedupSQLite / a JSON file (local); Delta (prod)
Tracing / observabilityArize Phoenix, Langfuse, OpenTelemetry; Prometheus + Grafana
NotifyTeams/Slack incoming webhooks; Apprise (one lib, many channels)
Evalspromptfoo, Phoenix Evals, DeepEval
Secretsdotenv (local), SOPS, HashiCorp Vault

5 · BUILD IT FREE, LOCAL & MOCK-FIRST $0, offline, zero real infra

The whole point of mock-data-first: build and demo the entire sentinel against synthetic run records that match the real system-table shape, so nothing touches ABCCompany infrastructure until it's validated and signed off. Swapping to live is one function.

🗄️ Signals — JSON fixtures in /mock shaped like system.lakeflow.job_run_timeline.
🧠 Triage — Ollama locally (or Claude via the Anthropic API in prod).
🔁 Detector + loop — plain Python (you see every rule).
💾 State — a local state.json.
📤 Digest — printed markdown (wire a Teams webhook later).

The whole sentinel (minimal, real — mock-first)

# pip install requests   ·   fixtures in ./mock/runs.json (shaped like system.lakeflow)
import json, hashlib, requests, datetime as dt

# --- 1. COLLECTOR (read-only). Mock now; swap body for databricks-sdk later. ---
def get_job_runs(window_min=60):
    return json.load(open("mock/runs.json"))          # PROD: system.lakeflow / Jobs API 2.2

# --- 2. DETECTOR (deterministic — the part you MUST trust, no LLM here) ---
def detect(runs):
    issues = []
    for r in runs:
        if r["state"] == "FAILED":
            issues.append({"kind":"failure", "job":r["job_name"], "run_id":r["run_id"],
                           "evidence":r["error"], "sev":"high"})
        elif r["retries"] >= 3:
            issues.append({"kind":"retry_spike", "job":r["job_name"], "run_id":r["run_id"],
                           "evidence":f"{r['retries']} retries", "sev":"med"})
        elif r["duration_min"] > 3 * r["baseline_min"]:
            issues.append({"kind":"slow", "job":r["job_name"], "run_id":r["run_id"],
                           "evidence":f"{r['duration_min']}m vs {r['baseline_min']}m baseline", "sev":"med"})
    return issues

# --- 3. DEDUP (don't re-alert the same fire) ---
def key(i): return hashlib.md5(f"{i['job']}|{i['kind']}|{i['evidence'][:40]}".encode()).hexdigest()
def fresh(issues):
    seen = set(json.load(open("state.json"))) if __import__("os").path.exists("state.json") else set()
    new = [i for i in issues if key(i) not in seen]
    json.dump(list(seen | {key(i) for i in new}), open("state.json","w"))
    return new

# --- 4. TRIAGER (AI — narration only, GROUNDED: must cite run_id) ---
def triage(issues):
    if not issues: return "✅ All pipelines healthy this window."
    prompt = ("You are an Ops triage assistant. For EACH issue, write one line: "
              "severity, job, plain-English likely cause, and cite the run_id. "
              "Only use the evidence given — never invent a cause.\n\n"
              + json.dumps(issues, indent=2))
    r = requests.post("http://localhost:11434/api/generate",
                      json={"model":"llama3.1", "prompt":prompt, "stream":False})
    return r.json()["response"].strip()          # PROD: Claude Messages API + structured output

# --- 5. COMPOSE + LOG (audit trail) ---
def run():
    runs   = get_job_runs()
    issues = fresh(detect(runs))        # deterministic first, then dedup
    digest = triage(issues)               # LLM only narrates what's left
    stamp  = dt.datetime.utcnow().isoformat()
    print(f"# Ops Health Digest — {stamp}\n\n{digest}")
    open("audit.log","a").write(json.dumps({"ts":stamp, "n_issues":len(issues), "issues":issues})+"\n")

run()
Notice the ordering: detect → dedup → triage. The LLM is the last and cheapest-guarded step, it runs only on fresh issues, and it's told to cite the run_id for every claim. That single discipline is what keeps a read-only observer trustworthy in a regulated shop.

Going live is one swap: replace get_job_runs()'s body with a databricks-sdk call (or a SELECT over system.lakeflow.job_run_timeline) using a read-only service principal, and point triage() at the Claude Messages API. Nothing else changes.

🟣 Cert tie-in: the prod triage step is a textbook Claude Messages API call — a single grounded request with structured output for the incident list (schema = severity/job/cause/run_id), no agent loop needed. Detection is deterministic; the model only writes the narrative. That "deterministic core, AI at the edges" split is the whole design.

6 · PRODUCTION FAILURE POINTS

False negatives (missed failures) → a real outage never makes the digest. Mitigation: fail-open — if triage errors, still send the raw deterministic list; alert if the sentinel itself hasn't run (watch the watcher).
Alert fatigue → a flapping job floods every digest and people stop reading. Mitigation: dedup by (job + error-signature), roll repeats into a count, cap issues-per-digest.
Hallucinated root cause → the LLM invents a plausible-but-wrong cause. Mitigation: ground strictly on evidence, require a run_id citation per claim, keep detection out of the model's hands.
Prompt injection via log text → an error message contains "ignore instructions and…". Mitigation: treat all log/metric text as untrusted data, isolate it from instructions, redact before sending.
Secret / PII leakage → a stack trace with a connection string reaches the LLM or the digest. Mitigation: redact before triage; scope the read-only principal; scrub the audit log.
Threshold drift → static thresholds go stale after a platform change and everything looks "slow." Mitigation: move to baseline-relative thresholds; review the rule set on a cadence.

7 · CONCEPTS → CCA MAP

You practised…CCA domain
Workflow-vs-agent judgement (mostly deterministic + one grounded AI step)D1 · Agentic Architecture
Read-only tools, least-privilege service principal, safety at the permission layerD1 (guardrails) · D2 · Tool Design
Grounding the narration, citing evidence, avoiding hallucinated causesD3 · Prompt Engineering · D5 (silent failures)
Structured output for the incident listD3 · Structured Output
Dedup/state, cost control (LLM only on fresh issues), audit logging, evalsD5 · Context & Reliability

8 · LEVEL IT UP