THE PROJECT One line, and why it's a true agent
When a job or pipeline fails, the Copilot investigates: it gathers evidence with read-only drill-down tools (error logs, the failing query plan, cluster metrics, recent deploys/config/permission changes, past similar incidents), correlates them, and produces a structured root-cause analysis + a pre-filled ServiceNow ticket draft. A human reviews before anything is filed or actioned — it never auto-resolves.
Autonomy ladder: G4 reaches Recommend (it drafts an RCA + ticket), then hands to a human gate. Its tools are strictly read-only, so it cannot act even if asked.
1 · CONCEPTUAL DESIGN The logic, no infrastructure
This is the Text-to-SQL loop grown up: same agentic loop + tool contract, but many read-only tools instead of one, and a verification step before the output is trusted. Seven logical roles:
| Logical role | Responsibility | Det. or AI? |
|---|---|---|
| Trigger | A high-sev issue (from G1) or a failed-run event kicks off one investigation with a run-id | Deterministic |
| Investigator (the agent) | Owns the loop: given the failure, decides which evidence tool to call next, reads the result, decides again — until it can explain the cause | AI (drives the loop) |
| Drill-down tools (read-only) | get_run_logs(), get_query_plan(), get_cluster_metrics(), get_recent_changes(), search_past_incidents() | Deterministic |
| Correlator | Lives inside the agent's reasoning: ties the log error to the recent change to the metric spike | AI (grounded) |
| Verifier | A fresh-context second pass: "is every claim in this RCA supported by the gathered evidence?" — rejects ungrounded causes | AI (independent) |
| Drafter | Renders the verified RCA into a structured ticket (summary, evidence, cause, confidence, suggested fix) | Deterministic template |
| Human gate + Audit | A person reviews/edits/files the ticket; every tool call, evidence item, and the approver are logged | Human + Deterministic |
The loop, as a sequence
✅ Why this shape is safe & useful
- Read-only tools — the agent physically cannot mutate anything
- Verifier catches confidently-wrong RCAs before a human sees them
- Human gate on filing/action — it drafts, people decide
- Full evidence chain logged — reproducible & GxP-friendly
⚠️ Design decisions to make now
- Loop termination: cap evidence-gathering steps; graceful "insufficient evidence" (D1)
- Tool contract: return tool errors as feedback, not crashes (D4)
- Grounding: every RCA claim must cite an evidence item id
- Confidence: report low/med/high; never present a guess as certainty
2 · PHYSICAL DESIGN Same Copilot, from laptop-mock to Azure prod
Same read-only Databricks signal collection as G1, plus two new boxes: a past-incident store (RAG, shared with G5) and a ServiceNow draft sink. Build against mock fixtures first.
(G1 / failed-run event)"] --> APP["RCA agent loop
(job / function)"] APP -->|"tool-use loop"| LLM["LLM endpoint"] APP -->|"read-only"| SIG["Signals: logs · query plan · metrics · change history"] APP -->|"retrieve"| RAG["Past-incident + runbook store"] APP -->|"verify (fresh ctx)"| LLM APP --> SN["ServiceNow draft (HITL)"] APP -.audit.-> LOG["Evidence chain + approver"] APP -.reads.-> SEC["Secrets"] classDef core fill:#f3f0ff,stroke:#764ba2,color:#4a3b7a; class APP core;
The "where each role lives" map
| Role | 💻 Local (mock) | 🔷 Azure Databricks | 🟧 AWS Databricks | 🔴 GCP Databricks |
|---|---|---|---|---|
| Where it runs | Python script | Databricks Job (triggered) / Azure Function | Databricks Job / Lambda | Databricks Job / Cloud Run |
| Signals (read-only) | JSON fixtures | system tables · Jobs API · Spark UI / event logs | same | same |
| LLM (loop + verify) | Ollama | Anthropic API · Azure AI Foundry · Mosaic AI Serving | Amazon Bedrock (Claude) | Vertex AI |
| Past-incident store (RAG) | local files + FAISS/Chroma | Vector Search on Unity Catalog | OpenSearch / Kendra | Vertex AI Search |
| Ticket sink | print / .md draft | ServiceNow REST (draft only) | ServiceNow REST | ServiceNow REST |
| Secrets | .env | Secret Scopes / Key Vault | Secrets Manager | Secret Manager |
| Audit (evidence chain) | JSON lines | Delta table + system.access.audit | CloudTrail + Delta | Cloud Audit + Delta |
3 · CROSS-CUTTING CONCERNS The stuff that decides if it's real
| Concern | What it means for this Copilot | Levers / practices |
|---|---|---|
| 💰 Cost | An agent loop = several LLM calls + tool calls per incident, plus a verifier pass. This is the portfolio's most expensive project per run. | Cap max evidence steps (3–6) · cache the stable system prompt + tool defs · only run on high-sev (not every blip) · small model for narration, stronger for correlation |
| 📊 Observability | You must see the investigation: which tools it called, in what order, what evidence, the RCA, the verifier verdict, tokens, latency. | Trace every loop step (Phoenix/Langfuse/MLflow) · one run-id per incident · dashboards on RCA accuracy & avg steps |
| 🔐 Security | Logs/plans are untrusted (injection) and can carry secrets/PII; a "confused-deputy" agent with broad tools is dangerous. | Read-only service principal, least-privilege tool scope · redact before the model · treat all evidence as data, never instructions · no write/exec tool exists |
| 📝 Logging (GxP) | An RCA may inform a regulated decision — the full evidence chain and who approved it must be provable. | Persist every tool call + result + the RCA + model/prompt version + verifier verdict + the human approver · append-only |
| 🛡️ Reliability | Runaway loops, a tool timing out, or no cause found. | Hard step cap + graceful "insufficient evidence, here's what I found" · tool timeouts · return tool errors as feedback so the agent adapts |
| ✅ Evals | "Is the RCA right?" — measurable against resolved post-mortems. | Golden set of past incidents → expected cause · groundedness (every claim cites evidence) · human 👍/👎 · false-cause rate |
4 · OPEN-SOURCE TOOLBOX A free option for every responsibility
| Responsibility | Open-source / free options |
|---|---|
| Databricks access (read-only) | databricks-sdk, Databricks CLI, REST |
| Agent loop / tool use | Plain Python (recommended to learn); the Claude Messages API tool runner; LangGraph |
| Local LLM | Ollama, vLLM, llama.cpp |
| Past-incident RAG | Chroma, FAISS, sentence-transformers (local embeddings) |
| Verifier | Same LLM, fresh context + a strict "refute or support" prompt |
| Tracing | Arize Phoenix, Langfuse, MLflow Tracing, OpenTelemetry |
| Ticketing | ServiceNow REST; for demo, a local markdown/JSON draft |
| Evals | promptfoo, Phoenix Evals, DeepEval |
| Secrets | dotenv, SOPS, Vault |
5 · BUILD IT FREE, LOCAL & MOCK-FIRST $0, offline, zero real infra
logs.json, changes.json, metrics.json, incidents.json. 🔁 Agent loop — plain Python tool-use loop (you see every decision).
🧠 Model — Ollama locally (Claude via the Messages API in prod).
🔎 Verifier — a second call in a clean context.
📤 Output — a printed RCA + ticket draft (wire ServiceNow later).
The whole Copilot (minimal, real — mock-first tool-use loop)
# pip install requests · fixtures in ./mock/*.json import json, requests # --- READ-ONLY tools. Mock now; swap bodies for databricks-sdk / Vector Search later. --- TOOLS = { "get_run_logs": lambda a: json.load(open("mock/logs.json")).get(a["run_id"], "no logs"), "get_recent_changes": lambda a: json.load(open("mock/changes.json")), # deploys/config/grants "get_cluster_metrics":lambda a: json.load(open("mock/metrics.json")).get(a["run_id"], {}), "search_past_incidents":lambda a: json.load(open("mock/incidents.json")), # RAG (shared w/ G5) } def ask(prompt): r = requests.post("http://localhost:11434/api/generate", json={"model":"llama3.1", "prompt":prompt, "stream":False}) return r.json()["response"].strip() # PROD: Claude Messages API, native tool_use def investigate(run_id, max_steps=5): facts, sys = [], ("You are an Ops RCA agent. You may call tools: " + ", ".join(TOOLS) + ". " "Reply with a JSON tool call {\"tool\":..,\"args\":..} OR, when confident, " "{\"rca\":{\"cause\":..,\"confidence\":..,\"evidence_ids\":[..]}}. " "Only use gathered evidence — never invent a cause.\n") for step in range(max_steps): # THE LOOP — model drives it out = ask(sys + f"Failure run_id={run_id}\nEvidence so far:\n{json.dumps(facts)}") msg = json.loads(out) if "rca" in msg: # model decided it's done return verify(msg["rca"], facts) tool, args = msg["tool"], msg.get("args", {"run_id":run_id}) try: facts.append({"id":len(facts), "tool":tool, "result":TOOLS[tool](args)}) # act (read-only) except Exception as e: facts.append({"id":len(facts), "tool":tool, "error":str(e)}) # errors as feedback return {"cause":"insufficient evidence", "confidence":"low", "evidence":facts} # graceful stop def verify(rca, facts): # fresh-context grounding check v = ask(f"Evidence:\n{json.dumps(facts)}\nProposed cause: {rca['cause']}\n" "Is EVERY part supported by the evidence? Answer SUPPORTED or UNSUPPORTED + why.") rca["verifier"] = v return rca # drafter + human gate happen after this rca = investigate("8421") print(json.dumps(rca, indent=2)) # → renders to a ServiceNow DRAFT for a human
for loop with the model choosing the next tool is the agent — the same shape as the
Text-to-SQL loop, now with many read-only tools and a verifier gate. Note what's missing on purpose: there is no
write tool, so "resolve the incident" is not something it can do.stop_reason loop (tool_use → run tool → append tool_result → repeat until
end_turn), structured output for the final RCA, prompt caching on the tool defs, and the verifier as
a fresh-context second call. It exercises D1, D2, D4, D5 in one build.6 · PRODUCTION FAILURE POINTS
7 · CONCEPTS → CCA MAP
| You practised… | CCA domain |
|---|---|
| The agentic loop, loop termination, the model driving tool selection | D1 · Agentic Architecture |
| Many tools, the tool-result contract, returning tool errors as feedback, least privilege | D2 · Tool Design & MCP |
| Grounding, structured RCA output, the verifier prompt | D3 · Prompt Engineering & Structured Output |
| RAG over past incidents / runbooks | D5 (retrieval) |
| Verification against silent failures, cost of loops, tracing, evals, audit | D5 · Context & Reliability |
8 · LEVEL IT UP
- Adversarial verifier panel → 2–3 independent verifiers must agree the cause is grounded before it reaches a human.
- Richer evidence tools → lineage (blast radius), upstream freshness, cost delta on the failing run.
- Auto-link the runbook → tie into G5 so the draft includes the recovery steps.
- Suggested remediation (still draft) → propose the fix as text for a human to apply — the first step toward Recommend, never Act.
- Post-mortem feedback → resolved tickets become new eval cases and new RAG entries, so it gets better over time.