Reading about agentic architecture only gets you so far. This is a curated shortlist of real
problems — each deliberately chosen because it forces you to learn a distinct set of AI-engineering and
agentic concepts in practice. We'll pick one, then produce a full deep-dive for it (see
what each deep-dive contains).
🎯
Selection criteria — every project here is: (1) a genuine problem people pay to
solve, (2) buildable as a
minimal but real implementation, (3) rich in
production failure modes
worth studying, and (4) mapped to concepts in the
CCA material and
Gen AI Fundamentals.
AT A GLANCE Every project — one crisp table
Design = the architecture deep-dive · Code = runnable implementation. (P1–P10 are the
learning-path idea cards further down; the rows below are the built-out projects.)
| Domain | Theme | Project | What it does | AI part | Design | Code |
| Agentic core | NL→SQL | P0 · Text-to-SQL Agent | English → SQL, runs it, reads the DB error, fixes & retries until it works | The self-correcting agent loop (writes/repairs SQL) | design · Claude API · UC | repo |
| Advanced | Tiered memory | Second-Brain Assistant | Personal work assistant over Redis + SQLite + vector + graph memory | Memory routing, retrieval fusion & all reasoning | design | — |
| Integration | MCP | Lakehouse Ops MCP Server | One FastMCP server exposing every MCP aspect (tools · resources · prompts · transports · sampling/elicitation) | Host's model consumes the server's tools/resources | design | repo |
| Foundations | API internals | MyOwnAnthropicServer | A minimal POST /v1/messages shell to learn every server responsibility | The LLM call is one stage; the rest is the server | design | — |
| Foundations | Design patterns | GenericAnthropicMessageClient | One robust client for every request/response case via Facade + middleware + Strategy | Wraps the SDK/model calls; policies are pluggable | design | — |
| Foundations | Architecture | Cross-Cutting Concerns | Logging/tracing/cost/redaction/audit as pluggable ports (hexagonal + DI) | Wraps every model call; adapters swap per environment | design | — |
| Ops | Observability | G1 · Health Sentinel | Watches Databricks jobs/pipelines; flags failures, retry spikes, slow runs | Triage narration, grounded (detection is deterministic) | design · Claude API | repo |
| Ops | Observability | G2 · SLA & Freshness Watchdog | Per-table freshness/SLA checks + a weekly reliability digest | Weekly digest narrative (checks are deterministic) | card | — |
| Ops | Capacity | G3 · Capacity Forecaster | Forecasts DBU demand & queueing; warns before saturation | Narrative brief over a classical forecast | card | — |
| Ops | Incident | G4 · Incident RCA Copilot | On failure, gathers read-only evidence, correlates, drafts an RCA + ServiceNow ticket | Agent loop over drill-down tools + verifier | design | repo |
| Ops | Knowledge | G5 · Runbook Agent | Grounded Q&A over runbooks & past incidents to cut MTTR | RAG answer generation (cited) | card | — |
| Ops | FinOps | G6 · Cost & Right-Sizing Advisor | Attributes DBU cost, recommends cluster right-sizing as a proposal | Right-sizing recommendation (human approves) | card | — |
| Ops | FinOps | G7 · Waste & Idle Guardrails | Auto-kill idle clusters, enforce policies & budgets | None — pure deterministic automation | card | — |
| Ops | Data quality | G8 · DQ & Anomaly Sentinel | DQ rules + anomaly triage; proposes new rules from patterns | Anomaly triage & rule suggestions | card | — |
| Ops | Change safety | G9 · Change-Risk Reviewer | CI review of PRs to notebooks/DLT/dbt; gates risky merges | Risk review (schema break, cost, tests) | card | — |
| Ops | Governance | G10 · UC Access Auditor | Scans Unity Catalog grants/lineage/PII; flags least-privilege violations | Audit-ready summaries & violation flags | card | — |
| Ops | Exec brief | G11 · Weekly Reliability Brief | One exec read-out across reliability, cost, DQ, incidents, access | Multi-source synthesis (metrics are deterministic) | card | — |
🟣
Cert note: the AI-part column is deliberately narrow — most Ops value is
deterministic, with
Claude at the edges (grounded narration, correlation, RAG). G7 uses
no AI at all. Runnable code for G1 & G4 lives
in
databricks-ops-copilot.
LEARNING PATH Build them in this order
flowchart LR
subgraph T1["Tier 1 · Foundations"]
P1["📄 Docs Q&A (RAG)"]
P2["📝 Notes → Actions"]
P3["🏷️ Classifier at scale"]
end
subgraph T2["Tier 2 · Agentic core"]
P4["🎧 Support Agent"]
P5["🗄️ NL→SQL Copilot"]
P6["🧾 Doc Extraction"]
end
subgraph T3["Tier 3 · Advanced"]
P7["🔬 Research Agent"]
P8["💻 Coding Agent"]
P9["📬 Ops Automation"]
P10["📊 Eval Harness"]
end
T1 --> T2 --> T3
classDef a fill:#e6fcf0,stroke:#2b8a3e,color:#155724;
classDef b fill:#fff4e0,stroke:#b5691a,color:#7a4a0a;
classDef c fill:#fdeaf0,stroke:#b0296b,color:#7a1f4a;
class P1,P2,P3 a; class P4,P5,P6 b; class P7,P8,P9,P10 c;
Tier 1 teaches the stack with no autonomy; Tier 2 introduces the agentic loop & tools; Tier 3 is multi-agent & long-horizon.
Tier 1 · Foundations
start here · no autonomy
Deterministic pipelines — one/few LLM calls, no agent loop. You learn RAG, structured outputs, evals, and cost without the complexity of autonomy.
P1 · 📄 Docs Q&A Assistant (RAG) ★ recommended first
Answer questions over a company's own docs/knowledge base, grounded & cited — the "hello world" of production AI, and the base of half the apps out there.
RAGembeddingsvector storechunkingcitationsgrounding
Teaches: the whole retrieval stack · Maps to D5, D3
⚠ Prod failures: stale index · bad chunking · low retrieval recall · confident hallucination · cost of re-embedding
P2 · 📝 Meeting Notes → Action Items
Turn a transcript into a structured summary + owned action items with due dates — a pure workflow, no agent needed.
structured outputsworkflow vs agentbatch APIevals
Teaches: when NOT to use an agent · Maps to D3, D1, D5
⚠ Prod failures: schema drift · long transcripts overflow context · silent extraction misses
P3 · 🏷️ Content Classifier at Scale
Classify/moderate a high volume of items (tickets, posts, reviews) cheaply and reliably.
model-tier choicebatchrate limitsevalsdrift
Teaches: cost/throughput engineering · Maps to D5
⚠ Prod failures: throughput vs rate limits · label drift · prompt-injection in user content · eval rot
Tier 2 · Agentic core
the agent loop + tools
Now the model decides and acts: tool use, structured actions, error-recovery loops, and human-in-the-loop for anything risky.
P4 · 🎧 Customer Support Agent ★ recommended 2nd
Resolve support requests: look up an order, check status, issue a refund — with human approval on the irreversible bits.
agentic looptool designhuman-in-the-loopreversibilityguardrailsescalation
Teaches: the core agent + safe actions · Maps to D1, D4
⚠ Prod failures: wrong refund issued · tool 400s break the loop · prompt injection triggers an action · runaway retries
P5 · 🗄️ NL→SQL Analytics Copilot
Turn "how many EU orders last quarter?" into SQL over a warehouse, run it, and answer — read-only and safe.
tool useschema in contextself-correctionguardrailsstructured outputs
Teaches: tool loops + error recovery · Maps to D4, D1, D5
⚠ Prod failures: wrong joins/hallucinated columns · runaway/expensive queries · SQL injection · retry storms on errors
P6 · 🧾 Document Extraction Pipeline
Extract structured fields from invoices/forms (PDF/images) into a validated schema, at scale.
multimodalstructured outputsvalidationbatchevals
Teaches: multimodal + strict schemas · Maps to D3, D5, D4
⚠ Prod failures: format drift breaks extraction · low-confidence fields · silent wrong values · cost per page
Tier 3 · Advanced & multi-agent
orchestration · long horizon
Multiple agents, long-running tasks, and real autonomy. Maximum learning, maximum failure surface.
P7 · 🔬 Multi-Source Research Agent ★ recommended 3rd
Answer a deep question by planning, searching many sources in parallel, verifying claims, and synthesizing a cited report.
multi-agentcoordinatorparallel fan-outresult mergingadversarial verify
Teaches: orchestration patterns end-to-end · Maps to D1, D4, D5
⚠ Prod failures: cost blowup from fan-out · hallucinated citations · sub-agent disagreement · no convergence
P8 · 💻 Autonomous Coding Agent
Given an issue, edit files, run tests, and iterate until green — a mini Claude-Code.
full agent looploop terminationpermission gatesMCPreliability
Teaches: long-horizon autonomy safely · Maps to D1, D2, D4
⚠ Prod failures: runaway loops · destructive edits · context exhaustion · flaky-test thrash · cost
P9 · 📬 Agentic Ops Automation
Triage incoming emails/alerts → classify → route → draft a reply/action, with humans on the loop (via MCP tools).
MCP integrationorchestrationHITLguardrailsmemory
Teaches: wiring real tools via MCP · Maps to D4, D1, D2
⚠ Prod failures: acting on injected instructions · over-broad tool scope · confused-deputy · missed escalations
P10 · 📊 Eval & Observability Harness
The meta-project: an LLM-as-judge eval suite + tracing/metrics that every other project needs.
evalsLLM-as-judgetracingSLOsregression
Teaches: how you know it works · Maps to D5
⚠ Prod failures: judge bias/same-model · flaky evals · no drift detection · unmeasured cost
🏭 Ops Innovation Portfolio · AI + Automation for Azure Databricks Operations
for an Ops Director · GxP-aware
💻
Runnable code: the portfolio has a companion repo —
github.com/basantchoudhary/databricks-ops-copilot
— with
G1 &
G4 implemented, mock-data-first (clone & run offline, zero installs), read-only, and Claude
wired through Databricks Model Serving. Design pages below; code in the repo.
A dedicated track for making a Databricks operations function more robust, reliable, and self-observing
— the kind of platform-Ops org that owns pipelines, clusters, cost, data quality, and incident response. It's built for a
regulated (pharma / GxP) environment, so the guiding rule is: deterministic where correctness must be
guaranteed, AI where judgment and language help — never the other way round.
🧭 Operating principles (read these first — they shape every project below)
- Read-only first. Every agent starts as an observer & advisor. It earns write/act
permissions only after it's been trusted for weeks — climb the autonomy ladder, don't skip it.
- Human-in-the-loop for any change. AI proposes; a human approves; a
deterministic tool executes. No agent mutates prod, cost, or access on its own.
- Deterministic guardrails, not prompts, for the rules that must hold. Idle-cluster kill, budget
caps, access policy — these are code/policy-as-code, never "please remember to…" in a prompt.
- Everything is audited & reproducible (GxP). Every recommendation and action logs its inputs,
evidence, model version, and the approver — audit-ready by design.
- Mock-data-first. Build and demo against synthetic system-table / job-run data, so nothing
touches real ABCCompany infrastructure until it's validated and signed off.
The autonomy ladder — where each project should sit
flowchart LR
O["1 · OBSERVE
collect signals
(read-only)"]:::a --> E["2 · EXPLAIN
plain-English RCA
& digests"]:::a
E --> R["3 · RECOMMEND
propose a fix
+ evidence"]:::b
R --> A["4 · APPROVE
human signs off
(HITL gate)"]:::b
A --> X["5 · ACT
deterministic tool
executes + logs"]:::c
classDef a fill:#e6fcf0,stroke:#2b8a3e,color:#155724;
classDef b fill:#fff4e0,stroke:#b5691a,color:#7a4a0a;
classDef c fill:#fdeaf0,stroke:#b0296b,color:#7a1f4a;
Start every capability at stage 1–2. Most value (and safety) is
in Observe → Explain → Recommend; only mature, low-risk actions graduate to stage 5.
Deterministic vs AI — how to decide what to build
| If the task is… | Build it as… | Why |
| A rule that must always hold (kill idle clusters > 30 min, block untagged jobs, enforce a budget cap) | Deterministic automation | Guaranteed, auditable, cheap, no hallucination surface. An LLM here only adds risk. |
| Structured detection with clear thresholds (freshness SLA breach, failure-rate spike, cost anomaly) | Deterministic + AI summary | Detection is math; AI only writes the human-readable narrative on top. |
| Correlating messy signals & explaining why (root-cause across logs, config changes, metrics) | AI agent (read-only tools) | Judgment + language over heterogeneous evidence — exactly where an agent earns its keep. |
| Answering "how do we handle X?" from runbooks & past incidents | RAG | Grounded, cited retrieval — no autonomy needed, low risk, fast MTTR win. |
| A recommendation a human will apply (right-size a cluster, add a DQ rule, tighten a grant) | Hybrid (AI proposes → human approves → deterministic applies) | AI's reasoning + a human gate + a deterministic executor = safe change management. |
🩺 Theme A · Observability & Reliability — know before it breaks
Continuously watch Databricks Jobs, DLT pipelines & streaming from the system tables + Jobs API. Deterministic core detects failures/latency/retry spikes; an agent clusters recurring failures, explains each in plain English, and points to the likely cause & runbook.
system.lakeflow job runsSpark metricsfailure clusteringRCA narrativeread-only
⚠ Watch: alert fatigue / noisy signals · hallucinated causes → always cite the evidence row · flapping jobs
G2 · SLA & Data-Freshness Watchdog Deterministic + AI digest
Declarative SLAs per Unity Catalog table ("gold.sales refreshed by 06:00, < 2h lag"). Deterministic checks fire on breach; a weekly AI reliability digest summarizes what slipped, trends, and the worst offenders for the Ops review.
freshness/lag SLOspolicy-as-codeweekly digesttrend narrative
Autonomy: Observe → Explain · Detection is pure code
⚠ Watch: SLA definitions drifting from reality · timezone/holiday calendars · digest that's ignored if too long
G3 · Capacity & Saturation Forecaster Deterministic ML + AI narrative
Forecast job durations, warehouse queueing, and DBU demand from historical run data; warn before saturation (month-end loads, study data spikes). Classical forecasting; AI writes the "here's what's coming and why" brief.
time-series forecastqueue depthwhat-ifearly warning
Autonomy: Observe → Recommend
⚠ Watch: over-trusting the forecast · regime changes after a platform migration · seasonality misread
🚨 Theme B · Incident Response & Knowledge — cut MTTR
G4 · Incident RCA Copilot AI Agent · read-only tools ✅ deep-dive ready
On a job/pipeline failure, the agent gathers evidence with deterministic read-only tools — error logs, Spark UI metrics, the failing query plan, recent notebook/config/permission changes — correlates them, and drafts a structured RCA + a pre-filled ServiceNow ticket. A human reviews before anything is filed or actioned.
evidence-gathering toolscorrelationServiceNow draftHITLaudit trail
⚠ Watch: confident-but-wrong RCA (silent failure) → verifier pass + cite every fact · over-broad tool scope · injected log content
G5 · Runbook & Tribal-Knowledge Agent RAG
Grounded Q&A over runbooks, past incident RCAs, architecture decisions, and platform docs — "how do we recover a failed DLT checkpoint?" answered with citations. Turns tribal knowledge into an always-on on-call assistant and slashes MTTR for known issues.
retrievalcitationsgroundingon-call assist
Autonomy: Explain only · zero write access · Maps to D5
⚠ Watch: stale runbooks → freshness policy · confident hallucination without a source · access-scoping the corpus
💰 Theme C · Cost / FinOps — DBU discipline
G6 · DBU Cost & Cluster Right-Sizing Advisor Hybrid
Deterministic cost attribution from system.billing.usage (by workspace, job, team, tag); an agent then recommends right-sizing — autoscaling bounds, Photon, spot/on-demand mix, serverless vs classic, SQL-warehouse sizing — as a change proposal with projected savings. A human applies it via a cluster-policy PR.
system.billing.usagecost attributionright-sizingsavings estimateapproval PR
Autonomy: Recommend → Approve → Act · Maps to D5
⚠ Watch: recommendation that hurts SLA to save cost · unattributed spend (untagged) · savings claims not verified after
G7 · Waste & Idle Guardrails Deterministic automation
Pure policy-as-code — auto-terminate idle/orphaned clusters, enforce cluster policies & mandatory cost tags, block oversized all-purpose clusters, and fire budget alerts. No AI at all: this is the textbook case where determinism is safer, cheaper, and audit-clean.
cluster policiesauto-terminationbudget alertstag enforcement
Autonomy: Act (bounded, rule-based) · teaches "not everything needs an agent"
⚠ Watch: killing a legitimately long job · policy too strict blocks teams · needs a documented exception path
🧪 Theme D · Data Quality & Change Safety — trust the data & the deploys
G8 · Data-Quality & Anomaly Sentinel Hybrid
Deterministic DQ rules (row counts, nulls, schema, referential integrity) via Lakehouse Monitoring / Great Expectations / DQX; an agent triages anomalies, explains likely drift causes, and proposes new rules from recurring patterns for a human to accept.
Lakehouse MonitoringDQX / GEdrift triagerule suggestions
Autonomy: Observe → Recommend
⚠ Watch: rule sprawl / noisy DQ · schema-valid-but-wrong values slipping through · alerting on expected variance
G9 · Pipeline Change-Risk Reviewer AI Agent · CI
Runs in Azure DevOps / GitHub CI on every PR to notebooks, DLT, dbt or SQL. Deterministic linters catch the mechanical issues; an agent reviews for breaking schema changes, missing tests, cost regressions, and non-idempotent writes, and posts a structured review. Risky changes gate the merge.
CI reviewschema-break detectioncost regressionmerge gateheadless
Autonomy: Recommend (blocks merge) · Maps to D2/D3
⚠ Watch: false-positive blocks eroding trust · reviewing generated/vendored code · secrets in diffs
🛡️ Theme E · Governance & Compliance — GxP & least privilege
G10 · Unity Catalog Governance & Access Auditor Hybrid · read-only
Deterministic scans of Unity Catalog grants, system.access.audit, lineage, external locations and PII tags; an agent produces audit-ready summaries, flags least-privilege violations & dormant access, and drafts the remediation for a data owner to approve. Evidence packaged for GxP review.
system.access.auditgrant reviewleast privilegelineage/PIIGxP evidence
Autonomy: Observe → Recommend · never changes a grant itself
⚠ Watch: over-revoking breaks a pipeline · audit summary must be verifiable, not just plausible · PII in the agent's own logs
G11 · Weekly Ops Reliability Brief AI over deterministic metrics
The capstone read-out: an agent assembles one exec-ready brief from all the above — reliability, cost, DQ, incidents, access posture — with trends, top risks, and recommended focus for the week. Deterministic metrics in; a crisp narrative out, for the Ops leadership review.
exec digestmulti-source synthesistrendsprioritization
Autonomy: Explain · Maps to D1 (synthesis)
⚠ Watch: numbers in the narrative not matching the source of truth → compute deterministically, let AI only phrase
🎯 Where she should start (lowest risk, fastest value):
G5 · Runbook Agent (pure RAG, zero autonomy, instant MTTR win) + G1 · Health Sentinel (read-only observe/explain).
Together they prove the pattern — grounded answers & safe observation — with no ability to touch production. From there,
climb the ladder: G6 cost recommendations, then G7 deterministic guardrails, then the GxP auditor G10.
Reference stack (all mock-data-first): Databricks system tables (billing/usage, access audit, job runs,
query history) + Jobs/REST API & SDK for read-only signals · Unity Catalog for governance/lineage ·
Lakehouse Monitoring / DQX for data quality · ServiceNow for ITSM tickets · optionally Sedai for autonomous
cost/perf optimization · Azure DevOps for CI · the Claude API (native tool use, adaptive thinking, structured
outputs) for every agent. Nothing writes to real infra until validated and signed off.
👉 Tell me which of G1–G11 to design first — I'll produce
the full deep-dive (architecture, deterministic-vs-AI split, read-only tool contracts, guardrails, GxP audit trail, mock-data
harness, and a runnable skeleton), same anatomy as the projects above.
DEEP-DIVE ANATOMY What we'll produce for the project you pick
Each chosen project gets a full, consistent treatment — not just code, but the thinking a real AI engineer does:
| # | Section | What's in it |
| 1 | The problem & why it matters | The real-world need, who has it, what "good" looks like |
| 2 | Requirements & success criteria | Functional + non-functional (latency, cost, accuracy), measurable |
| 3 | Architecture design | Component diagram, data flow, the key boxes and why |
| 4 | Design decisions & trade-offs | Model choice · RAG vs fine-tune · workflow vs agent · sync vs async — with the reasoning |
| 5 | Minimal implementation | A skeleton (pseudocode/real) that actually captures the core loop — small but honest |
| 6 | Cross-cutting concerns | Security & prompt injection · cost model · latency · observability · evals · guardrails |
| 7 | Production failure points | The catalog of what breaks in prod for this project — and the mitigation for each |
| 8 | Concepts → CCA map | Exactly which exam concepts this project makes concrete |
| 9 | Level it up | Extensions that turn the toy into something real |
MY RECOMMENDATION The 3-project spine
If you do just three, do these — together they cover the entire spectrum from "no autonomy" to "multi-agent," and every
cross-cutting concern shows up at least once:
1. 📄 P1 · Docs Q&A (RAG) — learn the retrieval stack & grounding (no autonomy).
2. 🎧 P4 · Support Agent — your first real agent: tools, human-in-the-loop, safe actions.
3. 🔬 P7 · Research Agent — multi-agent orchestration, parallelism, verification.
Then add 📊 P10 · Eval Harness as a companion — because in production, "how do you know it works?" is the question that never goes away.
👉 Which one should we design first? Tell me a project number (P1–P10) — I'll build the full deep-dive for it, then we move down the spine.