Home › AI Engineering Projects

🏗️ AI Engineering Projects — Catalog

Real problems, chosen to teach the concepts hands-on · pick one, then we design it end-to-end
✅ Deep-dive ready → P0 · Self-Correcting Text-to-SQL Agent
The smallest pure agent, designed end-to-end: conceptual → physical (across Local/Databricks/AWS/Azure/GCP) → cross-cutting (cost, security, observability, logging) → open-source toolbox → a $0 local build you actually run.
🟣 …and building it with the Claude API (native tool use) — the cert-prep version: the stop_reason loop, the tool_use/tool_result contract, prompt caching & adaptive thinking, each line mapped to a CCA domain. 🧱 …scaled to Databricks Unity Catalog (production) — grounding against thousands of tables, schema context & sample values, EXPLAIN validation, evals, cost/reliability/scale/observability, plus CLAUDE.md & Skills — all five CCA domains in one project. 🧠 Design ready → Second-Brain Assistant (tiered memory) — a toy that forces every memory store (Redis · SQLite · vector · graph) and context technique (routing, fusion, caching, context editing, compaction) to earn its place. Architecture, per-store memory design, failure points & cross-cutting. 🏭 Real-world track → Ops Innovation for Azure Databricks (G1–G11) — AI agents + deterministic automation to make a regulated Databricks Ops function robust & self-observing: health/RCA agents, FinOps, data quality, GxP governance — each placed on a read-only→approve→act autonomy ladder. 🔌 Learn MCP → Lakehouse Ops MCP Server (every aspect of MCP) — one tiny FastMCP server exposing all three primitives (tools · resources · prompts), both transports (stdio · streamable HTTP), and sampling/elicitation/roots + MCP security. Your D4 rehearsal; also answers which portfolio projects use MCP. 🏗️ Understand the API → MyOwnAnthropicServer (the Messages API, minimally) — design your own POST /v1/messages shell to see every server responsibility (auth · limits · validation · prompt assembly · caching · safety · usage · streaming) and how the LLM call is just one stage in the middle. 🧰 The caller's twin → GenericAnthropicMessageClient (design-patterns study) — one robust client that handles every case (6 stop reasons · errors · retries · rate limits · token-fit · context bloat · streaming · batch) via Facade + middleware pipeline + Strategy interfaces. How Tool Runner fits.

Reading about agentic architecture only gets you so far. This is a curated shortlist of real problems — each deliberately chosen because it forces you to learn a distinct set of AI-engineering and agentic concepts in practice. We'll pick one, then produce a full deep-dive for it (see what each deep-dive contains).

🎯 Selection criteria — every project here is: (1) a genuine problem people pay to solve, (2) buildable as a minimal but real implementation, (3) rich in production failure modes worth studying, and (4) mapped to concepts in the CCA material and Gen AI Fundamentals.

AT A GLANCE Every project — one crisp table

Design = the architecture deep-dive · Code = runnable implementation. (P1–P10 are the learning-path idea cards further down; the rows below are the built-out projects.)

DomainThemeProjectWhat it doesAI partDesignCode
Agentic coreNL→SQLP0 · Text-to-SQL AgentEnglish → SQL, runs it, reads the DB error, fixes & retries until it worksThe self-correcting agent loop (writes/repairs SQL)design · Claude API · UCrepo
AdvancedTiered memorySecond-Brain AssistantPersonal work assistant over Redis + SQLite + vector + graph memoryMemory routing, retrieval fusion & all reasoningdesign—
IntegrationMCPLakehouse Ops MCP ServerOne FastMCP server exposing every MCP aspect (tools · resources · prompts · transports · sampling/elicitation)Host's model consumes the server's tools/resourcesdesignrepo
FoundationsAPI internalsMyOwnAnthropicServerA minimal POST /v1/messages shell to learn every server responsibilityThe LLM call is one stage; the rest is the serverdesign—
FoundationsDesign patternsGenericAnthropicMessageClientOne robust client for every request/response case via Facade + middleware + StrategyWraps the SDK/model calls; policies are pluggabledesign—
FoundationsArchitectureCross-Cutting ConcernsLogging/tracing/cost/redaction/audit as pluggable ports (hexagonal + DI)Wraps every model call; adapters swap per environmentdesign—
OpsObservabilityG1 · Health SentinelWatches Databricks jobs/pipelines; flags failures, retry spikes, slow runsTriage narration, grounded (detection is deterministic)design · Claude APIrepo
OpsObservabilityG2 · SLA & Freshness WatchdogPer-table freshness/SLA checks + a weekly reliability digestWeekly digest narrative (checks are deterministic)card—
OpsCapacityG3 · Capacity ForecasterForecasts DBU demand & queueing; warns before saturationNarrative brief over a classical forecastcard—
OpsIncidentG4 · Incident RCA CopilotOn failure, gathers read-only evidence, correlates, drafts an RCA + ServiceNow ticketAgent loop over drill-down tools + verifierdesignrepo
OpsKnowledgeG5 · Runbook AgentGrounded Q&A over runbooks & past incidents to cut MTTRRAG answer generation (cited)card—
OpsFinOpsG6 · Cost & Right-Sizing AdvisorAttributes DBU cost, recommends cluster right-sizing as a proposalRight-sizing recommendation (human approves)card—
OpsFinOpsG7 · Waste & Idle GuardrailsAuto-kill idle clusters, enforce policies & budgetsNone — pure deterministic automationcard—
OpsData qualityG8 · DQ & Anomaly SentinelDQ rules + anomaly triage; proposes new rules from patternsAnomaly triage & rule suggestionscard—
OpsChange safetyG9 · Change-Risk ReviewerCI review of PRs to notebooks/DLT/dbt; gates risky mergesRisk review (schema break, cost, tests)card—
OpsGovernanceG10 · UC Access AuditorScans Unity Catalog grants/lineage/PII; flags least-privilege violationsAudit-ready summaries & violation flagscard—
OpsExec briefG11 · Weekly Reliability BriefOne exec read-out across reliability, cost, DQ, incidents, accessMulti-source synthesis (metrics are deterministic)card—
🟣 Cert note: the AI-part column is deliberately narrow — most Ops value is deterministic, with Claude at the edges (grounded narration, correlation, RAG). G7 uses no AI at all. Runnable code for G1 & G4 lives in databricks-ops-copilot.

LEARNING PATH Build them in this order

flowchart LR subgraph T1["Tier 1 · Foundations"] P1["📄 Docs Q&A (RAG)"] P2["📝 Notes → Actions"] P3["🏷️ Classifier at scale"] end subgraph T2["Tier 2 · Agentic core"] P4["🎧 Support Agent"] P5["🗄️ NL→SQL Copilot"] P6["🧾 Doc Extraction"] end subgraph T3["Tier 3 · Advanced"] P7["🔬 Research Agent"] P8["💻 Coding Agent"] P9["📬 Ops Automation"] P10["📊 Eval Harness"] end T1 --> T2 --> T3 classDef a fill:#e6fcf0,stroke:#2b8a3e,color:#155724; classDef b fill:#fff4e0,stroke:#b5691a,color:#7a4a0a; classDef c fill:#fdeaf0,stroke:#b0296b,color:#7a1f4a; class P1,P2,P3 a; class P4,P5,P6 b; class P7,P8,P9,P10 c;

Tier 1 teaches the stack with no autonomy; Tier 2 introduces the agentic loop & tools; Tier 3 is multi-agent & long-horizon.

Tier 1 · Foundations

start here · no autonomy

Deterministic pipelines — one/few LLM calls, no agent loop. You learn RAG, structured outputs, evals, and cost without the complexity of autonomy.

P1 · 📄 Docs Q&A Assistant (RAG) ★ recommended first

Answer questions over a company's own docs/knowledge base, grounded & cited — the "hello world" of production AI, and the base of half the apps out there.
RAGembeddingsvector storechunkingcitationsgrounding
Teaches: the whole retrieval stack · Maps to D5, D3
⚠ Prod failures: stale index · bad chunking · low retrieval recall · confident hallucination · cost of re-embedding

P2 · 📝 Meeting Notes → Action Items

Turn a transcript into a structured summary + owned action items with due dates — a pure workflow, no agent needed.
structured outputsworkflow vs agentbatch APIevals
Teaches: when NOT to use an agent · Maps to D3, D1, D5
⚠ Prod failures: schema drift · long transcripts overflow context · silent extraction misses

P3 · 🏷️ Content Classifier at Scale

Classify/moderate a high volume of items (tickets, posts, reviews) cheaply and reliably.
model-tier choicebatchrate limitsevalsdrift
Teaches: cost/throughput engineering · Maps to D5
⚠ Prod failures: throughput vs rate limits · label drift · prompt-injection in user content · eval rot

Tier 2 · Agentic core

the agent loop + tools

Now the model decides and acts: tool use, structured actions, error-recovery loops, and human-in-the-loop for anything risky.

P4 · 🎧 Customer Support Agent ★ recommended 2nd

Resolve support requests: look up an order, check status, issue a refund — with human approval on the irreversible bits.
agentic looptool designhuman-in-the-loopreversibilityguardrailsescalation
Teaches: the core agent + safe actions · Maps to D1, D4
⚠ Prod failures: wrong refund issued · tool 400s break the loop · prompt injection triggers an action · runaway retries

P5 · 🗄️ NL→SQL Analytics Copilot

Turn "how many EU orders last quarter?" into SQL over a warehouse, run it, and answer — read-only and safe.
tool useschema in contextself-correctionguardrailsstructured outputs
Teaches: tool loops + error recovery · Maps to D4, D1, D5
⚠ Prod failures: wrong joins/hallucinated columns · runaway/expensive queries · SQL injection · retry storms on errors

P6 · 🧾 Document Extraction Pipeline

Extract structured fields from invoices/forms (PDF/images) into a validated schema, at scale.
multimodalstructured outputsvalidationbatchevals
Teaches: multimodal + strict schemas · Maps to D3, D5, D4
⚠ Prod failures: format drift breaks extraction · low-confidence fields · silent wrong values · cost per page

Tier 3 · Advanced & multi-agent

orchestration · long horizon

Multiple agents, long-running tasks, and real autonomy. Maximum learning, maximum failure surface.

P7 · 🔬 Multi-Source Research Agent ★ recommended 3rd

Answer a deep question by planning, searching many sources in parallel, verifying claims, and synthesizing a cited report.
multi-agentcoordinatorparallel fan-outresult mergingadversarial verify
Teaches: orchestration patterns end-to-end · Maps to D1, D4, D5
⚠ Prod failures: cost blowup from fan-out · hallucinated citations · sub-agent disagreement · no convergence

P8 · 💻 Autonomous Coding Agent

Given an issue, edit files, run tests, and iterate until green — a mini Claude-Code.
full agent looploop terminationpermission gatesMCPreliability
Teaches: long-horizon autonomy safely · Maps to D1, D2, D4
⚠ Prod failures: runaway loops · destructive edits · context exhaustion · flaky-test thrash · cost

P9 · 📬 Agentic Ops Automation

Triage incoming emails/alerts → classify → route → draft a reply/action, with humans on the loop (via MCP tools).
MCP integrationorchestrationHITLguardrailsmemory
Teaches: wiring real tools via MCP · Maps to D4, D1, D2
⚠ Prod failures: acting on injected instructions · over-broad tool scope · confused-deputy · missed escalations

P10 · 📊 Eval & Observability Harness

The meta-project: an LLM-as-judge eval suite + tracing/metrics that every other project needs.
evalsLLM-as-judgetracingSLOsregression
Teaches: how you know it works · Maps to D5
⚠ Prod failures: judge bias/same-model · flaky evals · no drift detection · unmeasured cost

🏭 Ops Innovation Portfolio · AI + Automation for Azure Databricks Operations

for an Ops Director · GxP-aware
💻 Runnable code: the portfolio has a companion repo — github.com/basantchoudhary/databricks-ops-copilot — with G1 & G4 implemented, mock-data-first (clone & run offline, zero installs), read-only, and Claude wired through Databricks Model Serving. Design pages below; code in the repo.

A dedicated track for making a Databricks operations function more robust, reliable, and self-observing — the kind of platform-Ops org that owns pipelines, clusters, cost, data quality, and incident response. It's built for a regulated (pharma / GxP) environment, so the guiding rule is: deterministic where correctness must be guaranteed, AI where judgment and language help — never the other way round.

🧭 Operating principles (read these first — they shape every project below)
  • Read-only first. Every agent starts as an observer & advisor. It earns write/act permissions only after it's been trusted for weeks — climb the autonomy ladder, don't skip it.
  • Human-in-the-loop for any change. AI proposes; a human approves; a deterministic tool executes. No agent mutates prod, cost, or access on its own.
  • Deterministic guardrails, not prompts, for the rules that must hold. Idle-cluster kill, budget caps, access policy — these are code/policy-as-code, never "please remember to…" in a prompt.
  • Everything is audited & reproducible (GxP). Every recommendation and action logs its inputs, evidence, model version, and the approver — audit-ready by design.
  • Mock-data-first. Build and demo against synthetic system-table / job-run data, so nothing touches real ABCCompany infrastructure until it's validated and signed off.

The autonomy ladder — where each project should sit

flowchart LR O["1 · OBSERVE
collect signals
(read-only)"]:::a --> E["2 · EXPLAIN
plain-English RCA
& digests"]:::a E --> R["3 · RECOMMEND
propose a fix
+ evidence"]:::b R --> A["4 · APPROVE
human signs off
(HITL gate)"]:::b A --> X["5 · ACT
deterministic tool
executes + logs"]:::c classDef a fill:#e6fcf0,stroke:#2b8a3e,color:#155724; classDef b fill:#fff4e0,stroke:#b5691a,color:#7a4a0a; classDef c fill:#fdeaf0,stroke:#b0296b,color:#7a1f4a;

Start every capability at stage 1–2. Most value (and safety) is in Observe → Explain → Recommend; only mature, low-risk actions graduate to stage 5.

Deterministic vs AI — how to decide what to build

If the task is…Build it as…Why
A rule that must always hold (kill idle clusters > 30 min, block untagged jobs, enforce a budget cap)Deterministic automationGuaranteed, auditable, cheap, no hallucination surface. An LLM here only adds risk.
Structured detection with clear thresholds (freshness SLA breach, failure-rate spike, cost anomaly)Deterministic + AI summaryDetection is math; AI only writes the human-readable narrative on top.
Correlating messy signals & explaining why (root-cause across logs, config changes, metrics)AI agent (read-only tools)Judgment + language over heterogeneous evidence — exactly where an agent earns its keep.
Answering "how do we handle X?" from runbooks & past incidentsRAGGrounded, cited retrieval — no autonomy needed, low risk, fast MTTR win.
A recommendation a human will apply (right-size a cluster, add a DQ rule, tighten a grant)Hybrid (AI proposes → human approves → deterministic applies)AI's reasoning + a human gate + a deterministic executor = safe change management.

🩺 Theme A · Observability & Reliability — know before it breaks

G1 · Pipeline & Job Health Sentinel AI Agent · read-only ✅ deep-dive ready

Continuously watch Databricks Jobs, DLT pipelines & streaming from the system tables + Jobs API. Deterministic core detects failures/latency/retry spikes; an agent clusters recurring failures, explains each in plain English, and points to the likely cause & runbook.
system.lakeflow job runsSpark metricsfailure clusteringRCA narrativeread-only
Autonomy: Observe → Explain · Deterministic detection + AI triage · design → · 🟣 Claude API code →
⚠ Watch: alert fatigue / noisy signals · hallucinated causes → always cite the evidence row · flapping jobs

G2 · SLA & Data-Freshness Watchdog Deterministic + AI digest

Declarative SLAs per Unity Catalog table ("gold.sales refreshed by 06:00, < 2h lag"). Deterministic checks fire on breach; a weekly AI reliability digest summarizes what slipped, trends, and the worst offenders for the Ops review.
freshness/lag SLOspolicy-as-codeweekly digesttrend narrative
Autonomy: Observe → Explain · Detection is pure code
⚠ Watch: SLA definitions drifting from reality · timezone/holiday calendars · digest that's ignored if too long

G3 · Capacity & Saturation Forecaster Deterministic ML + AI narrative

Forecast job durations, warehouse queueing, and DBU demand from historical run data; warn before saturation (month-end loads, study data spikes). Classical forecasting; AI writes the "here's what's coming and why" brief.
time-series forecastqueue depthwhat-ifearly warning
Autonomy: Observe → Recommend
⚠ Watch: over-trusting the forecast · regime changes after a platform migration · seasonality misread

🚨 Theme B · Incident Response & Knowledge — cut MTTR

G4 · Incident RCA Copilot AI Agent · read-only tools ✅ deep-dive ready

On a job/pipeline failure, the agent gathers evidence with deterministic read-only tools — error logs, Spark UI metrics, the failing query plan, recent notebook/config/permission changes — correlates them, and drafts a structured RCA + a pre-filled ServiceNow ticket. A human reviews before anything is filed or actioned.
evidence-gathering toolscorrelationServiceNow draftHITLaudit trail
Autonomy: Explain → Recommend · never auto-resolves · Maps to D1/D2 · open deep-dive →
⚠ Watch: confident-but-wrong RCA (silent failure) → verifier pass + cite every fact · over-broad tool scope · injected log content

G5 · Runbook & Tribal-Knowledge Agent RAG

Grounded Q&A over runbooks, past incident RCAs, architecture decisions, and platform docs — "how do we recover a failed DLT checkpoint?" answered with citations. Turns tribal knowledge into an always-on on-call assistant and slashes MTTR for known issues.
retrievalcitationsgroundingon-call assist
Autonomy: Explain only · zero write access · Maps to D5
⚠ Watch: stale runbooks → freshness policy · confident hallucination without a source · access-scoping the corpus

💰 Theme C · Cost / FinOps — DBU discipline

G6 · DBU Cost & Cluster Right-Sizing Advisor Hybrid

Deterministic cost attribution from system.billing.usage (by workspace, job, team, tag); an agent then recommends right-sizing — autoscaling bounds, Photon, spot/on-demand mix, serverless vs classic, SQL-warehouse sizing — as a change proposal with projected savings. A human applies it via a cluster-policy PR.
system.billing.usagecost attributionright-sizingsavings estimateapproval PR
Autonomy: Recommend → Approve → Act · Maps to D5
⚠ Watch: recommendation that hurts SLA to save cost · unattributed spend (untagged) · savings claims not verified after

G7 · Waste & Idle Guardrails Deterministic automation

Pure policy-as-code — auto-terminate idle/orphaned clusters, enforce cluster policies & mandatory cost tags, block oversized all-purpose clusters, and fire budget alerts. No AI at all: this is the textbook case where determinism is safer, cheaper, and audit-clean.
cluster policiesauto-terminationbudget alertstag enforcement
Autonomy: Act (bounded, rule-based) · teaches "not everything needs an agent"
⚠ Watch: killing a legitimately long job · policy too strict blocks teams · needs a documented exception path

🧪 Theme D · Data Quality & Change Safety — trust the data & the deploys

G8 · Data-Quality & Anomaly Sentinel Hybrid

Deterministic DQ rules (row counts, nulls, schema, referential integrity) via Lakehouse Monitoring / Great Expectations / DQX; an agent triages anomalies, explains likely drift causes, and proposes new rules from recurring patterns for a human to accept.
Lakehouse MonitoringDQX / GEdrift triagerule suggestions
Autonomy: Observe → Recommend
⚠ Watch: rule sprawl / noisy DQ · schema-valid-but-wrong values slipping through · alerting on expected variance

G9 · Pipeline Change-Risk Reviewer AI Agent · CI

Runs in Azure DevOps / GitHub CI on every PR to notebooks, DLT, dbt or SQL. Deterministic linters catch the mechanical issues; an agent reviews for breaking schema changes, missing tests, cost regressions, and non-idempotent writes, and posts a structured review. Risky changes gate the merge.
CI reviewschema-break detectioncost regressionmerge gateheadless
Autonomy: Recommend (blocks merge) · Maps to D2/D3
⚠ Watch: false-positive blocks eroding trust · reviewing generated/vendored code · secrets in diffs

🛡️ Theme E · Governance & Compliance — GxP & least privilege

G10 · Unity Catalog Governance & Access Auditor Hybrid · read-only

Deterministic scans of Unity Catalog grants, system.access.audit, lineage, external locations and PII tags; an agent produces audit-ready summaries, flags least-privilege violations & dormant access, and drafts the remediation for a data owner to approve. Evidence packaged for GxP review.
system.access.auditgrant reviewleast privilegelineage/PIIGxP evidence
Autonomy: Observe → Recommend · never changes a grant itself
⚠ Watch: over-revoking breaks a pipeline · audit summary must be verifiable, not just plausible · PII in the agent's own logs

G11 · Weekly Ops Reliability Brief AI over deterministic metrics

The capstone read-out: an agent assembles one exec-ready brief from all the above — reliability, cost, DQ, incidents, access posture — with trends, top risks, and recommended focus for the week. Deterministic metrics in; a crisp narrative out, for the Ops leadership review.
exec digestmulti-source synthesistrendsprioritization
Autonomy: Explain · Maps to D1 (synthesis)
⚠ Watch: numbers in the narrative not matching the source of truth → compute deterministically, let AI only phrase
🎯 Where she should start (lowest risk, fastest value): G5 · Runbook Agent (pure RAG, zero autonomy, instant MTTR win) + G1 · Health Sentinel (read-only observe/explain). Together they prove the pattern — grounded answers & safe observation — with no ability to touch production. From there, climb the ladder: G6 cost recommendations, then G7 deterministic guardrails, then the GxP auditor G10.
Reference stack (all mock-data-first): Databricks system tables (billing/usage, access audit, job runs, query history) + Jobs/REST API & SDK for read-only signals · Unity Catalog for governance/lineage · Lakehouse Monitoring / DQX for data quality · ServiceNow for ITSM tickets · optionally Sedai for autonomous cost/perf optimization · Azure DevOps for CI · the Claude API (native tool use, adaptive thinking, structured outputs) for every agent. Nothing writes to real infra until validated and signed off.

👉 Tell me which of G1–G11 to design first — I'll produce the full deep-dive (architecture, deterministic-vs-AI split, read-only tool contracts, guardrails, GxP audit trail, mock-data harness, and a runnable skeleton), same anatomy as the projects above.

DEEP-DIVE ANATOMY What we'll produce for the project you pick

Each chosen project gets a full, consistent treatment — not just code, but the thinking a real AI engineer does:

#SectionWhat's in it
1The problem & why it mattersThe real-world need, who has it, what "good" looks like
2Requirements & success criteriaFunctional + non-functional (latency, cost, accuracy), measurable
3Architecture designComponent diagram, data flow, the key boxes and why
4Design decisions & trade-offsModel choice · RAG vs fine-tune · workflow vs agent · sync vs async — with the reasoning
5Minimal implementationA skeleton (pseudocode/real) that actually captures the core loop — small but honest
6Cross-cutting concernsSecurity & prompt injection · cost model · latency · observability · evals · guardrails
7Production failure pointsThe catalog of what breaks in prod for this project — and the mitigation for each
8Concepts → CCA mapExactly which exam concepts this project makes concrete
9Level it upExtensions that turn the toy into something real

MY RECOMMENDATION The 3-project spine

If you do just three, do these — together they cover the entire spectrum from "no autonomy" to "multi-agent," and every cross-cutting concern shows up at least once:

1. 📄 P1 · Docs Q&A (RAG) — learn the retrieval stack & grounding (no autonomy).
2. 🎧 P4 · Support Agent — your first real agent: tools, human-in-the-loop, safe actions.
3. 🔬 P7 · Research Agent — multi-agent orchestration, parallelism, verification.
Then add 📊 P10 · Eval Harness as a companion — because in production, "how do you know it works?" is the question that never goes away.

👉 Which one should we design first? Tell me a project number (P1–P10) — I'll build the full deep-dive for it, then we move down the spine.