Gen AI Fundamentals β€Ί AI Engineering Blueprint

πŸ—ΊοΈ AI Engineering Blueprint

Every conceptual requirement, and the physical stack that satisfies it β€” open source, Databricks, Azure, AWS, GCP
What this page is for. When you design an AI system, the hard part is not choosing a model β€” it is knowing what concerns exist at all, which ones apply at your scale, and what you would actually deploy for each. This is that checklist: conceptual requirement on the left, physical options across the platforms on the right. Use it as an architect's mind map and as a gap-finder in design reviews.

1 Β· Five kinds of engineering concept

These get used interchangeably and they are not the same thing. The clean way to tell them apart is to ask what unit you are designing, and which decision you are taking away from the model.

Discipline Unit of design You control Main levers Fails as Needed when
Prompt Engineering One message How you ask System prompt, role, few-shot examples, XML structure, chain-of-thought, prefill, output format Ambiguous instruction, wrong shape of answer, inconsistent tone Always β€” from the very first call
Context Engineering The context window What it sees Retrieval, chunking, ordering, compaction, summarisation, memory, cache breakpoints, token budget Context bloat, lost-in-the-middle, budget exhaustion, retrieved-but-irrelevant As soon as inputs exceed a hand-written prompt
Harness Engineering The runtime around the model What it can do Tool definitions, schema enforcement, validation, retries, circuit breakers, sandboxing, permissions, evals in CI Brittle integration, unbounded actions, no recovery path, silent failure The moment it touches production or real tools
Loop Engineering One iteration When it stops Termination conditions, max turns, budget caps, progress detection, self-verification, escalation Infinite loops, thrash, cost blow-up, confident stop before the job is done Anything agentic β€” the model decides its own next step
Graph Engineering The topology of steps Which path it takes Node decomposition, conditional edges, parallel branches, fan-out/fan-in, checkpoints, human-in-the-loop nodes, reusable subgraphs Hidden coupling, non-resumable runs, decomposition at the wrong altitude Multi-step or multi-agent, or when a run must survive a restart
The pattern to notice. Reading down that list, each discipline moves one more decision away from the model and into your system. Prompt engineering leaves almost everything to the model; graph engineering leaves it almost nothing but the judgement you cannot encode. Maturity in AI engineering is mostly about moving decisions leftward β€” deterministic by default, model by exception β€” and reaching for the next discipline only when the previous one genuinely runs out.
Common mistake: jumping to Graph and Loop engineering for a problem a good prompt and a schema would have solved. Every step down that ladder costs latency, money and debuggability. Earn it.

2 Β· How to read the mapping concept

The Need at column says at which level of ambition each concern becomes real. Not everything is required for a single LLM call, and pretending otherwise is how teams spend six months building a platform for a feature nobody has validated.

1 Single call β€” one request, one response W Workflow β€” fixed, predetermined steps S Skill β€” a packaged, reusable capability A Agent β€” the model chooses its own next step Β· not usually needed at that level
Shaded rows have no off-the-shelf answer for part of the requirement β€” the You must build in-house column says which part. 24 of the 37 rows carry one.
The pattern across that column. There is almost nothing you build entirely β€” but in two thirds of the table the judgement layer has no vendor. Vendors sell mechanisms: a retrieval engine, a guardrail runtime, an eval runner, a graph database. What none of them can supply is your definition β€” what counts as relevant, what is sensitive here, what a good answer looks like, which action needs a signature. Buying the mechanism and never writing the definition is the most common way an AI platform ends up impressive and useless.
Product names move fast β€” treat the platform columns as where to look, not as a procurement list. Verify current naming and regional availability before you commit to a design.

3 Β· Conceptual requirement β†’ physical stack map

⚠️ The third column is the one to read first: it names the part of each requirement that no vendor can supply. Shaded rows have one β€” 24 of 37 do. The platform columns to the right scroll sideways.
Conceptual requirement Need at ⚠️ You must build in-house Open source Databricks Azure AWS GCP
A Β· Harness & foundation
Model access & routingWhich model, and what happens when it is down? 1WSA β€” LiteLLM, OpenRouter, vLLM, Ollama Mosaic AI Gateway, Model Serving, external models Azure AI Foundry, Azure OpenAI, API Management Amazon Bedrock, SageMaker Vertex AI, Model Garden
Structured output & schema enforcementWill the answer parse, every time? 1WSA β€” Pydantic, Instructor, Outlines, JSON Schema Tool calling on Model Serving, UC functions Azure OpenAI structured outputs Bedrock tool use / converse API Vertex controlled generation
Tool design & registry (MCP)What can it call, and who declared that? Β·WSA The tools themselves β€” descriptions, schemas, granularity. No vendor knows your systems. MCP SDKs, FastMCP, OpenAPI specs UC functions, MCP on Databricks Apps AI Foundry tools, Logic Apps connectors Bedrock Agents action groups Vertex function calling, extensions
Prompt versioning & reproducibilityWhich prompt produced that answer last March? 1WSA The prompt library and its conventions Git, promptfoo, Langfuse MLflow prompt registry, model versions Prompt Flow, AI Foundry Bedrock prompt management Vertex AI prompt management
Config & secretsWhere do keys live, and who can read them? 1WSA β€” Vault, SOPS, dotenv Databricks Secrets, Unity Catalog Key Vault, App Configuration Secrets Manager, Parameter Store Secret Manager
B Β· Composition & control flow
Orchestration (graph engineering)What runs in what order, and can it resume? Β·WSA The topology β€” where the step boundaries fall LangGraph, Temporal, Prefect, Airflow, Dagster Workflows / Jobs, Delta Live Tables Durable Functions, Logic Apps Step Functions Workflows, Cloud Composer
Agent loop & terminationWhen does it stop, and how do we know it finished? Β·Β·SA Stop conditions, no-progress detection, budget caps Claude Agent SDK, LangGraph, AutoGen, CrewAI Mosaic AI Agent Framework AI Foundry Agent Service Bedrock Agents Vertex AI Agent Engine / ADK
State & checkpointingIf it dies at step 7, where does it restart? Β·WΒ·A β€” LangGraph checkpointer, Redis, Postgres Delta tables, Unity Catalog volumes Cosmos DB, Durable entities DynamoDB, S3 Firestore, Spanner
Concurrency, async & streamingDoes it feel fast, and does it scale? 1WSA β€” asyncio, SSE, WebSockets, Ray Model Serving streaming, serverless Azure OpenAI streaming, Container Apps Bedrock streaming, Lambda Vertex streaming, Cloud Run
Human-in-the-loop & approvalWho signs before it acts? Β·WSA The approval matrix β€” which action needs whose signature Build β€” approval queue + callback Databricks Apps, Workflows, review app Logic Apps approvals, Durable Functions Step Functions callback pattern Workflows callbacks
C Β· Context & knowledge
Context engineering & token budgetWhat occupies the window, in what order? 1WSA Everything. What enters the window, in what order, what is dropped first β€” no product decides this Token counting APIs, compaction, summarisation MLflow tracing token counts AI Foundry, Prompt Flow Bedrock converse token usage Vertex token counting
Retrieval / RAGWhere does the grounding come from? Β·WSA What counts as relevant in your domain LlamaIndex, Haystack, LangChain Vector Search + Unity Catalog Azure AI Search (hybrid + semantic) Bedrock Knowledge Bases, Kendra Vertex AI Search
Vector storeWhere do embeddings live? Β·WSA β€” pgvector, Qdrant, Weaviate, Milvus, FAISS, Chroma Databricks Vector Search AI Search, Cosmos DB vector OpenSearch Serverless, Aurora pgvector Vertex Vector Search, AlloyDB
Knowledge graphWhat are the relationships, not just the text? Β·Β·SA The ontology β€” the entities and relationships that matter here Neo4j, RDF/SPARQL, NetworkX, Apache AGE GraphFrames, Delta + UC lineage Cosmos DB Gremlin Amazon Neptune Spanner Graph
Chunking & indexing pipelineHow does source content become retrievable? Β·WSA Chunk boundaries and the metadata schema Unstructured, LlamaIndex parsers, Docling DLT / Jobs + Vector Search sync AI Search skillsets & indexers Bedrock KB ingestion, Glue Vertex AI Search ingestion, Dataflow
Memory (short & long term)What does it remember between turns and sessions? Β·Β·SA The retention policy β€” what is kept, summarised, forgotten Mem0, LangGraph memory, Redis, Zep Delta tables + UC, Lakebase Cosmos DB, Redis DynamoDB, ElastiCache Firestore, Memorystore
Caching (prompt & semantic)Are we paying twice for the same answer? 1WSA Cache key design β€” especially per-user partitioning Prompt caching, GPTCache, Redis Prompt caching on Model Serving Azure Cache for Redis, AOAI prompt caching Bedrock prompt caching, ElastiCache Vertex context caching, Memorystore
D Β· Quality & truth
Offline evalDid the change make it better or worse? 1WSA The dataset and the graders. The single largest in-house item, and the one that decides whether any of this works promptfoo, Ragas, DeepEval, Inspect AI MLflow Evaluate, Agent Evaluation AI Foundry evaluation SDK Bedrock model evaluation Vertex AI Evaluation service
Online eval, A/B & canaryIs it better for real users, not just the test set? Β·WSA The success metric Feature flags, Statsig OSS, GrowthBook MLflow + Lakehouse Monitoring, A/B on serving App Configuration, AI Foundry CloudWatch Evidently Vertex experiments
Hallucination control & groundingCan every claim be traced to a source? 1WSA The acceptance threshold β€” how grounded is grounded enough Ragas faithfulness, citation enforcement, verifier pass Agent Evaluation groundedness judges AI Foundry groundedness detection Bedrock Guardrails contextual grounding, KB citations Vertex grounding & check-grounding API
Feedback captureDo users think it helped? Β·WSA The taxonomy and what you do with it Build β€” thumbs + implicit signals Databricks Apps, review app, Delta App Insights custom events CloudWatch, DynamoDB BigQuery, Firestore
Regression testing in CIDoes the build fail when quality drops? 1WSA The thresholds that fail the build pytest, promptfoo CI, GitHub Actions Asset Bundles + MLflow in CI Azure DevOps / GitHub Actions CodeBuild, CodePipeline Cloud Build
E Β· Safety & security
Prompt injection defenceWhat if the retrieved document gives instructions? 1WSA Blast-radius limits on the untrusted path β€” detection is the second line LLM Guard, Rebuff, NeMo Guardrails Mosaic AI Gateway guardrails Azure AI Content Safety β€” Prompt Shields Bedrock Guardrails Model Armor
Guardrails (content, topic, policy)What must it never say or do? 1WSA The policies β€” what must never be said or done here Guardrails AI, NeMo Guardrails AI Gateway guardrails, UC policies Azure AI Content Safety Bedrock Guardrails Model Armor, safety filters
PII detection & redactionWhat leaves our boundary in the prompt? 1WSA Which categories are sensitive in your data Presidio, spaCy, regex rule sets AI Gateway PII, UC masking, Presidio in jobs Purview, Content Safety, Language PII Comprehend PII, Macie Sensitive Data Protection (DLP)
Identity & permission propagationDoes it see only what this user may see? 1WSA The on-behalf-of wiring and the parity tests OAuth2 / OIDC, Keycloak, on-behalf-of flows Unity Catalog, on-behalf-of, service principals Entra ID OBO flow IAM, Cognito identity pools IAM, Workload Identity Federation
Data governance & lineageWhat data went into that answer? Β·WSA β€” OpenLineage, DataHub, OpenMetadata Unity Catalog β€” lineage, tags, classification Microsoft Purview DataZone, Glue Data Catalog Dataplex
Sandboxing & code executionWhere does model-written code run? Β·Β·SA β€” Docker, gVisor, Firecracker, E2B Serverless compute, UC functions Container Apps dynamic sessions Bedrock code interpreter, Lambda Vertex code execution, Cloud Run
Audit trailCan we reconstruct this in three years? Β·WSA What is recorded, and for how long OpenTelemetry logs, append-only store UC audit logs, system tables Azure Monitor, Purview audit CloudTrail Cloud Audit Logs
F Β· Operations
Observability & tracingWhy did this run behave that way? 1WSA The span schema β€” prompt version, model, cost, quality score OpenTelemetry, OpenLLMetry, Langfuse, Phoenix MLflow Tracing, Lakehouse Monitoring Application Insights, AI Foundry tracing CloudWatch, X-Ray Cloud Trace, Cloud Monitoring
LoggingWhat happened, in order, with correlation? 1WSA β€” structlog, OpenTelemetry, Loki System tables, MLflow runs Azure Monitor Logs CloudWatch Logs Cloud Logging
Error handling, retries, circuit breakersWhat happens on the bad path? 1WSA β€” tenacity, pybreaker, Polly-style patterns Jobs retries, AI Gateway fallbacks API Management policies, Durable Functions Step Functions retry / catch Workflows retry policies
Rate limits & quotaWho gets throttled first when we hit the ceiling? 1WSA β€” Token bucket, Redis, Envoy, Kong AI Gateway rate limits API Management throttling, PTU Bedrock quotas, WAF Vertex quotas, Apigee
Performance & latencyIs it fast enough to be used? 1WSA β€” vLLM, batching, streaming, smaller models Provisioned throughput, serverless serving Provisioned throughput units (PTU) Bedrock provisioned throughput Vertex provisioned throughput
Cost & FinOpsWhat does one answer cost, and who pays? 1WSA The attribution model β€” spend to use case and business unit LiteLLM cost tracking, Langfuse, OpenCost System tables billing, tags, budget policies Cost Management, AOAI metrics Cost Explorer, Budgets Cloud Billing + BigQuery export
Deployment & CI/CD for AI assetsHow do prompts, tools and indexes ship? Β·WSA β€” GitHub Actions, GitOps, Argo Databricks Asset Bundles Azure DevOps, AI Foundry deployments CodePipeline Cloud Build, Cloud Deploy
Drift & production monitoringIs quality decaying quietly? Β·WSA The definition of drift and its alert thresholds Evidently, Phoenix, Langfuse metrics Lakehouse Monitoring AI Foundry monitoring, App Insights SageMaker Model Monitor, CloudWatch Vertex Model Monitoring

6 Β· Tool vs MCP vs Skill vs Agent concept

These four get argued about as if they were alternatives. They are not β€” they answer different questions, and the fastest way to see it is to ask what the model is missing.

Choice What it is Gives the model Choose it when Do not use it when
Workflow A predetermined sequence of steps you wrote Nothing β€” you decide the path You can write the steps down in advance. This is the default and it is under-used. The next step genuinely depends on what the previous one discovered
Tool One function the model may call, described by a schema A new ability β€” fetch, compute, act It lacks a capability: it cannot read your database or send that request It already can do it and simply does it badly β€” that is a prompt or skill problem
MCP server A standard way to expose tools, resources and prompts to any client Nothing new β€” it is distribution, not capability The same capability must be reused across apps, teams or hosts β€” or a third party provides it One app needs one function. An in-process tool is less machinery.
Skill Packaged instructions, conventions and assets, loaded on demand Know-how β€” how your organisation does this task It has the ability but not the procedure: your naming conventions, your review steps, your house style The gap is a missing capability β€” no amount of instruction grants access it does not have
Agent A loop where the model chooses the next call and when to stop Autonomy over the path The sequence cannot be known in advance β€” investigation, negotiation with reality, unknown depth You could have written the steps down. An agent is a workflow you chose not to design.

The decision, in order

  1. Can I write the steps down? β†’ Workflow. Stop here far more often than feels natural.
  2. Is a capability missing? β†’ Tool. Add the narrowest one that closes the gap.
  3. Will others need that same capability? β†’ Expose the tool over MCP. Otherwise keep it in-process.
  4. Does it have the ability but do the task wrongly? β†’ Skill. This is a knowledge gap, not a capability gap.
  5. Does the path depend on what it finds mid-run? β†’ Agent, with a loop budget and a stop condition.
The clarifying question: "Is this a can't or a doesn't know how?" A can't is a tool. A doesn't know how is a skill. Teams routinely build tools for problems that were skills, and end up with twelve narrow functions where one page of instructions would have worked.
Skill vs MCP is a false choice. They are orthogonal: MCP is how a capability is delivered, a skill is how a task should be done. A skill can happily instruct the model to use tools that arrive over MCP. If someone asks you to pick one, the question is wrong.
Cost of getting it wrong. Choosing agent when a workflow would do is the expensive mistake β€” it buys non-determinism, latency and a debugging problem, and it is the hardest thing to walk back once a demo has been seen. Choosing tool when you needed a skill is cheaper but accumulates: a sprawl of near-duplicate functions nobody can name.

7 Β· Memory & storage β€” when to use what trade-off

"Memory" is not one thing, and the store follows from the access pattern, not from the fact that AI is involved. Name the kind of memory first.

Kind of memory What it holds Lifetime Access pattern Right store Anti-pattern
Working / scratchpad Intermediate results inside one run The run Read/write by step In-process; LangGraph state; Redis if distributed Putting it in a vector store
Conversation history Turns in this session Session to days Get by session id, append, TTL Redis, Cosmos DB, DynamoDB, Firestore, Postgres Embedding every turn "just in case"
Semantic / long-term facts Documents, prior tickets, policies Months to years "Find things like this text" Vector store β€” pgvector, Qdrant, Weaviate, Milvus; Databricks Vector Search; AI Search; OpenSearch; Vertex Vector Search Using it for exact lookups it will get subtly wrong
Relationship / entity memory What depends on what; who owns what Long Multi-hop traversal Graph β€” Neo4j, Neptune, Spanner Graph, Cosmos Gremlin, Apache AGE Flattening a graph into chunks and hoping similarity recovers the edges
Episodic What happened, when, and what was decided Years β€” often regulated Append-only; query by time and entity Delta / Iceberg, BigQuery, event store, S3 + catalogue Keeping it only in application logs that rotate away
Procedural How we do things β€” conventions, runbooks Long, versioned Loaded on demand by task Git. Skills, prompts and runbooks as reviewed files Embedding your standards and hoping retrieval surfaces the right one
Cache Answers already paid for Minutes to weeks Exact key, or near-duplicate Provider prompt caching; Redis / Memorystore / ElastiCache; GPTCache for semantic Sharing a cache across users whose permissions differ

Choosing the retrieval substrate

Substrate Wins when the question is Loses when Typical choice
Vector"Anything similar to this description?" β€” vocabulary varies, phrasing is unpredictableThe query is a filter, an exact id, or an aggregatepgvector below ~1M vectors and you already run Postgres; a dedicated store above it
Keyword / BM25Exact error codes, identifiers, rare literal stringsThe same concept is worded differently across yearsOpenSearch, Elasticsearch, Postgres full-text
HybridBoth of the above β€” which is most real systemsYou have not measured which half is carrying the resultAzure AI Search, OpenSearch, Vertex AI Search, Databricks Vector Search
Graph"What is downstream of this?", "How are these connected?" β€” multi-hopYou only ever ask one-hop questions a join would answerNeo4j, Neptune, Spanner Graph
SQL / lakehouseFilters, joins, aggregates, "as at" historyThe user's phrasing must be matched rather than their filtersPostgres, Delta / Unity Catalog, BigQuery
The most common error in AI architecture: reaching for a vector store when the question was a filter or a join. "Show me open incidents for customer X" is WHERE, not cosine similarity. Vector search answers "similar to this text" and nothing else β€” everything else is a database you already know how to use.
Rule of thumb on scale: below roughly a million vectors, pgvector next to your existing relational data beats a separate vector service β€” because you get filters, joins and transactions for free, and one fewer system to secure, back up and reason about. Reach for a dedicated store when recall at scale, or ingest throughput, becomes the actual constraint.

8 Β· What to build, and what to buy trade-off

The You must build in-house column marks this row by row. Summarised, the test is simple: you must own anything that encodes your organisation's judgement, because no vendor can supply it and it is the part that actually differentiates the system.

Build in-house β€” nobody can sell you these

  • Eval dataset & graders β€” what a good answer is in your domain, with human labels
  • Relevance judgements β€” graded, not binary, and specific to your estate
  • Redaction categories β€” what counts as sensitive in your data
  • Policy & action classes β€” what an agent may do unattended, and what needs a signature
  • Golden path templates β€” the compliant route made the easy route
  • Cost attribution β€” mapping spend to business unit and use case
  • Prompt library & versioning conventions
  • Feedback taxonomy β€” what "this helped" means, and what you do with it

Buy or adopt β€” commodity, and moving fast

  • Model serving, routing and provisioned throughput
  • Vector stores and retrieval infrastructure
  • Tracing, logging and monitoring backends
  • Content safety and injection detection
  • Orchestration engines and checkpointing
  • Secret management and identity
  • Sandboxed execution environments
  • Eval runners β€” though not the eval data
The trap: building the commodity layer because it is more fun than labelling 200 incidents, and then having a beautiful platform you cannot prove is any good.

9 Β· Using this as a design review checklist practice

  1. Name the ambition. Single call, workflow, skill or agent? Everything else follows from that, and the honest answer is usually one rung lower than the pitch.
  2. Filter the table to the rows marked at that level. That is your minimum viable set of concerns.
  3. For each row, name the physical component you are actually deploying. "We'll figure out observability later" means you have chosen no observability.
  4. Find the rows with no owner. Those are your gaps, and they are usually in section E β€” safety and security are the concerns most often assumed rather than assigned.
  5. Challenge the ladder. For every agentic component, ask what would break if it were a workflow instead. If nothing would, make it a workflow.
Related: The AI Universe β€” Who Does What maps the vendors and layers; Anatomy of a Gen AI App walks the request path. This page is the checklist that sits between them.