1 Β· Five kinds of engineering concept
These get used interchangeably and they are not the same thing. The clean way to tell them apart is to ask what unit you are designing, and which decision you are taking away from the model.
| Discipline | Unit of design | You control | Main levers | Fails as | Needed when |
|---|---|---|---|---|---|
| Prompt Engineering | One message | How you ask | System prompt, role, few-shot examples, XML structure, chain-of-thought, prefill, output format | Ambiguous instruction, wrong shape of answer, inconsistent tone | Always β from the very first call |
| Context Engineering | The context window | What it sees | Retrieval, chunking, ordering, compaction, summarisation, memory, cache breakpoints, token budget | Context bloat, lost-in-the-middle, budget exhaustion, retrieved-but-irrelevant | As soon as inputs exceed a hand-written prompt |
| Harness Engineering | The runtime around the model | What it can do | Tool definitions, schema enforcement, validation, retries, circuit breakers, sandboxing, permissions, evals in CI | Brittle integration, unbounded actions, no recovery path, silent failure | The moment it touches production or real tools |
| Loop Engineering | One iteration | When it stops | Termination conditions, max turns, budget caps, progress detection, self-verification, escalation | Infinite loops, thrash, cost blow-up, confident stop before the job is done | Anything agentic β the model decides its own next step |
| Graph Engineering | The topology of steps | Which path it takes | Node decomposition, conditional edges, parallel branches, fan-out/fan-in, checkpoints, human-in-the-loop nodes, reusable subgraphs | Hidden coupling, non-resumable runs, decomposition at the wrong altitude | Multi-step or multi-agent, or when a run must survive a restart |
2 Β· How to read the mapping concept
The Need at column says at which level of ambition each concern becomes real. Not everything is required for a single LLM call, and pretending otherwise is how teams spend six months building a platform for a feature nobody has validated.
3 Β· Conceptual requirement β physical stack map
| Conceptual requirement | Need at | β οΈ You must build in-house | Open source | Databricks | Azure | AWS | GCP |
|---|---|---|---|---|---|---|---|
| A Β· Harness & foundation | |||||||
| Model access & routingWhich model, and what happens when it is down? | 1WSA | β | LiteLLM, OpenRouter, vLLM, Ollama | Mosaic AI Gateway, Model Serving, external models | Azure AI Foundry, Azure OpenAI, API Management | Amazon Bedrock, SageMaker | Vertex AI, Model Garden |
| Structured output & schema enforcementWill the answer parse, every time? | 1WSA | β | Pydantic, Instructor, Outlines, JSON Schema | Tool calling on Model Serving, UC functions | Azure OpenAI structured outputs | Bedrock tool use / converse API | Vertex controlled generation |
| Tool design & registry (MCP)What can it call, and who declared that? | Β·WSA | The tools themselves β descriptions, schemas, granularity. No vendor knows your systems. | MCP SDKs, FastMCP, OpenAPI specs | UC functions, MCP on Databricks Apps | AI Foundry tools, Logic Apps connectors | Bedrock Agents action groups | Vertex function calling, extensions |
| Prompt versioning & reproducibilityWhich prompt produced that answer last March? | 1WSA | The prompt library and its conventions | Git, promptfoo, Langfuse | MLflow prompt registry, model versions | Prompt Flow, AI Foundry | Bedrock prompt management | Vertex AI prompt management |
| Config & secretsWhere do keys live, and who can read them? | 1WSA | β | Vault, SOPS, dotenv | Databricks Secrets, Unity Catalog | Key Vault, App Configuration | Secrets Manager, Parameter Store | Secret Manager |
| B Β· Composition & control flow | |||||||
| Orchestration (graph engineering)What runs in what order, and can it resume? | Β·WSA | The topology β where the step boundaries fall | LangGraph, Temporal, Prefect, Airflow, Dagster | Workflows / Jobs, Delta Live Tables | Durable Functions, Logic Apps | Step Functions | Workflows, Cloud Composer |
| Agent loop & terminationWhen does it stop, and how do we know it finished? | Β·Β·SA | Stop conditions, no-progress detection, budget caps | Claude Agent SDK, LangGraph, AutoGen, CrewAI | Mosaic AI Agent Framework | AI Foundry Agent Service | Bedrock Agents | Vertex AI Agent Engine / ADK |
| State & checkpointingIf it dies at step 7, where does it restart? | Β·WΒ·A | β | LangGraph checkpointer, Redis, Postgres | Delta tables, Unity Catalog volumes | Cosmos DB, Durable entities | DynamoDB, S3 | Firestore, Spanner |
| Concurrency, async & streamingDoes it feel fast, and does it scale? | 1WSA | β | asyncio, SSE, WebSockets, Ray | Model Serving streaming, serverless | Azure OpenAI streaming, Container Apps | Bedrock streaming, Lambda | Vertex streaming, Cloud Run |
| Human-in-the-loop & approvalWho signs before it acts? | Β·WSA | The approval matrix β which action needs whose signature | Build β approval queue + callback | Databricks Apps, Workflows, review app | Logic Apps approvals, Durable Functions | Step Functions callback pattern | Workflows callbacks |
| C Β· Context & knowledge | |||||||
| Context engineering & token budgetWhat occupies the window, in what order? | 1WSA | Everything. What enters the window, in what order, what is dropped first β no product decides this | Token counting APIs, compaction, summarisation | MLflow tracing token counts | AI Foundry, Prompt Flow | Bedrock converse token usage | Vertex token counting |
| Retrieval / RAGWhere does the grounding come from? | Β·WSA | What counts as relevant in your domain | LlamaIndex, Haystack, LangChain | Vector Search + Unity Catalog | Azure AI Search (hybrid + semantic) | Bedrock Knowledge Bases, Kendra | Vertex AI Search |
| Vector storeWhere do embeddings live? | Β·WSA | β | pgvector, Qdrant, Weaviate, Milvus, FAISS, Chroma | Databricks Vector Search | AI Search, Cosmos DB vector | OpenSearch Serverless, Aurora pgvector | Vertex Vector Search, AlloyDB |
| Knowledge graphWhat are the relationships, not just the text? | Β·Β·SA | The ontology β the entities and relationships that matter here | Neo4j, RDF/SPARQL, NetworkX, Apache AGE | GraphFrames, Delta + UC lineage | Cosmos DB Gremlin | Amazon Neptune | Spanner Graph |
| Chunking & indexing pipelineHow does source content become retrievable? | Β·WSA | Chunk boundaries and the metadata schema | Unstructured, LlamaIndex parsers, Docling | DLT / Jobs + Vector Search sync | AI Search skillsets & indexers | Bedrock KB ingestion, Glue | Vertex AI Search ingestion, Dataflow |
| Memory (short & long term)What does it remember between turns and sessions? | Β·Β·SA | The retention policy β what is kept, summarised, forgotten | Mem0, LangGraph memory, Redis, Zep | Delta tables + UC, Lakebase | Cosmos DB, Redis | DynamoDB, ElastiCache | Firestore, Memorystore |
| Caching (prompt & semantic)Are we paying twice for the same answer? | 1WSA | Cache key design β especially per-user partitioning | Prompt caching, GPTCache, Redis | Prompt caching on Model Serving | Azure Cache for Redis, AOAI prompt caching | Bedrock prompt caching, ElastiCache | Vertex context caching, Memorystore |
| D Β· Quality & truth | |||||||
| Offline evalDid the change make it better or worse? | 1WSA | The dataset and the graders. The single largest in-house item, and the one that decides whether any of this works | promptfoo, Ragas, DeepEval, Inspect AI | MLflow Evaluate, Agent Evaluation | AI Foundry evaluation SDK | Bedrock model evaluation | Vertex AI Evaluation service |
| Online eval, A/B & canaryIs it better for real users, not just the test set? | Β·WSA | The success metric | Feature flags, Statsig OSS, GrowthBook | MLflow + Lakehouse Monitoring, A/B on serving | App Configuration, AI Foundry | CloudWatch Evidently | Vertex experiments |
| Hallucination control & groundingCan every claim be traced to a source? | 1WSA | The acceptance threshold β how grounded is grounded enough | Ragas faithfulness, citation enforcement, verifier pass | Agent Evaluation groundedness judges | AI Foundry groundedness detection | Bedrock Guardrails contextual grounding, KB citations | Vertex grounding & check-grounding API |
| Feedback captureDo users think it helped? | Β·WSA | The taxonomy and what you do with it | Build β thumbs + implicit signals | Databricks Apps, review app, Delta | App Insights custom events | CloudWatch, DynamoDB | BigQuery, Firestore |
| Regression testing in CIDoes the build fail when quality drops? | 1WSA | The thresholds that fail the build | pytest, promptfoo CI, GitHub Actions | Asset Bundles + MLflow in CI | Azure DevOps / GitHub Actions | CodeBuild, CodePipeline | Cloud Build |
| E Β· Safety & security | |||||||
| Prompt injection defenceWhat if the retrieved document gives instructions? | 1WSA | Blast-radius limits on the untrusted path β detection is the second line | LLM Guard, Rebuff, NeMo Guardrails | Mosaic AI Gateway guardrails | Azure AI Content Safety β Prompt Shields | Bedrock Guardrails | Model Armor |
| Guardrails (content, topic, policy)What must it never say or do? | 1WSA | The policies β what must never be said or done here | Guardrails AI, NeMo Guardrails | AI Gateway guardrails, UC policies | Azure AI Content Safety | Bedrock Guardrails | Model Armor, safety filters |
| PII detection & redactionWhat leaves our boundary in the prompt? | 1WSA | Which categories are sensitive in your data | Presidio, spaCy, regex rule sets | AI Gateway PII, UC masking, Presidio in jobs | Purview, Content Safety, Language PII | Comprehend PII, Macie | Sensitive Data Protection (DLP) |
| Identity & permission propagationDoes it see only what this user may see? | 1WSA | The on-behalf-of wiring and the parity tests | OAuth2 / OIDC, Keycloak, on-behalf-of flows | Unity Catalog, on-behalf-of, service principals | Entra ID OBO flow | IAM, Cognito identity pools | IAM, Workload Identity Federation |
| Data governance & lineageWhat data went into that answer? | Β·WSA | β | OpenLineage, DataHub, OpenMetadata | Unity Catalog β lineage, tags, classification | Microsoft Purview | DataZone, Glue Data Catalog | Dataplex |
| Sandboxing & code executionWhere does model-written code run? | Β·Β·SA | β | Docker, gVisor, Firecracker, E2B | Serverless compute, UC functions | Container Apps dynamic sessions | Bedrock code interpreter, Lambda | Vertex code execution, Cloud Run |
| Audit trailCan we reconstruct this in three years? | Β·WSA | What is recorded, and for how long | OpenTelemetry logs, append-only store | UC audit logs, system tables | Azure Monitor, Purview audit | CloudTrail | Cloud Audit Logs |
| F Β· Operations | |||||||
| Observability & tracingWhy did this run behave that way? | 1WSA | The span schema β prompt version, model, cost, quality score | OpenTelemetry, OpenLLMetry, Langfuse, Phoenix | MLflow Tracing, Lakehouse Monitoring | Application Insights, AI Foundry tracing | CloudWatch, X-Ray | Cloud Trace, Cloud Monitoring |
| LoggingWhat happened, in order, with correlation? | 1WSA | β | structlog, OpenTelemetry, Loki | System tables, MLflow runs | Azure Monitor Logs | CloudWatch Logs | Cloud Logging |
| Error handling, retries, circuit breakersWhat happens on the bad path? | 1WSA | β | tenacity, pybreaker, Polly-style patterns | Jobs retries, AI Gateway fallbacks | API Management policies, Durable Functions | Step Functions retry / catch | Workflows retry policies |
| Rate limits & quotaWho gets throttled first when we hit the ceiling? | 1WSA | β | Token bucket, Redis, Envoy, Kong | AI Gateway rate limits | API Management throttling, PTU | Bedrock quotas, WAF | Vertex quotas, Apigee |
| Performance & latencyIs it fast enough to be used? | 1WSA | β | vLLM, batching, streaming, smaller models | Provisioned throughput, serverless serving | Provisioned throughput units (PTU) | Bedrock provisioned throughput | Vertex provisioned throughput |
| Cost & FinOpsWhat does one answer cost, and who pays? | 1WSA | The attribution model β spend to use case and business unit | LiteLLM cost tracking, Langfuse, OpenCost | System tables billing, tags, budget policies | Cost Management, AOAI metrics | Cost Explorer, Budgets | Cloud Billing + BigQuery export |
| Deployment & CI/CD for AI assetsHow do prompts, tools and indexes ship? | Β·WSA | β | GitHub Actions, GitOps, Argo | Databricks Asset Bundles | Azure DevOps, AI Foundry deployments | CodePipeline | Cloud Build, Cloud Deploy |
| Drift & production monitoringIs quality decaying quietly? | Β·WSA | The definition of drift and its alert thresholds | Evidently, Phoenix, Langfuse metrics | Lakehouse Monitoring | AI Foundry monitoring, App Insights | SageMaker Model Monitor, CloudWatch | Vertex Model Monitoring |
6 Β· Tool vs MCP vs Skill vs Agent concept
These four get argued about as if they were alternatives. They are not β they answer different questions, and the fastest way to see it is to ask what the model is missing.
| Choice | What it is | Gives the model | Choose it when | Do not use it when |
|---|---|---|---|---|
| Workflow | A predetermined sequence of steps you wrote | Nothing β you decide the path | You can write the steps down in advance. This is the default and it is under-used. | The next step genuinely depends on what the previous one discovered |
| Tool | One function the model may call, described by a schema | A new ability β fetch, compute, act | It lacks a capability: it cannot read your database or send that request | It already can do it and simply does it badly β that is a prompt or skill problem |
| MCP server | A standard way to expose tools, resources and prompts to any client | Nothing new β it is distribution, not capability | The same capability must be reused across apps, teams or hosts β or a third party provides it | One app needs one function. An in-process tool is less machinery. |
| Skill | Packaged instructions, conventions and assets, loaded on demand | Know-how β how your organisation does this task | It has the ability but not the procedure: your naming conventions, your review steps, your house style | The gap is a missing capability β no amount of instruction grants access it does not have |
| Agent | A loop where the model chooses the next call and when to stop | Autonomy over the path | The sequence cannot be known in advance β investigation, negotiation with reality, unknown depth | You could have written the steps down. An agent is a workflow you chose not to design. |
The decision, in order
- Can I write the steps down? β Workflow. Stop here far more often than feels natural.
- Is a capability missing? β Tool. Add the narrowest one that closes the gap.
- Will others need that same capability? β Expose the tool over MCP. Otherwise keep it in-process.
- Does it have the ability but do the task wrongly? β Skill. This is a knowledge gap, not a capability gap.
- Does the path depend on what it finds mid-run? β Agent, with a loop budget and a stop condition.
7 Β· Memory & storage β when to use what trade-off
"Memory" is not one thing, and the store follows from the access pattern, not from the fact that AI is involved. Name the kind of memory first.
| Kind of memory | What it holds | Lifetime | Access pattern | Right store | Anti-pattern |
|---|---|---|---|---|---|
| Working / scratchpad | Intermediate results inside one run | The run | Read/write by step | In-process; LangGraph state; Redis if distributed | Putting it in a vector store |
| Conversation history | Turns in this session | Session to days | Get by session id, append, TTL | Redis, Cosmos DB, DynamoDB, Firestore, Postgres | Embedding every turn "just in case" |
| Semantic / long-term facts | Documents, prior tickets, policies | Months to years | "Find things like this text" | Vector store β pgvector, Qdrant, Weaviate, Milvus; Databricks Vector Search; AI Search; OpenSearch; Vertex Vector Search | Using it for exact lookups it will get subtly wrong |
| Relationship / entity memory | What depends on what; who owns what | Long | Multi-hop traversal | Graph β Neo4j, Neptune, Spanner Graph, Cosmos Gremlin, Apache AGE | Flattening a graph into chunks and hoping similarity recovers the edges |
| Episodic | What happened, when, and what was decided | Years β often regulated | Append-only; query by time and entity | Delta / Iceberg, BigQuery, event store, S3 + catalogue | Keeping it only in application logs that rotate away |
| Procedural | How we do things β conventions, runbooks | Long, versioned | Loaded on demand by task | Git. Skills, prompts and runbooks as reviewed files | Embedding your standards and hoping retrieval surfaces the right one |
| Cache | Answers already paid for | Minutes to weeks | Exact key, or near-duplicate | Provider prompt caching; Redis / Memorystore / ElastiCache; GPTCache for semantic | Sharing a cache across users whose permissions differ |
Choosing the retrieval substrate
| Substrate | Wins when the question is | Loses when | Typical choice |
|---|---|---|---|
| Vector | "Anything similar to this description?" β vocabulary varies, phrasing is unpredictable | The query is a filter, an exact id, or an aggregate | pgvector below ~1M vectors and you already run Postgres; a dedicated store above it |
| Keyword / BM25 | Exact error codes, identifiers, rare literal strings | The same concept is worded differently across years | OpenSearch, Elasticsearch, Postgres full-text |
| Hybrid | Both of the above β which is most real systems | You have not measured which half is carrying the result | Azure AI Search, OpenSearch, Vertex AI Search, Databricks Vector Search |
| Graph | "What is downstream of this?", "How are these connected?" β multi-hop | You only ever ask one-hop questions a join would answer | Neo4j, Neptune, Spanner Graph |
| SQL / lakehouse | Filters, joins, aggregates, "as at" history | The user's phrasing must be matched rather than their filters | Postgres, Delta / Unity Catalog, BigQuery |
WHERE, not cosine similarity.
Vector search answers "similar to this text" and nothing else β everything else is a database
you already know how to use.
pgvector next to
your existing relational data beats a separate vector service β because you get filters, joins and
transactions for free, and one fewer system to secure, back up and reason about. Reach for a dedicated
store when recall at scale, or ingest throughput, becomes the actual constraint.
8 Β· What to build, and what to buy trade-off
The You must build in-house column marks this row by row. Summarised, the test is simple: you must own anything that encodes your organisation's judgement, because no vendor can supply it and it is the part that actually differentiates the system.
Build in-house β nobody can sell you these
- Eval dataset & graders β what a good answer is in your domain, with human labels
- Relevance judgements β graded, not binary, and specific to your estate
- Redaction categories β what counts as sensitive in your data
- Policy & action classes β what an agent may do unattended, and what needs a signature
- Golden path templates β the compliant route made the easy route
- Cost attribution β mapping spend to business unit and use case
- Prompt library & versioning conventions
- Feedback taxonomy β what "this helped" means, and what you do with it
Buy or adopt β commodity, and moving fast
- Model serving, routing and provisioned throughput
- Vector stores and retrieval infrastructure
- Tracing, logging and monitoring backends
- Content safety and injection detection
- Orchestration engines and checkpointing
- Secret management and identity
- Sandboxed execution environments
- Eval runners β though not the eval data
9 Β· Using this as a design review checklist practice
- Name the ambition. Single call, workflow, skill or agent? Everything else follows from that, and the honest answer is usually one rung lower than the pitch.
- Filter the table to the rows marked at that level. That is your minimum viable set of concerns.
- For each row, name the physical component you are actually deploying. "We'll figure out observability later" means you have chosen no observability.
- Find the rows with no owner. Those are your gaps, and they are usually in section E β safety and security are the concerns most often assumed rather than assigned.
- Challenge the ladder. For every agentic component, ask what would break if it were a workflow instead. If nothing would, make it a workflow.