BIG PICTURE An LLM alone is not an application
A raw model just turns tokens into tokens. A production Gen AI application wraps that model in a set of components that give it your data, memory, tools, guardrails, and reliability. Almost every serious app — a support bot, a coding agent, a research assistant — is built from the same handful of building blocks.
DIAGRAM Reference architecture
flowchart TB
UI["👤 Client / UI"] --> ORCH
subgraph CORE["Application core"]
ORCH["🧭 Orchestrator / Agent loop"]
PROMPT["📝 Prompt & context assembly"]
ORCH <--> PROMPT
end
PROMPT -->|"query"| RAG
subgraph KNOW["Knowledge (RAG)"]
EMB["Embedding model"] --> VDB[("Vector store")]
end
RAG["Retriever"] --> EMB
VDB -->|"chunks"| PROMPT
PROMPT --> LLM["🧠 LLM (inference)"]
LLM <-->|"tool calls"| TOOLS["🔧 Tools / MCP"]
ORCH <--> MEM["💾 Memory (short + long term)"]
ORCH <--> CACHE["⚡ Prompt cache"]
LLM --> GUARD["🛡️ Guardrails / validation"] --> UI
ORCH -.emits.-> OBS["📊 Eval & observability"]
DATA[("📚 Your data sources")] -.ingest / chunk / embed.-> VDB
classDef core fill:#f3f0ff,stroke:#764ba2,color:#4a3b7a;
classDef ext fill:#eef6ff,stroke:#3b7bd0,color:#1f5fa8;
class ORCH,PROMPT core; class LLM,TOOLS,VDB,EMB,MEM,CACHE,GUARD,OBS ext;
COMPONENTS The building blocks
| # | Component | What it does | Example tech | Maps to CCA domain |
|---|---|---|---|---|
| 1 | Client / UI | Where the user types and sees results (chat, IDE, API) | Web app, CLI, Slack, IDE | — |
| 2 | Orchestrator / Agent loop | The control flow: decides steps, calls the model & tools until done | Agent SDK, LangChain, custom loop | D1 · Agentic Architecture |
| 3 | Prompt & context assembly | Builds the system prompt, few-shot examples, and injects retrieved data | Templates, XML tags | D3 · Prompt Engineering |
| 4 | LLM (inference) | The reasoning engine that generates responses | Claude, GPT, Gemini, Llama | D1 · D3 |
| 5 | Embeddings + Vector store (RAG) | Turns data into vectors and retrieves the relevant chunks at query time | Voyage/Cohere + Pinecone/pgvector | D5 · Context Management |
| 6 | Memory | Carries facts across turns (short-term) and sessions (long-term) | Summaries, memory tool, DB | D5 · Context Management |
| 7 | Tools / MCP | Lets the model fetch live data & take actions (search, DB, APIs) | Function calling, MCP servers | D4 · Tool Design & MCP |
| 8 | Caching | Reuses stable prompt prefixes to cut cost & latency | Prompt caching | D5 · Context Management |
| 9 | Guardrails / validation | Checks inputs & outputs: safety, format, permissions, spend caps | Classifiers, schemas, policies | D2 · D4 · D5 |
| 10 | Eval & observability | Measures quality, cost, latency; catches regressions & drift | Eval suites, tracing, logs | D5 · Observability |
| 11 | Data pipeline | Ingests, chunks, cleans & embeds source data into the vector store | ETL, Databricks, chunkers | D5 · Context Management |
Notice the right column: the components of a Gen AI app line up almost one-to-one with the
five CCA-F domains. That's the bridge — learn the building blocks here, then go deep on each in the CCA material.
TRADE-OFFS Three ways to give a model your knowledge
A recurring architecture decision: how do you get the model to know things it wasn't trained on? Three levers, increasing in weight:
| Approach | How | Best for | Cost / effort |
|---|---|---|---|
| Prompting / in-context | Put the info directly in the prompt | Small, per-request context | 🟢 Lowest |
| RAG | Retrieve relevant chunks from a vector store at query time | Large, changing knowledge bases | 🟡 Medium |
| Fine-tuning | Further-train the model weights on your data | Fixed style/format or narrow domain | 🔴 Highest |
Rule of thumb: start with prompting, add RAG when knowledge is large or changes often,
and reach for fine-tuning only when prompting + RAG can't get the behavior you need.
DATA FLOW Two phases: ingestion vs. query
RAG has two separate timelines that beginners often conflate:
🗂️ Ingestion (offline, ahead of time)
- Load source docs
- Chunk into passages
- Embed each chunk → vectors
- Store in the vector DB
⚡ Query (online, per request)
- Embed the user's question
- Search the vector DB for nearest chunks
- Inject chunks into the prompt
- Model answers, grounded in them
TAKEAWAYS
Remember: the LLM is one component, not the whole app — most engineering effort goes into the layers around it.
Remember: RAG ≠ fine-tuning. RAG adds knowledge at query time without changing the model; fine-tuning changes the weights.
Remember: tools/MCP give the model actions & live data; RAG gives it relevant documents; memory gives it continuity. Different jobs.