Gen AI Fundamentals › Anatomy of a Gen AI App

🧩 Anatomy of a Gen AI Application

The major components of a real product — and how they fit together

BIG PICTURE An LLM alone is not an application

A raw model just turns tokens into tokens. A production Gen AI application wraps that model in a set of components that give it your data, memory, tools, guardrails, and reliability. Almost every serious app — a support bot, a coding agent, a research assistant — is built from the same handful of building blocks.

DIAGRAM Reference architecture

flowchart TB UI["👤 Client / UI"] --> ORCH subgraph CORE["Application core"] ORCH["🧭 Orchestrator / Agent loop"] PROMPT["📝 Prompt & context assembly"] ORCH <--> PROMPT end PROMPT -->|"query"| RAG subgraph KNOW["Knowledge (RAG)"] EMB["Embedding model"] --> VDB[("Vector store")] end RAG["Retriever"] --> EMB VDB -->|"chunks"| PROMPT PROMPT --> LLM["🧠 LLM (inference)"] LLM <-->|"tool calls"| TOOLS["🔧 Tools / MCP"] ORCH <--> MEM["💾 Memory (short + long term)"] ORCH <--> CACHE["⚡ Prompt cache"] LLM --> GUARD["🛡️ Guardrails / validation"] --> UI ORCH -.emits.-> OBS["📊 Eval & observability"] DATA[("📚 Your data sources")] -.ingest / chunk / embed.-> VDB classDef core fill:#f3f0ff,stroke:#764ba2,color:#4a3b7a; classDef ext fill:#eef6ff,stroke:#3b7bd0,color:#1f5fa8; class ORCH,PROMPT core; class LLM,TOOLS,VDB,EMB,MEM,CACHE,GUARD,OBS ext;

COMPONENTS The building blocks

#ComponentWhat it doesExample techMaps to CCA domain
1Client / UIWhere the user types and sees results (chat, IDE, API)Web app, CLI, Slack, IDE—
2Orchestrator / Agent loopThe control flow: decides steps, calls the model & tools until doneAgent SDK, LangChain, custom loopD1 · Agentic Architecture
3Prompt & context assemblyBuilds the system prompt, few-shot examples, and injects retrieved dataTemplates, XML tagsD3 · Prompt Engineering
4LLM (inference)The reasoning engine that generates responsesClaude, GPT, Gemini, LlamaD1 · D3
5Embeddings + Vector store (RAG)Turns data into vectors and retrieves the relevant chunks at query timeVoyage/Cohere + Pinecone/pgvectorD5 · Context Management
6MemoryCarries facts across turns (short-term) and sessions (long-term)Summaries, memory tool, DBD5 · Context Management
7Tools / MCPLets the model fetch live data & take actions (search, DB, APIs)Function calling, MCP serversD4 · Tool Design & MCP
8CachingReuses stable prompt prefixes to cut cost & latencyPrompt cachingD5 · Context Management
9Guardrails / validationChecks inputs & outputs: safety, format, permissions, spend capsClassifiers, schemas, policiesD2 · D4 · D5
10Eval & observabilityMeasures quality, cost, latency; catches regressions & driftEval suites, tracing, logsD5 · Observability
11Data pipelineIngests, chunks, cleans & embeds source data into the vector storeETL, Databricks, chunkersD5 · Context Management
Notice the right column: the components of a Gen AI app line up almost one-to-one with the five CCA-F domains. That's the bridge — learn the building blocks here, then go deep on each in the CCA material.

TRADE-OFFS Three ways to give a model your knowledge

A recurring architecture decision: how do you get the model to know things it wasn't trained on? Three levers, increasing in weight:

ApproachHowBest forCost / effort
Prompting / in-contextPut the info directly in the promptSmall, per-request context🟢 Lowest
RAGRetrieve relevant chunks from a vector store at query timeLarge, changing knowledge bases🟡 Medium
Fine-tuningFurther-train the model weights on your dataFixed style/format or narrow domain🔴 Highest
Rule of thumb: start with prompting, add RAG when knowledge is large or changes often, and reach for fine-tuning only when prompting + RAG can't get the behavior you need.

DATA FLOW Two phases: ingestion vs. query

RAG has two separate timelines that beginners often conflate:

🗂️ Ingestion (offline, ahead of time)

  • Load source docs
  • Chunk into passages
  • Embed each chunk → vectors
  • Store in the vector DB

⚡ Query (online, per request)

  • Embed the user's question
  • Search the vector DB for nearest chunks
  • Inject chunks into the prompt
  • Model answers, grounded in them

TAKEAWAYS

Remember: the LLM is one component, not the whole app — most engineering effort goes into the layers around it.
Remember: RAG ≠ fine-tuning. RAG adds knowledge at query time without changing the model; fine-tuning changes the weights.
Remember: tools/MCP give the model actions & live data; RAG gives it relevant documents; memory gives it continuity. Different jobs.