Gen AI Fundamentals › Training vs Inference

⚙️ Training vs Inference — Building a Model vs Using One

One is how a model is made · the other is how you run it every day

BIG IDEA Two completely different phases

People blur "training" and "running" a model, but they're worlds apart. Training is the giant, one-time process that creates the model by learning its weights from data. Inference is what happens every time you send a prompt — the finished model, weights frozen, produces an answer. As an app builder you live almost entirely in inference; training is done by the labs.

🏗️ Training (make the model)

  • Done once by the lab (Anthropic, OpenAI…)
  • Huge cost: thousands of GPUs, weeks–months
  • Sets the weights
  • Result: a reusable model

⚡ Inference (use the model)

  • Happens on every request
  • Cost = tokens × price, milliseconds–seconds
  • Weights are frozen — read-only
  • What your app and this study material are about

DIAGRAM The model lifecycle

flowchart LR D["📚 Huge text corpus"] --> PT["Pre-training
next-token prediction"] PT --> BASE["Base model
(knows language, raw)"] BASE --> POST["Post-training
SFT + RLHF / alignment"] POST --> ALIGN["Aligned model
(helpful, safe, follows instructions)"] ALIGN --> DEP["Deploy behind an API"] DEP --> INF["⚡ Inference
your prompt → answer"] U["👤 You"] --> INF --> U classDef train fill:#f3f0ff,stroke:#764ba2,color:#4a3b7a; classDef run fill:#eef6ff,stroke:#3b7bd0,color:#1f5fa8; class PT,POST train; class DEP,INF run;

KEY POINT At inference, the model does not learn

This trips up almost everyone: when you chat with a model, it is not learning from you. The weights are fixed. Any "memory" of your earlier messages exists only because the conversation is re-sent with each call (the API is stateless). To actually change what a model knows or how it behaves, you must go back to a training step — or, far more commonly, work around it with prompting and RAG.

⚠️ Separate from weights: providers have data-use policies about whether your inputs may be stored or used to train future models. That's a policy/contract matter — it doesn't mean the model you're calling learns live.

THE LADDER Four ways to adapt a model — cheapest to heaviest

When you need a model to know or do something new, climb only as high as you must:

MethodWhat it changesEffort / costWhen to use
PromptingJust the input — no model change🟢 InstantSteering behavior, small per-request context
RAGAdds retrieved data at query time🟡 Moderate (build a pipeline)Large or changing private knowledge
Fine-tuningAdjusts the weights on your examples🟠 Heavier, slower to updateA fixed style/format the base won't hold
Pre-trainingBuilds a model from scratch🔴 Enormous (labs only)Almost never — you buy this, not build it
Practical order: prompt → add RAG → consider fine-tuning. 90%+ of real apps never need to touch the weights at all.

COST Why inference cost is the one you manage

Training cost is the lab's problem, amortized into the per-token price you pay. Inference is your recurring bill, and it scales with usage — so the whole of CCA Domain 5 (caching, batching, model tiers, token efficiency) is really about optimizing inference. Training a model is like building a factory; inference is paying for each item it makes.

TAKEAWAYS

Remember: training creates weights; inference uses them. You do inference.
Remember: the model doesn't learn from your prompts at inference — "memory" is just re-sent context.
Remember: to add knowledge, reach for RAG before fine-tuning; to change behavior/style, try prompting/examples before fine-tuning.

GLOSSARY

TermIn one sentence
TrainingThe process that sets a model's weights by learning from data.
InferenceRunning the finished model to answer a prompt.
Pre-trainingThe large first phase: learn language/knowledge via next-token prediction.
Fine-tuningA lighter follow-up that adapts a base model to a task or style.
SFTSupervised fine-tuning — training on curated good input→output examples.
RLHFReinforcement Learning from Human Feedback — aligning a model to human preferences.
Base vs aligned modelRaw text predictor vs the helpful, instruction-following assistant.
Foundation modelA large general model you adapt, rather than train yourself.
➡️ You've now got the core mental models. Head back to the Fundamentals hub or dive into the CCA-F material.