Not sure what to ask? Start here. Each question has a 2β3 sentence answer. Skim the whole thing once to build intuition, then dip back in when a term trips you up. Grouped by theme:
β Tokens & tokenization
What is a token?
A token is a chunk of text the model reads and writes β usually a whole short word or a piece of a longer one (e.g. "token", "ization"). Models don't see letters or words; they see sequences of tokens. You're billed per token, in and out.
How is the number of tokens calculated?
A component called the tokenizer splits your text into tokens using a fixed vocabulary learned during training (a "byte-pair encoding"-style scheme). Common character sequences become single tokens; rare ones get split into several. The token count is just how many pieces your text breaks into for that specific tokenizer.
Why does the same text produce different token counts on different models?
Because each model family has its own tokenizer and vocabulary, trained on different data. Claude, GPT, and Gemini each split text differently, so identical text can be, say, 1,000 tokens for one and 1,150 for another. Even a new version of the same provider can ship a new tokenizer that changes counts β so never assume counts transfer between models.
Roughly how many tokens is a word or a page?
Rule of thumb for English: ~1 token β 4 characters β ΒΎ of a word, so ~1.3 tokens per word. A single-spaced page (~500 words) is roughly ~650β750 tokens. These are estimates β always measure if it matters.
Why do code and non-English text use more tokens?
Tokenizers are optimized on mostly-English text, so English words often map to one token. Code (punctuation, indentation, rare identifiers) and non-English scripts break into many more, smaller tokens β sometimes one token per character. Same meaning, more tokens β more cost and more of your context window used.
How do I count tokens before sending a request?
Use the provider's token-counting endpoint (Anthropic exposes /v1/messages/count_tokens) or an official tokenizer library. It returns the exact input-token count for that model so you can budget before you pay. It's an estimate of input only β the model's output tokens aren't known until it generates.
Why do tokens matter so much?
Tokens are the unit of three things at once: cost (you pay per token), the context window (a token budget you can't exceed), and latency (more tokens = more to process and generate). Almost every optimization in Gen AI is really "use fewer/cheaper tokens."
Input tokens vs output tokens β what's the difference?
Input tokens are everything you send (system prompt, history, documents, tools). Output tokens are what the model generates. Output is usually priced several times higher than input, because it's produced one token at a time and is the expensive part.
β‘ Models & choosing one
What's the difference between a model family and a model (e.g. Opus / Sonnet / Haiku)?
A family (e.g. Claude) shares training and design; within it, sizes trade capability for speed/cost. Roughly: a big model (Opus) is smartest but slowest/priciest, a mid model (Sonnet) balances, and a small model (Haiku) is fastest/cheapest. Same "brain lineage," different weight class.
Why choose one model over another?
Match the model to the task's difficulty and constraints: hard reasoning β a bigger model; high-volume simple work (classify, extract) β a small fast one. Also weigh context-window size, latency needs, cost, and modality (text vs images). The pro move is a mix β a strong model for hard steps, a cheap one for bulk.
What is a context window and why does its size matter?
The context window is the maximum tokens a model can consider at once β input and output combined. A bigger window lets you fit more documents/history in one call, but doesn't remove cost (you pay for what you fill) and doesn't guarantee the model uses the middle well.
Bigger model vs smaller model β what actually changes?
Bigger models generally reason better on complex, ambiguous, multi-step tasks, but cost more and respond slower. Smaller models are cheaper and faster and are often good enough for well-scoped tasks. Don't pay for Opus-level reasoning to do a lookup a Haiku-level model nails.
When should I use a "reasoning" / extended-thinking model?
When the task needs multi-step logic, math, planning, or careful analysis β letting the model "think" before answering measurably improves accuracy there. For simple lookups, classification, or formatting it just adds latency and cost, so skip it.
What does "multimodal" mean?
A multimodal model handles more than text β commonly images (and sometimes audio/PDF) as input. If your task involves screenshots, documents, or photos, you need a model with that modality; a text-only model can't see them.
Open-weight vs closed/proprietary models β why pick each?
Open-weight (e.g. Llama, Mistral) can be downloaded and self-hosted β full control, data stays in-house, no per-token fee but you run the infra. Closed (Claude, GPT, Gemini) are accessed via API β top capability and zero ops, but you send data out and pay per token.
How do I know which model is actually "best" for me?
There's no universal best β run an eval. Build a small set of representative test cases with a grading method, try a few models, and compare quality vs cost vs latency on your task. Benchmarks are a starting hint, not an answer.
β’ Providers & clouds
What's the difference between a model provider (lab) and a cloud/gateway?
A provider/lab (Anthropic, OpenAI, Google) trains the model. A cloud gateway (AWS Bedrock, Azure AI Foundry, Google Vertex AI) hosts and resells many models β often including the same ones β with enterprise plumbing around them. See the AI Universe layer map.
Why choose one provider over another (Anthropic vs OpenAI vs Google)?
Compare on model capability for your task, price, latency, context window, safety posture, and features (caching, batch, tool use, structured outputs). Many teams also value ecosystem fit and support. The honest answer: benchmark 2β3 on your real workload rather than trusting a leaderboard.
Why access a model through Bedrock/Vertex/Foundry instead of the provider's own API?
To keep everything inside your existing cloud: unified billing, IAM/security, data residency, networking, and compliance you already trust. If your data and apps live in AWS/Azure/GCP, calling the model through that cloud's gateway avoids sending data to a separate vendor and simplifies governance.
Why would an enterprise pick Azure AI Foundry or AWS Bedrock specifically?
Usually because they're already committed to that cloud β contracts, credits, security reviews, and skills are in place. The gateway gives them multiple models (so they aren't locked to one lab), regional deployment, and central governance, all under one bill.
What does Databricks add on top?
Databricks sits where your data lives (the lakehouse). It adds model serving, RAG, and governance next to your data, so you can build Gen AI on your own tables without shipping data elsewhere. It's a data+AI platform, not a model lab.
Does the same model behave identically across providers?
Mostly yes on core quality β it's the same weights β but availability, versions, region, latency, price, quotas, and some features can differ, and a specific version may lag on one platform. Test on the platform you'll actually ship on.
β£ Cost & performance
What drives the cost of an LLM call?
Tokens Γ price. Cost = (input tokens Γ input rate) + (output tokens Γ output rate), and the rates depend on the model tier. So the three cost levers are: pick a cheaper model, send fewer input tokens, and generate fewer output tokens.
Why is output usually more expensive than input?
Input is processed in one pass, but output is generated one token at a time, each requiring a full forward pass β it's the compute-heavy part. So output tokens are typically priced several times higher than input tokens; keeping responses concise saves real money.
What is prompt caching and when does it save money?
Prompt caching stores a processed, static prompt prefix so repeated requests reuse it instead of re-processing β cached reads are ~90% cheaper and faster. It pays off when you send the same large system prompt / documents across many calls (agents, chatbots). It doesn't help if your prompt changes every time.
What is the Batch API and when should I use it?
The Batch API processes many requests asynchronously (results within a 24-hour window) at roughly half price. Perfect for non-urgent bulk work β evals, backfills, offline generation β but not for anything interactive where a user is waiting.
How do I reduce latency?
Use a smaller/faster model, stream the response (so first words appear immediately), cap output length, cache the prompt prefix, and trim input tokens. Time-to-first-token and total tokens are the two dials that matter most.
Why does a bigger model cost and take more?
More parameters means more computation per token, so each token is pricier and slower to produce. That's the core trade-off: you buy better reasoning with money and time, so only spend it where the task actually needs it.
β€ Prompting & context
What is a prompt, and what is a system prompt?
A prompt is the text you send to steer the model. The system prompt is a special top-level instruction that sets the model's role, rules, and tone for the whole conversation, separate from the user's turn-by-turn messages.
What's the difference between the context window and "memory"?
The context window is the model's short-term working memory for a single request β it forgets everything once the call ends. "Memory" is what you build around it (summaries, a database, a memory tool) to carry facts across turns and sessions. The API itself is stateless.
Why put important info at the start or end of a long prompt?
Models pay less attention to the middle of very long inputs (the "lost in the middle" effect). Placing key instructions and the actual question near the top and bottom β and asking the model to quote relevant passages first β improves accuracy on long documents.
What is few-shot prompting / using examples?
Showing the model a few inputβoutput examples of exactly what you want. It's often the single most effective way to lock in a format or style β "show, don't just tell." A few relevant, diverse examples usually beats a long paragraph of instructions.
Prompting vs fine-tuning β when do I use which?
Prompting is instant, free-ish, and iterable β always try it first. Fine-tuning retrains the model's weights on your data; it's heavier and slower to update, worth it only when prompting (and RAG) can't get the consistent behavior you need.
β₯ RAG & vector stores
What is an embedding?
An embedding is a list of numbers (a vector) that represents the meaning of a piece of text, produced by an embedding model. Texts with similar meaning get similar vectors, so you can measure "closeness" mathematically β the basis of semantic search.
What is a vector store / vector database?
A database (Pinecone, Weaviate, pgvector, Milvus, Chroma) built to store embeddings and find the nearest ones fast. Given a query vector, it returns the most similar stored chunks β the retrieval engine behind RAG.
What is RAG and why use it?
Retrieval-Augmented Generation: at query time, fetch the most relevant chunks of your data from a vector store and put them in the prompt, so the model answers from current, private knowledge it was never trained on. It reduces hallucination and keeps answers grounded β without retraining.
RAG vs fine-tuning vs long context β how do I choose?
Long context / prompting: small, per-request info β just paste it in. RAG: large or frequently-changing knowledge base β retrieve the relevant bits. Fine-tuning: you need a fixed style/behavior, not new facts. They're complementary, not either/or.
What is chunking?
Splitting documents into smaller passages ("chunks") before embedding, so retrieval returns focused, relevant pieces instead of whole files. Chunk size and overlap are tuning knobs: too big wastes context, too small loses meaning.
Does a bigger context window remove the need for RAG?
No. You could stuff more in, but it's slower, pricier, and less precise β and models still attend unevenly to huge inputs. RAG sends only what's relevant, which is usually cheaper and more accurate. Big windows help RAG; they don't replace it.
β¦ Agents, tools & MCP
What's an agent vs a single LLM call?
A single call is one prompt in, one answer out. An agent runs the model in a loop: it decides an action, calls a tool, reads the result, and repeats until the task is done. The model β not hard-coded logic β chooses the next step.
What is tool / function calling?
Giving the model a set of functions it can invoke (search, database query, send email). The model outputs a structured request to call a tool with arguments; your code runs it and returns the result, which the model then uses. It's how an LLM reaches live data and takes actions.
What is MCP and why does it matter?
Model Context Protocol is an open standard for connecting models to tools and data β the "USB-C port for AI." Instead of hand-wiring every integration per app, you run an MCP server once and any MCP-compatible app (Claude, IDEs, etc.) can use it. It standardizes the plumbing.
What's the difference between a tool, RAG, and memory?
Different jobs: tools give the model live data and actions; RAG gives it relevant documents to read; memory gives it continuity across turns. A capable app often uses all three together.
β§ Training vs inference
What's the difference between training and inference?
Training creates the model β a massive, one-time (expensive) process that sets its weights by learning from data. Inference is using the finished model to answer a prompt. As an app developer you almost always do inference, not training.
What is pre-training vs fine-tuning?
Pre-training teaches a model general language/knowledge from huge, broad data β this makes the "foundation model." Fine-tuning is a lighter follow-up that adapts that base to a specific behavior, domain, or format.
What is a "foundation model"?
A large, general-purpose model pre-trained on broad data that can be adapted to many tasks (via prompting, RAG, or fine-tuning). LLMs like Claude and GPT are foundation models; so are large image and embedding models.
What does the "B" in "70B" mean?
Billions of parameters β the tunable numbers (weights) the model learned during training. More parameters generally means more capacity to model complexity, but also more compute to run. It's a rough size indicator, not a quality guarantee.
Does the model learn from my prompts?
No β at inference the weights are frozen. The model doesn't remember or learn from your API calls; each request is independent, and any "memory" is something the app re-sends. (Separately, providers have data-use policies governing whether inputs may be used to improve future models β check them.)
β¨ Reliability & safety
Why do models "hallucinate," and how do I reduce it?
A model predicts plausible text, so when it lacks the facts it can state confident-sounding wrong answers. Reduce it by grounding (give it the source docs via RAG), allowing "I don't know," asking it to cite/quote evidence, and verifying critical outputs.
What is a rate limit?
A cap on how much you can call the API in a window β typically requests per minute and tokens per minute. Exceed it and you get a 429 error; the fix is to back off and retry, and to spread or queue load.
What are guardrails?
Checks around the model that keep it safe and on-spec: input/output validation, moderation, permission gates on tools, spend/rate caps, and human approval for risky actions. The model is powerful but not trusted blindly β guardrails bound what it can do.
What is an eval?
A repeatable test of model quality: a set of representative cases plus a grading method (code checks, an LLM-as-judge, or humans). Evals let you compare models, catch regressions when you change a prompt or model, and prove something actually got better.
β© A bit deeper
What is "temperature" (and top-p)?
Temperature controls randomness in how the model picks the next token: low (near 0) = focused, deterministic, repeatable; higher = more varied and creative. top-p (nucleus sampling) is a related dial that limits choices to the most probable options. Use low temperature for facts/code/extraction, higher for brainstorming.
What does a "stateless" API really mean?
The model keeps no memory between calls β every request must include the full conversation you want it to see. It doesn't "remember" your last message unless you resend it. That's why you own history management, and why longer conversations cost more (you re-send and re-pay for the history each turn).
Streaming vs non-streaming β what's the difference?
Non-streaming: you wait for the whole response, then get it at once. Streaming: tokens arrive as they're generated, so the user sees text appear immediately (lower perceived latency) and long outputs avoid timeouts. Same total tokens and cost β it just changes when you receive them.
What are "structured outputs" / JSON mode?
Ways to force the model to return data in an exact shape (e.g. valid JSON matching a schema) instead of free text, so your code can parse it reliably. Approaches include tool/function calling, prefilling, and a dedicated structured-output feature. Essential when the model's answer feeds another system.
Where does my data go β is it private?
When you call a hosted API your prompt leaves your machine to the provider (or the cloud gateway) to run inference. Weights don't learn from it, but whether inputs may be retained or used to improve models is governed by the provider's data-use and retention policy β read it, and for strict requirements use a cloud gateway (data stays in your cloud) or self-hosted open models.
What does a multi-agent system look like?
Instead of one agent doing everything, a coordinator delegates sub-tasks to specialized sub-agents (e.g. researcher, writer, checker), each with its own focused context and tools, then merges their results. It helps when tasks are separable or need different expertise β at the cost of more complexity and tokens. (This is the heart of CCA Domain 1.)
CHEAT SHEETS
Token rules of thumb (English)
| Text | β Tokens |
|---|---|
| 1 token | ~4 characters Β· ~ΒΎ word |
| 1 word | ~1.3 tokens |
| 1 sentence | ~15β30 tokens |
| 1 single-spaced page (~500 words) | ~650β750 tokens |
| Code & non-English | Noticeably more per word |
Quick "which to pick" guide
| Decision | Lean toward⦠|
|---|---|
| Hard reasoning / ambiguous task | A bigger model (e.g. Opus-tier) |
| High-volume, simple, latency-sensitive | A small fast model (e.g. Haiku-tier) |
| Need current/private knowledge | RAG (not fine-tuning) |
| Need a fixed style/format everywhere | Few-shot examples β then fine-tuning |
| Same big prompt reused a lot | Prompt caching |
| Bulk, non-urgent jobs | Batch API (~50% off) |
| Data must stay in your cloud | A cloud gateway (Bedrock / Foundry / Vertex) |