THE IDEA The model is one line; the rest is the server
The best way to understand what the Anthropic server does is to sketch your own. When you do, the punchline jumps out: calling the LLM is a single line in the middle — auth, rate limits, validation, prompt assembly, caching, safety, response shaping, billing, and streaming are all the server's job, before and after that line. This page designs that shell (not the transformer — that stays a black box) so you can see the responsibilities.
THE BIG PICTURE Three boxes, not one
People imagine "the API = the model." It's really three responsibilities, and you only own the outer one:
build messages · resend history
· loop on stop_reason · retry"]:::you S["ANTHROPIC SERVER
auth · limits · validate · assemble
· cache · safety · meter · stream"]:::srv M["THE MODEL
prefill + decode
(the transformer)"]:::mod C -->|"POST /v1/messages"| S -->|"prompt"| M M -->|"tokens"| S -->|"JSON response"| C classDef you fill:#e7ecff,stroke:#3b5bdb,color:#1e3a8a; classDef srv fill:#f3f0ff,stroke:#764ba2,color:#4a3b7a; classDef mod fill:#dcfce7,stroke:#15803d,color:#14532d;
MyOwnAnthropicServer is the middle box. The client box is what you already build (see the
agent loop); the model box we leave as model.generate().
DO YOU ALWAYS NEED THE SERVER? Three deployment shapes
Common question: "can the agent talk to the LLM directly, with no Anthropic-style server?" Yes — but be precise about what "direct" means. The server is a bundle of responsibilities, not a mandatory box. Who performs them (and whether you need them at all) depends on where the model runs:
all 13 stages"]:::srv --> a3["Model"]:::mod end subgraph B["B · SELF-HOSTED SERVER — you run it"] direction LR b1["Client / LangGraph"] --> b2["Your server
vLLM · TGI · Ollama"]:::srv --> b3["Model"]:::mod end subgraph C["C · IN-PROCESS — no server at all"] direction LR c1["Agent = the runtime
transformers · llama.cpp"] --> c3["Model (same process)"]:::mod end classDef srv fill:#f3f0ff,stroke:#764ba2,color:#4a3b7a; classDef mod fill:#dcfce7,stroke:#15803d,color:#14532d;
| A · Hosted API | B · Self-hosted server | C · In-process / embedded | |
|---|---|---|---|
| Backend examples | Anthropic, OpenAI, Bedrock, Vertex, Databricks Model Serving | vLLM, TGI, Ollama, LM Studio, Triton | HF transformers, llama-cpp-python, MLX |
| Who runs the "server" | the provider | you (ops burden) | nobody — it's a function call |
| Auth · limits · billing · safety | provider handles it | you build/operate what you need | usually N/A (single user) |
| Client examples | LangGraph, your app, the Anthropic SDK | LangGraph, your app | your script |
| Truly "direct to the LLM"? | No — HTTP to the provider's server | Sort of — your own local/edge server | Yes — same process, model.generate() |
| Trade-off | zero ops, pay per token, rate limits | control & privacy, but you operate it | simplest, but you own scaling & safety |
THE REQUEST LIFECYCLE 13 stages of one call
THE MINIMAL SERVER ~40 lines — one function per responsibility
# MyOwnAnthropicServer — a teaching shell for POST /v1/messages. # The transformer is a black box (model.generate). Everything AROUND it is the server's job. def handle_messages(http_request): # 1 · TRANSPORT — parse the HTTP body try: body = json.loads(http_request.body) except: return err(400, "invalid_request_error", "malformed JSON") # 2 · AUTH — who is this? (x-api-key, anthropic-version) → 401 / 403 org = authenticate(http_request.headers.get("x-api-key")) if not org: return err(401, "authentication_error", "invalid key") if not org.can_use(body.get("model")): return err(403, "permission_error", "no access") # 3 · RATE LIMIT / QUOTA — token buckets per org (RPM/ITPM/OTPM) → 429 + Retry-After if not rate_limiter.allow(org, body): return err(429, "rate_limit_error", retry_after=2) # 4 · SIZE — payload cap → 413 if too_large(body): return err(413, "request_too_large", "body too large") # 5 · VALIDATE — the big 400 layer, BEFORE the model ever runs bad = validate(body) # required fields · role alternation · tool_use<->tool_result if bad: return err(400, "invalid_request_error", bad) # · max_tokens<=limit · no temperature/budget_tokens # 6 · ROUTE — resolve model id -> engine; check the context window → 404 if unknown model = resolve_model(body["model"]) # 7 · ASSEMBLE PROMPT — render tools -> system -> messages into model format; count tokens prompt, n_in = render(body) # enforce max input (context-window-exceeded is a 400) # 8 · PROMPT CACHE — hash the stable prefix; reuse KV on a hit (skip prefill) kv, cache_read, cache_new = prompt_cache.lookup(prompt, body) # 9 · INFER — the ONE line that is "the model" (prefill + autoregressive decode) out = model.generate(prompt, kv, max_tokens = body["max_tokens"], # → stop_reason "max_tokens" stop_sequences = body.get("stop_sequences"), # → "stop_sequence" thinking = body.get("thinking"), # adaptive reasoning (billed as output) tools = body.get("tools"), # constrained decode → "tool_use" output_config = body.get("output_config")) # schema-constrained JSON # 10 · SAFETY — classifiers can override the output → stop_reason "refusal" if safety.blocks(body, out): out = refusal() # 11 · RESPONSE — content blocks (text/thinking/tool_use) + stop_reason + usage resp = build(out, stop_reason=out.stop_reason, usage=usage(n_in, out.n_out, cache_read, cache_new)) # the 4 usage fields # 12 · METER + LOG — record billing; attach request-id for tracing meter(org, resp.usage); log(http_request.id, org, resp) # 13 · RETURN — one JSON body, or a stream of SSE events if body["stream"] return sse_stream(resp) if body.get("stream") else json_response(resp)
RESPONSIBILITY TABLE stage → what it does → what breaks there
| # | Server responsibility | Fails with | CCA tie-in |
|---|---|---|---|
| 1 | HTTP transport & JSON parsing | 400 (malformed JSON) | D4 |
| 2 | Auth — API key → org/workspace & model access | 401 / 403 | D5 errors |
| 3 | Rate limiting — per-org token buckets (RPM/TPM) | 429 (+ Retry-After) | D5 · rate limits |
| 4 | Payload size limits | 413 | D5 |
| 5 | Validation — schema, message structure, tool contract, params | 400 (pre-model, unbilled) | D4 · tool contract |
| 6 | Model routing & capability/context-window check | 404 / 400 | D1 |
| 7 | Prompt assembly — render tools→system→messages; token counting | 400 (context exceeded) | D5 · tokens |
| 8 | Prompt caching — prefix hash, KV reuse, cache accounting | — | D5 · caching |
| 9 | Inference — prefill + decode, stop conditions, tool/structured decode | 200 + stop_reason | D1 · stop reasons |
| 10 | Safety — input/output classifiers | 200 + refusal | D5 |
| 11 | Response assembly — content blocks, stop_reason, usage | — | D1 · D5 |
| 12 | Metering / billing / logging (request-id) | — | D5 · cost, obs |
| 13 | Serialization & streaming (SSE events) | 500 / 529 on failure | D5 · reliability |
ZOOM: STAGE 9 what "call the LLM" actually hides
The one line model.generate() is itself a small pipeline — worth knowing for the exam even though you'd
never build it:
- Prefill — process the whole prompt in one pass, building the KV cache (this is what prompt caching reuses → the ~0.1× cache-read).
- Decode — generate output tokens one at a time, autoregressively (this is why output is slow & costs 5×).
- Stop conditions — halt on natural end (
end_turn), themax_tokenscap, astop_sequence, or a tool call (tool_use) → the server stamps the matchingstop_reason. - Constrained decoding — for
tools/output_config.format, sampling is restricted to tokens valid under the schema/grammar (this is why structured outputs can't produce malformed JSON — the shape is guaranteed at decode time). - Thinking — extra reasoning tokens generated before the answer, billed as output.
SCOPE what we deliberately DON'T build
model.generate(), which is exactly the boundary Anthropic exposes to you.CLIENT vs SERVER the responsibility split (and why statelessness)
| Your client does | The server does |
|---|---|
| Build & resend the full message history every turn | Treat each request as stateless — no session memory |
Loop on stop_reason, run tools, return tool_result | Decode the tool call, stamp tool_use, validate the returned pairing |
| Retry 429/5xx with backoff; set idempotency | Enforce limits, return typed errors, meter usage |
| Decide model, max_tokens, caching breakpoints | Route, cache, assemble, infer, shape the response |
CONCEPTS → CCA MAP
| You now understand… | CCA domain |
|---|---|
| Where 400 vs 401 vs 429 vs 200+stop_reason each originate | D5 · errors · who catches what |
| Why validation (incl. the tool contract) is pre-model & unbilled | D4 · tool design |
| Prompt assembly order (tools→system→messages) & caching | D5 · caching & cost |
| How stop_reason is decided; prefill vs decode | D1 · stop reasons |
| Statelessness, usage fields, streaming, metering | D1 · D5 |
LEVEL IT UP make the shell actually run
- Wrap a local model → replace
model.generate()with an Ollama call and servePOST /v1/messageswith FastAPI — a toy Messages-API-compatible server you can point the Anthropic SDK at (base_url). - Implement one real stage → e.g. a genuine
validate()that reproduces the 400 rules, or a real token-bucket rate limiter. - Add streaming → emit the SSE event sequence (
message_start → content_block_delta → message_stop). - Add a prefix cache → hash the stable prefix and skip re-work on a hit; watch your own
cache_readcounter.