AI Engineering Projects › MyOwnAnthropicServer

🏗️ MyOwnAnthropicServer — what the Messages API server actually does

Design your own minimal POST /v1/messages to see every responsibility Anthropic handles for you

THE IDEA The model is one line; the rest is the server

The best way to understand what the Anthropic server does is to sketch your own. When you do, the punchline jumps out: calling the LLM is a single line in the middle — auth, rate limits, validation, prompt assembly, caching, safety, response shaping, billing, and streaming are all the server's job, before and after that line. This page designs that shell (not the transformer — that stays a black box) so you can see the responsibilities.

🎯 This is the synthesis of the whole D5/D4 error & cost material: who catches what, cost, caching, and stop-reasons — all visible as stages of one request.

THE BIG PICTURE Three boxes, not one

People imagine "the API = the model." It's really three responsibilities, and you only own the outer one:

flowchart LR C["YOUR CLIENT
build messages · resend history
· loop on stop_reason · retry"]:::you S["ANTHROPIC SERVER
auth · limits · validate · assemble
· cache · safety · meter · stream"]:::srv M["THE MODEL
prefill + decode
(the transformer)"]:::mod C -->|"POST /v1/messages"| S -->|"prompt"| M M -->|"tokens"| S -->|"JSON response"| C classDef you fill:#e7ecff,stroke:#3b5bdb,color:#1e3a8a; classDef srv fill:#f3f0ff,stroke:#764ba2,color:#4a3b7a; classDef mod fill:#dcfce7,stroke:#15803d,color:#14532d;

MyOwnAnthropicServer is the middle box. The client box is what you already build (see the agent loop); the model box we leave as model.generate().

DO YOU ALWAYS NEED THE SERVER? Three deployment shapes

Common question: "can the agent talk to the LLM directly, with no Anthropic-style server?" Yes — but be precise about what "direct" means. The server is a bundle of responsibilities, not a mandatory box. Who performs them (and whether you need them at all) depends on where the model runs:

flowchart TB subgraph A["A · HOSTED API — the provider runs the server"] direction LR a1["Client / LangGraph"] --> a2["Provider server
all 13 stages"]:::srv --> a3["Model"]:::mod end subgraph B["B · SELF-HOSTED SERVER — you run it"] direction LR b1["Client / LangGraph"] --> b2["Your server
vLLM · TGI · Ollama"]:::srv --> b3["Model"]:::mod end subgraph C["C · IN-PROCESS — no server at all"] direction LR c1["Agent = the runtime
transformers · llama.cpp"] --> c3["Model (same process)"]:::mod end classDef srv fill:#f3f0ff,stroke:#764ba2,color:#4a3b7a; classDef mod fill:#dcfce7,stroke:#15803d,color:#14532d;
A · Hosted APIB · Self-hosted serverC · In-process / embedded
Backend examplesAnthropic, OpenAI, Bedrock, Vertex, Databricks Model ServingvLLM, TGI, Ollama, LM Studio, TritonHF transformers, llama-cpp-python, MLX
Who runs the "server"the provideryou (ops burden)nobody — it's a function call
Auth · limits · billing · safetyprovider handles ityou build/operate what you needusually N/A (single user)
Client examplesLangGraph, your app, the Anthropic SDKLangGraph, your appyour script
Truly "direct to the LLM"?No — HTTP to the provider's serverSort of — your own local/edge serverYes — same process, model.generate()
Trade-offzero ops, pay per token, rate limitscontrol & privacy, but you operate itsimplest, but you own scaling & safety
LangGraph doesn't remove the server. LangGraph / LangChain / CrewAI / LlamaIndex are client-side orchestrators — they decide what to call, not where the model runs. LangGraph → OpenAI is shape A (it calls OpenAI's API server over HTTP, which still does auth/limits/validation/safety). It only becomes "direct" when you point it at a local Ollama (B) or an in-process model (C).
The punchline: the 13 stages don't disappear in B and C — they either become your job (self-hosted, multi-user) or become unnecessary (single-user local: no auth, no rate limits, no billing). That's exactly why the hosted API exists — you rent those responsibilities instead of building them. MyOwnAnthropicServer is what you'd have to write to turn shape C back into shape A.

THE REQUEST LIFECYCLE 13 stages of one call

flowchart TD R["POST /v1/messages"] --> T["1 · parse HTTP/JSON"] --> A{"2 · auth"} A -->|bad| E401["401/403"]:::e A -->|ok| RL{"3 · rate limit"} RL -->|over| E429["429 +Retry-After"]:::e RL -->|ok| SZ{"4 · size"} SZ -->|too big| E413["413"]:::e SZ -->|ok| V{"5 · validate"} V -->|invalid| E400["400 (pre-model)"]:::e V -->|ok| RT["6 · route model"] --> AS["7 · assemble prompt + count tokens"] AS --> CA["8 · prompt cache lookup"] --> INF["9 · MODEL: prefill+decode"]:::m INF --> SF["10 · safety classifiers"] --> RS["11 · build response + stop_reason + usage"] RS --> MT["12 · meter + log"] --> OUT["13 · return / stream (SSE)"]:::ok classDef e fill:#fee2e2,stroke:#b91c1c,color:#7f1d1d; classDef m fill:#dcfce7,stroke:#15803d,color:#14532d; classDef ok fill:#dcfce7,stroke:#15803d,color:#14532d;

THE MINIMAL SERVER ~40 lines — one function per responsibility

# MyOwnAnthropicServer — a teaching shell for POST /v1/messages.
# The transformer is a black box (model.generate). Everything AROUND it is the server's job.

def handle_messages(http_request):
    # 1 · TRANSPORT — parse the HTTP body
    try:    body = json.loads(http_request.body)
    except: return err(400, "invalid_request_error", "malformed JSON")

    # 2 · AUTH — who is this? (x-api-key, anthropic-version)  → 401 / 403
    org = authenticate(http_request.headers.get("x-api-key"))
    if not org:                    return err(401, "authentication_error", "invalid key")
    if not org.can_use(body.get("model")): return err(403, "permission_error", "no access")

    # 3 · RATE LIMIT / QUOTA — token buckets per org (RPM/ITPM/OTPM)  → 429 + Retry-After
    if not rate_limiter.allow(org, body): return err(429, "rate_limit_error", retry_after=2)

    # 4 · SIZE — payload cap  → 413
    if too_large(body):        return err(413, "request_too_large", "body too large")

    # 5 · VALIDATE — the big 400 layer, BEFORE the model ever runs
    bad = validate(body)          # required fields · role alternation · tool_use<->tool_result
    if bad: return err(400, "invalid_request_error", bad)  # · max_tokens<=limit · no temperature/budget_tokens

    # 6 · ROUTE — resolve model id -> engine; check the context window  → 404 if unknown
    model = resolve_model(body["model"])

    # 7 · ASSEMBLE PROMPT — render tools -> system -> messages into model format; count tokens
    prompt, n_in = render(body)     # enforce max input (context-window-exceeded is a 400)

    # 8 · PROMPT CACHE — hash the stable prefix; reuse KV on a hit (skip prefill)
    kv, cache_read, cache_new = prompt_cache.lookup(prompt, body)

    # 9 · INFER — the ONE line that is "the model" (prefill + autoregressive decode)
    out = model.generate(prompt, kv,
        max_tokens      = body["max_tokens"],          # → stop_reason "max_tokens"
        stop_sequences  = body.get("stop_sequences"),  # → "stop_sequence"
        thinking        = body.get("thinking"),        # adaptive reasoning (billed as output)
        tools           = body.get("tools"),           # constrained decode → "tool_use"
        output_config   = body.get("output_config"))   # schema-constrained JSON

    # 10 · SAFETY — classifiers can override the output  → stop_reason "refusal"
    if safety.blocks(body, out): out = refusal()

    # 11 · RESPONSE — content blocks (text/thinking/tool_use) + stop_reason + usage
    resp = build(out, stop_reason=out.stop_reason,
        usage=usage(n_in, out.n_out, cache_read, cache_new))  # the 4 usage fields

    # 12 · METER + LOG — record billing; attach request-id for tracing
    meter(org, resp.usage); log(http_request.id, org, resp)

    # 13 · RETURN — one JSON body, or a stream of SSE events if body["stream"]
    return sse_stream(resp) if body.get("stream") else json_response(resp)
Read stages 1–8, then 10–13, and notice: only stage 9 is "the model." Everything else is the responsibility you get for free — and exactly what an exam question about "where does X happen?" is testing.

RESPONSIBILITY TABLE stage → what it does → what breaks there

#Server responsibilityFails withCCA tie-in
1HTTP transport & JSON parsing400 (malformed JSON)D4
2Auth — API key → org/workspace & model access401 / 403D5 errors
3Rate limiting — per-org token buckets (RPM/TPM)429 (+ Retry-After)D5 · rate limits
4Payload size limits413D5
5Validation — schema, message structure, tool contract, params400 (pre-model, unbilled)D4 · tool contract
6Model routing & capability/context-window check404 / 400D1
7Prompt assembly — render tools→system→messages; token counting400 (context exceeded)D5 · tokens
8Prompt caching — prefix hash, KV reuse, cache accounting—D5 · caching
9Inference — prefill + decode, stop conditions, tool/structured decode200 + stop_reasonD1 · stop reasons
10Safety — input/output classifiers200 + refusalD5
11Response assembly — content blocks, stop_reason, usage—D1 · D5
12Metering / billing / logging (request-id)—D5 · cost, obs
13Serialization & streaming (SSE events)500 / 529 on failureD5 · reliability

ZOOM: STAGE 9 what "call the LLM" actually hides

The one line model.generate() is itself a small pipeline — worth knowing for the exam even though you'd never build it:

SCOPE what we deliberately DON'T build

Not in scope: the transformer itself (attention, weights, the KV cache internals). That's the model, not the server — and reimplementing it teaches you nothing about the API's responsibilities. We keep it as an opaque model.generate(), which is exactly the boundary Anthropic exposes to you.

CLIENT vs SERVER the responsibility split (and why statelessness)

Your client doesThe server does
Build & resend the full message history every turnTreat each request as stateless — no session memory
Loop on stop_reason, run tools, return tool_resultDecode the tool call, stamp tool_use, validate the returned pairing
Retry 429/5xx with backoff; set idempotencyEnforce limits, return typed errors, meter usage
Decide model, max_tokens, caching breakpointsRoute, cache, assemble, infer, shape the response
Statelessness is a server responsibility you feel as a client cost: because the server keeps no session, you resend the whole history — which is why long chats get expensive (every past token re-billed). Build this shell and that trade-off stops being abstract.

CONCEPTS → CCA MAP

You now understand…CCA domain
Where 400 vs 401 vs 429 vs 200+stop_reason each originateD5 · errors · who catches what
Why validation (incl. the tool contract) is pre-model & unbilledD4 · tool design
Prompt assembly order (tools→system→messages) & cachingD5 · caching & cost
How stop_reason is decided; prefill vs decodeD1 · stop reasons
Statelessness, usage fields, streaming, meteringD1 · D5

LEVEL IT UP make the shell actually run

Doing even one stage for real turns the abstract "the API handles it" into muscle memory — which is exactly the level the CCA exam probes.