AI Engineering Projects › GenericAnthropicMessageClient

🧰 GenericAnthropicMessageClient — one robust client, every case

A design-patterns study: handle every request/response permutation of the Messages API behind one interface

THE IDEA The client-side twin of the server

If MyOwnAnthropicServer is everything the server does, this is everything the caller should do to be robust: handle all 6 stop reasons, the full error taxonomy, retries + backoff + idempotency, rate limits, token counting → max_tokens fit, context-bloat (trim / edit / compact), streaming, batch, structured outputs, tool loops, caching, and multi-backend routing — all behind one call. Your instinct is right: with code-to-interface design this collapses into a small set of pluggable pieces.

🎯 The lesson isn't "reinvent the SDK" — it's that a robust caller is a pipeline of policies, and each policy is an interface you can swap. That framing is exactly what the exam's reliability questions reward.

THE CASE MATRIX everything one call must survive

DimensionThe permutations to handle
stop_reasontool_use · end_turn · max_tokens · stop_sequence · pause_turn · refusal
Errors400 (never retry) · 401/403 · 413 · 429 (+Retry-After) · 500/529 (retry)
Retriesexponential backoff + jitter · honor Retry-After · idempotency on side effects
Token budgetcount_tokens pre-flight · shrink input or lower max_tokens to fit the window
Context bloatsliding window · context editing (clear tool results) · compaction
Execution modesync · streaming (SSE) · batch (~50%, async)
Outputfree text · structured (output_config.format) · tool arguments
Cost leversprompt caching breakpoints · model tier · effort / adaptive thinking
BackendAnthropic direct · Databricks Model Serving · Bedrock · Vertex
Try to handle all of this with if/elif in one function and you get a 400-line unmaintainable blob. Code-to-interface turns each row into a strategy you plug in.

DOES THIS EXIST? yes — in layers (and Tool Runner is one slice)

LayerAlready handlesGap it leaves
Anthropic SDKauto-retry 429/5xx + backoff · streaming helpers · messages.parse() · count_tokens · Batch APIyou still orchestrate stop_reasons, context, budgets, fallbacks
Tool Runner beta.messages.tool_runnerthe agentic loop — tool_use routing, execute, tool_result, per-turn hooksonly the tool loop — not batch / context-bloat / multi-backend
LiteLLM · AI gateways (Portkey, Cloudflare, Kong, Helicone)retries · fallbacks · rate-limit handling · caching · unified multi-provider APIgeneric & provider-agnostic — not Anthropic-shaped policies
Frameworks (LangChain/LangGraph, Instructor)orchestration, structured outputs, memoryheavy for "just a robust client"
So: the pieces exist; a thin house Facade that binds them to your policies (and treats all 6 stop_reasons + context-bloat + token-fit as first-class) is common in real codebases — and the best way to learn the surface. Tool Runner is the first-party attempt at the agentic-loop slice of exactly this idea.

THE DESIGN Facade + Middleware pipeline + Strategies

Three patterns do the heavy lifting: a Facade (one entry point), a Chain-of-Responsibility middleware pipeline (cross-cutting steps as an onion), and Strategy interfaces (each concern swappable).

flowchart LR APP["your code"] --> F["MessageClient (Facade)
send() · stream() · submit_batch()"]:::f F --> P["Middleware pipeline"]:::p subgraph P2["pipeline (onion order)"] direction TB m1["TokenBudget / ContextFit"] --> m2["CacheAnnotate"] --> m3["RateLimit"] --> m4["→ Backend.call ←"] --> m5["Retry"] --> m6["StopReasonRouter"] --> m7["OutputParser"] end P --> P2 P2 -. plugs in .-> STRAT["Strategies:
RetryPolicy · ContextPolicy · StopReasonHandler
OutputParser · BackendAdapter · ExecutionMode"]:::s classDef f fill:#e7ecff,stroke:#3b5bdb,color:#1e3a8a; classDef p fill:#f3f0ff,stroke:#764ba2,color:#4a3b7a; classDef s fill:#fff4e0,stroke:#b5691a,color:#7a4a0a;

PATTERN MAP concern → pattern

ConcernDesign pattern
One entry point hiding all complexityFacade
Cross-cutting steps (count, cache, rate-limit, retry, route, parse)Chain of Responsibility / middleware
Swappable retry / context / parse behaviourStrategy
Wrapping the base client with retry/logging/cacheDecorator
The agent loop skeleton with hook pointsTemplate Method + Observer (hooks)
Routing on stop_reasonState / dispatch table
Many backends behind one interfaceAdapter
Tool calls as first-class objectsCommand
Choosing sync / stream / batchStrategy + Factory
Fall back to another model/provider on failureChain of Responsibility

THE INTERFACES code-to-interface (Python Protocols)

from typing import Protocol, Iterator

# ADAPTER — one interface over every backend (Anthropic / Databricks / Bedrock).
class Backend(Protocol):
    def create(self, req) -> "Response": ...
    def stream(self, req) -> Iterator["Event"]: ...
    def count_tokens(self, req) -> int: ...

# STRATEGY — retry (429/5xx only; NEVER 400), backoff + jitter, honor Retry-After.
class RetryPolicy(Protocol):
    def should_retry(self, err, attempt) -> bool: ...
    def delay(self, err, attempt) -> float: ...

# STRATEGY — fit the request into the window BEFORE sending.
class ContextPolicy(Protocol):
    def fit(self, req, n_tokens, window) -> "Request": ...   # trim / clear tool_results / compact / lower max_tokens

# STATE / dispatch — turn a stop_reason into the next Action.
class StopReasonHandler(Protocol):
    def handle(self, resp, ctx) -> "Action": ...
        # tool_use   -> run tools, append tool_result, CONTINUE
        # end_turn   -> DONE
        # max_tokens -> CONTINUE (or raise cap) — not done!
        # refusal    -> RETURN graceful decline
        # pause_turn -> RESUME (resend)          | stop_sequence -> DONE

# STRATEGY — text / structured(schema) / tool-args.
class OutputParser(Protocol):
    def parse(self, resp): ...

# CHAIN OF RESPONSIBILITY — each step wraps the next (onion).
class Middleware(Protocol):
    def __call__(self, req, nxt) -> "Result": ...

# FACADE — one entry point; composes a backend + a pipeline + policies.
class MessageClient:
    def __init__(self, backend: Backend, pipeline: list[Middleware],
                 retry: RetryPolicy, context: ContextPolicy,
                 router: StopReasonHandler, parser: OutputParser): ...
    def send(self, req) -> "Result": ...        # runs the pipeline; loops on the router until DONE
    def stream(self, req) -> Iterator: ...       # same policies, streamed
    def submit_batch(self, reqs) -> "BatchHandle": ...  # same request shape, async ~50% cost
Every hard case from the matrix is now a named interface with one job. Swapping "sliding window" for "compaction" is a one-line policy change, not a rewrite — that's the whole point of coding to interfaces.

THE HEART stop_reason as a state machine

The messiest part — the loop — becomes a tiny dispatch once StopReasonHandler owns it:

stateDiagram-v2 [*] --> Call Call --> Route: 200 + stop_reason Route --> RunTools: tool_use RunTools --> Call: append tool_result Route --> Continue: max_tokens (truncated) Continue --> Call Route --> Resume: pause_turn Resume --> Call Route --> Done: end_turn / stop_sequence Route --> Decline: refusal Done --> [*] Decline --> [*]
The exam trap this design kills: forgetting a stop_reason. A dispatch table with a required case for all six (and a catch-all) makes "silently loop on max_tokens" impossible by construction.

VS TOOL RUNNER what each covers

🛠️ Tool Runner (first-party)

  • Automates the tool-use loop for tools you define
  • Per-turn hooks (approval, interception, result edit, retries)
  • Streaming, compaction support
  • Zero loop code — great for production agents

🧰 GenericMessageClient (your Facade)

  • Superset scope: also batch, context-fit, multi-backend, fallbacks, output-parse strategy
  • You own the policy interfaces — house rules in one place
  • More to build & maintain
  • Best as a thin wrapper that uses the SDK/Tool Runner inside
Recommended reality: don't reimplement retries/streaming the SDK already does — compose. Let the Facade delegate the tool loop to Tool Runner, retries to the SDK, and add only the policies they don't cover (context-fit, batch routing, fallbacks, budget). Build-your-own is for learning the surface; ship a thin wrapper.

CONCEPTS → CCA MAP

The client policy…CCA domain
All 6 stop_reasons as a dispatch/state machineD1 · stop reasons
Retry 429/5xx not 400 · backoff+jitter · idempotencyD5 · reliability
count_tokens → fit window; context editing / compactionD5 · context & tokens
Caching breakpoints · batch · model tier · effortD5 · cost
Tool loop, structured outputs, the tool contractD4 · tool design

LEVEL IT UP

🧩 The cross-cutting concerns (logging, tracing, cost, redaction, caching, audit…) are the client's constructor dependencies — each a pluggable port. Designed here: Cross-Cutting Concerns — pluggable ports.