THE IDEA The client-side twin of the server
If MyOwnAnthropicServer is everything the server does, this is everything the caller should do to be robust: handle all 6 stop reasons, the full error taxonomy, retries + backoff + idempotency, rate limits, token counting → max_tokens fit, context-bloat (trim / edit / compact), streaming, batch, structured outputs, tool loops, caching, and multi-backend routing — all behind one call. Your instinct is right: with code-to-interface design this collapses into a small set of pluggable pieces.
THE CASE MATRIX everything one call must survive
| Dimension | The permutations to handle |
|---|---|
| stop_reason | tool_use · end_turn · max_tokens · stop_sequence · pause_turn · refusal |
| Errors | 400 (never retry) · 401/403 · 413 · 429 (+Retry-After) · 500/529 (retry) |
| Retries | exponential backoff + jitter · honor Retry-After · idempotency on side effects |
| Token budget | count_tokens pre-flight · shrink input or lower max_tokens to fit the window |
| Context bloat | sliding window · context editing (clear tool results) · compaction |
| Execution mode | sync · streaming (SSE) · batch (~50%, async) |
| Output | free text · structured (output_config.format) · tool arguments |
| Cost levers | prompt caching breakpoints · model tier · effort / adaptive thinking |
| Backend | Anthropic direct · Databricks Model Serving · Bedrock · Vertex |
if/elif in one function and you get a 400-line unmaintainable
blob. Code-to-interface turns each row into a strategy you plug in.DOES THIS EXIST? yes — in layers (and Tool Runner is one slice)
| Layer | Already handles | Gap it leaves |
|---|---|---|
| Anthropic SDK | auto-retry 429/5xx + backoff · streaming helpers · messages.parse() · count_tokens · Batch API | you still orchestrate stop_reasons, context, budgets, fallbacks |
Tool Runner beta.messages.tool_runner | the agentic loop — tool_use routing, execute, tool_result, per-turn hooks | only the tool loop — not batch / context-bloat / multi-backend |
| LiteLLM · AI gateways (Portkey, Cloudflare, Kong, Helicone) | retries · fallbacks · rate-limit handling · caching · unified multi-provider API | generic & provider-agnostic — not Anthropic-shaped policies |
| Frameworks (LangChain/LangGraph, Instructor) | orchestration, structured outputs, memory | heavy for "just a robust client" |
THE DESIGN Facade + Middleware pipeline + Strategies
Three patterns do the heavy lifting: a Facade (one entry point), a Chain-of-Responsibility middleware pipeline (cross-cutting steps as an onion), and Strategy interfaces (each concern swappable).
send() · stream() · submit_batch()"]:::f F --> P["Middleware pipeline"]:::p subgraph P2["pipeline (onion order)"] direction TB m1["TokenBudget / ContextFit"] --> m2["CacheAnnotate"] --> m3["RateLimit"] --> m4["→ Backend.call ←"] --> m5["Retry"] --> m6["StopReasonRouter"] --> m7["OutputParser"] end P --> P2 P2 -. plugs in .-> STRAT["Strategies:
RetryPolicy · ContextPolicy · StopReasonHandler
OutputParser · BackendAdapter · ExecutionMode"]:::s classDef f fill:#e7ecff,stroke:#3b5bdb,color:#1e3a8a; classDef p fill:#f3f0ff,stroke:#764ba2,color:#4a3b7a; classDef s fill:#fff4e0,stroke:#b5691a,color:#7a4a0a;
PATTERN MAP concern → pattern
| Concern | Design pattern |
|---|---|
| One entry point hiding all complexity | Facade |
| Cross-cutting steps (count, cache, rate-limit, retry, route, parse) | Chain of Responsibility / middleware |
| Swappable retry / context / parse behaviour | Strategy |
| Wrapping the base client with retry/logging/cache | Decorator |
| The agent loop skeleton with hook points | Template Method + Observer (hooks) |
Routing on stop_reason | State / dispatch table |
| Many backends behind one interface | Adapter |
| Tool calls as first-class objects | Command |
| Choosing sync / stream / batch | Strategy + Factory |
| Fall back to another model/provider on failure | Chain of Responsibility |
THE INTERFACES code-to-interface (Python Protocols)
from typing import Protocol, Iterator # ADAPTER — one interface over every backend (Anthropic / Databricks / Bedrock). class Backend(Protocol): def create(self, req) -> "Response": ... def stream(self, req) -> Iterator["Event"]: ... def count_tokens(self, req) -> int: ... # STRATEGY — retry (429/5xx only; NEVER 400), backoff + jitter, honor Retry-After. class RetryPolicy(Protocol): def should_retry(self, err, attempt) -> bool: ... def delay(self, err, attempt) -> float: ... # STRATEGY — fit the request into the window BEFORE sending. class ContextPolicy(Protocol): def fit(self, req, n_tokens, window) -> "Request": ... # trim / clear tool_results / compact / lower max_tokens # STATE / dispatch — turn a stop_reason into the next Action. class StopReasonHandler(Protocol): def handle(self, resp, ctx) -> "Action": ... # tool_use -> run tools, append tool_result, CONTINUE # end_turn -> DONE # max_tokens -> CONTINUE (or raise cap) — not done! # refusal -> RETURN graceful decline # pause_turn -> RESUME (resend) | stop_sequence -> DONE # STRATEGY — text / structured(schema) / tool-args. class OutputParser(Protocol): def parse(self, resp): ... # CHAIN OF RESPONSIBILITY — each step wraps the next (onion). class Middleware(Protocol): def __call__(self, req, nxt) -> "Result": ... # FACADE — one entry point; composes a backend + a pipeline + policies. class MessageClient: def __init__(self, backend: Backend, pipeline: list[Middleware], retry: RetryPolicy, context: ContextPolicy, router: StopReasonHandler, parser: OutputParser): ... def send(self, req) -> "Result": ... # runs the pipeline; loops on the router until DONE def stream(self, req) -> Iterator: ... # same policies, streamed def submit_batch(self, reqs) -> "BatchHandle": ... # same request shape, async ~50% cost
THE HEART stop_reason as a state machine
The messiest part — the loop — becomes a tiny dispatch once StopReasonHandler owns it:
stop_reason. A dispatch table with a
required case for all six (and a catch-all) makes "silently loop on max_tokens" impossible by construction.VS TOOL RUNNER what each covers
🛠️ Tool Runner (first-party)
- Automates the tool-use loop for tools you define
- Per-turn hooks (approval, interception, result edit, retries)
- Streaming, compaction support
- Zero loop code — great for production agents
🧰 GenericMessageClient (your Facade)
- Superset scope: also batch, context-fit, multi-backend, fallbacks, output-parse strategy
- You own the policy interfaces — house rules in one place
- More to build & maintain
- Best as a thin wrapper that uses the SDK/Tool Runner inside
CONCEPTS → CCA MAP
| The client policy… | CCA domain |
|---|---|
| All 6 stop_reasons as a dispatch/state machine | D1 · stop reasons |
| Retry 429/5xx not 400 · backoff+jitter · idempotency | D5 · reliability |
| count_tokens → fit window; context editing / compaction | D5 · context & tokens |
| Caching breakpoints · batch · model tier · effort | D5 · cost |
| Tool loop, structured outputs, the tool contract | D4 · tool design |
LEVEL IT UP
- Build the vertical slice → Facade + a
RetryPolicy+ aStopReasonHandlerover the real SDK. That alone is a great portfolio piece. - Add a ContextPolicy → plug in sliding-window, then swap to context-editing, then compaction — same interface.
- Add a BackendAdapter → one for Anthropic-direct, one for Databricks Model Serving (base_url) — prove the Adapter.
- Add fallbacks → a chain: Opus → Sonnet → cached — Chain of Responsibility in action.
- Wrap Tool Runner → let the Facade delegate the loop; you only own the policies around it.