AI Engineering Projects β€Ί GenericMessageClient β€Ί Cross-Cutting Concerns

🧩 Cross-Cutting Concerns β€” pluggable ports

Keep logging, tracing, cost, security & more OUT of the core β€” behind interfaces, with swappable adapters

THE IDEA A concern is cross-cutting when it touches every call but isn't the logic

Logging, tracing, metrics, cost, redaction, caching, rate limits, audit β€” none of these are "call the model," yet every request needs them. If you scatter them through the core you get an unmaintainable, untestable mess. The fix is the same principle you liked: code to interface. The core depends only on a port (e.g. LogSink); where it physically logs is a swappable adapter (stdout / CloudWatch / Azure Monitor). That's ports-and-adapters (hexagonal) architecture + dependency injection, applied to GenericAnthropicMessageClient.

🎯 Two rules make this work: (1) the core imports interfaces, never concrete sinks (Dependency-Inversion). (2) The concrete adapters are wired in one place β€” the composition root. Swapping "log to file" for "log to CloudWatch" is then a one-line change, and tests inject fakes.

THE SHAPE core depends on ports; adapters plug in

flowchart LR CORE["MessageClient CORE
pure logic Β· depends only on PORTS"]:::core CORE --> L["port: LogSink"]:::p --> LA["stdout Β· JSON file Β· CloudWatch Β· Azure Monitor Β· Loki"] CORE --> T["port: Tracer"]:::p --> TA["Phoenix Β· Langfuse Β· MLflow Β· App Insights Β· X-Ray"] CORE --> M["port: MetricSink"]:::p --> MA["Prometheus Β· StatsD Β· CloudWatch Β· Cloud Monitoring"] CORE --> B["port: Budget"]:::p --> BA["in-mem counter Β· Redis Β· Delta ledger"] CORE --> S["port: SecretProvider"]:::p --> SA["dotenv Β· Key Vault Β· Secrets Manager Β· secret scopes"] CORE --> AU["port: AuditSink"]:::p --> AUA["JSONL Β· Log Analytics Β· CloudTrail Β· Delta"] classDef core fill:#e7ecff,stroke:#3b5bdb,color:#1e3a8a; classDef p fill:#f3f0ff,stroke:#764ba2,color:#4a3b7a;

The core never names a concrete sink. Point it at a different adapter and behaviour changes with zero core edits β€” the Dependency-Inversion Principle in practice.

THE FULL SET every cross-cutting concern, as a port

ConcernWhat it does on every callPort interfacePattern
Observability
πŸ“ LoggingStructured record of request/response/outcomeLogSink.emit()Port/Adapter
πŸ”Ž TracingA span per request & per loop step (parent/child)Tracer.span()Port + Observer
πŸ“Š Metricstokens, cost, latency, retries, cache-hit, successMetricSink.incr/observe()Port/Adapter
🧾 Audit (GxP)Immutable, provable record (who/what/when + versions)AuditSink.record()Port/Adapter
Cost & governance
πŸ’° Budget / cost accountingEstimate + enforce per-request/session/tenant spendBudget.check/record()Strategy + Port
🚦 Rate limiting / concurrencyClient-side throttle so you don't self-inflict 429sRateLimiter.acquire()Port/Adapter
πŸŽ›οΈ Feature flags / kill switchEnable/disable a model/tool/agent instantlyFlagSource.enabled()Port/Adapter
🏷️ Prompt / model versioningPin & log the exact prompt+model for reproducibilityPromptRegistry.get()Port/Adapter
Security & safety
πŸ” SecretsFetch keys/tokens from a vault, never hard-codeSecretProvider.get()Port/Adapter
🩹 Redaction (PII/secrets)Scrub sensitive data before log/trace/modelRedactor.scrub()Port + Decorator
πŸ›‘οΈ GuardrailsInput/output moderation & injection filteringGuardrailPolicy.check_in/out()Strategy + Port
Resilience & performance
♻️ Retry / circuit-breaker / fallbackSurvive transient errors; degrade gracefullyRetryPolicy, FallbackStrategy + Chain of Resp.
πŸ” IdempotencyKeys so a retry can't double-execute a side effectIdempotencyStorePort/Adapter
πŸ—„οΈ CachingClient-side response/semantic cache (beyond prompt caching)CacheStore.get/put()Port + Decorator
Configuration
βš™οΈ Config sourceTimeouts, model, budgets, TTLs β€” from env/file/vault/dynamicConfigSource.get()Port/Adapter
πŸ”Œ Backend / identityWhich endpoint + how to auth (key / OAuth / cloud IAM)Backend (adapter)Adapter

THE PORTS interfaces the core depends on (Python Protocols)

from typing import Protocol, Any

class LogSink(Protocol):
    def emit(self, record: dict) -> None: ...        # adapter decides WHERE (stdout/CloudWatch/...)

class Tracer(Protocol):
    def span(self, name: str, **attrs): ...             # context manager; nests per loop step

class MetricSink(Protocol):
    def incr(self, name: str, value: float = 1, **tags): ...
    def observe(self, name: str, value: float, **tags): ...

class Budget(Protocol):
    def check(self, est_tokens: int) -> bool: ...        # deny before the call if over
    def record(self, usage) -> None: ...              # debit after

class Redactor(Protocol):
    def scrub(self, obj: Any) -> Any: ...              # PII/secrets out BEFORE anything logs it

class SecretProvider(Protocol):
    def get(self, name: str) -> str: ...

class CacheStore(Protocol):
    def get(self, key: str): ...
    def put(self, key: str, value, ttl: int) -> None: ...

class AuditSink(Protocol):
    def record(self, event: dict) -> None: ...          # append-only, immutable

class RateLimiter(Protocol):
    def acquire(self, cost: int) -> None: ...          # block/throttle client-side

class GuardrailPolicy(Protocol):
    def check_input(self, req) -> None: ...
    def check_output(self, resp) -> None: ...

HOW THEY WEAVE IN the middleware order matters

Cross-cutting behaviour applies as middleware around the core call (Decorator/Chain-of-Responsibility). Order is a design decision, not an accident:

flowchart TB R["request"] --> RD["Redact (scrub secrets/PII)"] --> BG["Budget.check"] --> RL["RateLimiter.acquire"] RL --> CH["Cache.get (hit? short-circuit)"] --> GI["Guardrail.check_input"] --> TR["Tracer.span + Log start"] TR --> CALL["β†’ Backend.call (+ Retry) ←"]:::c --> GO["Guardrail.check_output"] --> BR["Budget.record + Metrics"] BR --> AU["Audit.record"] --> CP["Cache.put"] --> LOG["Log end"] --> OUT["result"] classDef c fill:#dcfce7,stroke:#15803d,color:#14532d;
Ordering traps: Redact must run first β€” before anything logs or traces the payload, or you leak PII into your telemetry. Budget/RateLimit before the call (no point paying to reject later). Cache before Guardrail-input? depends β€” cache the validated request. Getting the order right is half the design.

THE COMPOSITION ROOT wire concrete adapters in ONE place

# composition_root.py β€” the ONLY file that names concrete adapters (dependency injection)
client = MessageClient(
    backend = DatabricksBackend(host=cfg.host, token=secrets.get("DBX_TOKEN")),  # Adapter
    ports   = Ports(
        log    = CloudWatchLog()      if prod else StdoutLog(),        # swap WHERE we log, 1 line
        tracer = LangfuseTracer()     if prod else PhoenixTracer(),
        metric = PrometheusMetrics(),
        budget = RedisBudget(cap_usd=cfg.daily_cap),
        redact = PIIRedactor(),
        cache  = RedisCache()          if prod else InMemoryCache(),
        audit  = DeltaAudit()          if prod else JsonlAudit(),
        limiter= TokenBucket(rpm=cfg.rpm),
        guard  = CompositeGuardrail([InjectionFilter(), Moderation()]),
    ),
)
# Tests: pass FakeLog(), FakeBudget(), InMemoryCache() β€” no network, deterministic.
This is the payoff: the core logic is written once and never mentions CloudWatch, Redis, or Langfuse. Dev vs prod, local vs Azure, real vs fake β€” all decided here. That's testability and portability by construction.

PHYSICAL ADAPTERS the same port, per environment

PortπŸ’» LocalπŸ”· Azure🟧 AWSπŸ”΄ GCP🧱 Databricks
LogSinkstdout / JSON fileAzure Monitor / Log AnalyticsCloudWatch LogsCloud Loggingdriver logs / system tables
TracerPhoenix / LangfuseApp InsightsX-Ray + CloudWatchCloud TraceMLflow Tracing
MetricSinkPrometheus + GrafanaAzure MonitorCloudWatchCloud MonitoringMLflow / Lakehouse
SecretProviderdotenvKey VaultSecrets ManagerSecret ManagerDatabricks secret scopes
CacheStorein-memory / local RedisAzure Cache for RedisElastiCacheMemorystoreDelta / Redis
AuditSinkJSONL fileLog AnalyticsCloudTrail + S3Cloud Audit LogsDelta + system.access.audit
ConfigSourceenv / fileApp ConfigurationAppConfig / SSMRuntime Configjob params / widgets
Read a row to pick an adapter per environment; the core code is identical across all columns. This is the same "one interface, five homes" idea as the Text-to-SQL physical design.

CONCEPTS β†’ CCA MAP

The cross-cutting port…CCA domain
Logging, tracing, metrics, request-idD5 Β· observability
Budget, rate limiting, caching, prompt/model versioningD5 Β· cost
Secrets, redaction, guardrails (injection/moderation)D5 Β· guardrails Β· D4
Retry / circuit-breaker / fallback / idempotencyD5 Β· reliability
Audit trail (GxP), kill switchD5 Β· governance

LEVEL IT UP