AI Engineering Projects › Second-Brain Assistant (tiered memory)

🧠 Second-Brain Assistant — Tiered Memory & Context Engineering

A toy that forces every memory store + context technique to earn its place · Claude-native

THE PROJECT Why this one teaches everything

A long-running personal assistant you talk to across many sessions. It remembers facts, recalls your notes semantically, knows how people/projects relate, and manages long conversations. Each capability maps to a distinct, justified store — so you learn when to reach for each, not just how. The dialogue itself is where every context-bloat technique lives.

Deliberately toy & light/embedded (SQLite + Chroma + Kùzu in-process; Redis via brew or fakeredis) so you learn the concepts and APIs with almost no RAM/CPU cost — no Docker/Neo4j/Postgres tax.

WHAT IS THIS? The problem it solves

It's a personal work assistant — a "chief of staff" / second brain for one knowledge worker (you). A plain chatbot forgets everything when a session ends, so you re-explain your projects, people, and preferences every time. This assistant gives you continuity: it remembers your facts & preferences, recalls your past notes, tracks the people/projects you work with and how they connect, and keeps long conversations coherent. The value is recall across time — not a smarter single answer.

Not customer-facing — a private companion that keeps your working context so you never start from zero. The same shape generalises (a support agent = per-customer memory; a research assistant = a knowledge base), but here the "user" is one person, over months.

WALK-THROUGH A real conversation — what's stored where, and why

Two sessions, a few days apart. Watch each memory tier earn its place. SQLite Vector Graph Redis Context

Session 1 · Monday · Turn 1
YouI'm kicking off the ABCCompany observability demo with Priya and Rahul. Priya's leading it; target is next Friday.
App reads: fresh context — the router finds little to recall (facts/graph near-empty). → prompt = cached persona + (empty memory) + your turn.
Assistant (Claude)Got it — ABCCompany observability demo, Priya leading, Rahul on it, due next Friday. Want me to track blockers as they surface?
Reflect & write (async, after replying):
Graph nodes Priya, Rahul, Project "ABCCompany observability demo"; edges Priya-LEADS→, Rahul-WORKS_ON→, due=next Fri — why: roles/links you'll traverse later.
Vector episode "kicked off ABCCompany demo — Priya lead, Rahul, due next Fri" — why: semantic recall of the note later.
SQLite fact current_focus = "ABCCompany observability demo" — why: cheap exact lookup.
Redis session turn appended with a TTL — why: fast working memory.
Turn 2
YouRemember I prefer metrics dashboards over PDF reports.
AssistantNoted — I'll default to dashboards.
Write: SQLite preferences.output_format = "dashboard" — an explicit "remember" → a durable preference, upserted by key so a later change overrides it (no contradictions). why structured: exact, deterministic, shapes every future output.
Turn 3
YouPull the latest error logs for the ABCCompany ingestion job and summarise.
Tool use: Claude calls fetch_logs → the tool returns a large payload (context bloat).
AssistantThe job is failing on a null customer_id at line 42 — 3 rows affected.
Manage & write:
Context the big fetch_logs tool_result is cleared after use (context editing) — the dialogue stays intact. why: tool-result bloat.
Vector episode "ABCCompany ingestion failing: null customer_id, line 42 (Mon)".
Redis the summary is cached, key = hash(query + memory_version) — why: skip re-running an identical ask.
— session ends — Redis working memory expires (TTL). Durable memory (SQLite, Vector, Graph) persists.
Session 2 · Thursday (new session — working memory gone, durable memory persists)
YouWho's on the ABCCompany demo again, and when's it due?
Reads: router → Graph (who) + SQLite (due date). Graph traversal returns Priya (lead), Rahul; due = Friday. Fused into a small memory block placed after the cached persona.
AssistantPriya's leading it, Rahul's on it, and it's due this Friday.
why graph: "who's on X" needs traversal, not similarity — a vector store can't answer it.
Turn 5
YouWhat did I say was breaking in that ingestion job?
Reads: router → Vector (semantic recall). Top-K returns Monday's episode "null customer_id, line 42" above the similarity threshold.
AssistantYou noted it was failing on a null customer_id at line 42.
why vector: similarity recall of a past note — no exact key, no relationship to traverse.
Turn 6 · (many turns later in this session)
Context the dialogue has grown long → compaction summarises the earlier turns into a summary block (the full response.content is appended back). The detail is still safe in the Vector store. why: keep the window under the limit without losing recall.
Turn 7
YouActually the demo moved to the following Monday.
AssistantUpdated — the ABCCompany demo is now due Monday.
Write (handle staleness): SQLite due_date upserted (Friday → Monday, latest-wins, not appended) · Graph edge due_date updated · memory_version bumped → any cached answer about the due date is invalidated. why: you must never get the stale Friday date back.

So — what lives where, and why (from this flow)

StoreEarned its place atBecause
SQLitepref (T2), due date (T4, updated T7)exact facts you look up by key and overwrite
Vectorrecall (T5)"what did I say about…" — similarity recall of notes
Graphwho's-on (T4)traversal over people/projects — not similarity
Redissession & cache (T1, T3)fast ephemeral working memory + response cache with TTL
Context (API)big logs (T3), long session (T6)context editing (tool bloat) + compaction (dialogue)

PART A Conceptual architecture — the memory tiers

TierStoreHoldsRetrievalTeaches
Working / sessionRediscurrent session state, scratchpad, response cache, rate-limitby session key + TTLshort-term memory, caching, eviction
Profile / factsSQLiteprofile, preferences, durable key factsexact key / SQLstructured memory, write-vs-skip
Episodic / semanticVector (Chroma)past notes, chat snippets, docssimilarity (RAG)embeddings, chunking, recall
RelationalGraph (Kùzu)people · projects · entities & linkstraversal / Cyphergraph memory, multi-hop
In-conversationthe Claude APIthe live dialogue—compaction · context editing · caching · truncation

The turn lifecycle (this is the whole design)

flowchart TB U["User message"] --> RT{"Route: which memories matter?"} RT -->|"facts/prefs"| SQ[("SQLite")] RT -->|"'what did I say about…'"| VEC[("Vector / Chroma")] RT -->|"'who works with…'"| GR[("Graph / Kùzu")] RT -->|"session state"| RED[("Redis")] SQ --> FUSE VEC --> FUSE GR --> FUSE RED --> FUSE FUSE["Fuse → a small, bounded MEMORY CONTEXT block"] --> PROMPT["Assemble prompt:
[cached persona/tools] + memory block + dialogue"] PROMPT --> LLM["Claude (adaptive thinking, tools)"] LLM --> ANS["Reply to user"] LLM -.reflect.-> WR{"Anything worth remembering?"} WR -->|"durable fact"| SQ WR -->|"note/episode"| VEC WR -->|"new entity/link"| GR classDef s fill:#eef6ff,stroke:#3b7bd0,color:#1f5fa8; class FUSE,PROMPT,LLM s;

Two halves: read (route → retrieve → fuse → prompt) before the model answers, and write (reflect → persist) after. Keeping the fuse step small is the entire context-engineering game.

PART B The light/embedded stack

ResponsibilityPickWhy (footprint)
LLMClaude API — Opus 4.8 main, Haiku 4.5 for routing/extractionClaude-native; tier by task
Working/session + cacheRedis (brew) or fakeredis (in-process)tiny native, or zero-server for pure learning
Structured factsSQLite (stdlib)no server
Vector / embeddingsChroma (embedded) + sentence-transformers (local, free) or Voyage (prod)Claude has no embedding model — use a separate embedder
GraphKùzu (embedded, Cypher-like)no JVM/Neo4j server
Orchestrationplain Python + Anthropic SDK (manual loop)see the agent loop lessons (D1/D4)

PART C Memory implementation — per store

Redis — working/session memory + cache D5
SQLite — structured facts & preferences
Vector (Chroma) — episodic / semantic recall RAG
Graph (Kùzu) — relationships & multi-hop

PART D Context engineering — the heart of it

LeverWhat you buildClaude / CCA
Memory routingA cheap classifier (rules or Haiku) decides which stores a turn needs — or query all in parallel and fuse. Skipping stores saves tokens & latency.D1/D5
What to remember (write policy)A "reflect" step after each turn extracts durable facts/entities and routes them to the right store — and skips the ephemeral chatter. Not everything is worth remembering.D3
Retrieval fusionMerge results from 4 stores into ONE small memory context block with a per-store token budget (e.g. top-3 vector, N facts, a summarised graph neighbourhood). Place it AFTER the cached prefix.D5
Prompt cachingCache the stable persona + instructions + tool defs (byte-stable prefix); the volatile memory block & user turn go after the last breakpoint.prompt caching D5
Context editingTool calls (web fetch, doc read) return big payloads → clear stale tool_results while keeping the dialogue.context editing D5
CompactionWhen a single long session nears the window, summarise the dialogue (append response.content!). The detail is safe in the vector store.compaction D5
Safe truncationFallback sliding window that never orphans tool_use/tool_result pairs.D5
Golden rule of the design: durable knowledge lives in the stores; the context window holds only a small, fused, per-turn slice. The window is a workspace, not a filing cabinet.

PART E Failure points & challenges

FailureCauseMitigation
Memory poisoningThe model writes a wrong/harmful "fact" — recalled foreverValidate on write · store confidence + provenance · confirm high-stakes writes · decay/TTL low-confidence memories
Stale / conflicting factsA preference changed but the old one lingersUpsert by key (latest-wins) · timestamps · contradiction detection at write
Wrong-memory recallVector returns off-topic snippets → distraction & hallucinationSimilarity threshold · rerank · cap count · relevance filter before injecting
PII / secrets leakSensitive data written to memory, replayed into every sessionNever store secrets · redact/classify on write · encrypt at rest · per-user isolation
Cross-store inconsistencySame entity in SQLite, graph, and vector divergesA canonical entity id · one source of truth per fact type · write-through / reconciliation
Stale cacheResponse cache returns an answer after memory changedKey the cache by (query + memory version); bump the version on any write
Compaction lossThe summary drops a detail you neededCompact only the dialogue; keep raw episodes in the vector store; verify
LatencyQuerying 4 stores every turnRoute (skip stores) · query in parallel · cache · async writes (reflect after replying)
Cost / unbounded growthEmbeddings + storage grow foreverModel-tier (Haiku for routing/extraction) · dedup · decay/evict low-value memories · cache
Cold startEmpty memory for a new userDegrade gracefully; don't fabricate; ask & learn

PART F Cross-cutting design

ConcernHere
Security & privacyPer-user memory isolation, PII redaction on write, encryption at rest, a "forget me" path (GDPR-style delete across all four stores).
CostPrompt-cache the persona; Haiku for routing/extraction; dedup + decay memories; response cache; bounded per-store retrieval budget.
LatencyParallel store reads; route to skip stores; write memories asynchronously after the reply; cache.
ReliabilityAny store down → degrade (skip that memory tier, don't fail the turn); retries/backoff on the API; loop termination.
ObservabilityTrace each turn: which stores hit, what was retrieved & injected, tokens (incl. cache_read), what was written, latency per store.
Evals"Did it remember correctly?" — a golden set of (setup writes → later question → expected recall); measure recall precision, wrong-memory injection rate, and answer quality.

CERT MAP Claude features this exercises

In this projectClaude featureCCA
The read→generate→write loop, routing, reflect step, graceful degradationAgentic loopD1
Memory read/write tools; the Claude memory tool as one write path; is_error handlingTool use / memory toolD4/D5
Persona & instruction design; write-vs-skip; clarify; extraction promptsPrompt engineeringD3
Prompt caching · context editing · compaction · truncation · retrieval fusion · evals · observabilityContext & reliabilityD5
This project is the perfect companion to the D5 material: Caching & Memory · Context-bloat cheat sheet · runnable techniques script.