Skip to content

Instantly share code, notes, and snippets.

@kevinkiklee
Last active November 2, 2026 20:59
Show Gist options
  • Select an option

  • Save kevinkiklee/0962f62284afa3fab834f7a513b142c2 to your computer and use it in GitHub Desktop.

Select an option

Save kevinkiklee/0962f62284afa3fab834f7a513b142c2 to your computer and use it in GitHub Desktop.
Robin From Scratch — Design Specification

Robin From Scratch — Design Specification


Table of Contents

  1. What Is Robin?
  2. Phase 1: Foundation — Event Store + Scheduler
  3. Phase 2: Memory — Recall, Entities, Beliefs
  4. Phase 3: LLM Abstraction — Dispatcher & Providers
  5. Phase 4: Cognition — Capture, Biographer, Embedder, Dream
  6. Phase 5: Integrations — External Data Sources
  7. Phase 6: Surfaces — MCP, CLI, HTTP, Primer
  8. Phase 7: Agent System — Sandboxed Execution
  9. Phase 8: User Data & Extensions
  10. Hard-Won Lessons
  11. Glossary

1. What Is Robin?

Robin is a local-first personal AI assistant that learns about its user over time. It runs as a background daemon, captures interactions, pulls data from external services, and builds a structured understanding of the user's life — relationships, preferences, health, career, finances, creative work. It exposes this understanding to any AI coding tool (Claude Code, Cursor, Gemini CLI, etc.) via MCP servers.

A Typical Session

  1. The user opens their AI tool. It loads the primer — a compact context file (~2,500 tokens) with behavioral rules, active beliefs, a user profile, a knowledge index, and a learning digest.
  2. The user asks "what camera gear do I have?" The tool calls Robin's recall MCP tool; Robin searches the event store (lexical + vector) and returns ranked results.
  3. The user says "I sold the 24-120 last week." The tool calls believe with a topic and claim, superseding the old belief.
  4. The session ends; a hook posts the transcript to Robin's HTTP endpoint. The capture pipeline classifies it as personal and ingests it.
  5. On the next biographer tick, entities, relations, and belief candidates are extracted. Overnight, the dream job promotes qualifying candidates, updates entity profiles, and writes the daily journal.
  6. Tomorrow's primer reflects the updated knowledge.

Meanwhile, integrations poll external services (mail, calendar, health, finance) and the embedder vectorizes everything for semantic search — no user intervention.

Core Principles

  • Personal data only — Memory holds the user's life, not engineering artifacts. Enforced structurally by a closed personal-domain allowlist (Phase 4), not by ever-growing blocklists.
  • Local-first — All data lives on the user's machine. Cloud services are optional (LLM inference, external integrations).
  • Model-agnostic — Any LLM provider (cloud or local) behind capability-based routing.
  • Append-only events — Nothing is deleted, only superseded. Every belief, correction, and prediction is an immutable event with provenance.
  • Autonomous cognition — Background jobs extract, consolidate, and verify knowledge without user intervention.
  • Auditable — Every decision has a trail: event logs, agent transcripts, belief provenance, correction history.
  • Verified, not trusted — Autonomous work is checked by deterministic post-condition verifiers; claims of work without observable effects raise alerts.

What Robin Is NOT

Not a chatbot, web app, or cloud service. It has no conversational UI of its own. It is infrastructure that makes other AI tools smarter about the user.

What the Builder Needs

  • Persistent storage with full-text search (SQL + FTS or equivalent)
  • Vector similarity search (DB extension, separate index, or service)
  • At least one LLM for text generation and one for embeddings
  • A scheduling mechanism (long-running daemon preferred)
  • A process supervisor to restart the daemon on crash (launchd, systemd, Docker)
  • A way to expose tools to AI assistants (MCP servers or equivalent)
  • Structured logging (JSON)

Build Order

Phase Title Depends on
1 Foundation — Event Store + Scheduler —
2 Memory — Recall, Entities, Beliefs 1
3 LLM Abstraction —
4 Cognition 1, 2, 3
5 Integrations 1, 2
6 Surfaces 2, 5
7 Agent System 3, 6
8 User Data & Extensions all

Phases 1 and 3 are independent and can be built in parallel. Verify each phase's "Done when" checklist before starting the next — every later phase assumes the earlier ones actually work, and debugging cognition on top of a flaky scheduler wastes days.

Minimum viable Robin: the core MCP server needs only Phases 1–2. A thin slice — event store + lexical recall + remember/believe exposed over MCP + a hand-written primer file — is a usable memory system in days, before any LLM plumbing, integrations, or cognition exist. Build that slice first, use it daily, then layer the rest on. Only the extension server needs Phases 5 and 7.

Testing: write automated tests alongside each phase, collocated with the module. Anything time-dependent (deadlines, leases, staleness TTLs, cron) takes an injectable clock — wall-clock tests for this logic are flaky and slow. Treat each "Done when" item as an executable test, not a manual check.


2. Phase 1: Foundation

Goal: A running daemon with persistent storage, structured logging, config loading, and a job scheduler with crash recovery.

Configuration

Config Purpose
LLM config Role-to-provider mapping
Policies Power state, capture rules, network mode, agent budget caps, cognition kill-switches
Secrets API keys, OAuth tokens. Restrictive permissions, loaded into env, never logged.

Config files are human-editable (YAML/TOML/similar). Secrets are separate from non-secret config. Every process entry point (daemon, each MCP server, standalone scripts) must load secrets itself — see Lessons, §10.

Schema Migrations

Numbered, forward-only migration files applied transactionally at boot; the daemon records the applied version and refuses to run against a schema newer than it knows. Every later phase adds tables (candidates, ledger, alerts, habits, recommendations) — without a migration system from day one, each upgrade becomes manual surgery.

Event Store

An append-only log. Every meaningful occurrence becomes an event.

Event: id, ts (ISO-8601), kind (e.g. session.captured, belief.update, integration.tick), source, optional actor, status (ok/error/skipped), payload (JSON), optional content_ref (FK to content table).

Content table (separate so event scans never drag large blobs): id, ts, body (full text, FTS-indexed), embedding (NULL until the embedder fills it).

Nothing is deleted. Supersession is a new event referencing the prior event's ID.

Ingest interface: One function that writes an event + optional content body. Supports upsert-by-external-ID so re-ingesting the same external record (e.g. the same mail thread) updates in place instead of duplicating.

Scheduler

Tick-based; claims and executes one job at a time in-process.

Job: id, kind, status (pending/running/completed/failed), scheduled_at, leased_until, completed_at, error.

Behavior:

  • Ticks every 1 second; claims the oldest pending job with scheduled_at <= now (FIFO)
  • Sets a lease on the claimed job (protects against a second daemon instance; the owning daemon completes the job regardless of lease expiry, so the lease only needs to exceed dead-daemon detection time, ~5 min)
  • Executes the registered handler, records the result
  • Cron re-arming: after completion — success or failure — a cron job's next run is inserted. Transient failures can never permanently silence a schedule.
  • On boot, a new daemon proactively breaks expired leases rather than waiting for them to lapse

Crash recovery on boot:

  1. Detect and clear a stale PID file
  2. Reset jobs with expired leases (prior daemon died mid-run)
  3. Re-arm cron schedules that should have fired while down
  4. Handlers with partial progress (multi-chunk extraction) resume from their own persisted cursors — the scheduler only guarantees re-entry into the queue

Health Monitor

Runs every ~60s alongside the scheduler:

  • Verifies the scheduler heartbeat is recent; verifies DB reachability
  • A single CRITICAL observation is tolerated (slow handlers are legitimate); only sustained CRITICAL (30 min) triggers self-termination for supervisor respawn
  • Boot grace window (~2 min) suppresses false positives during startup
  • Maintains a persistent alert store: alerts are raised by invariant trips, integration failures, and verifier mismatches; they auto-resolve when the underlying condition next succeeds. Exposed via MCP/CLI so the user (and agents) can see open problems.

Power/Pause Gates

A configurable power state (active, paused, off) stored in policies, read/writable at runtime. When paused, the tick loop skips execution.

Phase 1 Done When

  • Daemon starts, loads config + secrets, opens DB, applies migrations (idempotently — a second boot is a no-op)
  • Cron handlers fire on time; jobs claim and execute FIFO
  • Jobs re-arm after completion, including after failure
  • Health monitor detects a stalled scheduler and terminates only after the sustained-critical window
  • Kill the daemon mid-job → restart → expired leases reset, crons re-arm
  • Structured logs for claims, completions, failures, health checks

3. Phase 2: Memory

Goal: Store, search, and structure knowledge about the user.

Recall (Search)

Mode How When
Lexical FTS on content bodies Exact terms, names
Vector Cosine similarity on embeddings Semantic queries
Hybrid Both, merged by reciprocal rank fusion Default

Vector search depends on the embedder (Phase 4); lexical-only recall is fully functional before that. Results are enriched with kind, age, source, and score. An optional rerank stage (the rerank LLM role) can re-score the top K — ship without it; hybrid fusion alone is good enough in practice.

Remember (Write)

The public write path: an event with kind, source, and content body. All external data enters through remember or the ingest interface.

Entity Graph

Entity: id, type (person, place, organization, service, topic, thing), canonical_name (case-normalized, deduped at storage), optional prose profile (generated later by dream). Fuzzy disambiguation ("Sam" vs "Sam Rivera") happens at extraction time, not storage time.

Relation: subject_id → predicate (verb phrase: lives-in, works-at, uses) → object_id, plus source_event_id and ts.

Operations: upsert (rejects blocked types), find by name/type, traverse N-hop, add relation.

Blocked entity types (storage backstop): upsert rejects dev-internal types outright. Seed the blocklist with: error, env_var, surface, table, directory, function, field, schema, method, command, system_component, pipeline, mechanism, configuration, specification, log, file, database, variable, test case, alias, format, effort_level, attribute, tag, version, os, software, library. This is the backstop; the primary defense is the domain allowlist at extraction time (Phase 4).

Entity hygiene (ongoing): even with gating, noise accumulates. Provide (a) a noise blocklist for names that keep re-entering, and (b) an entity-merge operation that folds duplicates (relations re-pointed, loser tombstoned). Run hygiene periodically, not as a one-off.

Beliefs (Supersedable Facts)

A durable, topic-keyed claim about the user. One active belief per topic; new beliefs supersede prior ones (last-write-wins by timestamp).

Belief: topic (kebab-case, e.g. home-location), claim (one declarative sentence), confidence (0–1), supersedes (prior belief ID), retracted (bool), provenance, sources (event IDs), verified_at.

Stored as belief.update events. "Current truth" = latest non-retracted event per topic (the belief head).

Belief candidates — drafts awaiting promotion: topic, claim, confidence, source_event_id, provenance, domain, status (pending/promoted/rejected/conflict). Drafted by the biographer, promoted by dream or manually. Expiry: pending > 14 days deleted; conflicts

30 days expired.

Provenance & Staleness

Class Meaning Staleness TTL
first-party User explicitly stated it Never
inferred Extracted/inferred from sessions 180 days
third-party Someone else said it about the user 120 days
external From an external data source / automated check 7 days
unknown Legacy or untracked 365 days

Beyond staleness flagging, apply read-time confidence decay for inferred beliefs (half-life on the order of the TTL, floored at ~50% of stored confidence). First-party never decays. Stale heads with a registered resolver (an automated re-checker keyed by topic prefix) get re-verified; the rest are flagged with a belief.stale event.

Corrections

When the assistant gets something wrong: what (the wrong statement), correction (the truth), optional context, optional topic.

  • Behavioral (no topic): a standing directive, surfaced in every primer.
  • Topic-linked (topic set): automatically retracts the belief head on that topic during the next dream pass. The correction IS the review.

Predictions

Forecasts for calibration: claim, confidence, deadline, outcome (right/wrong/unverifiable), brier_delta. Resolved manually or auto-marked unverifiable past deadline. Aggregate Brier scores reveal over- or under-confidence.

Recommendations & Habits

The same discipline as predictions, applied to advice. When the assistant gives the user a substantive recommendation (buy/skip/try/avoid a specific named thing), it logs a recommendation: claim, domain, ts, status. Then:

  • A nightly linker scans subsequent events for evidence the user acted on it and marks recommendations acted/ignored.
  • A periodic LLM scan backfills recommendations the assistant forgot to log.
  • Acted/ignored rates per domain feed calibration: which advice actually lands.

A soft habits store sits on top: recurring behavioral patterns graduate from "observed" to "graduated" habits, and retire when they stop appearing. This closes the advice→action→learning loop the way Brier scores close the prediction loop.

Phase 2 Done When

  • Events write via remember/ingest and search via lexical recall
  • Entities upsert (blocked types rejected), find, traverse; merge works
  • Beliefs write, query by topic (head), supersede, retract
  • Candidates insert, list, promote (writes belief.update), reject
  • Corrections record (behavioral + topic-linked)
  • Predictions and recommendations record and resolve

4. Phase 3: LLM Abstraction

Goal: A provider-agnostic LLM layer that routes by capability, enforces cost controls, and fails gracefully.

Capability-Based Routing

The system never calls "a model" — it calls a role:

Role Purpose
interactive User-facing conversational calls (quality + latency)
agentic Multi-turn tool-use loops (Phase 7)
reasoning Entity extraction, claim drafting, disambiguation
summarize Entity profiles, journal narratives
classify Cheap/fast triage
embed Vector embeddings
rerank Re-score recall results

Config maps each role to provider + model. Route everything to one provider to start; split later for cost/quality. Providers declare which roles they support; the dispatcher warns at startup on invalid assignments (e.g. an embed-only provider mapped to reasoning).

Provider Interface

Request: system prompt, messages, optional tools, optional output schema (structured JSON), max tokens, temperature, timeout override, cacheable hint. Response: text, structured output, tool calls, usage (input/output/cached tokens), cost USD, latency, provider name. Embed (separate): text(s) → vector(s).

Dispatcher

The single chokepoint for all LLM calls:

  1. Route by role
  2. Timeout every call (default ~5 min, overridable) — a hung provider must not wedge the scheduler
  3. Spend tracking — cumulative daily USD, configurable cap, resets at UTC midnight. Cap hit → clean error, callers degrade gracefully
  4. Lenient registration — missing API key logs a warning; the system runs degraded, not crashed
  5. Error classification — distinguish unreachable (timeout, connection refused, spend cap) from bad output (JSON parse, schema mismatch). This is load-bearing: unreachable must NOT advance processing cursors (retry when recovered); bad output MUST advance (one poison input never blocks the pipeline forever)

Phase 3 Done When

  • Two+ providers registered (one text, one embed); roles route via config
  • Invalid role assignment warns at startup
  • A deliberately slow provider triggers a timeout error
  • Past the daily cap, calls return a spend-cap error
  • Missing keys warn, don't crash
  • Unreachable vs bad-output classified correctly

5. Phase 4: Cognition

Goal: Background jobs that extract structure from raw text, vectorize content, and consolidate knowledge overnight.

Session Capture

When the user's AI tool finishes a session, the transcript reaches Robin via an HTTP hook, file watcher, or integration tick.

Pipeline:

  1. Receive transcript (user/assistant turns)
  2. Scope check — optional working-directory allowlist
  3. Skip rules — no assistant turns, empty content, one-word acknowledgments, tool-only output
  4. Dedup — hash the user turns; skip if seen
  5. Classify dev vs personal with a keyword-density heuristic (no LLM). Be conservative: only flag dev when dev signals heavily dominate (~3:1) — a mis-classification drops real personal facts, the worst error
  6. Ingest as a session.captured event

Two throttles that prevent runaway loops (see Lessons, §10): cap captures per tick (e.g. 20), and suppress self-capture — the daemon's own LLM/agent sessions must never re-enter the capture pipeline, or Robin recursively captures itself.

Domain Allowlist (the core gating decision)

v1 of this spec gated extraction with blocklists of dev noise. Production experience inverted this: blocklists lose by default (every novel dev concept requires a new rule), allowlists win by default (a novel dev concept is excluded by absence).

Define a closed set of personal domains, e.g.: health, finance, career, relationships, preferences, creative, travel, home, life_events, identity, directives. The biographer tags every extracted entity and claim with a domain; anything outside the set is dropped at extraction. directives is the one door adjacent to dev noise — it admits standing rules the user sets about how they work ("always use X email for commits"), gated by a durable-rule-vs-transient-task test in the prompt.

Keep the type blocklist (Phase 2) and quality filters as backstops. Ship a kill-switch policy flag so gating can be disabled for debugging.

Biographer (Entity & Belief Extraction)

A scheduled job (every 5–15 min) reading unprocessed session.captured and knowledge.doc events. Integration events (mail, calendar) are recallable but NOT biographer input — they're structured data; the biographer is for narrative text.

Pass 1 — Entities & relations: structured-output prompt → {entities: [{type, name, domain}], relations: [...]}. Then filter: domain allowlist, blocked types, and a quality filter dropping role markers, bare numbers, git SHAs, state flags, code identifiers, file paths, CLI flags, money amounts. A relation touching a dropped entity is itself dropped. When a name matches multiple existing entities, ask the LLM to disambiguate (pick one or declare new).

Pass 2 — Claim drafting: prompt for durable declarative facts → {claims: [{topic, claim, confidence, domain}]}. Exclude imperatives, transient details, speculation. Each surviving claim becomes a pending belief candidate. Cap ~20 candidates per event.

Preprocessing (before any LLM call): strip tool-output blocks and code fences, collapse consecutive same-role turns, drop very short turns; for assistant-heavy sessions keep user turns + final assistant turn. Knowledge docs are plain markdown — treat all content as user-authored.

Multi-tick chunking: large events split into chunks; process a bounded number per tick with a persisted cursor that survives restarts.

Deadline enforcement — inside the loops. The tick must accept a deadline and check it between every chunk, every claim batch, and every disambiguation call — not just at tick start. A deadline checked only between sessions lets one large session run 20+ minutes of sequential LLM calls, wedging the scheduler until the health monitor kills the daemon (see Lessons, §10). Also cap per-batch LLM timeouts so a single call can't consume the whole tick.

Error handling: LLM unreachable → do NOT advance the cursor. Bad output → DO advance. (Phase 3's classification, applied.)

Embedder

Every 1–2 min: query content rows WHERE embedding IS NULL, batch (~200), embed, write back, maintain the vector index. Never embed inline during ingest — semantic recall lags by at most one cycle; lexical recall is instant.

Ingest Docs

Every ~10 min: scan content directories for *.md, SHA-256 hash each; new path → ingest as knowledge.doc; changed hash → update in place and nullify the embedding; same hash → skip. Serves both recall and biographer extraction.

Dream (Nightly Consolidation)

Deterministic pass (no LLM):

  1. Resolve overdue predictions → unverifiable, compute Brier
  2. Roll up daily metrics (entities, handler activity, calibration)
  3. Detect narrative arcs — recurring themes in a rolling ~14-day window (e.g. cluster events sharing ≥2 entities; merge near-duplicate arcs by Jaccard similarity)
  4. Expire stale candidates (pending > 14 days; conflicts > 30 days)
  5. Belief freshness scan — per-provenance TTLs; resolver re-verify or belief.stale flag
  6. Replay corrections — retract belief heads for topic-linked corrections
  7. Promote candidates when ALL hold: confidence ≥ 0.8, candidate age ≥ 7 days (the age gate lets contradicting evidence arrive first), domain allowed, and no conflicting head on the topic. A contradicting candidate is flagged conflict for review, never auto-promoted
  8. Merge near-duplicate candidates (same topic, similar claim → keep higher-confidence)
  9. Link recommendations to actions; update habit observations

Synthesis pass (LLM, degrades gracefully to metrics-only): 10. Re-summarize hot entities (many new relations since last profile; cap per run, e.g. 25) 11. Generate the daily journal (narrative of the day + arcs) 12. Optional depth insights across arcs

Cognition Job Schedule

Job Cadence LLM Bound
Biographer 5–15 min reasoning tick deadline + per-batch timeout
Embedder 1–2 min embed per-batch timeout
Ingest docs ~10 min none —
Dream nightly summarize (optional) per-call timeouts

Phase 4 Done When

  • Capture receives, classifies, dedups, throttles, and never self-captures
  • Biographer extracts entities/relations/claims from sessions and docs, with domain gating, quality filtering, and disambiguation
  • Cursor semantics correct: unreachable retries, poison advances
  • A deliberately huge event cannot exceed the tick deadline (verify with a clock-injected test)
  • Embedder vectorizes in batches; ingest-docs dedups by hash
  • Dream: predictions resolved, candidates expired/promoted/conflict-flagged (with the age gate), corrections replayed, staleness flagged, journal written; runs to completion with no LLM available

6. Phase 5: Integrations

Goal: A framework for connecting Robin to external APIs and local data, with a tick lifecycle, on-demand actions, and user-extensible plugins.

Contract

Manifest: name, version, schedule (cron), permissions (memory read/write, network hosts, required secrets), optional actions (name, description, input schema).

Handler: optional tick(ctx) (periodic fetch + ingest), init, cleanup, health. Action-only integrations (no tick) are valid.

Context

Every handler receives: db, llm (dispatcher), state (per-integration DB-backed KV — OAuth tokens, cursors, watermarks), log, fetch, ingest(input), checkOutbound(input).

Tick Lifecycle

Scheduler claims the cron job → tick(ctx) runs inside a timeout (e.g. 120s) → fetch → ingest() (dedup by external ID/hash) → return {status, ingested?, message?} → completion event recorded → consecutive errors tracked, alert raised past a threshold.

Missing secrets: skip the integration with a warning; never crash the daemon.

OAuth Token Handling (hard-won — see Lessons, §10)

Integrations using OAuth refresh-token rotation need all of the following or they will strand the user in re-auth loops:

  • Single-flight refresh across processes. MCP servers and the daemon are separate processes sharing the same stored tokens. Two concurrent refreshes of a rotating token family revoke the grant. Use a cross-process lease (DB or lockfile) so exactly one process refreshes; others wait and re-read.
  • Never retry a refresh POST after a 5xx. The provider's gateway may have committed the rotation and eaten the response; replaying the old token revokes the family. Fail the tick, apply a cool-off, try fresh next tick.
  • Auth-death latch. When a grant is definitively dead (400/401 on refresh), latch the state and make every subsequent tick skip with a clear "run reauth <name>" message — don't hammer the provider.
  • reauth seeds state directly. The CLI re-auth flow must write the newly exchanged tokens into the integration's state (clearing the latch), not just into the env file.

Discretion (Outbound Guards)

Before any outbound write, checkOutbound() scans for PII (Luhn-validated card numbers, SSNs) and secrets (common API-key formats). A match blocks the write and logs an audit event.

Reference Integration Set

Integration Cadence Purpose
Mail 15 min Subjects, senders, snippets
Calendar 30 min Upcoming events
Issues/PRs (Linear, GitHub) 30 min Work items
Health wearable 30–60 min Sleep, recovery, workouts
Finance 15 min Quotes for tracked symbols
Weather 1 h Conditions at user's location

Start with ONE and get it working end-to-end before adding more.

User Extensions

Users drop a manifest + handler into the extensions directory; same contract as built-ins. Restart-to-reload is fine.

Phase 5 Done When

  • Built-in and extension integrations load and schedule
  • Tick timeout contains hung API calls; dedup prevents duplicates
  • Action-only integrations answer on-demand queries
  • Outbound guard blocks PII/secrets; missing secrets skip, don't crash
  • An OAuth integration survives: concurrent refresh attempts (single-flight), a 5xx mid-rotation (no replay), and a dead grant (latch + clean reauth)
  • One integration works end-to-end

7. Phase 6: Surfaces

Goal: Expose memory to AI tools (MCP), provide a setup/maintenance CLI, an HTTP hook endpoint, and session-start context (primer).

MCP Servers

Two servers keep the tool count manageable. They are separate processes from the daemon — each must independently load secrets and open the DB, and they serve stale code after a rebuild until the AI-tool session restarts (§10).

Core server (memory + cognition + self-model):

Tool Purpose
recall Search events (lexical/vector/hybrid)
remember Write an event
believe / recall_belief Write / query beliefs by topic
review_beliefs / resolve_belief_candidate List and settle pending/conflicted candidates
find_entity / get / list Entity search; fetch by ID; list events/entities/jobs/predictions
predict / recommend Log forecasts and recommendations
record_correction Log an error + fix
audit / explain Event log; narrative explanation of a decision
health / metrics / alerts Daemon health; calibration + counts; open alerts
journal Daily narrative from dream
power Read/write power state
skill Load reusable methodology playbooks (markdown)

Extension server (integrations + agent + graph):

Tool Purpose
Integration actions Generated from loaded integrations' action defs
related_entities N-hop graph traversal
resolve_prediction / resolve_recommendation Mark outcomes
integration_status Per-integration health
ingest Direct event write
agent / run Run an agentic task (Phase 7); run a registered job
check_action Pre-flight trust check for risky operations
update Update runtime policies/rules

CLI

Setup and maintenance, not a daily interface: init, doctor (DB, LLM, integrations, disk), reauth <name>, reindex, power <state>, plus read-outs (recommendations, habits, alerts).

HTTP Server

POST /hook/session-end (transcript intake), GET /health, GET /ready.

Bind to loopback only — this endpoint receives full session transcripts and must never be reachable off-machine. Bind 127.0.0.1 explicitly, not localhost: localhost can resolve dual-stack and land the listener on IPv6-only (::1), silently breaking clients that connect via 127.0.0.1.

Primer (Session-Start Context)

Deterministic assembly — no LLM, no staleness surprises. Budget ~10,000 chars (~2,500 tokens) total:

Section Content
Behavioral rules Corrections without a topic — standing directives
Learning digest ~350 chars of self-performance: prediction calibration, belief counts, agent spend, recommendation follow-through
Active beliefs Belief heads: topic + claim + confidence
User profile Inlined character/voice files
Knowledge index Titles of knowledge files
Pending items Counts: candidates, stale beliefs, open predictions

Put the digest before beliefs — it's short and high-signal; beliefs can be 7k+ chars and get truncated first.

Trigger: regenerate on demand from a session-start hook calling the CLI (freshest), or on a short cron as a fallback. Assembly is cheap and deterministic, so regenerating per-session is fine.

Delivery: a well-known file path the AI tool reads at session start (most portable). Hook-based injection is most seamless where the tool supports it.

Auto-Recall (Per-Turn Injection)

The primer is static per session; auto-recall is the second, dynamic layer. Where the AI tool supports a prompt-submit hook, run each user prompt through recall and inject the top few matching snippets into the turn's context — deterministic, no LLM call.

What production tuning taught: skip prompts too short to match (<~12 chars); filter the candidate pool to curated kinds (knowledge docs, beliefs) because raw transcripts and telemetry outrank curated content on raw relevance and drown it; slice long documents to the relevant section rather than injecting teaser truncations; cap the injected budget so snippets can't crowd out the primer. Mark injections clearly as background context, not instructions — and accept that they are partial: the AI tool should open the source file before answering anything load-bearing from a snippet.

Phase 6 Done When

  • Both MCP servers expose their tool sets; recall/remember/beliefs round-trip
  • Integration actions callable through the extension server
  • CLI initializes a fresh instance; doctor reports honestly
  • HTTP endpoint receives transcripts and triggers capture
  • Primer assembles under budget; a fresh session starts with it loaded

8. Phase 7: Agent System

Goal: A guarded primitive for multi-turn LLM tool-use loops with cost controls, isolation, verification, and full auditability.

The runAgent Primitive

All agentic execution flows through one guarded function. No other code path may run multi-turn LLM loops; shelling out to an AI CLI directly is banned.

Parameters: goal, surface (caller identity), allowedTools (explicit list, no "allow all"), maxTurns, timeoutMs, maxBudgetUsd, model, workdir.

Behavior:

  1. Pre-flight cap check — per-surface daily cap; over → return capped without spending a dollar
  2. Run the loop: goal + tools → LLM → tool calls → results → repeat
  3. Enforce limits: hard deadline (abort signal), turn counter, per-run budget — any limit → graceful stop
  4. Stream every message to a per-run JSONL transcript
  5. Record one ledger row: tokens, cost, status, surface, duration
  6. Return success / capped / timeout / error

Run Out-of-Process

The scheduler executes one job at a time; an agent run lasts up to 30 minutes. If a scheduled job calls runAgent inline, all cognition and integrations stall for the duration. Autonomous runs execute in a detached child process: the scheduled job spawns the runner and returns; the runner owns the run end-to-end (ledger, transcript, verifier). Two consequences to design for:

  • A detached child has no parent to report errors to — it must log to its own runner log file, and its exit is not release-on-exit for any lock it holds (hence the stale-stealable lockfile below).
  • The child inherits its environment from the supervisor, not your shell. Resolve the repo/workdir explicitly; a cwd default that works in dev will silently point elsewhere under the process supervisor.

Isolation & Safety

  • Tool allowlisting per run; write-capable tools are explicit opt-in
  • Worktree isolation: write-capable runs operate in an isolated copy (git worktree / temp dir / container); changes are a reviewable diff the caller applies or discards
  • OS sandboxing for write-capable runs (macOS seatbelt, Linux bubblewrap, container). Fail closed: can't sandbox → error, never run unsandboxed
  • Single-flight lock per surface (lockfile with holder PID + timestamp; a lock older than a staleness bound is considered abandoned and stolen, since a crashed holder never releases)
  • The agent's working directory is a default, not a boundary — only the sandbox is a boundary. Any handler that can write needs the sandbox even if its cwd looks harmless

Outcome Verification

Agents report success; sometimes they're wrong (or the model claims work it didn't do). After each run, a deterministic verifier per handler checks a post-condition against the world — no LLM:

  • "Research" → a research.brief event exists since run start
  • "KB curation" → a knowledge file mtime changed since run start
  • "Health remediation" → the worktree has a diff
  • Handlers with no checkable post-condition return unverifiable

fail (claimed work, no observable effect) records an outcome-mismatch and raises an alert. This one mechanism catches goal/toolset drift, permission regressions, and hallucinated success — cheaply.

Cost Controls

Layer Scope
Per-run budget One run
Per-surface daily cap All runs on a surface per day
Global daily cap All LLM calls (dispatcher, Phase 3)

Suggested starting points: $5/run, 30 turns, 30 min, $25–50/surface/day. Size per-run budgets to the most expensive legitimate run — a cap below one full run of your largest handler starves it silently (diagnose via ledger rows with status capped).

Handler Registry

Specialized handlers, not "do anything" agents. The production set, as a menu — start with 2–3:

Handler Purpose Mode
Self-improvement Refine extraction/belief quality worktree write
Research Multi-source web research → ingested brief read + ingest
Integration ops Exercise/fix integrations via their actions scoped
KB curation Organize, dedupe knowledge files scoped write
Belief reconciliation Hunt contradictions, propose fixes propose-only
Prediction calibration Resolve open predictions, report drift read + resolve
Gap fill Find knowledge gaps → research → fill scoped write
Dream enrichment Cross-session synthesis over the corpus propose-only
Life executor Real-world actions (reservations, rides) hard risk gate
Integration authoring "Integrate with X" → scaffold in a worktree worktree write
Health remediation Diagnose invariant trips; propose config/code fix worktree write
Daily brief Morning synthesis across integrations + memory read-only

Three lessons baked into this table: (1) propose-only handlers write proposals as events for review, never directly into beliefs; (2) a handler's goal text, allowlist, and verifier must agree — a goal that says "write a journal" with a read-only toolset fails forever, and only the verifier will tell you; (3) handlers that act in the real world (reservations, purchases, messages) sit behind a risk gate: a pre-flight check (check_action) classifies the action, and irreversible or spend-bearing actions require explicit user confirmation by policy — autonomy is for gathering and proposing, not for spending.

Phase 7 Done When

  • runAgent enforces budget, turns, timeout; cap pre-flight returns capped
  • An autonomous run executes detached — the scheduler keeps ticking during it, and the child's errors land in its runner log
  • Every run: JSONL transcript + ledger row
  • Allowlisting blocks unauthorized tools; worktree isolates writes; sandbox fails closed
  • Verifiers pass/fail correctly against seeded post-conditions; a fail raises an alert
  • One handler works end-to-end, verified

9. Phase 8: User Data & Extensions

Goal: Directory layout, extension system, first-run setup, multi-machine.

User Data Directory

user-data/
├── config/
│   ├── llm-config.*        # role → provider mapping
│   ├── policies.*          # power, capture, gating, agent caps
│   └── secrets/.env        # keys + tokens (restrictive perms)
├── state/db/               # database file(s)
├── content/
│   ├── profile/            # WHO the user is (character, relationships, people/)
│   ├── knowledge/          # WHAT the user knows (by topic)
│   └── artifacts/          # generated outputs
├── extensions/
│   ├── integrations/       # user-added integrations
│   ├── jobs/               # user-added scheduled jobs
│   └── skills/             # user-added methodology files
├── logs/
└── agent-runs/             # per-run JSONL transcripts

content/ enforces personal-data-only: no dev artifacts, code docs, or session logs. User skills shadow system skills of the same name.

First-Run Setup (init)

  1. Create user-data/ structure with sensible default configs
  2. Prompt for / detect LLM credentials
  3. Initialize DB, apply migrations
  4. Install process supervisor config
  5. Register MCP servers with the AI tool (or print instructions)
  6. Install session-capture hooks
  7. Run doctor to verify

Backups

content/ is the irreplaceable data — knowledge and profile files carry the durable memory. Back it up additively (never --delete-style mirroring; a local corruption must not propagate) to durable storage, plus secrets separately with restrictive permissions. The database is rebuildable from content + fresh sessions, but a periodic DB snapshot makes recovery fast. Record a success timestamp per backup leg and have doctor warn when a leg goes stale — a backup that silently stopped is the failure mode, not a backup that errors.

Multi-Machine

Sync content/ + extensions/ (private git, Syncthing); each machine runs its own daemon, DB, config, and secrets. The database is never synced — it is a derived index; a new machine's graph rebuilds organically from synced knowledge files + new sessions. Session history does not transfer; knowledge files carry the durable content.

Phase 8 Done When

  • init produces a complete instance; doctor verifies it
  • Extension integrations/jobs/skills load and run
  • Content files are indexed and searchable
  • Fresh machine: init + synced content → running Robin

10. Hard-Won Lessons

Every item here is a production failure from the reference implementation and the design change that fixed it. Build these in from the start.

1. Deadlines must bind inside work loops. A biographer tick that checked its deadline only between sessions ran 20+ minutes on one large session, starving the scheduler heartbeat until the health monitor killed the daemon — roughly hourly, and the restart re-claimed the same event. Check the deadline between every chunk, claim batch, and disambiguation call, and cap per-batch LLM timeouts. Symptom to watch for: "skipped cron jobs" that look like integration staleness are often daemon start/stop churn — check daemon lifecycle first.

2. Blocklists lose; allowlists win. Months of adding dev-noise rules (entity types, name patterns) never kept the graph clean — every novel dev concept needed a new rule. Inverting to a closed personal-domain allowlist fixed it structurally: novel noise is excluded by absence. Keep the blocklists only as backstops.

3. Separate processes, separate initialization. MCP servers are not the daemon. Each process must load secrets and open the DB itself; forgetting is a silent failure ("integration skips: key not set"). And after a rebuild, the daemon picks up new code on restart, but MCP servers run stale code until the AI-tool session restarts. Make doctor report which code version each process runs.

4. OAuth refresh-token rotation is hostile. Concurrent refreshes from separate processes revoke the token family → cross-process single-flight lease. A gateway 5xx may mean the rotation committed server-side → never replay a refresh POST. A dead grant should latch (skip ticks with a clear reauth message), and re-auth must seed tokens into integration state, not just the env file.

5. Suppress self-capture. Once Robin runs its own agents, their sessions look like user sessions. Without suppression + a per-tick capture cap, the system recursively ingests its own output.

6. Verify autonomous work deterministically. An agent that "succeeded" with a goal its toolset couldn't perform failed silently for weeks. Cheap post-condition verifiers (did a row appear? did a file change? is there a diff?) plus an alert on mismatch catch this class permanently.

7. Budget caps starve silently. A per-run budget below the cost of one legitimate run of your biggest handler doesn't error loudly — the run just stops mid-task with status capped. Size caps to the most expensive legitimate run and audit the ledger for capped rows.

8. Cursor discipline is the pipeline's immune system. The unreachable-vs-bad-output distinction (Phase 3) shows up everywhere: capture, biographer, embedder, integrations. Get it right once in the dispatcher's error taxonomy and honor it in every consumer.

9. Promotion needs an age gate. Auto-promoting high-confidence candidates immediately turns one misread session into durable false "truth." The 7-day age gate gives contradicting evidence time to arrive; conflicts flag for review instead of last-write-wins.

10. Restart after deploy, always. A long-running daemon serves stale in-memory code and prompts until restarted. Make restart cheap, idempotent, and part of the deploy habit.


11. Glossary

Term Definition
Alert A persistent record of a detected problem (invariant trip, integration failure, verifier mismatch); auto-resolves on next success.
Arc A recurring theme detected across recent events (shared-entity clustering over a rolling window).
Auto-recall Deterministic per-turn injection of recall snippets matched to the user's prompt, via a prompt-submit hook.
Belief / belief head A durable, topic-keyed claim stored as an event; the head is the latest non-retracted event on the topic — current truth.
Belief candidate A draft claim from the biographer awaiting promotion, rejection, or conflict review.
Biographer The cognition job extracting entities, relations, and claims from narrative text.
Correction A record of something the assistant got wrong + the fix; behavioral (standing directive) or topic-linked (retracts a belief).
Cron re-arming After any completion — including failure — a cron job's next run is inserted.
Disambiguation LLM resolution of an extracted name against existing entities.
Dispatcher The single chokepoint for LLM calls: role routing, timeouts, spend caps, error taxonomy.
Domain allowlist The closed set of personal domains extraction is limited to; the structural noise gate.
Dream Nightly consolidation: resolve predictions, promote candidates, replay corrections, link recommendations, write the journal.
Embedder Batch job turning NULL-embedding content rows into vectors.
Event An immutable record in the append-only store; everything is an event.
Habit A soft behavioral pattern that graduates with repeated observation and retires when it stops.
Ingest Write an event (+ content) with external-ID dedup.
Integration A module connecting Robin to an external source via tick/actions.
Lease A claim timestamp preventing concurrent daemons from running the same job.
Primer Deterministic session-start context: rules, digest, beliefs, profile, index.
Provenance A belief's evidence class (first-party, inferred, third-party, external, unknown); drives staleness and decay.
Recall Search over the event store (lexical, vector, hybrid).
Recommendation Logged advice whose follow-through is detected and scored, like predictions.
Resolver An automated re-checker registered by topic prefix that re-verifies stale beliefs.
runAgent The single guarded function for all multi-turn tool-use loops.
Single-flight A cross-process lock (with stale-steal) ensuring one holder: agent runs per surface, OAuth refreshes per integration.
Skill A markdown playbook encoding a reusable methodology.
Supersede Replace a topic's belief head with a newer event; the old one remains in the log.
Surface A caller identity for agent runs, each with its own daily cap.
Verifier A deterministic post-condition check that an agent run's claimed work is observable in the world.
Worktree An isolated copy of a working directory where write-capable agent runs produce reviewable diffs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment