- What Is Robin?
- Phase 1: Foundation — Event Store + Scheduler
- Phase 2: Memory — Recall, Entities, Beliefs
- Phase 3: LLM Abstraction — Dispatcher & Providers
- Phase 4: Cognition — Capture, Biographer, Embedder, Dream
- Phase 5: Integrations — External Data Sources
- Phase 6: Surfaces — MCP, CLI, HTTP, Primer
- Phase 7: Agent System — Sandboxed Execution
- Phase 8: User Data & Extensions
- Hard-Won Lessons
- Glossary
Robin is a local-first personal AI assistant that learns about its user over time. It runs as a background daemon, captures interactions, pulls data from external services, and builds a structured understanding of the user's life — relationships, preferences, health, career, finances, creative work. It exposes this understanding to any AI coding tool (Claude Code, Cursor, Gemini CLI, etc.) via MCP servers.
- The user opens their AI tool. It loads the primer — a compact context file (~2,500 tokens) with behavioral rules, active beliefs, a user profile, a knowledge index, and a learning digest.
- The user asks "what camera gear do I have?" The tool calls Robin's
recallMCP tool; Robin searches the event store (lexical + vector) and returns ranked results. - The user says "I sold the 24-120 last week." The tool calls
believewith a topic and claim, superseding the old belief. - The session ends; a hook posts the transcript to Robin's HTTP endpoint. The capture pipeline classifies it as personal and ingests it.
- On the next biographer tick, entities, relations, and belief candidates are extracted. Overnight, the dream job promotes qualifying candidates, updates entity profiles, and writes the daily journal.
- Tomorrow's primer reflects the updated knowledge.
Meanwhile, integrations poll external services (mail, calendar, health, finance) and the embedder vectorizes everything for semantic search — no user intervention.
- Personal data only — Memory holds the user's life, not engineering artifacts. Enforced structurally by a closed personal-domain allowlist (Phase 4), not by ever-growing blocklists.
- Local-first — All data lives on the user's machine. Cloud services are optional (LLM inference, external integrations).
- Model-agnostic — Any LLM provider (cloud or local) behind capability-based routing.
- Append-only events — Nothing is deleted, only superseded. Every belief, correction, and prediction is an immutable event with provenance.
- Autonomous cognition — Background jobs extract, consolidate, and verify knowledge without user intervention.
- Auditable — Every decision has a trail: event logs, agent transcripts, belief provenance, correction history.
- Verified, not trusted — Autonomous work is checked by deterministic post-condition verifiers; claims of work without observable effects raise alerts.
Not a chatbot, web app, or cloud service. It has no conversational UI of its own. It is infrastructure that makes other AI tools smarter about the user.
- Persistent storage with full-text search (SQL + FTS or equivalent)
- Vector similarity search (DB extension, separate index, or service)
- At least one LLM for text generation and one for embeddings
- A scheduling mechanism (long-running daemon preferred)
- A process supervisor to restart the daemon on crash (launchd, systemd, Docker)
- A way to expose tools to AI assistants (MCP servers or equivalent)
- Structured logging (JSON)
| Phase | Title | Depends on |
|---|---|---|
| 1 | Foundation — Event Store + Scheduler | — |
| 2 | Memory — Recall, Entities, Beliefs | 1 |
| 3 | LLM Abstraction | — |
| 4 | Cognition | 1, 2, 3 |
| 5 | Integrations | 1, 2 |
| 6 | Surfaces | 2, 5 |
| 7 | Agent System | 3, 6 |
| 8 | User Data & Extensions | all |
Phases 1 and 3 are independent and can be built in parallel. Verify each phase's "Done when" checklist before starting the next — every later phase assumes the earlier ones actually work, and debugging cognition on top of a flaky scheduler wastes days.
Minimum viable Robin: the core MCP server needs only Phases 1–2. A thin
slice — event store + lexical recall + remember/believe exposed over MCP +
a hand-written primer file — is a usable memory system in days, before any
LLM plumbing, integrations, or cognition exist. Build that slice first, use it
daily, then layer the rest on. Only the extension server needs Phases 5 and 7.
Testing: write automated tests alongside each phase, collocated with the module. Anything time-dependent (deadlines, leases, staleness TTLs, cron) takes an injectable clock — wall-clock tests for this logic are flaky and slow. Treat each "Done when" item as an executable test, not a manual check.
Goal: A running daemon with persistent storage, structured logging, config loading, and a job scheduler with crash recovery.
| Config | Purpose |
|---|---|
| LLM config | Role-to-provider mapping |
| Policies | Power state, capture rules, network mode, agent budget caps, cognition kill-switches |
| Secrets | API keys, OAuth tokens. Restrictive permissions, loaded into env, never logged. |
Config files are human-editable (YAML/TOML/similar). Secrets are separate from non-secret config. Every process entry point (daemon, each MCP server, standalone scripts) must load secrets itself — see Lessons, §10.
Numbered, forward-only migration files applied transactionally at boot; the daemon records the applied version and refuses to run against a schema newer than it knows. Every later phase adds tables (candidates, ledger, alerts, habits, recommendations) — without a migration system from day one, each upgrade becomes manual surgery.
An append-only log. Every meaningful occurrence becomes an event.
Event: id, ts (ISO-8601), kind (e.g. session.captured,
belief.update, integration.tick), source, optional actor, status
(ok/error/skipped), payload (JSON), optional content_ref (FK to
content table).
Content table (separate so event scans never drag large blobs): id, ts,
body (full text, FTS-indexed), embedding (NULL until the embedder fills
it).
Nothing is deleted. Supersession is a new event referencing the prior event's ID.
Ingest interface: One function that writes an event + optional content body. Supports upsert-by-external-ID so re-ingesting the same external record (e.g. the same mail thread) updates in place instead of duplicating.
Tick-based; claims and executes one job at a time in-process.
Job: id, kind, status (pending/running/completed/failed),
scheduled_at, leased_until, completed_at, error.
Behavior:
- Ticks every 1 second; claims the oldest pending job with
scheduled_at <= now(FIFO) - Sets a lease on the claimed job (protects against a second daemon instance; the owning daemon completes the job regardless of lease expiry, so the lease only needs to exceed dead-daemon detection time, ~5 min)
- Executes the registered handler, records the result
- Cron re-arming: after completion — success or failure — a cron job's next run is inserted. Transient failures can never permanently silence a schedule.
- On boot, a new daemon proactively breaks expired leases rather than waiting for them to lapse
Crash recovery on boot:
- Detect and clear a stale PID file
- Reset jobs with expired leases (prior daemon died mid-run)
- Re-arm cron schedules that should have fired while down
- Handlers with partial progress (multi-chunk extraction) resume from their own persisted cursors — the scheduler only guarantees re-entry into the queue
Runs every ~60s alongside the scheduler:
- Verifies the scheduler heartbeat is recent; verifies DB reachability
- A single CRITICAL observation is tolerated (slow handlers are legitimate); only sustained CRITICAL (30 min) triggers self-termination for supervisor respawn
- Boot grace window (~2 min) suppresses false positives during startup
- Maintains a persistent alert store: alerts are raised by invariant trips, integration failures, and verifier mismatches; they auto-resolve when the underlying condition next succeeds. Exposed via MCP/CLI so the user (and agents) can see open problems.
A configurable power state (active, paused, off) stored in policies,
read/writable at runtime. When paused, the tick loop skips execution.
- Daemon starts, loads config + secrets, opens DB, applies migrations (idempotently — a second boot is a no-op)
- Cron handlers fire on time; jobs claim and execute FIFO
- Jobs re-arm after completion, including after failure
- Health monitor detects a stalled scheduler and terminates only after the sustained-critical window
- Kill the daemon mid-job → restart → expired leases reset, crons re-arm
- Structured logs for claims, completions, failures, health checks
Goal: Store, search, and structure knowledge about the user.
| Mode | How | When |
|---|---|---|
| Lexical | FTS on content bodies | Exact terms, names |
| Vector | Cosine similarity on embeddings | Semantic queries |
| Hybrid | Both, merged by reciprocal rank fusion | Default |
Vector search depends on the embedder (Phase 4); lexical-only recall is fully
functional before that. Results are enriched with kind, age, source, and score.
An optional rerank stage (the rerank LLM role) can re-score the top K —
ship without it; hybrid fusion alone is good enough in practice.
The public write path: an event with kind, source, and content body. All
external data enters through remember or the ingest interface.
Entity: id, type (person, place, organization, service,
topic, thing), canonical_name (case-normalized, deduped at storage),
optional prose profile (generated later by dream). Fuzzy disambiguation
("Sam" vs "Sam Rivera") happens at extraction time, not storage time.
Relation: subject_id → predicate (verb phrase: lives-in,
works-at, uses) → object_id, plus source_event_id and ts.
Operations: upsert (rejects blocked types), find by name/type, traverse N-hop, add relation.
Blocked entity types (storage backstop): upsert rejects dev-internal types
outright. Seed the blocklist with: error, env_var, surface, table,
directory, function, field, schema, method, command,
system_component, pipeline, mechanism, configuration, specification,
log, file, database, variable, test case, alias, format,
effort_level, attribute, tag, version, os, software, library.
This is the
backstop; the primary defense is the domain allowlist at extraction time
(Phase 4).
Entity hygiene (ongoing): even with gating, noise accumulates. Provide (a) a noise blocklist for names that keep re-entering, and (b) an entity-merge operation that folds duplicates (relations re-pointed, loser tombstoned). Run hygiene periodically, not as a one-off.
A durable, topic-keyed claim about the user. One active belief per topic; new beliefs supersede prior ones (last-write-wins by timestamp).
Belief: topic (kebab-case, e.g. home-location), claim (one
declarative sentence), confidence (0–1), supersedes (prior belief ID),
retracted (bool), provenance, sources (event IDs), verified_at.
Stored as belief.update events. "Current truth" = latest non-retracted event
per topic (the belief head).
Belief candidates — drafts awaiting promotion: topic, claim,
confidence, source_event_id, provenance, domain, status
(pending/promoted/rejected/conflict). Drafted by the biographer,
promoted by dream or manually. Expiry: pending > 14 days deleted; conflicts
30 days expired.
| Class | Meaning | Staleness TTL |
|---|---|---|
first-party |
User explicitly stated it | Never |
inferred |
Extracted/inferred from sessions | 180 days |
third-party |
Someone else said it about the user | 120 days |
external |
From an external data source / automated check | 7 days |
unknown |
Legacy or untracked | 365 days |
Beyond staleness flagging, apply read-time confidence decay for inferred
beliefs (half-life on the order of the TTL, floored at ~50% of stored
confidence). First-party never decays. Stale heads with a registered
resolver (an automated re-checker keyed by topic prefix) get re-verified;
the rest are flagged with a belief.stale event.
When the assistant gets something wrong: what (the wrong statement),
correction (the truth), optional context, optional topic.
- Behavioral (no topic): a standing directive, surfaced in every primer.
- Topic-linked (topic set): automatically retracts the belief head on that topic during the next dream pass. The correction IS the review.
Forecasts for calibration: claim, confidence, deadline, outcome
(right/wrong/unverifiable), brier_delta. Resolved manually or
auto-marked unverifiable past deadline. Aggregate Brier scores reveal over-
or under-confidence.
The same discipline as predictions, applied to advice. When the assistant gives
the user a substantive recommendation (buy/skip/try/avoid a specific named
thing), it logs a recommendation: claim, domain, ts, status. Then:
- A nightly linker scans subsequent events for evidence the user acted on
it and marks recommendations
acted/ignored. - A periodic LLM scan backfills recommendations the assistant forgot to log.
- Acted/ignored rates per domain feed calibration: which advice actually lands.
A soft habits store sits on top: recurring behavioral patterns graduate from "observed" to "graduated" habits, and retire when they stop appearing. This closes the advice→action→learning loop the way Brier scores close the prediction loop.
- Events write via remember/ingest and search via lexical recall
- Entities upsert (blocked types rejected), find, traverse; merge works
- Beliefs write, query by topic (head), supersede, retract
- Candidates insert, list, promote (writes
belief.update), reject - Corrections record (behavioral + topic-linked)
- Predictions and recommendations record and resolve
Goal: A provider-agnostic LLM layer that routes by capability, enforces cost controls, and fails gracefully.
The system never calls "a model" — it calls a role:
| Role | Purpose |
|---|---|
interactive |
User-facing conversational calls (quality + latency) |
agentic |
Multi-turn tool-use loops (Phase 7) |
reasoning |
Entity extraction, claim drafting, disambiguation |
summarize |
Entity profiles, journal narratives |
classify |
Cheap/fast triage |
embed |
Vector embeddings |
rerank |
Re-score recall results |
Config maps each role to provider + model. Route everything to one provider to
start; split later for cost/quality. Providers declare which roles they
support; the dispatcher warns at startup on invalid assignments (e.g. an
embed-only provider mapped to reasoning).
Request: system prompt, messages, optional tools, optional output schema (structured JSON), max tokens, temperature, timeout override, cacheable hint. Response: text, structured output, tool calls, usage (input/output/cached tokens), cost USD, latency, provider name. Embed (separate): text(s) → vector(s).
The single chokepoint for all LLM calls:
- Route by role
- Timeout every call (default ~5 min, overridable) — a hung provider must not wedge the scheduler
- Spend tracking — cumulative daily USD, configurable cap, resets at UTC midnight. Cap hit → clean error, callers degrade gracefully
- Lenient registration — missing API key logs a warning; the system runs degraded, not crashed
- Error classification — distinguish unreachable (timeout, connection refused, spend cap) from bad output (JSON parse, schema mismatch). This is load-bearing: unreachable must NOT advance processing cursors (retry when recovered); bad output MUST advance (one poison input never blocks the pipeline forever)
- Two+ providers registered (one text, one embed); roles route via config
- Invalid role assignment warns at startup
- A deliberately slow provider triggers a timeout error
- Past the daily cap, calls return a spend-cap error
- Missing keys warn, don't crash
- Unreachable vs bad-output classified correctly
Goal: Background jobs that extract structure from raw text, vectorize content, and consolidate knowledge overnight.
When the user's AI tool finishes a session, the transcript reaches Robin via an HTTP hook, file watcher, or integration tick.
Pipeline:
- Receive transcript (user/assistant turns)
- Scope check — optional working-directory allowlist
- Skip rules — no assistant turns, empty content, one-word acknowledgments, tool-only output
- Dedup — hash the user turns; skip if seen
- Classify
devvspersonalwith a keyword-density heuristic (no LLM). Be conservative: only flagdevwhen dev signals heavily dominate (~3:1) — a mis-classification drops real personal facts, the worst error - Ingest as a
session.capturedevent
Two throttles that prevent runaway loops (see Lessons, §10): cap captures per tick (e.g. 20), and suppress self-capture — the daemon's own LLM/agent sessions must never re-enter the capture pipeline, or Robin recursively captures itself.
v1 of this spec gated extraction with blocklists of dev noise. Production experience inverted this: blocklists lose by default (every novel dev concept requires a new rule), allowlists win by default (a novel dev concept is excluded by absence).
Define a closed set of personal domains, e.g.: health, finance, career,
relationships, preferences, creative, travel, home, life_events,
identity, directives. The biographer tags every extracted entity and claim
with a domain; anything outside the set is dropped at extraction. directives
is the one door adjacent to dev noise — it admits standing rules the user
sets about how they work ("always use X email for commits"), gated by a
durable-rule-vs-transient-task test in the prompt.
Keep the type blocklist (Phase 2) and quality filters as backstops. Ship a kill-switch policy flag so gating can be disabled for debugging.
A scheduled job (every 5–15 min) reading unprocessed session.captured and
knowledge.doc events. Integration events (mail, calendar) are recallable but
NOT biographer input — they're structured data; the biographer is for
narrative text.
Pass 1 — Entities & relations: structured-output prompt →
{entities: [{type, name, domain}], relations: [...]}. Then filter: domain
allowlist, blocked types, and a quality filter dropping role markers, bare
numbers, git SHAs, state flags, code identifiers, file paths, CLI flags, money
amounts. A relation touching a dropped entity is itself dropped. When a name
matches multiple existing entities, ask the LLM to disambiguate (pick one
or declare new).
Pass 2 — Claim drafting: prompt for durable declarative facts →
{claims: [{topic, claim, confidence, domain}]}. Exclude imperatives,
transient details, speculation. Each surviving claim becomes a pending belief
candidate. Cap ~20 candidates per event.
Preprocessing (before any LLM call): strip tool-output blocks and code fences, collapse consecutive same-role turns, drop very short turns; for assistant-heavy sessions keep user turns + final assistant turn. Knowledge docs are plain markdown — treat all content as user-authored.
Multi-tick chunking: large events split into chunks; process a bounded number per tick with a persisted cursor that survives restarts.
Deadline enforcement — inside the loops. The tick must accept a deadline and check it between every chunk, every claim batch, and every disambiguation call — not just at tick start. A deadline checked only between sessions lets one large session run 20+ minutes of sequential LLM calls, wedging the scheduler until the health monitor kills the daemon (see Lessons, §10). Also cap per-batch LLM timeouts so a single call can't consume the whole tick.
Error handling: LLM unreachable → do NOT advance the cursor. Bad output → DO advance. (Phase 3's classification, applied.)
Every 1–2 min: query content rows WHERE embedding IS NULL, batch (~200),
embed, write back, maintain the vector index. Never embed inline during ingest
— semantic recall lags by at most one cycle; lexical recall is instant.
Every ~10 min: scan content directories for *.md, SHA-256 hash each; new
path → ingest as knowledge.doc; changed hash → update in place and nullify
the embedding; same hash → skip. Serves both recall and biographer extraction.
Deterministic pass (no LLM):
- Resolve overdue predictions →
unverifiable, compute Brier - Roll up daily metrics (entities, handler activity, calibration)
- Detect narrative arcs — recurring themes in a rolling ~14-day window (e.g. cluster events sharing ≥2 entities; merge near-duplicate arcs by Jaccard similarity)
- Expire stale candidates (pending > 14 days; conflicts > 30 days)
- Belief freshness scan — per-provenance TTLs; resolver re-verify or
belief.staleflag - Replay corrections — retract belief heads for topic-linked corrections
- Promote candidates when ALL hold: confidence ≥ 0.8, candidate age ≥ 7
days (the age gate lets contradicting evidence arrive first), domain
allowed, and no conflicting head on the topic. A contradicting candidate is
flagged
conflictfor review, never auto-promoted - Merge near-duplicate candidates (same topic, similar claim → keep higher-confidence)
- Link recommendations to actions; update habit observations
Synthesis pass (LLM, degrades gracefully to metrics-only): 10. Re-summarize hot entities (many new relations since last profile; cap per run, e.g. 25) 11. Generate the daily journal (narrative of the day + arcs) 12. Optional depth insights across arcs
| Job | Cadence | LLM | Bound |
|---|---|---|---|
| Biographer | 5–15 min | reasoning | tick deadline + per-batch timeout |
| Embedder | 1–2 min | embed | per-batch timeout |
| Ingest docs | ~10 min | none | — |
| Dream | nightly | summarize (optional) | per-call timeouts |
- Capture receives, classifies, dedups, throttles, and never self-captures
- Biographer extracts entities/relations/claims from sessions and docs, with domain gating, quality filtering, and disambiguation
- Cursor semantics correct: unreachable retries, poison advances
- A deliberately huge event cannot exceed the tick deadline (verify with a clock-injected test)
- Embedder vectorizes in batches; ingest-docs dedups by hash
- Dream: predictions resolved, candidates expired/promoted/conflict-flagged (with the age gate), corrections replayed, staleness flagged, journal written; runs to completion with no LLM available
Goal: A framework for connecting Robin to external APIs and local data, with a tick lifecycle, on-demand actions, and user-extensible plugins.
Manifest: name, version, schedule (cron), permissions (memory
read/write, network hosts, required secrets), optional actions (name,
description, input schema).
Handler: optional tick(ctx) (periodic fetch + ingest), init,
cleanup, health. Action-only integrations (no tick) are valid.
Every handler receives: db, llm (dispatcher), state (per-integration
DB-backed KV — OAuth tokens, cursors, watermarks), log, fetch,
ingest(input), checkOutbound(input).
Scheduler claims the cron job → tick(ctx) runs inside a timeout (e.g.
120s) → fetch → ingest() (dedup by external ID/hash) → return
{status, ingested?, message?} → completion event recorded → consecutive
errors tracked, alert raised past a threshold.
Missing secrets: skip the integration with a warning; never crash the daemon.
Integrations using OAuth refresh-token rotation need all of the following or they will strand the user in re-auth loops:
- Single-flight refresh across processes. MCP servers and the daemon are separate processes sharing the same stored tokens. Two concurrent refreshes of a rotating token family revoke the grant. Use a cross-process lease (DB or lockfile) so exactly one process refreshes; others wait and re-read.
- Never retry a refresh POST after a 5xx. The provider's gateway may have committed the rotation and eaten the response; replaying the old token revokes the family. Fail the tick, apply a cool-off, try fresh next tick.
- Auth-death latch. When a grant is definitively dead (400/401 on
refresh), latch the state and make every subsequent tick skip with a clear
"run
reauth <name>" message — don't hammer the provider. reauthseeds state directly. The CLI re-auth flow must write the newly exchanged tokens into the integration's state (clearing the latch), not just into the env file.
Before any outbound write, checkOutbound() scans for PII (Luhn-validated
card numbers, SSNs) and secrets (common API-key formats). A match blocks the
write and logs an audit event.
| Integration | Cadence | Purpose |
|---|---|---|
| 15 min | Subjects, senders, snippets | |
| Calendar | 30 min | Upcoming events |
| Issues/PRs (Linear, GitHub) | 30 min | Work items |
| Health wearable | 30–60 min | Sleep, recovery, workouts |
| Finance | 15 min | Quotes for tracked symbols |
| Weather | 1 h | Conditions at user's location |
Start with ONE and get it working end-to-end before adding more.
Users drop a manifest + handler into the extensions directory; same contract as built-ins. Restart-to-reload is fine.
- Built-in and extension integrations load and schedule
- Tick timeout contains hung API calls; dedup prevents duplicates
- Action-only integrations answer on-demand queries
- Outbound guard blocks PII/secrets; missing secrets skip, don't crash
- An OAuth integration survives: concurrent refresh attempts (single-flight), a 5xx mid-rotation (no replay), and a dead grant (latch + clean reauth)
- One integration works end-to-end
Goal: Expose memory to AI tools (MCP), provide a setup/maintenance CLI, an HTTP hook endpoint, and session-start context (primer).
Two servers keep the tool count manageable. They are separate processes from the daemon — each must independently load secrets and open the DB, and they serve stale code after a rebuild until the AI-tool session restarts (§10).
Core server (memory + cognition + self-model):
| Tool | Purpose |
|---|---|
recall |
Search events (lexical/vector/hybrid) |
remember |
Write an event |
believe / recall_belief |
Write / query beliefs by topic |
review_beliefs / resolve_belief_candidate |
List and settle pending/conflicted candidates |
find_entity / get / list |
Entity search; fetch by ID; list events/entities/jobs/predictions |
predict / recommend |
Log forecasts and recommendations |
record_correction |
Log an error + fix |
audit / explain |
Event log; narrative explanation of a decision |
health / metrics / alerts |
Daemon health; calibration + counts; open alerts |
journal |
Daily narrative from dream |
power |
Read/write power state |
skill |
Load reusable methodology playbooks (markdown) |
Extension server (integrations + agent + graph):
| Tool | Purpose |
|---|---|
| Integration actions | Generated from loaded integrations' action defs |
related_entities |
N-hop graph traversal |
resolve_prediction / resolve_recommendation |
Mark outcomes |
integration_status |
Per-integration health |
ingest |
Direct event write |
agent / run |
Run an agentic task (Phase 7); run a registered job |
check_action |
Pre-flight trust check for risky operations |
update |
Update runtime policies/rules |
Setup and maintenance, not a daily interface: init, doctor (DB, LLM,
integrations, disk), reauth <name>, reindex, power <state>, plus
read-outs (recommendations, habits, alerts).
POST /hook/session-end (transcript intake), GET /health, GET /ready.
Bind to loopback only — this endpoint receives full session transcripts
and must never be reachable off-machine. Bind 127.0.0.1 explicitly, not
localhost: localhost can resolve dual-stack and land the listener on
IPv6-only (::1), silently breaking clients that connect via 127.0.0.1.
Deterministic assembly — no LLM, no staleness surprises. Budget ~10,000 chars (~2,500 tokens) total:
| Section | Content |
|---|---|
| Behavioral rules | Corrections without a topic — standing directives |
| Learning digest | ~350 chars of self-performance: prediction calibration, belief counts, agent spend, recommendation follow-through |
| Active beliefs | Belief heads: topic + claim + confidence |
| User profile | Inlined character/voice files |
| Knowledge index | Titles of knowledge files |
| Pending items | Counts: candidates, stale beliefs, open predictions |
Put the digest before beliefs — it's short and high-signal; beliefs can be 7k+ chars and get truncated first.
Trigger: regenerate on demand from a session-start hook calling the CLI (freshest), or on a short cron as a fallback. Assembly is cheap and deterministic, so regenerating per-session is fine.
Delivery: a well-known file path the AI tool reads at session start (most portable). Hook-based injection is most seamless where the tool supports it.
The primer is static per session; auto-recall is the second, dynamic layer.
Where the AI tool supports a prompt-submit hook, run each user prompt through
recall and inject the top few matching snippets into the turn's context —
deterministic, no LLM call.
What production tuning taught: skip prompts too short to match (<~12 chars); filter the candidate pool to curated kinds (knowledge docs, beliefs) because raw transcripts and telemetry outrank curated content on raw relevance and drown it; slice long documents to the relevant section rather than injecting teaser truncations; cap the injected budget so snippets can't crowd out the primer. Mark injections clearly as background context, not instructions — and accept that they are partial: the AI tool should open the source file before answering anything load-bearing from a snippet.
- Both MCP servers expose their tool sets; recall/remember/beliefs round-trip
- Integration actions callable through the extension server
- CLI initializes a fresh instance;
doctorreports honestly - HTTP endpoint receives transcripts and triggers capture
- Primer assembles under budget; a fresh session starts with it loaded
Goal: A guarded primitive for multi-turn LLM tool-use loops with cost controls, isolation, verification, and full auditability.
All agentic execution flows through one guarded function. No other code path may run multi-turn LLM loops; shelling out to an AI CLI directly is banned.
Parameters: goal, surface (caller identity), allowedTools (explicit
list, no "allow all"), maxTurns, timeoutMs, maxBudgetUsd, model,
workdir.
Behavior:
- Pre-flight cap check — per-surface daily cap; over → return
cappedwithout spending a dollar - Run the loop: goal + tools → LLM → tool calls → results → repeat
- Enforce limits: hard deadline (abort signal), turn counter, per-run budget — any limit → graceful stop
- Stream every message to a per-run JSONL transcript
- Record one ledger row: tokens, cost, status, surface, duration
- Return
success/capped/timeout/error
The scheduler executes one job at a time; an agent run lasts up to 30 minutes. If a scheduled job calls runAgent inline, all cognition and integrations stall for the duration. Autonomous runs execute in a detached child process: the scheduled job spawns the runner and returns; the runner owns the run end-to-end (ledger, transcript, verifier). Two consequences to design for:
- A detached child has no parent to report errors to — it must log to its own runner log file, and its exit is not release-on-exit for any lock it holds (hence the stale-stealable lockfile below).
- The child inherits its environment from the supervisor, not your shell. Resolve the repo/workdir explicitly; a cwd default that works in dev will silently point elsewhere under the process supervisor.
- Tool allowlisting per run; write-capable tools are explicit opt-in
- Worktree isolation: write-capable runs operate in an isolated copy (git worktree / temp dir / container); changes are a reviewable diff the caller applies or discards
- OS sandboxing for write-capable runs (macOS seatbelt, Linux bubblewrap, container). Fail closed: can't sandbox → error, never run unsandboxed
- Single-flight lock per surface (lockfile with holder PID + timestamp; a lock older than a staleness bound is considered abandoned and stolen, since a crashed holder never releases)
- The agent's working directory is a default, not a boundary — only the sandbox is a boundary. Any handler that can write needs the sandbox even if its cwd looks harmless
Agents report success; sometimes they're wrong (or the model claims work it didn't do). After each run, a deterministic verifier per handler checks a post-condition against the world — no LLM:
- "Research" → a
research.briefevent exists since run start - "KB curation" → a knowledge file mtime changed since run start
- "Health remediation" → the worktree has a diff
- Handlers with no checkable post-condition return
unverifiable
fail (claimed work, no observable effect) records an outcome-mismatch and
raises an alert. This one mechanism catches goal/toolset drift, permission
regressions, and hallucinated success — cheaply.
| Layer | Scope |
|---|---|
| Per-run budget | One run |
| Per-surface daily cap | All runs on a surface per day |
| Global daily cap | All LLM calls (dispatcher, Phase 3) |
Suggested starting points: $5/run, 30 turns, 30 min, $25–50/surface/day. Size
per-run budgets to the most expensive legitimate run — a cap below one full
run of your largest handler starves it silently (diagnose via ledger rows with
status capped).
Specialized handlers, not "do anything" agents. The production set, as a menu — start with 2–3:
| Handler | Purpose | Mode |
|---|---|---|
| Self-improvement | Refine extraction/belief quality | worktree write |
| Research | Multi-source web research → ingested brief | read + ingest |
| Integration ops | Exercise/fix integrations via their actions | scoped |
| KB curation | Organize, dedupe knowledge files | scoped write |
| Belief reconciliation | Hunt contradictions, propose fixes | propose-only |
| Prediction calibration | Resolve open predictions, report drift | read + resolve |
| Gap fill | Find knowledge gaps → research → fill | scoped write |
| Dream enrichment | Cross-session synthesis over the corpus | propose-only |
| Life executor | Real-world actions (reservations, rides) | hard risk gate |
| Integration authoring | "Integrate with X" → scaffold in a worktree | worktree write |
| Health remediation | Diagnose invariant trips; propose config/code fix | worktree write |
| Daily brief | Morning synthesis across integrations + memory | read-only |
Three lessons baked into this table: (1) propose-only handlers write proposals
as events for review, never directly into beliefs; (2) a handler's goal text,
allowlist, and verifier must agree — a goal that says "write a journal" with a
read-only toolset fails forever, and only the verifier will tell you; (3)
handlers that act in the real world (reservations, purchases, messages) sit
behind a risk gate: a pre-flight check (check_action) classifies the
action, and irreversible or spend-bearing actions require explicit user
confirmation by policy — autonomy is for gathering and proposing, not for
spending.
- runAgent enforces budget, turns, timeout; cap pre-flight returns
capped - An autonomous run executes detached — the scheduler keeps ticking during it, and the child's errors land in its runner log
- Every run: JSONL transcript + ledger row
- Allowlisting blocks unauthorized tools; worktree isolates writes; sandbox fails closed
- Verifiers pass/fail correctly against seeded post-conditions; a fail raises an alert
- One handler works end-to-end, verified
Goal: Directory layout, extension system, first-run setup, multi-machine.
user-data/
├── config/
│ ├── llm-config.* # role → provider mapping
│ ├── policies.* # power, capture, gating, agent caps
│ └── secrets/.env # keys + tokens (restrictive perms)
├── state/db/ # database file(s)
├── content/
│ ├── profile/ # WHO the user is (character, relationships, people/)
│ ├── knowledge/ # WHAT the user knows (by topic)
│ └── artifacts/ # generated outputs
├── extensions/
│ ├── integrations/ # user-added integrations
│ ├── jobs/ # user-added scheduled jobs
│ └── skills/ # user-added methodology files
├── logs/
└── agent-runs/ # per-run JSONL transcripts
content/ enforces personal-data-only: no dev artifacts, code docs, or
session logs. User skills shadow system skills of the same name.
- Create
user-data/structure with sensible default configs - Prompt for / detect LLM credentials
- Initialize DB, apply migrations
- Install process supervisor config
- Register MCP servers with the AI tool (or print instructions)
- Install session-capture hooks
- Run
doctorto verify
content/ is the irreplaceable data — knowledge and profile files carry the
durable memory. Back it up additively (never --delete-style mirroring; a
local corruption must not propagate) to durable storage, plus secrets
separately with restrictive permissions. The database is rebuildable from
content + fresh sessions, but a periodic DB snapshot makes recovery fast.
Record a success timestamp per backup leg and have doctor warn when a leg
goes stale — a backup that silently stopped is the failure mode, not a backup
that errors.
Sync content/ + extensions/ (private git, Syncthing); each machine runs
its own daemon, DB, config, and secrets. The database is never synced — it
is a derived index; a new machine's graph rebuilds organically from synced
knowledge files + new sessions. Session history does not transfer; knowledge
files carry the durable content.
initproduces a complete instance;doctorverifies it- Extension integrations/jobs/skills load and run
- Content files are indexed and searchable
- Fresh machine:
init+ synced content → running Robin
Every item here is a production failure from the reference implementation and the design change that fixed it. Build these in from the start.
1. Deadlines must bind inside work loops. A biographer tick that checked its deadline only between sessions ran 20+ minutes on one large session, starving the scheduler heartbeat until the health monitor killed the daemon — roughly hourly, and the restart re-claimed the same event. Check the deadline between every chunk, claim batch, and disambiguation call, and cap per-batch LLM timeouts. Symptom to watch for: "skipped cron jobs" that look like integration staleness are often daemon start/stop churn — check daemon lifecycle first.
2. Blocklists lose; allowlists win. Months of adding dev-noise rules (entity types, name patterns) never kept the graph clean — every novel dev concept needed a new rule. Inverting to a closed personal-domain allowlist fixed it structurally: novel noise is excluded by absence. Keep the blocklists only as backstops.
3. Separate processes, separate initialization. MCP servers are not the
daemon. Each process must load secrets and open the DB itself; forgetting is a
silent failure ("integration skips: key not set"). And after a rebuild, the
daemon picks up new code on restart, but MCP servers run stale code until the
AI-tool session restarts. Make doctor report which code version each process
runs.
4. OAuth refresh-token rotation is hostile. Concurrent refreshes from separate processes revoke the token family → cross-process single-flight lease. A gateway 5xx may mean the rotation committed server-side → never replay a refresh POST. A dead grant should latch (skip ticks with a clear reauth message), and re-auth must seed tokens into integration state, not just the env file.
5. Suppress self-capture. Once Robin runs its own agents, their sessions look like user sessions. Without suppression + a per-tick capture cap, the system recursively ingests its own output.
6. Verify autonomous work deterministically. An agent that "succeeded" with a goal its toolset couldn't perform failed silently for weeks. Cheap post-condition verifiers (did a row appear? did a file change? is there a diff?) plus an alert on mismatch catch this class permanently.
7. Budget caps starve silently. A per-run budget below the cost of one
legitimate run of your biggest handler doesn't error loudly — the run just
stops mid-task with status capped. Size caps to the most expensive
legitimate run and audit the ledger for capped rows.
8. Cursor discipline is the pipeline's immune system. The unreachable-vs-bad-output distinction (Phase 3) shows up everywhere: capture, biographer, embedder, integrations. Get it right once in the dispatcher's error taxonomy and honor it in every consumer.
9. Promotion needs an age gate. Auto-promoting high-confidence candidates immediately turns one misread session into durable false "truth." The 7-day age gate gives contradicting evidence time to arrive; conflicts flag for review instead of last-write-wins.
10. Restart after deploy, always. A long-running daemon serves stale in-memory code and prompts until restarted. Make restart cheap, idempotent, and part of the deploy habit.
| Term | Definition |
|---|---|
| Alert | A persistent record of a detected problem (invariant trip, integration failure, verifier mismatch); auto-resolves on next success. |
| Arc | A recurring theme detected across recent events (shared-entity clustering over a rolling window). |
| Auto-recall | Deterministic per-turn injection of recall snippets matched to the user's prompt, via a prompt-submit hook. |
| Belief / belief head | A durable, topic-keyed claim stored as an event; the head is the latest non-retracted event on the topic — current truth. |
| Belief candidate | A draft claim from the biographer awaiting promotion, rejection, or conflict review. |
| Biographer | The cognition job extracting entities, relations, and claims from narrative text. |
| Correction | A record of something the assistant got wrong + the fix; behavioral (standing directive) or topic-linked (retracts a belief). |
| Cron re-arming | After any completion — including failure — a cron job's next run is inserted. |
| Disambiguation | LLM resolution of an extracted name against existing entities. |
| Dispatcher | The single chokepoint for LLM calls: role routing, timeouts, spend caps, error taxonomy. |
| Domain allowlist | The closed set of personal domains extraction is limited to; the structural noise gate. |
| Dream | Nightly consolidation: resolve predictions, promote candidates, replay corrections, link recommendations, write the journal. |
| Embedder | Batch job turning NULL-embedding content rows into vectors. |
| Event | An immutable record in the append-only store; everything is an event. |
| Habit | A soft behavioral pattern that graduates with repeated observation and retires when it stops. |
| Ingest | Write an event (+ content) with external-ID dedup. |
| Integration | A module connecting Robin to an external source via tick/actions. |
| Lease | A claim timestamp preventing concurrent daemons from running the same job. |
| Primer | Deterministic session-start context: rules, digest, beliefs, profile, index. |
| Provenance | A belief's evidence class (first-party, inferred, third-party, external, unknown); drives staleness and decay. |
| Recall | Search over the event store (lexical, vector, hybrid). |
| Recommendation | Logged advice whose follow-through is detected and scored, like predictions. |
| Resolver | An automated re-checker registered by topic prefix that re-verifies stale beliefs. |
| runAgent | The single guarded function for all multi-turn tool-use loops. |
| Single-flight | A cross-process lock (with stale-steal) ensuring one holder: agent runs per surface, OAuth refreshes per integration. |
| Skill | A markdown playbook encoding a reusable methodology. |
| Supersede | Replace a topic's belief head with a newer event; the old one remains in the log. |
| Surface | A caller identity for agent runs, each with its own daily cap. |
| Verifier | A deterministic post-condition check that an agent run's claimed work is observable in the world. |
| Worktree | An isolated copy of a working directory where write-capable agent runs produce reviewable diffs. |