Skip to content

Commit c75b8db

Browse files
author
Tehan
committed
feat(external-memory): Hindsight backend tee and unified external recall
Vertical sync of durable knowledge to an external Hindsight memory backend, orthogonal to upstream's local workspace sharing: each project keeps its own bank and sharing stays local-only, so the two axes never diverge. - Write path: historian fact promotion and `ctx_memory` mutations tee to the backend post-commit (update/archive/merge/verify all propagate). - Read path: three-slice recall at session start (project bank, user profile via scope:user, global via recall tags), rendered as an <external-memory> block with the profile slice merged into <user-profile>. - Cache safety: the recall snapshot rides m[1] as a delta and is explicitly subtracted from pressure-refold accounting, so it can never trigger an m[0] hard fold. `externalRecallHash` appears 0x inside `mustMaterialize` -- an invariant re-verified on every upstream rebase. - `ctx_search` gains an explicit "external" source (opt-in, 5s abort). - Full OpenCode + Pi parity. Deliberately never proposed upstream: it couples the plugin to an external service, and upstream is actively simplifying the memory system (see cortexkit#335). This commit also carries the union resolutions that let both fork features coexist -- shared fork migration lane, merged status/RPC/TUI surfaces, and the combined prompt-surface budget.
1 parent ba4968d commit c75b8db

84 files changed

Lines changed: 7179 additions & 266 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎CONFIGURATION.md‎

Lines changed: 70 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -592,6 +592,69 @@ Cross-session memory settings. All memories are scoped to the current project (i
592592
| `injection_budget_tokens` | `number` (500–20000) | `4000` | Token budget for memory injection into `<session-history>`. |
593593
| `auto_promote` | `boolean` | `true` | Promote eligible session facts to project memories automatically after historian or `/ctx-recomp` runs. When `false`, historian and recomp do not write any new memories — agents can still create memories explicitly via `ctx_memory write`, and existing memories continue to be injected and searched normally. |
594594
| `retrieval_count_promotion_threshold` | `number` | `3` | Retrievals needed before a memory is auto-promoted to permanent. |
595+
| `external` | `object` | See below | **User-config-only.** Long-term memory backend (Hindsight) — tees curated writes OUT and recalls them BACK once per session, plus an explicit-only `ctx_search` source. A cloned repo cannot redirect the endpoint or read a user's personal memory store. |
596+
597+
### `memory.external`
598+
599+
The `memory.external` block controls the **external memory backend** (Hindsight): a long-term companion store that holds curated memories OUT of the project (so they survive across projects, harness restarts, and (in future) the user's whole fleet) and recalls them BACK once per session. The local SQLite store remains the source of truth; the external store is a long-term companion.
600+
601+
**Security:** this entire block is **user-config-only**. `stripUnsafeProjectConfigFields()` in `src/config/project-security.ts` drops it from project-level config (parallel to `auto_update` and `sqlite`) — a cloned repo cannot redirect the endpoint, exfiltrate a user's personal memory store, or read a user's external memory by editing `magic-context.jsonc` in a project root. Set the block in `~/.config/opencode/magic-context.jsonc` (OpenCode) or `~/.pi/agent/magic-context.jsonc` (Pi).
602+
603+
```jsonc
604+
{
605+
"memory": {
606+
"external": {
607+
"provider": "off", // "off" (default) or "hindsight"
608+
"endpoint": "http://10.0.0.1:8889", // required when provider is hindsight; Hindsight base URL
609+
"api_key": "{env:HINDSIGHT_API_KEY}", // optional bearer token
610+
"project_bank": "mc-{name}-{id8}", // project bank name template; {name}=project basename, {id8}=first 8 chars of the project identity hash
611+
"main_bank": "user-memories", // required when provider is hindsight; bank for user + global scope. Assumed to pre-exist; never created or modified by the plugin
612+
"retain_sources": ["historian", "agent", "dreamer"], // which creation points tee (gate write-side, not read-side)
613+
"tags": [], // static tags attached to every retained item
614+
"recall": { // see `memory.external.recall` below
615+
"enabled": true,
616+
"timeout_ms": 3000,
617+
"max_tokens": 2048,
618+
"dedup_threshold": 0.85,
619+
"global_tags": [],
620+
"global_from_prompt": false,
621+
"search": true,
622+
"mental_models": true,
623+
"profile_mental_models": ["user-preferences"]
624+
}
625+
}
626+
}
627+
}
628+
```
629+
630+
| Field | Type | Default | Description |
631+
|-------|------|---------|-------------|
632+
| `provider` | `"hindsight"` \| `"off"` | `"off"` | Backend implementation. `"off"` disables the whole feature. `"hindsight"` activates tee-on-write + once-per-session recall + the explicit `ctx_search` source. |
633+
| `endpoint` | `string` | — | Required when `provider: "hindsight"`. Hindsight base URL (e.g. `http://10.0.0.1:8889`). Trailing slashes are stripped. SSRF-guarded. |
634+
| `api_key` | `string` | — | Optional bearer token. Supports `{env:VAR}` substitution. Never logged. |
635+
| `project_bank` | `string` (template) | `"mc-{name}-{id8}"` | Project-bank name template. Placeholders: `{name}` = sanitized project basename, `{id8}` = first 8 chars of the project identity hash. Created on first retain (PUT with the project bank mission). |
636+
| `main_bank` | `string` | — | Required when `provider: "hindsight"`. Bank for `user` and `global` scope items. **Assumed to pre-exist; the plugin never creates or modifies it.** |
637+
| `retain_sources` | `string[]` | `["historian","agent","dreamer"]` | Which creation points tee to the external backend. `historian` = fact promotion in `src/features/magic-context/memory/promotion.ts`. `agent` = `ctx_memory` `write` / `update` in `src/tools/ctx-memory/tools.ts`. `dreamer` = user-memory promotion in `src/features/magic-context/user-memory/review-user-memories.ts`. Read paths (recall, search, mental-models) and W2 correctives (archive/update/verify corrective propagation) are NOT gated by this list. |
638+
| `tags` | `string[]` | `[]` | Static tags attached to every retained item (in addition to the always-present `source:magic-context`, `category:<X>`, and the per-scope `project:<id>` / `scope:user` / `scope:global` tags). |
639+
| `recall` | `object` | See below | Unified read path — session-start recall + `ctx_search` external source. |
640+
641+
### `memory.external.recall`
642+
643+
| Field | Type | Default | Description |
644+
|-------|------|---------|-------------|
645+
| `enabled` | `boolean` | `true` | Session-start recall from the external backend, merged into the context injection. Late results ride the m[1] `<external-memory>` delta; the first m[0] render awaits up to `timeout_ms` (cache-cold path). |
646+
| `timeout_ms` | `number` (500–15000) | `3000` | Max wait for recall at the first render of a session (when the cache is already cold). Late results arrive as an m[1] delta on later passes. |
647+
| `max_tokens` | `number` (256–8192) | `2048` | Per-slice token budget — project / profile / global each get this cap. The external block is NOT charged to the history or local-memory budget; it is bounded solely by this × 3 slices. |
648+
| `dedup_threshold` | `number` (0.5–0.99) | `0.85` | Cosine similarity above which a recalled item is dropped as a duplicate of a local memory (or active user memory). Hash-only fallback when embeddings are unavailable. |
649+
| `global_tags` | `string[]` | `[]` | Tag filter for the global (main-bank) recall slice, matched with `tags_match: "any"` (untagged content INCLUDED). `[]` = no filter sent (full autoRecall replacement). |
650+
| `global_from_prompt` | `boolean` | `false` | Include an excerpt of the session's first user prompt in the global-slice recall query (the project name is always included). The first prompt is fixed for the session, so the query — and the frozen recall snapshot — stays deterministic across crash-recovery re-fires. |
651+
| `search` | `boolean` | `true` | Expose the `ctx_search` `"external"` source (project + main bank). **Explicit-only** — the auto-search hot path (every user prompt hint) NEVER hits it, even when this is `true`. |
652+
| `mental_models` | `boolean` | `true` | Use Hindsight mental models as the fast path for the project and profile recall slices (single GET, server-refreshed), falling back to full recall when absent/empty. The global slice always uses full recall. |
653+
| `profile_mental_models` | `string[]` | `["user-preferences"]` | Main-bank mental-model names (case-insensitive) used for the profile slice. The main bank is never modified by the plugin — create these manually. Project-bank mental models are seeded by the plugin (`project-conventions`, `project-decisions`) on the first successful retain. |
654+
655+
**Engine-agnostic interface.** `external-memory-provider.ts` defines a neutral `ExternalMemoryBackend` interface (mirrors the `EmbeddingProvider` pattern). Hindsight is the only shipped implementation today. Document identity is `mc:<scopeKey>:<category>:<hash>` (content-derived → idempotent re-retains upsert on the server; retries never duplicate). Bank routing: `scope: "project"` → `project_bank` (per project); `scope: "user"` and `scope: "global"` → `main_bank`. Project items carry `project:<id>` + `project-name:<basename>` tags; globals carry `scope:global` plus `origin-project:*` provenance tags and a `context` field that names the originating project so Hindsight's fact extractor links it as an entity.
656+
657+
**Failure semantics.** Hindsight calls are fire-and-forget; failures are logged, never thrown. The impl carries a circuit breaker (3 fails in 60s → open 5min → half-open probe) so a hung endpoint can't drag every plugin operation through its timeout. A 422 (memory-defense rejected content) is treated as a hard no and never retried; 404 on a missing bank is benign (returns empty for the slice). The bearer token is never logged. Cross-harness: OpenCode and Pi share the same banks (the impl is a single, neutral interface and the project bank name is project-derived, so both harnesses resolve the same bank).
595658

596659
---
597660

@@ -863,7 +926,13 @@ skill-memory:
863926
"injection_budget_tokens": 4000,
864927
"auto_promote": true,
865928
"auto_search": { "enabled": true, "score_threshold": 0.6, "min_prompt_chars": 20 },
866-
"git_commit_indexing": { "enabled": false, "since_days": 365, "max_commits": 2000 }
929+
"git_commit_indexing": { "enabled": false, "since_days": 365, "max_commits": 2000 },
930+
"external": { // USER-LEVEL ONLY — stripped from project config
931+
"provider": "hindsight",
932+
"endpoint": "http://10.0.0.1:8889",
933+
"api_key": "{env:HINDSIGHT_API_KEY}",
934+
"main_bank": "user-memories"
935+
}
867936
},
868937

869938
"sidekick": {

‎README.md‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -229,7 +229,7 @@ Because it runs during idle time, the dreamer pairs well with local models, even
229229

230230
*The right memory at the right moment.* Every turn, active project memories and the compacted session history are injected automatically and cache-stably. On demand, the agent reaches for:
231231

232-
- **`ctx_search`**: one query across project **memories**, raw **conversation** history, indexed **git commits**, parked **notes**, and promoted **primers**. Semantic embeddings with full-text fallback.
232+
- **`ctx_search`**: one query across project **memories**, raw **conversation** history, indexed **git commits**, parked **notes**, promoted **primers**, and (opt-in) the long-term **external** memory backend. Semantic embeddings with full-text fallback.
233233

234234
```
235235
ctx_search(query="why did we pick event sourcing for orders")
@@ -241,6 +241,8 @@ Because it runs during idle time, the dreamer pairs well with local models, even
241241

242242
Recall works **across sessions** (a new session inherits everything) and **across harnesses** (write a memory in OpenCode, retrieve it in Pi or OMP).
243243

244+
> **Long-term memory** *(opt-in, off by default)* tees curated writes (historian promotions, `ctx_memory` writes, dreamer user-memory promotions) to a Hindsight backend and recalls them BACK once per session as a `<external-memory>` block — across sessions, across harnesses, and across projects for the global slice. Same `ctx_search` "external" source (explicit-only, never on the auto-search hot path). User-config-only; a repo cannot redirect the endpoint. Configure under `memory.external`; see [CONFIGURATION.md](./CONFIGURATION.md#memoryexternal).
245+
>
244246
> **Auto search hints** *(on by default)* run a background `ctx_search` each turn and whisper a "vague recall" when something relevant exists — like almost remembering a note you took. It appends only compact fragments, never full content; set `memory.auto_search.enabled: false` to turn it off. **Git commit indexing** *(opt-in)* makes your project history semantically searchable as an additional `ctx_search` source — enable with `memory.git_commit_indexing.enabled: true`.
245247
246248
### Agent tools at a glance
@@ -249,7 +251,7 @@ Recall works **across sessions** (a new session inherits everything) and **acros
249251
|------|-------|-------------|
250252
| `ctx_reduce` | Context | Queue stale tagged content for removal, cache-aware |
251253
| `ctx_memory` | Capture | Write or delete durable cross-session memories |
252-
| `ctx_search` | Recall | Search memories, conversation history, git commits, notes, and primers |
254+
| `ctx_search` | Recall | Search memories, conversation history, git commits, notes, primers, and external long-term memory |
253255
| `ctx_expand` | Recall | Decompress a history range back to the transcript |
254256
| `ctx_note` | Recall | Deferred intentions and dreamer-evaluated smart notes |
255257
| `ctx_skill_note` | Recall | Write back a per-skill note (gotcha/discovery/fix/workflow) for future loads |

0 commit comments

Comments
 (0)