You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 9bc5511
Browse filesBrowse the repository at this point in the historyBrowse files
Tehan
committed
feat(external-memory): Hindsight backend tee and unified external recall
Vertical sync of durable knowledge to an external Hindsight memory backend,
orthogonal to upstream's local workspace sharing: each project keeps its own
bank and sharing stays local-only, so the two axes never diverge.
- Write path: historian fact promotion and `ctx_memory` mutations tee to the
backend post-commit (update/archive/merge/verify all propagate).
- Read path: three-slice recall at session start (project bank, user profile
via scope:user, global via recall tags), rendered as an <external-memory>
block with the profile slice merged into <user-profile>.
- Cache safety: the recall snapshot rides m[1] as a delta and is explicitly
subtracted from pressure-refold accounting, so it can never trigger an
m[0] hard fold. `externalRecallHash` appears 0x inside `mustMaterialize`
-- an invariant re-verified on every upstream rebase.
- `ctx_search` gains an explicit "external" source (opt-in, 5s abort).
- Full OpenCode + Pi parity.
Deliberately never proposed upstream: it couples the plugin to an external
service, and upstream is actively simplifying the memory system (see cortexkit#335).
This commit also carries the union resolutions that let both fork features
coexist -- shared fork migration lane, merged status/RPC/TUI surfaces, and
the combined prompt-surface budget.
Copy file name to clipboardExpand all lines: CONFIGURATION.md
+70-1Lines changed: 70 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -600,6 +600,69 @@ Cross-session memory settings. All memories are scoped to the current project (i
600
600
|`injection_budget_tokens`|`number` (500–20000) |`4000`| Token budget for memory injection into `<session-history>`. |
601
601
|`auto_promote`|`boolean`|`true`| Promote eligible session facts to project memories automatically after historian or `/ctx-recomp` runs. When `false`, historian and recomp do not write any new memories — agents can still create memories explicitly via `ctx_memory write`, and existing memories continue to be injected and searched normally. |
602
602
|`retrieval_count_promotion_threshold`|`number`|`3`| Retrievals needed before a memory is auto-promoted to permanent. |
603
+
|`external`|`object`| See below |**User-config-only.** Long-term memory backend (Hindsight) — tees curated writes OUT and recalls them BACK once per session, plus an explicit-only `ctx_search` source. A cloned repo cannot redirect the endpoint or read a user's personal memory store. |
604
+
605
+
### `memory.external`
606
+
607
+
The `memory.external` block controls the **external memory backend** (Hindsight): a long-term companion store that holds curated memories OUT of the project (so they survive across projects, harness restarts, and (in future) the user's whole fleet) and recalls them BACK once per session. The local SQLite store remains the source of truth; the external store is a long-term companion.
608
+
609
+
**Security:** this entire block is **user-config-only**. `stripUnsafeProjectConfigFields()` in `src/config/project-security.ts` drops it from project-level config (parallel to `auto_update` and `sqlite`) — a cloned repo cannot redirect the endpoint, exfiltrate a user's personal memory store, or read a user's external memory by editing `magic-context.jsonc` in a project root. Set the block in `~/.config/opencode/magic-context.jsonc` (OpenCode) or `~/.pi/agent/magic-context.jsonc` (Pi).
610
+
611
+
```jsonc
612
+
{
613
+
"memory": {
614
+
"external": {
615
+
"provider":"off", // "off" (default) or "hindsight"
616
+
"endpoint":"http://10.0.0.1:8889", // required when provider is hindsight; Hindsight base URL
"project_bank":"mc-{name}-{id8}", // project bank name template; {name}=project basename, {id8}=first 8 chars of the project identity hash
619
+
"main_bank":"user-memories", // required when provider is hindsight; bank for user + global scope. Assumed to pre-exist; never created or modified by the plugin
620
+
"retain_sources": ["historian", "agent", "dreamer"], // which creation points tee (gate write-side, not read-side)
621
+
"tags": [], // static tags attached to every retained item
622
+
"recall": { // see `memory.external.recall` below
623
+
"enabled":true,
624
+
"timeout_ms":3000,
625
+
"max_tokens":2048,
626
+
"dedup_threshold":0.85,
627
+
"global_tags": [],
628
+
"global_from_prompt":false,
629
+
"search":true,
630
+
"mental_models":true,
631
+
"profile_mental_models": ["user-preferences"]
632
+
}
633
+
}
634
+
}
635
+
}
636
+
```
637
+
638
+
| Field | Type | Default | Description |
639
+
|-------|------|---------|-------------|
640
+
|`provider`|`"hindsight"`\|`"off"`|`"off"`| Backend implementation. `"off"` disables the whole feature. `"hindsight"` activates tee-on-write + once-per-session recall + the explicit `ctx_search` source. |
641
+
|`endpoint`|`string`| — | Required when `provider: "hindsight"`. Hindsight base URL (e.g. `http://10.0.0.1:8889`). Trailing slashes are stripped. SSRF-guarded. |
|`project_bank`|`string` (template) |`"mc-{name}-{id8}"`| Project-bank name template. Placeholders: `{name}` = sanitized project basename, `{id8}` = first 8 chars of the project identity hash. Created on first retain (PUT with the project bank mission). |
644
+
|`main_bank`|`string`| — | Required when `provider: "hindsight"`. Bank for `user` and `global` scope items. **Assumed to pre-exist; the plugin never creates or modifies it.**|
645
+
|`retain_sources`|`string[]`|`["historian","agent","dreamer"]`| Which creation points tee to the external backend. `historian` = fact promotion in `src/features/magic-context/memory/promotion.ts`. `agent` = `ctx_memory``write` / `update` in `src/tools/ctx-memory/tools.ts`. `dreamer` = user-memory promotion in `src/features/magic-context/user-memory/review-user-memories.ts`. Read paths (recall, search, mental-models) and W2 correctives (archive/update/verify corrective propagation) are NOT gated by this list. |
646
+
|`tags`|`string[]`|`[]`| Static tags attached to every retained item (in addition to the always-present `source:magic-context`, `category:<X>`, and the per-scope `project:<id>` / `scope:user` / `scope:global` tags). |
|`enabled`|`boolean`|`true`| Session-start recall from the external backend, merged into the context injection. Late results ride the m[1]`<external-memory>` delta; the first m[0] render awaits up to `timeout_ms` (cache-cold path). |
654
+
|`timeout_ms`|`number` (500–15000) |`3000`| Max wait for recall at the first render of a session (when the cache is already cold). Late results arrive as an m[1] delta on later passes. |
655
+
|`max_tokens`|`number` (256–8192) |`2048`| Per-slice token budget — project / profile / global each get this cap. The external block is NOT charged to the history or local-memory budget; it is bounded solely by this × 3 slices. |
656
+
|`dedup_threshold`|`number` (0.5–0.99) |`0.85`| Cosine similarity above which a recalled item is dropped as a duplicate of a local memory (or active user memory). Hash-only fallback when embeddings are unavailable. |
657
+
|`global_tags`|`string[]`|`[]`| Tag filter for the global (main-bank) recall slice, matched with `tags_match: "any"` (untagged content INCLUDED). `[]` = no filter sent (full autoRecall replacement). |
658
+
|`global_from_prompt`|`boolean`|`false`| Include an excerpt of the session's first user prompt in the global-slice recall query (the project name is always included). The first prompt is fixed for the session, so the query — and the frozen recall snapshot — stays deterministic across crash-recovery re-fires. |
659
+
|`search`|`boolean`|`true`| Expose the `ctx_search``"external"` source (project + main bank). **Explicit-only** — the auto-search hot path (every user prompt hint) NEVER hits it, even when this is `true`. |
660
+
|`mental_models`|`boolean`|`true`| Use Hindsight mental models as the fast path for the project and profile recall slices (single GET, server-refreshed), falling back to full recall when absent/empty. The global slice always uses full recall. |
661
+
|`profile_mental_models`|`string[]`|`["user-preferences"]`| Main-bank mental-model names (case-insensitive) used for the profile slice. The main bank is never modified by the plugin — create these manually. Project-bank mental models are seeded by the plugin (`project-conventions`, `project-decisions`) on the first successful retain. |
662
+
663
+
**Engine-agnostic interface.**`external-memory-provider.ts` defines a neutral `ExternalMemoryBackend` interface (mirrors the `EmbeddingProvider` pattern). Hindsight is the only shipped implementation today. Document identity is `mc:<scopeKey>:<category>:<hash>` (content-derived → idempotent re-retains upsert on the server; retries never duplicate). Bank routing: `scope: "project"` → `project_bank` (per project); `scope: "user"` and `scope: "global"` → `main_bank`. Project items carry `project:<id>` + `project-name:<basename>` tags; globals carry `scope:global` plus `origin-project:*` provenance tags and a `context` field that names the originating project so Hindsight's fact extractor links it as an entity.
664
+
665
+
**Failure semantics.** Hindsight calls are fire-and-forget; failures are logged, never thrown. The impl carries a circuit breaker (3 fails in 60s → open 5min → half-open probe) so a hung endpoint can't drag every plugin operation through its timeout. A 422 (memory-defense rejected content) is treated as a hard no and never retried; 404 on a missing bank is benign (returns empty for the slice). The bearer token is never logged. Cross-harness: OpenCode and Pi share the same banks (the impl is a single, neutral interface and the project bank name is project-derived, so both harnesses resolve the same bank).
Copy file name to clipboardExpand all lines: README.md
+4-2Lines changed: 4 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -262,7 +262,7 @@ Because it runs during idle time, the dreamer pairs well with local models, even
262
262
263
263
*The right memory at the right moment.* Every turn, active project memories and the compacted session history are injected automatically and cache-stably. On demand, the agent reaches for:
264
264
265
-
-**`ctx_search`**: one query across project **memories**, raw **conversation** history, indexed **git commits**, parked **notes**, and promoted **primers**. Semantic embeddings with full-text fallback.
265
+
-**`ctx_search`**: one query across project **memories**, raw **conversation** history, indexed **git commits**, parked **notes**, promoted **primers**, and (opt-in) the long-term **external** memory backend. Semantic embeddings with full-text fallback.
266
266
267
267
```
268
268
ctx_search(query="why did we pick event sourcing for orders")
@@ -274,6 +274,8 @@ Because it runs during idle time, the dreamer pairs well with local models, even
274
274
275
275
Recall works **across sessions** (a new session inherits everything) and **across harnesses** (write a memory in OpenCode, retrieve it in Pi or OMP).
276
276
277
+
> **Long-term memory***(opt-in, off by default)* tees curated writes (historian promotions, `ctx_memory` writes, dreamer user-memory promotions) to a Hindsight backend and recalls them BACK once per session as a `<external-memory>` block — across sessions, across harnesses, and across projects for the global slice. Same `ctx_search` "external" source (explicit-only, never on the auto-search hot path). User-config-only; a repo cannot redirect the endpoint. Configure under `memory.external`; see [CONFIGURATION.md](./CONFIGURATION.md#memoryexternal).
278
+
>
277
279
> **Auto search hints***(on by default)* run a background `ctx_search` each turn and whisper a "vague recall" when something relevant exists — like almost remembering a note you took. It appends only compact fragments, never full content; set `memory.auto_search.enabled: false` to turn it off. **Git commit indexing***(opt-in)* makes your project history semantically searchable as an additional `ctx_search` source — enable with `memory.git_commit_indexing.enabled: true`.
278
280
279
281
### Agent tools at a glance
@@ -282,7 +284,7 @@ Recall works **across sessions** (a new session inherits everything) and **acros
282
284
|------|-------|-------------|
283
285
|`ctx_reduce`| Context | Queue stale tagged content for removal, cache-aware |
284
286
|`ctx_memory`| Capture | Write or delete durable cross-session memories |
0 commit comments