Skip to content

Commit f92c901

Browse files
committed
fix(pi-plugin): close council audit findings (#1, #3, #4, #5, #7) + Pi README
The Step 5b/5c council audit found several real config-flow gaps where the loaded magic-context.jsonc was being silently ignored by Pi's extension entry point. All confirmed findings are addressed here: #1 CRITICAL (unanimous 9/9): `protectedTags=0` was hardcoded into `applyPendingOperations` despite Step 5b shipping the config loader. Recent-turn drops were materializing mid-task, defeating the whole purpose of the protected-tag window. Now threads `config.protected_tags` (defaults to schema value 20) through `PiContextHandlerOptions`. #3 HIGH: `memoryEnabled: true` and `injectDocs: true` were hardcoded in the `before_agent_start` system-prompt injector. Memories + `<project-docs>` were getting injected even when the user had explicitly set `memory.enabled=false` or `dreamer.inject_docs=false`. Now respects both config keys. #4 HIGH: `package.json` declared `types: "dist/index.d.ts"` but the build emits no .d.ts files (the bundle has `@magic-context/core` inlined, declarations would be broken anyway). Removed the stale `types` field and the matching `exports['.'].types` entry — Pi doesn't load extensions through TypeScript, so consumers don't need type definitions. #5 HIGH: Cross-harness embedding model mismatch silently returned zero search results. Now warns at Pi startup if the project's stored embedding vectors were produced by a different model than what's currently configured, with explicit guidance toward the dreamer re-embedding sweep. #7 HIGH: Dreamer hardcoded `embeddingConfig: { provider: 'off' }` and `memoryEnabled: false`, making most dreamer tasks (consolidate, verify, improve) into no-ops. Now threads `config.embedding` and `config.memory.enabled` through `PiDreamerOptions`. #2 HIGH: Pi had no README. Created a comprehensive package-level README.md covering installation, configuration, slash commands, storage layout, cross-harness coherence, the `pi --print` mode limitation, and the architecture/module breakdown. Documents the intentional ctx_expand/ctx_reduce gap as a Pi-specific design choice rather than a missing feature. Tests: existing test that drops the second message now opts into `protectedTags: 0` since the new default (20) correctly defers the drop. `PiContextHandlerOptions.protectedTags` is optional with a 20 fallback so existing test fixtures don't need updating. All 1079 tests pass across all three packages.
1 parent 4d3c7f0 commit f92c901

6 files changed

Lines changed: 289 additions & 13 deletions

File tree

‎packages/pi-plugin/README.md‎

Lines changed: 183 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,183 @@
1+
# Magic Context — Pi extension
2+
3+
Cross-session memory and context management for [Pi coding agent](https://github.com/mariozechner/pi-mono). Shares the same SQLite database as the [OpenCode plugin](https://www.npmjs.com/package/@cortexkit/opencode-magic-context), so memories, embeddings, dreamer state, and project knowledge follow you across both harnesses.
4+
5+
> **Status:** v0.1.0 — first release. Production-ready for interactive Pi sessions; `pi --print` mode has a known limitation (see below).
6+
7+
---
8+
9+
## What it does
10+
11+
Magic Context is a context engine that keeps long Pi sessions productive by:
12+
13+
| Feature | What it does |
14+
|---|---|
15+
| **Tagging + drops** | Tags every assistant/user/tool message with `§N§ ` so you can drop specific turns later via `ctx_reduce` |
16+
| **Historian** | Background subagent compresses old conversation into compartments + facts at threshold pressure or commit boundaries |
17+
| **`<session-history>` injection** | Prepends compressed history into the system prompt every turn so the agent never loses context |
18+
| **Project memories** | Persistent cross-session knowledge store with embedding-based semantic search |
19+
| **Dreamer** | Scheduled background subagent that consolidates, verifies, archives, and improves stored memories |
20+
| **`/ctx-aug`** | On-demand sidekick that augments the next turn with relevant memories |
21+
| **Auto-search hint** | When user prompts mention previously-discussed topics, appends a compact memory hint |
22+
| **Note nudges** | Surface deferred intentions at natural work boundaries (commit, todo completion, historian publication) |
23+
| **Cross-harness sharing** | Memories written from OpenCode appear in Pi (and vice versa) for the same project |
24+
25+
---
26+
27+
## Installation
28+
29+
```bash
30+
# Install via npm
31+
npm install -g @cortexkit/pi-magic-context
32+
33+
# Or via Bun
34+
bun add -g @cortexkit/pi-magic-context
35+
```
36+
37+
Add the extension to your Pi config (`~/.pi/agent/settings.json`):
38+
39+
```jsonc
40+
{
41+
"extensions": [
42+
"@cortexkit/pi-magic-context"
43+
]
44+
}
45+
```
46+
47+
Then run the interactive setup wizard:
48+
49+
```bash
50+
magic-context-pi setup
51+
```
52+
53+
The wizard will:
54+
1. Create `~/.pi/agent/magic-context.jsonc` with sensible defaults
55+
2. Prompt you for historian, sidekick, and embedding model preferences
56+
3. Verify your model picks resolve correctly via Pi's CLI
57+
58+
To check installation health later:
59+
60+
```bash
61+
magic-context-pi doctor
62+
```
63+
64+
---
65+
66+
## Configuration
67+
68+
Magic Context reads two config files (in this priority order):
69+
70+
1. `$cwd/.pi/magic-context.jsonc` (project-level overrides)
71+
2. `~/.pi/agent/magic-context.jsonc` (user-level defaults)
72+
73+
Both are merged through a Zod schema. Invalid fields fall back to defaults — bad config never disables the plugin entirely.
74+
75+
### Minimal config
76+
77+
```jsonc
78+
{
79+
"$schema": "https://raw.githubusercontent.com/cortexkit/opencode-magic-context/master/assets/magic-context.schema.json",
80+
"enabled": true,
81+
"historian": {
82+
"model": "anthropic/claude-haiku-4-5"
83+
},
84+
"embedding": {
85+
"provider": "local"
86+
}
87+
}
88+
```
89+
90+
For the full configuration reference (including dreamer, sidekick, auto-search, and experimental features), see [CONFIGURATION.md](https://github.com/cortexkit/opencode-magic-context/blob/master/CONFIGURATION.md) in the main repository — the schema is shared between both plugins.
91+
92+
---
93+
94+
## Slash commands
95+
96+
All commands trigger `triggerTurn: false` (never sent to the LLM):
97+
98+
| Command | What it does |
99+
|---|---|
100+
| `/ctx-status` | Live token breakdown + queued ops + cache state |
101+
| `/ctx-flush` | Force-process pending ops queue |
102+
| `/ctx-recomp` | Rebuild compartments from raw history (heavy operation) |
103+
| `/ctx-dream` | Trigger a dream run on demand |
104+
| `/ctx-aug` | Augment your next prompt with sidekick-retrieved memories |
105+
106+
---
107+
108+
## Storage
109+
110+
Magic Context stores everything in a single shared SQLite database at:
111+
112+
```
113+
~/.local/share/cortexkit/magic-context/context.db
114+
```
115+
116+
This is the **same database** the OpenCode plugin uses. Tables are scoped by:
117+
- `harness` column (`'pi'` or `'opencode'`) for session-scoped data (tags, compartments, facts, notes)
118+
- `project_path` (resolved git root) for project-scoped data (memories, embeddings, dreamer runs)
119+
120+
So memories and dreamer state are shared across both harnesses for the same project; per-session tagging stays correctly attributed.
121+
122+
Storage failures are fatal — Magic Context will refuse to register hooks rather than run with ephemeral state, since that would let context grow unbounded across restarts.
123+
124+
---
125+
126+
## Cross-harness coherence
127+
128+
For semantic search to work across harnesses, both plugins must use the **same embedding model**. Magic Context detects mismatch on Pi startup and warns:
129+
130+
```
131+
WARN embedding model mismatch detected for project ...:
132+
stored vectors use "openai-compatible:Qwen/Qwen3-Embedding-8B" but Pi is configured with "local:Xenova/all-MiniLM-L6-v2".
133+
Cross-harness search will return zero results until vectors are re-embedded.
134+
```
135+
136+
Easiest fix: configure `embedding` once in `~/.pi/agent/magic-context.jsonc` (Pi) and `~/.config/opencode/magic-context.jsonc` (OpenCode) with identical settings.
137+
138+
---
139+
140+
## Tools available to the agent
141+
142+
| Tool | Action set | Purpose |
143+
|---|---|---|
144+
| `ctx_search` | n/a | Search memories + raw session history; returns ranked results with previews |
145+
| `ctx_memory` | `write`, `delete` | Manage project memories explicitly (most writes happen via dreamer instead) |
146+
| `ctx_note` | `read`, `write`, `update`, `dismiss` | Defer intentions for later — surfaced via note nudges at work boundaries |
147+
148+
`ctx_expand` and `ctx_reduce` from the OpenCode plugin are **intentionally not exposed on Pi** — they depend on raw OpenCode message ordinals, while Pi has its own message identity model. Drops still happen automatically via threshold-driven historian; you don't need an explicit `ctx_reduce` to trigger reduction.
149+
150+
---
151+
152+
## Known limitation: `pi --print` mode
153+
154+
Pi's `--print` mode is single-turn: the process exits immediately after `agent_end`. Magic Context fires historian as a background subagent (so the LLM call never blocks on summarization), but Pi's `agent_end` event uses synchronous listener fanout — the parent process exits while our async `await` is still pending, killing the spawned historian subprocess mid-run.
155+
156+
**Production users running interactive `pi` are unaffected.** The historian started on turn N completes during turn N+1's user-think time, the same pattern OpenCode uses.
157+
158+
For `--print` mode, the upstream fix is in progress: pi-coding-agent commit `9022a5b5 fix(agent): await subscribed event handlers` (shipped in `^0.70.5`). Once you're on that version, `--print` mode also works end-to-end.
159+
160+
---
161+
162+
## Architecture & implementation
163+
164+
This package is part of the [opencode-magic-context monorepo](https://github.com/cortexkit/opencode-magic-context). The Pi extension shares the core implementation (~7,500 lines) with the OpenCode plugin via the `@magic-context/core` workspace dependency, exposing only the Pi-specific adapter layer:
165+
166+
| Pi-specific module | Responsibility |
167+
|---|---|
168+
| `context-handler.ts` | Pi `pi.on("context", ...)` adapter — tags, drops, runs nudges and auto-search |
169+
| `subagent-runner.ts` | Spawns `pi --print --mode json --no-extensions ...` for historian/sidekick/dreamer |
170+
| `tools/` | Pi `pi.registerTool` wrappers around the shared tool implementations |
171+
| `commands/` | Pi `pi.registerCommand` wrappers for the five `/ctx-*` slash commands |
172+
| `dreamer/` | Pi-side adapter for the shared dreamer scheduler |
173+
| `system-prompt.ts` | Pi `before_agent_start` injector for `<session-history>`, `<project-memory>`, `<project-docs>` |
174+
| `config/` | Pi-convention config loader (`$cwd/.pi/magic-context.jsonc` + `~/.pi/agent/magic-context.jsonc`) |
175+
| `cli/` | `magic-context-pi setup` and `magic-context-pi doctor` |
176+
177+
For deeper architectural detail, see the main repo's [ARCHITECTURE.md](https://github.com/cortexkit/opencode-magic-context/blob/master/ARCHITECTURE.md).
178+
179+
---
180+
181+
## License
182+
183+
MIT — see [LICENSE](https://github.com/cortexkit/opencode-magic-context/blob/master/LICENSE).

‎packages/pi-plugin/package.json‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,6 @@
44
"type": "module",
55
"description": "Pi coding agent extension for Magic Context — cross-session memory and context management",
66
"main": "dist/index.js",
7-
"types": "dist/index.d.ts",
87
"bin": {
98
"magic-context-pi": "dist/cli.js"
109
},
@@ -60,7 +59,6 @@
6059
},
6160
"exports": {
6261
".": {
63-
"types": "./dist/index.d.ts",
6462
"import": "./dist/index.js"
6563
}
6664
},

‎packages/pi-plugin/src/context-handler.test.ts‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -74,6 +74,10 @@ describe("registerPiContextHandler", () => {
7474
registerPiContextHandler(fake.pi as never, {
7575
db,
7676
ctxReduceEnabled: true,
77+
// Disable protection so the immediate drop on tag #2 actually
78+
// materializes; otherwise the schema default (20) defers the
79+
// drop because tag #2 is in the protected window.
80+
protectedTags: 0,
7781
});
7882
const handler = fake.handlers.get("context") as (
7983
event: { messages: never[] },

‎packages/pi-plugin/src/context-handler.ts‎

Lines changed: 23 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -172,6 +172,20 @@ export interface PiContextHandlerOptions {
172172
* is registered. Step 5b's config loader will make it configurable.
173173
*/
174174
ctxReduceEnabled: boolean;
175+
/**
176+
* Number of most-recent tags treated as protected (mirrors OpenCode
177+
* `protected_tags`). Drops with tag IDs in the protected window are
178+
* deferred — `applyPendingOperations` requeues them as deferred so
179+
* they re-evaluate next pass instead of being lost. Critical for
180+
* keeping the agent's recent working context intact.
181+
*
182+
* Defaults from the schema to 20; can be 1-100. Optional so existing
183+
* test fixtures don't need updating; callers in production (`index.ts`)
184+
* always thread the loaded config value. A previous bug used a
185+
* hardcoded `0` here — the council audit caught that recent turns
186+
* were getting dropped mid-task.
187+
*/
188+
protectedTags?: number;
175189
/**
176190
* Optional historian wiring (Step 4b.3b). When omitted, the trigger
177191
* check is skipped — context events still tag + drop normally, and
@@ -285,6 +299,9 @@ export function registerPiContextHandler(
285299
projectIdentity,
286300
messages: event.messages,
287301
ctxReduceEnabled: options.ctxReduceEnabled,
302+
// Default to 20 (matches schema default) when caller doesn't
303+
// thread an explicit value — tests rely on this fallback.
304+
protectedTags: options.protectedTags ?? 20,
288305
});
289306

290307
// After tagging+drops have committed, check whether historian
@@ -545,6 +562,7 @@ interface RunPipelineArgs {
545562
projectIdentity: string;
546563
messages: Parameters<typeof createPiTranscript>[0];
547564
ctxReduceEnabled: boolean;
565+
protectedTags: number;
548566
}
549567

550568
async function runPipeline(args: RunPipelineArgs) {
@@ -568,12 +586,11 @@ async function runPipeline(args: RunPipelineArgs) {
568586
// numbers; the actual mutation goes through the TagTarget surface
569587
// we built above and lands in the underlying AgentMessage content.
570588
//
571-
// `protectedTags` is hardcoded to 0 for Pi here because we don't
572-
// have a magic-context.jsonc config loader yet; once 5b lands the
573-
// configured value flows in. 0 is the existing OpenCode default
574-
// for unprotected behavior — at this stage Pi sessions don't have
575-
// the recent-turn protection that primary OpenCode sessions get.
576-
applyPendingOperations(args.sessionId, args.db, targets, 0);
589+
// `protectedTags` flows from the loaded `magic-context.jsonc` config
590+
// (Step 5b) — drops in the protected window are deferred via
591+
// `applyPendingOperations` so the agent's recent working context
592+
// stays intact. Default is 20 (matches OpenCode); range is 1-100.
593+
applyPendingOperations(args.sessionId, args.db, targets, args.protectedTags);
577594

578595
// Apply persistent dropped/truncated tag statuses so cross-pass
579596
// drops survive even if pending_ops was already drained on a

‎packages/pi-plugin/src/dreamer/index.ts‎

Lines changed: 21 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,7 @@
1-
import type { DreamerConfig } from "@magic-context/core/config/schema/magic-context";
1+
import type {
2+
DreamerConfig,
3+
EmbeddingConfig,
4+
} from "@magic-context/core/config/schema/magic-context";
25
import { registerDreamProjectDirectory } from "@magic-context/core/features/magic-context/dreamer";
36
import type { ContextDatabase } from "@magic-context/core/features/magic-context/storage";
47
import { startDreamScheduleTimer } from "@magic-context/core/plugin/dream-timer";
@@ -10,6 +13,21 @@ export interface PiDreamerOptions {
1013
projectIdentity: string;
1114
/** Resolved DreamerConfig from loadPiConfig(). When .enabled is false, the function is a no-op. */
1215
config: DreamerConfig;
16+
/**
17+
* Council finding #7: dreamer needs the real embedding config so it can
18+
* (a) consolidate near-duplicate memories using cosine similarity and
19+
* (b) re-embed memory content when it gets rewritten by `improve`.
20+
* Hardcoded `{provider:"off"}` previously meant dreamer skipped both
21+
* paths even when the user had a real embedding model configured.
22+
*/
23+
embeddingConfig: EmbeddingConfig;
24+
/**
25+
* Council finding #7: dreamer needs the real memory.enabled gate so the
26+
* memory-promotion pipeline (consolidation + improve + archive) can
27+
* actually write to the project memory store. Hardcoded `false`
28+
* previously made dreamer's memory tasks a no-op.
29+
*/
30+
memoryEnabled: boolean;
1331
}
1432

1533
type DreamTimerRegistration = Parameters<typeof startDreamScheduleTimer>[0];
@@ -69,8 +87,8 @@ export function registerPiDreamerProject(opts: PiDreamerOptions): void {
6987
directory: opts.projectDir,
7088
client: createPiDreamerClient(opts),
7189
dreamerConfig: opts.config,
72-
embeddingConfig: { provider: "off" },
73-
memoryEnabled: false,
90+
embeddingConfig: opts.embeddingConfig,
91+
memoryEnabled: opts.memoryEnabled,
7492
experimentalUserMemories: opts.config.user_memories.enabled
7593
? {
7694
enabled: true,

‎packages/pi-plugin/src/index.ts‎

Lines changed: 58 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -305,6 +305,47 @@ export default async function (pi: ExtensionAPI): Promise<void> {
305305
: ""),
306306
);
307307

308+
// Council finding #5: warn loudly if Pi's configured embedding model
309+
// disagrees with the model that the project's stored embedding vectors
310+
// were produced under. Cross-harness search relies on cosine similarity
311+
// between vectors from the SAME model — a mismatch silently returns
312+
// zero hits because the embedding spaces are unrelated.
313+
//
314+
// We only warn (don't crash) because:
315+
// - The user may be intentionally rotating embedding models.
316+
// - Existing vectors will be re-embedded as part of the periodic
317+
// dreamer sweep + on-demand re-embedding when memories update.
318+
// - Hard-failing on mismatch would prevent the upgrade path from
319+
// ever completing.
320+
//
321+
// The warning surfaces in `magic-context.log` so users debugging
322+
// "why is search returning nothing?" see a clear pointer.
323+
if (config.embedding.provider !== "off") {
324+
try {
325+
const { getStoredModelId } = await import(
326+
"@magic-context/core/features/magic-context/memory/storage-memory-embeddings"
327+
);
328+
const { getEmbeddingModelId } = await import(
329+
"@magic-context/core/features/magic-context/memory/embedding"
330+
);
331+
const stored = getStoredModelId(db, projectIdentity);
332+
const current = getEmbeddingModelId();
333+
if (stored && current && stored !== current) {
334+
warn(
335+
`embedding model mismatch detected for project ${projectIdentity}: ` +
336+
`stored vectors use "${stored}" but Pi is configured with "${current}". ` +
337+
"Cross-harness search will return zero results until vectors are re-embedded. " +
338+
"Either restore the previous embedding model in magic-context.jsonc, or wait " +
339+
"for the dreamer's periodic embedding sweep to backfill new vectors.",
340+
);
341+
}
342+
} catch (err) {
343+
// Embedding-model lookup is best-effort — if it throws (e.g. DB
344+
// schema race during first boot), don't block plugin load.
345+
warn("embedding model coherence check failed (non-fatal):", err);
346+
}
347+
}
348+
308349
// Register the agent-facing tools. Reuses the same business logic
309350
// the OpenCode plugin uses (insertMemory, unifiedSearch, addNote, …)
310351
// via the shared cortexkit DB. Cross-harness memory sharing is automatic
@@ -328,6 +369,11 @@ export default async function (pi: ExtensionAPI): Promise<void> {
328369
registerPiContextHandler(pi, {
329370
db,
330371
ctxReduceEnabled: config.ctx_reduce_enabled,
372+
// `protected_tags` flows here from the loaded magic-context.jsonc
373+
// config (Step 5b). Defaults to schema value (20) when unset.
374+
// Council finding #1 (unanimous CRITICAL): a hardcoded `0` here
375+
// silently let recent-turn drops mid-task; use real config value.
376+
protectedTags: config.protected_tags ?? 20,
331377
historian: historianConfig,
332378
nudge: nudgeConfig,
333379
autoSearch: autoSearchConfig,
@@ -414,6 +460,12 @@ export default async function (pi: ExtensionAPI): Promise<void> {
414460
projectDir,
415461
projectIdentity,
416462
config: dreamerConfig,
463+
// Council finding #7: thread real embedding + memory config so
464+
// dreamer can do semantic dedup AND can write memory updates.
465+
// Previously hardcoded to off/false, making most dreamer tasks
466+
// useless on Pi.
467+
embeddingConfig: config.embedding,
468+
memoryEnabled: config.memory.enabled,
417469
});
418470
info(
419471
dreamerConfig.enabled
@@ -453,8 +505,12 @@ export default async function (pi: ExtensionAPI): Promise<void> {
453505
db,
454506
cwd: ctx.cwd,
455507
sessionId,
456-
memoryEnabled: true,
457-
injectDocs: true,
508+
// Council finding #3: respect memory.enabled and dreamer.inject_docs
509+
// from config. Hardcoded `true` previously meant the user's explicit
510+
// disable was ignored — memories injected into every prompt even
511+
// when memory.enabled=false. Same for project docs.
512+
memoryEnabled: config.memory.enabled,
513+
injectDocs: config.dreamer?.inject_docs ?? true,
458514
});
459515
if (!block) return;
460516
return { systemPrompt: `${event.systemPrompt}\n\n${block}` };

0 commit comments

Comments
 (0)