Skip to content

Commit 0e99ca0

Browse files
committed
fix(pi): round-2 audit — compaction boundary, reasoning replay channel, recomp/upgrade parity
Second blind audit round (1 council + 4 Oracles) on the round-1 Pi fixes. Caught a regression in my own round-1 change plus several deeper bugs; verified every finding against source and dismissed the false positives. REGRESSION fix (round-1 introduced): - findFirstKeptEntryId advanced past a folded-toolResult synthetic boundary, dropping that un-summarized tool-result run (neither summarized nor kept). Now DEFERS (returns null) when the kept-start ordinal is synthetic — nothing trims until a real entry heads the kept tail (Pi never cuts at a toolResult). HIGH/BLOCKER: - Synthetic-user compaction-trim failure (council UNANIMOUS #1): a compartment ending on `synth-user-<id>` never matched a live message, so the summarized prefix never trimmed -> duplicate content + overflow in tool-heavy sessions. trimPiMessagesToBoundary now resolves a synth-user cutoff to the underlying real toolResult entry id. - Publish/drain race: onPublished fired (deferred-materialization signal) before queueDropsForCompartmentalizedMessages, with an `await ensureProjectRegistered` between them — a concurrent context pass could materialize with empty pendingOps and strand the drops. queueDrops now runs before onPublished. - Reasoning replay/clear mutated args.messages while the transcript flushes a separate `working` copy at commit() — tagging/drops reassign working[idx] to fresh objects, so commit() discarded the reasoning mutation while the cleared- reasoning watermark advanced (defer-pass replay divergence -> cache bust). Reasoning now mutates the transcript working array (single flush channel). - No-cache contention fallback derived markers.maxMemoryId from a separate read than the rendered set -> possible duplicate memory across m[0]/m[1]. Watermark now derived from the frozen fallback set (parity with materializeM0Pi). - /ctx-recomp + /ctx-session-upgrade omitted ensureProjectRegistered (rebuilt compartments kept NULL p1_embedding) and the runner fallback chain (fallbackModels + session-model last resort). Both threaded through. - createPiHistorianClient ignored body.model, so every recomp fallback attempt re-ran the primary; now honors the per-attempt override. - /ctx-session-upgrade never advanced the Pi native compaction marker after a successful recomp -> branch grew unbounded post-upgrade. Extracted the marker advance into shared queueAndApplyPiRecompMarker, now called by both commands. - --no-session children (sidekick/dreamer) exposed session-scoped ctx_note / ctx_expand resolving the ephemeral child session (orphaned notes / empty expand). Dropped both for those children; kept project-scoped ctx_search / ctx_memory. Dismissed (verified false positives): council SOLO-HIGH "heuristics self-enable on defer passes" — Pi's shouldRunHeuristics gating is signal-equivalent to OpenCode's materializationRequested model; the pending/deferred-materialization signal IS a legitimate cache-bust trigger. Tests: plugin 1663, Pi 348 (+8 regression), 0 fail. tsc + biome clean.
1 parent 49a8b86 commit 0e99ca0

12 files changed

Lines changed: 389 additions & 160 deletions

‎packages/pi-plugin/src/commands/ctx-recomp.ts‎

Lines changed: 13 additions & 104 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,6 @@
11
import type { ExtensionAPI } from "@earendil-works/pi-coding-agent";
22
import { getCompartments } from "@magic-context/core/features/magic-context/compartment-storage";
3-
import {
4-
type ContextDatabase,
5-
clearPendingPiCompactionMarkerStateIf,
6-
setPendingPiCompactionMarkerState,
7-
} from "@magic-context/core/features/magic-context/storage";
3+
import type { ContextDatabase } from "@magic-context/core/features/magic-context/storage";
84
import { COMPARTMENT_AGENT_SYSTEM_PROMPT } from "@magic-context/core/hooks/magic-context/compartment-prompt";
95
import { executeContextRecompWithResult } from "@magic-context/core/hooks/magic-context/compartment-runner";
106
import {
@@ -15,17 +11,13 @@ import type { RawMessageProvider } from "@magic-context/core/hooks/magic-context
1511
import { setRawMessageProvider } from "@magic-context/core/hooks/magic-context/read-session-chunk";
1612
import { describeError } from "@magic-context/core/shared/error-message";
1713
import type { SubagentRunner } from "@magic-context/core/shared/subagent-runner";
18-
import { applyDeferredPiCompactionMarker } from "../compaction-marker-manager-pi";
1914
import {
20-
signalPiDeferredHistoryRefresh,
2115
signalPiHistoryRefresh,
2216
signalPiPendingMaterialization,
2317
} from "../context-handler";
24-
import {
25-
buildPiCompactionSummary,
26-
findFirstKeptEntryId,
27-
} from "../pi-historian-runner";
18+
import { ensureProjectRegisteredFromPiDirectory } from "../embedding-bootstrap";
2819
import { createPiHistorianClient } from "../pi-recomp-client-shared";
20+
import { queueAndApplyPiRecompMarker } from "../pi-recomp-marker";
2921
import { readPiSessionMessages } from "../read-session-pi";
3022
import { setMagicContextRecompActive, updateStatusLine } from "../status-line";
3123
import { resolveSessionId, sendCtxStatusMessage } from "./pi-command-utils";
@@ -154,6 +146,16 @@ export function registerCtxRecompCommand(
154146
historianTimeoutMs: deps.historianTimeoutMs,
155147
memoryEnabled: deps.memoryEnabled,
156148
autoPromote: deps.autoPromote,
149+
// Embedding substrate: register before the recomp publish path
150+
// calls embedAndStoreCompartments, else rebuilt rows keep NULL
151+
// p1_embedding (dropped from ctx_search / dreamer linkage).
152+
ensureProjectRegistered: ensureProjectRegisteredFromPiDirectory,
153+
// Recomp-runner model chain parity with OpenCode: configured
154+
// fallbacks + the session's own model as last-ditch retry.
155+
fallbackModels: deps.historianFallbacks,
156+
fallbackModelId: ctx.model
157+
? `${ctx.model.provider}/${ctx.model.id}`
158+
: undefined,
157159
},
158160
parsed.kind === "partial" ? { range: parsed.range } : {},
159161
);
@@ -196,99 +198,6 @@ export function registerCtxRecompCommand(
196198
});
197199
}
198200

199-
function queueAndApplyPiRecompMarker(args: {
200-
db: ContextDatabase;
201-
sessionId: string;
202-
ctx: unknown;
203-
}): void {
204-
const appendCompaction = resolvePiAppendCompaction(args.ctx);
205-
const readBranchEntries = resolvePiReadBranchEntries(args.ctx);
206-
if (!appendCompaction || !readBranchEntries) return;
207-
208-
const compartments = getCompartments(args.db, args.sessionId);
209-
const last = compartments[compartments.length - 1];
210-
if (!last) return;
211-
212-
let firstKeptEntryId: string | null = null;
213-
try {
214-
firstKeptEntryId = findFirstKeptEntryId(
215-
readBranchEntries(),
216-
last.endMessage,
217-
);
218-
} catch {
219-
firstKeptEntryId = null;
220-
}
221-
if (!firstKeptEntryId || last.endMessageId.length === 0) return;
222-
223-
const pending = {
224-
firstKeptEntryId,
225-
endMessageId: last.endMessageId,
226-
ordinal: last.endMessage,
227-
tokensBefore: 0,
228-
summary: buildPiCompactionSummary(compartments),
229-
publishedAt: Date.now(),
230-
};
231-
232-
setPendingPiCompactionMarkerState(args.db, args.sessionId, pending);
233-
const outcome = applyDeferredPiCompactionMarker(
234-
{ db: args.db, appendCompaction, readBranchEntries },
235-
args.sessionId,
236-
pending,
237-
);
238-
if (outcome.kind === "retryable-failure") {
239-
signalPiDeferredHistoryRefresh(args.sessionId);
240-
return;
241-
}
242-
if (
243-
!clearPendingPiCompactionMarkerStateIf(args.db, args.sessionId, pending)
244-
) {
245-
signalPiDeferredHistoryRefresh(args.sessionId);
246-
}
247-
}
248-
249-
function resolvePiAppendCompaction(
250-
ctx: unknown,
251-
):
252-
| ((
253-
summary: string,
254-
firstKeptEntryId: string,
255-
tokensBefore: number,
256-
details?: unknown,
257-
fromHook?: boolean,
258-
) => string | undefined)
259-
| undefined {
260-
const sm = (ctx as { sessionManager?: unknown })?.sessionManager as
261-
| {
262-
appendCompaction?: (
263-
summary: string,
264-
firstKeptEntryId: string,
265-
tokensBefore: number,
266-
details?: unknown,
267-
fromHook?: boolean,
268-
) => string | undefined;
269-
}
270-
| undefined;
271-
if (typeof sm?.appendCompaction !== "function") return undefined;
272-
return sm.appendCompaction.bind(sm);
273-
}
274-
275-
function resolvePiReadBranchEntries(
276-
ctx: unknown,
277-
): (() => unknown[]) | undefined {
278-
const sm = (ctx as { sessionManager?: unknown })?.sessionManager as
279-
| { getBranch?: () => unknown[] }
280-
| undefined;
281-
if (typeof sm?.getBranch !== "function") return undefined;
282-
return () => {
283-
try {
284-
const entries = sm.getBranch?.call(sm);
285-
return Array.isArray(entries) ? entries : [];
286-
} catch {
287-
return [];
288-
}
289-
};
290-
}
291-
292201
function parseRecompArgs(
293202
raw: string,
294203
):

‎packages/pi-plugin/src/commands/ctx-session-upgrade.ts‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,8 +18,10 @@ import {
1818
signalPiHistoryRefresh,
1919
signalPiPendingMaterialization,
2020
} from "../context-handler";
21+
import { ensureProjectRegisteredFromPiDirectory } from "../embedding-bootstrap";
2122
import { runPiMemoryMigration } from "../pi-memory-migration";
2223
import { createPiHistorianClient } from "../pi-recomp-client-shared";
24+
import { queueAndApplyPiRecompMarker } from "../pi-recomp-marker";
2325
import { readPiSessionMessages } from "../read-session-pi";
2426
import { setMagicContextRecompActive, updateStatusLine } from "../status-line";
2527
import { resolveSessionId, sendCtxStatusMessage } from "./pi-command-utils";
@@ -204,6 +206,17 @@ export function registerCtxSessionUpgradeCommand(
204206
historianTimeoutMs: deps.historianTimeoutMs,
205207
memoryEnabled: deps.memoryEnabled,
206208
autoPromote: deps.autoPromote,
209+
// Embedding substrate: without this the recomp publish path
210+
// no-ops embedAndStoreCompartments on an unregistered project,
211+
// leaving rebuilt compartments with NULL p1_embedding (dropped
212+
// from ctx_search / dreamer linkage). Parity with OpenCode.
213+
ensureProjectRegistered: ensureProjectRegisteredFromPiDirectory,
214+
// Recomp-runner model chain (parity with OpenCode
215+
// recomp-orchestrator): configured fallbacks + the session's
216+
// own model as the last-ditch retry, so an empty/invalid-but-
217+
// HTTP-200 historian primary escalates instead of failing.
218+
fallbackModels: deps.historianFallbacks,
219+
fallbackModelId: sessionMainModel,
207220
},
208221
{},
209222
);
@@ -230,6 +243,13 @@ export function registerCtxSessionUpgradeCommand(
230243
return;
231244
}
232245

246+
// Advance the Pi native compaction marker to the rebuilt boundary
247+
// so getBranch() trims the pre-upgrade branch. Without this the
248+
// upgrade republishes compartments but the entire pre-upgrade JSONL
249+
// branch stays visible and grows unbounded until a later incremental
250+
// historian pass advances the marker. Shared with /ctx-recomp.
251+
queueAndApplyPiRecompMarker({ db: deps.db, sessionId, ctx });
252+
233253
signalPiHistoryRefresh(sessionId);
234254
signalPiPendingMaterialization(sessionId);
235255

‎packages/pi-plugin/src/context-handler.ts‎

Lines changed: 12 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -2598,6 +2598,14 @@ async function runPipeline(args: RunPipelineArgs): Promise<RunPipelineResult> {
25982598
}
25992599

26002600
const transcript = createPiTranscript(args.messages, args.sessionId);
2601+
// Reasoning clearing/replay mutate `part.thinking` in place. They MUST target
2602+
// the transcript's `working` array (the channel commit() flushes), not the
2603+
// original `args.messages`: tagging/drops/caveman reassign working[idx] to
2604+
// fresh objects, so a reasoning mutation written to args.messages[idx] (a
2605+
// now-divergent object) would be discarded when commit() does
2606+
// source[idx] = working[idx], leaving the cleared-reasoning watermark ahead
2607+
// of the actual wire bytes → defer-pass replay divergence → cache bust.
2608+
const workingMessages = transcript.getWorkingMessages();
26012609
const currentTurnId = (() => {
26022610
const ids = buildPiMessageIdByIndex(
26032611
args.messages as PiAgentMessage[],
@@ -2801,14 +2809,14 @@ async function runPipeline(args: RunPipelineArgs): Promise<RunPipelineResult> {
28012809
const clearedReplay = replayClearedReasoningPi({
28022810
db: args.db,
28032811
sessionId: args.sessionId,
2804-
messages: args.messages,
2812+
messages: workingMessages,
28052813
messageIdToMaxTag,
28062814
piMessageStableId,
28072815
});
28082816
const inlineReplay = replayStrippedInlineThinkingPi({
28092817
db: args.db,
28102818
sessionId: args.sessionId,
2811-
messages: args.messages,
2819+
messages: workingMessages,
28122820
messageIdToMaxTag,
28132821
piMessageStableId,
28142822
});
@@ -2979,13 +2987,13 @@ async function runPipeline(args: RunPipelineArgs): Promise<RunPipelineResult> {
29792987
const meta = getOrCreateSessionMeta(args.db, args.sessionId);
29802988
const prevWatermark = meta.clearedReasoningThroughTag ?? 0;
29812989
const clearOutcome = clearOldReasoningPi({
2982-
messages: args.messages,
2990+
messages: workingMessages,
29832991
messageIdToMaxTag,
29842992
clearReasoningAge: args.reasoningClearing.clearReasoningAge,
29852993
piMessageStableId,
29862994
});
29872995
const stripOutcome = stripInlineThinkingPi({
2988-
messages: args.messages,
2996+
messages: workingMessages,
29892997
messageIdToMaxTag,
29902998
clearReasoningAge: args.reasoningClearing.clearReasoningAge,
29912999
piMessageStableId,

‎packages/pi-plugin/src/inject-compartments-pi.test.ts‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -104,6 +104,44 @@ describe("trimPiMessagesToBoundary", () => {
104104
expect(removed).toBe(3);
105105
expect(messages).toEqual([user("keep")]);
106106
});
107+
108+
it("resolves a synth-user-* cutoff to the underlying real toolResult entry id", () => {
109+
// A compartment ending on a folded-toolResult boundary carries
110+
// endMessageId = `synth-user-<realToolResultEntryId>`. The live array has
111+
// no message with that synthetic id — only the real toolResult (entry id
112+
// "tr-real"). Pre-fix, the cutoff never matched and NOTHING was trimmed
113+
// (history duplicated -> overflow). The fix strips the prefix and matches
114+
// the real toolResult, then the orphan sweep removes its paired assistant.
115+
const messages = [
116+
assistant(["call-a"]),
117+
result("call-a"),
118+
assistant([], "next turn"),
119+
user("keep"),
120+
];
121+
122+
const removed = __test.trimPiMessagesToBoundary(
123+
messages,
124+
["a", "tr-real", "a2", "keep"],
125+
"synth-user-tr-real",
126+
);
127+
128+
// toolResult "tr-real" (cutoff) + its paired assistant "call-a" (orphan
129+
// sweep) are removed; the later turn + keep survive.
130+
expect(removed).toBe(2);
131+
expect(messages.map((m) => m.role)).toEqual(["assistant", "user"]);
132+
expect((messages[1] as { content: string }).content).toBe("keep");
133+
});
134+
135+
it("returns 0 (no spurious trim) when a synth-user-* cutoff has no matching real entry", () => {
136+
const messages = [assistant(["call-a"]), user("keep")];
137+
const removed = __test.trimPiMessagesToBoundary(
138+
messages,
139+
["a", "keep"],
140+
"synth-user-nonexistent",
141+
);
142+
expect(removed).toBe(0);
143+
expect(messages.length).toBe(2);
144+
});
107145
});
108146

109147
function piState(sessionId: string, cwd: string) {

‎packages/pi-plugin/src/inject-compartments-pi.ts‎

Lines changed: 37 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,7 @@ import { buildKeyFilesBlock } from "@magic-context/core/hooks/magic-context/key-
6262
import { estimateTokens } from "@magic-context/core/hooks/magic-context/read-session-formatting";
6363
import type { MessageLike } from "@magic-context/core/hooks/magic-context/tag-messages";
6464
import { sessionLog as logSession } from "@magic-context/core/shared/logger";
65+
import { SYNTH_USER_ID_PREFIX } from "./read-session-pi";
6566

6667
/**
6768
* Pi message shapes — kept structurally compatible with
@@ -163,6 +164,22 @@ function buildMessageLikeProjection(
163164
* Mirrors the `messages.splice(0, cutoffIndex+1)` behavior the shared
164165
* `prepareCompartmentInjection` does on its (OpenCode) MessageLike[].
165166
*
167+
* # Synthetic-user (folded toolResult) cutoffs
168+
*
169+
* A compartment's `endMessageId` comes from `convertEntriesToRawMessages`,
170+
* which folds a run of `toolResult` entries into a synthetic-user RawMessage
171+
* with id `${SYNTH_USER_ID_PREFIX}<firstFoldedToolResultEntryId>`. The LIVE Pi
172+
* message array does NOT contain that synthetic id — folding is a historian-
173+
* chunking artifact only; the underlying toolResult messages are present as
174+
* real entries. So a raw `resolveStableId(msg) === cutoffMessageId` comparison
175+
* can never match a synth-user cutoff, `cutoffIndex` stays -1, and the
176+
* summarized prefix is never trimmed → duplicate content + overflow in
177+
* tool-heavy sessions. When the cutoff is synthetic, strip the prefix and match
178+
* against the underlying real toolResult entry id instead (the suffix is, by
179+
* construction, the real entry id of the first folded toolResult, which IS a
180+
* visible message). Trimming through that toolResult covers the whole folded
181+
* run because the bidirectional orphan sweep below removes the rest of the run.
182+
*
166183
* Returns the count of messages removed — used for log parity.
167184
*/
168185
function trimPiMessagesToBoundary(
@@ -171,10 +188,16 @@ function trimPiMessagesToBoundary(
171188
cutoffMessageId: string,
172189
): number {
173190
if (cutoffMessageId.length === 0) return 0;
191+
// Resolve a synthetic-user (folded toolResult) cutoff to the real entry id
192+
// of the underlying toolResult, which is what the live message carries.
193+
const effectiveCutoffId = cutoffMessageId.startsWith(SYNTH_USER_ID_PREFIX)
194+
? cutoffMessageId.slice(SYNTH_USER_ID_PREFIX.length)
195+
: cutoffMessageId;
196+
if (effectiveCutoffId.length === 0) return 0;
174197
let cutoffIndex = -1;
175198
for (let i = 0; i < piMessages.length; i++) {
176199
const msg = piMessages[i];
177-
if (msg && resolveStableId(msg, i, entryIds) === cutoffMessageId) {
200+
if (msg && resolveStableId(msg, i, entryIds) === effectiveCutoffId) {
178201
cutoffIndex = i;
179202
break;
180203
}
@@ -1033,6 +1056,19 @@ export function injectM0M1Pi(
10331056
["active", "permanent"],
10341057
markers.materializedAt,
10351058
);
1059+
// Align the m[1] memory watermark to the EXACT set rendered into
1060+
// this fallback m[0], by construction. readCurrentMarkers computed
1061+
// markers.maxMemoryId from its OWN separate getMemoriesByProject
1062+
// read; if a sibling inserted a memory between that read and this
1063+
// one, renderM1Pi(markers) could re-emit a memory already in m[0]
1064+
// as "new" (duplicate across m[0]/m[1]). Deriving the watermark
1065+
// from fallbackMemories — the same set renderM0Pi receives —
1066+
// removes the TOCTOU window (parity with materializeM0Pi's
1067+
// snapshot-aligned watermark).
1068+
markers.maxMemoryId = fallbackMemories.reduce(
1069+
(max, m) => (m.id > max ? m.id : max),
1070+
0,
1071+
);
10361072
let dpm = 1;
10371073
m0 = renderM0Pi(state, db, docs.renderedBlock, dpm, fallbackMemories);
10381074
let attempts = 0;

0 commit comments

Comments
 (0)