Skip to content

Commit edd6e06

Browse files
committed
fix(#336): historian children ride the age-gated orphan sweep instead of inline delete at publish — stale threshold covers timeout+retries+fallbacks+detached-writer grace; FK failures from late OpenCode part writes eliminated
Co-Authored-By: Alfonso <[email protected]>
2 parents a8d1beb + 51aabd1 commit edd6e06

13 files changed

Lines changed: 436 additions & 38 deletions

File tree

‎CONFIGURATION.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -167,7 +167,7 @@ Higher-tier models with longer cache windows benefit from a longer TTL. Setting
167167
| `execute_threshold_percentage` | `number` (20–90) or `object` | `65` | Context usage that forces queued ops to execute. Capped at 90% of the output-reserved safe window, leaving about 10% for mid-turn input growth. Supports per-model maps. |
168168
| `execute_threshold_tokens` | `object` (per-model map) | — | **Optional absolute-tokens variant of `execute_threshold_percentage`.** Per-model map (e.g. `{ "default": 150000, "github-copilot/gpt-5.2-codex": 40000 }`). When set for a model, overrides the percentage-based threshold for that model. Clamped to `90% × context_limit` with a warn log. Requires a resolvable context limit — falls through to percentage if unavailable. See below. |
169169
| `clear_reasoning_age` | `number` | `50` | Clear thinking/reasoning blocks older than N tags. |
170-
| `historian_timeout_ms` | `number` | `300000` | Timeout per historian call (ms). |
170+
| `historian_timeout_ms` | `number` | `600000` | Timeout per historian call (ms). |
171171
| `history_budget_percentage` | `number` (0.05–0.5) | `0.15` | Fraction of usable context (`context_limit × execute_threshold`) reserved for the history block. Triggers compression when exceeded. |
172172
| `compaction.enabled` | `boolean` | `true` | When `false`, use compaction-off mode: keep Magic Context's knowledge layer and let native compaction (or nothing) own the context window. Boot-resolved; restart after changing it. See below. |
173173
| `commit_cluster_trigger` | `object` | See below | Controls the commit-cluster historian trigger. |

‎assets/magic-context.schema.json‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1271,8 +1271,8 @@
12711271
"maximum": 0.5
12721272
},
12731273
"historian_timeout_ms": {
1274-
"default": 300000,
1275-
"description": "Timeout for each historian prompt call in milliseconds (default: 300000)",
1274+
"default": 600000,
1275+
"description": "Timeout for each historian prompt call in milliseconds (default: 600000)",
12761276
"type": "number",
12771277
"minimum": 60000
12781278
},

‎packages/docs/src/content/docs/reference/configuration.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -100,7 +100,7 @@ The background agent that condenses old conversation into compact history.
100100
| `historian.two_pass` | boolean | `false` | Run a second editor pass over historian output to clean low-signal U: lines and cross-compartment duplicates. Adds ~1 extra API call and ~1.3x cost per historian run. Useful for models without extended thinking support. (default: false) |
101101
| `historian.thinking_level` | `"off"` \\| `"minimal"` \\| `"low"` \\| `"medium"` \\| `"high"` \\| `"xhigh"` \\| `"max"` | — | Pi only: explicit thinking level passed as --thinking <level> to Pi historian subagent invocations. Required when using reasoning models (e.g. github-copilot/gpt-5.4) because Pi's default thinking-level resolution can pick a value the provider rejects. OpenCode users set variant instead. Valid: off \| minimal \| low \| medium \| high \| xhigh \| max |
102102
| `historian.disallowed_tools` | `"*"` \\| `"read"` \\| `"aft_outline"` \\| `"aft_zoom"` \\| `"aft_search"`[] | `[]` | OpenCode only. Tools to REMOVE from the historian's default allow-list [read, aft_outline, aft_zoom, aft_search]. Applies to both historian and historian-editor agents. Use ["*"] to strip all tool definitions from the model request — this prevents weak instruction-following models (e.g. mistral-small-latest) from entering tool-calling loops. Individual tool names remove just that tool. Note: a user-supplied historian.permission override can re-allow a tool that disallowed_tools removed — disallowed_tools sets the baseline, permission overrides take precedence. (default: []) |
103-
| `historian_timeout_ms` | number (60000–) | `300000` | Timeout for each historian prompt call in milliseconds (default: 300000) |
103+
| `historian_timeout_ms` | number (60000–) | `600000` | Timeout for each historian prompt call in milliseconds (default: 600000) |
104104
| `commit_cluster_trigger` | object | — | Commit-cluster trigger: fire historian when enough commit clusters accumulate in the unsummarized tail |
105105
| `commit_cluster_trigger.enabled` | boolean | `true` | Enable commit-cluster based historian triggering (default: true) |
106106
| `commit_cluster_trigger.min_clusters` | number (1–) | `3` | Minimum commit clusters required to trigger historian (min: 1, default: 3) |

‎packages/pi-plugin/src/pi-historian-runner.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -126,7 +126,7 @@ import {
126126
} from "./read-session-pi";
127127

128128
const HISTORIAN_AGENT_NAME = "magic-context-historian";
129-
const DEFAULT_HISTORIAN_TIMEOUT_MS = 120_000;
129+
const DEFAULT_HISTORIAN_TIMEOUT_MS = 600_000;
130130
const MAX_HISTORIAN_RETRIES = 2;
131131

132132
/** Keep historian alert noise to once per minute per session. */

‎packages/plugin/src/config/schema/magic-context.test.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,7 @@ describe("MagicContextConfigSchema", () => {
4141
overlay: true,
4242
},
4343
});
44+
expect(result.historian_timeout_ms).toBeGreaterThanOrEqual(10 * 60_000);
4445
expect(result.historian).toBeUndefined();
4546
expect(result.dreamer).toBeUndefined();
4647
expect(result.sidekick).toBeUndefined();

‎packages/plugin/src/config/schema/magic-context.ts‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ export const DEFAULT_EXECUTE_THRESHOLD_PERCENTAGE = 65;
1616
// escalation derives above the effective threshold and the 95% wall stays fixed.
1717
export const EXECUTE_THRESHOLD_CAP_MESSAGE =
1818
"execute_threshold is capped at 90% for cache safety: output capacity is reserved from the usable context window, and the remaining 10% absorbs mid-turn growth before the absolute 95% emergency wall. Use a value between 20 and 90.";
19-
export const DEFAULT_HISTORIAN_TIMEOUT_MS = 300_000;
19+
export const DEFAULT_HISTORIAN_TIMEOUT_MS = 600_000;
2020
export const DEFAULT_HISTORY_BUDGET_PERCENTAGE = 0.15;
2121

2222
export const DEFAULT_LOCAL_EMBEDDING_MODEL = "Xenova/all-MiniLM-L6-v2";
@@ -784,7 +784,7 @@ export const MagicContextConfigSchema = z
784784
.number()
785785
.min(60_000)
786786
.default(DEFAULT_HISTORIAN_TIMEOUT_MS)
787-
.describe("Timeout for each historian prompt call in milliseconds (default: 300000)"),
787+
.describe("Timeout for each historian prompt call in milliseconds (default: 600000)"),
788788
commit_cluster_trigger: z
789789
.object({
790790
enabled: z

‎packages/plugin/src/features/magic-context/dreamer/retrospective-orphan-sweep.test.ts‎

Lines changed: 69 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,8 @@ import { Database } from "../../../shared/sqlite";
66
import { closeQuietly } from "../../../shared/sqlite-helpers";
77
import {
88
CURATE_CHILD_TITLE,
9+
HISTORIAN_CHILD_TITLE,
10+
historianOrphanStaleMs,
911
MAINTAIN_DOCS_CHILD_TITLE,
1012
REFRESH_PRIMERS_CHILD_TITLE,
1113
RETROSPECTIVE_CHILD_TITLE,
@@ -50,6 +52,18 @@ describe("retrospectiveOrphanStaleMs", () => {
5052
});
5153
});
5254

55+
describe("historianOrphanStaleMs", () => {
56+
test("covers every outer, model-suggestion, and fallback attempt plus grace", () => {
57+
const timeoutMs = 10 * 60_000;
58+
const fallbackModelCount = 2;
59+
const fullAttemptBudget = timeoutMs * 3 * 2 * (fallbackModelCount + 1);
60+
61+
expect(historianOrphanStaleMs(timeoutMs, fallbackModelCount)).toBeGreaterThan(
62+
fullAttemptBudget,
63+
);
64+
});
65+
});
66+
5367
describe("sweepOrphanedRetrospectiveChildren", () => {
5468
const DIR = "/repo/project";
5569
const now = 10_000_000;
@@ -71,6 +85,7 @@ describe("sweepOrphanedRetrospectiveChildren", () => {
7185
test("deletes old privacy-sensitive children in this directory", async () => {
7286
db = makeOpencodeDb();
7387
// old orphans in this dir → swept
88+
insert(db, "old-historian", HISTORIAN_CHILD_TITLE, DIR, now - staleMs - 8);
7489
insert(db, "old-user-memories", USER_MEMORIES_CHILD_TITLE, DIR, now - staleMs - 7);
7590
insert(db, "old", RETROSPECTIVE_CHILD_TITLE, DIR, now - staleMs - 6);
7691
insert(db, "old-curate", CURATE_CHILD_TITLE, DIR, now - staleMs - 5);
@@ -90,8 +105,9 @@ describe("sweepOrphanedRetrospectiveChildren", () => {
90105
DIR,
91106
now - staleMs - 1,
92107
);
93-
// recent child (live run) → NOT swept
108+
// recent children (live or still draining detached writes) → NOT swept
94109
insert(db, "fresh", RETROSPECTIVE_CHILD_TITLE, DIR, now - 1000);
110+
insert(db, "fresh-historian", HISTORIAN_CHILD_TITLE, DIR, now - 1000);
95111
// old but a different title → NOT swept
96112
insert(db, "other-title", "magic-context-dream-verify", DIR, now - staleMs - 1);
97113
// old retrospective but ANOTHER directory → NOT swept
@@ -107,6 +123,7 @@ describe("sweepOrphanedRetrospectiveChildren", () => {
107123
});
108124

109125
expect(deleted).toEqual([
126+
"old-historian",
110127
"old-user-memories",
111128
"old",
112129
"old-curate",
@@ -115,7 +132,57 @@ describe("sweepOrphanedRetrospectiveChildren", () => {
115132
"old-compile",
116133
"old-confirm",
117134
]);
118-
expect(count).toBe(7);
135+
expect(count).toBe(8);
136+
});
137+
138+
test("keeps a recent historian child and sweeps it after the full attempt budget", async () => {
139+
db = makeOpencodeDb();
140+
const configuredStaleMs = historianOrphanStaleMs(10 * 60_000, 2);
141+
const historianNow = configuredStaleMs + 1_000_000;
142+
insert(
143+
db,
144+
"historian-still-draining",
145+
HISTORIAN_CHILD_TITLE,
146+
DIR,
147+
historianNow - configuredStaleMs + 1,
148+
);
149+
insert(
150+
db,
151+
"historian-budget-expired",
152+
HISTORIAN_CHILD_TITLE,
153+
DIR,
154+
historianNow - configuredStaleMs - 1,
155+
);
156+
const { client, deleted } = deleteClient();
157+
158+
const count = await sweepOrphanedRetrospectiveChildren({
159+
opencodeDb: db,
160+
client,
161+
sessionDirectory: DIR,
162+
staleMs: configuredStaleMs,
163+
now: historianNow,
164+
});
165+
166+
expect(count).toBe(1);
167+
expect(deleted).toEqual(["historian-budget-expired"]);
168+
});
169+
170+
test("preserves stale children when keep_subagents is enabled", async () => {
171+
db = makeOpencodeDb();
172+
insert(db, "kept-historian", HISTORIAN_CHILD_TITLE, DIR, now - staleMs - 1);
173+
const { client, deleted } = deleteClient();
174+
175+
const count = await sweepOrphanedRetrospectiveChildren({
176+
opencodeDb: db,
177+
client,
178+
sessionDirectory: DIR,
179+
staleMs,
180+
now,
181+
keepSubagents: true,
182+
});
183+
184+
expect(count).toBe(0);
185+
expect(deleted).toEqual([]);
119186
});
120187

121188
test("treats a delete error (404 / already removed) as success", async () => {

‎packages/plugin/src/features/magic-context/dreamer/retrospective-orphan-sweep.ts‎

Lines changed: 47 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -6,18 +6,20 @@ import type { DreamTaskName } from "./task-registry";
66
type OpencodeClient = PluginContext["client"];
77

88
/**
9-
* Privacy backstop for dreamer children that carry raw user or project text.
9+
* Age-gated backstop for internal children that carry raw user or project text.
1010
*
11-
* These child sessions normally delete themselves in `finally`, but a hard
12-
* SIGKILL/OOM BETWEEN session-create and that delete would leave their prompts
13-
* on disk. This sweep removes such crash-orphaned children.
11+
* Historian children deliberately remain after prompt completion because OpenCode
12+
* can still have detached writers persisting their final parts. Dreamer children
13+
* normally delete themselves in `finally`, but a hard SIGKILL/OOM can orphan them.
14+
* This sweep is the safe retirement path for both cases.
1415
*
1516
* CONCURRENCY: `session.delete` has no cross-process "active session" lease (OC
1617
* peer confirmed), so the ONLY safe filter is AGE — a child older than any
1718
* legitimate run cannot belong to a live run on another OpenCode process.
1819
* OpenCode sets `title` + `time_created` immediately at create (not lazily), so
1920
* the age gate is airtight. 404 on delete = already-swept = success.
2021
*/
22+
export const HISTORIAN_CHILD_TITLE = "magic-context-compartment";
2123
export const RETROSPECTIVE_CHILD_TITLE = "magic-context-dream-retrospective";
2224
export const USER_MEMORIES_CHILD_TITLE = "magic-context-dream-user-memories";
2325
export const CURATE_CHILD_TITLE = "magic-context-dream-curate";
@@ -51,6 +53,16 @@ export const PRIVACY_SENSITIVE_CHILD_TITLE_MATCHES: PrivacySensitiveChildTitleMa
5153
prefixes: [SMART_NOTE_COMPILE_CHILD_TITLE_PREFIX, SMART_NOTE_CONFIRM_CHILD_TITLE_PREFIX],
5254
};
5355

56+
export const HISTORIAN_CHILD_TITLE_MATCHES: PrivacySensitiveChildTitleMatches = {
57+
exact: [HISTORIAN_CHILD_TITLE],
58+
prefixes: [],
59+
};
60+
61+
export const INTERNAL_CHILD_TITLE_MATCHES: PrivacySensitiveChildTitleMatches = {
62+
exact: [HISTORIAN_CHILD_TITLE, ...PRIVACY_SENSITIVE_CHILD_TITLE_MATCHES.exact],
63+
prefixes: PRIVACY_SENSITIVE_CHILD_TITLE_MATCHES.prefixes,
64+
};
65+
5466
/** Stale threshold from task timeout(s): max(60min, maxTimeout×3) — comfortably
5567
* past every swept child type so a live child is never swept. */
5668
export function retrospectiveOrphanStaleMs(
@@ -67,14 +79,38 @@ export function retrospectiveOrphanStaleMs(
6779
return Math.max(60 * 60_000, timeoutMs * 3);
6880
}
6981

82+
const HISTORIAN_OUTER_ATTEMPTS = 3;
83+
const HISTORIAN_MODEL_SUGGESTION_ATTEMPTS = 2;
84+
const HISTORIAN_MAX_RETRY_BACKOFF_MS = 11_000;
85+
const HISTORIAN_DETACHED_WRITER_GRACE_MS = 15 * 60_000;
86+
87+
/**
88+
* Keep a historian child past its worst-case prompt budget. Each outer attempt
89+
* may traverse the primary plus every configured fallback, and every model can
90+
* consume a second timeout while following OpenCode's model-name suggestion.
91+
*/
92+
export function historianOrphanStaleMs(timeoutMs: number, fallbackModelCount: number): number {
93+
const timeout = Math.max(60_000, timeoutMs);
94+
const fallbackCount = Math.max(0, Math.floor(fallbackModelCount));
95+
const fullAttemptBudget =
96+
timeout *
97+
HISTORIAN_OUTER_ATTEMPTS *
98+
HISTORIAN_MODEL_SUGGESTION_ATTEMPTS *
99+
(fallbackCount + 1);
100+
return Math.max(
101+
60 * 60_000,
102+
fullAttemptBudget + HISTORIAN_MAX_RETRY_BACKOFF_MS + HISTORIAN_DETACHED_WRITER_GRACE_MS,
103+
);
104+
}
105+
70106
interface OrphanRow {
71107
id: string;
72108
time_created: number;
73109
}
74110

75111
/**
76-
* Delete crash-orphaned privacy-sensitive dreamer children for THIS project
77-
* directory when they are older than `staleMs`. Best-effort + fail-open: any
112+
* Delete stale internal children for THIS project directory when they are older
113+
* than `staleMs`. Best-effort + fail-open: any
78114
* DB/schema/API error is logged and skipped (never throws into the caller's
79115
* sweep). Returns the count deleted.
80116
*/
@@ -85,10 +121,11 @@ export async function sweepOrphanedRetrospectiveChildren(args: {
85121
staleMs: number;
86122
titleMatches?: PrivacySensitiveChildTitleMatches;
87123
now?: number;
124+
keepSubagents?: boolean;
88125
}): Promise<number> {
89126
const { opencodeDb, client, sessionDirectory, staleMs } = args;
90-
const titleMatches = args.titleMatches ?? PRIVACY_SENSITIVE_CHILD_TITLE_MATCHES;
91-
if (!opencodeDb) return 0;
127+
const titleMatches = args.titleMatches ?? INTERNAL_CHILD_TITLE_MATCHES;
128+
if (!opencodeDb || args.keepSubagents === true) return 0;
92129
const now = args.now ?? Date.now();
93130
const cutoff = now - staleMs;
94131

@@ -118,7 +155,7 @@ export async function sweepOrphanedRetrospectiveChildren(args: {
118155
.all(...titleParams, sessionDirectory, cutoff) as OrphanRow[];
119156
} catch (error) {
120157
// `session` table absent / schema drift / locked → skip silently.
121-
log(`[dreamer] retrospective orphan sweep: read skipped (${String(error)})`);
158+
log(`[magic-context] internal child orphan sweep: read skipped (${String(error)})`);
122159
return 0;
123160
}
124161
if (rows.length === 0) return 0;
@@ -134,7 +171,7 @@ export async function sweepOrphanedRetrospectiveChildren(args: {
134171
}
135172
}
136173
if (deleted > 0) {
137-
log(`[dreamer] swept ${deleted} crash-orphaned privacy-sensitive child session(s)`);
174+
log(`[magic-context] swept ${deleted} stale internal child session(s)`);
138175
}
139176
return deleted;
140177
}

‎packages/plugin/src/hooks/magic-context/compartment-runner-historian.ts‎

Lines changed: 7 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -325,9 +325,6 @@ async function runHistorianPrompt(args: {
325325
let agentSessionId: string | null = null;
326326
const startedAt = Date.now();
327327
let invocationRecorded = false;
328-
// Keep FAILED historian child sessions for debugging (the model output, the
329-
// exact prompt, and the error are all inspectable in the child session). Only
330-
// delete on SUCCESS, where the result is already persisted as a compartment.
331328
let outcomeOk = false;
332329

333330
const recordInvocation = (params: {
@@ -489,23 +486,15 @@ async function runHistorianPrompt(args: {
489486
error: `Historian failed while processing this session: ${desc.brief}`,
490487
};
491488
} finally {
492-
// Delete the child session ONLY on success. On failure, keep it so the
493-
// failed model output / prompt / error can be inspected for debugging
494-
// (the run is already recorded as failed in subagent_invocations +
495-
// historian_runs; the live child session is the missing piece). A periodic
496-
// sweep can GC old failed child sessions later if needed.
497-
if (agentSessionId && outcomeOk && !shouldKeepSubagents()) {
498-
await client.session.delete({ path: { id: agentSessionId } }).catch((e: unknown) => {
499-
shared.sessionLog(
500-
parentSessionId,
501-
"compartment agent: session cleanup failed",
502-
getErrorMessage(e),
503-
);
504-
});
505-
} else if (agentSessionId && (!outcomeOk || shouldKeepSubagents())) {
489+
if (agentSessionId) {
490+
const retentionReason = shouldKeepSubagents()
491+
? "keep_subagents"
492+
: outcomeOk
493+
? "prompt completed; cleanup deferred to the age-gated sweep"
494+
: "failed; cleanup deferred to the age-gated sweep";
506495
shared.sessionLog(
507496
parentSessionId,
508-
`historian: KEEPING child session ${agentSessionId} (${outcomeOk ? "keep_subagents" : "failed"}) — not deleted`,
497+
`historian: KEEPING child session ${agentSessionId} (${retentionReason}) — not deleted inline`,
509498
);
510499
}
511500
}

0 commit comments

Comments
 (0)