Skip to content

Commit c59635a

Browse files
steipetesteipete
authored
fix: avoid compaction checkpoint transcript copies (#86666)
1 parent 6814525 commit c59635a

16 files changed

Lines changed: 339 additions & 131 deletions

‎docs/concepts/compaction.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -106,14 +106,14 @@ The byte guard requires `truncateAfterCompaction: true`. Without transcript rota
106106

107107
### Successor transcripts
108108

109-
When `agents.defaults.compaction.truncateAfterCompaction` is enabled, OpenClaw does not rewrite the existing transcript in place. It creates a new active successor transcript from the compaction summary, preserved state, and unsummarized tail, then keeps the previous JSONL as the archived checkpoint source.
109+
When `agents.defaults.compaction.truncateAfterCompaction` is enabled, OpenClaw does not rewrite the existing transcript in place. It creates a new active successor transcript from the compaction summary, preserved state, and unsummarized tail, then records checkpoint metadata that points branch/restore flows at that compacted successor.
110110
Successor transcripts also drop exact duplicate long user turns that arrive
111111
inside a short retry window, so channel retry storms are not carried into the
112112
next active transcript after compaction.
113113

114-
Pre-compaction checkpoints are retained only while they stay below OpenClaw's
115-
checkpoint size cap; oversized active transcripts still compact, but OpenClaw
116-
skips the large debug snapshot instead of doubling disk usage.
114+
OpenClaw no longer writes separate `.checkpoint.*.jsonl` copies for new
115+
compactions. Existing legacy checkpoint files can still be used while referenced
116+
and are pruned by normal session cleanup.
117117

118118
### Compaction notices
119119

‎docs/reference/session-management-compaction.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -50,9 +50,9 @@ OpenClaw persists sessions in two layers:
5050
- Append-only transcript with tree structure (entries have `id` + `parentId`)
5151
- Stores the actual conversation + tool calls + compaction summaries
5252
- Used to rebuild the model context for future turns
53-
- Large pre-compaction debug checkpoints are skipped once the active
54-
transcript exceeds the checkpoint size cap, avoiding a second giant
55-
`.checkpoint.*.jsonl` copy.
53+
- Compaction checkpoints are metadata over the compacted successor
54+
transcript. New compactions do not write a second `.checkpoint.*.jsonl`
55+
copy.
5656

5757
Gateway history readers should avoid materializing the whole transcript unless
5858
the surface explicitly needs arbitrary historical access. First-page history,
@@ -357,8 +357,8 @@ OpenClaw also enforces a safety floor for embedded runs:
357357
disable.
358358
- When `agents.defaults.compaction.truncateAfterCompaction` is enabled,
359359
OpenClaw rotates the active transcript to a compacted successor JSONL after
360-
compaction. The old full transcript remains archived and linked from the
361-
compaction checkpoint instead of being rewritten in place.
360+
compaction. Branch/restore checkpoint actions use that compacted successor;
361+
legacy pre-compaction checkpoint files remain readable while referenced.
362362

363363
Why: leave enough headroom for multi-turn "housekeeping" (like memory writes) before compaction becomes unavoidable.
364364

‎docs/web/webchat.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@ Status: the macOS/iOS SwiftUI chat UI talks directly to the Gateway WebSocket.
2525
- The UI connects to the Gateway WebSocket and uses `chat.history`, `chat.send`, and `chat.inject`.
2626
- `chat.history` is bounded for stability: Gateway may truncate long text fields, omit heavy metadata, and replace oversized entries with `[chat.history omitted: message too large]`.
2727
- `chat.history` follows the active transcript branch for modern append-only session files, so abandoned rewrite branches and superseded prompt copies are not rendered in WebChat.
28-
- Compaction entries render as an explicit compacted-history divider. The divider explains that earlier turns are preserved in a checkpoint and links to the Sessions checkpoint controls, where operators can branch or restore the pre-compaction view when their permissions allow it.
28+
- Compaction entries render as an explicit compacted-history divider. The divider explains that the compacted transcript is preserved as a checkpoint and links to the Sessions checkpoint controls, where operators can branch or restore from that compacted view when their permissions allow it.
2929
- Control UI remembers the backing Gateway `sessionId` returned by `chat.history` and includes it on follow-up `chat.send` calls, so reconnects and page refreshes continue the same stored conversation unless the user starts or resets a session.
3030
- Control UI coalesces duplicate in-flight submits for the same session, message, and attachments before generating a new `chat.send` run id; the Gateway still dedupes repeated requests that reuse the same idempotency key.
3131
- Workspace startup files and pending `BOOTSTRAP.md` instructions are supplied through the agent system prompt's Project Context, not copied into the WebChat user message. Bootstrap truncation only adds a concise system-prompt recovery notice; detailed counts and config knobs stay on diagnostic surfaces.

‎src/config/sessions/disk-budget.test.ts‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -196,6 +196,7 @@ describe("enforceSessionDiskBudget", () => {
196196
dir,
197197
"keep.checkpoint.22222222-2222-4222-8222-222222222222.jsonl",
198198
);
199+
const referencedPostCompactionPath = path.join(dir, "keep-compacted.jsonl");
199200
const store: Record<string, SessionEntry> = {
200201
"agent:main:main": {
201202
sessionId,
@@ -212,7 +213,7 @@ describe("enforceSessionDiskBudget", () => {
212213
sessionFile: referencedCheckpointPath,
213214
leafId: "leaf",
214215
},
215-
postCompaction: { sessionId },
216+
postCompaction: { sessionId, sessionFile: referencedPostCompactionPath },
216217
},
217218
],
218219
},
@@ -221,6 +222,7 @@ describe("enforceSessionDiskBudget", () => {
221222
await fs.writeFile(transcriptPath, "k".repeat(80), "utf-8");
222223
await fs.writeFile(checkpointPath, "c".repeat(5000), "utf-8");
223224
await fs.writeFile(referencedCheckpointPath, "r".repeat(260), "utf-8");
225+
await fs.writeFile(referencedPostCompactionPath, "p".repeat(260), "utf-8");
224226

225227
const result = await enforceSessionDiskBudget({
226228
store,
@@ -235,6 +237,7 @@ describe("enforceSessionDiskBudget", () => {
235237
await expectPathExists(transcriptPath);
236238
await expectPathMissing(checkpointPath);
237239
await expectPathExists(referencedCheckpointPath);
240+
await expectPathExists(referencedPostCompactionPath);
238241
expectBudgetResult(result);
239242
expect(result.removedFiles).toBe(1);
240243
expect(result.removedEntries).toBe(0);

‎src/config/sessions/disk-budget.ts‎

Lines changed: 10 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -176,14 +176,16 @@ function resolveReferencedSessionArtifactPaths(params: {
176176
referenced.add(resolved);
177177
}
178178
for (const checkpoint of entry.compactionCheckpoints ?? []) {
179-
const checkpointFile = checkpoint.preCompaction.sessionFile?.trim();
180-
if (!checkpointFile) {
181-
continue;
182-
}
183-
const resolvedCheckpointPath = canonicalizePathForComparison(checkpointFile);
184-
const relative = path.relative(resolvedSessionsDir, resolvedCheckpointPath);
185-
if (relative && !relative.startsWith("..") && !path.isAbsolute(relative)) {
186-
referenced.add(resolvedCheckpointPath);
179+
const checkpointFiles = [
180+
checkpoint.preCompaction.sessionFile?.trim(),
181+
checkpoint.postCompaction.sessionFile?.trim(),
182+
].filter((filePath): filePath is string => Boolean(filePath));
183+
for (const checkpointFile of checkpointFiles) {
184+
const resolvedCheckpointPath = canonicalizePathForComparison(checkpointFile);
185+
const relative = path.relative(resolvedSessionsDir, resolvedCheckpointPath);
186+
if (relative && !relative.startsWith("..") && !path.isAbsolute(relative)) {
187+
referenced.add(resolvedCheckpointPath);
188+
}
187189
}
188190
}
189191
}

‎src/config/sessions/store.pruning.integration.test.ts‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -238,6 +238,7 @@ describe("Integration: saveSessionStore with pruning", () => {
238238
testDir,
239239
"fresh-session.checkpoint.22222222-2222-4222-8222-222222222222.jsonl",
240240
);
241+
const referencedPostCompactionPath = path.join(testDir, "fresh-session-compacted.jsonl");
241242
const store: Record<string, SessionEntry> = {
242243
fresh: {
243244
sessionId: "fresh-session",
@@ -254,7 +255,10 @@ describe("Integration: saveSessionStore with pruning", () => {
254255
sessionFile: referencedCheckpointPath,
255256
leafId: "leaf",
256257
},
257-
postCompaction: { sessionId: "fresh-session" },
258+
postCompaction: {
259+
sessionId: "fresh-session",
260+
sessionFile: referencedPostCompactionPath,
261+
},
258262
},
259263
],
260264
},
@@ -271,6 +275,7 @@ describe("Integration: saveSessionStore with pruning", () => {
271275
await fs.writeFile(storePath, JSON.stringify(store, null, 2), "utf-8");
272276
await fs.writeFile(referencedTranscript, "referenced", "utf-8");
273277
await fs.writeFile(referencedCheckpointPath, "referenced checkpoint", "utf-8");
278+
await fs.writeFile(referencedPostCompactionPath, "referenced post-compaction", "utf-8");
274279
await fs.writeFile(oldOrphanTranscript, "orphan transcript", "utf-8");
275280
await fs.writeFile(freshOrphanTranscript, "fresh orphan", "utf-8");
276281
await fs.writeFile(orphanRuntime, "orphan runtime", "utf-8");
@@ -279,6 +284,7 @@ describe("Integration: saveSessionStore with pruning", () => {
279284
for (const file of [
280285
referencedTranscript,
281286
referencedCheckpointPath,
287+
referencedPostCompactionPath,
282288
oldOrphanTranscript,
283289
orphanRuntime,
284290
orphanPointer,
@@ -312,6 +318,7 @@ describe("Integration: saveSessionStore with pruning", () => {
312318
await expectPathMissing(orphanCheckpoint);
313319
await expectPathExists(referencedTranscript);
314320
await expectPathExists(referencedCheckpointPath);
321+
await expectPathExists(referencedPostCompactionPath);
315322
await expectPathExists(freshOrphanTranscript);
316323
});
317324

‎src/gateway/server-methods/sessions.ts‎

Lines changed: 39 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -433,6 +433,33 @@ function cloneCheckpointSessionEntry(params: {
433433
};
434434
}
435435

436+
function resolveCheckpointForkSource(
437+
checkpoint: NonNullable<ReturnType<typeof getSessionCompactionCheckpoint>>,
438+
): { sourceFile: string; sourceLeafId?: string; totalTokens?: number } | null {
439+
const preCompactionFile = checkpoint.preCompaction.sessionFile?.trim();
440+
if (preCompactionFile) {
441+
return {
442+
sourceFile: preCompactionFile,
443+
sourceLeafId: checkpoint.preCompaction.leafId,
444+
totalTokens: checkpoint.tokensBefore,
445+
};
446+
}
447+
const postCompactionFile = checkpoint.postCompaction.sessionFile?.trim();
448+
if (!postCompactionFile) {
449+
return null;
450+
}
451+
const postCompactionLeafId =
452+
checkpoint.postCompaction.leafId ?? checkpoint.postCompaction.entryId;
453+
if (!postCompactionLeafId) {
454+
return null;
455+
}
456+
return {
457+
sourceFile: postCompactionFile,
458+
sourceLeafId: postCompactionLeafId,
459+
totalTokens: checkpoint.tokensAfter,
460+
};
461+
}
462+
436463
function isAgentMainSessionKey(cfg: OpenClawConfig, sessionKey: string): boolean {
437464
const parsed = parseAgentSessionKey(sessionKey);
438465
if (!parsed) {
@@ -1555,7 +1582,8 @@ export const sessionsHandlers: GatewayRequestHandlers = {
15551582
return;
15561583
}
15571584
const checkpoint = getSessionCompactionCheckpoint({ entry, checkpointId });
1558-
if (!checkpoint?.preCompaction.sessionFile) {
1585+
const forkSource = checkpoint ? resolveCheckpointForkSource(checkpoint) : null;
1586+
if (!checkpoint || !forkSource) {
15591587
respond(
15601588
false,
15611589
undefined,
@@ -1564,8 +1592,9 @@ export const sessionsHandlers: GatewayRequestHandlers = {
15641592
return;
15651593
}
15661594
const branchedSession = await forkCompactionCheckpointTranscriptAsync({
1567-
sourceFile: checkpoint.preCompaction.sessionFile,
1568-
sessionDir: path.dirname(checkpoint.preCompaction.sessionFile),
1595+
sourceFile: forkSource.sourceFile,
1596+
sourceLeafId: forkSource.sourceLeafId,
1597+
sessionDir: path.dirname(forkSource.sourceFile),
15691598
});
15701599
if (!branchedSession?.sessionFile) {
15711600
respond(
@@ -1583,7 +1612,7 @@ export const sessionsHandlers: GatewayRequestHandlers = {
15831612
nextSessionFile: branchedSession.sessionFile,
15841613
label,
15851614
parentSessionKey: canonicalKey,
1586-
totalTokens: checkpoint.tokensBefore,
1615+
totalTokens: forkSource.totalTokens,
15871616
});
15881617

15891618
await updateSessionStore(target.storePath, (store) => {
@@ -1654,7 +1683,8 @@ export const sessionsHandlers: GatewayRequestHandlers = {
16541683
return;
16551684
}
16561685
const checkpoint = getSessionCompactionCheckpoint({ entry, checkpointId });
1657-
if (!checkpoint?.preCompaction.sessionFile) {
1686+
const forkSource = checkpoint ? resolveCheckpointForkSource(checkpoint) : null;
1687+
if (!checkpoint || !forkSource) {
16581688
respond(
16591689
false,
16601690
undefined,
@@ -1677,8 +1707,9 @@ export const sessionsHandlers: GatewayRequestHandlers = {
16771707
}
16781708

16791709
const restoredSession = await forkCompactionCheckpointTranscriptAsync({
1680-
sourceFile: checkpoint.preCompaction.sessionFile,
1681-
sessionDir: path.dirname(checkpoint.preCompaction.sessionFile),
1710+
sourceFile: forkSource.sourceFile,
1711+
sourceLeafId: forkSource.sourceLeafId,
1712+
sessionDir: path.dirname(forkSource.sourceFile),
16821713
});
16831714
if (!restoredSession?.sessionFile) {
16841715
respond(
@@ -1692,7 +1723,7 @@ export const sessionsHandlers: GatewayRequestHandlers = {
16921723
currentEntry: entry,
16931724
nextSessionId: restoredSession.sessionId,
16941725
nextSessionFile: restoredSession.sessionFile,
1695-
totalTokens: checkpoint.tokensBefore,
1726+
totalTokens: forkSource.totalTokens,
16961727
preserveCompactionCheckpoints: true,
16971728
});
16981729

0 commit comments

Comments
 (0)