Skip to content

Commit b2b58a9

Browse files
committed
Merge remote-tracking branch 'origin/main' into fix/exec-ansi-split-chunks
2 parents 0bb3590 + ef91ce5 commit b2b58a9

31 files changed

Lines changed: 1510 additions & 488 deletions

docs/concepts/main-session.md

Lines changed: 11 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -60,20 +60,26 @@ continuity comes from layers around it:
6060
is enabled by default; any configured DM isolation turns it off unless you
6161
opt in explicitly. See [Memory configuration](/reference/memory-config).
6262

63-
## A rolling session, not an immortal one
63+
## A rolling session with durable history
6464

6565
The main session rolls forward through resets and compaction rather than
66-
growing forever:
66+
making the model carry its entire history at once:
6767

6868
- By default there is no automatic reset; compaction keeps the active context
6969
bounded while preserving the rolling session. Daily and idle resets are
7070
opt-in (see [Session management](/concepts/session)). On `/new` and `/reset`,
7171
the tail of the ending conversation is saved to daily memory notes, and the
72-
next session re-primes recent notes.
72+
next session re-primes recent notes. Reset assigns a new live session id but
73+
keeps the previous SQLite transcript searchable under the same main-session
74+
key.
7375
- When the conversation approaches the context window, compaction summarizes
7476
and continues in place — the transcript history stays in the session store.
75-
- The per-agent session store keeps archived transcripts until a disk budget
76-
(default 10 GB) evicts the oldest ones.
77+
- Session lists show the current live conversation, not every historical
78+
session id behind it.
79+
- When the per-agent store's physical database, WAL, and session artifacts
80+
exceed the disk budget (default 10 GB), OpenClaw extracts the oldest
81+
unreferenced history to a verified compressed archive before removing its
82+
database rows. Live, routed, and in-flight sessions are never budget victims.
7783

7884
## When you want isolation instead
7985

docs/docs_map.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2493,7 +2493,7 @@ Do not edit it by hand; run `pnpm docs:map:gen`.
24932493
- H2: Home
24942494
- H2: What flows into the main session
24952495
- H2: Memory across resets and conversations
2496-
- H2: A rolling session, not an immortal one
2496+
- H2: A rolling session with durable history
24972497
- H2: When you want isolation instead
24982498
- H2: Related
24992499

docs/reference/session-management-compaction.md

Lines changed: 4 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -53,19 +53,15 @@ Per agent, on the Gateway host (resolved via `src/config/sessions.ts`):
5353
| `maxDiskBytes` | `10gb` | per-agent sessions disk budget; `false` disables |
5454
| `highWaterBytes` | 80% of `maxDiskBytes` | target after budget cleanup |
5555

56-
Archived transcripts are kept by default and compressed with zstd (`*.jsonl.<reason>.<timestamp>.zst`) when the runtime supports it, so deleting or resetting a session never silently discards conversation history. The disk budget evicts the oldest archives first, before touching live sessions.
56+
Reset advances the live `sessionKey -> sessionId` mapping but keeps the previous SQLite session, transcript, trajectory, and search rows. That history remains searchable under the same session key; ordinary entry and session lists show only the new live mapping. Retained reset history is bounded by the disk budget, not by `resetArchiveRetention`, which only ages archive artifacts. Explicit deletion is different: it writes and verifies a compressed transcript archive (`*.jsonl.deleted.<timestamp>.zst` when zstd is available) before removing the deleted session's rows.
5757

58-
Active SQLite enforcement of `maxDiskBytes` measures session-row JSON plus transcript-event JSON bytes per session; legacy offline-maintenance enforcement measures files in the selected sessions directory.
58+
`maxDiskBytes` enforcement uses physical bytes: the per-agent SQLite main file, its `-wal` file, and counted files in the agent sessions directory. It never estimates row JSON sizes or subtracts logical row sizes from that total.
5959

6060
Gateway model-run probe sessions (keys matching `agent:*:explicit:model-run-<uuid>`) get a separate, fixed `24h` retention. This pruning is pressure-gated: it only runs when session-entry maintenance/cap pressure is reached, and only before the global stale-entry cleanup/cap step. Other explicit sessions do not use this retention.
6161

62-
Enforcement order for disk-budget cleanup (`mode: "enforce"`):
62+
When combined physical usage exceeds `maxDiskBytes`, `mode: "enforce"` first reclaims checkpointable database space, then removes the oldest retained reset/delete archives. If usage is still above `highWaterBytes`, it walks historical SQLite sessions by `sessions.updated_at`, oldest first. Historical means the session id is not referenced by a live session entry, a route target, or an admitted/in-flight run. For each victim, cleanup writes, fsyncs, and reads back the compressed archive before a write transaction removes the session row and its transcript, trajectory, active, index, and FTS projections. This includes sessions that contain trajectory events but no transcript events. Cleanup rechecks route, entry, and admission references at deletion time, remeasures physical usage after each archive or session victim, and stops at `highWaterBytes`.
6363

64-
1. Remove oldest archived transcript artifacts, orphan legacy artifacts, or orphan trajectory artifacts first.
65-
2. If still above target, evict oldest session entries and their transcript rows or trajectory artifacts.
66-
3. Repeat until usage is at or below `highWaterBytes`.
67-
68-
`mode: "warn"` reports potential evictions without mutating the store or files.
64+
Committed writes and deletion first land in the WAL. Cleanup checkpoints it so the WAL can shrink immediately, then uses incremental vacuum to return eligible free tail pages from the main file; pages that are not yet reclaimable stay in the main file and therefore remain counted on the next physical measurement. `mode: "warn"` reports the current physical overage without checkpointing, writing an archive, or deleting rows.
6965

7066
Run maintenance on demand:
7167

scripts/github/pr-ci-sweeper.d.mts

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,4 +16,5 @@ export function runPrCiSweeper(params: {
1616
core: Pick<Console, "info"> & { setFailed: (message: string) => void };
1717
dryRun?: boolean;
1818
appSlug?: string;
19+
now?: number;
1920
}): Promise<Array<{ number: number; sha: string; action: "refire" | "skip"; reason: string }>>;

scripts/github/pr-ci-sweeper.mjs

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -131,13 +131,21 @@ async function reopenWithRetry({ github, core, owner, repo, pullNumber }) {
131131
return false;
132132
}
133133

134-
export async function runPrCiSweeper({ github, context, core, dryRun = false, appSlug = "" }) {
134+
export async function runPrCiSweeper({
135+
github,
136+
context,
137+
core,
138+
dryRun = false,
139+
appSlug = "",
140+
// Injectable clock: fixture-based tests pin a fixed instant so lookback
141+
// classification cannot rot as wall-clock time passes the fixture dates.
142+
now = Date.now(),
143+
}) {
135144
const sweeperLogins = new Set(KNOWN_SWEEPER_LOGINS);
136145
if (appSlug) {
137146
sweeperLogins.add(`${appSlug}[bot]`);
138147
}
139148
const { owner, repo } = context.repo;
140-
const now = Date.now();
141149
const results = [];
142150
let refires = 0;
143151
const openPrs = await github.paginate(github.rest.pulls.list, {

src/config/sessions/artifacts.ts

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,11 @@ export function isSessionArchiveArtifactName(fileName: string): boolean {
3838
);
3939
}
4040

41+
/** Returns true for retained reset/delete transcript archives counted by the session budget. */
42+
export function isRetainedSessionTranscriptArchiveName(fileName: string): boolean {
43+
return hasArchiveSuffix(fileName, "deleted") || hasArchiveSuffix(fileName, "reset");
44+
}
45+
4146
/** Returns true for migration rollback archives retained beside their legacy source. */
4247
export function isMigrationArchiveArtifactName(fileName: string): boolean {
4348
return MIGRATION_ARCHIVE_RE.test(fileName);

src/config/sessions/cleanup-service.ts

Lines changed: 32 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -19,10 +19,13 @@ import {
1919
applySessionEntryLifecycleMutation,
2020
listSessionEntries,
2121
loadTranscriptEventsSync,
22-
previewSessionDiskBudget,
2322
purgeDeletedAgentSessionEntries,
2423
type SessionEntryLifecycleRemoval,
2524
} from "./session-accessor.js";
25+
import {
26+
enforceSqliteSessionHistoryDiskBudget,
27+
inspectSqliteSessionHistoryDiskBudget,
28+
} from "./session-history-eviction.js";
2629
import { resolveSqliteTargetFromSessionStorePath } from "./session-sqlite-target.js";
2730
import { cloneSessionStoreRecord } from "./store-cache.js";
2831
import { collectSessionMaintenancePreserveKeysForStore } from "./store-maintenance-preserve.js";
@@ -435,15 +438,13 @@ async function previewStoreCleanup(params: {
435438
keys: dmScopeRetiredKeys,
436439
});
437440
const diskBudgetPreview = fs.existsSync(resolveCleanupSqlitePath(params.target))
438-
? previewSessionDiskBudget({
441+
? await inspectSqliteSessionHistoryDiskBudget({
439442
agentId: params.target.agentId,
440-
store: previewStore,
441443
storePath: params.target.storePath,
442-
activeSessionKey: params.activeKey,
443-
preserveKeys: preserveSessionKeys,
444+
mode: params.mode,
444445
maintenance: params.maintenance,
445446
})
446-
: { diskBudget: null, removedKeys: new Set<string>() };
447+
: { diskBudget: null, wouldMutate: false };
447448
const diskBudget = diskBudgetPreview.diskBudget;
448449
const unreferencedArtifacts = await pruneUnreferencedSessionArtifacts({
449450
store: previewStore,
@@ -452,7 +453,7 @@ async function previewStoreCleanup(params: {
452453
dryRun: true,
453454
excludeCanonicalPaths: entryCleanupArtifactPaths,
454455
});
455-
const budgetEvictedKeys = diskBudgetPreview.removedKeys;
456+
const budgetEvictedKeys = new Set<string>();
456457
const beforeCount = Object.keys(beforeStore).length;
457458
const afterPreviewCount = Object.keys(previewStore).length;
458459
const wouldMutate =
@@ -463,7 +464,8 @@ async function previewStoreCleanup(params: {
463464
capped > 0 ||
464465
unreferencedArtifacts.removedFiles > 0 ||
465466
(diskBudget?.removedEntries ?? 0) > 0 ||
466-
(diskBudget?.removedFiles ?? 0) > 0;
467+
(diskBudget?.removedFiles ?? 0) > 0 ||
468+
diskBudgetPreview.wouldMutate;
467469

468470
const summary: SessionCleanupSummary = {
469471
agentId: params.target.agentId,
@@ -606,6 +608,12 @@ export async function runSessionsCleanup(params: {
606608
freedBytes: 0,
607609
olderThanMs: maintenance.pruneAfterMs,
608610
});
611+
const appliedDiskBudget = await enforceSqliteSessionHistoryDiskBudget({
612+
agentId: target.agentId,
613+
storePath: target.storePath,
614+
mode,
615+
maintenance,
616+
});
609617
const preview = previewResults.find(
610618
(result) => result.summary.storePath === target.storePath,
611619
);
@@ -631,8 +639,16 @@ export async function runSessionsCleanup(params: {
631639
}),
632640
dryRun: false,
633641
unreferencedArtifacts,
642+
diskBudget: appliedDiskBudget,
634643
wouldMutate:
635-
(preview?.summary.wouldMutate ?? false) || unreferencedArtifacts.removedFiles > 0,
644+
removedSessionKeys.size > 0 ||
645+
unreferencedArtifacts.removedFiles > 0 ||
646+
(appliedDiskBudget?.removedEntries ?? 0) > 0 ||
647+
(appliedDiskBudget?.removedFiles ?? 0) > 0 ||
648+
// Checkpoint/incremental-vacuum reclamation mutates the store
649+
// even when no session or archive was removed.
650+
(appliedDiskBudget != null &&
651+
appliedDiskBudget.totalBytesAfter < appliedDiskBudget.totalBytesBefore),
636652
applied: true,
637653
appliedCount: lifecycleResult.afterCount,
638654
}
@@ -649,16 +665,20 @@ export async function runSessionsCleanup(params: {
649665
pruned: appliedReport.pruned,
650666
capped: appliedReport.capped,
651667
unreferencedArtifacts,
652-
diskBudget: appliedReport.diskBudget,
668+
diskBudget: appliedDiskBudget,
653669
wouldMutate:
654670
missingApplied > 0 ||
655671
dmScopeRetiredApplied > 0 ||
656672
appliedReport.modelRunPruned > 0 ||
657673
appliedReport.pruned > 0 ||
658674
appliedReport.capped > 0 ||
659675
unreferencedArtifacts.removedFiles > 0 ||
660-
(appliedReport.diskBudget?.removedEntries ?? 0) > 0 ||
661-
(appliedReport.diskBudget?.removedFiles ?? 0) > 0,
676+
(appliedDiskBudget?.removedEntries ?? 0) > 0 ||
677+
(appliedDiskBudget?.removedFiles ?? 0) > 0 ||
678+
// Checkpoint/incremental-vacuum reclamation mutates the store
679+
// even when no session or archive was removed.
680+
(appliedDiskBudget != null &&
681+
appliedDiskBudget.totalBytesAfter < appliedDiskBudget.totalBytesBefore),
662682
applied: true,
663683
appliedCount: lifecycleResult.afterCount,
664684
};

src/config/sessions/disk-budget.test.ts

Lines changed: 47 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,12 @@ import {
1010
resolveTrajectoryPointerFilePath,
1111
} from "../../trajectory/paths.js";
1212
import { formatSessionArchiveTimestamp } from "./artifacts.js";
13-
import { enforceSessionDiskBudget, pruneUnreferencedSessionArtifacts } from "./disk-budget.js";
13+
import {
14+
enforceSessionDiskBudget,
15+
measureSessionPhysicalDiskUsage,
16+
pruneUnreferencedSessionArtifacts,
17+
} from "./disk-budget.js";
18+
import { resolveSqliteTargetFromSessionStorePath } from "./session-sqlite-target.js";
1419
import { saveSessionStore } from "./store.js";
1520
import type { SessionEntry } from "./types.js";
1621

@@ -54,6 +59,47 @@ function refreshPathBeforeSecondStat(targetPath: string): ReturnType<typeof vi.s
5459
}
5560

5661
describe("enforceSessionDiskBudget", () => {
62+
it("counts the SQLite main file and WAL as physical session usage", async () => {
63+
await withTempDir({ prefix: "openclaw-disk-budget-sqlite-" }, async (dir) => {
64+
const storePath = path.join(dir, "sessions.json");
65+
const databasePath = resolveSqliteTargetFromSessionStorePath(storePath).path;
66+
if (!databasePath) {
67+
throw new Error("expected a SQLite database path");
68+
}
69+
await fs.writeFile(databasePath, Buffer.alloc(321));
70+
await fs.writeFile(`${databasePath}-wal`, Buffer.alloc(654));
71+
72+
const usage = await measureSessionPhysicalDiskUsage(storePath);
73+
74+
expect(usage).toEqual({
75+
databaseMainBytes: 321,
76+
databaseWalBytes: 654,
77+
sessionFilesBytes: 0,
78+
totalBytes: 975,
79+
});
80+
});
81+
});
82+
83+
it("excludes migration archives from physical SQLite usage (#106875)", async () => {
84+
await withTempDir({ prefix: "openclaw-disk-budget-sqlite-" }, async (dir) => {
85+
const storePath = path.join(dir, "sessions.json");
86+
const databasePath = resolveSqliteTargetFromSessionStorePath(storePath).path;
87+
if (!databasePath) {
88+
throw new Error("expected a SQLite database path");
89+
}
90+
await fs.writeFile(databasePath, Buffer.alloc(100));
91+
// Rollback archives are recovery artifacts outside the session budget;
92+
// counting them would evict live history to pay for unreclaimable bytes.
93+
await fs.writeFile(path.join(dir, "legacy.jsonl.migrated"), Buffer.alloc(4096));
94+
await fs.writeFile(path.join(dir, "legacy.jsonl.migrated.2"), Buffer.alloc(4096));
95+
96+
const usage = await measureSessionPhysicalDiskUsage(storePath);
97+
98+
expect(usage.totalBytes).toBe(100);
99+
expect(usage.sessionFilesBytes).toBe(0);
100+
});
101+
});
102+
57103
it("excludes migration archives from the session disk budget (#106875)", async () => {
58104
await withTempDir({ prefix: "openclaw-disk-budget-" }, async (dir) => {
59105
const storePath = path.join(dir, "sessions.json");

src/config/sessions/disk-budget.ts

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,12 +14,14 @@ import {
1414
isCompactionCheckpointTranscriptFileName,
1515
isMigrationArchiveArtifactName,
1616
isPrimarySessionTranscriptFileName,
17+
isRetainedSessionTranscriptArchiveName,
1718
isSessionArchiveArtifactName,
1819
isSessionStoreTempArtifactName,
1920
SESSION_STORE_TEMP_STALE_MS,
2021
isTrajectorySessionArtifactName,
2122
} from "./artifacts.js";
2223
import { resolveSessionFilePath } from "./paths.js";
24+
import { resolveSqliteTargetFromSessionStorePath } from "./session-sqlite-target.js";
2325
import { projectSessionStoreForPersistence } from "./skill-prompt-blobs.js";
2426
import { shouldPreserveMaintenanceEntry } from "./store-maintenance.js";
2527
import type { SessionEntry } from "./types.js";
@@ -47,6 +49,13 @@ export type SessionUnreferencedArtifactSweepResult = {
4749
olderThanMs: number;
4850
};
4951

52+
export type SessionPhysicalDiskUsage = {
53+
databaseMainBytes: number;
54+
databaseWalBytes: number;
55+
sessionFilesBytes: number;
56+
totalBytes: number;
57+
};
58+
5059
type SessionDiskBudgetLogger = {
5160
warn: (message: string, context?: Record<string, unknown>) => void;
5261
info: (message: string, context?: Record<string, unknown>) => void;
@@ -248,6 +257,86 @@ async function readSessionsDirFiles(sessionsDir: string): Promise<SessionsDirFil
248257
return results.filter((file): file is SessionsDirFileStat => Boolean(file));
249258
}
250259

260+
async function readSqliteDatabaseFiles(storePath: string): Promise<SessionsDirFileStat[]> {
261+
const databasePath = resolveSqliteTargetFromSessionStorePath(storePath).path;
262+
if (!databasePath) {
263+
return [];
264+
}
265+
const files: SessionsDirFileStat[] = [];
266+
for (const filePath of [databasePath, `${databasePath}-wal`]) {
267+
const stat = await fs.promises.stat(filePath).catch(() => null);
268+
if (!stat?.isFile()) {
269+
continue;
270+
}
271+
files.push({
272+
path: filePath,
273+
canonicalPath: canonicalizePathForComparison(filePath),
274+
name: path.basename(filePath),
275+
size: stat.size,
276+
mtimeMs: stat.mtimeMs,
277+
});
278+
}
279+
return files;
280+
}
281+
282+
/** Measures current physical session artifacts plus the agent SQLite main file and WAL. */
283+
export async function measureSessionPhysicalDiskUsage(
284+
storePath: string,
285+
): Promise<SessionPhysicalDiskUsage> {
286+
const sessionsDirFiles = await readSessionsDirFiles(path.dirname(storePath));
287+
const promptBlobFiles = await readSessionPromptBlobFiles(path.dirname(storePath));
288+
const databaseFiles = await readSqliteDatabaseFiles(storePath);
289+
const databasePath = resolveSqliteTargetFromSessionStorePath(storePath).path;
290+
const databaseMainPath = databasePath ? canonicalizePathForComparison(databasePath) : undefined;
291+
const databaseWalPath = databasePath
292+
? canonicalizePathForComparison(`${databasePath}-wal`)
293+
: undefined;
294+
const uniqueFiles = new Map<string, SessionsDirFileStat>();
295+
for (const file of [...sessionsDirFiles, ...promptBlobFiles, ...databaseFiles]) {
296+
uniqueFiles.set(file.canonicalPath, file);
297+
}
298+
const databaseMainBytes = databaseMainPath ? (uniqueFiles.get(databaseMainPath)?.size ?? 0) : 0;
299+
const databaseWalBytes = databaseWalPath ? (uniqueFiles.get(databaseWalPath)?.size ?? 0) : 0;
300+
const totalBytes = [...uniqueFiles.values()].reduce((sum, file) => sum + file.size, 0);
301+
return {
302+
databaseMainBytes,
303+
databaseWalBytes,
304+
sessionFilesBytes: totalBytes - databaseMainBytes - databaseWalBytes,
305+
totalBytes,
306+
};
307+
}
308+
309+
export async function hasRetainedSessionTranscriptArchives(storePath: string): Promise<boolean> {
310+
const files = await readSessionsDirFiles(path.dirname(storePath));
311+
return files.some((file) => isRetainedSessionTranscriptArchiveName(file.name));
312+
}
313+
314+
/** Removes oldest retained reset/delete archives, remeasuring physical usage after each file. */
315+
export async function pruneSessionTranscriptArchivesToHighWater(params: {
316+
highWaterBytes: number;
317+
storePath: string;
318+
}): Promise<{ removedFiles: number; usage: SessionPhysicalDiskUsage }> {
319+
// Oldest-first is the hard-cap sacrifice order: under extreme pressure this
320+
// may prune an archive the current pass just extracted, which is preferred
321+
// over evicting additional sessions' searchable rows to spare a copy.
322+
const files = (await readSessionsDirFiles(path.dirname(params.storePath)))
323+
.filter((file) => isRetainedSessionTranscriptArchiveName(file.name))
324+
.toSorted((left, right) => left.mtimeMs - right.mtimeMs);
325+
let usage = await measureSessionPhysicalDiskUsage(params.storePath);
326+
let removedFiles = 0;
327+
for (const file of files) {
328+
if (usage.totalBytes <= params.highWaterBytes) {
329+
break;
330+
}
331+
if ((await removeFileIfExists(file.path)) <= 0) {
332+
continue;
333+
}
334+
removedFiles += 1;
335+
usage = await measureSessionPhysicalDiskUsage(params.storePath);
336+
}
337+
return { removedFiles, usage };
338+
}
339+
251340
async function readSessionPromptBlobFiles(sessionsDir: string): Promise<SessionsDirFileStat[]> {
252341
const root = path.join(sessionsDir, "skills-prompts", "sha256");
253342
const prefixEntries = await fs.promises.readdir(root, { withFileTypes: true }).catch(() => []);

0 commit comments

Comments
 (0)