fix(backup): clean up stale .tmp archives from interrupted runs before creating new backup#91663
Conversation
…e creating new backup When backup create is interrupted (timeout, SIGKILL, etc.), the temporary .tar.gz.<uuid>.tmp file is left behind. Over successive runs these accumulate and can consume significant disk space (12.6GB reported in issue openclaw#50442). Add cleanupStaleTempArchives() that scans the output directory for files matching the backup temp pattern (<basename>.<uuid>.tmp) and removes them before creating a new archive. The cleanup is non-fatal — errors from missing directories or permission issues are logged and execution continues. Fixes openclaw#50442
|
Thanks for the context here. I swept through the related work, and this is now duplicate or superseded. Close as superseded: the branch contains a useful archive-temp cleanup idea, but it is conflicting, has an active-writer safety defect, and the newer clean replacement PR covers the same archive leak plus the stale staging-directory half with owner-marker protection and sufficient proof. Root-cause cluster Members:
Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything. Canonical path: Use #95600 as the canonical landing path for guarded two-artifact backup temp cleanup, then close the related backup-temp reports after merge. So I’m closing this here and keeping the remaining discussion on #95600. Review detailsBest possible solution: Use #95600 as the canonical landing path for guarded two-artifact backup temp cleanup, then close the related backup-temp reports after merge. Do we have a high-confidence way to reproduce the issue? Yes, at source level. Current main creates the reported output-adjacent temp archive and staging directory and removes them only during an in-process Is this the best way to solve the issue? No for this PR as the landing path. The safer maintainable solution is the newer guarded startup sweep in #95600, which covers both artifact classes and avoids an age-only active-writer heuristic. Security review: Security review cleared: The diff touches backup source and tests only and introduces no dependency, workflow, credential, or supply-chain change; destructive deletion is tracked as availability merge risk. AGENTS.md: found and applied where relevant. What I checked:
Likely related people:
Codex review notes: model internal, reasoning high; reviewed against 4fc504d321b6. |
…ives Only remove .tmp files whose mtime is older than a 30 s staleness threshold. This protects against concurrently running backup processes that target the same output path — their actively-written temp archive will be preserved. Also reports how many active temp archives were skipped in the cleanup log message for observability.
|
@clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
|
Closing as superseded by #95600, which covers the same archive-temp cleanup plus stale staging-directory cleanup with owner-marker protection. Thanks for the review. |
Summary
When
openclaw backup createis interrupted (timeout, SIGKILL, process crash), the temporary.tar.gz.<uuid>.tmpfile created bybuildTempArchivePath()is left on disk. On successive runs, each interrupted backup leaves a new.tmpfile, accumulating and consuming significant disk space (12.6GB across 6 files reported in #50442).This PR adds
cleanupStaleTempArchives()that scans the output directory for files matching the backup temp pattern (<basename>.<uuid>.tmp) and removes stale ones before creating a new archive.Key safety property: Only
.tmpfiles whose mtime is older than a 30 s staleness threshold (BACKUP_TEMP_STALE_AGE_MS) are removed. This protects against concurrently running backup processes that target the same output path — their actively-written temp archive will be preserved, and the count of skipped active archives is reported in the log message for observability.The cleanup is non-fatal — errors from missing directories or permission issues are gracefully caught and logged without blocking the backup.
Changes
src/infra/backup-create.ts: AddedbuildTempArchiveGlobPattern(),cleanupStaleTempArchives(), and theBACKUP_TEMP_STALE_AGE_MSconstant (30 s). Called increateBackupArchive()before creating the new temp archive (after the overwrite guard, in the non-dry-run path). Exported viatestApifor testing.src/infra/backup-create.test.ts: Added 7 tests covering:.tmpfiles matching the UUID patternReal behavior proof
Behavior addressed: Stale
.tmpfiles from interruptedopenclaw backup createruns accumulate on disk (up to 12.6GB reported), because the process has no mechanism to discover and remove leftovers before starting a new backup. Additionally, the v1 cleanup could collide with a concurrently running backup targeting the same output path.Real environment tested: Node v22.22.0, Linux 4.19.112-2.el8.x86_64, commit 39c4cf6.
tsxv4.22.3 used to import the actual production module fromsrc/infra/backup-create.js.Exact steps or command run after this patch:
cleanupStaleTempArchivesfromsrc/infra/backup-create.js(real production module, not copied logic).tmpfile (mtime = now, simulating a concurrentopenclaw backup createwriter).tmpfiles (mtime = 150 s ago, sizes 5MB + 3MB + 2MB, simulating 3 interrupted runs)cleanupStaleTempArchives({ outputPath, log })with default 30 s staleness thresholdEvidence after fix (terminal capture):
$ npx tsx -e 'import fs from "node:fs/promises"; import os from "node:os"; import path from "node:path";
const { testApi } = await import("./src/infra/backup-create.js");
const { cleanupStaleTempArchives } = testApi;
const dir = await fs.mkdtemp(path.join(os.tmpdir(), "oc-proof-real-import-"));
const outputPath = path.join(dir, "openclaw-backup.tar.gz");
const activeTmp = outputPath + ".a1b2c3d4-e5f6-7890-abcd-ef1234567890.tmp";
await fs.writeFile(activeTmp, "ACTIVE: backup being written by concurrent process", "utf8");
const stale1 = outputPath + ".b2c3d4e5-f6a7-8901-bcde-f12345678901.tmp";
const stale2 = outputPath + ".c3d4e5f6-a7b8-9012-cdef-123456789012.tmp";
const stale3 = outputPath + ".d4e5f6a7-b8c9-0123-defa-234567890123.tmp";
await fs.writeFile(stale1, Buffer.alloc(5_242_880, "x")); await fs.writeFile(stale2, Buffer.alloc(3_145_728, "y")); await fs.writeFile(stale3, Buffer.alloc(2_097_152, "z"));
const oldTime = new Date(Date.now() - 150_000);
await fs.utimes(stale1, oldTime, oldTime); await fs.utimes(stale2, oldTime, oldTime); await fs.utimes(stale3, oldTime, oldTime);
console.log("Before cleanup:");
for (const f of await fs.readdir(dir)) { const s = await fs.stat(path.join(dir, f)); console.log(" ", f, "|", (s.size/1048576).toFixed(1)+"MB", "|", Math.round((Date.now()-s.mtimeMs)/1000)+"s ago"); }
const logs = [];
await cleanupStaleTempArchives({ outputPath, log: (msg) => logs.push(msg) });
console.log("\nLog:", logs[0]);
console.log("\nAfter cleanup:");
for (const f of await fs.readdir(dir)) console.log(" ", f);
await fs.rm(dir, { recursive: true, force: true });'
Before cleanup:
openclaw-backup.tar.gz.a1b2c3d4-e5f6-7890-abcd-ef1234567890.tmp | 0.0MB | 0s ago
openclaw-backup.tar.gz.b2c3d4e5-f6a7-8901-bcde-f12345678901.tmp | 5.0MB | 150s ago
openclaw-backup.tar.gz.c3d4e5f6-a7b8-9012-cdef-123456789012.tmp | 3.0MB | 150s ago
openclaw-backup.tar.gz.d4e5f6a7-b8c9-0123-defa-234567890123.tmp | 2.0MB | 150s ago
Log: Backup cleaned up 3 stale temp archives (10.0MB) from a previous interrupted run. (1 active temp archive skipped)
After cleanup:
openclaw-backup.tar.gz.a1b2c3d4-e5f6-7890-abcd-ef1234567890.tmp
exit code: 0
Observed result after fix:
.tmpfiles (10.0MB total) were removed.tmpfile (concurrent writer) was preserved — staleness guard workssrc/infra/backup-create.jsmodule path, not copied inlineWhat was not tested:
openclaw backup createCLI command with a full OpenClaw state directory (requires building the entire project and having real agent state)