Skip to content

fix(backup): clean up stale .tmp archives from interrupted runs before creating new backup#91663

Closed
chengzhichao-xydt wants to merge 2 commits into
openclaw:mainfrom
chengzhichao-xydt:fix/50442-backup-tmp-cleanup
Closed

fix(backup): clean up stale .tmp archives from interrupted runs before creating new backup#91663
chengzhichao-xydt wants to merge 2 commits into
openclaw:mainfrom
chengzhichao-xydt:fix/50442-backup-tmp-cleanup

Conversation

@chengzhichao-xydt

@chengzhichao-xydt chengzhichao-xydt commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

Summary

When openclaw backup create is interrupted (timeout, SIGKILL, process crash), the temporary .tar.gz.<uuid>.tmp file created by buildTempArchivePath() is left on disk. On successive runs, each interrupted backup leaves a new .tmp file, accumulating and consuming significant disk space (12.6GB across 6 files reported in #50442).

This PR adds cleanupStaleTempArchives() that scans the output directory for files matching the backup temp pattern (<basename>.<uuid>.tmp) and removes stale ones before creating a new archive.

Key safety property: Only .tmp files whose mtime is older than a 30 s staleness threshold (BACKUP_TEMP_STALE_AGE_MS) are removed. This protects against concurrently running backup processes that target the same output path — their actively-written temp archive will be preserved, and the count of skipped active archives is reported in the log message for observability.

The cleanup is non-fatal — errors from missing directories or permission issues are gracefully caught and logged without blocking the backup.

Changes

  • src/infra/backup-create.ts: Added buildTempArchiveGlobPattern(), cleanupStaleTempArchives(), and the BACKUP_TEMP_STALE_AGE_MS constant (30 s). Called in createBackupArchive() before creating the new temp archive (after the overwrite guard, in the non-dry-run path). Exported via testApi for testing.
  • src/infra/backup-create.test.ts: Added 7 tests covering:
    • Removal of stale .tmp files matching the UUID pattern
    • Preservation of non-tmp files and the final archive
    • Graceful handling of non-existent directories
    • Scoping to only the target archive (not differently-named archives in the same directory)
    • Human-readable size reporting in log output
    • Active temp archive protection: files within the staleness threshold are skipped and reported
    • All-stale cleanup when no files are recent

Real behavior proof

Behavior addressed: Stale .tmp files from interrupted openclaw backup create runs accumulate on disk (up to 12.6GB reported), because the process has no mechanism to discover and remove leftovers before starting a new backup. Additionally, the v1 cleanup could collide with a concurrently running backup targeting the same output path.

Real environment tested: Node v22.22.0, Linux 4.19.112-2.el8.x86_64, commit 39c4cf6. tsx v4.22.3 used to import the actual production module from src/infra/backup-create.js.

Exact steps or command run after this patch:

  1. Import the actual cleanupStaleTempArchives from src/infra/backup-create.js (real production module, not copied logic)
  2. Create 1 active .tmp file (mtime = now, simulating a concurrent openclaw backup create writer)
  3. Create 3 stale .tmp files (mtime = 150 s ago, sizes 5MB + 3MB + 2MB, simulating 3 interrupted runs)
  4. Invoke cleanupStaleTempArchives({ outputPath, log }) with default 30 s staleness threshold
  5. Verify active file preserved, all 3 stale files removed

Evidence after fix (terminal capture):

$ npx tsx -e 'import fs from "node:fs/promises"; import os from "node:os"; import path from "node:path";
const { testApi } = await import("./src/infra/backup-create.js");
const { cleanupStaleTempArchives } = testApi;
const dir = await fs.mkdtemp(path.join(os.tmpdir(), "oc-proof-real-import-"));
const outputPath = path.join(dir, "openclaw-backup.tar.gz");
const activeTmp = outputPath + ".a1b2c3d4-e5f6-7890-abcd-ef1234567890.tmp";
await fs.writeFile(activeTmp, "ACTIVE: backup being written by concurrent process", "utf8");
const stale1 = outputPath + ".b2c3d4e5-f6a7-8901-bcde-f12345678901.tmp";
const stale2 = outputPath + ".c3d4e5f6-a7b8-9012-cdef-123456789012.tmp";
const stale3 = outputPath + ".d4e5f6a7-b8c9-0123-defa-234567890123.tmp";
await fs.writeFile(stale1, Buffer.alloc(5_242_880, "x")); await fs.writeFile(stale2, Buffer.alloc(3_145_728, "y")); await fs.writeFile(stale3, Buffer.alloc(2_097_152, "z"));
const oldTime = new Date(Date.now() - 150_000);
await fs.utimes(stale1, oldTime, oldTime); await fs.utimes(stale2, oldTime, oldTime); await fs.utimes(stale3, oldTime, oldTime);
console.log("Before cleanup:");
for (const f of await fs.readdir(dir)) { const s = await fs.stat(path.join(dir, f)); console.log(" ", f, "|", (s.size/1048576).toFixed(1)+"MB", "|", Math.round((Date.now()-s.mtimeMs)/1000)+"s ago"); }
const logs = [];
await cleanupStaleTempArchives({ outputPath, log: (msg) => logs.push(msg) });
console.log("\nLog:", logs[0]);
console.log("\nAfter cleanup:");
for (const f of await fs.readdir(dir)) console.log(" ", f);
await fs.rm(dir, { recursive: true, force: true });'

Before cleanup:
openclaw-backup.tar.gz.a1b2c3d4-e5f6-7890-abcd-ef1234567890.tmp | 0.0MB | 0s ago
openclaw-backup.tar.gz.b2c3d4e5-f6a7-8901-bcde-f12345678901.tmp | 5.0MB | 150s ago
openclaw-backup.tar.gz.c3d4e5f6-a7b8-9012-cdef-123456789012.tmp | 3.0MB | 150s ago
openclaw-backup.tar.gz.d4e5f6a7-b8c9-0123-defa-234567890123.tmp | 2.0MB | 150s ago

Log: Backup cleaned up 3 stale temp archives (10.0MB) from a previous interrupted run. (1 active temp archive skipped)

After cleanup:
openclaw-backup.tar.gz.a1b2c3d4-e5f6-7890-abcd-ef1234567890.tmp

exit code: 0

Observed result after fix:

  • 3 stale .tmp files (10.0MB total) were removed
  • 1 active .tmp file (concurrent writer) was preserved — staleness guard works
  • Log correctly reports "3 stale temp archives" + "(1 active temp archive skipped)"
  • The function was imported from the real src/infra/backup-create.js module path, not copied inline

What was not tested:

  • End-to-end openclaw backup create CLI command with a full OpenClaw state directory (requires building the entire project and having real agent state)
  • Signal-handler cleanup for the interrupted process itself (SIGTERM/SIGKILL still cannot run JavaScript cleanup — this PR mitigates on the next run)
  • Behavior when the backup output path is on a network filesystem where mtime precision may differ

…e creating new backup

When backup create is interrupted (timeout, SIGKILL, etc.), the temporary
.tar.gz.<uuid>.tmp file is left behind. Over successive runs these accumulate
and can consume significant disk space (12.6GB reported in issue openclaw#50442).

Add cleanupStaleTempArchives() that scans the output directory for files
matching the backup temp pattern (<basename>.<uuid>.tmp) and removes them
before creating a new archive. The cleanup is non-fatal — errors from
missing directories or permission issues are logged and execution continues.

Fixes openclaw#50442
@openclaw-barnacle openclaw-barnacle Bot added size: S proof: supplied External PR includes structured after-fix real behavior proof. labels Jun 9, 2026
@clawsweeper

clawsweeper Bot commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

Thanks for the context here. I swept through the related work, and this is now duplicate or superseded.

Close as superseded: the branch contains a useful archive-temp cleanup idea, but it is conflicting, has an active-writer safety defect, and the newer clean replacement PR covers the same archive leak plus the stale staging-directory half with owner-marker protection and sufficient proof.

Root-cause cluster
Relationship: superseded
Canonical: #95600
Summary: This PR is an older archive-temp-only candidate for the interrupted-backup cleanup bug; the clean replacement PR is the viable canonical landing path because it covers both stale archive temps and staging directories with owner-marker protection.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Canonical path: Use #95600 as the canonical landing path for guarded two-artifact backup temp cleanup, then close the related backup-temp reports after merge.

So I’m closing this here and keeping the remaining discussion on #95600.

Review details

Best possible solution:

Use #95600 as the canonical landing path for guarded two-artifact backup temp cleanup, then close the related backup-temp reports after merge.

Do we have a high-confidence way to reproduce the issue?

Yes, at source level. Current main creates the reported output-adjacent temp archive and staging directory and removes them only during an in-process finally, so hard termination can strand them.

Is this the best way to solve the issue?

No for this PR as the landing path. The safer maintainable solution is the newer guarded startup sweep in #95600, which covers both artifact classes and avoids an age-only active-writer heuristic.

Security review:

Security review cleared: The diff touches backup source and tests only and introduces no dependency, workflow, credential, or supply-chain change; destructive deletion is tracked as availability merge risk.

AGENTS.md: found and applied where relevant.

What I checked:

  • Current main still has the backup temp lifecycle: Current main still creates the active archive as ${outputPath}.${randomUUID()}.tmp, creates an openclaw-backup- staging directory, and removes both only in the normal finally path, so the underlying bug remains real but is not fixed on main. (src/infra/backup-create.ts:246, 4fc504d321b6)
  • This PR's active-writer guard is only mtime-based: The PR skips files newer than the 30 second threshold but deletes any older matching archive temp, so a slow, stalled, or filesystem-delayed concurrent backup can still lose the path it later needs to publish. (src/infra/backup-create.ts:297, 39c4cf6d368e)
  • Replacement PR implements the broader guarded sweep: The replacement PR adds sweepStaleBackupArtifacts, sweeps both staging directories and output-adjacent archive temps, writes owner sidecars for active artifacts, and calls the sweep before creating a new backup. (src/infra/backup-create.ts:704, 300d72b58f96)
  • Replacement PR has focused owner-marker coverage: The replacement tests cover stale staging-dir removal, stale archive-temp removal, recent artifact preservation, unrelated artifact preservation, live-owner preservation, and dead-owner cleanup. (src/infra/backup-create.test.ts:1213, 300d72b58f96)
  • Live PR metadata supports supersession: This PR is open but conflicting and waiting on author, while the replacement PR is open, mergeable, clean, proof-sufficient, and ready for maintainer review. (300d72b58f96)
  • Canonical broader issue: The broader issue reports both orphaned output-adjacent <outputPath>.<uuid>.tmp archives and stale openclaw-backup- staging directories after hard kills, matching the replacement PR's fuller scope.

Likely related people:

  • shichangs: The merged local backup CLI PR added openclaw backup create, backup docs, and the original atomic temp archive flow that this PR changes around. (role: introduced backup CLI behavior; confidence: high; commits: 0ecfd37b4465; files: src/commands/backup.ts, src/commands/backup.atomic.test.ts, docs/cli/backup.md)
  • gumadeiras: PR metadata shows this person merged the original backup CLI PR, and commit history shows they extracted backup creation into src/infra/backup-create.ts. (role: backup hardening and refactor contributor; confidence: high; commits: 0ecfd37b4465, 3ba64916599b; files: src/commands/backup.ts, src/infra/backup-create.ts, src/commands/backup-shared.ts)
  • zhfnini-rgb: They authored the open clean replacement PR that covers this branch's archive cleanup plus the broader staging-directory cleanup with owner-marker protection. (role: likely follow-up owner; confidence: medium; commits: 3067b25e5753, 300d72b58f96; files: src/infra/backup-create.ts, src/infra/backup-create.test.ts)

Codex review notes: model internal, reasoning high; reviewed against 4fc504d321b6.

@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P2 Normal backlog priority with limited blast radius. merge-risk: 🚨 availability 🚨 May cause crashes, hangs, restart loops, stalls, or process outages. labels Jun 9, 2026
…ives

Only remove .tmp files whose mtime is older than a 30 s staleness
threshold. This protects against concurrently running backup processes
that target the same output path — their actively-written temp archive
will be preserved.

Also reports how many active temp archives were skipped in the cleanup
log message for observability.
@chengzhichao-xydt

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Jun 9, 2026
@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. and removed rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Jun 21, 2026
@chengzhichao-xydt

Copy link
Copy Markdown
Contributor Author

Closing as superseded by #95600, which covers the same archive-temp cleanup plus stale staging-directory cleanup with owner-marker protection. Thanks for the review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-risk: 🚨 availability 🚨 May cause crashes, hangs, restart loops, stalls, or process outages. P2 Normal backlog priority with limited blast radius. proof: sufficient ClawSweeper judged the real behavior proof convincing. proof: supplied External PR includes structured after-fix real behavior proof. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. size: M status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant