Summary

• Retry backup tar writes on a fresh deterministic temp archive path after EOF-class live-write races.
• Publish the temp archive path from the successful attempt instead of always using the first .tmp path.
• Clean all deterministic retry temp archive paths after success or failure so stale .retry-N files do not remain.

Change Type

• Bug fix
• Feature
• Refactor
• Docs
• Security
• Chore

Scope

• Gateway
• Skills
• Auth
• Memory
• Integrations
• API
• UI/UX
• CI
• Backup

Real behavior proof

Behavior addressed: Backup archive retries no longer reuse a temp archive path that failed cleanup with EBUSY; each retry writes to a fresh .retry-N path and returns the successful temp path.

Environment tested: Linux workspace with Node running TypeScript sources via node --import tsx on branch fix/101382-backup-temp-retry.

Steps run after the patch: Called the actual testApi.writeTarArchiveWithRetry production helper from src/infra/backup-create.ts, forced the first two cleanup attempts to throw EBUSY, and let the third tar attempt succeed.

Evidence after fix:

node --import tsx --no-warnings -e '
import fs from "node:fs/promises";
import { testApi } from "./src/infra/backup-create.js";

const tempArchivePath = "/tmp/openclaw-proof-backup.tar.gz.tmp";
const attemptedPaths = [];
const cleanupCalls = [];
const logLines = [];
let attempt = 0;
const realRm = fs.rm.bind(fs);
fs.rm = async (targetPath, options) => {
  const key = String(targetPath);
  cleanupCalls.push(key);
  if (key === tempArchivePath || key === `${tempArchivePath}.retry-2`) {
    throw Object.assign(new Error("resource busy"), { code: "EBUSY" });
  }
  return realRm(targetPath, options);
};
try {
  const completedPath = await testApi.writeTarArchiveWithRetry({
    tempArchivePath,
    runTar: async (attemptTempArchivePath) => {
      attempt += 1;
      attemptedPaths.push(attemptTempArchivePath);
      if (attempt < 3) {
        throw Object.assign(new Error("did not encounter expected EOF"), {
          path: "/tmp/openclaw-state/sessions/live.jsonl",
        });
      }
    },
    log: (message) => logLines.push(message),
    sleepMs: async () => undefined,
  });
  console.log(JSON.stringify({ attemptedPaths, completedPath, cleanupCalls, logLines }, null, 2));
} finally {
  fs.rm = realRm;
}
'

Output:

{
  "attemptedPaths": [
    "/tmp/openclaw-proof-backup.tar.gz.tmp",
    "/tmp/openclaw-proof-backup.tar.gz.tmp.retry-2",
    "/tmp/openclaw-proof-backup.tar.gz.tmp.retry-3"
  ],
  "completedPath": "/tmp/openclaw-proof-backup.tar.gz.tmp.retry-3",
  "cleanupCalls": [
    "/tmp/openclaw-proof-backup.tar.gz.tmp",
    "/tmp/openclaw-proof-backup.tar.gz.tmp.retry-2",
    "/tmp/openclaw-proof-backup.tar.gz.tmp",
    "/tmp/openclaw-proof-backup.tar.gz.tmp.retry-2"
  ],
  "logLines": [
    "Backup archiver could not remove temp archive /tmp/openclaw-proof-backup.tar.gz.tmp between retries: EBUSY. Continuing.",
    "Backup archiver hit a live-write race on /tmp/openclaw-state/sessions/live.jsonl (attempt 1/3); retrying in 10s.",
    "Backup archiver could not remove temp archive /tmp/openclaw-proof-backup.tar.gz.tmp.retry-2 between retries: EBUSY. Continuing.",
    "Backup archiver hit a live-write race on /tmp/openclaw-state/sessions/live.jsonl (attempt 2/3); retrying in 20s."
  ]
}

Observed result: The production retry helper attempted .tmp, then .tmp.retry-2, then .tmp.retry-3, and returned the successful .tmp.retry-3 path while logging the two EBUSY cleanup failures.

Not tested: Native Windows file-lock behavior was not run in this Linux environment; the proof exercises the same production retry and cleanup path with real EBUSY errors injected at the fs.rm boundary.

Root Cause

• The retry loop always reused the original temp archive path even when cleanup of the failed archive raised EBUSY.
• createBackupArchive always published the original temp path, so the retry helper could not safely switch to a fresh temp path.

Regression Test Plan

• Added coverage in src/infra/backup-create.test.ts for EBUSY cleanup followed by a successful retry on .retry-2.
• Added coverage in src/infra/backup-create.test.ts for cleanup of retry temp paths when a later attempt fails.
• Added command-level coverage in src/commands/backup.atomic.test.ts proving intermediate retry temp archives are removed after cleanup races across multiple retries.

Verification

• node scripts/run-vitest.mjs src/infra/backup-create.test.ts src/commands/backup.atomic.test.ts passed.
• git diff --check passed.
• pnpm check:test-types passed.

Impact Assessment

• Scope: backup archive creation only; the change is concentrated in the shared tar retry helper and its publishing call site.
• Downstream: existing backup callers still receive the same final archive path; only internal temp archive paths vary between retry attempts.
• Edge cases: handles cleanup failure, later non-EOF failure, final success, and final cleanup of deterministic retry temp paths.
• Hard-coded values: no new environment assumptions; retry suffixes derive from the existing temp archive path and existing retry count.
• Existing fixes: competing PRs did not isolate retry attempts from a locked temp archive path.
• Import paths: no new imports were added.

User-visible / Behavior Changes

Backups that hit a live-write EOF race and cannot remove the failed temp archive can still complete by writing the next attempt to a fresh retry temp path.

Security Impact

• New permissions: No
• Secrets handling changed: No
• New network calls: No

Closes #101382