Skip to content

Commit f5037ae

Browse files
authored
🤖 fix: claim non-host background spawn names with an exclusive marker create, not mkdir (coder#7502)
## Summary On non-host runtimes (SSH, Coder, Docker, devcontainer), a background spawn now claims its record directory with an exclusive create of an empty `.spawn-claim` file inside it, not with `mkdir`. Before, two backends that spawned the same name at the same time could both claim one directory on a host whose `mkdir` is uutils coreutils 0.10.0 (Ubuntu 25.10 and 26.04). Then both processes wrote one `output.log`, `meta.json` and `exit_code`. ## Background `claimRuntimeSpawnDir` relied on `mkdir` (no `-p`) failing when the directory exists. uutils coreutils mkdir before 0.11.0 can exit 0 for both of two concurrent creates of one directory: uutils/coreutils#13970, "mkdir: exits 0 despite EEXIST from the kernel under concurrent invocation (breaks mkdir-as-mutex)", closed on 2026-08-28. This host runs `mkdir (uutils coreutils) 0.10.0`. CI's unit jobs run on Ubuntu 22.04 with GNU coreutils, so CI does not see the bug. This is the first failure of coder#7256. On origin/main b7b487e with bun 1.4.3, I ran `backgroundProcessManager.test.ts` 20 times, 6 at a time (load 5.50 at start, 22.33 at end). 2 of 20 runs failed. One of them was "gives concurrent same-name spawns from two backends distinct directories on a non-host runtime" (Expected 2, Received 1, line 513). With instrumentation, both claim scripts logged `ok .../race-job` 80 µs apart. ## Implementation The claim script now does three steps per candidate name: 1. If the name exists (`[ -e ]` or `[ -L ]`), the name is held. This check comes first because a buggy `mkdir` can exit 0 for a directory that exists. 2. `mkdir` makes the directory. A failed `mkdir` on a name that now exists means another claimer was faster, so the name is held. Any other failure exits 1, as before. 3. `(set -C; : > "$dir/.spawn-claim")` decides the claim. The noclobber redirect opens the file with O_EXCL, so exactly one claimer creates it. `strace` shows `openat(AT_FDCWD, "sx/m", O_WRONLY|O_CREAT|O_EXCL, 0666)` for both dash and bash on this host. `releaseRuntimeSpawnDir` (the undo of the claim for a spawn that a seal refuses) now removes the marker before its `rmdir`. Without that, `rmdir` fails on the marker, and the leftover directory holds the name and fails the remote crash-orphan probe closed. The existing test "removes the claimed record directory when a seal refuses a non-host spawn" covers this. My search of the remote runtime scripts for `(if|until|while|!|&&|\|\|) *mkdir [^-]`, `mkdir [^-p]` and `mkdir (-m N |--)` (without `-p`) found no other mkdir claim. ## Validation 1. The new test "gives two backends distinct directories when the runtime's mkdir exits 0 for an existing directory" puts a fake `mkdir` first on PATH for the claim exec. The fake waits until both claims reach it, then runs `mkdir -p`, so it always exits 0. On main the test failed 3 of 3 runs (Expected 2, Received 1). With the fix it passed 10 of 10 runs. The existing concurrency test is unchanged. 2. A shell stress test on this host (uutils mkdir 0.10.0, dash): 1000 pairs of concurrent claims of one fresh name. The old script gave 204 of 1000 double claims. The new script gave 0 of 1000, with no errors. 3. `make test` on this branch: 28293 pass, 0 fail. 4. Tripwire `bun scripts/perf/loopTestBaseline.ts --runs 3 --base origin/main`: `loop-tests: 0 trips` (compare-75d7a84e48-049762441e-r3, host load 12.27 at start and 4.83 at end). 5. `make mutate-check` exits 3 (baseline failure, which does not block). The runner drops `backgroundProcessManager.test.ts` because one run takes about 255 s, over its 120 s `BASELINE_TIMEOUT`. Without that file, 8 mutants survive in `claimRuntimeSpawnDir`. All 8 are on lines this PR does not change: the two asserts and the error-message text. 6. On main 4049cf0 plus this PR plus the test fixes of coder#7513, `backgroundProcessManager.test.ts` failed 0 of 20 runs, 6 at a time (load 12.41 at start and 8.57 at end). On origin/main b7b487e it failed 2 of 20. ## Upgrade and downgrade I checked the mix with the old and new claim scripts in a shell on this host: 1. An old Xum that claims a name a new Xum made: its `mkdir` fails, so the name is held. An old Xum reads and lists the record as before: the crash-orphan probe and the output reads look only at `meta.json`, `exit_code` and `output.log`, so the marker is inert. The failed-spawn cleanup in `spawnProcess` removes the whole directory with `rm -rf`, marker included. Xum does not prune remote records, in old or new versions. 2. A new Xum that claims a name an old Xum made: the `[ -e ]` check holds the name. 3. An old and a new backend that claim one name at the same time: with GNU mkdir, `mkdir` picks the winner as before. With uutils mkdir 0.10.0 both can still win, which is the same as two old backends today. ## Risks Low. The change is limited to the non-host claim and its undo. A remote host whose shell lacks `set -C` would fail the claim with exit 1, so the spawn aborts with an error instead of sharing a directory. `set -C` is POSIX. Refs coder#7256 --- _Generated with `xum` • Model: `anthropic:claude-opus-5-5` • Thinking: `high` • Cost: `$76.41`_ <!-- mux-attribution: model=anthropic:claude-opus-5-5 thinking=high costs=76.41 -->
1 parent fbbce35 commit f5037ae

3 files changed

Lines changed: 88 additions & 13 deletions

File tree

‎src/node/services/backgroundProcessExecutor.ts‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -157,8 +157,8 @@ export interface SpawnOptions {
157157
/** Host-namespace paths to translate before injecting as environment variables. */
158158
pathEnv?: Record<string, string>;
159159
/**
160-
* The caller created the record directory for this spawn (BackgroundProcessManager's atomic
161-
* mkdir claim on non-host runtimes, #4889), so a failure before the spawn may remove it.
160+
* The caller created the record directory for this spawn (BackgroundProcessManager's
161+
* exclusive claim on non-host runtimes, #4889), so a failure before the spawn may remove it.
162162
*/
163163
recordDirClaimed?: boolean;
164164
}

‎src/node/services/backgroundProcessManager.test.ts‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -516,6 +516,65 @@ describe("BackgroundProcessManager", () => {
516516
}
517517
});
518518

519+
it("gives two backends distinct directories when the runtime's mkdir exits 0 for an existing directory", async () => {
520+
// uutils coreutils mkdir before 0.11.0 (Ubuntu 25.10, 26.04) can exit 0 for both of two
521+
// concurrent creates of one directory (uutils/coreutils#13970, #7256). This fake mkdir
522+
// always behaves that way: it waits until both claims reached it, then runs `mkdir -p`.
523+
// So mkdir cannot pick the winner, and only the exclusive marker create can.
524+
const realMkdir = Bun.which("mkdir");
525+
if (realMkdir == null) throw new Error("the test needs a mkdir on PATH");
526+
const fakeBin = await fs.mkdtemp(path.join(os.tmpdir(), "bgpm-fake-mkdir-"));
527+
const arrivals = path.join(fakeBin, "arrivals");
528+
await fs.mkdir(arrivals);
529+
await fs.writeFile(
530+
path.join(fakeBin, "mkdir"),
531+
[
532+
"#!/bin/sh",
533+
`case "$1" in -*) exec ${realMkdir} "$@" ;; esac`,
534+
`: > ${arrivals}/$$`,
535+
"i=0",
536+
`while [ "$(ls ${arrivals} | wc -l)" -lt 2 ] && [ "$i" -lt 100 ]; do sleep 0.05; i=$((i + 1)); done`,
537+
`exec ${realMkdir} -p "$@"`,
538+
].join("\n"),
539+
{ mode: 0o755 }
540+
);
541+
const base = new LocalRuntime(process.cwd());
542+
const remote = new Proxy({} as Runtime, {
543+
get(_target, prop) {
544+
if (prop === "exec") {
545+
return (command: string, opts: never) =>
546+
base.exec(
547+
command.includes("__MUX_SPAWN_NAME_CLAIMED__")
548+
? `PATH=${fakeBin}:$PATH\n${command}`
549+
: command,
550+
opts
551+
);
552+
}
553+
const value = (base as unknown as Record<PropertyKey, unknown>)[prop];
554+
return typeof value === "function"
555+
? (value as (...args: unknown[]) => unknown).bind(base)
556+
: value;
557+
},
558+
});
559+
const other = new BackgroundProcessManager(bgOutputDir);
560+
try {
561+
const results = await Promise.all(
562+
[manager, other].map((m) =>
563+
m.spawn(remote, testWorkspaceId, "sleep 5", {
564+
cwd: process.cwd(),
565+
displayName: "uutils-race-job",
566+
})
567+
)
568+
);
569+
const dirs = results.map((r) => (r.success ? r.outputDir : r.error));
570+
expect(results.every((r) => r.success)).toBe(true);
571+
expect(new Set(dirs).size).toBe(2);
572+
} finally {
573+
await other.cleanup(testWorkspaceId);
574+
await fs.rm(fakeBin, { recursive: true, force: true });
575+
}
576+
});
577+
519578
it("removes the claimed record directory when a seal refuses a non-host spawn", async () => {
520579
// A cleanup, archive or removal that seals the workspace while the remote claim runs
521580
// refuses the spawn after its mkdir; the empty claimed directory must not stay behind.

‎src/node/services/backgroundProcessManager.ts‎

Lines changed: 27 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,9 @@ const SPAWN_NAME_LOCK_TIMEOUT_MS = 30_000;
4646
// Candidate names one non-host claim exec tries in order (#5485): a name used in k earlier
4747
// sessions costs ceil((k + 1) / batch) exec round-trips instead of k + 1.
4848
const RUNTIME_SPAWN_NAME_CLAIM_BATCH = 8;
49+
// Empty file whose exclusive create inside a non-host record directory decides the name claim
50+
// (claimRuntimeSpawnDir). Record readers only look for meta.json and exit_code, so it is inert.
51+
const RUNTIME_SPAWN_CLAIM_MARKER_FILENAME = ".spawn-claim";
4952
// Upper bound on a fail-closed cleanup() (archive, removal) waiting for pending spawns and
5053
// migrations, shared by both of its drains (#5477). Each wait is normally bounded by its own
5154
// steps (the 30 s name lock, 10 s claim execs), but a hung runtime probe or spawnProcess call
@@ -2301,7 +2304,7 @@ export class BackgroundProcessManager extends EventEmitter<BackgroundProcessMana
23012304
// reusing a settled one may clobber a record the other backend still tracks (see
23022305
// recordDirIsFree). Host-local records are probed on the local filesystem; all other
23032306
// layouts (SSH/Coder, Docker, devcontainer) live in the runtime's exec namespace and are
2304-
// claimed through the runtime with an atomic mkdir instead (claimRuntimeSpawnDir).
2307+
// claimed through the runtime with an exclusive marker create instead (claimRuntimeSpawnDir).
23052308
// Cross-process claim (#4873): two backends on one XUM_ROOT (desktop + `xum server`) have
23062309
// separate in-memory reservations but share the host records root, so both could probe a
23072310
// name as free and spawn into one directory. Host-local spawns therefore hold a
@@ -3441,9 +3444,12 @@ export class BackgroundProcessManager extends EventEmitter<BackgroundProcessMana
34413444
* records are NOT host-local (SSH/Coder, Docker, devcontainer, multi-project — see
34423445
* spawnRecordsAreHostLocal): the record layout lives in the runtime's exec namespace, so the
34433446
* name is claimed through the runtime. No host file lock can serialise backends there, so
3444-
* the claim is a plain `mkdir` (no -p) of the record directory: it is atomic on the target
3445-
* filesystem, so of two backends spawning the same name (or one spawning after the other)
3446-
* exactly one creates the directory, and the other moves on to a suffixed name (#4889).
3447+
* the claim is an exclusive create of an empty marker file in the record directory: the
3448+
* shell's noclobber redirect (`set -C`) opens it with O_EXCL, so of two backends spawning
3449+
* the same name (or one spawning after the other) exactly one creates the marker, and the
3450+
* other moves on to a suffixed name (#4889). `mkdir` only makes the directory and never
3451+
* decides the claim: uutils coreutils mkdir before 0.11.0 (Ubuntu 25.10 and 26.04) can exit 0
3452+
* for both of two concurrent creates of one directory (uutils/coreutils#13970, #7256).
34473453
*
34483454
* An existing directory is never reused, not even a settled one with an exit marker: nothing
34493455
* on the runtime says which backend spawned it, another backend on this Xum root may still
@@ -3470,13 +3476,20 @@ export class BackgroundProcessManager extends EventEmitter<BackgroundProcessMana
34703476
try {
34713477
const tempDir = await runtime.tempDir();
34723478
const workspaceDir = `${tempDir}/${BG_OUTPUT_SUBDIR}/${workspaceId}`;
3473-
// `[ -L ]` too: a dangling symlink planted at the name makes mkdir fail but `[ -e ]` false.
3474-
// Any other mkdir failure exits 1, so the caller aborts instead of skipping the name.
3479+
// A name that exists before the claim is held, whatever made it: the check comes first
3480+
// because a buggy mkdir can exit 0 for a directory that exists. `[ -L ]` too: a dangling
3481+
// symlink planted at the name makes mkdir fail but `[ -e ]` false. A directory that
3482+
// appeared between the check and the mkdir is held by whoever created its marker.
3483+
// Any other mkdir or marker failure exits 1, so the caller aborts instead of skipping
3484+
// the name. A backend without the marker (older Xum) still claims with mkdir alone.
3485+
const marker = `"$1"/${RUNTIME_SPAWN_CLAIM_MARKER_FILENAME}`;
34753486
const script = [
34763487
`mkdir -p ${quotePathForShell(workspaceDir)} || exit 1`,
34773488
`claim() {`,
3478-
` if mkdir "$1" 2>/dev/null; then echo "__MUX_SPAWN_NAME_CLAIMED__:$2:"; exit 0; fi`,
3479-
` [ -e "$1" ] || [ -L "$1" ] || exit 1`,
3489+
` if [ -e "$1" ] || [ -L "$1" ]; then return 0; fi`,
3490+
` if ! mkdir "$1" 2>/dev/null; then [ -e "$1" ] || [ -L "$1" ] || exit 1; return 0; fi`,
3491+
` if (set -C; : > ${marker}) 2>/dev/null; then echo "__MUX_SPAWN_NAME_CLAIMED__:$2:"; exit 0; fi`,
3492+
` [ -e ${marker} ] || [ -L ${marker} ] || exit 1`,
34803493
`}`,
34813494
...candidates.map(
34823495
(candidate, index) =>
@@ -3508,8 +3521,9 @@ export class BackgroundProcessManager extends EventEmitter<BackgroundProcessMana
35083521
}
35093522

35103523
/**
3511-
* Undo claimRuntimeSpawnDir for a spawn that is refused before anything ran. `rmdir` removes
3512-
* only an empty directory, so it cannot delete output another process wrote. Best-effort: a
3524+
* Undo claimRuntimeSpawnDir for a spawn that is refused before anything ran: remove the claim
3525+
* marker, then `rmdir`, which removes only an empty directory, so it cannot delete output
3526+
* another process wrote. Best-effort: a
35133527
* leftover empty directory only costs a suffixed name and fails the remote orphan probe closed.
35143528
*/
35153529
private async releaseRuntimeSpawnDir(
@@ -3520,7 +3534,9 @@ export class BackgroundProcessManager extends EventEmitter<BackgroundProcessMana
35203534
try {
35213535
const tempDir = await runtime.tempDir();
35223536
const processDir = `${tempDir}/${BG_OUTPUT_SUBDIR}/${workspaceId}/${processId}`;
3523-
await execBuffered(runtime, `rmdir ${quotePathForShell(processDir)}`, {
3537+
const marker = `${processDir}/${RUNTIME_SPAWN_CLAIM_MARKER_FILENAME}`;
3538+
const script = `rm -f -- ${quotePathForShell(marker)}; rmdir -- ${quotePathForShell(processDir)}`;
3539+
await execBuffered(runtime, script, {
35243540
cwd: "/tmp",
35253541
timeout: 10,
35263542
});

0 commit comments

Comments
 (0)