Repository navigation
Commit f5037ae
authored
🤖 fix: claim non-host background spawn names with an exclusive marker create, not mkdir (coder#7502)
## Summary
On non-host runtimes (SSH, Coder, Docker, devcontainer), a background
spawn now claims its record directory with an exclusive create of an
empty `.spawn-claim` file inside it, not with `mkdir`. Before, two
backends that spawned the same name at the same time could both claim
one directory on a host whose `mkdir` is uutils coreutils 0.10.0 (Ubuntu
25.10 and 26.04). Then both processes wrote one `output.log`,
`meta.json` and `exit_code`.
## Background
`claimRuntimeSpawnDir` relied on `mkdir` (no `-p`) failing when the
directory exists. uutils coreutils mkdir before 0.11.0 can exit 0 for
both of two concurrent creates of one directory: uutils/coreutils#13970,
"mkdir: exits 0 despite EEXIST from the kernel under concurrent
invocation (breaks mkdir-as-mutex)", closed on 2026-08-28. This host
runs `mkdir (uutils coreutils) 0.10.0`. CI's unit jobs run on Ubuntu
22.04 with GNU coreutils, so CI does not see the bug.
This is the first failure of coder#7256. On origin/main b7b487e with bun
1.4.3, I ran `backgroundProcessManager.test.ts` 20 times, 6 at a time
(load 5.50 at start, 22.33 at end). 2 of 20 runs failed. One of them was
"gives concurrent same-name spawns from two backends distinct
directories on a non-host runtime" (Expected 2, Received 1, line 513).
With instrumentation, both claim scripts logged `ok .../race-job` 80 µs
apart.
## Implementation
The claim script now does three steps per candidate name:
1. If the name exists (`[ -e ]` or `[ -L ]`), the name is held. This
check comes first because a buggy `mkdir` can exit 0 for a directory
that exists.
2. `mkdir` makes the directory. A failed `mkdir` on a name that now
exists means another claimer was faster, so the name is held. Any other
failure exits 1, as before.
3. `(set -C; : > "$dir/.spawn-claim")` decides the claim. The noclobber
redirect opens the file with O_EXCL, so exactly one claimer creates it.
`strace` shows `openat(AT_FDCWD, "sx/m", O_WRONLY|O_CREAT|O_EXCL, 0666)`
for both dash and bash on this host.
`releaseRuntimeSpawnDir` (the undo of the claim for a spawn that a seal
refuses) now removes the marker before its `rmdir`. Without that,
`rmdir` fails on the marker, and the leftover directory holds the name
and fails the remote crash-orphan probe closed. The existing test
"removes the claimed record directory when a seal refuses a non-host
spawn" covers this.
My search of the remote runtime scripts for `(if|until|while|!|&&|\|\|)
*mkdir [^-]`, `mkdir [^-p]` and `mkdir (-m N |--)` (without `-p`) found
no other mkdir claim.
## Validation
1. The new test "gives two backends distinct directories when the
runtime's mkdir exits 0 for an existing directory" puts a fake `mkdir`
first on PATH for the claim exec. The fake waits until both claims reach
it, then runs `mkdir -p`, so it always exits 0. On main the test failed
3 of 3 runs (Expected 2, Received 1). With the fix it passed 10 of 10
runs. The existing concurrency test is unchanged.
2. A shell stress test on this host (uutils mkdir 0.10.0, dash): 1000
pairs of concurrent claims of one fresh name. The old script gave 204 of
1000 double claims. The new script gave 0 of 1000, with no errors.
3. `make test` on this branch: 28293 pass, 0 fail.
4. Tripwire `bun scripts/perf/loopTestBaseline.ts --runs 3 --base
origin/main`: `loop-tests: 0 trips` (compare-75d7a84e48-049762441e-r3,
host load 12.27 at start and 4.83 at end).
5. `make mutate-check` exits 3 (baseline failure, which does not block).
The runner drops `backgroundProcessManager.test.ts` because one run
takes about 255 s, over its 120 s `BASELINE_TIMEOUT`. Without that file,
8 mutants survive in `claimRuntimeSpawnDir`. All 8 are on lines this PR
does not change: the two asserts and the error-message text.
6. On main 4049cf0 plus this PR plus the test fixes of coder#7513,
`backgroundProcessManager.test.ts` failed 0 of 20 runs, 6 at a time
(load 12.41 at start and 8.57 at end). On origin/main b7b487e it
failed 2 of 20.
## Upgrade and downgrade
I checked the mix with the old and new claim scripts in a shell on this
host:
1. An old Xum that claims a name a new Xum made: its `mkdir` fails, so
the name is held. An old Xum reads and lists the record as before: the
crash-orphan probe and the output reads look only at `meta.json`,
`exit_code` and `output.log`, so the marker is inert. The failed-spawn
cleanup in `spawnProcess` removes the whole directory with `rm -rf`,
marker included. Xum does not prune remote records, in old or new
versions.
2. A new Xum that claims a name an old Xum made: the `[ -e ]` check
holds the name.
3. An old and a new backend that claim one name at the same time: with
GNU mkdir, `mkdir` picks the winner as before. With uutils mkdir 0.10.0
both can still win, which is the same as two old backends today.
## Risks
Low. The change is limited to the non-host claim and its undo. A remote
host whose shell lacks `set -C` would fail the claim with exit 1, so the
spawn aborts with an error instead of sharing a directory. `set -C` is
POSIX.
Refs coder#7256
---
_Generated with `xum` • Model: `anthropic:claude-opus-5-5` • Thinking:
`high` • Cost: `$76.41`_
<!-- mux-attribution: model=anthropic:claude-opus-5-5 thinking=high
costs=76.41 -->1 parent fbbce35 commit f5037ae
3 files changed
Lines changed: 88 additions & 13 deletions
File tree
- src/node/services
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
157 | 157 | | |
158 | 158 | | |
159 | 159 | | |
160 | | - | |
161 | | - | |
| 160 | + | |
| 161 | + | |
162 | 162 | | |
163 | 163 | | |
164 | 164 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
516 | 516 | | |
517 | 517 | | |
518 | 518 | | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
| 523 | + | |
| 524 | + | |
| 525 | + | |
| 526 | + | |
| 527 | + | |
| 528 | + | |
| 529 | + | |
| 530 | + | |
| 531 | + | |
| 532 | + | |
| 533 | + | |
| 534 | + | |
| 535 | + | |
| 536 | + | |
| 537 | + | |
| 538 | + | |
| 539 | + | |
| 540 | + | |
| 541 | + | |
| 542 | + | |
| 543 | + | |
| 544 | + | |
| 545 | + | |
| 546 | + | |
| 547 | + | |
| 548 | + | |
| 549 | + | |
| 550 | + | |
| 551 | + | |
| 552 | + | |
| 553 | + | |
| 554 | + | |
| 555 | + | |
| 556 | + | |
| 557 | + | |
| 558 | + | |
| 559 | + | |
| 560 | + | |
| 561 | + | |
| 562 | + | |
| 563 | + | |
| 564 | + | |
| 565 | + | |
| 566 | + | |
| 567 | + | |
| 568 | + | |
| 569 | + | |
| 570 | + | |
| 571 | + | |
| 572 | + | |
| 573 | + | |
| 574 | + | |
| 575 | + | |
| 576 | + | |
| 577 | + | |
519 | 578 | | |
520 | 579 | | |
521 | 580 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
46 | 46 | | |
47 | 47 | | |
48 | 48 | | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
49 | 52 | | |
50 | 53 | | |
51 | 54 | | |
| |||
2301 | 2304 | | |
2302 | 2305 | | |
2303 | 2306 | | |
2304 | | - | |
| 2307 | + | |
2305 | 2308 | | |
2306 | 2309 | | |
2307 | 2310 | | |
| |||
3441 | 3444 | | |
3442 | 3445 | | |
3443 | 3446 | | |
3444 | | - | |
3445 | | - | |
3446 | | - | |
| 3447 | + | |
| 3448 | + | |
| 3449 | + | |
| 3450 | + | |
| 3451 | + | |
| 3452 | + | |
3447 | 3453 | | |
3448 | 3454 | | |
3449 | 3455 | | |
| |||
3470 | 3476 | | |
3471 | 3477 | | |
3472 | 3478 | | |
3473 | | - | |
3474 | | - | |
| 3479 | + | |
| 3480 | + | |
| 3481 | + | |
| 3482 | + | |
| 3483 | + | |
| 3484 | + | |
| 3485 | + | |
3475 | 3486 | | |
3476 | 3487 | | |
3477 | 3488 | | |
3478 | | - | |
3479 | | - | |
| 3489 | + | |
| 3490 | + | |
| 3491 | + | |
| 3492 | + | |
3480 | 3493 | | |
3481 | 3494 | | |
3482 | 3495 | | |
| |||
3508 | 3521 | | |
3509 | 3522 | | |
3510 | 3523 | | |
3511 | | - | |
3512 | | - | |
| 3524 | + | |
| 3525 | + | |
| 3526 | + | |
3513 | 3527 | | |
3514 | 3528 | | |
3515 | 3529 | | |
| |||
3520 | 3534 | | |
3521 | 3535 | | |
3522 | 3536 | | |
3523 | | - | |
| 3537 | + | |
| 3538 | + | |
| 3539 | + | |
3524 | 3540 | | |
3525 | 3541 | | |
3526 | 3542 | | |
| |||
0 commit comments