Repository navigation
Commit 87d60fd
authored
fix(PG-956): the run-slot tests take a lock as the Mac does on CI's ubuntu-26.04 (#1076)
## TL;DR
- 🧪 The run-slot lock tests stop failing now and then on GitHub's Linux runner. One such failure held up a ship.
- 🔍 The cause was the runner's own `mkdir` command, which can tell two programs that each made the same folder. The tests now use the runner's other `mkdir`, which tells only one, as the Mac's does.
## Summary
```diff
tests/scripts/ops/run-slot-fake.ts, machine()
slowChild("python3", bin)
+ if (GNU_MKDIR) // Bun.which("gnumkdir"): ubuntu-26.04 only
+ bin/mkdir = exec gnumkdir "$@"
```
- GitHub's `ubuntu-latest` is ubuntu-26.04, whose `/usr/bin/mkdir` is uutils coreutils 0.10.0 (`rust-coreutils` 0.10.0-1ubuntu2~26.04.1). Its `create_dir` checks `path.exists()`, then maps a failed `mkdir(2)` to success with `Err(_) if path.is_dir() => Ok(())`, even without `-p`. So when two processes race, both exit 0.
- run-slot.sh takes every lock with a plain `mkdir`: the gauges refresh lock, the run slots and the model slots. On that runner, two looks can each hold the gauges lock and both run `top`: "eight looks with no reading yet run top once" got 6.
- The image keeps GNU's mkdir as `gnumkdir`, and each fake machine's bin now runs that one wherever it exists. On the Mac and on GNU images nothing changes.
- `.agents/testing.md`, Traps: one new trap, the racing `mkdir` and how a test avoids it.
**Decisions:**
- **The fix is in the test, not in run-slot.sh.** run-slot.sh runs for real only on the Mac, whose BSD `mkdir` is atomic: 60 rounds of the eight-look case there each ran one `top`. Only CI runs the script on Linux.
- **The dead-lock takeover case stays out of this PR.** It is a separate race, not yet reproduced alone (Evidence, #959's row), now #1077, so this PR keeps to the two "run top once" cases #956 names.
- **The reported lead is refuted.** The lead was that a reading stamped with the look's start time goes stale after a refresh slower than 10 s. The red run took 1.6 s, and six one-second tops in 1.6 s means they ran side by side. A stale stamp cannot do that, while two lock holders can.
- **Nothing is posted upstream.** uutils fixed this in 0.11.0 (uutils/coreutils#13970, uutils/coreutils#13986), and Ubuntu's request to bring that fix to 26.04 (LP #2171333) already has a patch waiting for a sponsor. A comment would add no new facts. Once 26.04 takes that patch, the shim is harmless and can go.
## Evidence
- **Cause, in an `ubuntu:26.04` container:** 8 racing `mkdir L`, 200 rounds each:
| mkdir | rounds with more than one winner |
| --- | --- |
| `/usr/bin/mkdir` (uutils 0.10.0) | 33 of 200, and 38 of 200 on a second run |
| `gnumkdir` (GNU 9.7) | 0 of 200 |
- **No repro without uutils:** the eight-look case ran exactly one `top` in 60 of 60 rounds on the Mac (BSD mkdir) and 80 of 80 on `oven/bun:debian` (GNU mkdir, bash 5.2).
- **Before / after in `ubuntu:26.04`** (`--cpus 2`, Linux bun 1.4.4):
| run | canary's fake | this head's fake |
| --- | --- | --- |
| probe: 60 rounds of 8 looks with no reading, rounds that ran `top` more than once | 10 of 60 (2 or 3 tops) | 0 of 60 |
| `run-slot-gauges-lock.test.ts`, 20 runs, the two "run top once" cases | failed in 6 of 20: the stale-reading case (line 46) in runs 1, 14 and 15, the no-reading case (line 52) in runs 5, 7 and 13 | failed in 0 of 20 |
- **At this head on the Mac:** 20 runs in a row of `bun run test:file tests/scripts/ops/run-slot-gauges-lock.test.ts`: 20 of 20 passed, each 10 pass, 0 fail. `bun run check-types:tests`: no errors. `bun run test:skills`: all 13 passed.
- **Sightings on CI:**
- canary at d216f23, Suite (8): "eight looks with no reading yet run top once", expected 1, received 6, in 1631 ms.
- #1063 at 1af2e05, Suite (7) (job 127580442051), ubuntu-26.04: "eight looks at a stale reading run top once", expected 1, received 2, in 1526 ms. The same case fails 3 of 20 on canary's fake above and 0 of 20 at this head.
- #959 at 3006e0a, attempt 1, Suite (5) (job 127180040487): line 124 of the same file, "eight looks at a dead lock together each read the stale or the new reading", expected true, received false, in 1627 ms.
**Not this race, and not fixed here.** That case still failed 1 of 20 runs at this head in `ubuntu:26.04` (run 12, line 125), with GNU's `mkdir` in the fake. A probe of the case alone found 0 bad rounds in 60 with GNU's `mkdir` and 0 in 60 with all 104 `gnu*` tools on PATH, so no tool swap is shown to fix it. It is #1077.
- **doc-sync:** the change touches the test fake alone. `git grep -n -iE "mkdir"` over `.agents`, `scripts/README.md`, `scripts/ops/README.md`, `scripts/hooks/README.md`, `docs` and the core skills finds no line that calls `mkdir` atomic. The new trap in `.agents/testing.md` is the one place that says it is not on ubuntu-26.04.
## Merge Danger
**Door:** two-way
A revert brings back the uutils `mkdir` in the fake, and the flake with it.
**Blast Radius:** small
Tests only: every `run-slot*.test.ts` machine on a runner that has `gnumkdir`. run-slot.sh and every other script are unchanged.
## Links
- Closes #956
- uutils/coreutils#13970 (the report) and uutils/coreutils#13986 (the fix, in 0.11.0)
- #1077: the dead-lock takeover race, left out of this PR
- Ubuntu: https://bugs.launchpad.net/ubuntu/+source/rust-coreutils/+bug/2171333
🤖 Generated with [Claude Code](https://claude.com/claude-code)1 parent 25501e1 commit 87d60fd
2 files changed
Lines changed: 9 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
66 | 66 | | |
67 | 67 | | |
68 | 68 | | |
| 69 | + | |
| 70 | + | |
69 | 71 | | |
70 | 72 | | |
71 | 73 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
30 | 30 | | |
31 | 31 | | |
32 | 32 | | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
33 | 36 | | |
34 | 37 | | |
35 | 38 | | |
| |||
133 | 136 | | |
134 | 137 | | |
135 | 138 | | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
136 | 143 | | |
137 | 144 | | |
138 | 145 | | |
| |||
0 commit comments