Skip to content

Commit 87d60fd

Browse files
authored
fix(PG-956): the run-slot tests take a lock as the Mac does on CI's ubuntu-26.04 (#1076)
## TL;DR - 🧪 The run-slot lock tests stop failing now and then on GitHub's Linux runner. One such failure held up a ship. - 🔍 The cause was the runner's own `mkdir` command, which can tell two programs that each made the same folder. The tests now use the runner's other `mkdir`, which tells only one, as the Mac's does. ## Summary ```diff tests/scripts/ops/run-slot-fake.ts, machine() slowChild("python3", bin) + if (GNU_MKDIR) // Bun.which("gnumkdir"): ubuntu-26.04 only + bin/mkdir = exec gnumkdir "$@" ``` - GitHub's `ubuntu-latest` is ubuntu-26.04, whose `/usr/bin/mkdir` is uutils coreutils 0.10.0 (`rust-coreutils` 0.10.0-1ubuntu2~26.04.1). Its `create_dir` checks `path.exists()`, then maps a failed `mkdir(2)` to success with `Err(_) if path.is_dir() => Ok(())`, even without `-p`. So when two processes race, both exit 0. - run-slot.sh takes every lock with a plain `mkdir`: the gauges refresh lock, the run slots and the model slots. On that runner, two looks can each hold the gauges lock and both run `top`: "eight looks with no reading yet run top once" got 6. - The image keeps GNU's mkdir as `gnumkdir`, and each fake machine's bin now runs that one wherever it exists. On the Mac and on GNU images nothing changes. - `.agents/testing.md`, Traps: one new trap, the racing `mkdir` and how a test avoids it. **Decisions:** - **The fix is in the test, not in run-slot.sh.** run-slot.sh runs for real only on the Mac, whose BSD `mkdir` is atomic: 60 rounds of the eight-look case there each ran one `top`. Only CI runs the script on Linux. - **The dead-lock takeover case stays out of this PR.** It is a separate race, not yet reproduced alone (Evidence, #959's row), now #1077, so this PR keeps to the two "run top once" cases #956 names. - **The reported lead is refuted.** The lead was that a reading stamped with the look's start time goes stale after a refresh slower than 10 s. The red run took 1.6 s, and six one-second tops in 1.6 s means they ran side by side. A stale stamp cannot do that, while two lock holders can. - **Nothing is posted upstream.** uutils fixed this in 0.11.0 (uutils/coreutils#13970, uutils/coreutils#13986), and Ubuntu's request to bring that fix to 26.04 (LP #2171333) already has a patch waiting for a sponsor. A comment would add no new facts. Once 26.04 takes that patch, the shim is harmless and can go. ## Evidence - **Cause, in an `ubuntu:26.04` container:** 8 racing `mkdir L`, 200 rounds each: | mkdir | rounds with more than one winner | | --- | --- | | `/usr/bin/mkdir` (uutils 0.10.0) | 33 of 200, and 38 of 200 on a second run | | `gnumkdir` (GNU 9.7) | 0 of 200 | - **No repro without uutils:** the eight-look case ran exactly one `top` in 60 of 60 rounds on the Mac (BSD mkdir) and 80 of 80 on `oven/bun:debian` (GNU mkdir, bash 5.2). - **Before / after in `ubuntu:26.04`** (`--cpus 2`, Linux bun 1.4.4): | run | canary's fake | this head's fake | | --- | --- | --- | | probe: 60 rounds of 8 looks with no reading, rounds that ran `top` more than once | 10 of 60 (2 or 3 tops) | 0 of 60 | | `run-slot-gauges-lock.test.ts`, 20 runs, the two "run top once" cases | failed in 6 of 20: the stale-reading case (line 46) in runs 1, 14 and 15, the no-reading case (line 52) in runs 5, 7 and 13 | failed in 0 of 20 | - **At this head on the Mac:** 20 runs in a row of `bun run test:file tests/scripts/ops/run-slot-gauges-lock.test.ts`: 20 of 20 passed, each 10 pass, 0 fail. `bun run check-types:tests`: no errors. `bun run test:skills`: all 13 passed. - **Sightings on CI:** - canary at d216f23, Suite (8): "eight looks with no reading yet run top once", expected 1, received 6, in 1631 ms. - #1063 at 1af2e05, Suite (7) (job 127580442051), ubuntu-26.04: "eight looks at a stale reading run top once", expected 1, received 2, in 1526 ms. The same case fails 3 of 20 on canary's fake above and 0 of 20 at this head. - #959 at 3006e0a, attempt 1, Suite (5) (job 127180040487): line 124 of the same file, "eight looks at a dead lock together each read the stale or the new reading", expected true, received false, in 1627 ms. **Not this race, and not fixed here.** That case still failed 1 of 20 runs at this head in `ubuntu:26.04` (run 12, line 125), with GNU's `mkdir` in the fake. A probe of the case alone found 0 bad rounds in 60 with GNU's `mkdir` and 0 in 60 with all 104 `gnu*` tools on PATH, so no tool swap is shown to fix it. It is #1077. - **doc-sync:** the change touches the test fake alone. `git grep -n -iE "mkdir"` over `.agents`, `scripts/README.md`, `scripts/ops/README.md`, `scripts/hooks/README.md`, `docs` and the core skills finds no line that calls `mkdir` atomic. The new trap in `.agents/testing.md` is the one place that says it is not on ubuntu-26.04. ## Merge Danger **Door:** two-way A revert brings back the uutils `mkdir` in the fake, and the flake with it. **Blast Radius:** small Tests only: every `run-slot*.test.ts` machine on a runner that has `gnumkdir`. run-slot.sh and every other script are unchanged. ## Links - Closes #956 - uutils/coreutils#13970 (the report) and uutils/coreutils#13986 (the fix, in 0.11.0) - #1077: the dead-lock takeover race, left out of this PR - Ubuntu: https://bugs.launchpad.net/ubuntu/+source/rust-coreutils/+bug/2171333 🤖 Generated with [Claude Code](https://claude.com/claude-code)
1 parent 25501e1 commit 87d60fd

2 files changed

Lines changed: 9 additions & 0 deletions

File tree

‎.agents/testing.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,8 @@ Keep one case on the real watcher, appending again until a change lands inside a
6666
A speed test bounds work rather than milliseconds, since a bound in milliseconds fails when every core is busy: it caps the steps `stepsOf()` (`tests/scripts/hooks/pace.ts`) reads from `stepsTaken()` in `scripts/hooks/shell.ts`, which counts every `tick()`.
6767
For a native step no tick counts (a regex, a string built whole), it bounds the step's CPU time over a same-length `echo` with `overBaseline()` (`tests/scripts/hooks/pace.ts`), since a busy machine stretches a long call's wall time far more than a short baseline's (x23 to x590 at load 44, measured).
6868
A timer or backstop case asserts the refusal under a test timeout tied to the constant it guards (the hung Docker case in `tests/scripts/hooks/no-mac-web-build.test.ts`).
69+
GitHub's ubuntu-26.04 runner runs uutils 0.10.0 as `mkdir`, which reports success when another process made the folder between its check and its own mkdir, so a shell lock taken with `mkdir` can have two holders there: 8 racing `mkdir`s gave more than one winner in 38 of 200 rounds, and GNU's, which the image keeps as `gnumkdir`, in none (measured).
70+
A test that races such a lock puts `gnumkdir` first on its child's PATH as `mkdir` (`GNU_MKDIR` in `tests/scripts/ops/run-slot-fake.ts`), so it checks the lock as the Mac's atomic `mkdir` runs it.
6971
A test that lists items keeps to the items it made, filtering the list by their ids (the "an update starts unread" test in `tests/api/hono/src/lib/dashboard/inbox.test.ts`), since a file's tests share one database and another test's items can land in its list: with the clock 30 days ahead, one test's list took four older updates another test had left.
7072
`Bun.which(name)` reads the PATH the process started with, so code under test that looks a tool up that way runs the real tool wherever one is installed and misses a stub the test prepends to `process.env.PATH`: look it up with `Bun.which(name, { PATH: process.env.PATH ?? "" })` (`packages/scripts/src/ops-ext-sync/sync.ts`), or take the lookup as a dependency the test passes in.
7173

‎tests/scripts/ops/run-slot-fake.ts‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,9 @@ export const stop = async (p: Subprocess) => {
3030
await p.exited
3131
}
3232

33+
// Ubuntu 26.04's default mkdir, uutils 0.10.0, reports success when another process made the folder between its own check and its mkdir (fixed in uutils 0.11.0), so two looks could each take one of run-slot.sh's mkdir locks; the image keeps GNU's as gnumkdir, which fails there as the Mac's mkdir does, so a machine's fake bin runs that one where it exists
34+
const GNU_MKDIR = Bun.which("gnumkdir")
35+
3336
export const dirs: string[] = []
3437
afterAll(() => {
3538
for (const dir of dirs) rmSync(dir, { force: true, recursive: true })
@@ -133,6 +136,10 @@ esac
133136
chmodSync(join(bin, "bun"), 0o755)
134137
// SLOW_CHILD puts a slowed python3 in the fake bin folder, which PATH reads first
135138
slowChild("python3", bin)
139+
if (GNU_MKDIR) {
140+
writeFileSync(join(bin, "mkdir"), `#!/bin/bash\nexec ${GNU_MKDIR} "$@"\n`)
141+
chmodSync(join(bin, "mkdir"), 0o755)
142+
}
136143
const env = (extra: Record<string, string> = {}) => ({
137144
CLAUDE_CONFIG_DIR: config,
138145
PLAYGROUND_DATA_DIR: join(config, "playground"),

0 commit comments

Comments
 (0)