Skip to content

Commit 0f4b7c4

Browse files
committed
ci(unit): run npm single-threaded so node 24's exit cannot deadlock it
The merge queue's unit (node 24) job hung in npm ci after the install had finished and ejected #3176 (run 43399170758) and #3182 (43420654281); it had done so four times in September. The cause is a deadlock in node 24's process.exit(): npm calls it, node joins V8's worker threads, and a worker parked mid-Maglev-compile waits for a garbage collection only the joining main thread could run (nodejs/node#64274, fix proposed in #66171, unreleased). gdb stacks of hung npm ci runs in the sandbox show exactly that; on the runners a hung run had the same three threads in futex waits, and npm's debug log ended in its exit handler. npm's own process now runs with --single-threaded, which turns off V8's background tasks, so no compile can park. The flag is refused in NODE_OPTIONS, so npm-cli.js is run directly and npm's children run as before. On the runners: plain npm ci hung 4 of 60; the flagged form hung 0 of 145 (60 in the same run, 43407363473, and 85 in 43421579091), at no measurable cost (median 57 s plain, 56 s flagged). In the sandbox, with stdio on pipes: 4 of 34 before, 0 of 20 after. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01DPMMjGGdttTJWpWrHaaNL2
1 parent fa0d387 commit 0f4b7c4

3 files changed

Lines changed: 62 additions & 3 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 19 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -48,8 +48,16 @@ permissions:
4848
# WHAT THE 15m CAP ON `unit` DOES AND DOES NOT BUY — say it plainly, because the
4949
# obvious reading is wrong: it catches THREE of those four. The 664s wedge finishes
5050
# inside the cap and would still tie up a runner for 11 minutes. A cap is a ceiling
51-
# on the worst case, not a detector for the failure mode; the wedge itself is
52-
# unfixed and untracked here.
51+
# on the worst case, not a detector for the failure mode.
52+
#
53+
# THE WEDGE'S CAUSE, found 2026-11-13: a deadlock in node 24's process.exit(), not
54+
# in npm and not in this repo. npm prints its whole summary, calls process.exit(),
55+
# and the exit joins V8's worker threads while one of them is parked mid-compile
56+
# (Maglev), waiting for a garbage collection that only the joining main thread
57+
# could run (nodejs/node#64274; fix proposed in #66171, unreleased). The `unit`
58+
# job's install now runs npm itself with `--single-threaded`, which removes the
59+
# background compile; see that step, and engineering/gotchas/ci.md "npm ci on
60+
# node 24 prints everything, then never exits" for the stacks and the counts.
5361
#
5462
# THE DISTRIBUTION, and how to reproduce it. Every job across the last 100 completed
5563
# `ci.yml` runs (`pull_request` + `merge_group`), sampled 2026-09-02T12:17Z:
@@ -362,7 +370,15 @@ jobs:
362370
uses: ./.github/actions/apt-cached
363371
with:
364372
packages: zsh
365-
- run: npm ci
373+
# npm's own process runs with V8's background tasks off (`--single-threaded`). On node 24 a plain `npm ci` can
374+
# finish its work and then never exit: npm calls process.exit(), the exit joins V8's worker threads, and a worker
375+
# parked mid-compile waits for a garbage collection only the joining main thread could run (nodejs/node#64274,
376+
# fix proposed in #66171, in no release yet). With no background compile there is nothing to park. The flag
377+
# cannot go in NODE_OPTIONS (node refuses it there), so npm-cli.js is run directly; npm's children, the
378+
# `prepare` build among them, run as before. The cause, the stacks and the counts are in
379+
# engineering/gotchas/ci.md, "npm ci on node 24 prints everything, then never exits".
380+
- name: npm ci (npm itself single-threaded, against the node 24 exit deadlock)
381+
run: node --single-threaded "$(dirname "$(readlink -f "$(command -v npm)")")/npm-cli.js" ci
366382
timeout-minutes: 10 # an npm ci that never exits (run 40590390608 hung 14 min) fails here, not at the job cap
367383
# dist/ and the docs-site bundles are BUILT, not committed (.gitignore), so a
368384
# checkout does not have them and everything below that loads the engine
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
- **CI:** the merge queue's `unit (node 24)` job no longer hangs in `npm ci` after the install has finished. The cause was a deadlock in node 24's `process.exit()` (nodejs/node#64274): a background compile waited for a garbage collection that the exiting main thread never ran. The job now runs npm itself with V8's background tasks off (`--single-threaded`).

‎engineering/gotchas/ci.md‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -915,3 +915,45 @@ this file is the detail. Entry shape and the rule for adding one are in the [gui
915915
prints the same refusal.
916916
- **Fix:** an `actions/setup-node` step with `node-version-file: .nvmrc` before `npm ci` in the container job, which puts the repo's Node first
917917
on PATH. Any new container job needs the same step.
918+
919+
## npm ci on node 24 prints everything, then never exits
920+
921+
- **Symptom:** `unit (node 24)` (merge queue only) is cancelled or times out in its `npm ci` step. The log carries the
922+
whole install, `prepare`'s `build OK`, `added N packages, and audited M`, the funding line, the audit summary and the
923+
`npm warn install-scripts` lines, then nothing until the step's 10-minute timeout (`The action 'Run npm ci' has timed out after 10 minutes`; since
924+
2026-11-13 the step is named `npm ci (npm itself single-threaded, …)`), and
925+
the cleanup lists one orphan, `npm ci`. Node 22 never does it. Seen in September (runs 33582248943, 33586223061,
926+
33519483686, 33593380463, all the same last line) and on 2026-11-13, when it ejected #3176 from the queue (run
927+
43399170758).
928+
- **Cause:** a deadlock in node 24's `process.exit()`, not in npm and not in this repo. npm's exit handler calls
929+
`process.exit()`; node then shuts its platform down and joins V8's worker threads while the isolate is still alive.
930+
If a worker is mid-compile in the background (Maglev) and runs out of room, it parks in
931+
`CollectionBarrier::AwaitCollectionBackground` waiting for a garbage collection only the main thread can run, and the
932+
main thread is the one blocked in the join. The `gdb` stacks of a hung `npm ci` (node 24.22.1, npm 11.19.1):
933+
- main thread: `uv_thread_join` ← `node::WorkerThreadsTaskRunner::Shutdown` ← `node::NodePlatform::Shutdown` ←
934+
`node::DefaultProcessExitHandlerInternal` ← `node::Environment::Exit`;
935+
- `V8Worker`: `CollectionBarrier::AwaitCollectionBackground` ← `HeapAllocator::AllocateRawWithLightRetrySlowPath` ←
936+
`MaglevCodeGenerator::BuildCodeObject` ← `MaglevCompiler::Compile`.
937+
938+
Upstream: nodejs/node#64274, fix proposed in nodejs/node#66171, in no release on 2026-11-13.
939+
- **How to reproduce:** loop `npm ci` on node 24 and treat a run past 200 s as hung. On the runners (ubuntu-24.04,
940+
node 24.22.2), the probe on scratch PR #3179 hung 4 of 60 plain runs with output redirected to a FILE (run
941+
43407363473). In the cloud sandbox (node 24.22.1) a file never hung in 9 runs, but stdout and stderr on PIPES
942+
(`npm ci 2> >(cat >e) | cat >o`) hung 4 of 34 (3 of 14, then 1 of 20 in a loop that alternated with the fix), so
943+
locally, use pipes. A hung process is in state `Sl` with wchan `futex_do_wait`, not `ep_poll`, so a JS-level handle
944+
dump never runs; attach `gdb -p <pid> -batch -ex "thread apply all bt"` for the stacks above (taken locally; on the
945+
runners the probe captured the threads' kernel futex stacks, and npm's debug log of a hung run ends `verbose exit 0`
946+
/ `info ok`, which npm writes from its `exit` handler, so process.exit() had been called).
947+
- **Fix:** the `unit` job runs npm's own process with V8's background tasks off:
948+
`node --single-threaded "$(dirname "$(readlink -f "$(command -v npm)")")/npm-cli.js" ci`. With no background compile
949+
there is nothing to park. Measured after: 0 of 20 locally (the alternating loop), 0 of 145 on the runners: 60 in the same run as the
950+
4 of 60 (the other arm), 85 more in run 43421579091. At the plain rate, 0 of 145 by chance is about 1 in 20,000. It costs nothing measurable: the 56 finished plain runs took 57 s at the median and 66 s at p90, the 60 flagged
951+
runs 56 s and 69 s. The flag cannot go in `NODE_OPTIONS` (node refuses `--single-threaded`, `--no-maglev` and
952+
`--no-concurrent-sparkplug` there), so it is passed on the command line of npm itself, and npm's children run as
953+
before. `--no-maglev --no-concurrent-sparkplug` is the narrower mitigation others report; one reproduction upstream
954+
still hung with it, so it was not chosen.
955+
- **Not a fix:** a longer timeout or a retry. The 10-minute step cap only bounds the cost.
956+
- **Any node 24 process that calls `process.exit()` is exposed,** not only npm. Nothing else in CI runs on node 24 except
957+
this job's later steps, which have shown no such hang; if one appears, the same stacks will say so.
958+
- **Removable when:** the fix in nodejs/node#66171 (or another for #64274) ships in the node 24 line CI uses; then the
959+
step goes back to plain `npm ci`.

0 commit comments

Comments
 (0)