Skip to content

Commit d2c5e5a

Browse files
committed
fix(ci): stop fetch-ant hanging in process.exit() on macOS (solardev-xyz#544)
Node 24's process.exit() joins V8's background compiler threads without disposing the isolate; a Maglev/Sparkplug job parked waiting for a GC at that moment never wakes, so the exit hangs at 0% CPU (nodejs/node#64274). fetch-ant.js called process.exit(0) right after 'All downloads complete.' and hung the macOS e2e-onboarding-identity leg in 6 of 1075 jobs since 2026-09-15; every other macOS download step, all of which exit naturally, hung 0 times in ~6,600 runs. - fetch-ant.js / fetch-arti.js: set process.exitCode and return instead of calling process.exit(), on success and failure. - scripts/ci/hang-watchdog.sh: run a command under a deadline; on expiry print the process table and a stack sample (macOS sample, Linux /proc stacks) of its tree, kill the tree and exit 124. - Every ant:download step: hang-watchdog at 300 s plus a 7-minute step timeout-minutes backstop, instead of a silent 20-minute job cancel. - Tests: no process.exit( in either script; fetch-ant.js run end to end against a stand-in release exits 0/1 on its own; the watchdog's pass-through, deadline, tree-kill and usage paths.
1 parent 60837dc commit d2c5e5a

8 files changed

Lines changed: 384 additions & 12 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1201,7 +1201,12 @@ jobs:
12011201

12021202
- name: Download Ant (antd) binary
12031203
# Version pinned in scripts/fetch-ant.js so CI tests exactly what releases ship.
1204-
run: npm run ant:download
1204+
# Normally ~5 s. hang-watchdog fails the step after 5 min with a
1205+
# process table and stack samples instead of a silent 20-minute
1206+
# job cancel (#544); timeout-minutes is the backstop.
1207+
timeout-minutes: 7
1208+
shell: bash
1209+
run: bash scripts/ci/hang-watchdog.sh 300 npm run ant:download
12051210
env:
12061211
GITHUB_TOKEN: ${{ github.token }}
12071212

@@ -1326,7 +1331,12 @@ jobs:
13261331

13271332
- name: Download Ant (antd) binary
13281333
# Version pinned in scripts/fetch-ant.js so CI tests exactly what releases ship.
1329-
run: npm run ant:download
1334+
# Normally ~5 s. hang-watchdog fails the step after 5 min with a
1335+
# process table and stack samples instead of a silent 20-minute
1336+
# job cancel (#544); timeout-minutes is the backstop.
1337+
timeout-minutes: 7
1338+
shell: bash
1339+
run: bash scripts/ci/hang-watchdog.sh 300 npm run ant:download
13301340
env:
13311341
GITHUB_TOKEN: ${{ github.token }}
13321342

‎.github/workflows/release.yml‎

Lines changed: 18 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -287,7 +287,12 @@ jobs:
287287
GITHUB_TOKEN: ${{ github.token }}
288288

289289
- name: Download Ant binaries
290-
run: npm run ant:download
290+
# Normally ~5 s. hang-watchdog fails the step after 5 min with a
291+
# process table and stack samples instead of a silent 20-minute
292+
# job cancel (#544); timeout-minutes is the backstop.
293+
timeout-minutes: 7
294+
shell: bash
295+
run: bash scripts/ci/hang-watchdog.sh 300 npm run ant:download
291296
env:
292297
GITHUB_TOKEN: ${{ github.token }}
293298

@@ -533,7 +538,12 @@ jobs:
533538
GITHUB_TOKEN: ${{ github.token }}
534539

535540
- name: Download Ant binaries
536-
run: npm run ant:download
541+
# Normally ~5 s. hang-watchdog fails the step after 5 min with a
542+
# process table and stack samples instead of a silent 20-minute
543+
# job cancel (#544); timeout-minutes is the backstop.
544+
timeout-minutes: 7
545+
shell: bash
546+
run: bash scripts/ci/hang-watchdog.sh 300 npm run ant:download
537547
env:
538548
GITHUB_TOKEN: ${{ github.token }}
539549

@@ -1130,7 +1140,12 @@ jobs:
11301140
GITHUB_TOKEN: ${{ github.token }}
11311141

11321142
- name: Download Ant binaries
1133-
run: npm run ant:download
1143+
# Normally ~5 s. hang-watchdog fails the step after 5 min with a
1144+
# process table and stack samples instead of a silent 20-minute
1145+
# job cancel (#544); timeout-minutes is the backstop.
1146+
timeout-minutes: 7
1147+
shell: bash
1148+
run: bash scripts/ci/hang-watchdog.sh 300 npm run ant:download
11341149
env:
11351150
GITHUB_TOKEN: ${{ github.token }}
11361151

‎.github/workflows/startup-smoke.yml‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -57,7 +57,12 @@ jobs:
5757
uses: ./.github/actions/install-playwright-deps
5858

5959
- name: Download Ant (antd) binary
60-
run: npm run ant:download
60+
# Normally ~5 s. hang-watchdog fails the step after 5 min with a
61+
# process table and stack samples instead of a silent 20-minute
62+
# job cancel (#544); timeout-minutes is the backstop.
63+
timeout-minutes: 7
64+
shell: bash
65+
run: bash scripts/ci/hang-watchdog.sh 300 npm run ant:download
6166
env:
6267
GITHUB_TOKEN: ${{ github.token }}
6368

‎scripts/ci/hang-watchdog.sh‎

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
#!/usr/bin/env bash
2+
#
3+
# Run one CI command under a wall-clock deadline, and when it blows the
4+
# deadline, record what is still alive before killing it (#544).
5+
#
6+
# scripts/ci/hang-watchdog.sh <seconds> <command> [args...]
7+
#
8+
# A step-level `timeout-minutes` already stops a hang, but it does so silently:
9+
# the log ends, then "The operation was canceled", and the runner reaps the
10+
# leftover process during job cleanup without saying what it was doing. That is
11+
# all #544 had to go on — `npm run ant:download` printed its last line, called
12+
# `process.exit(0)`, and sat for 19 minutes. This wrapper fails the step at the
13+
# deadline instead, after printing:
14+
#
15+
# - the process table (`ps`), so the log names what is still running —
16+
# npm, node, a tar/unzip child, something else holding the step's pipe;
17+
# - a stack sample of every process in the command's process tree: `sample`
18+
# on macOS (it ships with the OS), the kernel wait channel and stack on
19+
# Linux. A native hang (a thread join, a blocked syscall) is invisible
20+
# from JavaScript, so the stack is the only place it shows.
21+
#
22+
# Exit status: the command's own status, or 124 (as `timeout` uses) when the
23+
# deadline expired. The command runs in its own process group so the whole
24+
# tree, not just the top process, is signalled.
25+
#
26+
# Pair it with a step `timeout-minutes` a little above the deadline, as a
27+
# backstop for a watchdog that cannot itself run (e.g. the runner losing the
28+
# machine).
29+
30+
set -uo pipefail
31+
32+
if [ "$#" -lt 2 ] || ! [[ "$1" =~ ^[0-9]+$ ]]; then
33+
echo "usage: $0 <seconds> <command> [args...]" >&2
34+
exit 2
35+
fi
36+
37+
deadline="$1"
38+
shift
39+
40+
# `set -m` puts the background command in a process group of its own, whose id
41+
# is its pid, so `kill -- -<pid>` reaches every descendant.
42+
set -m
43+
"$@" &
44+
pid=$!
45+
set +m
46+
47+
# Every pid whose ancestry leads back to $1 (including $1 itself).
48+
descendants() {
49+
local root="$1"
50+
ps -axo pid=,ppid= 2>/dev/null | awk -v root="$root" '
51+
{ parent[$1] = $2; pids[NR] = $1 }
52+
END {
53+
for (i in pids) {
54+
p = pids[i]
55+
for (q = p; q != "" && q != 0 && q != 1; q = parent[q]) {
56+
if (q == root) { print p; break }
57+
}
58+
}
59+
}'
60+
}
61+
62+
diagnose() {
63+
echo "::group::hang-watchdog: process table"
64+
ps -axo pid,ppid,pgid,etime,stat,%cpu,command 2>/dev/null || ps -ef || ps
65+
echo "::endgroup::"
66+
local p
67+
for p in $(descendants "$pid"); do
68+
echo "::group::hang-watchdog: stack of pid $p ($(ps -o command= -p "$p" 2>/dev/null))"
69+
if command -v sample >/dev/null 2>&1; then
70+
# macOS: 2 s of samples, every thread, symbolicated. Node's release
71+
# binaries use the hardened runtime, which can refuse an unprivileged
72+
# sampler; hosted runners have passwordless sudo for the retry.
73+
{ sample "$p" 2 -mayDie 2>&1 || sudo -n sample "$p" 2 -mayDie 2>&1; } | sed -n '1,400p'
74+
elif [ -r "/proc/$p/status" ]; then
75+
grep -E '^(State|Threads):' "/proc/$p/status"
76+
for t in /proc/"$p"/task/*; do
77+
echo "thread ${t##*/}: wchan=$(cat "$t/wchan" 2>/dev/null) $(cat "$t/stat" 2>/dev/null | awk '{print "state=" $3}')"
78+
cat "$t/stack" 2>/dev/null | sed 's/^/ /'
79+
done
80+
else
81+
echo "(no stack sampler on this platform)"
82+
fi
83+
echo "::endgroup::"
84+
done
85+
}
86+
87+
start=$SECONDS
88+
while kill -0 "$pid" 2>/dev/null; do
89+
if [ $((SECONDS - start)) -ge "$deadline" ]; then
90+
echo "::error title=hang-watchdog::'$*' still running after ${deadline}s; dumping processes and killing it (see #544)"
91+
diagnose
92+
kill -TERM -- "-$pid" 2>/dev/null || kill -TERM "$pid" 2>/dev/null
93+
sleep 5
94+
kill -KILL -- "-$pid" 2>/dev/null || kill -KILL "$pid" 2>/dev/null
95+
wait "$pid" 2>/dev/null
96+
exit 124
97+
fi
98+
sleep 1
99+
done
100+
101+
wait "$pid"

‎scripts/ci/hang-watchdog.test.js‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
/**
2+
* scripts/ci/hang-watchdog.sh (#544): a command that finishes keeps its own
3+
* exit status; one that outlives the deadline is reported (process table plus
4+
* a stack of every process in its tree) and killed — the whole tree, not just
5+
* the top process — with status 124.
6+
*/
7+
8+
const { spawnSync } = require('child_process');
9+
const path = require('path');
10+
11+
const SCRIPT = path.join(__dirname, 'hang-watchdog.sh');
12+
const maybe = process.platform === 'win32' ? describe.skip : describe;
13+
14+
function watchdog(args, timeout = 30_000) {
15+
return spawnSync('bash', [SCRIPT, ...args], { encoding: 'utf8', timeout });
16+
}
17+
18+
function alive(pid) {
19+
try {
20+
process.kill(pid, 0);
21+
return true;
22+
} catch {
23+
return false;
24+
}
25+
}
26+
27+
maybe('hang-watchdog.sh', () => {
28+
test('passes a finished command through, exit status and output included', () => {
29+
const result = watchdog(['10', 'bash', '-c', 'echo done; exit 3']);
30+
expect(result.status).toBe(3);
31+
expect(result.stdout).toBe('done\n');
32+
});
33+
34+
test('a command past the deadline is diagnosed, killed with its children, and fails 124', () => {
35+
// The child records its own pid and its grandchild's, then hangs.
36+
const result = watchdog(['2', 'bash', '-c', 'sleep 300 & echo "pids $$ $!"; wait']);
37+
expect(result.error).toBeUndefined();
38+
expect(result.status).toBe(124);
39+
expect(result.stdout).toMatch(/::error title=hang-watchdog::.*still running after 2s/);
40+
expect(result.stdout).toMatch(/hang-watchdog: process table/);
41+
const [, parent, child] = result.stdout.match(/pids (\d+) (\d+)/);
42+
expect(result.stdout).toContain(`hang-watchdog: stack of pid ${parent} `);
43+
expect(result.stdout).toContain(`hang-watchdog: stack of pid ${child} `);
44+
expect(alive(Number(parent))).toBe(false);
45+
expect(alive(Number(child))).toBe(false);
46+
});
47+
48+
test('rejects a missing or non-numeric deadline', () => {
49+
expect(watchdog(['soon', 'true']).status).toBe(2);
50+
expect(watchdog(['10']).status).toBe(2);
51+
});
52+
});

‎scripts/fetch-ant.js‎

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -277,11 +277,19 @@ async function main() {
277277
}
278278

279279
console.log('All downloads complete.');
280-
process.exit(0);
281280
} catch (err) {
282281
console.error('Error:', err);
283-
process.exit(1);
282+
process.exitCode = 1;
284283
}
284+
// No process.exit() here, on either path (#544): Node 24's process.exit()
285+
// joins V8's background compiler threads without tearing the isolate down,
286+
// and a Maglev/Sparkplug job parked waiting for a main-thread GC at that
287+
// moment never wakes, so the exit hangs forever at 0% CPU
288+
// (nodejs/node#64274, fix pending in nodejs/node#66171). That hung this
289+
// step on macOS CI in ~0.5% of runs, after "All downloads complete." was
290+
// already printed. Returning lets the process end the normal way, which
291+
// disposes the isolate first; nothing here holds the event loop open (the
292+
// deadline timers are unref'd and keep-alive sockets do not ref the loop).
285293
}
286294

287295
if (require.main === module) {

‎scripts/fetch-arti.js‎

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -312,10 +312,12 @@ async function main() {
312312
'Arti has no clean prebuilt-binary distribution, so it is built from\n' +
313313
'crates.io. Install Rust (https://rustup.rs) and re-run, or set CARGO_BIN.\n'
314314
);
315-
process.exit(1);
315+
process.exitCode = 1;
316+
return;
316317
}
317318
if (!checkRustVersion(cargoVersionOutput)) {
318-
process.exit(1);
319+
process.exitCode = 1;
320+
return;
319321
}
320322

321323
const target = platformKey();
@@ -375,13 +377,16 @@ async function main() {
375377
}
376378
}
377379

378-
process.exit(ok ? 0 : 1);
380+
// Not process.exit(): on Node 24 it can hang forever joining V8's
381+
// background compiler threads — see the note at the end of
382+
// scripts/fetch-ant.js's main() (#544, nodejs/node#64274).
383+
process.exitCode = ok ? 0 : 1;
379384
}
380385

381386
if (require.main === module) {
382387
main().catch((err) => {
383388
console.error(`\nError: ${err.message}`);
384-
process.exit(1);
389+
process.exitCode = 1;
385390
});
386391
}
387392

0 commit comments

Comments
 (0)