Skip to content

Commit ed7942d

Browse files
authored
lane-broker 0.42.0: node processes run with --no-maglev --no-concurrent-sparkplug (Node 24 process.exit deadlock, BRAIN-577); ships the BRAIN-576 silence watchdog (#123)
lane-broker 0.42.0. lane-broker node processes run with `--no-maglev --no-concurrent-sparkplug` (BRAIN-577). This release also ships the BRAIN-576 silence watchdog (#122). Why: twice in an hour on the Mac's lane-broker gates, a supervisor wrote its result and then hung forever in `process.exit`. The main thread was in `NodePlatform::Shutdown` → `uv_thread_join` while a V8 worker sat in `MaglevConcurrentDispatcher::JobTask::Run` → `CollectionBarrier::AwaitCollectionBackground`. In production that means a finished lane that never returns. This is a Node 24/26 bug ([nodejs/node#64274](nodejs/node#64274); the fix [#66171](nodejs/node#66171) is unmerged), so no Node version fixes it yet. Another project measured 6/1019 hangs going to 0/1019 with both compilers off. What changed: - `src/node-exit-safe.js`: one flag list. `childExitSafeFlags()` is for spawned node children. They inherit `NODE_OPTIONS` but not our execArgv, so only an operator's `NODE_OPTIONS` choice suppresses a flag. - Supervisors (`run.js`), `lane reconcile` (`status.js`) and `remote-pipeline` (`remote-runner.js`) get the flags in argv. - The `lane` and `lane-store` CLIs disable the compilers at startup with `v8.setFlagsFromString`, as their first import. That covers code compiled after startup, but not node's bootstrap; the README notes this. - README: a "Node runtime flags" section with the upstream links. - Version 0.42.0. Verified: - `tests/node-exit-safe-flags.test.js`: a live supervisor's `ps` command line carries both flags; the operator override; a child gets both even when the parent was started with them. - 94/94 on the spawn-path files (implementer), 10/10 targeted (orchestrator). - Canaries: empty flag list (implementer) and the parent-execArgv filter reintroduced (orchestrator). Both failed by name and were restored. - Push gate, full suite: ℹ tests 2202 ℹ pass 2199 fail 0. No exit deadlock this run; the BRAIN-576 gate without this fix hit it once. Implemented-by: Sonnet 🤖 Generated with [Claude Code](https://claude.com/claude-code)
1 parent 00a0dc6 commit ed7942d

11 files changed

Lines changed: 100 additions & 6 deletions

‎README.md‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2068,6 +2068,17 @@ did not start. Bringing load back down (stopping stray sessions, `pkill`ing
20682068
runaway processes) is a human job; `lane status` shows the current load and
20692069
the queue so you know what's waiting and why.
20702070

2071+
## Node runtime flags (BRAIN-577)
2072+
2073+
Every lane-broker node process (the CLI, the supervisor, spawned `lane` helpers) runs with
2074+
`--no-maglev --no-concurrent-sparkplug`. Node 24 can deadlock in `process.exit()` while a concurrent
2075+
Maglev or Sparkplug compile job is in flight, which left finished supervisors hung forever
2076+
([nodejs/node#64274](https://github.com/nodejs/node/issues/64274); fix in
2077+
[#66171](https://github.com/nodejs/node/pull/66171), not yet released). Self-spawned processes get the
2078+
flags in argv; `bin/lane.js` and `bin/lane-store.js` set them at startup with `v8.setFlagsFromString`
2079+
(`src/node-exit-safe.js`), so only node's own bootstrap code is not covered. A `--maglev`/`--no-maglev` (or
2080+
sparkplug equivalent) in `execArgv` or `NODE_OPTIONS` is respected.
2081+
20712082
## State
20722083

20732084
`$LANE_BROKER_STATE` (default `~/.cache/lane-broker`): `leases/`, `queue/`,

‎bin/lane-store.js‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
#!/usr/bin/env node
2+
import '../src/node-exit-safe-boot.js';
23
import { storeCli } from '../src/store/cli.js';
34

45
storeCli(process.argv.slice(2)).then(

‎bin/lane.js‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,6 @@
11
#!/usr/bin/env node
2+
// First import on purpose (ESM hoists imports): BRAIN-577, see src/node-exit-safe.js.
3+
import '../src/node-exit-safe-boot.js';
24
import { runCommand } from '../src/run.js';
35
import { statusCommand } from '../src/status.js';
46
import { suggestCommand } from '../src/suggest.js';

‎package-lock.json‎

Lines changed: 2 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎package.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
{
22
"name": "@andrewpopov/lane-broker",
3-
"version": "0.41.2",
3+
"version": "0.42.0",
44
"description": "Machine-wide test-lane coordinator: serializes conflicting test lanes, caps concurrent heavy lanes, and refuses to start new work while the machine is overloaded.",
55
"author": "Andrew Popov",
66
"license": "MIT",

‎src/node-exit-safe-boot.js‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
import { disableConcurrentJit } from './node-exit-safe.js';
2+
3+
disableConcurrentJit();

‎src/node-exit-safe.js‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
// BRAIN-577: Node 24 can deadlock in process.exit() when a concurrent Maglev/Sparkplug compile job is
2+
// still running (nodejs/node#64274). Every lane-broker node process turns both compilers off.
3+
import v8 from 'node:v8';
4+
5+
const SWITCHES = ['maglev', 'concurrent-sparkplug'];
6+
7+
export const NODE_EXIT_SAFE_FLAGS = SWITCHES.map((name) => `--no-${name}`);
8+
9+
const isSet = (arg, name) => arg === `--${name}` || arg === `--no-${name}`;
10+
11+
/** The exit-safe flags the operator has not already chosen a value for, in `argv`/`NODE_OPTIONS`. */
12+
export function exitSafeFlagsFor(argv = process.execArgv, nodeOptions = process.env.NODE_OPTIONS ?? '') {
13+
const given = [...argv, ...nodeOptions.split(/\s+/)];
14+
return NODE_EXIT_SAFE_FLAGS.filter((flag, i) => !given.some((arg) => isSet(arg, SWITCHES[i])));
15+
}
16+
17+
/** For a node child this process spawns: a child inherits NODE_OPTIONS but never our execArgv, so only an
18+
* operator's NODE_OPTIONS choice counts (a parent started with --no-maglev must still pass it on). */
19+
export const childExitSafeFlags = () => exitSafeFlagsFor([], process.env.NODE_OPTIONS ?? '');
20+
21+
/** In-process form, for entrypoints whose shebang cannot carry flags. Only affects compilations that
22+
* start after this call (node's own bootstrap code is already compiled). */
23+
export function disableConcurrentJit() {
24+
for (const flag of exitSafeFlagsFor()) v8.setFlagsFromString(flag);
25+
}

‎src/remote-runner.js‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@ import fs from 'node:fs';
22
import os from 'node:os';
33
import path from 'node:path';
44
import crypto from 'node:crypto';
5+
import { childExitSafeFlags } from './node-exit-safe.js';
56
import { execFileSync } from 'node:child_process';
67
import { fileURLToPath } from 'node:url';
78
import { pipeline } from 'node:stream/promises';
@@ -564,7 +565,7 @@ export async function remoteExecCommand({ root = defaultRemoteRoot(), stdin = pr
564565
maxBytes: runnerCfg.remoteDepsCacheMaxBytes,
565566
},
566567
});
567-
pipelineCmd = [process.execPath, laneBinPath, 'remote-pipeline', ticketDir];
568+
pipelineCmd = [process.execPath, ...childExitSafeFlags(), laneBinPath, 'remote-pipeline', ticketDir];
568569
}
569570

570571
const remotePriority = remotePriorityFrom(header);

‎src/run.js‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@ import fs from 'node:fs';
22
import os from 'node:os';
33
import path from 'node:path';
44
import crypto from 'node:crypto';
5+
import { childExitSafeFlags } from './node-exit-safe.js';
56
import { spawn, execFileSync } from 'node:child_process';
67
import { fileURLToPath } from 'node:url';
78
import { ensureStateDirs, paths, stateHome, readJsonSafe, appendHistory, LockTimeoutError, MigrationInProgressError, assertNotMigrating, isUuid } from './state.js';
@@ -700,7 +701,7 @@ export async function runCommand({
700701
try {
701702
supervisor = spawnSupervisor(
702703
process.execPath,
703-
[supervisorPath],
704+
[...childExitSafeFlags(), supervisorPath],
704705
{
705706
detached: true,
706707
// A detached caller must never hold the supervisor's pipes -- `ID=$(lane

‎src/status.js‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
import fs from 'node:fs';
22
import path from 'node:path';
3+
import { childExitSafeFlags } from './node-exit-safe.js';
34
import { spawn } from 'node:child_process';
45
import { fileURLToPath } from 'node:url';
56
import { ensureStateDirs, paths, withLock, bootId, readJsonSafe, readDrainMarker, readRemoteDrain } from './state.js';
@@ -681,7 +682,7 @@ function spawnReconcileSweep(status) {
681682
if (process.env.LANE_BROKER_TEST_NO_STATUS_RECONCILE === '1') return;
682683
if (!status.remote?.some((r) => r.orphaned && !r.moveInterrupted)) return;
683684
try {
684-
const child = spawn(process.execPath, [LANE_BIN, 'reconcile'], { detached: true, stdio: 'ignore', env: process.env });
685+
const child = spawn(process.execPath, [...childExitSafeFlags(), LANE_BIN, 'reconcile'], { detached: true, stdio: 'ignore', env: process.env });
685686
child.on('error', () => {});
686687
child.unref();
687688
} catch {

0 commit comments

Comments
 (0)