Repository navigation
fix: CLI and Gateway processes can hang forever at exit on Node 24 and 26 - #163788
Conversation
On Node 24 and 26, process.exit() joins V8's platform workers without disposing the isolate. A concurrent Maglev or Sparkplug compile job parked waiting for a main-thread GC then never finishes, so CLI commands, the hook relay, or a stopping Gateway can sit at 0% CPU forever after printing their output (nodejs/node#64274; fix pending in nodejs/node#66171). Every OpenClaw executable entrypoint (CLI entry, Gateway/service index, native hook relay, macOS node worker) now turns off Maglev and concurrent Sparkplug at startup when it runs as main, matching Node 22's tiering and the existing Vitest policy. Library imports keep stock V8; explicit --maglev or --concurrent-sparkplug flags still win. Testbox A/B (4 vCPU, Node 24.19, e2e CLI child replay, 8-way): 6/1019 hung with defaults, 0/1019 with the policy; re-enabling only Maglev hung 2/1019 and only concurrent Sparkplug 12/944, so both flags are needed.
|
🦞👀 Pull request received. I will update this pull request when review starts. ClawSweeper review completeClawSweeper finished reviewing this revision. The review result is being finalized. |
|
Codex review: needs maintainer review before merge. Reviewed October 2, 2026, 5:15 PM ET / 21:15 UTC. ClawSweeper reviewWhat this changesThe PR disables two Node background compilers at OpenClaw executable startup to prevent exit hangs, preserves explicit operator flags, and documents the behavior. Merge readiness✅ Ready for maintainer review This PR remains useful: current main lacks the mitigation, and the supplied production-path replay supports it. No blocking patch defect was found. Likely related people: steipete and SunnyShu0925 (medium-confidence routing candidates). Priority: P1 Review scores
Verification
How this fits togetherOpenClaw's executable bootstrap prepares Node before CLI commands, Gateway services, hook relays, and the macOS node worker run. Its compiler policy affects whether those processes can finish shutdown after their work and output complete. flowchart TD
A[Executable invocation] --> B[OpenClaw bootstrap]
B --> C[Node runtime and explicit flags]
C --> D[Disable unsafe background compilers]
D --> E[Command or service runs]
E --> F[Output and resource cleanup]
F --> G[Process exits]
Before mergeNone. Agent review detailsSecurityNone. PR surfaceSource +36, Docs +4. Total +40 across 6 files. View PR surface stats
Review metricsNone. Technical reviewBest possible solution: Use one executable-bootstrap compiler policy while retaining operator opt-ins, then retire it when supported Node versions contain the upstream shutdown fix. Do we have a high-confidence way to reproduce the issue? Yes: the supplied Node 24.19 Testbox replay identifies a concrete, intermittently failing CLI-child exit path with debugger stacks. This read-only review did not execute a current-main reproduction. Is this the best way to solve the issue? Yes: a shared policy at executable startup addresses the evidenced compiler deadlock without replacing lifecycle cleanup or macOS CA handling. AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning medium; reviewed against 6347c0c51250. LabelsLabel changes:
Label justifications:
EvidenceWhat I checked:
Likely related people:
Rating scale
Overall follows the weaker of proof and patch quality. Workflow
|
|
Merged via squash.
|
…enclaw#163788) On Node 24 and 26, process.exit() joins V8's platform workers without disposing the isolate. A concurrent Maglev or Sparkplug compile job parked waiting for a main-thread GC then never finishes, so CLI commands, the hook relay, or a stopping Gateway can sit at 0% CPU forever after printing their output (nodejs/node#64274; fix pending in nodejs/node#66171). Every OpenClaw executable entrypoint (CLI entry, Gateway/service index, native hook relay, macOS node worker) now turns off Maglev and concurrent Sparkplug at startup when it runs as main, matching Node 22's tiering and the existing Vitest policy. Library imports keep stock V8; explicit --maglev or --concurrent-sparkplug flags still win. Testbox A/B (4 vCPU, Node 24.19, e2e CLI child replay, 8-way): 6/1019 hung with defaults, 0/1019 with the policy; re-enabling only Maglev hung 2/1019 and only concurrent Sparkplug 12/944, so both flags are needed.
…enclaw#163788) On Node 24 and 26, process.exit() joins V8's platform workers without disposing the isolate. A concurrent Maglev or Sparkplug compile job parked waiting for a main-thread GC then never finishes, so CLI commands, the hook relay, or a stopping Gateway can sit at 0% CPU forever after printing their output (nodejs/node#64274; fix pending in nodejs/node#66171). Every OpenClaw executable entrypoint (CLI entry, Gateway/service index, native hook relay, macOS node worker) now turns off Maglev and concurrent Sparkplug at startup when it runs as main, matching Node 22's tiering and the existing Vitest policy. Library imports keep stock V8; explicit --maglev or --concurrent-sparkplug flags still win. Testbox A/B (4 vCPU, Node 24.19, e2e CLI child replay, 8-way): 6/1019 hung with defaults, 0/1019 with the policy; re-enabling only Maglev hung 2/1019 and only concurrent Sparkplug 12/944, so both flags are needed. (cherry picked from commit 1aa0aeb) Co-authored-by: RomneyDa <[email protected]>
Related: #163788 ## What Problem This Solves Fixes commands hanging after output during forced Node shutdown without retaining the blanket Maglev and concurrent-Sparkplug workaround. ## User Impact CLI commands and the Gateway finish owned work and cleanup before exiting naturally, with Node's default compiler settings restored. Exit status, Unix signal termination, output, explicit compiler overrides, and system-CA trust are preserved. The Gateway still joins native skills-watcher retirement. This can retain synchronous FSEvents shutdown latency on macOS; the separate upstream certificate-loader stall is not fixed. Native macOS/Windows qualification is not claimed. ## Why This Change Was Made Existing lifecycle owners retain admitted work, native workers, child transports, updater recovery, and watchdogs through actual completion. The boundary guard prevents new forced normal exits. Cleanup dependencies are retained before a command can replace its package, without eagerly importing them into CLI bootstrap. No new configuration, dependency, or schema is introduced. ## Evidence - Earlier full pnpm build passed on composed tree faaafbe6b1f47e707f8392243898525990f3a452 (the terminal repair plus pinned CI base 97e81ce), including the canonical CLI-bootstrap import gate, in 4m35.7s. Explicit/requested exits restore terminal state after cleanup; ordinary completion stays reset-free. The build retains a Control UI startup-JS size warning; no budget was raised or waived. - The built packaged CLI completed a real-PTY model round trip and exited naturally with status 0. Eight dual-TTY failure probes and twelve piped/dual-TTY telemetry probes passed. These used the built executable and existing helpers, not a claimed canonical Vitest smoke run. - The composed memory helper parses with one run/execute binding and passes 20 publication tests. The final runtime, one-shot, and cleanup suites passed 40 tests, including red proof of premature watcher loading. Broker regressions: 84 passed; real-process negative controls cover IPC disconnect and Unix signal propagation after output settlement. - The SIGTERM capture fixture retains its 20-second parent deadline and persistence assertions: red was 99 instead of 143; repaired case passed in 708ms (11.19s wrapper). The latest message/update fixture matrix passed 25/25 (179.52s): synthetic pending resources release only after observed stall diagnostics and join completion; ordered exit listeners remain checked. These later repairs change test/support files only, not production code. - Latest pinned-main integration preserved QA error reporting without forced exit: 204 QA tests, 10 reconciliation tests, and 6 real service-child relay tests passed. The reconciliation case failed before removing an obsolete admission close. Scoped types, lint, and three-pass review are complete; this integration did not repeat the full application build. - CI repair qualification: 96 tests across seven files passed (196.15s); final QA integration: 34 tests across three files passed (43.28s). Scoped types, type-aware lint, formatting, and mock-export checking passed. Fresh independent P0–P2 reviews found no actionable issues. - Browser startup recovery: all three saved/chat/toast cases passed (35.743s). Output and local-chat shutdown regressions were exercised with the rebuilt CLI. Fixture repairs preserve native completion, privacy, output, and fallback-model assertions. - Historical pre-integration proof: 115 affected test files; 137 built-executable cases with two platform skips; package-integrity checks; Linux Node 24.19.0 compiler activity and real Gateway SIGTERM-to-natural-exit in 175ms; published 2026.9.9 driver update with readiness and durable/incognito state checks. [Historical verification](https://github.com/openclaw/openclaw/actions/runs/38073218249). Compiler traces are functional witnesses, not throughput benchmarks or universal deadlock proof. Current-head CI run 38106144930 is red: 12 failing test jobs are attributed to the tested baseline (294b2e3), with complete job and source-bound evidence for native admission. They cover browser catalog/focus and a stale Details selector, channel mocks, Codex import/steering, and Claude discovery fixtures. None is claimed fixed or green by this PR. The original Gateway read-admission and node-wire timeout observations remain unreproduced, not declared fixed. Earlier local smoke preparation was refused by managed-artifact separation; no guard was bypassed. Requested and directed by @steipete. ## Work sessions - [Original discussion with @steipete](https://team.openclaw.ai/chat/roboclaw/dashboard/fcef23f8-a69b-4b60-9869-3bd40ac48588) - [Implementation and verification](https://team.openclaw.ai/chat/roboclaw/subagent/c0adc270-c23a-4139-b3b7-00852b7db66f) ## Worked on by - @steipete <!-- openclaw-publication:09df66dc-cd63-4093-b782-154eeacdaa8f --> --- [View the OpenClaw team session](https://team.openclaw.ai/chat/roboclaw/subagent/c0adc270-c23a-4139-b3b7-00852b7db66f) Co-authored-by: steipete <[email protected]>
Related: openclaw#163788 ## What Problem This Solves Fixes commands hanging after output during forced Node shutdown without retaining the blanket Maglev and concurrent-Sparkplug workaround. ## User Impact CLI commands and the Gateway finish owned work and cleanup before exiting naturally, with Node's default compiler settings restored. Exit status, Unix signal termination, output, explicit compiler overrides, and system-CA trust are preserved. The Gateway still joins native skills-watcher retirement. This can retain synchronous FSEvents shutdown latency on macOS; the separate upstream certificate-loader stall is not fixed. Native macOS/Windows qualification is not claimed. ## Why This Change Was Made Existing lifecycle owners retain admitted work, native workers, child transports, updater recovery, and watchdogs through actual completion. The boundary guard prevents new forced normal exits. Cleanup dependencies are retained before a command can replace its package, without eagerly importing them into CLI bootstrap. No new configuration, dependency, or schema is introduced. ## Evidence - Earlier full pnpm build passed on composed tree faaafbe6b1f47e707f8392243898525990f3a452 (the terminal repair plus pinned CI base 5706528), including the canonical CLI-bootstrap import gate, in 4m35.7s. Explicit/requested exits restore terminal state after cleanup; ordinary completion stays reset-free. The build retains a Control UI startup-JS size warning; no budget was raised or waived. - The built packaged CLI completed a real-PTY model round trip and exited naturally with status 0. Eight dual-TTY failure probes and twelve piped/dual-TTY telemetry probes passed. These used the built executable and existing helpers, not a claimed canonical Vitest smoke run. - The composed memory helper parses with one run/execute binding and passes 20 publication tests. The final runtime, one-shot, and cleanup suites passed 40 tests, including red proof of premature watcher loading. Broker regressions: 84 passed; real-process negative controls cover IPC disconnect and Unix signal propagation after output settlement. - The SIGTERM capture fixture retains its 20-second parent deadline and persistence assertions: red was 99 instead of 143; repaired case passed in 708ms (11.19s wrapper). The latest message/update fixture matrix passed 25/25 (179.52s): synthetic pending resources release only after observed stall diagnostics and join completion; ordered exit listeners remain checked. These later repairs change test/support files only, not production code. - Latest pinned-main integration preserved QA error reporting without forced exit: 204 QA tests, 10 reconciliation tests, and 6 real service-child relay tests passed. The reconciliation case failed before removing an obsolete admission close. Scoped types, lint, and three-pass review are complete; this integration did not repeat the full application build. - CI repair qualification: 96 tests across seven files passed (196.15s); final QA integration: 34 tests across three files passed (43.28s). Scoped types, type-aware lint, formatting, and mock-export checking passed. Fresh independent P0–P2 reviews found no actionable issues. - Browser startup recovery: all three saved/chat/toast cases passed (35.743s). Output and local-chat shutdown regressions were exercised with the rebuilt CLI. Fixture repairs preserve native completion, privacy, output, and fallback-model assertions. - Historical pre-integration proof: 115 affected test files; 137 built-executable cases with two platform skips; package-integrity checks; Linux Node 24.19.0 compiler activity and real Gateway SIGTERM-to-natural-exit in 175ms; published 2026.9.9 driver update with readiness and durable/incognito state checks. [Historical verification](https://github.com/openclaw/openclaw/actions/runs/38073218249). Compiler traces are functional witnesses, not throughput benchmarks or universal deadlock proof. Current-head CI run 38106144930 is red: 12 failing test jobs are attributed to the tested baseline (960ccb8), with complete job and source-bound evidence for native admission. They cover browser catalog/focus and a stale Details selector, channel mocks, Codex import/steering, and Claude discovery fixtures. None is claimed fixed or green by this PR. The original Gateway read-admission and node-wire timeout observations remain unreproduced, not declared fixed. Earlier local smoke preparation was refused by managed-artifact separation; no guard was bypassed. Requested and directed by @steipete. ## Work sessions - [Original discussion with @steipete](https://team.openclaw.ai/chat/roboclaw/dashboard/fcef23f8-a69b-4b60-9869-3bd40ac48588) - [Implementation and verification](https://team.openclaw.ai/chat/roboclaw/subagent/c0adc270-c23a-4139-b3b7-00852b7db66f) ## Worked on by - @steipete <!-- openclaw-publication:09df66dc-cd63-4093-b782-154eeacdaa8f --> --- [View the OpenClaw team session](https://team.openclaw.ai/chat/roboclaw/subagent/c0adc270-c23a-4139-b3b7-00852b7db66f) Co-authored-by: steipete <[email protected]>
Related: nodejs/node#64274 (upstream fix pending in nodejs/node#66171)
What Problem This Solves
Fixes:
openclawcommands (and the hook relay or a stopping Gateway) can hang forever at 0% CPU after they have already printed their output, on Node 24 and 26.User Impact
User impact: CLI commands, Codex hook relays, and Gateway stop/restart exits finish instead of occasionally wedging forever. No configuration changes; operators who want V8's Maglev or concurrent Sparkplug back can pass
--maglev/--concurrent-sparkplugtonode.Release note: CLI, hook relay, and Gateway processes no longer hang at exit on Node 24/26 (V8 compile-job deadlock in
process.exit()).Why This Change Was Made
process.exit()joins V8's background workers without disposing the isolate. If a concurrent Maglev or Sparkplug compile job is waiting for a main-thread garbage collection at that moment, the join never returns. Node 24 turned both compilers on by default; Node 22 had neither, and our Vitest workers already run with both off for this reason.Every OpenClaw executable entrypoint (
entry, the Gateway/serviceindex, the native hook relay, the macOS node worker) now switches both off at startup when it runs as the main module. Library imports keep stock V8, Bun is skipped, and explicit operator flags win. V8 reads both flags per decision, so setting them at entry takes effect (--trace-opt: Maglev compiles inmodels list --jsondrop from 689 to 50, all before the entry runs).Alternatives checked: letting the loop drain via
process.exitCodeworks for probed commands (0–139 ms) but macOS system-CA natural exit hangs in Keychain shutdown (#113112) and ~246 directruntime.exitcall sites remain, including thegateway healthroute that hung. Adding flags to the respawn plan would miss the hook relay,gateway run/the service,gateway status, and the TUI.Evidence
Blacksmith Testbox, 4 vCPU, Node 24.19, replay of the
test/cli-json-stdoutGateway CLI children (gateway health --json, fresh state per run, 8-way). Every hang'sgdbstack: main thread inWorkerThreadsTaskRunner::Shutdown→uv_thread_join, a V8 worker parked inCollectionBarrier::AwaitCollectionBackground.Cost (same boxes): concurrent Sparkplug off is free; Maglev off costs about +5% CLI wall time sequentially and +10–20% at 8-way on 4 vCPUs. Gateway start→ready +1–5% sequential and faster under 6-way load (p50 15.0/16.0 s vs 20.7/17.4 s); SIGTERM→exit unchanged; the log-redaction hot-path benchmark is unchanged (steady state still reaches TurboFan). Gateway start/SIGTERM cycles: 0/240 exit hangs either way.
Also: Codex autoreview scoped-clean,
tsgo:core, oxlint/oxfmt on touched files, runtime build including the bootstrap-import check. Warm-home commands without the e2e environment did not reproduce the hang on Linux (0/4000 both arms); the hang was also observed on macOS CLI children and in the CI e2e shard.No unit test: the contract is process-level (a mocked
setFlagsFromStringtest would only mirror the call); the A/B above exercises the real entrypoint. Follow-up: Gateway-spawned workers with their own entry files are not covered; retire this policy once supported Node floors ship nodejs/node#66171.