Skip to content

fix(dev-stack): identify the dev-server deaths as a Node v24.15.0 stack-corruption bug - #224

Merged
natestarner merged 1 commit into
mainfrom
fix/node-2415-stack-corruption
Sep 2, 2026
Merged

natestarner merged 1 commit into
mainfrom
fix/node-2415-stack-corruption

Conversation

@natestarner

Copy link
Copy Markdown
Owner

SOLVED. The Vite dev server has been dying mid-e2e-run since 2026-08-17 with rc=127, no error output, and a healthy backend beside it. Four investigations, four different conclusions, none of which held.

It was never Vite. Node v24.15.0 corrupts its own stack under concurrent load on Windows. Windows' /GS stack-cookie check catches the corruption and fail-fasts the process instantly and silently — which is why there was never a message, a dialog, or a Windows Error Reporting event.

The evidence that settled it

A full minidump (ProcDump — no admin needed), parsed for its exception record and module list:

code            : 0xC0000409
fast-fail code  : 2  (STACK_COOKIE_CHECK_FAILURE)
faulting module : C:\Program Files\nodejs\node.exe  +0x21F2189

Subcode 2, not 7. That distinction is the whole answer — code 7 (FATAL_APP_EXIT) is abort(), a deliberate panic; code 2 is the stack cookie found overwritten, i.e. real memory corruption.

Then A/B on this repo's own e2e suite at E2E_WORKERS=11:

Node Crashes
v24.15.0 (currently installed) 2 in 3 — plus 3 earlier hunts that each crashed on attempt 1
v24.14.1 0 in 8
v24.20.0 LTS 0 in 8

Corroborated upstream: nodejs/node#62991 reports intermittent native crashes on Windows in v24.15.0, with v24.14.1 called out as known-good.

Eliminated by measurement, not argument

  • Vite — reproduces identically on Vite 7.3.6 (rollup + esbuild) and 8.2.1 (rolldown). The rolldown binding is loaded in the process, but the fault is in node.exe.
  • Rust panic — reproduced under RUST_BACKTRACE=full; no panic message.
  • V8 fatal error / OOM — --report-on-fatalerror produced no report, and the flag was verified to fire on a real V8 OOM.
  • Vite's own memory — 572 MB peak against Node's multi-GB ceiling, falling in the final seconds; handles and threads flat.
  • Playwright teardown — no webServer in playwright.config.ts; workers keep cycling 10+ s after Vite dies.
  • Job objects / console CTRL — everything is inJob=True, including the backend that survives.

What changes

  • scripts/check-node-version.sh — warns at up.sh time when the running Node is on the known-bad list. A warning, not a hard failure: someone with only the bad build installed still needs to run the stack, and the ledger now types these crashes correctly. Add to the list rather than rediscovering this.
  • supervise-server.js — use a shell only for a .cmd. Spawning a real .exe through cmd.exe breaks on any space in its path and tripped DEP0190 on every start.
  • dev-stack.md — replaced the speculation with the answer, and retired the two inferences that cost the most.
  • docs/incidents/2026-09-01-… — the full narrative, including why each earlier investigation went wrong.

The two inferences that cost the most

  • "A crash is ruled out — zero WER events." __fastfail bypasses exception handling by design and produces no WER event. Absence of evidence from a mechanism built to produce none.
  • "Memory is ruled out — commit was comfortable." True but irrelevant: it measured host commit charge, a different quantity from the failure.

CI is deliberately untouched

The bug is Windows-only (upstream confirms Linux/macOS unaffected) and CI runs on ubuntu, so pinning node-version there would be churn that fixes nothing.

Verified

Full e2e suite 215 passed after the change. The guard fires correctly on the installed v24.15.0.

Action for the machine: install any Node other than v24.15.0 — v24.14.1 and v24.20.0 LTS are both verified clean here. That is the actual fix; this PR makes sure nobody loses another week to it.

Handbook: unaffected — local dev tooling, no user-facing surface.

SOLVED. The Vite dev server has been dying mid-e2e-run since 2026-08-17 with
"rc=127", no error output, and a healthy backend beside it. It was never Vite.

Node v24.15.0 corrupts its own stack under concurrent load on Windows. Windows'
/GS stack-cookie check catches the corruption and fail-fasts the process
instantly and silently -- which is why there was never a message, a dialog, or a
Windows Error Reporting event.

Evidence
- Minidump (ProcDump, no admin), parsed for its exception record and modules:
      code            : 0xC0000409
      fast-fail code  : 2  (STACK_COOKIE_CHECK_FAILURE)
      faulting module : C:\Program Files\nodejs\node.exe
  Subcode 2, not 7. Code 7 (FATAL_APP_EXIT) is abort() -- a deliberate panic.
  Code 2 is the stack cookie found overwritten: real memory corruption.
- A/B on this repo's own e2e suite at E2E_WORKERS=11:
      v24.15.0   2 crashes in 3 runs  (+3 earlier hunts, each crashed run 1)
      v24.14.1   0 crashes in 8 runs
      v24.20.0   0 crashes in 8 runs
- Reproduces identically on Vite 7.3.6 AND Vite 8.2.1, so neither the new
  toolchain nor its Rust native addons are implicated.
- Rust panic ruled out with RUST_BACKTRACE=full (no panic message); V8 fatal
  error ruled out with --report-on-fatalerror (no report, and the flag was
  verified to fire on a real V8 OOM).
- Upstream: nodejs/node#62991 -- v24.15.0 native
  crashes on Windows, v24.14.1 called out as known-good. Windows-only, so CI on
  ubuntu was never at risk and is deliberately left alone.

What changes
- scripts/check-node-version.sh: warns at up.sh time when the running Node is on
  the known-bad list. A warning, not a hard failure -- someone with only the bad
  build installed still needs to run the stack, and the ledger now types these
  crashes correctly. Add to the list rather than rediscovering this.
- supervise-server.js: use a shell only for a .cmd. Spawning a real .exe through
  cmd.exe breaks on any space in its path and tripped DEP0190 on every start.
- dev-stack.md: replaced the speculation with the answer, and retired the two
  inferences that cost the most -- "no WER event, therefore not a crash" (
  __fastfail produces none by design) and "commit was comfortable, therefore not
  memory" (it measured host commit, a different quantity).
- docs/incidents/2026-09-01-...: the full narrative, including why four previous
  investigations each reached a different wrong conclusion.

Verified: full e2e suite 215 passed after the change.

Co-Authored-By: Claude Opus 5 <[email protected]>
@natestarner
natestarner merged commit f6e6a94 into main Sep 2, 2026
17 checks passed
@natestarner
natestarner deleted the fix/node-2415-stack-corruption branch September 2, 2026 02:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant