Repository navigation
fix(dev-stack): identify the dev-server deaths as a Node v24.15.0 stack-corruption bug - #224
Merged
Merged
Conversation
SOLVED. The Vite dev server has been dying mid-e2e-run since 2026-08-17 with
"rc=127", no error output, and a healthy backend beside it. It was never Vite.
Node v24.15.0 corrupts its own stack under concurrent load on Windows. Windows'
/GS stack-cookie check catches the corruption and fail-fasts the process
instantly and silently -- which is why there was never a message, a dialog, or a
Windows Error Reporting event.
Evidence
- Minidump (ProcDump, no admin), parsed for its exception record and modules:
code : 0xC0000409
fast-fail code : 2 (STACK_COOKIE_CHECK_FAILURE)
faulting module : C:\Program Files\nodejs\node.exe
Subcode 2, not 7. Code 7 (FATAL_APP_EXIT) is abort() -- a deliberate panic.
Code 2 is the stack cookie found overwritten: real memory corruption.
- A/B on this repo's own e2e suite at E2E_WORKERS=11:
v24.15.0 2 crashes in 3 runs (+3 earlier hunts, each crashed run 1)
v24.14.1 0 crashes in 8 runs
v24.20.0 0 crashes in 8 runs
- Reproduces identically on Vite 7.3.6 AND Vite 8.2.1, so neither the new
toolchain nor its Rust native addons are implicated.
- Rust panic ruled out with RUST_BACKTRACE=full (no panic message); V8 fatal
error ruled out with --report-on-fatalerror (no report, and the flag was
verified to fire on a real V8 OOM).
- Upstream: nodejs/node#62991 -- v24.15.0 native
crashes on Windows, v24.14.1 called out as known-good. Windows-only, so CI on
ubuntu was never at risk and is deliberately left alone.
What changes
- scripts/check-node-version.sh: warns at up.sh time when the running Node is on
the known-bad list. A warning, not a hard failure -- someone with only the bad
build installed still needs to run the stack, and the ledger now types these
crashes correctly. Add to the list rather than rediscovering this.
- supervise-server.js: use a shell only for a .cmd. Spawning a real .exe through
cmd.exe breaks on any space in its path and tripped DEP0190 on every start.
- dev-stack.md: replaced the speculation with the answer, and retired the two
inferences that cost the most -- "no WER event, therefore not a crash" (
__fastfail produces none by design) and "commit was comfortable, therefore not
memory" (it measured host commit, a different quantity).
- docs/incidents/2026-09-01-...: the full narrative, including why four previous
investigations each reached a different wrong conclusion.
Verified: full e2e suite 215 passed after the change.
Co-Authored-By: Claude Opus 5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
SOLVED. The Vite dev server has been dying mid-e2e-run since 2026-08-17 with
rc=127, no error output, and a healthy backend beside it. Four investigations, four different conclusions, none of which held.It was never Vite. Node v24.15.0 corrupts its own stack under concurrent load on Windows. Windows'
/GSstack-cookie check catches the corruption and fail-fasts the process instantly and silently — which is why there was never a message, a dialog, or a Windows Error Reporting event.The evidence that settled it
A full minidump (ProcDump — no admin needed), parsed for its exception record and module list:
Subcode 2, not 7. That distinction is the whole answer — code 7 (
FATAL_APP_EXIT) isabort(), a deliberate panic; code 2 is the stack cookie found overwritten, i.e. real memory corruption.Then A/B on this repo's own e2e suite at
E2E_WORKERS=11:Corroborated upstream: nodejs/node#62991 reports intermittent native crashes on Windows in v24.15.0, with v24.14.1 called out as known-good.
Eliminated by measurement, not argument
rolldownbinding is loaded in the process, but the fault is innode.exe.RUST_BACKTRACE=full; no panic message.--report-on-fatalerrorproduced no report, and the flag was verified to fire on a real V8 OOM.webServerinplaywright.config.ts; workers keep cycling 10+ s after Vite dies.inJob=True, including the backend that survives.What changes
scripts/check-node-version.sh— warns atup.shtime when the running Node is on the known-bad list. A warning, not a hard failure: someone with only the bad build installed still needs to run the stack, and the ledger now types these crashes correctly. Add to the list rather than rediscovering this.supervise-server.js— use a shell only for a.cmd. Spawning a real.exethroughcmd.exebreaks on any space in its path and trippedDEP0190on every start.dev-stack.md— replaced the speculation with the answer, and retired the two inferences that cost the most.docs/incidents/2026-09-01-…— the full narrative, including why each earlier investigation went wrong.The two inferences that cost the most
__fastfailbypasses exception handling by design and produces no WER event. Absence of evidence from a mechanism built to produce none.CI is deliberately untouched
The bug is Windows-only (upstream confirms Linux/macOS unaffected) and CI runs on ubuntu, so pinning
node-versionthere would be churn that fixes nothing.Verified
Full e2e suite 215 passed after the change. The guard fires correctly on the installed v24.15.0.
Action for the machine: install any Node other than v24.15.0 — v24.14.1 and v24.20.0 LTS are both verified clean here. That is the actual fix; this PR makes sure nobody loses another week to it.
Handbook: unaffected — local dev tooling, no user-facing surface.