Skip to content

Commit f6e6a94

Browse files
natestarnerclaude
andauthored
fix(dev-stack): identify the dev-server deaths as a Node v24.15.0 bug (#224)
SOLVED. The Vite dev server has been dying mid-e2e-run since 2026-08-17 with "rc=127", no error output, and a healthy backend beside it. It was never Vite. Node v24.15.0 corrupts its own stack under concurrent load on Windows. Windows' /GS stack-cookie check catches the corruption and fail-fasts the process instantly and silently -- which is why there was never a message, a dialog, or a Windows Error Reporting event. Evidence - Minidump (ProcDump, no admin), parsed for its exception record and modules: code : 0xC0000409 fast-fail code : 2 (STACK_COOKIE_CHECK_FAILURE) faulting module : C:\Program Files\nodejs\node.exe Subcode 2, not 7. Code 7 (FATAL_APP_EXIT) is abort() -- a deliberate panic. Code 2 is the stack cookie found overwritten: real memory corruption. - A/B on this repo's own e2e suite at E2E_WORKERS=11: v24.15.0 2 crashes in 3 runs (+3 earlier hunts, each crashed run 1) v24.14.1 0 crashes in 8 runs v24.20.0 0 crashes in 8 runs - Reproduces identically on Vite 7.3.6 AND Vite 8.2.1, so neither the new toolchain nor its Rust native addons are implicated. - Rust panic ruled out with RUST_BACKTRACE=full (no panic message); V8 fatal error ruled out with --report-on-fatalerror (no report, and the flag was verified to fire on a real V8 OOM). - Upstream: nodejs/node#62991 -- v24.15.0 native crashes on Windows, v24.14.1 called out as known-good. Windows-only, so CI on ubuntu was never at risk and is deliberately left alone. What changes - scripts/check-node-version.sh: warns at up.sh time when the running Node is on the known-bad list. A warning, not a hard failure -- someone with only the bad build installed still needs to run the stack, and the ledger now types these crashes correctly. Add to the list rather than rediscovering this. - supervise-server.js: use a shell only for a .cmd. Spawning a real .exe through cmd.exe breaks on any space in its path and tripped DEP0190 on every start. - dev-stack.md: replaced the speculation with the answer, and retired the two inferences that cost the most -- "no WER event, therefore not a crash" ( __fastfail produces none by design) and "commit was comfortable, therefore not memory" (it measured host commit, a different quantity). - docs/incidents/2026-09-01-...: the full narrative, including why four previous investigations each reached a different wrong conclusion. Verified: full e2e suite 215 passed after the change. Co-authored-by: Claude Opus 5 <[email protected]>
1 parent 67fedf9 commit f6e6a94

6 files changed

Lines changed: 206 additions & 53 deletions

File tree

‎.claude/rules/dev-stack.md‎

Lines changed: 56 additions & 52 deletions
Original file line numberDiff line numberDiff line change
@@ -30,61 +30,65 @@ on that console at once. That is what killed Vite mid-run for months, and it pre
3030
`Start-Process` substitute reverted twice — once for the pair, once for the frontend alone.
3131
Don't "unify" the two launches for symmetry.
3232

33-
## 2026-09-01 — there are TWO failure modes, and `rc=127` could never tell them apart
34-
35-
**Everything above this section was reasoning about a broken instrument.** Both halves of the
36-
diagnostic were measured wrong, and fixing them changed the answer:
37-
38-
1. **`rc=127` carries no information.** bash collapses every abnormal end to wait status 32512.
39-
Measured the same day: an external `Stop-Process -Force` (native `0xFFFFFFFF`) and a fail-fast
40-
abort inside Vite (native `0xC0000409`) both arrive at bash as "exit 127".
41-
2. **The `[[frontend exited rc=…]]` marker had never once been written.** Zero occurrences in any
42-
`frontend.log`, against 8 in `backend.log`. Git Bash's `bash.exe`, spawned with
43-
`DETACHED_PROCESS`, silently discards its own stdout, while a native Windows child writing the
44-
**same** inherited handle succeeds — so Vite's output landed, the log looked healthy, and
45-
`e2e.sh`'s "no marker means something killed it" fired every single time regardless of truth.
46-
47-
3. **`intent=unexpected` was wrong for a large share of the ledger.** `down.sh` drops a sentinel
48-
before killing, and a death is "planned" only if that sentinel is under 60s old — but `up.sh`
49-
used to **delete the sentinel the moment both ports answered**, ~20s in, while the dying server
50-
records its own exit asynchronously. Under load that lag is tens of seconds: a stop measured at
51-
20:33:17.3 — 1.3s after `down.sh` wrote the sentinel — did not reach the ledger until 20:33:44,
52-
by which point the breadcrumb was gone. So **ordinary `/run-local` restarts were filed as
53-
unexpected deaths.** Tells: 23 of the "unexpected" frontend deaths have `chrome=0` (no test run
54-
in flight, i.e. exactly what a restart looks like), and one batch shows four servers across
55-
three worktrees dying inside the same second — a machine-wide shutdown, all marked unexpected.
56-
`up.sh` now lets the sentinel age out instead.
57-
58-
**So the ledger's 66 "mysteries" were inflated.** With the native code recorded, the only death so
59-
far confirmed natural — mid-e2e, no fresh sentinel anywhere — carried `0xC0000409`, a fail-fast
60-
abort **inside** the process. Every `0xFFFFFFFF` (external kill) instance examined has been
61-
accounted for: `down.sh` doing its job, or a deliberate verification kill. **An external killer has
62-
not been demonstrated to exist.** Do not go hunting one without a sample that survives this
63-
sentinel check.
64-
65-
**Read the native code, never `rc=127`** — and check `intent` before treating anything as a bug.
66-
`0xFFFFFFFF` = something killed it (usually `down.sh`, so expect `intent=planned`); `0xC0000409` =
67-
it aborted itself; `0xC0000005` = native crash; a small integer = a real self-exit.
33+
## SOLVED 2026-09-01 — it was a Node.js bug, not Vite
34+
35+
**Node v24.15.0 corrupts its own stack under concurrent load on Windows.** Windows' `/GS`
36+
stack-cookie check catches it and fail-fasts the process instantly and silently, which is why
37+
there was never an error message, a crash dialog, or a WER event. Everything above this section
38+
was reasoning about instruments that could not see it. Full narrative:
39+
`docs/incidents/2026-09-01-vite-dev-server-node-stack-corruption.md`.
40+
41+
Minidump, parsed for its exception record and module list:
42+
43+
```
44+
code : 0xC0000409
45+
fast-fail code : 2 (STACK_COOKIE_CHECK_FAILURE) <- NOT 7 (abort); real memory corruption
46+
faulting module : C:\Program Files\nodejs\node.exe
47+
```
48+
49+
A/B on this repo's own e2e suite at `E2E_WORKERS=11`:
50+
51+
| Node | Crashes |
52+
|---|---|
53+
| **v24.15.0** | **2 in 3** (plus three earlier hunts that each crashed on the first run) |
54+
| v24.14.1 | 0 in 8 |
55+
| v24.20.0 | 0 in 8 |
56+
57+
Reproduces identically on Vite **7.3.6** and Vite **8.2.1**, so it is neither the new toolchain
58+
nor its Rust native addons. Upstream: [nodejs/node#62991](https://github.com/nodejs/node/issues/62991)
59+
(Windows-only; Linux/macOS unaffected, so CI on ubuntu was never at risk).
60+
61+
`scripts/check-node-version.sh` warns at `up.sh` time when the running Node is on the known-bad
62+
list. **Add to that list rather than rediscovering this**, and don't pin CI for it — the bug is
63+
Windows-only and CI does not run there.
64+
65+
### Read the native exit code, never `rc=127`
66+
67+
`scripts/supervise-server.js` records the real code, because bash collapses every abnormal end to
68+
127 — an external `Stop-Process -Force` and a fail-fast crash are indistinguishable through it.
69+
70+
| Native | Meaning |
71+
|---|---|
72+
| `0xFFFFFFFF` | `TerminateProcess(-1)` — something killed it. Usually `down.sh`, so expect `intent=planned` |
73+
| `0xC0000409` | fail-fast. **Check the subcode**: 2 = stack corruption, 7 = deliberate `abort()` |
74+
| `0xC0000005` | native access violation |
75+
| small integer | a real self-exit |
6876

6977
### What this retires
7078

71-
- *"A crash is ruled out — zero Application Error / WER events."* **Wrong inference.**
72-
`__fastfail` (`0xC0000409`) terminates without running exception handlers, so it produces no WER
73-
event by design. Absence of WER never ruled a crash out, and one of the two modes is a crash.
74-
- *"Host commit-charge exhaustion is ruled out — commit 49–60% at every death."* Stale. Deaths are
75-
now recorded at **84–92%** commit with 24–76 concurrent Chrome processes. Commit is not the sole
76-
cause (23 deaths happened with `chrome=0`), but the old numbers no longer describe the picture.
77-
- *"The console-CTRL event is the root cause, fixed by `detach-launch.js`."* Detaching was correct
78-
and should stay, but it **did not reduce the death rate**: 5, 9 on either side of 2026-08-18 and
79-
unchanged after. It also introduced the lost-marker bug above.
80-
81-
The npm-shim finding still stands (127 appeared with npm removed) — and is now explained: 127 was
82-
never about npm, it is just what bash reports for any abnormal end.
83-
84-
Ruled out by instruments, not argument — don't re-derive these: Playwright teardown (there is no
85-
`webServer` in `playwright.config.ts`, and its workers keep cycling for 10+ s after Vite dies) and
86-
Vite's own memory (peak 572 MB private against Node's multi-GB ceiling, and *falling* in the final
87-
seconds, with handles and threads flat).
79+
- *"A crash is ruled out — zero Application Error / WER events."* **Wrong inference, and the most
80+
expensive one.** `__fastfail` bypasses exception handling *by design* and produces no WER event.
81+
- *"Memory is ruled out — commit was comfortable."* True but irrelevant: it measured host commit
82+
charge. Vite's own peak was 572 MB against Node's multi-GB ceiling, and falling at the end.
83+
- *"The console-CTRL event is the root cause, fixed by `detach-launch.js`."* Detaching is correct
84+
and stays, but it did not reduce the death rate (5, 9 either side of 2026-08-18) and it is what
85+
broke the exit marker.
86+
- *"There are 66 unexplained deaths."* A large share were routine `/run-local` restarts misfiled
87+
as unexpected — `up.sh` deleted the planned-stop sentinel before the dying server could read it.
88+
89+
Still ruled out by instruments, not argument: Playwright teardown (no `webServer` in
90+
`playwright.config.ts`; its workers keep cycling 10+ s after Vite dies) and job objects
91+
(everything is `inJob=True`, including the backend that survives).
8892

8993
## Detaching means nothing reaps it for you
9094

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
# 2026-09-01 — The Vite dev server died mid-run for months. It was a Node.js bug.
2+
3+
**Symptom.** The frontend dev server vanished partway through an e2e run, leaving `rc=127` in the
4+
death ledger, no error output, a healthy backend beside it, and ~60 red specs that read exactly
5+
like a code regression. First recorded 2026-08-17; still happening 2026-09-01. Four separate
6+
investigations, four different conclusions, none of which held.
7+
8+
**Root cause.** **Node.js v24.15.0 corrupts its own stack under concurrent load on Windows.**
9+
Windows' `/GS` stack-cookie check catches the corruption and fail-fasts the process instantly and
10+
silently — which is why there was never an error message, never a crash dialog, and never a
11+
Windows Error Reporting event.
12+
13+
It was never Vite, never npm, never the console, never memory.
14+
15+
## Why it took four attempts
16+
17+
Every investigation was reading instruments that could not answer the question:
18+
19+
1. **`rc=127` carries no information.** bash collapses *every* abnormal end to wait status 32512.
20+
Measured: an external `Stop-Process -Force` (native `0xFFFFFFFF`) and a fail-fast crash
21+
(native `0xC0000409`) both arrive as "exit 127". The one number everyone reasoned from could
22+
not distinguish a crash from a kill.
23+
2. **The `[[frontend exited rc=…]]` marker had never once been written.** Zero occurrences in any
24+
`frontend.log`, against 8 in `backend.log`. Git Bash's `bash.exe`, spawned with
25+
`DETACHED_PROCESS`, silently discards its own stdout, while a native Windows child writing the
26+
**same** inherited handle succeeds. So Vite's output landed and the log looked healthy, while
27+
the line explaining the death never appeared — and `e2e.sh`'s "no marker means something killed
28+
it" fired every time regardless of the truth.
29+
3. **`intent=unexpected` was wrong for much of the ledger.** `up.sh` deleted the planned-stop
30+
sentinel ~20s in, while the dying server records asynchronously — tens of seconds under load.
31+
Ordinary `/run-local` restarts were therefore filed as unexplained deaths. 23 of the
32+
"unexpected" deaths have `chrome=0` (no test run in flight at all), and one batch shows four
33+
servers across three worktrees dying inside the same second: a machine-wide shutdown, every row
34+
marked unexpected.
35+
36+
Two "ruled out by instruments" conclusions were also wrong:
37+
38+
- *"A crash is ruled out — zero WER events."* `__fastfail` bypasses exception handling **by
39+
design** and produces no WER event. Absence of WER never ruled a crash out.
40+
- *"Memory is ruled out — commit was comfortable."* True but irrelevant; it measured host commit
41+
charge, which is a different quantity from the failure. Vite's own peak was 572 MB against
42+
Node's multi-GB ceiling, and *falling* in the final seconds.
43+
44+
## The evidence that settled it
45+
46+
A full minidump (ProcDump, no admin needed) parsed for its exception record and module list:
47+
48+
```
49+
code : 0xC0000409
50+
fast-fail code : 2 (STACK_COOKIE_CHECK_FAILURE)
51+
faulting module : C:\Program Files\nodejs\node.exe +0x21F2189
52+
```
53+
54+
Fast-fail **code 2**, not code 7. That distinction is the whole answer: code 7
55+
(`FATAL_APP_EXIT`) is `abort()` — a deliberate panic. Code 2 is the `/GS` cookie check finding
56+
the stack overwritten, i.e. genuine memory corruption inside `node.exe`.
57+
58+
Then A/B, identical load (`E2E_WORKERS=11`, full e2e suite):
59+
60+
| Node | Crashes |
61+
|---|---|
62+
| **v24.15.0** | **2 in 3** (plus 3 earlier hunts that each crashed on attempt 1) |
63+
| v24.14.1 | **0 in 8** |
64+
| v24.20.0 | see `dev-stack.md` for the version this repo pins |
65+
66+
Also eliminated along the way, each by measurement rather than argument:
67+
68+
- **Vite** — reproduces identically on Vite **7.3.6** (rollup + esbuild) and Vite **8.2.1**
69+
(rolldown), so neither the new toolchain nor its Rust native addons are implicated. The
70+
`rolldown` binding is loaded in the process but the fault is in `node.exe`.
71+
- **Rust panic** — reproduced with `RUST_BACKTRACE=full`; no panic message.
72+
- **V8 fatal error / OOM** — `--report-on-fatalerror` produced no report (verified the flag *does*
73+
fire on a real V8 OOM).
74+
- **Playwright teardown** — no `webServer` in `playwright.config.ts`; workers keep cycling for 10+
75+
seconds after Vite dies.
76+
- **Job objects / console CTRL events** — everything is `inJob=True` including the backend, which
77+
survives.
78+
79+
Upstream: [nodejs/node#62991](https://github.com/nodejs/node/issues/62991) reports intermittent
80+
native crashes on Windows in v24.15.0 specifically, with v24.14.1 called out as known-good.
81+
Different exit code in that report, same version, platform and class.
82+
83+
## Takeaways
84+
85+
1. **Pin the Node version, and refuse to run on a known-bad one.** A floating `node-version: '24'`
86+
in CI and whatever is installed locally is how a bad patch release walked in. `up.sh` now warns
87+
when the running Node is on the known-bad list.
88+
2. **Never diagnose one of these from `rc=127`.** Read the native exit code that
89+
`scripts/supervise-server.js` now records: `0xFFFFFFFF` = something killed it (usually
90+
`down.sh`, so expect `intent=planned`); `0xC0000409` = it fail-fasted — check the fast-fail
91+
subcode before assuming `abort()`; `0xC0000005` = native crash; a small integer = a real
92+
self-exit.
93+
3. **A diagnostic that can only return one answer is worse than none.** For three weeks the
94+
tooling asserted "something killed it" on every occurrence, which is what sent each
95+
investigation looking for a killer that did not exist.
96+
4. **Absence of evidence from a mechanism that is designed not to produce evidence proves
97+
nothing.** "No WER event, therefore not a crash" was the single most costly inference here.

‎docs/incidents/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@ need the "why" or the debugging story.
3434
| 2026-08-19 | [Creating an exercise stopped opening it, online only](2026-08-19-exercise-create-navigation-lost-online.md) — an awaited invalidation evicted the optimistic row before it was selected; invisible in all three degraded modes, and the flow had no online coverage | Log screen / offline |
3535
| 2026-08-31 | [The boot white screen came back, and the 2026-08-25 fix's recovery path didn't help](2026-08-31-boot-white-screen-recurrence.md) — "Try again" re-threw against the same restored state; root cause still not identified, but a real `/login` link plus an outside-React watchdog now guarantee a way out regardless | Frontend / resilience |
3636
| 2026-08-31 | [X-Forwarded-For spoofing bypassed every per-IP rate limit](2026-08-31-xff-spoofing-bypassed-per-ip-rate-limits.md) — the 2026-07-17 fix traded a shared-bucket bug for a header-spoofing one; confirmed live against lower before the fix | Auth / infra / security |
37+
| 2026-09-01 | [The Vite dev server died mid-run for months — it was a Node.js bug](2026-09-01-vite-dev-server-node-stack-corruption.md) — Node v24.15.0 corrupts its own stack on Windows and fail-fasts silently; four investigations failed because `rc=127`, the exit marker and the planned/unexpected tag were all broken | Local dev tooling |
3738

3839
## Adding a new incident
3940

‎scripts/check-node-version.sh‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
#!/usr/bin/env bash
2+
# Refuse to be surprised by a known-bad Node.js build.
3+
#
4+
# 2026-09-01: the Vite dev server had been dying mid-e2e-run for weeks with no error output. It
5+
# was not Vite. Node v24.15.0 corrupts its own stack under concurrent load on Windows; the /GS
6+
# stack-cookie check catches it and fail-fasts the process instantly and SILENTLY (native exit
7+
# 0xC0000409, fast-fail subcode 2 = STACK_COOKIE_CHECK_FAILURE, confirmed from a minidump).
8+
#
9+
# Measured on this repo's own e2e suite at E2E_WORKERS=11:
10+
# v24.15.0 -> 2 crashes in 3 runs (plus 3 earlier hunts that crashed on the first run)
11+
# v24.14.1 -> 0 crashes in 8 runs
12+
# Reproduces identically on Vite 7 and Vite 8, so it is Node, not the toolchain.
13+
# Upstream: https://github.com/nodejs/node/issues/62991
14+
#
15+
# Full narrative: docs/incidents/2026-09-01-vite-dev-server-node-stack-corruption.md
16+
#
17+
# Deliberately a WARNING, not a hard failure: a developer who has only the bad build installed
18+
# still needs to be able to run the stack, and the death ledger now types these crashes correctly
19+
# (scripts/supervise-server.js records the native exit code). The point is that nobody should ever
20+
# again spend a week chasing a phantom because the tooling stayed quiet about it.
21+
#
22+
# Usage: bash scripts/check-node-version.sh (sourced or called; never exits non-zero)
23+
24+
KNOWN_BAD="v24.15.0"
25+
26+
_nv="$(node --version 2>/dev/null || echo unknown)"
27+
28+
for _bad in $KNOWN_BAD; do
29+
if [ "$_nv" = "$_bad" ]; then
30+
echo "" >&2
31+
echo " !! Node $_nv is a KNOWN-BAD build for this project." >&2
32+
echo " It corrupts its own stack under load on Windows and the dev server dies" >&2
33+
echo " mid-run with no error output (native 0xC0000409, silent by design)." >&2
34+
echo " Measured here: 2 crashes in 3 e2e runs, vs 0 in 8 on v24.14.1." >&2
35+
echo "" >&2
36+
echo " Fix: install any Node other than $_bad (v24.14.1 and the current 24.x LTS are" >&2
37+
echo " both verified clean here), then restart the stack." >&2
38+
echo " Details: docs/incidents/2026-09-01-vite-dev-server-node-stack-corruption.md" >&2
39+
echo "" >&2
40+
fi
41+
done
42+
43+
unset _nv _bad

‎scripts/supervise-server.js‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,11 @@ if (!logPath || !label || !command) {
3636
// cmd.exe between us and npm -- which is harmless for our purpose: the probe measured the native
3737
// code propagating intact up through BOTH cmd.exe and npm-cli.js (all three read 0xC0000409 on a
3838
// crash). bash is the only link in the chain that mangles it, and it is no longer in the path.
39-
const child = spawn(command, args, { stdio: 'inherit', shell: process.platform === 'win32' });
39+
// A shell is needed ONLY to launch a .cmd (npm). Spawning a real .exe through cmd.exe breaks on
40+
// any space in its path ("C:/Program Files/..." arrives as the token "C:/Program") and trips
41+
// DEP0190. So use the shell only when the target is not an executable.
42+
const needsShell = process.platform === 'win32' && !/\.exe$/i.test(command);
43+
const child = spawn(command, args, { stdio: 'inherit', shell: needsShell });
4044

4145
child.on('exit', (code, signal) => {
4246
const when = new Date().toISOString().replace(/\.\d+Z$/, '');

‎scripts/up.sh‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -165,6 +165,10 @@ _open_log() {
165165
echo "=============================================================================="
166166
} >> "$log"
167167
}
168+
# A known-bad Node build is the one failure mode that looks exactly like a code regression, so
169+
# say so BEFORE anything starts rather than after an e2e run has gone red.
170+
bash "$SCRIPT_DIR/check-node-version.sh" || true
171+
168172
_open_log backend.log
169173
_open_log frontend.log
170174

0 commit comments

Comments
 (0)