Skip to content

Commit 51067f3

Browse files
fix(ci): economyEngine.test.js IPC flake bit again after #3222 (isolation alone wasn't the fix) (#3276)
PR #3222's fix (run the file alone in its own pass) theorized the "Unable to deserialize cloned data" test-runner bug needed concurrent IPC traffic from other workers to trigger. That theory was disproven the same evening: the isolated pass itself failed with the identical error twice more (CI runs 30836291715 attempt 1 and 30840936886), both with zero other workers running. One of the two was masked in `gh run list` by a subsequent green rerun and only surfaced by checking run_attempt > 1 directly via the GitHub REST API. The bug (nodejs/node#64061, fixed upstream but not yet in our pinned Node 24.x line) is triggered by this file's IPC payload size/shape alone, not by concurrency. Running it "alone" only changed the odds of hitting the exact buffer boundary; it never removed the vulnerable code path (the file still ran as a child worker streaming results back via socket + structured-clone). Fix: run this one file with --test-isolation=none, which executes it in the same process as the node --test invocation instead of spawning a child worker. No child worker means no IPC round-trip for this file, so processRawBuffer/FileTest.parseMessage (present in every crash stack trace) are never invoked for it - the bug class cannot trigger regardless of payload size or concurrent CI load. Scope is limited to this one file/pass; the other ~340 backend test files keep default process-per-file isolation unchanged. Verified locally: 25/25 green runs with --test-isolation=none, 25/25 green with default isolation (no local repro - expected, the bug is rare and CI-Linux-weighted), failure-propagation confirmed correct (a deliberately failing test still exits 1), and the per-file FileTest wrapper line from the crash traces is structurally absent from --test-isolation=none output. 4x full npm test (4988/4988) green after the change. No dependency, lockfile, or package.json changes. Refs #3172 Co-authored-by: Nicolai Dolmer <[email protected]> Co-authored-by: Claude <[email protected]>
1 parent 6ae1d6a commit 51067f3

2 files changed

Lines changed: 111 additions & 23 deletions

File tree

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
# Postmortem · 2026-08-04 · #3172 recurrence: concurrency theory disproven, fixed with --test-isolation=none
2+
3+
## Hvad skete der?
4+
PR #3222 (2026-08-03 eftermiddag) fixede #3172 ved at køre
5+
`lib/economyEngine.test.js` alene FØRST i egen pass, ud fra teorien at
6+
"Unable to deserialize cloned data due to invalid or unsupported version"
7+
kun opstod når mange testfil-workers strømmede IPC-data samtidigt til
8+
parent-processen. 5 lokale stress-runs var grønne, og fixet blev merget.
9+
10+
Samme aften (3/8) fejlede backend-tests-jobbet igen 2 gange med PRÆCIS samme
11+
fejl, BEGGE gange i selve Pass 1 — den isolerede kørsel af filen ALENE, uden
12+
en eneste anden worker aktiv:
13+
- Run 30836291715, attempt 1, 17:20 (branch fix/2041-clarity-stitching) —
14+
maskeret i `gh run list`, fordi en automatisk/manuel re-run (attempt 2)
15+
bagefter var grøn, så run'ets SLUT-konklusion viste "success". Fundet kun
16+
ved at scanne `run_attempt > 1` via GitHub REST API direkte.
17+
- Run 30840936886, 18:22 (branch feat/3188-datatable-sort) — synlig direkte
18+
i `gh run list --status failure`.
19+
20+
## Rod-årsag (korrigeret)
21+
Concurrency-teorien fra #3222 var forkert. Buggen (nodejs/node#64061, en
22+
signed/unsigned-fejl i test-runnerens `processRawBuffer`) trigges af selve
23+
STØRRELSEN/FORMEN af `economyEngine.test.js`'s IPC-payload — ikke af
24+
samtidig trafik fra andre workers. At isolere filen fra andre workers
25+
reducerede sandsynligvis kun TIMINGEN af buffer-chunks nok til at sænke
26+
frekvensen, men fjernede aldrig selve den sårbare kodesti: filen kører
27+
STADIG som en separat child-worker der streamer resultater tilbage via
28+
socket + structured-clone, uanset om den er alene eller ej.
29+
30+
## Fix
31+
`backend/scripts/run-tests.js` Pass 1 kører nu med
32+
`--test-isolation=none`, som får node:test til at eksekvere filen i SAMME
33+
proces som `node --test`-invokeringen i stedet for at spawne en
34+
child-worker. Uden en child-worker er der intet IPC-round-trip for denne
35+
fil, så `processRawBuffer`/`FileTest.parseMessage` (begge i hver eneste
36+
crash-stack-trace) kaldes aldrig for filen — bug-klassen kan strukturelt
37+
ikke trigges, uanset payload-størrelse eller samtidig CI-belastning.
38+
39+
Scope bevidst begrænset til Pass 1 (kun denne ene fil). Pass 2 (~340 øvrige
40+
filer) beholder standard proces-pr-fil isolation uændret — at slå isolation
41+
fra for HELE suiten ville risikere global-state-lækage mellem urelaterede
42+
testfiler, en langt større blast radius end nødvendigt for ét payload.
43+
44+
## Verifikation
45+
- 25/25 kørsler af `node --test --test-isolation=none lib/economyEngine.test.js`
46+
lokalt: grønt hver gang, 105/105 tests.
47+
- 25/25 kørsler med standard isolation (baseline, ingen lokal reproduktion —
48+
ventet, buggen er sjælden og primært set på CI's Ubuntu-runner).
49+
- Bekræftet exit-kode-korrekthed: en kunstigt fejlende test under
50+
`--test-isolation=none` exit'er stadig 1 (fejl undertrykkes ikke).
51+
- Bekræftet strukturelt: uden isolation vises INGEN "✔/✖ lib/economyEngine.test.js
52+
(Xms)"-wrapper-linje i output (den linje der kom fra `FileTest.parseMessage`
53+
i alle crash-traces) — beviser den sårbare kodesti er bypasset, ikke bare
54+
undgået ved timing.
55+
- 4× fuld `npm test` (4988/4988) grønt lokalt efter ændringen.
56+
57+
## Læring
58+
En "kør den isoleret"-fix er kun en RIGTIG fix hvis isolationen faktisk
59+
fjerner den mekanisme buggen lever i. Her fjernede "isoleret proces" kun
60+
KONKURRENCEN om IPC-båndbredde, men beholdt selve IPC-mekanismen (socket +
61+
structured-clone) som buggen sidder i — deraf den falske tryghed fra 5/5
62+
lokale grønne stress-runs (samme sjældne-timing-problem som i CI, bare
63+
lavere sandsynlighed lokalt). Når rod-årsagen er en bug i en SPECIFIK
64+
kode-mekanisme (her: cross-process IPC-deserialisering), er den robuste fix
65+
at fjerne selve mekanismen for det sårbare payload (`--test-isolation=none`),
66+
ikke at ændre hvornår/med-hvem den kører. Bonus: `gh run list` viser kun
67+
SIDSTE attempt's konklusion — en flake der bliver maskeret af en efterfølgende
68+
grøn re-run er usynlig uden at tjekke `run_attempt > 1` via GitHub's REST API
69+
direkte (`gh api .../runs?created=...`).

‎backend/scripts/run-tests.js‎

Lines changed: 42 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -1,28 +1,44 @@
11
// #3172: node:internal/test_runner intermittently corrupts worker-IPC data
2-
// ("Unable to deserialize cloned data due to invalid or unsupported version")
3-
// only when MANY test-file workers stream results back to the parent test
4-
// runner process at the same time. Root cause upstream: nodejs/node#64061,
5-
// fixed by nodejs/node#64706 (merged 2026-07-26) — a signed/unsigned bug in
6-
// the test runner's internal `processRawBuffer` that mishandles large IPC
7-
// buffers. That fix has not reached our pinned Node 24.x line yet, so we
8-
// isolate the offending file instead of waiting on a Node bump.
2+
// ("Unable to deserialize cloned data due to invalid or unsupported version").
3+
// Root cause upstream: nodejs/node#64061, fixed by nodejs/node#64706 (merged
4+
// 2026-07-26) — a signed/unsigned bug in the test runner's internal
5+
// `processRawBuffer` that mishandles large IPC buffers streamed from a
6+
// per-file test worker back to the parent process. That fix has not reached
7+
// our pinned Node 24.x line yet.
98
//
109
// lib/economyEngine.test.js is by far the largest backend test file (100+
1110
// top-level tests, 6500+ lines, ~250KB) — its per-worker result payload is
12-
// large enough to be the one that trips the bug when it lands concurrently
13-
// with the other ~340 test-file workers. Isolated runs of the file are
14-
// 102/102 green every time; the flake has only ever been observed in the
15-
// full suite (#3169-CI, #3171-CI).
11+
// the one large enough to trip the bug.
1612
//
17-
// Fix: run the large file alone FIRST (no concurrent worker traffic to race
18-
// against), then run every other backend test file exactly as before (same
19-
// flags, same default concurrency). Any CLI args npm forwards (e.g.
20-
// `--test-reporter=spec` from `npm test -- --test-reporter=spec`) are passed
21-
// through to both passes unchanged.
13+
// A first fix (2026-08-03, PR #3222) ran this file alone in its own pass,
14+
// theorizing the bug needed concurrent IPC traffic from other workers to
15+
// trigger. That theory was WRONG: the isolated pass itself failed with the
16+
// exact same deserialize error twice more the same evening (CI runs
17+
// 30836291715 attempt 1 @17:20 and 30840936886 @18:22, both on the isolated
18+
// Pass 1, zero other workers running). The bug is triggered by this file's
19+
// payload size/shape alone, not by concurrency — isolating it from other
20+
// workers only lowered the odds of hitting the exact buffer boundary that
21+
// trips `processRawBuffer`, it didn't remove the vulnerable code path.
2222
//
23-
// Deliberately NOT using `--test-concurrency=1` for the whole suite: that
24-
// would serialize all ~344 files and materially slow down CI/local runs for
25-
// a bug that only needs ONE file kept out of the concurrent pool.
23+
// Actual fix: run this one file with `--test-isolation=none`, which makes
24+
// node:test execute it in the *same* process instead of spawning a child
25+
// worker that streams results back over a socket via structured-clone IPC.
26+
// With no IPC round-trip for this file, `processRawBuffer`/`FileTest.parseMessage`
27+
// are never invoked for it, so the bug class cannot trigger regardless of
28+
// payload size or concurrent load. Verified locally: 25/25 consecutive green
29+
// runs, exit codes and failure-propagation confirmed correct (a deliberately
30+
// failing test still exits 1), and the per-file `FileTest` wrapper line that
31+
// appeared in every crash stack trace is absent from `--test-isolation=none`
32+
// output — confirming the vulnerable path is structurally bypassed, not just
33+
// avoided by timing.
34+
//
35+
// `--test-isolation=none` is scoped to Pass 1 ONLY (this one file). Pass 2
36+
// (all other ~340 files) keeps the default process-per-file isolation
37+
// unchanged — disabling isolation suite-wide would risk state/global leakage
38+
// between unrelated test files, a much bigger blast radius than this one
39+
// file needs. Any CLI args npm forwards (e.g. `--test-reporter=spec` from
40+
// `npm test -- --test-reporter=spec`) are passed through to both passes
41+
// unchanged, in addition to the isolation flag on Pass 1.
2642
import { spawnSync } from "node:child_process";
2743
import { readdirSync } from "node:fs";
2844
import path from "node:path";
@@ -44,10 +60,10 @@ function findTestFiles(dir, out) {
4460
return out;
4561
}
4662

47-
function runNodeTest(extraArgs, files) {
63+
function runNodeTest(extraArgs, files, isolationArgs = []) {
4864
const result = spawnSync(
4965
process.execPath,
50-
["--test", "--import", "./test-setup.js", ...extraArgs, ...files],
66+
["--test", "--import", "./test-setup.js", ...isolationArgs, ...extraArgs, ...files],
5167
{ stdio: "inherit", cwd: backendRoot },
5268
);
5369
if (result.error) throw result.error;
@@ -77,8 +93,11 @@ const restFilesRelative = allTestFilesAbsolute
7793
.filter((file) => file !== isolatedAbsolute)
7894
.map((file) => path.relative(backendRoot, file));
7995

80-
console.log(`\n[run-tests] Pass 1/2: ${ISOLATED_RELATIVE} in isolation (#3172 worker-IPC flake guard)\n`);
81-
const isolatedStatus = runNodeTest(extraArgs, [ISOLATED_RELATIVE]);
96+
console.log(
97+
`\n[run-tests] Pass 1/2: ${ISOLATED_RELATIVE} with --test-isolation=none ` +
98+
"(#3172 worker-IPC deserialize-bug guard — no child worker, no IPC, bug class cannot trigger)\n",
99+
);
100+
const isolatedStatus = runNodeTest(extraArgs, [ISOLATED_RELATIVE], ["--test-isolation=none"]);
82101
if (isolatedStatus !== 0) {
83102
process.exit(isolatedStatus);
84103
}

0 commit comments

Comments
 (0)