You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
ci: the Node 22 test run intermittently dies in the runner's own IPC decoder ("Unable to deserialize cloned data") #383
On Node 22 the full test run intermittently fails with an exception raised inside
the test runner itself, not inside a test:
not ok 82 - packages/daemon/__tests__/cli-rules.test.ts
failureType: 'uncaughtException'
error: 'Unable to deserialize cloned data due to invalid or unsupported version.'
stack: |-
#processRawBuffer (node:internal/test_runner/runner:354:20)
FileTest.parseMessage (node:internal/test_runner/runner:290:27)
Socket.<anonymous> (node:internal/test_runner/runner:400:15)
#processRawBuffer is the runner's own IPC decoder for the messages a test file
sends back to the parent. The failure is in that channel, so the file it names is
whichever one happened to be streaming at the time — it is not a property of cli-rules.test.ts.
What was measured (Node 22.23.2, darwin-arm64, and ubuntu-latest in CI):
cli-rules.test.ts on its own: green, three runs out of three.
The full suite (npm test): one run fail 1, the next run fail 0, with no
change to a single line in between.
In CI it hit the same file twice on consecutive pushes, then passed on rerun.
Node 24 has not shown it in any run so far.
Cost: every occurrence burns a CI rerun, and it trains everyone to re-run a red
build instead of reading it — which is exactly how a real failure gets waved
through.
Worth investigating:
Whether a newer 22.x has this fixed upstream (the CI matrix uses '22', so it
already tracks the newest patch).
Whether lowering --test-concurrency for the Node 22 job removes it, and what
that costs in wall-clock.
Whether one specific test writes something to stdout that the decoder trips
over — the trace points at chunk boundaries, not at content, but that has not
been ruled out.
Correcting the claim in the issue body, with three more data points.
I wrote: "the file it names is whichever one happened to be streaming at the time — it is not a property of cli-rules.test.ts." That does not hold. It has now hit cli-rules.test.ts three times in a row in CI, on three unrelated pushes (61d9e5a, 14a1db2, 5cb4200), always on Node 22, always as not ok 82/83. Whatever the mechanism, this file is involved reproducibly.
What has since been ruled out:
Not reproducible locally. Node 22.23.2 on darwin-arm64: the file alone runs green 3/3; the full suite ran green 5/5 in a row just now. The earlier local fail 1 I recorded has not recurred.
Not output volume or encoding. The file's own TAP output is 4220 bytes with zero non-printable characters.
Not an abrupt child exit. There is no process.exit anywhere in rules-cmd.ts or in the CLI modules the test imports.
What that leaves: something specific to the CI runner — most plausibly its lower core count, since --test-concurrency defaults to the CPU count and the failure only ever appears in the parallel full run.
Next step when someone picks this up: pin --test-concurrency for the Node 22 job to the runner's core count minus one and see whether it disappears, before touching the test itself. Cost of the status quo is one rerun per push, which trains everyone to re-run a red build instead of reading it.
Update, and a possible connection — stated as a possibility, not a finding.
While chasing this I ran the full suite on Node 22 at forced concurrency levels and hit a different failure at --test-concurrency=2 and =3: floor-acts.test.ts, a real assertion failure, not a runner crash. It turned out to be a genuine bug (liveIntent broke a tie on a millisecond-resolution timestamp, so the earlier act won when two were written in the same millisecond) and is fixed in 6d4a50b.
On the push after that fix, CI went green on both Node 22 and Node 24 — the cli-rules.test.ts deserialisation crash did not appear, after four consecutive appearances.
I do not claim these are causally linked. They are different failure classes: one is an assertion inside a test, the other an uncaught exception in the runner's IPC decoder. One green run after four red ones is not evidence, and a failure that only ever showed up in CI can stay away for unrelated reasons.
What it does mean: the next occurrence is worth waiting for before spending more on this. If it does not come back over the next handful of pushes, this can probably be closed as a symptom of the crash-during-teardown that the floor-acts failure caused. If it does come back, the concurrency lead in the comment above is still the first thing to try.
Wieder aufgetreten: Run 33154594824 (Push 15c8c99, 28.08.). Gleiche Signatur — uncaughtException in #processRawBuffer, „Unable to deserialize cloned data", diesmal als not ok 83 auf cli-rules.test.ts; Node 24 im selben Run grün.
Damit ist die Frage vom 28.08. beantwortet: Der floor-acts-Tiebreak-Fix (6d4a50b) hat damit nichts zu tun — der Crash tritt danach weiter auf. Rerun des Node-22-Jobs angestoßen.
Root cause found, and it is ours — not a runner-side lottery.
node --test pipes only child.stdout into #processRawBuffer; child.stderr goes through a separate readline.Interface and can never corrupt the frame stream. So the # [bastra.embeddings] … lines are ruled out — those are console.error, i.e. stderr.
What is not ruled out: cli-rules.test.ts calls cmdRules in-process, and rules-cmd.ts writes 8 unguarded ✓ … status lines to stdout. ✓ is E2 9C 93. Node's decoder reads the frame length signed (runner.js:334), so a byte >= 0x80 landing at the length offset yields a negative length, the "wait for more data" guard does not fire, and v8 deserialises garbage. That is upstream nodejs/node#64061; the fix (nodejs/node#64706, >>> 0) landed in v24.20.0 and is not in any 22.x, with no backport label — so Node 22 stays exposed.
This also corrects the issue body: the file named is not incidental. cli-rules.test.ts is the only one of our six CLI test files that does not capture process.stdout.write — embeddings-cmd, wizard, bridges-*, install-vault-firstrun and map-daemon-start all do.
Measured on Node 22.18.0 / darwin-arm64, full suite:
arm
IPC crashes
wall-clock
default concurrency
9 / 20
13.5 s
--test-concurrency=1
0 / 10
71.7 s
stdout captured in the test
0 / 20
13.5 s
All 9 crashes named cli-rules.test.ts, each directly after the # ✓ wrote … lines. Fisher exact on 9/20 vs 0/20 is p ≈ 0.0008. The earlier "5/5 green locally" note was simply too small a sample.
So the concurrency lead from the comment above is dropped: it costs 5.3× wall-clock and only suppresses the symptom. Node 24.16.0 ran 15/15 green unguarded — same defect in the code, so not proof of immunity, just a much lower rate under this workload, and 24.20.0 fixes it outright.
Fixed in d3dd732 by capturing stdout in withProject, matching what the other five CLI test files already do. The wider class (26 test files importing CLI modules unguarded, 15 CLI modules emitting non-ASCII status lines) is cut as #414.
Activity
n0mad-ai commentedon Aug 28, 2026
Correcting the claim in the issue body, with three more data points.
I wrote: "the file it names is whichever one happened to be streaming at the time — it is not a property of
cli-rules.test.ts." That does not hold. It has now hitcli-rules.test.tsthree times in a row in CI, on three unrelated pushes (61d9e5a,14a1db2,5cb4200), always on Node 22, always asnot ok 82/83. Whatever the mechanism, this file is involved reproducibly.What has since been ruled out:
fail 1I recorded has not recurred.process.exitanywhere inrules-cmd.tsor in the CLI modules the test imports.What that leaves: something specific to the CI runner — most plausibly its lower core count, since
--test-concurrencydefaults to the CPU count and the failure only ever appears in the parallel full run.Next step when someone picks this up: pin
--test-concurrencyfor the Node 22 job to the runner's core count minus one and see whether it disappears, before touching the test itself. Cost of the status quo is one rerun per push, which trains everyone to re-run a red build instead of reading it.fix(daemon): zwei Affirmations in derselben Millisekunde — die später…
n0mad-ai commentedon Aug 28, 2026
Update, and a possible connection — stated as a possibility, not a finding.
While chasing this I ran the full suite on Node 22 at forced concurrency levels and hit a different failure at
--test-concurrency=2and=3:floor-acts.test.ts, a real assertion failure, not a runner crash. It turned out to be a genuine bug (liveIntentbroke a tie on a millisecond-resolution timestamp, so the earlier act won when two were written in the same millisecond) and is fixed in6d4a50b.On the push after that fix, CI went green on both Node 22 and Node 24 — the
cli-rules.test.tsdeserialisation crash did not appear, after four consecutive appearances.I do not claim these are causally linked. They are different failure classes: one is an assertion inside a test, the other an uncaught exception in the runner's IPC decoder. One green run after four red ones is not evidence, and a failure that only ever showed up in CI can stay away for unrelated reasons.
What it does mean: the next occurrence is worth waiting for before spending more on this. If it does not come back over the next handful of pushes, this can probably be closed as a symptom of the crash-during-teardown that the floor-acts failure caused. If it does come back, the concurrency lead in the comment above is still the first thing to try.
Leaving open and watching.
n0mad-ai commentedon Aug 28, 2026
Wieder aufgetreten: Run 33154594824 (Push 15c8c99, 28.08.). Gleiche Signatur —
uncaughtExceptionin#processRawBuffer, „Unable to deserialize cloned data", diesmal alsnot ok 83aufcli-rules.test.ts; Node 24 im selben Run grün.Damit ist die Frage vom 28.08. beantwortet: Der floor-acts-Tiebreak-Fix (6d4a50b) hat damit nichts zu tun — der Crash tritt danach weiter auf. Rerun des Node-22-Jobs angestoßen.
fix(daemon-tests): cli-rules kapselt stdout — rohe ✓-Zeilen crashten …
n0mad-ai commentedon Aug 28, 2026
Root cause found, and it is ours — not a runner-side lottery.
node --testpipes onlychild.stdoutinto#processRawBuffer;child.stderrgoes through a separatereadline.Interfaceand can never corrupt the frame stream. So the# [bastra.embeddings] …lines are ruled out — those areconsole.error, i.e. stderr.What is not ruled out:
cli-rules.test.tscallscmdRulesin-process, andrules-cmd.tswrites 8 unguarded✓ …status lines to stdout.✓isE2 9C 93. Node's decoder reads the frame length signed (runner.js:334), so a byte >= 0x80 landing at the length offset yields a negative length, the "wait for more data" guard does not fire, and v8 deserialises garbage. That is upstream nodejs/node#64061; the fix (nodejs/node#64706,>>> 0) landed in v24.20.0 and is not in any 22.x, with no backport label — so Node 22 stays exposed.This also corrects the issue body: the file named is not incidental.
cli-rules.test.tsis the only one of our six CLI test files that does not captureprocess.stdout.write—embeddings-cmd,wizard,bridges-*,install-vault-firstrunandmap-daemon-startall do.Measured on Node 22.18.0 / darwin-arm64, full suite:
--test-concurrency=1All 9 crashes named
cli-rules.test.ts, each directly after the# ✓ wrote …lines. Fisher exact on 9/20 vs 0/20 is p ≈ 0.0008. The earlier "5/5 green locally" note was simply too small a sample.So the concurrency lead from the comment above is dropped: it costs 5.3× wall-clock and only suppresses the symptom. Node 24.16.0 ran 15/15 green unguarded — same defect in the code, so not proof of immunity, just a much lower rate under this workload, and 24.20.0 fixes it outright.
Fixed in d3dd732 by capturing stdout in
withProject, matching what the other five CLI test files already do. The wider class (26 test files importing CLI modules unguarded, 15 CLI modules emitting non-ASCII status lines) is cut as #414.fix(tests): the test shim keeps raw stdout off the runner's frame pipe (