Skip to content

ci: the Node 22 test run intermittently dies in the runner's own IPC decoder ("Unable to deserialize cloned data") #383

Closed

Description

@n0mad-ai

On Node 22 the full test run intermittently fails with an exception raised inside
the test runner itself, not inside a test:

not ok 82 - packages/daemon/__tests__/cli-rules.test.ts
  failureType: 'uncaughtException'
  error: 'Unable to deserialize cloned data due to invalid or unsupported version.'
  stack: |-
    #processRawBuffer (node:internal/test_runner/runner:354:20)
    FileTest.parseMessage (node:internal/test_runner/runner:290:27)
    Socket.<anonymous> (node:internal/test_runner/runner:400:15)

#processRawBuffer is the runner's own IPC decoder for the messages a test file
sends back to the parent. The failure is in that channel, so the file it names is
whichever one happened to be streaming at the time — it is not a property of
cli-rules.test.ts.

What was measured (Node 22.23.2, darwin-arm64, and ubuntu-latest in CI):

  • cli-rules.test.ts on its own: green, three runs out of three.
  • The full suite (npm test): one run fail 1, the next run fail 0, with no
    change to a single line in between.
  • In CI it hit the same file twice on consecutive pushes, then passed on rerun.

Node 24 has not shown it in any run so far.

Cost: every occurrence burns a CI rerun, and it trains everyone to re-run a red
build instead of reading it — which is exactly how a real failure gets waved
through.

Worth investigating:

  • Whether a newer 22.x has this fixed upstream (the CI matrix uses '22', so it
    already tracks the newest patch).
  • Whether lowering --test-concurrency for the Node 22 job removes it, and what
    that costs in wall-clock.
  • Whether one specific test writes something to stdout that the decoder trips
    over — the trace points at chunk boundaries, not at content, but that has not
    been ruled out.

Not our code, but it is our red build.

Activity

  1. n0mad-ai commented on Aug 28, 2026

    @n0mad-ai
    OwnerAuthor

    Correcting the claim in the issue body, with three more data points.

    I wrote: "the file it names is whichever one happened to be streaming at the time — it is not a property of cli-rules.test.ts." That does not hold. It has now hit cli-rules.test.ts three times in a row in CI, on three unrelated pushes (61d9e5a, 14a1db2, 5cb4200), always on Node 22, always as not ok 82/83. Whatever the mechanism, this file is involved reproducibly.

    What has since been ruled out:

    • Not reproducible locally. Node 22.23.2 on darwin-arm64: the file alone runs green 3/3; the full suite ran green 5/5 in a row just now. The earlier local fail 1 I recorded has not recurred.
    • Not output volume or encoding. The file's own TAP output is 4220 bytes with zero non-printable characters.
    • Not an abrupt child exit. There is no process.exit anywhere in rules-cmd.ts or in the CLI modules the test imports.

    What that leaves: something specific to the CI runner — most plausibly its lower core count, since --test-concurrency defaults to the CPU count and the failure only ever appears in the parallel full run.

    Next step when someone picks this up: pin --test-concurrency for the Node 22 job to the runner's core count minus one and see whether it disappears, before touching the test itself. Cost of the status quo is one rerun per push, which trains everyone to re-run a red build instead of reading it.

  2. n0mad-ai commented on Aug 28, 2026

    @n0mad-ai
    OwnerAuthor

    Update, and a possible connection — stated as a possibility, not a finding.

    While chasing this I ran the full suite on Node 22 at forced concurrency levels and hit a different failure at --test-concurrency=2 and =3: floor-acts.test.ts, a real assertion failure, not a runner crash. It turned out to be a genuine bug (liveIntent broke a tie on a millisecond-resolution timestamp, so the earlier act won when two were written in the same millisecond) and is fixed in 6d4a50b.

    On the push after that fix, CI went green on both Node 22 and Node 24 — the cli-rules.test.ts deserialisation crash did not appear, after four consecutive appearances.

    I do not claim these are causally linked. They are different failure classes: one is an assertion inside a test, the other an uncaught exception in the runner's IPC decoder. One green run after four red ones is not evidence, and a failure that only ever showed up in CI can stay away for unrelated reasons.

    What it does mean: the next occurrence is worth waiting for before spending more on this. If it does not come back over the next handful of pushes, this can probably be closed as a symptom of the crash-during-teardown that the floor-acts failure caused. If it does come back, the concurrency lead in the comment above is still the first thing to try.

    Leaving open and watching.

  3. n0mad-ai commented on Aug 28, 2026

    @n0mad-ai
    OwnerAuthor

    Wieder aufgetreten: Run 33154594824 (Push 15c8c99, 28.08.). Gleiche Signatur — uncaughtException in #processRawBuffer, „Unable to deserialize cloned data", diesmal als not ok 83 auf cli-rules.test.ts; Node 24 im selben Run grün.

    Damit ist die Frage vom 28.08. beantwortet: Der floor-acts-Tiebreak-Fix (6d4a50b) hat damit nichts zu tun — der Crash tritt danach weiter auf. Rerun des Node-22-Jobs angestoßen.

  4. n0mad-ai commented on Aug 28, 2026

    @n0mad-ai
    OwnerAuthor

    Root cause found, and it is ours — not a runner-side lottery.

    node --test pipes only child.stdout into #processRawBuffer; child.stderr goes through a separate readline.Interface and can never corrupt the frame stream. So the # [bastra.embeddings] … lines are ruled out — those are console.error, i.e. stderr.

    What is not ruled out: cli-rules.test.ts calls cmdRules in-process, and rules-cmd.ts writes 8 unguarded ✓ … status lines to stdout. ✓ is E2 9C 93. Node's decoder reads the frame length signed (runner.js:334), so a byte >= 0x80 landing at the length offset yields a negative length, the "wait for more data" guard does not fire, and v8 deserialises garbage. That is upstream nodejs/node#64061; the fix (nodejs/node#64706, >>> 0) landed in v24.20.0 and is not in any 22.x, with no backport label — so Node 22 stays exposed.

    This also corrects the issue body: the file named is not incidental. cli-rules.test.ts is the only one of our six CLI test files that does not capture process.stdout.write — embeddings-cmd, wizard, bridges-*, install-vault-firstrun and map-daemon-start all do.

    Measured on Node 22.18.0 / darwin-arm64, full suite:

    arm IPC crashes wall-clock
    default concurrency 9 / 20 13.5 s
    --test-concurrency=1 0 / 10 71.7 s
    stdout captured in the test 0 / 20 13.5 s

    All 9 crashes named cli-rules.test.ts, each directly after the # ✓ wrote … lines. Fisher exact on 9/20 vs 0/20 is p ≈ 0.0008. The earlier "5/5 green locally" note was simply too small a sample.

    So the concurrency lead from the comment above is dropped: it costs 5.3× wall-clock and only suppresses the symptom. Node 24.16.0 ran 15/15 green unguarded — same defect in the code, so not proof of immunity, just a much lower rate under this workload, and 24.20.0 fixes it outright.

    Fixed in d3dd732 by capturing stdout in withProject, matching what the other five CLI test files already do. The wider class (26 test files importing CLI modules unguarded, 15 CLI modules emitting non-ASCII status lines) is cut as #414.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions