Skip to content

test/editor-patch.test.ts dies silently mid-file under the parallel suite, twice in ~15 runs #405

Closed

Description

@bbertucc

Iris Maintainer Agent here.

test/editor-patch.test.ts has died twice under the full npm test run, in a way that no assertion explains. Filing it because a silent non-zero exit is the one failure mode that can also hide a real one, and because it is rare enough that the next person to see it will assume it is a flake and move on.

What it looks like

✖ failing tests:

test at test/editor-patch.test.ts:1:1
✖ test/editor-patch.test.ts (1585.773916ms)
  'test failed'

'test failed' with no assertion, no diff, and no stderr at all — the run was captured with 2>&1 and a grep for Error|error:|stderr|Warning across the whole log matched nothing from that file. That message is what the runner reports when a test file's process exits non-zero without having reported a failure, so the child died rather than failed.

It died part-way through. The last line it printed was a block that gave its IMAGE to another block is not re-seated either, which is test 40 of 47, and the totals show 1550 registered against 1558 — the ~8 tests after that point were never registered.

Observations so far

Seen on branch retire-prd-399 (once), branch docs-concise-400 (once)
File alone (node --test test/editor-patch.test.ts) 47/47 pass
Full suite, repeated on docs-concise-400 9 consecutive clean runs, 1558–1559/1559, exit 0, empty stderr

So it is roughly 2 in ~15 observed full-suite runs and does not reproduce on demand. It appeared before any of the #400 work existed, and neither branch touched the editor pipeline or this test, so it is not either branch's diff.

What I ruled out

  • Shared temp state. The file's harness takes mkdtempSync(join(tmpdir(), "iris-patch-")) per call and removes it in a finally, so two parallel files cannot collide on a path.
  • An explicit exit. No process.exit anywhere in the file.

What it is probably worth looking at

node --test runs files in parallel up to the core count (the passing runs sit around 434% CPU), so the candidates are resource pressure or a signal — both of which produce exactly this signature: a dead child, no output. An unhandled rejection would normally print, which argues against it, but a rejection raised inside the harness's router.complete callback after its test has already settled is the one shape worth checking, since assert.ok(shown, …) throws in there.

Two cheap things that would settle it, neither of which needs the failure to be reproducible on demand:

  1. Run the suite with --test-reporter=tap, which prints the child's exitCode and signal in the YAML diagnostic. The spec reporter throws both away, which is why two occurrences have told us nothing. (I've had a loop doing this on docs-concise-400; 8 runs, no failure, so the diagnostic is still uncaptured.)
  2. If it turns out to be a signal under memory pressure, --test-concurrency is the knob, and CI is where it will bite — a silent child death there reads as a normal red build with an unhelpful message.

Noted on #404, which is where I hit it the second time.

Activity

  1. bbertucc commented on Sep 5, 2026

    @bbertucc
    MemberAuthor

    Iris Maintainer Agent here.

    Settled, and it is not this test file. It is a SIGSEGV inside node's own garbage collector. PR #409.

    The signature was reproducible without reproducing the failure. A throwaway file that kills itself mid-run renders under spec as, character for character, what this issue reports:

    ✖ /tmp/diagprobe/dies.test.ts (105.300666ms)
      'test failed'
    

    The same run under tap:

    not ok 1 - /tmp/diagprobe/dies.test.ts
      failureType: 'testCodeFailure'
      exitCode: ~
      signal: 'SIGKILL'
    

    So 'test failed' with no assertion and no stderr is the runner's report of a child that died without reporting — and spec throws away the two fields that say which. That closes the "no assertion explains it" part of this issue: nothing was hidden, the reporter discarded it. Also worth correcting from the issue body: spec is the default even when stdout is not a TTY (checked by piping through cat), so CI has been losing it too.

    What died. ~/Library/Logs/DiagnosticReports holds six node-*.ips crash reports over the four days here — Sep 1 12:45, Sep 3 14:31, Sep 4 13:48, 15:51, 18:53, 19:14 — every one a SIGSEGV in a child of node, all with the same stack and the same frame offsets:

    v8::internal::ClearStaleLeftTrimmedPointerVisitor::VisitRootPointers
    v8::internal::InternalFrame::Iterate → Isolate::Iterate → Heap::IterateRoots
    v8::internal::MarkCompactCollector::MarkRoots → MarkLiveObjects → CollectGarbage
    

    Faulting on 0xe and 0x6. Those are tagged Smis (7 and 3), so V8 is treating a frame slot holding a small integer as a heap pointer while marking roots. Node v24.16.0, macOS 26.3.1.

    Each guess in this issue, against evidence rather than argument:

    • Resource pressure / a signal from outside — no. The machine's seven JetsamEvent-*.ips (08-29 02:34 … 09-04 06:56) coincide with none of the six crashes. macOS did not kill it; it crashed.
    • A rejection inside router.complete — no. That exits non-zero with output. This takes SIGSEGV with a native stack and no JS frames.
    • Our own left-trimming — no, and this was the one lead that could have pointed back at Iris. ClearStaleLeftTrimmedPointerVisitor only has work after an array is left-trimmed, which in V8 means Array.prototype.shift. The entire tree has one .shift() — src/util/queue.ts:95 — and test/editor-patch.test.ts does not reach it (util/queue is imported only by src/util/requestLimits.ts, src/routes/sessions.ts and test/queue.test.ts). The trim is node's own.
    • --test-concurrency as the knob — plausible as a mitigation, but it would be tuning against a crash we cannot yet attribute to a run. Not doing it on this evidence.

    What I am not claiming. These six reports are not tied to the two failures in this issue — nothing retained links a crash report to a suite run, and four of the six fall in windows where the suite reported all-pass, so at least some belong to other node processes. The mechanism is certain; the attribution of any single occurrence is not. Which is exactly why the fix is to make the next one name itself rather than to chase this one.

    So #409 does two things and stops there: npm test now prints signal/exitCode when a child dies, at the failure and again as the last line of the run (both CI workflows read npm test through tail, -120 and -40, so an early death would otherwise be above the cut); and CONTRIBUTING.md plus the issue-to-pr.yml prompt stop telling people npm test is a bare node --test, which would route around it.

    It leaves the crash upstream where it belongs. If it recurs after #409 it will arrive with SIGSEGV on the line, and a fresh .ips to attach to a node report — which is the thing this issue could not produce in ~17 runs of trying.

    Two corrections to my own reading in the body above: I wrote that the ~8 tests after the last printed one "were never registered". They were registered — registration is synchronous at module load — they were never run, which is why the pass count came out short rather than the suite going red on them. And "roughly 2 in ~15 observed full-suite runs" is a rate for observed suite failures, not for the crash; the crash reports say the segfault happens more often than the suite notices.

  2. added a commit that references this issue on Sep 5, 2026
    013b6e2
  3. bbertucc commented on Sep 9, 2026

    @bbertucc
    MemberAuthor

    Iris Maintainer Agent here.

    Reopening: the reporter this issue asked for has caught it, and the answer is not a test change. The children die of SIGSEGV inside V8's garbage collector.

    It happened again on run 5 of a 10-run npm test loop this morning (worktree-strip-boundary-439-440, tree clean, PR #444's merged head), and test/spec-with-signals.mjs printed the diagnostic the spec reporter used to throw away:

    ℹ tests 1641
    ℹ pass 1639
    ℹ fail 2
    
    ✖ failing tests:
    
    test at test/editor-patch.test.ts:1:1
    ✖ test/editor-patch.test.ts (1615.239708ms)
      'test failed'
    
    test at test/editor-salvage.test.ts:1:1
    ✖ test/editor-salvage.test.ts (2072.226792ms)
      'test failed'
    
    ‼ dead child processes in this run (see #405):
      - test/editor-patch.test.ts: the process was killed by SIGSEGV without reporting a failure
      - test/editor-salvage.test.ts: the process was killed by SIGSEGV without reporting a failure
    

    Same signature as the original report: 'test failed' with no assertion, and 1,641 of 1,649 registered — the ~8 tests after the point of death never existed. editor-patch.test.ts is the file this issue was filed about; editor-salvage.test.ts is new, and it died in the same run.

    The crash reports, which say where

    macOS wrote one .ips per dead child, both at the same second, and they are identical:

    files ~/Library/Logs/DiagnosticReports/node-2026-09-09-075605{,.000}.ips
    pids 67014 and 67016
    exception EXC_BAD_ACCESS (SIGSEGV), KERN_INVALID_ADDRESS at 0x000000000000000e
    launched / crashed 07:55:59 / 07:56:05 — six seconds in
    node / OS v24.16.0, macOS 26.3.1 (25D771280a), ARM-64

    The faulting stack, top down:

    v8::internal::ClearStaleLeftTrimmedPointerVisitor::VisitRootPointers(...)
    v8::internal::InternalFrame::Iterate(v8::internal::RootVisitor*) const
    v8::internal::Isolate::Iterate(...)
    v8::internal::Heap::IterateRoots(...)
    v8::internal::MarkCompactCollector::MarkRoots(...)
    v8::internal::MarkCompactCollector::MarkLiveObjects()
    v8::internal::MarkCompactCollector::CollectGarbage()
    v8::internal::Heap::MarkCompact()
    ...
    v8::internal::StackGuard::HandleInterrupts(...)
    

    A mark-compact GC, entered from a stack-guard interrupt, dereferencing 0xe while iterating roots. Every frame is in the node image — the only other images in the report are dyld and two libsystem stubs, and the JS below the interrupt is generic trampolines. So neither test file is doing anything the crash can be attributed to: they died because they were the two processes GC happened to be running in.

    That resolves the two candidates this issue named. It is not an unhandled rejection in the harness, and it is not a kill under memory pressure — a SIGKILL is what that would look like. It is a V8 bug, reached under the parallelism node --test uses.

    Rate

    this morning 1 of 10 loop runs, plus 2 earlier full runs the same hour, so 1 of 12
    this issue's original count 2 in ~15
    node --test test/editor-patch.test.ts alone 47/47, as before

    Consistent at roughly 1 run in 8–12, and still not reproducible on demand.

    What I am NOT doing

    No test change, because there is no honest one. Lowering --test-concurrency would make it rarer without making it not happen, and it would slow every clean run to hide a crash that is not ours. The decisions this leaves are a human's: bump or pin Node (v24.16.0 is what crashed; the symbol above is the thing to search a Node/V8 changelog for), and decide whether CI should retry a run whose only failures are dead children — in CI this reads as a normal red build, which is the part that costs someone an afternoon.

    What #405's reporter bought is that the next occurrence anywhere, including CI, names the signal in the log instead of 'test failed'. That part worked exactly as intended, which is why this is worth reopening rather than refiling.

  4. bbertucc commented on Sep 11, 2026

    @bbertucc
    MemberAuthor

    Iris Maintainer Agent here.

    One more dated occurrence, since the rate is the only thing still accumulating on this issue.

    Hit it today (2026-09-11) on worktree-cleanup-pass while running the gates for #461's round 3. Same signature, same file, and the reporter named it without being asked:

    ✖ test/editor-patch.test.ts (1704.060916ms)
      'test failed'
    
    ‼ dead child process in this run (see #405):
      - test/editor-patch.test.ts: the process was killed by SIGSEGV without reporting a failure
    

    One child this time, not two. Node v24.16.0, macOS 26.3.1 — unchanged from the 2026-09-09 report, so nothing about the environment has moved.

    The diff it happened on is two comments (src/config.ts and test/review-cap.test.ts), which is as strong a "not the branch's code" control as this failure gets. The re-run was 1694/1694, exit 0.

    Rate, added to what is already here: 1 of 4 full-suite runs in this session's window (three earlier runs today were clean). That keeps it in the same 1-in-8-to-12 band across a bigger denominator rather than moving it.

    Nothing new to diagnose — the V8 GC stack in the comment above is still the answer, and the two open decisions are still a human's: whether to bump or pin Node off v24.16.0, and whether CI should retry a run whose only failures are dead children. Recording this only so the next reader can see the frequency has not changed and the diagnostic keeps working outside a deliberate loop.

  5. bbertucc commented on Sep 14, 2026

    @bbertucc
    MemberAuthor

    Iris Maintainer Agent here.

    Closing this. The crash has an upstream name, and the mitigation shipped in #477 (72bd975).

    What it is

    nodejs/node#62393, fixed in V8 by CL
    0b94a9fd23ba: "[leaptiering] Fix BaselineOutOfLinePrologue builtin … which tried to preserve
    kJavaScriptCallDispatchHandleRegister even on configurations where it's not used which resulted in a
    random value on the stack discoverable by GC."

    That is this issue's stack exactly. BaselineOutOfLinePrologue is Sparkplug's prologue; the stray
    register value is a small integer; ClearStaleLeftTrimmedPointerVisitor reads it as a heap pointer
    while marking roots and faults at 0xe — the address in every report here. Nothing in
    test/editor-patch.test.ts was ever wrong, which is why the file kept changing and the deaths kept
    happening.

    The fix

    npm test now runs with --no-sparkplug. No Sparkplug code means that builtin is never entered,
    so the path is gone. It costs nothing measurable: 55.8 s mean either way over two full runs each here,
    and the reviewer measured ~3% on Linux x64 over the two heaviest jsdom files, against a 22-minute
    review step. This suite is jsdom- and I/O-bound; the baseline tier was buying it very little.

    The flag is on the test script only, deliberately — npm start and npm run dev keep the Sparkplug
    path, because one dev server dying of SIGSEGV is loud, where a dead test child reads as a clean run
    with a short pass count. That asymmetry is the whole reason this issue was hard.

    A test pins the flag and its position. node --test "test/*.test.ts" --no-sparkplug exits 0, warns
    about nothing, and the child's execArgv does not carry the flag — anything after the positional goes
    to the runner, not to V8. So the pin asserts --no-sparkplug comes before --test.

    test/spec-with-signals.mjs stays. It is what named this crash instead of costing an afternoon, and it
    is what would catch the next dead child of any cause — including this one, if the flag is ever dropped
    early.

    The two decisions this issue was holding — both are no

    • Do not bump or pin Node. The backport is
      open, not landed — rebased onto v24.x-staging
      after v24.21.0, CI green, waiting. No released 24.x has the fix, and one upstream report has the
      crash live on v26.7.0, which corrects the earlier "fixed in 25.0.0" reading. .nvmrc stays at
      24. Separately: this machine runs v24.16.0 while .nvmrc gives CI v24.21.0. Updating
      locally is ordinary hygiene, not a fix for this.
    • CI needs no dead-child retry. Every occurrence — the five local reports and every report
      upstream — is macOS arm64, and every workflow here runs ubuntu-latest. Retrying a failure never
      seen on that platform would be masking on speculation, and with Sparkplug off there is nothing left
      to retry.

    One correction to the rate on this issue

    Ten node-*.ips reports are retained on this machine. Five carry this stack — 09-08 21:48,
    09-09 07:56, 09-09 13:02, 09-11 15:23, 09-13 20:19. The other five are a different failure:
    SIGABRT out of v8::internal::Heap::FatalProcessOutOfMemory, i.e. heap exhaustion, clustered in
    pairs and triples. Worth separating so the next reader does not fold them into this crash's frequency.

    When to undo this

    Drop --no-sparkplug and its pin test together when nodejs/node#65753 ships in a 24.x release.
    That removal condition is written where the flag is, in
    CONTRIBUTING.md § Development.
    Reopen this if a dead child appears with the flag on — that would be a different bug, and the reporter
    will say so.

    Thanks for keeping the dated occurrences coming on this one; the rate is what made it worth chasing to
    upstream rather than retrying around.

    🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions