Skip to content

run-tests.cjs defaults --test-force-exit on: executed tests are dropped from the report at rc=0, so every reported test count is a lower bound #4031

Closed as duplicate

Description

@0xdhx

Pre-submission checklist

  • I have searched existing issues and this bug has not already been reported.

Closest prior art, all distinct: #1051 / #869 (the Windows hang that motivated the flag), #1054 (the PR that added it), #409 (run-tests.cjs exits 0 for a misspelled --suite — same vacuous-green class, different mechanism), #2455 / #2465 (chunk timeout handling).

GSD Version

1.11.0. scripts/run-tests.cjs is byte-identical at tag v1.11.0 and at next = b811ea16fc4a044dc16b36af91d7cee9d5375727, so this is live for release users, not only on next.

Runtime

Claude Code

Operating System

WSL

Node.js Version

v24.14.1 (also reproduced on v22.22.2)

Shell

/bin/bash

Installation Method

Repo development checkout of next. This is the repository's own test runner (scripts/run-tests.cjs), not the installed package.


What happened?

scripts/run-tests.cjs passes --test-force-exit to every node --test chunk by default. Under that flag, a chunk's tests execute and their results are then silently discarded before the report — the runner prints a smaller pass count and exits 0.

The default:

const forceExit = nodeMajor >= 22 && !process.env.RUN_TESTS_NO_FORCE_EXIT;

const forceExit = nodeMajor >= 22 && !process.env.RUN_TESTS_NO_FORCE_EXIT;

The spawn:

gsd-core/scripts/run-tests.cjs

Lines 1007 to 1014 in b811ea1

execFileSync(
process.execPath,
['--test', ...(forceExit ? ['--test-force-exit'] : []), concurrency, ...chunks[i]],
{
stdio: 'inherit',
env: { ...process.env },
timeout: chunkTimeoutMs,
},

execFileSync(
  process.execPath,
  ['--test', ...(forceExit ? ['--test-force-exit'] : []), concurrency, ...chunks[i]],
  { stdio: 'inherit', env: { ...process.env }, timeout: chunkTimeoutMs },
)

The rationale comment above L961 (L946–L958) attributes the flag to the Windows post-test hang in #1051 (a recurrence of #869) and describes it as exiting "once all known tests have finished". The dropped tests are known — registered synchronously at module load — and finished — they demonstrably executed. So this sits outside the documented semantics of the flag, and it is not a case of the flag doing its job on a leaked handle.

The consequence: every pass/fail count this repository's test runner reports is a lower bound. A green run does not mean the suite ran. It means what arrived at the reporter passed.

What did you expect?

Either every registered, executed test appears in the report, or the runner fails loudly when it cannot account for one. A run that executes a test, discards its result, and exits 0 is indistinguishable from a run in which that test never existed.

Steps to reproduce

A. Minimal synthetic reproduction — no GSD code, no subprocesses, no leaked handles.

This matters because it removes this repository's test content from the causal story entirely: no describe blocks, no asynchronous registration, no child processes, no open handles. Just synchronous top-level test() calls.

mkdir -p /tmp/nodetest-repro && cd /tmp/nodetest-repro
for f in a b c d e f g h i j; do
  { for i in $(seq 1 199); do echo "require('node:test')('${f}_t${i}', () => {});"; done
    echo "require('node:test')('${f}_zz_sentinel', () => { require('node:fs').writeFileSync(process.env.SENTINEL_DIR + '/${f}', 'ran'); });"
  } > "t_${f}.test.cjs"
done
# 10 files x 200 synchronous top-level tests = 2000 registered.
# The last test of each file writes a file to disk, so "did it actually run?" is answerable
# independently of the report.

export SENTINEL_DIR=/tmp/nodetest-repro/sent
rm -rf "$SENTINEL_DIR"; mkdir -p "$SENTINEL_DIR"

# --test-concurrency=4 is what run-tests.cjs uses by default on Linux/macOS (2 on Windows).
node --test --test-force-exit --test-concurrency=4 t_*.test.cjs; echo "rc=$?"
ls "$SENTINEL_DIR"    # every file that actually reached its last test

Compare the reporter's ℹ tests total against 2000, and compare the sentinel names present in the report against the files present in $SENTINEL_DIR.

B. Through this repository's own runner, on a checkout of next:

node scripts/run-tests.cjs --files-from <a file list>          # force-exit on  (default)
RUN_TESTS_NO_FORCE_EXIT=1 node scripts/run-tests.cjs --files-from <same list>   # off

Error output / logs

Synthetic case A, Node v24.14.1, Linux, 2000 registered tests. Every run below exited 0 with fail 0 and empty stderr:

invocation                                              runs truncated   worst reported
--test-force-exit --test-concurrency=4  (Linux/macOS default)   15 / 15   1644 / 2000
--test-force-exit --test-concurrency=2  (Windows default)       14 / 15   1803 / 2000
--test-force-exit --test-concurrency=1                           6 / 32   1909 / 2000
no flag,          --test-concurrency=4                           0 / 15   2000 / 2000
no flag,          --test-concurrency=1                           0 / 20   2000 / 2000

At the concurrency the runner actually uses, this is not intermittent — it happened on every run.

The results are lost after the tests execute. Three consecutive runs at the default concurrency, checking each file's on-disk sentinel against the report:

run 1: rc=0  reported 1913/2000  fail 0  stderr empty
   tail sentinel missing from report for files: a b
   ... all of which had WRITTEN their sentinel file to disk

run 2: rc=0  reported 1676/2000  fail 0  stderr empty
   tail sentinel missing from report for files: a b c d e f g h j
   ... all nine had WRITTEN their sentinel file to disk

run 3: rc=0  reported 1923/2000  fail 0  stderr empty
   tail sentinel missing from report for files: f g
   ... both had WRITTEN their sentinel file to disk

In run 2, nine of ten files ran to their final test and the reporter accounted for none of those nine.

Shape of the loss: contiguous per-file tails, several files per run. Diffing reported test names against a clean run, one truncated run lost exactly tests 145–200 of one file and 166–200 of another — 91 results, every one a contiguous run to end-of-file.

On this repository's real suite (seven test files, 925 tests, 182 suites), same machine:

node scripts/run-tests.cjs (force-exit on, default)      rc=0  tests 863, 844, 912
same + RUN_TESTS_NO_FORCE_EXIT=1                         rc=0  tests 925, 925
node --test --test-force-exit --test-concurrency=1 $FILES rc=0  tests 884 / 906 / 922 / 925
node --test                   --test-concurrency=1 $FILES rc=0  tests 925
node22 --test --test-force-exit --test-concurrency=1 $FILES  rc=0  tests 810   (v22.22.2)

The tests that vanished there were also a contiguous tail: the last 41 tests of a file with zero describe blocks and 109 synchronous top-level test() registrations.

How often does this happen?

Every time (100% reproducible) at the runner's default concurrency on Linux/macOS in the synthetic case; the number of results lost varies run to run.

Impact

Moderate — Feature is broken but I have a workaround.

The runner still catches failures it receives, so this is not a blocker. It is a correctness hole in the reporting itself: a count can silently shrink, and CI has no way to notice.

Workaround (if any)

RUN_TESTS_NO_FORCE_EXIT=1 restores complete reporting (925/925 and 2000/2000, every run). It is not a fix — it reintroduces the ~150 s per-chunk Windows post-test hang that #1051/#1054 added the flag to solve.

Additional context

What this is not arguing. Not that adopting --test-force-exit was wrong — the Windows hang was real and the flag fixed it. Not that any test in this repository is faulty; the synthetic reproduction has no GSD code in it at all. And not, yet, a Node bug report: the root cause is in Node's test runner, and while the reproduction above is now minimal enough to carry one, no nodejs/node issue has been filed. If one is, it should link here.

Suggested fixes, ordered by how much of the #1051 fix they preserve:

  1. Per-chunk registered-vs-reported cross-check. The runner already knows each chunk's file list. Comparing a count of top-level test(/it(/describe( registrations per file — or, more robustly, a second pass parsing --test-reporter=tap for # Subtest: per file — against the reporter's per-file totals turns a dropped tail into a loud failure rather than a smaller number. Keeps force-exit everywhere.
  2. Scope the flag to the platform that needed it. forceExit = process.platform === 'win32' && nodeMajor >= 22 && !process.env.RUN_TESTS_NO_FORCE_EXIT removes the exposure from the Linux/macOS lanes at no cost to them, since the only documented downside of running without it is the Windows-specific hang. Windows keeps the exposure — which is why this pairs well with (1) rather than replacing it.
  3. Re-run a chunk whose reported total moved. Weakest: detects only nondeterminism between two runs of the same chunk, and doubles wall-clock on the retry.

Given that the loss is ~100% at the shipped default concurrency rather than a rare flake, (2) applied now with (1) behind it seems the strongest combination — but the choice of shape is the maintainers'.

Explicitly out of scope here. Whether --test-force-exit can also swallow a failing result. With the sentinel changed to assert.fail(...), four runs all reported the failure and exited 1, so no truncated-with-failure run was observed. That is 0-for-4 — a hint that a failing child takes a different exit path, not evidence. What is established is that passing results are silently dropped, which is enough to make every reported count a lower bound.

Activity

  1. 0xdhx commented on Aug 29, 2026

    @0xdhx
    ContributorAuthor

    Correction to my own report, and it changes the calculus here.

    The issue body says "no nodejs/node issue has been filed." That is wrong, and it was wrong when I wrote it. The root cause is already filed as nodejs/node#64833 — opened 2026-07-30, a month before this issue, describing the same defect in the same terms: --test-force-exit with --test-concurrency > 1 silently loses verdicts and still exits 0. I missed it on my first search; apologies for the noise.

    The root cause is established there, and it is more specific than what I wrote. Under process isolation each test file's child streams its results to the parent over a pipe. The force-exit flush loop added in nodejs/node#55099 only drains reporter destinations that expose a close() method — a pipe has none, so that branch resolves immediately without flushing and the parent process.exit()s with bytes still buffered.

    That corrects a claim I made above. I had concluded this was not a flush problem because the truncation reproduces with the parent's stdout going to a file and to a pty, not only to a pipe. That reasoning was about the wrong pipe: the parent's own stdout destination is irrelevant, because the child-to-parent channel is a pipe in every case under process isolation.

    One datum from my testing that supports the mechanism and that I did not see in that thread — removing the child-process boundary removes the loss entirely:

    node --test --experimental-test-isolation=none    --test-force-exit  ...   2000/2000, 5 runs, no loss
    node --test --experimental-test-isolation=process --test-force-exit  ...   truncated on every run
    

    Same files, same flag, same machine. With no child process there is no pipe to leave unflushed.

    Two competing fixes are open upstream, both REVIEW_REQUIRED as of 2026-08-29:

    What this means for this issue. The exposure described above is unchanged and stays live until a Node fix ships and this repo's floor moves onto it — so this is not a "close it, upstream has it" situation. But it does reprice the suggested fixes: option 2 (scoping forceExit to win32) is unaffected and still removes the Linux/macOS exposure today at no cost, while option 1 (the registered-vs-reported cross-check) is now a question of whether it is worth carrying a detector for a defect with a fix already in review rather than an indefinite one. That trade is yours; I have no view on your release cadence.

  2. trek-e commented on Aug 29, 2026

    @trek-e
    Collaborator

    This was generated by AI during triage.

    Diagnosis

    Reproduced: Verified statically only (did not re-run the synthetic repro). Read the current scripts/run-tests.cjs at next and confirmed the mechanism exactly: forceExit (line 1074) has no process.platform gate and, since engines.node now requires >=24.0.0 (per the code's own comment), it is effectively unconditional — true for every supported Node version and platform unless RUN_TESTS_NO_FORCE_EXIT=1 is set. The spawn (line 1213) unconditionally splices --test-force-exit into every chunk's argv when forceExit is true. This matches the issue's claim precisely. Your own follow-up comment supplies independent, external confirmation of the exact mechanism: nodejs/node#64833, with two competing upstream fixes (#64857, #64875) both still REVIEW_REQUIRED.

    Root cause: scripts/run-tests.cjs:1074 (the forceExit computation, inside main()) and scripts/run-tests.cjs:1213 (the execFileSync argv splice). No platform gate; the flag is unconditionally applied. Given the confirmed upstream defect, this silently truncates the reporter's count of already-executed, already-passing tests while still exiting 0.

    Introduced by: commit 4698b3e349, fix(#1051): force-exit + per-chunk timeout for the windows full-test lane; close leaked test handles (#1054), 2026-06-11. git blame shows the line unchanged since — long-standing, not a recent regression. The five most recent commits touching this file (fix(#4012), chunk-hang instrumentation / ndjson reporter) are orthogonal — they diagnose timeout kills and never touch the forceExit computation, so they neither fix nor mask this.

    Blast radius: High. Affects every gsd-test / scripts/run-tests.cjs invocation on every supported Node version and platform (not just Windows) — every chunked run's reported pass/fail count, and by extension every green verdict this repo's own push/PR gates rely on, is potentially a lower bound rather than an exact count. Silent: no crash, no nonzero exit, fail 0.

    Coupled: None found via graph co-change query (MemDB backend was unavailable this session — embedded owner had exited); git log shows only the orthogonal #4012 work touching this file recently.

    Regression test: Author N synthetic .test.cjs files with many synchronous top-level test() calls each, last one writing a sentinel file to disk; run through the runner's actual chunk-spawn path at its default concurrency and assert (a) reported total equals the exact registered count, and (b) every on-disk sentinel has a corresponding "passed" entry in the report. Expected to fail today per the issue's own data; a fix (e.g. scoping forceExit to win32, and/or a registered-vs-reported cross-check) should turn it green or loud rather than silently short.

  3. trek-e commented on Aug 29, 2026

    @trek-e
    Collaborator

    Agent Brief

    Category: Bug — silent test-result loss (test infrastructure).

    Summary: This repo's own test runner (scripts/run-tests.cjs) can silently drop already-executed, already-passing tests from its reported count while still exiting 0, because it unconditionally passes --test-force-exit to every node --test chunk on every platform, and Node's --test-force-exit has a confirmed upstream defect (nodejs/node#64833) that loses buffered results under process isolation with concurrency > 1.

    Current behavior: Every chunked test run (i.e. every gsd-test / run-tests.cjs invocation, on any platform, on any currently-supported Node version) force-exits after all tests finish, regardless of platform. At the runner's default concurrency, a contiguous tail of a chunk's tests can execute, pass, and write their side effects, yet never appear in the reporter's total — and the run still reports fail 0 and exits 0. There is no signal, loud or quiet, that a result was dropped.

    Desired behavior: Either (a) the reported test count is always the exact number of registered/executed tests — the runner fails loudly if it cannot account for one — or (b) the exposure to the underlying Node defect is scoped down to only the platform that actually needs --test-force-exit (the reporter's own history says this was added for a Windows-specific post-test hang), so unaffected platforms stop paying the truncation risk at all. A fix does not need to remove --test-force-exit entirely — the Windows hang it defends against is real.

    Key interfaces: scripts/run-tests.cjs — the forceExit local (computed once in main()) and the execFileSync chunk-spawn argv that conditionally includes --test-force-exit; the RUN_TESTS_NO_FORCE_EXIT environment-variable escape hatch already exists and its existing semantics (used by this repo's own hang-regression test) must be preserved by any fix.

    Acceptance criteria:

    Out of scope: Fixing the upstream Node defect itself (tracked at nodejs/node#64833 with two candidate patches under review) — this repo's fix should be defensive/scoping, not a Node core patch. Also out of scope: the #4012 chunk-timeout/ndjson-reporter diagnostic feature (unrelated mechanism — that's for timeout kills, not result-count integrity) and whether --test-force-exit can also swallow a failing result (reporter's own testing found no evidence of that; only passing results were observed dropped).

  4. added
    bugSomething isn't working
    and removed
    needs-triageNew issue awaiting maintainer review
    on Aug 29, 2026
  5. trek-e commented on Sep 11, 2026

    @trek-e
    Collaborator

    Closing as a duplicate of #4631 — consolidating at the seam rather than fixing per-symptom.

    This is not a dismissal of the report. Your specifics are preserved verbatim in that epic's Absorbed issues table, and its Done when checklist requires a failing-first regression test for this exact defect before the epic can close.

    Why it is being consolidated: a scan of all 69 open confirmed-bug issues (2026-09-11) found this one shares a single root cause with seven other open issues — a gate hand-rolls a narrow derivation of what it should evaluate, the derivation comes back empty, and "empty" is reported as "pass". Fixing each occurrence where it was found leaves the remaining call sites carrying the same bug, which is the pattern that produced this cluster in the first place.

    The epic delivers one resolveEvaluationScope(), evidence read through the existing document seams instead of per-gate regex, and an Evidence type that distinguishes "none" from "could not read" so a gate can no longer pass vacuously, plus a machine-enforced anti-divergence ratchet so a new hand-written copy cannot be introduced — the approach that worked for #1372 (markdown sectionizer), #2143 (table + bounded mutation) and #2121 (phase-id), each of which closed its bug class structurally.

    Track the fix at #4631.

  6. added
    duplicateThis issue or pull request already exists
    on Sep 11, 2026
  7. added 8 commits that reference this issue on Oct 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingconfirmed-bugVerified reproducible bugduplicateThis issue or pull request already exists

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions