Repository navigation
Description
What happened?
The CI job full test (windows-latest, 22) (the test-full job in .github/workflows/test.yml, timeout-minutes: 20) intermittently gets CANCELLED at the 20-minute wall-clock cap (conclusion=cancelled, with no failed test step). This red-blocks otherwise-green PRs — it is a false-negative gate.
This is a recurrence of #869 (which raised the cap 15→20m as a stopgap; the durable fix was deferred). The job is now hitting the new 20m cap.
What did you expect?
The Windows unit lane should finish well under the cap, and a genuine hang should fail with a clear diagnostic — never silently consume the wall-clock budget and surface as an opaque cancelled.
Steps to reproduce
- Open any PR that triggers the
test-fullmatrix. - Observe
full test (windows-latest, 22)(npm run test:unit→scripts/run-tests.cjs --suite unit). - Intermittently the job is CANCELLED at 20m with no failed test step. Re-running the identical commit frequently passes (~15m).
Canonical evidence — PR #1047, run 27362003027, job 80851641619 (2026-06-11):
scripts/run-tests.cjs splits the ~656 unit files into chunks (Windows CreateProcess 32,767-char command-line cap, #3597) and runs each chunk sequentially as execFileSync(process.execPath, ['--test', concurrency, ...chunkFiles]) with no per-chunk timeout.
The overrun is NOT uniform slowness and NOT a single slow test (every individual test runs <1s). It is two windows of ~150s and ~146s of pure silence at chunk boundaries:
- Gap 1: 16:39:30 → 16:42:00 (150s), immediately after
ok 613 - _applyRuntimeRewrites(trae) — install-path bare-form (#983). - Gap 2: 16:46:30 → 16:48:56 (146s), immediately after
ok 347 - execute-plan.md routing instruction (#1979).
After each silence the NEXT chunk's node --test prints the GSD banner — i.e. the silence is the previous chunk's child process failing to EXIT for ~150s after its last test printed. Baseline unit step is already 11m54s–13m44s, so ≥2 such stalls push the job past 20m → cancel.
Root cause (diagnosed)
A unit test leaves an open event-loop handle, so node --test will not exit until the handle drains; on Windows that drain takes ~150s. Confirmed leak sources:
tests/perf-316-state-lock-buffer-alloc.test.cjs— createsnew Worker(...)threads (lines ~169, ~214) that are never.terminate()d; on the timeout/error path a worker can stay alive (blocked inAtomics.wait/ a 30s lock-acquire retry) after the test finishes.tests/locking-bugs-1909-1916-1925-1927.test.cjs—spawn()s child processes (lines ~207, ~340, ~460) wrapped in promises but never.kill()d on the reject/timeout path; theirstdio: 'pipe'channels hold the parent event loop open until the children self-exit (up to ~10s each).
Secondary: the stale comment in scripts/run-tests.cjs blaming synckit worker_threads is obsolete — synckit/the SDK bridge was removed (bridge-collapse). gsd-tools now runs as a subprocess.
Error output / logs
ok 613 - _applyRuntimeRewrites(trae) — install-path bare-form (#983)
# (150s of silence — chunk's node --test process not exiting)
<GSD ASCII banner> # next chunk starts
...
Error: The operation was canceled. # job killed at 20m cap
Impact
Major — a false-negative gate that red-blocks merge-ready PRs and forces manual re-runs.
How often does this happen?
Sometimes / intermittent (probabilistic — depends on whether ≥2 chunk stalls land in the same run).
Workaround
Re-run the cancelled job (often passes the second time). Not durable.
Proposed fix
- Add
--test-force-exit(Node ≥22;enginesalready requires ≥22.0.0) to thenode --testinvocation inscripts/run-tests.cjs— forces process exit once all tests finish regardless of lingering handles. Primary durable fix. - Fix the two leaking tests (terminate the workers; kill the spawned children on the error path) so the leak doesn't exist in the first place.
- Add a per-chunk timeout to the
execFileSynccall with a clear diagnostic message, so a chunk that hangs can never silently eat the job budget — it fails loudly instead of being cancelled. - Refresh the stale
synckitcomment.
Sharding the Windows full lane (the deferred #869 follow-up) is intentionally not included here — force-exit + leak fixes should bring the lane comfortably under the cap, and one-concern-per-PR.
Activity
github-actions commentedon Jun 11, 2026
Branch
fix/1051-ci-full-test-windows-latest-22-intermittcreated.git fetch origin && git checkout fix/1051-ci-full-test-windows-latest-22-intermittfix(#1051): force-exit + per-chunk timeout for the windows full-test …
ci(#1212): shard windows full-test lane to remove timeout cliff (#1222)
ci(#2952): shard the full test lane instead of widening its cap (#2960)
fix(open-gsd#4012): a killed chunk now names the file that was hanging
fix(#5170): test-file children write their results to a blocking stdo…
fix(#5170): run-tests fails a chunk whose registered tests did not al…
fix(#5170): test-file children write their results to a blocking stdo…
fix(#5170): run-tests fails a chunk whose registered tests did not al…
fix(#5170): test-file children write their results to a blocking stdo…
refactor(#5170): typed gate evidence and a verdict-driven exit status…