Skip to content

ci: full test (windows-latest, 22) intermittently CANCELLED at 20m cap — leaked test handle stalls node --test ~150s at chunk boundaries #1051

Closed

Description

@trek-e

What happened?

The CI job full test (windows-latest, 22) (the test-full job in .github/workflows/test.yml, timeout-minutes: 20) intermittently gets CANCELLED at the 20-minute wall-clock cap (conclusion=cancelled, with no failed test step). This red-blocks otherwise-green PRs — it is a false-negative gate.

This is a recurrence of #869 (which raised the cap 15→20m as a stopgap; the durable fix was deferred). The job is now hitting the new 20m cap.

What did you expect?

The Windows unit lane should finish well under the cap, and a genuine hang should fail with a clear diagnostic — never silently consume the wall-clock budget and surface as an opaque cancelled.

Steps to reproduce

  1. Open any PR that triggers the test-full matrix.
  2. Observe full test (windows-latest, 22) (npm run test:unit → scripts/run-tests.cjs --suite unit).
  3. Intermittently the job is CANCELLED at 20m with no failed test step. Re-running the identical commit frequently passes (~15m).

Canonical evidence — PR #1047, run 27362003027, job 80851641619 (2026-06-11):

scripts/run-tests.cjs splits the ~656 unit files into chunks (Windows CreateProcess 32,767-char command-line cap, #3597) and runs each chunk sequentially as execFileSync(process.execPath, ['--test', concurrency, ...chunkFiles]) with no per-chunk timeout.

The overrun is NOT uniform slowness and NOT a single slow test (every individual test runs <1s). It is two windows of ~150s and ~146s of pure silence at chunk boundaries:

  • Gap 1: 16:39:30 → 16:42:00 (150s), immediately after ok 613 - _applyRuntimeRewrites(trae) — install-path bare-form (#983).
  • Gap 2: 16:46:30 → 16:48:56 (146s), immediately after ok 347 - execute-plan.md routing instruction (#1979).

After each silence the NEXT chunk's node --test prints the GSD banner — i.e. the silence is the previous chunk's child process failing to EXIT for ~150s after its last test printed. Baseline unit step is already 11m54s–13m44s, so ≥2 such stalls push the job past 20m → cancel.

Root cause (diagnosed)

A unit test leaves an open event-loop handle, so node --test will not exit until the handle drains; on Windows that drain takes ~150s. Confirmed leak sources:

  • tests/perf-316-state-lock-buffer-alloc.test.cjs — creates new Worker(...) threads (lines ~169, ~214) that are never .terminate()d; on the timeout/error path a worker can stay alive (blocked in Atomics.wait / a 30s lock-acquire retry) after the test finishes.
  • tests/locking-bugs-1909-1916-1925-1927.test.cjs — spawn()s child processes (lines ~207, ~340, ~460) wrapped in promises but never .kill()d on the reject/timeout path; their stdio: 'pipe' channels hold the parent event loop open until the children self-exit (up to ~10s each).

Secondary: the stale comment in scripts/run-tests.cjs blaming synckit worker_threads is obsolete — synckit/the SDK bridge was removed (bridge-collapse). gsd-tools now runs as a subprocess.

Error output / logs

ok 613 - _applyRuntimeRewrites(trae) — install-path bare-form (#983)
# (150s of silence — chunk's node --test process not exiting)
<GSD ASCII banner>   # next chunk starts
...
Error: The operation was canceled.   # job killed at 20m cap

Impact

Major — a false-negative gate that red-blocks merge-ready PRs and forces manual re-runs.

How often does this happen?

Sometimes / intermittent (probabilistic — depends on whether ≥2 chunk stalls land in the same run).

Workaround

Re-run the cancelled job (often passes the second time). Not durable.

Proposed fix

  1. Add --test-force-exit (Node ≥22; engines already requires ≥22.0.0) to the node --test invocation in scripts/run-tests.cjs — forces process exit once all tests finish regardless of lingering handles. Primary durable fix.
  2. Fix the two leaking tests (terminate the workers; kill the spawned children on the error path) so the leak doesn't exist in the first place.
  3. Add a per-chunk timeout to the execFileSync call with a clear diagnostic message, so a chunk that hangs can never silently eat the job budget — it fails loudly instead of being cancelled.
  4. Refresh the stale synckit comment.

Sharding the Windows full lane (the deferred #869 follow-up) is intentionally not included here — force-exit + leak fixes should bring the lane comfortably under the cap, and one-concern-per-PR.

Activity

  1. github-actions commented on Jun 11, 2026

    @github-actions
    Contributor

    Branch fix/1051-ci-full-test-windows-latest-22-intermitt created.

    git fetch origin && git checkout fix/1051-ci-full-test-windows-latest-22-intermitt
  2. added 2 commits that reference this issue on Jun 11, 2026
  3. added a commit that references this issue on Aug 1, 2026
  4. added a commit that references this issue on Aug 29, 2026
  5. added 6 commits that reference this issue on Oct 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingconfirmed-bugVerified reproducible bug

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions