Skip to content

CLI CI: integration shard containing mtc-federation-governance-cli.test.js flakes with vitest onTaskUpdate timeout #4

Description

@chainlesschain

Summary

__tests__/integration/mtc-federation-governance-cli.test.js is a single 41-test file that runs ~110s on Linux CI / ~200s on Windows CI / ~347s on local Windows. All 41 tests pass — only the vitest worker→main onTaskUpdate RPC heartbeat times out during long quiet stretches between subprocess results.

Why sharding can't fix it

vitest's --shard=k/n hashes files into buckets. This single file dominates whichever bucket it lands in:

Layout Heavy shard Ubuntu macOS Windows
4 shards (c4c4f0d8a) 4/4 109s ❌ 136s ❌ 213s ❌
8 shards (b52c2f427) 8/8 112s ❌ passed 203s ❌

Bumping shards just relocates the failure; doesn't shrink the file. Wall-times barely move because the heavy file dominates regardless of how many siblings sit alongside it.

Evidence

  • CLI CI run 25321545472 on b52c2f427 — 49/51 jobs green; only integration shard 8/8 on Ubuntu + Windows red
  • Both red shards reported Test Files 5 passed (5) followed by Error: [vitest-worker]: Timeout calling "onTaskUpdate" — pure heartbeat saturation, not test-correctness regression
  • Local repro on Windows: 41/41 pass in 347s, then unhandled error on RPC timeout

Proposed fixes (pick one)

A. Split the test file (highest leverage, most engineering work)

  • Break the 41 tests across 3-5 smaller files grouped by concern (3-of-3 vs threshold, M-of-N validation, error paths, etc.)
  • Each smaller file completes within heartbeat window
  • Root cause cure

B. Dedicated CI job for this file (smallest workflow change)

  • Add an integration-heavy job in .github/workflows/_cli-test.yml that runs only mtc-federation-governance-cli.test.js
  • Make the matrix integration shards exclude that file via --exclude or explicit file list
  • File still flakes in isolation but cleaner to debug + can have its own timeout/retry settings

C. Reduce subprocess spawn count in the test

  • 41 subprocess spawns is heavy; many tests likely set up similar federations with one differing assertion
  • Refactor to share federation setup across describes, asserting different shapes within
  • Reduces the ~85s baseline significantly

Out of scope (already tried)

  • Bumping shards from 4 → 8 (above)
  • Shorter testTimeout / hookTimeout (this isn't the issue — tests pass; reporter RPC times out)
  • --silent=true (already in place; doesn't help intra-file gaps)
  • Increasing vitest's onTaskUpdate timeout — value is hardcoded in node_modules/vitest/dist/chunks/rpc.*.js, no config knob

Workaround until fixed

Re-run the failing CLI CI shard. Other shards will be green; the same heavy file lands deterministically in the same shard so re-running just gives it another chance to slip under the timeout (it sometimes does on macOS, never on Linux/Windows in our recent runs).

Context

Documented inline in .github/workflows/_cli-test.yml integration job comment as of commit 016df8a0d. This is the residual after a successful sharding migration that took CLI CI from 0/3 OS jobs green (pre-shard, glob batching) to 49/51 jobs green (96%) on b52c2f427.

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions