test-stability: eliminate timeout flakiness in the web test suite (zero-flake requirement) #1596

Activity

1. cliffhall

   self-assigned this

   on Jul 2, 2026
2. cliffhall

   added

   v2Issues and PRs for v2Issues and PRs for v2

   on Jul 2, 2026
3. cliffhall commented on Jul 2, 2026

   

   cliffhall

   on Jul 2, 2026

   MemberAuthor

   More actions

   Smoke test & AGENTS.md audit — PR #1597

   Independent verification (fresh detached checkout of origin/v2/1596-test-stability @ dd781206, clean npm install at root). Goal: independently confirm zero test-content flakiness rather than trust the implementer's runs.

   Test-only confirmation

   git diff origin/v2/main...HEAD --name-only — 7 files, all tests, no product source touched (clients/web/src/** non-test and core/ untouched):

   • clients/web/src/components/groups/PromptArgumentsForm/PromptArgumentsForm.test.tsx
   • clients/web/src/components/groups/ServerConfigModal/ServerConfigModal.test.tsx
   • clients/web/src/components/groups/ServerImportConfigModal/ServerImportConfigModal.test.tsx
   • clients/web/src/components/groups/ServerImportJsonModal/ServerImportJsonModal.test.tsx
   • clients/web/src/components/screens/ResourcesScreen/ResourcesScreen.test.tsx
   • clients/web/src/components/views/InspectorView/InspectorView.test.tsx
   • clients/web/src/test/integration/mcp/inspectorClient.test.ts

   Run-by-run evidence (clients/web)

   #	Command	Load	Result
   1	npm run test:coverage	solo	229 files / 3237 tests passed, gate clean
   2	npm run test:coverage (A)	concurrent w/ #3	3237 passed — see note ¹
   3	npm run test:coverage (B)	concurrent w/ #2	3237 passed — see note ¹
   4	npm run test:coverage	solo	exit 0, 3237 passed, 0 threshold errors
   5	npm run test:coverage	under load: 8 CPU burners saturating all cores	exit 0, 3237 passed, 0 threshold errors
   6	npm run validate (web)	(residual load)	exit 0, 191 files / 2438 passed
   7	npm run test:integration	solo	exit 0, 38 files / 799 passed

   Targeted stress on the previously-flaky files:

   • PromptArgumentsForm.test.tsx (debounce/completions, most timing-sensitive) — 12/12 runs green (20 tests each).
   • InspectorView.test.tsx — 5/5 green (44 tests each).
   • All 6 touched files run individually — 150 tests, all pass.

   Test-content failures reproduced across every run above: ZERO. ✅

   ¹ Concurrent-coverage note (not a flake): running two test:coverage invocations in the same working dir simultaneously produced coverage-gate errors on files not touched by this PR (AppRenderer.tsx, ContentViewer.tsx, ResourceLink.tsx) — while all 3237 tests still passed in both. This is a v8 coverage-temp-dir collision (two collectors clobbering the same coverage/ output), a known measurement artifact, not test flakiness. The isolated form of "under load" — a solo coverage run against 8 CPU burners (row 5) — passed the gate clean, so the load dimension is covered without the collision.

   Also observed: vitest run with a 5+ file positional filter and/or background node processes intermittently prints No test files found (zero tests execute — a vitest v4 CLI discovery quirk). Every file passes standalone and the full unfiltered suite is immune (rows 1–5). CI runs the full project, never a positional filter, so this does not affect the PR.

   Mutation check (proves the test is load-bearing) ✅

   Temporarily removed the clearTimeout(existing) in PromptArgumentsForm.tsx handleFocus, then ran "cancels a pending debounce timer when the input is re-focused":

   × cancels a pending debounce timer when the input is re-focused
   AssertionError: expected 1 to be +0 // Object.is equality
   

   The un-cancelled debounce fired during the fake-timer advance → test failed exactly as intended. Source restored; working tree verified clean (git status --porcelain empty).

   AGENTS.md compliance

   • Mantine: N/A (test-only diff).
   • TypeScript / test hygiene: no any, no .skip/.only/.todo/xit/xdescribe, no eslint-disable, no @ts-ignore/@ts-expect-error in the diff.
   • Deterministic, not timeout inflation: fix uses userEvent.setup({ delay: null }) (94 sites), awaited conditions (findBy/waitFor, 10 sites), vi.useFakeTimers() for the debounce-cancel test, and vi.waitFor around the out-of-band getStderrLogs() in the integration test. 12 wall-clock setTimeout sleeps removed. No global testTimeout/hookTimeout inflation — the only timeout literal is a scoped SETTLE = { timeout: 3000 } passed to individual waitFor calls.
   • Coverage gate: test:coverage passes solo and under 8-core CPU load with 0 threshold errors (≥90 on all four dimensions holds). Touched files are test files (not themselves gate targets); the components they exercise stayed ≥90.

   Verdict: PASS ✅

   Zero test-content flakes reproduced across 5 independent full-suite green runs (2 concurrent + 1 under 8-core CPU load) plus validate, integration, and 17+ targeted stress iterations. The debounce-cancel test is verified load-bearing. Diff is test-only, deterministic, and AGENTS.md-compliant.

   Helper: under-load coverage harness (saved to /tmp, not committed)

   # Solo coverage under full CPU saturation (8 burners on an 8-core box)
   cd clients/web
   NCPU=$(sysctl -n hw.ncpu)
   PIDS=""
   for i in $(seq 1 $NCPU); do
     ( node -e 'const e=Date.now()+300000; while(Date.now()<e){Math.sqrt(Math.random()*9e9);}' ) &
     PIDS="$PIDS $!"
   done
   npm run test:coverage; echo "exit=$?"
   kill $PIDS 2>/dev/null; pkill -f "Math.sqrt"   # note: kill the child, not just the subshell
   
   # Targeted repetition of the most timing-sensitive file
   for n in $(seq 1 12); do
     npx vitest run src/components/groups/PromptArgumentsForm/PromptArgumentsForm.test.tsx
   done
4. cliffhall

   linked a pull request that will close this issuetest(web): eliminate timeout flakiness in the web test suite #1597on Jul 2, 2026
5. cliffhall commented on Jul 2, 2026

   

   cliffhall

   on Jul 2, 2026

   MemberAuthor

   More actions

   🔗 Implemented by PR #1597. Its code is already integrated into the Wave 1 rollup branch 1579-wave-1 (see rollup PR #1601 / verification #1600). #1597 currently targets v2/main and will auto-show as Merged once the rollup lands in v2/main. The Development panel doesn't link it now because the closing keyword is on a non-default base branch.
6. cliffhall

   closed this as completedon Jul 12, 2026
7. pull

   added a commit that references this issue on Jul 28, 2026

   test(web): eliminate timeout flakiness in the web test suite (closes m…

   ...

   This issue will close once commit 4bad110 is merged into the 'main' branch.

   4bad110
8. cliffhall

   added this to the v2.0.0 milestone on Jul 28, 2026
9. cliffhall

   added 3 commits that reference this issue on Sep 12, 2026

   fix: address Copilot review round 4

   ...

   7b794de

   fix: address Copilot review round 4

   ...

   9cff77e

   test: address Copilot review — tighten progress-token guard and de-fl…

   ...

   fb28b0f
10. pull

    added a commit that references this issue on Sep 16, 2026

    fix: state every test-gate timeout instead of inheriting a library de…

    ...

    58e53d5
11. Yaimsputnik5

    added a commit that references this issue on Nov 13, 2026

    Wait for the asserted count, not the first match, in SkillsScreen ver…

    ...

    965570b