Skip to content

CI: e2e-mock required check fails intermittently on 30s timeouts caused by real-network model-catalog fetch stalls #2044

Open

Description

@edelauna

Problem (one or two sentences)

The required e2e-mock CI check fails intermittently. One test times out after 30s while 85 or more tests in the same run pass.

Context (who is affected and when)

Every PR that runs the e2e.yml workflow is affected. A transient runner-network stall during the mock run fails the whole job and holds the PR review gate. Three unrelated branches hit this on 2026-10-13:

  • PR Fix interrupted child redelegation #1905, run 38343639461, branch fix/interrupted-child-redelegation, test Markdown List Rendering > Should render unordered lists with bullets in chat
  • Run 38391536768, branch vps2/f6-e2e-viewstate, test Roo Code View State > sidebar and tab panel keep mode isolated through the real ContextProxy singleton
  • Run 38388064934, same branch and test, failed again 30 minutes later

All three branches passed on later runs with no related code change. The PR #1905 commit touches no fetcher, model-cache, or e2e harness code, and the same test passed in 534ms at the prior head in run 38329310237.

Reproduction steps

Not reproducible on demand. The pattern from the failing runs:

  1. Mock e2e starts and the extension host activates.
  2. The host runs background model-catalog refreshes against real endpoints (OpenRouter, Vercel AI Gateway, NanoGPT).
  3. A runner-network stall makes those fetches hang. Each fetch holds a socket until the 15s abort timer fires (MODEL_CATALOG_FETCH_TIMEOUT_MS = 15_000). A second wave follows immediately, so the stall window reaches about 30s.
  4. The test task in progress never issues its first API request during this window.
  5. waitUntilCompleted rejects with Timeout after 30s.

Evidence from the PR #1905 run job log:

  • 03:52:48 test task starts, checkpoints initialize.
  • 03:52:49 to 03:53:02: no task-loop activity at all.
  • 03:53:02 and 03:53:17: two waves of [getModels] ... AbortError: This operation was aborted for openrouter, plus requestRouterModels failures. 16 hang-abort lines in the failing run, zero in the passing run.
  • 03:53:18: Timeout after 30s for the test. The two sibling tests then pass in 768ms and 1273ms with the identical start-and-wait flow.

Expected result

Mock e2e never depends on the real network. A transient runner-network stall must not fail the required check.

Actual result

Real-network catalog fetches run during mock e2e. When they hang, they block task startup for about 30s, one test fails, and the whole required job fails.

Variations tried

Re-run of the failed job. The PR #1905 re-run was started manually; no code change was involved.

App Version

CI only. e2e-mock job in .github/workflows/e2e.yml, GitHub-hosted ubuntu-latest runners.

Relevant logs or errors

Error: Timeout after 30s
	at Timeout._onTimeout (out/suite/utils.js:15:20)

[getModels] Failed to fetch models in modelCache for openrouter: AbortError: This operation was aborted
	at Timeout._onTimeout (node:internal/abort_controller:127:7)

Suggested fixes

  1. Extend the mock fast-fallback to all real-network catalog fetchers under mock e2e. Today only the DeepSeek fetcher honors E2E_MOCK_MODEL_LIST_FALLBACK. OpenRouter, Vercel AI Gateway, and NanoGPT have no equivalent.
  2. Alternatively, add aimock fixtures for the catalog endpoints so mock runs never touch the real network.
  3. Optionally, add one retry to the e2e poll helper or raise its 30s ceiling, so a single stall cannot fail the required check.

Activity

  1. edelauna commented on Oct 13, 2026

    @edelauna
    ContributorAuthor

    Provenance check after the initial report:

    Conclusion: the base fragility (single-shot 30s wait, real-network dependence during mock e2e) predates #1906. The refactor did not introduce the flake, but it added a background real-network fetch fan-out that widens the stall window to the full 30s budget when the runner network hiccups. Failure counts in this sample: 1 timeout flake in 100 runs before the merge, 3 timeout flakes in one day after it. The sample is small, so treat the frequency shift as indicative, not proven.

    The suggested fixes stay valid. Extending the mock fallback or stubbing catalog endpoints removes the new stall source; a retry in the poll helper covers the pre-existing one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions