Skip to content

ci: e2e (windows) job is chronically flaky under load (different specs fail each run) #189

Closed

Description

@alltomatos

The `e2e (windows)` CI job has failed with zero actual test failures on two consecutive runs of the same commit (PR #161, run 34401629220): all failures are Playwright "flaky" (fail once/twice, pass on retry), with a different set of unrelated specs each time:

  • Run 1: `open-file-expand-folder`, `file-browser-sidebar-tab-switch`, `review-line-comment`, `session-timeline-lifecycle-state`, `session-timeline-projection`, `session-timeline-reasoning-projection`, `session-timeline-tool-projection`, `session-timeline-tool-state`, `terminal-composer-focus`, `smoke/session-timeline` — 9 flaky, 88 passed
  • Run 2 (rerun of the same commit): `file-browser-sidebar-tab-switch`, `open-file-expand-folder`, `review-line-comment`, plus others — 11 flaky, 86 passed

No spec repeats as a hard failure across runs; different specs each time, spread across completely unrelated feature areas (file browser, review pane, session timeline, terminal). This is the signature of the Windows e2e runner being resource-starved (memory/CPU contention slowing down Playwright's own internal waits past their thresholds), not a real regression in any of them.

Playwright's default behavior treats any "flaky" (needed-a-retry) test as a non-zero exit code for the whole `test:e2e:local` script, so the CI check goes red even when 0 tests hard-fail.

Not investigated further — flagging so this stops looking like "the PR broke something" on every red `e2e (windows)` check. Possible directions: bump the Windows e2e runner tier/resources, increase Playwright's default expect/action timeouts for Windows specifically, or treat flaky-but-passed as non-fatal in CI (`--reporter` / retry accounting change).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions