Repository navigation
Move E2E tests out of PR pipeline to release/nightly #2397
Description
Problem
E2E tests on every PR are expensive and provide low signal:
- 42% failure rate — nearly half of all E2E runs fail
- ~892 CI hours spent on PR E2E over 4 months (714 runs × ~75 min avg)
- ~96 hours of developer time wasted waiting on flaky retests
- Only ~22% of runs (upper bound) could have caught a real bug
- At most 60 unique bugs caught in 4 months across 714 runs — realistically closer to 20-30
- Most actual code issues are caught earlier by code review (AI bots and human reviewers), not by E2E
Current state (2026-06-08)
Consolidated test PR #2500 is now closed (2026-05-14) — every piece shipped via its own focused PR, so the preview snapshot is no longer needed.
Program is ~85% complete: every retry-masked bug fix and test-suite improvement has shipped, and the install gate has been hardened so single-attempt CI will actually fail on a broken release. The two keystone PRs remain open: #2558 (drop-retries) was rebased and re-ran CI today (2026-06-08) — it now also waits out the gateway.bats tenant teardown — but is still REVIEW_REQUIRED; #2559 (TIA + release E2E workflow) is unchanged and still awaiting review. Separately, #2826 opens a new exploration: migrating the app E2E suite from BATS to declarative Chainsaw.
Shipped — bugs the 3× retry was hiding
Audit of 30 successful PR runs found deterministic bugs that retry was masking. Each got a focused fix:
| PR | What | Status |
|---|---|---|
| #2508 | installer: drop chart-defined Namespace; caller pre-applies (Helm namespace-ownership conflict on cold install, 17/17 hit rate) |
MERGED 2026-05-05 |
| #2509 | operator: 5 HR generation knobs + Strategy=RetryOnFailure, RetryInterval=30s (replaces the flux reconcile --force workaround for seaweedfs-system HR 2-min timeout) |
MERGED 2026-05-05 |
| #2528 | harbor: bucket-secret values-driven; HR gated on BucketInfo via Flux valuesFrom |
MERGED 2026-05-04 |
| #2529 | objectstorage-controller: retry on Bucket update conflict during BucketAccess reconcile |
MERGED 2026-05-04 |
Shipped — test-suite improvements
| PR | What | Status |
|---|---|---|
| #2553 | cozyreport: collect Flux / cert-manager / host context + summary.txt (machine-readable "what's broken now") |
MERGED 2026-05-04 |
| #2554 | e2e: replace fixed timeouts with event-driven backstops across 16 bats files |
MERGED 2026-05-06 |
| #2555 | virtual-machine: vminstance disk delete-recreate race + bump VM IP/ready timeouts |
MERGED 2026-05-06 |
| #2556 | e2e: pre-pull timing-sensitive platform images before install |
MERGED 2026-05-04 |
| #2557 | installer,e2e: configurable helmReleaseInterval + parallel post-install prep (LINSTOR / SC / MetalLB runs alongside platform reconcile wait, eliminates the ~10-min Flux dead zone in Install Cozystack) |
MERGED 2026-05-07 |
Shipped — additional E2E hardening (post 2026-05-13)
Continuation of the program scope after the May-13 status update:
| PR | What | Status |
|---|---|---|
| #2690 | fix(e2e-sandbox): use curl -fsSL when downloading mc (fail fast on a bad download instead of silently producing a broken image) |
MERGED 2026-05-20 |
| #2724 | test(e2e): pre-pull platform images via staged-busybox DaemonSet (DaemonSet-based extension of #2556; pre-warms the pull on every node in parallel before install) |
MERGED 2026-05-26 |
| #2728 | test(e2e): ouroboros fold + CSI/NFS ordering + OIDC keycloakInternalUrl |
MERGED 2026-05-27 |
Shipped — additional E2E hardening (post 2026-05-28)
| PR | What | Status |
|---|---|---|
| #2824 | test(e2e): fail the install test when HelmReleases are not Ready — the backgrounded kubectl wait fan-out discarded child exit codes and the final check never exited non-zero, so the readiness gate was toothless. Prerequisite for meaningful single-attempt CI (#2558): the gate has to actually fail before retries can be dropped. |
LANDED in main 2026-06-04 (commit 67b1a960d; rebase-merged, GitHub shows the PR closed) |
| #2601 | refactor(seaweedfs): split seaweedfs-system into seaweedfs-db + seaweedfs-system — retires the slow-HR install path that #2509's RetryOnFailure knob was working around |
MERGED 2026-05-28 |
| #2602 | feat(flux): upgrade to v2.8.0 + chart fixes for strict SSA & kstatus (folds the #2612 kubevirt TPM fix) — reduces install-time reconcile flake |
MERGED 2026-06-02 |
Open — keystone PRs
| PR | What | Status |
|---|---|---|
| #2558 | ci: drop 3× retry on Run E2E + Install Cozystack (keeps the 3× retry only on Prepare environment, pure infra). On failure now captures kubectl get hr -A -o wide + events. Keystone — relies on all the bug-fixes above to keep CI green without retries. |
OPEN, mergeable, REVIEW_REQUIRED. Rebased + re-ran E2E 2026-06-08; now also waits out the gateway.bats tenant teardown |
| #2559 | ci: TIA E2E (default-on, full-e2e opt-out) + release E2E workflow. This is the direct implementation of this issue's original proposal: graph-aware test selection on PRs + a separate release-branch E2E run. |
OPEN, mergeable, REVIEW_REQUIRED (no change since 2026-05-28) |
Open — additional E2E tooling
| PR | What | Status |
|---|---|---|
| #2740 | feat(hack): improve check-readiness.sh — wait/core/parallel modes and broader coverage |
OPEN |
| #2755 | feat(hack): rewrite check-readiness as a Go command with test suite (supersedes #2740 once landed) |
OPEN |
Exploration — declarative E2E (Chainsaw)
| PR | What | Status |
|---|---|---|
| #2826 | test(e2e): pilot migrating app E2E from BATS to Kyverno Chainsaw (CNCF, KUTTL successor). Ports postgres + bucket as examples under hack/e2e-chainsaw/, duplicating (not replacing) the BATS originals. Declarative tests are easier to select per-PR (complements the TIA work in #2559) and remove a class of bash-level flakiness. |
OPEN, draft |
In-flight — deterministic bug E2E surfaced
| PR | What | Status |
|---|---|---|
| #2842 | fix(platform): order cozystack-basics after the APIs its ValidatingAdmissionPolicies type-check — kube-controller-manager's VAP status type-checker nil-derefs when a policy matches a CRD whose schema isn't established yet, CrashLooping KCM on a cold install. Exactly the kind of deterministic, ordering-dependent failure that retries used to paper over. |
OPEN, touched 2026-06-08 |
Followups
- feat(api): expose HelmRelease generation knobs as cozystack-api flags #2571 —
feat(api): bringcozystack-api'sconvertApplicationToHelmReleaseto HR-knob parity with the operator (followup to feat(operator): expose HelmRelease generation knobs (interval, retry-interval, timeouts, max-history) #2509). OPEN (currently conflicting — needs rebase). - chore: address review followups from #2509 and #2553 #2577 —
chore: address review followups from feat(operator): expose HelmRelease generation knobs (interval, retry-interval, timeouts, max-history) #2509 (operator min-duration validator) + feat(cozyreport): collect Flux/cert-manager/host context + summary.txt #2553 (cozyreport pipefail / OOMKilled lastState). OPEN, draft. - e2e: move per-test resource cleanup into bats teardown() #2583 —
e2e: move per-test resource cleanup into bats teardown(). Carved out from review of fix(virtual-machine): vminstance disk race + bump VM IP/ready timeouts #2555. OPEN issue.
Expected impact (revisited)
- Per-PR E2E becomes diff-aware via feat(ci): test-impact analysis E2E (default-on) + release E2E workflow #2559 — docs / chart-only / unrelated-app PRs run a minimal selection (or zero) bats files; full suite remains the default until the label gating is promoted after soak.
- Single-attempt CI via ci(e2e): drop 3x retry on Run E2E + Install Cozystack, wait out gateway.bats tenant teardown #2558 reclaims the masked failure-rate; the bug-fixes shipped in Stage 1/2 plus the hardened install gate (test(e2e): fail install test when HelmReleases are not Ready #2824) keep deterministic regressions from re-emerging once retries are gone.
- Release-branch E2E workflow (part of feat(ci): test-impact analysis E2E (default-on) + release E2E workflow #2559) covers the integration-regression surface that previously came from PR E2E.
- Chainsaw migration (test(e2e): migrate app suite from BATS to Chainsaw and wire into CI #2826), if adopted, makes per-PR test selection cleaner and removes bash-level flakiness — a longer-horizon complement to TIA.
Related: prior E2E stabilization efforts
- [tests] Fix flaky E2E timeouts, add diagnostics and retry cleanup #2255 —
[tests] Fix flaky E2E timeouts, add diagnostics and retry cleanup(Andrei Kvapil (@kvaps), open) - [tests] Stabilize E2E kubernetes tests #2262 —
[tests] Stabilize E2E kubernetes tests(Aleksei Sviridkin (@lexfrei), merged) - [tests] Fix Kafka E2E test timeout and retry race condition #2358 —
[tests] Fix Kafka E2E test timeout and retry race condition(Aleksei Sviridkin (@lexfrei), merged) - fix(e2e): increase HelmRelease readiness timeout for kubernetes test #2033 —
fix(e2e): increase HelmRelease readiness timeout for kubernetes test(Aleksei Sviridkin (@lexfrei), merged) - fix(e2e): make kubernetes test retries effective by cleaning up stale resources #2062 —
fix(e2e): make kubernetes test retries effective by cleaning up stale resources(Aleksei Sviridkin (@lexfrei), merged) - [e2e] fix timeouts for capi and keycloak #858 —
[e2e] fix timeouts for capi and keycloak(Andrei Kvapil (@kvaps), merged)
Data (original baseline, Dec 2025 – Apr 2026)
Analysis based on 773 E2E runs from Dec 2025 – Apr 2026:
| Metric | Value |
|---|---|
| E2E runs analyzed | 773 |
| Clean first-try successes | 375 (49%) |
| Flaky (passed on retest, no code change) | 77 (10%) |
| Final failures | 321 (42%) |
| Failures resolved by code fix (upper bound) | 169 events / 60 branches |
| Realistic bugs caught by E2E | ~20-30 |
| CI hours on PR E2E | ~892h |
| Cost per bug caught | ~15-36 CI hours |
Reacted by dosubot
Metadata
Metadata
Labels
Issues or PRs related to CI workflows, GitHub Actions, automationIssues or PRs related to testing (e2e, bats, unit tests)Categorizes issue or PR as related to a new featureMust be staffed and worked on either currently, or very soon, ideally in time for the next releaseIndicates an issue is ready to be actively worked on
Type
Projects
- Status
Activity
myasnikovdaniil commentedon Apr 24, 2026
Alternative / complement: path-based Test Impact Analysis on PR
The current proposal is binary — run full E2E nightly/release, nothing on PR. A middle-ground worth considering is Test Impact Analysis (TIA): run only the tests whose code paths a PR actually touches.
Why it fits this repo
The source↔test mapping is already explicit in the layout:
packages/apps/<name>/→hack/e2e-apps/<name>.bats(1:1 for postgres, mariadb, kafka, …)packages/system/*,packages/core/*,hack/*,api/*, root manifests → shared infra → full E2Edocs/,dashboard/, workflow docs → skip E2ESo a rules-based selector on changed paths recovers ~80% of TIA's benefit without coverage instrumentation (which is awkward for bash/bats + Helm anyway).
Selector sketch
packages/apps/<X>/<X>.batspackages/extra/<X>/<X>.bats(if exists)docs/,dashboard/, non-test.github/packages/system/*,packages/core/*,hack/,api/, rootTrade-offs vs "move to nightly only"
/retestcycles shrink because the flakiness surface per PR shrinkssystem/monitoringchange that breakspostgres.bats) — needs explicit escape hatches or a conservative fallback-to-fullImplementation cost
~50 lines: a
dorny/paths-filter(or equivalent) step inpull-requests.yamlthat sets a matrix of bats files to run, with a "run all" fallback when shared infra paths match.Suggestion
Not an alternative to moving full E2E off PR — a complement. Keep the nightly/release full run, and add path-based selection on PR so single-package changes still get targeted verification without the 75 min tax.
References:
myasnikovdaniil commentedon Jun 22, 2026
Closing — the core objective is shipped and live on
main, and the replacement gate is proven green.Shipped (merged 2026-06-09):
Run E2E/Install Cozystack.full-e2eopt-out), so docs / chart-only / unrelated-app PRs no longer pay for the full suite.Full-suite coverage now runs on the release-cut PR — confirmed green. The v1.5.0 cut (#2970) ran
E2E Teststo success on the digest-bake commit (e1aaa90e), which touchespackages/core/and so escalates to the full suite under TIA. The standalonerelease-e2e.yamltag run on that same commit was red — a workflow-level issue, not a product regression — so it's a redundant, non-blocking alarm, being retired in #2978.Spun-off follow-ups: Chainsaw migration → #2979, per-test cleanup into bats
teardown()→ #2583.Not built here: a scheduled nightly full-suite run against
mainbetween releases (coverage currently lands at release-cut time). If we want periodic main coverage, let's open a dedicated issue.ci(tests): remove redundant Release E2E workflow (#2978)
ci(tests): remove redundant Release E2E workflow