[Epic] Diff-driven & scenario-based E2E testing for Cozystack #2847

Goal

Make Cozystack's E2E testing fast, scenario-driven, and trustworthy enough that we can cut releases faster and with more confidence. Two complementary tracks: (1) diff-driven install so a PR's CI installs only what the changed app actually needs, and (2) scenario coverage (install, upgrade, per-app workloads) so the major usage paths are continuously exercised.

Status as of 2026-10-01

Checked against main at bc400dff1.

Three tasks are done, the .deps pivot, the install selector (select-install.sh, #3279) and the install smoke scenario (Track C). Two are obsolete as written, the test-minimal bundle and the COZYSTACK_ENABLED_PACKAGES wiring, because the design moved to select-install.sh --disabled in #4022. Update 2026-11-10: #4022 merged on 2026-10-07 and both PR lanes pass select-install.sh --disabled into the install, so real installs use its closure now.

The PR lanes changed under this epic. #4020 (merged 2026-09-07) moved the PR lane and the fork lane to Talos containers and they run the full suite there, QEMU stays in nightly and e2e-tag. Install takes about 28 min on the container lane, not the ~18 min this epic started from. I measured 62 PR e2e jobs from 2026-09-28 to 09-30 (install step median 1684 s), and Run E2E tests was full-shaped in 73% of them, scoped in 23% and skipped in 5%, classified by step duration. Install is paid in all three cases. The replay in #3817 of the last 150 merges to main agrees, 70.7% full-suite after its change, 80.0% before. So most runs escalate and install selection can only save on the scoped quarter. The window is three days and heavy on re-runs, so read it as a rough number.

That answers the measurement the track was waiting for, at least roughly. The "~60% of PRs touch one app" payoff does not hold. What is left to decide is whether a saving on a quarter of runs pays for the wiring, and that needs a look at the escalation side first, #4008 (broad selector tier) is open for that.

Track B is #3276, still open, conflicting and CHANGES_REQUESTED (2026-09-29). It needs a rework on the container lane, not a rebase.

Rationale

Every E2E run installs the isp-full bundle before any app test. A diff-driven-install design proposed computing the minimal install set from the PR diff using per-app PackageSource dependency lists, targeting a drop from the install time for PRs that touch one app. The selector script and the test-selection foundation it builds on landed in #2559. This epic operationalizes that design and folds in the in-flight upgrade-test work so install, upgrade and app scenarios are tracked under one roof.

The "~60% of PRs touch one app" and "~6-8 min" numbers were the original design estimates and were never measured on their own. The measurements in the status above say most runs escalate to the full suite, as docs/agents/e2e-testing.md (the TIA section) already said.

Test selection is default-on since #2559, with full-e2e as the opt-out that forces the whole suite. There is no e2e-tia label and never was. Install selection, when it is wired, has to pick its own default.

Supersedes #98 and #759. Both predate the current Package/PackageSource/bundles.enabledPackages model, and their goals (a basic install smoke test, scenario-based regression coverage) are covered by the tasks below.

Tasks

Track A, diff-driven install

• Land hack/e2e-apps/install.deps plus per-app .deps leaf files with a static validator. Superseded by test(e2e): diff-driven install selector select-install.sh #3279. The selector reuses the real PackageSource graph in packages/core/platform/sources/ directly, so there are no parallel dependency files to keep in sync.
• Add hack/select-install.sh and hack/select-install_test.bats. Delivered in test(e2e): diff-driven install selector select-install.sh #3279. It maps the suite names select-e2e.sh emits back to their *-application PackageSource, walks spec.variants[].dependsOn forward to the transitive closure (seeding cozystack.cozystack-engine), and has a --validate mode for graph reachability, cycles and the suite-to-source mapping. It runs on every PR through make unit-tests, and hack/select-e2e_test.bats asserts the suite-to-source round trip in both directions (ci: select-e2e maps only *-application sources, so kuberture and securitygroup suites are never selected by dependency #3665).
• Land feat(platform, tests): select E2E packages and declare controller dependencies #4022: select-install.sh --disabled <suites> emits the bundles.disabledPackages list for an ordinary variant, plus the mongodb-application and seaweedfs-application dependsOn edges it needs and the kubernetes-nodes-application mapping. Merged on 2026-10-07.
• Add the test-minimal bundle variant and the bundles.testMinimal knob. Superseded by feat(platform, tests): select E2E packages and declare controller dependencies #4022. packages/core/platform/templates/bundles/ still holds only iaas, naas, paas and system, and nothing needs a new one.
• Wire hack/e2e-install-cozystack.bats to read COZYSTACK_ENABLED_PACKAGES. Obsolete mechanism. The install Package is generated by hack/e2e-platform-packages.sh now, and it already emits a disabledPackages list for the container lane (LINSTOR and kubevirt-cdi). The wiring task is the next one.
• Wire select-install.sh --disabled into hack/e2e-platform-packages.sh and add a Select install components step to the PR and fork workflows. Done for the PR and fork lanes, which are where install time is paid on every push. e2e-tag, nightly and e2e-proxmox do not get it. full_suite_pattern in hack/select-e2e.sh lists four of the five Chainsaw workflows and not e2e-proxmox. A full-suite answer from select-e2e.sh does not become an empty disable list, the selector still drops the packages no suite needs.
• Capture install timing durably: hack/install-summary.sh or a step in cozyreport.sh writing install.duration_seconds and the HR timings, with a regression detector. Not started, hack/install-summary.sh does not exist. The 62-job number above was read by hand from step durations (gh run view --json jobs), so it is not reproducible from the repo. Keep this task only if we want the number to stay visible.
• Document install-selection semantics in .github/PULL_REQUEST_TEMPLATE.md and docs/agents/e2e-testing.md. Test-selection semantics are already in the TIA section of that doc, including the escalation lists, the full-e2e label and the per-path coverage rule. The PR template has no e2e section.
• Soak (no code change, ~2 weeks): median install time on scoped versus full runs, require at least 50% reduction on app-only runs and zero green-on-full / red-on-minimal divergences before promotion. The wiring merged on 2026-10-07, no soak data is published yet.
• Promote to default in a separate PR after soak data is published. Test selection is already default-on, so this is about the install side only.

Track B, upgrade scenario

• Land test(e2e): add chainsaw-native release upgrade testing lane #3276, the Chainsaw upgrade lane (install previous, seed canary data, upgrade, assert survival, data integrity, HR reconciliation and migration stamp) as a parallel non-blocking upgrade-e2e job. Open, conflicting, CHANGES_REQUESTED on 2026-09-29. The review lists what has to change: commit trailers the contributing rules don't allow (13 of 19 commits), a lane built on QEMU and on artifacts the PR pipeline no longer produces, a baseline release that lacks the linstor drbd.enabled switch so it still needs QEMU, and its own HR wait next to cozy_wait_all_helmreleases_ready. test(e2e): wire the 1.5→1.6 etcd upgrade job into CI as the authoritative regression guard #3316 was closed as stale on 2026-10-01 (it asked for a 1.5 to 1.6 job, v1.6.4 is stable and main is v1.7.0-alpha.3). e2e: no upgrade test keeps a VMI-only hotplug volume attached across the switch to declarative hotplug #4463 (hotplug volume across an upgrade) needs this lane too.
• Make the upgrade lane share the install telemetry so install and upgrade timings live in one place. e2e-tag.yaml already uploads cozyreport.tgz. Jobs to sequence against are rc-e2e, generate-changelog and update-website-docs in tags.yaml, and changelog and website-docs in promote-rc.yaml.

Track C, install smoke scenario (replaces #98)

• hack/e2e-install-cozystack.bats is the canonical install smoke scenario that Add simple test with kind #98 asked for. docs/agents/e2e-testing.md (the "install gate must have teeth" section) names it as the install gate with the fail-if-any-HR-not-Ready contract, implemented by cozy_wait_all_helmreleases_ready. The kind + paas-hosted approach from Add simple test with kind #98 is obsolete.

Related work tracked elsewhere

• test(e2e): migrate the E2E app suite from BATS to Kyverno Chainsaw #2979, BATS to Chainsaw migration. Closed 2026-08-14. 23 suites are runnable now, backup and kubernetes-proxmox are parked as chainsaw-test.yaml.disabled. The two monitoring OIDC BATS suites that were never ported were deleted in test(e2e): remove unused monitoring suites and audit BATS runners #4019, and test(e2e): cover monitoring OIDC modes with a Chainsaw suite #4678 (2026-10-05) added a Chainsaw suite for monitoring OIDC in hack/e2e-chainsaw/monitoring/.
• select-e2e.sh: a per-suite test edit silently defeats the full-suite escalation #3330, e2e: an empty TIA selection concludes "E2E Tests" green with no suite run #3392, ci: select-e2e maps only *-application sources, so kuberture and securitygroup suites are never selected by dependency #3665, selection bugs fixed by fix(ci): close the silent-green gaps in e2e selection and the helm unit sweep #3670 and fix(ci): decide E2E test coverage per changed path #3441. Two follow-ups: select-e2e: a per-suite path naming no runnable suite has its escalation swallowed by any other selection #3816 (a per-suite path naming no runnable suite swallowed another escalation, fixed in fix(ci): close e2e suite selection gaps in select-e2e and select-install #4634 on 2026-10-01 together with the seaweedfs mapping and chainsaw-test.yml discovery) and e2e test selection: a package that maps to some suites never escalates, so suites that also cover it are dropped silently #3985 (the ingress mapping is fixed in fix(networking): prevent webhook admission outages during namespace churn #3997, the structural part is still open).
• test(e2e): lightweight E2E cluster (kind / Talos-in-Docker) for diff-scoped PR runs #3238, lightweight E2E cluster. Delivered as test(e2e): run the merge-gating lanes on Talos containers #4020 in another shape, Talos containers for every PR, and closed.
• Audit e2e Chainsaw post-HR-Ready asserts for redundancy under Flux v2.8 kstatus #2653, post-HR-Ready wait audit, now on the Chainsaw suites.
• ci(workflows): run the E2E suite on approval instead of on every push #4005 (run the suite on approval), test(e2e): add a broad selector tier that withholds the tenant-k8s suites #4008 (broad selector tier), test(e2e): stop bringing up the monitoring instance nothing looks at #4023 (stop bringing up the monitoring instance in e2e). ci(workflows): run the E2E suite on approval instead of on every push #4005 and test(e2e): add a broad selector tier that withholds the tenant-k8s suites #4008 are open, test(e2e): stop bringing up the monitoring instance nothing looks at #4023 merged on 2026-10-02. All three cut PR-lane cost without touching what suites assert.
• No CI coverage for the isp-hosted platform variant #3390, no CI coverage for the isp-hosted variant, closed on 2026-10-05 by ci(e2e): add a daily isp-hosted install lane on kind #4712, a daily isp-hosted install lane on kind. feat(platform): add slim installation variants #4595 merged slim variants that no suite runs.

Prerequisites

• feat(ci): test-impact analysis E2E (default-on) + release E2E workflow #2559, merged 2026-06-09. hack/select-e2e.sh and hack/select-e2e_test.bats on main, invoked from pull-requests.yaml and e2e-fork.yaml.

Out of scope

• Rewriting the BATS harness or the Talos-based runner provisioning.
• Production install-path behaviour: the default, isp-full, isp-hosted and isp-full-generic variants are untouched.