Repository navigation
ci(e2e): run nightly and release-tag e2e on the container lane - #4828
Merged
Merged
Conversation
Nightly and the rc gate still booted the management nodes as QEMU guests, the substrate the merge-gating lanes left behind. On it the tenant Kubernetes suites have missed their node-join deadline on most nights since 2026-10-02, each failure spends the whole deadline, and the job reaches its timeout before the alphabetically later suites run. Over the same days those suites passed on every PR run that selected them, where the nodes are Talos containers and a tenant worker sits one virtualization level shallower. Both lanes now use the PR lanes' container substrate. Nightly's e2e no longer waits for the nocloud disk, which it does not boot; the disk is still built and published. A manual e2e-tag dispatch against a tag that predates the container lane fails early and names the ref to dispatch from instead. DRBD, the replicated StorageClass and the Cozystack Talos node image lose their last e2e coverage with this. The docs and comments that pointed at nightly and the rc gate as that coverage now say so. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
The comment on every lane claimed the budget fits two tenant Kubernetes suites failing at their node-join deadline. A green full suite on the container lane already spends most of it, and a suite that runs into the deadline costs about 25 minutes more than one that passes, so the claim does not hold. State the real bound so a cancelled run is read as a red suite, not as a budget to raise. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
Aleksei Sviridkin (lexfrei)
requested a review
from Andrei Kvapil (kvaps)
as a code owner
October 9, 2026 16:25
Aleksei Sviridkin (lexfrei)
added
the
kind/backport
Aleksei Sviridkin (lexfrei)
requested review from
IvanHunters,
Timofei Larkin (lllamnyp),
myasnikovdaniil and
Timur Tukaev (tym83)
as code owners
October 9, 2026 16:25
github-actions
Bot
added
area/testing
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configuration
📒 Files selected for processing (11)
Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 7 remain after this review. 📝 WalkthroughWalkthroughNightly and tagged E2E jobs now use Talos containers instead of booting the nocloud disk. The workflows prepare and remove container clusters, use local storage with DRBD disabled, and update checks and documentation to describe the resulting lane coverage. ChangesE2E lane transition
Priority: ➖ Normal Estimated code review effort: 3 (Moderate) | ~20 minutes Change: Other Sequence Diagram(s)sequenceDiagram
participant NightlyE2EWorkflow
participant GitHubRunner
participant GHCR
participant ContainerCluster
participant E2ESuite
NightlyE2EWorkflow->>GitHubRunner: Start E2E after mirror
GitHubRunner->>GHCR: Log in and stage the published closure
GitHubRunner->>ContainerCluster: Load modules and run prepare-env-container
GitHubRunner->>ContainerCluster: Install with DRBD disabled and local storage
GitHubRunner->>E2ESuite: Run the full E2E suite
GitHubRunner->>ContainerCluster: Delete the container cluster
Merge Risk: ⚪ Minimal · up to The container-lane transition is mergeable after normal workflow checks. The loss of disk- and DRBD-specific CI coverage is documented and intentional. 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
IvanHunters
approved these changes
Oct 9, 2026
Aleksei Sviridkin (lexfrei)
enabled auto-merge
October 9, 2026 16:26
Contributor
Author
|
The |
|
Git push to origin failed for release-1.6 with exitcode 1 |
1 of 11 tasks
myasnikovdaniil
added a commit
that referenced
this pull request
Oct 14, 2026
…e container lane (#4913) ## What this PR does Backport of #4828 to `release-1.6`. PR lane on this line runs on Talos containers since the backport of #4020, but `e2e-tag.yaml` (the rc gate that `tags.yaml` calls for every rc from `release-1.6`) and `nightly.yaml` still boot QEMU guests. Every rc tag run here since v1.6.2-rc.1 is red. v1.6.4-rc.1 failed with kubernetes-latest and kubernetes-previous spending about 32 minutes each at node-join deadline (https://github.com/cozystack/cozystack/actions/runs/36124680874), while the same 41 suites passed on container lane in the v1.6.4 promotion PR, in a 93 minute job (https://github.com/cozystack/cozystack/actions/runs/36383277279). e2e plumbing on `release-1.6` diverged from `main` after #4020, so the cherry-pick of the first commit has conflicts resolved against this branch, and there is one prerequisite commit: - `e2e-tag.yaml` and `nightly.yaml` get the same host module step, `prepare-env-container`, endpoint and storage class env and two-part teardown as `pull-requests.yaml` here. `e2e-tag.yaml` still requires the release to carry its nocloud disk but doesn't download it anymore, and a dispatch against a tag cut before container lane (v1.6.3 and older) fails early and says to dispatch with `--ref <tag>`. - nightly moves to 32-vCPU runner, because `hack/e2e-container-up.sh` refuses a 24-vCPU host (`main` did the same earlier in db2ddfa). Tests: `platform-packages_test.bats` checks container lane env on all three lanes, `promote-gate-contract.bats` covers the new guard (on `main` it lives in `collect-images_test.bats`, there is no such file here). Differences from `main`: every lane keeps `timeout-minutes: 180`, the follow-up comment fix about 215-minute budget is not included (this branch has no such comment and no such value), fork lane wording is dropped because there is no fork split here, docs edits cover only paragraphs that exist on this branch. Coverage lost on this line: DRBD and the `replicated` StorageClass, Cozystack Talos node image with its extensions, CDI stock 600M worker memory limit. Docs and comments that pointed at the rc run as that coverage now say so. `nightly.yaml` runs only from `main`, so the change there is for parity. `tags.yaml` takes `e2e-tag.yaml` from the commit of the rc tag, so this has to merge before v1.6.5-rc.1 is cut. If rc is cut first, dispatching `e2e-tag.yaml` from `release-1.6` with that tag gives the same evidence. `make bats-unit-tests` passes locally (487 ok, base has 486), but the workflows themselves were not run yet. This stays a draft until `e2e-tag.yaml` dispatched from this branch against v1.6.4-rc.1 is green, I will put the run link here. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note ci(e2e): on release-1.6, the release-tag e2e gate and nightly run on Talos containers instead of QEMU guests. DRBD, the replicated StorageClass and the Cozystack Talos node image are no longer covered by any e2e lane on this line. ```
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
Nightly and the release-tag e2e (
e2e-tag.yaml, the rc gate) now run on the Talos-container substrate the PR lanes moved to in #4020. This is the interim step from #4642, not the whole issue, so it does not close it.On QEMU the nightly full suite has been cancelled at its 215-minute timeout on every night since 2026-10-01 except 2026-10-07. From 2026-10-02 on, kubernetes-latest and kubernetes-previous failed their tenant node-join deadline almost every night, each failure took about 35 minutes, and the suites after them alphabetically never ran. In the 18 PR runs since 2026-10-03 that I checked and that ran them, they passed every time, at 7 to 10 minutes each. A nightly dispatched from this branch (run 37934155231) passed all 62 tests in 193 minutes. The kubernetes suites took 9 to 11 minutes there.
Changes:
nightly.yamlande2e-tag.yamlget the same host module step,prepare-env-container, endpoint and storage-class env and two-part teardown aspull-requests.yaml. Nightly's e2e no longer waits for the nocloud disk, which it does not boot. The disk is still built and published.e2e-tag.yamlstill requires the release to carry its disk but no longer downloads it. A dispatch against a tag cut before the container lane (v1.6.3 and older, v1.7.0-alpha.1) fails early and says to dispatch with--ref <tag>. Whentags.yamlcalls it, it runs the copy from the tag's own commit, so the guard does not fire there.timeout-minutes: 215in all four e2e workflows claimed the budget covers two failing tenant Kubernetes suites. It does not, and the comment now says so. The value is unchanged.platform-packages_test.batsnow checks the container-lane env on all four workflows, andcollect-images_test.batscovers the new guard.What loses coverage: DRBD and the
replicatedStorageClass (Immediate binding, tenant StorageClass propagation, RWX), the Cozystack Talos node image with its extensions, and the stock 600M CDI importer limit. None of them can run on containers, so no CI lane tests them until #4642 gets a lane with a kernel per node. The docs and comments that pointed at nightly and rc as that coverage now say this.Release lines keep their own
e2e-tag.yaml, so rc tags on release-1.6 stay on QEMU until this is backported. That is whykind/backportis on this PR. Leftover QEMU-only text and diagnostics are collected in #4825.A dispatch of
e2e-tag.yamlfrom this branch againstv1.7.0-alpha.3is running as run 37958852914. I'll post the result here.Screenshots
Downstream repositories
Release note
Summary by CodeRabbit