Skip to content

Commit 94a202b

Browse files
test(e2e): run the merge-gating lanes on Talos containers (#4020)
Replaces QEMU substrate with Talos containers on both lanes that gate a merge. srv1-srv3 run as containers, so tenant worker sits at L2 instead of L3. Measured in CI on commits where both substrates ran the same tree: | | qemu | container | |---|---|---| | `kubernetes-latest` / `-previous` | 1143-2040s, red on every vgif-absent runner | 466-809s, green in both strata | | whole job | | 24-49 min faster | Almost all of it is in Chainsaw. Install takes about half an hour either way. Node-join soft-red gate goes with it. It was a prosthetic for nested virt: a worker that registers in minutes elsewhere could miss any deadline this test can afford, and #3513 established the discriminator is host `kvm_amd` `vgif`, not anything about the product. Remove the nesting level and there is nothing left to tolerate, so `hack/e2e-node-join-soft-red.sh`, the marker and the `soft_red` outputs are gone from all four workflows. ### What moves off the per-PR path DRBD, and with it `replicated` StorageClass, its `Immediate` binding mode, tenant StorageClass propagation (applies only to remotely-accessible classes), and cozystack talos node image with its extensions. None of it is lost - `nightly.yaml` and `e2e-tag.yaml` still run the suite on QEMU, so all four keep nightly and release-candidate coverage. Live migration is not on that list because nothing tests it today anyway. One more belongs on the list and was missing from it: the nocloud disk build. `build-talos` no longer runs `make -C packages/core/talos talos-nocloud`, so `images/talos/profiles/nocloud.yaml` is compiled by `nightly.yaml` and by the tag build's `make assets` rather than by anything a PR runs, and `promote-rc.yaml` still hard-fails a promotion whose rc release has no `nocloud-amd64.raw.xz`. Nothing in that chain breaks, because the tag build supplies the asset as before. What a PR no longer catches is narrow: it still compiles `installer.yaml` through the same imager, and the two profiles are generated by the same script and differ only in `platform`, `kind`, `imageOptions` and `outFormat`. ### Product fixes Four, all found by measurement while building the lane, each on its own commit: The linstor `drbd.enabled` gate. Without it drbd-logger sidecar exits, satellite is never Ready, piraeus registers zero nodes and every PVC hangs Pending behind a cluster that looks healthy. linstor `--strict-topology`. Without it external-provisioner passes every topology segment as `requisite`, so a node-pinned volume can be provisioned away from its pod and the pod is then unschedulable forever with nothing erroring anywhere. kubevirt-cdi importer resources are configurable now. Default restates CDI's own values exactly so production behaviour does not move, and the lane raises the ceiling through its own Package. vm-disk binds standalone disks immediately. A VMDisk on `local` silently never populated, while the same disk on `replicated` did. ### Where the run stands The lane has gone green twice, most recently on `cc70cef7b` where `E2E (in-tree)` took 2h3m against its 215m cap with 45 Chainsaw tests passing. The measurements quoted elsewhere in this description come from the first of those runs, on a commit that has since been rebased away: the two tenant-Kubernetes suites came in at 569.72s (`kubernetes-latest`) and 470.19s (`kubernetes-previous`) against a 67m operation. Both runs answer the four earlier reds, all of which were the lua-protobuf segfault from #4012 once `vminstance`/`vmdisk` were fixed on the branch. ghcr.io mirror removal is carried here too and is the same change as #4007, whichever lands first makes the other a no-op. ### What review changed `drbd.enabled=false` now refuses to render together with `talos.enabled=false`. It removed the logger sidecar only, while the two DRBD initContainers piraeus contributes unconditionally are deleted by the Talos configuration alone, so that combination produced a satellite that can never reach Ready. `isp-full-generic` sets `talos.enabled=false`, so it was reachable and not theoretical. vm-disk keeps unconditional immediate binding, a standalone disk has no consumer to wait for. What it costs on a node-pinned class is in the chart README now, and the annotation is pinned per source type and on the existing-DataVolume path where `helm upgrade` retrofits it. The rest is the harness holding its own promises: HelmRelease gate reads are bounded and its minimum-count half is pinned by a test that fails without it, two Chainsaw ops sit above the deadlines they contain, the fallback-StorageClass render restores `replicated` when it fails, container lane compose project and pool files are per-checkout, and three comments that said the opposite of the code now say what is true (QEMU is still wired into nightly and e2e-tag, the lane replaces two Packages and not one, the argument-count assertion cannot see upstream drift). ### Release note ```release-note fix(linstor): CSI provisioning is constrained to the node the scheduler chose (`--strict-topology`), and the DRBD sidecar can be switched off with `drbd.enabled` on substrates whose kernel cannot run DRBD. On an existing cluster the new default re-templates `spec.csiController.podTemplate`, so piraeus rolls `linstor-csi-controller` once during the upgrade. `drbd.enabled=false` requires `talos.enabled=true` and the chart now refuses the other combination at render time, because only the Talos satellite configuration removes the DRBD initContainers. fix(kubevirt-cdi): CDI worker pod resources are configurable through `importerResources`, which reaches the importer, the uploader and the host-assisted cloner. The default restates CDI's own built-in values exactly, so an install that sets nothing behaves as before. fix(vm-disk): every disk requests immediate binding, not only `upload` sources, so a standalone disk on a WaitForFirstConsumer StorageClass is populated instead of waiting for a consumer that may never arrive. On a node-pinned class such as `local` the volume then binds where CDI's worker pod was scheduled rather than where the VM will run; see the chart README before choosing such a class for VM disks. The default `replicated` is unaffected. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added container-based end-to-end testing with local storage support. * Added optional backup-access preflight checks for database backup examples. * Added configurable CDI worker resources and LINSTOR storage settings. * Added configurable immediate storage binding for VM disks. * Added stronger readiness checks, diagnostics, and parallel test execution. * **Bug Fixes** * Improved cleanup failure reporting, capacity validation, and missing-image detection. * Improved FoundationDB health verification and storage placement behavior. * **Removed** * Removed GHCR mirror support and soft-red node-join tolerance. * Removed standalone OIDC render test suites. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 67a8674 + 62f2234 commit 94a202b

80 files changed

Lines changed: 4388 additions & 4322 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/e2e-fork.yaml‎

Lines changed: 59 additions & 41 deletions
Original file line numberDiff line numberDiff line change
@@ -540,17 +540,12 @@ jobs:
540540
name: E2E (fork run)
541541
needs: [resolve, publish]
542542
if: ${{ needs.resolve.outputs.should_run == 'true' }}
543-
# What the gate decided, promoted to a job output because the job's RESULT
544-
# can no longer answer it: a red the lane tolerated finishes `success`, so
545-
# the status this lane reports would otherwise call that run a pass.
546-
outputs:
547-
soft_red: ${{ steps.e2e_tests.outputs.soft_red }}
543+
# Talos nodes as containers, the same substrate pull-requests.yaml uses; the
544+
# per-node 8 CPU / 24 GiB in hack/e2e-compose.yaml is why the class is this
545+
# one. See the `e2e` job there for the full sizing note.
548546
runs-on: oracle-vm-32cpu-128gb-x86-64
549547
# Sized above what the tenant-Kubernetes pair can spend on a failing run, and
550-
# held to one figure across every lane that runs the suite. That bound, the
551-
# measured run the rest of the margin comes from, and what neither of them
552-
# covers are all in hack/node-join-soft-red_test.bats rather than restated
553-
# here in four workflows that would then drift apart.
548+
# held to one figure across every lane that runs the suite.
554549
timeout-minutes: 215
555550
permissions:
556551
contents: read
@@ -582,13 +577,6 @@ jobs:
582577
- name: Set up toolchain (yq)
583578
run: |
584579
command -v yq >/dev/null || { sudo curl -sSL -o /usr/local/bin/yq https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 && sudo chmod +x /usr/local/bin/yq; }
585-
- name: Download Talos image
586-
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
587-
with:
588-
name: talos-image
589-
path: _out/assets
590-
github-token: ${{ secrets.GITHUB_TOKEN }}
591-
run-id: ${{ needs.resolve.outputs.run_id }}
592580
- name: Download pr.patch
593581
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
594582
with:
@@ -606,15 +594,50 @@ jobs:
606594
run: |
607595
yq -i ".cozystackOperator.platformSourceUrl = \"oci://${REGISTRY}/cozystack-packages\"" packages/core/installer/values.yaml
608596
yq -i ".cozystackOperator.platformSourceRef = \"digest=${PACKAGES_DIGEST}\"" packages/core/installer/values.yaml
597+
# Mandatory, and up front: the nodes take their kernel from this host and
598+
# Talos cannot load modules in container mode, so a module missing here is
599+
# missing for the whole run and surfaces later as a CNI or storage failure.
600+
# See the same step in pull-requests.yaml for the full rationale.
601+
- name: Prepare container-lane host
602+
run: |
603+
set -eu
604+
for m in openvswitch zfs; do
605+
if grep -q "^${m} " /proc/modules; then
606+
echo "${m}: already loaded"
607+
continue
608+
fi
609+
if [ "${m}" = zfs ]; then
610+
sudo apt-get update -qq
611+
sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq zfsutils-linux
612+
fi
613+
sudo modprobe "${m}" || echo "${m}: modprobe failed"
614+
done
615+
missing=''
616+
for m in openvswitch zfs; do
617+
grep -q "^${m} " /proc/modules || missing="${missing} ${m}"
618+
done
619+
if [ -n "${missing}" ]; then
620+
echo "::error title=container-lane host::kernel module(s) could not be loaded:${missing}"
621+
echo "This is a host problem, not a product failure: openvswitch backs the" >&2
622+
echo "kube-ovn dataplane and zfs backs every LINSTOR pool, and container-mode" >&2
623+
echo "Talos cannot load either one itself." >&2
624+
exit 1
625+
fi
626+
grep -E '^(openvswitch|zfs) ' /proc/modules
609627
- name: Prepare environment
610628
run: |
611629
attempt=0
612-
until make SANDBOX_NAME=$SANDBOX_NAME prepare-env; do
630+
until make SANDBOX_NAME=$SANDBOX_NAME prepare-env-container; do
613631
attempt=$((attempt + 1))
614632
if [ $attempt -ge 3 ]; then echo "Attempt $attempt failed, exiting..."; exit 1; fi
615633
echo "Attempt $attempt failed, retrying..."
634+
make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME delete-cluster-container || true
616635
done
617636
- name: Install Cozystack into sandbox
637+
env:
638+
COZY_APISERVER_ENDPOINT: https://192.168.123.11:6443
639+
COZY_LINSTOR_DRBD_ENABLED: "false"
640+
COZY_E2E_STORAGE_CLASS: local
618641
run: |
619642
if ! make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME install-cozystack; then
620643
echo "❌ Install failed (no retry — diagnostics below)"
@@ -674,6 +697,10 @@ jobs:
674697
id: e2e_tests
675698
if: ${{ steps.select.outputs.skip != 'true' }}
676699
env:
700+
# Must match the install step above: installing on `local` while
701+
# templating the Chainsaw fixtures with `replicated` fails deep inside
702+
# a suite instead of at the mismatch.
703+
COZY_E2E_STORAGE_CLASS: local
677704
SELECTED_APPS: ${{ steps.select.outputs.apps }}
678705
FULL_E2E: ${{ needs.resolve.outputs.full_e2e }}
679706
run: |
@@ -699,19 +726,6 @@ jobs:
699726
kubectl get bucketaccesses.objectstorage.k8s.io -A -o custom-columns="NAMESPACE:.metadata.namespace,NAME:.metadata.name,GRANTED:.status.accessGranted,CLAIM:.spec.bucketClaimName" 2>&1
700727
' || true
701728
echo "::endgroup::"
702-
# One failure the lane does not block on, and only one: the tenant
703-
# node-join deadline. The test itself stays red -- Chainsaw ran its
704-
# whole catch above and the report records the failure -- and this
705-
# decides only whether the job does. It blocks on anything it cannot
706-
# attribute to that deadline, including a report it could not read.
707-
if hack/e2e-node-join-soft-red.sh "$SANDBOX_NAME"; then
708-
# Recorded because this exit makes the job succeed, and the status
709-
# this lane reports back to the pull request is derived from that
710-
# result: without this the required check would say the suite
711-
# passed on a run where it failed.
712-
echo "soft_red=true" >>"$GITHUB_OUTPUT"
713-
exit 0
714-
fi
715729
exit 1
716730
fi
717731
echo "✅ All selected E2E tests passed"
@@ -739,9 +753,21 @@ jobs:
739753
name: cozyreport
740754
path: _out/cozyreport.tgz
741755
if-no-files-found: warn
756+
# Both halves, and in this order: the compose project owns the nodes and
757+
# the zpools outlive it.
742758
- name: Tear down sandbox
743759
if: always()
744-
run: make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME delete || true
760+
run: |
761+
# Annotated, not fatal: same reasoning as the in-tree lane. The runner
762+
# is discarded after the job, so a dirty host has no next tenant, and
763+
# failing here would post the required "E2E Tests" status as a failure
764+
# indistinguishable from a real one.
765+
cleanup_rc=0
766+
make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME delete-cluster-container || cleanup_rc=$?
767+
make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME delete || true
768+
if [ "$cleanup_rc" -ne 0 ]; then
769+
echo "::error title=sandbox teardown::teardown exited $cleanup_rc; the host may hold a stale compose project or zpool"
770+
fi
745771
746772
# Flip the pending "E2E Tests" commit status posted by `resolve` to its
747773
# terminal state. Runs on failure too, so a failed publish or e2e turns the
@@ -769,22 +795,14 @@ jobs:
769795
env:
770796
PUBLISH_RESULT: ${{ needs.publish.result }}
771797
E2E_RESULT: ${{ needs.e2e.result }}
772-
E2E_SOFT_RED: ${{ needs.e2e.outputs.soft_red }}
773798
HEAD_SHA: ${{ needs.resolve.outputs.head_sha }}
774799
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
775800
with:
776801
script: |
777-
const { PUBLISH_RESULT, E2E_RESULT, E2E_SOFT_RED, HEAD_SHA, RUN_URL } = process.env;
802+
const { PUBLISH_RESULT, E2E_RESULT, HEAD_SHA, RUN_URL } = process.env;
778803
const { owner, repo } = context.repo;
779804
let state, description;
780-
if (E2E_RESULT === 'success' && E2E_SOFT_RED === 'true') {
781-
// The suite did NOT pass: the tenant node-join deadline expired and
782-
// the lane decided not to block on it. `success` is the decision;
783-
// this text is the only place a reader standing in front of the
784-
// merge button learns what it was a decision about.
785-
state = 'success';
786-
description = 'fork e2e soft: node-join deadline missed, suite failed, merge not blocked';
787-
} else if (E2E_RESULT === 'success') {
805+
if (E2E_RESULT === 'success') {
788806
state = 'success';
789807
description = 'fork e2e passed (privileged run)';
790808
} else if (PUBLISH_RESULT !== 'success') {

‎.github/workflows/e2e-tag.yaml‎

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -135,10 +135,7 @@ jobs:
135135
# agent while the full platform installs across all three guests.
136136
runs-on: oracle-vm-32cpu-128gb-x86-64
137137
# Sized above what the tenant-Kubernetes pair can spend on a failing run, and
138-
# held to one figure across every lane that runs the suite. That bound, the
139-
# measured run the rest of the margin comes from, and what neither of them
140-
# covers are all in hack/node-join-soft-red_test.bats rather than restated
141-
# here in four workflows that would then drift apart.
138+
# held to one figure across every lane that runs the suite.
142139
timeout-minutes: 215
143140
permissions:
144141
contents: read

‎.github/workflows/nightly.yaml‎

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -228,10 +228,7 @@ jobs:
228228
# hack/sandbox-runner-headroom.bats.
229229
runs-on: oracle-vm-32cpu-128gb-x86-64
230230
# Sized above what the tenant-Kubernetes pair can spend on a failing run, and
231-
# held to one figure across every lane that runs the suite. That bound, the
232-
# measured run the rest of the margin comes from, and what neither of them
233-
# covers are all in hack/node-join-soft-red_test.bats rather than restated
234-
# here in four workflows that would then drift apart.
231+
# held to one figure across every lane that runs the suite.
235232
timeout-minutes: 215
236233
permissions:
237234
contents: read

0 commit comments

Comments
 (0)