Skip to content

Commit 6d07a75

Browse files
test(e2e): run the merge-gating lanes on Talos containers
The srv1-srv3 nodes of both lanes that gate a merge -- the same-repo one in pull-requests.yaml and the fork one in e2e-fork.yaml -- run as Talos containers instead of QEMU guests, so a tenant worker sits at L2 rather than L3. Measured in CI on commits where both substrates ran the same tree: whole-job wall clock down 24-49 minutes, essentially all of it in Chainsaw, while the two tenant-Kubernetes suites go from 1143-2040s to 466-809s and stop depending on the runner's vgif flag. The node-join soft-red gate goes with the substrate, which is what it was for. It existed because nested virtualisation on the shared runners degraded far enough that a worker registering in minutes elsewhere could miss any deadline this test could afford; #3513 established that the discriminator was the host's kvm_amd vgif flag, and removing a nesting level removes what it acted on. So hack/e2e-node-join-soft-red.sh, the SOFT-RED-node-join.txt marker and the soft_red job outputs are gone from all four workflows, and a missed deadline is an ordinary red again. The ::warning annotation survives, because the reason a run went red should be legible without opening the log, and so does the 124-versus-everything -else test, because "the workers were slow" and "the wait never ran" send a reader to different places. Done by swapping the substrate inside the existing e2e jobs rather than by promoting a separate container job. Those jobs carry more than a substrate -- the GitHub App token, the report-overrun guard, the images list, the SSH breakpoint -- and promoting a job that never had them would have dropped four features without saying so. Three properties of container mode drive the design, and each fails quietly rather than loudly. machine.kernel.modules is a silent no-op -- kernel_module_spec.go returns early on ModeContainer with no error and no event -- so the host loads openvswitch and zfs before the nodes start and the workflow step asserts both rather than trusting them; a missing module surfaces an hour later looking like a CNI or a storage regression. Docker gives a privileged container its own tmpfs /dev seeded once at creation, so /dev/kvm works but ZFS zvols created later never appear and linstor-csi fails ControllerPublishVolume; the compose file binds the host devtmpfs, which is also why talosctl cluster create docker can never carry LINSTOR. And there is no ARP VIP, so 192.168.123.11 stands in for the QEMU lane's .10. The QEMU machinery stays in the tree and keeps running: nightly.yaml and e2e-tag.yaml still call `make prepare-env`, so DRBD, the replicated StorageClass, its Immediate binding mode, the tenant StorageClass propagation that only applies to remotely-accessible classes, and the Cozystack Talos node image with its extensions all keep nightly and release-candidate coverage. None of them is exercised per pull request any more. That is the trade. Nothing on the PR path builds or downloads a nocloud disk now, so build-talos drops the talos-nocloud step and the talos-image artifact. build-talos itself stays: finalize consumes its digest fragment. One property this does not fix, recorded beside the runner class because it will be read off a red run eventually: a container node reports the HOST's CPU and memory as capacity, and hack/e2e-container-up.sh counters that with kubelet systemReserved. That corrects the scheduler and not what a pod reads from /proc, so a workload sizing itself from /proc/cpuinfo or /proc/meminfo is configured for the whole runner. Also carried here because the same files carry them: the ghcr.io pull-through mirror removal (see #4007 -- whichever lands first makes the other a no-op), the authoritative HelmRelease readiness gate, the backup-credential preflight, the prepull guard against an image-less container fragment, and folding the two standalone OIDC suites into the latest-version tenant cluster. Signed-off-by: Myasnikov Daniil <[email protected]>
1 parent bf0d8c8 commit 6d07a75

56 files changed

Lines changed: 3478 additions & 4246 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/e2e-fork.yaml‎

Lines changed: 53 additions & 41 deletions
Original file line numberDiff line numberDiff line change
@@ -540,17 +540,12 @@ jobs:
540540
name: E2E (fork run)
541541
needs: [resolve, publish]
542542
if: ${{ needs.resolve.outputs.should_run == 'true' }}
543-
# What the gate decided, promoted to a job output because the job's RESULT
544-
# can no longer answer it: a red the lane tolerated finishes `success`, so
545-
# the status this lane reports would otherwise call that run a pass.
546-
outputs:
547-
soft_red: ${{ steps.e2e_tests.outputs.soft_red }}
543+
# Talos nodes as containers, the same substrate pull-requests.yaml uses; the
544+
# per-node 8 CPU / 24 GiB in hack/e2e-compose.yaml is why the class is this
545+
# one. See the `e2e` job there for the full sizing note.
548546
runs-on: oracle-vm-32cpu-128gb-x86-64
549547
# Sized above what the tenant-Kubernetes pair can spend on a failing run, and
550-
# held to one figure across every lane that runs the suite. That bound, the
551-
# measured run the rest of the margin comes from, and what neither of them
552-
# covers are all in hack/node-join-soft-red_test.bats rather than restated
553-
# here in four workflows that would then drift apart.
548+
# held to one figure across every lane that runs the suite.
554549
timeout-minutes: 215
555550
permissions:
556551
contents: read
@@ -582,13 +577,6 @@ jobs:
582577
- name: Set up toolchain (yq)
583578
run: |
584579
command -v yq >/dev/null || { sudo curl -sSL -o /usr/local/bin/yq https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 && sudo chmod +x /usr/local/bin/yq; }
585-
- name: Download Talos image
586-
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
587-
with:
588-
name: talos-image
589-
path: _out/assets
590-
github-token: ${{ secrets.GITHUB_TOKEN }}
591-
run-id: ${{ needs.resolve.outputs.run_id }}
592580
- name: Download pr.patch
593581
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
594582
with:
@@ -606,15 +594,50 @@ jobs:
606594
run: |
607595
yq -i ".cozystackOperator.platformSourceUrl = \"oci://${REGISTRY}/cozystack-packages\"" packages/core/installer/values.yaml
608596
yq -i ".cozystackOperator.platformSourceRef = \"digest=${PACKAGES_DIGEST}\"" packages/core/installer/values.yaml
597+
# Mandatory, and up front: the nodes take their kernel from this host and
598+
# Talos cannot load modules in container mode, so a module missing here is
599+
# missing for the whole run and surfaces later as a CNI or storage failure.
600+
# See the same step in pull-requests.yaml for the full rationale.
601+
- name: Prepare container-lane host
602+
run: |
603+
set -eu
604+
for m in openvswitch zfs; do
605+
if grep -q "^${m} " /proc/modules; then
606+
echo "${m}: already loaded"
607+
continue
608+
fi
609+
if [ "${m}" = zfs ]; then
610+
sudo apt-get update -qq
611+
sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq zfsutils-linux
612+
fi
613+
sudo modprobe "${m}" || echo "${m}: modprobe failed"
614+
done
615+
missing=''
616+
for m in openvswitch zfs; do
617+
grep -q "^${m} " /proc/modules || missing="${missing} ${m}"
618+
done
619+
if [ -n "${missing}" ]; then
620+
echo "::error title=container-lane host::kernel module(s) could not be loaded:${missing}"
621+
echo "This is a host problem, not a product failure: openvswitch backs the" >&2
622+
echo "kube-ovn dataplane and zfs backs every LINSTOR pool, and container-mode" >&2
623+
echo "Talos cannot load either one itself." >&2
624+
exit 1
625+
fi
626+
grep -E '^(openvswitch|zfs) ' /proc/modules
609627
- name: Prepare environment
610628
run: |
611629
attempt=0
612-
until make SANDBOX_NAME=$SANDBOX_NAME prepare-env; do
630+
until make SANDBOX_NAME=$SANDBOX_NAME prepare-env-container; do
613631
attempt=$((attempt + 1))
614632
if [ $attempt -ge 3 ]; then echo "Attempt $attempt failed, exiting..."; exit 1; fi
615633
echo "Attempt $attempt failed, retrying..."
634+
make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME delete-cluster-container || true
616635
done
617636
- name: Install Cozystack into sandbox
637+
env:
638+
COZY_APISERVER_ENDPOINT: https://192.168.123.11:6443
639+
COZY_LINSTOR_DRBD_ENABLED: "false"
640+
COZY_E2E_STORAGE_CLASS: local
618641
run: |
619642
if ! make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME install-cozystack; then
620643
echo "❌ Install failed (no retry — diagnostics below)"
@@ -674,6 +697,10 @@ jobs:
674697
id: e2e_tests
675698
if: ${{ steps.select.outputs.skip != 'true' }}
676699
env:
700+
# Must match the install step above: installing on `local` while
701+
# templating the Chainsaw fixtures with `replicated` fails deep inside
702+
# a suite instead of at the mismatch.
703+
COZY_E2E_STORAGE_CLASS: local
677704
SELECTED_APPS: ${{ steps.select.outputs.apps }}
678705
FULL_E2E: ${{ needs.resolve.outputs.full_e2e }}
679706
run: |
@@ -699,19 +726,6 @@ jobs:
699726
kubectl get bucketaccesses.objectstorage.k8s.io -A -o custom-columns="NAMESPACE:.metadata.namespace,NAME:.metadata.name,GRANTED:.status.accessGranted,CLAIM:.spec.bucketClaimName" 2>&1
700727
' || true
701728
echo "::endgroup::"
702-
# One failure the lane does not block on, and only one: the tenant
703-
# node-join deadline. The test itself stays red -- Chainsaw ran its
704-
# whole catch above and the report records the failure -- and this
705-
# decides only whether the job does. It blocks on anything it cannot
706-
# attribute to that deadline, including a report it could not read.
707-
if hack/e2e-node-join-soft-red.sh "$SANDBOX_NAME"; then
708-
# Recorded because this exit makes the job succeed, and the status
709-
# this lane reports back to the pull request is derived from that
710-
# result: without this the required check would say the suite
711-
# passed on a run where it failed.
712-
echo "soft_red=true" >>"$GITHUB_OUTPUT"
713-
exit 0
714-
fi
715729
exit 1
716730
fi
717731
echo "✅ All selected E2E tests passed"
@@ -739,9 +753,15 @@ jobs:
739753
name: cozyreport
740754
path: _out/cozyreport.tgz
741755
if-no-files-found: warn
756+
# Both halves, and in this order: the compose project owns the nodes and
757+
# the zpools outlive it.
742758
- name: Tear down sandbox
743759
if: always()
744-
run: make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME delete || true
760+
run: |
761+
cleanup_rc=0
762+
make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME delete-cluster-container || cleanup_rc=$?
763+
make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME delete || true
764+
exit "$cleanup_rc"
745765
746766
# Flip the pending "E2E Tests" commit status posted by `resolve` to its
747767
# terminal state. Runs on failure too, so a failed publish or e2e turns the
@@ -769,22 +789,14 @@ jobs:
769789
env:
770790
PUBLISH_RESULT: ${{ needs.publish.result }}
771791
E2E_RESULT: ${{ needs.e2e.result }}
772-
E2E_SOFT_RED: ${{ needs.e2e.outputs.soft_red }}
773792
HEAD_SHA: ${{ needs.resolve.outputs.head_sha }}
774793
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
775794
with:
776795
script: |
777-
const { PUBLISH_RESULT, E2E_RESULT, E2E_SOFT_RED, HEAD_SHA, RUN_URL } = process.env;
796+
const { PUBLISH_RESULT, E2E_RESULT, HEAD_SHA, RUN_URL } = process.env;
778797
const { owner, repo } = context.repo;
779798
let state, description;
780-
if (E2E_RESULT === 'success' && E2E_SOFT_RED === 'true') {
781-
// The suite did NOT pass: the tenant node-join deadline expired and
782-
// the lane decided not to block on it. `success` is the decision;
783-
// this text is the only place a reader standing in front of the
784-
// merge button learns what it was a decision about.
785-
state = 'success';
786-
description = 'fork e2e soft: node-join deadline missed, suite failed, merge not blocked';
787-
} else if (E2E_RESULT === 'success') {
799+
if (E2E_RESULT === 'success') {
788800
state = 'success';
789801
description = 'fork e2e passed (privileged run)';
790802
} else if (PUBLISH_RESULT !== 'success') {

‎.github/workflows/e2e-tag.yaml‎

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -135,10 +135,7 @@ jobs:
135135
# agent while the full platform installs across all three guests.
136136
runs-on: oracle-vm-32cpu-128gb-x86-64
137137
# Sized above what the tenant-Kubernetes pair can spend on a failing run, and
138-
# held to one figure across every lane that runs the suite. That bound, the
139-
# measured run the rest of the margin comes from, and what neither of them
140-
# covers are all in hack/node-join-soft-red_test.bats rather than restated
141-
# here in four workflows that would then drift apart.
138+
# held to one figure across every lane that runs the suite.
142139
timeout-minutes: 215
143140
permissions:
144141
contents: read

‎.github/workflows/nightly.yaml‎

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -228,10 +228,7 @@ jobs:
228228
# hack/sandbox-runner-headroom.bats.
229229
runs-on: oracle-vm-32cpu-128gb-x86-64
230230
# Sized above what the tenant-Kubernetes pair can spend on a failing run, and
231-
# held to one figure across every lane that runs the suite. That bound, the
232-
# measured run the rest of the margin comes from, and what neither of them
233-
# covers are all in hack/node-join-soft-red_test.bats rather than restated
234-
# here in four workflows that would then drift apart.
231+
# held to one figure across every lane that runs the suite.
235232
timeout-minutes: 215
236233
permissions:
237234
contents: read

0 commit comments

Comments
 (0)