Skip to content

Commit da28ea7

Browse files
lexfreilexfrei
committed
ci(e2e): run nightly and release-tag e2e on the container lane
Nightly and the rc gate still booted the management nodes as QEMU guests, the substrate the merge-gating lanes left behind. On it the tenant Kubernetes suites have missed their node-join deadline on most nights since 2026-10-02, each failure spends the whole deadline, and the job reaches its timeout before the alphabetically later suites run. Over the same days those suites passed on every PR run that selected them, where the nodes are Talos containers and a tenant worker sits one virtualization level shallower. Both lanes now use the PR lanes' container substrate. Nightly's e2e no longer waits for the nocloud disk, which it does not boot; the disk is still built and published. A manual e2e-tag dispatch against a tag that predates the container lane fails early and names the ref to dispatch from instead. DRBD, the replicated StorageClass and the Cozystack Talos node image lose their last e2e coverage with this. The docs and comments that pointed at nightly and the rc gate as that coverage now say so. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent aa804aa commit da28ea7

10 files changed

Lines changed: 166 additions & 84 deletions

File tree

‎.github/workflows/e2e-tag.yaml‎

Lines changed: 61 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -11,8 +11,10 @@ name: E2E Release Tag
1111
# This lane validates published tags only; draft and promote-time validation are
1212
# explicitly out of scope because release checks run at RC publication time.
1313
# Prerelease tags use the matching digest-pinned staging branch, while stable
14-
# tags use their pinned merge tree directly. In both cases the nocloud disk
15-
# comes from the published release identified by the same tag.
14+
# tags use their pinned merge tree directly. The nodes run as Talos containers
15+
# booted from the upstream image, so the release's nocloud disk is required to
16+
# exist but is not what the suite runs on. It is still required because users
17+
# install from it, and promote-rc.yaml checks it for an rc only.
1618

1719
on:
1820
workflow_call:
@@ -37,8 +39,8 @@ concurrency:
3739
group: e2e-tag-${{ inputs.tag }}
3840
cancel-in-progress: false
3941

40-
# Listing published releases, reading Git refs, downloading a release asset, and
41-
# checking out the pinned tree require only repository contents read access.
42+
# Listing published releases, reading Git refs and checking out the pinned tree
43+
# require only repository contents read access.
4244
# Draft releases are not part of this lane's contract. The public GHCR images
4345
# referenced by that tree need no package token.
4446
#
@@ -57,7 +59,6 @@ jobs:
5759
runs-on: ubuntu-latest
5860
timeout-minutes: 10
5961
outputs:
60-
disk_id: ${{ steps.resolve.outputs.disk_id }}
6162
tree_ref: ${{ steps.resolve.outputs.tree_ref }}
6263
steps:
6364
- name: Validate tag and resolve release
@@ -123,16 +124,15 @@ jobs:
123124
throw error;
124125
}
125126
126-
core.setOutput('disk_id', String(disk.id));
127127
core.setOutput('tree_ref', treeRef);
128128
core.info(`Resolved ${tag}: tree '${treeRef}', disk asset ${disk.id}`);
129129
130130
e2e:
131131
name: E2E ${{ inputs.tag }} (full suite)
132132
needs: resolve
133-
# The sandbox boots three 8-vCPU / 24-GiB QEMU guests. This runner leaves
134-
# enough host headroom for QEMU, networking, containerd, and the runner
135-
# agent while the full platform installs across all three guests.
133+
# The sandbox runs three 8-vCPU / 24-GiB Talos containers. This runner
134+
# leaves enough host headroom for networking, containerd, and the runner
135+
# agent while the full platform installs across all three nodes.
136136
runs-on: oracle-vm-32cpu-128gb-x86-64
137137
# Sized above what the tenant-Kubernetes pair can spend on a failing run, and
138138
# held to one figure across every lane that runs the suite.
@@ -150,45 +150,74 @@ jobs:
150150
fetch-depth: 1
151151
persist-credentials: false
152152

153-
- name: Download disk asset from release
154-
env:
155-
DISK_ID: ${{ needs.resolve.outputs.disk_id }}
156-
GH_TOKEN: ${{ github.token }}
157-
run: |
158-
mkdir -p _out/assets
159-
curl --fail-with-body --silent --show-error --location \
160-
-H "Authorization: Bearer ${GH_TOKEN}" \
161-
-H "Accept: application/octet-stream" \
162-
-o _out/assets/nocloud-amd64.raw.xz \
163-
"https://api.github.com/repos/${GITHUB_REPOSITORY}/releases/assets/${DISK_ID}"
164-
165153
- name: Set sandbox ID
166154
env:
167155
TAG: ${{ inputs.tag }}
168156
run: echo "SANDBOX_NAME=cozy-e2e-sandbox-$(echo "${GITHUB_REPOSITORY}:${GITHUB_WORKFLOW}:${TAG}" | sha256sum | cut -c1-10)" >> "$GITHUB_ENV"
169157

158+
# tags.yaml calls the e2e-tag.yaml from the tag's own commit, so this can
159+
# only fire on a dispatch from a newer ref against an older tag.
160+
- name: Require a container-lane tree
161+
env:
162+
TAG: ${{ inputs.tag }}
163+
run: |
164+
if [ ! -f hack/e2e-compose.yaml ]; then
165+
echo "::error title=e2e-tag::${TAG} predates the container lane; dispatch e2e-tag.yaml with --ref ${TAG} to run it on the lane it shipped with"
166+
exit 1
167+
fi
168+
170169
- name: Prepare workspace
171170
run: |
172171
rm -rf "/tmp/$SANDBOX_NAME"
173172
cp -r "${{ github.workspace }}" "/tmp/$SANDBOX_NAME"
174173
175-
# This is the only retried stage: Talos image unpacking, QEMU startup, and
176-
# sandbox networking are pure runner infrastructure, not product logic.
174+
# Talos cannot load kernel modules in container mode, so the nodes get
175+
# exactly what this host has loaded. Why each module, and why failing
176+
# here is better than an hour later: the same step in pull-requests.yaml.
177+
- name: Prepare container-lane host
178+
run: |
179+
set -eu
180+
for m in openvswitch zfs; do
181+
grep -q "^${m} " /proc/modules && continue
182+
if [ "${m}" = zfs ]; then
183+
sudo apt-get update -qq
184+
sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq zfsutils-linux
185+
fi
186+
sudo modprobe "${m}" || echo "${m}: modprobe failed"
187+
done
188+
missing=''
189+
for m in openvswitch zfs; do
190+
grep -q "^${m} " /proc/modules || missing="${missing} ${m}"
191+
done
192+
if [ -n "${missing}" ]; then
193+
echo "::error title=container-lane host::kernel module(s) could not be loaded:${missing}"
194+
exit 1
195+
fi
196+
197+
# This is the only retried stage: Talos container startup and sandbox
198+
# networking are pure runner infrastructure, not product logic.
177199
- name: Prepare environment
178200
run: |
179201
cd "/tmp/$SANDBOX_NAME"
180202
attempt=0
181-
until make SANDBOX_NAME=$SANDBOX_NAME prepare-env; do
203+
until make SANDBOX_NAME="$SANDBOX_NAME" prepare-env-container; do
182204
attempt=$((attempt + 1))
183205
if [ $attempt -ge 3 ]; then
184206
echo "Attempt $attempt failed, exiting..."
185207
exit 1
186208
fi
187209
echo "Attempt $attempt failed, retrying..."
210+
make -C packages/core/testing SANDBOX_NAME="$SANDBOX_NAME" delete-cluster-container || true
188211
done
189212
echo "Prepare environment completed after $((attempt + 1)) attempts"
190213
191214
- name: Install Cozystack into sandbox
215+
env:
216+
# No ARP VIP in container mode: .11 is a plain control-plane node.
217+
COZY_APISERVER_ENDPOINT: https://192.168.123.11:6443
218+
# The nodes share the runner kernel and cannot load DRBD.
219+
COZY_LINSTOR_DRBD_ENABLED: "false"
220+
COZY_E2E_STORAGE_CLASS: local
192221
run: |
193222
cd "/tmp/$SANDBOX_NAME"
194223
if ! make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME install-cozystack; then
@@ -218,6 +247,8 @@ jobs:
218247
# CHAINSAW_SUITES is the testing Makefile's full-suite mode.
219248
- name: Run E2E tests (full suite)
220249
id: e2e_tests
250+
env:
251+
COZY_E2E_STORAGE_CLASS: local
221252
run: |
222253
cd "/tmp/$SANDBOX_NAME"
223254
if ! make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME test-chainsaw CHAINSAW_SUITES=""; then
@@ -404,13 +435,17 @@ jobs:
404435
Resume from inside: `breakpoint resume`. Otherwise the breakpoint
405436
exits 10 minutes after the last SSH session disconnects.
406437
438+
# The compose project owns the nodes and the zpools outlive it, so both
439+
# halves go, in this order.
407440
- name: Tear down sandbox
408441
if: always()
409-
run: make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME delete || true
442+
run: |
443+
make -C packages/core/testing SANDBOX_NAME="$SANDBOX_NAME" delete-cluster-container || true
444+
make -C packages/core/testing SANDBOX_NAME="$SANDBOX_NAME" delete || true
410445
411446
- name: Remove workspace
412447
if: always()
413-
run: rm -rf "/tmp/$SANDBOX_NAME"
448+
run: rm -rf "/tmp/${SANDBOX_NAME:?}"
414449

415450
- name: Summarize outcome
416451
if: always()

‎.github/workflows/nightly.yaml‎

Lines changed: 48 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,8 @@ name: Nightly
1010
# cozy-installer chart are re-published on GHCR so the whole install closure is
1111
# self-contained there. The only thing genuinely built is the Talos nocloud disk
1212
# (assembled from upstream profiles, independent of the registry), published as a
13-
# GHCR OCI artifact so a nightly is installable on real hardware, not just e2e.
13+
# GHCR OCI artifact so a nightly is installable on real hardware. The nightly
14+
# e2e does not boot it.
1415
#
1516
# No GitHub release and no api/apps/v1alpha1 Go-module tag are created — a
1617
# nightly is an installable artifact set on GHCR, nothing more. retention.yaml
@@ -233,13 +234,16 @@ jobs:
233234
oras tag "${DST_REGISTRY}/cozystack-nocloud:${VERSION}" nightly
234235
echo "Published cozystack-nocloud:${VERSION} (+ :nightly) to GHCR."
235236
236-
# ── Full e2e against the published GHCR closure (chart + images + disk).
237+
# ── Full e2e against the published GHCR closure (chart + images), on the
238+
# same Talos-container substrate as the PR lanes. The nodes boot the
239+
# upstream Talos image, so the nocloud disk built above is published but
240+
# not exercised here, and e2e does not wait for it.
237241
e2e:
238242
name: E2E (full suite)
239-
needs: [mirror, build-disk]
240-
# Same sandbox as the other e2e lanes: three QEMU guests at `-smp 8` commit
241-
# 24 vCPU for the whole run, so the runner class has to be strictly larger,
242-
# leaving the host its own cycles for QEMU's emulation and IO threads,
243+
needs: [mirror]
244+
# Same sandbox as the other e2e lanes: three nodes at 8 vCPU / 24 GiB each
245+
# (hack/e2e-compose.yaml) commit 24 vCPU for the whole run, so the runner
246+
# class has to be strictly larger, leaving the host its own cycles for
243247
# containerd and the runner agent. Enforced by
244248
# hack/sandbox-runner-headroom.bats.
245249
runs-on: oracle-vm-32cpu-128gb-x86-64
@@ -260,16 +264,12 @@ jobs:
260264
uses: fluxcd/flux2/action@04acaec6161ac4fb1a82ffafa88901c03271d34f # v2.8.6
261265
with:
262266
version: "2.8.6"
263-
- name: Set up toolchain (oras)
264-
run: |
265-
command -v oras >/dev/null || { curl -fsSL https://github.com/oras-project/oras/releases/download/v1.2.0/oras_1.2.0_linux_amd64.tar.gz | sudo tar -xz -C /usr/local/bin oras; }
266267
- name: Login to GHCR
267268
env:
268269
GHCR_USER: ${{ github.actor }}
269270
GHCR_TOKEN: ${{ secrets.GITHUB_TOKEN }}
270271
run: |
271272
echo "$GHCR_TOKEN" | docker login "${DST_REGISTRY%%/*}" -u "$GHCR_USER" --password-stdin
272-
echo "$GHCR_TOKEN" | oras login "${DST_REGISTRY%%/*}" -u "$GHCR_USER" --password-stdin
273273
274274
# Overlay the GHCR-rewritten tree (operator image, .e2e.image and every
275275
# component now point at GHCR), then pin the installer chart to the GHCR
@@ -288,27 +288,49 @@ jobs:
288288
yq -i ".cozystackOperator.platformSourceUrl = \"oci://${DST_REGISTRY}/cozystack-packages\"" packages/core/installer/values.yaml
289289
yq -i ".cozystackOperator.platformSourceRef = \"digest=${PACKAGES_DIGEST}\"" packages/core/installer/values.yaml
290290
291-
- name: Fetch nocloud disk from GHCR
292-
env:
293-
VERSION: ${{ needs.mirror.outputs.version }}
294-
run: |
295-
set -eu
296-
mkdir -p _out/assets
297-
oras pull "${DST_REGISTRY}/cozystack-nocloud:${VERSION}" --output _out/assets
298-
299291
- name: Set sandbox ID
300292
run: echo "SANDBOX_NAME=cozy-e2e-sandbox-$(echo "${GITHUB_REPOSITORY}:${GITHUB_WORKFLOW}:${GITHUB_REF}" | sha256sum | cut -c1-10)" >> "$GITHUB_ENV"
301293

294+
# Talos cannot load kernel modules in container mode, so the nodes get
295+
# exactly what this host has loaded. Why each module, and why failing
296+
# here is better than an hour later: the same step in pull-requests.yaml.
297+
- name: Prepare container-lane host
298+
run: |
299+
set -eu
300+
for m in openvswitch zfs; do
301+
grep -q "^${m} " /proc/modules && continue
302+
if [ "${m}" = zfs ]; then
303+
sudo apt-get update -qq
304+
sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq zfsutils-linux
305+
fi
306+
sudo modprobe "${m}" || echo "${m}: modprobe failed"
307+
done
308+
missing=''
309+
for m in openvswitch zfs; do
310+
grep -q "^${m} " /proc/modules || missing="${missing} ${m}"
311+
done
312+
if [ -n "${missing}" ]; then
313+
echo "::error title=container-lane host::kernel module(s) could not be loaded:${missing}"
314+
exit 1
315+
fi
316+
302317
- name: Prepare environment
303318
run: |
304319
attempt=0
305-
until make SANDBOX_NAME=$SANDBOX_NAME prepare-env; do
320+
until make SANDBOX_NAME="$SANDBOX_NAME" prepare-env-container; do
306321
attempt=$((attempt + 1))
307322
if [ $attempt -ge 3 ]; then echo "Attempt $attempt failed, exiting..."; exit 1; fi
308323
echo "Attempt $attempt failed, retrying..."
324+
make -C packages/core/testing SANDBOX_NAME="$SANDBOX_NAME" delete-cluster-container || true
309325
done
310326
311327
- name: Install Cozystack into sandbox
328+
env:
329+
# No ARP VIP in container mode: .11 is a plain control-plane node.
330+
COZY_APISERVER_ENDPOINT: https://192.168.123.11:6443
331+
# The nodes share the runner kernel and cannot load DRBD.
332+
COZY_LINSTOR_DRBD_ENABLED: "false"
333+
COZY_E2E_STORAGE_CLASS: local
312334
run: |
313335
if ! make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME install-cozystack; then
314336
echo "❌ Install failed (no retry — diagnostics below)"
@@ -332,6 +354,8 @@ jobs:
332354
# Always the FULL app suite — a nightly has no diff to scope against, so
333355
# Test Impact Analysis does not apply (empty CHAINSAW_SUITES = whole suite).
334356
- name: Run E2E tests (full suite)
357+
env:
358+
COZY_E2E_STORAGE_CLASS: local
335359
run: |
336360
if ! make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME test-chainsaw CHAINSAW_SUITES=""; then
337361
echo "❌ Chainsaw E2E failed (see the assertion diffs or LEAKED-PODS lines above and diagnostics below)"
@@ -399,6 +423,10 @@ jobs:
399423
path: _out/cozyreport.tgz
400424
if-no-files-found: warn
401425

426+
# The compose project owns the nodes and the zpools outlive it, so both
427+
# halves go, in this order.
402428
- name: Tear down sandbox
403429
if: always()
404-
run: make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME delete || true
430+
run: |
431+
make -C packages/core/testing SANDBOX_NAME="$SANDBOX_NAME" delete-cluster-container || true
432+
make -C packages/core/testing SANDBOX_NAME="$SANDBOX_NAME" delete || true

‎.github/workflows/pull-requests.yaml‎

Lines changed: 6 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -883,7 +883,7 @@ jobs:
883883
# nocloud disk asset and export its id; the container lane boots no disk,
884884
# so nothing in this workflow reads that id any more and resolving it only
885885
# added a way to fail a release PR over an asset it does not use. The
886-
# asset is still resolved where it is still consumed, in e2e-tag.yaml.
886+
# asset's presence is still checked in e2e-tag.yaml and promote-rc.yaml.
887887
- name: Verify draft release exists (release PR)
888888
if: contains(github.event.pull_request.labels.*.name, 'release')
889889
uses: actions/github-script@f28e40c7f34bde8b3046d885e986cb6290c5673b # v7
@@ -903,15 +903,11 @@ jobs:
903903
}
904904
905905
906-
# In-tree e2e for SAME-REPO PRs, on Talos nodes running as CONTAINERS. The
907-
# QEMU substrate this lane used to boot is off the per-PR path only: it is
908-
# still wired into nightly.yaml and e2e-tag.yaml, both of which run `make
909-
# prepare-env` -> `make -C packages/core/testing prepare-cluster`. That target
910-
# and hack/e2e-prepare-cluster.bats are therefore live code, not leftovers,
911-
# and deleting them takes nightly and every release-candidate run with them.
912-
# They carry the coverage containers cannot: DRBD, and with it the
913-
# `replicated` StorageClass, its Immediate binding mode and the tenant
914-
# StorageClass propagation that only applies to remotely-accessible classes.
906+
# In-tree e2e for SAME-REPO PRs, on Talos nodes running as CONTAINERS, like
907+
# every other e2e lane. What containers cannot carry has no lane at all:
908+
# DRBD, and with it the `replicated` StorageClass, its Immediate binding mode
909+
# and the tenant StorageClass propagation that only applies to
910+
# remotely-accessible classes.
915911
#
916912
# Deliberately NOT named "E2E Tests": the
917913
# required "E2E Tests" context is a commit status posted by the `e2e-report`

0 commit comments

Comments
 (0)