Skip to content

Commit 428fbcb

Browse files
committed
test(e2e): run a Kairos worker pool in the kubernetes-latest suite
Kairos pools join through a different bootstrap provider, a different worker disk and a kubeadm join, none of which the Talos pools of the suite exercise. After the tenant cluster passes its own checks, the suite adds a pool on the built-in kairos NodeGroupClass and requires its Machine to link a Ready Node that carries a KubeVirt providerID and the cluster-namespace annotation, and a pod to run on it. Each wait on the management side collects the pool's disk import, machines, health check, events and guest console before it fails, so a registry that refuses the disk is told apart from a worker that booted and never joined. The step runs inside the E2E job, which had about 16 minutes left under its 215-minute cap on a full run of this branch. A red Kairos step would end the job as cancelled and lose its log and report, so the cap moves to 255 minutes in every lane that runs the suite, and the step's ceiling is sized so that the old cap plus the step stays under the new one; the sum is written beside the step. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent 9b1abe2 commit 428fbcb

12 files changed

Lines changed: 411 additions & 18 deletions

File tree

‎.github/workflows/e2e-fork.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -543,7 +543,7 @@ jobs:
543543
# Covers a green full suite. It does not cover a tenant-Kubernetes suite
544544
# running into its node-join deadline, so a red there can end as a
545545
# cancelled job. Held to one figure across every lane that runs the suite.
546-
timeout-minutes: 215
546+
timeout-minutes: 255
547547
permissions:
548548
contents: read
549549
packages: read

‎.github/workflows/e2e-tag.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -137,7 +137,7 @@ jobs:
137137
# Covers a green full suite. It does not cover a tenant-Kubernetes suite
138138
# running into its node-join deadline, so a red there can end as a
139139
# cancelled job. Held to one figure across every lane that runs the suite.
140-
timeout-minutes: 215
140+
timeout-minutes: 255
141141
permissions:
142142
contents: read
143143
checks: write # Breakpoint action updates its dedicated "Breakpoint Open" Check Run.
@@ -292,7 +292,7 @@ jobs:
292292
# collect-report packages cozyreport plus per-failed-test crust-gather
293293
# snapshots captured before Chainsaw cleanup.
294294
#
295-
# A ceiling of its own, not the job's 215 minutes. `Upload cozyreport.tgz`
295+
# A ceiling of its own, not the job's 255 minutes. `Upload cozyreport.tgz`
296296
# is the next step and the collector writes its tarball at the very end of
297297
# its run, so a read that never returns spends what is left of the job
298298
# budget, the job is cancelled, and the upload never runs -- the artifact

‎.github/workflows/nightly.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -250,7 +250,7 @@ jobs:
250250
# Covers a green full suite. It does not cover a tenant-Kubernetes suite
251251
# running into its node-join deadline, so a red there can end as a
252252
# cancelled job. Held to one figure across every lane that runs the suite.
253-
timeout-minutes: 215
253+
timeout-minutes: 255
254254
permissions:
255255
contents: read
256256
packages: read
@@ -385,7 +385,7 @@ jobs:
385385
path: _out/chainsaw-report.xml
386386
if-no-files-found: ignore
387387

388-
# A ceiling of its own, not the job's 215 minutes. `Upload cozyreport.tgz`
388+
# A ceiling of its own, not the job's 255 minutes. `Upload cozyreport.tgz`
389389
# is the next step and the collector writes its tarball at the very end of
390390
# its run, so a read that never returns spends what is left of the job
391391
# budget, the job is cancelled, and the upload never runs -- the artifact

‎.github/workflows/pull-requests.yaml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1053,7 +1053,7 @@ jobs:
10531053
# Covers a green full suite. It does not cover a tenant-Kubernetes suite
10541054
# running into its node-join deadline, so a red there can end as a
10551055
# cancelled job. Held to one figure across every lane that runs the suite.
1056-
timeout-minutes: 215
1056+
timeout-minutes: 255
10571057
permissions:
10581058
contents: read
10591059
packages: read
@@ -1324,7 +1324,7 @@ jobs:
13241324

13251325
# ▸ Collect debug information (always runs)
13261326
#
1327-
# A ceiling of its own, not the job's 215 minutes. `Upload
1327+
# A ceiling of its own, not the job's 255 minutes. `Upload
13281328
# cozyreport.tgz` is the next step and the collector writes its tarball
13291329
# at the very end of its run, so a read that never returns spends what
13301330
# is left of the job budget, the job is cancelled, and the upload never
@@ -1335,7 +1335,7 @@ jobs:
13351335
# 30 minutes is measured against the step: it takes 3-42s across sampled
13361336
# runs, on failed e2e jobs as well as green ones. It RESERVES nothing --
13371337
# `timeout-minutes` caps a step, it does not carve budget out of the job,
1338-
# so a suite that has already burned 210 of the 215 leaves this step five
1338+
# so a suite that has already burned 250 of the 255 leaves this step five
13391339
# minutes and the upload is cancelled with everything else. What the cap
13401340
# buys is that a hung read cannot be the reason.
13411341
# `continue-on-error` because the ceiling must not change what the job

‎docs/agents/e2e-testing.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -241,11 +241,11 @@ A HelmRelease install waits through kstatus unless the ApplicationDefinition car
241241

242242
The first floor is the app's own first boot. For a chart that sets `startupProbe` it is `initialDelaySeconds + (failureThreshold - 1) * periodSeconds`; for a chart that does not, it is whatever its operator needs, and only a measurement will say. mariadb is the case in this tree where this floor is real rather than nominal: its chart grants a 310s startup budget, so a wait that fails earlier fails a boot the probe was still willing to wait for.
243243

244-
The second floor is the release's own install timeout, and it is usually the binding one. Where to read it depends on who created the release, and getting this wrong under-budgets rather than over-budgets. For a release cozystack-api generated from an ApplicationDefinition the default is 10m (`pkg/cmd/server/start.go`), overridden per definition by `release.cozystack.io/helm-install-timeout` — `kubernetes-rd` at 20m, `kubernetes-nodes-rd`, `tenant-rd` and `monitoring-rd` at 15m, `kafka-rd` at 7h, a floor no suite clears inside the E2E job's 215-minute cap; the first two also disable the wait, so for `kubernetes` the override bounds its blocking post-install hook rather than a kstatus poll, and `kubernetes-nodes`, which renders no post-install hook, has nothing in its install for the override to bound. For a HelmRelease an app chart renders itself there is no ApplicationDefinition to read and that default never applies: the floor is the `spec.install.timeout` on the template that renders it, which takes precedence over its `spec.timeout`, and where a template sets neither, helm-controller's own documented default of 5m applies instead. There is deliberately no count of that population here. Read the value off the template you are waiting on: they range from 5m to 30m, they are spread across `packages/apps`, `packages/extra` and `packages/system`, and a release carrying `spec.kubeConfig` installs into a tenant cluster and is not one of these at all. A number in this sentence would be an obligation to revisit it whenever a template is added, and its three qualifiers are each easy to drop from a recursive grep. The nested `<name>-system` releases the app charts wrap their workloads in are all of this second kind. A HelmRelease that cozystack-operator builds for a `Package` is a third kind: its `spec.install.timeout` comes from the operator's `--helmrelease-install-timeout`, 10m unless `cozystackOperator.helmReleaseInstallTimeout` in the installer chart overrides it. At or below whichever floor applies, a failed install is reported as a bare assert timeout, because the release has not yet recorded why it failed.
244+
The second floor is the release's own install timeout, and it is usually the binding one. Where to read it depends on who created the release, and getting this wrong under-budgets rather than over-budgets. For a release cozystack-api generated from an ApplicationDefinition the default is 10m (`pkg/cmd/server/start.go`), overridden per definition by `release.cozystack.io/helm-install-timeout` — `kubernetes-rd` at 20m, `kubernetes-nodes-rd`, `tenant-rd` and `monitoring-rd` at 15m, `kafka-rd` at 7h, a floor no suite clears inside the E2E job's 255-minute cap; the first two also disable the wait, so for `kubernetes` the override bounds its blocking post-install hook rather than a kstatus poll, and `kubernetes-nodes`, which renders no post-install hook, has nothing in its install for the override to bound. For a HelmRelease an app chart renders itself there is no ApplicationDefinition to read and that default never applies: the floor is the `spec.install.timeout` on the template that renders it, which takes precedence over its `spec.timeout`, and where a template sets neither, helm-controller's own documented default of 5m applies instead. There is deliberately no count of that population here. Read the value off the template you are waiting on: they range from 5m to 30m, they are spread across `packages/apps`, `packages/extra` and `packages/system`, and a release carrying `spec.kubeConfig` installs into a tenant cluster and is not one of these at all. A number in this sentence would be an obligation to revisit it whenever a template is added, and its three qualifiers are each easy to drop from a recursive grep. The nested `<name>-system` releases the app charts wrap their workloads in are all of this second kind. A HelmRelease that cozystack-operator builds for a `Package` is a third kind: its `spec.install.timeout` comes from the operator's `--helmrelease-install-timeout`, 10m unless `cozystackOperator.helmReleaseInstallTimeout` in the installer chart overrides it. At or below whichever floor applies, a failed install is reported as a bare assert timeout, because the release has not yet recorded why it failed.
245245

246246
What holds the install open is anything in the rendered manifests kstatus reports InProgress, which is wider than a `Ready` condition. kstatus carries built-in rules for a list of kinds — the workloads, but also `Service`, `Pod`, `PersistentVolumeClaim` and `CustomResourceDefinition` — and falls through to reading `Ready` only for kinds it has no rule for. Read that list rather than recalling it: `legacyTypes` in `pkg/kstatus/status/core.go` of `fluxcd/cli-utils`, at whatever version the helm-controller image in `internal/fluxinstall/manifests/fluxcd.yaml` was built against — nothing in this repository vendors it, so there is no local copy to open and the version has to come from that image. It does not divide the way intuition suggests, because `Secret`, `ConfigMap` and `batch/CronJob` are in the table and map to always-ready, so being in it is not the same as gating on it. The practical consequence is that "renders no CR publishing `Ready`" does not exempt a release, and `tcp-balancer` and `vpn` show why: each renders a `WorkloadMonitor`, whose status type declares no `conditions` at all, so kstatus falls past the `Ready` fallback and reads that CR as Current — and then gates the install on the Deployment beside it through a built-in rule. The CR is present and contributes nothing; the workload is what holds the install open.
247247

248-
Every mistake in this derivation under-budgets, which is the harmful direction — but the floor is not free either, because a wait that must exceed its release's install timeout means a genuinely failing suite burns that long before it says so, and `.chainsaw.yaml` already warns about diagnostics eating the job's own 215m — for scale, the timeouts written on the mariadb suite's own steps come to 120m of that, before the defaults its untimed steps inherit from `.chainsaw.yaml`, and a job cancelled at the cap loses both its log and `cozyreport.tgz`. Raise a suite when a failed install there needs to say why; leave one alone when the diagnosis is not worth ten minutes added to every red run in that suite. That is the trade, and it is per suite — "only the ones kstatus polls" is not a filter, because almost everything polls. Two suites are worth reading before writing a new wait, and deliberately not as a list of every wait and its floor: a hand-written list of them is incomplete in a way its author cannot see, and goes stale without saying so — a `grep` for the call misses the invocations split across lines, and reading for it by eye misses the single-release waits among the `--all` ones. Enumerate with `grep -rnE 'kubectl +wait.*(^|[^[:alnum:]])hr([^[:alnum:]]|$)' hack/` and the chainsaw asserts on `kind: HelmRelease`, not from memory and not from here. mariadb's 11m in `hack/e2e-chainsaw/mariadb/chainsaw-test.yaml` is the worked example of the derivation above. harbor's two asserts show why one literal cannot serve a whole suite: `harbor-test` is generated and clears its 600s floor at 11m, while `harbor-test-system` is rendered by the app chart with a `spec.timeout` of 15m and needs 16m.
248+
Every mistake in this derivation under-budgets, which is the harmful direction — but the floor is not free either, because a wait that must exceed its release's install timeout means a genuinely failing suite burns that long before it says so, and `.chainsaw.yaml` already warns about diagnostics eating the job's own 255m — for scale, the timeouts written on the mariadb suite's own steps come to 120m of that, before the defaults its untimed steps inherit from `.chainsaw.yaml`, and a job cancelled at the cap loses both its log and `cozyreport.tgz`. Raise a suite when a failed install there needs to say why; leave one alone when the diagnosis is not worth ten minutes added to every red run in that suite. That is the trade, and it is per suite — "only the ones kstatus polls" is not a filter, because almost everything polls. Two suites are worth reading before writing a new wait, and deliberately not as a list of every wait and its floor: a hand-written list of them is incomplete in a way its author cannot see, and goes stale without saying so — a `grep` for the call misses the invocations split across lines, and reading for it by eye misses the single-release waits among the `--all` ones. Enumerate with `grep -rnE 'kubectl +wait.*(^|[^[:alnum:]])hr([^[:alnum:]]|$)' hack/` and the chainsaw asserts on `kind: HelmRelease`, not from memory and not from here. mariadb's 11m in `hack/e2e-chainsaw/mariadb/chainsaw-test.yaml` is the worked example of the derivation above. harbor's two asserts show why one literal cannot serve a whole suite: `harbor-test` is generated and clears its 600s floor at 11m, while `harbor-test-system` is rendered by the app chart with a `spec.timeout` of 15m and needs 16m.
249249

250250
Two things this derivation does not give you. Readiness *may* span a first boot rather than always doing so — a poll landing before the operator's first status write reads a condition-less resource as Current and can end the wait early, which is why a suite keeps an endpoint assert after the HR-Ready one rather than treating it as redundant. And a number copied from a neighbouring suite says nothing about the release you are waiting on, even where it clears the neighbour's floor: the floor depends on who rendered the release, and kafka's assert sits under its own on purpose. Scan `hack/e2e-chainsaw/*/chainsaw-test.yaml` for HR-Ready asserts and hold each against its own release's floor plus the start margin; the result is left to that scan rather than written out here, because a list of them goes stale silently while the scan does not. Chainsaw asserts are not the whole surface: `wait_hr_ready` calls under `examples/backups/` carry the same floor, not all of them from a `run-all.sh` and not all of them with a literal budget — a call that passes none takes the helper's own default, which a grep for the number misses — so scan both places. Derive the budget from the release you are waiting on.
251251

‎hack/cozyreport.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -121,7 +121,7 @@
121121
# the number; do not expect to find it written here.
122122
#
123123
# TWO ceilings sit outside it, and they fail differently. The job is capped at
124-
# `timeout-minutes: 215`, and reaching that cancels every step still to come --
124+
# `timeout-minutes: 255`, and reaching that cancels every step still to come --
125125
# the artifact upload among them. The step running this script is capped at 30 of
126126
# its own, and reaching THAT ends only this step: the uploads after it still run,
127127
# and `continue-on-error` on that step keeps the overrun out of the job's result,

‎hack/cozytest.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -208,7 +208,7 @@ _cozy_on_exit() {
208208
# capture would read as a complete one listing fewer crash-loops than
209209
# actually occurred. This handler's whole worst case is ~1350s (330 + 390
210210
# + 630, each leg including its 30s kill grace) against a job capped at
211-
# 215 minutes with no per-step timeout —
211+
# 255 minutes with no per-step timeout —
212212
# deliberate, for the same reason recorded in hack/e2e-chainsaw/.chainsaw.yaml:
213213
# a bounded, honest wait beats a kill that collects nothing.
214214
if [ "$_prev_rc" -ne 0 ]; then

‎hack/e2e-chainsaw/.chainsaw.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -66,14 +66,14 @@ spec:
6666
# script: it SIGKILLs the shell alone, orphaned children hold the pipe open,
6767
# and the op overruns (#4268 measured 606s under a 300s ceiling), so the
6868
# nominal ~5m was a figure a busy catch could exceed, not a cap. Tests run
69-
# sequentially, and the E2E job is capped at 215 timeout-minutes. A cascade
69+
# sequentially, and the E2E job is capped at 255 timeout-minutes. A cascade
7070
# reddening many suites can therefore spend a large share of the job in
7171
# diagnostics. That is deliberate — the old ceiling
7272
# bought its speed by collecting nothing at all — and it is bounded: both legs
7373
# only approach their ceilings on a genuinely degraded cluster, a systemic
7474
# failure usually reds the install gate and skips the E2E step entirely, and
7575
# the report-upload steps are `if: always()`. That last point does NOT extend
76-
# to a job timeout: a 215-minute cap ends the job `cancelled`, and the ~4m48s
76+
# to a job timeout: a 255-minute cap ends the job `cancelled`, and the ~4m48s
7777
# grace window is consumed by the earlier `Collect report` step, so both
7878
# cozyreport.tgz and the job log are lost (measured on runs 29557003429 and
7979
# 29568562337). If this ever does start truncating runs, bound the number of

0 commit comments

Comments
 (0)