You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 428fbcb
Browse filesBrowse the repository at this point in the historyBrowse files
test(e2e): run a Kairos worker pool in the kubernetes-latest suite
Kairos pools join through a different bootstrap provider, a different
worker disk and a kubeadm join, none of which the Talos pools of the
suite exercise. After the tenant cluster passes its own checks, the
suite adds a pool on the built-in kairos NodeGroupClass and requires
its Machine to link a Ready Node that carries a KubeVirt providerID and
the cluster-namespace annotation, and a pod to run on it. Each wait on
the management side collects the pool's disk import, machines, health
check, events and guest console before it fails, so a registry that
refuses the disk is told apart from a worker that booted and never
joined.
The step runs inside the E2E job, which had about 16 minutes left
under its 215-minute cap on a full run of this branch. A red Kairos
step would end the job as cancelled and lose its log and report, so
the cap moves to 255 minutes in every lane that runs the suite, and
the step's ceiling is sized so that the old cap plus the step stays
under the new one; the sum is written beside the step.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>
Copy file name to clipboardExpand all lines: docs/agents/e2e-testing.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -241,11 +241,11 @@ A HelmRelease install waits through kstatus unless the ApplicationDefinition car
241
241
242
242
The first floor is the app's own first boot. For a chart that sets `startupProbe` it is `initialDelaySeconds + (failureThreshold - 1) * periodSeconds`; for a chart that does not, it is whatever its operator needs, and only a measurement will say. mariadb is the case in this tree where this floor is real rather than nominal: its chart grants a 310s startup budget, so a wait that fails earlier fails a boot the probe was still willing to wait for.
243
243
244
-
The second floor is the release's own install timeout, and it is usually the binding one. Where to read it depends on who created the release, and getting this wrong under-budgets rather than over-budgets. For a release cozystack-api generated from an ApplicationDefinition the default is 10m (`pkg/cmd/server/start.go`), overridden per definition by `release.cozystack.io/helm-install-timeout` — `kubernetes-rd` at 20m, `kubernetes-nodes-rd`, `tenant-rd` and `monitoring-rd` at 15m, `kafka-rd` at 7h, a floor no suite clears inside the E2E job's 215-minute cap; the first two also disable the wait, so for `kubernetes` the override bounds its blocking post-install hook rather than a kstatus poll, and `kubernetes-nodes`, which renders no post-install hook, has nothing in its install for the override to bound. For a HelmRelease an app chart renders itself there is no ApplicationDefinition to read and that default never applies: the floor is the `spec.install.timeout` on the template that renders it, which takes precedence over its `spec.timeout`, and where a template sets neither, helm-controller's own documented default of 5m applies instead. There is deliberately no count of that population here. Read the value off the template you are waiting on: they range from 5m to 30m, they are spread across `packages/apps`, `packages/extra` and `packages/system`, and a release carrying `spec.kubeConfig` installs into a tenant cluster and is not one of these at all. A number in this sentence would be an obligation to revisit it whenever a template is added, and its three qualifiers are each easy to drop from a recursive grep. The nested `<name>-system` releases the app charts wrap their workloads in are all of this second kind. A HelmRelease that cozystack-operator builds for a `Package` is a third kind: its `spec.install.timeout` comes from the operator's `--helmrelease-install-timeout`, 10m unless `cozystackOperator.helmReleaseInstallTimeout` in the installer chart overrides it. At or below whichever floor applies, a failed install is reported as a bare assert timeout, because the release has not yet recorded why it failed.
244
+
The second floor is the release's own install timeout, and it is usually the binding one. Where to read it depends on who created the release, and getting this wrong under-budgets rather than over-budgets. For a release cozystack-api generated from an ApplicationDefinition the default is 10m (`pkg/cmd/server/start.go`), overridden per definition by `release.cozystack.io/helm-install-timeout` — `kubernetes-rd` at 20m, `kubernetes-nodes-rd`, `tenant-rd` and `monitoring-rd` at 15m, `kafka-rd` at 7h, a floor no suite clears inside the E2E job's 255-minute cap; the first two also disable the wait, so for `kubernetes` the override bounds its blocking post-install hook rather than a kstatus poll, and `kubernetes-nodes`, which renders no post-install hook, has nothing in its install for the override to bound. For a HelmRelease an app chart renders itself there is no ApplicationDefinition to read and that default never applies: the floor is the `spec.install.timeout` on the template that renders it, which takes precedence over its `spec.timeout`, and where a template sets neither, helm-controller's own documented default of 5m applies instead. There is deliberately no count of that population here. Read the value off the template you are waiting on: they range from 5m to 30m, they are spread across `packages/apps`, `packages/extra` and `packages/system`, and a release carrying `spec.kubeConfig` installs into a tenant cluster and is not one of these at all. A number in this sentence would be an obligation to revisit it whenever a template is added, and its three qualifiers are each easy to drop from a recursive grep. The nested `<name>-system` releases the app charts wrap their workloads in are all of this second kind. A HelmRelease that cozystack-operator builds for a `Package` is a third kind: its `spec.install.timeout` comes from the operator's `--helmrelease-install-timeout`, 10m unless `cozystackOperator.helmReleaseInstallTimeout` in the installer chart overrides it. At or below whichever floor applies, a failed install is reported as a bare assert timeout, because the release has not yet recorded why it failed.
245
245
246
246
What holds the install open is anything in the rendered manifests kstatus reports InProgress, which is wider than a `Ready` condition. kstatus carries built-in rules for a list of kinds — the workloads, but also `Service`, `Pod`, `PersistentVolumeClaim` and `CustomResourceDefinition` — and falls through to reading `Ready` only for kinds it has no rule for. Read that list rather than recalling it: `legacyTypes` in `pkg/kstatus/status/core.go` of `fluxcd/cli-utils`, at whatever version the helm-controller image in `internal/fluxinstall/manifests/fluxcd.yaml` was built against — nothing in this repository vendors it, so there is no local copy to open and the version has to come from that image. It does not divide the way intuition suggests, because `Secret`, `ConfigMap` and `batch/CronJob` are in the table and map to always-ready, so being in it is not the same as gating on it. The practical consequence is that "renders no CR publishing `Ready`" does not exempt a release, and `tcp-balancer` and `vpn` show why: each renders a `WorkloadMonitor`, whose status type declares no `conditions` at all, so kstatus falls past the `Ready` fallback and reads that CR as Current — and then gates the install on the Deployment beside it through a built-in rule. The CR is present and contributes nothing; the workload is what holds the install open.
247
247
248
-
Every mistake in this derivation under-budgets, which is the harmful direction — but the floor is not free either, because a wait that must exceed its release's install timeout means a genuinely failing suite burns that long before it says so, and `.chainsaw.yaml` already warns about diagnostics eating the job's own 215m — for scale, the timeouts written on the mariadb suite's own steps come to 120m of that, before the defaults its untimed steps inherit from `.chainsaw.yaml`, and a job cancelled at the cap loses both its log and `cozyreport.tgz`. Raise a suite when a failed install there needs to say why; leave one alone when the diagnosis is not worth ten minutes added to every red run in that suite. That is the trade, and it is per suite — "only the ones kstatus polls" is not a filter, because almost everything polls. Two suites are worth reading before writing a new wait, and deliberately not as a list of every wait and its floor: a hand-written list of them is incomplete in a way its author cannot see, and goes stale without saying so — a `grep` for the call misses the invocations split across lines, and reading for it by eye misses the single-release waits among the `--all` ones. Enumerate with `grep -rnE 'kubectl +wait.*(^|[^[:alnum:]])hr([^[:alnum:]]|$)' hack/` and the chainsaw asserts on `kind: HelmRelease`, not from memory and not from here. mariadb's 11m in `hack/e2e-chainsaw/mariadb/chainsaw-test.yaml` is the worked example of the derivation above. harbor's two asserts show why one literal cannot serve a whole suite: `harbor-test` is generated and clears its 600s floor at 11m, while `harbor-test-system` is rendered by the app chart with a `spec.timeout` of 15m and needs 16m.
248
+
Every mistake in this derivation under-budgets, which is the harmful direction — but the floor is not free either, because a wait that must exceed its release's install timeout means a genuinely failing suite burns that long before it says so, and `.chainsaw.yaml` already warns about diagnostics eating the job's own 255m — for scale, the timeouts written on the mariadb suite's own steps come to 120m of that, before the defaults its untimed steps inherit from `.chainsaw.yaml`, and a job cancelled at the cap loses both its log and `cozyreport.tgz`. Raise a suite when a failed install there needs to say why; leave one alone when the diagnosis is not worth ten minutes added to every red run in that suite. That is the trade, and it is per suite — "only the ones kstatus polls" is not a filter, because almost everything polls. Two suites are worth reading before writing a new wait, and deliberately not as a list of every wait and its floor: a hand-written list of them is incomplete in a way its author cannot see, and goes stale without saying so — a `grep` for the call misses the invocations split across lines, and reading for it by eye misses the single-release waits among the `--all` ones. Enumerate with `grep -rnE 'kubectl +wait.*(^|[^[:alnum:]])hr([^[:alnum:]]|$)' hack/` and the chainsaw asserts on `kind: HelmRelease`, not from memory and not from here. mariadb's 11m in `hack/e2e-chainsaw/mariadb/chainsaw-test.yaml` is the worked example of the derivation above. harbor's two asserts show why one literal cannot serve a whole suite: `harbor-test` is generated and clears its 600s floor at 11m, while `harbor-test-system` is rendered by the app chart with a `spec.timeout` of 15m and needs 16m.
249
249
250
250
Two things this derivation does not give you. Readiness *may* span a first boot rather than always doing so — a poll landing before the operator's first status write reads a condition-less resource as Current and can end the wait early, which is why a suite keeps an endpoint assert after the HR-Ready one rather than treating it as redundant. And a number copied from a neighbouring suite says nothing about the release you are waiting on, even where it clears the neighbour's floor: the floor depends on who rendered the release, and kafka's assert sits under its own on purpose. Scan `hack/e2e-chainsaw/*/chainsaw-test.yaml` for HR-Ready asserts and hold each against its own release's floor plus the start margin; the result is left to that scan rather than written out here, because a list of them goes stale silently while the scan does not. Chainsaw asserts are not the whole surface: `wait_hr_ready` calls under `examples/backups/` carry the same floor, not all of them from a `run-all.sh` and not all of them with a literal budget — a call that passes none takes the helper's own default, which a grep for the number misses — so scan both places. Derive the budget from the release you are waiting on.
0 commit comments