Skip to content

Commit c34b0bd

Browse files
committed
ci(e2e): leave more time for sandbox cleanup
A successful full suite can reach teardown too close to the job limit, leaving less time than cleanup normally takes. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent 91f03dd commit c34b0bd

10 files changed

Lines changed: 34 additions & 28 deletions

File tree

‎.github/workflows/e2e-fork.yaml‎

Lines changed: 2 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -540,10 +540,8 @@ jobs:
540540
# per-node 8 CPU / 24 GiB in hack/e2e-compose.yaml is why the class is this
541541
# one. See the `e2e` job there for the full sizing note.
542542
runs-on: oracle-vm-32cpu-128gb-x86-64
543-
# Covers a green full suite. It does not cover a tenant-Kubernetes suite
544-
# running into its node-join deadline, so a red there can end as a
545-
# cancelled job. Held to one figure across every lane that runs the suite.
546-
timeout-minutes: 215
543+
# The job limit includes cleanup; it does not reserve time for it.
544+
timeout-minutes: 265
547545
permissions:
548546
contents: read
549547
packages: read

‎.github/workflows/pull-requests.yaml‎

Lines changed: 4 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -950,10 +950,8 @@ jobs:
950950
# scheduler and not what a pod reads from /proc, so a workload that sizes
951951
# itself from /proc/cpuinfo or /proc/meminfo is configured for the whole host.
952952
runs-on: oracle-vm-32cpu-128gb-x86-64
953-
# Covers a green full suite. It does not cover a tenant-Kubernetes suite
954-
# running into its node-join deadline, so a red there can end as a
955-
# cancelled job. Held to one figure across every lane that runs the suite.
956-
timeout-minutes: 215
953+
# The job limit includes cleanup; it does not reserve time for it.
954+
timeout-minutes: 265
957955
permissions:
958956
contents: read
959957
packages: read
@@ -1224,7 +1222,7 @@ jobs:
12241222

12251223
# ▸ Collect debug information (always runs)
12261224
#
1227-
# A ceiling of its own, not the job's 215 minutes. `Upload
1225+
# A ceiling of its own, not the job's 265 minutes. `Upload
12281226
# cozyreport.tgz` is the next step and the collector writes its tarball
12291227
# at the very end of its run, so a read that never returns spends what
12301228
# is left of the job budget, the job is cancelled, and the upload never
@@ -1235,7 +1233,7 @@ jobs:
12351233
# 30 minutes is measured against the step: it takes 3-42s across sampled
12361234
# runs, on failed e2e jobs as well as green ones. It RESERVES nothing --
12371235
# `timeout-minutes` caps a step, it does not carve budget out of the job,
1238-
# so a suite that has already burned 210 of the 215 leaves this step five
1236+
# so a suite that has already burned 260 of the 265 leaves this step five
12391237
# minutes and the upload is cancelled with everything else. What the cap
12401238
# buys is that a hung read cannot be the reason.
12411239
# `continue-on-error` because the ceiling must not change what the job

‎docs/agents/e2e-testing.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -245,11 +245,11 @@ A HelmRelease install waits through kstatus unless the ApplicationDefinition car
245245

246246
The first floor is the app's own first boot. For a chart that sets `startupProbe` it is `initialDelaySeconds + (failureThreshold - 1) * periodSeconds`; for a chart that does not, it is whatever its operator needs, and only a measurement will say. mariadb is the case in this tree where this floor is real rather than nominal: its chart grants a 310s startup budget, so a wait that fails earlier fails a boot the probe was still willing to wait for.
247247

248-
The second floor is the release's own install timeout, and it is usually the binding one. Where to read it depends on who created the release, and getting this wrong under-budgets rather than over-budgets. For a release cozystack-api generated from an ApplicationDefinition the default is 10m (`pkg/cmd/server/start.go`), overridden per definition by `release.cozystack.io/helm-install-timeout` — `kubernetes-rd` at 20m, `kubernetes-nodes-rd`, `tenant-rd` and `monitoring-rd` at 15m, `kafka-rd` at 7h, a floor no suite clears inside the E2E job's 215-minute cap; the first two also disable the wait, so for `kubernetes` the override bounds its blocking post-install hook rather than a kstatus poll, and `kubernetes-nodes`, which renders no post-install hook, has nothing in its install for the override to bound. For a HelmRelease an app chart renders itself there is no ApplicationDefinition to read and that default never applies: the floor is the `spec.install.timeout` on the template that renders it, which takes precedence over its `spec.timeout`, and where a template sets neither, helm-controller's own documented default of 5m applies instead. There is deliberately no count of that population here. Read the value off the template you are waiting on: they range from 5m to 30m, they are spread across `packages/apps`, `packages/extra` and `packages/system`, and a release carrying `spec.kubeConfig` installs into a tenant cluster and is not one of these at all. A number in this sentence would be an obligation to revisit it whenever a template is added, and its three qualifiers are each easy to drop from a recursive grep. The nested `<name>-system` releases the app charts wrap their workloads in are all of this second kind. A HelmRelease that cozystack-operator builds for a `Package` is a third kind: its `spec.install.timeout` comes from the operator's `--helmrelease-install-timeout`, 10m unless `cozystackOperator.helmReleaseInstallTimeout` in the installer chart overrides it. At or below whichever floor applies, a failed install is reported as a bare assert timeout, because the release has not yet recorded why it failed.
248+
The second floor is the release's own install timeout, and it is usually the binding one. Where to read it depends on who created the release, and getting this wrong under-budgets rather than over-budgets. For a release cozystack-api generated from an ApplicationDefinition the default is 10m (`pkg/cmd/server/start.go`), overridden per definition by `release.cozystack.io/helm-install-timeout` — `kubernetes-rd` at 20m, `kubernetes-nodes-rd`, `tenant-rd` and `monitoring-rd` at 15m, `kafka-rd` at 7h, a floor no suite clears inside the E2E job's time limit; the first two also disable the wait, so for `kubernetes` the override bounds its blocking post-install hook rather than a kstatus poll, and `kubernetes-nodes`, which renders no post-install hook, has nothing in its install for the override to bound. For a HelmRelease an app chart renders itself there is no ApplicationDefinition to read and that default never applies: the floor is the `spec.install.timeout` on the template that renders it, which takes precedence over its `spec.timeout`, and where a template sets neither, helm-controller's own documented default of 5m applies instead. There is deliberately no count of that population here. Read the value off the template you are waiting on: they range from 5m to 30m, they are spread across `packages/apps`, `packages/extra` and `packages/system`, and a release carrying `spec.kubeConfig` installs into a tenant cluster and is not one of these at all. A number in this sentence would be an obligation to revisit it whenever a template is added, and its three qualifiers are each easy to drop from a recursive grep. The nested `<name>-system` releases the app charts wrap their workloads in are all of this second kind. A HelmRelease that cozystack-operator builds for a `Package` is a third kind: its `spec.install.timeout` comes from the operator's `--helmrelease-install-timeout`, 10m unless `cozystackOperator.helmReleaseInstallTimeout` in the installer chart overrides it. At or below whichever floor applies, a failed install is reported as a bare assert timeout, because the release has not yet recorded why it failed.
249249

250250
What holds the install open is anything in the rendered manifests kstatus reports InProgress, which is wider than a `Ready` condition. kstatus carries built-in rules for a list of kinds — the workloads, but also `Service`, `Pod`, `PersistentVolumeClaim` and `CustomResourceDefinition` — and falls through to reading `Ready` only for kinds it has no rule for. Read that list rather than recalling it: `legacyTypes` in `pkg/kstatus/status/core.go` of `fluxcd/cli-utils`, at whatever version the helm-controller image in `internal/fluxinstall/manifests/fluxcd.yaml` was built against — nothing in this repository vendors it, so there is no local copy to open and the version has to come from that image. It does not divide the way intuition suggests, because `Secret`, `ConfigMap` and `batch/CronJob` are in the table and map to always-ready, so being in it is not the same as gating on it. The practical consequence is that "renders no CR publishing `Ready`" does not exempt a release, and `tcp-balancer` and `vpn` show why: each renders a `WorkloadMonitor`, whose status type declares no `conditions` at all, so kstatus falls past the `Ready` fallback and reads that CR as Current — and then gates the install on the Deployment beside it through a built-in rule. The CR is present and contributes nothing; the workload is what holds the install open.
251251

252-
Every mistake in this derivation under-budgets, which is the harmful direction — but the floor is not free either, because a wait that must exceed its release's install timeout means a genuinely failing suite burns that long before it says so, and `.chainsaw.yaml` already warns about diagnostics eating the job's own 215m — for scale, the timeouts written on the mariadb suite's own steps come to 120m of that, before the defaults its untimed steps inherit from `.chainsaw.yaml`, and a job cancelled at the cap loses both its log and `cozyreport.tgz`. Raise a suite when a failed install there needs to say why; leave one alone when the diagnosis is not worth ten minutes added to every red run in that suite. That is the trade, and it is per suite — "only the ones kstatus polls" is not a filter, because almost everything polls. Two suites are worth reading before writing a new wait, and deliberately not as a list of every wait and its floor: a hand-written list of them is incomplete in a way its author cannot see, and goes stale without saying so — a `grep` for the call misses the invocations split across lines, and reading for it by eye misses the single-release waits among the `--all` ones. Enumerate with `grep -rnE 'kubectl +wait.*(^|[^[:alnum:]])hr([^[:alnum:]]|$)' hack/` and the chainsaw asserts on `kind: HelmRelease`, not from memory and not from here. mariadb's 11m in `hack/e2e-chainsaw/mariadb/chainsaw-test.yaml` is the worked example of the derivation above. harbor's two asserts show why one literal cannot serve a whole suite: `harbor-test` is generated and clears its 600s floor at 11m, while `harbor-test-system` is rendered by the app chart with a `spec.timeout` of 15m and needs 16m.
252+
Every mistake in this derivation under-budgets, which is the harmful direction — but the floor is not free either, because a wait that must exceed its release's install timeout means a genuinely failing suite burns that long before it says so, and `.chainsaw.yaml` already warns about diagnostics eating the job's own time limit — for scale, the timeouts written on the mariadb suite's own steps come to 120m of that, before the defaults its untimed steps inherit from `.chainsaw.yaml`, and a job cancelled at the cap loses both its log and `cozyreport.tgz`. Raise a suite when a failed install there needs to say why; leave one alone when the diagnosis is not worth ten minutes added to every red run in that suite. That is the trade, and it is per suite — "only the ones kstatus polls" is not a filter, because almost everything polls. Two suites are worth reading before writing a new wait, and deliberately not as a list of every wait and its floor: a hand-written list of them is incomplete in a way its author cannot see, and goes stale without saying so — a `grep` for the call misses the invocations split across lines, and reading for it by eye misses the single-release waits among the `--all` ones. Enumerate with `grep -rnE 'kubectl +wait.*(^|[^[:alnum:]])hr([^[:alnum:]]|$)' hack/` and the chainsaw asserts on `kind: HelmRelease`, not from memory and not from here. mariadb's 11m in `hack/e2e-chainsaw/mariadb/chainsaw-test.yaml` is the worked example of the derivation above. harbor's two asserts show why one literal cannot serve a whole suite: `harbor-test` is generated and clears its 600s floor at 11m, while `harbor-test-system` is rendered by the app chart with a `spec.timeout` of 15m and needs 16m.
253253

254254
Two things this derivation does not give you. Readiness *may* span a first boot rather than always doing so — a poll landing before the operator's first status write reads a condition-less resource as Current and can end the wait early, which is why a suite keeps an endpoint assert after the HR-Ready one rather than treating it as redundant. And a number copied from a neighbouring suite says nothing about the release you are waiting on, even where it clears the neighbour's floor: the floor depends on who rendered the release, and kafka's assert sits under its own on purpose. Scan `hack/e2e-chainsaw/*/chainsaw-test.yaml` for HR-Ready asserts and hold each against its own release's floor plus the start margin; the result is left to that scan rather than written out here, because a list of them goes stale silently while the scan does not. Chainsaw asserts are not the whole surface: `wait_hr_ready` calls under `examples/backups/` carry the same floor, not all of them from a `run-all.sh` and not all of them with a literal budget — a call that passes none takes the helper's own default, which a grep for the number misses — so scan both places. Derive the budget from the release you are waiting on.
255255

‎hack/cozyreport.bats‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -966,6 +966,19 @@ STUB
966966
}
967967
}
968968

969+
@test "PR E2E jobs leave room for cleanup after a long successful run" {
970+
for wf in pull-requests e2e-fork; do
971+
minutes="$(yq --unwrapScalar '.jobs.e2e."timeout-minutes"' "$HACK_DIR/../.github/workflows/$wf.yaml")"
972+
case "$minutes" in
973+
''|*[!0-9]*) echo "FAIL: $wf E2E job timeout is not a whole number: $minutes"; false ;;
974+
esac
975+
if [ "$minutes" -lt 223 ]; then
976+
echo "FAIL: $wf E2E job budget $minutes minutes cannot fit a 211m15s run and 11m25s cleanup"
977+
false
978+
fi
979+
done
980+
}
981+
969982
@test "the step that runs this collector has a ceiling of its own everywhere" {
970983
# Every bound inside the script protects one read. None of them protects the
971984
# STEP: the tarball is written on the collector's last line, the upload is the

‎hack/cozyreport.sh‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -120,8 +120,8 @@
120120
# it sits UNDER this composition deliberately. Add the terms yourself if you need
121121
# the number; do not expect to find it written here.
122122
#
123-
# TWO ceilings sit outside it, and they fail differently. The job is capped at
124-
# `timeout-minutes: 215`, and reaching that cancels every step still to come --
123+
# TWO ceilings sit outside it, and they fail differently. Reaching the job's
124+
# `timeout-minutes` cancels every step still to come --
125125
# the artifact upload among them. The step running this script is capped at 30 of
126126
# its own, and reaching THAT ends only this step: the uploads after it still run,
127127
# and `continue-on-error` on that step keeps the overrun out of the job's result,

‎hack/cozytest.sh‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -207,8 +207,8 @@ _cozy_on_exit() {
207207
# 12 × 20s log reads), so it should not fire; if it does, a silently cut
208208
# capture would read as a complete one listing fewer crash-loops than
209209
# actually occurred. This handler's whole worst case is ~1350s (330 + 390
210-
# + 630, each leg including its 30s kill grace) against a job capped at
211-
# 215 minutes with no per-step timeout —
210+
# + 630, each leg including its 30s kill grace) against a job with no
211+
# per-step timeout —
212212
# deliberate, for the same reason recorded in hack/e2e-chainsaw/.chainsaw.yaml:
213213
# a bounded, honest wait beats a kill that collects nothing.
214214
if [ "$_prev_rc" -ne 0 ]; then

‎hack/e2e-chainsaw/.chainsaw.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -66,7 +66,7 @@ spec:
6666
# script: it SIGKILLs the shell alone, orphaned children hold the pipe open,
6767
# and the op overruns (#4268 measured 606s under a 300s ceiling), so the
6868
# nominal ~5m was a figure a busy catch could exceed, not a cap. Tests run
69-
# sequentially, and the E2E job is capped at 215 timeout-minutes. A cascade
69+
# sequentially, and each E2E job has a time limit. A cascade
7070
# reddening many suites can therefore spend a large share of the job in
7171
# diagnostics. That is deliberate — the old ceiling
7272
# bought its speed by collecting nothing at all — and it is bounded: both legs

‎hack/e2e-chainsaw/computeplane/chainsaw-test.yaml‎

Lines changed: 2 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -35,10 +35,8 @@
3535
# and the teardown. This is the third suite in a full run that brings up a
3636
# tenant cluster, next to kubernetes-latest and kubernetes-previous. A failure
3737
# that stalls worker join everywhere reddens all three, and their red paths
38-
# together can exceed the E2E job's 215-minute cap, which cancels the job and
39-
# loses its log and report. The cap is sized for the red path of the
40-
# kubernetes pair only; this suite is not counted in it, and that is an
41-
# accepted risk.
38+
# together can exceed the E2E job's time limit, which cancels the job and
39+
# loses its log and report.
4240
apiVersion: chainsaw.kyverno.io/v1alpha1
4341
kind: Test
4442
metadata:

‎hack/e2e-chainsaw/kubernetes-latest/chainsaw-test.yaml‎

Lines changed: 4 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -77,11 +77,10 @@ spec:
7777
# The OIDC lifecycle folded onto this cluster by the 5th argument joins
7878
# that list rather than being priced into the ceiling: it adds up to 1m
7979
# of job wait, a 600s cozy_wait_helmrelease_upgrade and another 1m of job
80-
# wait, all on the passing path. Raising the ceiling to cover them is not
81-
# available -- the guard in hack/run-kubernetes-serial-console_test.bats
82-
# holds COZY_OP_CEILING equal to BOTH suites' `timeout:`, and two suites
83-
# at 79m plus their 22m teardowns exceed the job's own 215-minute cap.
84-
# What is available is a measurement: on the green run of this branch
80+
# wait, all on the passing path. The guard in
81+
# hack/run-kubernetes-serial-console_test.bats holds COZY_OP_CEILING equal
82+
# to BOTH suites' `timeout:`; both suites and their teardowns share the
83+
# job's time limit. On the green run of this branch
8584
# (33724179053) the whole Test took 569.72s against this 4020s op, with
8685
# kubernetes-previous, which carries no OIDC fold, at 470.19s. The
8786
# ceilings only bind when something is already hung.

‎hack/e2e-chainsaw/vminstance/chainsaw-test.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -473,7 +473,7 @@ spec:
473473
# scripts) so their worst-case sum stays under this op timeout, and a hung
474474
# wait trips the script's own named "Timeout waiting for ..." rather than a
475475
# bare `signal: killed` (docs/agents/e2e-testing.md). Sized small on purpose:
476-
# this suite is execution.parallel:1 and shares the job's 215m cap with every
476+
# this suite is execution.parallel:1 and shares the job's time limit with every
477477
# other Test in it and their finally blocks — an oversized op timeout lets the
478478
# job cap fire first and lose the report before any op timeout can.
479479
- script:
@@ -650,7 +650,7 @@ spec:
650650
# timeout, so a hung wait trips the script's own named fail-fast, not
651651
# Chainsaw's bare `signal: killed`; and
652652
# it is sized small on purpose, since this suite (execution.parallel:1) shares
653-
# the 215m job cap with the other Tests and the finally blocks — an oversized
653+
# the job's time limit with the other Tests and the finally blocks — an oversized
654654
# op timeout lets the cap fire first and lose the report. BSL setup is in this
655655
# step (not a separate one) so the finally below always tears the BSL down.
656656
- script:

0 commit comments

Comments
 (0)