Skip to content

Commit 5edf362

Browse files
ci(e2e): raise the E2E job limit from 215 to 265 minutes
A green full Chainsaw run no longer fits in 215 minutes. Nightly has been cancelled at the limit on each of the last four runs, and pull requests that select the full suite end the same way while every suite that got to run passed. All four jobs that run the suite (pull-requests, e2e-fork, e2e-tag, nightly) move to 265, which keeps them on one figure. Comments that state the cap follow it. The kubernetes-latest note on why the op ceiling cannot grow is restated so its arithmetic still holds. Part of #5261. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <[email protected]>
1 parent 9978771 commit 5edf362

9 files changed

Lines changed: 15 additions & 14 deletions

File tree

‎.github/workflows/e2e-fork.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -543,7 +543,7 @@ jobs:
543543
# Covers a green full suite. It does not cover a tenant-Kubernetes suite
544544
# running into its node-join deadline, so a red there can end as a
545545
# cancelled job. Held to one figure across every lane that runs the suite.
546-
timeout-minutes: 215
546+
timeout-minutes: 265
547547
permissions:
548548
contents: read
549549
packages: read

‎.github/workflows/e2e-tag.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -137,7 +137,7 @@ jobs:
137137
# Covers a green full suite. It does not cover a tenant-Kubernetes suite
138138
# running into its node-join deadline, so a red there can end as a
139139
# cancelled job. Held to one figure across every lane that runs the suite.
140-
timeout-minutes: 215
140+
timeout-minutes: 265
141141
permissions:
142142
contents: read
143143
checks: write # Breakpoint action updates its dedicated "Breakpoint Open" Check Run.
@@ -292,7 +292,7 @@ jobs:
292292
# collect-report packages cozyreport plus per-failed-test crust-gather
293293
# snapshots captured before Chainsaw cleanup.
294294
#
295-
# A ceiling of its own, not the job's 215 minutes. `Upload cozyreport.tgz`
295+
# A ceiling of its own, not the job's 265 minutes. `Upload cozyreport.tgz`
296296
# is the next step and the collector writes its tarball at the very end of
297297
# its run, so a read that never returns spends what is left of the job
298298
# budget, the job is cancelled, and the upload never runs -- the artifact

‎.github/workflows/nightly.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -250,7 +250,7 @@ jobs:
250250
# Covers a green full suite. It does not cover a tenant-Kubernetes suite
251251
# running into its node-join deadline, so a red there can end as a
252252
# cancelled job. Held to one figure across every lane that runs the suite.
253-
timeout-minutes: 215
253+
timeout-minutes: 265
254254
permissions:
255255
contents: read
256256
packages: read
@@ -385,7 +385,7 @@ jobs:
385385
path: _out/chainsaw-report.xml
386386
if-no-files-found: ignore
387387

388-
# A ceiling of its own, not the job's 215 minutes. `Upload cozyreport.tgz`
388+
# A ceiling of its own, not the job's 265 minutes. `Upload cozyreport.tgz`
389389
# is the next step and the collector writes its tarball at the very end of
390390
# its run, so a read that never returns spends what is left of the job
391391
# budget, the job is cancelled, and the upload never runs -- the artifact

‎.github/workflows/pull-requests.yaml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -953,7 +953,7 @@ jobs:
953953
# Covers a green full suite. It does not cover a tenant-Kubernetes suite
954954
# running into its node-join deadline, so a red there can end as a
955955
# cancelled job. Held to one figure across every lane that runs the suite.
956-
timeout-minutes: 215
956+
timeout-minutes: 265
957957
permissions:
958958
contents: read
959959
packages: read
@@ -1224,7 +1224,7 @@ jobs:
12241224

12251225
# ▸ Collect debug information (always runs)
12261226
#
1227-
# A ceiling of its own, not the job's 215 minutes. `Upload
1227+
# A ceiling of its own, not the job's 265 minutes. `Upload
12281228
# cozyreport.tgz` is the next step and the collector writes its tarball
12291229
# at the very end of its run, so a read that never returns spends what
12301230
# is left of the job budget, the job is cancelled, and the upload never
@@ -1235,7 +1235,7 @@ jobs:
12351235
# 30 minutes is measured against the step: it takes 3-42s across sampled
12361236
# runs, on failed e2e jobs as well as green ones. It RESERVES nothing --
12371237
# `timeout-minutes` caps a step, it does not carve budget out of the job,
1238-
# so a suite that has already burned 210 of the 215 leaves this step five
1238+
# so a suite that has already burned 260 of the 265 leaves this step five
12391239
# minutes and the upload is cancelled with everything else. What the cap
12401240
# buys is that a hung read cannot be the reason.
12411241
# `continue-on-error` because the ceiling must not change what the job

‎hack/cozyreport.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -121,7 +121,7 @@
121121
# the number; do not expect to find it written here.
122122
#
123123
# TWO ceilings sit outside it, and they fail differently. The job is capped at
124-
# `timeout-minutes: 215`, and reaching that cancels every step still to come --
124+
# `timeout-minutes: 265`, and reaching that cancels every step still to come --
125125
# the artifact upload among them. The step running this script is capped at 30 of
126126
# its own, and reaching THAT ends only this step: the uploads after it still run,
127127
# and `continue-on-error` on that step keeps the overrun out of the job's result,

‎hack/cozytest.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -208,7 +208,7 @@ _cozy_on_exit() {
208208
# capture would read as a complete one listing fewer crash-loops than
209209
# actually occurred. This handler's whole worst case is ~1350s (330 + 390
210210
# + 630, each leg including its 30s kill grace) against a job capped at
211-
# 215 minutes with no per-step timeout —
211+
# 265 minutes with no per-step timeout —
212212
# deliberate, for the same reason recorded in hack/e2e-chainsaw/.chainsaw.yaml:
213213
# a bounded, honest wait beats a kill that collects nothing.
214214
if [ "$_prev_rc" -ne 0 ]; then

‎hack/e2e-chainsaw/.chainsaw.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -66,14 +66,14 @@ spec:
6666
# script: it SIGKILLs the shell alone, orphaned children hold the pipe open,
6767
# and the op overruns (#4268 measured 606s under a 300s ceiling), so the
6868
# nominal ~5m was a figure a busy catch could exceed, not a cap. Tests run
69-
# sequentially, and the E2E job is capped at 215 timeout-minutes. A cascade
69+
# sequentially, and the E2E job is capped at 265 timeout-minutes. A cascade
7070
# reddening many suites can therefore spend a large share of the job in
7171
# diagnostics. That is deliberate — the old ceiling
7272
# bought its speed by collecting nothing at all — and it is bounded: both legs
7373
# only approach their ceilings on a genuinely degraded cluster, a systemic
7474
# failure usually reds the install gate and skips the E2E step entirely, and
7575
# the report-upload steps are `if: always()`. That last point does NOT extend
76-
# to a job timeout: a 215-minute cap ends the job `cancelled`, and the ~4m48s
76+
# to a job timeout: a 265-minute cap ends the job `cancelled`, and the ~4m48s
7777
# grace window is consumed by the earlier `Collect report` step, so both
7878
# cozyreport.tgz and the job log are lost (measured on runs 29557003429 and
7979
# 29568562337). If this ever does start truncating runs, bound the number of

‎hack/e2e-chainsaw/computeplane/chainsaw-test.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@
3535
# and the teardown. This is the third suite in a full run that brings up a
3636
# tenant cluster, next to kubernetes-latest and kubernetes-previous. A failure
3737
# that stalls worker join everywhere reddens all three, and their red paths
38-
# together can exceed the E2E job's 215-minute cap, which cancels the job and
38+
# together can exceed the E2E job's 265-minute cap, which cancels the job and
3939
# loses its log and report. The cap is sized for the red path of the
4040
# kubernetes pair only; this suite is not counted in it, and that is an
4141
# accepted risk.

‎hack/e2e-chainsaw/kubernetes-latest/chainsaw-test.yaml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -80,7 +80,8 @@ spec:
8080
# wait, all on the passing path. Raising the ceiling to cover them is not
8181
# available -- the guard in hack/run-kubernetes-serial-console_test.bats
8282
# holds COZY_OP_CEILING equal to BOTH suites' `timeout:`, and two suites
83-
# at 79m plus their 22m teardowns exceed the job's own 215-minute cap.
83+
# at 79m plus their 22m teardowns take 202m, which with the install and
84+
# the rest of a full run does not fit the job's own 265-minute cap.
8485
# What is available is a measurement: on the green run of this branch
8586
# (33724179053) the whole Test took 569.72s against this 4020s op, with
8687
# kubernetes-previous, which carries no OIDC fold, at 470.19s. The

0 commit comments

Comments
 (0)