Skip to content

Commit b0e0f38

Browse files
test(e2e): keep the chainsaw ops above the deadlines they contain
.chainsaw.yaml states the rule and says why: an op that expires before its inner bounds do is a SIGKILL on the process group, which takes out the diagnostics the failure branch exists to print, while an inner timeout leaves a partial capture and lets the remaining steps run. refresh-and-assert-full-health, added on this branch, breaks it. The poll deadline is 270s, the last in-loop read can start just under it and run 25s, and the failure branch then reads again for another 25s: 320s of inner bounds under a 5m op. Raised to 6m. verify-processes-and-security-context breaks it harder and did so before this branch: 240s for the process counts, then 120s for connectionString and 120s for generations.reconciled, sequentially, is 480s under the same 5m. Fixed here rather than filed, because it is the same defect one step away, found by applying the rule to the neighbour. Raised to 9m. The kubernetes-latest ceiling gets the measurement instead of a change. The OIDC lifecycle folded onto that cluster adds up to 1m + 600s + 1m of passing-path waits that the 67m op does not price, and raising the op is not available: a guard holds COZY_OP_CEILING equal to both suites' timeout, and two suites at 79m plus their teardowns exceed the job's 215-minute cap. On the green run of this branch the whole Test took 569.72s against the 4020s op, with kubernetes-previous at 470.19s. The ceilings only bind once something is already hung, which is what the existing residual note (#3666) says; now it says it with a number. Signed-off-by: Myasnikov Daniil <[email protected]> Assisted-By: LLM
1 parent ec7f172 commit b0e0f38

2 files changed

Lines changed: 25 additions & 2 deletions

File tree

‎hack/e2e-chainsaw/foundationdb/chainsaw-test.yaml‎

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -82,7 +82,13 @@ spec:
8282
- name: verify-processes-and-security-context
8383
try:
8484
- script:
85-
timeout: 5m
85+
# Three sequential inner deadlines -- 240s for the process counts, then
86+
# 120s for connectionString and 120s for generations.reconciled -- sum to
87+
# 480s. Pre-existing overrun, found by applying the same .chainsaw.yaml
88+
# rule to the step next door: at 5m the op fired first and took the whole
89+
# script out by SIGKILL, so a slow-but-recovering cluster reported
90+
# `signal: killed` instead of naming the stage that was still waiting.
91+
timeout: 9m
8692
content: |
8793
set -eu
8894
ns="$NAMESPACE"
@@ -131,7 +137,12 @@ spec:
131137
- name: refresh-and-assert-full-health
132138
try:
133139
- script:
134-
timeout: 5m
140+
# 270s poll deadline + one 25s in-loop read that may start just under it
141+
# + one 25s diagnostic read on the failure branch = 320s of inner bounds.
142+
# Keep the op above that sum, per .chainsaw.yaml: at 5m Chainsaw SIGKILLs
143+
# the script instead, and the diagnostic that the failure branch exists
144+
# to print is exactly what gets lost.
145+
timeout: 6m
135146
content: |
136147
set -eu
137148
ns="$NAMESPACE"

‎hack/e2e-chainsaw/kubernetes-latest/chainsaw-test.yaml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -72,6 +72,18 @@ spec:
7272
# diagnostic runs, and the pre-wait halves of the four readings above spend
7373
# inside that bringup rather than inside this phase. Those residuals are
7474
# tracked in cozystack/cozystack#3666.
75+
#
76+
# The OIDC lifecycle folded onto this cluster by the 5th argument joins
77+
# that list rather than being priced into the ceiling: it adds up to 1m
78+
# of job wait, a 600s cozy_wait_helmrelease_upgrade and another 1m of job
79+
# wait, all on the passing path. Raising the ceiling to cover them is not
80+
# available -- the guard in hack/run-kubernetes-serial-console_test.bats
81+
# holds COZY_OP_CEILING equal to BOTH suites' `timeout:`, and two suites
82+
# at 79m plus their 22m teardowns exceed the job's own 215-minute cap.
83+
# What is available is a measurement: on the green run of this branch
84+
# (33724179053) the whole Test took 569.72s against this 4020s op, with
85+
# kubernetes-previous, which carries no OIDC fold, at 470.19s. The
86+
# ceilings only bind when something is already hung.
7587
timeout: 67m
7688
# Co-locate the tenant crust-gather snapshot (captured by
7789
# run-kubernetes.sh on failure) under the same snapshots/<test>/ dir as

0 commit comments

Comments
 (0)