Skip to content

Commit a7ac37b

Browse files
lexfreilexfrei
authored
test(e2e): let HR-Ready waits outlast the install timeout of the release they wait on (#4471)
## What this PR does Stacked on #4468, which holds the raised backup waits to a start margin; review that one first. This PR's own diff is the six commits on top of it. Most HR-Ready waits in the E2E suites gave up before the release they were waiting on could fail its install. cozystack-api gives every release it generates a 600s install timeout, and the Chainsaw asserts in front of those releases sat at 5 or 6 minutes. When an install genuinely failed, the suite reported a bare assert timeout against a release that was still installing, and the reason the install failed, which the release records when its own timeout fires, was never reached. Red runs failed just as often as they should; they only said less. I raised every HR-Ready wait above the install timeout of its own release, by the 60s start margin #4468 introduces: - Chainsaw asserts on releases cozystack-api generates go to 11m, across bucket, clickhouse, etcd, external-dns, foundationdb, harbor, mongodb, openbao, opensearch, postgres, qdrant, rabbitmq, redis, valkey and vminstance. - Asserts on nested `<name>-system` releases take their floor from the template that renders them: 11m for external-dns, openbao and qdrant (10m `spec.timeout`), 16m for `harbor-test-system` (15m). `harbor-test` sat exactly on its 600s floor, which expires first because the assert's clock starts before the install's; it goes to 11m too. - The kuberture Package release takes its 10m from cozystack-operator and goes to 11m. - In the backup walkthroughs, the bucket waits go from 300s to 660s in clickhouse, kafka, mariadb, mongodb, postgres, rabbitmq and redis, and the two postgres waits go from 360s to 660s. The kafka bucket wait passed no budget and silently took the helper's 300s default, which is why earlier scans for literal budgets missed it. Each Chainsaw op that runs a walkthrough grows by exactly what its waits gained (clickhouse 78m, kafka 46m, mariadb 48m, mongodb 42m, postgres 66m, rabbitmq 48m, redis 36m), so a failure at any later wait keeps the headroom it had and is still reported by the script instead of being cut off by the op. Two waits stay as they are, on purpose: - kafka: `kafka-rd` sets a 7h install timeout, which no budget inside the 215-minute E2E job can clear. The kafka assert and the Kafka waits in the kafka and kafka-metadata walkthroughs keep their numbers, and the assert now says why in-line. - `vm-instance-test`: `vm-instance-rd` disables the install wait, so its release goes Ready as soon as helm has applied and there is no floor to clear. Earlier lists of this problem included it by mistake. The cost falls on red runs only: a suite whose release never goes Ready now spends up to six more minutes before it reports, and in exchange the report carries the release's own reason. Green runs are unchanged. Suites run one after another in a single job capped at 215 minutes, so a systemic failure that reddens many suites now reaches that cap sooner, and a job cancelled at the cap loses its log and `cozyreport.tgz`. I took that trade because a failed install that says why is worth more than a red run that ends a few minutes earlier. `docs/agents/e2e-testing.md` stopped being true in several places once the waits were raised: it described most of the tree as under the floor, mariadb as the only suite raised, the walkthrough bucket waits as under it, and harbor's asserts as sitting on and under theirs. I rewrote those sentences, and the one that pointed at "every wait above the band" of 5-6m, kept the start-margin wording from #4468, and added that a walkthrough wait passing no budget takes the helper's default. The comments in `hack/mariadb-first-boot-waits.bats` that quoted the old op and bucket figures are restated; its existing checks and constants are unchanged, because every op grew by exactly its bucket delta and each remainder still sits at exactly its constant. I also added postgres to the table that file holds the walkthroughs to, so its raised waits and its op cannot drift back apart unnoticed, and a test that holds each table suite's bucket wait to the bucket release's floor, read from the api-server default with `bucket-rd` checked for an override; the op check alone could not catch a bucket wait dropping back, since a lower wait only widens the slack. redis stays out: its walkthrough waits on no redis HelmRelease, only on the bucket and on the sentinel master, so the table's floor check has nothing to read there. How I checked it, since E2E cannot run locally: - I re-ran the enumeration from the sizing section of `docs/agents/e2e-testing.md` over every `hack/e2e-chainsaw/*/chainsaw-test.yaml` and every `wait_hr_ready` call under `examples/backups/`, literal or default, and held each against its own release's floor. Every `Ready=True` assert and every `wait_hr_ready` call a suite drives now clears its floor, except the kafka ones named above. The opt-in etcd round-trip uses bare `kubectl wait hr` calls at 300s instead of that helper; they are listed in #4470. The `Ready=False` assert on the rejected redis upgrade is not a readiness wait and is left out. - `make bats-unit-mariadb-first-boot-waits` passes, 31 of 31. For the new postgres row, putting the op back to 50m fails only the postgres slack test, and a postgres wait at 659s fails only the postgres floor test. Setting any one table suite's bucket wait to 659s fails only the bucket test, naming only that suite, and an install-timeout override on `bucket-rd` fails it too. I ran the mutations under plain `bats` so a fail-fast runner could not hide a second failure. - `hack/select-e2e.sh` over this diff selects bucket, clickhouse, etcd, external-dns, foundationdb, harbor, kafka, kuberture, mariadb, mongodb, openbao, opensearch, postgres, qdrant, rabbitmq, redis, valkey and vminstance. Out of scope, and tracked in #4470: the tenant addon waits in `hack/e2e-chainsaw/_lib/run-kubernetes.sh` (an edit there escalates the whole E2E run), the redis and kafka-metadata `wait_hr_ready` helpers that print no reason on timeout, and the opt-in etcd round-trip's bare `kubectl wait` calls. Known and left for later, none of them changed here: - Only the mariadb Chainsaw asserts and the walkthroughs in the guard's table are held by a test. The other raised asserts, and the redis and kafka bucket waits, rest on literals with in-line reasons; #4470 covers extending the guard. - The Kafka waits in `examples/backups/kafka/04-create-kafka.sh`, `07-restore-to-copy.sh` and `kafka-metadata/run-all.sh` stay at 300s against the 7h floor without an in-line reason at the call site. The doc's rule asks that of Chainsaw asserts only, and the kafka Chainsaw assert carries it. The postgres waits are the ones #3748 is about. Their budget now clears the install-timeout floor and a test holds it there; what a CNPG first boot costs on this platform is still unmeasured, so that issue stays open. Closes #3771 Closes #3753 ### Screenshots Not a UI change. ### Downstream repositories I walked the trigger map in `docs/agents/contributing.md` against the diff. It touches E2E suites, backup walkthrough scripts, a unit-lane bats file and an agent doc, none of which any downstream repository restates. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(tests): HelmRelease readiness waits in the E2E suites and backup walkthroughs now outlast the install timeout of the release they wait on, so a failed install reports its own reason instead of a bare timeout. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Tests** * Extended readiness and backup round-trip test timeouts across multiple services to accommodate slower startup and provisioning. * Added checks to verify that bucket readiness waits meet configured installation-timeout requirements. * **Documentation** * Updated testing guidance to clarify how readiness waits relate to release-specific timeouts and overall test budgets. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 360f03b + a0af2fd commit a7ac37b

27 files changed

Lines changed: 213 additions & 87 deletions

File tree

‎docs/agents/e2e-testing.md‎

Lines changed: 5 additions & 5 deletions
Large diffs are not rendered by default.

‎examples/backups/clickhouse/03-create-bucket.sh‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,9 @@ spec:
2727
EOF
2828

2929
log_substep "Waiting for bucket HelmRelease to be Ready..."
30-
wait_hr_ready "bucket-${BUCKET_NAME}" 300
30+
# 660s, above this generated release's 600s install timeout; see "Sizing an
31+
# HR-Ready budget" in docs/agents/e2e-testing.md.
32+
wait_hr_ready "bucket-${BUCKET_NAME}" 660
3133
kubectl -n "$NAMESPACE" wait bucketclaims.objectstorage.k8s.io "bucket-${BUCKET_NAME}" --for=jsonpath='{.status.bucketReady}'=true --timeout=180s
3234
# Cozystack's bucket app provisions a BucketAccess named "<bucket-name>-backup"
3335
# (the "-backup" suffix is the BucketAccessClass name); the BucketInfo Secret

‎examples/backups/kafka/03-create-bucket.sh‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,9 @@ spec:
2626
readonly: false
2727
EOF
2828

29-
wait_hr_ready "bucket-${BUCKET_NAME}"
29+
# 660s, above this generated release's 600s install timeout; see "Sizing an
30+
# HR-Ready budget" in docs/agents/e2e-testing.md.
31+
wait_hr_ready "bucket-${BUCKET_NAME}" 660
3032
kubectl -n "$NAMESPACE" wait bucketclaims.objectstorage.k8s.io "bucket-${BUCKET_NAME}" --for=jsonpath='{.status.bucketReady}'=true --timeout=300s
3133
# Cozystack's bucket app provisions a BucketAccess named "<bucket-name>-backup"
3234
# (the "-backup" suffix is the BucketAccessClass name); the BucketInfo Secret

‎examples/backups/mariadb/run-all.sh‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -48,7 +48,9 @@ wait_app_grant_ready() {
4848

4949
print_header "Step 00: Provision Bucket '${BUCKET_NAME}' in ${NAMESPACE}"
5050
kubectl -n "$NAMESPACE" apply -f "$SCRIPT_DIR/00-bucket.yaml"
51-
wait_hr_ready "bucket-${BUCKET_NAME}" 300
51+
# 660s, above this generated release's 600s install timeout; see "Sizing an
52+
# HR-Ready budget" in docs/agents/e2e-testing.md.
53+
wait_hr_ready "bucket-${BUCKET_NAME}" 660
5254
wait_for_field bucketclaims.objectstorage.k8s.io "bucket-${BUCKET_NAME}" \
5355
'{.status.bucketReady}' true "$NAMESPACE" 300
5456
wait_for_field bucketaccesses.objectstorage.k8s.io "bucket-${BUCKET_NAME}-${BUCKET_USER}" \

‎examples/backups/mongodb/run-all.sh‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,9 @@ subst() {
4444

4545
print_header "Step 00: Provision Bucket '${BUCKET_NAME}' in ${NAMESPACE}"
4646
kubectl -n "$NAMESPACE" apply -f "$SCRIPT_DIR/00-bucket.yaml"
47-
wait_hr_ready "bucket-${BUCKET_NAME}" 300
47+
# 660s, above this generated release's 600s install timeout; see "Sizing an
48+
# HR-Ready budget" in docs/agents/e2e-testing.md.
49+
wait_hr_ready "bucket-${BUCKET_NAME}" 660
4850
wait_for_field bucketclaims.objectstorage.k8s.io "bucket-${BUCKET_NAME}" \
4951
'{.status.bucketReady}' true "$NAMESPACE" 300
5052
wait_for_field bucketaccesses.objectstorage.k8s.io "bucket-${BUCKET_NAME}-${BUCKET_USER}" \

‎examples/backups/postgres/run-all.sh‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,9 @@ subst() {
3838

3939
print_header "Step 00: Provision Bucket '${BUCKET_NAME}' in ${NAMESPACE}"
4040
kubectl -n "$NAMESPACE" apply -f "$SCRIPT_DIR/00-bucket.yaml"
41-
wait_hr_ready "bucket-${BUCKET_NAME}" 300
41+
# 660s, above this generated release's 600s install timeout; see "Sizing an
42+
# HR-Ready budget" in docs/agents/e2e-testing.md.
43+
wait_hr_ready "bucket-${BUCKET_NAME}" 660
4244
wait_for_field bucketclaims.objectstorage.k8s.io "bucket-${BUCKET_NAME}" \
4345
'{.status.bucketReady}' true "$NAMESPACE" 300
4446
wait_for_field bucketaccesses.objectstorage.k8s.io "bucket-${BUCKET_NAME}-backup" \
@@ -123,7 +125,9 @@ materialise_backup_secrets "$PG_SRC_NAME"
123125

124126
print_header "Step 05b: Deploy source Postgres '${PG_SRC_NAME}' and wait for it to be healthy"
125127
subst 05-postgres-src.yaml | kubectl -n "$NAMESPACE" apply -f -
126-
wait_hr_ready "postgres-${PG_SRC_NAME}" 360
128+
# 660s, above this generated release's 600s install timeout; see "Sizing an
129+
# HR-Ready budget" in docs/agents/e2e-testing.md.
130+
wait_hr_ready "postgres-${PG_SRC_NAME}" 660
127131
wait_for_field clusters.postgresql.cnpg.io "$PG_SRC_CLUSTER" \
128132
'{.status.phase}' 'Cluster in healthy state' "$NAMESPACE" 360
129133
# The 'demo' database and 'app' user are created by the chart's init Job, not
@@ -167,7 +171,8 @@ materialise_backup_secrets "$PG_TARGET_NAME"
167171
kubectl -n "$NAMESPACE" apply -f "$SCRIPT_DIR/30-postgres-target.yaml"
168172
# Let the target's first install settle before the RestoreJob driver suspends
169173
# its HelmRelease — suspending an HR mid-install races helm-controller.
170-
wait_hr_ready "postgres-${PG_TARGET_NAME}" 360
174+
# 660s, as for the source above.
175+
wait_hr_ready "postgres-${PG_TARGET_NAME}" 660
171176
kubectl -n "$NAMESPACE" apply -f "$SCRIPT_DIR/40-restorejob-to-copy.yaml"
172177
wait_for_field restorejobs.backups.cozystack.io "$RESTOREJOB_TOCOPY_NAME" \
173178
'{.status.phase}' Succeeded "$NAMESPACE" 1200 Failed

‎examples/backups/rabbitmq/run-all.sh‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,9 @@ print_header "RabbitMQ definitions backup/restore demo (namespace: $NAMESPACE)"
4343
# --- Bucket ------------------------------------------------------------------
4444
print_header "Step 00: Provision Bucket '${BUCKET_NAME}' in ${NAMESPACE}"
4545
kubectl -n "$NAMESPACE" apply -f "$SCRIPT_DIR/00-bucket.yaml"
46-
wait_hr_ready "bucket-${BUCKET_NAME}" 300
46+
# 660s, above this generated release's 600s install timeout; see "Sizing an
47+
# HR-Ready budget" in docs/agents/e2e-testing.md.
48+
wait_hr_ready "bucket-${BUCKET_NAME}" 660
4749
wait_for_field bucketclaims.objectstorage.k8s.io "bucket-${BUCKET_NAME}" \
4850
'{.status.bucketReady}' true "$NAMESPACE" 300
4951
wait_for_field bucketaccesses.objectstorage.k8s.io "bucket-${BUCKET_NAME}-${BUCKET_USER}" \

‎examples/backups/redis/run-all.sh‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,9 @@ print_header "Redis backup/restore demo (namespace: $NAMESPACE)"
4141
# --- Bucket ------------------------------------------------------------------
4242
print_header "Step 00: Provision Bucket '${BUCKET_NAME}' in ${NAMESPACE}"
4343
kubectl -n "$NAMESPACE" apply -f "$SCRIPT_DIR/bucket.yaml"
44-
wait_hr_ready "bucket-${BUCKET_NAME}" 300
44+
# 660s, above this generated release's 600s install timeout; see "Sizing an
45+
# HR-Ready budget" in docs/agents/e2e-testing.md.
46+
wait_hr_ready "bucket-${BUCKET_NAME}" 660
4547
wait_for_field bucketclaims.objectstorage.k8s.io "bucket-${BUCKET_NAME}" \
4648
'{.status.bucketReady}' true "$NAMESPACE" 300
4749
wait_for_field bucketaccesses.objectstorage.k8s.io "bucket-${BUCKET_NAME}-${BUCKET_USER}" \

‎hack/e2e-chainsaw/bucket/chainsaw-test.yaml‎

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -39,8 +39,10 @@ spec:
3939

4040
- name: wait-bucket-ready
4141
try:
42+
# 11m: above the 600s install timeout cozystack-api gives this release;
43+
# see "Sizing an HR-Ready budget" in docs/agents/e2e-testing.md.
4244
- assert:
43-
timeout: 6m
45+
timeout: 11m
4446
resource:
4547
apiVersion: helm.toolkit.fluxcd.io/v2
4648
kind: HelmRelease
@@ -254,8 +256,10 @@ spec:
254256

255257
- name: wait-buckets-ready
256258
try:
259+
# 11m each: above the 600s install timeout cozystack-api gives these
260+
# releases; see "Sizing an HR-Ready budget" in docs/agents/e2e-testing.md.
257261
- assert:
258-
timeout: 6m
262+
timeout: 11m
259263
resource:
260264
apiVersion: helm.toolkit.fluxcd.io/v2
261265
kind: HelmRelease
@@ -265,7 +269,7 @@ spec:
265269
(conditions[?type == 'Ready']):
266270
- status: "True"
267271
- assert:
268-
timeout: 6m
272+
timeout: 11m
269273
resource:
270274
apiVersion: helm.toolkit.fluxcd.io/v2
271275
kind: HelmRelease
@@ -275,7 +279,7 @@ spec:
275279
(conditions[?type == 'Ready']):
276280
- status: "True"
277281
- assert:
278-
timeout: 6m
282+
timeout: 11m
279283
resource:
280284
apiVersion: helm.toolkit.fluxcd.io/v2
281285
kind: HelmRelease

‎hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml‎

Lines changed: 9 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -25,8 +25,10 @@ spec:
2525

2626
- name: wait-helmrelease-ready
2727
try:
28+
# 11m: above the 600s install timeout cozystack-api gives this release;
29+
# see "Sizing an HR-Ready budget" in docs/agents/e2e-testing.md.
2830
- assert:
29-
timeout: 6m
31+
timeout: 11m
3032
resource:
3133
apiVersion: helm.toolkit.fluxcd.io/v2
3234
kind: HelmRelease
@@ -135,8 +137,9 @@ spec:
135137

136138
- name: wait-helmrelease-ready
137139
try:
140+
# Same 11m as clickhouse-test above.
138141
- assert:
139-
timeout: 6m
142+
timeout: 11m
140143
resource:
141144
apiVersion: helm.toolkit.fluxcd.io/v2
142145
kind: HelmRelease
@@ -336,13 +339,13 @@ spec:
336339
# Inner budgets, sized against what CI actually takes rather than guessed
337340
# generously — the whole bucket suite completes in ~37s and a backup-enabled
338341
# ClickHouse plus Keeper reaches Ready in ~98s (measured on run 30353894332):
339-
# 03 750s = 300 bucket HR + 180 claim + 180 access + 90 S3 preflight
342+
# 03 1110s = 660 bucket HR + 180 claim + 180 access + 90 S3 preflight
340343
# 04 960s = 660 HR + 300 StatefulSet
341344
# 05 480s = BackupJob to Succeeded
342345
# 06 480s = in-place RestoreJob
343346
# 07 1440s = 660 HR + 300 StatefulSet + 480 to-copy RestoreJob
344-
# sum = 4110s (68m30s)
345-
# 72m therefore leaves ~3m30s of headroom for the kubectl apply / exec SQL
347+
# sum = 4470s (74m30s)
348+
# 78m therefore leaves ~3m30s of headroom for the kubectl apply / exec SQL
346349
# calls interleaved between those waits (convention 6: justified in-line).
347350
# The 660s HR waits are sized in "Sizing an HR-Ready budget",
348351
# docs/agents/e2e-testing.md.
@@ -352,7 +355,7 @@ spec:
352355
# wait_hr_ready on Stalled=True — so a genuinely broken backup reports in
353356
# seconds. The happy path is minutes.
354357
- script:
355-
timeout: 72m
358+
timeout: 78m
356359
content: |
357360
set -eu
358361
cd ../../..

0 commit comments

Comments
 (0)