Repository navigation
test(e2e): HR-Ready waits sit under the install timeout of the release they wait on #4404
Description
Many e2e waits for a HelmRelease to become Ready give up before the release's own install timeout does. When that install fails, the suite reports a bare assert timeout, because the release has not recorded its failure reason yet. The wait's clock also starts earlier: at the step, while the install clock starts only after the Application is reconciled and the chart artifact is fetched.
The floor is the effective install timeout of the release. It is read in this order: spec.install.timeout, then spec.timeout, then the 5m helm-controller default. A release that cozystack-api generates from an ApplicationDefinition gets 10m from pkg/cmd/server/start.go. The release.cozystack.io/helm-install-timeout annotation overrides it: kubernetes 20m, kubernetes-nodes, tenant and monitoring 15m, kafka 7h. A release rendered by an app chart itself, like the nested <name>-system ones, gets whatever its template sets. A Package release gets the cozystack-operator --helmrelease-install-timeout, 10m by default.
#3769 adds a section on sizing these budgets to docs/agents/e2e-testing.md, with a scan that enumerates them. Below is that scan run against origin/main at 8d37e0b.
Chainsaw HR-Ready asserts at or under their floor
This is every assert on a HelmRelease with Ready=True in hack/e2e-chainsaw/*/chainsaw-test.yaml, except harbor, which has its own section below. 31 asserts in 17 suites.
file:line assert release floor
hack/e2e-chainsaw/bucket/chainsaw-test.yaml:37 6m bucket-test 10m
hack/e2e-chainsaw/bucket/chainsaw-test.yaml:191 6m bucket-cap1 10m
hack/e2e-chainsaw/bucket/chainsaw-test.yaml:201 6m bucket-cap2 10m
hack/e2e-chainsaw/bucket/chainsaw-test.yaml:211 6m bucket-cap3 10m
hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml:28 6m clickhouse-test 10m
hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml:138 6m clickhouse-test-backup 10m
hack/e2e-chainsaw/etcd/chainsaw-test.yaml:71 6m etcd 10m
hack/e2e-chainsaw/external-dns/chainsaw-test.yaml:26 6m test 10m
hack/e2e-chainsaw/external-dns/chainsaw-test.yaml:36 5m test-system 10m (chart spec.timeout)
hack/e2e-chainsaw/external-dns/chainsaw-test.yaml:78 6m test-prefix 10m
hack/e2e-chainsaw/external-dns/chainsaw-test.yaml:88 5m test-prefix-system 10m (chart spec.timeout)
hack/e2e-chainsaw/foundationdb/chainsaw-test.yaml:33 5m foundationdb-test 10m
hack/e2e-chainsaw/kafka/chainsaw-test.yaml:37 6m kafka-test 7h (kafka-rd override)
hack/e2e-chainsaw/kuberture/chainsaw-test.yaml:46 5m kuberture 10m (cozystack-operator default)
hack/e2e-chainsaw/mariadb/chainsaw-test.yaml:25 6m mariadb-test 10m
hack/e2e-chainsaw/mariadb/chainsaw-test.yaml:316 6m mariadb-single 10m
hack/e2e-chainsaw/mongodb/chainsaw-test.yaml:30 6m mongodb-test 10m
hack/e2e-chainsaw/openbao/chainsaw-test.yaml:61 6m openbao-test 10m
hack/e2e-chainsaw/openbao/chainsaw-test.yaml:71 5m openbao-test-system 10m (chart spec.timeout)
hack/e2e-chainsaw/opensearch/chainsaw-test.yaml:127 6m opensearch-test 10m
hack/e2e-chainsaw/postgres/chainsaw-test.yaml:31 6m postgres-test 10m
hack/e2e-chainsaw/qdrant/chainsaw-test.yaml:76 6m qdrant-test 10m
hack/e2e-chainsaw/qdrant/chainsaw-test.yaml:86 5m qdrant-test-system 10m (chart spec.timeout)
hack/e2e-chainsaw/rabbitmq/chainsaw-test.yaml:26 6m rabbitmq-test 10m
hack/e2e-chainsaw/redis/chainsaw-test.yaml:24 6m redis-test 10m
hack/e2e-chainsaw/redis/chainsaw-test.yaml:119 6m redis-secure 10m
hack/e2e-chainsaw/redis/chainsaw-test.yaml:536 6m redis-flip 10m
hack/e2e-chainsaw/valkey/chainsaw-test.yaml:24 6m valkey-test 10m
hack/e2e-chainsaw/vminstance/chainsaw-test.yaml:23 6m vm-disk-test 10m
hack/e2e-chainsaw/vminstance/chainsaw-test.yaml:73 6m vm-disk-vmi-test 10m
hack/e2e-chainsaw/vminstance/chainsaw-test.yaml:106 6m vm-instance-test 10m
#3769 raises both mariadb asserts to 11m, so they leave this list when it merges. No suite can clear the kafka floor inside the 215m E2E job, so that one is a known trade. Two things are left out on purpose. redis/chainsaw-test.yaml:574 asserts Ready=False on a rejected upgrade, so it is not a readiness wait. The kubectl wait hr calls in scripts are listed by a separate grep in the doc: grep -rnE 'kubectl +wait.*(^|[^[:alnum:]])hr([^[:alnum:]]|$)' hack/.
harbor
Harbor uses one literal for two asserts with different floors.
hack/e2e-chainsaw/harbor/chainsaw-test.yaml:151 10m harbor-test 10m equal, so the assert expires first
hack/e2e-chainsaw/harbor/chainsaw-test.yaml:190 10m harbor-test-system 15m chart spec.timeout in packages/apps/harbor/templates/harbor.yaml
The first one sits exactly on its floor, and that behaves the same as being under it. The second is five minutes under. The chart renders the nested release with its own 15m, so the 10m default never applies to it.
Backup walkthrough waits
Every wait_hr_ready call in examples/backups/ is under its floor too. These scripts run inside the Chainsaw op of the matching suite, so the same trade applies.
examples/backups/clickhouse/03-create-bucket.sh:30 300s bucket release 10m
examples/backups/clickhouse/04-create-clickhouse.sh:57 300s clickhouse source 10m
examples/backups/clickhouse/07-restore-to-copy.sh:70 300s clickhouse target 10m
examples/backups/kafka-metadata/run-all.sh:36 300s kafka source 7h
examples/backups/kafka-metadata/run-all.sh:113 300s kafka target 7h
examples/backups/mariadb/run-all.sh:52 300s bucket release 10m
examples/backups/mariadb/run-all.sh:139 300s mariadb source 10m
examples/backups/mariadb/run-all.sh:174 300s mariadb target 10m
examples/backups/mongodb/run-all.sh:47 300s bucket release 10m
examples/backups/mongodb/run-all.sh:78 300s mongodb source 10m
examples/backups/mongodb/run-all.sh:105 300s mongodb target 10m
examples/backups/postgres/run-all.sh:41 300s bucket release 10m
examples/backups/postgres/run-all.sh:126 360s postgres source 10m
examples/backups/postgres/run-all.sh:170 360s postgres target 10m
examples/backups/rabbitmq/run-all.sh:46 300s bucket release 10m
examples/backups/rabbitmq/run-all.sh:133 300s rabbitmq source 10m
examples/backups/rabbitmq/run-all.sh:179 300s rabbitmq target 10m
examples/backups/redis/run-all.sh:44 300s bucket release 10m
#3769 raises the two mariadb instance waits to 660s. The mariadb bucket wait stays at 300s.
Why not just raise all of them
A wait above its floor gets a diagnosis: the release records why its install failed before the wait gives up. The cost is extra time on every red run of that suite. Suites run one after another in a single 215m job, and a job cancelled at the cap loses both its log and cozyreport.tgz. A walkthrough wait also has the Chainsaw op around it. Raising the wait alone makes more slow runs end in SIGKILL instead of a report, and raising the op adds its own minutes. That trade is made per suite, so this issue is a list to decide from, not a request to raise everything.
Separate defect: two helpers print no reason on timeout
The redis and kafka-metadata walkthroughs have their own wait_hr_ready, and on timeout they print only Timeout waiting for HelmRelease/<name> to become Ready. There is no condition, no reason and no release history. The copy shared by clickhouse, mariadb, mongodb, postgres and rabbitmq prints the Ready message on origin/main. #3769 extends that copy with every condition and its reason, the release history, and separate lines for a release that never appeared and for a kubectl that could not answer. The redis and kafka-metadata helpers are not copies of it, so they get none of this. Fixing them does not depend on the budgets above: it makes a timeout readable whatever the number is.
Activity
lexfrei commentedon Sep 23, 2026
#3769 now covers more of this list than the body says. Once it merges, these lines are off it: the two mariadb Chainsaw asserts (
mariadb:25,mariadb:316), and the database waits in the mariadb, mongodb, rabbitmq and clickhouse walkthroughs (mariadb/run-all.sh:139,174,mongodb/run-all.sh:78,105,rabbitmq/run-all.sh:133,179,clickhouse/04-create-clickhouse.sh:57,clickhouse/07-restore-to-copy.sh:70), each raised to 660s with its Chainsaw op grown by the same amount. The bucket waits, postgres, kafka-metadata, harbor, the other Chainsaw asserts and the redis and kafka-metadata diagnostic gap are unchanged.fix(tests): let the mariadb waits cover a first boot (#3769)
lexfrei commentedon Sep 23, 2026
Duplicate of #3771, which I missed when I searched before filing. #3771 has the same finding from August, when fourteen suites were counted. The enumeration above is the refreshed one against 8d37e0b, and it adds the backup walkthrough waits, the harbor pair and the redis and kafka-metadata diagnostic gap. #3769 has since merged and retired the mariadb, mongodb, rabbitmq and clickhouse walkthrough waits and both mariadb asserts. Closing this one so the discussion stays in #3771.