Audit e2e Chainsaw post-HR-Ready asserts for redundancy under Flux v2.8 kstatus #2653

Activity

1. myasnikovdaniil

   added

   kind/cleanupCategorizes issue or PR as related to cleanup of code, process, or technical debtCategorizes issue or PR as related to cleanup of code, process, or technical debt

   area/testingIssues or PRs related to testing (e2e, bats, unit tests)Issues or PRs related to testing (e2e, bats, unit tests)

   on May 14, 2026
2. myasnikovdaniil

   self-assigned this

   on May 28, 2026
3. Aleksei Sviridkin (lexfrei)

   added

   triage/acceptedIndicates an issue is ready to be actively worked onIndicates an issue is ready to be actively worked on

   priority/backlogGeneral backlog priority. Lower than priority/important-longtermGeneral backlog priority. Lower than priority/important-longterm

   on Jul 1, 2026
4. myasnikovdaniil commented on Jul 9, 2026

   

   myasnikovdaniil

   on Jul 9, 2026

   ContributorAuthor

   More actions

   Audit results — 22 hack/e2e-apps/*.bats files

   Walked all 22 files (not ~18). 82 total kubectl wait invocations:

   Bucket	Count
   KEEP — HelmRelease --for=condition=ready gate	24
   KEEP — CR / operator-async (CRD-Established, CR conditions, functional Job/probe)	13
   KEEP — teardown --for=delete	11
   DROP candidate — post-HR wait on a standard resource kstatus subsumes	23

   The DROP-candidate count (23) matches the ~21 estimate. Total is 82 rather than ~36 because the estimate excluded all of gateway.bats (13 waits) and every --for=delete teardown wait.

   ⚠️ Key finding: only 11 of the 23 are safe to drop on this issue's premise

   A cozystack app HR's inventory is the CR, so HR Ready == CR Ready. kstatus health-checks the CR's condition, not the operator-created StatefulSet/PVC downstream of it. And etcd.bats:175 already documents in-tree that the etcd operator flips its CR Ready early (Available=True at ClusterDiscovered, before all members are up) — exactly why that test gates on readyMembers=3 rather than the HR. So a post-HR wait on an operator-created workload is not guaranteed redundant.

   Tier 1 — high-confidence DROP (chart renders the resource → in HR inventory → kstatus genuinely subsumes it):

   kuberture.bats:90     deployment kuberture (controller)
   qdrant.bats:26,28     sts + pvc (no operator; chart-rendered sts)
   openbao.bats:59,61    sts + pvc (chart-rendered sts)
   harbor.bats:84,86,88  deploy core/registry/portal (chart Deployments)
   mariadb.bats:49,108   deployment mariadb-$name-metrics (confirm chart-rendered)
   

   Tier 2 — verify-first DROP (operator-created downstream of a chart CR; remove only after confirming the operator gates CR Ready on the workload):

   mariadb.bats:45,104   sts (mariadb-operator)
   redis.bats:26,28,30   pvc/deploy/sts (spotahome)
   clickhouse.bats:44,48 sts (Altinity)
   mongodb.bats:40       sts (Percona)
   kafka.bats:49,51      pvc (Strimzi)
   postgres.bats:48      job init-job (only if in HR inventory, not a Helm hook)
   

   Tier 3 — redundant with a sibling wait, not with the HR:

   vminstance.bats:33,68  pvc Bound (redundant with the preceding `dv --for=condition=ready`)
   

   Recommendation

   Ship a first PR removing Tier 1 + the 2 Tier-3 vminstance PVCs (each with a one-line "chart-renders-it / redundant-with-dv" justification). Follow up on Tier 2 only after a live describe confirms each operator sets CR Ready after its rollout. Leave all 24 HR gates, 13 CR/functional waits, and 11 teardown deletes untouched.
5. myasnikovdaniil

   changed the title [-]Audit e2e bats post-HR-Ready waits for redundancy under Flux v2.8 kstatus[/-] [+]Audit e2e Chainsaw post-HR-Ready asserts for redundancy under Flux v2.8 kstatus[/+] on Oct 1, 2026
6. myasnikovdaniil commented on Oct 1, 2026

   

   myasnikovdaniil

   on Oct 1, 2026

   ContributorAuthor

   More actions

   Re-checked this against main (8799a4e). App suites moved from hack/e2e-apps/*.bats to Chainsaw in hack/e2e-chainsaw/ (only two monitoring bats files left in e2e-apps), so the 82 waits and file:line references in the audit above don't point at anything anymore.

   New baseline is 23 suites, 56 Test documents and 312 try operations (counted per document with yq): 184 assert, 66 script, 41 apply, 14 error, 7 delete. 34 of the asserts are on HelmRelease, 33 wait for Ready=True and one expects Ready=False (redis TLS flip rejection). Imperative waits left in suite YAML are only in gateway, 7 until kubectl loops and 1 kubectl wait.

   Candidates for this issue are 72 asserts on Deployment, StatefulSet, PVC, Pod, Job, Service and Endpoints, 69 of them after an HR assert (gateway and securitygroup have none). Service ports and Endpoints (25) plus Job and Pod (7) check shape or function that kstatus doesn't see, so they stay. The other 37 are workload readiness waits:

   • 10 in a nested -system HR: external-dns Deployment (2), harbor Deployment (3), kuberture Deployment, qdrant sts and PVC, openbao sts and PVC
   • 22 operator-created from a CR: mariadb (4), redis (7), valkey (4), clickhouse (3), mongodb, kafka PVC (2), rabbitmq
   • 2 vminstance PVC asserts that follow a DataVolume Ready=True assert in the same step
   • 3 that check rollout state and not readiness (opensearch sts twice, second assert in redis-tls-flip-rejected), these stay

   qdrant, openbao and external-dns charts render a nested HelmRelease and no workload (checked with helm template, no disableWait on it), so they are redundant through the nested HR's own health check, not because the chart renders them. I didn't confirm on a live cluster that the parent HR waits for the nested one, this is the hop to check before dropping anything. Harbor doesn't render with default values and kuberture is a system package, didn't check those two.

   mariadb metrics Deployment was Tier 1 pending a chart check and it is not chart-rendered (chart has only the MariaDB CR), so it moves to the operator group. Also mariadb sts, mariadb metrics Deployment and clickhouse replica sts assert status.replicas, that is pod count and not readiness, so dropping them loses an existence check.

   First PR then is the 10 nested-HR asserts plus the 2 vminstance PVC asserts, after the live check. Operator group only after describe on each, same as before.
7. myasnikovdaniil commented on Oct 1, 2026

   

   myasnikovdaniil

   on Oct 1, 2026

   Last edited by myasnikovdaniil

   ContributorAuthor

   More actions

   Checked the nested HR hop on a dev cluster (helm-controller v1.5.0). Installed a Qdrant app with a non-existent storageClass, so the PVC stays Pending and the nested -system HR can't become Ready.

   Parent HR stayed Ready=Unknown/Progressing for the whole 10m install timeout, then failed with timeout waiting for: [HelmRelease/tenant-root/qdrant-nested-wait-probe-system status: 'InProgress']. Nested HR was cycling between Progressing and InstallFailed (failed early due to stalled resources: StatefulSet ... Failed) the whole time. The message lists only the nested HR, so that is what the parent waits on, and on a normal install parent HR Ready covers the nested HR's StatefulSet and PVC.

   openbao and external-dns render the same nested HR and no workload (helm template), I didn't run them. harbor and kuberture are still unchecked. Upgrade path is not covered (9 of 101 upgraded HRs on that cluster have empty status.inventory, didn't check why), e2e only installs so it doesn't matter here.

   Correction to what I wrote here first: the qdrant sts and PVC asserts can't just go. The sts assert also pins the app.kubernetes.io/instance label that the catch selector relies on, and the PVC assert is what shows the PVC existed before the reclaim-on-delete check looks for its absence (same in kafka). For qdrant only readyReplicas: 1 is redundant.
8. myasnikovdaniil commented on Oct 2, 2026

   

   myasnikovdaniil

   on Oct 2, 2026

   ContributorAuthor

   More actions

   First step is up: #4652. It drops the readyReplicas checks in qdrant and external-dns (the instance-label pins stay) and the two vminstance PVC Bound asserts. The three suites pass on CI.

   Correction to the baseline comment above: openbao sts and PVC asserts don't belong in the redundant group. They run after the init and unseal step, and that step also carries the teardown finally, so they stay. The numbers there are from 8799a4e, several suites changed on main since and a kafka-metadata suite was added.

   What is left here: the operator-created workloads (mariadb, redis, valkey, clickhouse, mongodb, kafka, rabbitmq), each after a live describe, and harbor and kuberture, where I haven't checked the nested HR.
9. Aleksei Sviridkin (lexfrei)

   added a commit that references this issue on Oct 2, 2026

   test(e2e): drop readiness asserts that HelmRelease Ready already cove…

   ...

   42faf54