Skip to content

Commit 1ea3168

Browse files
committed
fix(e2e): wait for the OIDC releases to finish installing
Both OIDC suites asserted render-side facts that hold a few seconds after the tenant Kubernetes CR is applied, and then ended. The install action for that release was still running at that point: the kind sets helm-install-disable-wait, so the install does not wait for the in-tenant addon HelmReleases, but it does still wait for the chart's two blocking post-install hook Jobs, and one of them polls for the Kamaji-issued admin kubeconfig before writing the tenant ClusterRoleBindings. A Helm release cannot be uninstalled while its install action is running, so ending the test there handed the remainder of the install to the cleanup phase on top of the teardown itself, against a single five-minute budget. Both suites were reported red on a cleanup step that tests nothing, seconds after every assertion had passed. Wait for the release on the last step instead: HelmRelease at Ready, then the OIDC bootstrap Job at status.succeeded. The ordering is what makes the second assert cheap -- a release cannot be Ready while a blocking hook Job is still running, so by then the Job is settled and reading it costs a poll. Asserting it also closes real missing coverage: that Job applies the per-user ClusterRoleBindings inside the tenant and patches the OIDC kubeconfig Secret, and neither suite checked that it ran. The two suites get different budgets deliberately. The System suite waits 20m, bounded on both sides by the install path rather than by a CI budget: below by a worst-case backoff on the other hook Job's non-optional token Secret mount, above by the kind's own helm-install-timeout, past which helm-controller abandons the install so a longer wait cannot observe a success that is no longer coming. The CustomConfig suite keeps a 5m wait, which removes the common case but not the worst one; matching them would add another copy of the same wait to a failure path that is already the most expensive thing either suite contributes, and the room has to come from bounding the collectors underneath it first. Note what this costs: a failing suite is now considerably more expensive than it was, because a failure in try runs catch handlers and a cleanup overrun did not. Every catch op in both suites carries an explicit timeout so that cost is a stated number rather than an inherited one, and the reds being replaced were false ones while a red that now costs the full wait means the install genuinely did not finish. State the rule in the e2e conventions, where the list of justified longer waits also gains the System suite's 20m, with the caveat that nothing checks that list and it therefore records cases rather than a maximum. Add a unit guard holding future fixtures to the rule, written to POSIX sh because cozytest.sh sources it under /bin/sh, which is dash on the runner and bash on a workstation. It discovers every Chainsaw Test that applies a Kubernetes CR through apply.file and requires both asserts on the last step, by name and by field, in that order, with both budgets positive and the Job's both shorter than the Ready wait and under an absolute ceiling. That ceiling is not redundant with the comparison: a relative bound loses its grip as the value it refers to grows, and raising a wait must not widen what may hide beneath it. Correct a claim both fixtures carried about an empty nodeGroups install producing no workers. The MachineDeployment does render with replicas: 0, but the chart's cluster-autoscaler then scales the default group to one worker in response to unschedulable in-tenant addon Pods, and CI events carry the whole chain. A helm-unittest comment describing the same case as render-side is corrected too, since waiting for the install is what stops that being true. This removes the term that was crossing the budget, which is not the larger of the two. On the run these numbers come from, the install remainder billed to cleanup was 57-117s while the teardown itself was 137-197s of the same 254s: taking the smaller term out is what brings the total under the 300s budget, and the bigger half is left unbounded. So a slow enough teardown can still cross that budget on its own, and that bound belongs to the chart rather than to an e2e test. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent cfaf2e0 commit 1ea3168

8 files changed

Lines changed: 1504 additions & 45 deletions

File tree

‎docs/agents/e2e-testing.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,7 @@ Trap-based cleanup was the single biggest source of false failures in the BATS s
5050

5151
- Resources a test creates that Chainsaw cannot reclaim — because a controller, not the test, owns them (e.g. the Velero `Backup`, `BackupStorageLocation`, and credentials secret in `cozy-velero` in the disabled `hack/e2e-chainsaw/backup/` suite) — must be pruned explicitly. Do not `|| true` over a stuck delete and leave stale state for the next run.
5252
- For nested tenants, tear down **child → parent with a hard wait for deletion between** each. Deleting the parent while a child is still uninstalling wedges the parent's cleanup Job on the child namespace, and both stuck uninstalls occupy helm-controller workers past the end of the test — starving whichever suite runs next. The kubernetes suites encode this ordering in `hack/e2e-chainsaw/_lib/run-kubernetes.sh`.
53+
- A test that applies a tenant `Kubernetes` CR must not end while that release is still installing. A Helm release cannot be uninstalled while its install action is running, and for this kind that action outlives the render-side asserts by design: `release.cozystack.io/helm-install-disable-wait` on the `ApplicationDefinition` drops the readiness wait, but the chart's blocking post-install hook Jobs still hold it, and one polls for the Kamaji-issued admin kubeconfig. A test that stops at its asserts therefore hands the remainder of the install to the `cleanup` budget on top of the teardown, and is reported red on a step that tests nothing. Wait for the release on the last step instead; checklist item 9 states the shape, and `hack/chainsaw-tenant-cluster-cleanup.bats` enforces it for a suite that applies the CR through `apply.file`. Two shapes are outside the guard's reach and hold to the rule by review: a suite that creates its tenant cluster from a script — `kubernetes-latest` and `kubernetes-previous` do, via a heredoc in `_lib/run-kubernetes.sh` — and a fixture that inlines the CR as `apply.resource:`, which leaves no manifest for either direction of the walk to find. This does not soften anything above it — cleanup stays bounded and blocking, and a genuinely stuck uninstall still fails.
5354

5455
### 5. The install gate must have teeth
5556

@@ -156,9 +157,10 @@ The suite is pinned to Chainsaw **v0.2.15** (the latest release as of May 2026);
156157
3. Imperative-only waits live in a `script` step; any bare `sleep` carries a `TODO(e2e-replace-fixed-timeouts):` justification.
157158
4. No test-level `EXIT`/`RETURN` trap — rely on Chainsaw cleanup; a self-contained trap is allowed inside a single `script` step (port-forward / temp dir), and inside an explicit subshell within a BATS `@test`, where it does not displace the runner's own handler. A BATS file that still carries test-level traps declares the count itself in `# EXIT-TRAP DEBT: N`; do not ask for one to be removed without checking that line, because the count is exact in both directions.
158159
5. Controller-created artifacts the test cannot reclaim are pruned explicitly; nested tenants delete child → parent with a wait-for-deletion between.
159-
6. Standard HR-Ready assert timeout is **5–6m**; longer waits (harbor 10m, NFS 10m, VM image pulls, platform-wide install 15m) are justified in-line.
160+
6. Standard HR-Ready assert timeout is **5–6m**; longer waits (harbor 10m, NFS 10m, VM image pulls, platform-wide install 15m, the `kubernetes-oidc-system` tenant-cluster install 20m) are justified in-line. Nothing checks this list, so read it as the cases someone recorded rather than as the tree's maximum; the budgets themselves are in the fixtures.
160161
7. Failure path attaches scoped diagnostics via a `catch:` block, never a silent pass.
161162
8. If it touches parent-HR behavior, add the `status.history` remediation guard (`hack/e2e-chainsaw/_lib/remediation-guard.sh`).
163+
9. A test that applies a tenant `Kubernetes` CR waits for its release to finish installing on its **last** step — `HelmRelease/<release>` at `Ready`, then `Job/<release>-oidc-bootstrap` at `status.succeeded` when the CR's `spec.oidc.mode` resolves to anything other than `None` — because a release whose install action is still running cannot be uninstalled, so whatever is left of the install is charged to Chainsaw's `cleanup` budget on top of the teardown. The second assert is a read rather than a wait: a release cannot be `Ready` while a blocking hook Job is still running, so by then the Job is in its final state, and pinning it says the fixture's payload actually ran rather than only that the install returned. The two are not interchangeable, and reversing them is what the order rule prevents: with the Job first, a broken install is waited out twice — once for a `succeeded` that never comes, then again by the Ready assert on its own budget. The Job's short timeout is not what prevents that; it is there because an assert reading a settled object has no use for an install-sized budget. The guard holds that timeout to an absolute ceiling as well as to a comparison against the Ready budget, because the comparison alone loses its grip as the Ready budget grows — under a 20m wait a 15m Job budget is still "shorter", and raising a wait must not widen what may hide beneath it. `hack/chainsaw-tenant-cluster-cleanup.bats` enforces this for a suite that applies the CR through `apply.file`; a script-driven suite is outside its reach and holds to the rule by review.
162164

163165
## In-flight direction (not yet the merged standard)
164166

0 commit comments

Comments
 (0)