Repository navigation
Commit 1ea3168
committed
fix(e2e): wait for the OIDC releases to finish installing
Both OIDC suites asserted render-side facts that hold a few seconds
after the tenant Kubernetes CR is applied, and then ended. The install
action for that release was still running at that point: the kind sets
helm-install-disable-wait, so the install does not wait for the
in-tenant addon HelmReleases, but it does still wait for the chart's
two blocking post-install hook Jobs, and one of them polls for the
Kamaji-issued admin kubeconfig before writing the tenant
ClusterRoleBindings.
A Helm release cannot be uninstalled while its install action is
running, so ending the test there handed the remainder of the install
to the cleanup phase on top of the teardown itself, against a single
five-minute budget. Both suites were reported red on a cleanup step
that tests nothing, seconds after every assertion had passed.
Wait for the release on the last step instead: HelmRelease at Ready,
then the OIDC bootstrap Job at status.succeeded. The ordering is what
makes the second assert cheap -- a release cannot be Ready while a
blocking hook Job is still running, so by then the Job is settled and
reading it costs a poll. Asserting it also closes real missing
coverage: that Job applies the per-user ClusterRoleBindings inside the
tenant and patches the OIDC kubeconfig Secret, and neither suite
checked that it ran.
The two suites get different budgets deliberately. The System suite
waits 20m, bounded on both sides by the install path rather than by a
CI budget: below by a worst-case backoff on the other hook Job's
non-optional token Secret mount, above by the kind's own
helm-install-timeout, past which helm-controller abandons the install
so a longer wait cannot observe a success that is no longer coming.
The CustomConfig suite keeps a 5m wait, which removes the common case
but not the worst one; matching them would add another copy of the
same wait to a failure path that is already the most expensive thing
either suite contributes, and the room has to come from bounding the
collectors underneath it first.
Note what this costs: a failing suite is now considerably more
expensive than it was, because a failure in try runs catch handlers
and a cleanup overrun did not. Every catch op in both suites carries
an explicit timeout so that cost is a stated number rather than an
inherited one, and the reds being replaced were false ones while a red
that now costs the full wait means the install genuinely did not
finish.
State the rule in the e2e conventions, where the list of justified
longer waits also gains the System suite's 20m, with the caveat that
nothing checks that list and it therefore records cases rather than a
maximum. Add a unit guard holding future fixtures to the rule, written
to POSIX sh because cozytest.sh sources it under /bin/sh, which is
dash on the runner and bash on a workstation. It discovers every
Chainsaw Test that applies a Kubernetes CR through apply.file and
requires both asserts on the last step, by name and by field, in that
order, with both budgets positive and the Job's both shorter than the
Ready wait and under an absolute ceiling. That ceiling is not
redundant with the comparison: a relative bound loses its grip as the
value it refers to grows, and raising a wait must not widen what may
hide beneath it.
Correct a claim both fixtures carried about an empty nodeGroups
install producing no workers. The MachineDeployment does render with
replicas: 0, but the chart's cluster-autoscaler then scales the
default group to one worker in response to unschedulable in-tenant
addon Pods, and CI events carry the whole chain. A helm-unittest
comment describing the same case as render-side is corrected too,
since waiting for the install is what stops that being true.
This removes the term that was crossing the budget, which is not the
larger of the two. On the run these numbers come from, the install
remainder billed to cleanup was 57-117s while the teardown itself was
137-197s of the same 254s: taking the smaller term out is what brings
the total under the 300s budget, and the bigger half is left
unbounded. So a slow enough teardown can still cross that budget on
its own, and that bound belongs to the chart rather than to an e2e
test.
Assisted-By: Claude <[email protected]>
Signed-off-by: Aleksei Sviridkin <[email protected]>1 parent cfaf2e0 commit 1ea3168
8 files changed
Lines changed: 1504 additions & 45 deletions
File tree
- docs/agents
- hack
- e2e-chainsaw
- kubernetes-oidc-customconfig
- kubernetes-oidc-system
- packages/apps/kubernetes/tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
50 | 50 | | |
51 | 51 | | |
52 | 52 | | |
| 53 | + | |
53 | 54 | | |
54 | 55 | | |
55 | 56 | | |
| |||
156 | 157 | | |
157 | 158 | | |
158 | 159 | | |
159 | | - | |
| 160 | + | |
160 | 161 | | |
161 | 162 | | |
| 163 | + | |
162 | 164 | | |
163 | 165 | | |
164 | 166 | | |
| |||
0 commit comments