Repository navigation
Commit d367846
authored
test(e2e): cover the computeplane module end to end (#4713)
## What this PR does
This adds an e2e suite for the ComputePlane tenant module, which had
only render tests so far. The suite lives in
`hack/e2e-chainsaw/computeplane/`. It turns the module on in its own
tenant, `tenant-cplane`, so the shared `tenant-test` is never toggled.
That tenant is a child of `tenant-root`, not of `tenant-test`, because a
tenant two levels down cannot reach the etcd it inherits from root
(#4726).
What it checks on a real cluster:
- the module release installs, and the cluster release it renders keeps
the pinned addons and the `internal.cozystack.io/computeplane` label;
- the pool release gets only the keys from the module's per-pool
allowlist (the patch sends a `nameOverride` through the ComputePlane API
and checks it is dropped);
- Kamaji writes `computeplane-cluster-admin-kubeconfig`, and the tenant
can't get at it: the Secret is not labelled `tenantresource=true`,
`tenantsecrets` does not list it to the tenant's super-admin group, and
`kubectl auth can-i get` answers `no` for that group (the read was
possible until #4722);
- cert-manager and ingress-nginx come up inside the cluster, and the
Proxied ingress Service gets a ready endpoint;
- the module and cluster releases have no remediation cycle in
`status.history`;
- on teardown, the in-cluster addon releases are gone before the
KamajiControlPlane starts deleting;
- turning the module off removes all of it: releases, in-cluster addons,
control plane, Secret, worker VMs and disks, and the LINSTOR pools go
back to their pre-suite level.
The issue names the Secret `computeplane-admin-kubeconfig`. The module
writes `computeplane-cluster-admin-kubeconfig`, so the suite checks that
name.
The issue also lists fluxcd as a pinned addon. That is out of date. The
module runs no in-cluster Flux on purpose
(`packages/extra/computeplane/templates/cluster.yaml:108-110`), so the
suite checks cert-manager and ingress-nginx only.
One part of the issue is not covered here. The `placement: ComputePlane`
routing in cozystack-api doesn't exist yet, so there is no catalog app
to run on the cluster. The issue ties that to "together with, or right
after" this test, so I left it for the routing work, tracked in #4715.
Things the review found outside this diff, each filed on its own: the
module release's 10m install timeout against a slow cluster bringup
(#4714), the module docs not matching the per-pool allowlist (#4716),
and the LINSTOR helpers in `_lib` running `kubectl exec` without a
timeout (#4111).
The suite brings its own pool: `md0`, one worker, same sizing as the
kubernetes suites. The default `md0` that the module renders for empty
`nodeGroups` asks for the `replicated` StorageClass. The container lane
doesn't have it, so on this lane the default pool stays covered by the
render tests only.
ComputePlane workers can't use the in-sandbox Talos image cache. The
pool allowlist drops `talos`, so the worker pulls its disk from the
public Image Factory (#4700). If the worker join or an addon release
times out, the catch block prints the worker image source and points at
that issue, so a red run caused by the factory is easy to tell apart.
The suite itself needs no entry in `hack/select-e2e.sh`, because
`computeplane-application` maps to it by name. That source depends on
`kubernetes-application` and re-sources the kubernetes charts. So PRs
that touch the kubernetes charts or ingress-nginx now select
`computeplane` too, which I estimate adds 12-15 minutes to a green run
(kubernetes-previous takes about 8 minutes with two workers, plus
teardown). A red run can take much longer, because the waits sit above
the release install timeouts. The suite header has the numbers. It is
the third suite that brings up a tenant cluster, and the 215m E2E job
cap is sized for the red path of the kubernetes pair only. If worker
join breaks everywhere, all three go red and can run past the cap, which
cancels the job and loses its log. I left that as an accepted risk.
`hack/select-e2e_test.bats` now expects `computeplane` in the
ingress-nginx selection.
The computeplane source also owns `apps/kubernetes-nodes`. Before this
PR a pool chart edit reached no suite and ran everything. With the new
suite alone it would have selected only `computeplane`, which never
creates `KubernetesNodes` natively. So `kubernetes-nodes-application`
now maps to `kubernetes-latest kubernetes-previous computeplane` in
`hack/select-e2e.sh`, and two new tests pin that for a pool chart edit
and a `kubernetes-nodes-rd` edit.
The suite is not in the leak-guard allowlist. Its teardown already fails
when a TenantControlPlane sticks on its finalizer (#4489), because the
admin kubeconfig Secret it owns stays behind, so an exemption would only
hide other leaks. The README suite list now names the suite.
JFYI, not a defect claim: the Proxied ingress Service from the
kubernetes chart has `internal.cozystack.io/tenantresource: "true"`. So
the tenant sees it, even though `computeplane-rd` has `services.include:
[]`. The module doesn't promise to hide it.
How I checked it. `chainsaw lint` passes at v0.2.15, the version the
sandbox pins. I also ran the suite against an envtest apiserver, with
fake controllers applying the real `helm template` output of the tenant
and computeplane charts. The intact run is green. Each of these
breakages turns it red at the step meant to catch it: label removed,
addon `exposeMethod` changed, a key outside the pool allowlist, the
allowlist `pick` replaced by the whole group, stale Ready from an older
generation, wrong Secret name, Secret labelled `tenantresource`, Secret
listed in `tenantsecrets`, a direct RBAC grant on the Secret,
ingress-nginx addon not Ready, no ready ingress endpoint, an
`uninstalled` history entry, the control plane deleted before the addon
releases, an addon release or worker VM left after teardown.
Closes #3369
Closes #4747
### Screenshots
Not a UI change.
### Downstream repositories
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
test(e2e): add an end-to-end suite for the ComputePlane tenant module
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Tests**
* Added end-to-end coverage for enabling and disabling the ComputePlane
tenant module, including generated releases, pool value filtering, admin
credential access, addon and cluster readiness, and resource cleanup.
* Kubernetes nodes changes now select the ComputePlane suite alongside
the latest and previous Kubernetes suites.
* **Documentation**
* Added the ComputePlane suite to the documented end-to-end test suite
list.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->7 files changed
Lines changed: 768 additions & 4 deletions
File tree
- hack
- e2e-chainsaw
- computeplane
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
28 | 28 | | |
29 | 29 | | |
30 | 30 | | |
| 31 | + | |
31 | 32 | | |
32 | 33 | | |
33 | 34 | | |
| |||
335 | 336 | | |
336 | 337 | | |
337 | 338 | | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
21 | | - | |
| 21 | + | |
22 | 22 | | |
23 | 23 | | |
24 | 24 | | |
| |||
55 | 55 | | |
56 | 56 | | |
57 | 57 | | |
58 | | - | |
| 58 | + | |
59 | 59 | | |
60 | 60 | | |
61 | 61 | | |
| |||
0 commit comments