Skip to content

Commit d367846

Browse files
authored
test(e2e): cover the computeplane module end to end (#4713)
## What this PR does This adds an e2e suite for the ComputePlane tenant module, which had only render tests so far. The suite lives in `hack/e2e-chainsaw/computeplane/`. It turns the module on in its own tenant, `tenant-cplane`, so the shared `tenant-test` is never toggled. That tenant is a child of `tenant-root`, not of `tenant-test`, because a tenant two levels down cannot reach the etcd it inherits from root (#4726). What it checks on a real cluster: - the module release installs, and the cluster release it renders keeps the pinned addons and the `internal.cozystack.io/computeplane` label; - the pool release gets only the keys from the module's per-pool allowlist (the patch sends a `nameOverride` through the ComputePlane API and checks it is dropped); - Kamaji writes `computeplane-cluster-admin-kubeconfig`, and the tenant can't get at it: the Secret is not labelled `tenantresource=true`, `tenantsecrets` does not list it to the tenant's super-admin group, and `kubectl auth can-i get` answers `no` for that group (the read was possible until #4722); - cert-manager and ingress-nginx come up inside the cluster, and the Proxied ingress Service gets a ready endpoint; - the module and cluster releases have no remediation cycle in `status.history`; - on teardown, the in-cluster addon releases are gone before the KamajiControlPlane starts deleting; - turning the module off removes all of it: releases, in-cluster addons, control plane, Secret, worker VMs and disks, and the LINSTOR pools go back to their pre-suite level. The issue names the Secret `computeplane-admin-kubeconfig`. The module writes `computeplane-cluster-admin-kubeconfig`, so the suite checks that name. The issue also lists fluxcd as a pinned addon. That is out of date. The module runs no in-cluster Flux on purpose (`packages/extra/computeplane/templates/cluster.yaml:108-110`), so the suite checks cert-manager and ingress-nginx only. One part of the issue is not covered here. The `placement: ComputePlane` routing in cozystack-api doesn't exist yet, so there is no catalog app to run on the cluster. The issue ties that to "together with, or right after" this test, so I left it for the routing work, tracked in #4715. Things the review found outside this diff, each filed on its own: the module release's 10m install timeout against a slow cluster bringup (#4714), the module docs not matching the per-pool allowlist (#4716), and the LINSTOR helpers in `_lib` running `kubectl exec` without a timeout (#4111). The suite brings its own pool: `md0`, one worker, same sizing as the kubernetes suites. The default `md0` that the module renders for empty `nodeGroups` asks for the `replicated` StorageClass. The container lane doesn't have it, so on this lane the default pool stays covered by the render tests only. ComputePlane workers can't use the in-sandbox Talos image cache. The pool allowlist drops `talos`, so the worker pulls its disk from the public Image Factory (#4700). If the worker join or an addon release times out, the catch block prints the worker image source and points at that issue, so a red run caused by the factory is easy to tell apart. The suite itself needs no entry in `hack/select-e2e.sh`, because `computeplane-application` maps to it by name. That source depends on `kubernetes-application` and re-sources the kubernetes charts. So PRs that touch the kubernetes charts or ingress-nginx now select `computeplane` too, which I estimate adds 12-15 minutes to a green run (kubernetes-previous takes about 8 minutes with two workers, plus teardown). A red run can take much longer, because the waits sit above the release install timeouts. The suite header has the numbers. It is the third suite that brings up a tenant cluster, and the 215m E2E job cap is sized for the red path of the kubernetes pair only. If worker join breaks everywhere, all three go red and can run past the cap, which cancels the job and loses its log. I left that as an accepted risk. `hack/select-e2e_test.bats` now expects `computeplane` in the ingress-nginx selection. The computeplane source also owns `apps/kubernetes-nodes`. Before this PR a pool chart edit reached no suite and ran everything. With the new suite alone it would have selected only `computeplane`, which never creates `KubernetesNodes` natively. So `kubernetes-nodes-application` now maps to `kubernetes-latest kubernetes-previous computeplane` in `hack/select-e2e.sh`, and two new tests pin that for a pool chart edit and a `kubernetes-nodes-rd` edit. The suite is not in the leak-guard allowlist. Its teardown already fails when a TenantControlPlane sticks on its finalizer (#4489), because the admin kubeconfig Secret it owns stays behind, so an exemption would only hide other leaks. The README suite list now names the suite. JFYI, not a defect claim: the Proxied ingress Service from the kubernetes chart has `internal.cozystack.io/tenantresource: "true"`. So the tenant sees it, even though `computeplane-rd` has `services.include: []`. The module doesn't promise to hide it. How I checked it. `chainsaw lint` passes at v0.2.15, the version the sandbox pins. I also ran the suite against an envtest apiserver, with fake controllers applying the real `helm template` output of the tenant and computeplane charts. The intact run is green. Each of these breakages turns it red at the step meant to catch it: label removed, addon `exposeMethod` changed, a key outside the pool allowlist, the allowlist `pick` replaced by the whole group, stale Ready from an older generation, wrong Secret name, Secret labelled `tenantresource`, Secret listed in `tenantsecrets`, a direct RBAC grant on the Secret, ingress-nginx addon not Ready, no ready ingress endpoint, an `uninstalled` history entry, the control plane deleted before the addon releases, an addon release or worker VM left after teardown. Closes #3369 Closes #4747 ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note test(e2e): add an end-to-end suite for the ComputePlane tenant module ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Tests** * Added end-to-end coverage for enabling and disabling the ComputePlane tenant module, including generated releases, pool value filtering, admin credential access, addon and cluster readiness, and resource cleanup. * Kubernetes nodes changes now select the ComputePlane suite alongside the latest and previous Kubernetes suites. * **Documentation** * Added the ComputePlane suite to the documented end-to-end test suite list. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 3fc45ed + d35493a commit d367846

7 files changed

Lines changed: 768 additions & 4 deletions

File tree

‎hack/chainsaw-op-budgets.bats‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@ GATEWAY_SUITE="hack/e2e-chainsaw/gateway/chainsaw-test.yaml"
2828
GATEWAY_STEP="parent-child-and-route-drive-listener-set"
2929
REDIS_SUITE="hack/e2e-chainsaw/redis/chainsaw-test.yaml"
3030
REDIS_STEP="verify-tenant-ca-projection"
31+
COMPUTEPLANE_SUITE="hack/e2e-chainsaw/computeplane/chainsaw-test.yaml"
3132

3233
# Seconds the gateway op keeps above its summed waits for what carries no cap
3334
# of its own: three applies and one label read.
@@ -335,3 +336,15 @@ rc=0") printf '%s\n' "$rt_out" >&2; echo "the step passed with a hung read" >&2;
335336
[ "$rt_took" -lt 15 ] || { echo "the step took ${rt_took}s against a 2s budget; the read was not clamped" >&2; exit 1; }
336337
rm -rf "$rt_dir"
337338
}
339+
340+
@test "every kubectl in the computeplane scripts runs under its own timeout" {
341+
# The suite's budget comments sum op ceilings and inner bounds, which hold
342+
# only while no read can outlive them; kubectl's own --timeout does not
343+
# count, it leaves the requests before the wait unbounded. Calls inside
344+
# sourced helpers are the helpers' to bound and are not seen here.
345+
cp_scripts=$(yq '.spec.steps[] | (.try[]?, .catch[]?, .finally[]?) | .script.content // ""' "$COMPUTEPLANE_SUITE")
346+
printf '%s\n' "$cp_scripts" | grep -q 'kubectl ' || { echo "no kubectl found in the scripts of $COMPUTEPLANE_SUITE; the check reads nothing" >&2; exit 1; }
347+
cp_bare=$(printf '%s\n' "$cp_scripts" | awk '/^[[:space:]]*#/ { next }
348+
{ line = " " $0; all = gsub(/[^-_[:alnum:]]kubectl /, "&", line); ok = gsub(/timeout (-k [0-9]+ )?([0-9]+|[$][(][(][^)]*[)][)]) kubectl /, "&", line); if (all != ok) print }')
349+
[ -z "$cp_bare" ] || { printf '%s\n' "$cp_bare" >&2; echo "kubectl without a timeout in the computeplane scripts" >&2; exit 1; }
350+
}
Lines changed: 163 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,163 @@
1+
#!/usr/bin/env bats
2+
# -----------------------------------------------------------------------------
3+
# Unit tests for hack/e2e-chainsaw/computeplane/kcp-hold.sh, the finalizer the
4+
# computeplane teardown holds the KamajiControlPlane with. A release that
5+
# reports success without removing the entry leaves the control plane and its
6+
# tenant stuck in deletion for good, so every path that cannot prove the entry
7+
# is gone has to fail.
8+
#
9+
# kubectl is a stub on PATH: `get` prints $FIX/kcp.json when it exists (an
10+
# empty answer is --ignore-not-found on an absent object) and exits with
11+
# $FIX/get-rc, and `patch` records its JSON patch in $FIX/patch and exits with
12+
# $FIX/patch-rc.
13+
#
14+
# cozytest.sh runs the bodies under sh and puts `return 0` in front of every
15+
# bare `}`, so the helpers whose status matters close with `} # name`.
16+
#
17+
# Run with: hack/cozytest.sh hack/computeplane-kcp-hold_test.bats
18+
# -----------------------------------------------------------------------------
19+
20+
HACK_DIR="$(cd "$(dirname "${BATS_TEST_FILENAME:-$0}")" && pwd)"
21+
KH_LIB="$HACK_DIR/e2e-chainsaw/computeplane/kcp-hold.sh"
22+
export KH_LIB
23+
24+
HOLD=e2e.cozystack.io/teardown-order
25+
26+
kh() {
27+
PATH="$FIX/bin:$PATH" sh -c '. "$KH_LIB"; "$@"' kh "$@"
28+
} # kh
29+
30+
new_fixture() {
31+
FIX=$(mktemp -d)
32+
export FIX
33+
mkdir -p "$FIX/bin"
34+
cat >"$FIX/bin/kubectl" <<'EOF'
35+
#!/bin/sh
36+
case " $* " in
37+
*" get "*)
38+
[ ! -f "$FIX/kcp.json" ] || cat "$FIX/kcp.json"
39+
exit "$(cat "$FIX/get-rc" 2>/dev/null || echo 0)" ;;
40+
*" patch "*)
41+
for a; do last=$a; done
42+
printf '%s' "$last" >"$FIX/patch"
43+
exit "$(cat "$FIX/patch-rc" 2>/dev/null || echo 0)" ;;
44+
esac
45+
echo "kubectl stub: unexpected call: $*" >&2
46+
exit 99
47+
EOF
48+
chmod +x "$FIX/bin/kubectl"
49+
} # new_fixture
50+
51+
# kcp_with FINALIZERS_JSON — the object the stub returns; "null" for none.
52+
kcp_with() {
53+
printf '{"metadata":{"name":"computeplane-cluster","finalizers":%s}}' "$1" | jq 'del(.metadata.finalizers | select(. == null))' >"$FIX/kcp.json"
54+
} # kcp_with
55+
56+
patch_is() {
57+
[ -f "$FIX/patch" ] || { echo "no patch was sent" >&2; return 1; }
58+
got=$(jq -cS . "$FIX/patch")
59+
want=$(printf '%s' "$1" | jq -cS .)
60+
[ "$got" = "$want" ] || { echo "patch: got $got, want $want" >&2; return 1; }
61+
} # patch_is
62+
63+
@test "hold creates the finalizer list when the object has none" {
64+
new_fixture
65+
kcp_with null
66+
kh kcp_hold
67+
patch_is "[{\"op\":\"add\",\"path\":\"/metadata/finalizers\",\"value\":[\"$HOLD\"]}]"
68+
rm -rf "$FIX"
69+
}
70+
71+
@test "hold appends behind a test of the existing list" {
72+
new_fixture
73+
kcp_with '["a.example/x"]'
74+
kh kcp_hold
75+
patch_is "[{\"op\":\"test\",\"path\":\"/metadata/finalizers\",\"value\":[\"a.example/x\"]},{\"op\":\"add\",\"path\":\"/metadata/finalizers/-\",\"value\":\"$HOLD\"}]"
76+
rm -rf "$FIX"
77+
}
78+
79+
@test "hold returns 2 and patches nothing when the object is absent" {
80+
new_fixture
81+
rc=0
82+
kh kcp_hold || rc=$?
83+
[ "$rc" -eq 2 ] || { echo "rc=$rc, want 2" >&2; exit 1; }
84+
[ ! -f "$FIX/patch" ] || { echo "patched an absent object" >&2; exit 1; }
85+
rm -rf "$FIX"
86+
}
87+
88+
@test "hold fails when the patch fails" {
89+
new_fixture
90+
kcp_with null
91+
echo 1 >"$FIX/patch-rc"
92+
if kh kcp_hold; then echo "hold succeeded on a failed patch" >&2; exit 1; fi
93+
rm -rf "$FIX"
94+
}
95+
96+
@test "hold fails without patching when the object cannot be read as JSON" {
97+
new_fixture
98+
echo 'not json' >"$FIX/kcp.json"
99+
if kh kcp_hold; then echo "hold succeeded on unreadable input" >&2; exit 1; fi
100+
[ ! -f "$FIX/patch" ] || { echo "patched with $(cat "$FIX/patch")" >&2; exit 1; }
101+
rm -rf "$FIX"
102+
}
103+
104+
@test "hold fails with 1, not the absent object's 2, when the read fails" {
105+
new_fixture
106+
echo 1 >"$FIX/get-rc"
107+
rc=0
108+
kh kcp_hold || rc=$?
109+
[ "$rc" -eq 1 ] || { echo "rc=$rc, want 1" >&2; exit 1; }
110+
[ ! -f "$FIX/patch" ] || { echo "patched after a failed read" >&2; exit 1; }
111+
rm -rf "$FIX"
112+
}
113+
114+
@test "release removes its own entry by index behind a test of that index" {
115+
new_fixture
116+
kcp_with "[\"a.example/x\",\"$HOLD\",\"b.example/y\"]"
117+
out=$(kh kcp_release)
118+
patch_is "[{\"op\":\"test\",\"path\":\"/metadata/finalizers/1\",\"value\":\"$HOLD\"},{\"op\":\"remove\",\"path\":\"/metadata/finalizers/1\"}]"
119+
[ -n "$out" ] || { echo "release printed nothing" >&2; exit 1; }
120+
rm -rf "$FIX"
121+
}
122+
123+
@test "release succeeds without patching when the entry is not there" {
124+
new_fixture
125+
kcp_with '["a.example/x"]'
126+
kh kcp_release
127+
[ ! -f "$FIX/patch" ] || { echo "patched with $(cat "$FIX/patch")" >&2; exit 1; }
128+
rm -rf "$FIX"
129+
}
130+
131+
@test "release succeeds without patching when the object is gone" {
132+
new_fixture
133+
kh kcp_release
134+
[ ! -f "$FIX/patch" ] || { echo "patched an absent object" >&2; exit 1; }
135+
rm -rf "$FIX"
136+
}
137+
138+
@test "release fails silently on stdout when the patch fails" {
139+
new_fixture
140+
kcp_with "[\"$HOLD\"]"
141+
echo 1 >"$FIX/patch-rc"
142+
rc=0
143+
out=$(kh kcp_release) || rc=$?
144+
[ "$rc" -eq 1 ] || { echo "rc=$rc, want 1" >&2; exit 1; }
145+
[ -z "$out" ] || { echo "printed '$out' on a failed release" >&2; exit 1; }
146+
rm -rf "$FIX"
147+
}
148+
149+
@test "release fails when the object cannot be read as JSON" {
150+
new_fixture
151+
echo 'not json' >"$FIX/kcp.json"
152+
if kh kcp_release; then echo "release reported the finalizer gone without reading the object" >&2; exit 1; fi
153+
rm -rf "$FIX"
154+
}
155+
156+
@test "release fails when the read fails" {
157+
new_fixture
158+
kcp_with "[\"$HOLD\"]"
159+
echo 1 >"$FIX/get-rc"
160+
if kh kcp_release; then echo "release reported the finalizer gone after a failed read" >&2; exit 1; fi
161+
[ ! -f "$FIX/patch" ] || { echo "patched after a failed read" >&2; exit 1; }
162+
rm -rf "$FIX"
163+
}

‎hack/e2e-chainsaw/README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ hack/e2e-chainsaw/
1818
└── ...
1919
```
2020

21-
Suites: `postgres`, `bucket`, `mariadb`, `mongodb`, `redis`, `valkey`, `qdrant`, `clickhouse`, `kafka`, `rabbitmq`, `etcd`, `openbao`, `harbor`, `foundationdb`, `external-dns`, `kuberture`, `vminstance`, `gateway`, `opensearch`, `monitoring`, `kubernetes-latest`, `kubernetes-previous`, `securitygroup`, `seaweedfs`.
21+
Suites: `postgres`, `bucket`, `mariadb`, `mongodb`, `redis`, `valkey`, `qdrant`, `clickhouse`, `kafka`, `kafka-metadata`, `rabbitmq`, `etcd`, `openbao`, `harbor`, `foundationdb`, `external-dns`, `kuberture`, `vminstance`, `gateway`, `opensearch`, `monitoring`, `kubernetes-latest`, `kubernetes-previous`, `securitygroup`, `seaweedfs`, `computeplane`.
2222

2323
## What Chainsaw buys over the BATS suite
2424

@@ -55,7 +55,7 @@ Declarative suites assert on `status.conditions` and concrete fields. Inherently
5555

5656
## Running locally
5757

58-
Requires a cluster with Cozystack installed and a `tenant-test` namespace (the environment `hack/e2e-install-cozystack.bats` produces). `bucket` and `seaweedfs` additionally need `mc`, `nc`, and `jq` on the host; `kuberture`/`openbao`/`harbor`/`vminstance` need `jq`.
58+
Requires a cluster with Cozystack installed and a `tenant-test` namespace (the environment `hack/e2e-install-cozystack.bats` produces). `bucket` and `seaweedfs` additionally need `mc`, `nc`, and `jq` on the host; `kuberture`/`openbao`/`foundationdb`/`securitygroup`/`computeplane`/`kubernetes-latest`/`kubernetes-previous`/`vminstance` need `jq`.
5959

6060
```bash
6161
# install chainsaw

0 commit comments

Comments
 (0)