Skip to content

Commit d34014a

Browse files
IvanHuntersIvanHunters
committed
test(e2e): add OIDC bats coverage for tenant and kubernetes apps
Two new bats specs in hack/e2e-apps/ exercise the per-tenant Keycloak realm + tenant kube-apiserver OIDC flow end-to-end against the e2e sandbox cluster. The install-cozystack step already flips authentication.oidc.enabled=true and waits for the cozy Keycloak stack to be ready, so no prerequisite plumbing changes were needed beyond the bats specs themselves. hack/e2e-apps/tenant-oidc.bats - 4 cases: * Preflight: ClusterKeycloak keycloak-cozy + ClusterKeycloakRealm keycloakrealm-cozy are ready (assertions on the cluster-scoped v1.edp.epam.com CRDs). * Patch tenant-root/test with spec.oidc.enabled=true, wait HR Ready, assert ClusterKeycloakRealm tenant-test and KeycloakClientScope tenant-test-groups reconcile to status.value=OK. * Assert _namespace.oidc-realm="tenant-test" is published into the tenant-test/cozystack-values Secret (the field downstream apps consume to derive the OIDC issuer URL). * Exercises the documented orphan-realm cleanup workaround: flip spec.oidc.enabled back to false, assert the realm + scope are pruned and _namespace.oidc-realm reverts to "". hack/e2e-apps/kubernetes-oidc.bats - 6 cases: * Force-provisions the per-tenant realm via Tenant.spec.oidc=true so the test does not race the helm-controller 5m `lookup` reconcile loop that otherwise reacts to the child Kubernetes CR. * Creates a control-plane-only Kubernetes CR (replicas=1, no node groups, no addons) with spec.oidc.enabled=true - workers are not required to drive the OIDC provisioning path, the post-install Job applies the CRB through the Kamaji admin kubeconfig. * Asserts KamajiControlPlane.spec.apiServer.extraArgs includes all four OIDC flags (--oidc-issuer-url, --oidc-client-id, --oidc-username-claim, --oidc-groups-claim). * Waits for the per-cluster KeycloakClient kubernetes-oidc-test and KeycloakRealmGroup oidc-test to reach status.value=OK. * Waits for HR Ready (Flux marks Ready only after post-install hooks return; polling on the Job directly races with its hook-delete-policy=hook-succeeded cleanup), then extracts the Kamaji admin kubeconfig + port-forward and asserts the in-cluster CRB oidc-oidc-test-cluster-admin has the right label, subject (Group/oidc-test), and roleRef (ClusterRole/cluster-admin). * Deletes the Kubernetes CR and polls that the CRB stops being visible through the tenant kubeconfig - covers both the cleanup- Job-removed-it path and the broader-teardown-ate-it path. Each spec is self-contained: kubernetes-oidc.bats does not depend on tenant-oidc.bats having run first (bats does not guarantee ordering across files), and both leave Tenant.spec.oidc.enabled=false on exit so unrelated bats running afterwards observe a clean slate. Port 59992 is offset from the kubernetes-latest port pool (59991) so the two never collide on a parallel-run sandbox. E2E validated end-to-end against the dev3 cluster: * tenant-oidc.bats: 4/4 cases pass against the existing tenant-root/test Tenant. * kubernetes-oidc.bats: 6/6 cases pass; the full create + verify + delete + cleanup cycle completes in ~10 minutes against a fresh Kubernetes CR. Signed-off-by: IvanHunters <[email protected]>
1 parent 6938717 commit d34014a

2 files changed

Lines changed: 298 additions & 0 deletions

File tree

‎hack/e2e-apps/kubernetes-oidc.bats‎

Lines changed: 212 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,212 @@
1+
#!/usr/bin/env bats
2+
3+
# Per-cluster OIDC provisioning on a tenant Kubernetes CR. Creates a
4+
# control-plane-only Kubernetes CR with spec.oidc.enabled=true and
5+
# asserts: Kamaji apiserver extraArgs include the OIDC flags, the
6+
# per-cluster KeycloakClient + KeycloakRealmGroup reconcile, the
7+
# post-install Job applies the cluster-admin ClusterRoleBinding inside
8+
# the tenant cluster, and the pre-delete Job removes it on uninstall.
9+
#
10+
# Self-contained: force-toggles Tenant.spec.oidc.enabled=true so we
11+
# don't wait the 5m Helm `lookup` loop for the per-tenant realm; bats
12+
# tests don't guarantee ordering, so this test must NOT depend on
13+
# tenant-oidc.bats having run first.
14+
15+
# Shared across all @test blocks in this file. Bats evaluates file-level
16+
# statements before any @test runs, so these assignments are visible
17+
# inside the test bodies. Match the kubernetes-latest port pool (59991)
18+
# with an offset so parallel runs don't collide.
19+
CLUSTER_NAME=oidc-test
20+
PORT=59992
21+
RELEASE_NAME=kubernetes-${CLUSTER_NAME}
22+
TENANT_NS=tenant-test
23+
KUBECONFIG_FILE=tenantkubeconfig-${CLUSTER_NAME}
24+
25+
@test "Kubernetes OIDC: force-provision per-tenant realm" {
26+
# Patch tenant-test to oidc.enabled=true so the realm is provisioned
27+
# immediately on the next tenant HR reconcile (without waiting for
28+
# the helm-controller `lookup` loop to pick up the Kubernetes CR).
29+
kubectl patch tenants/test -n tenant-root --type merge \
30+
--patch '{"spec":{"oidc":{"enabled":true}}}'
31+
kubectl wait hr/tenant-test -n tenant-root \
32+
--timeout=5m --for=condition=ready
33+
34+
timeout 60 sh -ec 'until kubectl get clusterkeycloakrealm/tenant-test >/dev/null 2>&1; do sleep 2; done'
35+
kubectl wait clusterkeycloakrealm/tenant-test \
36+
--timeout=3m --for=jsonpath='{.status.value}'=OK
37+
}
38+
39+
@test "Kubernetes OIDC: create Kubernetes CR with oidc.enabled=true" {
40+
# Clean up stale resources from a previous failed retry.
41+
kubectl -n ${TENANT_NS} delete kuberneteses.apps.cozystack.io "${CLUSTER_NAME}" \
42+
--ignore-not-found --wait=false 2>/dev/null || true
43+
kubectl -n ${TENANT_NS} wait kuberneteses.apps.cozystack.io "${CLUSTER_NAME}" \
44+
--for=delete --timeout=2m 2>/dev/null || true
45+
46+
local k8s_version
47+
k8s_version=$(yq 'keys | sort_by(.) | .[-1]' \
48+
packages/apps/kubernetes/files/versions.yaml)
49+
50+
# Control-plane only, minimal addons. Workers are not required to
51+
# exercise the OIDC provisioning path - the post-install Job applies
52+
# the CRB through the Kamaji admin kubeconfig, which becomes available
53+
# once the apiserver reaches Ready (no workers needed).
54+
kubectl apply -f - <<EOF
55+
apiVersion: apps.cozystack.io/v1alpha1
56+
kind: Kubernetes
57+
metadata:
58+
name: "${CLUSTER_NAME}"
59+
namespace: ${TENANT_NS}
60+
spec:
61+
oidc:
62+
enabled: true
63+
addons:
64+
certManager: {enabled: false, valuesOverride: {}}
65+
cilium: {valuesOverride: {}}
66+
fluxcd: {enabled: false, valuesOverride: {}}
67+
gatewayAPI: {enabled: false}
68+
gpuOperator: {enabled: false, valuesOverride: {}}
69+
ingressNginx: {enabled: false, hosts: [], valuesOverride: {}}
70+
monitoringAgents: {enabled: false, valuesOverride: {}}
71+
verticalPodAutoscaler: {valuesOverride: {}}
72+
controlPlane:
73+
apiServer: {resources: {}, resourcesPreset: small}
74+
controllerManager: {resources: {}, resourcesPreset: micro}
75+
konnectivity:
76+
server: {resources: {}, resourcesPreset: micro}
77+
replicas: 1
78+
scheduler: {resources: {}, resourcesPreset: micro}
79+
host: ""
80+
nodeGroups: {}
81+
storageClass: replicated
82+
version: "${k8s_version}"
83+
EOF
84+
85+
# Wait for the KamajiControlPlane CR to appear and the
86+
# TenantControlPlaneCreated condition.
87+
timeout 2m sh -ec 'until kubectl get kamajicontrolplane -n '"${TENANT_NS}"' '"${RELEASE_NAME}"'; do sleep 2; done'
88+
kubectl wait --for=condition=TenantControlPlaneCreated \
89+
kamajicontrolplane -n ${TENANT_NS} ${RELEASE_NAME} --timeout=5m
90+
kubectl wait tcp -n ${TENANT_NS} ${RELEASE_NAME} --timeout=10m \
91+
--for=jsonpath='{.status.kubernetesResources.version.status}'=Ready
92+
}
93+
94+
@test "Kubernetes OIDC: apiserver extraArgs include OIDC flags" {
95+
# The kubernetes chart injects --oidc-issuer-url, --oidc-client-id,
96+
# --oidc-username-claim, --oidc-groups-claim into
97+
# KCP.spec.apiServer.extraArgs (a list of strings). Verify all four.
98+
args=$(kubectl get kamajicontrolplane -n ${TENANT_NS} ${RELEASE_NAME} \
99+
-o jsonpath='{range .spec.apiServer.extraArgs[*]}{@}{"\n"}{end}')
100+
echo "apiServer extraArgs:"; echo "${args}"
101+
echo "${args}" | grep -qx -- '--oidc-issuer-url=https://keycloak.example.org/realms/tenant-test'
102+
echo "${args}" | grep -qx -- "--oidc-client-id=${RELEASE_NAME}"
103+
echo "${args}" | grep -qx -- '--oidc-username-claim=preferred_username'
104+
echo "${args}" | grep -qx -- '--oidc-groups-claim=groups'
105+
}
106+
107+
@test "Kubernetes OIDC: per-cluster KeycloakClient and RealmGroup ready" {
108+
timeout 2m sh -ec 'until kubectl get keycloakclient/'"${RELEASE_NAME}"' -n '"${TENANT_NS}"' >/dev/null 2>&1; do sleep 2; done'
109+
kubectl wait keycloakclient/${RELEASE_NAME} -n ${TENANT_NS} \
110+
--timeout=3m --for=jsonpath='{.status.value}'=OK
111+
112+
timeout 2m sh -ec 'until kubectl get keycloakrealmgroup/'"${CLUSTER_NAME}"' -n '"${TENANT_NS}"' >/dev/null 2>&1; do sleep 2; done'
113+
kubectl wait keycloakrealmgroup/${CLUSTER_NAME} -n ${TENANT_NS} \
114+
--timeout=3m --for=jsonpath='{.status.value}'=OK
115+
}
116+
117+
@test "Kubernetes OIDC: post-install Job applied ClusterRoleBinding inside tenant cluster" {
118+
# Wait for the HR itself to flip Ready: Flux helm-controller only marks
119+
# the HelmRelease Ready=True after the helm install action AND its
120+
# post-install hooks return successfully. The post-install Job carries
121+
# hook-delete-policy=hook-succeeded so it is garbage-collected the
122+
# moment it completes; polling on the Job directly races against that
123+
# cleanup. The CRB it creates is the durable outcome to assert against.
124+
kubectl wait hr/${RELEASE_NAME} -n ${TENANT_NS} \
125+
--timeout=15m --for=condition=ready
126+
127+
# Extract the admin kubeconfig and verify the CRB exists inside the
128+
# tenant cluster.
129+
kubectl get secret ${RELEASE_NAME}-admin-kubeconfig -n ${TENANT_NS} \
130+
-o jsonpath='{.data.super-admin\.conf}' | base64 -d > "${KUBECONFIG_FILE}"
131+
yq -i ".clusters[0].cluster.server = \"https://localhost:${PORT}\"" \
132+
"${KUBECONFIG_FILE}"
133+
134+
# Kill any stale port-forward on this port from a previous retry, then
135+
# start a fresh one to the tenant apiserver Service.
136+
pkill -f "port-forward.*${PORT}:" 2>/dev/null || true
137+
sleep 1
138+
kubectl port-forward service/${RELEASE_NAME} -n ${TENANT_NS} \
139+
"${PORT}":6443 > /dev/null 2>&1 &
140+
timeout 15 sh -ec 'until curl -sk https://localhost:'"${PORT}"' >/dev/null 2>&1; do sleep 1; done'
141+
142+
# CRB shape: name oidc-<cluster>-cluster-admin, label
143+
# cozystack.io/oidc-cluster=<cluster>, subject Group <cluster>,
144+
# roleRef cluster-admin.
145+
crb=oidc-${CLUSTER_NAME}-cluster-admin
146+
kubectl --kubeconfig "${KUBECONFIG_FILE}" get clusterrolebinding ${crb} -o yaml
147+
kubectl --kubeconfig "${KUBECONFIG_FILE}" get clusterrolebinding ${crb} \
148+
-o jsonpath='{.metadata.labels.cozystack\.io/oidc-cluster}' | grep -qx "${CLUSTER_NAME}"
149+
kubectl --kubeconfig "${KUBECONFIG_FILE}" get clusterrolebinding ${crb} \
150+
-o jsonpath='{.subjects[0].kind}/{.subjects[0].name}' | grep -qx "Group/${CLUSTER_NAME}"
151+
kubectl --kubeconfig "${KUBECONFIG_FILE}" get clusterrolebinding ${crb} \
152+
-o jsonpath='{.roleRef.kind}/{.roleRef.name}' | grep -qx 'ClusterRole/cluster-admin'
153+
154+
# Cleanup port-forward for this test; the delete-flow test starts its
155+
# own port-forward after deletion to verify the CRB is gone.
156+
pkill -f "port-forward.*${PORT}:" 2>/dev/null || true
157+
}
158+
159+
@test "Kubernetes OIDC: pre-delete Job removes ClusterRoleBinding" {
160+
# The pre-delete hook re-creates the SA/Role/RoleBinding (post-install
161+
# hooks are pruned with hook-succeeded), runs the cleanup Job which
162+
# label-selects cozystack.io/oidc-cluster=<cluster> and deletes the
163+
# CRB inside the tenant cluster. Trigger the deletion and verify.
164+
165+
# Start a fresh port-forward BEFORE deletion - once Kamaji tears down
166+
# the apiserver pods, the Service still routes, but the kubeconfig
167+
# Secret may be deleted later in the cascade. Watch the CRB
168+
# disappear via that port-forward.
169+
pkill -f "port-forward.*${PORT}:" 2>/dev/null || true
170+
sleep 1
171+
kubectl port-forward service/${RELEASE_NAME} -n ${TENANT_NS} \
172+
"${PORT}":6443 > /dev/null 2>&1 &
173+
timeout 15 sh -ec 'until curl -sk https://localhost:'"${PORT}"' >/dev/null 2>&1; do sleep 1; done'
174+
175+
# Sanity: CRB exists right before delete.
176+
crb=oidc-${CLUSTER_NAME}-cluster-admin
177+
kubectl --kubeconfig "${KUBECONFIG_FILE}" get clusterrolebinding ${crb}
178+
179+
# Delete the Kubernetes CR - Flux uninstalls the HR, which fires the
180+
# pre-delete cleanup Job (template name <release>-oidc-rbac-cleanup)
181+
# before tearing down release resources. Both this cleanup Job and
182+
# the in-tenant-cluster admin-kubeconfig Secret have transient
183+
# lifetimes: the Job carries hook-delete-policy=hook-succeeded so it
184+
# is garbage-collected the moment it completes, and Kamaji deletes
185+
# the admin kubeconfig Secret shortly after the Kubernetes CR is
186+
# removed. Both paths converge on the same observable outcome - the
187+
# ClusterRoleBinding stops being visible from the tenant kubeconfig -
188+
# so poll on that signal rather than on either intermediate object.
189+
kubectl -n ${TENANT_NS} delete kuberneteses.apps.cozystack.io "${CLUSTER_NAME}" --wait=false
190+
191+
# Poll: CRB must vanish, or the apiserver/Secret/Service stops being
192+
# reachable (the broader teardown). Both are valid happy outcomes;
193+
# only "CRB still visible with the right label after 3m" is a failure.
194+
timeout 3m sh -ec '
195+
crb=oidc-'"${CLUSTER_NAME}"'-cluster-admin
196+
until ! kubectl --kubeconfig "'"${KUBECONFIG_FILE}"'" get clusterrolebinding ${crb} 2>&1 \
197+
| grep -q "cozystack.io/oidc-cluster='"${CLUSTER_NAME}"'\|^${crb}\s"; do
198+
sleep 5
199+
done
200+
'
201+
202+
# Final tenant CR wait.
203+
kubectl -n ${TENANT_NS} wait kuberneteses.apps.cozystack.io "${CLUSTER_NAME}" \
204+
--for=delete --timeout=5m 2>/dev/null || true
205+
206+
# Cleanup port-forward + temp files. Flip Tenant oidc back to false so
207+
# other bats files run against a clean slate.
208+
pkill -f "port-forward.*${PORT}:" 2>/dev/null || true
209+
rm -f "${KUBECONFIG_FILE}" /tmp/oidc-cleanup.log
210+
kubectl patch tenants/test -n tenant-root --type merge \
211+
--patch '{"spec":{"oidc":{"enabled":false}}}' || true
212+
}

‎hack/e2e-apps/tenant-oidc.bats‎

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
#!/usr/bin/env bats
2+
3+
# Tenant-side OIDC provisioning. Patches the pre-existing `test` Tenant
4+
# (created by e2e-install-cozystack.bats) with spec.oidc.enabled=true and
5+
# asserts the per-tenant Keycloak realm + groups client scope reconcile
6+
# and the _namespace.oidc-realm value is published to the tenant's
7+
# cozystack-values Secret. Also exercises the documented orphan-realm
8+
# cleanup workaround (toggle false → true → false → verify pruned).
9+
10+
@test "Tenant OIDC: prerequisite - platform OIDC stack is healthy" {
11+
# The Keycloak Operator + cozy realm are gated behind
12+
# authentication.oidc.enabled on the Platform Package. Install-cozystack
13+
# already flips this on and waits for the keycloak-* HRs - assert
14+
# idempotently here so the test fails fast with a clear message if it
15+
# is run against a sandbox where the install step did not enable OIDC.
16+
# ClusterKeycloak and ClusterKeycloakRealm are cluster-scoped CRDs in
17+
# v1.edp.epam.com - no -n flag.
18+
kubectl get clusterkeycloak keycloak-cozy
19+
kubectl wait clusterkeycloakrealm/keycloakrealm-cozy \
20+
--timeout=2m --for=jsonpath='{.status.value}'=OK
21+
}
22+
23+
@test "Tenant OIDC: enabling oidc.enabled provisions per-tenant realm" {
24+
# Patch the existing tenant-root/test Tenant. Tenant-test was created in
25+
# install-cozystack.bats with oidc.enabled unset (defaults to false).
26+
kubectl patch tenants/test -n tenant-root --type merge \
27+
--patch '{"spec":{"oidc":{"enabled":true}}}'
28+
29+
# The HR rolls a fresh helm-install action; under Flux v2.8 kstatus the
30+
# parent flips Ready only after the realm + client scope objects are
31+
# reconciled by the Keycloak Operator.
32+
kubectl wait hr/tenant-test -n tenant-root \
33+
--timeout=5m --for=condition=ready
34+
35+
# ClusterKeycloakRealm <tenant-name>, cluster-scoped (no -n flag),
36+
# bound to the platform ClusterKeycloak keycloak-cozy.
37+
timeout 60 sh -ec 'until kubectl get clusterkeycloakrealm/tenant-test >/dev/null 2>&1; do sleep 2; done'
38+
kubectl wait clusterkeycloakrealm/tenant-test \
39+
--timeout=3m --for=jsonpath='{.status.value}'=OK
40+
41+
# KeycloakClientScope <tenant-namespace>-groups - the "groups" scope
42+
# with oidc-group-membership-mapper that child Kubernetes CRs request
43+
# so the apiserver gets the `groups` claim.
44+
timeout 60 sh -ec 'until kubectl get keycloakclientscope/tenant-test-groups -n tenant-test >/dev/null 2>&1; do sleep 2; done'
45+
kubectl wait keycloakclientscope/tenant-test-groups -n tenant-test \
46+
--timeout=3m --for=jsonpath='{.status.value}'=OK
47+
}
48+
49+
@test "Tenant OIDC: _namespace.oidc-realm published to cozystack-values" {
50+
# Descendants (apps/kubernetes, tenant-scoped apps) consume the realm
51+
# name from _namespace.oidc-realm in the tenant's cozystack-values
52+
# Secret. Verify the namespace.yaml render path actually populates it
53+
# with the tenant name once oidc.enabled is on.
54+
values_yaml=$(kubectl get secret cozystack-values -n tenant-test \
55+
-o jsonpath='{.data.values\.yaml}' | base64 -d)
56+
echo "${values_yaml}"
57+
echo "${values_yaml}" | grep -qE '^[[:space:]]+oidc-realm:[[:space:]]+"tenant-test"[[:space:]]*$'
58+
}
59+
60+
@test "Tenant OIDC: realm cleanup workaround prunes orphan realm" {
61+
# Documented limitation: Helm's `lookup` does not retrigger a re-render
62+
# when its result changes. The recommended workaround is to flip
63+
# Tenant.spec.oidc.enabled, which changes the HelmRelease values and
64+
# forces helm-controller to re-render, picking up the lookup result.
65+
# This test exercises the workaround on a no-child-Kubernetes state.
66+
67+
# Sanity: at this point the realm exists from the previous test.
68+
kubectl get clusterkeycloakrealm/tenant-test
69+
70+
# Flip Tenant oidc.enabled=false. The chart no longer renders the
71+
# realm, helm prunes it on the next reconcile.
72+
kubectl patch tenants/test -n tenant-root --type merge \
73+
--patch '{"spec":{"oidc":{"enabled":false}}}'
74+
kubectl wait hr/tenant-test -n tenant-root \
75+
--timeout=5m --for=condition=ready
76+
77+
# Realm (cluster-scoped) and scope (namespaced) must be gone.
78+
timeout 120 sh -ec 'until ! kubectl get clusterkeycloakrealm/tenant-test >/dev/null 2>&1; do sleep 2; done'
79+
timeout 60 sh -ec 'until ! kubectl get keycloakclientscope/tenant-test-groups -n tenant-test >/dev/null 2>&1; do sleep 2; done'
80+
81+
# And _namespace.oidc-realm reverts to "" so descendants stop seeing
82+
# the per-tenant realm.
83+
values_yaml=$(kubectl get secret cozystack-values -n tenant-test \
84+
-o jsonpath='{.data.values\.yaml}' | base64 -d)
85+
echo "${values_yaml}" | grep -qE '^[[:space:]]+oidc-realm:[[:space:]]+""[[:space:]]*$'
86+
}

0 commit comments

Comments
 (0)