Skip to content

Commit 6938717

Browse files
committed
fix(api): regenerate deepcopy for OIDC types + docs + drop unused label
Three follow-ups surfaced by the upstream CI verify-codegen check and a fresh adversarial review: * api/apps/v1alpha1/{kubernetes,tenant}/zz_generated.deepcopy.go — regenerated via hack/update-codegen.sh. cozyvalues-gen produces types.go but does not touch controller-gen deepcopy output; the new `OIDC` struct field on `ConfigSpec` therefore needed manual codegen. CI's "Verify generated code is up to date" job was failing on this drift. * docs/oidc-tenant.md — new operator-facing document covering enable flow, kubelogin setup, and the four known limitations (orphan realm on last-child-removed without a values bump because helm-controller does not re-render on lookup changes; no caBundle / self-signed Keycloak support; hardcoded JWT username/groups claims; CRB orphan on runtime oidc.enabled=true->false toggle since Helm hooks only fire on install/upgrade/delete). * apps/tenant/templates/namespace.yaml — drop the namespace.cozystack.io/ oidc-realm label. It was dead code: no NetworkPolicy or other selector consumes it. The corresponding `_namespace.oidc-realm` field in the cozystack-values Secret stays — that one is actually consumed by apps/kubernetes to derive the OIDC issuer URL. Signed-off-by: IvanHunters <[email protected]>
1 parent 4d80b91 commit 6938717

4 files changed

Lines changed: 268 additions & 1 deletion

File tree

‎api/apps/v1alpha1/kubernetes/zz_generated.deepcopy.go‎

Lines changed: 16 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎api/apps/v1alpha1/tenant/zz_generated.deepcopy.go‎

Lines changed: 16 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎docs/oidc-tenant.md‎

Lines changed: 236 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,236 @@
1+
# Tenant kube-apiserver OIDC authentication
2+
3+
This document covers the operator workflow for enabling OIDC
4+
authentication on tenant Kubernetes clusters managed by Cozystack, and
5+
the limitations of the current implementation.
6+
7+
## Overview
8+
9+
Each Cozystack tenant can have its own Keycloak realm provisioned by the
10+
`apps/tenant` chart, dedicated to the tenant's kube-apiservers and
11+
tenant-scoped applications. The platform `cozy` realm stays dedicated to
12+
the management cluster (dashboard, kubeapps, mgmt kubectl) and is not
13+
touched by this feature.
14+
15+
When the `Kubernetes` CR opts into OIDC (`spec.oidc.enabled: true`):
16+
17+
1. The `apps/tenant` chart sees the request via a `lookup` on the
18+
tenant's namespace, provisions `ClusterKeycloakRealm <tenant>` and a
19+
`groups` `KeycloakClientScope`, and publishes `_namespace.oidc-realm`
20+
to the tenant's `cozystack-values` Secret.
21+
2. The `apps/kubernetes` chart picks up `_namespace.oidc-realm`,
22+
creates a per-cluster public `KeycloakClient kubernetes-<cluster>`
23+
with an audience scope, creates `KeycloakRealmGroup <cluster>`,
24+
wires the tenant kube-apiserver to the realm via
25+
`KamajiControlPlane.spec.apiServer.extraArgs`, and runs a
26+
post-install/post-upgrade `Job` that materialises a
27+
`ClusterRoleBinding` inside the tenant cluster binding the realm
28+
group to the built-in `cluster-admin` ClusterRole.
29+
3. Operators grant or revoke access by adding or removing users from
30+
the realm group in Keycloak.
31+
32+
## Prerequisites
33+
34+
- Platform-level OIDC must be enabled
35+
(`authentication.oidc.enabled: true` in the platform values →
36+
`_cluster.oidc-enabled=true`). The chart fails render with an
37+
actionable message if this is missing.
38+
- A publicly resolvable platform DNS name with a valid TLS certificate
39+
on the Keycloak ingress (so the tenant apiserver can validate the
40+
OIDC issuer over HTTPS). Self-signed Keycloak deployments are not
41+
supported in this iteration — see Limitations below.
42+
43+
## Enable OIDC on a tenant cluster
44+
45+
```yaml
46+
apiVersion: apps.cozystack.io/v1alpha1
47+
kind: Kubernetes
48+
metadata:
49+
name: prod-a
50+
namespace: tenant-acme
51+
spec:
52+
...
53+
oidc:
54+
enabled: true
55+
```
56+
57+
On the next `apps/tenant` reconcile (≤ 5 min) the realm `tenant-acme`
58+
appears. On the next `apps/kubernetes` reconcile the kube-apiserver
59+
gets OIDC flags and the in-cluster ClusterRoleBinding is created.
60+
61+
Operators do **not** need to pre-toggle `Tenant.spec.oidc.enabled` —
62+
the parent chart auto-provisions the realm when any child Kubernetes CR
63+
requests OIDC. `Tenant.spec.oidc.enabled=true` is only useful when
64+
tenant-scoped applications (Grafana, etc.) need the realm before the
65+
first Kubernetes CR exists.
66+
67+
## Create a user and grant access
68+
69+
In Keycloak (the tenant realm — e.g. `tenant-acme`):
70+
71+
1. Create a user, set a non-temporary password, mark email verified.
72+
2. Add the user to the realm group named after the cluster (e.g.
73+
`prod-a`). One membership = full kubectl access to that cluster.
74+
75+
To revoke access, remove the user from the group.
76+
77+
## Wire kubectl with kubelogin
78+
79+
The end-user installs `kubectl oidc-login`:
80+
81+
```bash
82+
brew install int128/kubelogin/kubelogin
83+
# or: kubectl krew install oidc-login
84+
```
85+
86+
Then writes a kubeconfig pointing at the tenant apiserver and using
87+
the OIDC issuer:
88+
89+
```yaml
90+
apiVersion: v1
91+
kind: Config
92+
clusters:
93+
- name: prod-a
94+
cluster:
95+
server: https://prod-a.acme.example.com:443
96+
certificate-authority-data: <base64 cluster CA, copy from
97+
admin kubeconfig Secret>
98+
contexts:
99+
- name: prod-a
100+
context:
101+
cluster: prod-a
102+
user: oidc
103+
current-context: prod-a
104+
users:
105+
- name: oidc
106+
user:
107+
exec:
108+
apiVersion: client.authentication.k8s.io/v1
109+
command: kubectl
110+
args:
111+
- oidc-login
112+
- get-token
113+
- --oidc-issuer-url=https://keycloak.acme.example.com/realms/tenant-acme
114+
- --oidc-client-id=kubernetes-prod-a
115+
```
116+
117+
A ready-to-paste version of this snippet (with the live cluster CA and
118+
issuer URL filled in) is printed in the chart `NOTES.txt`:
119+
120+
```bash
121+
helm get notes -n tenant-acme prod-a
122+
```
123+
124+
Running `kubectl get pods` opens the browser, logs the user into
125+
Keycloak, returns the id_token, and the apiserver authenticates the
126+
request based on the `groups` claim.
127+
128+
## Limitations
129+
130+
### Realm cleanup is not automatic after the last child OIDC cluster is removed
131+
132+
The `apps/tenant` chart uses Helm's `lookup` function to discover
133+
whether any child `Kubernetes` CR has `spec.oidc.enabled=true`.
134+
Helm-controller does **not** re-render a chart when a `lookup` result
135+
changes — it only re-renders when the chart source artifact or the
136+
HelmRelease values change. Consequently, deleting the last
137+
`Kubernetes` CR with OIDC enabled does **not** trigger an `apps/tenant`
138+
re-render, and the orphan `ClusterKeycloakRealm` stays in the tenant
139+
namespace.
140+
141+
To force cleanup, the operator can:
142+
143+
- Explicitly toggle `Tenant.spec.oidc.enabled=true` and then back to
144+
`false`. Each toggle changes the HelmRelease values, which triggers
145+
a re-render with the up-to-date lookup result. After the second
146+
toggle, the chart no longer renders the realm and Helm prunes it.
147+
- Or wait for the next platform upgrade that bumps any chart-affecting
148+
source — the realm cleanup happens for free as a side effect.
149+
150+
### Self-signed Keycloak is not supported
151+
152+
The tenant apiserver validates the OIDC issuer over HTTPS using the
153+
system trust store inside the Kamaji apiserver pod (typically the
154+
upstream Kubernetes apiserver base image). If the platform Keycloak
155+
ingress uses a private CA, the apiserver fails the TLS handshake and
156+
all OIDC logins return 401. The chart does not currently expose a
157+
`caBundle` field — public DNS with a valid certificate (e.g. via
158+
cert-manager + Let's Encrypt) is required.
159+
160+
### JWT claims are not configurable
161+
162+
`--oidc-username-claim` is fixed to `preferred_username` and
163+
`--oidc-groups-claim` is fixed to `groups`. These match the Keycloak
164+
defaults; deployments using non-default claim mappings would need a
165+
chart change.
166+
167+
### Runtime toggle of `oidc.enabled` from `true` to `false`
168+
169+
Helm hooks only fire on install / upgrade / delete, not on values
170+
changes. If an operator flips `Kubernetes.spec.oidc.enabled` from
171+
`true` to `false`, the chart stops rendering the in-cluster
172+
`ClusterRoleBinding` Job but the existing binding inside the tenant
173+
cluster is not removed. The apiserver also drops the OIDC arguments on
174+
the next reconcile, so the orphan binding is inert (no realm group
175+
matches against the now-disabled OIDC path). Manual cleanup:
176+
177+
```bash
178+
kubectl --kubeconfig=<admin-kubeconfig> delete clusterrolebinding \
179+
--selector cozystack.io/oidc-cluster=<cluster>
180+
```
181+
182+
### CI / headless access requires manual KeycloakClient patch
183+
184+
The chart-rendered `KeycloakClient` is public and does **not** enable
185+
`directAccessGrantsEnabled` (password grant). This is correct for
186+
browser-flow logins. For CI pipelines that need a non-interactive
187+
token, the cluster-admin can patch the client on the live cluster:
188+
189+
```bash
190+
kubectl -n <tenant-namespace> patch keycloakclient kubernetes-<cluster> \
191+
--type=merge --patch '{"spec":{"directAccess":true}}'
192+
```
193+
194+
This is intentionally not the default — interactive flow stays
195+
recommended for human users.
196+
197+
## Troubleshooting
198+
199+
### Apiserver returns 401 with valid token
200+
201+
- Check the apiserver flags in the Kamaji pod:
202+
203+
```bash
204+
kubectl --context=mgmt -n <tenant-ns> get pod \
205+
-l kamaji.clastix.io/name=<cluster> \
206+
-o jsonpath='{.items[0].spec.containers[?(@.name=="kube-apiserver")].args}' | \
207+
tr ',' '\n' | grep oidc
208+
```
209+
210+
- Confirm the issuer URL matches the realm — decode the id_token and
211+
compare `iss` against the `--oidc-issuer-url` flag.
212+
- Confirm `aud` in the token equals `--oidc-client-id`. Mismatch is
213+
the most common cause when running multiple clusters in the same
214+
realm.
215+
216+
### Apiserver returns 403 for a user that is in the right group
217+
218+
- Check the in-cluster `ClusterRoleBinding`:
219+
220+
```bash
221+
kubectl --kubeconfig=<admin-kubeconfig> get clusterrolebinding \
222+
--selector cozystack.io/oidc-cluster=<cluster>
223+
```
224+
225+
- The bootstrap Job runs as a `post-install` / `post-upgrade` hook;
226+
check its logs in the management cluster:
227+
228+
```bash
229+
kubectl --context=mgmt -n <tenant-ns> logs \
230+
job/<release-name>-oidc-rbac
231+
```
232+
233+
### Realm or scope objects stuck after CR deletion
234+
235+
See "Realm cleanup is not automatic" under Limitations. Operator
236+
intervention required (toggle `Tenant.spec.oidc`).

‎packages/apps/tenant/templates/namespace.yaml‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -119,7 +119,6 @@ metadata:
119119
namespace.cozystack.io/gateway: {{ $gateway | quote }}
120120
namespace.cozystack.io/monitoring: {{ $monitoring | quote }}
121121
namespace.cozystack.io/seaweedfs: {{ $seaweedfs | quote }}
122-
namespace.cozystack.io/oidc-realm: {{ $oidcRealm | quote }}
123122
namespace.cozystack.io/host: {{ $computedHost | quote }}
124123
{{- with $schedulingClass }}
125124
scheduler.cozystack.io/scheduling-class: {{ . | quote }}

0 commit comments

Comments
 (0)