|
| 1 | +# Tenant kube-apiserver OIDC authentication |
| 2 | + |
| 3 | +This document covers the operator workflow for enabling OIDC |
| 4 | +authentication on tenant Kubernetes clusters managed by Cozystack, and |
| 5 | +the limitations of the current implementation. |
| 6 | + |
| 7 | +## Overview |
| 8 | + |
| 9 | +Each Cozystack tenant can have its own Keycloak realm provisioned by the |
| 10 | +`apps/tenant` chart, dedicated to the tenant's kube-apiservers and |
| 11 | +tenant-scoped applications. The platform `cozy` realm stays dedicated to |
| 12 | +the management cluster (dashboard, kubeapps, mgmt kubectl) and is not |
| 13 | +touched by this feature. |
| 14 | + |
| 15 | +When the `Kubernetes` CR opts into OIDC (`spec.oidc.enabled: true`): |
| 16 | + |
| 17 | +1. The `apps/tenant` chart sees the request via a `lookup` on the |
| 18 | + tenant's namespace, provisions `ClusterKeycloakRealm <tenant>` and a |
| 19 | + `groups` `KeycloakClientScope`, and publishes `_namespace.oidc-realm` |
| 20 | + to the tenant's `cozystack-values` Secret. |
| 21 | +2. The `apps/kubernetes` chart picks up `_namespace.oidc-realm`, |
| 22 | + creates a per-cluster public `KeycloakClient kubernetes-<cluster>` |
| 23 | + with an audience scope, creates `KeycloakRealmGroup <cluster>`, |
| 24 | + wires the tenant kube-apiserver to the realm via |
| 25 | + `KamajiControlPlane.spec.apiServer.extraArgs`, and runs a |
| 26 | + post-install/post-upgrade `Job` that materialises a |
| 27 | + `ClusterRoleBinding` inside the tenant cluster binding the realm |
| 28 | + group to the built-in `cluster-admin` ClusterRole. |
| 29 | +3. Operators grant or revoke access by adding or removing users from |
| 30 | + the realm group in Keycloak. |
| 31 | + |
| 32 | +## Prerequisites |
| 33 | + |
| 34 | +- Platform-level OIDC must be enabled |
| 35 | + (`authentication.oidc.enabled: true` in the platform values → |
| 36 | + `_cluster.oidc-enabled=true`). The chart fails render with an |
| 37 | + actionable message if this is missing. |
| 38 | +- A publicly resolvable platform DNS name with a valid TLS certificate |
| 39 | + on the Keycloak ingress (so the tenant apiserver can validate the |
| 40 | + OIDC issuer over HTTPS). Self-signed Keycloak deployments are not |
| 41 | + supported in this iteration — see Limitations below. |
| 42 | + |
| 43 | +## Enable OIDC on a tenant cluster |
| 44 | + |
| 45 | +```yaml |
| 46 | +apiVersion: apps.cozystack.io/v1alpha1 |
| 47 | +kind: Kubernetes |
| 48 | +metadata: |
| 49 | + name: prod-a |
| 50 | + namespace: tenant-acme |
| 51 | +spec: |
| 52 | + ... |
| 53 | + oidc: |
| 54 | + enabled: true |
| 55 | +``` |
| 56 | +
|
| 57 | +On the next `apps/tenant` reconcile (≤ 5 min) the realm `tenant-acme` |
| 58 | +appears. On the next `apps/kubernetes` reconcile the kube-apiserver |
| 59 | +gets OIDC flags and the in-cluster ClusterRoleBinding is created. |
| 60 | + |
| 61 | +Operators do **not** need to pre-toggle `Tenant.spec.oidc.enabled` — |
| 62 | +the parent chart auto-provisions the realm when any child Kubernetes CR |
| 63 | +requests OIDC. `Tenant.spec.oidc.enabled=true` is only useful when |
| 64 | +tenant-scoped applications (Grafana, etc.) need the realm before the |
| 65 | +first Kubernetes CR exists. |
| 66 | + |
| 67 | +## Create a user and grant access |
| 68 | + |
| 69 | +In Keycloak (the tenant realm — e.g. `tenant-acme`): |
| 70 | + |
| 71 | +1. Create a user, set a non-temporary password, mark email verified. |
| 72 | +2. Add the user to the realm group named after the cluster (e.g. |
| 73 | + `prod-a`). One membership = full kubectl access to that cluster. |
| 74 | + |
| 75 | +To revoke access, remove the user from the group. |
| 76 | + |
| 77 | +## Wire kubectl with kubelogin |
| 78 | + |
| 79 | +The end-user installs `kubectl oidc-login`: |
| 80 | + |
| 81 | +```bash |
| 82 | +brew install int128/kubelogin/kubelogin |
| 83 | +# or: kubectl krew install oidc-login |
| 84 | +``` |
| 85 | + |
| 86 | +Then writes a kubeconfig pointing at the tenant apiserver and using |
| 87 | +the OIDC issuer: |
| 88 | + |
| 89 | +```yaml |
| 90 | +apiVersion: v1 |
| 91 | +kind: Config |
| 92 | +clusters: |
| 93 | +- name: prod-a |
| 94 | + cluster: |
| 95 | + server: https://prod-a.acme.example.com:443 |
| 96 | + certificate-authority-data: <base64 cluster CA, copy from |
| 97 | + admin kubeconfig Secret> |
| 98 | +contexts: |
| 99 | +- name: prod-a |
| 100 | + context: |
| 101 | + cluster: prod-a |
| 102 | + user: oidc |
| 103 | +current-context: prod-a |
| 104 | +users: |
| 105 | +- name: oidc |
| 106 | + user: |
| 107 | + exec: |
| 108 | + apiVersion: client.authentication.k8s.io/v1 |
| 109 | + command: kubectl |
| 110 | + args: |
| 111 | + - oidc-login |
| 112 | + - get-token |
| 113 | + - --oidc-issuer-url=https://keycloak.acme.example.com/realms/tenant-acme |
| 114 | + - --oidc-client-id=kubernetes-prod-a |
| 115 | +``` |
| 116 | + |
| 117 | +A ready-to-paste version of this snippet (with the live cluster CA and |
| 118 | +issuer URL filled in) is printed in the chart `NOTES.txt`: |
| 119 | + |
| 120 | +```bash |
| 121 | +helm get notes -n tenant-acme prod-a |
| 122 | +``` |
| 123 | + |
| 124 | +Running `kubectl get pods` opens the browser, logs the user into |
| 125 | +Keycloak, returns the id_token, and the apiserver authenticates the |
| 126 | +request based on the `groups` claim. |
| 127 | + |
| 128 | +## Limitations |
| 129 | + |
| 130 | +### Realm cleanup is not automatic after the last child OIDC cluster is removed |
| 131 | + |
| 132 | +The `apps/tenant` chart uses Helm's `lookup` function to discover |
| 133 | +whether any child `Kubernetes` CR has `spec.oidc.enabled=true`. |
| 134 | +Helm-controller does **not** re-render a chart when a `lookup` result |
| 135 | +changes — it only re-renders when the chart source artifact or the |
| 136 | +HelmRelease values change. Consequently, deleting the last |
| 137 | +`Kubernetes` CR with OIDC enabled does **not** trigger an `apps/tenant` |
| 138 | +re-render, and the orphan `ClusterKeycloakRealm` stays in the tenant |
| 139 | +namespace. |
| 140 | + |
| 141 | +To force cleanup, the operator can: |
| 142 | + |
| 143 | +- Explicitly toggle `Tenant.spec.oidc.enabled=true` and then back to |
| 144 | + `false`. Each toggle changes the HelmRelease values, which triggers |
| 145 | + a re-render with the up-to-date lookup result. After the second |
| 146 | + toggle, the chart no longer renders the realm and Helm prunes it. |
| 147 | +- Or wait for the next platform upgrade that bumps any chart-affecting |
| 148 | + source — the realm cleanup happens for free as a side effect. |
| 149 | + |
| 150 | +### Self-signed Keycloak is not supported |
| 151 | + |
| 152 | +The tenant apiserver validates the OIDC issuer over HTTPS using the |
| 153 | +system trust store inside the Kamaji apiserver pod (typically the |
| 154 | +upstream Kubernetes apiserver base image). If the platform Keycloak |
| 155 | +ingress uses a private CA, the apiserver fails the TLS handshake and |
| 156 | +all OIDC logins return 401. The chart does not currently expose a |
| 157 | +`caBundle` field — public DNS with a valid certificate (e.g. via |
| 158 | +cert-manager + Let's Encrypt) is required. |
| 159 | + |
| 160 | +### JWT claims are not configurable |
| 161 | + |
| 162 | +`--oidc-username-claim` is fixed to `preferred_username` and |
| 163 | +`--oidc-groups-claim` is fixed to `groups`. These match the Keycloak |
| 164 | +defaults; deployments using non-default claim mappings would need a |
| 165 | +chart change. |
| 166 | + |
| 167 | +### Runtime toggle of `oidc.enabled` from `true` to `false` |
| 168 | + |
| 169 | +Helm hooks only fire on install / upgrade / delete, not on values |
| 170 | +changes. If an operator flips `Kubernetes.spec.oidc.enabled` from |
| 171 | +`true` to `false`, the chart stops rendering the in-cluster |
| 172 | +`ClusterRoleBinding` Job but the existing binding inside the tenant |
| 173 | +cluster is not removed. The apiserver also drops the OIDC arguments on |
| 174 | +the next reconcile, so the orphan binding is inert (no realm group |
| 175 | +matches against the now-disabled OIDC path). Manual cleanup: |
| 176 | + |
| 177 | +```bash |
| 178 | +kubectl --kubeconfig=<admin-kubeconfig> delete clusterrolebinding \ |
| 179 | + --selector cozystack.io/oidc-cluster=<cluster> |
| 180 | +``` |
| 181 | + |
| 182 | +### CI / headless access requires manual KeycloakClient patch |
| 183 | + |
| 184 | +The chart-rendered `KeycloakClient` is public and does **not** enable |
| 185 | +`directAccessGrantsEnabled` (password grant). This is correct for |
| 186 | +browser-flow logins. For CI pipelines that need a non-interactive |
| 187 | +token, the cluster-admin can patch the client on the live cluster: |
| 188 | + |
| 189 | +```bash |
| 190 | +kubectl -n <tenant-namespace> patch keycloakclient kubernetes-<cluster> \ |
| 191 | + --type=merge --patch '{"spec":{"directAccess":true}}' |
| 192 | +``` |
| 193 | + |
| 194 | +This is intentionally not the default — interactive flow stays |
| 195 | +recommended for human users. |
| 196 | + |
| 197 | +## Troubleshooting |
| 198 | + |
| 199 | +### Apiserver returns 401 with valid token |
| 200 | + |
| 201 | +- Check the apiserver flags in the Kamaji pod: |
| 202 | + |
| 203 | + ```bash |
| 204 | + kubectl --context=mgmt -n <tenant-ns> get pod \ |
| 205 | + -l kamaji.clastix.io/name=<cluster> \ |
| 206 | + -o jsonpath='{.items[0].spec.containers[?(@.name=="kube-apiserver")].args}' | \ |
| 207 | + tr ',' '\n' | grep oidc |
| 208 | + ``` |
| 209 | + |
| 210 | +- Confirm the issuer URL matches the realm — decode the id_token and |
| 211 | + compare `iss` against the `--oidc-issuer-url` flag. |
| 212 | +- Confirm `aud` in the token equals `--oidc-client-id`. Mismatch is |
| 213 | + the most common cause when running multiple clusters in the same |
| 214 | + realm. |
| 215 | + |
| 216 | +### Apiserver returns 403 for a user that is in the right group |
| 217 | + |
| 218 | +- Check the in-cluster `ClusterRoleBinding`: |
| 219 | + |
| 220 | + ```bash |
| 221 | + kubectl --kubeconfig=<admin-kubeconfig> get clusterrolebinding \ |
| 222 | + --selector cozystack.io/oidc-cluster=<cluster> |
| 223 | + ``` |
| 224 | + |
| 225 | +- The bootstrap Job runs as a `post-install` / `post-upgrade` hook; |
| 226 | + check its logs in the management cluster: |
| 227 | + |
| 228 | + ```bash |
| 229 | + kubectl --context=mgmt -n <tenant-ns> logs \ |
| 230 | + job/<release-name>-oidc-rbac |
| 231 | + ``` |
| 232 | + |
| 233 | +### Realm or scope objects stuck after CR deletion |
| 234 | + |
| 235 | +See "Realm cleanup is not automatic" under Limitations. Operator |
| 236 | +intervention required (toggle `Tenant.spec.oidc`). |
0 commit comments