Skip to content

Commit 257bbbb

Browse files
docs(kubernetes): add OIDC authentication guide (#596)
## Summary Adds a new guide under `content/en/docs/next/kubernetes/`: **OIDC authentication for kubectl**. Documents how to enable per-user OIDC authentication and RBAC on tenant Kubernetes clusters via the new `spec.oidc` selector on the `Kubernetes` resource. Covers all three modes (`None`, `System`, `CustomConfig`), how to retrieve and use the chart-generated kubeconfig, the toggle-off cleanup mechanic, prerequisites, gotchas (legacy `--oidc-*` flag collision), and what is deliberately out of scope for this feature (per-tenant realms, IdP federation into `cozy`, cross-cluster SSO). Ships alongside cozystack/cozystack#3044, the code PR that adds the `spec.oidc` selector to the `Kubernetes` chart. Full architectural rationale lives in cozystack/community#24. Only adds to `docs/next/`; version-specific pages (`v1.5`, ...) get the guide when the code PR lands in that release. ## Test plan - [x] `HUGO_ENVIRONMENT=development hugo` renders the site cleanly (1686 pages, zero warnings); page reachable at `/docs/next/kubernetes/oidc-authentication/`. - [x] `weight: 45` sits between the section index (40) and `gpu-sharing.md` (50); appears in the intended sidebar slot. - [x] YAML examples (`spec.oidc.mode: System`, `CustomConfig` inline, `CustomConfig` secretRef, `kubectl krew install oidc-login`) verified end-to-end on a dev cluster against the code PR. - [x] `docs/next/` is skipped by the production build (as designed via `config/production/hugo.yaml`); Netlify deploy-preview will show it. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Added a new guide for OIDC-based `kubectl` authentication in Cozystack-managed Kubernetes clusters. * Explains per-cluster audience isolation, available `spec.oidc.mode` options, and provides configuration examples for platform-managed and custom identity providers. * Covers how to obtain and use kubeconfig for sign-in, authentication workflow with the login plugin, prerequisites/gotchas (including legacy flag incompatibility), and cleanup behavior when OIDC is disabled or clusters are deleted. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 5fa55f7 + 526de66 commit 257bbbb

2 files changed

Lines changed: 298 additions & 0 deletions

File tree

Lines changed: 149 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,149 @@
1+
---
2+
title: "OIDC authentication for kubectl"
3+
linkTitle: "OIDC authentication"
4+
description: "Give tenant users per-identity kubectl access to a Managed Kubernetes cluster with per-cluster audience isolation."
5+
weight: 45
6+
---
7+
8+
Tenant Kubernetes clusters can authenticate `kubectl` users through OIDC instead of the shared static admin kubeconfig. Each user then has their own identity, per-user audit, and RBAC that can be revoked by disabling the account — not by rotating a shared certificate.
9+
10+
The identity model is deliberately **per-cluster** rather than per-tenant: each tenant Kubernetes cluster gets its own OIDC audience, and a token minted for cluster A is rejected by cluster B's apiserver. That gives you cross-cluster isolation without provisioning a Keycloak realm per tenant. The full rationale (why per-cluster audience and not per-tenant realm; how it relates to Keycloak Organizations; what BYO-OIDC looks like) is in the [design proposal](https://github.com/cozystack/community/pull/24).
11+
12+
{{% alert color="info" %}}
13+
The static `kubernetes-<cluster>-admin-kubeconfig` Secret in the tenant namespace stays available as a break-glass path regardless of whether OIDC is enabled.
14+
{{% /alert %}}
15+
16+
## Modes
17+
18+
`spec.oidc.mode` picks the identity source:
19+
20+
- **`None`** — the default. No OIDC; only the static admin kubeconfig works. Existing clusters render identically to before.
21+
- **`System`** — trust the platform `cozy` Keycloak realm via a per-cluster public client and audience binding. Users are the ones a Cozystack platform admin already provisioned in `cozy`; the tenant does not manage a directory of its own.
22+
- **`CustomConfig`** — trust a tenant-supplied issuer directly (BYO IdP: Okta, Auth0, a customer's own Keycloak). `cozy` is not in the path.
23+
24+
## Enable OIDC — `System` mode
25+
26+
```yaml
27+
apiVersion: apps.cozystack.io/v1alpha1
28+
kind: Kubernetes
29+
metadata:
30+
name: prod
31+
namespace: tenant-acme
32+
spec:
33+
oidc:
34+
mode: System
35+
users:
36+
37+
role: admin # binds to ClusterRole/cluster-admin
38+
39+
role: view # binds to ClusterRole/view
40+
# ...
41+
```
42+
43+
Cozystack provisions:
44+
45+
- A per-cluster `KeycloakClient` in the `cozy` realm, with `clientId` set to `<namespace>-kubernetes-<cluster-name>` (for the CR above: `tenant-acme-kubernetes-prod`). `public: true`, PKCE required, redirect URIs locked to `localhost:8000` and `localhost:18000` (the `kubectl oidc-login` defaults).
46+
- A per-cluster `KeycloakClientScope` whose audience mapper pins the token's `aud` claim to that same `clientId`.
47+
- A structured `AuthenticationConfiguration` (`apiserver.config.k8s.io/v1beta1`) on the tenant kube-apiserver, pointing at the `cozy` issuer and the per-cluster audience.
48+
- One `ClusterRoleBinding` inside the tenant cluster for each `users[]` entry — `admin` → `cluster-admin`, `view` → `view`. The chart uses your `users[].email` value as the `User:` subject and matches it against the token's `email` claim.
49+
50+
Removing a user from `users[]` prunes their `ClusterRoleBinding` on the next reconcile.
51+
52+
### Prerequisite
53+
54+
`System` mode requires the platform-level OIDC feature (`authentication.oidc.enabled` at the Cozystack platform values). If the flag is off, the chart hard-fails the render with a clear message. Ask a Cozystack platform admin to enable it, or use `CustomConfig`.
55+
56+
## Enable OIDC — `CustomConfig` mode
57+
58+
Bring your own issuer. Two supply paths, **mutually exclusive**:
59+
60+
```yaml
61+
spec:
62+
oidc:
63+
mode: CustomConfig
64+
customConfig:
65+
config: |
66+
apiVersion: apiserver.config.k8s.io/v1beta1
67+
kind: AuthenticationConfiguration
68+
jwt:
69+
- issuer:
70+
url: https://idp.acme.example
71+
certificateAuthority: |
72+
-----BEGIN CERTIFICATE-----
73+
...
74+
-----END CERTIFICATE-----
75+
audiences:
76+
- cozystack-prod
77+
claimMappings:
78+
username:
79+
claim: email
80+
prefix: ""
81+
groups:
82+
claim: groups
83+
prefix: ""
84+
users:
85+
86+
role: admin
87+
```
88+
89+
…or via a pre-existing Secret in the tenant namespace (you create it separately, e.g. under your own GitOps repository, so the AuthenticationConfiguration does not live inside the `Kubernetes` CR):
90+
91+
```yaml
92+
spec:
93+
oidc:
94+
mode: CustomConfig
95+
customConfig:
96+
secretRef:
97+
name: acme-byo-authn-config # Secret with a `config.yaml` key holding the AuthenticationConfiguration
98+
```
99+
100+
Setting both `config` and `secretRef.name` (or neither) fails the render. In `CustomConfig` mode no Keycloak objects are provisioned in `cozy`; the tenant apiserver trusts the tenant-supplied issuer directly.
101+
102+
Ensure your BYO issuer emits the `email` claim in the JWT. Every conformant OIDC provider does when the client requests the `email` scope. If you distribute a hand-crafted kubeconfig instead of using the chart-generated one, remember to include `--oidc-extra-scope=email` in the `kubectl oidc-login` exec block.
103+
104+
## Get the kubeconfig
105+
106+
In `System` mode, Cozystack writes a ready-to-use kubeconfig into a `kubernetes-<cluster>-oidc-kubeconfig` Secret in the tenant namespace (the same namespace where the `Kubernetes` resource lives). It's exposed to the dashboard alongside the admin kubeconfig, and you can also fetch it directly:
107+
108+
```shell
109+
kubectl --namespace tenant-acme get secret kubernetes-prod-oidc-kubeconfig \
110+
--output=jsonpath='{.data.kubeconfig}' | base64 -d > prod.kubeconfig
111+
```
112+
113+
The file contains the tenant CA (extracted from the Kamaji-issued admin kubeconfig at reconcile time), the external apiserver URL, and a `kubectl oidc-login` exec block wired to your per-cluster client.
114+
115+
In `CustomConfig` mode no kubeconfig Secret is generated — you distribute the OIDC kubeconfig out-of-band from your own IdP configuration.
116+
117+
## Sign in
118+
119+
Install the `oidc-login` kubectl plugin once:
120+
121+
```shell
122+
kubectl krew install oidc-login
123+
```
124+
125+
Then use the OIDC kubeconfig — the first request triggers the browser flow:
126+
127+
```shell
128+
kubectl --kubeconfig prod.kubeconfig get pods --all-namespaces
129+
```
130+
131+
`kubectl oidc-login` opens Keycloak's login page on `localhost:8000` (falling back to `localhost:18000`), captures the token, and caches it locally. Subsequent calls are silent until the token expires.
132+
133+
## Toggling OIDC off
134+
135+
Setting `spec.oidc.mode` back to `None` — or deleting the `Kubernetes` resource entirely — reconciles a cleanup pass that removes the tenant apiserver's `--authentication-config` flag, deletes the chart-owned OIDC Secrets, and drops the per-cluster Keycloak client and audience scope. `ClusterRoleBindings` labelled by the release are also removed from the tenant cluster (best-effort during pre-delete, since the tenant apiserver may already be tearing down).
136+
137+
## Prerequisites and gotchas
138+
139+
- **Don't mix with legacy `--oidc-*` flags.** The tenant kube-apiserver refuses to boot if both `--authentication-config` (injected by `spec.oidc`) and any legacy `--oidc-*` flag are set. If you previously wired OIDC by hand through `controlPlane.apiServer.extraArgs`, remove those flags before enabling `spec.oidc`. The chart fails the render with a pointer to this migration.
140+
- **`oidc-login` plugin required.** Without `kubectl krew install oidc-login` the exec block errors out client-side. The plugin is a documented prerequisite.
141+
- **`emailVerified: true` when provisioning Keycloak users.** Phase 1 does not add a `claimValidationRules` entry to the rendered `AuthenticationConfiguration` — so `email_verified` is not chart-enforced. Set `emailVerified: true` on the `KeycloakRealmUser` (or complete the email-verify flow through the Keycloak UI) so the identity holding a given `users[].email` is guaranteed authentic. The `cozy` realm's default `duplicateEmails: false` prevents a second account from claiming an already-registered address. If the issuer explicitly emits `email_verified: false` on a token the apiserver rejects it (k8s upstream behaviour); a missing claim is treated as verified. CEL `claimValidationRules` to make this a hard gate is a follow-up hardening path.
142+
- **Custom issuer with a self-signed CA.** In `CustomConfig` mode you can supply the CA inline under `issuer.certificateAuthority`. The legacy `--oidc-*` flag path could not.
143+
144+
## What's out of scope for this feature
145+
146+
- **Per-tenant Keycloak realms.** Managed multi-tenant identity (a hosted directory the tenant self-administers) is a separate proposal, evaluated against Keycloak Organizations. Track it in the [community proposal](https://github.com/cozystack/community/pull/24).
147+
- **Federating an external IdP into the platform `cozy` realm.** BYO-for-Cozystack-itself is a distinct problem — this feature is BYO-for-a-managed-service.
148+
- **Cross-cluster SSO inside one tenant.** By design: each cluster has its own audience, which is the per-cluster isolation primitive.
149+
- **RFC 8693 token exchange.** Possible future optimisation; not required for the per-cluster client + audience model.
Lines changed: 149 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,149 @@
1+
---
2+
title: "OIDC authentication for kubectl"
3+
linkTitle: "OIDC authentication"
4+
description: "Give tenant users per-identity kubectl access to a Managed Kubernetes cluster with per-cluster audience isolation."
5+
weight: 45
6+
---
7+
8+
Tenant Kubernetes clusters can authenticate `kubectl` users through OIDC instead of the shared static admin kubeconfig. Each user then has their own identity, per-user audit, and RBAC that can be revoked by disabling the account — not by rotating a shared certificate.
9+
10+
The identity model is deliberately **per-cluster** rather than per-tenant: each tenant Kubernetes cluster gets its own OIDC audience, and a token minted for cluster A is rejected by cluster B's apiserver. That gives you cross-cluster isolation without provisioning a Keycloak realm per tenant. The full rationale (why per-cluster audience and not per-tenant realm; how it relates to Keycloak Organizations; what BYO-OIDC looks like) is in the [design proposal](https://github.com/cozystack/community/pull/24).
11+
12+
{{% alert color="info" %}}
13+
The static `kubernetes-<cluster>-admin-kubeconfig` Secret in the tenant namespace stays available as a break-glass path regardless of whether OIDC is enabled.
14+
{{% /alert %}}
15+
16+
## Modes
17+
18+
`spec.oidc.mode` picks the identity source:
19+
20+
- **`None`** — the default. No OIDC; only the static admin kubeconfig works. Existing clusters render identically to before.
21+
- **`System`** — trust the platform `cozy` Keycloak realm via a per-cluster public client and audience binding. Users are the ones a Cozystack platform admin already provisioned in `cozy`; the tenant does not manage a directory of its own.
22+
- **`CustomConfig`** — trust a tenant-supplied issuer directly (BYO IdP: Okta, Auth0, a customer's own Keycloak). `cozy` is not in the path.
23+
24+
## Enable OIDC — `System` mode
25+
26+
```yaml
27+
apiVersion: apps.cozystack.io/v1alpha1
28+
kind: Kubernetes
29+
metadata:
30+
name: prod
31+
namespace: tenant-acme
32+
spec:
33+
oidc:
34+
mode: System
35+
users:
36+
37+
role: admin # binds to ClusterRole/cluster-admin
38+
39+
role: view # binds to ClusterRole/view
40+
# ...
41+
```
42+
43+
Cozystack provisions:
44+
45+
- A per-cluster `KeycloakClient` in the `cozy` realm, with `clientId` set to `<namespace>-kubernetes-<cluster-name>` (for the CR above: `tenant-acme-kubernetes-prod`). `public: true`, PKCE required, redirect URIs locked to `localhost:8000` and `localhost:18000` (the `kubectl oidc-login` defaults).
46+
- A per-cluster `KeycloakClientScope` whose audience mapper pins the token's `aud` claim to that same `clientId`.
47+
- A structured `AuthenticationConfiguration` (`apiserver.config.k8s.io/v1beta1`) on the tenant kube-apiserver, pointing at the `cozy` issuer and the per-cluster audience.
48+
- One `ClusterRoleBinding` inside the tenant cluster for each `users[]` entry — `admin` → `cluster-admin`, `view` → `view`. The chart uses your `users[].email` value as the `User:` subject and matches it against the token's `email` claim.
49+
50+
Removing a user from `users[]` prunes their `ClusterRoleBinding` on the next reconcile.
51+
52+
### Prerequisite
53+
54+
`System` mode requires the platform-level OIDC feature (`authentication.oidc.enabled` at the Cozystack platform values). If the flag is off, the chart hard-fails the render with a clear message. Ask a Cozystack platform admin to enable it, or use `CustomConfig`.
55+
56+
## Enable OIDC — `CustomConfig` mode
57+
58+
Bring your own issuer. Two supply paths, **mutually exclusive**:
59+
60+
```yaml
61+
spec:
62+
oidc:
63+
mode: CustomConfig
64+
customConfig:
65+
config: |
66+
apiVersion: apiserver.config.k8s.io/v1beta1
67+
kind: AuthenticationConfiguration
68+
jwt:
69+
- issuer:
70+
url: https://idp.acme.example
71+
certificateAuthority: |
72+
-----BEGIN CERTIFICATE-----
73+
...
74+
-----END CERTIFICATE-----
75+
audiences:
76+
- cozystack-prod
77+
claimMappings:
78+
username:
79+
claim: email
80+
prefix: ""
81+
groups:
82+
claim: groups
83+
prefix: ""
84+
users:
85+
86+
role: admin
87+
```
88+
89+
…or via a pre-existing Secret in the tenant namespace (you create it separately, e.g. under your own GitOps repository, so the AuthenticationConfiguration does not live inside the `Kubernetes` CR):
90+
91+
```yaml
92+
spec:
93+
oidc:
94+
mode: CustomConfig
95+
customConfig:
96+
secretRef:
97+
name: acme-byo-authn-config # Secret with a `config.yaml` key holding the AuthenticationConfiguration
98+
```
99+
100+
Setting both `config` and `secretRef.name` (or neither) fails the render. In `CustomConfig` mode no Keycloak objects are provisioned in `cozy`; the tenant apiserver trusts the tenant-supplied issuer directly.
101+
102+
Ensure your BYO issuer emits the `email` claim in the JWT. Every conformant OIDC provider does when the client requests the `email` scope. If you distribute a hand-crafted kubeconfig instead of using the chart-generated one, remember to include `--oidc-extra-scope=email` in the `kubectl oidc-login` exec block.
103+
104+
## Get the kubeconfig
105+
106+
In `System` mode, Cozystack writes a ready-to-use kubeconfig into a `kubernetes-<cluster>-oidc-kubeconfig` Secret in the tenant namespace (the same namespace where the `Kubernetes` resource lives). It's exposed to the dashboard alongside the admin kubeconfig, and you can also fetch it directly:
107+
108+
```shell
109+
kubectl --namespace tenant-acme get secret kubernetes-prod-oidc-kubeconfig \
110+
--output=jsonpath='{.data.kubeconfig}' | base64 -d > prod.kubeconfig
111+
```
112+
113+
The file contains the tenant CA (extracted from the Kamaji-issued admin kubeconfig at reconcile time), the external apiserver URL, and a `kubectl oidc-login` exec block wired to your per-cluster client.
114+
115+
In `CustomConfig` mode no kubeconfig Secret is generated — you distribute the OIDC kubeconfig out-of-band from your own IdP configuration.
116+
117+
## Sign in
118+
119+
Install the `oidc-login` kubectl plugin once:
120+
121+
```shell
122+
kubectl krew install oidc-login
123+
```
124+
125+
Then use the OIDC kubeconfig — the first request triggers the browser flow:
126+
127+
```shell
128+
kubectl --kubeconfig prod.kubeconfig get pods --all-namespaces
129+
```
130+
131+
`kubectl oidc-login` opens Keycloak's login page on `localhost:8000` (falling back to `localhost:18000`), captures the token, and caches it locally. Subsequent calls are silent until the token expires.
132+
133+
## Toggling OIDC off
134+
135+
Setting `spec.oidc.mode` back to `None` — or deleting the `Kubernetes` resource entirely — reconciles a cleanup pass that removes the tenant apiserver's `--authentication-config` flag, deletes the chart-owned OIDC Secrets, and drops the per-cluster Keycloak client and audience scope. `ClusterRoleBindings` labelled by the release are also removed from the tenant cluster (best-effort during pre-delete, since the tenant apiserver may already be tearing down).
136+
137+
## Prerequisites and gotchas
138+
139+
- **Don't mix with legacy `--oidc-*` flags.** The tenant kube-apiserver refuses to boot if both `--authentication-config` (injected by `spec.oidc`) and any legacy `--oidc-*` flag are set. If you previously wired OIDC by hand through `controlPlane.apiServer.extraArgs`, remove those flags before enabling `spec.oidc`. The chart fails the render with a pointer to this migration.
140+
- **`oidc-login` plugin required.** Without `kubectl krew install oidc-login` the exec block errors out client-side. The plugin is a documented prerequisite.
141+
- **`emailVerified: true` when provisioning Keycloak users.** Phase 1 does not add a `claimValidationRules` entry to the rendered `AuthenticationConfiguration` — so `email_verified` is not chart-enforced. Set `emailVerified: true` on the `KeycloakRealmUser` (or complete the email-verify flow through the Keycloak UI) so the identity holding a given `users[].email` is guaranteed authentic. The `cozy` realm's default `duplicateEmails: false` prevents a second account from claiming an already-registered address. If the issuer explicitly emits `email_verified: false` on a token the apiserver rejects it (k8s upstream behaviour); a missing claim is treated as verified. CEL `claimValidationRules` to make this a hard gate is a follow-up hardening path.
142+
- **Custom issuer with a self-signed CA.** In `CustomConfig` mode you can supply the CA inline under `issuer.certificateAuthority`. The legacy `--oidc-*` flag path could not.
143+
144+
## What's out of scope for this feature
145+
146+
- **Per-tenant Keycloak realms.** Managed multi-tenant identity (a hosted directory the tenant self-administers) is a separate proposal, evaluated against Keycloak Organizations. Track it in the [community proposal](https://github.com/cozystack/community/pull/24).
147+
- **Federating an external IdP into the platform `cozy` realm.** BYO-for-Cozystack-itself is a distinct problem — this feature is BYO-for-a-managed-service.
148+
- **Cross-cluster SSO inside one tenant.** By design: each cluster has its own audience, which is the per-cluster isolation primitive.
149+
- **RFC 8693 token exchange.** Possible future optimisation; not required for the per-cluster client + audience model.

0 commit comments

Comments
 (0)