Skip to content

Commit 3fb8ec5

Browse files
authored
feat(monitoring): per-instance OIDC selector for the Grafana instance (Phase 1) (#3176)
<!-- Follow-up to #3044 — same design family, different subject. --> ## What this PR does Adds a Phase-1 OIDC selector to the Grafana instance shipped by `packages/system/monitoring/`, mirroring the shape of the tenant kube-apiserver PR ([#3044](#3044)) so that operators can use one mental model for both. Rationale in [cozystack/community#24](cozystack/community#24) — per-cluster (here: per-Monitoring-release) audience isolation on the flat `cozy` realm, no per-tenant realm. Authorization is **app-side** via a chart-owned `users:` map reconciled by a post-install/post-upgrade Job into Grafana org membership. The chart does NOT own Keycloak directory objects (no `KeycloakRealmGroup`s) and does NOT rely on Grafana's `role_attribute_path` — same design shape as #3044's ClusterRoleBinding reconcile, adapted for Grafana orgs. See `docs/oidc-grafana.md` for the full rationale. The `Monitoring` CR grows a new field: ```yaml spec: oidc: mode: None | System | CustomConfig # default None customConfig: # only for CustomConfig config: {} # inline grafana.ini keys, or... secretRef: name: "" # ...an operator-managed Secret users: # optional; ignored in mode=None - email: [email protected] role: Admin | Editor | Viewer ``` ### Modes - **`None`** (default) — no OIDC; `admin_user` / `admin_password` Secret is the only auth path. Existing releases render byte-identical. - **`System`** — chart provisions, in the release namespace of the Monitoring CR: - `KeycloakClient` (`public: false`, confidential, `secret` sourced from a chart-owned Secret via the EDP `$<secret>:<key>` reference; `redirectUris` locked to `https://grafana.<host>/login/generic_oauth`). - `KeycloakClientScope` with an `oidc-audience-mapper` pinning `id_token.aud` to the per-release clientId — the isolation primitive. - Persistent Kubernetes Secret carrying the confidential `client-secret` (random on first install, preserved via `lookup`+fallback, same pattern as `packages/system/dashboard`). - Grafana CR's `spec.config.auth.generic_oauth` section wired to the `cozy` realm issuer + per-release audience scope, with two chart-forced settings the users-Job depends on: `skip_org_role_sync: "true"` (login never overwrites the Job's org-role assignments) and `oauth_allow_insecure_email_lookup: "true"` (OIDC identity binds to the pre-provisioned local account by email). - `GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET` env on the Grafana Deployment sourced from the Secret. - Post-install / post-upgrade **users-Job** reconciling `spec.oidc.users` into Grafana's Main Org.: pre-provisions a local account per listed email, `POST /api/orgs/1/users` + `PATCH /api/orgs/1/users/{userId}` to converge role changes, prunes every non-admin Main-Org member whose email is not in the list. - Server-level `GrafanaAdmin` promotion (`allow_assign_grafana_admin`) is out of scope for Phase 1. Every Grafana instance (platform and tenant) caps at org-level `Admin`. - **`CustomConfig`** — tenant supplies the whole `[auth.generic_oauth]` payload; `cozy` is not in the path. Two mutually exclusive paths: - `customConfig.config: {...}` — inline map of `grafana.ini` keys. The chart **merges** the two chart-forced settings (`skip_org_role_sync`, `oauth_allow_insecure_email_lookup`) on top so the users-Job contract holds. The operator can NOT override those two (merge wins). - `customConfig.secretRef.name: ...` — operator-owned Secret with an `auth.ini` fragment; the chart mounts it under `/etc/grafana/oidc` and wires `GF_PATHS_CUSTOM_INI`, and does NOT emit `auth.generic_oauth` under `spec.config` (the ini fragment is authoritative). Because the chart cannot inject the two settings into a mounted ini, `spec.oidc.users` is NOT supported in this branch — the chart hard-fails render on that combination. ### Fail-close guards - `mode: System` without `_cluster.oidc-enabled == "true"` (the platform-level `authentication.oidc.enabled` flag) → hard-fail with an actionable message pointing at the platform values. - `mode: System` on a cluster where the EDP Keycloak operator CRDs (`v1.edp.epam.com/v1`) are not registered → hard-fail (symmetric on `oidc-keycloak.yaml` and `grafana.yaml`). - `mode: CustomConfig` with both or neither of `config` / `secretRef.name` → hard-fail. - `mode: CustomConfig` with `secretRef.name` AND non-empty `users` → hard-fail (chart cannot inject the two forced settings into an operator-mounted ini; forcing an explicit choice avoids silent contract loss). - `spec.oidc.users[]` entry with missing / unknown `role` or malformed `email` → hard-fail at render time so the users-Job never fires with a body that would 400 on Grafana's admin API. ### Break-glass posture `admin_user` / `admin_password` and `disable_login_form: false` are unchanged in every mode. Locking the form off under `mode: System` is a documented follow-up hardening (see `docs/oidc-grafana.md`). ### What's NOT in this PR (documented as out of scope) - Per-tenant Keycloak realms (Phase 2 territory, tracked in community#24). - Full-logout through Keycloak's end-session endpoint (`backend-logout-url` and matching Keycloak client attribute). - CEL `claimValidationRules` gate on `email_verified` — layered guarantees are described in `docs/oidc-grafana.md`. - Multi-issuer composition on one Monitoring release (mutually exclusive modes). - Server-level `GrafanaAdmin` promotion. ### Docs - New `docs/oidc-grafana.md` — full operator guide (parallel to `docs/oidc-tenant.md`). - Website page follows in a paired PR against cozystack/website (`docs/monitoring-oidc-authentication` branch, under `content/en/docs/next/operations/services/monitoring/oidc-authentication.md`). ### Tests - 33 `helm-unittest` cases in `tests/oidc_test.yaml` + `tests/oidc_crd_missing_test.yaml` cover: all three modes; defensive nil paths (`oidc: ~`); System-mode Keycloak client + audience scope shape; CustomConfig chart-forced merge (positive assertion) AND `mode=CustomConfig inline — operator cannot override chart-forced OIDC settings` (regression guard); CustomConfig secretRef + `users` incompat fail-fast; `spec.oidc.users[].role` missing/unknown fail-fast; users-Job shape (hook annotations, RBAC-free, activeDeadlineSeconds, `ttlSecondsAfterFinished: 3600`, admin credential wiring); no chart-owned `KeycloakRealmGroup`s in any mode; no `role_attribute_path` in Grafana config. - Render-side bats in `hack/e2e-apps/monitoring-oidc-{system,customconfig}.bats` — mirror the tenant kube-apiserver's e2e shape (no live browser flow — deferred to a follow-up integration suite, same as #3044). Asserts on live cluster: users-Job appears with `DESIRED_USERS_JSON` populated, Grafana config carries `skip_org_role_sync=true` and no `role_attribute_path`, no chart-owned `KeycloakRealmGroup`s land in `cozy`. ### Screenshots N/A — no UI change on default values; the visual delta is Grafana's own "Sign in with Keycloak" button appearing under the login form when a release opts in. ### Release note ```release-note feat(monitoring): Each `Monitoring` CR can now opt its Grafana instance into OIDC via a flat selector — `spec.oidc.mode: System | CustomConfig | None` (default `None`). `System` trusts the platform `cozy` realm via a per-instance confidential Keycloak client + audience binding. Authorization is app-side: `spec.oidc.users: [{email, role: Admin|Editor|Viewer}]` is reconciled into Grafana's Main Org. by a chart-owned post-install/post-upgrade Job (pre-provision → org add → PATCH role → prune stale). `CustomConfig` accepts a tenant-supplied `[auth.generic_oauth]` payload — inline as a map (chart merges the two contract-critical settings `skip_org_role_sync` + `oauth_allow_insecure_email_lookup` on top) or as a Secret with an `auth.ini` key (in which case `spec.oidc.users` is not supported and the chart fails render on that combination). Server-level `GrafanaAdmin` promotion is out of scope for Phase 1. The `admin_user`/`admin_password` Secret stays a documented break-glass path in every mode. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added OIDC authentication for Monitoring Grafana with `None`, platform-managed `System`, and tenant-controlled `CustomConfig` modes. * System mode provisions platform-backed OIDC for each instance (including persistent client credentials) and can optionally promote matching users to server-level `GrafanaAdmin`. * CustomConfig supports inline OAuth settings or Secret-backed `auth.ini` (with mutual exclusivity validation).
2 parents 215bd43 + 5d13c15 commit 3fb8ec5

24 files changed

Lines changed: 2894 additions & 9 deletions

‎docs/oidc-grafana.md‎

Lines changed: 410 additions & 0 deletions
Large diffs are not rendered by default.

‎docs/oidc-tenant.md‎

Lines changed: 26 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -195,10 +195,10 @@ expires.
195195
the apiserver rejects tokens at request time; the admin kubeconfig
196196
keeps working as break-glass.
197197
- **`emailVerified` on Keycloak users is a prescriptive requirement,
198-
not a chart-enforced one.** Phase 1 does not emit any
199-
`claimValidationRules` in the rendered `AuthenticationConfiguration`,
200-
so the chart itself is not the gate. The layered guarantees you rely
201-
on instead:
198+
not a chart-enforced one.** The chart emits a
199+
`claimValidationRules` entry keyed on `groups` (see next bullet),
200+
but not one keyed on `email_verified`. The layered guarantees you
201+
rely on instead:
202202
1. Provision users with `emailVerified: true` (via `KeycloakRealmUser`
203203
or the Keycloak UI's email-verify flow) so no unverified identity
204204
ever holds the email you name in `users[].email`.
@@ -214,6 +214,28 @@ expires.
214214
the rendered System-mode config would elevate item 3 to a hard gate;
215215
that is a reasonable follow-up hardening but is out of scope for
216216
Phase 1.
217+
218+
- **Cross-tenant isolation gate.** The rendered
219+
`AuthenticationConfiguration` carries a CEL `claimValidationRules`
220+
entry on the `groups` claim that rejects any token whose caller is
221+
not a member of at least one of this tenant's four Keycloak groups:
222+
223+
```
224+
<namespace>-view <namespace>-use <namespace>-admin <namespace>-super-admin
225+
```
226+
227+
The tenant chart (`packages/apps/tenant/templates/keycloakgroups.yaml`)
228+
provisions these groups in the shared `cozy` realm. Without this
229+
gate, any authenticated cozy-realm user could point `kubectl
230+
oidc-login` at another tenant's cluster, land at
231+
`system:authenticated`, and enumerate the discovery surface
232+
(`kubectl api-resources`, OpenAPI schemata) — RBAC would default-deny
233+
named-resource access, but discovery still leaks. Symmetric with the
234+
Grafana-side `allowed_groups` gate on the Monitoring chart's
235+
`auth.generic_oauth` block (see `docs/oidc-grafana.md`). CustomConfig
236+
mode does NOT get this gate injected — the tenant's own
237+
AuthenticationConfiguration is authoritative and the tenant is
238+
responsible for their own claim-side guards.
217239
- **`kubectl` without the `oidc-login` plugin** — the exec block errors
218240
out client-side; install the plugin.
219241
Lines changed: 202 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,202 @@
1+
#!/usr/bin/env bats
2+
3+
# Phase-1 OIDC CustomConfig selector e2e for the Grafana instance,
4+
# render-side only.
5+
#
6+
# What is exercised on a live cluster: that the new
7+
# `spec.oidc.mode: CustomConfig` field is admitted; that cozystack-api
8+
# accepts the updated schema; that the HelmRelease renders; that the
9+
# operator-supplied auth.generic_oauth payload lands on the Grafana
10+
# CR; and, crucially, that NO Keycloak objects are created in the
11+
# `cozy` realm when the tenant brings their own issuer.
12+
13+
# cozytest.sh (the e2e runner) is not real bats: it never invokes
14+
# setup()/teardown(). Cleanup belongs in cozy_cleanup(), which runs at
15+
# suite exit and on the first failing test. Per-test isolation is done
16+
# inline at the top of each @test.
17+
#
18+
# Naming: the extra/monitoring chart is a singleton per tenant
19+
# namespace and enforces `.Release.Name == "monitoring"` in
20+
# templates/check-release-name.yaml. So the Monitoring CR (and its
21+
# operator-produced outer HelmRelease) MUST be named `monitoring`;
22+
# the inner packages/system/monitoring HelmRelease that carries the
23+
# OIDC templates is then named `${outer}-system`, i.e. `monitoring-system`.
24+
CR_NAME="monitoring"
25+
INNER_REL="${CR_NAME}-system"
26+
BYO_SECRET="mon-oidc-byo-oauth"
27+
28+
cleanup_mon() {
29+
kubectl -n tenant-test delete monitoring.apps.cozystack.io "${CR_NAME}" \
30+
--ignore-not-found --wait=false 2>/dev/null || true
31+
kubectl -n tenant-test wait monitoring.apps.cozystack.io "${CR_NAME}" \
32+
--for=delete --timeout=2m 2>/dev/null || true
33+
kubectl -n tenant-test delete secret "${BYO_SECRET}" \
34+
--ignore-not-found 2>/dev/null || true
35+
}
36+
37+
cozy_cleanup() { cleanup_mon; }
38+
39+
@test "Monitoring CR accepts spec.oidc.mode=CustomConfig with inline config" {
40+
cleanup_mon
41+
kubectl apply -f - <<EOF
42+
apiVersion: apps.cozystack.io/v1alpha1
43+
kind: Monitoring
44+
metadata:
45+
name: ${CR_NAME}
46+
namespace: tenant-test
47+
spec:
48+
host: ""
49+
metricsStorages:
50+
- name: shortterm
51+
retentionPeriod: "3d"
52+
deduplicationInterval: "15s"
53+
storage: 10Gi
54+
storageClassName: replicated
55+
logsStorages:
56+
- name: generic
57+
retentionPeriod: "1"
58+
storage: 10Gi
59+
storageClassName: replicated
60+
oidc:
61+
mode: CustomConfig
62+
customConfig:
63+
config:
64+
client_id: byo-monitoring
65+
client_secret: byo-secret
66+
auth_url: https://idp.example.test/auth
67+
token_url: https://idp.example.test/token
68+
api_url: https://idp.example.test/userinfo
69+
users:
70+
71+
role: Viewer
72+
EOF
73+
74+
# Outer HR (created by cozystack-operator from the CR) shares the CR name.
75+
timeout 60 sh -ec 'until kubectl -n tenant-test get hr "'"${CR_NAME}"'" >/dev/null 2>&1; do sleep 2; done'
76+
# Inner HR (rendered by extra/monitoring) is ${outer}-system.
77+
timeout 120 sh -ec 'until kubectl -n tenant-test get hr "'"${INNER_REL}"'" >/dev/null 2>&1; do sleep 2; done'
78+
timeout 180 sh -ec 'until kubectl -n tenant-test get grafana grafana >/dev/null 2>&1; do sleep 5; done'
79+
}
80+
81+
@test "Grafana CR carries the operator's auth.generic_oauth payload verbatim" {
82+
# Poll the specific field, not just CR existence — otherwise we may
83+
# grab a stale Grafana CR that hasn't yet been reconciled with the
84+
# inner HR's CustomConfig values.
85+
timeout 180 sh -ec '
86+
until [ "$(kubectl -n tenant-test get grafana grafana \
87+
-o jsonpath="{.spec.config.auth\.generic_oauth.client_id}" 2>/dev/null)" = "byo-monitoring" ]; do
88+
sleep 5
89+
done
90+
'
91+
92+
client_id=$(kubectl -n tenant-test get grafana grafana \
93+
-o jsonpath='{.spec.config.auth\.generic_oauth.client_id}')
94+
echo "client_id: ${client_id}"
95+
[ "${client_id}" = "byo-monitoring" ]
96+
97+
auth_url=$(kubectl -n tenant-test get grafana grafana \
98+
-o jsonpath='{.spec.config.auth\.generic_oauth.auth_url}')
99+
echo "auth_url: ${auth_url}"
100+
[ "${auth_url}" = "https://idp.example.test/auth" ]
101+
102+
# No cozy realm in the picture.
103+
echo "${auth_url}" | grep -vq "realms/cozy" || {
104+
echo "auth_url unexpectedly references the cozy realm"
105+
return 1
106+
}
107+
}
108+
109+
@test "No KeycloakClient / Scope are created in the cozy realm" {
110+
# Only meaningful when the EDP CRDs exist on the cluster.
111+
if ! kubectl api-resources --api-group=v1.edp.epam.com >/dev/null 2>&1; then
112+
skip "EDP Keycloak operator CRDs not present on this cluster"
113+
fi
114+
115+
# clientId helper: printf "%s-%s" .Release.Namespace .Release.Name in
116+
# the inner chart -> tenant-test-monitoring-system.
117+
CID="tenant-test-${INNER_REL}"
118+
119+
# Give any reconciler a moment; then assert neither cozy-realm object
120+
# exists. Chart-owned KeycloakRealmGroups were removed entirely (see
121+
# design note in docs/oidc-grafana.md) so no group assertions here.
122+
sleep 5
123+
! kubectl -n tenant-test get keycloakclient.v1.edp.epam.com "${CID}" 2>/dev/null
124+
! kubectl -n tenant-test get keycloakclientscope.v1.edp.epam.com "${CID}-audience" 2>/dev/null
125+
}
126+
127+
@test "secretRef variant mounts operator Secret under /etc/grafana/oidc" {
128+
# Operator-owned Secret carrying a ready-made ini fragment.
129+
kubectl -n tenant-test create secret generic "${BYO_SECRET}" \
130+
--from-literal=auth.ini='[auth.generic_oauth]
131+
enabled = true
132+
name = ByoIdp
133+
client_id = byo-monitoring
134+
client_secret = byo-secret
135+
' --dry-run=client -o yaml | kubectl apply -f -
136+
137+
# Update the Monitoring CR in place: swap customConfig.config (inline)
138+
# for customConfig.secretRef.name. Delete + immediate apply races
139+
# kubectl's create-vs-patch decision against the API server's grace-
140+
# period cache and can hit "NotFound" during patch, so let the operator
141+
# reconcile the mode change on the same CR instead.
142+
kubectl apply -f - <<EOF
143+
apiVersion: apps.cozystack.io/v1alpha1
144+
kind: Monitoring
145+
metadata:
146+
name: ${CR_NAME}
147+
namespace: tenant-test
148+
spec:
149+
host: ""
150+
metricsStorages:
151+
- name: shortterm
152+
retentionPeriod: "3d"
153+
deduplicationInterval: "15s"
154+
storage: 10Gi
155+
storageClassName: replicated
156+
logsStorages:
157+
- name: generic
158+
retentionPeriod: "1"
159+
storage: 10Gi
160+
storageClassName: replicated
161+
oidc:
162+
mode: CustomConfig
163+
customConfig:
164+
secretRef:
165+
name: ${BYO_SECRET}
166+
EOF
167+
168+
timeout 60 sh -ec 'until kubectl -n tenant-test get hr "'"${CR_NAME}"'" >/dev/null 2>&1; do sleep 2; done'
169+
timeout 120 sh -ec 'until kubectl -n tenant-test get hr "'"${INNER_REL}"'" >/dev/null 2>&1; do sleep 2; done'
170+
timeout 180 sh -ec 'until kubectl -n tenant-test get grafana grafana >/dev/null 2>&1; do sleep 5; done'
171+
172+
# The Grafana CR from @test 1 already exists and carries the inline
173+
# auth.generic_oauth block; wait for the chart to reconcile the swap
174+
# to secretRef by polling for the target state (volume mount present
175+
# AND inline block absent) instead of asserting immediately. 600s
176+
# window is sized for a full inner-HR upgrade cycle + grafana-
177+
# operator reconcile — 180s is too tight when the sandbox is warm-
178+
# loaded, cf. the intermittent CI failure.
179+
timeout 600 sh -ec '
180+
until [ -n "$(kubectl -n tenant-test get grafana grafana -o jsonpath="{.spec.deployment.spec.template.spec.containers[?(@.name==\"grafana\")].volumeMounts[?(@.name==\"oidc-custom-ini\")].mountPath}")" ]; do sleep 5; done
181+
'
182+
timeout 120 sh -ec '
183+
until [ -z "$(kubectl -n tenant-test get grafana grafana -o jsonpath="{.spec.config.auth\.generic_oauth}")" ]; do sleep 5; done
184+
'
185+
186+
# spec.config carries no auth.generic_oauth section in this branch.
187+
section=$(kubectl -n tenant-test get grafana grafana \
188+
-o jsonpath='{.spec.config.auth\.generic_oauth}')
189+
echo "spec.config.auth.generic_oauth: ${section:-<empty>}"
190+
[ -z "${section}" ]
191+
192+
# The volume + mount + env are on the Grafana CR's deployment override.
193+
mount=$(kubectl -n tenant-test get grafana grafana \
194+
-o jsonpath='{.spec.deployment.spec.template.spec.containers[?(@.name=="grafana")].volumeMounts[?(@.name=="oidc-custom-ini")].mountPath}')
195+
echo "mount: ${mount}"
196+
[ "${mount}" = "/etc/grafana/oidc" ]
197+
198+
vol=$(kubectl -n tenant-test get grafana grafana \
199+
-o jsonpath='{.spec.deployment.spec.template.spec.volumes[?(@.name=="oidc-custom-ini")].secret.secretName}')
200+
echo "secret volume: ${vol}"
201+
[ "${vol}" = "${BYO_SECRET}" ]
202+
}

0 commit comments

Comments
 (0)