Skip to content

Commit f24b563

Browse files
lexfreilexfrei
committed
fix(platform): route OIDC backend calls through the in-cluster Keycloak by default
authentication.oidc.keycloakInternalUrl defaulted to an empty string, which left oauth2-proxy performing OIDC discovery against the external Keycloak hostname. That request resolves through public DNS and lands on the root ingress controller, which does not serve yet on a fresh install: the root tenant's IngressClass only exists once tenant-root is ready, and tenant-root is still reconciling while the dashboard starts. Discovery has nothing to talk to, the gatekeeper fails, and the dashboard never comes up. The bundled Keycloak is the only supported provider — the external hostname is templated as keycloak.<host> and the realm is fixed — so backend-to-backend calls have no reason to leave the cluster. Default the value to http://keycloak-http.cozy-keycloak.svc:8080/realms/cozy, the same Service address keycloak-configure already uses. The dashboard and linstor-gui gatekeepers then run with --skip-oidc-discovery and send token, jwks, userinfo and logout requests to the Service, while browser redirects stay on the external host. Those calls travel as plaintext HTTP between pods, as keycloak-configure's admin calls to the same address already do. The value remains overridable for a Keycloak that does not live at the standard in-cluster address, and setting it back to an empty string restores external discovery. Existing OIDC clusters that never set it get a gatekeeper rollout onto the internal path on upgrade. Enabling OIDC swaps the dashboard's token-proxy container for oauth2-proxy, so the E2E install now drives that swap to completion: it waits for the auth-proxy container to appear, then gates on the gatekeeper rollout. Readiness alone proves nothing there — the HelmRelease is still Ready from the token-proxy install, and the Deployment keeps the old ReplicaSet Available while a new pod crashloops, so only the rollout catches a regression back to external discovery. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent 5d881f1 commit f24b563

5 files changed

Lines changed: 373 additions & 12 deletions

File tree

‎hack/e2e-install-cozystack.bats‎

Lines changed: 51 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -328,16 +328,60 @@ EOF
328328
}
329329

330330
@test "Keycloak OIDC stack is healthy" {
331-
# keycloakInternalUrl makes oauth2-proxy skip OIDC discovery and route
332-
# backend calls (token/jwks/userinfo/logout) through the in-cluster
333-
# keycloak Service. Without it the dashboard gatekeeper crashloops on
334-
# DNS lookup of keycloak.example.org -- the e2e host placeholder does
335-
# not resolve, and under Flux v2.8 kstatus the gatekeeper Deployment
336-
# then flips to Failed and stalls the dashboard HelmRelease.
337-
kubectl patch package cozystack.cozystack-platform --type merge -p '{"spec":{"components":{"platform":{"values":{"authentication":{"oidc":{"enabled":true,"keycloakInternalUrl":"http://keycloak-http.cozy-keycloak.svc:8080/realms/cozy"}}}}}}}'
331+
# Only oidc.enabled is set here: keycloakInternalUrl defaults to the in-cluster
332+
# keycloak Service, which makes oauth2-proxy skip OIDC discovery and route the
333+
# backend calls (token/jwks/userinfo/logout) through that Service.
334+
kubectl patch package cozystack.cozystack-platform --type merge -p '{"spec":{"components":{"platform":{"values":{"authentication":{"oidc":{"enabled":true}}}}}}}'
338335

339336
timeout 120 sh -ec 'until kubectl get hr -n cozy-keycloak keycloak keycloak-configure keycloak-operator >/dev/null 2>&1; do sleep 1; done'
340337
kubectl wait hr/keycloak hr/keycloak-configure hr/keycloak-operator -n cozy-keycloak --timeout=20m --for=condition=ready
338+
339+
# Enabling OIDC swaps the dashboard's token-proxy container for oauth2-proxy,
340+
# so the dashboard is the consumer that proves the internal-URL default works.
341+
# The install-time `kubectl wait hr --all -A` ran before this test flipped the
342+
# flag and only ever saw the token-proxy shape, so nothing has re-checked the
343+
# dashboard on the OIDC path.
344+
#
345+
# Waiting on hr/dashboard directly would be vacuous: it is still Ready=True
346+
# from the token-proxy install, so `--for=condition=ready` returns instantly
347+
# against the stale condition, before Flux has even consumed the patched
348+
# values. Gate on an observable that exists ONLY on the OIDC path instead --
349+
# the auth-proxy container appearing in the gatekeeper Deployment -- which is
350+
# reached only after the new values are rendered.
351+
timeout 600 sh -ec 'until kubectl get deploy/incloud-web-gatekeeper -n cozy-dashboard -o jsonpath="{.spec.template.spec.containers[*].name}" 2>/dev/null | grep -qw auth-proxy; do sleep 5; done' || {
352+
echo "=== gatekeeper never re-rendered with the auth-proxy container after enabling OIDC ==="
353+
echo "=== the patched values likely never reached the dashboard HelmRelease ==="
354+
kubectl get package cozystack.cozystack-platform -o yaml 2>&1 || true
355+
kubectl get hr/dashboard -n cozy-dashboard -o yaml 2>&1 || true
356+
kubectl get deploy/incloud-web-gatekeeper -n cozy-dashboard -o yaml 2>&1 || true
357+
false
358+
}
359+
360+
# Then wait for the ROLLOUT, not for condition=available. The Deployment has
361+
# replicas: 1 and maxUnavailable: 25%, which rounds down to 0, so Kubernetes
362+
# keeps the old token-proxy pod up while the new one starts: Available stays
363+
# True on the strength of the OLD ReplicaSet even as the new auth-proxy pod
364+
# crashloops. `rollout status` is the check that only succeeds once the
365+
# UPDATED pod is available.
366+
#
367+
# That is what gives this case teeth: were the default to regress to the
368+
# external hostname, oauth2-proxy would do OIDC discovery against
369+
# keycloak.example.org -- the e2e host placeholder does not resolve -- and
370+
# crashloop, failing the rollout instead of shipping the regression.
371+
kubectl rollout status deploy/incloud-web-gatekeeper -n cozy-dashboard --timeout=10m || {
372+
echo "=== deploy/incloud-web-gatekeeper rollout did not complete after enabling OIDC ==="
373+
kubectl get deploy/incloud-web-gatekeeper -n cozy-dashboard -o yaml 2>&1 || true
374+
echo "=== cozy-dashboard pods ==="
375+
kubectl get pods -n cozy-dashboard -o wide 2>&1 || true
376+
echo "=== auth-proxy logs ==="
377+
kubectl logs -n cozy-dashboard -l app.kubernetes.io/name=gatekeeper --all-containers --tail=50 2>&1 || true
378+
false
379+
}
380+
381+
# Not the vacuous wait described above: past the rollout gate the HelmRelease
382+
# has necessarily been re-reconciled, so Ready here means the whole upgrade
383+
# converged -- not just the one Deployment this test watched.
384+
kubectl wait hr/dashboard -n cozy-dashboard --timeout=10m --for=condition=ready
341385
}
342386

343387
@test "Aggregated API rejects Tenant name with dashes" {

‎packages/core/platform/tests/apps_oidc_keycloak_internal_url_test.yaml‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
suite: apps.yaml OIDC keycloak-internal-url platform → tenant value wiring
2+
templates:
3+
- templates/apps.yaml
4+
5+
release:
6+
name: cozy-platform
7+
namespace: cozy-system
8+
9+
# authentication.oidc.keycloakInternalUrl rides the _cluster channel as
10+
# keycloak-internal-url and is read by the dashboard and linstor-gui
11+
# gatekeepers (packages/system/{dashboard,linstor-gui}/templates/gatekeeper.yaml).
12+
# A non-empty value switches oauth2-proxy to --skip-oidc-discovery and sends
13+
# the backend calls (token, jwks, userinfo, logout) straight to the Keycloak
14+
# Service, so the dashboard no longer depends on the root ingress serving the
15+
# external Keycloak hostname before it can start. The default must therefore be
16+
# the in-cluster Service address, and an operator override — including an
17+
# override back to "" — must still reach the consumers untouched.
18+
19+
tests:
20+
- it: writes the in-cluster keycloak Service address by default
21+
asserts:
22+
- matchRegex:
23+
path: stringData["values.yaml"]
24+
pattern: 'keycloak-internal-url:\s*"http://keycloak-http\.cozy-keycloak\.svc:8080/realms/cozy"'
25+
26+
- it: writes the in-cluster default even when OIDC is enabled
27+
set:
28+
authentication.oidc.enabled: true
29+
asserts:
30+
- matchRegex:
31+
path: stringData["values.yaml"]
32+
pattern: 'oidc-enabled:\s*"true"'
33+
- matchRegex:
34+
path: stringData["values.yaml"]
35+
pattern: 'keycloak-internal-url:\s*"http://keycloak-http\.cozy-keycloak\.svc:8080/realms/cozy"'
36+
37+
- it: honors an operator override pointing at a non-standard keycloak placement
38+
set:
39+
authentication.oidc.enabled: true
40+
authentication.oidc.keycloakInternalUrl: http://keycloak.example.test:8080/realms/other
41+
asserts:
42+
- matchRegex:
43+
path: stringData["values.yaml"]
44+
pattern: 'keycloak-internal-url:\s*"http://keycloak\.example\.test:8080/realms/other"'
45+
46+
- it: keeps the empty-string escape hatch, restoring external OIDC discovery
47+
set:
48+
authentication.oidc.enabled: true
49+
authentication.oidc.keycloakInternalUrl: ""
50+
asserts:
51+
- matchRegex:
52+
path: stringData["values.yaml"]
53+
pattern: 'keycloak-internal-url:\s*""'

‎packages/core/platform/values.yaml‎

Lines changed: 15 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -330,11 +330,21 @@ authentication:
330330
enabled: false
331331
insecureSkipVerify: false
332332
keycloakExtraRedirectUri: ""
333-
# Internal URL to access KeyCloak realm for backend-to-backend requests (bypasses external DNS).
334-
# When set, oauth2-proxy uses --skip-oidc-discovery and routes all backend calls (token, jwks,
335-
# userinfo, logout) through this URL while keeping browser redirects on the external URL.
336-
# Example: http://keycloak-http.cozy-keycloak.svc:8080/realms/cozy
337-
keycloakInternalUrl: ""
333+
# In-cluster URL of the KeyCloak realm, used for backend-to-backend requests. oauth2-proxy runs
334+
# with --skip-oidc-discovery and sends every backend call (token, jwks, userinfo, logout) to this
335+
# URL, while browser-facing redirects keep pointing at the external host.
336+
#
337+
# This is the default path: the bundled KeyCloak is the only supported provider, so those calls
338+
# have no reason to leave the cluster. Taking the external hostname instead would route them
339+
# through public DNS and the root ingress — which does not serve yet on a fresh install, leaving
340+
# the dashboard permanently down.
341+
#
342+
# The keycloak-http Service has no TLS listener, so these backend calls travel as plaintext HTTP
343+
# between pods, as keycloak-configure's own admin calls to the same address already do.
344+
#
345+
# Override only for a KeyCloak that does not live at the standard in-cluster address. Setting it
346+
# to an empty string restores full OIDC discovery against the external issuer.
347+
keycloakInternalUrl: "http://keycloak-http.cozy-keycloak.svc:8080/realms/cozy"
338348
# Gateway API configuration
339349
gateway:
340350
# Enable Gateway API support across the platform. When true:

‎packages/system/dashboard/tests/gatekeeper_test.yaml‎

Lines changed: 131 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,34 @@ tests:
5151
path: spec.template.spec.containers[0].ports[0].containerPort
5252
value: 8000
5353

54+
# authentication.oidc.enabled defaults to false while keycloakInternalUrl now
55+
# defaults to the in-cluster Service, so THIS pair — OIDC off, internal URL
56+
# set — is what most clusters actually render. The internal-URL block is
57+
# nested inside the oidc-enabled guard, so it cannot leak into the
58+
# token-proxy branch today; pin that, because a refactor which hoists the
59+
# block out would otherwise ship silently.
60+
- it: oidc-enabled=false ignores a set keycloak-internal-url
61+
set:
62+
_cluster:
63+
root-host: example.org
64+
oidc-enabled: "false"
65+
keycloak-internal-url: http://keycloak-http.cozy-keycloak.svc:8080/realms/cozy
66+
tokenProxy:
67+
image: example.invalid/token-proxy:test
68+
asserts:
69+
- equal:
70+
path: spec.template.spec.containers[0].name
71+
value: token-proxy
72+
- notContains:
73+
path: spec.template.spec.containers[0].args
74+
content: --skip-oidc-discovery
75+
- notContains:
76+
path: spec.template.spec.containers[0].args
77+
content: --redeem-url=http://keycloak-http.cozy-keycloak.svc:8080/realms/cozy/protocol/openid-connect/token
78+
- notContains:
79+
path: spec.template.spec.containers[0].args
80+
content: --oidc-jwks-url=http://keycloak-http.cozy-keycloak.svc:8080/realms/cozy/protocol/openid-connect/certs
81+
5482
- it: oidc-enabled=true renders auth-proxy with /ping probes and no startupProbe
5583
set:
5684
_cluster:
@@ -74,3 +102,106 @@ tests:
74102
- equal:
75103
path: spec.template.spec.containers[0].ports[0].containerPort
76104
value: 8000
105+
106+
# keycloak-internal-url arrives on the _cluster channel from
107+
# authentication.oidc.keycloakInternalUrl in packages/core/platform. Its
108+
# default is the in-cluster Keycloak Service, so the value asserted below is
109+
# what a stock cluster ships: oauth2-proxy skips OIDC discovery and talks to
110+
# the Service for token/jwks/userinfo/logout, which keeps the dashboard
111+
# independent of the root ingress serving keycloak.<host> at startup. The
112+
# browser-facing URLs (redirect, login) must stay on the external host —
113+
# routing them to a cluster-internal address would send the user's browser to
114+
# an unreachable URL.
115+
- it: default in-cluster keycloak URL skips discovery and keeps browser URLs external
116+
set:
117+
_cluster:
118+
root-host: example.org
119+
oidc-enabled: "true"
120+
keycloak-internal-url: http://keycloak-http.cozy-keycloak.svc:8080/realms/cozy
121+
asserts:
122+
- contains:
123+
path: spec.template.spec.containers[0].args
124+
content: --skip-oidc-discovery
125+
- contains:
126+
path: spec.template.spec.containers[0].args
127+
content: --redeem-url=http://keycloak-http.cozy-keycloak.svc:8080/realms/cozy/protocol/openid-connect/token
128+
- contains:
129+
path: spec.template.spec.containers[0].args
130+
content: --oidc-jwks-url=http://keycloak-http.cozy-keycloak.svc:8080/realms/cozy/protocol/openid-connect/certs
131+
- contains:
132+
path: spec.template.spec.containers[0].args
133+
content: --validate-url=http://keycloak-http.cozy-keycloak.svc:8080/realms/cozy/protocol/openid-connect/userinfo
134+
- contains:
135+
path: spec.template.spec.containers[0].args
136+
content: --backend-logout-url=http://keycloak-http.cozy-keycloak.svc:8080/realms/cozy/protocol/openid-connect/logout?id_token_hint={id_token}
137+
- contains:
138+
path: spec.template.spec.containers[0].args
139+
content: --redirect-url=https://dashboard.example.org/oauth2/callback
140+
- contains:
141+
path: spec.template.spec.containers[0].args
142+
content: --login-url=https://keycloak.example.org/realms/cozy/protocol/openid-connect/auth
143+
- contains:
144+
path: spec.template.spec.containers[0].args
145+
content: --oidc-issuer-url=https://keycloak.example.org/realms/cozy
146+
147+
- it: an explicit keycloak-internal-url override wins over the default
148+
set:
149+
_cluster:
150+
root-host: example.org
151+
oidc-enabled: "true"
152+
keycloak-internal-url: http://keycloak.example.test:8080/realms/other
153+
asserts:
154+
- contains:
155+
path: spec.template.spec.containers[0].args
156+
content: --skip-oidc-discovery
157+
- contains:
158+
path: spec.template.spec.containers[0].args
159+
content: --redeem-url=http://keycloak.example.test:8080/realms/other/protocol/openid-connect/token
160+
- contains:
161+
path: spec.template.spec.containers[0].args
162+
content: --oidc-jwks-url=http://keycloak.example.test:8080/realms/other/protocol/openid-connect/certs
163+
- contains:
164+
path: spec.template.spec.containers[0].args
165+
content: --validate-url=http://keycloak.example.test:8080/realms/other/protocol/openid-connect/userinfo
166+
- notContains:
167+
path: spec.template.spec.containers[0].args
168+
content: --redeem-url=http://keycloak-http.cozy-keycloak.svc:8080/realms/cozy/protocol/openid-connect/token
169+
170+
# The escape hatch: an operator who sets keycloakInternalUrl back to "" gets
171+
# the historical shape — full OIDC discovery against the external issuer, no
172+
# backend URLs pinned. It must keep working, so pin it.
173+
#
174+
# --skip-oidc-discovery carries the whole meaning here. All five in-cluster
175+
# args live inside one {{- if $keycloakInternalUrl }} block in the template,
176+
# so its absence structurally guarantees the absence of the other four; the
177+
# notContains assertions below then pin the regression that actually threatens
178+
# this branch — the in-cluster default leaking back in. Asserting an arg count
179+
# instead would couple this OIDC test to every unrelated gatekeeper flag.
180+
- it: empty keycloak-internal-url restores external OIDC discovery
181+
set:
182+
_cluster:
183+
root-host: example.org
184+
oidc-enabled: "true"
185+
keycloak-internal-url: ""
186+
asserts:
187+
- notContains:
188+
path: spec.template.spec.containers[0].args
189+
content: --skip-oidc-discovery
190+
- contains:
191+
path: spec.template.spec.containers[0].args
192+
content: --oidc-issuer-url=https://keycloak.example.org/realms/cozy
193+
- contains:
194+
path: spec.template.spec.containers[0].args
195+
content: --backend-logout-url=https://keycloak.example.org/realms/cozy/protocol/openid-connect/logout?id_token_hint={id_token}
196+
- notContains:
197+
path: spec.template.spec.containers[0].args
198+
content: --redeem-url=http://keycloak-http.cozy-keycloak.svc:8080/realms/cozy/protocol/openid-connect/token
199+
- notContains:
200+
path: spec.template.spec.containers[0].args
201+
content: --oidc-jwks-url=http://keycloak-http.cozy-keycloak.svc:8080/realms/cozy/protocol/openid-connect/certs
202+
- notContains:
203+
path: spec.template.spec.containers[0].args
204+
content: --validate-url=http://keycloak-http.cozy-keycloak.svc:8080/realms/cozy/protocol/openid-connect/userinfo
205+
- notContains:
206+
path: spec.template.spec.containers[0].args
207+
content: --backend-logout-url=http://keycloak-http.cozy-keycloak.svc:8080/realms/cozy/protocol/openid-connect/logout?id_token_hint={id_token}

0 commit comments

Comments
 (0)