Skip to content

Commit 4bf9581

Browse files
authored
fix(kubernetes): render the aggregation-layer UID header for tenant OIDC (#3539)
## What this PR does Tenant clusters with `spec.oidc.mode: System` or `CustomConfig` authenticate users fine, but a UID does not survive the hop to an aggregated API server. The tenant kube-apiserver publishes `requestheader-uid-headers` into the `extension-apiserver-authentication` ConfigMap only when the flag is set, and the chart never set it. The aggregation layer still sends `X-Remote-Uid`, but the extension server trusts only the header names published in that ConfigMap, so it ignores the one it gets and the call arrives without a UID. The field documentation even told operators to add the flag by hand via `controlPlane.apiServer.extraArgs`. It forwards a UID rather than creating one. ServiceAccount tokens carry one, and for those the aggregation hop starts working. Users authenticated by `spec.oidc.mode: System` still have none, because the generated AuthenticationConfiguration maps `username` and `groups` only; giving them one means mapping a claim, which changes the identity of users who have none today, and is tracked separately. The chart now renders `--requestheader-uid-headers=X-Remote-Uid` on v1.32 and newer, and adds `--feature-gates=RemoteRequestHeaderUID=true` on v1.32, where the gate is still alpha and the apiserver rejects the flag without it. From v1.33 the gate is beta and on by default, so only the header flag is rendered. v1.31 gets nothing: the flag does not exist there, and writing it by hand stops the apiserver from starting. `--emulated-version` in the same passthrough moves that decision, because the gate resolves through the versioned spec at the emulated version. Rendering the gate on every version from v1.32 rather than on v1.32 alone was the alternative, and it would remove the migration step entirely, because an opt-out would then drop two entries instead of one and move the entry count by itself. It loses on what it leaves behind: a redundant gate term on every cluster, which stops the apiserver from starting on the day the gate is retired from the known list. The design also prefers rendering over failing for the same reason the fail branches are narrow, since a cluster stranded on a HelmRelease that no longer converges is a worse outcome than one that never gets the UID fix. The chart reads an operator entry the way the apiserver reads it: header list elements matched whole and untrimmed and folded as `EqualFold` folds them, gate names compared exactly, gate values resolved as `strconv.ParseBool` resolves them, `AllAlpha` and `AllBeta` blankets honoured, an empty value treated as no entry. On v1.32 an entry that says nothing about `RemoteRequestHeaderUID` gets a second one carrying the operator's own terms plus `RemoteRequestHeaderUID=true`. Carrying those terms into the merged copy is what makes it land correctly whether the tenant control plane collapses duplicate flags into the last one or hands both to the apiserver, which merges them in the order it reads them. Tests cover the render matrix over mode, version and operator input: every fail path with its exact message, the ownership and merge paths, duplicate entries in both orders, blanket gates, empty values, and case variants on the header name and on the gate value. The latest-version tenant lane creates its tenant with `spec.oidc.mode: System` and asserts the flag on the rendered KamajiControlPlane, so the tenant apiserver starts with the flag in CI. Nothing asserts the UID arriving at an aggregated apiserver; that needs an extension server inside the tenant and is not in this change. The layer where the count-keyed reset actually bites is the tenant Deployment's arguments, and asserting there needs a live tenant control plane, so it is not in this change either. The rule this follows is that a configuration which renders and boots today keeps rendering. Both flags are new, so an `extraArgs` shape reaching this code was rendering before it existed, and stranding a running tenant on a HelmRelease that no longer converges is a worse outcome than one that never gets the fix. A few shapes do stop rendering, and reaching any of them takes writing `--requestheader-uid-headers` into the passthrough by hand, attempting exactly what the chart now does and getting it wrong; in each the kube-apiserver fails option validation and never starts, so nothing that was working is lost. `docs/oidc-tenant.md` covers these shapes. One known limitation is in there and worth naming here: on v1.33 and newer, adding the documented opt-out is count-neutral in `extraArgs`, and so is removing it again, and on v1.33 and v1.34 adding or removing `--emulated-version=1.31`. The pinned Kamaji build rewrites a running apiserver's arguments only when that count changes and otherwise keeps every flag that has left the list, so either the header flag stays beside a now-disabled gate or the disabled gate stays beside the header flag, and the apiserver refuses to start. Changing the entry count in the same edit is the mitigation, and the fix belongs to the pin rather than to this chart, tracked in #3541. The paths that need no operator edit are clear: turning the feature on for a default tenant moves the count, and a tenant already carrying the opt-out renders byte-identically before and after. Out of scope: `mode: None` still publishes no UID header even though ServiceAccount tokens carry UIDs. That is pre-existing, and widening the condition changes the render for every default tenant. Tracked separately, along with the claim mapping above. Closes #3377 ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change Checked the trigger map against the diff: no package added or removed, no schema field, enum or default changed (description text only), no ApplicationDefinition surface touched, no hack/ layout moved. The managed-apps reference page picks up the new field description through release-time README regeneration. ### Release note ```release-note fix(kubernetes): tenant clusters with `spec.oidc.mode` set to `System` or `CustomConfig` now render `--requestheader-uid-headers=X-Remote-Uid` on the tenant kube-apiserver, plus the `RemoteRequestHeaderUID` feature gate on v1.32, so that an aggregated API server can read the caller's UID out of the request headers. Without that flag the tenant apiserver never publishes `requestheader-uid-headers`, and an aggregated API server ignores the UID header even for callers that have one, ServiceAccount tokens among them. It forwards a UID rather than creating one: users authenticated by `spec.oidc.mode: System` still have none, because the generated AuthenticationConfiguration maps `username` and `groups` only, and that is tracked separately. Adding the flag by hand through `controlPlane.apiServer.extraArgs` is no longer necessary, and an entry of your own is respected: the chart steps aside when yours already carries `X-Remote-Uid` or already enables the gate, merges `RemoteRequestHeaderUID=true` into a copy of a v1.32 `--feature-gates` entry that says nothing about the gate, and renders neither flag when you switch the gate off. The render fails where a `--requestheader-uid-headers` list of your own would stop the kube-apiserver from starting anyway: one that omits `X-Remote-Uid`, one with an empty element, and a list of your own while the gate is off. On v1.33 and newer there is a migration step, and it applies in both directions: the pinned Kamaji build rewrites a running apiserver's arguments only when the number of `extraArgs` entries changes, and otherwise keeps every flag that has left the list. Adding the documented opt-out keeps that number the same, so the header flag already on the Deployment stays beside the now-disabled gate. Removing the opt-out later keeps it the same too, so the disabled gate stays on the Deployment beside the header flag the chart renders again, although your spec no longer shows it. Adding or removing `--emulated-version=1.31` has the same shape both ways on v1.33 and v1.34. The apiserver refuses to start in each case, so change the entry count in the same edit. ```
2 parents 08bbed3 + 187e4ce commit 4bf9581

12 files changed

Lines changed: 2500 additions & 56 deletions

File tree

‎api/apps/v1alpha1/kubernetes/types.go‎

Lines changed: 2 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎docs/oidc-tenant.md‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -126,6 +126,66 @@ in CustomConfig mode: the issuer and clientId are inside the
126126
operator-supplied config and are not knowable to the chart. Distribute
127127
the OIDC kubeconfig out-of-band.
128128

129+
## Aggregated API servers
130+
131+
Authenticating a user is only half of the path. A call that lands on an aggregated API server — an `APIService` backed by an extension server rather than by the core apiserver — reaches it over the aggregation layer, which forwards the caller's identity in request headers. The UID travels in `X-Remote-Uid`, which the aggregation layer sends whenever the `RemoteRequestHeaderUID` gate is on, and the extension server trusts that header only when the tenant kube-apiserver has published it under `requestheader-uid-headers` in the `extension-apiserver-authentication` ConfigMap. Without that key the extension server ignores the header, and the call is served as if the caller had no UID, even though `kubectl` against the core apiserver works fine. Anything on the far side that reads the UID, an authorizer or an audit record, sees an empty value.
132+
133+
Carrying the UID across that hop is a property of the aggregation layer, not of OIDC: a caller authenticated by any means loses their UID the same way. The chart nonetheless renders these flags only when `spec.oidc.mode` is not `None`, which keeps the blast radius to clusters opting into a new feature. A cluster that has never left `mode: None` therefore still loses the UID at the extension server.
134+
135+
Be clear about who this helps. The flag forwards a UID the caller already has; it does not create one. ServiceAccount tokens carry a UID, and so do other identities that set one, and for those the aggregation hop starts working. **OIDC users authenticated through `spec.oidc.mode: System` do not get a UID from this change**, because the AuthenticationConfiguration the chart generates maps `username` and `groups` only. With no `claimMappings.uid` and no CEL expression for it, upstream returns an empty UID (`plugin/pkg/authenticator/token/oidc/oidc.go`, `getUID`, read at v0.35.0), so there is nothing for the header to carry. Giving those users a UID means mapping a claim, which changes the identity of users who currently have none, and is tracked separately rather than done here. In `CustomConfig` mode an operator-supplied configuration that maps `uid` does produce one, and then this change carries it.
136+
137+
Whenever `spec.oidc.mode` is not `None`, the chart renders the flags that publish it, alongside `--authentication-config`:
138+
139+
| `spec.version` | rendered |
140+
| --- | --- |
141+
| `v1.31` | nothing — the flag does not exist upstream, and an unknown flag stops the apiserver from starting |
142+
| `v1.32` | `--requestheader-uid-headers=X-Remote-Uid` plus `--feature-gates=RemoteRequestHeaderUID=true` |
143+
| `v1.33` and newer | `--requestheader-uid-headers=X-Remote-Uid` |
144+
145+
The gate is rendered on `v1.32` only, where `RemoteRequestHeaderUID` is still alpha and the apiserver rejects the flag while the gate is off. From `v1.33` the gate is beta and on by default, so rendering it would buy nothing and would collide with a gate list of your own.
146+
147+
The version in that table is the effective one, not the image tag. `--emulated-version` in `controlPlane.apiServer.extraArgs` moves it: the gate resolves through the versioned spec at the emulated version, so `--emulated-version=1.32` puts `RemoteRequestHeaderUID` back to alpha-and-off on a `v1.35` cluster, and the chart follows by rendering the gate alongside the header flag. Below `1.32` the gate has no spec at all, `featureGate.Set()` reports it PreAlpha and refuses to enable it, so the chart renders neither flag.
148+
149+
Two values the chart ignores are still not harmless. An element naming another component, `--emulated-version=wardle=1.2`, says nothing about the tenant kube-apiserver, but the apiserver rejects a component it has not registered, `component not registered: wardle`, and does not start. A value whose minor is above your `spec.version` is ignored too, because emulation only moves the effective version down, and there the apiserver checks against a range rather than a ceiling: it rejects one above its own binary or below the floor it still supports, naming the range it accepts. In both cases the control plane will not start whatever the chart renders.
150+
151+
Adding either flag through `controlPlane.apiServer.extraArgs` by hand is no longer necessary, and an existing cluster that does keeps working: the chart skips its own copy when your entry already carries `X-Remote-Uid` or already enables the gate.
152+
153+
Where your entry does not already satisfy the requirement, the chart works around it rather than failing. On `v1.32` a `--feature-gates` entry of yours that says nothing about `RemoteRequestHeaderUID` gets a second entry rendered after it, carrying your terms plus `RemoteRequestHeaderUID=true`. That decides the gate whether the tenant control plane collapses duplicate flags into the last one or hands both to the apiserver, which merges them in the order it reads them; carrying your terms into the merged entry is what makes the first reading safe.
154+
155+
If you turned the gate off on purpose, with `RemoteRequestHeaderUID=false` or a blanket `AllAlpha=false` on `v1.32` and `AllBeta=false` from `v1.33`, the chart renders neither flag and the render is byte-identical to the one before this feature existed. You keep losing the UID at the aggregation layer, which is what the opt-out asks for. On a cluster that is already running, adding the opt-out and removing it again each need one more edit alongside, below.
156+
157+
The running control plane is a separate matter, and on `v1.33` and newer this needs care. The tenant control plane rewrites the apiserver's arguments from scratch only when the *number* of `extraArgs` entries changes. Otherwise it starts from the arguments already on the running Deployment and lays the new list over them, so a flag that has left `extraArgs` stays behind. That is behaviour of the Kamaji build this repo pins in `packages/system/kamaji` rather than a property of Kubernetes, and upstream has since changed it. It bites in both directions:
158+
159+
- Adding the opt-out adds one entry while the chart drops one, so the count does not change, and the `--requestheader-uid-headers=X-Remote-Uid` already on the running Deployment stays there beside your now-disabled gate.
160+
- Removing the opt-out entry to turn the feature back on drops one entry while the chart adds one. The count again does not change, so your `RemoteRequestHeaderUID=false` or `AllBeta=false` stays on the Deployment beside the header flag the chart now renders, although your CR no longer shows it.
161+
- `--emulated-version=1.31` has the same shape both ways on `v1.33` and `v1.34`. Adding it leaves the header flag behind at an emulated version where the gate is off; removing it leaves the emulation behind beside the header flag the chart now renders. From `v1.35` that value is below the oldest version the apiserver can emulate, and it refuses to start on it in any case.
162+
163+
The apiserver refuses to start in each of these combinations.
164+
165+
Change the entry count in the same edit to force the rewrite, whichever way you are going: any second flag you were going to add or drop anyway will do, or set `spec.oidc.mode: None`, let it converge, and turn it back on. On `v1.32` the count changes by itself as long as the chart renders the gate too, because it then drops or adds two entries rather than one; where the gate is already yours, the chart renders the header flag alone and the rule above applies as it does on `v1.33`. That migration is the whole mitigation, and it stays necessary for as long as the pinned build keys the rewrite on the entry count; the fix belongs to that pin rather than to this chart, and is tracked in cozystack/cozystack#3541. The paths that need no operator edit are clear: turning the feature on for a default tenant moves the count, and a tenant already carrying the opt-out renders byte-identically before and after.
166+
167+
The render still fails where a uid-headers list of your own describes a kube-apiserver that already refuses to start: a list that omits `X-Remote-Uid`, a list with an element that is empty once trimmed, which a trailing comma leaves behind, and a list alongside a gate setting that turns `RemoteRequestHeaderUID` off. The chart cannot rescue any of them by withholding its own flag, because the flag the apiserver rejects is yours. The error message names the fix, which beats a crash-looping control plane.
168+
169+
Outside those shapes, nothing that renders today stops rendering: these flags are new, so every other `extraArgs` shape reaching this code was rendering before it existed. Setups whose gate is genuinely already on are left alone, whether you named it explicitly or switched it on with a blanket `AllAlpha=true`.
170+
171+
The gate check reads your entry the way the apiserver does.
172+
173+
- An element may name a component, because `--feature-gates` is a colon-separated multimap: `kube:RemoteRequestHeaderUID=false` is the same opinion as `RemoteRequestHeaderUID=false`, while an element naming another component is not about the tenant kube-apiserver and is ignored, on the same terms as an unregistered component in `--emulated-version` above.
174+
- The chart writes its own term in whichever spelling you used, because the apiserver rejects a value mixing the bare and `kube:` forms.
175+
- Past that it is one `Name=value` element at a time, spaces trimmed on both sides, and the value resolved the way `strconv.ParseBool` resolves it, so `1`, `t` and `T` mean the same as `true`, and `0`, `f` and `F` the same as `false`.
176+
- Blanket settings count, because the apiserver applies them to every gate you have not named explicitly. On `v1.32` a `--feature-gates=AllAlpha=true` genuinely switches `RemoteRequestHeaderUID` on, so the chart treats it as yours and leaves it alone; from `v1.33` the mirror image applies, and `AllBeta=false` reads as your opt-out.
177+
- Naming the gate wins either way: `AllAlpha=true,RemoteRequestHeaderUID=false` is an opt-out on `v1.32`, and `AllBeta=false,RemoteRequestHeaderUID=true` leaves the gate on from `v1.33`, because an explicit entry beats a blanket one whichever order they appear in.
178+
- Gate names are compared exactly, as the apiserver looks them up, so `remoterequestheaderuid=true` is not an opinion about `RemoteRequestHeaderUID`; the apiserver would reject that spelling as an unrecognized gate anyway.
179+
- An entry with an empty value is not an entry as far as the chart is concerned: it reads the last entry that carries a value.
180+
181+
Do not rely on an empty value to cancel an earlier entry, and this has nothing to do with OIDC: the tenant control plane collapses `extraArgs` by flag name, again a property of the pinned build rather than of Kubernetes, and writes an empty value back as a bare flag, so a trailing `--requestheader-uid-headers=` reaches the apiserver as `--requestheader-uid-headers`, which it refuses to start on. Drop the entry you do not want instead of emptying it.
182+
183+
Quotes stop the chart from reading your entry at all. `--requestheader-uid-headers` and `--emulated-version` are both CSV-parsed by the apiserver, which strips a surrounding pair of double quotes before it sees the value, and the chart does not reproduce that. Rather than act on a value it may be reading differently, it renders neither flag, exactly as it did before these flags existed. That applies to the entry the chart reads, the last one on that flag that carries a value; quotes on an earlier entry that a later one replaces change nothing. Drop the quotes if you want the chart to read the entry.
184+
185+
The uid-headers list is read without trimming, because the apiserver does not trim it either: `--requestheader-uid-headers=X-Remote-Uid , Other` does not contain `X-Remote-Uid` as far as the apiserver is concerned, and the chart treats it the same way rather than accepting a value the apiserver will reject. The emptiness check is the one that trims, again because the apiserver's does: in `X-Remote-Uid, ` the second element is empty once trimmed, and the apiserver refuses the whole list with `empty value in "requestheader-uid-headers"` although `X-Remote-Uid` is in it.
186+
187+
On `v1.31` the chart renders neither flag, and the UID guards above do not run, so a uid-headers or `--feature-gates` entry of yours passes through untouched. The unrelated `--oidc-*` and `--authentication-config` collision guards still apply there as everywhere else. Aggregated API servers cannot see the caller's UID on `v1.31`; move the cluster to `v1.32` or newer if you need them to. Writing `--requestheader-uid-headers` yourself does not work around it — the flag does not exist on `v1.31`, and an unknown flag stops the apiserver from starting.
188+
129189
## Users and RBAC
130190

131191
`users[]` is a flat list. Each entry produces a single

‎hack/e2e-chainsaw/_lib/run-kubernetes.sh‎

Lines changed: 22 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -139,10 +139,27 @@ EOF
139139

140140
cozy_oidc_bindings() {
141141
local test_name="$1"
142-
kubectl --kubeconfig "tenantkubeconfig-${test_name}" get clusterrolebindings \
142+
local bindings
143+
bindings=$(kubectl --kubeconfig "tenantkubeconfig-${test_name}" get clusterrolebindings \
143144
--selector="app.kubernetes.io/managed-by=cozystack-oidc,app.kubernetes.io/instance=kubernetes-${test_name}" \
144-
-o 'jsonpath={range .items[*]}{.subjects[0].name}{"\t"}{.roleRef.name}{"\n"}{end}' |
145-
sort
145+
-o 'jsonpath={range .items[*]}{.subjects[0].name}{"\t"}{.roleRef.name}{"\n"}{end}') || return 1
146+
printf '%s' "${bindings}" | sort
147+
}
148+
149+
cozy_assert_oidc_apiserver_flags() {
150+
local release="$1"
151+
local extra_args
152+
153+
# Captured because a pipeline is single-use: a second grep on it reads nothing.
154+
extra_args=$(kubectl -n tenant-test get kamajicontrolplane "${release}" \
155+
-o jsonpath='{.spec.apiServer.extraArgs}')
156+
printf '%s\n' "${extra_args}" |
157+
grep -qF -- '--authentication-config=/etc/kubernetes/authentication-config/config.yaml'
158+
# --feature-gates=RemoteRequestHeaderUID=true is deliberately not asserted:
159+
# the chart renders it only on v1.32, while this lane takes the highest
160+
# version in the map, which is above that.
161+
printf '%s\n' "${extra_args}" |
162+
grep -qF -- '--requestheader-uid-headers=X-Remote-Uid'
146163
}
147164

148165
cozy_assert_oidc_system() {
@@ -154,9 +171,7 @@ cozy_assert_oidc_system() {
154171
kubectl -n tenant-test wait job "${release}-oidc-bootstrap" \
155172
--for=condition=complete --timeout=1m
156173

157-
kubectl -n tenant-test get kamajicontrolplane "${release}" \
158-
-o jsonpath='{.spec.apiServer.extraArgs}' |
159-
grep -qF -- '--authentication-config=/etc/kubernetes/authentication-config/config.yaml'
174+
cozy_assert_oidc_apiserver_flags "${release}"
160175

161176
authn_config=$(kubectl -n tenant-test get secret "${release}-oidc-authn-config" \
162177
-o jsonpath='{.data.config\.yaml}' | base64 -d)
@@ -206,9 +221,7 @@ cozy_switch_and_assert_oidc_custom_config() {
206221
kubectl -n tenant-test wait job "${release}-oidc-bootstrap" \
207222
--for=condition=complete --timeout=1m
208223

209-
kubectl -n tenant-test get kamajicontrolplane "${release}" \
210-
-o jsonpath='{.spec.apiServer.extraArgs}' |
211-
grep -qF -- '--authentication-config=/etc/kubernetes/authentication-config/config.yaml'
224+
cozy_assert_oidc_apiserver_flags "${release}"
212225

213226
authn_config=$(kubectl -n tenant-test get secret "${release}-oidc-authn-config" \
214227
-o jsonpath='{.data.config\.yaml}' | base64 -d)

0 commit comments

Comments
 (0)