Skip to content

Commit d870b39

Browse files
committed
fix(kubernetes): render the aggregation-layer UID header for tenant OIDC
A tenant kube-apiserver publishes requestheader-uid-headers into the extension-apiserver-authentication ConfigMap only when the flag is set, and the chart never set it. Without that key the request-header authenticator on an aggregated apiserver does not know which header carries the UID and ignores it, so a caller who has a UID is served without one. Operators had to add the flag by hand through the controlPlane.apiServer.extraArgs escape hatch. This forwards a UID the caller already has and does not create one. ServiceAccount tokens carry a UID and gain from it. OIDC users authenticated through spec.oidc.mode: System do not: the AuthenticationConfiguration the chart generates maps username and groups only, and with no claimMappings.uid upstream returns an empty UID (plugin/pkg/authenticator/token/oidc/oidc.go, getUID, read at v0.35.0), so there is nothing to carry. Mapping a claim would give those users a UID and change the identity of users who have none today, which is a separate decision and is tracked separately. The half read rather than run: that an existing UID is lost without the published key comes from the controller writing the ConfigMap and the request-header authenticator consuming it, not from a live cluster. That OIDC users have no UID is unambiguous in the source named above. Render --requestheader-uid-headers=X-Remote-Uid whenever OIDC is on, plus --feature-gates=RemoteRequestHeaderUID=true on v1.32, where the gate is still alpha and the apiserver rejects the flag without it. v1.31 gets nothing: the flag does not exist there and an unknown flag is fatal to the apiserver. The deciding version is the effective one, not the image tag. --emulated-version in the same passthrough moves it down, and the gate resolves at the emulated version through featureSpecAtEmulationVersion (featureSpecAtEmulationAndMinCompatVersion by v1.35), so an emulated 1.32 needs the gate line even on a v1.35 cluster, and below 1.32 the gate is PreAlpha and cannot be enabled at all, which leaves nothing safe to render. Both flags are new, so every extraArgs shape reaching this code was rendering before it existed, and turning one of those into a render error would strand a running tenant on a HelmRelease that stops converging while its control plane carries on. The chart therefore yields to an operator entry that already satisfies the requirement, renders nothing at all when the operator turned RemoteRequestHeaderUID off, and otherwise merges its gate term into a copy of the operator's --feature-gates value. Carrying that value into the merged entry decides the gate whether the tenant control plane collapses duplicate flags into the last one or hands both to the apiserver, which merges them in the order it reads them. Operator entries are read the way the apiserver reads them: header list elements matched whole and untrimmed, folded as EqualFold folds them, and refused when one is empty after trimming, as checkForWhiteSpaceOnly refuses it; gate names compared exactly; gate values resolved as strconv.ParseBool resolves them; blanket AllAlpha and AllBeta honoured; an empty value treated as no entry at all. Each flag is judged by its last entry carrying a value. The apiserver's CSV reader strips quotes the chart does not parse, so quotes on that entry make the chart render nothing, while quotes on an earlier entry the collapse drops do not. The render fails only where a uid-headers list of the operator's describes a control plane that already refuses to start: one that omits X-Remote-Uid, one with an element that is empty after trimming, and any such list while the gate is off. Withholding the chart's own flag rescues none of them, because the flag the apiserver rejects is the operator's. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent 41ef4a6 commit d870b39

3 files changed

Lines changed: 2278 additions & 0 deletions

File tree

‎packages/apps/kubernetes/templates/cluster.yaml‎

Lines changed: 266 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -255,6 +255,51 @@ metadata:
255255
{{- if and $oidcEnabled (has "authentication-config" $userVolumeNames) }}
256256
{{- fail "controlPlane.apiServer.extraVolumes: volume name \"authentication-config\" is reserved by the chart for OIDC AuthenticationConfiguration when spec.oidc.mode is not None — rename your volume." }}
257257
{{- end }}
258+
{{- /*
259+
Aggregation-layer UID plumbing. What is rendered per version, why an
260+
opt-out renders nothing rather than failing, and the migration note live
261+
in docs/oidc-tenant.md. That table is the one to update; there is
262+
deliberately no copy of it here.
263+
264+
Anchors for the split, read at v1.32.13 and v1.35.6:
265+
staging/src/k8s.io/apiserver/pkg/features/kube_features.go for the gate's
266+
versioned spec, and .../server/options/authentication.go Validate() for
267+
the refusals the guards below mirror. The version that decides is the
268+
EFFECTIVE one, which --emulated-version moves down.
269+
270+
The merged gate entry carries the operator's own value, in their own
271+
prefix spelling, because that is correct whether duplicate flags collapse
272+
by name (it is last and wins) or are preserved (featureGate.Set() merges
273+
it into the earlier one).
274+
275+
Two things do depend on the collapse, pinned by packages/system/kamaji,
276+
whose Makefile takes the newest -edge tag on `make update`. Reading only
277+
the LAST entry of each flag cuts both ways: handed every copy the apiserver
278+
merges them, so an earlier explicit setting the chart never saw would reach
279+
the gate. Failing on a uid-headers list without X-Remote-Uid is the other:
280+
handed both copies pflag appends, and the union passes caseInsensitiveHas.
281+
The opt-out path is unaffected, since the chart emits nothing there.
282+
283+
Ownership is resolved after the arg loop because the last entry of each
284+
flag, and the interaction between them, are known only once every arg has
285+
been seen.
286+
*/}}
287+
{{- $uidHeaderUnsupported := false }}
288+
{{- $uidGateRequired := false }}
289+
{{- $emulatedKubeVersion := "" }}
290+
{{- $uidGateOptedOut := false }}
291+
{{- $uidHeaderArgSeen := "" }}
292+
{{- $uidHeaderQuoted := false }}
293+
{{- $emulatedQuoted := false }}
294+
{{- $uidHeaderConforms := false }}
295+
{{- $uidHeaderHasEmpty := false }}
296+
{{- $gateArgValue := "" }}
297+
{{- $gateUsesKubePrefix := false }}
298+
{{- $gateArgSeen := "" }}
299+
{{- $uidGateSetting := "" }}
300+
{{- $allBetaSetting := "" }}
301+
{{- $allAlphaSetting := "" }}
302+
{{- $uidArgs := list }}
258303
{{- /*
259304
Legacy `--oidc-*` flags in the user-supplied extraArgs are mutually
260305
exclusive with `--authentication-config`. The kube-apiserver refuses
@@ -276,6 +321,224 @@ metadata:
276321
{{- if hasPrefix "--authentication-config" $arg }}
277322
{{- fail (printf "controlPlane.apiServer.extraArgs contains %q while spec.oidc.mode is %q — the chart already injects --authentication-config= for OIDC. Drop your explicit flag and let spec.oidc.mode: CustomConfig own the AuthenticationConfiguration (customConfig.config or customConfig.secretRef.name), or leave spec.oidc.mode: None and keep managing the flag yourself." $arg $.Values.oidc.mode) }}
278323
{{- end }}
324+
{{- if hasPrefix "--requestheader-uid-headers=" $arg }}
325+
{{- /*
326+
An empty value is not an opinion: readAsCSV returns an empty slice and
327+
Validate() only inspects a non-empty list, so the chart reads the last
328+
entry carrying one. Where the chart renders nothing for that flag the
329+
operator's empty value survives the collapse and reaches the apiserver as
330+
a BARE flag, which takes the next sorted argument as its value, and
331+
Validate() refuses the result: not this code's to repair, and why docs
332+
tell operators to delete such an entry rather than empty it.
333+
334+
Only --flag=value is matched, and the space-separated spelling does not
335+
survive the collapse, which keys on "=". Written as two elements the flag
336+
comes back bare and takes whichever argument sorts after it as its value;
337+
written as one element the whole string becomes the key and reaches pflag
338+
as an unknown flag. The apiserver refuses to start either way.
339+
340+
Match whole ELEMENTS and do NOT trim them: caseInsensitiveHas compares with
341+
strings.EqualFold and pflag's CSV reader keeps surrounding spaces, so
342+
"X-Remote-Uid-Extra" and "X-Remote-Uid " both fail there. Being more
343+
permissive than upstream would skip the chart's flag for a list the
344+
apiserver rejects. The emptiness test is the one that trims, because
345+
checkForWhiteSpaceOnly does, and it refuses the list whatever else it holds.
346+
*/}}
347+
{{- $headerValue := trimPrefix "--requestheader-uid-headers=" $arg }}
348+
{{- /*
349+
Both flags this block reads are pflag StringSlice values, whose CSV reader
350+
strips a pair of surrounding double quotes before the apiserver sees them.
351+
Reproducing CSV here would be a second parser to keep correct, so a quoted
352+
value in the entry the chart reads means it cannot know what the apiserver
353+
will read and renders nothing at all, which is what it did before these
354+
flags existed. Quotes are tracked per flag and reset with each entry, like
355+
every other reading here: an earlier entry the collapse drops says nothing
356+
about the one that reaches the apiserver.
357+
358+
That split is the rule for the whole block: fail only where the operator's
359+
own entry would stop the apiserver from starting, and withhold silently
360+
where the cluster keeps running and only misses the fix.
361+
*/}}
362+
{{- if ne $headerValue "" }}
363+
{{- $uidHeaderQuoted = contains "\"" $headerValue }}
364+
{{- $uidHeaders := list }}
365+
{{- $uidHeaderHasEmpty = false }}
366+
{{- range $h := splitList "," $headerValue }}
367+
{{- $uidHeaders = append $uidHeaders (lower $h) }}
368+
{{- if eq (trim $h) "" }}
369+
{{- $uidHeaderHasEmpty = true }}
370+
{{- end }}
371+
{{- end }}
372+
{{- $uidHeaderArgSeen = $arg }}
373+
{{- $uidHeaderConforms = has "x-remote-uid" $uidHeaders }}
374+
{{- end }}
375+
{{- end }}
376+
{{- if hasPrefix "--feature-gates=" $arg }}
377+
{{- /*
378+
Values resolve as strconv.ParseBool resolves them (0/f/F, 1/t/T), names
379+
compare exactly, and both only after the component prefix is off: the flag
380+
is a colon-separated multimap parsed above featureGate.Set(), which is the
381+
layer `known[Feature(k)]` belongs to.
382+
383+
A blanket AllAlpha/AllBeta counts: setUnsetAlphaGates and setUnsetBetaGates
384+
skip a key an explicit entry already wrote, so the explicit one wins in
385+
either map-iteration order.
386+
*/}}
387+
{{- $gateValue := trimPrefix "--feature-gates=" $arg }}
388+
{{- if ne $gateValue "" }}
389+
{{- /*
390+
Per ARGUMENT, not per element: only the surviving --feature-gates entry
391+
decides the spelling, so a prefix seen in an earlier entry must not leak
392+
into a merge onto a later unprefixed one.
393+
*/}}
394+
{{- $gateUsesKubePrefix = false }}
395+
{{- $gateValues := dict }}
396+
{{- range $g := splitList "," $gateValue }}
397+
{{- /*
398+
--feature-gates is a colon-separated multimap: each element is
399+
component:name=value, and an element with no component belongs to kube.
400+
Reading the whole "kube:Name" as the gate name misses every prefixed
401+
spelling, and appending an unprefixed term next to a prefixed one is
402+
rejected outright -- componentGlobalsRegistry.Set() refuses a mix of the
403+
empty prefix and kube: in one flag (registry.go, read at v1.35.6).
404+
*/}}
405+
{{- $element := $g }}
406+
{{- if contains ":" $element }}
407+
{{- $component := trim (index (splitList ":" $element) 0) }}
408+
{{- /*
409+
Only a literal "kube" keys under the kube component. An empty component,
410+
as in ":Foo=true", keys under "" exactly like an unprefixed element, so
411+
answering it with a kube:-prefixed term would create the mix the registry
412+
rejects. A foreign component keeps its prefix and so never matches one of
413+
the gate names below.
414+
*/}}
415+
{{- if eq $component "kube" }}
416+
{{- $gateUsesKubePrefix = true }}
417+
{{- end }}
418+
{{- if or (eq $component "kube") (eq $component "") }}
419+
{{- $element = trimPrefix (printf "%s:" (index (splitList ":" $element) 0)) $element }}
420+
{{- end }}
421+
{{- end }}
422+
{{- $kv := splitList "=" $element }}
423+
{{- if eq (len $kv) 2 }}
424+
{{- $gname := trim (index $kv 0) }}
425+
{{- $gval := trim (index $kv 1) }}
426+
{{- if has $gval (list "1" "t" "T" "true" "TRUE" "True") }}
427+
{{- $gateValues = set $gateValues $gname "true" }}
428+
{{- else if has $gval (list "0" "f" "F" "false" "FALSE" "False") }}
429+
{{- $gateValues = set $gateValues $gname "false" }}
430+
{{- end }}
431+
{{- end }}
432+
{{- end }}
433+
{{- $gateArgSeen = $arg }}
434+
{{- $gateArgValue = $gateValue }}
435+
{{- $uidGateSetting = dig "RemoteRequestHeaderUID" "" $gateValues }}
436+
{{- $allBetaSetting = dig "AllBeta" "" $gateValues }}
437+
{{- $allAlphaSetting = dig "AllAlpha" "" $gateValues }}
438+
{{- end }}
439+
{{- end }}
440+
{{- if hasPrefix "--emulated-version=" $arg }}
441+
{{- /*
442+
Emulation moves the version the gate resolves at, and the flags below have
443+
to follow it rather than the image tag. featureSpecAtEmulationVersion
444+
(featureSpecAtEmulationAndMinCompatVersion by v1.35) walks the versioned
445+
specs down to the emulated version, so RemoteRequestHeaderUID
446+
is alpha-and-off at an emulated 1.32 on any binary, and PreAlpha below that,
447+
where featureGate.Set() refuses to enable it at all.
448+
449+
The value is a CSV of component=version, and an element naming no component
450+
is kube's. Anything else names another component and is none of our
451+
business. An unparseable version is left alone, the way an unparseable gate
452+
value is: the apiserver rejects it on its own terms.
453+
*/}}
454+
{{- $emulatedValue := trimPrefix "--emulated-version=" $arg }}
455+
{{- if ne $emulatedValue "" }}
456+
{{- /*
457+
Reset per argument, for the reason given at the --feature-gates parser.
458+
*/}}
459+
{{- $emulatedKubeVersion = "" }}
460+
{{- $emulatedQuoted = contains "\"" $emulatedValue }}
461+
{{- range $e := splitList "," $emulatedValue }}
462+
{{- $parts := splitList "=" (trim $e) }}
463+
{{- if eq (len $parts) 1 }}
464+
{{- $emulatedKubeVersion = trim (index $parts 0) }}
465+
{{- else if and (eq (len $parts) 2) (eq (trim (index $parts 0)) "kube") }}
466+
{{- $emulatedKubeVersion = trim (index $parts 1) }}
467+
{{- end }}
468+
{{- end }}
469+
{{- end }}
470+
{{- end }}
471+
{{- end }}
472+
{{- /*
473+
Emulation can only move the effective version down, so the lower of the two
474+
decides. A tenant asking to emulate something newer than its own image is
475+
rejected by the apiserver; reading the lower one keeps the chart from
476+
rendering a flag that version cannot take.
477+
*/}}
478+
{{- $chartMinorParts := splitList "." (trimPrefix "v" .Values.version) }}
479+
{{- $chartMinor := atoi (index $chartMinorParts 1) }}
480+
{{- $effectiveMinor := $chartMinor }}
481+
{{- if ne $emulatedKubeVersion "" }}
482+
{{- $emulatedParts := splitList "." (trimPrefix "v" $emulatedKubeVersion) }}
483+
{{- if and (ge (len $emulatedParts) 2) (regexMatch "^[0-9]+$" (index $emulatedParts 1)) }}
484+
{{- $emulatedMinor := atoi (index $emulatedParts 1) }}
485+
{{- if lt $emulatedMinor $effectiveMinor }}
486+
{{- $effectiveMinor = $emulatedMinor }}
487+
{{- end }}
488+
{{- end }}
489+
{{- end }}
490+
{{- /*
491+
The fail messages are migration instructions, so they name the version the
492+
decision was taken at, which is not the image tag when emulation moved it.
493+
*/}}
494+
{{- $versionForMessage := .Values.version }}
495+
{{- if and (ne $emulatedKubeVersion "") (ne $effectiveMinor $chartMinor) }}
496+
{{- $versionForMessage = printf "%s emulating %s" .Values.version $emulatedKubeVersion }}
497+
{{- end }}
498+
{{- $uidHeaderUnsupported = le $effectiveMinor 31 }}
499+
{{- $uidGateRequired = eq $effectiveMinor 32 }}
500+
{{- if eq $uidGateSetting "false" }}
501+
{{- $uidGateOptedOut = true }}
502+
{{- else if ne $uidGateSetting "true" }}
503+
{{- if $uidGateRequired }}
504+
{{- if eq $allAlphaSetting "false" }}
505+
{{- $uidGateOptedOut = true }}
506+
{{- end }}
507+
{{- else if eq $allBetaSetting "false" }}
508+
{{- $uidGateOptedOut = true }}
509+
{{- end }}
510+
{{- end }}
511+
{{- if not (or $uidHeaderUnsupported $uidHeaderQuoted $emulatedQuoted) }}
512+
{{- if $uidGateOptedOut }}
513+
{{- if ne $uidHeaderArgSeen "" }}
514+
{{- fail (printf "controlPlane.apiServer.extraArgs contains %q alongside %q, while spec.oidc.mode is %q on Kubernetes %s — the kube-apiserver rejects a --requestheader-uid-headers list while RemoteRequestHeaderUID is disabled, so the control plane would refuse to start. Drop one of the two: your header flag, or the gate setting that turns RemoteRequestHeaderUID off." $uidHeaderArgSeen $gateArgSeen $.Values.oidc.mode $versionForMessage) }}
515+
{{- end }}
516+
{{- else }}
517+
{{- if $uidHeaderHasEmpty }}
518+
{{- fail (printf "controlPlane.apiServer.extraArgs contains %q while spec.oidc.mode is %q — the kube-apiserver rejects a uid-headers list with an empty or whitespace-only element, so the control plane would refuse to start. Remove the empty element from your entry, or drop the entry and let the chart render the flag." $uidHeaderArgSeen $.Values.oidc.mode) }}
519+
{{- end }}
520+
{{- if and (ne $uidHeaderArgSeen "") (not $uidHeaderConforms) }}
521+
{{- fail (printf "controlPlane.apiServer.extraArgs contains %q while spec.oidc.mode is %q — the chart injects --requestheader-uid-headers=X-Remote-Uid so the authenticated user UID survives the aggregation layer, and the kube-apiserver rejects a uid-headers list that omits X-Remote-Uid. Add X-Remote-Uid to your entry, or drop the entry and let the chart render the flag." $uidHeaderArgSeen $.Values.oidc.mode) }}
522+
{{- end }}
523+
{{- if eq $uidHeaderArgSeen "" }}
524+
{{- $uidArgs = append $uidArgs "--requestheader-uid-headers=X-Remote-Uid" }}
525+
{{- end }}
526+
{{- if and $uidGateRequired (not (or (eq $uidGateSetting "true") (eq $allAlphaSetting "true"))) }}
527+
{{- if ne $gateArgValue "" }}
528+
{{- /*
529+
Match the operator's spelling: a prefixed value must get a prefixed term,
530+
or the apiserver rejects the mixed flag before any gate is read.
531+
*/}}
532+
{{- $ownTerm := "RemoteRequestHeaderUID=true" }}
533+
{{- if $gateUsesKubePrefix }}
534+
{{- $ownTerm = "kube:RemoteRequestHeaderUID=true" }}
535+
{{- end }}
536+
{{- $uidArgs = append $uidArgs (printf "--feature-gates=%s,%s" $gateArgValue $ownTerm) }}
537+
{{- else }}
538+
{{- $uidArgs = append $uidArgs "--feature-gates=RemoteRequestHeaderUID=true" }}
539+
{{- end }}
540+
{{- end }}
541+
{{- end }}
279542
{{- end }}
280543
{{- end }}
281544
spec:
@@ -299,6 +562,9 @@ spec:
299562
{{- end }}
300563
{{- if $oidcEnabled }}
301564
- --authentication-config=/etc/kubernetes/authentication-config/config.yaml
565+
{{- with $uidArgs }}
566+
{{- toYaml . | nindent 6 }}
567+
{{- end }}
302568
{{- end }}
303569
{{- if not (or .Values.controlPlane.apiServer.extraArgs $oidcEnabled) }} []{{ end }}
304570
extraVolumeMounts:

‎packages/apps/kubernetes/tests/apiserver_authentication_test.yaml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,10 @@ tests:
6262
###############################################
6363

6464
- it: passes apiServer extraArgs through to the KamajiControlPlane
65+
# Runs under the default oidc.mode=None, where the chart injects nothing of
66+
# its own and the passthrough is verbatim. The collision guards on
67+
# --authentication-config and --requestheader-uid-headers are gated on
68+
# oidc.mode != None and are covered in oidc_test.yaml.
6569
release:
6670
name: test-k8s
6771
namespace: tenant-test

0 commit comments

Comments
 (0)