Skip to content

Commit 1baf126

Browse files
authored
fix(tenant): inherit full ancestor label chain from parent namespace (cozystack#2912)
## Summary `packages/apps/tenant/templates/namespace.yaml` labels every tenant namespace with `tenant.cozystack.io/<ancestor>` for each of its ancestors. These labels drive the `<ancestor>-egress` `CiliumClusterwideNetworkPolicy` (`networkpolicy.yaml`): an ancestor tenant's pods may reach a namespace only if that namespace carries `tenant.cozystack.io/<ancestor>`. The chain was derived by splitting `.Release.Namespace`, which only ever reached one level up. A direct child of `tenant-root` got `tenant-root` (its parent namespace *is* `tenant-root`), but a grandchild — `tenant-ktj-htdev` created under `tenant-ktj` — got only `tenant-ktj` and dropped `tenant-root` entirely. Without `tenant.cozystack.io/tenant-root` on a nested namespace, the root ingress controller cannot reach the proxy service in a nested `kubernetes` tenant, surfacing as `upstream timed out (110) while connecting to upstream` in the nginx logs. ## Fix Derive the ancestor chain deterministically from the namespace name instead of from any cluster lookup. Every tenant descends from `tenant-root`, so that label is emitted unconditionally; the remaining ancestors are encoded in the namespace name itself, because `tenant.name` builds each child as `<parent-namespace>-<word>` — so every progressive dash-prefix of the name is a real ancestor namespace (`tenant-ktj-htdev` yields `tenant-ktj` and `tenant-ktj-htdev`). The logic lives in a new `tenant.ancestorTenantLabels` helper: ```yaml labels: {{- include "tenant.ancestorTenantLabels" $tenantName | nindent 4 }} ``` This needs no cluster state, renders identically offline (so it is unit-testable), and converges on every reconcile regardless of the order in which parent and child `HelmRelease`s reconcile — unlike a parent-namespace `lookup`, which helm-controller does not re-render on and which is ordering-dependent. The `ownerReference` `lookup` is kept but made non-fatal: during a real reconcile the parent namespace always exists (the tenant `HelmRelease` and its `cozystack-values` Secret live in it), so the `ownerReference` is always emitted; when the lookup is empty (offline / unit test) it is simply omitted so the labels still render. Migration `49` backfills the labels on existing clusters so they heal on upgrade regardless of per-tenant `HelmRelease` state (e.g. suspended or failed releases), and `migrations.targetVersion` is bumped to `50` so the runner iterates it. The migration derives the same chain from each namespace name, is idempotent (`kubectl label --overwrite`) and best-effort (per-namespace failures are tolerated so a transient apiserver error cannot abort the upgrade). ## Test plan - [x] `helm template packages/apps/tenant` at depth 1/2/3 — emits exactly the expected ancestor set at each depth: depth-1 `{tenant-root, tenant-ktj}`, depth-2 adds `tenant-ktj-htdev`, depth-3 adds `tenant-ktj-htdev-foo`. No missing, no stray labels. - [x] `helm unittest packages/apps/tenant` — new `tests/namespace_ancestor_labels_test.yaml` pins the chain at depths 1/2/3 plus a negative assertion; whole suite green. - [x] `bats hack/cozystack-version-stamp.bats` — migration `49` stamps the version through the shared `stamp_cozystack_version` helper (all 10 tests pass, including the guard that migrations >= 42 must use the helper). - [x] Migration `49` ancestor derivation verified under `/bin/sh` to match the helm output byte-for-byte, and dry-runs cleanly under `set -euo pipefail`. - [ ] Live cluster: create a 3-level tenant (`root -> ktj -> htdev`), verify `tenant-ktj-htdev` has all three `tenant.cozystack.io/*` labels. - [ ] Live cluster: create a `kubernetes` tenant under `htdev` with `addons.ingressNginx.hosts`, verify an external HTTP request reaches the tenant ingress controller via the root-ingress proxy (no `upstream timed out`). ## Closes - Closes cozystack#2810 - Closes cozystack#2171 ```release-note fix(tenant): tenant namespaces now carry the full `tenant.cozystack.io/<ancestor>` label chain (including `tenant-root`) at any nesting depth, restoring root-ingress reachability to nested tenants; a migration backfills the labels on existing clusters ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved tenant namespace label inheritance across multi-level tenant hierarchies, ensuring all ancestor tenant labels are consistently applied. * **New Features** * Standardized ancestor label generation to reliably include the full tenant chain for derived tenant namespaces. * **Tests** * Added regression coverage for ancestor label presence and absence at multiple hierarchy depths. * **Chores** * Added a migration to backfill missing ancestor labels for existing tenant namespaces during platform upgrades. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents c5ff217 + 5996404 commit 1baf126

5 files changed

Lines changed: 207 additions & 12 deletions

File tree

‎packages/apps/tenant/templates/_helpers.tpl‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,38 @@
5353
typing, so the "key absent" form is what distinguishes "unset"
5454
from explicit `false`.
5555
*/}}
56+
{{/*
57+
tenant.ancestorTenantLabels emits the full set of
58+
`tenant.cozystack.io/<ancestor>: ""` namespace labels for the tenant whose
59+
namespace name is passed as the single argument (e.g. "tenant-ktj-htdev").
60+
61+
Every tenant descends from tenant-root, so that label is always emitted.
62+
The remaining ancestors are encoded in the namespace name itself: tenant.name
63+
constructs each child namespace as `<parent-namespace>-<word>`, so every
64+
progressive dash-prefix of the name is a real ancestor namespace
65+
(tenant-ktj-htdev -> tenant-ktj, tenant-ktj-htdev). The last prefix is the
66+
tenant's own name, so its self-label is included too.
67+
68+
Deriving the chain from the name (rather than from a lookup of the parent
69+
namespace's labels) is deterministic: it needs no cluster state, renders
70+
identically offline, and converges on every reconcile regardless of the
71+
order in which parent and child HelmReleases reconcile. It replaces an
72+
earlier splitList over `.Release.Namespace` that only ever reached one level
73+
up and therefore dropped tenant-root for tenants at depth >= 2 — breaking the
74+
`<ancestor>-egress` CiliumClusterwideNetworkPolicy that grants an ancestor
75+
reachability to its descendants via the `tenant.cozystack.io/<ancestor>`
76+
namespace label.
77+
*/}}
78+
{{- define "tenant.ancestorTenantLabels" -}}
79+
{{- $parts := splitList "-" . -}}
80+
tenant.cozystack.io/tenant-root: ""
81+
{{- range $i, $v := $parts }}
82+
{{- if ne $i 0 }}
83+
{{ printf "tenant.cozystack.io/%s: \"\"" (join "-" (slice $parts 0 (add $i 1))) }}
84+
{{- end }}
85+
{{- end }}
86+
{{- end -}}
87+
5688
{{- define "tenant.gatewayEffective" -}}
5789
{{- if kindIs "invalid" .Values.gateway -}}
5890
false

‎packages/apps/tenant/templates/namespace.yaml‎

Lines changed: 13 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,14 @@
1-
{{/* Lookup for namespace uid (needed for ownerReferences) */}}
1+
{{/*
2+
Lookup the parent namespace, used below to stamp the child namespace's
3+
ownerReference uid. During a real helm-controller reconcile the parent
4+
namespace (.Release.Namespace) always exists, so the ownerReference is
5+
always emitted. The lookup returns empty only when rendering offline
6+
(helm template / helm unittest), where the ownerReference is simply omitted
7+
so the rest of the template — notably the ancestor labels below, which are
8+
derived purely from the namespace name and need no cluster state — still
9+
renders and can be unit-tested.
10+
*/}}
211
{{- $existingNS := lookup "v1" "Namespace" "" .Release.Namespace }}
3-
{{- if not $existingNS }}
4-
{{- fail (printf "error lookup existing namespace: %s" .Release.Namespace) }}
5-
{{- end }}
612

713
{{- if ne (include "tenant.name" .) "tenant-root" }}
814
{{/* Compute namespace values once for use in both Secret and labels */}}
@@ -75,13 +81,7 @@ metadata:
7581
name: {{ $tenantName }}
7682
{{- if hasPrefix "tenant-" .Release.Namespace }}
7783
labels:
78-
tenant.cozystack.io/{{ $tenantName }}: ""
79-
{{- $parts := splitList "-" .Release.Namespace }}
80-
{{- range $i, $v := $parts }}
81-
{{- if ne $i 0 }}
82-
tenant.cozystack.io/{{ join "-" (slice $parts 0 (add $i 1)) }}: ""
83-
{{- end }}
84-
{{- end }}
84+
{{- include "tenant.ancestorTenantLabels" $tenantName | nindent 4 }}
8585
{{/* Labels for network policies */}}
8686
namespace.cozystack.io/etcd: {{ $etcd | quote }}
8787
namespace.cozystack.io/ingress: {{ $ingress | quote }}
@@ -93,6 +93,7 @@ metadata:
9393
scheduler.cozystack.io/scheduling-class: {{ . | quote }}
9494
{{- end }}
9595
alpha.kubevirt.io/auto-memory-limits-ratio: "1.0"
96+
{{- if $existingNS }}
9697
ownerReferences:
9798
- apiVersion: v1
9899
blockOwnerDeletion: true
@@ -101,6 +102,7 @@ metadata:
101102
name: {{ .Release.Namespace }}
102103
uid: {{ $existingNS.metadata.uid }}
103104
{{- end }}
105+
{{- end }}
104106
---
105107
apiVersion: v1
106108
kind: Secret
Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
suite: tenant namespace ancestor labels
2+
# Regression cover for the tenant.cozystack.io/<ancestor> namespace labels
3+
# emitted by namespace.yaml. These labels drive the <ancestor>-egress
4+
# CiliumClusterwideNetworkPolicy (networkpolicy.yaml): an ancestor tenant is
5+
# allowed to reach any namespace carrying tenant.cozystack.io/<ancestor>, so
6+
# every tenant namespace MUST carry the label of each of its ancestors,
7+
# tenant-root included, at every depth.
8+
#
9+
# The labels are derived from the namespace name by tenant.ancestorTenantLabels
10+
# (see _helpers.tpl) with no cluster lookup, so helm-unittest can render them
11+
# offline — the ownerReference lookup is the only cluster-coupled part of
12+
# namespace.yaml and is skipped when the parent namespace lookup is empty.
13+
templates:
14+
- templates/namespace.yaml
15+
set:
16+
_cluster:
17+
root-host: example.com
18+
tests:
19+
- it: depth-1 tenant (direct child of root) carries tenant-root and its own label
20+
release:
21+
name: tenant-ktj
22+
namespace: tenant-root
23+
asserts:
24+
- documentIndex: 0
25+
isKind:
26+
of: Namespace
27+
- documentIndex: 0
28+
equal:
29+
path: metadata.name
30+
value: tenant-ktj
31+
- documentIndex: 0
32+
equal:
33+
path: metadata.labels["tenant.cozystack.io/tenant-root"]
34+
value: ""
35+
- documentIndex: 0
36+
equal:
37+
path: metadata.labels["tenant.cozystack.io/tenant-ktj"]
38+
value: ""
39+
40+
- it: depth-2 tenant carries the full ancestor chain including tenant-root
41+
release:
42+
name: tenant-htdev
43+
namespace: tenant-ktj
44+
asserts:
45+
- documentIndex: 0
46+
equal:
47+
path: metadata.name
48+
value: tenant-ktj-htdev
49+
- documentIndex: 0
50+
equal:
51+
path: metadata.labels["tenant.cozystack.io/tenant-root"]
52+
value: ""
53+
- documentIndex: 0
54+
equal:
55+
path: metadata.labels["tenant.cozystack.io/tenant-ktj"]
56+
value: ""
57+
- documentIndex: 0
58+
equal:
59+
path: metadata.labels["tenant.cozystack.io/tenant-ktj-htdev"]
60+
value: ""
61+
62+
- it: depth-3 tenant carries every ancestor from root down to itself
63+
release:
64+
name: tenant-foo
65+
namespace: tenant-ktj-htdev
66+
asserts:
67+
- documentIndex: 0
68+
equal:
69+
path: metadata.name
70+
value: tenant-ktj-htdev-foo
71+
- documentIndex: 0
72+
equal:
73+
path: metadata.labels["tenant.cozystack.io/tenant-root"]
74+
value: ""
75+
- documentIndex: 0
76+
equal:
77+
path: metadata.labels["tenant.cozystack.io/tenant-ktj"]
78+
value: ""
79+
- documentIndex: 0
80+
equal:
81+
path: metadata.labels["tenant.cozystack.io/tenant-ktj-htdev"]
82+
value: ""
83+
- documentIndex: 0
84+
equal:
85+
path: metadata.labels["tenant.cozystack.io/tenant-ktj-htdev-foo"]
86+
value: ""
87+
# A tenant must NOT claim descendants it does not have: the depth-3
88+
# namespace is not an ancestor of any deeper tenant, so no stray label.
89+
- documentIndex: 0
90+
notExists:
91+
path: metadata.labels["tenant.cozystack.io/tenant-ktj-htdev-foo-bar"]
Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
#!/bin/sh
2+
# Migration 49 --> 50
3+
# Backfill tenant.cozystack.io/<ancestor> labels on existing tenant namespaces.
4+
#
5+
# Tenant chart versions before this release derived the ancestor label chain by
6+
# splitting .Release.Namespace, which only ever reached one level up and dropped
7+
# the transitive ancestors (notably tenant.cozystack.io/tenant-root) for tenants
8+
# at depth >= 2. That broke the <ancestor>-egress CiliumClusterwideNetworkPolicy
9+
# (packages/apps/tenant/templates/networkpolicy.yaml): it grants an ancestor
10+
# reachability to a descendant namespace only when the descendant carries
11+
# tenant.cozystack.io/<ancestor>, so a missing tenant-root label left the root
12+
# ingress controller unable to reach nested tenants (upstream timeouts, #2810 /
13+
# #2171).
14+
#
15+
# The chart fix makes every tenant namespace self-compute its full ancestor
16+
# chain from its own name on the next reconcile. This migration backfills the
17+
# labels immediately so clusters heal without waiting on each tenant
18+
# HelmRelease to re-reconcile (e.g. suspended or failed releases).
19+
#
20+
# Best-effort by design: the worst case if a relabel is skipped is the
21+
# pre-existing missing-label bug this PR set out to fix, which is strictly no
22+
# worse than the status quo. A transient apiserver hiccup must therefore NOT
23+
# abort the whole platform upgrade, so the namespace list and per-namespace
24+
# relabel tolerate failure (|| true) rather than exiting non-zero. Idempotent:
25+
# kubectl label --overwrite only adds/refreshes the listed labels and never
26+
# removes others.
27+
28+
set -euo pipefail
29+
30+
# shellcheck source=lib/cozystack-version.sh
31+
. "$(dirname "$0")/lib/cozystack-version.sh"
32+
33+
# Every tenant lives in a namespace named tenant-<...>. tenant-root is the
34+
# implicit chain root and carries no tenant.cozystack.io/* label (the chart
35+
# omits the label block for it), so it is skipped. The `|| true` keeps a
36+
# transient apiserver error from aborting the upgrade.
37+
tenant_ns=$(kubectl get namespaces \
38+
-o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' \
39+
| grep '^tenant-' || true)
40+
41+
for ns in $tenant_ns; do
42+
[ "$ns" = "tenant-root" ] && continue
43+
44+
# Build the ancestor label list from the namespace name: every tenant
45+
# descends from tenant-root, then each progressive dash-prefix of the name
46+
# (skipping the bare "tenant" segment) is a real ancestor namespace, the last
47+
# being the tenant itself. e.g. tenant-ktj-htdev -> tenant-root, tenant-ktj,
48+
# tenant-ktj-htdev.
49+
labels="tenant.cozystack.io/tenant-root="
50+
acc=""
51+
IFS='-'
52+
for seg in $ns; do
53+
if [ -z "$acc" ]; then
54+
acc="$seg"
55+
else
56+
acc="$acc-$seg"
57+
fi
58+
[ "$acc" = "tenant" ] && continue
59+
labels="$labels tenant.cozystack.io/$acc="
60+
done
61+
unset IFS
62+
63+
echo "Backfilling ancestor labels on namespace $ns: $labels"
64+
# shellcheck disable=SC2086 # word splitting of $labels into flags is intended
65+
kubectl label namespace "$ns" $labels --overwrite || true
66+
done
67+
68+
# Stamp version via the shared helper so the cozystack-version ConfigMap keeps
69+
# the platform.cozystack.io/no-delete label (see lib/cozystack-version.sh).
70+
stamp_cozystack_version 50

‎packages/core/platform/values.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ sourceRef:
66
migrations:
77
enabled: false
88
image: ghcr.io/cozystack/cozystack/platform-migrations:v1.5.0@sha256:8bf61f17e99eb4a3365394e3d97ac52a1d68cc84aa104894ecb09de25721dc1d
9-
targetVersion: 49
9+
targetVersion: 50
1010
# Bundle deployment configuration
1111
bundles:
1212
system:

0 commit comments

Comments
 (0)