Skip to content

Commit d3e3677

Browse files
fix(installer): host cozy-system labeler hook in kube-system
Re-introduces the pre-install/pre-upgrade Job hook (cozy-system-labeler) that was added by 3a87485/4754f57f and dropped by be0a668, but moves it into kube-system to dodge the PSA admission denial that broke the in-namespace approach: pods "cozy-system-labeler-..." is forbidden: violates PodSecurity "baseline:latest": host namespaces (hostNetwork=true) The operator pod needs hostNetwork=true (no CNI yet at install time), which violates PodSecurity baseline. cozy-system therefore needs enforce=privileged before the operator pod can be admitted. A labeler pod inside cozy-system would itself need hostNetwork=true and hit the same admission denial — Job admission denial means no pod is ever created, .status.failed never increments and backoffLimit never trips, helm waits for Complete=True until its budget elapses. `kube-system` is PSA-exempt on Talos (defaults.exemptions.namespaces in the apiserver PodSecurityConfiguration) and unlabeled on vanilla kubeadm/kind/k3s, so hostNetwork pods are admitted there. The hook's ServiceAccount is bound to a ClusterRole scoped via `resourceNames: ["cozy-system"]` to the one target namespace, keeping the cluster permission grant minimal. Also revert the e2e bats workaround (ddc429c) that pre-applied cozy-system manually before helm install: the labeler hook handles the labels in-chart now, so the bats can go back to plain `helm upgrade --install --namespace cozy-system --create-namespace ...`. Mirrors the corresponding fix on PR #2508 (fix/installer-namespace-bootstrap) so the two branches converge on the same resolution. Verified end-to-end in CI on this branch: cold install passes admission, operator pod comes up, all downstream E2E tests pass. Assisted-By: Claude <[email protected]> Signed-off-by: Myasnikov Daniil <[email protected]>
1 parent 3d7155c commit d3e3677

2 files changed

Lines changed: 125 additions & 20 deletions

File tree

‎hack/e2e-install-cozystack.bats‎

Lines changed: 3 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -28,30 +28,13 @@
2828
}
2929

3030
@test "Install Cozystack" {
31-
# Pre-create the cozy-system namespace with the labels the operator pod needs.
32-
# The chart no longer ships a Namespace resource (helm v3's --create-namespace
33-
# collides with chart-defined Namespace). On Talos clusters that enforce PSA
34-
# baseline-by-default, the cozystack-operator pod (hostNetwork=true) is
35-
# rejected unless the namespace is labelled enforce=privileged BEFORE the
36-
# pod is scheduled. A pre-install hook can't bootstrap that label because
37-
# the hook itself runs in the same namespace and faces the same PSA gate.
38-
# So apply the namespace from the caller (this bats / production install docs)
39-
# — same pattern as kube-prometheus-stack, argo-cd, cert-manager.
40-
kubectl apply -f - <<'EOF'
41-
apiVersion: v1
42-
kind: Namespace
43-
metadata:
44-
name: cozy-system
45-
labels:
46-
cozystack.io/system: "true"
47-
cozystack.io/deletion-protected: "true"
48-
pod-security.kubernetes.io/enforce: privileged
49-
EOF
50-
5131
# Install cozy-installer chart (operator installs CRDs on startup via --install-crds).
32+
# The chart's pre-install hook patches PSA + cozystack labels onto cozy-system
33+
# after --create-namespace bootstraps it.
5234
helm upgrade installer packages/core/installer \
5335
--install \
5436
--namespace cozy-system \
37+
--create-namespace \
5538
--set cozystackOperator.helmReleaseInterval=30s \
5639
--wait \
5740
--timeout 2m
Lines changed: 122 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,122 @@
1+
{{/*
2+
Pre-install/upgrade hook that labels the cozy-system namespace with the
3+
PodSecurity and cozystack identity labels.
4+
5+
Two chicken-and-egg constraints to thread:
6+
7+
1. helm v3 cannot adopt a namespace pre-created by --create-namespace
8+
because it lacks helm meta annotations; without --create-namespace,
9+
helm fails to write its release-secret because the namespace does
10+
not exist yet. The chart no longer ships a Namespace cozy-system
11+
resource; callers run helm with --create-namespace and the namespace
12+
is born bare (no labels).
13+
14+
2. The operator pod needs hostNetwork=true (no CNI yet at install time),
15+
which violates PodSecurity baseline. So cozy-system needs
16+
enforce=privileged before the operator pod is admitted. But a labeler
17+
pod inside cozy-system would itself need hostNetwork=true and hit
18+
the same admission denial. The hook therefore lives in kube-system,
19+
which is PSA-exempt on Talos (defaults.exemptions.namespaces in the
20+
apiserver PodSecurityConfiguration) and unlabeled on vanilla
21+
kubeadm/kind/k3s. The Job's SA is bound to a ClusterRole scoped via
22+
resourceNames to "cozy-system" only, so the cluster permission grant
23+
is minimal.
24+
*/}}
25+
---
26+
apiVersion: v1
27+
kind: ServiceAccount
28+
metadata:
29+
name: cozy-system-labeler
30+
namespace: kube-system
31+
annotations:
32+
helm.sh/hook: pre-install,pre-upgrade
33+
helm.sh/hook-weight: "-200"
34+
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
35+
---
36+
apiVersion: rbac.authorization.k8s.io/v1
37+
kind: ClusterRole
38+
metadata:
39+
name: cozy-system-labeler
40+
annotations:
41+
helm.sh/hook: pre-install,pre-upgrade
42+
helm.sh/hook-weight: "-180"
43+
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
44+
rules:
45+
- apiGroups: [""]
46+
resources: ["namespaces"]
47+
resourceNames: ["cozy-system"]
48+
verbs: ["get", "patch"]
49+
---
50+
apiVersion: rbac.authorization.k8s.io/v1
51+
kind: ClusterRoleBinding
52+
metadata:
53+
name: cozy-system-labeler
54+
annotations:
55+
helm.sh/hook: pre-install,pre-upgrade
56+
helm.sh/hook-weight: "-160"
57+
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
58+
subjects:
59+
- kind: ServiceAccount
60+
name: cozy-system-labeler
61+
namespace: kube-system
62+
roleRef:
63+
kind: ClusterRole
64+
name: cozy-system-labeler
65+
apiGroup: rbac.authorization.k8s.io
66+
---
67+
apiVersion: batch/v1
68+
kind: Job
69+
metadata:
70+
name: cozy-system-labeler
71+
namespace: kube-system
72+
annotations:
73+
helm.sh/hook: pre-install,pre-upgrade
74+
helm.sh/hook-weight: "-100"
75+
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
76+
spec:
77+
backoffLimit: 6
78+
template:
79+
spec:
80+
serviceAccountName: cozy-system-labeler
81+
restartPolicy: OnFailure
82+
# The hook runs BEFORE Cilium / kube-ovn / any CNI is installed, so the
83+
# cluster's nodes are NotReady-NoSchedule and have no pod network. Mirror
84+
# the cozystack-operator deployment's scheduling: hostNetwork=true so the
85+
# pod doesn't need CNI, plus the same tolerations the operator uses.
86+
hostNetwork: true
87+
tolerations:
88+
- key: "node.kubernetes.io/not-ready"
89+
operator: "Exists"
90+
- key: "node.kubernetes.io/unreachable"
91+
operator: "Exists"
92+
- key: "node.cilium.io/agent-not-ready"
93+
operator: "Exists"
94+
- key: "node.cloudprovider.kubernetes.io/uninitialized"
95+
operator: "Exists"
96+
containers:
97+
- name: kubectl
98+
image: alpine/k8s:1.32.0
99+
imagePullPolicy: IfNotPresent
100+
{{- if eq .Values.cozystackOperator.variant "talos" }}
101+
env:
102+
# Talos KubePrism endpoint — same as cozystack-operator uses to reach
103+
# the apiserver before CNI is up.
104+
- name: KUBERNETES_SERVICE_HOST
105+
value: "localhost"
106+
- name: KUBERNETES_SERVICE_PORT
107+
value: "7445"
108+
{{- else if eq .Values.cozystackOperator.variant "generic" }}
109+
env:
110+
- name: KUBERNETES_SERVICE_HOST
111+
value: {{ required "cozystack.apiServerHost is required in generic mode" .Values.cozystack.apiServerHost | quote }}
112+
- name: KUBERNETES_SERVICE_PORT
113+
value: {{ .Values.cozystack.apiServerPort | quote }}
114+
{{- end }}
115+
command:
116+
- kubectl
117+
- label
118+
- --overwrite
119+
- namespace
120+
- cozy-system
121+
- cozystack.io/system=true
122+
- pod-security.kubernetes.io/enforce=privileged

0 commit comments

Comments
 (0)