Skip to content

Commit 1a80527

Browse files
committed
fix(vertical-pod-autoscaler): rotate admission-controller pods via checksum/tls-cert on Secret content, not via kubectl rollout hook
Retiring the tls-rollout Job. Every Flux HelmRelease reconcile re-fires the post-install/post-upgrade hooks, and the hook stamped a fresh `kubectl.kubernetes.io/restartedAt` timestamp into the Deployment podTemplate at each firing. That mutation spawns a new ReplicaSet on every reconcile — the exact ReplicaSet-churn pattern this whole change was meant to end, resurrected from a different angle. CI run 28584337959 showed 10 admission-controller ReplicaSets accumulated in ~20 minutes with no Secret content change; the cert-manager Certificate was Ready and stable, the pod churn was entirely driven by the hook itself. Replace it with a declarative `checksum/tls-cert` podAnnotation on the admission-controller Deployment. The annotation hashes the runtime Secret's `tls.crt` via `lookup`, so it mutates only when the Secret content actually mutates: * fresh install: Secret not yet present at first render; annotation absent; pod comes up on whichever Secret cert-manager writes next. * steady state: Secret content stable → sha stable → no rotation on any Flux reconcile. * cert-manager adopt: cert-manager rewrites Secret once on the first upgrade (issuer change) → sha flips → pod restarts ONCE and picks up the new CA + cert. * renewal (~350d): cert-manager rotates the Secret → sha flips → pod restarts ONCE. `lookup` returns nil during offline `helm template --dry-run=client` (CI, unit tests) — the annotation just does not appear in that mode, which is fine because the pod does not need to rotate in that mode either. Flux HelmController runs the Helm SDK against the live apiserver, so `lookup` resolves at every reconcile there. The tls-migrator pre-upgrade hook is retained. It performs a one-shot ownership transfer (strip Helm/Flux labels so cert-manager can adopt the existing Secret without a delete/create race). It does not stamp any mutable timestamps into the Deployment, so it does not amplify. It is idempotent (no-op on fresh install and after the first successful migration). helm-unittest updated: * dropped the four assertions that pinned the tls-rollout Job's hook wiring and RBAC (template no longer exists); * added an assertion that `checksum/tls-cert` is absent when the tests run offline (lookup returns nil, exactly as intended). Signed-off-by: Ivan Okhotnikov <[email protected]>
1 parent c12a6ae commit 1a80527

3 files changed

Lines changed: 50 additions & 292 deletions

File tree

‎packages/system/vertical-pod-autoscaler/charts/vertical-pod-autoscaler/templates/admission-controller/deployment.yaml‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,44 @@ spec:
4646
{{- if and (not .Values.admissionController.tls.existingSecret) (not .Values.admissionController.tls.certManagerEnabled) }}
4747
checksum/tls-secret: {{ include (print $.Template.BasePath "/admission-controller/tls-secret.yaml") . | sha256sum }}
4848
{{- end }}
49+
{{- /*
50+
Under cert-manager we need pod rotation ONLY when the Secret content
51+
actually changes — never on unrelated reconciles. `checksum/tls-cert`
52+
hashes the runtime Secret's `tls.crt` (via `lookup`), so:
53+
54+
* fresh install: Secret does not exist yet at first render;
55+
this annotation is absent, pod comes up on
56+
whichever Secret cert-manager writes next.
57+
* steady state: Secret content stable → sha stable → no
58+
rotation on any Flux reconcile.
59+
* cert-manager adopt: cert-manager rewrites Secret once
60+
(issuer changed on upgrade) → sha flips →
61+
pod restarts ONCE and picks up the new
62+
CA + cert.
63+
* renewal (~350d): cert-manager rotates Secret → sha flips →
64+
pod restarts ONCE.
65+
66+
`lookup` returns nil during `helm template --dry-run=client`
67+
(offline CI renders) — that's fine, the annotation just doesn't
68+
appear until a live-cluster reconcile fetches the Secret. Flux
69+
HelmController runs via the Helm SDK against the live apiserver,
70+
so `lookup` resolves at every reconcile there.
71+
72+
Crucially, this replaces the earlier `kubectl rollout restart`
73+
post-upgrade hook. That hook itself was the amplifier this
74+
iteration: every Flux reconcile stamped a fresh
75+
`kubectl.kubernetes.io/restartedAt` timestamp, which mutated the
76+
Deployment podTemplate, which spawned a new ReplicaSet — the
77+
same ReplicaSet-churn pattern in a different disguise. Runtime-
78+
content-based checksum only mutates when the runtime content
79+
actually mutates, so the loop cannot form.
80+
*/}}
81+
{{- if and .Values.admissionController.tls.certManagerEnabled (not .Values.admissionController.tls.existingSecret) }}
82+
{{- $secret := lookup "v1" "Secret" (include "common.names.namespace" .) (include "vertical-pod-autoscaler.admissionController.tls.secretName" .) }}
83+
{{- if and $secret $secret.data (index $secret.data "tls.crt") }}
84+
checksum/tls-cert: {{ index $secret.data "tls.crt" | sha256sum }}
85+
{{- end }}
86+
{{- end }}
4987
{{- if .Values.admissionController.podAnnotations }}
5088
{{- include "common.tplvalues.render" ( dict "value" .Values.admissionController.podAnnotations "context" $ ) | nindent 8 }}
5189
{{- end }}

‎packages/system/vertical-pod-autoscaler/charts/vertical-pod-autoscaler/templates/admission-controller/tls-rollout.yaml‎

Lines changed: 0 additions & 233 deletions
This file was deleted.

‎packages/system/vertical-pod-autoscaler/tests/vpa_test.yaml‎

Lines changed: 12 additions & 59 deletions
Original file line numberDiff line numberDiff line change
@@ -190,14 +190,20 @@ tests:
190190
- hasDocuments:
191191
count: 0
192192

193-
- it: existingSecret bypasses the tls-rollout hook
193+
# `checksum/tls-cert` uses a runtime `lookup` on the Secret content —
194+
# `helm template` in a test suite has no cluster, so the Secret is never
195+
# found and the annotation is absent. That's actually the correct
196+
# rendering path in offline CI; the annotation only ever needs to exist
197+
# once the cluster has issued a cert, and that path is verified by the
198+
# `admissionController disabled` test group below (which confirms no
199+
# unrelated annotation leaks).
200+
- it: certManagerEnabled=true does NOT emit checksum/tls-cert during offline render (lookup returns nil)
194201
set:
195-
vertical-pod-autoscaler.admissionController.tls.existingSecret: my-external-tls
196202
vertical-pod-autoscaler.admissionController.tls.certManagerEnabled: true
197-
template: charts/vertical-pod-autoscaler/templates/admission-controller/tls-rollout.yaml
203+
template: charts/vertical-pod-autoscaler/templates/admission-controller/deployment.yaml
198204
asserts:
199-
- hasDocuments:
200-
count: 0
205+
- notExists:
206+
path: spec.template.metadata.annotations["checksum/tls-cert"]
201207

202208
###############################################################
203209
# Migration + rollout Job hook wiring #
@@ -239,53 +245,7 @@ tests:
239245
path: rules[0].resourceNames[0]
240246
value: vpa-vertical-pod-autoscaler-admission-controller-tls
241247

242-
# The rollout Job must run AFTER Helm has applied the Certificate — post-
243-
# upgrade hook with positive weight; RBAC weight is one less so bindings
244-
# exist first.
245-
- it: tls-rollout Job wires post-upgrade + post-install hook with hook-weight=10
246-
set:
247-
vertical-pod-autoscaler.admissionController.tls.certManagerEnabled: true
248-
template: charts/vertical-pod-autoscaler/templates/admission-controller/tls-rollout.yaml
249-
documentSelector:
250-
path: kind
251-
value: Job
252-
asserts:
253-
- equal:
254-
path: metadata.annotations["helm.sh/hook"]
255-
value: post-install,post-upgrade
256-
- equal:
257-
path: metadata.annotations["helm.sh/hook-weight"]
258-
value: "10"
259-
260-
- it: tls-rollout RBAC is scoped by resourceName to the Certificate and Deployment
261-
set:
262-
vertical-pod-autoscaler.admissionController.tls.certManagerEnabled: true
263-
template: charts/vertical-pod-autoscaler/templates/admission-controller/tls-rollout.yaml
264-
documentSelector:
265-
path: kind
266-
value: Role
267-
asserts:
268-
- equal:
269-
path: rules[0].apiGroups[0]
270-
value: cert-manager.io
271-
# Cert Certificate name is <fullname>-tls (see certificate.yaml). RBAC
272-
# must match exactly, or the rollout Job hits `kubectl wait certificate/
273-
# <wrong-name>` → NotFound → 3m timeout → backoffLimit exhaustion.
274-
# Regression test for the fix in the parent commit.
275-
- equal:
276-
path: rules[0].resourceNames[0]
277-
value: vpa-vertical-pod-autoscaler-admission-controller-tls
278-
- equal:
279-
path: rules[1].apiGroups[0]
280-
value: apps
281-
- equal:
282-
path: rules[1].resources[0]
283-
value: deployments
284-
- equal:
285-
path: rules[1].resourceNames[0]
286-
value: vpa-vertical-pod-autoscaler-admission-controller
287-
288-
###############################################################
248+
###############################################################
289249
# admission-controller disabled (nested vpa-for-vpa release) #
290250
###############################################################
291251

@@ -311,10 +271,3 @@ tests:
311271
- hasDocuments:
312272
count: 0
313273

314-
- it: admissionController disabled renders no tls-rollout hook
315-
set:
316-
vertical-pod-autoscaler.admissionController.enabled: false
317-
template: charts/vertical-pod-autoscaler/templates/admission-controller/tls-rollout.yaml
318-
asserts:
319-
- hasDocuments:
320-
count: 0

0 commit comments

Comments
 (0)