Repository navigation
fix(admission): reload the webhook TLS certificate on rotation - #22
Conversation
The admission webhook served HTTPS with http.ListenAndServeTLS, which reads the cert and key files once at startup and never again. After the mounted TLS Secret is rotated (cert-manager renewal or a manual re-issue) the pod keeps presenting the old leaf certificate while the apiserver's caBundle already trusts the new CA, so the handshake fails with an x509 error and, with failurePolicy: Ignore, the webhook silently stops assigning the scheduler until it is restarted. Serve the certificate through tls.Config.GetCertificate backed by a reloader that re-reads the files on each handshake and reloads the pair when their contents change, detected by a content hash so a Secret rotation via atomic ..data symlink swap is always observed. A transient read failure keeps serving the last known-good certificate. The -tls-cert-file/-tls-key-file flags are unchanged. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
kvaps
left a comment
There was a problem hiding this comment.
LGTM. Clean, dependency-free fix.
The certificate is now served through tls.Config.GetCertificate backed by a content-hash reloader, so a Secret rotation (cert-manager renewal or a manual re-issue) is picked up on the next handshake with no restart. Reading the files by path follows the atomic ..data symlink swap, and a torn/partial read across the swap falls back to the last known-good certificate and retries — the process never goes certificate-less.
Nice test coverage: a kubelet-accurate ..data symlink-swap reproduction, real TLS handshakes over a loopback listener, keep-last-good on read/parse errors, fail-fast at startup, and concurrent access under the race detector. Verified locally on my side too: go build ./..., go vet ./... and go test -race ./... all pass. The MinVersion: tls.VersionTLS12 and ReadHeaderTimeout hardening is a welcome bonus.
Thanks for the thorough work!
…sion webhook (#3201) ## What this PR does Re-vendors the `linstor-scheduler` chart from upstream **0.3.1** (appVersion **v0.3.6**). Upstream now ships the admission webhook natively (contributed via [piraeusdatastore/helm-charts#94](piraeusdatastore/helm-charts#94)), so the loose downstream admission templates — which round-tripped every Pod through an old `k8s.io/api` and silently stripped fields they did not know about, the `image` volume source and the native-sidecar `restartPolicy` among them, breaking KubeVirt containerDisk VMs and native sidecars cluster-wide — are dropped in favour of the upstream copy. - The admission webhook comes from the upstream chart now; the previous downstream copy and its patch are removed. `make update` pulls 0.3.1 and applies only `cozystack-scheduler-deployment.patch` (the scheduler pod component label and the distro-suffix-stripped kube-scheduler image tag, neither of which is upstream yet), so vendoring stays deterministic and reproducible. - The upstream admission Deployment keeps the same name but changes its immutable pod selector, so an in-place upgrade of an existing install would fail with `field is immutable`. Migration 52 deletes the old admission Deployment before the upgrade so the new one is recreated cleanly (`targetVersion` 52 → 53). - appVersion v0.3.6 also carries the webhook's TLS certificate hot-reload fix ([piraeusdatastore/linstor-scheduler-extender#22](piraeusdatastore/linstor-scheduler-extender#22)): the webhook used to load its certificate once at startup, so after the mounted Secret was rotated it kept serving a stale certificate and — with `failurePolicy: Ignore` — silently stopped mutating until it was restarted. It now reloads on rotation, so no external cert-reload workaround is needed here. - Wrapper values keep `replicaCount: 2`, now that the admission Pods have their own PodDisruptionBudget and no longer share the scheduler's. - Adds a helm-unittest suite covering the admission resources, PodDisruptionBudget selector isolation, least-privilege RBAC, the TLS provisioning modes and their rejection paths, the scheduler image-tag strip, and backward compatibility with the webhook disabled. Closes #3198. ### Screenshots N/A — no UI changes. ### Downstream repositories I walked the trigger map against the diff, file by file. It touches `packages/system/linstor-scheduler/` and, under `packages/core/platform/`, only migration 52 and the `migrations.targetVersion` bump — an internal, release-managed counter, not a user-facing `spec.components.platform.values.*` key documented on the website. No trigger matches. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(linstor-scheduler): the LINSTOR scheduler admission webhook no longer strips unknown Pod fields (the `image` volume source and native-sidecar `restartPolicy`), which had broken KubeVirt containerDisk VMs and native sidecars, and it now reloads its TLS certificate after rotation instead of silently failing. The chart is re-vendored from upstream 0.3.1; an automatic upgrade migration replaces the admission Deployment whose immutable selector changed. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Expanded the optional Linstor Scheduler admission webhook with richer configuration, including automated `schedulerName` mutation for LINSTOR CSI-backed PVC Pods. * **Improvements** * Updated the Linstor Scheduler Helm chart to **0.3.1** (scheduler app **v0.3.6**), including updated admission webhook behavior, safer defaults, and **2 replicas**. * Platform migrations now advance to **version 53**, with upgrade-safe cleanup of the existing admission deployment. * **Documentation** * Updated Helm README and post-install notes with detailed guidance for admission webhook and TLS modes. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Problem
The
linstor-scheduler-admissionwebhook serves HTTPS withhttp.ListenAndServeTLS, which reads-tls-cert-fileand-tls-key-fileexactly once at startup and never looks at them again. When the mounted TLS Secret is renewed — cert-manager rotates it on its own schedule, and a manual re-issue does the same — the running pod keeps presenting the old leaf certificate while the apiserver'sMutatingWebhookConfigurationcaBundlealready trusts the new CA. The TLS handshake then fails with an x509 "unknown authority" error. Because the webhook is registered withfailurePolicy: Ignore, the apiserver swallows that failure and admits pods unmutated, so the webhook silently stops assigning thelinstorscheduler until someone restarts the Deployment. That silent gap is what people work around today by bolting an external cert-reloader (a sidecar or operator that restarts the pod when the Secret changes) onto the deployment.Fix
I serve the certificate dynamically through
tls.Config.GetCertificateinstead of loading it once. A smallcertreload.Reloadercaches the parsedtls.Certificateand, on each handshake, re-reads the cert and key files and reloads the pair when their contents change. Detection is content-based (a SHA-256 of the file bytes) rather than size or mtime: Kubernetes rotates a mounted Secret by atomically swapping the..datasymlink in the mount directory, and reading the files by path follows that symlink to the new target, so a rotation is always observed — even when the renewed certificate happens to have the same byte length and a preserved modification time. A transient reload failure, for example a half-written file or a cert/key pair momentarily read across the swap, is logged and the last known-good certificate keeps being served, with a retry on the next handshake, so the process never goes certificate-less on a bad read. The-tls-cert-file/-tls-key-fileflags keep their existing meaning, and the server now useshttp.Server{TLSConfig: ...}withListenAndServeTLS("", ""), the idiomatic way to driveGetCertificate. Moving to an explicithttp.Serveralso lets me setMinVersion: tls.VersionTLS12and aReadHeaderTimeout. The change is dependency-free — only the standard library and the logger already in use.With this in place the webhook survives certificate rotation on its own, and the external cert-reload workaround is no longer needed.
Tests
pkg/certreloadhas unit tests that reproduce kubelet's atomic..datasymlink swap and assert the reloader picks up the new certificate (serial A → B); drive real TLS handshakes over a loopback listener to provecrypto/tlsactually calls the callback and a freshly connecting client sees the rotated certificate; cover in-place rotation; assert a corrupt or missing file leaves the last known-good certificate in place while a later valid pair is loaded; assert unchanged files return the cached certificate without re-parsing; and assert construction fails fast when the files are absent or not a valid key pair. A concurrent test exercises the mutex under the race detector.go build ./...,go vet ./...andgo test -race ./...all pass.DCO
The commit is
Signed-off-by:and GPG-signed.