Skip to content

fix(admission): reload the webhook TLS certificate on rotation - #22

Merged
kvaps merged 1 commit into
piraeusdatastore:masterfrom
lexfrei:feat/admission-cert-hot-reload
Jul 16, 2026
Merged

kvaps merged 1 commit into
piraeusdatastore:masterfrom
lexfrei:feat/admission-cert-hot-reload

Conversation

@lexfrei

@lexfrei lexfrei commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Problem

The linstor-scheduler-admission webhook serves HTTPS with http.ListenAndServeTLS, which reads -tls-cert-file and -tls-key-file exactly once at startup and never looks at them again. When the mounted TLS Secret is renewed — cert-manager rotates it on its own schedule, and a manual re-issue does the same — the running pod keeps presenting the old leaf certificate while the apiserver's MutatingWebhookConfiguration caBundle already trusts the new CA. The TLS handshake then fails with an x509 "unknown authority" error. Because the webhook is registered with failurePolicy: Ignore, the apiserver swallows that failure and admits pods unmutated, so the webhook silently stops assigning the linstor scheduler until someone restarts the Deployment. That silent gap is what people work around today by bolting an external cert-reloader (a sidecar or operator that restarts the pod when the Secret changes) onto the deployment.

Fix

I serve the certificate dynamically through tls.Config.GetCertificate instead of loading it once. A small certreload.Reloader caches the parsed tls.Certificate and, on each handshake, re-reads the cert and key files and reloads the pair when their contents change. Detection is content-based (a SHA-256 of the file bytes) rather than size or mtime: Kubernetes rotates a mounted Secret by atomically swapping the ..data symlink in the mount directory, and reading the files by path follows that symlink to the new target, so a rotation is always observed — even when the renewed certificate happens to have the same byte length and a preserved modification time. A transient reload failure, for example a half-written file or a cert/key pair momentarily read across the swap, is logged and the last known-good certificate keeps being served, with a retry on the next handshake, so the process never goes certificate-less on a bad read. The -tls-cert-file / -tls-key-file flags keep their existing meaning, and the server now uses http.Server{TLSConfig: ...} with ListenAndServeTLS("", ""), the idiomatic way to drive GetCertificate. Moving to an explicit http.Server also lets me set MinVersion: tls.VersionTLS12 and a ReadHeaderTimeout. The change is dependency-free — only the standard library and the logger already in use.

With this in place the webhook survives certificate rotation on its own, and the external cert-reload workaround is no longer needed.

Tests

pkg/certreload has unit tests that reproduce kubelet's atomic ..data symlink swap and assert the reloader picks up the new certificate (serial A → B); drive real TLS handshakes over a loopback listener to prove crypto/tls actually calls the callback and a freshly connecting client sees the rotated certificate; cover in-place rotation; assert a corrupt or missing file leaves the last known-good certificate in place while a later valid pair is loaded; assert unchanged files return the cached certificate without re-parsing; and assert construction fails fast when the files are absent or not a valid key pair. A concurrent test exercises the mutex under the race detector. go build ./..., go vet ./... and go test -race ./... all pass.

DCO

The commit is Signed-off-by: and GPG-signed.

The admission webhook served HTTPS with http.ListenAndServeTLS, which
reads the cert and key files once at startup and never again. After the
mounted TLS Secret is rotated (cert-manager renewal or a manual re-issue)
the pod keeps presenting the old leaf certificate while the apiserver's
caBundle already trusts the new CA, so the handshake fails with an x509
error and, with failurePolicy: Ignore, the webhook silently stops
assigning the scheduler until it is restarted.

Serve the certificate through tls.Config.GetCertificate backed by a
reloader that re-reads the files on each handshake and reloads the pair
when their contents change, detected by a content hash so a Secret
rotation via atomic ..data symlink swap is always observed. A transient
read failure keeps serving the last known-good certificate. The
-tls-cert-file/-tls-key-file flags are unchanged.

Assisted-By: Claude <[email protected]>
Signed-off-by: Aleksei Sviridkin <[email protected]>

@kvaps kvaps left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Clean, dependency-free fix.

The certificate is now served through tls.Config.GetCertificate backed by a content-hash reloader, so a Secret rotation (cert-manager renewal or a manual re-issue) is picked up on the next handshake with no restart. Reading the files by path follows the atomic ..data symlink swap, and a torn/partial read across the swap falls back to the last known-good certificate and retries — the process never goes certificate-less.

Nice test coverage: a kubelet-accurate ..data symlink-swap reproduction, real TLS handshakes over a loopback listener, keep-last-good on read/parse errors, fail-fast at startup, and concurrent access under the race detector. Verified locally on my side too: go build ./..., go vet ./... and go test -race ./... all pass. The MinVersion: tls.VersionTLS12 and ReadHeaderTimeout hardening is a welcome bonus.

Thanks for the thorough work!

@kvaps
kvaps merged commit 2ebb158 into piraeusdatastore:master Jul 16, 2026
1 check passed
lexfrei added a commit to cozystack/cozystack that referenced this pull request Jul 16, 2026
…sion webhook (#3201)

## What this PR does

Re-vendors the `linstor-scheduler` chart from upstream **0.3.1**
(appVersion **v0.3.6**). Upstream now ships the admission webhook
natively (contributed via
[piraeusdatastore/helm-charts#94](piraeusdatastore/helm-charts#94)),
so the loose downstream admission templates — which round-tripped every
Pod through an old `k8s.io/api` and silently stripped fields they did
not know about, the `image` volume source and the native-sidecar
`restartPolicy` among them, breaking KubeVirt containerDisk VMs and
native sidecars cluster-wide — are dropped in favour of the upstream
copy.

- The admission webhook comes from the upstream chart now; the previous
downstream copy and its patch are removed. `make update` pulls 0.3.1 and
applies only `cozystack-scheduler-deployment.patch` (the scheduler pod
component label and the distro-suffix-stripped kube-scheduler image tag,
neither of which is upstream yet), so vendoring stays deterministic and
reproducible.
- The upstream admission Deployment keeps the same name but changes its
immutable pod selector, so an in-place upgrade of an existing install
would fail with `field is immutable`. Migration 52 deletes the old
admission Deployment before the upgrade so the new one is recreated
cleanly (`targetVersion` 52 → 53).
- appVersion v0.3.6 also carries the webhook's TLS certificate
hot-reload fix
([piraeusdatastore/linstor-scheduler-extender#22](piraeusdatastore/linstor-scheduler-extender#22)):
the webhook used to load its certificate once at startup, so after the
mounted Secret was rotated it kept serving a stale certificate and —
with `failurePolicy: Ignore` — silently stopped mutating until it was
restarted. It now reloads on rotation, so no external cert-reload
workaround is needed here.
- Wrapper values keep `replicaCount: 2`, now that the admission Pods
have their own PodDisruptionBudget and no longer share the scheduler's.
- Adds a helm-unittest suite covering the admission resources,
PodDisruptionBudget selector isolation, least-privilege RBAC, the TLS
provisioning modes and their rejection paths, the scheduler image-tag
strip, and backward compatibility with the webhook disabled.

Closes #3198.

### Screenshots

N/A — no UI changes.

### Downstream repositories

I walked the trigger map against the diff, file by file. It touches
`packages/system/linstor-scheduler/` and, under
`packages/core/platform/`, only migration 52 and the
`migrations.targetVersion` bump — an internal, release-managed counter,
not a user-facing `spec.components.platform.values.*` key documented on
the website. No trigger matches.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
fix(linstor-scheduler): the LINSTOR scheduler admission webhook no longer strips unknown Pod fields (the `image` volume source and native-sidecar `restartPolicy`), which had broken KubeVirt containerDisk VMs and native sidecars, and it now reloads its TLS certificate after rotation instead of silently failing. The chart is re-vendored from upstream 0.3.1; an automatic upgrade migration replaces the admission Deployment whose immutable selector changed.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Expanded the optional Linstor Scheduler admission webhook with richer
configuration, including automated `schedulerName` mutation for LINSTOR
CSI-backed PVC Pods.
* **Improvements**
* Updated the Linstor Scheduler Helm chart to **0.3.1** (scheduler app
**v0.3.6**), including updated admission webhook behavior, safer
defaults, and **2 replicas**.
* Platform migrations now advance to **version 53**, with upgrade-safe
cleanup of the existing admission deployment.
* **Documentation**
* Updated Helm README and post-install notes with detailed guidance for
admission webhook and TLS modes.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants