Repository navigation
Commit ea15a20
committed
fix(cert-manager): move the webhook off the kubelet's port (#3359)
## What this PR does
The cert-manager webhook listens on port 10250 — the kubelet's port.
When a connection to the webhook Service resolves to a node IP rather
than the pod IP, it reaches the kubelet, which answers the TLS handshake
with its own node serving certificate. The API server then rejects every
cert-manager admission call in the cluster:
```text
failed calling webhook "webhook.cert-manager.io": ... x509: certificate is valid for srv3, not cert-manager-webhook.cozy-cert-manager.svc
```
The certificate is valid — it is simply the kubelet's, issued for the
node name. That signature is what separates this from the cainjector
failure mode, which reports `certificate signed by unknown authority`;
that possibility was checked and ruled out on the run where this was
observed. Here it blocked the whole platform bootstrap and pointed the
operator at cert-manager rather than at the routing layer.
The value comes straight from the vendored chart, where upstream chose
it deliberately for a different constraint: on GKE private clusters the
API server may only reach nodes on 443 and 10250. cozystack did not
override it.
Moving the webhook to 10260 is upstream's own remedy for this exact
error. The cert-manager troubleshooting guide documents the identical
signature — under a different cause, EKS Fargate, where each microVM
runs a kubelet that already holds 10250 — and prescribes `--set
webhook.securePort=10260`.
This does not fix the misroute, whose root cause is still unestablished;
the run carries no dataplane state, so candidates such as a stale
service map during bootstrap remain unproven. What it removes is the
trap that turns a rare, transient misroute into a cluster-wide failure
attributed to the wrong component. With the collision gone, the same
misroute surfaces as a connection refusal.
The override lives in the package-root values file, which `make update`
leaves alone, so it survives re-vendoring.
### Upgrade safety
The rendered webhook Deployment has one replica and no explicit
strategy, so the defaults apply: `maxUnavailable` 0 and `maxSurge` 1.
The replacement Pod must become Ready before the old one is removed. The
Service targets a named port rather than a number, so during the overlap
the EndpointSlice controller routes to each Pod on the port it actually
serves. There is no window with zero ready endpoints, and no maintenance
step is required. Had the Service pinned the port numerically, this
upgrade itself would have broken cluster-wide admission.
### Not addressed here
`external-secrets-operator` runs its webhook on 10250 with the same
`failurePolicy: Fail`, so it carries the identical exposure and deserves
the same treatment; `metrics-server` also uses 10250 but is an
APIService rather than an admission webhook. Both are left out to keep
this change to one component.
Closes #3355
### Release note
```release-note
fix(cert-manager): move the admission webhook off port 10250, which the kubelet also serves. A connection misrouted to a node answered with the kubelet's certificate, failing every cert-manager admission call cluster-wide with a misleading x509 error.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Configured the cert-manager webhook to use secure port 10260.
* Added coverage to verify secure port configuration, HTTPS service
routing, and disabled host networking.
* **Documentation**
* Added configuration comments explaining the secure port selection and
its role in avoiding service routing conflicts.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
(cherry picked from commit 3affec9)
(cherry picked from commit ee8344d7bf782a608574ff3a064cb5c766ac49ff)1 parent 711f09f commit ea15a20
2 files changed
Lines changed: 96 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
10 | 42 | | |
11 | 43 | | |
12 | 44 | | |
| |||
0 commit comments