Skip to content

Commit ea15a20

Browse files
fix(cert-manager): move the webhook off the kubelet's port (#3359)
## What this PR does The cert-manager webhook listens on port 10250 — the kubelet's port. When a connection to the webhook Service resolves to a node IP rather than the pod IP, it reaches the kubelet, which answers the TLS handshake with its own node serving certificate. The API server then rejects every cert-manager admission call in the cluster: ```text failed calling webhook "webhook.cert-manager.io": ... x509: certificate is valid for srv3, not cert-manager-webhook.cozy-cert-manager.svc ``` The certificate is valid — it is simply the kubelet's, issued for the node name. That signature is what separates this from the cainjector failure mode, which reports `certificate signed by unknown authority`; that possibility was checked and ruled out on the run where this was observed. Here it blocked the whole platform bootstrap and pointed the operator at cert-manager rather than at the routing layer. The value comes straight from the vendored chart, where upstream chose it deliberately for a different constraint: on GKE private clusters the API server may only reach nodes on 443 and 10250. cozystack did not override it. Moving the webhook to 10260 is upstream's own remedy for this exact error. The cert-manager troubleshooting guide documents the identical signature — under a different cause, EKS Fargate, where each microVM runs a kubelet that already holds 10250 — and prescribes `--set webhook.securePort=10260`. This does not fix the misroute, whose root cause is still unestablished; the run carries no dataplane state, so candidates such as a stale service map during bootstrap remain unproven. What it removes is the trap that turns a rare, transient misroute into a cluster-wide failure attributed to the wrong component. With the collision gone, the same misroute surfaces as a connection refusal. The override lives in the package-root values file, which `make update` leaves alone, so it survives re-vendoring. ### Upgrade safety The rendered webhook Deployment has one replica and no explicit strategy, so the defaults apply: `maxUnavailable` 0 and `maxSurge` 1. The replacement Pod must become Ready before the old one is removed. The Service targets a named port rather than a number, so during the overlap the EndpointSlice controller routes to each Pod on the port it actually serves. There is no window with zero ready endpoints, and no maintenance step is required. Had the Service pinned the port numerically, this upgrade itself would have broken cluster-wide admission. ### Not addressed here `external-secrets-operator` runs its webhook on 10250 with the same `failurePolicy: Fail`, so it carries the identical exposure and deserves the same treatment; `metrics-server` also uses 10250 but is an APIService rather than an admission webhook. Both are left out to keep this change to one component. Closes #3355 ### Release note ```release-note fix(cert-manager): move the admission webhook off port 10250, which the kubelet also serves. A connection misrouted to a node answered with the kubelet's certificate, failing every cert-manager admission call cluster-wide with a misleading x509 error. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Configured the cert-manager webhook to use secure port 10260. * Added coverage to verify secure port configuration, HTTPS service routing, and disabled host networking. * **Documentation** * Added configuration comments explaining the secure port selection and its role in avoiding service routing conflicts. <!-- end of auto-generated comment: release notes by coderabbit.ai --> (cherry picked from commit 3affec9) (cherry picked from commit ee8344d7bf782a608574ff3a064cb5c766ac49ff)
1 parent 711f09f commit ea15a20

2 files changed

Lines changed: 96 additions & 0 deletions

File tree

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
suite: cert-manager webhook secure port
2+
3+
# Keeps the webhook off the kubelet's port. The values.yaml override this
4+
# guards carries the reasoning; it is not repeated here.
5+
#
6+
# The first test is the regression guard: it fails if the override is dropped
7+
# or if `make update` re-vendors a chart that ignores it, and it asserts the
8+
# listener flag together with the advertised containerPort so the two cannot
9+
# drift apart. The other two pin preconditions the fix depends on and already
10+
# hold on their own — that the port lives in the Pod's network namespace
11+
# rather than the node's, and that the Service reaches it by name, which is
12+
# what lets a port change roll out without dropping an endpoint.
13+
#
14+
# An upstream rename of the key needs no test: the vendored values.schema.json
15+
# sets additionalProperties: false, so the override would fail helm outright
16+
# rather than be silently ignored.
17+
18+
templates:
19+
- charts/cert-manager/templates/webhook-deployment.yaml
20+
- charts/cert-manager/templates/webhook-service.yaml
21+
22+
release:
23+
name: cert-manager
24+
namespace: cozy-cert-manager
25+
26+
tests:
27+
- it: serves the webhook on 10260, not the kubelet's 10250
28+
template: charts/cert-manager/templates/webhook-deployment.yaml
29+
asserts:
30+
- isKind:
31+
of: Deployment
32+
- contains:
33+
path: spec.template.spec.containers[0].args
34+
content: --secure-port=10260
35+
- equal:
36+
path: spec.template.spec.containers[0].ports[0]
37+
value:
38+
name: https
39+
protocol: TCP
40+
containerPort: 10260
41+
42+
- it: keeps the webhook off the host network, so the port lives in the Pod netns
43+
template: charts/cert-manager/templates/webhook-deployment.yaml
44+
asserts:
45+
# The template emits the key only when the value is truthy, so absence
46+
# is the only shape a disabled hostNetwork can take here. isNullOrEmpty
47+
# would not do: it fails on a path that does not exist.
48+
- notExists:
49+
path: spec.template.spec.hostNetwork
50+
51+
- it: resolves the Service to the webhook container by name, not by number
52+
template: charts/cert-manager/templates/webhook-service.yaml
53+
asserts:
54+
- isKind:
55+
of: Service
56+
- equal:
57+
path: spec.ports[0].name
58+
value: https
59+
- equal:
60+
path: spec.ports[0].port
61+
value: 443
62+
- equal:
63+
path: spec.ports[0].targetPort
64+
value: https

‎packages/system/cert-manager/values.yaml‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,38 @@ cert-manager:
77
cpu: 50m
88
memory: 128Mi
99
webhook:
10+
# Upstream defaults securePort to 10250, the port the kubelet serves on.
11+
# That default is chosen for GKE private clusters, where apiservers may
12+
# reach nodes only on 443 and 10250; it costs nothing there and is unsafe
13+
# here. When a connection to the webhook Service resolves to a node IP
14+
# rather than a Pod IP, it reaches the kubelet, which completes the TLS
15+
# handshake with its node serving certificate, and the apiserver rejects
16+
# every cert-manager admission call cluster-wide with "x509: certificate
17+
# is valid for <node>, not cert-manager-webhook.cozy-cert-manager.svc".
18+
# The certificate is valid — it is simply the wrong server's — so the
19+
# error reads as a cert-manager fault and hides the misroute that caused
20+
# it. Serving on a port nothing else answers on turns that same misroute
21+
# into a connection refusal, which names the real problem. This does not
22+
# prevent the misroute; it removes the collision that makes one
23+
# catastrophic and misattributed.
24+
#
25+
# What has to hold for 10260 to be an improvement is that nothing answers
26+
# on it in the node's network namespace, since the misroute lands on a
27+
# node IP. Nothing does, in either cluster this chart is installed into.
28+
# On management nodes the only host-network workload of the kind that
29+
# might is local-ccm, which links none of the cloud-provider framework
30+
# that binds 10258 and 10260 by default, and binds no TCP port at all.
31+
# No cloud-controller-manager runs on a tenant node either: the kubevirt
32+
# one does build on that framework, but it is a pod-networked Deployment
33+
# in the management cluster, so whatever it binds stays in a Pod's
34+
# namespace. Anything host-network added later to either cluster has to
35+
# keep clear of the port.
36+
#
37+
# Under webhook.hostNetwork the webhook itself would join that namespace,
38+
# where the clusterwide policy in packages/system/cilium-networkpolicy
39+
# denies 10250 from the world but has no entry for 10260. Enabling
40+
# hostNetwork means revisiting that policy.
41+
securePort: 10260
1042
resources:
1143
limits:
1244
cpu: 200m

0 commit comments

Comments
 (0)