Skip to content

Commit df85632

Browse files
committed
fix(cert-manager): move the webhook off the kubelet's port
The vendored chart defaults webhook.securePort to 10250, which is the port the kubelet serves on. Upstream picks that value deliberately, for GKE private clusters where apiservers may reach nodes only on 443 and 10250; it is free of consequence there and unsafe here. When a connection to the webhook Service resolves to a node IP instead of a Pod IP, it reaches the kubelet, which completes the TLS handshake with its node serving certificate. The apiserver then rejects every cert-manager admission call in the cluster with x509: certificate is valid for <node>, not cert-manager-webhook.cozy-cert-manager.svc and, because the webhook is failurePolicy: Fail, that stalls platform bootstrap. The certificate is valid; it is simply the wrong server's. The error names cert-manager and points away from the routing layer that actually failed, which makes a rare misroute expensive to diagnose. Serve on 10260 instead, the port upstream's troubleshooting guide prescribes for this exact error. It documents the failure on EKS Fargate, where a microVM forwards every port to the pod except the one its own kubelet already holds; the cause there is understood and ours is not, but the shape is the same, down to the certificate that comes back naming the node. What the fix needs of the port is that nothing answers on it in a node's network namespace, and nothing in the platform does. The same misroute is then refused at connect time, which is an honest and immediately diagnosable error. This does not prevent the misroute, whose cause is unestablished; it removes the collision that turns one into a cluster-wide failure attributed to the wrong component. The Service targets the container port by name, so it follows the change without an edit, and a rolling update resolves the old and new ports per Pod rather than dropping the endpoint. Tests pin the listener flag together with the advertised containerPort so the two cannot drift apart. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent 7e206e2 commit df85632

2 files changed

Lines changed: 96 additions & 0 deletions

File tree

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
suite: cert-manager webhook secure port
2+
3+
# Keeps the webhook off the kubelet's port. The values.yaml override this
4+
# guards carries the reasoning; it is not repeated here.
5+
#
6+
# The first test is the regression guard: it fails if the override is dropped
7+
# or if `make update` re-vendors a chart that ignores it, and it asserts the
8+
# listener flag together with the advertised containerPort so the two cannot
9+
# drift apart. The other two pin preconditions the fix depends on and already
10+
# hold on their own — that the port lives in the Pod's network namespace
11+
# rather than the node's, and that the Service reaches it by name, which is
12+
# what lets a port change roll out without dropping an endpoint.
13+
#
14+
# An upstream rename of the key needs no test: the vendored values.schema.json
15+
# sets additionalProperties: false, so the override would fail helm outright
16+
# rather than be silently ignored.
17+
18+
templates:
19+
- charts/cert-manager/templates/webhook-deployment.yaml
20+
- charts/cert-manager/templates/webhook-service.yaml
21+
22+
release:
23+
name: cert-manager
24+
namespace: cozy-cert-manager
25+
26+
tests:
27+
- it: serves the webhook on 10260, not the kubelet's 10250
28+
template: charts/cert-manager/templates/webhook-deployment.yaml
29+
asserts:
30+
- isKind:
31+
of: Deployment
32+
- contains:
33+
path: spec.template.spec.containers[0].args
34+
content: --secure-port=10260
35+
- equal:
36+
path: spec.template.spec.containers[0].ports[0]
37+
value:
38+
name: https
39+
protocol: TCP
40+
containerPort: 10260
41+
42+
- it: keeps the webhook off the host network, so the port lives in the Pod netns
43+
template: charts/cert-manager/templates/webhook-deployment.yaml
44+
asserts:
45+
# The template emits the key only when the value is truthy, so absence
46+
# is the only shape a disabled hostNetwork can take here. isNullOrEmpty
47+
# would not do: it fails on a path that does not exist.
48+
- notExists:
49+
path: spec.template.spec.hostNetwork
50+
51+
- it: resolves the Service to the webhook container by name, not by number
52+
template: charts/cert-manager/templates/webhook-service.yaml
53+
asserts:
54+
- isKind:
55+
of: Service
56+
- equal:
57+
path: spec.ports[0].name
58+
value: https
59+
- equal:
60+
path: spec.ports[0].port
61+
value: 443
62+
- equal:
63+
path: spec.ports[0].targetPort
64+
value: https

‎packages/system/cert-manager/values.yaml‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,38 @@ cert-manager:
77
cpu: 50m
88
memory: 128Mi
99
webhook:
10+
# Upstream defaults securePort to 10250, the port the kubelet serves on.
11+
# That default is chosen for GKE private clusters, where apiservers may
12+
# reach nodes only on 443 and 10250; it costs nothing there and is unsafe
13+
# here. When a connection to the webhook Service resolves to a node IP
14+
# rather than a Pod IP, it reaches the kubelet, which completes the TLS
15+
# handshake with its node serving certificate, and the apiserver rejects
16+
# every cert-manager admission call cluster-wide with "x509: certificate
17+
# is valid for <node>, not cert-manager-webhook.cozy-cert-manager.svc".
18+
# The certificate is valid — it is simply the wrong server's — so the
19+
# error reads as a cert-manager fault and hides the misroute that caused
20+
# it. Serving on a port nothing else answers on turns that same misroute
21+
# into a connection refusal, which names the real problem. This does not
22+
# prevent the misroute; it removes the collision that makes one
23+
# catastrophic and misattributed.
24+
#
25+
# What has to hold for 10260 to be an improvement is that nothing answers
26+
# on it in the node's network namespace, since the misroute lands on a
27+
# node IP. Nothing does, in either cluster this chart is installed into.
28+
# On management nodes the only host-network workload of the kind that
29+
# might is local-ccm, which links none of the cloud-provider framework
30+
# that binds 10258 and 10260 by default, and binds no TCP port at all.
31+
# No cloud-controller-manager runs on a tenant node either: the kubevirt
32+
# one does build on that framework, but it is a pod-networked Deployment
33+
# in the management cluster, so whatever it binds stays in a Pod's
34+
# namespace. Anything host-network added later to either cluster has to
35+
# keep clear of the port.
36+
#
37+
# Under webhook.hostNetwork the webhook itself would join that namespace,
38+
# where the clusterwide policy in packages/system/cilium-networkpolicy
39+
# denies 10250 from the world but has no entry for 10260. Enabling
40+
# hostNetwork means revisiting that policy.
41+
securePort: 10260
1042
resources:
1143
limits:
1244
cpu: 200m

0 commit comments

Comments
 (0)