Skip to content

Commit 982d6d4

Browse files
fix(cert-manager): move the webhook off the kubelet's port
Backport of #3359 onto release-1.5. The cert-manager webhook listened on 10250, the port the kubelet also serves. When a connection to the webhook Service resolved to a node IP rather than a Pod IP it reached the kubelet, which completed the TLS handshake with its own node serving certificate, and the API server then rejected every cert-manager admission call cluster-wide: failed calling webhook "webhook.cert-manager.io": ... x509: certificate is valid for srv3, not cert-manager-webhook.cozy-cert-manager.svc The certificate is valid — it is simply the wrong server's — so the error reads as a cert-manager fault and hides the misroute that caused it. Moving the webhook to 10260 is upstream's own remedy for this signature. It does not fix the misroute; it removes the collision that turns a rare transient one into a cluster-wide outage attributed to the wrong component. The override lives in the package-root values file, which `make update` leaves alone, so it survives re-vendoring. Stacked on the #3199 backport (#3202): release-1.5 ships an EMPTY packages/system/cert-manager/values.yaml, so every resource override in that file arrived after v1.5.2. The bot's cherry-pick of this fix conflicted for that reason alone — nothing to merge into — and applies cleanly once the cainjector backport is in place. Upgrade safety is unchanged from the original: one replica, no explicit strategy, so maxUnavailable 0 / maxSurge 1 apply and the replacement Pod must be Ready before the old one goes. The Service targets a NAMED port, so during the overlap each Pod is reached on the port it actually serves and there is no window with zero ready endpoints. Closes #3355 (cherry picked from commit 3affec9) Assisted-By: Claude <[email protected]> Signed-off-by: Myasnikov Daniil <[email protected]>
1 parent 711f09f commit 982d6d4

2 files changed

Lines changed: 96 additions & 0 deletions

File tree

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
suite: cert-manager webhook secure port
2+
3+
# Keeps the webhook off the kubelet's port. The values.yaml override this
4+
# guards carries the reasoning; it is not repeated here.
5+
#
6+
# The first test is the regression guard: it fails if the override is dropped
7+
# or if `make update` re-vendors a chart that ignores it, and it asserts the
8+
# listener flag together with the advertised containerPort so the two cannot
9+
# drift apart. The other two pin preconditions the fix depends on and already
10+
# hold on their own — that the port lives in the Pod's network namespace
11+
# rather than the node's, and that the Service reaches it by name, which is
12+
# what lets a port change roll out without dropping an endpoint.
13+
#
14+
# An upstream rename of the key needs no test: the vendored values.schema.json
15+
# sets additionalProperties: false, so the override would fail helm outright
16+
# rather than be silently ignored.
17+
18+
templates:
19+
- charts/cert-manager/templates/webhook-deployment.yaml
20+
- charts/cert-manager/templates/webhook-service.yaml
21+
22+
release:
23+
name: cert-manager
24+
namespace: cozy-cert-manager
25+
26+
tests:
27+
- it: serves the webhook on 10260, not the kubelet's 10250
28+
template: charts/cert-manager/templates/webhook-deployment.yaml
29+
asserts:
30+
- isKind:
31+
of: Deployment
32+
- contains:
33+
path: spec.template.spec.containers[0].args
34+
content: --secure-port=10260
35+
- equal:
36+
path: spec.template.spec.containers[0].ports[0]
37+
value:
38+
name: https
39+
protocol: TCP
40+
containerPort: 10260
41+
42+
- it: keeps the webhook off the host network, so the port lives in the Pod netns
43+
template: charts/cert-manager/templates/webhook-deployment.yaml
44+
asserts:
45+
# The template emits the key only when the value is truthy, so absence
46+
# is the only shape a disabled hostNetwork can take here. isNullOrEmpty
47+
# would not do: it fails on a path that does not exist.
48+
- notExists:
49+
path: spec.template.spec.hostNetwork
50+
51+
- it: resolves the Service to the webhook container by name, not by number
52+
template: charts/cert-manager/templates/webhook-service.yaml
53+
asserts:
54+
- isKind:
55+
of: Service
56+
- equal:
57+
path: spec.ports[0].name
58+
value: https
59+
- equal:
60+
path: spec.ports[0].port
61+
value: 443
62+
- equal:
63+
path: spec.ports[0].targetPort
64+
value: https

‎packages/system/cert-manager/values.yaml‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,38 @@ cert-manager:
77
cpu: 50m
88
memory: 128Mi
99
webhook:
10+
# Upstream defaults securePort to 10250, the port the kubelet serves on.
11+
# That default is chosen for GKE private clusters, where apiservers may
12+
# reach nodes only on 443 and 10250; it costs nothing there and is unsafe
13+
# here. When a connection to the webhook Service resolves to a node IP
14+
# rather than a Pod IP, it reaches the kubelet, which completes the TLS
15+
# handshake with its node serving certificate, and the apiserver rejects
16+
# every cert-manager admission call cluster-wide with "x509: certificate
17+
# is valid for <node>, not cert-manager-webhook.cozy-cert-manager.svc".
18+
# The certificate is valid — it is simply the wrong server's — so the
19+
# error reads as a cert-manager fault and hides the misroute that caused
20+
# it. Serving on a port nothing else answers on turns that same misroute
21+
# into a connection refusal, which names the real problem. This does not
22+
# prevent the misroute; it removes the collision that makes one
23+
# catastrophic and misattributed.
24+
#
25+
# What has to hold for 10260 to be an improvement is that nothing answers
26+
# on it in the node's network namespace, since the misroute lands on a
27+
# node IP. Nothing does, in either cluster this chart is installed into.
28+
# On management nodes the only host-network workload of the kind that
29+
# might is local-ccm, which links none of the cloud-provider framework
30+
# that binds 10258 and 10260 by default, and binds no TCP port at all.
31+
# No cloud-controller-manager runs on a tenant node either: the kubevirt
32+
# one does build on that framework, but it is a pod-networked Deployment
33+
# in the management cluster, so whatever it binds stays in a Pod's
34+
# namespace. Anything host-network added later to either cluster has to
35+
# keep clear of the port.
36+
#
37+
# Under webhook.hostNetwork the webhook itself would join that namespace,
38+
# where the clusterwide policy in packages/system/cilium-networkpolicy
39+
# denies 10250 from the world but has no entry for 10260. Enabling
40+
# hostNetwork means revisiting that policy.
41+
securePort: 10260
1042
resources:
1143
limits:
1244
cpu: 200m

0 commit comments

Comments
 (0)