Skip to content

Commit 02c6942

Browse files
authored
fix(kubeovn-webhook): restrict the webhook port to the API server (#3955)
## What this PR does Closes the exposure behind GHSA-g883-q79m-8225 by constraining who can reach the kube-ovn webhook's serving port. The `/mutate-pods` handler returns namespace-derived annotations to any caller that reaches it, and until now nothing stopped an arbitrary in-cluster pod from being that caller. ### Why the client-certificate path does not close it #2698 added `--client-ca-file`, deliberately off by default, as observation mode. Turning it on requires the API server to **present** a certificate, which it does only when its `AdmissionConfiguration` names a `kubeConfigFile` for the webhook plugins. Checked against a running Cozystack control plane rather than assumed: the `AdmissionConfiguration` there carries a `PodSecurity` plugin block and nothing else, so no client certificate is offered to admission webhooks at all. Flipping to `RequireAndVerifyClientCert` on such a cluster would reject the API server itself, and with `failurePolicy: Fail` that stops pod creation cluster-wide. This matters for how the remaining work was scoped: the plan had been to observe issuer logs for a day and then pin a CA. No amount of observation changes the outcome, because the absence is a property of the control plane's configuration, not of the traffic. Making `--client-ca-file` usable is a separate, larger change to how the platform configures the API server, and it is not a prerequisite for closing the exposure. ### What the policy does Denies pod traffic and world traffic to the serving port, and needs no carve-out for the legitimate caller: the API server runs on the host network, so it is not a Cilium endpoint and `fromEndpoints` never matches it. Unlike the ingress-nginx admission webhook this component exists only on the management cluster, so there is no konnectivity-tunnelled caller to exempt either — and consequently no forgeable pod label anywhere in the selector. It is a `CiliumNetworkPolicy` rather than a core `NetworkPolicy`, and that distinction is the reason it works at all: `packages/system/kubeovn/values.yaml` sets `ENABLE_NP: false`, so kube-ovn programs no ACLs and a core NetworkPolicy would be created and never enforced. The values comment that claimed NetworkPolicy was simply not an option is corrected — it was true only of the core kind. `webhookPolicy.enabled` lets a cluster whose networking variant ships no Cilium turn the object off. Deny-only with default-deny explicitly disabled, matching the ingress-nginx admission policy: putting these pods behind a default-deny would mean enumerating every port they serve and breaking traffic on the first one forgotten. ### Verification Eight helm-unittest cases cover the port, the all-namespaces span, the absence of any label-based exemption, the world deny, the disabled default-deny, the Cilium kind, the selector and the off switch. The rendered object was additionally accepted by a live API server via `kubectl apply --dry-run=server`, so the CRD validates it as written. Not verified: that a denied pod actually fails to reach the port at runtime. That needs a live negative test against the policy, which is the same gap the ingress-nginx admission policy carries. ### Screenshots Not applicable — no UI change. ### Downstream repositories Walked the trigger map against the diff. Nothing is reached: the change is one chart template, its values and its tests under `packages/system/`, with no package added or removed under `packages/apps/` or `packages/extra/`, no `packages/core/platform/values.yaml` key, no platform variant or bundle change, nothing under `hack/`, and no node prerequisite. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(kubeovn-webhook): the kube-ovn webhook's serving port is now closed to in-cluster pods and external traffic by a CiliumNetworkPolicy. It previously answered any in-cluster pod with namespace-derived annotations (GHSA-g883-q79m-8225). Disable with webhookPolicy.enabled=false on clusters whose networking variant ships no Cilium. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a Cilium network policy to protect the namespace-annotation webhook. * The policy is enabled by default and restricts access to the webhook port, blocking unauthorized in-cluster and external traffic. * Standardized the webhook port across its deployment, service, and network policy configuration. * **Bug Fixes** * Improved webhook network isolation when standard Kubernetes NetworkPolicy enforcement is unavailable. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents eef95f4 + a7cef35 commit 02c6942

8 files changed

Lines changed: 234 additions & 6 deletions

File tree

‎.github/workflows/pr-labeler.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -99,6 +99,7 @@ jobs:
9999
'vpn': 'area/networking',
100100
'metallb': 'area/networking',
101101
'kube-ovn': 'area/networking',
102+
'kubeovn-webhook': 'area/networking',
102103
'tcp-balancer': 'area/networking',
103104
'securitygroups': 'area/networking',
104105
'cozy-proxy': 'area/networking',

‎packages/system/kubeovn-webhook/templates/_helpers.tpl‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,3 +5,14 @@ kube-ovn-webhook
55
{{- define "namespace-annotation-webhook.fullname" -}}
66
kube-ovn-webhook
77
{{- end }}
8+
9+
{{/*
10+
The port the webhook serves on. Defined once and consumed by the container
11+
port, the Service targetPort and the CiliumNetworkPolicy, because those three
12+
must agree: the policy has default-deny disabled, so a policy pointed at a port
13+
nothing serves denies nothing while the real port stays open -- a drift that
14+
removes the control silently rather than breaking anything visibly.
15+
*/}}
16+
{{- define "namespace-annotation-webhook.port" -}}
17+
8443
18+
{{- end }}

‎packages/system/kubeovn-webhook/templates/deployment.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,7 @@ spec:
3939
cpu: 100m
4040
memory: 128Mi
4141
ports:
42-
- containerPort: 8443
42+
- containerPort: {{ include "namespace-annotation-webhook.port" . }}
4343
name: https
4444
volumeMounts:
4545
- name: webhook-certs
Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
{{- if .Values.webhookPolicy.enabled }}
2+
---
3+
{{/*
4+
The /mutate-pods handler answers whoever asks it with namespace-derived
5+
annotations, and only the API server has any business calling it
6+
(GHSA-g883-q79m-8225). Client-certificate verification cannot close this on
7+
its own: enabling --client-ca-file requires the API server to *present* a
8+
certificate, which it does only when its AdmissionConfiguration names a
9+
kubeConfigFile for the webhook plugins. A stock Cozystack control plane ships
10+
a PodSecurity plugin block and nothing else, so no certificate is offered and
11+
RequireAndVerifyClientCert would reject the API server itself -- with
12+
failurePolicy: Fail that stops pod creation cluster-wide. Until that changes,
13+
the network is the only place the caller can actually be constrained.
14+
15+
This is a CiliumNetworkPolicy, not a core NetworkPolicy, and the distinction
16+
is the whole reason it works: packages/system/kubeovn/values.yaml sets
17+
ENABLE_NP: false, so kube-ovn programs no ACLs and a core NetworkPolicy would
18+
be created and never enforced. Cilium enforces its own policies regardless,
19+
which is why the platform already relies on them elsewhere.
20+
21+
Deny-only, with default-deny explicitly disabled. A Cilium rule carrying an
22+
ingress section puts the selected endpoints behind a default-deny, which here
23+
would mean enumerating every port these pods serve and silently breaking
24+
traffic on the first one forgotten. Subtracting one port leaves everything
25+
else exactly as it was.
26+
27+
The API server needs no carve-out, and this was confirmed on a live kube-ovn
28+
cluster rather than reasoned about: cilium monitor on the webhook's endpoint
29+
shows the API server's connection arriving from the node InternalIP with
30+
identity "host" when it originates on the same node and "kube-apiserver" when
31+
it originates on another, never from the kube-ovn join subnet and never as
32+
"world". Neither identity is a Cilium endpoint, so fromEndpoints cannot match
33+
it and the world deny does not apply to it. Unlike the ingress-nginx admission
34+
webhook, this component exists only on the management cluster, so there is no
35+
konnectivity-tunnelled caller to exempt either.
36+
37+
Note the scope this actually has: it removes in-cluster pods and external
38+
traffic, not every possible caller. Any host-network workload on the
39+
management cluster carries the same host / remote-node identity that exempts
40+
the API server and keeps its reach.
41+
*/}}
42+
apiVersion: cilium.io/v2
43+
kind: CiliumNetworkPolicy
44+
metadata:
45+
name: {{ include "namespace-annotation-webhook.name" . }}
46+
namespace: {{ .Release.Namespace }}
47+
spec:
48+
endpointSelector:
49+
matchLabels:
50+
app.kubernetes.io/name: {{ include "namespace-annotation-webhook.name" . }}
51+
app.kubernetes.io/instance: {{ .Release.Name }}
52+
enableDefaultDeny:
53+
ingress: false
54+
ingressDeny:
55+
{{- /*
56+
Every pod in the cluster. The namespace key is matched with Exists so the rule
57+
spans all namespaces: a plain fromEndpoints in a namespaced
58+
CiliumNetworkPolicy would only ever select pods sharing this policy's
59+
namespace, which would leave the webhook open to every other one.
60+
*/}}
61+
- fromEndpoints:
62+
- matchExpressions:
63+
- key: k8s:io.kubernetes.pod.namespace
64+
operator: Exists
65+
toPorts:
66+
- ports:
67+
- port: {{ include "namespace-annotation-webhook.port" . | quote }}
68+
protocol: TCP
69+
- fromEntities:
70+
- world
71+
toPorts:
72+
- ports:
73+
- port: {{ include "namespace-annotation-webhook.port" . | quote }}
74+
protocol: TCP
75+
{{- end }}

‎packages/system/kubeovn-webhook/templates/service.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ spec:
99
type: ClusterIP
1010
ports:
1111
- port: 443
12-
targetPort: 8443
12+
targetPort: {{ include "namespace-annotation-webhook.port" . }}
1313
protocol: TCP
1414
name: https
1515
selector:
Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
suite: kubeovn-webhook exposure
2+
release:
3+
name: kubeovn-webhook
4+
namespace: cozy-kubeovn
5+
templates:
6+
- templates/networkpolicy.yaml
7+
tests:
8+
- it: denies pod traffic to the webhook port
9+
# The /mutate-pods handler returns namespace-derived annotations to any
10+
# caller that reaches it (GHSA-g883-q79m-8225), so closing the port to pods
11+
# is the whole point of the object.
12+
asserts:
13+
- equal:
14+
path: spec.ingressDeny[0].toPorts[0].ports[0].port
15+
value: "8443"
16+
- equal:
17+
path: spec.ingressDeny[0].toPorts[0].ports[0].protocol
18+
value: TCP
19+
20+
- it: spans every namespace rather than only the release namespace
21+
# A bare fromEndpoints in a namespaced CiliumNetworkPolicy selects only pods
22+
# in the policy's own namespace, which would leave the webhook reachable
23+
# from every other one — the exact gap being closed.
24+
asserts:
25+
- contains:
26+
path: spec.ingressDeny[0].fromEndpoints[0].matchExpressions
27+
content:
28+
key: k8s:io.kubernetes.pod.namespace
29+
operator: Exists
30+
31+
- it: carries no label-based exemption
32+
# There is no caller to carve out: the API server runs on the host network
33+
# and is therefore not a Cilium endpoint, so fromEndpoints never matches it.
34+
# A label exemption would be both unnecessary and forgeable — any workload
35+
# can set a pod label on itself, and with default-deny disabled, falling out
36+
# of the deny is the entire access decision.
37+
asserts:
38+
- lengthEqual:
39+
path: spec.ingressDeny[0].fromEndpoints[0].matchExpressions
40+
count: 1
41+
- lengthEqual:
42+
path: spec.ingressDeny[0].fromEndpoints
43+
count: 1
44+
45+
- it: denies world traffic to the webhook port
46+
asserts:
47+
- contains:
48+
path: spec.ingressDeny[1].fromEntities
49+
content: world
50+
- equal:
51+
path: spec.ingressDeny[1].toPorts[0].ports[0].port
52+
value: "8443"
53+
54+
- it: does not put the webhook behind a default deny
55+
# Enabling default-deny would require enumerating every port these pods
56+
# serve and would break traffic on the first one omitted.
57+
asserts:
58+
- equal:
59+
path: spec.enableDefaultDeny.ingress
60+
value: false
61+
- isNull:
62+
path: spec.ingress
63+
64+
- it: is a CiliumNetworkPolicy rather than a core NetworkPolicy
65+
# kube-ovn runs with ENABLE_NP: false, so a core NetworkPolicy would be
66+
# created and never enforced. Cilium enforces its own policies regardless.
67+
asserts:
68+
- isKind:
69+
of: CiliumNetworkPolicy
70+
- equal:
71+
path: apiVersion
72+
value: cilium.io/v2
73+
74+
- it: selects only the webhook pods
75+
asserts:
76+
- equal:
77+
path: spec.endpointSelector.matchLabels["app.kubernetes.io/instance"]
78+
value: kubeovn-webhook
79+
80+
- it: can be disabled where no Cilium is present to enforce it
81+
set:
82+
webhookPolicy.enabled: false
83+
asserts:
84+
- hasDocuments:
85+
count: 0
Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
suite: kubeovn-webhook port agreement
2+
release:
3+
name: kubeovn-webhook
4+
namespace: cozy-kubeovn
5+
templates:
6+
- templates/networkpolicy.yaml
7+
- templates/deployment.yaml
8+
- templates/service.yaml
9+
tests:
10+
- it: protects exactly the port the webhook serves and the Service routes to
11+
# The regression guard for the one drift that fails open. The policy runs
12+
# with default-deny disabled, so a policy port that disagrees with the
13+
# container port denies a port nothing serves and leaves the real one
14+
# reachable -- the control disappears without anything breaking visibly.
15+
# All three render from namespace-annotation-webhook.port for that reason.
16+
asserts:
17+
- equal:
18+
path: spec.ingressDeny[0].toPorts[0].ports[0].port
19+
value: "8443"
20+
template: templates/networkpolicy.yaml
21+
- equal:
22+
path: spec.ingressDeny[1].toPorts[0].ports[0].port
23+
value: "8443"
24+
template: templates/networkpolicy.yaml
25+
- equal:
26+
path: spec.template.spec.containers[0].ports[0].containerPort
27+
value: 8443
28+
template: templates/deployment.yaml
29+
- equal:
30+
path: spec.ports[0].targetPort
31+
value: 8443
32+
template: templates/service.yaml

‎packages/system/kubeovn-webhook/values.yaml‎

Lines changed: 28 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -4,10 +4,12 @@ routes: ""
44
#
55
# The handler returns namespace-derived annotations to whoever asks it, so the
66
# only thing that keeps it from answering an arbitrary in-cluster pod is
7-
# authenticating the caller (GHSA-g883-q79m-8225). NetworkPolicy is not an
8-
# option here: packages/system/kubeovn/values.yaml sets ENABLE_NP: false, so
9-
# kube-ovn-controller programs no ACLs and a NetworkPolicy object would be
10-
# created and never enforced.
7+
# authenticating the caller (GHSA-g883-q79m-8225), or constraining who can
8+
# reach it at all -- see webhookPolicy below, which does the latter. A *core*
9+
# NetworkPolicy is not an option here: packages/system/kubeovn/values.yaml sets
10+
# ENABLE_NP: false, so kube-ovn-controller programs no ACLs and the object would
11+
# be created and never enforced. A CiliumNetworkPolicy is enforced by Cilium
12+
# regardless, which is the path webhookPolicy takes.
1113
#
1214
# Empty (the default) = certificates are requested and logged, never required.
1315
# Nothing is rejected, so this cannot break admission. Read the webhook log:
@@ -27,3 +29,25 @@ routes: ""
2729
# observation above can run, not so it can be switched on.
2830
clientCAFile: ""
2931
image: ghcr.io/cozystack/cozystack/kubeovn-webhook:v1.6.0@sha256:593c324a38db59495497c5b68b90cad25dfe6753c73b7a287b3280e26f70a15f
32+
33+
# Restrict who may reach the webhook's serving port.
34+
#
35+
# The handler returns namespace-derived annotations to any caller, and only the
36+
# API server should be calling it. clientCAFile above cannot enforce that on a
37+
# stock control plane -- the API server presents no client certificate unless
38+
# its AdmissionConfiguration names a kubeConfigFile for the webhook plugins,
39+
# and a stock Cozystack control plane ships only a PodSecurity block. Until
40+
# that changes, this policy is the enforcement.
41+
#
42+
# Rendered as a CiliumNetworkPolicy on purpose: kube-ovn runs with
43+
# ENABLE_NP: false, so a core NetworkPolicy object would be created and never
44+
# enforced. Set enabled: false on a cluster whose networking variant ships no
45+
# Cilium, where the object would have no controller to act on it.
46+
# The port is deliberately not configurable here. It has to agree with the
47+
# container port and the Service targetPort, and because the policy runs with
48+
# default-deny disabled, a disagreement fails open: the policy would deny a port
49+
# nothing serves while the real one stayed reachable, removing this control
50+
# without breaking anything visible. All three read one definition instead --
51+
# see namespace-annotation-webhook.port in templates/_helpers.tpl.
52+
webhookPolicy:
53+
enabled: true

0 commit comments

Comments
 (0)