Skip to content

Commit 6a289cf

Browse files
test(cilium): pin the service-LB assumption both webhook denies rest on
Both admission-webhook deny policies name only their webhook's backend port. That covers a pod dialling the ClusterIP because Cilium translates a service to its backends before it enforces egress policy -- in v1.19.5, bpf/bpf_lxc.c runs __per_packet_lb_svc_xlate_4 and lb4_local() and only then tail-calls into handle_ipv4_from_lxc where policy_can_egress4 evaluates the rule. That ordering exists only while Cilium performs the DNAT, and the only reason it does is cilium's kubeProxyReplacement: true, set against an upstream chart default of "false". Set it false and the DNAT moves downstream of Cilium's egress hook: the packet arrives with dst = ClusterIP, matches no endpoint identity, and with default-deny off it is allowed. Both webhooks regain a pod -reachable ClusterIP path while the backend deny keeps working, so nothing fails and nothing is logged. Same shape as the port disagreement namespace-annotation-webhook.port already exists to prevent, and it had no pin. The suite asserts the rendered cilium-config key rather than .Values, so an upstream bump that renames it or stops honouring the override is caught too, and it spells out each of the five variant valuesFiles combinations from sources/networking.yaml so a future variant file that overrides the value reds here. Proven both ways: flipping values.yaml reds all six cases, and adding the override to values-kilo.yaml alone reds exactly the cilium-kilo case. The suite also records why a target-side ingressDeny is not available as a belt-and-braces fallback. A target-side rule needs a peer selector covering pods, and the peer selector is what carries namespace churn -- the failure being removed here. pkg/policy/selectorcache.go UpdateIdentities notifies a selector only when the churned identities were in its own cached selections, and pkg/policy/l4.go IdentitySelectionUpdated then writes the policy map of every endpoint the rule's endpointSelector selects, which for a target-side rule is the fail-closed webhook pods. The one churn-free peer is the wildcard, which Cilium collapses to a single ANY-identity entry and skips identity updates for -- but ANY covers reserved:host, remote-node and kube-apiserver, so it would deny the API server. There is no third shape. Refs #3983. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <[email protected]>
1 parent 042ac27 commit 6a289cf

7 files changed

Lines changed: 211 additions & 2 deletions

File tree

Lines changed: 148 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,148 @@
1+
suite: cilium owns service load-balancing
2+
3+
# This pins one value, and it is pinned because two other packages depend on it
4+
# in a way that fails OPEN if it is lost. Read this before changing
5+
# kubeProxyReplacement in values.yaml or in any variant values file.
6+
#
7+
# packages/system/ingress-nginx and packages/system/kubeovn-webhook each ship a
8+
# CiliumClusterwideNetworkPolicy that denies pod egress to their admission
9+
# webhook. Both name only the webhook's BACKEND port (8443) and select the
10+
# webhook pods by label; neither names the Service frontend, because Cilium
11+
# cannot express a frontend in a deny rule -- it resolves `toServices` only
12+
# inside allow rules, and an egressDeny whose sole destination is an unresolved
13+
# toServices reaches the datapath as a wildcard, denying every destination on
14+
# that port for every selected pod.
15+
#
16+
# Naming the backend alone still covers a pod dialling the ClusterIP, because
17+
# Cilium translates the service before it enforces egress policy: in Cilium
18+
# v1.19.5, bpf/bpf_lxc.c runs __per_packet_lb_svc_xlate_4 and lb4_local() and
19+
# only then tail-calls CILIUM_CALL_IPV4_FROM_LXC_CONT into handle_ipv4_from_lxc,
20+
# where policy_can_egress4 evaluates the rule. By that point the destination is
21+
# the backend pod on 8443, which is what the rules match.
22+
#
23+
# That ordering exists only while Cilium is the thing doing the DNAT, and the
24+
# only reason it is today is the value asserted below: kubeProxyReplacement,
25+
# "true" here against an upstream chart default of "false". Set it false and the
26+
# ClusterIP DNAT moves downstream of Cilium's egress hook (kube-proxy in the
27+
# host netns, or kube-ovn's OVN load balancer). The packet then reaches
28+
# policy_can_egress4 with dst = ClusterIP, which resolves to no endpoint
29+
# identity and cannot match toEndpoints; both policies run with
30+
# enableDefaultDeny: false, so falling out of the deny IS the access decision
31+
# and the packet is allowed. Every pod regains <webhook>.<ns>.svc:443 while the
32+
# backend-port deny keeps working and nothing reports a change.
33+
#
34+
# Re-adding a target-side ingressDeny on the webhook pods as a second line of
35+
# defence does not work, and that is recorded here because it is the obvious
36+
# next idea. To deny pods, a target-side rule needs a peer selector covering
37+
# them, and the peer selector is what carries namespace churn -- which is the
38+
# failure this whole arrangement was built to remove (#3983):
39+
#
40+
# * pkg/policy/selectorcache.go, SelectorCache.UpdateIdentities ->
41+
# updateSelections, which notifies a selector's users only when the churned
42+
# identities were in ITS cached selections;
43+
# * pkg/policy/l4.go, L4Filter.IdentitySelectionUpdated -> AccumulateMapChanges,
44+
# which writes the policy map of every endpoint the rule's endpointSelector
45+
# selects -- for a target-side rule, the fail-closed webhook pods.
46+
#
47+
# So a cluster-wide non-wildcard peer (`namespace Exists`, or a kube-system
48+
# split) reattaches every namespace identity to the webhook endpoint's own
49+
# policy, which is exactly the shape that was removed. The one churn-free peer
50+
# is the wildcard `{}`: pkg/policy/l4.go collapses it to a single ANY-identity
51+
# map entry and returns early from IdentitySelectionUpdated, so identity churn
52+
# never touches it. But ANY covers the reserved identities too -- host,
53+
# remote-node, kube-apiserver -- so it would deny the API server and stop
54+
# admission cluster-wide. There is no third shape, which is why the dependency
55+
# is pinned here instead of being hedged there.
56+
#
57+
# The other half of the guard is behavioural rather than static: the
58+
# admission-webhook-deny-still-enforced case in
59+
# hack/e2e-chainsaw/gateway/chainsaw-test.yaml connects to the admission
60+
# Service's ClusterIP from a pod and requires the connect to fail, so the effect
61+
# is asserted against a live cluster and not only reasoned about here.
62+
#
63+
# Asserted on the rendered ConfigMap key rather than on .Values, so an upstream
64+
# bump that renames the key or stops honouring our override is caught too --
65+
# that is silent in exactly the same way, and the same argument the
66+
# agent_resources_test.yaml suite next door makes for its own assertions.
67+
#
68+
# One variant per test, spelling out the valuesFiles combination that
69+
# packages/core/platform/sources/networking.yaml actually installs, so a future
70+
# variant file that overrides the value reddens here rather than shipping. Guest
71+
# clusters install this chart with no variant values file at all
72+
# (sources/kubernetes-application.yaml), which the defaults case covers.
73+
release:
74+
name: cilium
75+
namespace: cozy-cilium
76+
tests:
77+
- it: chart defaults keep Cilium in charge of service load-balancing
78+
template: charts/cilium/templates/cilium-configmap.yaml
79+
documentSelector:
80+
path: kind
81+
value: ConfigMap
82+
asserts:
83+
- equal:
84+
path: data["kube-proxy-replacement"]
85+
value: "true"
86+
87+
- it: the cilium variant keeps it
88+
template: charts/cilium/templates/cilium-configmap.yaml
89+
values:
90+
- ../values-talos.yaml
91+
documentSelector:
92+
path: kind
93+
value: ConfigMap
94+
asserts:
95+
- equal:
96+
path: data["kube-proxy-replacement"]
97+
value: "true"
98+
99+
- it: the cilium-kilo variant keeps it
100+
template: charts/cilium/templates/cilium-configmap.yaml
101+
values:
102+
- ../values-talos.yaml
103+
- ../values-kilo.yaml
104+
documentSelector:
105+
path: kind
106+
value: ConfigMap
107+
asserts:
108+
- equal:
109+
path: data["kube-proxy-replacement"]
110+
value: "true"
111+
112+
- it: the cilium-generic variant keeps it
113+
template: charts/cilium/templates/cilium-configmap.yaml
114+
values:
115+
- ../values-apparmor.yaml
116+
documentSelector:
117+
path: kind
118+
value: ConfigMap
119+
asserts:
120+
- equal:
121+
path: data["kube-proxy-replacement"]
122+
value: "true"
123+
124+
- it: the kubeovn-cilium variant keeps it
125+
template: charts/cilium/templates/cilium-configmap.yaml
126+
values:
127+
- ../values-talos.yaml
128+
- ../values-kubeovn.yaml
129+
documentSelector:
130+
path: kind
131+
value: ConfigMap
132+
asserts:
133+
- equal:
134+
path: data["kube-proxy-replacement"]
135+
value: "true"
136+
137+
- it: the kubeovn-cilium-generic variant keeps it
138+
template: charts/cilium/templates/cilium-configmap.yaml
139+
values:
140+
- ../values-apparmor.yaml
141+
- ../values-kubeovn.yaml
142+
documentSelector:
143+
path: kind
144+
value: ConfigMap
145+
asserts:
146+
- equal:
147+
path: data["kube-proxy-replacement"]
148+
value: "true"

‎packages/system/cilium/values.yaml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,14 @@
11
cilium:
2+
# Load-bearing beyond service routing, and pinned for that reason. The
3+
# admission-webhook deny policies in packages/system/ingress-nginx and
4+
# packages/system/kubeovn-webhook name only their webhook's backend port, which
5+
# covers a pod dialling the ClusterIP only while Cilium is the thing that
6+
# performs the service DNAT -- and only this value makes it so, against an
7+
# upstream chart default of "false". Setting it false does not break those
8+
# policies visibly; it reopens the ClusterIP path to both webhooks while the
9+
# backend deny keeps working. tests/kube_proxy_replacement_test.yaml asserts
10+
# the rendered value across the defaults and every variant valuesFiles
11+
# combination, and carries the whole argument.
212
kubeProxyReplacement: true
313
# Upstream cilium with hostFirewall enabled and the IPv6 datapath disabled
414
# (ipv6.enabled=false, the default here) drops ALL node IPv6 before any

‎packages/system/ingress-nginx/templates/admission-webhook-egress-policy.yaml‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,28 @@ carries toPorts (pkg/policy/types/policyentry.go). Such a rule denies every
4343
destination on that port for every selected endpoint -- a cluster-wide
4444
egress outage, not a tighter deny.
4545

46+
That ordering is Cilium's own, and it is conditional: it holds while Cilium owns
47+
service load-balancing, which on every cozystack variant it does because
48+
packages/system/cilium/values.yaml sets kubeProxyReplacement: true against an
49+
upstream chart default of "false". Set that false and the ClusterIP DNAT moves
50+
downstream of Cilium's egress hook (kube-proxy in the host netns, or kube-ovn's
51+
OVN load balancer): the packet reaches policy_can_egress4 with dst = ClusterIP,
52+
matches no endpoint identity, and -- default-deny being off -- is allowed. Pods
53+
regain <admission>.<ns>.svc:443 while the backend deny below keeps working, and
54+
nothing reports the change.
55+
56+
Two things keep that from happening quietly.
57+
packages/system/cilium/tests/kube_proxy_replacement_test.yaml pins the value
58+
across the chart defaults and all five variant valuesFiles combinations, so
59+
flipping it reddens the cilium package's own suite; it also carries the full
60+
argument, including why a target-side ingressDeny cannot be re-added here as a
61+
fallback (its peer selector is what carries namespace churn, and the one
62+
churn-free peer is a wildcard that would deny the API server too). And the
63+
deny-still-enforced case in hack/e2e-chainsaw/gateway/chainsaw-test.yaml
64+
connects to the admission ClusterIP from a pod and requires that connect to
65+
fail, so the effect is asserted against a live cluster rather than only reasoned
66+
about here.
67+
4668
The first rule covers every workload outside kube-system. The second covers
4769
kube-system except the konnectivity-agent ServiceAccount: a Kamaji-hosted
4870
tenant cluster reaches in-cluster webhooks through that pod, and Kamaji sets

‎packages/system/ingress-nginx/values.yaml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -113,5 +113,14 @@ ingress-nginx:
113113
# without an Event or a render-time message to look at. Set enabled: false to
114114
# opt out deliberately, which is what separates "this cluster has no Cilium"
115115
# from "this deny was not wanted here".
116+
#
117+
# The policy denies the webhook's backend port only, which covers a pod dialling
118+
# the Service ClusterIP because Cilium translates a service to its backends
119+
# before it enforces egress policy. That depends on Cilium owning service
120+
# load-balancing -- packages/system/cilium/values.yaml kubeProxyReplacement,
121+
# true there against an upstream default of "false". It is pinned by
122+
# packages/system/cilium/tests/kube_proxy_replacement_test.yaml, which carries
123+
# the argument; flip it and the ClusterIP path to this webhook reopens while the
124+
# backend deny keeps working.
116125
admissionWebhookPolicy:
117126
enabled: true

‎packages/system/kubeovn-webhook/templates/_helpers.tpl‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,9 @@ removes the control silently rather than breaking anything visibly.
1616
The Service frontend port is deliberately NOT defined here. The policy denies
1717
the backend port only, which already covers ClusterIP-addressed traffic because
1818
Cilium translates a service to its backends before enforcing egress policy, so
19-
there is no second value that has to stay in agreement.
19+
there is no second value that has to stay in agreement. That translation order
20+
is itself a pinned assumption -- see
21+
packages/system/cilium/tests/kube_proxy_replacement_test.yaml.
2022
*/}}
2123
{{- define "namespace-annotation-webhook.port" -}}
2224
8443

‎packages/system/kubeovn-webhook/templates/networkpolicy.yaml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,18 @@ WILDCARD when the rule carries toPorts (pkg/policy/types/policyentry.go). That
3535
denies EVERY destination on that port for every selected endpoint -- a
3636
cluster-wide egress outage rather than a tighter deny.
3737

38+
That ordering is Cilium's own and it is conditional: it holds while Cilium owns
39+
service load-balancing, which on every cozystack variant it does because
40+
packages/system/cilium/values.yaml sets kubeProxyReplacement: true against an
41+
upstream chart default of "false". Set that false and the ClusterIP DNAT moves
42+
downstream of Cilium's egress hook, the packet reaches policy_can_egress4 with
43+
dst = ClusterIP, matches no endpoint identity, and -- default-deny being off --
44+
is allowed, restoring pod access to /mutate-pods with nothing to look at.
45+
packages/system/cilium/tests/kube_proxy_replacement_test.yaml pins that value
46+
and carries the full argument, including why a target-side ingressDeny is not
47+
available as a fallback: its peer selector is what carries namespace churn, and
48+
the only churn-free peer is a wildcard that would deny the API server too.
49+
3850
The API server and host-network workloads are not selected: a rule carrying an
3951
endpointSelector can never match the host endpoint, because Cilium routes host
4052
policy through nodeSelector instead (pkg/policy/rule.go, matchesSubject). This

‎packages/system/kubeovn-webhook/values.yaml‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,12 @@ image: ghcr.io/cozystack/cozystack/kubeovn-webhook:v1.6.0@sha256:593c324a38db594
5151
# target-side world deny both read that one definition -- see
5252
# namespace-annotation-webhook.port in templates/_helpers.tpl. Only the backend
5353
# port is denied; Cilium translates a service to its backends before enforcing
54-
# egress policy, so ClusterIP-addressed traffic is matched there too.
54+
# egress policy, so ClusterIP-addressed traffic is matched there too. That last
55+
# step depends on Cilium owning service load-balancing --
56+
# packages/system/cilium/values.yaml kubeProxyReplacement, true there against an
57+
# upstream default of "false", pinned by
58+
# packages/system/cilium/tests/kube_proxy_replacement_test.yaml, which carries
59+
# the argument. Flip it and the ClusterIP path to /mutate-pods reopens while the
60+
# backend deny keeps working.
5561
webhookPolicy:
5662
enabled: true

0 commit comments

Comments
 (0)