|
| 1 | +suite: cilium owns service load-balancing |
| 2 | + |
| 3 | +# This pins one value, and it is pinned because two other packages depend on it |
| 4 | +# in a way that fails OPEN if it is lost. Read this before changing |
| 5 | +# kubeProxyReplacement in values.yaml or in any variant values file. |
| 6 | +# |
| 7 | +# packages/system/ingress-nginx and packages/system/kubeovn-webhook each ship a |
| 8 | +# CiliumClusterwideNetworkPolicy that denies pod egress to their admission |
| 9 | +# webhook. Both name only the webhook's BACKEND port (8443) and select the |
| 10 | +# webhook pods by label; neither names the Service frontend, because Cilium |
| 11 | +# cannot express a frontend in a deny rule -- it resolves `toServices` only |
| 12 | +# inside allow rules, and an egressDeny whose sole destination is an unresolved |
| 13 | +# toServices reaches the datapath as a wildcard, denying every destination on |
| 14 | +# that port for every selected pod. |
| 15 | +# |
| 16 | +# Naming the backend alone still covers a pod dialling the ClusterIP, because |
| 17 | +# Cilium translates the service before it enforces egress policy: in Cilium |
| 18 | +# v1.19.5, bpf/bpf_lxc.c runs __per_packet_lb_svc_xlate_4 and lb4_local() and |
| 19 | +# only then tail-calls CILIUM_CALL_IPV4_FROM_LXC_CONT into handle_ipv4_from_lxc, |
| 20 | +# where policy_can_egress4 evaluates the rule. By that point the destination is |
| 21 | +# the backend pod on 8443, which is what the rules match. |
| 22 | +# |
| 23 | +# That ordering exists only while Cilium is the thing doing the DNAT, and the |
| 24 | +# only reason it is today is the value asserted below: kubeProxyReplacement, |
| 25 | +# "true" here against an upstream chart default of "false". Set it false and the |
| 26 | +# ClusterIP DNAT moves downstream of Cilium's egress hook (kube-proxy in the |
| 27 | +# host netns, or kube-ovn's OVN load balancer). The packet then reaches |
| 28 | +# policy_can_egress4 with dst = ClusterIP, which resolves to no endpoint |
| 29 | +# identity and cannot match toEndpoints; both policies run with |
| 30 | +# enableDefaultDeny: false, so falling out of the deny IS the access decision |
| 31 | +# and the packet is allowed. Every pod regains <webhook>.<ns>.svc:443 while the |
| 32 | +# backend-port deny keeps working and nothing reports a change. |
| 33 | +# |
| 34 | +# Re-adding a target-side ingressDeny on the webhook pods as a second line of |
| 35 | +# defence does not work, and that is recorded here because it is the obvious |
| 36 | +# next idea. To deny pods, a target-side rule needs a peer selector covering |
| 37 | +# them, and the peer selector is what carries namespace churn -- which is the |
| 38 | +# failure this whole arrangement was built to remove (#3983): |
| 39 | +# |
| 40 | +# * pkg/policy/selectorcache.go, SelectorCache.UpdateIdentities -> |
| 41 | +# updateSelections, which notifies a selector's users only when the churned |
| 42 | +# identities were in ITS cached selections; |
| 43 | +# * pkg/policy/l4.go, L4Filter.IdentitySelectionUpdated -> AccumulateMapChanges, |
| 44 | +# which writes the policy map of every endpoint the rule's endpointSelector |
| 45 | +# selects -- for a target-side rule, the fail-closed webhook pods. |
| 46 | +# |
| 47 | +# So a cluster-wide non-wildcard peer (`namespace Exists`, or a kube-system |
| 48 | +# split) reattaches every namespace identity to the webhook endpoint's own |
| 49 | +# policy, which is exactly the shape that was removed. The one churn-free peer |
| 50 | +# is the wildcard `{}`: pkg/policy/l4.go collapses it to a single ANY-identity |
| 51 | +# map entry and returns early from IdentitySelectionUpdated, so identity churn |
| 52 | +# never touches it. But ANY covers the reserved identities too -- host, |
| 53 | +# remote-node, kube-apiserver -- so it would deny the API server and stop |
| 54 | +# admission cluster-wide. There is no third shape, which is why the dependency |
| 55 | +# is pinned here instead of being hedged there. |
| 56 | +# |
| 57 | +# The other half of the guard is behavioural rather than static: the |
| 58 | +# admission-webhook-deny-still-enforced case in |
| 59 | +# hack/e2e-chainsaw/gateway/chainsaw-test.yaml connects to the admission |
| 60 | +# Service's ClusterIP from a pod and requires the connect to fail, so the effect |
| 61 | +# is asserted against a live cluster and not only reasoned about here. |
| 62 | +# |
| 63 | +# Asserted on the rendered ConfigMap key rather than on .Values, so an upstream |
| 64 | +# bump that renames the key or stops honouring our override is caught too -- |
| 65 | +# that is silent in exactly the same way, and the same argument the |
| 66 | +# agent_resources_test.yaml suite next door makes for its own assertions. |
| 67 | +# |
| 68 | +# One variant per test, spelling out the valuesFiles combination that |
| 69 | +# packages/core/platform/sources/networking.yaml actually installs, so a future |
| 70 | +# variant file that overrides the value reddens here rather than shipping. Guest |
| 71 | +# clusters install this chart with no variant values file at all |
| 72 | +# (sources/kubernetes-application.yaml), which the defaults case covers. |
| 73 | +release: |
| 74 | + name: cilium |
| 75 | + namespace: cozy-cilium |
| 76 | +tests: |
| 77 | + - it: chart defaults keep Cilium in charge of service load-balancing |
| 78 | + template: charts/cilium/templates/cilium-configmap.yaml |
| 79 | + documentSelector: |
| 80 | + path: kind |
| 81 | + value: ConfigMap |
| 82 | + asserts: |
| 83 | + - equal: |
| 84 | + path: data["kube-proxy-replacement"] |
| 85 | + value: "true" |
| 86 | + |
| 87 | + - it: the cilium variant keeps it |
| 88 | + template: charts/cilium/templates/cilium-configmap.yaml |
| 89 | + values: |
| 90 | + - ../values-talos.yaml |
| 91 | + documentSelector: |
| 92 | + path: kind |
| 93 | + value: ConfigMap |
| 94 | + asserts: |
| 95 | + - equal: |
| 96 | + path: data["kube-proxy-replacement"] |
| 97 | + value: "true" |
| 98 | + |
| 99 | + - it: the cilium-kilo variant keeps it |
| 100 | + template: charts/cilium/templates/cilium-configmap.yaml |
| 101 | + values: |
| 102 | + - ../values-talos.yaml |
| 103 | + - ../values-kilo.yaml |
| 104 | + documentSelector: |
| 105 | + path: kind |
| 106 | + value: ConfigMap |
| 107 | + asserts: |
| 108 | + - equal: |
| 109 | + path: data["kube-proxy-replacement"] |
| 110 | + value: "true" |
| 111 | + |
| 112 | + - it: the cilium-generic variant keeps it |
| 113 | + template: charts/cilium/templates/cilium-configmap.yaml |
| 114 | + values: |
| 115 | + - ../values-apparmor.yaml |
| 116 | + documentSelector: |
| 117 | + path: kind |
| 118 | + value: ConfigMap |
| 119 | + asserts: |
| 120 | + - equal: |
| 121 | + path: data["kube-proxy-replacement"] |
| 122 | + value: "true" |
| 123 | + |
| 124 | + - it: the kubeovn-cilium variant keeps it |
| 125 | + template: charts/cilium/templates/cilium-configmap.yaml |
| 126 | + values: |
| 127 | + - ../values-talos.yaml |
| 128 | + - ../values-kubeovn.yaml |
| 129 | + documentSelector: |
| 130 | + path: kind |
| 131 | + value: ConfigMap |
| 132 | + asserts: |
| 133 | + - equal: |
| 134 | + path: data["kube-proxy-replacement"] |
| 135 | + value: "true" |
| 136 | + |
| 137 | + - it: the kubeovn-cilium-generic variant keeps it |
| 138 | + template: charts/cilium/templates/cilium-configmap.yaml |
| 139 | + values: |
| 140 | + - ../values-apparmor.yaml |
| 141 | + - ../values-kubeovn.yaml |
| 142 | + documentSelector: |
| 143 | + path: kind |
| 144 | + value: ConfigMap |
| 145 | + asserts: |
| 146 | + - equal: |
| 147 | + path: data["kube-proxy-replacement"] |
| 148 | + value: "true" |
0 commit comments