Skip to content

Commit 4d7568e

Browse files
committed
fix(cilium): take cilium's own addresses from a block kube-ovn does not hand out
On the kube-ovn variants Cilium allocates no pod addresses, but the agent still takes its router IP and the Gateway API Ingress IP from its allocation CIDR. In ipam.mode kubernetes that is Node.spec.podCIDR, which the documented installs slice from the same range kube-ovn allocates pods from, so kube-ovn sooner or later hands a pod an address Cilium already holds. Cilium now runs ipam.mode cluster-pool on the kube-ovn variants, with a /29 per node from the last /21 of networking.podCIDR, and kube-ovn excludes that /21 through --default-exclude-ips. The block stays inside the pod CIDR on purpose. Envoy reaches Gateway API backends from the node's Ingress IP, and kube-ovn masquerades host-originated traffic only when the source lies in one of its subnets, so a block outside them such as 100.65.0.0/16 breaks Gateway API traffic that crosses nodes. A post-upgrade hook moves existing clusters. It adds the block and every node's current Ingress IP to excludeIps of the default subnet, because the agent keeps a former Ingress IP in its ipcache as reserved:ingress. It stops there if kube-ovn has already handed out addresses from the block. Then, one node at a time, it clears the inherited /24 from the CiliumNode, waits for the operator to allocate from the block and restarts the agent. Assisted-by: LLM Signed-off-by: Yan Bondarenko <[email protected]>
1 parent 9620be0 commit 4d7568e

10 files changed

Lines changed: 1280 additions & 8 deletions

File tree

‎hack/cilium-ipam-migration.bats‎

Lines changed: 415 additions & 0 deletions
Large diffs are not rendered by default.

‎packages/core/platform/templates/_helpers.tpl‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -184,3 +184,53 @@ cozystack-scheduler (emitted with its own variant in both branches)
184184
common-packages and has no cilium.io CRD for the controller to watch. */ -}}
185185
{{include "cozystack.platform.package.default" (list "cozystack.securitygroup-controller" $root) }}
186186
{{- end }}
187+
188+
{{/*
189+
The block Cilium takes its own addresses from on the kube-ovn variants, the
190+
last /21 of the pod CIDR, which kube-ovn is told to exclude. Called with the
191+
pod CIDR, "cozystack.platform.ciliumPool" emits the block as a CIDR for the
192+
Cilium operator and "cozystack.platform.ciliumPoolRange" as a first..last range
193+
for kube-ovn excludeIps.
194+
195+
The block has to sit inside the pod CIDR rather than next to it. Envoy reaches
196+
Gateway API backends from the node's Ingress IP, and kube-ovn masquerades
197+
host-originated traffic only when its source lies in one of its subnets.
198+
Outside them the backend's reply is routed to the backend's own node and lost.
199+
The end of the range is the part kube-ovn reaches last, since it hands out
200+
free addresses in ascending order and reuses released ones only once the rest
201+
are gone. A /21 cut into /29s serves 256 nodes, the most a /16 cluster CIDR
202+
sliced into /24s ever allowed.
203+
*/}}
204+
{{- define "cozystack.platform.ciliumPoolBounds" -}}
205+
{{- $cidr := toString . -}}
206+
{{- if not (regexMatch "^[0-9]{1,3}(\\.[0-9]{1,3}){3}/[0-9]{1,2}$" $cidr) -}}
207+
{{- fail (printf "networking.podCIDR %q is not an IPv4 CIDR" $cidr) -}}
208+
{{- end -}}
209+
{{- $parts := splitList "/" $cidr -}}
210+
{{- $prefix := atoi (last $parts) -}}
211+
{{- if gt $prefix 20 -}}
212+
{{- fail (printf "networking.podCIDR %s is too small. The kube-ovn variants reserve its last /21 for Cilium, so it needs a prefix of /20 or shorter" $cidr) -}}
213+
{{- end -}}
214+
{{- $address := 0 -}}
215+
{{- range splitList "." (first $parts) -}}
216+
{{- $address = add (mul $address 256) (atoi .) -}}
217+
{{- end -}}
218+
{{- $size := 1 -}}
219+
{{- range until (sub 32 $prefix | int) -}}
220+
{{- $size = mul $size 2 -}}
221+
{{- end -}}
222+
{{- $first := add (sub $address (mod $address $size)) (sub $size 2048) -}}
223+
{{- printf "%s %s" (include "cozystack.platform.ipv4" $first) (include "cozystack.platform.ipv4" (add $first 2047)) -}}
224+
{{- end }}
225+
226+
{{- define "cozystack.platform.ipv4" -}}
227+
{{- printf "%d.%d.%d.%d" (div . 16777216) (mod (div . 65536) 256) (mod (div . 256) 256) (mod . 256) -}}
228+
{{- end }}
229+
230+
{{- define "cozystack.platform.ciliumPool" -}}
231+
{{- printf "%s/21" (first (splitList " " (include "cozystack.platform.ciliumPoolBounds" .))) -}}
232+
{{- end }}
233+
234+
{{- define "cozystack.platform.ciliumPoolRange" -}}
235+
{{- join ".." (splitList " " (include "cozystack.platform.ciliumPoolBounds" .)) -}}
236+
{{- end }}

‎packages/core/platform/templates/bundles/system.yaml‎

Lines changed: 23 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -22,17 +22,25 @@
2222
{{- if dig "encryption" "enabled" false .Values.networking -}}
2323
{{- $_ := set $kubeovnDict "func" (dict "ENABLE_OVN_IPSEC" true) -}}
2424
{{- end -}}
25-
{{- $kubeovnValues := dict "kube-ovn" $kubeovnDict -}}
26-
{{- $_ := set $networkingComponents "kubeovn" (dict "values" $kubeovnValues) -}}
2725
{{- /* For Talos (isp-full, isp-slim): use KubePrism endpoint and disable cgroup autoMount */ -}}
2826
{{- /* gateway.http2 is threaded explicitly in BOTH states: inline HelmRelease
2927
values override the system/cilium valuesFiles default (enableAlpn: true),
3028
so an operator's http2: false actually reaches the dataplane. */ -}}
31-
{{- $ciliumValues := dict "cilium" (dict
29+
{{- $cilium := dict
3230
"k8sServiceHost" "localhost"
3331
"k8sServicePort" "7445"
3432
"cgroup" (dict "autoMount" (dict "enabled" false))
35-
"gatewayAPI" (dict "enableAlpn" .Values.gateway.http2)) -}}
33+
"gatewayAPI" (dict "enableAlpn" .Values.gateway.http2) -}}
34+
{{- /* Cilium's own addresses come from a block of the pod CIDR that kube-ovn
35+
does not hand out, see cozystack.platform.ciliumPool. The slim variants
36+
run no kube-ovn and keep Cilium allocating pods from Node.spec.podCIDR. */ -}}
37+
{{- if not (include "cozystack.platform.slim" $) -}}
38+
{{- $_ := set $kubeovnDict "networking" (dict "EXCLUDE_IPS" (include "cozystack.platform.ciliumPoolRange" .Values.networking.podCIDR)) -}}
39+
{{- $_ := set $cilium "ipam" (dict "operator" (dict "clusterPoolIPv4PodCIDRList" (list (include "cozystack.platform.ciliumPool" .Values.networking.podCIDR)))) -}}
40+
{{- end -}}
41+
{{- $kubeovnValues := dict "kube-ovn" $kubeovnDict -}}
42+
{{- $_ := set $networkingComponents "kubeovn" (dict "values" $kubeovnValues) -}}
43+
{{- $ciliumValues := dict "cilium" $cilium -}}
3644
{{- $_ := set $networkingComponents "cilium" (dict "values" $ciliumValues) -}}
3745
{{- end -}}
3846
{{include "cozystack.platform.package.networking" (list $ $networkingComponents "kubeovn-cilium" "cilium") }}
@@ -112,15 +120,22 @@
112120
{{- if dig "encryption" "enabled" false .Values.networking -}}
113121
{{- $_ := set $kubeovnDict "func" (dict "ENABLE_OVN_IPSEC" true) -}}
114122
{{- end -}}
115-
{{- $kubeovnValues := dict "kube-ovn" $kubeovnDict -}}
116-
{{- $_ := set $networkingComponents "kubeovn" (dict "values" $kubeovnValues) -}}
117123
{{- /* Cilium configuration - for generic k8s, always enable cgroup autoMount */ -}}
118124
{{- /* gateway.http2 threading: same rationale as the isp-full branch above. */ -}}
119-
{{- $ciliumValues := dict "cilium" (dict
125+
{{- $cilium := dict
120126
"k8sServiceHost" $apiHost
121127
"k8sServicePort" $apiPort
122128
"cgroup" (dict "autoMount" (dict "enabled" true))
123-
"gatewayAPI" (dict "enableAlpn" .Values.gateway.http2)) -}}
129+
"gatewayAPI" (dict "enableAlpn" .Values.gateway.http2) -}}
130+
{{- /* Cilium's address block, same as the isp-full branch above, from the pod
131+
CIDR the cluster actually runs with. */ -}}
132+
{{- if not (include "cozystack.platform.slim" $) -}}
133+
{{- $_ := set $kubeovnDict "networking" (dict "EXCLUDE_IPS" (include "cozystack.platform.ciliumPoolRange" $podCIDR)) -}}
134+
{{- $_ := set $cilium "ipam" (dict "operator" (dict "clusterPoolIPv4PodCIDRList" (list (include "cozystack.platform.ciliumPool" $podCIDR)))) -}}
135+
{{- end -}}
136+
{{- $kubeovnValues := dict "kube-ovn" $kubeovnDict -}}
137+
{{- $_ := set $networkingComponents "kubeovn" (dict "values" $kubeovnValues) -}}
138+
{{- $ciliumValues := dict "cilium" $cilium -}}
124139
{{- $_ := set $networkingComponents "cilium" (dict "values" $ciliumValues) -}}
125140
{{- end -}}
126141
{{- /* Use kubeovn-cilium-generic / cilium-generic (no values-talos.yaml) */ -}}
Lines changed: 133 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,133 @@
1+
suite: Cilium address block on the kube-ovn variants
2+
templates:
3+
- templates/bundles/system.yaml
4+
release:
5+
name: cozystack
6+
namespace: cozy-system
7+
tests:
8+
# The block has to be the same in both places: Cilium allocates its router and
9+
# Ingress IPs from it, and kube-ovn must never hand one of those to a pod.
10+
- it: hands the last /21 of the default pod CIDR to Cilium and excludes it in kube-ovn
11+
set:
12+
bundles:
13+
system:
14+
enabled: true
15+
variant: isp-full
16+
documentSelector:
17+
path: metadata.name
18+
value: cozystack.networking
19+
asserts:
20+
- equal:
21+
path: spec.variant
22+
value: kubeovn-cilium
23+
- equal:
24+
path: spec.components.cilium.values.cilium.ipam.operator.clusterPoolIPv4PodCIDRList
25+
value:
26+
- 10.244.248.0/21
27+
- equal:
28+
path: spec.components.kubeovn.values.kube-ovn.networking.EXCLUDE_IPS
29+
value: 10.244.248.0..10.244.255.255
30+
31+
- it: follows a custom pod CIDR on the generic variant
32+
set:
33+
bundles:
34+
system:
35+
enabled: true
36+
variant: isp-full-generic
37+
networking:
38+
podCIDR: 10.42.0.0/16
39+
documentSelector:
40+
path: metadata.name
41+
value: cozystack.networking
42+
asserts:
43+
- equal:
44+
path: spec.variant
45+
value: kubeovn-cilium-generic
46+
- equal:
47+
path: spec.components.cilium.values.cilium.ipam.operator.clusterPoolIPv4PodCIDRList
48+
value:
49+
- 10.42.248.0/21
50+
- equal:
51+
path: spec.components.kubeovn.values.kube-ovn.networking.EXCLUDE_IPS
52+
value: 10.42.248.0..10.42.255.255
53+
54+
- it: takes the end of a wider pod CIDR and ignores host bits in the network address
55+
set:
56+
bundles:
57+
system:
58+
enabled: true
59+
variant: isp-full
60+
networking:
61+
podCIDR: 10.100.7.1/12
62+
documentSelector:
63+
path: metadata.name
64+
value: cozystack.networking
65+
asserts:
66+
- equal:
67+
path: spec.components.cilium.values.cilium.ipam.operator.clusterPoolIPv4PodCIDRList
68+
value:
69+
- 10.111.248.0/21
70+
- equal:
71+
path: spec.components.kubeovn.values.kube-ovn.networking.EXCLUDE_IPS
72+
value: 10.111.248.0..10.111.255.255
73+
74+
- it: keeps the kube-ovn ipv4 overrides next to the exclusion
75+
set:
76+
bundles:
77+
system:
78+
enabled: true
79+
variant: isp-full
80+
documentSelector:
81+
path: metadata.name
82+
value: cozystack.networking
83+
asserts:
84+
- equal:
85+
path: spec.components.kubeovn.values.kube-ovn.ipv4.POD_CIDR
86+
value: 10.244.0.0/16
87+
- equal:
88+
path: spec.components.cilium.values.cilium.k8sServiceHost
89+
value: localhost
90+
91+
- it: refuses a pod CIDR too small to give up a /21
92+
set:
93+
bundles:
94+
system:
95+
enabled: true
96+
variant: isp-full
97+
networking:
98+
podCIDR: 10.244.0.0/21
99+
asserts:
100+
- failedTemplate:
101+
errorPattern: "networking.podCIDR 10.244.0.0/21 is too small"
102+
103+
- it: refuses a pod CIDR that is not an IPv4 CIDR
104+
set:
105+
bundles:
106+
system:
107+
enabled: true
108+
variant: isp-full
109+
networking:
110+
podCIDR: fd00::/64
111+
asserts:
112+
- failedTemplate:
113+
errorPattern: "is not an IPv4 CIDR"
114+
115+
# The slim variants run Cilium alone. It allocates pod addresses there from
116+
# Node.spec.podCIDR, so neither the block nor kube-ovn belongs in them.
117+
- it: leaves the slim variant's Cilium IPAM alone
118+
set:
119+
bundles:
120+
system:
121+
enabled: true
122+
variant: isp-slim
123+
documentSelector:
124+
path: metadata.name
125+
value: cozystack.networking
126+
asserts:
127+
- equal:
128+
path: spec.variant
129+
value: cilium
130+
- notExists:
131+
path: spec.components.cilium.values.cilium.ipam
132+
- notExists:
133+
path: spec.components.kubeovn

‎packages/core/platform/values.yaml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -84,6 +84,9 @@ bundles:
8484
# Network configuration
8585
networking:
8686
clusterDomain: "cozy.local"
87+
# On the kube-ovn variants the last /21 of podCIDR is reserved for Cilium's
88+
# own router and Gateway API Ingress addresses and excluded from kube-ovn
89+
# allocation, so podCIDR needs a prefix of /20 or shorter.
8790
podCIDR: "10.244.0.0/16"
8891
podGateway: "10.244.0.1"
8992
serviceCIDR: "10.96.0.0/16"

0 commit comments

Comments
 (0)