Skip to content

Commit 87a6407

Browse files
authored
fix(cilium)!: take cilium's own addresses from a block kube-ovn does not hand out (#4778)
## What this PR does fixes #3740. on the kube-ovn variants cilium allocates no pod ips, but the agent still takes its router ip (`cilium_host`, `CiliumInternalIP`) and the gateway api ingress ip from its own allocation cidr. with `ipam.mode: kubernetes` that's `Node.spec.podCIDR`, a /24 out of the same /16 kube-ovn hands pods from, so sooner or later kube-ovn gives a pod an address cilium already holds. we hit it on three clusters, details in #3740 (comment). **what changes** - the kube-ovn variants run cilium with `ipam.mode: cluster-pool`, a /29 per node out of the last /21 of `networking.podCIDR` (`10.244.248.0/21` for the default /16). the platform derives the block from `networking.podCIDR` and passes the same range to kube-ovn as `--default-exclude-ips`, so a fresh install never hands it to pods. - the block sits inside the pod cidr on purpose. envoy reaches gateway api backends from the node's ingress ip, and kube-ovn masquerades host originated traffic only when the source is in `ovn40subnets`. with the `100.65.0.0/16` pool from the issue a nodeport through the gateway answered 200 only on the backend's node and 503 `upstream connect error ... connection timeout` on the other two. with the block inside the pod cidr it's 200 from all three. worth a look on the ci side too. the ci sandbox sets talos `podSubnets` to `100.65.0.0/16`, so cilium's ingress ips there already come from outside `ovn40subnets`, and the gateway e2e checks Accepted, Programmed and the lb service but sends no request through a listener, so it wouldn't notice. with this pr the sandbox takes its block from the pod cidr like everyone else. - the end of the /16 is what kube-ovn reaches last. it hands out the lowest free address and reuses released ones only when the free range runs dry (`pkg/ipam/subnet.go` and `pkg/ipam/ip_range_list.go` at v1.15.26). a /21 cut into /29s covers 256 nodes. - the slim variants and guest clusters stay on `ipam.mode: kubernetes`. cilium is the only cni there and `Node.spec.podCIDR` is the pod range. - `networking.podCIDR` now needs /20 or wider on the kube-ovn variants, otherwise the render fails and says why. an install that rendered before can stop rendering at upgrade, hence the `!`. a /21 cut into /29s also caps these variants at 256 nodes, a fresh install included. **existing clusters** a post-upgrade hook in the cilium chart moves them (`templates/ipam-migration.yaml`, `files/ipam-migration.sh`). before it touches anything it checks that the block has a /29 for every CiliumNode and that it lies inside kube-ovn's default subnet, and stops with both cidrs printed otherwise. a node cleared past the pool's capacity would lose pod networking on its next agent restart, and a block outside the subnet puts the agents outside `ovn40subnets` again. it has to be post-upgrade. platform migrations run as the platform's pre-upgrade hook, while every agent still runs `ipam.mode: kubernetes` and writes `Node.spec.podCIDR` back into a cleared CiliumNode (`pkg/nodediscovery/nodediscovery.go` at v1.19.5), so the move can only happen after the cilium release switched the mode. the job runs on the host network against the agent's own api endpoint (kubeprism on talos), because it restarts the agents. 1. adds the block and every node's current ingress ip to `excludeIps` of the default subnet, in one json patch with a `test` op, and reads the subnet again on conflict, since kube-ovn rewrites that list into its own order right after a patch. the old ingress ip goes in because cilium 1.19.5 keeps a former ingress ip in the agent's ipcache as `reserved:ingress` after the move. we saw 2 to 3 such entries per node hours later. old router ips leave the ipcache and go back to kube-ovn. 2. stops right there if an address in the block is still allocated, names it, and moves no agent. it first checks the subnet status it patched against, then waits until `status.v4availableIPrange` stops offering the block, which is kube-ovn having applied the exclusion, and then reads the `ips.kubeovn.io` objects of the default subnet. the status alone can't answer it, kube-ovn leaves excluded addresses out of `v4usingIPrange` (`GetSubnetIPRangeString` in `pkg/ipam/ipam.go` at v1.15.26), so a pod still in the block would be invisible on a retry where the exclusion is already in place. recreate those pods and the next upgrade retry goes through. 3. one node at a time, clears the inherited /24 from `spec.ipam.podCIDRs` (again with a `test` op, so a concurrent writer fails the patch instead of losing an update), waits for the operator to allocate a /29 from the block and restarts that node's agent. a node that isn't ready is skipped and its agent picks up the block when it starts. nodes already moved are left alone, so a retry after a timeout just continues. the hook runs inside the cilium upgrade, so where restarting every agent takes longer than the upgrade timeout, 10m by default, the release first reports a failed upgrade and the move finishes on the retry. running pods keep their addresses. every agent restarts twice on upgrade, once when the config change rolls the daemonset and once from the hook, and while it restarts host to pod traffic on that node blips for a few seconds because the router ip moves. a pod that sits on a router ip today is reachable again after the move, the node no longer holds that address. a pod on a node's current ingress ip keeps it and that node keeps seeing it as `reserved:ingress`, so recreate it. the hook logs those pods by name. ### Validation - `helm unittest` passes cilium 23/23, platform 246/246 and kube-ovn 22/22. the new suites pin the rendered `cilium-config` keys per variant, the hook shape, env and rbac, and the platform wiring for the talos and generic branches, a custom pod cidr, host bits in the network address, the refusals for a too small or non ipv4 cidr, and the slim variant. - on a trial merge with main at cbc9d0c, where #3849 runs `hack/*.bats` under real bats 1.14 with `set -u`, the 27 bats tests pass after the second commit loads `test_helper`, the strict setup audit is clean, and helm unittest passes cilium 23/23, platform 263/263 and kube-ovn 22/22. - `hack/cilium-ipam-migration.bats` passes 27/27 and runs the real script against a fake kubectl that models the operator, the agent daemonset and kube-ovn publishing the subnet status. thirteen mutations turn it red each (exclusion after the move, no ingress exclusion, no retry, no status check, an ingress ip inside the block listed on its own, no `test` op, no wait for kube-ovn, no ip object check, no subnet filter on the ip objects, no capacity check, a block outside the subnet let through, overlap turned into containment, no log of ingress ip holders). the containment one is pinned by a usage range that crosses the start of the block, `10.244.247.250-10.244.248.3`. - live on a 3 node talos test cluster with cozystack v1.6.4, cilium 1.19.5 and kube-ovn v1.15.26 - moved from `kubernetes` to the block, exclusion first and then the node moves this script does. 3 nodes in about 40 seconds, every agent took its router and ingress ip from its /29, no pod changed its address. - gateway api through a nodeport 9/9 answered 200 and through the loadbalancer ip 6/6, against 3/9 with `100.65.0.0/16`. - the final hook job on the moved cluster is a no-op in 2 seconds, including the ip object read with the job's own rbac. the same ip object pipeline over that cluster's real objects finds 77 in a populated /24 and none in the block. - the exact `test` plus `replace` patch on `ovn-default` goes through, kube-ovn re-sorts the list right after, and the same patch against the old list is rejected. that's the conflict path the retry covers. - shellcheck is clean and `hack/check-commit-trailers.sh` passes. - not run here. the upgrade move from an older release in ci, e2e only covers the fresh install. ### Screenshots not a ui change. ### Downstream repositories walked the trigger map against the diff. `packages/core/platform/values.yaml` only gains a comment, but `networking.podCIDR` gets a new constraint on the kube-ovn variants, which the hand written key table on the website should say. the ansible role sets `networking.podCIDR` to `10.42.0.0/16`, which yields `10.42.248.0/21` and needs nothing there. the talm preset keeps `podSubnets: 10.244.0.0/16`, which no longer matters on these variants because cilium stops reading `Node.spec.podCIDR` there. the terraform provider doesn't model platform values. - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: cozystack/website#739 - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(cilium)!: on the kube-ovn variants cilium now takes its router and gateway api ingress ips from the last /21 of networking.podCIDR, which kube-ovn no longer hands out, so pods stop landing on addresses cilium holds. existing clusters move on upgrade through a post-upgrade hook, and each cilium agent restarts twice, once for the config change and once in the hook. networking.podCIDR needs /20 or wider on these variants, and a /21 of /29s caps them at 256 nodes. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Kube-OVN-enabled variants now reserve the final `/21` of the pod CIDR for Cilium router and Gateway API Ingress addresses, excluding it from Kube-OVN pod allocation. Pod CIDRs must be IPv4 with a prefix of `/20` or shorter. * Added a post-upgrade migration that moves existing Cilium IP allocations into the cluster pool and updates affected node networking. * **Documentation** * Documented the pod CIDR requirements and node capacity limit for Kube-OVN variants. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 76c030d + f944eef commit 87a6407

10 files changed

Lines changed: 1539 additions & 8 deletions

File tree

‎hack/cilium-ipam-migration.bats‎

Lines changed: 563 additions & 0 deletions
Large diffs are not rendered by default.

‎packages/core/platform/templates/_helpers.tpl‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -184,3 +184,53 @@ cozystack-scheduler (emitted with its own variant in both branches)
184184
common-packages and has no cilium.io CRD for the controller to watch. */ -}}
185185
{{include "cozystack.platform.package.default" (list "cozystack.securitygroup-controller" $root) }}
186186
{{- end }}
187+
188+
{{/*
189+
The block Cilium takes its own addresses from on the kube-ovn variants, the
190+
last /21 of the pod CIDR, which kube-ovn is told to exclude. Called with the
191+
pod CIDR, "cozystack.platform.ciliumPool" emits the block as a CIDR for the
192+
Cilium operator and "cozystack.platform.ciliumPoolRange" as a first..last range
193+
for kube-ovn excludeIps.
194+
195+
The block has to sit inside the pod CIDR rather than next to it. Envoy reaches
196+
Gateway API backends from the node's Ingress IP, and kube-ovn masquerades
197+
host-originated traffic only when its source lies in one of its subnets.
198+
Outside them the backend's reply is routed to the backend's own node and lost.
199+
The end of the range is the part kube-ovn reaches last, since it hands out
200+
free addresses in ascending order and reuses released ones only once the rest
201+
are gone. A /21 cut into /29s serves 256 nodes, the most a /16 cluster CIDR
202+
sliced into /24s ever allowed.
203+
*/}}
204+
{{- define "cozystack.platform.ciliumPoolBounds" -}}
205+
{{- $cidr := toString . -}}
206+
{{- if not (regexMatch "^[0-9]{1,3}(\\.[0-9]{1,3}){3}/[0-9]{1,2}$" $cidr) -}}
207+
{{- fail (printf "networking.podCIDR %q is not an IPv4 CIDR" $cidr) -}}
208+
{{- end -}}
209+
{{- $parts := splitList "/" $cidr -}}
210+
{{- $prefix := atoi (last $parts) -}}
211+
{{- if gt $prefix 20 -}}
212+
{{- fail (printf "networking.podCIDR %s is too small. The kube-ovn variants reserve its last /21 for Cilium, so it needs a prefix of /20 or shorter" $cidr) -}}
213+
{{- end -}}
214+
{{- $address := 0 -}}
215+
{{- range splitList "." (first $parts) -}}
216+
{{- $address = add (mul $address 256) (atoi .) -}}
217+
{{- end -}}
218+
{{- $size := 1 -}}
219+
{{- range until (sub 32 $prefix | int) -}}
220+
{{- $size = mul $size 2 -}}
221+
{{- end -}}
222+
{{- $first := add (sub $address (mod $address $size)) (sub $size 2048) -}}
223+
{{- printf "%s %s" (include "cozystack.platform.ipv4" $first) (include "cozystack.platform.ipv4" (add $first 2047)) -}}
224+
{{- end }}
225+
226+
{{- define "cozystack.platform.ipv4" -}}
227+
{{- printf "%d.%d.%d.%d" (div . 16777216) (mod (div . 65536) 256) (mod (div . 256) 256) (mod . 256) -}}
228+
{{- end }}
229+
230+
{{- define "cozystack.platform.ciliumPool" -}}
231+
{{- printf "%s/21" (first (splitList " " (include "cozystack.platform.ciliumPoolBounds" .))) -}}
232+
{{- end }}
233+
234+
{{- define "cozystack.platform.ciliumPoolRange" -}}
235+
{{- join ".." (splitList " " (include "cozystack.platform.ciliumPoolBounds" .)) -}}
236+
{{- end }}

‎packages/core/platform/templates/bundles/system.yaml‎

Lines changed: 23 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -22,17 +22,25 @@
2222
{{- if dig "encryption" "enabled" false .Values.networking -}}
2323
{{- $_ := set $kubeovnDict "func" (dict "ENABLE_OVN_IPSEC" true) -}}
2424
{{- end -}}
25-
{{- $kubeovnValues := dict "kube-ovn" $kubeovnDict -}}
26-
{{- $_ := set $networkingComponents "kubeovn" (dict "values" $kubeovnValues) -}}
2725
{{- /* For Talos (isp-full, isp-slim): use KubePrism endpoint and disable cgroup autoMount */ -}}
2826
{{- /* gateway.http2 is threaded explicitly in BOTH states: inline HelmRelease
2927
values override the system/cilium valuesFiles default (enableAlpn: true),
3028
so an operator's http2: false actually reaches the dataplane. */ -}}
31-
{{- $ciliumValues := dict "cilium" (dict
29+
{{- $cilium := dict
3230
"k8sServiceHost" "localhost"
3331
"k8sServicePort" "7445"
3432
"cgroup" (dict "autoMount" (dict "enabled" false))
35-
"gatewayAPI" (dict "enableAlpn" .Values.gateway.http2)) -}}
33+
"gatewayAPI" (dict "enableAlpn" .Values.gateway.http2) -}}
34+
{{- /* Cilium's own addresses come from a block of the pod CIDR that kube-ovn
35+
does not hand out, see cozystack.platform.ciliumPool. The slim variants
36+
run no kube-ovn and keep Cilium allocating pods from Node.spec.podCIDR. */ -}}
37+
{{- if not (include "cozystack.platform.slim" $) -}}
38+
{{- $_ := set $kubeovnDict "networking" (dict "EXCLUDE_IPS" (include "cozystack.platform.ciliumPoolRange" .Values.networking.podCIDR)) -}}
39+
{{- $_ := set $cilium "ipam" (dict "operator" (dict "clusterPoolIPv4PodCIDRList" (list (include "cozystack.platform.ciliumPool" .Values.networking.podCIDR)))) -}}
40+
{{- end -}}
41+
{{- $kubeovnValues := dict "kube-ovn" $kubeovnDict -}}
42+
{{- $_ := set $networkingComponents "kubeovn" (dict "values" $kubeovnValues) -}}
43+
{{- $ciliumValues := dict "cilium" $cilium -}}
3644
{{- $_ := set $networkingComponents "cilium" (dict "values" $ciliumValues) -}}
3745
{{- end -}}
3846
{{include "cozystack.platform.package.networking" (list $ $networkingComponents "kubeovn-cilium" "cilium") }}
@@ -112,15 +120,22 @@
112120
{{- if dig "encryption" "enabled" false .Values.networking -}}
113121
{{- $_ := set $kubeovnDict "func" (dict "ENABLE_OVN_IPSEC" true) -}}
114122
{{- end -}}
115-
{{- $kubeovnValues := dict "kube-ovn" $kubeovnDict -}}
116-
{{- $_ := set $networkingComponents "kubeovn" (dict "values" $kubeovnValues) -}}
117123
{{- /* Cilium configuration - for generic k8s, always enable cgroup autoMount */ -}}
118124
{{- /* gateway.http2 threading: same rationale as the isp-full branch above. */ -}}
119-
{{- $ciliumValues := dict "cilium" (dict
125+
{{- $cilium := dict
120126
"k8sServiceHost" $apiHost
121127
"k8sServicePort" $apiPort
122128
"cgroup" (dict "autoMount" (dict "enabled" true))
123-
"gatewayAPI" (dict "enableAlpn" .Values.gateway.http2)) -}}
129+
"gatewayAPI" (dict "enableAlpn" .Values.gateway.http2) -}}
130+
{{- /* Cilium's address block, same as the isp-full branch above, from the pod
131+
CIDR the cluster actually runs with. */ -}}
132+
{{- if not (include "cozystack.platform.slim" $) -}}
133+
{{- $_ := set $kubeovnDict "networking" (dict "EXCLUDE_IPS" (include "cozystack.platform.ciliumPoolRange" $podCIDR)) -}}
134+
{{- $_ := set $cilium "ipam" (dict "operator" (dict "clusterPoolIPv4PodCIDRList" (list (include "cozystack.platform.ciliumPool" $podCIDR)))) -}}
135+
{{- end -}}
136+
{{- $kubeovnValues := dict "kube-ovn" $kubeovnDict -}}
137+
{{- $_ := set $networkingComponents "kubeovn" (dict "values" $kubeovnValues) -}}
138+
{{- $ciliumValues := dict "cilium" $cilium -}}
124139
{{- $_ := set $networkingComponents "cilium" (dict "values" $ciliumValues) -}}
125140
{{- end -}}
126141
{{- /* Use kubeovn-cilium-generic / cilium-generic (no values-talos.yaml) */ -}}
Lines changed: 133 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,133 @@
1+
suite: Cilium address block on the kube-ovn variants
2+
templates:
3+
- templates/bundles/system.yaml
4+
release:
5+
name: cozystack
6+
namespace: cozy-system
7+
tests:
8+
# The block has to be the same in both places: Cilium allocates its router and
9+
# Ingress IPs from it, and kube-ovn must never hand one of those to a pod.
10+
- it: hands the last /21 of the default pod CIDR to Cilium and excludes it in kube-ovn
11+
set:
12+
bundles:
13+
system:
14+
enabled: true
15+
variant: isp-full
16+
documentSelector:
17+
path: metadata.name
18+
value: cozystack.networking
19+
asserts:
20+
- equal:
21+
path: spec.variant
22+
value: kubeovn-cilium
23+
- equal:
24+
path: spec.components.cilium.values.cilium.ipam.operator.clusterPoolIPv4PodCIDRList
25+
value:
26+
- 10.244.248.0/21
27+
- equal:
28+
path: spec.components.kubeovn.values.kube-ovn.networking.EXCLUDE_IPS
29+
value: 10.244.248.0..10.244.255.255
30+
31+
- it: follows a custom pod CIDR on the generic variant
32+
set:
33+
bundles:
34+
system:
35+
enabled: true
36+
variant: isp-full-generic
37+
networking:
38+
podCIDR: 10.42.0.0/16
39+
documentSelector:
40+
path: metadata.name
41+
value: cozystack.networking
42+
asserts:
43+
- equal:
44+
path: spec.variant
45+
value: kubeovn-cilium-generic
46+
- equal:
47+
path: spec.components.cilium.values.cilium.ipam.operator.clusterPoolIPv4PodCIDRList
48+
value:
49+
- 10.42.248.0/21
50+
- equal:
51+
path: spec.components.kubeovn.values.kube-ovn.networking.EXCLUDE_IPS
52+
value: 10.42.248.0..10.42.255.255
53+
54+
- it: takes the end of a wider pod CIDR and ignores host bits in the network address
55+
set:
56+
bundles:
57+
system:
58+
enabled: true
59+
variant: isp-full
60+
networking:
61+
podCIDR: 10.100.7.1/12
62+
documentSelector:
63+
path: metadata.name
64+
value: cozystack.networking
65+
asserts:
66+
- equal:
67+
path: spec.components.cilium.values.cilium.ipam.operator.clusterPoolIPv4PodCIDRList
68+
value:
69+
- 10.111.248.0/21
70+
- equal:
71+
path: spec.components.kubeovn.values.kube-ovn.networking.EXCLUDE_IPS
72+
value: 10.111.248.0..10.111.255.255
73+
74+
- it: keeps the kube-ovn ipv4 overrides next to the exclusion
75+
set:
76+
bundles:
77+
system:
78+
enabled: true
79+
variant: isp-full
80+
documentSelector:
81+
path: metadata.name
82+
value: cozystack.networking
83+
asserts:
84+
- equal:
85+
path: spec.components.kubeovn.values.kube-ovn.ipv4.POD_CIDR
86+
value: 10.244.0.0/16
87+
- equal:
88+
path: spec.components.cilium.values.cilium.k8sServiceHost
89+
value: localhost
90+
91+
- it: refuses a pod CIDR too small to give up a /21
92+
set:
93+
bundles:
94+
system:
95+
enabled: true
96+
variant: isp-full
97+
networking:
98+
podCIDR: 10.244.0.0/21
99+
asserts:
100+
- failedTemplate:
101+
errorPattern: "networking.podCIDR 10.244.0.0/21 is too small"
102+
103+
- it: refuses a pod CIDR that is not an IPv4 CIDR
104+
set:
105+
bundles:
106+
system:
107+
enabled: true
108+
variant: isp-full
109+
networking:
110+
podCIDR: fd00::/64
111+
asserts:
112+
- failedTemplate:
113+
errorPattern: "is not an IPv4 CIDR"
114+
115+
# The slim variants run Cilium alone. It allocates pod addresses there from
116+
# Node.spec.podCIDR, so neither the block nor kube-ovn belongs in them.
117+
- it: leaves the slim variant's Cilium IPAM alone
118+
set:
119+
bundles:
120+
system:
121+
enabled: true
122+
variant: isp-slim
123+
documentSelector:
124+
path: metadata.name
125+
value: cozystack.networking
126+
asserts:
127+
- equal:
128+
path: spec.variant
129+
value: cilium
130+
- notExists:
131+
path: spec.components.cilium.values.cilium.ipam
132+
- notExists:
133+
path: spec.components.kubeovn

‎packages/core/platform/values.yaml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -84,6 +84,10 @@ bundles:
8484
# Network configuration
8585
networking:
8686
clusterDomain: "cozy.local"
87+
# On the kube-ovn variants the last /21 of podCIDR is reserved for Cilium's
88+
# own router and Gateway API Ingress addresses and excluded from kube-ovn
89+
# allocation, so podCIDR needs a prefix of /20 or shorter. Each node takes a
90+
# /29 of that block, which caps these variants at 256 nodes.
8791
podCIDR: "10.244.0.0/16"
8892
podGateway: "10.244.0.1"
8993
serviceCIDR: "10.96.0.0/16"

0 commit comments

Comments
 (0)