Skip to content

Commit a2d0e4c

Browse files
fix(cilium): pass the CiliumNode list to jq on stdin in the IPAM hook
The hook fed the whole `kubectl get ciliumnodes -o json` to jq as one argument. Cilium copies every Node label onto its CiliumNode, 22 to 30 KiB per node on Talos or with NFD and GPU labels, so the list passes the 128 KiB Linux allows a single argument within six nodes. jq then fails to start with "Argument list too long", the hook Job fails before it moves any node, and the cilium HelmRelease loops on UpgradeFailed with the HelmReleases behind it stuck in DependencyNotReady. The hook renders on every upgrade of cilium in cluster-pool mode, so a cluster that already finished the migration hits the same failure on its next values change. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <[email protected]>
1 parent 4087218 commit a2d0e4c

2 files changed

Lines changed: 33 additions & 4 deletions

File tree

‎hack/cilium-ipam-migration.bats‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -488,6 +488,33 @@ pod_cidrs_of() {
488488
rm -rf "$STATE"
489489
}
490490

491+
@test "a CiliumNode list larger than one argument may hold still moves every node and names the pod on an old Ingress IP" {
492+
init_state
493+
add_ip web-0.tenant-c ovn-default 10.244.1.135
494+
add_ip web-1.tenant-c ovn-default 10.244.4.135
495+
add_ip db-0.tenant-c ovn-default 10.244.9.9
496+
# Cilium copies every Node label onto the CiliumNode, which takes six real
497+
# ones past the 128 KiB Linux allows a single argument. One annotation stands
498+
# in for the labels and keeps the list over that limit even printed compact.
499+
for i in 1 2 3 4 5 6; do
500+
add_node "n$i" "10.244.$i.0/24" "10.244.$i.223" "10.244.$i.135"
501+
jq '.metadata.annotations.padding = ("x" * 24576)' "$STATE/ciliumnodes/n$i.json" > "$STATE/node.tmp"
502+
mv "$STATE/node.tmp" "$STATE/ciliumnodes/n$i.json"
503+
done
504+
[ "$(kubectl get ciliumnodes -o json | jq -c . | wc -c)" -gt 131072 ]
505+
"$SCRIPT" > "$STATE/out" 2>&1
506+
# Each pod is named with the node whose Ingress IP it holds, and a pod on an
507+
# address that is no node's Ingress IP is not named at all.
508+
grep -q "web-0.tenant-c holds 10.244.1.135, the Ingress IP of n1 before the move" "$STATE/out"
509+
grep -q "web-1.tenant-c holds 10.244.4.135, the Ingress IP of n4 before the move" "$STATE/out"
510+
if grep -q "db-0.tenant-c" "$STATE/out"; then
511+
cat "$STATE/out"
512+
false
513+
fi
514+
[ "$(pod_cidrs_of n6)" = '["10.244.248.40/29"]' ]
515+
rm -rf "$STATE"
516+
}
517+
491518
@test "a cluster without kube-ovn skips the exclusion and still moves its nodes" {
492519
init_state
493520
rm "$STATE/subnet.json"

‎packages/system/cilium/files/ipam-migration.sh‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -240,10 +240,12 @@ exclude_from_kube_ovn() {
240240
fi
241241
# A pod that holds a node's current Ingress IP keeps the address after the
242242
# move, and that node keeps it in its ipcache as reserved:ingress, so the pod
243-
# stays unreachable from there until it is recreated.
244-
jq -r --argjson cn "$ciliumnodes" '
245-
([$cn.items[] | select(.spec.ingress.ipv4) | {key: .spec.ingress.ipv4, value: .metadata.name}] | from_entries) as $ingress
246-
| .[] | select($ingress[.ip]) | "\(.name) \(.ip) \($ingress[.ip])"' <<<"$ipobjects" \
243+
# stays unreachable from there until it is recreated. The CiliumNode list goes
244+
# in on stdin, because the Node labels Cilium copies onto every CiliumNode take
245+
# it past the 128 KiB a single argument may hold within a handful of nodes.
246+
jq -rn 'input as $cn
247+
| ([$cn.items[] | select(.spec.ingress.ipv4) | {key: .spec.ingress.ipv4, value: .metadata.name}] | from_entries) as $ingress
248+
| input[] | select($ingress[.ip]) | "\(.name) \(.ip) \($ingress[.ip])"' <<<"$ciliumnodes"$'\n'"$ipobjects" \
247249
| while read -r ipname ip node; do
248250
if ! in_any "$POOL" "$ip"; then
249251
log "$ipname holds $ip, the Ingress IP of $node before the move. Recreate it once the run is done, $node keeps that address as reserved:ingress"

0 commit comments

Comments
 (0)