Skip to content

Commit e591ba6

Browse files
fix(cilium): pass the CiliumNode list to jq on stdin in the IPAM hook (#5212)
## What this PR does Fixes #4968. The cilium IPAM migration hook passed the whole `kubectl get ciliumnodes -o json` to `jq` as one argument, and Linux allows 128 KiB for a single argument. Cilium copies Node labels onto every CiliumNode (22 to 30 KiB per node with Talos, NFD or GPU labels), so the list is over the limit within six nodes. `jq` can't start (`Argument list too long`), the hook Job fails before it moves any node, `cilium` HelmRelease loops on `UpgradeFailed` and everything that depends on networking stays in `DependencyNotReady`. The hook is rendered on every upgrade of cilium in `cluster-pool` mode, so a cluster that already finished the migration hits the same on its next values change. The failing step only logs the pods that still hold an old Ingress IP, but it runs before the node loop, so on a cluster that is not migrated yet it blocks the migration itself. Now the list goes to `jq` on stdin together with the IP objects, same as the other `jq` calls in the script get their input. The new test in `hack/cilium-ipam-migration.bats` makes six CiliumNodes over the limit (one 24 KiB annotation each instead of labels, and it checks the list is over 128 KiB even printed compact). It fails on the old script with status 126 and passes now. It also checks that each pod is named with the node whose Ingress IP it holds and that a pod on an unrelated address is not named, because a join that always names the first node, or reports every IP object, passed the old tests. I did not take two suggestions from the issue. Skipping the hook when every CiliumNode is in the pool changes what one existing test pins (the hook adds the pool to `excludeIps` in that case) and doesn't help clusters that are not migrated yet. A value to disable the hook leaves the default broken. `cozystack-operator` rewriting `disableHooks` and helm-controller rolling the DaemonSet on every retry are other components. The bug is in release-1.6 and release-1.7 as well. ### Downstream repositories I walked the trigger map against the diff (one hook script and its bats test): no package, values, schema, default or output name changed. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(cilium): the IPAM migration hook no longer fails with "Argument list too long" on clusters with more than a few nodes, which blocked the cilium upgrade and every package that depends on networking. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Bug Fixes** * Improved IP address migration handling for environments with large node lists, avoiding failures caused by operating system argument-size limits. * Migration continues to identify affected pods by node and distinguish unrelated pods. * **Tests** * Added coverage for migrations involving large node lists, multiple IP objects, and unrelated pods. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Backported-from: ab13f9f
2 parents 07f3716 + b28a2e2 commit e591ba6

2 files changed

Lines changed: 33 additions & 4 deletions

File tree

‎hack/cilium-ipam-migration.bats‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -488,6 +488,33 @@ pod_cidrs_of() {
488488
rm -rf "$STATE"
489489
}
490490

491+
@test "a CiliumNode list larger than one argument may hold still moves every node and names the pod on an old Ingress IP" {
492+
init_state
493+
add_ip web-0.tenant-c ovn-default 10.244.1.135
494+
add_ip web-1.tenant-c ovn-default 10.244.4.135
495+
add_ip db-0.tenant-c ovn-default 10.244.9.9
496+
# Cilium copies every Node label onto the CiliumNode, which takes six real
497+
# ones past the 128 KiB Linux allows a single argument. One annotation stands
498+
# in for the labels and keeps the list over that limit even printed compact.
499+
for i in 1 2 3 4 5 6; do
500+
add_node "n$i" "10.244.$i.0/24" "10.244.$i.223" "10.244.$i.135"
501+
jq '.metadata.annotations.padding = ("x" * 24576)' "$STATE/ciliumnodes/n$i.json" > "$STATE/node.tmp"
502+
mv "$STATE/node.tmp" "$STATE/ciliumnodes/n$i.json"
503+
done
504+
[ "$(kubectl get ciliumnodes -o json | jq -c . | wc -c)" -gt 131072 ]
505+
"$SCRIPT" > "$STATE/out" 2>&1
506+
# Each pod is named with the node whose Ingress IP it holds, and a pod on an
507+
# address that is no node's Ingress IP is not named at all.
508+
grep -q "web-0.tenant-c holds 10.244.1.135, the Ingress IP of n1 before the move" "$STATE/out"
509+
grep -q "web-1.tenant-c holds 10.244.4.135, the Ingress IP of n4 before the move" "$STATE/out"
510+
if grep -q "db-0.tenant-c" "$STATE/out"; then
511+
cat "$STATE/out"
512+
false
513+
fi
514+
[ "$(pod_cidrs_of n6)" = '["10.244.248.40/29"]' ]
515+
rm -rf "$STATE"
516+
}
517+
491518
@test "a cluster without kube-ovn skips the exclusion and still moves its nodes" {
492519
init_state
493520
rm "$STATE/subnet.json"

‎packages/system/cilium/files/ipam-migration.sh‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -240,10 +240,12 @@ exclude_from_kube_ovn() {
240240
fi
241241
# A pod that holds a node's current Ingress IP keeps the address after the
242242
# move, and that node keeps it in its ipcache as reserved:ingress, so the pod
243-
# stays unreachable from there until it is recreated.
244-
jq -r --argjson cn "$ciliumnodes" '
245-
([$cn.items[] | select(.spec.ingress.ipv4) | {key: .spec.ingress.ipv4, value: .metadata.name}] | from_entries) as $ingress
246-
| .[] | select($ingress[.ip]) | "\(.name) \(.ip) \($ingress[.ip])"' <<<"$ipobjects" \
243+
# stays unreachable from there until it is recreated. The CiliumNode list goes
244+
# in on stdin, because the Node labels Cilium copies onto every CiliumNode take
245+
# it past the 128 KiB a single argument may hold within a handful of nodes.
246+
jq -rn 'input as $cn
247+
| ([$cn.items[] | select(.spec.ingress.ipv4) | {key: .spec.ingress.ipv4, value: .metadata.name}] | from_entries) as $ingress
248+
| input[] | select($ingress[.ip]) | "\(.name) \(.ip) \($ingress[.ip])"' <<<"$ciliumnodes"$'\n'"$ipobjects" \
247249
| while read -r ipname ip node; do
248250
if ! in_any "$POOL" "$ip"; then
249251
log "$ipname holds $ip, the Ingress IP of $node before the move. Recreate it once the run is done, $node keeps that address as reserved:ingress"

0 commit comments

Comments
 (0)