cilium-ipam-migration hook fails with "Argument list too long" when the CiliumNode list exceeds 128 KiB

Summary

The cilium-ipam-migration post-upgrade hook added by #4778 (backported to v1.6.5 in #4918) fails with /usr/bin/jq: Argument list too long on clusters whose CiliumNode list is larger than 128 KiB. It fails before any node is moved, so the migration never happens. The cilium HelmRelease then loops on UpgradeFailed, and every package that depends on networking stays DependencyNotReady: the platform upgrade stalls behind Cilium.

Where

packages/system/cilium/files/ipam-migration.sh, line 244 on main (unchanged since dc362fa):

jq -r --argjson cn "$ciliumnodes" '
  ([$cn.items[] | select(.spec.ingress.ipv4) | {key: .spec.ingress.ipv4, value: .metadata.name}] | from_entries) as $ingress
  | .[] | select($ingress[.ip]) | "\(.name) \(.ip) \($ingress[.ip])"' <<<"$ipobjects"

$ciliumnodes is the full kubectl get ciliumnodes -o json. Passed as a single argument, it is bounded by Linux MAX_ARG_STRLEN (128 KiB per argument), not by ARG_MAX.

Why the list gets that large

Cilium copies the Node labels onto each CiliumNode. On our nodes those labels weigh about 30 KB per node (Node Feature Discovery and GPU operator labels), so the list reaches 408 KB for 12 nodes. A 3-node lab cluster stays under the limit, which is why the hook passed there.

Observed on v1.6.0 → v1.6.5 (`isp-full`, 12 nodes)

2026-10-19T18:05:07Z subnet ovn-default already excludes the pool and the current Ingress IPs
/bin/bash: line 244: /usr/bin/jq: Argument list too long

The Job fails (backoffLimit: 0), Helm reports post-upgrade hooks failed, and with RetryOnFailure the hook reruns and fails every 30 s.
The agents already run with ipam.mode: cluster-pool, but every CiliumNode keeps its old /24 (allocator not configured for the requested CIDR). Networking keeps working on the old addresses, so nothing breaks, but nothing moves either.
About 130 HelmReleases sat in DependencyNotReady until the cilium release succeeded.
Setting spec.upgrade.disableHooks: true on the HelmRelease does not help on its own: cozystack-operator rewrites spec.upgrade within a second.

Workaround we used

Run the script by hand, unchanged except for that call, which reads the list from a file:
```
cnfile=$(mktemp); printf '%s' "$ciliumnodes" > "$cnfile"
jq -r --slurpfile cnw "$cnfile" '($cnw[0]) as $cn | ...' <<<"$ipobjects"
```
It moved the 12 nodes one by one in about 3 minutes.
Scale cozystack-operator to 0, set disableHooks: true on the cilium HelmRelease, wait for UpgradeSucceeded, then scale the operator back up. The operator removes disableHooks again, and no new upgrade is triggered because chart and values are unchanged.

Suggested fix

Never pass the CiliumNode list as an argument. Use --slurpfile or --rawfile from a temp file (the hook already mounts a writable /tmp), or fold both inputs into a single jq call fed through stdin. A test with a CiliumNode list over 128 KiB would catch a regression. The other jq calls in the script take their input through herestrings, which have no size limit; only this --argjson is affected.

As long as this is not fixed, any later upgrade that reruns the hook on a large cluster will fail the same way.

Independently hit this on a 30-node v1.6.3 → v1.6.5 (isp-full) cluster — same failure at the same jq --argjson call, but with a different amplifier that may be worth documenting alongside NFD/GPU labels: Talos. Each of our Talos v1.13.9 nodes carries ~405 labels (extension modules, CPU timer / TSC, kubevirt node-labeller, etc., ~21.5 KB of labels per Node), and Cilium copies them all onto the CiliumNode. The full kubectl get ciliumnodes -o json (managedFields stripped) came to 879,680 bytes for 30 nodes — workers ~22 KB each, so the limit is hit around node 6, long before any reasonable cluster size.

Confirming the rest of the report: agents ran ipam.mode: cluster-pool with every CiliumNode still holding its old /24, networking stayed healthy on the old addresses the whole time, and ~84 HelmReleases sat in DependencyNotReady overnight while the hook job retried every ~30 s. Also confirming spec.upgrade.disableHooks: true is rewritten by cozystack-operator within a second — patching it directly is not viable.

Full workaround that worked for us (cluster is now fully upgraded, 106/106 HelmReleases Ready):

Ran the migration script by hand with only that call changed (mktemp + printf > $cnfile + jq --slurpfile). 30/30 nodes moved into the new /21 pool in ~7 minutes (~13 s per node), agents restarted one at a time, zero failures. Note: manually running it while the hook job keeps retrying is safe — the hook dies at the jq line before touching anything, so there is no concurrent-writer risk.
To get the cilium release past its broken hook: kubectl -n cozy-system scale deploy cozystack-operator --replicas=0 → patch spec.upgrade.disableHooks: true on the cilium HelmRelease → annotate reconcile.fluxcd.io/requestedAt to force a reconcile. With the operator paused, the patch sticks; the release upgrades without hooks and went UpgradeSucceeded in ~15 s. Scale the operator back to 1 afterwards. We left disableHooks: true in place — harmless as long as the chart/values don't change (no further upgrade is triggered), and may also be the safer state for clusters that already migrated manually.
After that, the cascade cleared in ~40 minutes. The only slow tail was unrelated to this bug: the multus DaemonSet re-roll pulled a 195 MB image per node through our registry proxy (~2.5 min/node), which exceeded the 10 m helm upgrade timeout once and produced a transient UpgradeFailed until the DS rollout caught up — self-healing via retries, just noisy on large clusters.

One more data point for the fix: of all jq invocations in the script only this --argjson is affected (the rest read from herestrings), so a --slurpfile/--rawfile (or stdin) variant plus a unit test feeding a >128 KiB list should close it completely. Happy to share our patched script if useful.

Adding a data point from a small cluster, because it shows the threshold is crossed even
without label inflation.

Environment: Cozystack v1.6.5, Talos 1.13, Kubernetes 1.33, 6 nodes (3 control-plane +
3 workers), plain nodes — no NFD/GPU labels involved.

$ kubectl get ciliumnodes -o json | wc -c
168118                     # 6 nodes, ~28 KiB per CiliumNode
$ jq -r --argjson cn "$(kubectl get ciliumnodes -o json)" '1' <<<'[]'
bash: /usr/bin/jq: Argument list too long

So ~28 KiB per CiliumNode is already enough on its own — the copied labels are an amplifier,
not the trigger.

How we hit it: we changed an unrelated Cilium value (gatewayAPI.externalTrafficPolicy)
through Package/cozystack.networking. Any values change re-runs the upgrade, so the hook
fires and fails.

State of the migration: already applied and correct on our side — every CiliumNode has
a /29 from the pool 10.244.248.0/21, kube-ovn already excludes the pool (the
subnet ovn-default already excludes the pool and the current Ingress IPs line prints before
the failure). Only the advisory step — listing pods that still hold pre-migration Ingress IPs
— dies, which is why the release looks broken while nothing is actually left to do.

Impact with helm-controller: upgradeFailures went 3 → 6, one attempt every ~5 min, and
each attempt re-applies the chart, so with rollOutCiliumPods: true the Cilium DaemonSet rolled
again on a live cluster every 5 minutes. We parked it with:

kubectl -n cozy-cilium patch helmrelease cilium --type=merge -p '{"spec":{"suspend":true}}'

After suspending: nodes Ready, Cilium agents 6/6, operator 1/1, no traffic impact (all
manifests were already applied). Not a fix, but a safe parking position — and clearly better
than the repeated CNI rollouts.

For whichever fix gets picked: the projection below is 217 bytes for the same 6 nodes and
works, so the advisory step can stay --argjson-based:

jq -c '[.items[] | select(.spec.ingress.ipv4) | {name: .metadata.name, ip: .spec.ingress.ipv4}]'

Another data point from a v1.6.5 cluster (9 nodes, CiliumNode list ~232 KB, labels ~28 KB per node).

Here the migration itself had already completed on the 1.6.5 upgrade (ipam cluster-pool, every CiliumNode
holds a /29 from the pool, ovn-default already excludes the pool). The hook still runs on every later
helm upgrade of cilium (it is rendered whenever ipam.mode == cluster-pool), so any change that touches the
cilium values afterwards — in our case publishing.apiServerEndpoint, which lands in _cluster of every
package — loops the HelmRelease on UpgradeFailed and leaves ~80 dependent HelmReleases DependencyNotReady.
It started failing once the cluster grew from 6 to 9 nodes.

Workaround that worked for us (safe only because the migration is done): scale cozystack-operator to 0
(it resets the HelmRelease spec immediately), set spec.upgrade.disableHooks: true on cozy-cilium/cilium,
reconcile, wait for UpgradeSucceeded, remove the flag, scale the operator back.

Two suggestions besides the --argjson fix (--slurpfile / reading $ciliumnodes from stdin or a file):

make the hook a no-op once every CiliumNode already has an in-pool spec.ipam.podCIDRs (check that first,
before the expensive checks), or
expose a value to disable the hook after the migration.

fix(kubernetes): move the OIDC users out of the bootstrap Job's script (

fix(cilium): pass the CiliumNode list to jq on stdin in the IPAM hook (…

fix(kubernetes): move the OIDC users out of the bootstrap Job's script (

fix(cilium): pass the CiliumNode list to jq on stdin in the IPAM hook (…

fix(kubernetes): move the OIDC users out of the bootstrap Job's script (

Navigation Menu

Repository navigation

cilium-ipam-migration hook fails with "Argument list too long" when the CiliumNode list exceeds 128 KiB #4968

Summary

Where

Why the list gets that large

Observed on v1.6.0 → v1.6.5 (`isp-full`, 12 nodes)

Workaround we used

Suggested fix

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

cilium-ipam-migration hook fails with "Argument list too long" when the CiliumNode list exceeds 128 KiB #4968

Description

Summary

Where

Why the list gets that large

Observed on v1.6.0 → v1.6.5 (isp-full, 12 nodes)

Workaround we used

Suggested fix

Activity

zhangxiaopeng-zgci commented on Oct 22, 2026

hjkbnm911 commented on Oct 25, 2026

alexey8312 commented on Oct 31, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Observed on v1.6.0 → v1.6.5 (`isp-full`, 12 nodes)