1.20.0: BPF verifier rejects ipv6_hdrlen_with_fraginfo() as a global function on kernel 5.10 — datapath never initialises, CNI cannot create endpoints #47672

Is there an existing issue for this?

I searched open and closed issues for ipv6_hdrlen_with_fraginfo, the verifier message, and 1.20 datapath/WireGuard load failures, and found nothing covering this.

What happened?

Upgrading 1.19.6 → 1.20.0 on a kernel 5.10 cluster, every cilium agent fails to initialise its datapath in a permanent retry loop. The eBPF collection is rejected by the verifier:

Verifier error: program tail_nodeport_nat_egress_ipv6: load program: invalid argument:
  Arg#1 type PTR in ipv6_hdrlen_with_fraginfo() is not supported yet. (3 line(s) omitted)

level=warn msg="Failed to initialize datapath, retrying later"
  module=agent.datapath.orchestrator
  error="failed to load wireguard programs: loading eBPF collection into the kernel:
         program tail_nodeport_nat_egress_ipv6: load program: invalid argument:
         Arg#1 type PTR in ipv6_hdrlen_with_fraginfo() is not supported yet."
  retryDelay=10s

The same rejection occurs for tail_nodeport_nat_ingress_ipv6 and tail_handle_snat_fwd_ipv6 — the IPv6 NAT and WireGuard paths.

Because datapath init never completes, the agent API never becomes ready and the CNI plugin times out, so no new pod can obtain networking:

Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network
  for sandbox "...": plugin type="cilium-cni" failed (add): unable to create endpoint:
  Cilium API client timeout exceeded

Already-running pods keep working, since their datapath was programmed before the upgrade — so this does not present as an immediate outage. But the cluster can no longer heal itself: every restart, eviction, cron job or failover from that point produces a pod stuck in ContainerCreating. All three nodes were affected simultaneously.

Rolling back to 1.19.6 restored full function immediately.

Likely cause

bpf/lib/ipv6.h changed between the two releases:

// v1.19.6
static __always_inline int ipv6_hdrlen_with_fraginfo(struct __ctx_buff *ctx,
                                                     __u8 *nexthdr,
                                                     fraginfo_t *fraginfo)

// v1.20.0
__noinline __weak
int ipv6_hdrlen_with_fraginfo(const struct __ctx_buff *ctx, __u8 *nexthdr, fraginfo_t *fraginfo)

__noinline __weak makes this a global BPF function, so the verifier type-checks its arguments where previously it was inlined and never inspected. Accepting a context pointer as a global-function argument requires either a verifier new enough to infer it, or the __arg_ctx BTF tag — and __arg_ctx does not appear anywhere in the tree.

bpf/lib/nat.h has three further __noinline __weak conversions in 1.20.0, so other paths may be affected on the same kernels.

This appears to be kernel-version dependent, which is presumably why it was not caught in CI. The documented base requirement for 1.20 is Linux kernel >= 5.10, and this cluster is on 5.10 — i.e. within the supported range.

Cilium Version

1.20.0 (v1.20.0-450c5314)   -- broken
1.19.6 (v1.19.6-9a898243)   -- works

Kernel Version

5.10.0-39-amd64 x86_64
Debian GNU/Linux 11 (bullseye)

Kubernetes Version

v1.36.1+k0s
containerd://2.3.1

Regression

Yes — 1.19.6 works on this exact cluster and configuration; 1.20.0 does not.

Relevant configuration

Relevant cilium-config values (dual-stack with WireGuard node encryption, which is what pulls in the rejected IPv6 NAT/WireGuard programs):

routing-mode              tunnel
tunnel-protocol           vxlan
enable-ipv4               true
enable-ipv6               true
enable-wireguard          true
encrypt-node              true
kube-proxy-replacement    true
enable-ipv6-masquerade    true
ipam                      kubernetes
identity-allocation-mode  crd

I would expect single-stack clusters, or clusters without WireGuard, not to load these programs and therefore not to hit this.

How can we reproduce the issue?

1. Kernel 5.10 nodes (Debian 11 here).
2. Cilium 1.19.6 with the configuration above — dual-stack, WireGuard with encrypt-node, tunnel/VXLAN, kube-proxy replacement.
3. Upgrade to 1.20.0.
4. Every agent logs the verifier rejection and loops on Failed to initialize datapath; cilium-dbg status reports cilium-health daemon unreachable, and newly scheduled pods stay in ContainerCreating with Cilium API client timeout exceeded.