Repository navigation
Gateway API + kubernetes (host-scope) IPAM: per-node Ingress IP can be re-allocated to a pod after an agent restart (sandbox rejected "IP already in use") #46799
Description
Summary
With ipam: kubernetes (host-scope IPAM) and Gateway API enabled (enable-envoy-config=true), the per-node Cilium Ingress IP (CiliumNode.spec.ingress.ipv4, held by the reserved:ingress endpoint) is carved from the node Pod CIDR, but its reservation is not robust across a cilium-agent restart. After a restart the host-scope allocation bitmap is rebuilt empty and the Ingress IP is re-reserved via a path decoupled from local-endpoint restore; if the CNI server serves a pod ADD in that window, the pod can be handed the Ingress IP. The agent then rejects the pod sandbox with [PUT /endpoint/{id}][400] "IP <ingress-ip> is already in use", and the pod is stuck in ContainerCreating.
I hit this in Cozystack (a CNCF platform) E2E, where it intermittently wedges pods during cluster bring-up — reporting it upstream as a real downstream consumer of this configuration.
Cilium version
v1.19.5 (the mechanism is present throughout the v1.19 line; I could not find a released version that fixes it).
Configuration
- IPAM:
kubernetes(host-scope, per-nodeNode.spec.podCIDR) - Gateway API: enabled →
enable-envoy-config: "true" - Legacy ingress controller: disabled
gatewayAPI.hostNetwork.enabled: false(per-tenant LoadBalancer-type Services)
Mechanism (source references, tag v1.19.5)
- The per-node Ingress IP is allocated by the agent, gated solely on
EnableEnvoyConfig:daemon/infraendpoints/infra_ip_allocation.go→allocateIngressIPs()→AllocateNextFamilyWithoutSyncUpstream(IPv4, "ingress", PoolDefault()); the matching endpoint is created bydaemon/infraendpoints/ingress_endpoint.go→registerIngressEndpoint(same gate). The IP comes fromipam.PoolDefault(), which in host-scope mode is the node Pod CIDR. - In host-scope IPAM the allocator is a single in-memory bitmap with no persistence:
pkg/ipam/hostscope.go(newHostScopeAllocator→ipallocator.NewCIDRRange;RestoreFinished()is a no-op). On every agent start the bitmap is empty and must be re-populated. - After a restart the bitmap is re-populated by two decoupled paths: restored local endpoints (
daemon/cmd/endpoint_restore.go→allocateIPsLocked→AllocateIPWithoutSyncUpstream(..., "<ep> [restored]")) and the Ingress IP (allocateIngressIPs, reading the value from the restoredLocalNode/CiliumNode.spec.ingress.ipv4). Because these passes are not ordered against the CNI server beginning to serve pod ADDs, a pod ADD can be handed the Ingress IP before it is re-reserved. The Ingress IP is then held simultaneously by thereserved:ingressendpoint and a pod → the sandbox is rejected.
Impact
A pod can be left permanently wedged in ContainerCreating until it is rescheduled onto a different IP — and on a small per-node Pod CIDR, rescheduling can re-hit the same IP. Anything that restarts the agent (rollouts, node pressure, an operator restart) can re-enter the race. It surfaces as a FailedCreatePodSandBox event whose message IP equals CiliumNode.spec.ingress.ipv4 for that node.
Suggested fixes (any one)
- Reserve
spec.ingress.ipv4in the host-scope allocation bitmap before the CNI server starts serving pod allocations (treat it like a restored endpoint IP). - During endpoint restore, reconcile the restored
reserved:ingressendpoint's IP into the bitmap so the Ingress IP and pod IPs share one consistent reservation pass. - Source the per-node Ingress IP from a dedicated range outside the Pod CIDR so it can never collide with pod IPAM.
How to reproduce (sketch)
- Cilium v1.19.x,
ipam: kubernetes, Gateway API enabled, a small per-node Pod CIDR. - Create Gateways so the per-node
reserved:ingressIP is allocated; noteCiliumNode.spec.ingress.ipv4for a node. - Restart that node's
cilium-agentwhile pods are being created on the node (churn). - Observe a pod stuck in
ContainerCreatingwithFailedCreatePodSandBox … "IP <ingress-ip> is already in use"where<ingress-ip>equals that node'sCiliumNode.spec.ingress.ipv4.
Activity
fix(e2e): skip cilium-agent restart for the reserved:ingress IP varia…
squeed commentedon Jul 1, 2026
I'm surprised this hasn't happened before. Do you have any indication this is a regression?
lexfrei commentedon Jul 1, 2026
No hard evidence of regression on my end — I don't have data from pre-1.19 releases to compare against. What I can confirm: it reproduced with the identical signature on both 1.19.3 and 1.19.5 in our CI, so it isn't something introduced by our own recent Cilium bump. It's not high-frequency in isolation, and part of why it took a while to pin down is that our CI also has a second, unrelated, older orphaned-endpoint-leak flake (#38313) that produces the exact same "IP already in use" surface error — the two got conflated until I isolated this one by matching the disputed IP against
CiliumNode.spec.ingress.ipv4.youngnick commentedon Jul 17, 2026
This does sound like an issue, but I'm interested in how you opened this issue, @lexfrei. All the available issue templates should automatically set labels for you, which will help issues get seen by the correct folks more quickly. What path did you use to open this issue? I'd like us to be able to make sure that the labels (like
kind/bugetc) get set no matter the creation path.lexfrei commentedon Jul 17, 2026
Via the GitHub API (
ghCLI), not the web UI. I do follow the issue template when opening via the API — I have the model fill it out by its structure — but the API path doesn't apply the labels the template would auto-set in the web UI. I noticed that specific gap at creation time and judged it not worth switching to the browser over; @squeed setkind/bugandkind/community-reportby hand afterwards. I'll keep it in mind for this repo going forward.Disclosure while I'm here: this issue was drafted with an LLM under my direction (AIL:3) — I define the problem and the mechanism, the model drafts, and I read and edit every word before it goes out, same as on my PRs. Per the AI policy I should have declared that in the body; doing it now.
On closing the gap itself: as an external contributor I can't set labels directly either —
--labelis silently dropped without triage rights, so the template is the only path a drive-by contributor has to labels at all, and the API bypasses just that part. So closing this properly would need something server-side — e.g. a bot that labels anything opened without labels. For what it's worth, on the Cozystack side we currently use Dosu for this but aren't fully happy with it, and we've been kicking around moving to a nightly CronJob that sweeps issues and fills in whatever labels are missing — no strong opinion to offer yet, just sharing where a fellow CNCF project is on the same problem.youngnick commentedon Jul 20, 2026
Thanks for that @lexfrei, I've been seeing it a bit more, especially with agent-opened issues, and wanted to understand where the issue was coming from.
jrajahalme commentedon Jul 20, 2026
I was able to reproduce a pod getting stuck with "IP ... is already in use" error (with Codex running the repro) with these conditions:
Some observations:
--max-podswould prevent this from happening in the first placeI have issued #47360 to make Ingress IP restoration more robust, but that is mostly orthogonal with this issue. Should look into if we could make Endpoint PUT request to fail out earlier if the expiration UUID is unknown); this would improve diagnostics but not make the pod sandbox creation succeed in this case.
test(e2e): make kuberture split-horizon probes survive Cilium ingress…
test(e2e): make kuberture split-horizon probes survive Cilium ingress…
lexfrei commentedon Aug 11, 2026
Correcting my own report: the downstream motivation I gave for this issue is wrong. The source analysis is not, and I want to be precise about which half I am withdrawing.
What I wrote was "I hit this in Cozystack E2E, where it intermittently wedges pods during cluster bring-up". That is not what I hit.
In that platform cilium is not the IPAM. It runs chained behind kube-ovn (
cni.chainingMode: generic-veth,chainingTarget: kube-ovn), so pod addresses are allocated by kube-ovn and cilium only registers an endpoint for whatever address the chain hands it. kube-ovn allocates flat from its own subnet across all nodes and has no notion ofNode.spec.podCIDR. Those two ranges were configured over the same prefix, so the address cilium had reserved for its per-node Ingress endpoint was, from kube-ovn's side, an ordinary free address in the pool. Sooner or later it went to a pod, and the agent refused the sandbox.No agent restart is involved and no window is needed. The evidence names both sides:
cilium-dbg endpoint liston the node showsreserved:ingressholding the address, equal to that node'sCiliumNode.spec.ingress.ipv4, andovn-controlleron the same node logs a claim on it for the pod that never got a sandbox. It is a configuration overlap between two allocators, and it is fixed downstream by making the ranges disjoint.What stands is everything in the body that comes from reading the code, and none of it depends on my deployment:
allocateIngressIPs()is gated only onEnableEnvoyConfigand takes the address fromipam.PoolDefault(), which in host-scope mode is the node Pod CIDR.RestoreFinished()is a no-op, so it starts empty on every agent start.For a cluster where cilium does run its own IPAM with Gateway API enabled, that window is still readable from the source, and the three fixes I suggested still apply to it. I am not asking for anything to be done with the issue either way; I just do not want it carried by a reproduction that turned out to be a different bug.
What I can no longer offer is that reproduction. So the issue now rests on source reading rather than on an observed failure, which is weaker evidence than what I originally presented, and worth weighing as such.
github-actions commentedon Oct 10, 2026
This issue has been automatically marked as stale because it has not
had recent activity. It will be closed if no further activity occurs.