Cilium can't create new endpoint : "IP ipv4:10.17.239.232 is already in use"

Is there an existing issue for this?

I have searched the existing issues

Version

v1.16.3

What happened?

In our production environment using calico chaining, cilium fail to create endpoint and pod remain stuck in ContainerCreating state for a long time with IP ipv4:10.17.239.232 is already in use error.

This happens when a lot of pods are creating at the same time such has lot of jobs, airflow dags, etc...

Here is our configuration :

  agent-not-ready-taint-key: node.cilium.io/agent-not-ready
  arping-refresh-period: 30s
  auto-direct-node-routes: "false"
  bpf-events-drop-enabled: "true"
  bpf-events-policy-verdict-enabled: "true"
  bpf-events-trace-enabled: "true"
  bpf-lb-acceleration: disabled
  bpf-lb-external-clusterip: "false"
  bpf-lb-map-max: "65536"
  bpf-lb-sock: "false"
  bpf-lb-sock-terminate-pod-connections: "false"
  bpf-map-dynamic-size-ratio: "0.0025"
  bpf-policy-map-max: "16384"
  bpf-root: /sys/fs/bpf
  cgroup-root: /run/cilium/cgroupv2
  cilium-endpoint-gc-interval: 5m0s
  cluster-id: "0"
  cluster-name: default
  cluster-pool-ipv4-cidr: 10.66.64.0/18
  cluster-pool-ipv4-mask-size: "24"
  clustermesh-enable-endpoint-sync: "false"
  clustermesh-enable-mcs-api: "false"
  cni-chaining-mode: generic-veth
  controller-group-metrics: write-cni-file sync-host-ips sync-lb-maps-with-k8s-services
  custom-cni-conf: "true"
  datapath-mode: veth
  debug: "false"
  debug-verbose: ""
  direct-routing-skip-unreachable: "false"
  dnsproxy-socket-linger-timeout: "10"
  egress-gateway-reconciliation-trigger-interval: 1s
  enable-auto-protect-node-port-range: "true"
  enable-bpf-clock-probe: "false"
  enable-endpoint-health-checking: "false"
  enable-external-ips: "false"
  enable-health-check-loadbalancer-ip: "false"
  enable-health-check-nodeport: "true"
  enable-health-checking: "true"
  enable-host-legacy-routing: "true"
  enable-host-port: "false"
  enable-hubble: "true"
  enable-identity-mark: "false"
  enable-ipv4: "true"
  enable-ipv4-big-tcp: "false"
  enable-ipv4-masquerade: "false"
  enable-ipv6: "false"
  enable-ipv6-big-tcp: "false"
  enable-ipv6-masquerade: "false"
  enable-k8s-networkpolicy: "false"
  enable-k8s-terminating-endpoint: "true"
  enable-l2-neigh-discovery: "true"
  enable-l7-proxy: "true"
  enable-local-node-route: "false"
  enable-local-redirect-policy: "false"
  enable-masquerade-to-route-source: "false"
  enable-metrics: "true"
  enable-node-port: "false"
  enable-node-selector-labels: "false"
  enable-policy: default
  enable-runtime-device-detection: "true"
  enable-sctp: "false"
  enable-svc-source-range-check: "true"
  enable-tcx: "true"
  enable-vtep: "false"
  enable-well-known-identities: "false"
  enable-xt-socket-fallback: "true"
  envoy-base-id: "0"
  envoy-keep-cap-netbindservice: "false"
  external-envoy-proxy: "false"
  hubble-disable-tls: "false"
  hubble-export-file-max-backups: "5"
  hubble-export-file-max-size-mb: "10"
  hubble-listen-address: :4244
  hubble-metrics: drop
  hubble-metrics-server: :9091
  hubble-socket-path: /var/run/cilium/hubble.sock
  hubble-tls-cert-file: /var/lib/cilium/tls/hubble/server.crt
  hubble-tls-client-ca-files: /var/lib/cilium/tls/hubble/client-ca.crt
  hubble-tls-key-file: /var/lib/cilium/tls/hubble/server.key
  identity-allocation-mode: crd
  identity-gc-interval: 15m0s
  identity-heartbeat-timeout: 30m0s
  install-no-conntrack-iptables-rules: "false"
  ipam: cluster-pool
  ipam-cilium-node-update-rate: 5s
  k8s-client-burst: "20"
  k8s-client-qps: "10"
  k8s-kubeconfig-path: /etc/kubernetes/admin.conf
  k8s-require-ipv4-pod-cidr: "false"
  k8s-require-ipv6-pod-cidr: "false"
  kube-proxy-replacement: "false"
  kube-proxy-replacement-healthz-bind-address: ""
  max-connected-clusters: "255"
  mesh-auth-enabled: "false"
  mesh-auth-gc-interval: 5m0s
  mesh-auth-queue-size: "1024"
  mesh-auth-rotated-identities-queue-size: "1024"
  monitor-aggregation: medium
  monitor-aggregation-flags: all
  monitor-aggregation-interval: 5s
  nat-map-stats-entries: "32"
  nat-map-stats-interval: 30s
  node-port-bind-protection: "true"
  nodeport-addresses: ""
  nodes-gc-interval: 5m0s
  operator-api-serve-addr: 127.0.0.1:9234
  operator-prometheus-serve-addr: :9963
  policy-cidr-match-mode: ""
  preallocate-bpf-maps: "false"
  prometheus-serve-addr: :9962
  proxy-connect-timeout: "2"
  proxy-idle-timeout-seconds: "60"
  proxy-max-connection-duration-seconds: "0"
  proxy-max-requests-per-connection: "0"
  proxy-prometheus-port: "9964"
  proxy-xff-num-trusted-hops-egress: "0"
  proxy-xff-num-trusted-hops-ingress: "0"
  read-cni-conf: /tmp/cni-configuration/cni-config
  remove-cilium-node-taints: "true"
  routing-mode: native
  service-no-backend-response: reject
  set-cilium-is-up-condition: "true"
  set-cilium-node-taints: "true"
  synchronize-k8s-nodes: "true"
  tofqdns-dns-reject-response-code: refused
  tofqdns-enable-dns-compression: "true"
  tofqdns-endpoint-max-ip-per-hostname: "50"
  tofqdns-idle-connection-grace-period: 0s
  tofqdns-max-deferred-connection-deletes: "10000"
  tofqdns-proxy-response-max-delay: 100ms
  unmanaged-pod-watcher-interval: "15"
  vtep-cidr: ""
  vtep-endpoint: ""
  vtep-mac: ""
  vtep-mask: ""
  write-cni-conf-when-ready: /host/etc/cni/net.d/05-cilium.conflist

What happens is cilium trying to create the endpoint with this log :

time="2025-03-19T10:24:07Z" level=info msg="Create endpoint request" addressing="&{10.17.239.232   fe80::b05e:daff:fe49:5ec4  }" containerID=dc3e99d542096b45280639688e782b244a24cecf013d3efa77c78c7c5c6d0fe2 containerInterface=eth0 datapathConfiguration="&{false true false true true 0xc01378ae74}" interface=cali5b7b5eaa75d k8sPodName=team-infrastructure/calico-vault-db-injector-canary-worker10-459 k8sUID=5afc1abd-4cc3-4830-b5d8-97fd4d1e1495 labels="[]" subsys=daemon sync-build=true

But cilium get an error :

time="2025-03-19T10:24:07Z" level=warning msg="Creation of endpoint failed due to invalid data" ciliumEndpointName=/ containerID= containerInterface= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=0 error="IP ipv4:10.17.239.232 is already in use" ipv4= ipv6= k8sPodName=/ subsys=daemon

It look like cilium has trouble with the amount of load and can no't respond to some query :

kubectl -n kube-system exec -it cilium-w5fnc -- cilium-dbg endpoint list
Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), clean-cilium-state (init), install-cni-binaries (init)

Error: cannot get endpoint list: Cilium API client timeout exceeded

command terminated with exit code 1

How can we reproduce the issue?

To reproduce, you need to install cilium with helm and calico chaining (not sure its relate to chaining)

Schedule a big amount of pods (we did by creating 500 pods) on the same nodes

Cilium Version

1.16.3

Kernel Version

5.15.0-122-generic

Kubernetes Version

v1.30.4

Regression

No response

Sysdump

dump is too big to uploaded, and we don't want to leak any data from this.
If you need any specific informations, we can add to this issue but we prefere to ensure not critical data could be inside.

Relevant log output

Mar 19 10:26:27 kubernetes1-worker10.pr.par5 kubelet[1309026]: E0319 10:26:27.611011 1309026 remote_runtime.go:193] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"3e4271f523ed1bffef07fc1cf8454093e9dfed49a50de3037fa0856b77dbbc87\": plugin type=\"cilium-cni\" failed (add): unable to create endpoint: [PUT /endpoint/{id}][400] putEndpointIdInvalid  IP ipv4:10.17.239.250 is already in use"
Mar 19 10:26:27 kubernetes1-worker10.pr.par5kubelet[1309026]: E0319 10:26:27.611123 1309026 kuberuntime_sandbox.go:72] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"3e4271f523ed1bffef07fc1cf8454093e9dfed49a50de3037fa0856b77dbbc87\": plugin type=\"cilium-cni\" failed (add): unable to create endpoint: [PUT /endpoint/{id}][400] putEndpointIdInvalid  IP ipv4:10.17.239.250 is already in use" pod="team-infrastructure/calico-vault-db-injector-canary-worker10-438"
Mar 19 10:26:27 kubernetes1-worker10.pr.par5 kubelet[1309026]: E0319 10:26:27.611165 1309026 kuberuntime_manager.go:1166] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"3e4271f523ed1bffef07fc1cf8454093e9dfed49a50de3037fa0856b77dbbc87\": plugin type=\"cilium-cni\" failed (add): unable to create endpoint: [PUT /endpoint/{id}][400] putEndpointIdInvalid  IP ipv4:10.17.239.250 is already in use" pod="team-infrastructure/calico-vault-db-injector-canary-worker10-438"
Mar 19 10:26:27 kubernetes1-worker10.pr.par5 kubelet[1309026]: E0319 10:26:27.611236 1309026 pod_workers.go:1298] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"calico-vault-db-injector-canary-worker10-438_team-infrastructure(e4cb950e-2a60-405d-b467-7c804e81ec19)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"calico-vault-db-injector-canary-worker10-438_team-infrastructure(e4cb950e-2a60-405d-b467-7c804e81ec19)\\\": rpc error: code = Unknown desc = failed to setup network for sandbox \\\"3e4271f523ed1bffef07fc1cf8454093e9dfed49a50de3037fa0856b77dbbc87\\\": plugin type=\\\"cilium-cni\\\" failed (add): unable to create endpoint: [PUT /endpoint/{id}][400] putEndpointIdInvalid  IP ipv4:10.17.239.250 is already in use\"" pod="team-infrastructure/calico-vault-db-injector-canary-worker10-438" podUID="e4cb950e-2a60-405d-b467-7c804e81ec19"
Mar 19 10:26:27 kubernetes1-worker10.pr.par5 kubelet[1309026]: I0319 10:26:27.616604 1309026 kubelet.go:2449] "SyncLoop UPDATE" source="api" pods=["team-infrastructure/calico-vault-db-injector-canary-worker10-440"]
Mar 19 10:26:27 kubernetes1-worker10.pr.par5 kubelet[1309026]: I0319 10:26:27.713318 1309026 kubelet.go:2449] "SyncLoop UPDATE" source="api" pods=["team-infrastructure/calico-vault-db-injector-canary-worker10-424"]
Mar 19 10:26:27 kubernetes1-worker10.pr.par5 kubelet[1309026]: E0319 10:26:27.737096 1309026 remote_runtime.go:193] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"9ea9685ce2ff7c3b5c53ee20e27b13801992ee4f04842c981f5aaa90acd31fda\": plugin type=\"cilium-cni\" failed (add): unable to create endpoint: [PUT /endpoint/{id}][400] putEndpointIdInvalid  IP ipv4:10.17.239.122 is already in use"
Mar 19 10:26:27 kubernetes1-worker10.pr.par5 kubelet[1309026]: E0319 10:26:27.737201 1309026 kuberuntime_sandbox.go:72] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"9ea9685ce2ff7c3b5c53ee20e27b13801992ee4f04842c981f5aaa90acd31fda\": plugin type=\"cilium-cni\" failed (add): unable to create endpoint: [PUT /endpoint/{id}][400] putEndpointIdInvalid  IP ipv4:10.17.239.122 is already in use" pod="team-infrastructure/calico-vault-db-injector-canary-worker10-364"
Mar 19 10:26:27 kubernetes1-worker10.pr.par5 kubelet[1309026]: E0319 10:26:27.737238 1309026 kuberuntime_manager.go:1166] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"9ea9685ce2ff7c3b5c53ee20e27b13801992ee4f04842c981f5aaa90acd31fda\": plugin type=\"cilium-cni\" failed (add): unable to create endpoint: [PUT /endpoint/{id}][400] putEndpointIdInvalid  IP ipv4:10.17.239.122 is already in use" pod="team-infrastructure/calico-vault-db-injector-canary-worker10-364"
Mar 19 10:26:27 kubernetes1-worker10.pr.par5  kubelet[1309026]: E0319 10:26:27.737317 1309026 pod_workers.go:1298] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"calico-vault-db-injector-canary-worker10-364_team-infrastructure(c329f135-6cf3-4fc5-b1df-b4c1a73c89ee)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"calico-vault-db-injector-canary-worker10-364_team-infrastructure(c329f135-6cf3-4fc5-b1df-b4c1a73c89ee)\\\": rpc error: code = Unknown desc = failed to setup network for sandbox \\\"9ea9685ce2ff7c3b5c53ee20e27b13801992ee4f04842c981f5aaa90acd31fda\\\": plugin type=\\\"cilium-cni\\\" failed (add): unable to create endpoint: [PUT /endpoint/{id}][400] putEndpointIdInvalid  IP ipv4:10.17.239.122 is already in use\"" pod="team-infrastructure/calico-vault-db-injector-canary-worker10-364" podUID="c329f135-6cf3-4fc5-b1df-b4c1a73c89ee

Anything else?

No response

Cilium Users Document

Are you a user of Cilium? Please add yourself to the Users doc

Code of Conduct

I agree to follow this project's Code of Conduct

Hello, we are still facing the issue.

Maybe a sysdump can help you in a more deep investigation :

sysdump.zip

We upgraded in version 1.17.3 with this fix : #38784 and still facing the same issue with cilium

Attached is another sysdump from last night.

Behavior was:

all pods could no longer schedule themselves a node and
the number of pods continued to grow
logs showed a bunch of errors like

level=warning msg="Creation of endpoint failed due to invalid data" ciliumEndpointName=/ containerID= containerInterface= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=0 error="IP ipv4:10.17.128.74 is already in use". But doing cilium ip get 10.17.128.74/32 returned nothing during the incident.
level=error msg="Error while reloading endpoint BPF program" [..] error="context deadline exceeded"

The only way to resolve the issue was, as usual, to restart cilium on that specific nodes. No loss of services because BPF is persisted on host and cilium pod resumes its state after restart.

It happened a node with Cilium 1.17.3
Sysdump is here: cilium-sysdump-20250423-081836 2.zip. But I had to remove some files to be able to upload it (it was too big).

I've tried to take a look at some of the stack:

$ cat gops-cilium-sfh2m-cilium-agent-20250423-081836-stack.txt | grep sync.Mutex.Lock | wc -l
     324
$ cat gops-cilium-sfh2m-cilium-agent-20250423-081836-stack.txt | grep sync.Mutex.Lock -A30 | grep .removeEndpoint | wc -l
     140
cat gops-cilium-sfh2m-cilium-agent-20250423-081836-stack.txt | grep sync.Mutex.Lock -A30 | grep .UpdatePolicyMaps | wc -l
      86

Could it be a lock somewhere that forbid the pod to schedule ?

I believe due to

--api-rate-limit=endpoint-create=rate-limit:200/s,parallel-requests:200,auto-adjust:false

and the rate of pod creations you have, these symptoms that you are experiencing are expected. Cilium is unable to keep up with the amount of load due that it is configured for. This is why Cilium's default for endpoint creations is 4/s and this is well above that. You might want to consider reducing the rate of endpoint creations.

Thanks for the insights!

Is there any documented or recommended limit for endpoint creation rate? The default 4/s seems quite low for our use case. If you have any tuning guidance around this (like a maximum value), I’d be happy to test different values under our load and contribute a PR to the documentation (maybe under the "Tuning" page).

For context: we're running on bare-metal nodes with 96 cores and 256 GB RAM, so we're comfortable dedicating substantial resources to Cilium.

After analyzing a recent pprof, I noticed that a significant amount of CPU is spent in conntrack GC, especially during endpoint deletion. Here’s a flamegraph snapshot showing that:

To mitigate this, I’ve made the following changes:

cilium-endpoint-gc-interval=1m: so unused endpoints are cleaned up more regularly, avoiding long GC bursts that delay IP release and trigger IP already in use issues.
conntrack-gc-interval=1m: to spread out the load of deleting large numbers of endpoints from our short-lived workloads.
bpf-ct-timeout-regular-tcp=1m: to minimize stale entries in the conntrack table and reduce GC workload.

Happy to iterate based on your recommendations — and again, I’m more than willing to benchmark and document best practices if that helps the broader community.

I see in the pprof results that the CT GC is running with a iterative dump of the map which is much slower than batching. In v1.18 (which is not released yet) we have improved this: #36288. If you are in a position to try that out, it would be good to know if it remedies high churn like your case.

Another option to reduce the impact of CT GC is to decouple it from running on each endpoint creation / deletion, and rather batch up X endpoint creations/deletions to 1 CT GC cycle. This requires code changes however. We'd be open to a PR 🙂

This issue has been automatically marked as stale because it has not
had recent activity. It will be closed if no further activity occurs.

This issue has not seen any activity since it was marked stale.
Closing.

test(e2e): self-heal Cilium orphaned-endpoint leak during install

test(e2e): self-heal Cilium orphaned-endpoint leak across install and…

test(e2e): escalate cilium leak healer to agent restart on leaked-IP …

Navigation Menu

Repository navigation

Cilium can't create new endpoint : "IP ipv4:10.17.239.232 is already in use" #38313

Is there an existing issue for this?

Version

What happened?

How can we reproduce the issue?

Cilium Version

Kernel Version

Kubernetes Version

Regression

Sysdump

Relevant log output

Anything else?

Cilium Users Document

Code of Conduct

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Cilium can't create new endpoint : "IP ipv4:10.17.239.232 is already in use" #38313

Description

Is there an existing issue for this?

Version

What happened?

How can we reproduce the issue?

Cilium Version

Kernel Version

Kubernetes Version

Regression

Sysdump

Relevant log output

Anything else?

Cilium Users Document

Code of Conduct

Activity

SoulKyu commented on Apr 15, 2025

SoulKyu commented on Apr 22, 2025

Solvik commented on Apr 23, 2025

christarazi commented on Apr 24, 2025

Solvik commented on Apr 24, 2025

christarazi commented on Apr 24, 2025

github-actions commented on Jun 24, 2025

github-actions commented on Jul 8, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions