Current Behavior
Occasionally, an ingress controller pod will time out when attempting to access the Lease resource used for leadership coordination and lose leadership. This results in the termination of the apisix-ingress-controller process in the manager container and a subsequent restart of the container, but not the termination of the pod itself. When this same ingress controller pod is selected as the leader again, single pod deployments seem to cause *_conf_version (e.g. upstreams_conf_version) errors to occur during syncing for the resources associated with those single pod deployments, which start immediately after the new leader pod starts syncing. This syncing error prevents the updating of the upstream IP address associated with all single pod deployments and causes 502/504 errors on the user end until:
- Any of the affected single pod deployments are scaled up to two or more pods
- Either
apisix or the affected apisix-ingress-controller pods are restarted
It's not clear to me why scaling up resolves the issue. Restarting the apisix pods resets the *_conf_version values in their configurations to 0 which allows the apisix-ingress-controller to write new values, and restarting the apisix-ingress-controller pods seems to correct the internal Unix timestamp that they use for *_conf_version values. There appear to be no differences between the configurations reflected in the ADC configurations viewable through the debug API or the debug logs before and after the apisix-ingress-controller process restarts, so it seems like the issue is isolated to the manager container and the apisix-ingress-controller process within. The Unix timestamp within the *_conf_version value reported in the error only seem to correspond to a successful sync on the leader pod at that time.
#2708 may be experiencing a similar issue but in this case, only single pod deployments are affected.
Expected Behavior
Single pod deployments are able to have their upstream IPs updated correctly even when a pod in which the apisix-ingress-controller process in the manager container terminates becomes the ingress controller leader again.
Error Logs
Lease timeout and process restart:
2026-05-21T11:06:19.710Z E0521 11:06:19.710497 1 leaderelection.go:429] Failed to update lock optimistically: Put "https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/ingress-apisix/leases/apisix-ingress-controller-leader?timeout=10s": context deadline exceeded (Client.Timeout exceeded while awaiting headers), falling back to slow path apisix-ingress-controller-aaaaaaaaaa-aaaaa manager
2026-05-21T11:06:29.709Z E0521 11:06:29.709691 1 leaderelection.go:436] error retrieving resource lock ingress-apisix/apisix-ingress-controller-leader: Get "https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/ingress-apisix/leases/apisix-ingress-controller-leader?timeout=10s": context deadline exceeded apisix-ingress-controller-aaaaaaaaaa-aaaaa manager
2026-05-21T11:06:29.709Z I0521 11:06:29.709728 1 leaderelection.go:297] failed to renew lease ingress-apisix/apisix-ingress-controller-leader: context deadline exceeded apisix-ingress-controller-aaaaaaaaaa-aaaaa manager
2026-05-21T11:06:29.710Z Error: leader election lost apisix-ingress-controller-aaaaaaaaaa-aaaaa manager
2026-05-21T11:06:29.710Z 2026-05-21T11:06:29.709Z DEBUG controller-runtime.events recorder/recorder.go:104 apisix-ingress-controller-aaaaaaaaaa-aaaaa_4f765088-329e-487e-b94d-b9b5f782252b stopped leading {"type": "Normal", "object": {"kind":"Lease","namespace":"ingress-apisix","name":"apisix-ingress-controller-leader","uid":"49eeacf3-2112-429d-b2ad-13c653d548e5","apiVersion":"coordination.k8s.io/v1","resourceVersion":"1540593399"}, "reason": "LeaderElection"} apisix-ingress-controller-aaaaaaaaaa-aaaaa manager
2026-05-21T11:06:29.710Z Usage:
apisix-ingress-controller [command] [flags]
apisix-ingress-controller [command] apisix-ingress-controller-aaaaaaaaaa-aaaaa manager
2026-05-21T11:06:29.710Z Available Commands:
completion Generate the autocompletion script for the specified shell
help Help about any command
version version for apisix-ingress-controller apisix-ingress-controller-aaaaaaaaaa-aaaaa manager
2026-05-21T11:06:29.710Z Flags:
-c, --config-path string configuration file path for apisix-ingress-controller
--controller-name string The name of the controller (default "apisix.apache.org/apisix-ingress-controller")
--health-probe-bind-address string The address the probe endpoint binds to. (default ":8081")
-h, --help help for apisix-ingress-controller
--log-level string The log level for apisix-ingress-controller (default "info")
--metrics-bind-address string The address the metrics endpoint binds to. Use :8443 for HTTPS or :8080 for HTTP, or leave as 0 to disable the metrics service. (default "0")
-v, --version version for apisix-ingress-controller apisix-ingress-controller-aaaaaaaaaa-aaaaa manager
2026-05-21T11:06:29.710Z Use "apisix-ingress-controller [command] --help" for more information about a command. apisix-ingress-controller-aaaaaaaaaa-aaaaa manager
2026-05-21T11:06:29.710Z Error: leader election lost apisix-ingress-controller-aaaaaaaaaa-aaaaa manager
2026-05-21T11:06:29.709Z INFO controller-runtime manager/internal.go:538 Stopping and waiting for non leader election runnables apisix-ingress-controller-aaaaaaaaaa-aaaaa manager
2026-05-21T11:06:30.238Z INFO root/root.go:125 controller start configuration {"config": {"log_level":"debug","controller_name":"apisix.apache.org/apisix-ingress-controller","leader_election_id":"apisix-ingress-controller-leader","metrics_addr":":8080","server_addr":":9092","enable_server":false,"enable_http2":false,"probe_addr":":8081","secure_metrics":false,"leader_election":{"lease_duration":"30s","renew_deadline":"20s","retry_period":"2s"},"exec_adc_timeout":"15s","provider":{"type":"apisix-standalone","sync_period":"1m0s","init_sync_delay":"20m0s"},"webhook":{"enable":true,"tls_cert_file":"tls.crt","tls_key_file":"tls.key","tls_cert_dir":"/certs","port":9443},"disable_gateway_api":true}} apisix-ingress-controller-aaaaaaaaaa-aaaaa manager
Leadership change:
2026-05-21T11:06:30.454Z I0521 11:06:30.454706 1 leaderelection.go:257] attempting to acquire leader lease ingress-apisix/apisix-ingress-controller-leader... apisix-ingress-controller-aaaaaaaaaa-aaaaa manager
2026-05-21T11:06:41.883Z I0521 11:06:41.878897 1 leaderelection.go:271] successfully acquired lease ingress-apisix/apisix-ingress-controller-leader apisix-ingress-controller-bbbbbbbbbb-bbbbb manager
2026-05-21T11:06:41.879Z INFO provider apisix/provider.go:254 starting provider, waiting for readiness apisix-ingress-controller-bbbbbbbbbb-bbbbb manager
2026-05-21T11:06:41.879Z INFO status.updater status/updater.go:131 started status update handler apisix-ingress-controller-bbbbbbbbbb-bbbbb manager
2026-05-21T11:06:41.879Z DEBUG controller-runtime.events recorder/recorder.go:104 apisix-ingress-controller-bbbbbbbbbb-bbbbb_66e0b0a2-e092-4c01-8a56-2cce983bb382 became leader {"type": "Normal", "object": {"kind":"Lease","namespace":"ingress-apisix","name":"apisix-ingress-controller-leader","uid":"49eeacf3-2112-429d-b2ad-13c653d548e5","apiVersion":"coordination.k8s.io/v1","resourceVersion":"1540594430"}, "reason": "LeaderElection"} apisix-ingress-controller-bbbbbbbbbb-bbbbb manager
Leadership returns to original pod:
2026-05-22T14:44:54.036Z I0522 14:44:54.036708 1 leaderelection.go:257] attempting to acquire leader lease ingress-apisix/apisix-ingress-controller-leader... apisix-ingress-controller-bbbbbbbbbb-bbbbb manager
2026-05-22T14:45:24.996Z I0522 14:45:24.996003 1 leaderelection.go:271] successfully acquired lease ingress-apisix/apisix-ingress-controller-leader apisix-ingress-controller-aaaaaaaaaa-aaaaa manager
2026-05-22T14:45:24.996Z DEBUG controller-runtime.events recorder/recorder.go:104 apisix-ingress-controller-aaaaaaaaaa-aaaaa_ae6f86ca-a178-46ec-b99f-e3c301bcda8d became leader {"type": "Normal", "object": {"kind":"Lease","namespace":"ingress-apisix","name":"apisix-ingress-controller-leader","uid":"49eeacf3-2112-429d-b2ad-13c653d548e5","apiVersion":"coordination.k8s.io/v1","resourceVersion":"1543653763"}, "reason": "LeaderElection"} apisix-ingress-controller-aaaaaaaaaa-aaaaa manager
2026-05-22T14:45:24.996Z INFO provider apisix/provider.go:254 starting provider, waiting for readiness apisix-ingress-controller-aaaaaaaaaa-aaaaa manager
2026-05-22T14:45:24.996Z INFO status.updater status/updater.go:131 started status update handler apisix-ingress-controller-aaaaaaaaaa-aaaaa manager
Sync failure:
2026-05-22T14:50:26.156Z DEBUG provider.executor client/executor.go:303 received HTTP response from ADC Server {"server": "http://10.10.10.10:9180,http://10.10.10.20:9180", "status": 202, "response": "{\"status\":\"all_failed\",\"total_resources\":2,\"success_count\":0,\"failed_count\":2,\"success\":[],\"failed\":[{\"server\":\"http://10.10.10.10:9180\",\"event\":{},\"failed_at\":\"2026-05-22T14:50:26.000Z\",\"reason\":\"upstreams_conf_version must be greater than or equal to (1779434128737)\",\"response\":{\"status\":400,\"headers\":{\"date\":\"Fri, 22 May 2026 14:50:26 GMT\",\"content-type\":\"application/json\",\"transfer-encoding\":\"chunked\",\"connection\":\"keep-alive\",\"server\":\"APISIX/3.16.0\",\"access-control-allow-origin\":\"*\",\"access-control-allow-credentials\":\"true\",\"access-control-expose-headers\":\"*\",\"access-control-max-age\":\"3600\"},\"data\":{\"error_msg\":\"upstreams_conf_version must be greater than or equal to (1779434128737)\"}}},{\"server\":\"http://10.10.10.20:9180\",\"event\":{},\"failed_at\":\"2026-05-22T14:50:26.000Z\",\"reason\":\"upstreams_conf_version must be greater than or equal to (1779434128737)\",\"response\":{\"status\":400,\"headers\":{\"date\":\"Fri, 22 May 2026 14:50:26 GMT\",\"content-type\":\"application/json\",\"transfer-encoding\":\"chunked\",\"connection\":\"keep-alive\",\"server\":\"APISIX/3.16.0\",\"access-control-allow-origin\":\"*\",\"access-control-allow-credentials\":\"true\",\"access-control-expose-headers\":\"*\",\"access-control-max-age\":\"3600\"},\"data\":{\"error_msg\":\"upstreams_conf_version must be greater than or equal to (1779434128737)\"}}}]}"} apisix-ingress-controller-aaaaaaaaaa-aaaaa manager
Steps to Reproduce
The bug's dependence on the apisix-ingress-controller process timing out when getting the Lease resource has made it difficult to reproduce, but these are the steps:
- Scale target deployment to 1 pod and the apisix-ingress-controller deployment to 2 pods
- Wait for one of the apisix-ingress-controller processes in the manager container to self-terminate in the leader pod
- If the new leader pod selected isn't the same as the previous, terminate the leader until leadership returns to the previous pod
- Kill the target deployment's pod, spawning a new pod
- upstream_conf_version error
Environment
- APISIX version: 3.16.0
- APISIX Ingress controller version: 2.0.1
- APISIX Helm chart version: 2.14.0
- Kubernetes cluster version: 1.34.2
Current Behavior
Occasionally, an ingress controller pod will time out when attempting to access the
Leaseresource used for leadership coordination and lose leadership. This results in the termination of theapisix-ingress-controllerprocess in themanagercontainer and a subsequent restart of the container, but not the termination of the pod itself. When this same ingress controller pod is selected as the leader again, single pod deployments seem to cause*_conf_version(e.g.upstreams_conf_version) errors to occur during syncing for the resources associated with those single pod deployments, which start immediately after the new leader pod starts syncing. This syncing error prevents the updating of the upstream IP address associated with all single pod deployments and causes 502/504 errors on the user end until:apisixor the affectedapisix-ingress-controllerpods are restartedIt's not clear to me why scaling up resolves the issue. Restarting the
apisixpods resets the*_conf_versionvalues in their configurations to 0 which allows theapisix-ingress-controllerto write new values, and restarting theapisix-ingress-controllerpods seems to correct the internal Unix timestamp that they use for*_conf_versionvalues. There appear to be no differences between the configurations reflected in the ADC configurations viewable through the debug API or the debug logs before and after theapisix-ingress-controllerprocess restarts, so it seems like the issue is isolated to themanagercontainer and theapisix-ingress-controllerprocess within. The Unix timestamp within the*_conf_versionvalue reported in the error only seem to correspond to a successful sync on the leader pod at that time.#2708 may be experiencing a similar issue but in this case, only single pod deployments are affected.
Expected Behavior
Single pod deployments are able to have their upstream IPs updated correctly even when a pod in which the
apisix-ingress-controllerprocess in themanagercontainer terminates becomes the ingress controller leader again.Error Logs
Lease timeout and process restart:
Leadership change:
Leadership returns to original pod:
Sync failure:
Steps to Reproduce
The bug's dependence on the apisix-ingress-controller process timing out when getting the Lease resource has made it difficult to reproduce, but these are the steps:
Environment