Skip to content

fix: rebuild the ADC baseline on leader acquisition#2785

Merged
AlinsRan merged 2 commits into
apache:masterfrom
AlinsRan:fix/adc-cachekey-leader-nonce
Jul 17, 2026
Merged

fix: rebuild the ADC baseline on leader acquisition#2785
AlinsRan merged 2 commits into
apache:masterfrom
AlinsRan:fix/adc-cachekey-leader-nonce

Conversation

@AlinsRan

@AlinsRan AlinsRan commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

Fixes #2774.

Problem

The ADC server is a sidecar that outlives the controller process. When the pod loses the lease, the manager container restarts but the sidecar does not, so ADC keeps its per-cacheKey baseline — the last synced content plus the conf_version it generated.

Meanwhile another pod, as leader, bumps *_conf_version in APISIX. When the first pod is elected again, ADC diffs against its stale baseline and pushes a conf_version older than what APISIX now holds. APISIX standalone requires it to be monotonic and rejects the whole push:

upstreams_conf_version must be greater than or equal to (1779434128737)

From then on every sync fails and nothing lands — single-pod backends 502/504 once rescheduled. Restarting the pod clears it (the sidecar dies with it); the manager restarting on its own does not.

Fix

ADC ≥ 0.27.0 accepts bypassCache on /sync: it drops the baseline and re-derives it from the data plane. The baseline is rebuilt in two places:

  • On leader acquisition — the real fix. provider.Start() only runs once elected, and the first sync of each cacheKey rebuilds before pushing. Staleness can only enter on a leadership change, so this is where it is answered.
  • On a conf_version rejection — a safety net for a desync no leadership change explains (another writer, an evicted cache entry). Matched by the field name, not the message wording.

Everything else is untouched: unrelated errors don't trigger a rebuild, non-standalone mode has no conf_version, and a normal sync is byte-for-byte what it was (bypassCache is omitempty).

ADC_VERSION → 0.27.1.

Important

The ADC sidecar image (from the Helm chart, not this repo) must also move to ≥ 0.27.0, or the rebuild is rejected as an unknown field and the fix is a no-op.

Test

test/e2e/apisix/conf_version.go reproduces the bug: bump every *_conf_version via the Admin API, then assert an Ingress update still lands. Fails on master, passes with the fix. Verified against APISIX 3.17.0 + ADC 0.27.1.

@johannes-engler-mw

Copy link
Copy Markdown
Contributor

@AlinsRan How is the status of this PR, its impacting us massively. Happy to contribute if you like.

@juzhiyuan

Copy link
Copy Markdown
Member

@johannes-engler-mw I will follow up this PR next week.

@AlinsRan
AlinsRan force-pushed the fix/adc-cachekey-leader-nonce branch from 7b12ae4 to 4bb4cf0 Compare July 13, 2026 05:37
@AlinsRan AlinsRan changed the title fix: rotate ADC cacheKey nonce on leader acquisition fix: rebuild the ADC baseline on leader acquisition Jul 13, 2026
@AlinsRan
AlinsRan force-pushed the fix/adc-cachekey-leader-nonce branch from 30f72cd to 714884f Compare July 13, 2026 07:44
@AlinsRan AlinsRan changed the title fix: rebuild the ADC baseline on leader acquisition fix: rebuild the ADC baseline when the data plane rejects a stale conf_version Jul 13, 2026
@AlinsRan
AlinsRan force-pushed the fix/adc-cachekey-leader-nonce branch 5 times, most recently from 5321cc3 to 1965f0d Compare July 14, 2026 01:00
@AlinsRan AlinsRan changed the title fix: rebuild the ADC baseline when the data plane rejects a stale conf_version fix: rebuild the ADC baseline on leader acquisition Jul 14, 2026
AlinsRan added 2 commits July 14, 2026 13:40
The ADC server runs as a sidecar that outlives the controller process. Losing the
lease terminates the manager container but not the sidecar, so the baseline ADC
holds per cacheKey -- the last synced content plus the conf_version it generated --
survives a leadership change. Once the pod is elected again, ADC diffs against that
frozen snapshot and carries over conf_versions older than the ones the leader in
between has already pushed. APISIX standalone requires *_conf_version to be
monotonic and rejects the request:

    upstreams_conf_version must be greater than or equal to (1779434128737)

The data plane validates the configuration as a whole, so every later sync keeps
failing and no update lands at all, endpoint updates included. Backends running a
single pod then return 502/504 once that pod is rescheduled, while multi-pod
backends keep serving on their remaining endpoints.

Winning the election is the one moment a baseline from an earlier term can enter the
picture, so that is where to answer it: the provider, which only runs once elected,
puts every baseline in doubt, and the first sync of each cacheKey re-derives it from
the data plane through ADC's bypassCache before anything is pushed from it. Only a
sync ADC accepts settles the question, so a rebuild that never landed is attempted
again rather than assumed.

Prevention cannot cover a desync no leadership change explains -- another writer on
this data plane, something we did not foresee -- and the only way any of that shows
itself is the data plane refusing a conf_version. So that rejection rebuilds the
baseline and pushes once more. It is recognised by the field it names rather than by
the sentence around it: conf_version belongs to the standalone admin API, we send
those keys ourselves, while the prose around them is APISIX's to reword. This is a
net, not the fix: nothing else rebuilds on it, and the reported bug does not come
back if it ever stops firing.

bypassCache needs ADC >= 0.27.0, so bump ADC_VERSION to 0.27.1. The sidecar image
the chart deploys has to move with it, or the rebuild is answered with a schema
error instead of healing.

Fixes apache#2774
Bump every *_conf_version the data plane holds straight through its Admin API, which
is what a leader in between would have left behind: it pushed while this pod was on
standby, so APISIX ends up holding versions newer than the ones the ADC sidecar
generated -- and the sidecar survives a leadership change, so it keeps diffing
against that older baseline.

Nothing restarts during the spec, so the baseline the controller rebuilt when it was
elected is already current by the time the versions are injected. What the spec
therefore exercises is the safety net: the push that follows is refused by the data
plane, and only re-reading the data plane gets the Ingress update to land.

Without the fix it reproduces the reported failure. The sync carries over the
conf_versions of the resource types its diff does not touch, and APISIX rejects the
whole configuration:

    plugins_conf_version must be greater than or equal to (1783927244865)

Expose the data plane Admin API to the specs through the tunnel the scaffold already
forwards; reading and writing the configuration APISIX actually holds is what makes
the desync reproducible from a test.
@AlinsRan
AlinsRan force-pushed the fix/adc-cachekey-leader-nonce branch from 1965f0d to a939a84 Compare July 14, 2026 05:40
@AlinsRan
AlinsRan merged commit 6fc49d3 into apache:master Jul 17, 2026
26 checks passed
@AlinsRan
AlinsRan deleted the fix/adc-cachekey-leader-nonce branch July 17, 2026 16:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: Leader pod lease timeout leads to syncing errors that result in single pod deployments not receiving IP updates

5 participants