fix: rebuild the ADC baseline on leader acquisition#2785
Merged
Conversation
Contributor
|
@AlinsRan How is the status of this PR, its impacting us massively. Happy to contribute if you like. |
Member
|
@johannes-engler-mw I will follow up this PR next week. |
AlinsRan
force-pushed
the
fix/adc-cachekey-leader-nonce
branch
from
July 13, 2026 05:37
7b12ae4 to
4bb4cf0
Compare
AlinsRan
force-pushed
the
fix/adc-cachekey-leader-nonce
branch
from
July 13, 2026 07:44
30f72cd to
714884f
Compare
AlinsRan
force-pushed
the
fix/adc-cachekey-leader-nonce
branch
5 times, most recently
from
July 14, 2026 01:00
5321cc3 to
1965f0d
Compare
The ADC server runs as a sidecar that outlives the controller process. Losing the
lease terminates the manager container but not the sidecar, so the baseline ADC
holds per cacheKey -- the last synced content plus the conf_version it generated --
survives a leadership change. Once the pod is elected again, ADC diffs against that
frozen snapshot and carries over conf_versions older than the ones the leader in
between has already pushed. APISIX standalone requires *_conf_version to be
monotonic and rejects the request:
upstreams_conf_version must be greater than or equal to (1779434128737)
The data plane validates the configuration as a whole, so every later sync keeps
failing and no update lands at all, endpoint updates included. Backends running a
single pod then return 502/504 once that pod is rescheduled, while multi-pod
backends keep serving on their remaining endpoints.
Winning the election is the one moment a baseline from an earlier term can enter the
picture, so that is where to answer it: the provider, which only runs once elected,
puts every baseline in doubt, and the first sync of each cacheKey re-derives it from
the data plane through ADC's bypassCache before anything is pushed from it. Only a
sync ADC accepts settles the question, so a rebuild that never landed is attempted
again rather than assumed.
Prevention cannot cover a desync no leadership change explains -- another writer on
this data plane, something we did not foresee -- and the only way any of that shows
itself is the data plane refusing a conf_version. So that rejection rebuilds the
baseline and pushes once more. It is recognised by the field it names rather than by
the sentence around it: conf_version belongs to the standalone admin API, we send
those keys ourselves, while the prose around them is APISIX's to reword. This is a
net, not the fix: nothing else rebuilds on it, and the reported bug does not come
back if it ever stops firing.
bypassCache needs ADC >= 0.27.0, so bump ADC_VERSION to 0.27.1. The sidecar image
the chart deploys has to move with it, or the rebuild is answered with a schema
error instead of healing.
Fixes apache#2774
Bump every *_conf_version the data plane holds straight through its Admin API, which
is what a leader in between would have left behind: it pushed while this pod was on
standby, so APISIX ends up holding versions newer than the ones the ADC sidecar
generated -- and the sidecar survives a leadership change, so it keeps diffing
against that older baseline.
Nothing restarts during the spec, so the baseline the controller rebuilt when it was
elected is already current by the time the versions are injected. What the spec
therefore exercises is the safety net: the push that follows is refused by the data
plane, and only re-reading the data plane gets the Ingress update to land.
Without the fix it reproduces the reported failure. The sync carries over the
conf_versions of the resource types its diff does not touch, and APISIX rejects the
whole configuration:
plugins_conf_version must be greater than or equal to (1783927244865)
Expose the data plane Admin API to the specs through the tunnel the scaffold already
forwards; reading and writing the configuration APISIX actually holds is what makes
the desync reproducible from a test.
AlinsRan
force-pushed
the
fix/adc-cachekey-leader-nonce
branch
from
July 14, 2026 05:40
1965f0d to
a939a84
Compare
nic-6443
approved these changes
Jul 15, 2026
shreemaan-abhishek
approved these changes
Jul 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2774.
Problem
The ADC server is a sidecar that outlives the controller process. When the pod loses the lease, the manager container restarts but the sidecar does not, so ADC keeps its per-
cacheKeybaseline — the last synced content plus theconf_versionit generated.Meanwhile another pod, as leader, bumps
*_conf_versionin APISIX. When the first pod is elected again, ADC diffs against its stale baseline and pushes aconf_versionolder than what APISIX now holds. APISIX standalone requires it to be monotonic and rejects the whole push:From then on every sync fails and nothing lands — single-pod backends 502/504 once rescheduled. Restarting the pod clears it (the sidecar dies with it); the manager restarting on its own does not.
Fix
ADC ≥ 0.27.0 accepts
bypassCacheon/sync: it drops the baseline and re-derives it from the data plane. The baseline is rebuilt in two places:provider.Start()only runs once elected, and the first sync of each cacheKey rebuilds before pushing. Staleness can only enter on a leadership change, so this is where it is answered.conf_versionrejection — a safety net for a desync no leadership change explains (another writer, an evicted cache entry). Matched by the field name, not the message wording.Everything else is untouched: unrelated errors don't trigger a rebuild, non-standalone mode has no
conf_version, and a normal sync is byte-for-byte what it was (bypassCacheisomitempty).ADC_VERSION→ 0.27.1.Important
The ADC sidecar image (from the Helm chart, not this repo) must also move to ≥ 0.27.0, or the rebuild is rejected as an unknown field and the fix is a no-op.
Test
test/e2e/apisix/conf_version.goreproduces the bug: bump every*_conf_versionvia the Admin API, then assert an Ingress update still lands. Fails onmaster, passes with the fix. Verified against APISIX 3.17.0 + ADC 0.27.1.