Repository navigation
atenet-router no longer stalls 15s on every start - #3346
Merged
Eitan Yarmush (EItanya) merged 1 commit intoOct 29, 2026
Merged
Eitan Yarmush (EItanya) merged 1 commit into
Eitan Yarmush (EItanya) merged 1 commit into
Conversation
Chuang Wang (chuangw6)
requested review from
Lior Lieberman (LiorLieberman) and
Bowei Du (bowei)
October 28, 2026 18:14
Chuang Wang (chuangw6)
force-pushed
the
fix-xds-ads-sds-deadlock
branch
from
October 29, 2026 17:30
7531f48 to
9c28707
Compare
Lior Lieberman (LiorLieberman)
previously approved these changes
Oct 29, 2026
Lior Lieberman (LiorLieberman)
left a comment
Collaborator
There was a problem hiding this comment.
thanks, is this
Envoy asks for config in a fixed order: clusters, then the secrets the clusters need, then listeners, then the secrets the listeners need. Over state-of-the-world xDS, the router's ADS cache answers a secret request only if it names every secret. Envoy's first request names only the 2 cluster secrets, so it went unanswered until Envoy's 15s initial_fetch_timeout. The router Envoy now uses delta xDS. The delta path has no such check and still orders updates across resource types. Signed-off-by: Chuang Wang <[email protected]>
Chuang Wang (chuangw6)
force-pushed
the
fix-xds-ads-sds-deadlock
branch
from
October 29, 2026 21:00
9c28707 to
f32d3d3
Compare
Lior Lieberman (LiorLieberman)
approved these changes
Oct 29, 2026
Dmitry Berkovich (dberkov)
approved these changes
Oct 29, 2026
Eitan Yarmush (EItanya)
approved these changes
Oct 29, 2026
1 of 2 tasks
Dmitry Berkovich (dberkov)
pushed a commit
that referenced
this pull request
Nov 2, 2026
<!-- The PR title becomes the release note. Write it for users. --> Fixes #3268 ### Problem Envoy asks for config in a fixed order: clusters, then the 2 secrets the clusters need, then listeners, then the 2 secrets the listeners need. It won't ask for listeners until every cluster is ready. Over state-of-the-world xDS, the router's ADS cache [answers a request that lists names only if it names every resource](https://github.com/envoyproxy/go-control-plane/blob/71637ad69bbc5f51fbb2562e612a4365292804a5/pkg/cache/v3/simple.go#L491). Envoy's first secret request names only the 2 cluster secrets, so it was ignored. Envoy waited out its 15s `initial_fetch_timeout`, marked the cluster ready anyway, then asked for listeners and all 4 secrets. With one replica, that meant 15s with no ingress on every start. ### Solution The router Envoy now uses delta xDS (`api_type: DELTA_GRPC`). The delta path in go-control-plane has no such check, and with `ads=true` it still orders updates across resource types, which we'll want if the router ever serves dynamic config. ### Result With real Envoy v1.39.5, startup goes from 15.3s to 0.35s, with no timeout. `TestXdsServer_AnswersClusterSecretsBeforeListeners` checks the manifest uses delta and replays the startup over a delta stream. It fails on main. AI use: AI-assisted. I reviewed the change and ran it against real Envoy. - [x] Tests pass - [x] Docs updated (none needed) Signed-off-by: Chuang Wang <[email protected]> (cherry picked from commit 62efb70) Signed-off-by: Eitan Yarmush <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #3268
Problem
Envoy asks for config in a fixed order: clusters, then the 2 secrets the
clusters need, then listeners, then the 2 secrets the listeners need. It
won't ask for listeners until every cluster is ready.
Over state-of-the-world xDS, the router's ADS cache answers a request that
lists names only if it names every resource.
Envoy's first secret request names only the 2 cluster secrets, so it was
ignored. Envoy waited out its 15s
initial_fetch_timeout, marked thecluster ready anyway, then asked for listeners and all 4 secrets. With one
replica, that meant 15s with no ingress on every start.
Solution
The router Envoy now uses delta xDS (
api_type: DELTA_GRPC). The deltapath in go-control-plane has no such check, and with
ads=trueit stillorders updates across resource types, which we'll want if the router ever
serves dynamic config.
Result
With real Envoy v1.39.5, startup goes from 15.3s to 0.35s, with no
timeout.
TestXdsServer_AnswersClusterSecretsBeforeListenerschecks themanifest uses delta and replays the startup over a delta stream. It fails
on main.
AI use: AI-assisted. I reviewed the change and ran it against real Envoy.