Skip to content

atenet-router no longer stalls 15s on every start - #3346

Merged
Eitan Yarmush (EItanya) merged 1 commit into
agent-substrate:mainfrom
chuangw6:fix-xds-ads-sds-deadlock
Oct 29, 2026
Merged

Eitan Yarmush (EItanya) merged 1 commit into
agent-substrate:mainfrom
chuangw6:fix-xds-ads-sds-deadlock

Conversation

@chuangw6

@chuangw6 Chuang Wang (chuangw6) commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #3268

Problem

Envoy asks for config in a fixed order: clusters, then the 2 secrets the
clusters need, then listeners, then the 2 secrets the listeners need. It
won't ask for listeners until every cluster is ready.

Over state-of-the-world xDS, the router's ADS cache answers a request that
lists names only if it names every resource
.
Envoy's first secret request names only the 2 cluster secrets, so it was
ignored. Envoy waited out its 15s initial_fetch_timeout, marked the
cluster ready anyway, then asked for listeners and all 4 secrets. With one
replica, that meant 15s with no ingress on every start.

Solution

The router Envoy now uses delta xDS (api_type: DELTA_GRPC). The delta
path in go-control-plane has no such check, and with ads=true it still
orders updates across resource types, which we'll want if the router ever
serves dynamic config.

Result

With real Envoy v1.39.5, startup goes from 15.3s to 0.35s, with no
timeout. TestXdsServer_AnswersClusterSecretsBeforeListeners checks the
manifest uses delta and replays the startup over a delta stream. It fails
on main.

AI use: AI-assisted. I reviewed the change and ran it against real Envoy.

  • Tests pass
  • Docs updated (none needed)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks, is this

Comment thread cmd/atenet/internal/router/xds.go Resolved
Envoy asks for config in a fixed order: clusters, then the secrets the
clusters need, then listeners, then the secrets the listeners need. Over
state-of-the-world xDS, the router's ADS cache answers a secret request
only if it names every secret. Envoy's first request names only the 2
cluster secrets, so it went unanswered until Envoy's 15s
initial_fetch_timeout.

The router Envoy now uses delta xDS. The delta path has no such check
and still orders updates across resource types.

Signed-off-by: Chuang Wang <[email protected]>
@EItanya
Eitan Yarmush (EItanya) added this pull request to the merge queue Oct 29, 2026
Merged via the queue into agent-substrate:main with commit 62efb70 Oct 29, 2026
5 checks passed
Dmitry Berkovich (dberkov) pushed a commit that referenced this pull request Nov 2, 2026
<!-- The PR title becomes the release note. Write it for users. -->

Fixes #3268

### Problem

Envoy asks for config in a fixed order: clusters, then the 2 secrets the
clusters need, then listeners, then the 2 secrets the listeners need. It
won't ask for listeners until every cluster is ready.

Over state-of-the-world xDS, the router's ADS cache [answers a request
that
lists names only if it names every
resource](https://github.com/envoyproxy/go-control-plane/blob/71637ad69bbc5f51fbb2562e612a4365292804a5/pkg/cache/v3/simple.go#L491).
Envoy's first secret request names only the 2 cluster secrets, so it was
ignored. Envoy waited out its 15s `initial_fetch_timeout`, marked the
cluster ready anyway, then asked for listeners and all 4 secrets. With
one
replica, that meant 15s with no ingress on every start.

### Solution

The router Envoy now uses delta xDS (`api_type: DELTA_GRPC`). The delta
path in go-control-plane has no such check, and with `ads=true` it still
orders updates across resource types, which we'll want if the router
ever
serves dynamic config.

### Result

With real Envoy v1.39.5, startup goes from 15.3s to 0.35s, with no
timeout. `TestXdsServer_AnswersClusterSecretsBeforeListeners` checks the
manifest uses delta and replays the startup over a delta stream. It
fails
on main.

AI use: AI-assisted. I reviewed the change and ran it against real
Envoy.

- [x] Tests pass
- [x] Docs updated (none needed)

Signed-off-by: Chuang Wang <[email protected]>
(cherry picked from commit 62efb70)
Signed-off-by: Eitan Yarmush <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

atenet-router xDS server sets ads=true on SnapshotCache, deadlocking Envoy SDS startup for 15s on every router boot

4 participants