Repository navigation
atenet-router: raise the tunnel circuit breaker to 1m - #3087
Merged
Lior Lieberman (LiorLieberman) merged 1 commit intoOct 22, 2026
Merged
Lior Lieberman (LiorLieberman) merged 1 commit into
Lior Lieberman (LiorLieberman) merged 1 commit into
Conversation
Yufan Su (yufan-su)
requested review from
Lior Lieberman (LiorLieberman),
Bowei Du (bowei) and
yanavlasov
October 21, 2026 20:36
Bowei Du (bowei)
previously approved these changes
Oct 21, 2026
Collaborator
Is this the case? I thought yanavlasov mentioned its connections to the internal_listener? |
Lior Lieberman (LiorLieberman)
requested changes
Oct 21, 2026
Lior Lieberman (LiorLieberman)
left a comment
Lior Lieberman (LiorLieberman)
left a comment
Collaborator
There was a problem hiding this comment.
looks like it is used in buildActorTunnelCluster and other buildActorConnectCluster - which is used both for the internal_listener AND the proxy-->atunnel.
Should we split those configs?
I think increasing it for the proxy-->atunnel needs more thinking.. as this is real circut breakers we should think carefully about. The internal_listener ones are less important
Collaborator
Author
Yes we can split into two limit But I think we should increase the |
Yufan Su (yufan-su)
force-pushed
the
raise-tunnel-circuit-breaker
branch
from
October 22, 2026 00:13
6b3207a to
9f4923f
Compare
Yufan Su (yufan-su)
force-pushed
the
raise-tunnel-circuit-breaker
branch
from
October 22, 2026 01:08
9f4923f to
b01a268
Compare
yanavlasov
approved these changes
Oct 22, 2026
Collaborator
|
LGTM from me. |
Lior Lieberman (LiorLieberman)
approved these changes
Oct 22, 2026
M.Besher Massri (besher-massri)
pushed a commit
to besher-massri/substrate
that referenced
this pull request
Oct 22, 2026
…e#3087) At 1k nodes and 200k actors, atenet-router starts failing ingress with `503 upstream connect error or disconnect/reset before headers. reset reason: remote connection failure`. The requests that fail are the ones that need a new tunnel. `actor_connect` refuses the tunnel once a circuit breaker set by `tunnelCircuitBreakerLimit` (65536) is full. - Split the limit into tunnelCircuitBreakerLimit and actorConnectCircuitBreakerLimit, so each cluster can be tuned on its own. - Set both to 1048576. For actor_connect that covers 11,000 workers on up to about 95 Envoy threads. Leaving the breakers unset would mean Envoy's default of 1024. - Tuning actor_connect's limits is tracked in agent-substrate#3100. - [ ] Tests pass - [ ] Docs updated
2 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
At 1k nodes and 200k actors, atenet-router starts failing ingress with
503 upstream connect error or disconnect/reset before headers. reset reason: remote connection failure. The requests that fail are the ones that need a newtunnel.
actor_connectrefuses the tunnel once a circuit breaker set bytunnelCircuitBreakerLimit(65536) is full.Split the limit into tunnelCircuitBreakerLimit and
actorConnectCircuitBreakerLimit, so each cluster can be tuned on its own.
Set both to 1048576. For actor_connect that covers 11,000 workers on up to
about 95 Envoy threads. Leaving the breakers unset would mean Envoy's default
of 1024.
Tuning actor_connect's limits is tracked in atenet-router: size actor_connect's circuit breakers #3100.
Tests pass
Docs updated