You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[BUG] Successful HTTP failover hides repeated failures of the initial upstream from its circuit breaker #8156
When the initial upstream A fails and failover to B succeeds, the circuit breaker records the logical request's final success against A. A repeatedly failing node therefore remains eligible for initial traffic instead of reaching the configured failure threshold.
Confirmed on master at 1d0133477634f62a5093be10014c96f9f835c645. Suggested priority: P1.
Reproduction
Configure a Divide selector with upstreams A and B, retryStrategy = failover and retry = 1.
Make A return HTTP 503 for every request and B return HTTP 200 with a normal body.
Select A as the initial target on four consecutive requests. The real-network fixture uses the existing specify-domain header with A's host:port to make that first choice deterministic; failover then selects B.
Observe A's breaker after all four requests finish.
Production DividePlugin + NettyHttpClientPlugin + NettyClientMessageWriter, using two real local Reactor Netty HTTP servers, produced:
A actual HTTP 503 responses = 4
B successful HTTP 200 responses = 4
all four client response bodies = "B success"
A breaker blocking = false
The default failure threshold is three. A separate transport fixture reproduces the same outcome with A failing with ConnectException, so the problem is not specific to HTTP 503 classification.
Expected behavior
Record each failed attempt against the upstream that actually failed, even if a later failover makes the logical request successful. A's failures should open A's breaker, while B's success should be recorded against B.
Cause
DividePlugin.java records A's outcome around the entire downstream chain, which includes retry/failover and final response writing.
The first A failure is recovered before that outer recorder receives a terminal signal.
DefaultRetryStrategy.java records retry targets independently, but the initial attempt has no equivalent independent failure recording. Completion after B succeeds is therefore treated as A's success.
Suggested fix and regression coverage
Move failure/success recording to a per-attempt lifecycle keyed to the selected upstream. Record A's failed attempt before starting B, and record B's own outcome independently. Ensure aggregate request completion cannot clear failures attributed to another target.
Cover repeated A failure/B success, A failure/B failure, multiple failover targets, CURRENT retries, body errors, cancellation and half-open probes. Keep probe attempts exclusive and prevent duplicate outcome recording.
Validation
Both the real HTTP servers and the independent connection-failure fixture reproduce the problem at the commit above. The existing targeted test run passed 136 tests.
Activity
konlue commentedon Oct 30, 2026
/assign
fix(divide,httpclient): record each attempt's breaker outcome instead…