Skip to content

[BUG] Successful HTTP failover hides repeated failures of the initial upstream from its circuit breaker #8156

Open

Description

@Aias00

Problem

When the initial upstream A fails and failover to B succeeds, the circuit breaker records the logical request's final success against A. A repeatedly failing node therefore remains eligible for initial traffic instead of reaching the configured failure threshold.

Confirmed on master at 1d0133477634f62a5093be10014c96f9f835c645. Suggested priority: P1.

Reproduction

  1. Configure a Divide selector with upstreams A and B, retryStrategy = failover and retry = 1.
  2. Make A return HTTP 503 for every request and B return HTTP 200 with a normal body.
  3. Select A as the initial target on four consecutive requests. The real-network fixture uses the existing specify-domain header with A's host:port to make that first choice deterministic; failover then selects B.
  4. Observe A's breaker after all four requests finish.

Production DividePlugin + NettyHttpClientPlugin + NettyClientMessageWriter, using two real local Reactor Netty HTTP servers, produced:

A actual HTTP 503 responses = 4
B successful HTTP 200 responses = 4
all four client response bodies = "B success"
A breaker blocking = false

The default failure threshold is three. A separate transport fixture reproduces the same outcome with A failing with ConnectException, so the problem is not specific to HTTP 503 classification.

Expected behavior

Record each failed attempt against the upstream that actually failed, even if a later failover makes the logical request successful. A's failures should open A's breaker, while B's success should be recorded against B.

Cause

  • DividePlugin.java records A's outcome around the entire downstream chain, which includes retry/failover and final response writing.
  • The first A failure is recovered before that outer recorder receives a terminal signal.
  • DefaultRetryStrategy.java records retry targets independently, but the initial attempt has no equivalent independent failure recording. Completion after B succeeds is therefore treated as A's success.

Suggested fix and regression coverage

Move failure/success recording to a per-attempt lifecycle keyed to the selected upstream. Record A's failed attempt before starting B, and record B's own outcome independently. Ensure aggregate request completion cannot clear failures attributed to another target.

Cover repeated A failure/B success, A failure/B failure, multiple failover targets, CURRENT retries, body errors, cancellation and half-open probes. Keep probe attempts exclusive and prevent duplicate outcome recording.

Validation

Both the real HTTP servers and the independent connection-failure fixture reproduce the problem at the commit above. The existing targeted test run passed 136 tests.

Activity

  1. konlue commented on Oct 30, 2026

    @konlue
    Contributor

    /assign

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions