Skip to content

[BUG] HTTP failover releases leastActive inflight accounting before the response body finishes #8158

Closed

Description

@Aias00

Problem

For HTTP failover, the actual retry target B returns to its idle inflight baseline as soon as response headers arrive, even while B's response body is still streaming. The failed initial target A remains reserved until the complete logical request finishes.

This makes least-active routing observe the opposite of the actual workload: the serving node looks idle and the failed node looks busy.

Confirmed on master at 1d0133477634f62a5093be10014c96f9f835c645. Suggested priority: P2.

Reproduction

  1. Configure Divide with loadBalance = leastActive, retryStrategy = failover and retry = 1.
  2. Let A return HTTP 503 and B return a successful response with a first body chunk followed by a gate that keeps the body open.
  3. Start a request whose initial choice is A and which fails over to B.
  4. After NettyClientMessageWriter starts consuming B's body, inspect the two upstreams' inflight counters before opening the gate.

The fixture runs production Divide, Netty HTTP forwarding, failover and response writing against real local Reactor Netty HTTP servers. It expands the discovered pool from A-only to A+B between initial selection and forwarding to make the initial choice deterministic. It normalizes both counter baselines to zero so the outstanding reservations are directly visible.

Observed:

while B's response body remains open:
  request still running = true
  initial A inflight = 1
  actual serving B inflight = 0

after B's final body chunk and request completion:
  A inflight = 0
  B inflight = 0

The incorrect value during body transfer is the problem; both counters eventually returning to baseline does not make the active-request measurement correct.

Expected behavior

Release A's reservation when A's attempt fails. Hold B's reservation until B's actual response handling finishes, fails or is cancelled, including its response body.

Cause

  • DefaultRetryStrategy.java attaches retry inflight release to the Mono that produces response headers.
  • Response-body writing happens later, after AbstractHttpClientPlugin.execute() passes the response to the downstream response plugin.
  • DividePlugin.java attaches the initial target's release around the entire logical request instead.

P2C uses the same header-bound release in the retry path and needs analogous lifecycle coverage; the real-network counter reproduction here specifically validates leastActive.

Suggested fix and regression coverage

Bind reservations to the full lifecycle of each actual upstream attempt. Include response-body completion, error and cancellation. Keep initial-target and failover-target ownership separate, including synchronous failures and topology updates.

Add a regression with A failure and a gated B response. Assert counters both while the body is open and after completion/cancellation, rather than checking only the eventual baseline.

Validation

The behavior above was reproduced with real HTTP servers at the specified commit. Existing targeted HTTP, Divide and response tests passed as part of a 136-test run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions