[Feature][Load Balancer] Add opt-in adaptive scoring for HTTP upstream selection #8300

Goal

Add opt-in adaptive scoring for HTTP upstream selection in Divide, building on ShenYu's existing P2C behavior. Evaluate whether recent latency, in-flight work and classified upstream outcomes improve selection under heterogeneous capacity and changing load. No throughput or tail-latency improvement is assumed before benchmarks.

This is a target-selection feature. #7930 separately owns adaptive concurrency admission; this issue does not introduce another limiter, queue or connection pool.

Current baseline and motivation

Reviewed master 6222cf6cbe95ef22dafed415ab67bab95b87a53b:

• P2cLoadBalancer.load already combines smoothed lag and in-flight count: (sqrt(lag) + 1) * inflight, plus idle exploration. The proposal must demonstrate a useful increment over this baseline rather than claim ShenYu lacks adaptive selection altogether.
• Divide's feedback calculation measures lag from the upstream's shared lastPicked. By inspection, overlapping calls can overwrite that timestamp: A starts at 0 ms, B selects the same upstream at 90 ms, and A completes at 100 ms; the expression can attribute 10 ms to A. This is a source-level illustration, not a new network reproduction.
• LoadBalancer SPI currently exposes selection; Divide applies strategy-specific completion callbacks. Precise feedback requires the actual transport attempt and its response-body lifetime, not only the enclosing logical request.

Ownership and dependencies

• [Task][HTTP Client] Add optional observations for upstream attempts and response completion #8209 owns optional standard-HTTP attempt/body observations for Netty and WebClient. Reuse its identities, timing/outcome facts and read-only observation contract; do not replace its ownership with a second telemetry pipeline.
• [BUG] HTTP failover releases leastActive inflight accounting before the response body finishes #8158 / open PR fix(divide): bind failover inflight accounting to each attempt's response lifetime (#8158) #8251 own initial/failover-target reservation lifetimes. Correct attribution is a prerequisite; consume the final merged contract rather than reimplement the fix here or assume the open PR is already delivered.
• [Feature] Unify upstream eligibility reasons across discovery, health and request feedback #8064 owns endpoint eligibility composition. This feature ranks only already eligible candidates and must not re-admit unhealthy, draining or breaker-excluded endpoints.
• [Volunteer] <spring cloud plugin support peak ewma load balance algorithm> #5205 concerns Peak EWMA for the Spring Cloud plugin. Coordinate algorithm definitions and reusable components, while keeping this first delivery limited to HTTP/Divide.
• [BUG] shortestResponse does not track inflight requests and ignores growing queue depth #7973 separately owns shortestResponse in-flight accounting. Do not bundle that strategy's repair into this feature.
• Historical discussion Refactor: Decouple load balance metrics collection from DividePlugin #6426 covered metrics callbacks. Reconcile with the current code and [Task][HTTP Client] Add optional observations for upstream attempts and response completion #8209 before changing public SPI; its closed status does not replace review of current cancellation/body semantics.

First delivery

1. Accurate, bounded feedback. Keep a monotonic start time per actual transport subscription/attempt, including retries to the same target and failover to another target. Record headers-ready and response-terminal times separately. For initial delivery, enforce adaptive scoring only for supported non-streaming HTTP workloads; explicitly define handling/bypass for SSE and long-lived streams. Resource ownership must still extend through body completion, error or cancellation on every supported path.
2. Explainable selection. Specify a scoring formula and comparison direction using recent/smoothed upstream latency, active attempts and effective configured/warmup weight. Define a bounded recent window or time decay, minimum samples, timeout handling and upstream failure penalties. Fast 5xx/transport failures must not make a node look attractive merely because they finish quickly. Keep application/client 4xx, authentication rejection, local queue/admission delay and downstream cancellation distinct from upstream failure.
3. Cold start and recovery. Define unknown/stale/invalid samples explicitly, a safe fallback to existing selection, and bounded exploration/recovery. Preserve configured weights, zero-weight semantics and warmup; avoid permanent starvation or oscillation after a slow period. Document whether latency includes upstream pool acquisition and how that affects interpretation.
4. Opt-in integration. Preserve existing configurations, custom load-balancer names and ordinary P2C behavior. Propose the smallest explicit activation mechanism, such as a separate SPI name or versioned opt-in settings; include a rollback path and optional shadow/observation mode. Reuse health, Canary partitions, sticky affinity and breaker admission before adaptive comparison. Sticky-bound requests must still contribute accurate attempt feedback even when they bypass adaptive selection.
5. Scoped lifecycle and configuration. Bound state by a documented upstream/workload identity, reusing existing selector/namespace and upstream identity contracts. Preserve history across equivalent discovery refreshes, distinguish replacement instances, and clean up removed rules/upstreams. Any public SPI additions must preserve external implementations and have safe defaults. Apply Admin validation, data-sync and gateway behavior consistently; update seed/upgrade metadata for every supported DB dialect if configuration metadata changes.

Do not require backend instrumentation for this MVP. Dubbo's provider-load feedback can be considered later as an optional, trusted, bounded-freshness signal. Missing CPU/load feedback is unknown, never an invented zero load. No external metrics fetch, Redis operation, blocking callback or unbounded allocation may be added to each selection.

Acceptance criteria

• A deterministic overlap test reproduces the shared-timestamp problem and proves that A and B retain their own elapsed times using a monotonic clock.
• Real local HTTP servers cover both Netty and WebClient: success, connection/header failure, delayed/failed body, timeout and cancellation. Initial, CURRENT retry and failover attempts use the correct target and reserve/release once, including singleton pools. An open body must not make its serving target appear idle.
• Known latency differences and increasing concurrency affect selection as documented; fast failures, insufficient samples and unknown/stale metrics do not produce an erroneously preferred target. Cold start, idle probing, gradual recovery and sustained load changes remain bounded.
• Disabled/shadow modes preserve routing decisions and response/body subscription behavior. Canary, sticky bindings, warmup, breaker probes and discovery churn retain their existing semantics. State remains bounded under concurrent updates and removal/re-addition.
• Versioned benchmarks compare current P2C, leastActive and weighted random against the proposal: equal/unequal node capacity, a slow node, a fast-failing node, burst traffic and recovery. Record offered load, successful throughput, p95/p99, error rate, fairness, gateway CPU/memory and selection overhead. Include regressions and distributions; use [Task][Performance] Establish reproducible benchmarks and track gateway optimizations #7650's benchmark infrastructure where applicable.
• Publish low-cardinality diagnostics for score components, sample age/count, fallback and exploration reasons. Exclude request IDs, raw URLs, credentials and unbounded tenant/application labels. Clearly state that observations are per gateway, not a cluster-wide hard load limit.

Scope limits and later work

WebSocket sessions, gRPC/Dubbo-specific feedback, LLM TTFT/TPOT scoring, GPU metrics and global in-flight coordination require separate protocol/workload contracts before support is advertised. AI candidate/model pools remain owned by #8046 and inference-aware routing evaluation by #8054. A latency score cannot substitute for model capability, tenant authorization, quality, cost or residency policy.

References

• Dubbo AdaptiveLoadBalance, AdaptiveMetrics, and consumer feedback filter.
• Higress AI AdaptiveScore is a related workload-specific design, not the name of Dubbo's adaptive implementation or a formula to copy unchanged.