Skip to content

Commit 829872e

Browse files
rahul2393rahul2393
authored
feat(grpc-gcp): penalize retryable channel errors (#14219)
With dynamic scaling enabled, a channel returning retryable errors (UNAVAILABLE / RESOURCE_EXHAUSTED) kept its low active-stream count and so kept attracting picks: the pool had no signal that a channel was unhealthy until its streams actually piled up, and scale-up never triggered for a pool that was failing rather than busy. **Change** - **Error penalty on retryable failures.** A call completing with UNAVAILABLE or RESOURCE_EXHAUSTED adds `errorPenaltyStep` (new pool option, default 5) of synthetic load to that channel, capped at `maxRpcPerChannel`. The penalty **decays linearly to zero** over `errorPenaltyDuration` (default 5s), so a recovered channel re-enters picking gradually instead of at a cliff; a new retryable error restores the accumulated penalty and resets the window. Picker reads are lock-free and use a single clock snapshot per pick. - **Scale-up integration.** Penalty load counts toward the dynamic scale-up trigger with a strict capacity comparison, and applying a penalty signals the scale-up check eagerly — so a pool that is failing-but-idle still grows (bounded: one channel at a time). - **Single net-delta aggregate accounting.** Every update to the shared penalty aggregate is computed under the channel lock and applied as one atomic delta, so lock-free readers never observe an intermediate state; a channel leaving the pool clears its contribution. The aggregate deliberately holds the undecayed value until expiry as a conservative scale-up bound. - **Configuration.** `setErrorPenaltyStep(0)` disables penalties (negative rejected); `errorPenaltyDuration` must be strictly positive.
1 parent 34d3803 commit 829872e

4 files changed

Lines changed: 871 additions & 35 deletions

File tree

0 commit comments

Comments
 (0)