Add the PoolQueue middleware by olix0r · Pull Request #2540 · linkerd/linkerd2-proxy

olix0r · 2023-12-01T01:25:33Z

Tower's Buffer middleware is used to make a Service shareable across tasks. Within Linkerd, we've augmented the Buffer with failfast behavior, etc, as a Queue middleware. This queue plays a vital role in dispatching requests to load balancers, ensuring that load is shed when a balancer has no available endpoints.

The tower Balancer additionally holds a Stream of endpoint updates that are processed as the balancer is driven to readiness. Crucially, this means that updates to the balancer can only occur while requests are being processed. We cannot eagerly drop defunct endpoints, nor can we eagerly connect to new endpoints. We have observed situations where long-lived, mostly-idle balancers can buffer discovery updates indefinitely, bloating memory and even forcing backpressure to the control plane.

To correct this behavior, this change introduces the PoolQueue middleware. The PoolQueue is based on the tower's Buffer, but instead of composing over an inner Service, it composes over an inner Pool.

Pool is a new interface that provides an additional interface to update the pool's members and to drive all pending endpoints in the pool to be ready (decoupling the semantics of Service::poll_ready and Pool::poll_pool). Pool implementations will typically hold a ReadyCache of inner endpoints (as the tower balancer does). This change, however, does not include the concrete implementation of a Pool to be replace the balancer. A p2c pool will be introduced in a followup change.

This change has the added benefit of simplifying the endpoint discovery pipeline. We currently process Updates (including an array of endpoints) from the discovery API and convert that into a stream of discrete endpoint updates for the balancer, requiring redundant caching. The Pool interface processes Updates directly, so there is no need for the extra translation.

This basic implementation does not yet include metrics, though it includes copious diagnostic logging.

Tower's Buffer middleware is used to make a Service shareable across tasks. Within Linkerd, we've augmented the Buffer with failfast behavior, etc, as a Queue middleware. This queue plays a vital role in dispatching requests to load balancers, ensuring that load is shed when a balancer has no available endpoints. The tower Balancer additionally holds a Stream of endpoint updates that are processed as the balancer is driven to readiness. Crucially, this means that updates to the balancer can only occur while requests are being processed. We cannot eagerly drop defunct endpoints, nor can we eagerly connect to new endpoints. We have observed situations where long-lived, mostly-idle balancers can buffer discovery updates indefinitely, bloating memory and even forcing backpressure to the control plane. To correct this behavior, this change introduces the PoolQueue middleware. The PoolQueue is based on the tower's Buffer, but instead of composing over an inner Service, it composes over an inner Pool. Pool is a new interface that provides an additional interface to update the pool's members and to drive all pending endpoints in the pool to be ready (decoupling the semantics of Service::poll_ready and Pool::poll_pool). Pool implementations will typically hold a ReadyCache of inner endpoints (as the tower balancer does). This change, however, does not include the concrete implementation of a Pool to be replace the balancer. A p2c pool will be introduced in a followup change. This change has the added benefit of simplifying the endpoint discovery pipeline. We currently process Updates (including an array of endpoints) from the discovery API and convert that into a stream of discrete endpoint updates for the balancer, requiring redundant caching. The Pool interface processes Updates directly, so there is no need for the extra translation.

codecov · 2023-12-01T01:47:13Z

Codecov Report

Attention: 29 lines in your changes are missing coverage. Please review.

Comparison is base (e0ffbc7) 68.00% compared to head (2ea8cfa) 68.24%.

Files	Patch %	Lines
linkerd/proxy/pool/src/worker.rs	86.08%	16 Missing ⚠️
linkerd/proxy/pool/src/service.rs	77.77%	6 Missing ⚠️
linkerd/proxy/pool/src/future.rs	71.42%	4 Missing ⚠️
linkerd/proxy/pool/src/error.rs	66.66%	2 Missing ⚠️
linkerd/proxy/pool/src/message.rs	90.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2540      +/-   ##
==========================================
+ Coverage   68.00%   68.24%   +0.23%     
==========================================
  Files         338      344       +6     
  Lines       15975    16171     +196     
==========================================
+ Hits        10864    11036     +172     
- Misses       5111     5135      +24

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Following #2540, which introduces a new PoolQueue and Pool interface, this change introduces a P2cPool implementation that replaces Tower's p2c balancer (using the same underlying ReadyCache and p2c implementations). This balancer implementation is currently unused. It will be integrated in a follow-up change.

hawkw

This looks great! I had some fairly minor comments but didn't notice any significant concerns.

linkerd/proxy/pool/src/error.rs

linkerd/proxy/pool/src/future.rs

linkerd/proxy/pool/src/lib.rs

linkerd/proxy/pool/src/service.rs

linkerd/proxy/pool/src/worker.rs

Remove the error::Closed variant, since it indicates that the inner task panicked. It seems preferable to propagate the panic than to handle it gracefully.

Following #2540, which introduces a new PoolQueue and Pool interface, this change introduces a P2cPool implementation that replaces Tower's p2c balancer (using the same underlying ReadyCache and p2c implementations). This balancer implementation is currently unused. It will be integrated in a follow-up change.

This change culminates recent work to restructure the balancer to use a PoolQueue so that balancer changes may occur independently of request processing. This replaces independent discovery buffering so that the balancer task is responsible for polling discovery streams without independent buffering. Requests are buffered and processed as soon as the pool has available backends. Fail-fast circuit breaking is enforced on the balancer's queue so that requests can't get stuck in a queue indefinitely. In general, the new balancer is instrumented directly with metrics, and the relevant metric name prefix and labelset is provided by the stack. In addition to detailed queue metrics including request (in-queue) latency histograms, but also failfast states, discovery updates counts, and balancer endpoint pool sizes. --- * outbound: Move queues into the concrete stack (linkerd/linkerd2-proxy#2539) * metrics: Remove unused features (linkerd/linkerd2-proxy#2542) * Add the PoolQueue middleware (linkerd/linkerd2-proxy#2540) * ci: Fixup codecov config (linkerd/linkerd2-proxy#2545) * ci: Cancel prior runs (linkerd/linkerd2-proxy#2546) * ci: Skip ARM builds during non-release CI (linkerd/linkerd2-proxy#2547) * deps: Update tokio, tonic, and prost (linkerd/linkerd2-proxy#2544) * build(deps): bump tj-actions/changed-files from 40.2.0 to 40.2.1 (linkerd/linkerd2-proxy#2549) * metrics: Use prometheus-client for proxy_build_info (linkerd/linkerd2-proxy#2551) * balance: Add a p2c Pool implementation (linkerd/linkerd2-proxy#2541) * metrics: Export process metrics using prometheus-client (linkerd/linkerd2-proxy#2552) * linkerd_identity: split `linkerd_identity::Id` into DNS and URI variants (linkerd/linkerd2-proxy#2538) * outbound: Move HTTP balancer into its own module (linkerd/linkerd2-proxy#2554) * app: Setup prom registry for use in balancers (linkerd/linkerd2-proxy#2555) * vscode: Move workspace settings to devcontainer (linkerd/linkerd2-proxy#2557) * build(deps): bump tj-actions/changed-files from 40.2.1 to 40.2.2 (linkerd/linkerd2-proxy#2556) * balance: Instrument metrics in pool balancer (linkerd/linkerd2-proxy#2558) * Enable PoolQueue balancer (linkerd/linkerd2-proxy#2559) Signed-off-by: Oliver Gould <[email protected]>

olix0r requested a review from a team as a code owner December 1, 2023 01:25

olix0r force-pushed the ver/poolq branch from 9526c18 to 2ddf01e Compare December 1, 2023 01:27

lints

55dc059

olix0r mentioned this pull request Dec 1, 2023

balance: Add a p2c Pool implementation #2541

Merged

thiserror

220a7fe

hawkw approved these changes Dec 1, 2023

View reviewed changes

olix0r added 11 commits December 1, 2023 21:08

Merge branch 'main' into ver/poolq

32d7322

comments

9be44ab

use a rwlock for shared errors

79dd7b4

fixup tracing, comments

4c7c4c1

comment

16caec9

log pool errors

9becd32

Move error handling and request draining into a dedicated helper

9cf23b5

Assert that errors are propagated

f57eb03

Remove the error::Closed variant, since it indicates that the inner task panicked. It seems preferable to propagate the panic than to handle it gracefully.

Improve testing for error cases

a061326

commentary/clarity

a8ee87c

fixup mock semantics

2ea8cfa

olix0r merged commit 1b2039d into main Dec 2, 2023

olix0r deleted the ver/poolq branch December 2, 2023 19:48

olix0r mentioned this pull request Dec 13, 2023

proxy: v2.215.0 linkerd/linkerd2#11750

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add the PoolQueue middleware#2540

Add the PoolQueue middleware#2540
olix0r merged 14 commits intomainfrom
ver/poolq

olix0r commented Dec 1, 2023 •

edited

Loading

Uh oh!

codecov bot commented Dec 1, 2023 •

edited

Loading

Uh oh!

hawkw left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

olix0r commented Dec 1, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Dec 1, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

hawkw left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

olix0r commented Dec 1, 2023 •

edited

Loading

codecov bot commented Dec 1, 2023 •

edited

Loading