balance: Move endpoint state gauge into balancer by olix0r · Pull Request #2598 · linkerd/linkerd2-proxy

olix0r · 2023-12-29T06:32:03Z

There are a few problems:

Controller clients do not includer endpoint state metrics, as outbound stacks do.
Controller client balancer metrics are not currently scoped with a balancer_ prefix.
Endpoint state metrics should be an implementation detail of the balancer crate and not something that requires explicit instrumentation in the app.
The balancer endpoint gauges module should be in an independent crate so it can be tested and reused independently.

This change removes a test of the outbound stack, since we can no longer query the metrics registry in the same way.

The behavior has been manually confirmed:

:; linkerd diagnostics proxy-metrics -n emojivoto deploy/web | grep _state
control_identity_balancer_endpoints{endpoint_state="pending",addr="linkerd-identity-headless.linkerd.svc.cluster.local:8080"} 0
control_identity_balancer_endpoints{endpoint_state="ready",addr="linkerd-identity-headless.linkerd.svc.cluster.local:8080"} 0
control_destination_balancer_endpoints{endpoint_state="pending",addr="linkerd-dst-headless.linkerd.svc.cluster.local:8086"} 0
control_destination_balancer_endpoints{endpoint_state="ready",addr="linkerd-dst-headless.linkerd.svc.cluster.local:8086"} 1
control_policy_balancer_endpoints{endpoint_state="pending",addr="linkerd-policy.linkerd.svc.cluster.local:8090"} 0
control_policy_balancer_endpoints{endpoint_state="ready",addr="linkerd-policy.linkerd.svc.cluster.local:8090"} 1
outbound_http_balancer_endpoints{endpoint_state="pending",,backend_name="emoji-svc",backend_port="8080",backend_section_name=""} 0
outbound_http_balancer_endpoints{endpoint_state="pending",...,backend_name="voting-svc",backend_port="8080",backend_section_name=""} 0
outbound_http_balancer_endpoints{endpoint_state="ready",,backend_name="emoji-svc",backend_port="8080",backend_section_name=""} 1
outbound_http_balancer_endpoints{endpoint_state="ready",...,backend_name="voting-svc",backend_port="8080",backend_section_name=""} 1

codecov · 2023-12-29T06:43:00Z

Codecov Report

Merging #2598 (bf97d87) into main (7104542) will decrease coverage by 0.21%.
The diff coverage is 94.73%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2598      +/-   ##
==========================================
- Coverage   67.99%   67.79%   -0.21%     
==========================================
  Files         330      328       -2     
  Lines       14986    14963      -23     
==========================================
- Hits        10190    10144      -46     
- Misses       4796     4819      +23

Files	Coverage Δ
linkerd/app/core/src/control.rs	`91.57% <100.00%> (+0.18%)`	⬆️
linkerd/app/outbound/src/http/concrete.rs	`60.17% <ø> (ø)`
linkerd/app/outbound/src/http/concrete/balance.rs	`89.06% <ø> (-0.65%)`	⬇️
linkerd/app/outbound/src/metrics.rs	`64.06% <ø> (-29.88%)`	⬇️
linkerd/pool/p2c/src/lib.rs	`90.62% <ø> (ø)`
linkerd/proxy/balance/src/lib.rs	`97.36% <100.00%> (+2.63%)`	⬆️
linkerd/proxy/balance/gauge-endpoints/src/lib.rs	`93.87% <93.87%> (ø)`

... and 3 files with indirect coverage changes

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7104542...bf97d87. Read the comment docs.

There are a few problems: * Controller clients do not includer endpoint state metrics, as outbound stacks do. * Controller client balancer metrics are not currently scoped with a balancer_ prefix. * Endpoint state metrics should be an implementation detail of the balancer crate and not something that requires explicit instrumentation in the app. * The balancer endpoint gauges module should be in an independent crate so it can be tested and reused independently. This change removes a test of the outbound stack, since we can no longer query the metrics registry in the same way. The behavior has been manually confirmed: :; linkerd diagnostics proxy-metrics -n emojivoto deploy/web | grep _state control_identity_balancer_endpoints{endpoint_state="pending",addr="linkerd-identity-headless.linkerd.svc.cluster.local:8080"} 0 control_identity_balancer_endpoints{endpoint_state="ready",addr="linkerd-identity-headless.linkerd.svc.cluster.local:8080"} 0 control_destination_balancer_endpoints{endpoint_state="pending",addr="linkerd-dst-headless.linkerd.svc.cluster.local:8086"} 0 control_destination_balancer_endpoints{endpoint_state="ready",addr="linkerd-dst-headless.linkerd.svc.cluster.local:8086"} 1 control_policy_balancer_endpoints{endpoint_state="pending",addr="linkerd-policy.linkerd.svc.cluster.local:8090"} 0 control_policy_balancer_endpoints{endpoint_state="ready",addr="linkerd-policy.linkerd.svc.cluster.local:8090"} 1 outbound_http_balancer_endpoints{endpoint_state="pending",,backend_name="emoji-svc",backend_port="8080",backend_section_name=""} 0 outbound_http_balancer_endpoints{endpoint_state="pending",...,backend_name="voting-svc",backend_port="8080",backend_section_name=""} 0 outbound_http_balancer_endpoints{endpoint_state="ready",,backend_name="emoji-svc",backend_port="8080",backend_section_name=""} 1 outbound_http_balancer_endpoints{endpoint_state="ready",...,backend_name="voting-svc",backend_port="8080",backend_section_name=""} 1

* build(deps): bump itertools from 0.10.5 to 0.11.0 (linkerd/linkerd2-proxy#2594) * build(deps): bump async-trait from 0.1.68 to 0.1.75 (linkerd/linkerd2-proxy#2595) * pool: Decompose the pool and balancer crates (linkerd/linkerd2-proxy#2597) * balance: Move endpoint state gauge into balancer (linkerd/linkerd2-proxy#2598) * cargo: Remove cyclic meshtls dependency (linkerd/linkerd2-proxy#2602) * build(deps): bump mime from 0.3.16 to 0.3.17 (linkerd/linkerd2-proxy#2599) * build(deps): bump parking_lot_core from 0.9.5 to 0.9.9 (linkerd/linkerd2-proxy#2600) * build(deps): bump prost-build from 0.12.1 to 0.12.3 (linkerd/linkerd2-proxy#2601) * outbound: Update route backend metrics implementation (linkerd/linkerd2-proxy#2603) * deps: Update to indexmap v2 (linkerd/linkerd2-proxy#2604) * build(deps): bump actions/download-artifact from 3.0.2 to 4.1.0 (linkerd/linkerd2-proxy#2569) * deps: h2 v0.3.22 (linkerd/linkerd2-proxy#2605) * tracing: Ensure that INFO-level spans are preserved (linkerd/linkerd2-proxy#2611) * build(deps): bump serde from 1.0.185 to 1.0.193 (linkerd/linkerd2-proxy#2606) * build(deps): bump tokio-boring from 3.0.4 to 3.1.0 (linkerd/linkerd2-proxy#2607) * build(deps): bump deranged from 0.3.10 to 0.3.11 (linkerd/linkerd2-proxy#2608) * build(deps): bump axum from 0.6.11 to 0.6.20 (linkerd/linkerd2-proxy#2609) * build(deps): bump proc-macro2 from 1.0.69 to 1.0.74 (linkerd/linkerd2-proxy#2610) * build(deps): bump ahash from 0.8.6 to 0.8.7 (linkerd/linkerd2-proxy#2612) * build(deps): bump cc from 1.0.79 to 1.0.83 (linkerd/linkerd2-proxy#2613) * build(deps): bump scopeguard from 1.1.0 to 1.2.0 (linkerd/linkerd2-proxy#2614) * build(deps): bump io-lifetimes from 1.0.10 to 1.0.11 (linkerd/linkerd2-proxy#2616) * build(deps): bump pem from 3.0.2 to 3.0.3 (linkerd/linkerd2-proxy#2615) * build(deps): bump anyhow from 1.0.76 to 1.0.79 (linkerd/linkerd2-proxy#2619) * build(deps): bump socket2 from 0.4.9 to 0.5.5 (linkerd/linkerd2-proxy#2622) * build(deps): bump libfuzzer-sys from 0.4.6 to 0.4.7 (linkerd/linkerd2-proxy#2620) * build(deps): bump tempfile from 3.5.0 to 3.6.0 (linkerd/linkerd2-proxy#2621) * build(deps): bump ryu from 1.0.13 to 1.0.16 (linkerd/linkerd2-proxy#2623) * identity: Update metrics to follow OpenMetrics best practices (linkerd/linkerd2-proxy#2617) * build(deps): bump tokio from 1.34.0 to 1.35.1 (linkerd/linkerd2-proxy#2627) * build(deps): bump tracing from 0.1.37 to 0.1.40 (linkerd/linkerd2-proxy#2628) * build(deps): bump slab from 0.4.8 to 0.4.9 (linkerd/linkerd2-proxy#2629) * build(deps): bump unicode-bidi from 0.3.11 to 0.3.14 (linkerd/linkerd2-proxy#2630) * build(deps): bump tokio-stream from 0.1.12 to 0.1.14 (linkerd/linkerd2-proxy#2632) * build(deps): bump boring-sys from 3.0.4 to 3.1.0 (linkerd/linkerd2-proxy#2633) * build(deps): bump rcgen from 0.11.3 to 0.12.0 (linkerd/linkerd2-proxy#2635) * build(deps): bump trust-dns-resolver from 0.22.0 to 0.23.2 (linkerd/linkerd2-proxy#2631) * build(deps): bump memchr from 2.6.4 to 2.7.1 (linkerd/linkerd2-proxy#2637) * build(deps): bump pin-project from 1.0.12 to 1.1.3 (linkerd/linkerd2-proxy#2638) * build(deps): bump futures from 0.3.28 to 0.3.30 (linkerd/linkerd2-proxy#2639) * build(deps): bump rangemap from 1.3.0 to 1.4.0 (linkerd/linkerd2-proxy#2640) * build(deps): bump actions/download-artifact from 4.1.0 to 4.1.1 (linkerd/linkerd2-proxy#2636) * build(deps): bump thingbuf from 0.1.3 to 0.1.4 (linkerd/linkerd2-proxy#2642) * build(deps): bump rustix from 0.36.16 to 0.36.17 (linkerd/linkerd2-proxy#2643) * build(deps): bump httpdate from 1.0.2 to 1.0.3 (linkerd/linkerd2-proxy#2645) * build(deps): bump num_cpus from 1.15.0 to 1.16.0 (linkerd/linkerd2-proxy#2646) * Change inbound port check log level to debug. (linkerd/linkerd2-proxy#2625) * docs: Fix bad reference link (linkerd/linkerd2-proxy#2647) * identity: add spire identity client (linkerd/linkerd2-proxy#2580) * config:add spire client config (linkerd/linkerd2-proxy#2641) * discovery: consume server_name and UriLikeIdentity from proto (linkerd/linkerd2-proxy#2618) * build(deps): bump h2 from 0.3.22 to 0.3.24 (linkerd/linkerd2-proxy#2660) * build(deps): bump procfs from 0.15.1 to 0.16.0 (linkerd/linkerd2-proxy#2649) * build(deps): bump async-trait from 0.1.75 to 0.1.77 (linkerd/linkerd2-proxy#2650) * build(deps): bump semver from 1.0.20 to 1.0.21 (linkerd/linkerd2-proxy#2651) * build(deps): bump smallvec from 1.10.0 to 1.13.1 (linkerd/linkerd2-proxy#2661) * build(deps): bump either from 1.8.1 to 1.9.0 (linkerd/linkerd2-proxy#2652) * build(deps): bump actions/upload-artifact from 4.0.0 to 4.2.0 (linkerd/linkerd2-proxy#2658) * build(deps): bump shlex from 1.1.0 to 1.3.0 (linkerd/linkerd2-proxy#2664) * build(deps): bump DavidAnson/markdownlint-cli2-action (linkerd/linkerd2-proxy#2656) * build(deps): bump EmbarkStudios/cargo-deny-action from 1.5.5 to 1.5.10 (linkerd/linkerd2-proxy#2665) * build(deps): bump serde from 1.0.193 to 1.0.195 (linkerd/linkerd2-proxy#2670) * build(deps): bump clang-sys from 1.6.0 to 1.7.0 (linkerd/linkerd2-proxy#2668) * build(deps): bump zerocopy from 0.7.31 to 0.7.32 (linkerd/linkerd2-proxy#2666) * build(deps): bump unicode-ident from 1.0.6 to 1.0.12 (linkerd/linkerd2-proxy#2667) * build(deps): bump actions/upload-artifact from 4.2.0 to 4.3.0 (linkerd/linkerd2-proxy#2671) * build(deps): bump prettyplease from 0.2.15 to 0.2.16 (linkerd/linkerd2-proxy#2673) * build(deps): bump getrandom from 0.2.8 to 0.2.12 (linkerd/linkerd2-proxy#2674) * build(deps): bump which from 4.4.0 to 4.4.2 (linkerd/linkerd2-proxy#2675) * build(deps): bump sharded-slab from 0.1.4 to 0.1.7 (linkerd/linkerd2-proxy#2676) * build(deps): bump EmbarkStudios/cargo-deny-action from 1.5.10 to 1.5.11 (linkerd/linkerd2-proxy#2672) * build(deps): bump tj-actions/changed-files from 41.0.1 to 42.0.0 (linkerd/linkerd2-proxy#2657) Signed-off-by: Oliver Gould <[email protected]>

* proxy: v2.220.0 * build(deps): bump itertools from 0.10.5 to 0.11.0 (linkerd/linkerd2-proxy#2594) * build(deps): bump async-trait from 0.1.68 to 0.1.75 (linkerd/linkerd2-proxy#2595) * pool: Decompose the pool and balancer crates (linkerd/linkerd2-proxy#2597) * balance: Move endpoint state gauge into balancer (linkerd/linkerd2-proxy#2598) * cargo: Remove cyclic meshtls dependency (linkerd/linkerd2-proxy#2602) * build(deps): bump mime from 0.3.16 to 0.3.17 (linkerd/linkerd2-proxy#2599) * build(deps): bump parking_lot_core from 0.9.5 to 0.9.9 (linkerd/linkerd2-proxy#2600) * build(deps): bump prost-build from 0.12.1 to 0.12.3 (linkerd/linkerd2-proxy#2601) * outbound: Update route backend metrics implementation (linkerd/linkerd2-proxy#2603) * deps: Update to indexmap v2 (linkerd/linkerd2-proxy#2604) * build(deps): bump actions/download-artifact from 3.0.2 to 4.1.0 (linkerd/linkerd2-proxy#2569) * deps: h2 v0.3.22 (linkerd/linkerd2-proxy#2605) * tracing: Ensure that INFO-level spans are preserved (linkerd/linkerd2-proxy#2611) * build(deps): bump serde from 1.0.185 to 1.0.193 (linkerd/linkerd2-proxy#2606) * build(deps): bump tokio-boring from 3.0.4 to 3.1.0 (linkerd/linkerd2-proxy#2607) * build(deps): bump deranged from 0.3.10 to 0.3.11 (linkerd/linkerd2-proxy#2608) * build(deps): bump axum from 0.6.11 to 0.6.20 (linkerd/linkerd2-proxy#2609) * build(deps): bump proc-macro2 from 1.0.69 to 1.0.74 (linkerd/linkerd2-proxy#2610) * build(deps): bump ahash from 0.8.6 to 0.8.7 (linkerd/linkerd2-proxy#2612) * build(deps): bump cc from 1.0.79 to 1.0.83 (linkerd/linkerd2-proxy#2613) * build(deps): bump scopeguard from 1.1.0 to 1.2.0 (linkerd/linkerd2-proxy#2614) * build(deps): bump io-lifetimes from 1.0.10 to 1.0.11 (linkerd/linkerd2-proxy#2616) * build(deps): bump pem from 3.0.2 to 3.0.3 (linkerd/linkerd2-proxy#2615) * build(deps): bump anyhow from 1.0.76 to 1.0.79 (linkerd/linkerd2-proxy#2619) * build(deps): bump socket2 from 0.4.9 to 0.5.5 (linkerd/linkerd2-proxy#2622) * build(deps): bump libfuzzer-sys from 0.4.6 to 0.4.7 (linkerd/linkerd2-proxy#2620) * build(deps): bump tempfile from 3.5.0 to 3.6.0 (linkerd/linkerd2-proxy#2621) * build(deps): bump ryu from 1.0.13 to 1.0.16 (linkerd/linkerd2-proxy#2623) * identity: Update metrics to follow OpenMetrics best practices (linkerd/linkerd2-proxy#2617) * build(deps): bump tokio from 1.34.0 to 1.35.1 (linkerd/linkerd2-proxy#2627) * build(deps): bump tracing from 0.1.37 to 0.1.40 (linkerd/linkerd2-proxy#2628) * build(deps): bump slab from 0.4.8 to 0.4.9 (linkerd/linkerd2-proxy#2629) * build(deps): bump unicode-bidi from 0.3.11 to 0.3.14 (linkerd/linkerd2-proxy#2630) * build(deps): bump tokio-stream from 0.1.12 to 0.1.14 (linkerd/linkerd2-proxy#2632) * build(deps): bump boring-sys from 3.0.4 to 3.1.0 (linkerd/linkerd2-proxy#2633) * build(deps): bump rcgen from 0.11.3 to 0.12.0 (linkerd/linkerd2-proxy#2635) * build(deps): bump trust-dns-resolver from 0.22.0 to 0.23.2 (linkerd/linkerd2-proxy#2631) * build(deps): bump memchr from 2.6.4 to 2.7.1 (linkerd/linkerd2-proxy#2637) * build(deps): bump pin-project from 1.0.12 to 1.1.3 (linkerd/linkerd2-proxy#2638) * build(deps): bump futures from 0.3.28 to 0.3.30 (linkerd/linkerd2-proxy#2639) * build(deps): bump rangemap from 1.3.0 to 1.4.0 (linkerd/linkerd2-proxy#2640) * build(deps): bump actions/download-artifact from 4.1.0 to 4.1.1 (linkerd/linkerd2-proxy#2636) * build(deps): bump thingbuf from 0.1.3 to 0.1.4 (linkerd/linkerd2-proxy#2642) * build(deps): bump rustix from 0.36.16 to 0.36.17 (linkerd/linkerd2-proxy#2643) * build(deps): bump httpdate from 1.0.2 to 1.0.3 (linkerd/linkerd2-proxy#2645) * build(deps): bump num_cpus from 1.15.0 to 1.16.0 (linkerd/linkerd2-proxy#2646) * Change inbound port check log level to debug. (linkerd/linkerd2-proxy#2625) * docs: Fix bad reference link (linkerd/linkerd2-proxy#2647) * identity: add spire identity client (linkerd/linkerd2-proxy#2580) * config:add spire client config (linkerd/linkerd2-proxy#2641) * discovery: consume server_name and UriLikeIdentity from proto (linkerd/linkerd2-proxy#2618) * build(deps): bump h2 from 0.3.22 to 0.3.24 (linkerd/linkerd2-proxy#2660) * build(deps): bump procfs from 0.15.1 to 0.16.0 (linkerd/linkerd2-proxy#2649) * build(deps): bump async-trait from 0.1.75 to 0.1.77 (linkerd/linkerd2-proxy#2650) * build(deps): bump semver from 1.0.20 to 1.0.21 (linkerd/linkerd2-proxy#2651) * build(deps): bump smallvec from 1.10.0 to 1.13.1 (linkerd/linkerd2-proxy#2661) * build(deps): bump either from 1.8.1 to 1.9.0 (linkerd/linkerd2-proxy#2652) * build(deps): bump actions/upload-artifact from 4.0.0 to 4.2.0 (linkerd/linkerd2-proxy#2658) * build(deps): bump shlex from 1.1.0 to 1.3.0 (linkerd/linkerd2-proxy#2664) * build(deps): bump DavidAnson/markdownlint-cli2-action (linkerd/linkerd2-proxy#2656) * build(deps): bump EmbarkStudios/cargo-deny-action from 1.5.5 to 1.5.10 (linkerd/linkerd2-proxy#2665) * build(deps): bump serde from 1.0.193 to 1.0.195 (linkerd/linkerd2-proxy#2670) * build(deps): bump clang-sys from 1.6.0 to 1.7.0 (linkerd/linkerd2-proxy#2668) * build(deps): bump zerocopy from 0.7.31 to 0.7.32 (linkerd/linkerd2-proxy#2666) * build(deps): bump unicode-ident from 1.0.6 to 1.0.12 (linkerd/linkerd2-proxy#2667) * build(deps): bump actions/upload-artifact from 4.2.0 to 4.3.0 (linkerd/linkerd2-proxy#2671) * build(deps): bump prettyplease from 0.2.15 to 0.2.16 (linkerd/linkerd2-proxy#2673) * build(deps): bump getrandom from 0.2.8 to 0.2.12 (linkerd/linkerd2-proxy#2674) * build(deps): bump which from 4.4.0 to 4.4.2 (linkerd/linkerd2-proxy#2675) * build(deps): bump sharded-slab from 0.1.4 to 0.1.7 (linkerd/linkerd2-proxy#2676) * build(deps): bump EmbarkStudios/cargo-deny-action from 1.5.10 to 1.5.11 (linkerd/linkerd2-proxy#2672) * build(deps): bump tj-actions/changed-files from 41.0.1 to 42.0.0 (linkerd/linkerd2-proxy#2657) Signed-off-by: Oliver Gould <[email protected]> * Bump helm version * +changes * Update CHANGES.md Co-authored-by: Alejandro Pedraza <[email protected]> --------- Signed-off-by: Oliver Gould <[email protected]> Co-authored-by: Alejandro Pedraza <[email protected]>

olix0r requested a review from a team as a code owner December 29, 2023 06:32

olix0r force-pushed the ver/endpoint-gauge branch from 7af6a0e to bf97d87 Compare December 29, 2023 15:38

olix0r merged commit d6d3c18 into main Dec 29, 2023

olix0r deleted the ver/endpoint-gauge branch December 29, 2023 15:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

balance: Move endpoint state gauge into balancer#2598

balance: Move endpoint state gauge into balancer#2598
olix0r merged 1 commit intomainfrom
ver/endpoint-gauge

olix0r commented Dec 29, 2023

Uh oh!

codecov bot commented Dec 29, 2023 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

olix0r commented Dec 29, 2023

Uh oh!

codecov bot commented Dec 29, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

codecov bot commented Dec 29, 2023 •

edited

Loading