trace: update tracing-subscriber dependency to 0.2.1#426
Merged
Conversation
This release makes a number of significant performance improvements when filtering events emitted through the `log` compatibility layer (such as from `hyper` and `h2`). See tokio-rs/tracing#586 for details. Since these performance improvements apply to cases where a filter _disables_ an event, as well as to enabling events, this should improve the proxy's performance with the default log configuration, especially given that `h2` and `hyper` emit a _lot_ of `trace`-level logs in hot paths. Ignoring those logs should now have a much lower overhead. I'm going to do some benchmarking to quantify the performance improvement from this change, so I'll add benchmark results when my tests complete. In addition, 0.2.1 fixes a bug where `EnvFilter`'s filter selection differed from the `env_logger` behavior it was intended to emulate. If the most specific filter directive that applies to a given event did not enable that event's level, filter selection would continue to try decreasingly-specific filters, and would only disable the event if **no** filters enabled it (see tokio-rs/tracing#512). In practice, this means that if you set a filter like ``` LINKERD2_PROXY_LOG=warn,linkerd=debug,linkerd2_metrics=info ``` in an attempt to disable the `debug`-level events in the `linkerd2_metrics` crate, they would still be enabled by the `linkerd=debug` directive. This is now fixed. Signed-off-by: Eliza Weisman <[email protected]>
Contributor
Author
|
preliminary latency benchmarks: |
olix0r
approved these changes
Feb 17, 2020
seanmonstar
approved these changes
Feb 17, 2020
olix0r
added a commit
to linkerd/linkerd2
that referenced
this pull request
Feb 19, 2020
This release includes the results from continued profiling & performance analysis. In addition to modifying internals to prevent unwarranted memory growth, we've introduced new metrics to aid in debugging and diagnostics: a new `request_errors_total` metric exposes the number of requests that receive synthesized responses due to proxy errors; and a suite of `stack_*` metrics expose proxy internals that can help us identify unexpected behavior. --- * trace: update `tracing-subscriber` dependency to 0.2.1 (linkerd/linkerd2-proxy#426) * Reimplement the Lock middleware with tokio::sync (linkerd/linkerd2-proxy#427) * Add the request_errors_total metric (linkerd/linkerd2-proxy#417) * Expose the number of service instances in the proxy (linkerd/linkerd2-proxy#428) * concurrency-limit: Share a limit across Services (linkerd/linkerd2-proxy#429) * profiling: add benchmark and profiling scripts (linkerd/linkerd2-proxy#406) * http-box: Box HTTP payloads via middleware (linkerd/linkerd2-proxy#430) * lock: Generalize to protect a guarded value (linkerd/linkerd2-proxy#431)
olix0r
added a commit
to linkerd/linkerd2
that referenced
this pull request
Feb 19, 2020
This release includes the results from continued profiling & performance analysis. In addition to modifying internals to prevent unwarranted memory growth, we've introduced new metrics to aid in debugging and diagnostics: a new `request_errors_total` metric exposes the number of requests that receive synthesized responses due to proxy errors; and a suite of `stack_*` metrics expose proxy internals that can help us identify unexpected behavior. --- * trace: update `tracing-subscriber` dependency to 0.2.1 (linkerd/linkerd2-proxy#426) * Reimplement the Lock middleware with tokio::sync (linkerd/linkerd2-proxy#427) * Add the request_errors_total metric (linkerd/linkerd2-proxy#417) * Expose the number of service instances in the proxy (linkerd/linkerd2-proxy#428) * concurrency-limit: Share a limit across Services (linkerd/linkerd2-proxy#429) * profiling: add benchmark and profiling scripts (linkerd/linkerd2-proxy#406) * http-box: Box HTTP payloads via middleware (linkerd/linkerd2-proxy#430) * lock: Generalize to protect a guarded value (linkerd/linkerd2-proxy#431)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This release makes a number of significant performance improvements when
filtering events emitted through the
logcompatibility layer (such asfrom
hyperandh2). See tokio-rs/tracing#586 for details.Since these performance improvements apply to cases where a filter
disables an event, as well as to enabling events, this should improve
the proxy's performance with the default log configuration, especially
given that
h2andhyperemit a lot oftrace-level logs in hotpaths. Ignoring those logs should now have a much lower overhead.
In addition, 0.2.1 fixes a bug where
EnvFilter's filter selectiondiffered from the
env_loggerbehavior it was intended to emulate. Ifthe most specific filter directive that applies to a given event did not
enable that event's level, filter selection would continue to try
decreasingly-specific filters, and would only disable the event if
no filters enabled it (see tokio-rs/tracing#512).
In practice, this means that if you set a filter like
in an attempt to disable the
debug-level events in thelinkerd2_metricscrate, they would still be enabled by thelinkerd=debugdirective.This is now fixed.
Signed-off-by: Eliza Weisman [email protected]