Merged
Conversation
Currently, the `serve` function spawns tasks `instrument`ed with a
`tracing` span for each accepted connection. This span is at the `DEBUG`
level, so it's disabled with the default tracing configuration. This is
fine, as the per-connection context is relatively verbose and is mainly
useful for debugging.
However, we also rely on this span for propagating the `INFO`-level
spans which indicate which part of the proxy an event occurred in
(inbound, outbound, etc). When the `DEBUG` span is enabled, it will be a
child of these spans, so they are propagated to the spawned tasks.
However, when the `DEBUG` span is _not_ enabled, nothing propagates the
`INFO` spans. Since the default `tracing` configuration enables `INFO`
but not `DEBUG`, we want those spans to be propagated to the tasks
spawned in `serve`.
This commit fixes the missing spans by moving the spawn inside of the
`Span::in_scope` call, and using `in_current_span` rather than
`instrument`. Now, if the per-connection `DEBUG` span is enabled, it
will be the current span...but if it isn't, the `INFO` span will still
be current, so the task will still have the `INFO` span as part of its
span context regardless.
Alternatively, we could have fixed this by changing the `instrument()`
call to:
```rust
.instrument(span)
.in_current_span()
```
so that the task is always spawned in both the `DEBUG` span _and_ the
current span. However, this is a bit less efficient, as it wraps the
tasks in both spans even when the `INFO` span is not needed, so every
time the task is polled, we would enter both spans.
olix0r
reviewed
Sep 4, 2021
olix0r
approved these changes
Sep 4, 2021
Contributor
Author
|
upstream PR for a small API addition that should help avoid this in the future: tokio-rs/tracing#1538 |
olix0r
added a commit
to linkerd/linkerd2
that referenced
this pull request
Sep 9, 2021
This release improves error handling so that HTTP metrics include 5XX responses for common errors. Logging has also been improved to ensure `inbound` and `outbound` spans are always present in log messages. Outbound tap has been fixed to include route labels when service profiles are configured. --- * Set tracing spans on policy client (linkerd/linkerd2-proxy#1241) * build(deps): bump tokio-util from 0.6.7 to 0.6.8 (linkerd/linkerd2-proxy#1240) * core: fix missing spans in `serve` tasks (linkerd/linkerd2-proxy#1243) * build(deps): bump thiserror from 1.0.28 to 1.0.29 (linkerd/linkerd2-proxy#1244) * orig-proto: Avoid emiting HTTP/2 errors for upgraded requests (linkerd/linkerd2-proxy#1245) * Fix route labels on outbound tap metadata (linkerd/linkerd2-proxy#1247) * errors: Support contextual error handling strategies (linkerd/linkerd2-proxy#1246)
olix0r
added a commit
to linkerd/linkerd2
that referenced
this pull request
Sep 9, 2021
This release improves error handling so that HTTP metrics include 5XX responses for common errors. Logging has also been improved to ensure `inbound` and `outbound` spans are always present in log messages. Outbound tap has been fixed to include route labels when service profiles are configured. --- * Set tracing spans on policy client (linkerd/linkerd2-proxy#1241) * build(deps): bump tokio-util from 0.6.7 to 0.6.8 (linkerd/linkerd2-proxy#1240) * core: fix missing spans in `serve` tasks (linkerd/linkerd2-proxy#1243) * build(deps): bump thiserror from 1.0.28 to 1.0.29 (linkerd/linkerd2-proxy#1244) * orig-proto: Avoid emiting HTTP/2 errors for upgraded requests (linkerd/linkerd2-proxy#1245) * Fix route labels on outbound tap metadata (linkerd/linkerd2-proxy#1247) * errors: Support contextual error handling strategies (linkerd/linkerd2-proxy#1246)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Currently, the
servefunction spawns tasksinstrumented with atracingspan for each accepted connection. This span is at theDEBUGlevel, so it's disabled with the default tracing configuration. This is
fine, as the per-connection context is relatively verbose and is mainly
useful for debugging.
However, we also rely on this span for propagating the
INFO-levelspans which indicate which part of the proxy an event occurred in
(inbound, outbound, etc). When the
DEBUGspan is enabled, it will be achild of these spans, so they are propagated to the spawned tasks.
However, when the
DEBUGspan is not enabled, nothing propagates theINFOspans. Since the defaulttracingconfiguration enablesINFObut not
DEBUG, we want those spans to be propagated to the tasksspawned in
serve.This commit fixes the missing spans by moving the spawn inside of the
Span::in_scopecall, and usingin_current_spanrather thaninstrument. Now, if the per-connectionDEBUGspan is enabled, itwill be the current span...but if it isn't, the
INFOspan will stillbe current, so the task will still have the
INFOspan as part of itsspan context regardless.
Alternatively, we could have fixed this by changing the
instrument()call to:
so that the task is always spawned in both the
DEBUGspan and thecurrent span. However, this is a bit less efficient, as it wraps the
tasks in both spans even when the
INFOspan is not needed, so everytime the task is polled, we would enter both spans.