Refactor DiscoveryService to handle changes to settings and agent upgrade#3154
Merged
Conversation
This comment was marked as resolved.
This comment was marked as resolved.
This comment has been minimized.
This comment has been minimized.
andrewlock
force-pushed
the
andrew/discovery-service
branch
from
September 2, 2022 14:22
ad2e672 to
8ec51b7
Compare
This comment was marked as resolved.
This comment was marked as resolved.
This comment has been minimized.
This comment has been minimized.
pierotibou
approved these changes
Sep 2, 2022
pierotibou
left a comment
Contributor
There was a problem hiding this comment.
:nice: Thanks a lot for doing it
| await oldManager.Telemetry.DisposeAsync(sendAppClosingTelemetry: false).ConfigureAwait(false); | ||
| } | ||
|
|
||
| var discoveryReplaced = false; |
Contributor
There was a problem hiding this comment.
:oops: I had completely forgotten about settings refresh when reviewing previous PRs. Thanks
shurivich
reviewed
Sep 4, 2022
shurivich
reviewed
Sep 4, 2022
shurivich
reviewed
Sep 4, 2022
shurivich
approved these changes
Sep 4, 2022
shurivich
left a comment
Contributor
There was a problem hiding this comment.
Looks good, just some small comments.
This comment was marked as outdated.
This comment was marked as outdated.
shurivich
reviewed
Sep 6, 2022
This comment was marked as outdated.
This comment was marked as outdated.
zacharycmontoya
force-pushed
the
zach/stats/discovery-update
branch
2 times, most recently
from
September 7, 2022 19:29
4a765a2 to
fa817bd
Compare
andrewlock
force-pushed
the
andrew/discovery-service
branch
3 times, most recently
from
September 8, 2022 10:16
f9db4c5 to
a5ccb9f
Compare
This was referenced Sep 8, 2022
andrewlock
force-pushed
the
andrew/discovery-service
branch
from
September 8, 2022 13:57
a5ccb9f to
ef1df6a
Compare
This comment has been minimized.
This comment has been minimized.
andrewlock
force-pushed
the
andrew/discovery-service
branch
from
September 9, 2022 07:22
ef1df6a to
ebc9dee
Compare
This comment has been minimized.
This comment has been minimized.
Remove the static instance so that the lifetime is managed by the TracerManagerFactory instead and can respond to changes in exporter settings
The live debugger and RCM manager could still do with improvements, and ideally should be moved to TracerManagerFactory to respond to changes in ExporterSettings, but that's not _required_ here.
- Unregister discoveryService callback on RCM shut down - Add try..catch to change notification
andrewlock
force-pushed
the
andrew/discovery-service
branch
from
September 9, 2022 08:40
ebc9dee to
4293941
Compare
Member
Author
Benchmarks Report 🐌Benchmarks for #3154 compared to master:
The following thresholds were used for comparing the benchmark speeds:
Allocation changes below 0.5% are ignored. Benchmark detailsBenchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.AppSecBodyBenchmark - Slower
|
| Benchmark | diff/base | Base Median (ns) | Diff Median (ns) | Modality |
|---|---|---|---|---|
| Benchmarks.Trace.AppSecBodyBenchmark.BodyExtractorSimpleBody‑netcoreapp3.1 | 1.122 | 219.81 | 246.67 |
Raw results
| Branch | Method | Toolchain | Mean | StdError | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
|---|---|---|---|---|---|---|---|---|---|
| master | AllCycleSimpleBody |
net472 | 206ns | 0.529ns | 2.05ns | 0.0675 | 0 | 0 | 425 B |
| master | AllCycleSimpleBody |
netcoreapp3.1 | 247ns | 0.433ns | 1.68ns | 0.00585 | 0 | 0 | 424 B |
| master | AllCycleMoreComplexBody |
net472 | 194ns | 0.135ns | 0.524ns | 0.0637 | 0 | 0 | 401 B |
| master | AllCycleMoreComplexBody |
netcoreapp3.1 | 232ns | 0.4ns | 1.55ns | 0.00545 | 0 | 0 | 400 B |
| master | BodyExtractorSimpleBody |
net472 | 259ns | 0.278ns | 1.04ns | 0.0573 | 0 | 0 | 361 B |
| master | BodyExtractorSimpleBody |
netcoreapp3.1 | 220ns | 0.253ns | 0.981ns | 0.00371 | 0 | 0 | 272 B |
| master | BodyExtractorMoreComplexBody |
net472 | 14.4μs | 9.84ns | 38.1ns | 1.2 | 0.0216 | 0 | 7.62 KB |
| master | BodyExtractorMoreComplexBody |
netcoreapp3.1 | 12.4μs | 13.4ns | 51.9ns | 0.0931 | 0 | 0 | 6.75 KB |
| #3154 | AllCycleSimpleBody |
net472 | 186ns | 0.145ns | 0.542ns | 0.0675 | 0 | 0 | 425 B |
| #3154 | AllCycleSimpleBody |
netcoreapp3.1 | 235ns | 0.356ns | 1.38ns | 0.0058 | 0 | 0 | 424 B |
| #3154 | AllCycleMoreComplexBody |
net472 | 185ns | 0.181ns | 0.677ns | 0.0637 | 0 | 0 | 401 B |
| #3154 | AllCycleMoreComplexBody |
netcoreapp3.1 | 233ns | 0.26ns | 0.972ns | 0.00549 | 0 | 0 | 400 B |
| #3154 | BodyExtractorSimpleBody |
net472 | 266ns | 0.653ns | 2.53ns | 0.0574 | 0 | 0 | 361 B |
| #3154 | BodyExtractorSimpleBody |
netcoreapp3.1 | 247ns | 0.289ns | 1.08ns | 0.00372 | 0 | 0 | 272 B |
| #3154 | BodyExtractorMoreComplexBody |
net472 | 14.6μs | 14.5ns | 54.4ns | 1.21 | 0.0218 | 0 | 7.62 KB |
| #3154 | BodyExtractorMoreComplexBody |
netcoreapp3.1 | 11.6μs | 10.3ns | 37.2ns | 0.0926 | 0 | 0 | 6.75 KB |
Benchmarks.Trace.AspNetCoreBenchmark - Unknown 🤷 Same allocations ✔️
Raw results
| Branch | Method | Toolchain | Mean | StdError | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
|---|---|---|---|---|---|---|---|---|---|
| master | SendRequest |
net472 | 0ns | 0ns | 0ns | 0 | 0 | 0 | 0 b |
| master | SendRequest |
netcoreapp3.1 | 183μs | 168ns | 651ns | 0.274 | 0 | 0 | 20.57 KB |
| #3154 | SendRequest |
net472 | 0ns | 0ns | 0ns | 0 | 0 | 0 | 0 b |
| #3154 | SendRequest |
netcoreapp3.1 | 184μs | 250ns | 968ns | 0.184 | 0 | 0 | 20.57 KB |
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️
Raw results
| Branch | Method | Toolchain | Mean | StdError | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
|---|---|---|---|---|---|---|---|---|---|
| master | ExecuteNonQuery |
net472 | 1.86μs | 0.749ns | 2.9ns | 0.15 | 0.000931 | 0 | 947 B |
| master | ExecuteNonQuery |
netcoreapp3.1 | 1.45μs | 0.576ns | 2.23ns | 0.0123 | 0 | 0 | 936 B |
| #3154 | ExecuteNonQuery |
net472 | 1.88μs | 1.31ns | 5.06ns | 0.15 | 0.000939 | 0 | 947 B |
| #3154 | ExecuteNonQuery |
netcoreapp3.1 | 1.38μs | 0.579ns | 2.24ns | 0.0124 | 0 | 0 | 936 B |
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️
Raw results
| Branch | Method | Toolchain | Mean | StdError | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
|---|---|---|---|---|---|---|---|---|---|
| master | CallElasticsearch |
net472 | 2.53μs | 0.543ns | 1.96ns | 0.183 | 0 | 0 | 1.16 KB |
| master | CallElasticsearch |
netcoreapp3.1 | 1.53μs | 0.352ns | 1.32ns | 0.0146 | 0 | 0 | 1.1 KB |
| master | CallElasticsearchAsync |
net472 | 2.66μs | 0.901ns | 3.37ns | 0.204 | 0 | 0 | 1.29 KB |
| master | CallElasticsearchAsync |
netcoreapp3.1 | 1.64μs | 0.414ns | 1.6ns | 0.0164 | 0 | 0 | 1.22 KB |
| #3154 | CallElasticsearch |
net472 | 2.62μs | 1.36ns | 5.1ns | 0.183 | 0 | 0 | 1.16 KB |
| #3154 | CallElasticsearch |
netcoreapp3.1 | 1.56μs | 0.657ns | 2.46ns | 0.0147 | 0 | 0 | 1.1 KB |
| #3154 | CallElasticsearchAsync |
net472 | 2.85μs | 1.6ns | 6ns | 0.205 | 0 | 0 | 1.29 KB |
| #3154 | CallElasticsearchAsync |
netcoreapp3.1 | 1.64μs | 0.643ns | 2.41ns | 0.0165 | 0 | 0 | 1.22 KB |
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️
Raw results
| Branch | Method | Toolchain | Mean | StdError | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
|---|---|---|---|---|---|---|---|---|---|
| master | ExecuteAsync |
net472 | 2.66μs | 4.92ns | 17.7ns | 0.225 | 0 | 0 | 1.42 KB |
| master | ExecuteAsync |
netcoreapp3.1 | 1.75μs | 2.52ns | 9.41ns | 0.0179 | 0 | 0 | 1.34 KB |
| #3154 | ExecuteAsync |
net472 | 2.77μs | 6.79ns | 26.3ns | 0.224 | 0 | 0 | 1.42 KB |
| #3154 | ExecuteAsync |
netcoreapp3.1 | 1.66μs | 3.73ns | 14.5ns | 0.018 | 0 | 0 | 1.34 KB |
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️
Raw results
| Branch | Method | Toolchain | Mean | StdError | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
|---|---|---|---|---|---|---|---|---|---|
| master | SendAsync |
net472 | 5.62μs | 11ns | 39.7ns | 0.439 | 0 | 0 | 2.77 KB |
| master | SendAsync |
netcoreapp3.1 | 3.55μs | 7.73ns | 29.9ns | 0.0349 | 0 | 0 | 2.6 KB |
| #3154 | SendAsync |
net472 | 5.61μs | 8.3ns | 31.1ns | 0.439 | 0 | 0 | 2.77 KB |
| #3154 | SendAsync |
netcoreapp3.1 | 3.66μs | 13.6ns | 52.7ns | 0.0361 | 0 | 0 | 2.6 KB |
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️
Raw results
| Branch | Method | Toolchain | Mean | StdError | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
|---|---|---|---|---|---|---|---|---|---|
| master | EnrichedLog |
net472 | 3.12μs | 2.16ns | 8.09ns | 0.287 | 0 | 0 | 1.81 KB |
| master | EnrichedLog |
netcoreapp3.1 | 2.55μs | 0.893ns | 3.34ns | 0.0254 | 0 | 0 | 1.85 KB |
| #3154 | EnrichedLog |
net472 | 3.08μs | 3.95ns | 15.3ns | 0.287 | 0 | 0 | 1.81 KB |
| #3154 | EnrichedLog |
netcoreapp3.1 | 2.43μs | 0.843ns | 3.15ns | 0.0243 | 0 | 0 | 1.85 KB |
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️
Raw results
| Branch | Method | Toolchain | Mean | StdError | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
|---|---|---|---|---|---|---|---|---|---|
| master | EnrichedLog |
net472 | 151μs | 187ns | 725ns | 0.688 | 0.229 | 0 | 4.65 KB |
| master | EnrichedLog |
netcoreapp3.1 | 114μs | 73.8ns | 276ns | 0.0567 | 0 | 0 | 4.49 KB |
| #3154 | EnrichedLog |
net472 | 150μs | 103ns | 397ns | 0.674 | 0.225 | 0 | 4.65 KB |
| #3154 | EnrichedLog |
netcoreapp3.1 | 117μs | 203ns | 759ns | 0.0591 | 0 | 0 | 4.49 KB |
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️
Raw results
| Branch | Method | Toolchain | Mean | StdError | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
|---|---|---|---|---|---|---|---|---|---|
| master | EnrichedLog |
net472 | 5.5μs | 8.16ns | 31.6ns | 0.568 | 0.00271 | 0 | 3.59 KB |
| master | EnrichedLog |
netcoreapp3.1 | 4.33μs | 4.34ns | 15ns | 0.0523 | 0 | 0 | 3.91 KB |
| #3154 | EnrichedLog |
net472 | 5.63μs | 15ns | 58.1ns | 0.57 | 0.00277 | 0 | 3.59 KB |
| #3154 | EnrichedLog |
netcoreapp3.1 | 4.29μs | 7.56ns | 29.3ns | 0.0535 | 0 | 0 | 3.91 KB |
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️
Raw results
| Branch | Method | Toolchain | Mean | StdError | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
|---|---|---|---|---|---|---|---|---|---|
| master | SendReceive |
net472 | 2.26μs | 1.05ns | 3.65ns | 0.218 | 0 | 0 | 1.37 KB |
| master | SendReceive |
netcoreapp3.1 | 1.77μs | 0.462ns | 1.73ns | 0.0177 | 0 | 0 | 1.32 KB |
| #3154 | SendReceive |
net472 | 2.33μs | 0.956ns | 3.58ns | 0.218 | 0 | 0 | 1.37 KB |
| #3154 | SendReceive |
netcoreapp3.1 | 1.77μs | 1.31ns | 4.91ns | 0.0178 | 0 | 0 | 1.32 KB |
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️
Raw results
| Branch | Method | Toolchain | Mean | StdError | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
|---|---|---|---|---|---|---|---|---|---|
| master | EnrichedLog |
net472 | 5.07μs | 1.54ns | 5.76ns | 0.353 | 0 | 0 | 2.23 KB |
| master | EnrichedLog |
netcoreapp3.1 | 4.22μs | 0.846ns | 3.17ns | 0.0234 | 0 | 0 | 1.8 KB |
| #3154 | EnrichedLog |
net472 | 5.23μs | 2.79ns | 10.8ns | 0.353 | 0 | 0 | 2.23 KB |
| #3154 | EnrichedLog |
netcoreapp3.1 | 4.26μs | 1.48ns | 5.54ns | 0.0232 | 0 | 0 | 1.8 KB |
Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️
Raw results
| Branch | Method | Toolchain | Mean | StdError | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
|---|---|---|---|---|---|---|---|---|---|
| master | StartFinishSpan |
net472 | 1.16μs | 0.359ns | 1.34ns | 0.129 | 0 | 0 | 810 B |
| master | StartFinishSpan |
netcoreapp3.1 | 926ns | 0.412ns | 1.54ns | 0.0102 | 0 | 0 | 760 B |
| master | StartFinishScope |
net472 | 1.35μs | 0.341ns | 1.32ns | 0.141 | 0 | 0 | 891 B |
| master | StartFinishScope |
netcoreapp3.1 | 1.14μs | 0.405ns | 1.4ns | 0.012 | 0 | 0 | 880 B |
| #3154 | StartFinishSpan |
net472 | 1.14μs | 0.429ns | 1.66ns | 0.129 | 0 | 0 | 810 B |
| #3154 | StartFinishSpan |
netcoreapp3.1 | 926ns | 0.449ns | 1.68ns | 0.0102 | 0 | 0 | 760 B |
| #3154 | StartFinishScope |
net472 | 1.41μs | 0.731ns | 2.83ns | 0.141 | 0 | 0 | 891 B |
| #3154 | StartFinishScope |
netcoreapp3.1 | 1.03μs | 0.409ns | 1.53ns | 0.0118 | 0 | 0 | 880 B |
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️
Raw results
| Branch | Method | Toolchain | Mean | StdError | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
|---|---|---|---|---|---|---|---|---|---|
| master | RunOnMethodBegin |
net472 | 1.52μs | 0.558ns | 2.09ns | 0.141 | 0 | 0 | 891 B |
| master | RunOnMethodBegin |
netcoreapp3.1 | 1.19μs | 0.577ns | 2.16ns | 0.0117 | 0 | 0 | 880 B |
| #3154 | RunOnMethodBegin |
net472 | 1.48μs | 1.05ns | 4.07ns | 0.142 | 0 | 0 | 891 B |
| #3154 | RunOnMethodBegin |
netcoreapp3.1 | 1.14μs | 0.512ns | 1.85ns | 0.012 | 0 | 0 | 880 B |
5 tasks
This was referenced Sep 15, 2022
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary of changes
DiscoveryService.Instancein favour of instance onTracerManagerDiscoveryServicewhen exporter settings change (i.e. config in code scenario)DiscoveryServiceto "subscription" model to handle dynamic changes in config.Reason for change
This refactor handles three main issues in previous implementation
Implementation details
IDiscoveryServiceis now implemented as:If you need config, you call
SubscribeToChanges()and pass in a delegate to be executed every time the config changes. If the intial config has already been fetched, this immediately executes the callback (I chose this approach over exposing the "current"AgentConfigurationonIDiscoveryServiceas it actually makes consuming config easier).Updated existing usages of
IDiscoveryServiceto the new pattern. This was simple for theStatsAggregatorbut a bit trickier for RCM and LiveDebugger, as they still need to be moved to TracerManagerFactory initialization (or similar). Nevertheless, I think this is an improvementTest coverage
All existing tests (e.g.
StatsTestsetc) + unit tests for theDiscoveryServicebehaviour itself. As part of this, I extracted theTestApiRequestFactoryhelpers I've used elsewhere (and which I need in another branch too).Other details
This is a requirement for the Data Streams Monitoring work. So as to not conflict with his changes, I've rebased this branch on top of @zacharycmontoya's PR and incorporated his changes: #3152