Skip to content

Refactor DiscoveryService to handle changes to settings and agent upgrade#3154

Merged
andrewlock merged 7 commits into
masterfrom
andrew/discovery-service
Sep 9, 2022
Merged

Refactor DiscoveryService to handle changes to settings and agent upgrade#3154
andrewlock merged 7 commits into
masterfrom
andrew/discovery-service

Conversation

@andrewlock

Copy link
Copy Markdown
Member

Summary of changes

  • Removed static DiscoveryService.Instance in favour of instance on TracerManager
  • Handle recreation and tear down of DiscoveryService when exporter settings change (i.e. config in code scenario)
  • Redesign DiscoveryService to "subscription" model to handle dynamic changes in config.

Reason for change

This refactor handles three main issues in previous implementation

  • The previous implementation would not honour config in code.
  • Changes to the agent (e.g. it's upgraded while app is running) would not be honoured
  • Harder to test in integration tests due to static state

Implementation details

IDiscoveryService is now implemented as:

internal interface IDiscoveryService
{
    void SubscribeToChanges(Action<AgentConfiguration> callback);
    void RemoveSubscription(Action<AgentConfiguration> callback);
    Task DisposeAsync();
}

If you need config, you call SubscribeToChanges() and pass in a delegate to be executed every time the config changes. If the intial config has already been fetched, this immediately executes the callback (I chose this approach over exposing the "current" AgentConfiguration on IDiscoveryService as it actually makes consuming config easier).

Updated existing usages of IDiscoveryService to the new pattern. This was simple for the StatsAggregator but a bit trickier for RCM and LiveDebugger, as they still need to be moved to TracerManagerFactory initialization (or similar). Nevertheless, I think this is an improvement

Test coverage

All existing tests (e.g. StatsTests etc) + unit tests for the DiscoveryService behaviour itself. As part of this, I extracted the TestApiRequestFactory helpers I've used elsewhere (and which I need in another branch too).

Other details

This is a requirement for the Data Streams Monitoring work. So as to not conflict with his changes, I've rebased this branch on top of @zacharycmontoya's PR and incorporated his changes: #3152

@andrewlock
andrewlock requested review from a team as code owners September 2, 2022 11:43
@github-actions

This comment was marked as resolved.

@andrewlock

This comment has been minimized.

@github-actions

This comment was marked as resolved.

@andrewlock

This comment has been minimized.

@pierotibou pierotibou left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:nice: Thanks a lot for doing it

await oldManager.Telemetry.DisposeAsync(sendAppClosingTelemetry: false).ConfigureAwait(false);
}

var discoveryReplaced = false;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:oops: I had completely forgotten about settings refresh when reviewing previous PRs. Thanks

Comment thread tracer/src/Datadog.Trace/Agent/DiscoveryService/DiscoveryService.cs
Comment thread tracer/src/Datadog.Trace/Agent/DiscoveryService/DiscoveryService.cs Outdated
Comment thread tracer/src/Datadog.Trace/Agent/DiscoveryService/DiscoveryService.cs Outdated

@shurivich shurivich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, just some small comments.

@github-actions

This comment was marked as outdated.

Comment thread tracer/src/Datadog.Trace/Agent/DiscoveryService/DiscoveryService.cs Outdated
@github-actions

This comment was marked as outdated.

@zacharycmontoya
zacharycmontoya force-pushed the zach/stats/discovery-update branch 2 times, most recently from 4a765a2 to fa817bd Compare September 7, 2022 19:29
@andrewlock
andrewlock force-pushed the andrew/discovery-service branch 3 times, most recently from f9db4c5 to a5ccb9f Compare September 8, 2022 10:16
@andrewlock
andrewlock force-pushed the andrew/discovery-service branch from a5ccb9f to ef1df6a Compare September 8, 2022 13:57
@andrewlock

This comment has been minimized.

Base automatically changed from zach/stats/discovery-update to master September 8, 2022 18:01
@andrewlock
andrewlock force-pushed the andrew/discovery-service branch from ef1df6a to ebc9dee Compare September 9, 2022 07:22
@andrewlock

This comment has been minimized.

Remove the static instance so that the lifetime is managed by the TracerManagerFactory instead and can respond to changes in exporter settings
The live debugger and RCM manager could still do with improvements, and ideally should be moved to TracerManagerFactory to respond to changes in ExporterSettings, but that's not _required_ here.
- Unregister discoveryService callback on RCM shut down
- Add try..catch to change notification
@andrewlock
andrewlock force-pushed the andrew/discovery-service branch from ebc9dee to 4293941 Compare September 9, 2022 08:40
@andrewlock

Copy link
Copy Markdown
Member Author

Benchmarks Report 🐌

Benchmarks for #3154 compared to master:

  • 1 benchmarks are slower, with geometric mean 1.122
  • All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net472 721μs 592ns 2.29μs 0.361 0 0 3.18 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 471μs 186ns 697ns 0 0 0 2.59 KB
#3154 WriteAndFlushEnrichedTraces net472 720μs 558ns 2.16μs 0.359 0 0 3.18 KB
#3154 WriteAndFlushEnrichedTraces netcoreapp3.1 470μs 410ns 1.53μs 0 0 0 2.58 KB
Benchmarks.Trace.AppSecBodyBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #3154

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.AppSecBodyBenchmark.BodyExtractorSimpleBody‑netcoreapp3.1 1.122 219.81 246.67

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net472 206ns 0.529ns 2.05ns 0.0675 0 0 425 B
master AllCycleSimpleBody netcoreapp3.1 247ns 0.433ns 1.68ns 0.00585 0 0 424 B
master AllCycleMoreComplexBody net472 194ns 0.135ns 0.524ns 0.0637 0 0 401 B
master AllCycleMoreComplexBody netcoreapp3.1 232ns 0.4ns 1.55ns 0.00545 0 0 400 B
master BodyExtractorSimpleBody net472 259ns 0.278ns 1.04ns 0.0573 0 0 361 B
master BodyExtractorSimpleBody netcoreapp3.1 220ns 0.253ns 0.981ns 0.00371 0 0 272 B
master BodyExtractorMoreComplexBody net472 14.4μs 9.84ns 38.1ns 1.2 0.0216 0 7.62 KB
master BodyExtractorMoreComplexBody netcoreapp3.1 12.4μs 13.4ns 51.9ns 0.0931 0 0 6.75 KB
#3154 AllCycleSimpleBody net472 186ns 0.145ns 0.542ns 0.0675 0 0 425 B
#3154 AllCycleSimpleBody netcoreapp3.1 235ns 0.356ns 1.38ns 0.0058 0 0 424 B
#3154 AllCycleMoreComplexBody net472 185ns 0.181ns 0.677ns 0.0637 0 0 401 B
#3154 AllCycleMoreComplexBody netcoreapp3.1 233ns 0.26ns 0.972ns 0.00549 0 0 400 B
#3154 BodyExtractorSimpleBody net472 266ns 0.653ns 2.53ns 0.0574 0 0 361 B
#3154 BodyExtractorSimpleBody netcoreapp3.1 247ns 0.289ns 1.08ns 0.00372 0 0 272 B
#3154 BodyExtractorMoreComplexBody net472 14.6μs 14.5ns 54.4ns 1.21 0.0218 0 7.62 KB
#3154 BodyExtractorMoreComplexBody netcoreapp3.1 11.6μs 10.3ns 37.2ns 0.0926 0 0 6.75 KB
Benchmarks.Trace.AspNetCoreBenchmark - Unknown 🤷 Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net472 0ns 0ns 0ns 0 0 0 0 b
master SendRequest netcoreapp3.1 183μs 168ns 651ns 0.274 0 0 20.57 KB
#3154 SendRequest net472 0ns 0ns 0ns 0 0 0 0 b
#3154 SendRequest netcoreapp3.1 184μs 250ns 968ns 0.184 0 0 20.57 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net472 1.86μs 0.749ns 2.9ns 0.15 0.000931 0 947 B
master ExecuteNonQuery netcoreapp3.1 1.45μs 0.576ns 2.23ns 0.0123 0 0 936 B
#3154 ExecuteNonQuery net472 1.88μs 1.31ns 5.06ns 0.15 0.000939 0 947 B
#3154 ExecuteNonQuery netcoreapp3.1 1.38μs 0.579ns 2.24ns 0.0124 0 0 936 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net472 2.53μs 0.543ns 1.96ns 0.183 0 0 1.16 KB
master CallElasticsearch netcoreapp3.1 1.53μs 0.352ns 1.32ns 0.0146 0 0 1.1 KB
master CallElasticsearchAsync net472 2.66μs 0.901ns 3.37ns 0.204 0 0 1.29 KB
master CallElasticsearchAsync netcoreapp3.1 1.64μs 0.414ns 1.6ns 0.0164 0 0 1.22 KB
#3154 CallElasticsearch net472 2.62μs 1.36ns 5.1ns 0.183 0 0 1.16 KB
#3154 CallElasticsearch netcoreapp3.1 1.56μs 0.657ns 2.46ns 0.0147 0 0 1.1 KB
#3154 CallElasticsearchAsync net472 2.85μs 1.6ns 6ns 0.205 0 0 1.29 KB
#3154 CallElasticsearchAsync netcoreapp3.1 1.64μs 0.643ns 2.41ns 0.0165 0 0 1.22 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net472 2.66μs 4.92ns 17.7ns 0.225 0 0 1.42 KB
master ExecuteAsync netcoreapp3.1 1.75μs 2.52ns 9.41ns 0.0179 0 0 1.34 KB
#3154 ExecuteAsync net472 2.77μs 6.79ns 26.3ns 0.224 0 0 1.42 KB
#3154 ExecuteAsync netcoreapp3.1 1.66μs 3.73ns 14.5ns 0.018 0 0 1.34 KB
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net472 5.62μs 11ns 39.7ns 0.439 0 0 2.77 KB
master SendAsync netcoreapp3.1 3.55μs 7.73ns 29.9ns 0.0349 0 0 2.6 KB
#3154 SendAsync net472 5.61μs 8.3ns 31.1ns 0.439 0 0 2.77 KB
#3154 SendAsync netcoreapp3.1 3.66μs 13.6ns 52.7ns 0.0361 0 0 2.6 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 3.12μs 2.16ns 8.09ns 0.287 0 0 1.81 KB
master EnrichedLog netcoreapp3.1 2.55μs 0.893ns 3.34ns 0.0254 0 0 1.85 KB
#3154 EnrichedLog net472 3.08μs 3.95ns 15.3ns 0.287 0 0 1.81 KB
#3154 EnrichedLog netcoreapp3.1 2.43μs 0.843ns 3.15ns 0.0243 0 0 1.85 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 151μs 187ns 725ns 0.688 0.229 0 4.65 KB
master EnrichedLog netcoreapp3.1 114μs 73.8ns 276ns 0.0567 0 0 4.49 KB
#3154 EnrichedLog net472 150μs 103ns 397ns 0.674 0.225 0 4.65 KB
#3154 EnrichedLog netcoreapp3.1 117μs 203ns 759ns 0.0591 0 0 4.49 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 5.5μs 8.16ns 31.6ns 0.568 0.00271 0 3.59 KB
master EnrichedLog netcoreapp3.1 4.33μs 4.34ns 15ns 0.0523 0 0 3.91 KB
#3154 EnrichedLog net472 5.63μs 15ns 58.1ns 0.57 0.00277 0 3.59 KB
#3154 EnrichedLog netcoreapp3.1 4.29μs 7.56ns 29.3ns 0.0535 0 0 3.91 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net472 2.26μs 1.05ns 3.65ns 0.218 0 0 1.37 KB
master SendReceive netcoreapp3.1 1.77μs 0.462ns 1.73ns 0.0177 0 0 1.32 KB
#3154 SendReceive net472 2.33μs 0.956ns 3.58ns 0.218 0 0 1.37 KB
#3154 SendReceive netcoreapp3.1 1.77μs 1.31ns 4.91ns 0.0178 0 0 1.32 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 5.07μs 1.54ns 5.76ns 0.353 0 0 2.23 KB
master EnrichedLog netcoreapp3.1 4.22μs 0.846ns 3.17ns 0.0234 0 0 1.8 KB
#3154 EnrichedLog net472 5.23μs 2.79ns 10.8ns 0.353 0 0 2.23 KB
#3154 EnrichedLog netcoreapp3.1 4.26μs 1.48ns 5.54ns 0.0232 0 0 1.8 KB
Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net472 1.16μs 0.359ns 1.34ns 0.129 0 0 810 B
master StartFinishSpan netcoreapp3.1 926ns 0.412ns 1.54ns 0.0102 0 0 760 B
master StartFinishScope net472 1.35μs 0.341ns 1.32ns 0.141 0 0 891 B
master StartFinishScope netcoreapp3.1 1.14μs 0.405ns 1.4ns 0.012 0 0 880 B
#3154 StartFinishSpan net472 1.14μs 0.429ns 1.66ns 0.129 0 0 810 B
#3154 StartFinishSpan netcoreapp3.1 926ns 0.449ns 1.68ns 0.0102 0 0 760 B
#3154 StartFinishScope net472 1.41μs 0.731ns 2.83ns 0.141 0 0 891 B
#3154 StartFinishScope netcoreapp3.1 1.03μs 0.409ns 1.53ns 0.0118 0 0 880 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net472 1.52μs 0.558ns 2.09ns 0.141 0 0 891 B
master RunOnMethodBegin netcoreapp3.1 1.19μs 0.577ns 2.16ns 0.0117 0 0 880 B
#3154 RunOnMethodBegin net472 1.48μs 1.05ns 4.07ns 0.142 0 0 891 B
#3154 RunOnMethodBegin netcoreapp3.1 1.14μs 0.512ns 1.85ns 0.012 0 0 880 B

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants