[Profiler] Use shared runtime id in the profiler by gleocadie · Pull Request #2635 · DataDog/dd-trace-dotnet

gleocadie · 2022-04-01T16:26:05Z

Summary of changes

Reason for change

To finally have the Code Hotspot feature, the profiler needs to send runtime-id along with the profiles. Moreover, the runtime-id must be the same as the one used by the Tracer.

Implementation details

On windows, the profiler and tracer are started by the native proxy. This component also acts as a central point to share data/information. In our case, the native proxy will be source of truth for the runtime-id. In a previous PR, we made sure that the Tracer relies on the native proxy to get the runtime-id.

Now, the profiler will request the runtime-id for a specific AppDomainID to the native proxy.

On non-IIS applications, the runtime-id will be the same whatever AppDomainID is passed.
For IIS applications, the native proxy will provide one runtime-id per AppDomainID.

Note: In the linux case, we have a fallback, we let the profiler compute the runtime-id

In IIS case, we may have 1 process (w3wp) but multiple web applications. To handle this case, we will create and send as many requests as applications we have.
One pain point, the current version of the libddprof API we use, does not allow us to attach tags to a request. Tags are added to exporter instance. This forces us to create one exporter per application. In libddprof future releases, this is fixed.

Test coverage

Add new integration tests.

Other details

- dispatch sample per their runtime id - when exporting, send as many requests as applications we have (1 runtime id == 1 application)

add comment improve tests

andrewlock · 2022-04-07T01:03:44Z

Benchmarks Report 🐌

Benchmarks for #2635 compared to master:

4 benchmarks are faster, with geometric mean 1.152
All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Allocated
master	`WriteAndFlushEnrichedTraces`	net472	950μs	4.24μs	15.3μs	3.16 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	733μs	3.1μs	11.2μs	2.57 KB
#2635	`WriteAndFlushEnrichedTraces`	net472	915μs	3.2μs	12μs	3.16 KB
#2635	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	719μs	2.53μs	9.78μs	2.57 KB

Benchmarks.Trace.AppSecBodyBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`AllCycleSimpleBody`	net472	315ns	1.46ns	5.48ns	0.0519	0	329 B
master	`AllCycleSimpleBody`	netcoreapp3.1	387ns	1.61ns	6.22ns	0.00444	0	328 B
master	`AllCycleMoreComplexBody`	net472	307ns	1.45ns	5.62ns	0.0481	0	305 B
master	`AllCycleMoreComplexBody`	netcoreapp3.1	375ns	1.37ns	4.93ns	0.00411	0	304 B
master	`BodyExtractorSimpleBody`	net472	454ns	1.98ns	7.41ns	0.0569	0	361 B
master	`BodyExtractorSimpleBody`	netcoreapp3.1	463ns	2.06ns	8.23ns	0.0039	0	272 B
master	`BodyExtractorMoreComplexBody`	net472	25.5μs	110ns	605ns	1.19	0.0131	7.62 KB
master	`BodyExtractorMoreComplexBody`	netcoreapp3.1	21.5μs	81.4ns	305ns	0.0971	0	6.75 KB
#2635	`AllCycleSimpleBody`	net472	304ns	1.49ns	6.49ns	0.0519	0	329 B
#2635	`AllCycleSimpleBody`	netcoreapp3.1	374ns	1.92ns	9ns	0.00449	0	328 B
#2635	`AllCycleMoreComplexBody`	net472	284ns	0.895ns	3.35ns	0.0482	0	305 B
#2635	`AllCycleMoreComplexBody`	netcoreapp3.1	406ns	6.35ns	63.5ns	0.00417	0	304 B
#2635	`BodyExtractorSimpleBody`	net472	420ns	2.13ns	9.03ns	0.0571	0	361 B
#2635	`BodyExtractorSimpleBody`	netcoreapp3.1	472ns	2.46ns	11.8ns	0.00365	0	272 B
#2635	`BodyExtractorMoreComplexBody`	net472	23.9μs	79.3ns	307ns	1.19	0.0117	7.62 KB
#2635	`BodyExtractorMoreComplexBody`	netcoreapp3.1	20.6μs	104ns	478ns	0.0934	0	6.75 KB

Benchmarks.Trace.AspNetCoreBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #2635

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.AspNetCoreBenchmark.SendRequest‑netcoreapp3.1	1.195	349,203.56	292,238.27

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net472	0ns	0ns	0ns	0	0 b
master	`SendRequest`	netcoreapp3.1	350μs	1.61μs	6.42μs	0.17	19.74 KB
#2635	`SendRequest`	net472	0ns	0ns	0ns	0	0 b
#2635	`SendRequest`	netcoreapp3.1	292μs	1.13μs	4.24μs	0.146	19.74 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`ExecuteNonQuery`	net472	1.93μs	11ns	84.1ns	0.0934	0.00103	594 B
master	`ExecuteNonQuery`	netcoreapp3.1	1.56μs	5.48ns	20.5ns	0.00868	0	632 B
#2635	`ExecuteNonQuery`	net472	1.74μs	8.19ns	31.7ns	0.0943	0.000874	594 B
#2635	`ExecuteNonQuery`	netcoreapp3.1	1.49μs	3.7ns	13.3ns	0.00845	0	632 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`CallElasticsearch`	net472	3.03μs	14.5ns	59.8ns	0.127	0	803 B
master	`CallElasticsearch`	netcoreapp3.1	1.79μs	10ns	75.1ns	0.0105	0	792 B
master	`CallElasticsearchAsync`	net472	3.05μs	15.3ns	68.2ns	0.148	0.00151	939 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.91μs	9.16ns	37.8ns	0.0132	0	912 B
#2635	`CallElasticsearch`	net472	2.82μs	13.4ns	61.3ns	0.127	0	803 B
#2635	`CallElasticsearch`	netcoreapp3.1	1.85μs	6.64ns	23ns	0.0105	0	792 B
#2635	`CallElasticsearchAsync`	net472	3.08μs	16.5ns	92ns	0.148	0.00151	939 B
#2635	`CallElasticsearchAsync`	netcoreapp3.1	1.81μs	10.1ns	63.2ns	0.0123	0	912 B

Benchmarks.Trace.GraphQLBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #2635

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync‑net472	1.131	3,397.40	3,003.50

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`ExecuteAsync`	net472	3.41μs	16.9ns	71.8ns	0.167	0.00167	1.06 KB
master	`ExecuteAsync`	netcoreapp3.1	2.13μs	9.34ns	36.2ns	0.0147	0	1.03 KB
#2635	`ExecuteAsync`	net472	3.02μs	12ns	51.1ns	0.167	0.00154	1.06 KB
#2635	`ExecuteAsync`	netcoreapp3.1	2μs	6.94ns	29.4ns	0.0139	0	1.03 KB

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net472	7.24μs	37.2ns	167ns	0.361	2.28 KB
master	`SendAsync`	netcoreapp3.1	5.15μs	16.4ns	63.7ns	0.0303	2.21 KB
#2635	`SendAsync`	net472	6.84μs	22.3ns	86.3ns	0.359	2.28 KB
#2635	`SendAsync`	netcoreapp3.1	5.04μs	17.5ns	65.4ns	0.0301	2.21 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net472	3.73μs	21.8ns	192ns	0.227	1.45 KB
master	`EnrichedLog`	netcoreapp3.1	3.15μs	11.8ns	44.1ns	0.0207	1.53 KB
#2635	`EnrichedLog`	net472	3.38μs	12.4ns	46.2ns	0.228	1.45 KB
#2635	`EnrichedLog`	netcoreapp3.1	3.09μs	4.65ns	16.1ns	0.0213	1.53 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net472	302μs	1.28μs	4.97μs	0.314	0.157	4.33 KB
master	`EnrichedLog`	netcoreapp3.1	244μs	1.13μs	4.37μs	0	0	4.21 KB
#2635	`EnrichedLog`	net472	292μs	1.36μs	5.27μs	0.428	0.143	4.33 KB
#2635	`EnrichedLog`	netcoreapp3.1	233μs	1.06μs	4.09μs	0	0	4.21 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net472	8.21μs	38ns	142ns	0.504	3.23 KB
master	`EnrichedLog`	netcoreapp3.1	6.79μs	30.9ns	164ns	0.0499	3.6 KB
#2635	`EnrichedLog`	net472	7.84μs	39.7ns	182ns	0.504	3.23 KB
#2635	`EnrichedLog`	netcoreapp3.1	6.57μs	33.6ns	150ns	0.0483	3.6 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`SendReceive`	net472	2.39μs	10.9ns	40.8ns	0.16	0.00121	1.01 KB
master	`SendReceive`	netcoreapp3.1	2.09μs	9.8ns	38ns	0.0139	0	1.01 KB
#2635	`SendReceive`	net472	2.41μs	13.3ns	83ns	0.159	0	1.01 KB
#2635	`SendReceive`	netcoreapp3.1	2.24μs	10.8ns	61.3ns	0.0137	0	1.01 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net472	6.44μs	20.4ns	79.1ns	0.292	1.87 KB
master	`EnrichedLog`	netcoreapp3.1	5.23μs	12.8ns	47.9ns	0.021	1.49 KB
#2635	`EnrichedLog`	net472	5.92μs	23.7ns	82ns	0.292	1.87 KB
#2635	`EnrichedLog`	netcoreapp3.1	5.11μs	23.5ns	91ns	0.0205	1.49 KB

Benchmarks.Trace.SpanBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #2635

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑netcoreapp3.1	1.149	991.01	862.43
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net472	1.136	1,248.07	1,099.05

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net472	964ns	5.14ns	27.2ns	0.0716	457 B
master	`StartFinishSpan`	netcoreapp3.1	991ns	4.58ns	17.7ns	0.00611	456 B
master	`StartFinishScope`	net472	1.24μs	6.9ns	42.5ns	0.0841	538 B
master	`StartFinishScope`	netcoreapp3.1	1.13μs	6.58ns	57.4ns	0.0079	576 B
#2635	`StartFinishSpan`	net472	885ns	2.75ns	10.7ns	0.0718	457 B
#2635	`StartFinishSpan`	netcoreapp3.1	871ns	4.59ns	22ns	0.00635	456 B
#2635	`StartFinishScope`	net472	1.1μs	4.92ns	18.4ns	0.0844	538 B
#2635	`StartFinishScope`	netcoreapp3.1	1.02μs	4.2ns	16.3ns	0.00799	576 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net472	1.38μs	6.84ns	29ns	0.0838	538 B
master	`RunOnMethodBegin`	netcoreapp3.1	1.24μs	6.47ns	42.4ns	0.00798	576 B
#2635	`RunOnMethodBegin`	net472	1.29μs	3.84ns	14.9ns	0.0844	538 B
#2635	`RunOnMethodBegin`	netcoreapp3.1	1.18μs	5.47ns	21.2ns	0.00761	576 B

andrewlock · 2022-04-07T01:07:53Z

Code Coverage Report 📊

✔️ Merging #2635 into master will not change line coverage
✔️ Merging #2635 into master will not change branch coverage
✔️ Merging #2635 into master will will decrease complexity by 2

	master	#2635	Change
Lines	`13462` / `18467`	`13480` / `18467`
Lines %	`73%`	`73%`	`0%` ✔️
Branches	`7782` / `11018`	`7798` / `11017`
Branches %	`71%`	`71%`	`0%` ✔️
Complexity	`12309`	`12307`	`-2` ✔️

View the full report for further details:

Datadog.Trace Breakdown ✔️

	master	#2635	Change
Lines %	`73%`	`73%`	`0%` ✔️
Branches %	`71%`	`71%`	`0%` ✔️
Complexity	`12309`	`12307`	`-2` ✔️

The following classes have significant coverage changes.

File	Line coverage change	Branch coverage change	Complexity change
Datadog.Trace.Agent.StreamFactories.TcpStreamFactory	`-90%` ⛔	`0%` ✔️	`0` ✔️
Datadog.Trace.Telemetry.JsonHttpClientTelemetryTransport	`-26%` ⛔	`-38%` ⛔	`0` ✔️
Datadog.Trace.HttpOverStreams.TraceAgentHttpHeaderHelper	`-20%` ⛔	`0%` ✔️	`0` ✔️
Datadog.Trace.AsyncLocalScopeManager	`-8%` ⛔	`0%` ✔️	`0` ✔️
Datadog.Trace.Agent.TracesTransportStrategy	`-4%` ⚠️	`15%` ✔️	`-1` ✔️

View the full reports for further details:

chrisnas

LGTM

simplify the asserts

gleocadie added 2 commits April 1, 2022 15:57

Update the exporter class

753677d

- dispatch sample per their runtime id - when exporting, send as many requests as applications we have (1 runtime id == 1 application)

Implement RuntimeIdStore

19109c3

gleocadie added the area:profiler Issues related to the continous-profiler label Apr 1, 2022

gleocadie force-pushed the gleocadie/use-shared-runtime-id-in-the-profiler branch from 9e95e6e to fc56937 Compare April 6, 2022 07:28

Validate runtime id between profiler and tracer

4d0efc0

gleocadie force-pushed the gleocadie/use-shared-runtime-id-in-the-profiler branch from fc56937 to 22a17b0 Compare April 6, 2022 09:11

Fix linux build

3518bd6

gleocadie force-pushed the gleocadie/use-shared-runtime-id-in-the-profiler branch from 22a17b0 to 3518bd6 Compare April 6, 2022 09:36

gleocadie added 3 commits April 6, 2022 16:12

fix crash

36503c2

Use monitoring home for ASAN Tests

317d8e1

Fix asan test

9761df2

gleocadie force-pushed the gleocadie/use-shared-runtime-id-in-the-profiler branch 2 times, most recently from 4fb1aa5 to 80544c6 Compare April 6, 2022 14:28

Few fixes

713db3a

gleocadie force-pushed the gleocadie/use-shared-runtime-id-in-the-profiler branch 4 times, most recently from 250cf08 to 7bbdf51 Compare April 6, 2022 19:54

Fix buggy bits test and ASAN test in CI

2faa64f

gleocadie force-pushed the gleocadie/use-shared-runtime-id-in-the-profiler branch from 7bbdf51 to 2faa64f Compare April 6, 2022 20:10

gleocadie marked this pull request as ready for review April 6, 2022 20:56

gleocadie requested a review from a team as a code owner April 6, 2022 20:56

gleocadie force-pushed the gleocadie/use-shared-runtime-id-in-the-profiler branch from 2faa64f to 93d04a3 Compare April 6, 2022 20:57

Fix up

73c50c0

add comment improve tests

gleocadie force-pushed the gleocadie/use-shared-runtime-id-in-the-profiler branch from 93d04a3 to 73c50c0 Compare April 6, 2022 22:06

gleocadie requested a review from a team as a code owner April 6, 2022 23:54

Fix codehotspot test

ffadd54

gleocadie force-pushed the gleocadie/use-shared-runtime-id-in-the-profiler branch from 4443124 to ffadd54 Compare April 7, 2022 06:59

chrisnas requested changes Apr 7, 2022

View reviewed changes

gleocadie added 2 commits April 7, 2022 10:22

Address PR comment

333cf35

Try making the test less flaky

f18f9da

chrisnas approved these changes Apr 7, 2022

View reviewed changes

Fix x86 build

cdb6067

simplify the asserts

gleocadie merged commit fb1bb82 into master Apr 7, 2022

gleocadie deleted the gleocadie/use-shared-runtime-id-in-the-profiler branch April 7, 2022 14:23

github-actions Bot added this to the vNext milestone Apr 7, 2022

Uh oh!

Conversation

gleocadie commented Apr 1, 2022 • edited by chrisnas Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

Uh oh!

andrewlock commented Apr 7, 2022

Benchmarks Report 🐌

Benchmark details

Raw results

Raw results

Faster 🎉 in #2635

Raw results

Raw results

Raw results

Faster 🎉 in #2635

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #2635

Raw results

Raw results

Uh oh!

andrewlock commented Apr 7, 2022

Code Coverage Report 📊

Datadog.Trace Breakdown ✔️

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chrisnas left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

gleocadie commented Apr 1, 2022 •

edited by chrisnas

Loading