Switch .NET tracer to injecting both base64 & binary headers by veerbia · Pull Request #6448 · DataDog/dd-trace-dotnet

veerbia · 2024-12-16T18:24:33Z

Summary of changes

Switch .NET tracer to injecting both base64 & binary headers
Binary headers can be disabled via a config variable: DD_DATA_STREAMS_LEGACY_HEADERS=false (defaults to true)
.NET tracer, when extracting the context from headers looks at both base64 & binary headers

Reason for change

Data Streams previously used binary encoding in Kafka headers. This was causing issues in cross language communication because of the difference in negative byte handling. That’s why we switched to base64 encoding.
Today, .NET is the only remaining tracer using binary encoding for Kafka, SQS & RabbitMQ.

This is causing 3 issues:

Communication between .NET & Java breaks, since Java doesn’t support negative bytes in Kafka headers
Manual instrumentation breaks between .NET and any other language, since manual instrumentation uses base64 encoding
.NET can’t consume payloads from any other tracers, since the other tracers encode in base64

Also, byte headers are causing a crash of the .NET application (not reproduced yet).

Implementation details

Injects base64-encoded headers by default to ensure cross-language compatibility.
Binary headers are injected only if DD_DATA_STREAMS_LEGACY_HEADERS (default: true) is enabled for backward compatibility.
Prefers base64 headers for extraction. If base64 is unavailable or malformed, it falls back to binary headers when legacy headers are enabled.
A new configuration setting DD_DATA_STREAMS_LEGACY_HEADERS controls whether binary headers are included.

Test coverage

I've added tests for the following cases:

Inject_WhenLegacyHeadersDisabled_DoesNotIncludeBinaryHeader
- Ensures that when legacy headers are disabled, the binary header is not included in the injected headers.
Extract_WhenBothHeadersPresent_PrefersBase64Header
- Confirms that when both Base64 and binary headers are present, the Base64 header is preferred for extraction.
InjectedHeaders_HaveCorrectFormat
- Validates that injected headers are in the correct format, especially checking if the Base64-encoded header is valid and correctly decoded.
InjectHeaders_WhenLegacyHeadersDisabled_DoesNotIncludeLegacyHeader
- Ensures that legacy headers are excluded when legacy header support is disabled.
Inject_WhenLegacyHeadersEnabled_IncludesBothHeaders
- Confirms that both Base64 and binary headers are injected when legacy header support is enabled.
Extract_WhenBase64HeaderIsMalformed_ReturnsFallbackToBinary
- Verifies that if the Base64 header is malformed, the system falls back to using the binary header for extraction.

Other details

This appears to be handled already and doing it here just dupes the content. And it didn't add comma's between the JSON fields which ccause JSON deserialization errors. Additionally, it was re-encoding the Base64 again.

so that it isn't disabled.

This is added already below, but initially it wasn't obivous because the legacy data streams was always false.

…hared/ContextPropagation.cs

github-actions · 2024-12-16T18:25:20Z

Snapshots difference summary

The following differences have been observed in committed snapshots. It is meant to help the reviewer.
The diff is simplistic, so please check some files anyway while we improve it.

8 occurrences of :

-      aws.queue.name: MyAsyncSQSQueue,
-      aws.queue.url: http://localhost:00000/000000000000/MyAsyncSQSQueue,
+      aws.queue.name: MyAsyncSQSQueue2,
+      aws.queue.url: http://localhost:00000/000000000000/MyAsyncSQSQueue2,

2 occurrences of :

-      http.url: http://localhost:00000/000000000000/MyAsyncSQSQueue,
+      http.url: http://localhost:00000/000000000000/MyAsyncSQSQueue2,
[...]
-      queuename: MyAsyncSQSQueue,
+      queuename: MyAsyncSQSQueue2,

2 occurrences of :

-      queuename: MyAsyncSQSQueue,
+      queuename: MyAsyncSQSQueue2,

2 occurrences of :

-      http.url: http://localhost:00000/000000000000/MyAsyncSQSQueue,
+      http.url: http://localhost:00000/000000000000/MyAsyncSQSQueue2,
[...]
-      peer.service: MyAsyncSQSQueue,
-      queuename: MyAsyncSQSQueue,
+      peer.service: MyAsyncSQSQueue2,
+      queuename: MyAsyncSQSQueue2,

2 occurrences of :

-      peer.service: MyAsyncSQSQueue,
-      queuename: MyAsyncSQSQueue,
+      peer.service: MyAsyncSQSQueue2,
+      queuename: MyAsyncSQSQueue2,

datadog-ddstaging · 2024-12-16T19:00:01Z

Datadog Report

Branch report: veerbia/dsm-base64-migration
Commit report: 4c7e223
Test service: dd-trace-dotnet

✅ 0 Failed, 463613 Passed, 2849 Skipped, 19h 33m 27.3s Total Time

andrewlock · 2024-12-16T20:06:19Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6448) - mean (68ms)  : 66, 71
     .   : milestone, 68,
    master - mean (68ms)  : 66, 71
     .   : milestone, 68,

    section CallTarget+Inlining+NGEN
    This PR (6448) - mean (978ms)  : 956, 1001
     .   : milestone, 978,
    master - mean (977ms)  : 956, 998
     .   : milestone, 977,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6448) - mean (107ms)  : 105, 109
     .   : milestone, 107,
    master - mean (107ms)  : 105, 109
     .   : milestone, 107,

    section CallTarget+Inlining+NGEN
    This PR (6448) - mean (678ms)  : 661, 695
     .   : milestone, 678,
    master - mean (678ms)  : 663, 692
     .   : milestone, 678,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6448) - mean (91ms)  : 89, 93
     .   : milestone, 91,
    master - mean (91ms)  : 89, 93
     .   : milestone, 91,

    section CallTarget+Inlining+NGEN
    This PR (6448) - mean (633ms)  : 615, 651
     .   : milestone, 633,
    master - mean (635ms)  : 621, 649
     .   : milestone, 635,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6448) - mean (194ms)  : 189, 198
     .   : milestone, 194,
    master - mean (194ms)  : 189, 199
     .   : milestone, 194,

    section CallTarget+Inlining+NGEN
    This PR (6448) - mean (1,105ms)  : 1070, 1140
     .   : milestone, 1105,
    master - mean (1,098ms)  : 1069, 1127
     .   : milestone, 1098,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6448) - mean (278ms)  : 272, 284
     .   : milestone, 278,
    master - mean (278ms)  : 273, 283
     .   : milestone, 278,

    section CallTarget+Inlining+NGEN
    This PR (6448) - mean (874ms)  : 845, 903
     .   : milestone, 874,
    master - mean (878ms)  : 850, 906
     .   : milestone, 878,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6448) - mean (268ms)  : 260, 276
     .   : milestone, 268,
    master - mean (267ms)  : 263, 272
     .   : milestone, 267,

    section CallTarget+Inlining+NGEN
    This PR (6448) - mean (852ms)  : 817, 887
     .   : milestone, 852,
    master - mean (851ms)  : 824, 877
     .   : milestone, 851,

andrewlock · 2024-12-16T22:58:48Z

Benchmarks Report for tracer 🐌

Benchmarks for #6448 compared to master:

1 benchmarks are faster, with geometric mean 1.118
1 benchmarks are slower, with geometric mean 1.181
1 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	8.03μs	44.8ns	280ns	0.0168	0.00842	0	5.62 KB
master	`StartStopWithChild`	netcoreapp3.1	9.86μs	55.9ns	379ns	0.0251	0.01	0	5.8 KB
master	`StartStopWithChild`	net472	16.4μs	56.9ns	221ns	1.05	0.327	0.106	6.21 KB
#6448	`StartStopWithChild`	net6.0	8.05μs	41.7ns	220ns	0.0279	0.00796	0	5.61 KB
#6448	`StartStopWithChild`	netcoreapp3.1	10μs	55.8ns	335ns	0.0247	0.0099	0	5.8 KB
#6448	`StartStopWithChild`	net472	16.6μs	70.1ns	272ns	1.04	0.302	0.0899	6.2 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	478μs	321ns	1.25μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	669μs	355ns	1.38μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	857μs	450ns	1.68μs	0.431	3.3 KB
#6448	`WriteAndFlushEnrichedTraces`	net6.0	506μs	342ns	1.32μs	0	2.7 KB
#6448	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	640μs	357ns	1.38μs	0	2.7 KB
#6448	`WriteAndFlushEnrichedTraces`	net472	855μs	385ns	1.49μs	0.425	3.3 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	155μs	934ns	9.11μs	0.145	14.47 KB
master	`SendRequest`	netcoreapp3.1	169μs	992ns	9.09μs	0.173	17.27 KB
master	`SendRequest`	net472	0ns	0ns	0ns	0	0 b
#6448	`SendRequest`	net6.0	152μs	1.12μs	11.2μs	0.151	14.47 KB
#6448	`SendRequest`	netcoreapp3.1	171μs	1.04μs	10.2μs	0.16	17.27 KB
#6448	`SendRequest`	net472	0.00177ns	0.000774ns	0.0029ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #6448

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑netcoreapp3.1	41.65 KB	41.89 KB	243 B	0.58%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	572μs	2.72μs	11.5μs	0.581	0	0	41.73 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	663μs	3.38μs	20.3μs	0.379	0	0	41.65 KB
master	`WriteAndFlushEnrichedTraces`	net472	833μs	2.98μs	10.8μs	8.33	2.5	0.417	53.27 KB
#6448	`WriteAndFlushEnrichedTraces`	net6.0	543μs	2.71μs	12.1μs	0.551	0	0	41.57 KB
#6448	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	672μs	3.75μs	25.7μs	0.317	0	0	41.89 KB
#6448	`WriteAndFlushEnrichedTraces`	net472	813μs	3.36μs	12.6μs	8.36	2.39	0.398	53.29 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`ExecuteNonQuery`	net6.0	1.34μs	1.63ns	6.08ns	0.014	0	1.02 KB
master	`ExecuteNonQuery`	netcoreapp3.1	1.79μs	1.58ns	6.12ns	0.0133	0	1.02 KB
master	`ExecuteNonQuery`	net472	2.13μs	1.5ns	5.6ns	0.157	0.00105	987 B
#6448	`ExecuteNonQuery`	net6.0	1.33μs	1.42ns	5.5ns	0.0145	0	1.02 KB
#6448	`ExecuteNonQuery`	netcoreapp3.1	1.8μs	1.44ns	5.57ns	0.0134	0	1.02 KB
#6448	`ExecuteNonQuery`	net472	2.04μs	2.36ns	9.14ns	0.157	0.00101	987 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.33μs	0.421ns	1.52ns	0.0133	976 B
master	`CallElasticsearch`	netcoreapp3.1	1.64μs	1.13ns	4.21ns	0.0132	976 B
master	`CallElasticsearch`	net472	2.54μs	2.48ns	9.62ns	0.157	995 B
master	`CallElasticsearchAsync`	net6.0	1.3μs	3.34ns	13ns	0.0135	952 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.67μs	0.985ns	3.82ns	0.0132	1.02 KB
master	`CallElasticsearchAsync`	net472	2.66μs	2.27ns	8.5ns	0.166	1.05 KB
#6448	`CallElasticsearch`	net6.0	1.3μs	0.876ns	3.39ns	0.0132	976 B
#6448	`CallElasticsearch`	netcoreapp3.1	1.5μs	1.13ns	4.25ns	0.013	976 B
#6448	`CallElasticsearch`	net472	2.47μs	1.17ns	4.23ns	0.158	995 B
#6448	`CallElasticsearchAsync`	net6.0	1.29μs	0.924ns	3.46ns	0.0129	952 B
#6448	`CallElasticsearchAsync`	netcoreapp3.1	1.72μs	1.56ns	6.04ns	0.0137	1.02 KB
#6448	`CallElasticsearchAsync`	net472	2.64μs	1.51ns	5.63ns	0.166	1.05 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.34μs	0.67ns	2.6ns	0.0133	952 B
master	`ExecuteAsync`	netcoreapp3.1	1.68μs	0.626ns	2.43ns	0.0131	952 B
master	`ExecuteAsync`	net472	1.81μs	0.718ns	2.78ns	0.145	915 B
#6448	`ExecuteAsync`	net6.0	1.34μs	0.42ns	1.57ns	0.0134	952 B
#6448	`ExecuteAsync`	netcoreapp3.1	1.68μs	0.981ns	3.67ns	0.0125	952 B
#6448	`ExecuteAsync`	net472	1.81μs	0.669ns	2.59ns	0.145	915 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.46μs	1.41ns	5.46ns	0.0311	2.31 KB
master	`SendAsync`	netcoreapp3.1	5.55μs	1.45ns	5.44ns	0.039	2.85 KB
master	`SendAsync`	net472	7.37μs	2.3ns	8.92ns	0.494	3.12 KB
#6448	`SendAsync`	net6.0	4.34μs	1.57ns	5.86ns	0.0324	2.31 KB
#6448	`SendAsync`	netcoreapp3.1	5.25μs	2.94ns	11.4ns	0.0392	2.85 KB
#6448	`SendAsync`	net472	7.36μs	2.35ns	8.81ns	0.495	3.12 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.54μs	0.991ns	3.71ns	0.023	1.64 KB
master	`EnrichedLog`	netcoreapp3.1	2.18μs	5.29ns	20.5ns	0.0216	1.64 KB
master	`EnrichedLog`	net472	2.71μs	1.18ns	4.57ns	0.249	1.57 KB
#6448	`EnrichedLog`	net6.0	1.66μs	0.898ns	3.36ns	0.0233	1.64 KB
#6448	`EnrichedLog`	netcoreapp3.1	2.29μs	0.817ns	3.06ns	0.0221	1.64 KB
#6448	`EnrichedLog`	net472	2.7μs	0.839ns	3.25ns	0.25	1.57 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	119μs	153ns	592ns	0	0	4.28 KB
master	`EnrichedLog`	netcoreapp3.1	121μs	140ns	540ns	0	0	4.28 KB
master	`EnrichedLog`	net472	150μs	228ns	884ns	0.675	0.225	4.46 KB
#6448	`EnrichedLog`	net6.0	120μs	116ns	448ns	0	0	4.28 KB
#6448	`EnrichedLog`	netcoreapp3.1	124μs	196ns	759ns	0	0	4.28 KB
#6448	`EnrichedLog`	net472	153μs	165ns	640ns	0.68	0.227	4.46 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	3.02μs	0.832ns	3.11ns	0.031	2.2 KB
master	`EnrichedLog`	netcoreapp3.1	4.19μs	3.91ns	15.2ns	0.0294	2.2 KB
master	`EnrichedLog`	net472	4.87μs	1.82ns	7.03ns	0.319	2.02 KB
#6448	`EnrichedLog`	net6.0	3.11μs	1.04ns	4.02ns	0.0311	2.2 KB
#6448	`EnrichedLog`	netcoreapp3.1	4.22μs	2.97ns	11.5ns	0.0294	2.2 KB
#6448	`EnrichedLog`	net472	4.9μs	2.06ns	7.99ns	0.32	2.02 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	1.44μs	0.785ns	2.94ns	0.0159	1.14 KB
master	`SendReceive`	netcoreapp3.1	1.88μs	1.56ns	5.82ns	0.0152	1.14 KB
master	`SendReceive`	net472	2.21μs	0.867ns	3.01ns	0.184	1.16 KB
#6448	`SendReceive`	net6.0	1.32μs	0.662ns	2.56ns	0.0164	1.14 KB
#6448	`SendReceive`	netcoreapp3.1	1.87μs	1.95ns	7.56ns	0.0151	1.14 KB
#6448	`SendReceive`	net472	2.18μs	1.06ns	4.09ns	0.183	1.16 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.79μs	0.86ns	3.33ns	0.0222	1.6 KB
master	`EnrichedLog`	netcoreapp3.1	4.02μs	1.3ns	4.85ns	0.0225	1.65 KB
master	`EnrichedLog`	net472	4.55μs	3.92ns	15.2ns	0.323	2.04 KB
#6448	`EnrichedLog`	net6.0	2.66μs	0.58ns	2.25ns	0.0226	1.6 KB
#6448	`EnrichedLog`	netcoreapp3.1	3.89μs	1.67ns	6.23ns	0.0214	1.65 KB
#6448	`EnrichedLog`	net472	4.4μs	3.16ns	12.2ns	0.324	2.04 KB

Benchmarks.Trace.SpanBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #6448

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net472	1.118	924.38	827.14

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	407ns	0.154ns	0.576ns	0.00802	576 B
master	`StartFinishSpan`	netcoreapp3.1	543ns	0.315ns	1.22ns	0.00782	576 B
master	`StartFinishSpan`	net472	702ns	0.585ns	2.27ns	0.0915	578 B
master	`StartFinishScope`	net6.0	556ns	0.232ns	0.869ns	0.00977	696 B
master	`StartFinishScope`	netcoreapp3.1	711ns	0.489ns	1.83ns	0.00939	696 B
master	`StartFinishScope`	net472	923ns	0.52ns	2.01ns	0.104	658 B
#6448	`StartFinishSpan`	net6.0	400ns	0.256ns	0.888ns	0.00807	576 B
#6448	`StartFinishSpan`	netcoreapp3.1	565ns	0.599ns	2.24ns	0.00793	576 B
#6448	`StartFinishSpan`	net472	669ns	0.722ns	2.8ns	0.0918	578 B
#6448	`StartFinishScope`	net6.0	552ns	0.524ns	2.03ns	0.00978	696 B
#6448	`StartFinishScope`	netcoreapp3.1	691ns	0.509ns	1.91ns	0.00949	696 B
#6448	`StartFinishScope`	net472	826ns	1.05ns	4.07ns	0.104	658 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #6448

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑netcoreapp3.1	1.181	896.90	1,058.94

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	630ns	0.423ns	1.58ns	0.00979	696 B
master	`RunOnMethodBegin`	netcoreapp3.1	897ns	0.46ns	1.78ns	0.00943	696 B
master	`RunOnMethodBegin`	net472	1.11μs	0.573ns	2.22ns	0.104	658 B
#6448	`RunOnMethodBegin`	net6.0	651ns	0.524ns	2.03ns	0.00975	696 B
#6448	`RunOnMethodBegin`	netcoreapp3.1	1.06μs	2.33ns	9.04ns	0.009	696 B
#6448	`RunOnMethodBegin`	net472	1.13μs	0.891ns	3.45ns	0.105	658 B

andrewlock · 2024-12-16T23:53:01Z

Throughput/Crank Report ⚡

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (6448) (11.096M)   : 0, 11096382
    master (11.176M)   : 0, 11175779
    benchmarks/2.9.0 (11.033M)   : 0, 11032866

    section Automatic
    This PR (6448) (7.312M)   : 0, 7312245
    master (7.188M)   : 0, 7187812
    benchmarks/2.9.0 (7.786M)   : 0, 7785853

    section Trace stats
    master (7.592M)   : 0, 7592225

    section Manual
    master (11.202M)   : 0, 11201615

    section Manual + Automatic
    This PR (6448) (6.810M)   : 0, 6809676
    master (6.624M)   : 0, 6623622

    section DD_TRACE_ENABLED=0
    master (10.184M)   : 0, 10184081

gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (6448) (9.561M)   : 0, 9561017
    master (9.299M)   : 0, 9299402
    benchmarks/2.9.0 (9.495M)   : 0, 9494821

    section Automatic
    This PR (6448) (6.353M)   : 0, 6353019
    master (6.339M)   : 0, 6338804

    section Trace stats
    master (6.490M)   : 0, 6490416

    section Manual
    master (9.438M)   : 0, 9437615

    section Manual + Automatic
    This PR (6448) (5.940M)   : 0, 5940038
    master (5.979M)   : 0, 5979389

    section DD_TRACE_ENABLED=0
    master (8.785M)   : 0, 8784619

gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (6448) (10.036M)   : 0, 10035673
    master (9.992M)   : 0, 9991636
    benchmarks/2.9.0 (10.020M)   : 0, 10019592

    section Automatic
    This PR (6448) (6.297M)   : 0, 6297190
    master (6.531M)   : 0, 6530893
    benchmarks/2.9.0 (7.255M)   : 0, 7255257

    section Trace stats
    master (7.153M)   : 0, 7152931

    section Manual
    master (10.055M)   : 0, 10055115

    section Manual + Automatic
    This PR (6448) (6.160M)   : 0, 6159823
    master (5.851M)   : 0, 5850726

    section DD_TRACE_ENABLED=0
    master (9.524M)   : 0, 9524080

robertpi · 2024-12-17T11:51:42Z

Replaces: #6257

## Summary of changes - Switch .NET tracer to injecting both base64 & binary headers - Binary headers can be disabled via a config variable: DD_DATA_STREAMS_LEGACY_HEADERS=false (defaults to true) - .NET tracer, when extracting the context from headers looks at both base64 & binary headers ## Reason for change Data Streams previously used binary encoding in Kafka headers. This was causing issues in cross language communication because of the difference in negative byte handling. That’s why we switched to base64 encoding. Today, .NET is the only remaining tracer using binary encoding for Kafka, SQS & RabbitMQ. This is causing 3 issues: - Communication between .NET & Java breaks, since Java doesn’t support negative bytes in Kafka headers - Manual instrumentation breaks between .NET and any other language, since manual instrumentation uses base64 encoding - .NET can’t consume payloads from any other tracers, since the other tracers encode in base64 Also, byte headers are causing a crash of the [.NET application](https://datadog.zendesk.com/agent/tickets/1824351) (not reproduced yet). ## Implementation details - Injects base64-encoded headers by default to ensure cross-language compatibility. - Binary headers are injected only if `DD_DATA_STREAMS_LEGACY_HEADERS` (default: true) is enabled for backward compatibility. - Prefers base64 headers for extraction. If base64 is unavailable or malformed, it falls back to binary headers when legacy headers are enabled. - A new configuration setting DD_DATA_STREAMS_LEGACY_HEADERS controls whether binary headers are included. ## Test coverage I've added tests for the following cases: - Inject_WhenLegacyHeadersDisabled_DoesNotIncludeBinaryHeader - Ensures that when legacy headers are disabled, the binary header is not included in the injected headers. - Extract_WhenBothHeadersPresent_PrefersBase64Header - Confirms that when both Base64 and binary headers are present, the Base64 header is preferred for extraction. - InjectedHeaders_HaveCorrectFormat - Validates that injected headers are in the correct format, especially checking if the Base64-encoded header is valid and correctly decoded. - InjectHeaders_WhenLegacyHeadersDisabled_DoesNotIncludeLegacyHeader - Ensures that legacy headers are excluded when legacy header support is disabled. - Inject_WhenLegacyHeadersEnabled_IncludesBothHeaders - Confirms that both Base64 and binary headers are injected when legacy header support is enabled. - Extract_WhenBase64HeaderIsMalformed_ReturnsFallbackToBinary - Verifies that if the Base64 header is malformed, the system falls back to using the binary header for extraction. ## Other details   --------- Co-authored-by: Andrew Lock <[email protected]> Co-authored-by: Steven Bouwkamp <[email protected]>

veerbia and others added 30 commits November 7, 2024 15:28

switch .NET tracer to injecting both base64 & binary headers

4431a45

fix context propagation issues

8361b37

Merge branch 'master' into veerbia/base64migration

2abc2c9

Merge branch 'master' into veerbia/base64migration

703d9b5

Merge branch 'master' into veerbia/base64migration

5373d94

add newline

fd9cdc3

fix tryfromb64string

d9f8cc9

change default to true

f4f6af4

Merge branch 'master' into veerbia/base64migration

1c5bbd9

test fixes

b721d66

rm trail ws

111682b

rm trail ws

721a8fe

move private member to bottom

523a3cb

fix static before non static

6dfb7ff

fix tests

598fe6a

modify test approach

f7d971d

move to only checking b64

bd60e8b

Merge branch 'master' into veerbia/base64migration

ec78781

replace appendformat in ContextPropagation.cs

49d5f88

replace appendformat in tracer check

a9da986

rm semicolon

21577f7

rm log debug in DataStreamsManager.cs

d186dee

Extract Tracer.Instance to make testing easier

dc3d43c

Merge branch 'master' into veerbia/base64migration

166660c

try to resolve null ptr exception

1c8c29b

Update snapshots for SQS

bec777b

Remove DSM JSON injection

be18958

This appears to be handled already and doing it here just dupes the content. And it didn't add comma's between the JSON fields which ccause JSON deserialization errors. Additionally, it was re-encoding the Base64 again.

Set IsDataStreamsLegacyHeadersEnabled

47d92fe

so that it isn't disabled.

Remove duplicate PropagationKey header

f48d1dd

This is added already below, but initially it wasn't obivous because the legacy data streams was always false.

Update tracer/src/Datadog.Trace/ClrProfiler/AutoInstrumentation/AWS/S…

6e9062a

…hared/ContextPropagation.cs

rm immutabletracersettings.cs

381f958

veerbia requested review from a team as code owners December 16, 2024 18:24

veerbia requested a review from bouwkast December 16, 2024 18:25

Merge branch 'master' into veerbia/dsm-base64-migration

1f9df68

bouwkast approved these changes Dec 16, 2024

View reviewed changes

bouwkast requested review from bouwkast and lucaspimentel December 16, 2024 19:40

bouwkast reviewed Dec 16, 2024

View reviewed changes

Comment thread tracer/test/Datadog.Trace.ClrProfiler.IntegrationTests/DataStreamsMonitoringKafkaTests.cs

Comment thread tracer/test/Datadog.Trace.ClrProfiler.IntegrationTests/DataStreamsMonitoringKafkaTests.cs

veerbia and others added 5 commits December 16, 2024 16:11

fix rabbitmq tests

95a8651

fix kafka tests

6c686f4

revert kafka tests

1aaa925

fix kafka tests to involve minimal changes

9ef5bd8

Merge branch 'master' into veerbia/dsm-base64-migration

4c7e223

robertpi mentioned this pull request Dec 17, 2024

Switch .NET tracer to injecting both base64 & binary headers #6257

Closed

bouwkast self-requested a review December 17, 2024 13:22

bouwkast approved these changes Dec 17, 2024

View reviewed changes

lucaspimentel approved these changes Dec 17, 2024

View reviewed changes

veerbia merged commit 9870dbe into master Dec 17, 2024

veerbia deleted the veerbia/dsm-base64-migration branch December 17, 2024 16:40

github-actions Bot added this to the vNext-v3 milestone Dec 17, 2024

andrewlock added type:enhancement Improvement to an existing feature area:data-streams-monitoring labels Dec 18, 2024

Uh oh!

Conversation

veerbia commented Dec 16, 2024

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

Uh oh!

github-actions Bot commented Dec 16, 2024

Snapshots difference summary

Uh oh!

datadog-ddstaging Bot commented Dec 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Datadog Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

andrewlock commented Dec 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Execution-Time Benchmarks Report ⏱️

Uh oh!

andrewlock commented Dec 16, 2024

Benchmarks Report for tracer 🐌

Benchmark details

Raw results

Raw results

Raw results

More allocations ⚠️ in #6448

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #6448

Raw results

Slower ⚠️ in #6448

Raw results

Uh oh!

andrewlock commented Dec 16, 2024

Throughput/Crank Report ⚡

Uh oh!

robertpi commented Dec 17, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

datadog-ddstaging Bot commented Dec 16, 2024 •

edited

Loading

andrewlock commented Dec 16, 2024 •

edited

Loading