Verify version of glibc targeted in native tracer/profiler#6104
Conversation
Execution-Time Benchmarks Report ⏱️Execution-time results for samples comparing the following branches/commits: Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:
Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard. Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph). gantt
title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (6104) - mean (70ms) : 68, 73
. : milestone, 70,
master - mean (71ms) : 68, 74
. : milestone, 71,
section CallTarget+Inlining+NGEN
This PR (6104) - mean (1,107ms) : 1077, 1137
. : milestone, 1107,
master - mean (1,114ms) : 1089, 1139
. : milestone, 1114,
gantt
title Execution time (ms) FakeDbCommand (.NET Core 3.1)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (6104) - mean (109ms) : 106, 112
. : milestone, 109,
master - mean (110ms) : 107, 113
. : milestone, 110,
section CallTarget+Inlining+NGEN
This PR (6104) - mean (775ms) : 760, 790
. : milestone, 775,
master - mean (776ms) : 762, 790
. : milestone, 776,
gantt
title Execution time (ms) FakeDbCommand (.NET 6)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (6104) - mean (94ms) : 90, 97
. : milestone, 94,
master - mean (93ms) : 90, 95
. : milestone, 93,
section CallTarget+Inlining+NGEN
This PR (6104) - mean (729ms) : 709, 748
. : milestone, 729,
master - mean (730ms) : 717, 744
. : milestone, 730,
gantt
title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (6104) - mean (191ms) : 187, 195
. : milestone, 191,
master - mean (190ms) : 187, 194
. : milestone, 190,
section CallTarget+Inlining+NGEN
This PR (6104) - mean (1,215ms) : 1184, 1246
. : milestone, 1215,
master - mean (1,199ms) : 1180, 1218
. : milestone, 1199,
gantt
title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (6104) - mean (277ms) : 274, 281
. : milestone, 277,
master - mean (275ms) : 270, 279
. : milestone, 275,
section CallTarget+Inlining+NGEN
This PR (6104) - mean (945ms) : 925, 965
. : milestone, 945,
master - mean (939ms) : 921, 957
. : milestone, 939,
gantt
title Execution time (ms) HttpMessageHandler (.NET 6)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (6104) - mean (267ms) : 258, 277
. : milestone, 267,
master - mean (264ms) : 260, 267
. : milestone, 264,
section CallTarget+Inlining+NGEN
This PR (6104) - mean (933ms) : 905, 961
. : milestone, 933,
master - mean (960ms) : 851, 1070
. : milestone, 960,
|
Datadog ReportBranch report: ✅ 0 Failed, 369629 Passed, 2357 Skipped, 16h 55m 14.95s Total Time |
Throughput/Crank Report ⚡Throughput results for AspNetCoreSimpleController comparing the following branches/commits: Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red. Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards! gantt
title Throughput Linux x64 (Total requests)
dateFormat X
axisFormat %s
section Baseline
This PR (6104) (11.203M) : 0, 11203149
master (11.072M) : 0, 11071593
benchmarks/2.9.0 (11.081M) : 0, 11080577
section Automatic
This PR (6104) (7.334M) : 0, 7333864
master (7.300M) : 0, 7300144
benchmarks/2.9.0 (7.732M) : 0, 7732233
section Trace stats
master (7.747M) : 0, 7746563
section Manual
master (11.039M) : 0, 11038752
section Manual + Automatic
This PR (6104) (6.784M) : 0, 6784483
master (6.779M) : 0, 6779324
section DD_TRACE_ENABLED=0
master (10.270M) : 0, 10269736
gantt
title Throughput Linux arm64 (Total requests)
dateFormat X
axisFormat %s
section Baseline
This PR (6104) (9.584M) : 0, 9583994
master (9.551M) : 0, 9550662
benchmarks/2.9.0 (9.798M) : 0, 9798067
section Automatic
This PR (6104) (6.557M) : 0, 6557286
master (6.526M) : 0, 6526183
section Trace stats
master (6.825M) : 0, 6824918
section Manual
master (9.511M) : 0, 9510563
section Manual + Automatic
This PR (6104) (4.745M) : crit ,0, 4745497
master (6.042M) : 0, 6042167
section DD_TRACE_ENABLED=0
master (8.958M) : 0, 8958244
gantt
title Throughput Windows x64 (Total requests)
dateFormat X
axisFormat %s
section Baseline
This PR (6104) (10.100M) : 0, 10099991
master (10.136M) : 0, 10135995
benchmarks/2.9.0 (10.067M) : 0, 10067315
section Automatic
This PR (6104) (6.794M) : 0, 6793546
master (6.765M) : 0, 6765109
benchmarks/2.9.0 (7.552M) : 0, 7552193
section Trace stats
master (7.398M) : 0, 7398168
section Manual
master (10.039M) : 0, 10039465
section Manual + Automatic
This PR (6104) (6.138M) : 0, 6138373
master (6.241M) : 0, 6241272
section DD_TRACE_ENABLED=0
master (9.564M) : 0, 9564233
|
Benchmarks Report for tracer 🐌Benchmarks for #6104 compared to master:
The following thresholds were used for comparing the benchmark speeds:
Allocation changes below 0.5% are ignored. Benchmark detailsBenchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.ElasticsearchBenchmark - Faster 🎉 Same allocations ✔️
|
| Benchmark | base/diff | Base Median (ns) | Diff Median (ns) | Modality |
|---|---|---|---|---|
| Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearch‑net6.0 | 1.151 | 1,314.85 | 1,142.80 |
Raw results
| Branch | Method | Toolchain | Mean | StdError | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
|---|---|---|---|---|---|---|---|---|---|
| master | CallElasticsearch |
net6.0 | 1.32μs | 0.378ns | 1.42ns | 0.0139 | 0 | 0 | 976 B |
| master | CallElasticsearch |
netcoreapp3.1 | 1.55μs | 0.732ns | 2.74ns | 0.0131 | 0 | 0 | 976 B |
| master | CallElasticsearch |
net472 | 2.51μs | 1.37ns | 5.31ns | 0.157 | 0 | 0 | 995 B |
| master | CallElasticsearchAsync |
net6.0 | 1.26μs | 0.764ns | 2.86ns | 0.0132 | 0 | 0 | 952 B |
| master | CallElasticsearchAsync |
netcoreapp3.1 | 1.62μs | 0.796ns | 3.08ns | 0.0139 | 0 | 0 | 1.02 KB |
| master | CallElasticsearchAsync |
net472 | 2.58μs | 0.729ns | 2.73ns | 0.167 | 0 | 0 | 1.05 KB |
| #6104 | CallElasticsearch |
net6.0 | 1.14μs | 0.65ns | 2.43ns | 0.0136 | 0 | 0 | 976 B |
| #6104 | CallElasticsearch |
netcoreapp3.1 | 1.51μs | 1.67ns | 6.24ns | 0.0134 | 0 | 0 | 976 B |
| #6104 | CallElasticsearch |
net472 | 2.44μs | 1.64ns | 6.37ns | 0.157 | 0 | 0 | 995 B |
| #6104 | CallElasticsearchAsync |
net6.0 | 1.3μs | 0.635ns | 2.38ns | 0.0131 | 0 | 0 | 952 B |
| #6104 | CallElasticsearchAsync |
netcoreapp3.1 | 1.54μs | 1.33ns | 4.97ns | 0.0138 | 0 | 0 | 1.02 KB |
| #6104 | CallElasticsearchAsync |
net472 | 2.6μs | 1.95ns | 7.56ns | 0.167 | 0 | 0 | 1.05 KB |
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️
Raw results
| Branch | Method | Toolchain | Mean | StdError | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
|---|---|---|---|---|---|---|---|---|---|
| master | ExecuteAsync |
net6.0 | 1.33μs | 0.71ns | 2.75ns | 0.0135 | 0 | 0 | 952 B |
| master | ExecuteAsync |
netcoreapp3.1 | 1.61μs | 0.699ns | 2.71ns | 0.0128 | 0 | 0 | 952 B |
| master | ExecuteAsync |
net472 | 1.8μs | 0.943ns | 3.4ns | 0.145 | 0 | 0 | 915 B |
| #6104 | ExecuteAsync |
net6.0 | 1.24μs | 0.774ns | 3ns | 0.0135 | 0 | 0 | 952 B |
| #6104 | ExecuteAsync |
netcoreapp3.1 | 1.69μs | 1.61ns | 6.03ns | 0.0126 | 0 | 0 | 952 B |
| #6104 | ExecuteAsync |
net472 | 1.8μs | 0.43ns | 1.61ns | 0.145 | 0 | 0 | 915 B |
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️
Raw results
| Branch | Method | Toolchain | Mean | StdError | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
|---|---|---|---|---|---|---|---|---|---|
| master | SendAsync |
net6.0 | 4.37μs | 2.01ns | 7.78ns | 0.0305 | 0 | 0 | 2.22 KB |
| master | SendAsync |
netcoreapp3.1 | 5.15μs | 3.54ns | 13.7ns | 0.036 | 0 | 0 | 2.76 KB |
| master | SendAsync |
net472 | 7.73μs | 2.64ns | 10.2ns | 0.498 | 0 | 0 | 3.15 KB |
| #6104 | SendAsync |
net6.0 | 4.1μs | 1.88ns | 7.03ns | 0.0308 | 0 | 0 | 2.22 KB |
| #6104 | SendAsync |
netcoreapp3.1 | 5.06μs | 2.28ns | 8.2ns | 0.0352 | 0 | 0 | 2.76 KB |
| #6104 | SendAsync |
net472 | 7.94μs | 2.17ns | 8.4ns | 0.496 | 0 | 0 | 3.15 KB |
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️
Raw results
| Branch | Method | Toolchain | Mean | StdError | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
|---|---|---|---|---|---|---|---|---|---|
| master | EnrichedLog |
net6.0 | 1.52μs | 0.98ns | 3.8ns | 0.0233 | 0 | 0 | 1.64 KB |
| master | EnrichedLog |
netcoreapp3.1 | 2.2μs | 0.563ns | 2.11ns | 0.0221 | 0 | 0 | 1.64 KB |
| master | EnrichedLog |
net472 | 2.62μs | 1.16ns | 4.49ns | 0.249 | 0 | 0 | 1.57 KB |
| #6104 | EnrichedLog |
net6.0 | 1.46μs | 0.915ns | 3.54ns | 0.0233 | 0 | 0 | 1.64 KB |
| #6104 | EnrichedLog |
netcoreapp3.1 | 2.37μs | 2.4ns | 8.97ns | 0.0223 | 0 | 0 | 1.64 KB |
| #6104 | EnrichedLog |
net472 | 2.56μs | 1.03ns | 3.72ns | 0.249 | 0 | 0 | 1.57 KB |
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️
Raw results
| Branch | Method | Toolchain | Mean | StdError | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
|---|---|---|---|---|---|---|---|---|---|
| master | EnrichedLog |
net6.0 | 115μs | 188ns | 728ns | 0 | 0 | 0 | 4.28 KB |
| master | EnrichedLog |
netcoreapp3.1 | 118μs | 92.9ns | 335ns | 0 | 0 | 0 | 4.28 KB |
| master | EnrichedLog |
net472 | 147μs | 165ns | 641ns | 0.664 | 0.221 | 0 | 4.46 KB |
| #6104 | EnrichedLog |
net6.0 | 115μs | 112ns | 436ns | 0 | 0 | 0 | 4.28 KB |
| #6104 | EnrichedLog |
netcoreapp3.1 | 120μs | 187ns | 726ns | 0.0598 | 0 | 0 | 4.28 KB |
| #6104 | EnrichedLog |
net472 | 147μs | 67ns | 242ns | 0.658 | 0.219 | 0 | 4.46 KB |
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️
Raw results
| Branch | Method | Toolchain | Mean | StdError | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
|---|---|---|---|---|---|---|---|---|---|
| master | EnrichedLog |
net6.0 | 2.97μs | 0.945ns | 3.66ns | 0.0304 | 0 | 0 | 2.2 KB |
| master | EnrichedLog |
netcoreapp3.1 | 4.18μs | 1.67ns | 6.46ns | 0.0294 | 0 | 0 | 2.2 KB |
| master | EnrichedLog |
net472 | 4.86μs | 1.22ns | 4.58ns | 0.32 | 0 | 0 | 2.02 KB |
| #6104 | EnrichedLog |
net6.0 | 3.1μs | 1.21ns | 4.69ns | 0.0315 | 0 | 0 | 2.2 KB |
| #6104 | EnrichedLog |
netcoreapp3.1 | 4.12μs | 1.1ns | 4.1ns | 0.0289 | 0 | 0 | 2.2 KB |
| #6104 | EnrichedLog |
net472 | 4.88μs | 2.05ns | 7.67ns | 0.319 | 0 | 0 | 2.02 KB |
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️
Raw results
| Branch | Method | Toolchain | Mean | StdError | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
|---|---|---|---|---|---|---|---|---|---|
| master | SendReceive |
net6.0 | 1.3μs | 0.631ns | 2.36ns | 0.0163 | 0 | 0 | 1.14 KB |
| master | SendReceive |
netcoreapp3.1 | 1.78μs | 0.815ns | 3.16ns | 0.0154 | 0 | 0 | 1.14 KB |
| master | SendReceive |
net472 | 2.14μs | 1.5ns | 5.8ns | 0.183 | 0 | 0 | 1.16 KB |
| #6104 | SendReceive |
net6.0 | 1.34μs | 0.873ns | 3.38ns | 0.0161 | 0 | 0 | 1.14 KB |
| #6104 | SendReceive |
netcoreapp3.1 | 1.73μs | 0.902ns | 3.37ns | 0.015 | 0 | 0 | 1.14 KB |
| #6104 | SendReceive |
net472 | 2.02μs | 0.527ns | 1.97ns | 0.183 | 0.00101 | 0 | 1.16 KB |
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️
Raw results
| Branch | Method | Toolchain | Mean | StdError | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
|---|---|---|---|---|---|---|---|---|---|
| master | EnrichedLog |
net6.0 | 2.84μs | 0.863ns | 3.34ns | 0.0227 | 0 | 0 | 1.6 KB |
| master | EnrichedLog |
netcoreapp3.1 | 3.79μs | 12ns | 46.3ns | 0.0225 | 0 | 0 | 1.65 KB |
| master | EnrichedLog |
net472 | 4.38μs | 1.55ns | 6.01ns | 0.322 | 0 | 0 | 2.04 KB |
| #6104 | EnrichedLog |
net6.0 | 2.68μs | 1.16ns | 4.33ns | 0.0227 | 0 | 0 | 1.6 KB |
| #6104 | EnrichedLog |
netcoreapp3.1 | 3.88μs | 12.3ns | 47.7ns | 0.0212 | 0 | 0 | 1.65 KB |
| #6104 | EnrichedLog |
net472 | 4.3μs | 1.11ns | 4.16ns | 0.324 | 0 | 0 | 2.04 KB |
Benchmarks.Trace.SpanBenchmark - Slower ⚠️ Same allocations ✔️
Slower ⚠️ in #6104
Benchmark
diff/base
Base Median (ns)
Diff Median (ns)
Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑netcoreapp3.1
1.181
668.45
789.37
| Benchmark | diff/base | Base Median (ns) | Diff Median (ns) | Modality |
|---|---|---|---|---|
| Benchmarks.Trace.SpanBenchmark.StartFinishScope‑netcoreapp3.1 | 1.181 | 668.45 | 789.37 |
Raw results
| Branch | Method | Toolchain | Mean | StdError | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
|---|---|---|---|---|---|---|---|---|---|
| master | StartFinishSpan |
net6.0 | 403ns | 0.134ns | 0.484ns | 0.00809 | 0 | 0 | 576 B |
| master | StartFinishSpan |
netcoreapp3.1 | 633ns | 0.473ns | 1.77ns | 0.00791 | 0 | 0 | 576 B |
| master | StartFinishSpan |
net472 | 690ns | 0.27ns | 0.936ns | 0.0916 | 0 | 0 | 578 B |
| master | StartFinishScope |
net6.0 | 484ns | 0.31ns | 1.2ns | 0.00989 | 0 | 0 | 696 B |
| master | StartFinishScope |
netcoreapp3.1 | 668ns | 0.711ns | 2.76ns | 0.00939 | 0 | 0 | 696 B |
| master | StartFinishScope |
net472 | 857ns | 0.218ns | 0.816ns | 0.105 | 0 | 0 | 658 B |
| #6104 | StartFinishSpan |
net6.0 | 396ns | 0.198ns | 0.766ns | 0.00807 | 0 | 0 | 576 B |
| #6104 | StartFinishSpan |
netcoreapp3.1 | 613ns | 1.79ns | 6.91ns | 0.00801 | 0 | 0 | 576 B |
| #6104 | StartFinishSpan |
net472 | 675ns | 0.549ns | 2.13ns | 0.0917 | 0 | 0 | 578 B |
| #6104 | StartFinishScope |
net6.0 | 475ns | 0.244ns | 0.945ns | 0.00979 | 0 | 0 | 696 B |
| #6104 | StartFinishScope |
netcoreapp3.1 | 789ns | 0.39ns | 1.51ns | 0.00937 | 0 | 0 | 696 B |
| #6104 | StartFinishScope |
net472 | 846ns | 0.546ns | 2.04ns | 0.104 | 0 | 0 | 658 B |
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️
Raw results
| Branch | Method | Toolchain | Mean | StdError | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
|---|---|---|---|---|---|---|---|---|---|
| master | RunOnMethodBegin |
net6.0 | 664ns | 0.288ns | 1.12ns | 0.00968 | 0 | 0 | 696 B |
| master | RunOnMethodBegin |
netcoreapp3.1 | 874ns | 0.535ns | 2ns | 0.00922 | 0 | 0 | 696 B |
| master | RunOnMethodBegin |
net472 | 1.11μs | 0.391ns | 1.51ns | 0.104 | 0 | 0 | 658 B |
| #6104 | RunOnMethodBegin |
net6.0 | 649ns | 0.363ns | 1.41ns | 0.0097 | 0 | 0 | 696 B |
| #6104 | RunOnMethodBegin |
netcoreapp3.1 | 958ns | 0.676ns | 2.62ns | 0.00922 | 0 | 0 | 696 B |
| #6104 | RunOnMethodBegin |
net472 | 1.12μs | 0.546ns | 2.04ns | 0.104 | 0 | 0 | 658 B |
…ufficient" This reverts commit 2980c8b.
Summary of changes
Bail out of tracing if glibc < 2.17 on x64 or < 2.23 on arm64Reason for change
The native tracer and profiler are compiled against glibc/musl and as such have minimum versions they require - the musl version is generally not an issue, but glibc can be. The required glibc version is generally controlled by the system on which it's built. For our native build we have
2.172.28If we run on a glibc version that is too low, then we don't fail to load because we use
RTLD_LAZYlazy binding (instead ofRTLD_NOW):Ultimately, this means that we might load but crash later when a symbol that is not available is required.
To guard against that, we could add checks for the glibc version into the native loader, and bail out if we detect a low glibc version.
Unfortunately, things get tricky with arm64, where we technically require
2.28. .NET 5-8 only require 2.23, and the popular Amazon Linux 2 distro only requires 2.26. So if we were to block <2.28 that could be a significant problem.The reason this isn't a problem today is that while the profiler requires
2.28, it's not currently supported on arm64. The native tracer, meanwhile, currently only requires2.18.The longer term solution is to fix our min glibc dependency to match .NET's
2.23requirement, but that requires a significant investment, likely using a rootfs build technique like the .NET runtime does or something like https://github.com/wheybags/glibc_version_header.In the meantime, we will bail out on the following:
2.172.23. Note not2.28This will be fine as long as
2.23.This PR enforces the former condition by verifying the required glibc version in the native binary. We cannot enforce the latter currently, so will need to bear this in mind when we enable arm64 profiling.Given that those thresholds are also the minimum for .NET, we will not verify them in the native loader directly, instead we will rely on tests to assert we don't exceed 2.23 in the native tracer.Implementation details
The glibc check in the native loader uses the same technique as the SSI injector in https://github.com/DataDog/auto_inject/pull/433 (which is compatible with musl, and even busybox) to retrieve the glibc version.To retrieve the actual glibc symbols used by the native library, it's effectively running the following command:
nm --with-symbol-versions -D <TRACERFILE> | grep GLIBC | sed 's/^.*GLIBC_//' | sort -u -t "." -k1,1nr -k2,2nr -k3,3nr | head -1Which is obviously easier to follow in C# 😉 The
nmcommand returns a list of symbols, similar to this:We just grab the suffix and find the highest: that's our real minimum glibc version.
One complication is that the version of
nmin centos7 is too old, so it doesn't work 🙄 So we have to perform the check on the debian distro.Test coverage
I've tested this all manually, and I'll run against the full smoke tests to be sure. But it's kind of hard to test in practice, given we're basically limiting our self to .NET's glibc version now.
Other details
Given that final statement, a reasonable question is "why bother then"? Which is reasonable - I started down this route precisely because we actually require
2.28. So we can always park this if we want.EDIT: Yeah, decided to drop the check in the native loader in the end