Skip to content

Add IpAddressObfuscationUtil for use with client-side-stats#8418

Merged
andrewlock merged 8 commits into
masterfrom
andrew/client-side-stats/ip-obfuscation
Apr 16, 2026
Merged

Add IpAddressObfuscationUtil for use with client-side-stats#8418
andrewlock merged 8 commits into
masterfrom
andrew/client-side-stats/ip-obfuscation

Conversation

@andrewlock

@andrewlock andrewlock commented Apr 7, 2026

Copy link
Copy Markdown
Member

Summary of changes

Adds the IpAddressObfuscationUtil helper for use by client-side stats

Reason for change

It's not part of the RFC, but the agent does quantization of IPaddresses, which could appear in peer tags (which are used for stats calculations). Without this quantization, we would end up with an explosion of buckets

Implementation details

Used 🤖 to port the implementation from the Go agent, including the tests. Then iterated on it repeatedly to try to bring the allocations down as low as possible.

⚠️ There's still one code path which is kinda nasty where we try to parse if the tag is a valid IP v6, as it causes an allocation on .NET Framework, but I couldn't find a way to avoid that. Given all the allocations we're going to get anyway, I think we will just have to accept it for practical reasons, unless other people feel strongly

Test coverage

Grabbed the quantization unit tests from the agent

Other details

Pre-requisite for client-side-stats improvements. Part of a stack:

@andrewlock
andrewlock requested a review from a team as a code owner April 7, 2026 12:26
@andrewlock andrewlock added the area:tracer The core tracer library (Datadog.Trace, does not include OpenTracing, native code, or integrations) label Apr 7, 2026
chatgpt-codex-connector[bot]
chatgpt-codex-connector Bot reviewed Apr 7, 2026
View reviewed changes

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f00d1b65a5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread tracer/src/Datadog.Trace/Processors/IpAddressObfuscationUtil.cs Outdated
@pr-commenter

pr-commenter Bot commented Apr 7, 2026 •
edited
Loading

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-04-14 11:13:59

Comparing candidate commit f0895ac in PR branch andrew/client-side-stats/ip-obfuscation with baseline commit 7f212c3 in branch master.

Found 0 performance improvements and 1 performance regressions! Performance is the same for 26 metrics, 0 unstable metrics, 87 known flaky benchmarks.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:Benchmarks.Trace.HttpClientBenchmark.SendAsync net472

  • 🟥 throughput [-6109.101op/s; -5733.600op/s] or [-6.974%; -6.545%]

Known flaky benchmarks

These benchmarks are marked as flaky and will not trigger a failure. Modify FLAKY_BENCHMARKS_REGEX to control which benchmarks are marked as flaky.

scenario:Benchmarks.Trace.ActivityBenchmark.StartStopWithChild net472

  • ignore allocated_mem [+0 bytes; +0 bytes] or [-0.008%; +0.005%]
  • ignore execution_time [-1022.993µs; -102.866µs] or [-0.508%; -0.051%]
  • ignore throughput [+1018.172op/s; +1466.452op/s] or [+1.207%; +1.739%]

scenario:Benchmarks.Trace.ActivityBenchmark.StartStopWithChild net6.0

  • ignore allocated_mem [+0 bytes; +0 bytes] or [-0.006%; +0.007%]
  • ignore execution_time [-1733.074µs; +1767.503µs] or [-0.865%; +0.882%]
  • 🟩 throughput [+10224.392op/s; +12479.264op/s] or [+8.594%; +10.489%]

scenario:Benchmarks.Trace.ActivityBenchmark.StartStopWithChild netcoreapp3.1

  • ignore allocated_mem [+0 bytes; +0 bytes] or [-0.003%; +0.007%]
  • ignore execution_time [-202.551µs; +2016.108µs] or [-0.102%; +1.014%]
  • ignore throughput [-378.586op/s; +836.605op/s] or [-0.385%; +0.851%]

scenario:Benchmarks.Trace.AgentWriterBenchmark.WriteAndFlushEnrichedTraces net472

  • ignore allocated_mem [-20 bytes; -19 bytes] or [-0.613%; -0.600%]
  • 🟥 execution_time [+307.353ms; +311.788ms] or [+152.519%; +154.720%]
  • ignore throughput [-3.917op/s; +0.494op/s] or [-0.705%; +0.089%]

scenario:Benchmarks.Trace.AgentWriterBenchmark.WriteAndFlushEnrichedTraces net6.0

  • ignore allocated_mem [+0 bytes; +0 bytes] or [-0.009%; +0.002%]
  • 🟥 execution_time [+376.277ms; +379.297ms] or [+297.282%; +299.668%]
  • ignore throughput [+14.394op/s; +16.699op/s] or [+1.898%; +2.202%]

scenario:Benchmarks.Trace.AgentWriterBenchmark.WriteAndFlushEnrichedTraces netcoreapp3.1

  • ignore allocated_mem [+0 bytes; +0 bytes] or [-0.009%; +0.002%]
  • 🟥 execution_time [+393.083ms; +396.336ms] or [+347.864%; +350.743%]
  • ignore throughput [-3.798op/s; -0.395op/s] or [-0.536%; -0.056%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleMoreComplexBody net472

  • 🟥 allocated_mem [+1.308KB; +1.308KB] or [+27.529%; +27.541%]
  • ignore execution_time [-234.041µs; +386.993µs] or [-0.117%; +0.193%]
  • ignore throughput [-3842.295op/s; -3463.534op/s] or [-2.990%; -2.695%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleMoreComplexBody net6.0

  • 🟥 allocated_mem [+471 bytes; +472 bytes] or [+9.977%; +9.987%]
  • 🟩 execution_time [-15.998ms; -11.793ms] or [-7.471%; -5.508%]
  • ignore throughput [+4543.339op/s; +7323.268op/s] or [+3.316%; +5.346%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleMoreComplexBody netcoreapp3.1

  • 🟥 allocated_mem [+1.272KB; +1.272KB] or [+27.502%; +27.510%]
  • ignore execution_time [-13.529ms; -9.332ms] or [-6.442%; -4.444%]
  • ignore throughput [+130.913op/s; +2424.023op/s] or [+0.118%; +2.192%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody net472

  • 🟥 allocated_mem [+1.307KB; +1.307KB] or [+105.746%; +105.759%]
  • ignore execution_time [-867.476µs; -318.891µs] or [-0.432%; -0.159%]
  • 🟥 throughput [-246939.048op/s; -243902.533op/s] or [-25.214%; -24.904%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody net6.0

  • 🟥 allocated_mem [+471 bytes; +472 bytes] or [+38.558%; +38.566%]
  • 🟩 execution_time [-26.586ms; -21.731ms] or [-11.856%; -9.691%]
  • 🟥 throughput [-81778.464op/s; -58175.951op/s] or [-8.737%; -6.215%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody netcoreapp3.1

  • 🟥 allocated_mem [+1.272KB; +1.272KB] or [+105.292%; +105.304%]
  • ignore execution_time [-3.746ms; +1.707ms] or [-1.870%; +0.852%]
  • 🟥 throughput [-131040.817op/s; -113245.489op/s] or [-18.828%; -16.271%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorMoreComplexBody net472

  • ignore allocated_mem [+0 bytes; +0 bytes] or [-0.007%; +0.003%]
  • ignore execution_time [-1024.339µs; +37.564µs] or [-0.511%; +0.019%]
  • ignore throughput [-622.366op/s; +166.112op/s] or [-0.419%; +0.112%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorMoreComplexBody net6.0

  • ignore allocated_mem [+0 bytes; +0 bytes] or [-0.004%; +0.003%]
  • ignore execution_time [-1497.665µs; +2053.198µs] or [-0.756%; +1.036%]
  • 🟩 throughput [+11863.372op/s; +14874.800op/s] or [+7.548%; +9.465%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorMoreComplexBody netcoreapp3.1

  • ignore allocated_mem [+0 bytes; +0 bytes] or [-0.007%; +0.003%]
  • ignore execution_time [+2.758ms; +6.765ms] or [+1.406%; +3.449%]
  • 🟩 throughput [+7629.570op/s; +10300.468op/s] or [+6.078%; +8.206%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody net472

  • ignore allocated_mem [+0 bytes; +0 bytes] or [-0.008%; +0.004%]
  • ignore execution_time [-244.394µs; +50.425µs] or [-0.122%; +0.025%]
  • ignore throughput [+92803.950op/s; +102203.046op/s] or [+2.823%; +3.109%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody net6.0

  • ignore allocated_mem [+0 bytes; +0 bytes] or [-0.006%; +0.008%]
  • ignore execution_time [-2.643ms; -1.903ms] or [-1.307%; -0.941%]
  • 🟩 throughput [+461759.836op/s; +478761.666op/s] or [+15.397%; +15.964%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody netcoreapp3.1

  • ignore allocated_mem [+0 bytes; +0 bytes] or [-0.004%; +0.004%]
  • 🟩 execution_time [-19.788ms; -15.336ms] or [-9.122%; -7.069%]
  • 🟩 throughput [+234542.910op/s; +288973.662op/s] or [+9.310%; +11.470%]

scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeArgs net472

  • ignore allocated_mem [+0 bytes; +2 bytes] or [-0.001%; +0.007%]
  • 🟥 execution_time [+299.039ms; +300.279ms] or [+149.420%; +150.039%]
  • ignore throughput [+136.794op/s; +163.545op/s] or [+1.511%; +1.806%]

scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeArgs net6.0

  • ignore allocated_mem [-1 bytes; +2 bytes] or [-0.004%; +0.008%]
  • 🟥 execution_time [+300.860ms; +304.267ms] or [+151.724%; +153.443%]
  • ignore throughput [+309.745op/s; +523.955op/s] or [+2.369%; +4.007%]

scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeArgs netcoreapp3.1

  • ignore allocated_mem [-1 bytes; +2 bytes] or [-0.004%; +0.008%]
  • 🟥 execution_time [+300.238ms; +302.707ms] or [+151.236%; +152.480%]
  • ignore throughput [+110.526op/s; +236.606op/s] or [+1.067%; +2.284%]

scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeLegacyArgs net472

  • ignore allocated_mem [+2 bytes; +3 bytes] or [+0.137%; +0.150%]
  • 🟥 execution_time [+297.457ms; +298.143ms] or [+146.099%; +146.436%]
  • ignore throughput [+4.632op/s; +14.039op/s] or [+0.123%; +0.372%]

scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeLegacyArgs net6.0

  • ignore allocated_mem [+0 bytes; +0 bytes] or [-0.004%; +0.009%]
  • 🟥 execution_time [+295.231ms; +297.865ms] or [+144.328%; +145.615%]
  • ignore throughput [+50.917op/s; +93.737op/s] or [+0.740%; +1.362%]

scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeLegacyArgs netcoreapp3.1

  • ignore allocated_mem [+0 bytes; +0 bytes] or [-0.004%; +0.009%]
  • 🟥 execution_time [+302.775ms; +304.902ms] or [+151.326%; +152.390%]
  • ignore throughput [-55.517op/s; -4.871op/s] or [-1.102%; -0.097%]

scenario:Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmark net472

  • ignore allocated_mem [+0 bytes; +0 bytes] or [+nan%; +nan%]
  • ignore execution_time [+2.492µs; +6.126µs] or [+0.512%; +1.258%]
  • ignore throughput [-25.288op/s; -10.290op/s] or [-1.232%; -0.501%]

scenario:Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmark net6.0

  • ignore allocated_mem [+0 bytes; +0 bytes] or [-0.000%; +0.010%]
  • ignore execution_time [+12.857µs; +39.467µs] or [+2.949%; +9.052%]
  • ignore throughput [-196.048op/s; -76.304op/s] or [-8.523%; -3.317%]

scenario:Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmark netcoreapp3.1

  • ignore allocated_mem [+0 bytes; +0 bytes] or [-0.003%; +0.003%]
  • ignore execution_time [+6.518µs; +28.450µs] or [+1.396%; +6.096%]
  • ignore throughput [-141.288op/s; -60.661op/s] or [-6.522%; -2.800%]

scenario:Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmarkWithAttack net472

  • ignore allocated_mem [+0 bytes; +0 bytes] or [+nan%; +nan%]
  • ignore execution_time [-6.988µs; -3.190µs] or [-1.887%; -0.861%]
  • ignore throughput [+24.143op/s; +51.589op/s] or [+0.894%; +1.911%]

scenario:Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmarkWithAttack net6.0

  • ignore allocated_mem [+0 bytes; +0 bytes] or [-0.003%; +0.007%]
  • 🟥 execution_time [+23.126µs; +46.793µs] or [+7.383%; +14.939%]
  • 🟥 throughput [-434.424op/s; -235.327op/s] or [-13.542%; -7.336%]

scenario:Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmarkWithAttack netcoreapp3.1

  • ignore allocated_mem [+0 bytes; +0 bytes] or [-0.003%; +0.003%]
  • ignore execution_time [-15.431µs; +6.935µs] or [-4.221%; +1.897%]
  • ignore throughput [-85.195op/s; +48.259op/s] or [-3.057%; +1.732%]

scenario:Benchmarks.Trace.AspNetCoreBenchmark.SendRequest net472

  • ignore allocated_mem [+0 bytes; +0 bytes] or [+nan%; +nan%]
  • 🟥 execution_time [+299.782ms; +300.426ms] or [+149.622%; +149.943%]
  • ignore throughput [-1297035.760op/s; -734239.997op/s] or [-0.650%; -0.368%]

scenario:Benchmarks.Trace.AspNetCoreBenchmark.SendRequest net6.0

  • ignore allocated_mem [+106 bytes; +108 bytes] or [+0.592%; +0.603%]
  • 🟥 execution_time [+413.234ms; +417.877ms] or [+448.996%; +454.041%]
  • 🟩 throughput [+1103.804op/s; +1232.392op/s] or [+9.070%; +10.127%]

scenario:Benchmarks.Trace.AspNetCoreBenchmark.SendRequest netcoreapp3.1

  • ignore allocated_mem [+21 bytes; +23 bytes] or [+0.103%; +0.114%]
  • 🟥 execution_time [+363.132ms; +368.298ms] or [+275.723%; +279.645%]
  • 🟩 throughput [+621.507op/s; +819.346op/s] or [+6.017%; +7.932%]

scenario:Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces net472

  • 🟥 allocated_mem [+2.837KB; +2.841KB] or [+5.040%; +5.048%]
  • unstable execution_time [+398.194ms; +469.286ms] or [+183.086%; +215.773%]
  • 🟥 throughput [-559.660op/s; -502.627op/s] or [-50.711%; -45.543%]

scenario:Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces net6.0

  • ignore allocated_mem [-1.268KB; -1.266KB] or [-2.991%; -2.985%]
  • unstable execution_time [+182.035ms; +316.986ms] or [+77.576%; +135.086%]
  • 🟥 throughput [-744.292op/s; -660.865op/s] or [-49.644%; -44.080%]

scenario:Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces netcoreapp3.1

  • ignore allocated_mem [+786 bytes; +789 bytes] or [+1.858%; +1.865%]
  • unstable execution_time [+309.988ms; +335.563ms] or [+185.409%; +200.705%]
  • 🟥 throughput [-411.350op/s; -375.002op/s] or [-28.642%; -26.111%]

scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSlice net472

  • ignore allocated_mem [+0 bytes; +0 bytes] or [+nan%; +nan%]
  • ignore execution_time [-68.949µs; -55.049µs] or [-3.470%; -2.770%]
  • ignore throughput [+14.525op/s; +18.205op/s] or [+2.886%; +3.617%]

scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSlice net6.0

  • ignore allocated_mem [+0 bytes; +0 bytes] or [+nan%; +nan%]
  • 🟥 execution_time [+97.971µs; +121.480µs] or [+6.730%; +8.345%]
  • 🟥 throughput [-52.026op/s; -42.556op/s] or [-7.573%; -6.195%]

scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSlice netcoreapp3.1

  • ignore allocated_mem [+0 bytes; +0 bytes] or [+nan%; +nan%]
  • ignore execution_time [-170.482µs; -84.398µs] or [-5.931%; -2.936%]
  • ignore throughput [+10.853op/s; +28.508op/s] or [+3.120%; +8.194%]

scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSliceWithPool net472

  • ignore allocated_mem [+0 bytes; +0 bytes] or [+nan%; +nan%]
  • ignore execution_time [-18.525µs; -11.990µs] or [-1.600%; -1.036%]
  • ignore throughput [+9.207op/s; +14.135op/s] or [+1.066%; +1.637%]

scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSliceWithPool net6.0

  • ignore allocated_mem [+0 bytes; +0 bytes] or [+nan%; +nan%]
  • ignore execution_time [-55.087µs; -44.582µs] or [-5.109%; -4.135%]
  • ignore throughput [+40.650op/s; +50.250op/s] or [+4.383%; +5.418%]

scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSliceWithPool netcoreapp3.1

  • ignore allocated_mem [+0 bytes; +0 bytes] or [+nan%; +nan%]
  • ignore execution_time [+0.317µs; +8.618µs] or [+0.017%; +0.462%]
  • ignore throughput [-2.456op/s; -0.074op/s] or [-0.458%; -0.014%]

scenario:Benchmarks.Trace.CharSliceBenchmark.OriginalCharSlice net472

  • ignore allocated_mem [-43 bytes; +21 bytes] or [-0.007%; +0.003%]
  • ignore execution_time [+52.638µs; +72.713µs] or [+2.056%; +2.840%]
  • ignore throughput [-10.645op/s; -7.745op/s] or [-2.725%; -1.983%]

scenario:Benchmarks.Trace.CharSliceBenchmark.OriginalCharSlice net6.0

  • ignore allocated_mem [-38 bytes; +46 bytes] or [-0.006%; +0.007%]
  • ignore execution_time [-135.827µs; -92.450µs] or [-6.881%; -4.683%]
  • 🟩 throughput [+27.010op/s; +38.625op/s] or [+5.332%; +7.625%]

scenario:Benchmarks.Trace.CharSliceBenchmark.OriginalCharSlice netcoreapp3.1

  • ignore allocated_mem [-42 bytes; +23 bytes] or [-0.007%; +0.004%]
  • ignore execution_time [-129.684µs; -88.744µs] or [-3.289%; -2.250%]
  • ignore throughput [+6.099op/s; +8.702op/s] or [+2.405%; +3.432%]

scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearch net472

  • ignore allocated_mem [+0 bytes; +0 bytes] or [-0.001%; +0.008%]
  • 🟥 execution_time [+303.690ms; +305.196ms] or [+152.932%; +153.691%]
  • ignore throughput [+9269.549op/s; +10765.630op/s] or [+2.983%; +3.464%]

scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearch net6.0

  • ignore allocated_mem [+0 bytes; +0 bytes] or [-0.006%; +0.004%]
  • 🟥 execution_time [+301.307ms; +303.233ms] or [+150.986%; +151.951%]
  • 🟩 throughput [+40106.416op/s; +50768.732op/s] or [+6.323%; +8.004%]

scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearch netcoreapp3.1

  • ignore allocated_mem [+0 bytes; +0 bytes] or [-0.006%; +0.004%]
  • 🟥 execution_time [+302.627ms; +305.926ms] or [+152.027%; +153.684%]
  • ignore throughput [+13102.844op/s; +21597.940op/s] or [+2.760%; +4.550%]

scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearchAsync net472

  • ignore allocated_mem [+0 bytes; +0 bytes] or [-0.008%; +0.004%]
  • 🟥 execution_time [+301.117ms; +302.753ms] or [+151.211%; +152.033%]
  • ignore throughput [+2813.174op/s; +5765.684op/s] or [+0.942%; +1.932%]

scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearchAsync net6.0

  • ignore allocated_mem [+0 bytes; +0 bytes] or [-0.007%; +0.006%]
  • 🟥 execution_time [+299.286ms; +301.753ms] or [+147.984%; +149.204%]
  • 🟩 throughput [+31746.553op/s; +38521.131op/s] or [+5.115%; +6.207%]

scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearchAsync netcoreapp3.1

  • ignore allocated_mem [+0 bytes; +0 bytes] or [-0.006%; +0.004%]
  • 🟥 execution_time [+303.250ms; +306.964ms] or [+153.700%; +155.583%]
  • ignore throughput [+1977.243op/s; +10092.775op/s] or [+0.427%; +2.179%]

scenario:Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync net472

  • ignore allocated_mem [+0 bytes; +1 bytes] or [+0.108%; +0.119%]
  • 🟥 execution_time [+302.585ms; +304.397ms] or [+151.870%; +152.780%]
  • ignore throughput [-5764.678op/s; -3299.913op/s] or [-1.495%; -0.856%]

scenario:Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync net6.0

  • ignore allocated_mem [+0 bytes; +0 bytes] or [-0.006%; +0.006%]
  • 🟥 execution_time [+300.773ms; +302.646ms] or [+149.908%; +150.841%]
  • 🟩 throughput [+56929.558op/s; +62001.381op/s] or [+11.304%; +12.311%]

scenario:Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync netcoreapp3.1

  • ignore allocated_mem [+0 bytes; +0 bytes] or [-0.006%; +0.006%]
  • 🟥 execution_time [+300.305ms; +302.891ms] or [+149.399%; +150.686%]
  • ignore throughput [-11610.758op/s; -6387.207op/s] or [-2.748%; -1.512%]

scenario:Benchmarks.Trace.ILoggerBenchmark.EnrichedLog net472

  • ignore allocated_mem [+0 bytes; +0 bytes] or [-0.003%; +0.006%]
  • ignore execution_time [-1266.022µs; -427.106µs] or [-0.629%; -0.212%]
  • ignore throughput [+443.589op/s; +1571.551op/s] or [+0.178%; +0.632%]

scenario:Benchmarks.Trace.ILoggerBenchmark.EnrichedLog net6.0

  • ignore allocated_mem [+0 bytes; +0 bytes] or [-0.005%; +0.003%]
  • 🟩 execution_time [-15.665ms; -11.943ms] or [-7.284%; -5.554%]
  • 🟩 throughput [+19040.543op/s; +25941.089op/s] or [+5.223%; +7.116%]

scenario:Benchmarks.Trace.ILoggerBenchmark.EnrichedLog netcoreapp3.1

  • ignore allocated_mem [+0 bytes; +0 bytes] or [-0.004%; +0.008%]
  • ignore execution_time [-0.249ms; +3.705ms] or [-0.125%; +1.858%]
  • ignore throughput [+2062.621op/s; +7822.005op/s] or [+0.753%; +2.855%]

scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark net472

  • ignore allocated_mem [-4.459KB; -4.431KB] or [-1.623%; -1.613%]
  • unstable execution_time [+12.790µs; +53.999µs] or [+3.159%; +13.338%]
  • ignore throughput [-279.430op/s; -72.409op/s] or [-11.245%; -2.914%]

scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark net6.0

  • 🟩 allocated_mem [-17.954KB; -17.933KB] or [-6.549%; -6.542%]
  • unstable execution_time [-35.497µs; +17.795µs] or [-7.016%; +3.517%]
  • ignore throughput [-56.623op/s; +128.166op/s] or [-2.826%; +6.396%]

scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark netcoreapp3.1

  • 🟩 allocated_mem [-18.863KB; -18.846KB] or [-6.876%; -6.870%]
  • ignore execution_time [-51.122µs; +5.763µs] or [-8.859%; +0.999%]
  • ignore throughput [-3.607op/s; +151.514op/s] or [-0.206%; +8.656%]

scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark net472

  • ignore allocated_mem [-2 bytes; +2 bytes] or [-0.005%; +0.006%]
  • ignore execution_time [-289.280ns; +1309.947ns] or [-0.501%; +2.269%]
  • ignore throughput [-349.819op/s; +99.127op/s] or [-2.019%; +0.572%]

scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark net6.0

  • ignore allocated_mem [-4 bytes; +0 bytes] or [-0.010%; -0.001%]
  • 🟥 execution_time [+6.598µs; +10.703µs] or [+15.596%; +25.298%]
  • 🟥 throughput [-4807.519op/s; -3018.036op/s] or [-20.238%; -12.705%]

scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark netcoreapp3.1

  • ignore allocated_mem [-1 bytes; +1 bytes] or [-0.002%; +0.002%]
  • unstable execution_time [-15.476µs; -8.468µs] or [-24.011%; -13.137%]
  • 🟩 throughput [+2263.911op/s; +3737.087op/s] or [+13.890%; +22.928%]

scenario:Benchmarks.Trace.Log4netBenchmark.EnrichedLog net472

  • ignore allocated_mem [+2 bytes; +3 bytes] or [+0.061%; +0.072%]
  • 🟥 execution_time [+303.103ms; +304.127ms] or [+153.205%; +153.723%]
  • ignore throughput [-151.889op/s; -131.296op/s] or [-2.538%; -2.194%]

scenario:Benchmarks.Trace.Log4netBenchmark.EnrichedLog net6.0

  • ignore allocated_mem [-1 bytes; +0 bytes] or [-0.027%; -0.017%]
  • 🟥 execution_time [+303.788ms; +306.626ms] or [+154.627%; +156.071%]
  • ignore throughput [-96.305op/s; -19.400op/s] or [-1.194%; -0.241%]

scenario:Benchmarks.Trace.Log4netBenchmark.EnrichedLog netcoreapp3.1

  • ignore allocated_mem [-1 bytes; +0 bytes] or [-0.027%; -0.017%]
  • 🟥 execution_time [+298.977ms; +301.552ms] or [+149.675%; +150.964%]
  • ignore throughput [-141.933op/s; -77.157op/s] or [-1.808%; -0.983%]

scenario:Benchmarks.Trace.RedisBenchmark.SendReceive net472

  • ignore allocated_mem [+0 bytes; +0 bytes] or [-0.005%; +0.005%]
  • ignore execution_time [-1022.836µs; -274.227µs] or [-0.510%; -0.137%]
  • ignore throughput [-17989.453op/s; -16715.962op/s] or [-4.980%; -4.627%]

scenario:Benchmarks.Trace.RedisBenchmark.SendReceive net6.0

  • ignore allocated_mem [+0 bytes; +0 bytes] or [-0.004%; +0.007%]
  • ignore execution_time [-714.644µs; -55.564µs] or [-0.357%; -0.028%]
  • 🟩 throughput [+48466.445op/s; +52653.280op/s] or [+9.174%; +9.966%]

scenario:Benchmarks.Trace.RedisBenchmark.SendReceive netcoreapp3.1

  • ignore allocated_mem [+0 bytes; +0 bytes] or [-0.004%; +0.006%]
  • ignore execution_time [+1.754ms; +5.448ms] or [+0.889%; +2.762%]
  • ignore throughput [-4453.945op/s; +4306.901op/s] or [-1.054%; +1.019%]

scenario:Benchmarks.Trace.SerilogBenchmark.EnrichedLog net472

  • ignore allocated_mem [+0 bytes; +0 bytes] or [-0.005%; +0.006%]
  • 🟥 execution_time [+298.510ms; +300.275ms] or [+148.780%; +149.660%]
  • ignore throughput [-3040.803op/s; -1466.635op/s] or [-2.008%; -0.968%]

scenario:Benchmarks.Trace.SerilogBenchmark.EnrichedLog net6.0

  • ignore allocated_mem [+0 bytes; +0 bytes] or [+0.000%; +0.009%]
  • 🟥 execution_time [+302.021ms; +303.512ms] or [+151.661%; +152.409%]
  • ignore throughput [+4376.368op/s; +6048.177op/s] or [+1.903%; +2.630%]

scenario:Benchmarks.Trace.SerilogBenchmark.EnrichedLog netcoreapp3.1

  • ignore allocated_mem [+0 bytes; +0 bytes] or [-0.004%; +0.003%]
  • 🟥 execution_time [+303.117ms; +305.862ms] or [+153.721%; +155.114%]
  • ignore throughput [+2505.661op/s; +4347.566op/s] or [+1.411%; +2.449%]

scenario:Benchmarks.Trace.SingleSpanAspNetCoreBenchmark.SingleSpanAspNetCore net472

  • ignore allocated_mem [+0 bytes; +0 bytes] or [+nan%; +nan%]
  • 🟥 execution_time [+299.834ms; +300.480ms] or [+149.559%; +149.881%]
  • 🟩 throughput [+61334972.972op/s; +61570095.875op/s] or [+44.668%; +44.839%]

scenario:Benchmarks.Trace.SingleSpanAspNetCoreBenchmark.SingleSpanAspNetCore net6.0

  • ignore allocated_mem [+66 bytes; +68 bytes] or [+0.388%; +0.398%]
  • unstable execution_time [+382.161ms; +415.868ms] or [+475.285%; +517.205%]
  • 🟩 throughput [+1031.176op/s; +1207.663op/s] or [+7.972%; +9.336%]

scenario:Benchmarks.Trace.SingleSpanAspNetCoreBenchmark.SingleSpanAspNetCore netcoreapp3.1

  • ignore allocated_mem [+0 bytes; +0 bytes] or [+nan%; +nan%]
  • 🟥 execution_time [+299.527ms; +300.433ms] or [+149.397%; +149.849%]
  • ignore throughput [+1749384.016op/s; +2686803.334op/s] or [+0.775%; +1.190%]

scenario:Benchmarks.Trace.SpanBenchmark.StartFinishScope net472

  • ignore allocated_mem [+0 bytes; +0 bytes] or [-0.005%; +0.006%]
  • ignore execution_time [-648.959µs; +862.516µs] or [-0.325%; +0.431%]
  • ignore throughput [-20432.452op/s; -14783.050op/s] or [-2.280%; -1.650%]

scenario:Benchmarks.Trace.SpanBenchmark.StartFinishScope net6.0

  • ignore allocated_mem [+0 bytes; +0 bytes] or [-0.005%; +0.007%]
  • ignore execution_time [-4.340ms; -3.215ms] or [-2.126%; -1.575%]
  • 🟩 throughput [+87167.027op/s; +98165.132op/s] or [+8.139%; +9.165%]

scenario:Benchmarks.Trace.SpanBenchmark.StartFinishScope netcoreapp3.1

  • ignore allocated_mem [+0 bytes; +0 bytes] or [-0.003%; +0.006%]
  • ignore execution_time [+2.901ms; +7.027ms] or [+1.468%; +3.556%]
  • ignore throughput [+27356.793op/s; +47668.762op/s] or [+3.166%; +5.518%]

scenario:Benchmarks.Trace.SpanBenchmark.StartFinishSpan net472

  • ignore allocated_mem [+0 bytes; +0 bytes] or [-0.000%; +0.009%]
  • ignore execution_time [+158.714µs; +625.354µs] or [+0.079%; +0.313%]
  • ignore throughput [-21592.857op/s; -18505.889op/s] or [-1.977%; -1.694%]

scenario:Benchmarks.Trace.SpanBenchmark.StartFinishSpan net6.0

  • ignore allocated_mem [+0 bytes; +0 bytes] or [-0.004%; +0.004%]
  • ignore execution_time [+6.483ms; +10.580ms] or [+3.378%; +5.512%]
  • ignore throughput [+62540.768op/s; +94018.996op/s] or [+4.841%; +7.277%]

scenario:Benchmarks.Trace.SpanBenchmark.StartFinishSpan netcoreapp3.1

  • ignore allocated_mem [+0 bytes; +0 bytes] or [-0.001%; +0.008%]
  • ignore execution_time [-3.649ms; -2.181ms] or [-1.793%; -1.072%]
  • 🟩 throughput [+68109.294op/s; +77739.843op/s] or [+6.764%; +7.721%]

scenario:Benchmarks.Trace.SpanBenchmark.StartFinishTwoScopes net472

  • ignore allocated_mem [+0 bytes; +0 bytes] or [-0.008%; +0.002%]
  • ignore execution_time [-883.320µs; +362.027µs] or [-0.440%; +0.180%]
  • ignore throughput [+7385.904op/s; +10813.677op/s] or [+1.646%; +2.409%]

scenario:Benchmarks.Trace.SpanBenchmark.StartFinishTwoScopes net6.0

  • ignore allocated_mem [+0 bytes; +0 bytes] or [-0.004%; +0.009%]
  • ignore execution_time [-1018.678µs; +679.657µs] or [-0.509%; +0.339%]
  • 🟩 throughput [+51374.698op/s; +56166.334op/s] or [+9.329%; +10.199%]

scenario:Benchmarks.Trace.SpanBenchmark.StartFinishTwoScopes netcoreapp3.1

  • ignore allocated_mem [+0 bytes; +0 bytes] or [-0.006%; +0.004%]
  • ignore execution_time [+0.148ms; +4.224ms] or [+0.074%; +2.122%]
  • ignore throughput [+10958.455op/s; +20836.748op/s] or [+2.453%; +4.664%]

scenario:Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin net472

  • ignore allocated_mem [+0 bytes; +0 bytes] or [-0.005%; +0.006%]
  • ignore execution_time [-1186.747µs; -233.615µs] or [-0.591%; -0.116%]
  • ignore throughput [-35848.633op/s; -32360.397op/s] or [-5.247%; -4.736%]

scenario:Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin net6.0

  • ignore allocated_mem [+0 bytes; +0 bytes] or [-0.005%; +0.007%]
  • ignore execution_time [-1430.082µs; +2050.287µs] or [-0.715%; +1.026%]
  • 🟩 throughput [+67996.729op/s; +86709.066op/s] or [+7.597%; +9.688%]

scenario:Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin netcoreapp3.1

  • ignore allocated_mem [+0 bytes; +0 bytes] or [-0.005%; +0.005%]
  • ignore execution_time [+1.904ms; +5.800ms] or [+0.967%; +2.946%]
  • ignore throughput [+21028.675op/s; +35991.062op/s] or [+2.936%; +5.025%]

@dd-trace-dotnet-ci-bot

dd-trace-dotnet-ci-bot Bot commented Apr 7, 2026 •
edited
Loading

Copy link
Copy Markdown

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing This PR (8418) and master.

⚠️ Potential regressions detected

HttpMessageHandler

Metric Master (Mean ± 95% CI) Current (Mean ± 95% CI) Change Status
.NET Framework 4.8 - Baseline
duration190.29 ± (190.30 - 190.99) ms208.17 ± (207.73 - 208.78) ms+9.4%❌⬆️
.NET Framework 4.8 - Bailout
duration193.46 ± (193.52 - 193.92) ms211.17 ± (211.03 - 211.54) ms+9.2%❌⬆️
.NET Framework 4.8 - CallTarget+Inlining+NGEN
duration1139.43 ± (1138.81 - 1144.75) ms1207.36 ± (1207.21 - 1214.94) ms+6.0%❌⬆️
Full Metrics Comparison

FakeDbCommand

Metric Master (Mean ± 95% CI) Current (Mean ± 95% CI) Change Status
.NET Framework 4.8 - Baseline
duration72.00 ± (71.93 - 72.25) ms71.82 ± (71.84 - 72.12) ms-0.2%✅
.NET Framework 4.8 - Bailout
duration75.72 ± (75.64 - 76.02) ms75.52 ± (75.36 - 75.72) ms-0.3%✅
.NET Framework 4.8 - CallTarget+Inlining+NGEN
duration1069.15 ± (1071.14 - 1080.01) ms1070.62 ± (1070.06 - 1076.80) ms+0.1%✅⬆️
.NET Core 3.1 - Baseline
process.internal_duration_ms22.29 ± (22.26 - 22.32) ms22.14 ± (22.11 - 22.17) ms-0.7%✅
process.time_to_main_ms83.82 ± (83.62 - 84.01) ms82.75 ± (82.57 - 82.94) ms-1.3%✅
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed10.92 ± (10.92 - 10.93) MB10.92 ± (10.91 - 10.92) MB-0.1%✅
runtime.dotnet.threads.count12 ± (12 - 12)12 ± (12 - 12)+0.0%✅
.NET Core 3.1 - Bailout
process.internal_duration_ms22.31 ± (22.27 - 22.34) ms22.01 ± (21.98 - 22.04) ms-1.3%✅
process.time_to_main_ms85.12 ± (84.96 - 85.29) ms83.96 ± (83.77 - 84.16) ms-1.4%✅
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed10.96 ± (10.96 - 10.97) MB10.94 ± (10.93 - 10.94) MB-0.2%✅
runtime.dotnet.threads.count13 ± (13 - 13)13 ± (13 - 13)+0.0%✅
.NET Core 3.1 - CallTarget+Inlining+NGEN
process.internal_duration_ms227.08 ± (225.83 - 228.34) ms225.67 ± (224.47 - 226.88) ms-0.6%✅
process.time_to_main_ms518.37 ± (517.13 - 519.61) ms520.18 ± (518.83 - 521.54) ms+0.3%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed48.42 ± (48.39 - 48.45) MB48.44 ± (48.41 - 48.47) MB+0.0%✅⬆️
runtime.dotnet.threads.count28 ± (28 - 28)28 ± (28 - 28)+0.0%✅
.NET 6 - Baseline
process.internal_duration_ms21.04 ± (21.00 - 21.08) ms20.97 ± (20.94 - 21.01) ms-0.3%✅
process.time_to_main_ms72.40 ± (72.24 - 72.56) ms72.10 ± (71.94 - 72.26) ms-0.4%✅
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed10.63 ± (10.63 - 10.63) MB10.62 ± (10.61 - 10.62) MB-0.1%✅
runtime.dotnet.threads.count10 ± (10 - 10)10 ± (10 - 10)+0.0%✅
.NET 6 - Bailout
process.internal_duration_ms21.00 ± (20.97 - 21.04) ms20.91 ± (20.88 - 20.94) ms-0.4%✅
process.time_to_main_ms73.79 ± (73.63 - 73.95) ms73.23 ± (73.10 - 73.37) ms-0.8%✅
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed10.68 ± (10.68 - 10.69) MB10.75 ± (10.74 - 10.75) MB+0.6%✅⬆️
runtime.dotnet.threads.count11 ± (11 - 11)11 ± (11 - 11)+0.0%✅
.NET 6 - CallTarget+Inlining+NGEN
process.internal_duration_ms385.43 ± (383.54 - 387.32) ms385.58 ± (383.34 - 387.82) ms+0.0%✅⬆️
process.time_to_main_ms517.75 ± (516.81 - 518.69) ms521.79 ± (520.70 - 522.89) ms+0.8%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed50.08 ± (50.05 - 50.11) MB50.08 ± (50.05 - 50.11) MB-0.0%✅
runtime.dotnet.threads.count28 ± (28 - 28)28 ± (28 - 28)+0.1%✅⬆️
.NET 8 - Baseline
process.internal_duration_ms19.18 ± (19.14 - 19.21) ms19.22 ± (19.19 - 19.25) ms+0.2%✅⬆️
process.time_to_main_ms71.25 ± (71.08 - 71.42) ms71.34 ± (71.19 - 71.48) ms+0.1%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed7.69 ± (7.68 - 7.70) MB7.68 ± (7.68 - 7.69) MB-0.1%✅
runtime.dotnet.threads.count10 ± (10 - 10)10 ± (10 - 10)+0.0%✅
.NET 8 - Bailout
process.internal_duration_ms19.17 ± (19.13 - 19.20) ms19.19 ± (19.16 - 19.22) ms+0.1%✅⬆️
process.time_to_main_ms72.34 ± (72.21 - 72.48) ms72.26 ± (72.16 - 72.36) ms-0.1%✅
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed7.73 ± (7.72 - 7.74) MB7.74 ± (7.73 - 7.74) MB+0.1%✅⬆️
runtime.dotnet.threads.count11 ± (11 - 11)11 ± (11 - 11)+0.0%✅
.NET 8 - CallTarget+Inlining+NGEN
process.internal_duration_ms306.20 ± (303.93 - 308.48) ms303.61 ± (301.22 - 306.01) ms-0.8%✅
process.time_to_main_ms478.84 ± (477.97 - 479.70) ms481.49 ± (480.78 - 482.21) ms+0.6%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed37.11 ± (37.09 - 37.14) MB37.09 ± (37.07 - 37.12) MB-0.1%✅
runtime.dotnet.threads.count27 ± (27 - 27)27 ± (27 - 27)-0.4%✅

HttpMessageHandler

Metric Master (Mean ± 95% CI) Current (Mean ± 95% CI) Change Status
.NET Framework 4.8 - Baseline
duration190.29 ± (190.30 - 190.99) ms208.17 ± (207.73 - 208.78) ms+9.4%❌⬆️
.NET Framework 4.8 - Bailout
duration193.46 ± (193.52 - 193.92) ms211.17 ± (211.03 - 211.54) ms+9.2%❌⬆️
.NET Framework 4.8 - CallTarget+Inlining+NGEN
duration1139.43 ± (1138.81 - 1144.75) ms1207.36 ± (1207.21 - 1214.94) ms+6.0%❌⬆️
.NET Core 3.1 - Baseline
process.internal_duration_ms185.20 ± (184.92 - 185.49) ms199.05 ± (198.74 - 199.37) ms+7.5%✅⬆️
process.time_to_main_ms79.54 ± (79.37 - 79.70) ms85.86 ± (85.63 - 86.09) ms+7.9%✅⬆️
runtime.dotnet.exceptions.count3 ± (3 - 3)3 ± (3 - 3)+0.0%✅
runtime.dotnet.mem.committed16.18 ± (16.16 - 16.21) MB16.00 ± (15.97 - 16.03) MB-1.1%✅
runtime.dotnet.threads.count20 ± (20 - 20)20 ± (19 - 20)-0.1%✅
.NET Core 3.1 - Bailout
process.internal_duration_ms184.57 ± (184.34 - 184.81) ms198.88 ± (198.61 - 199.16) ms+7.8%✅⬆️
process.time_to_main_ms80.97 ± (80.86 - 81.09) ms87.21 ± (86.97 - 87.45) ms+7.7%✅⬆️
runtime.dotnet.exceptions.count3 ± (3 - 3)3 ± (3 - 3)+0.0%✅
runtime.dotnet.mem.committed16.16 ± (16.07 - 16.26) MB16.08 ± (16.06 - 16.10) MB-0.5%✅
runtime.dotnet.threads.count21 ± (20 - 21)21 ± (20 - 21)+0.1%✅⬆️
.NET Core 3.1 - CallTarget+Inlining+NGEN
process.internal_duration_ms392.10 ± (390.63 - 393.58) ms408.47 ± (407.01 - 409.93) ms+4.2%✅⬆️
process.time_to_main_ms504.22 ± (503.13 - 505.31) ms538.20 ± (536.55 - 539.84) ms+6.7%✅⬆️
runtime.dotnet.exceptions.count3 ± (3 - 3)3 ± (3 - 3)+0.0%✅
runtime.dotnet.mem.committed58.64 ± (58.42 - 58.86) MB59.24 ± (59.19 - 59.30) MB+1.0%✅⬆️
runtime.dotnet.threads.count30 ± (30 - 30)30 ± (30 - 30)+0.3%✅⬆️
.NET 6 - Baseline
process.internal_duration_ms189.67 ± (189.36 - 189.98) ms203.73 ± (203.27 - 204.20) ms+7.4%✅⬆️
process.time_to_main_ms69.29 ± (69.14 - 69.45) ms74.51 ± (74.25 - 74.77) ms+7.5%✅⬆️
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%✅
runtime.dotnet.mem.committed15.86 ± (15.68 - 16.03) MB16.33 ± (16.30 - 16.35) MB+3.0%✅⬆️
runtime.dotnet.threads.count18 ± (18 - 18)19 ± (19 - 19)+7.9%✅⬆️
.NET 6 - Bailout
process.internal_duration_ms188.26 ± (188.11 - 188.41) ms203.70 ± (203.39 - 204.01) ms+8.2%✅⬆️
process.time_to_main_ms70.24 ± (70.19 - 70.30) ms75.89 ± (75.70 - 76.08) ms+8.0%✅⬆️
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%✅
runtime.dotnet.mem.committed16.00 ± (15.83 - 16.17) MB16.33 ± (16.30 - 16.36) MB+2.1%✅⬆️
runtime.dotnet.threads.count19 ± (18 - 19)20 ± (20 - 20)+7.9%✅⬆️
.NET 6 - CallTarget+Inlining+NGEN
process.internal_duration_ms594.85 ± (591.84 - 597.87) ms598.04 ± (595.72 - 600.36) ms+0.5%✅⬆️
process.time_to_main_ms508.09 ± (507.18 - 508.99) ms538.79 ± (537.64 - 539.95) ms+6.0%✅⬆️
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%✅
runtime.dotnet.mem.committed61.67 ± (61.58 - 61.77) MB61.75 ± (61.65 - 61.85) MB+0.1%✅⬆️
runtime.dotnet.threads.count30 ± (30 - 30)31 ± (31 - 31)+1.0%✅⬆️
.NET 8 - Baseline
process.internal_duration_ms185.99 ± (185.74 - 186.25) ms202.25 ± (201.80 - 202.69) ms+8.7%✅⬆️
process.time_to_main_ms68.37 ± (68.25 - 68.49) ms74.78 ± (74.50 - 75.06) ms+9.4%✅⬆️
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%✅
runtime.dotnet.mem.committed11.72 ± (11.65 - 11.79) MB11.66 ± (11.63 - 11.68) MB-0.5%✅
runtime.dotnet.threads.count18 ± (18 - 18)19 ± (19 - 19)+3.5%✅⬆️
.NET 8 - Bailout
process.internal_duration_ms185.71 ± (185.49 - 185.92) ms201.51 ± (201.23 - 201.79) ms+8.5%✅⬆️
process.time_to_main_ms69.62 ± (69.56 - 69.69) ms75.77 ± (75.57 - 75.97) ms+8.8%✅⬆️
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%✅
runtime.dotnet.mem.committed11.67 ± (11.58 - 11.77) MB11.73 ± (11.71 - 11.75) MB+0.5%✅⬆️
runtime.dotnet.threads.count19 ± (18 - 19)20 ± (20 - 20)+6.3%✅⬆️
.NET 8 - CallTarget+Inlining+NGEN
process.internal_duration_ms521.08 ± (518.52 - 523.65) ms522.50 ± (518.30 - 526.70) ms+0.3%✅⬆️
process.time_to_main_ms466.62 ± (465.98 - 467.26) ms497.17 ± (496.24 - 498.09) ms+6.5%✅⬆️
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%✅
runtime.dotnet.mem.committed50.75 ± (50.72 - 50.78) MB50.79 ± (50.74 - 50.83) MB+0.1%✅⬆️
runtime.dotnet.threads.count30 ± (30 - 30)30 ± (30 - 30)+0.5%✅⬆️
Comparison explanation

Execution-time benchmarks measure the whole time it takes to execute a program, and are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are highlighted in **red**. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

Duration charts
FakeDbCommand (.NET Framework 4.8) 
gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8418) - mean (72ms)  : 70, 74
    master - mean (72ms)  : 70, 74

    section Bailout
    This PR (8418) - mean (76ms)  : 74, 77
    master - mean (76ms)  : 74, 78

    section CallTarget+Inlining+NGEN
    This PR (8418) - mean (1,073ms)  : 1024, 1123
    master - mean (1,076ms)  : 1010, 1141
Loading
FakeDbCommand (.NET Core 3.1) 
gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8418) - mean (111ms)  : 108, 115
    master - mean (114ms)  : 110, 117

    section Bailout
    This PR (8418) - mean (113ms)  : 109, 116
    master - mean (115ms)  : 112, 118

    section CallTarget+Inlining+NGEN
    This PR (8418) - mean (784ms)  : 765, 803
    master - mean (784ms)  : 764, 805
Loading
FakeDbCommand (.NET 6) 
gantt
    title Execution time (ms) FakeDbCommand (.NET 6)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8418) - mean (99ms)  : 95, 103
    master - mean (100ms)  : 96, 103

    section Bailout
    This PR (8418) - mean (100ms)  : 98, 103
    master - mean (101ms)  : 98, 104

    section CallTarget+Inlining+NGEN
    This PR (8418) - mean (936ms)  : 903, 969
    master - mean (932ms)  : 899, 964
Loading
FakeDbCommand (.NET 8) 
gantt
    title Execution time (ms) FakeDbCommand (.NET 8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8418) - mean (98ms)  : 95, 101
    master - mean (98ms)  : 94, 103

    section Bailout
    This PR (8418) - mean (99ms)  : 97, 101
    master - mean (99ms)  : 97, 101

    section CallTarget+Inlining+NGEN
    This PR (8418) - mean (819ms)  : 773, 864
    master - mean (814ms)  : 781, 848
Loading
HttpMessageHandler (.NET Framework 4.8) 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8418) - mean (208ms)  : 202, 214
    master - mean (191ms)  : 187, 194

    section Bailout
    This PR (8418) - mean (211ms)  : crit, 209, 214
    master - mean (194ms)  : 192, 196

    section CallTarget+Inlining+NGEN
    This PR (8418) - mean (1,211ms)  : crit, 1155, 1267
    master - mean (1,142ms)  : 1100, 1184
Loading
HttpMessageHandler (.NET Core 3.1) 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8418) - mean (294ms)  : 290, 298
    master - mean (273ms)  : 269, 277

    section Bailout
    This PR (8418) - mean (295ms)  : crit, 292, 299
    master - mean (273ms)  : 270, 276

    section CallTarget+Inlining+NGEN
    This PR (8418) - mean (975ms)  : 952, 999
    master - mean (924ms)  : 895, 953
Loading
HttpMessageHandler (.NET 6) 
gantt
    title Execution time (ms) HttpMessageHandler (.NET 6)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8418) - mean (287ms)  : 281, 293
    master - mean (267ms)  : 262, 272

    section Bailout
    This PR (8418) - mean (288ms)  : crit, 284, 292
    master - mean (266ms)  : 264, 268

    section CallTarget+Inlining+NGEN
    This PR (8418) - mean (1,165ms)  : 1133, 1198
    master - mean (1,133ms)  : 1090, 1176
Loading
HttpMessageHandler (.NET 8) 
gantt
    title Execution time (ms) HttpMessageHandler (.NET 8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8418) - mean (288ms)  : 279, 296
    master - mean (264ms)  : 261, 267

    section Bailout
    This PR (8418) - mean (287ms)  : crit, 283, 292
    master - mean (265ms)  : 261, 268

    section CallTarget+Inlining+NGEN
    This PR (8418) - mean (1,054ms)  : 990, 1118
    master - mean (1,019ms)  : 980, 1058
Loading

@andrewlock andrewlock mentioned this pull request Apr 7, 2026
Merged
@lucaspimentel
lucaspimentel requested a review from Copilot April 8, 2026 03:59
Copilot AI reviewed Apr 8, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds an IP address quantization utility to obfuscate peer-tag IPs (reducing stats cardinality) and ports the corresponding unit tests from the Go agent.

Changes:

  • Introduces IpAddressObfuscationUtil to detect/obfuscate IPs (with scheme/prefix handling) and deduplicate entries.
  • Adds unit tests covering a variety of IPv4/IPv6/hostname formats and comma-separated peer lists.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File Description
tracer/src/Datadog.Trace/Processors/IpAddressObfuscationUtil.cs Implements IP detection + obfuscation + deduplication logic for peer tags
tracer/test/Datadog.Trace.Tests/TraceProcessors/IpAddressObfuscationUtilTests.cs Adds ported test cases validating expected quantization behavior

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread tracer/src/Datadog.Trace/Processors/IpAddressObfuscationUtil.cs Outdated
Comment thread tracer/src/Datadog.Trace/Processors/IpAddressObfuscationUtil.cs Outdated
Comment thread tracer/src/Datadog.Trace/Processors/IpAddressObfuscationUtil.cs Outdated
Comment thread tracer/test/Datadog.Trace.Tests/TraceProcessors/IpAddressObfuscationUtilTests.cs
@andrewlock
andrewlock force-pushed the andrew/client-side-stats/ip-obfuscation branch from f00d1b6 to 77f3510 Compare April 8, 2026 10:41
@andrewlock
andrewlock requested review from a team as code owners April 8, 2026 10:41
@andrewlock
andrewlock force-pushed the andrew/client-side-stats/char-helpers branch from 8c15dbe to 46be799 Compare April 8, 2026 10:41
@andrewlock
andrewlock force-pushed the andrew/client-side-stats/ip-obfuscation branch from 77f3510 to 536d246 Compare April 8, 2026 10:57
Base automatically changed from andrew/client-side-stats/char-helpers to master April 8, 2026 14:23
@andrewlock
andrewlock force-pushed the andrew/client-side-stats/ip-obfuscation branch from 536d246 to a44d3e6 Compare April 8, 2026 14:28
This was referenced Apr 9, 2026
Merged
Merged
Closed
Merged
Closed
NachoEchevarria
NachoEchevarria reviewed Apr 13, 2026
View reviewed changes
Comment thread tracer/src/Datadog.Trace/Processors/IpAddressObfuscationUtil.cs Outdated
#if NETCOREAPP
return ushort.TryParse(s, out _);
#else
if (s.Length == 0)

@NachoEchevarria NachoEchevarria Apr 13, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NIT: In the !NETCOREAPP case, we are accepting as valid port values > 64k

@andrewlock andrewlock Apr 13, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. I think that's maybe something we should just live with, as it seems like more hassle (allocation) than it's worth to try to handle it? 🤔 An "easy" fix would be to reject s.Length > 5 too 🤔 That way we would still be allowing 65,535 - 99,999, which are not valid, but we would reject anything bigger. Seems like a reasonable trade off to me, wdyt?

@andrewlock andrewlock Apr 16, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I made that change here: f0895ac, I think it's probably good enough TBH

NachoEchevarria
NachoEchevarria approved these changes Apr 13, 2026
View reviewed changes

@NachoEchevarria NachoEchevarria left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

@andrewlock
andrewlock force-pushed the andrew/client-side-stats/ip-obfuscation branch from a44d3e6 to f0895ac Compare April 14, 2026 10:08
bouwkast
bouwkast approved these changes Apr 15, 2026
View reviewed changes

@bouwkast bouwkast left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't do a super thorough review of it compared to the go version, but looks good to me

@andrewlock
andrewlock merged commit b176976 into master Apr 16, 2026
140 checks passed
@andrewlock
andrewlock deleted the andrew/client-side-stats/ip-obfuscation branch April 16, 2026 07:50
@github-actions github-actions Bot added this to the vNext-v3 milestone Apr 16, 2026
andrewlock added a commit that referenced this pull request Apr 20, 2026
@andrewlock @claude @lucaspimentel
Implement Client-Side Stats (CSS) 1.2.0 (#8420)
014e5d0 
## Summary of changes

Updates the existing client-side-stats implementation to match version
1.2.0 [as defined in the
RFC](https://datadoghq.atlassian.net/wiki/spaces/APM/pages/6378947571/Client-Side+Stats+v1.2.0)
and as [implemented in the
agent](https://github.com/DataDog/datadog-agent/blob/c1d67a906f4c594654600760da1eea4c8037471a/pkg/proto/datadog/trace/stats.proto#L83).

## Reason for change

Our implementation is severely lagging the latest implementation in the
agent. This hasn't been a big deal, as it's not documented and not
enabled by default, but we'd like to fix the implementation to make it
usable.

## Implementation details

This was driven almost entirely by 🤖, by comparing our existing
implementation to the RFC, and also taking the agent/go implementation
as the canonical implementation.

> Implementing this in .NET highlighted a number of missing aspects in
the RFC, which I've raised elsewhere, and aim to get incorporated into
the RFC.

At a high level, the PR contains the following changes:

- Stats Wire Format & Serialization
- **Added new aggregation dimensions**: `SpanKind`, `IsTraceRoot` (as
Trilean), `HTTPMethod`, `HTTPEndpoint`, `GRPCStatusCode`,
`ServiceSource`, `PeerTags` to both the aggregation key and the msgpack
wire format
- **Fixed `GRPCStatusCode` type**: Changed from `int` to `string` to
match Go agent's wire format (agent was returning 400 Bad Request in
system tests)
- **gRPC status code extraction**: Checks 4 tag names in priority order
(`rpc.grpc.status_code`, `grpc.code`, `rpc.grpc.status.code`,
`grpc.status.code`)
- **Stochastic rounding**: `Hits`, `Errors`, `Duration`, `TopLevelHits`
accumulated as `double` (weighted by sampling rate) then rounded
probabilistically to `int64` for serialization
- **Duration weighting**: Durations are now multiplied by sampling
weight (`1/rate`), matching Go agent behavior
- **Default env**: Serializes `"unknown-env"` when environment is not
configured
  - **`git_commit_sha`**: Added as optional field in stats payload
  - **`Service`**: Added as top-level field in stats payload
- **Empty bucket suppression**: `HasHits()` check prevents sending
payloads with zero-hit buckets (stale keys retained for sketch reuse)
- Bucket Timing
- **10-second alignment**: Bucket `Start` timestamps aligned to
10-second boundaries (`ts - ts % 10_000_000_000`) matching Go tracer's
`alignTs`
- **Removed unused `StartTime`** property (only `Start` as aligned
nanoseconds)
- Agent Discovery (`/info` Endpoint)
- **`peer_tags`**: Parsed, sorted, deduplicated; used for peer tag
extraction on client/producer/consumer spans
- **`span_kinds_stats_computed`**: Parsed to override eligible span
kinds
  - **`obfuscation_version`**: Parsed for obfuscation negotiation
- **Trace filters**: Parsed `filter_tags`, `filter_tags_regex`,
`ignore_resources` from `/info`
- Trace Filtering
- **`TraceFilter` implementation**: Evaluates agent-configured filters
(exact tags, regex tags, resource patterns) against root spans before
stats computation
- **Tag-only filters**: Handles filter entries that match on tag key
presence without a specific value
- SQL Obfuscation
- **Operator splitters**: Added `* / = < > ! & ^ % ~ ? @ : #` as token
splitters (matching Go agent's `go-sqllexer` `isOperator()`) so queries
like `WHERE id='1'` are properly obfuscated
- **Whitespace normalization**: Post-processing pass adds spaces around
comparison operators (`=`, `<`, `>`, `!`) adjacent to `?` placeholders,
matching Go agent's normalizer output (e.g., `id='1'` → `id = ?`)
- **Obfuscation gating**: Only runs when `obfuscation_version` is
negotiated with agent; sends `Datadog-Obfuscation-Version` header
- Peer Tags
- **IP quantization**: Peer tag values run through
`IpAddressObfuscationUtil.QuantizePeerIpAddresses()` replacing
non-allowed IPs with `"blocked-ip-address"`
- **Base service handling**: Internal/missing-kind spans with
non-default service name use `_dd.base_service:{serviceName}` as sole
peer tag
- **FNV-1a hashing**: Peer tags hashed with null-byte separators for
aggregation key
- Other
- **No retries for stats**: Stats sends are fire-and-forget (retry limit
= 0)
- **Synthetics detection**: Uses `StartsWith("synthetics")` prefix
matching (not exact match)
- **Mock agent updates**: Test mock agent returns `obfuscation_version`
in `/info` response

## Test coverage

There's a _lot_ going on in this PR, because we were so far behind. I
_could_ split this into implementing individual features, but there
would be a lot of duplication between PRs, and it didn't seem like it
would be that easy to track. At least with this big bang we can compare
directly against the system tests etc.

The existing system tests for stats computation were checked, and made
to pass (which identified a number of hidden expectations which will be
added to the RFC). I'll create a PR to enable these in the system-tests
repo

## Other details

Part of a stack

- #8417
- #8418

There are still some _theoretical_ gaps between the go implementation
and the .NET implementation, but I _think_ these are non-issues in
_most_ cases:


  | Area | Gap | Impact |
  |------|-----|--------|
| gRPC status code normalization | Go agent normalizes string statuses
(e.g., `"CANCELLED"` → `"1"`); .NET passes raw tag value | Stats
mismatch if gRPC library uses string-form status codes |
| HTTP status code tags | Go agent checks both `http.status_code` and
`http.response.status_code` (OTel convention); .NET only checks
`http.status_code` | OTel spans using newer convention would get `0` in
.NET |
| Duration precision truncation | Go agent uses float bit masking; .NET
uses integer shifting — both target ~10 bits but may produce slightly
different values | Minor histogram differences |
| SQL obfuscation | Go agent uses full tokenizer + normalizer; .NET uses
character-level splitter with targeted normalization around comparison
operators only | Complex SQL with unusual formatting may produce
different resource strings |
| `HTTP_method` / `HTTP_endpoint` | .NET always populates from span
tags; Go agent only populates from newer OTel pipeline paths | Creates
different aggregation keys — Go groups all methods/routes together in
default path |
| `_top_level` metric | Go checks both `_top_level` and `_dd.top_level`;
.NET only checks `_dd.top_level` | Spans using older metric name would
be missed by .NET |
| `span_derived_primary_tags` | Still in Go agent code but implemented
in v1.3.0 RFC, which was reverted; removed from .NET | No current
impact; may need to re-add if spec reverts back |

There is another big elephant in the room, which is perf. The peer tags,
in particular, currently requires a _bunch_ of allocation. I'd rather
defer trying to fight against that to another PR if possible, unless
anyone has some clear ideas 😄

Another aspect I'm not sure about is how this interacts with
@zacharycmontoya's recent work to publish OTLP stats. I took a random
guess and fought the refactoring, but need to verify it.

---------

Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>
Co-authored-by: Lucas Pimentel <[email protected]>
andrewlock added a commit that referenced this pull request Apr 20, 2026
@andrewlock
Extract ContainsSpanId to SpanCollection (#8435)
aefdbef 
## Summary of changes

Extracts `ContainsSpanId` from `TraceChunkModel` to `SpanCollection`

## Reason for change

I need to find a given span in a `SpanCollection` for client-side-stats.
The algorithm used by `TraceChunkModel` does it "properly" (by stating
from the last span when looking for a root span), so it seems to make
sense to move this method to `SpanCollection`. Additionally, this
essentially hides some of the internals about the `SpanCollection` type,
so seems like a better place for it anyway.

## Implementation details

- Move the `IndexOf()` method from `TraceChunkModel` to `SpanCollection`
- Switch it to `Contains()` instead, seeing as the index is never
actually used AFAICT. It's easy to switch it back later if we _do_ need
it.

## Test coverage

Added some unit tests for the method

## Other details


Part of a stack

- #8417
- #8418
- #8420
andrewlock added a commit that referenced this pull request Apr 21, 2026
@andrewlock
Fix processing pipeline for client-side-stats (#8436)
2e20358 
## Summary of changes

Updates the processing pipeline for client side stats

## Reason for change

The bulk of the CSS 1.2.0 work was done in #8420, but the
span-processing pipeline wasn't the same as the Go agent:

| Step | Go Agent  | .NET Tracer                            |
| ---- | --------- | -------------------------------------- |
| 1    | Normalize | Normalize + Obfuscate (`ProcessTrace`) |
| 2    | Filter    | Filter + Sample (`ShouldKeepTrace`)    |
| 3    | Obfuscate |                                        |
| 4    | Sample    |                                        |
| 5    | Stats     | Stats (`AddRange`)                     |


## Implementation details

The difference was mostly due to the way we had separate
`ProcessTrace()` and `ShouldKeepTrace()` methods. By merging them, we
can encapsulate the pipeline, and ensure that we mirror the way the Go
agent processes traces.

Additionally, there are two different mechanisms for dropping spans now
- p0drops and "trace filtering", so added a new metric tag
`reason:trace_filter` for it.

Additionally, made some small optimizations:
- No need for an array of `ITraceProcessor` when it only contains one
- Updated `TraceFilter` algorithm to correctly look for local trace
root, starting from last span
- Made `_statsAggregator` non-null in `AgentWriter` to eliminate a bunch
of null checks

## Test coverage

Covered by existing tests.

## Other details

Part of a stack:
- #8417
- #8418
- #8420
- #8435
andrewlock added a commit that referenced this pull request Apr 21, 2026
@andrewlock
Improve peer-tag calculation fast-path performance (#8445)
269954c 
## Summary of changes

Improves the performance of peer-tag hash calculation for "fast path"
cases (where there's an existing bucket)

## Reason for change

We have to calculate the hash of all peer tags as part of
client-side-stats bucketing calculations, which involves encoding them
as utf-8. Additionally, when we _send_ the buckets, we have to send the
tags as utf-8.

In the initial CSS 1.2.0 implementation, we encoded the tags every-time
we ran a calculation, which would allocate for every span that had peer
tags, generally quite expensive. In this PR, we switch to doing the
encoding twice: once with a zero-allocation implementation (amortized 0
on .NET Framework) to calculate the hash, and then, _if_ we need the
"real" encoded tags, then we do that encoding again.

## Implementation details

- Split the peer tags work in two, once to calculate the peer tags hash,
once to get the actual tags as `key:value`
- In the hash-calculation stage, we can use `stackalloc` for .NET Core,
and array pool implementation for .NET Framework etc
- As additional optimizations, we also 
- Pre-encode the peer tag _keys_ to utf-8, so that we only need to do
those once for the hash calculation.
- Pass key details from `BuildKey` to `GetEncodedPeerTags` (i.e. is this
a "base service" only tag, if so, what's the tag value, otherwise how
big does the peer tag list need to be)
- If client-side stats is disabled, don't bother doing all the
pre-calculation (or for trace filters)

There are some possible future optimizations, _not_ implemented in this
PR:
- Track which tags require IP quantization to avoid re-doing it. (Maybe
we should allow-list the quantization anyway, so that it only applies to
specific tags? Or alternatively, block list?)
- Instead of doing the encoding of `tag:value`, allow writing the
pre-computed `byte[]` to `MessagePackBinary` and appending the `value`.
This is doable, but requires updating the `MessagePackBinary`
implementation to support it, so I considered it out of scope for now

## Test coverage

Mostly covered by existing tests, but added some additional unit tests
that directly compare the hashing to values used in Go agent tests.

Additionally did some benchmarking. The key thing is that the
`ClientSpanWithPeerTags` path is zero-allocation (and it's nice that the
slow-path is still lower allocation than before, even if it's slower
over all)

| Method | Runtime | Mean | Error | Allocated |
| -------------------------------------------------- |
------------------ | ----------: | ---------: | --------: |
| BuildKey_SimpleSpan_Before | .NET 10.0 | 44.34 ns | 0.906 ns | - |
| BuildKey_SimpleSpan_After | .NET 10.0 | 39.17 ns | 0.445 ns | - |
| BuildKey_SimpleSpan_Before | .NET 6.0 | 93.13 ns | 1.837 ns | - |
| BuildKey_SimpleSpan_After | .NET 6.0 | 87.56 ns | 0.839 ns | - |
| BuildKey_SimpleSpan_Before | .NET Core 3.1 | 192.53 ns | 2.572 ns | -
|
| BuildKey_SimpleSpan_After | .NET Core 3.1 | 217.32 ns | 4.362 ns | - |
| BuildKey_SimpleSpan_Before | .NET Framework 4.8 | 131.98 ns | 2.621 ns
| - |
| BuildKey_SimpleSpan_After | .NET Framework 4.8 | 175.20 ns | 7.647 ns
| - |
| | | | | |
| BuildKey_ClientSpanNoPeerTags_Before | .NET 10.0 | 160.62 ns | 2.622
ns | - |
| BuildKey_ClientSpanNoPeerTags_After | .NET 10.0 | 151.27 ns | 1.995 ns
| - |
| BuildKey_ClientSpanNoPeerTags_Before | .NET 6.0 | 243.62 ns | 3.199 ns
| - |
| BuildKey_ClientSpanNoPeerTags_After | .NET 6.0 | 232.55 ns | 2.975 ns
| - |
| BuildKey_ClientSpanNoPeerTags_Before | .NET Core 3.1 | 550.47 ns |
9.998 ns | - |
| BuildKey_ClientSpanNoPeerTags_After | .NET Core 3.1 | 680.42 ns |
10.022 ns | - |
| BuildKey_ClientSpanNoPeerTags_Before | .NET Framework 4.8 | 442.88 ns
| 8.854 ns | - |
| BuildKey_ClientSpanNoPeerTags_After | .NET Framework 4.8 | 682.12 ns |
13.468 ns | - |
| | | | | |
| BuildKey_ClientSpanWithPeerTags_Before | .NET 10.0 | 800.82 ns |
15.062 ns | 840 B |
| BuildKey_ClientSpanWithPeerTags_After | .NET 10.0 | 575.04 ns | 5.604
ns | - |
| BuildKey_ClientSpanWithPeerTags_Before | .NET 6.0 | 1,076.07 ns |
13.124 ns | 840 B |
| BuildKey_ClientSpanWithPeerTags_After | .NET 6.0 | 1,440.26 ns |
28.755 ns | - |
| BuildKey_ClientSpanWithPeerTags_Before | .NET Core 3.1 | 1,531.27 ns |
13.509 ns | 840 B |
| BuildKey_ClientSpanWithPeerTags_After | .NET Core 3.1 | 1,774.35 ns |
89.601 ns | - |
| BuildKey_ClientSpanWithPeerTags_Before | .NET Framework 4.8 | 1,682.15
ns | 17.082 ns | 859 B |
| BuildKey_ClientSpanWithPeerTags_After | .NET Framework 4.8 | 2,040.90
ns | 162.895 ns | - |
| | | | | |
| BuildKey_ClientSpanWithPeerTags_GetEncodedPeerTags | .NET 10.0 |
1,141.85 ns | 9.980 ns | 760 B |
| BuildKey_ClientSpanWithPeerTags_GetEncodedPeerTags | .NET 6.0 |
2,720.55 ns | 53.097 ns | 760 B |
| BuildKey_ClientSpanWithPeerTags_GetEncodedPeerTags | .NET Core 3.1 |
3,534.28 ns | 141.234 ns | 760 B |
| BuildKey_ClientSpanWithPeerTags_GetEncodedPeerTags | .NET Framework
4.8 | 3,438.80 ns | 84.352 ns | 778 B |


<details><summary>Details</summary>
<p>

```csharp
// <copyright file="StatsAggregatorBenchmark.cs" company="Datadog">
// Unless explicitly stated otherwise all files in this repository are licensed under the Apache 2 License.
// This product includes software developed at Datadog (https://www.datadoghq.com/). Copyright 2017 Datadog, Inc.
// </copyright>

using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using BenchmarkDotNet.Attributes;
using Datadog.Trace;
using Datadog.Trace.Agent;
using Datadog.Trace.Agent.DiscoveryService;
using Datadog.Trace.Configuration;
using Datadog.Trace.Tagging;

namespace Benchmarks.Trace;

/// <summary>
/// StatsAggregator.BuildKey benchmarks
/// </summary>
[MemoryDiagnoser]
[BenchmarkCategory(Constants.TracerCategory)]
public class StatsAggregatorBenchmark
{
    private static readonly List<StatsAggregator.PeerTagKey> PeerTagKeys =
    [
        new("_dd.base_service"),
        new("aws.queue.name"),
        new("aws.queue.url"),
        new("aws.s3.bucket"),
        new("aws.stream.name"),
        new("bucketname"),
        new("db.couchbase.seed.nodes"),
        new("db.hostname"),
        new("db.instance"),
        new("db.system"),
        new("messaging.destination"),
        new("messaging.kafka.bootstrap.servers"),
        new("messaging.rabbitmq.exchange"),
        new("messaging.system"),
        new("network.destination.name"),
        new("peer.hostname"),
        new("peer.service"),
        new("server.address"),
        new("topic"),
    ];

    private StatsAggregator _aggregator;
    private Span _simpleSpan;
    private Span _clientSpanNoPeerTags;
    private Span _clientSpanWithPeerTags;
    private StatsAggregationKey _key;
    private List<byte[]> _encoded;

    [GlobalSetup]
    public void GlobalSetup()
    {
        _aggregator = new StatsAggregator(
            new NoOpApi(),
            new TracerSettings(),
            NullDiscoveryService.Instance,
            isOtlp: false);

        var now = DateTimeOffset.UtcNow;

        // Simple span: no span kind, no peer tags — exercises the "internal" fast path
        _simpleSpan = CreateSpan(now, "web-service", "web.request", "GET /api/users", "web");

        // Client span with SpanKind but no matching peer tags — iterates peer tag keys but finds nothing
        var clientTags = new HttpTags { HttpMethod = "GET", HttpStatusCode = "200" };
        _clientSpanNoPeerTags = CreateSpan(now, "web-service", "http.request", "GET /api/orders", "http", clientTags);

        // Client span with several peer tags set — the most expensive path
        var peerTags = new HttpTags { HttpMethod = "POST", HttpStatusCode = "200" };
        _clientSpanWithPeerTags = CreateSpan(now, "web-service", "http.request", "POST /api/data", "http", peerTags);
        _clientSpanWithPeerTags.Tags.SetTag("peer.service", "remote-service");
        _clientSpanWithPeerTags.Tags.SetTag("db.instance", "i-1234");
        _clientSpanWithPeerTags.Tags.SetTag("db.system", "postgres");
        _clientSpanWithPeerTags.Tags.SetTag("server.address", "db.example.com");
        _clientSpanWithPeerTags.Tags.SetTag("network.destination.name", "db.example.com");

    }

    [GlobalCleanup]
    public void GlobalCleanup()
    {
        _aggregator.DisposeAsync().GetAwaiter().GetResult();
        var value = _key;
        var encoded = _encoded;
    }

    /// <summary>
    /// BuildKey for a simple span with no span kind (internal fast-path, no peer tag iteration).
    /// </summary>
    [Benchmark]
    public void BuildKey_SimpleSpan()
    {
        _key = _aggregator.BuildKey(_simpleSpan, PeerTagKeys, out _);
    }

    /// <summary>
    /// BuildKey for a client span that has no matching peer tags (iterates all peer tag keys, finds none).
    /// </summary>
    [Benchmark]
    public void BuildKey_ClientSpanNoPeerTags()
    {
        _key = _aggregator.BuildKey(_clientSpanNoPeerTags, PeerTagKeys, out _);
    }

    /// <summary>
    /// BuildKey for a client span with several peer tags set (UTF-8 encoding + FNV hashing).
    /// </summary>
    [Benchmark]
    public void BuildKey_ClientSpanWithPeerTags()
    {
        _key = _aggregator.BuildKey(_clientSpanWithPeerTags, PeerTagKeys, out var results);
    }

    /// <summary>
    /// BuildKey for a client span with several peer tags set (UTF-8 encoding + FNV hashing).
    /// </summary>
    [Benchmark]
    public void BuildKey_ClientSpanWithPeerTags_GetEncodedPeerTags()
    {
        _key = _aggregator.BuildKey(_clientSpanWithPeerTags, PeerTagKeys, out var results);
        _encoded = StatsAggregator.GetEncodedPeerTags(_clientSpanWithPeerTags, PeerTagKeys, in results);
    }

    private static Span CreateSpan(DateTimeOffset start, string serviceName, string operationName, string resourceName, string type, ITags tags = null)
    {
        var tracer = Benchmarks.Trace.Asm.EmptyDatadogTracer.Instance;
        var traceContext = new TraceContext(tracer);
        var context = new SpanContext(null, traceContext, serviceName);
        var span = new Span(context, start, tags);
        span.OperationName = operationName;
        span.ResourceName = resourceName;
        span.Type = type;
        return span;
    }

    private sealed class NoOpApi : IApi
    {
        public TracesEncoding TracesEncoding => TracesEncoding.DatadogV0_4;

        public Task<bool> Ping() => Task.FromResult(true);

        public Task<bool> SendTracesAsync(ArraySegment<byte> traces, int numberOfTraces, bool statsComputationEnabled, long numberOfDroppedP0Traces, long numberOfDroppedP0Spans, bool apmTracingEnabled = true)
            => Task.FromResult(true);

        public Task<bool> SendStatsAsync(StatsBuffer stats, long bucketDuration, int tracerObfuscationVersion)
            => Task.FromResult(true);
    }
}
```

</p>
</details> 


## Other details


Part of a stack:
- #8417
- #8418
- #8420
- #8435
- #8436
- #8444
@andrewlock andrewlock mentioned this pull request Apr 22, 2026
Merged
andrewlock added a commit that referenced this pull request Apr 23, 2026
@andrewlock
Remove agent check in client-side-stats (#8503)
7c74848 
## Summary of changes

Removes the minimum-agent version check from the client-side-stats check

## Reason for change

We didn't have it before, which means people who previously had stats,
suddenly won't have them. Also other agents (like bottlecap etc) don't
have the version. And we haven't figured out how to handle that yet.
While CSS is off by default, this seems like the best option to take.

## Implementation details

Remove the code that is checking for CSS support.

## Test coverage

Covered by existing (and removed a test for the behaviour)

## Other details

Follows on from 
- #8417
- #8418
- #8420
- #8435
- #8436
- #8444
- #8445
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments
@chatgpt-codex-connector chatgpt-codex-connector[bot] chatgpt-codex-connector[bot] left review comments
@bouwkast bouwkast bouwkast approved these changes
@NachoEchevarria NachoEchevarria NachoEchevarria approved these changes

Assignees

No one assigned

Labels

area:client-side-stats area:tracer The core tracer library (Datadog.Trace, does not include OpenTracing, native code, or integrations)

Projects

None yet

Milestone

3.42.0

Development

Successfully merging this pull request may close these issues.

4 participants

@andrewlock @bouwkast @NachoEchevarria