Skip to content

Apply cardinality limits to Client-Side stats implementation#8824

Merged
andrewlock merged 11 commits into
masterfrom
andrew/css-cardinality-limits
Jul 2, 2026
Merged

Apply cardinality limits to Client-Side stats implementation#8824
andrewlock merged 11 commits into
masterfrom
andrew/css-cardinality-limits

Conversation

@andrewlock

@andrewlock andrewlock commented Jun 24, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Summary of changes

Implements the Client-Side Stats cardinality control RFC

Reason for change

Currently, we have a risk of unbounded memory growth caused by cardinality explosion. This has not been a big issue so far as CSS is off by default for .NET. However, resource name explosion (in particular), as well as span-derived primary tags and peer tags have the potential to cause OOM issues. This PR adds strict (configurable) caps to prevent issues from runaway memory growth.

Implementation details

Largely broken into separate distinct commits

  • Convert StatsAggregationKey to readonly record struct, seeing as that's basically what it was anyway
  • Make IsTraceRoot nullable (required for "overflow bucket")
  • Make resource length limit configurable (to support "big_resource" feature from agent)
  • Add per-field cardinality limitation to resource, httpendoint, peertags, additional tags
    • Per-field limits stop a single very high cardinality field rendering everything else useless
    • Only added to these 4 fields for now, but we can expand if necessary
  • Add whole-key cardinality limit
    • Essentially a cap on the maximum number of buckets we'll allocate
    • All additional stats flow into an "overflow" bucket
  • Add telemetry/heath-metrics/logging for tracking overflow occurences
  • Add some tests

Test coverage

Added a bunch of unit tests. Waiting on system tests to confirm the implementation.

Other details

Part of a stack

https://datadoghq.atlassian.net/browse/APMLP-1446

@andrewlock andrewlock added area:tracer The core tracer library (Datadog.Trace, does not include OpenTracing, native code, or integrations) type:new-feature area:client-side-stats labels Jun 24, 2026
This was referenced Jun 24, 2026
Merged
Merged
Merged
@pr-commenter

pr-commenter Bot commented Jun 24, 2026 •
edited
Loading

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-07-02 11:32:59

Comparing candidate commit e146b42 in PR branch andrew/css-cardinality-limits with baseline commit dcedc21 in branch master.

📊 Benchmarking dashboard

Found 0 performance improvements and 1 performance regressions! Performance is the same for 71 metrics, 0 unstable metrics, 60 known flaky benchmarks, 66 flaky benchmarks without significant changes.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:Benchmarks.Trace.DbCommandBenchmark.ExecuteNonQuery net472

  • 🟥 throughput [-22082.412op/s; -18032.961op/s] or [-6.220%; -5.079%]

Known flaky benchmarks

These benchmarks are marked as flaky and will not trigger a failure. Modify FLAKY_BENCHMARKS_REGEX to control which benchmarks are marked as flaky.

scenario:Benchmarks.Trace.ActivityBenchmark.StartStopWithChild net472

  • 🟥 throughput [-7753.083op/s; -7099.440op/s] or [-9.193%; -8.418%]

scenario:Benchmarks.Trace.ActivityBenchmark.StartStopWithChild netcoreapp3.1

  • 🟥 throughput [-6257.066op/s; -5127.430op/s] or [-6.362%; -5.213%]

scenario:Benchmarks.Trace.AgentWriterBenchmark.WriteAndFlushEnrichedTraces net472

  • 🟥 execution_time [+301.448ms; +304.679ms] or [+149.589%; +151.192%]
  • 🟥 throughput [-50.687op/s; -46.105op/s] or [-9.120%; -8.295%]

scenario:Benchmarks.Trace.AgentWriterBenchmark.WriteAndFlushEnrichedTraces net6.0

  • 🟥 execution_time [+379.463ms; +381.326ms] or [+299.799%; +301.271%]
  • 🟩 throughput [+84.926op/s; +89.717op/s] or [+11.197%; +11.829%]

scenario:Benchmarks.Trace.AgentWriterBenchmark.WriteAndFlushEnrichedTraces netcoreapp3.1

  • 🟥 execution_time [+395.285ms; +397.818ms] or [+349.813%; +352.054%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleMoreComplexBody net472

  • 🟥 allocated_mem [+1.308KB; +1.308KB] or [+27.528%; +27.540%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleMoreComplexBody net6.0

  • 🟥 allocated_mem [+471 bytes; +472 bytes] or [+9.976%; +9.987%]
  • 🟩 execution_time [-16.203ms; -12.020ms] or [-7.568%; -5.614%]
  • 🟩 throughput [+7705.846op/s; +10484.269op/s] or [+5.625%; +7.653%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleMoreComplexBody netcoreapp3.1

  • 🟥 allocated_mem [+1.272KB; +1.272KB] or [+27.500%; +27.510%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody net472

  • 🟥 allocated_mem [+1.307KB; +1.307KB] or [+105.743%; +105.758%]
  • 🟥 throughput [-255596.165op/s; -252705.848op/s] or [-26.098%; -25.803%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody net6.0

  • 🟥 allocated_mem [+471 bytes; +472 bytes] or [+38.557%; +38.566%]
  • 🟩 execution_time [-26.944ms; -22.085ms] or [-12.016%; -9.849%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody netcoreapp3.1

  • 🟥 allocated_mem [+1.272KB; +1.272KB] or [+105.288%; +105.304%]
  • 🟥 throughput [-151195.088op/s; -134307.917op/s] or [-21.724%; -19.297%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorMoreComplexBody net6.0

  • 🟩 throughput [+9647.554op/s; +12535.670op/s] or [+6.139%; +7.976%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorMoreComplexBody netcoreapp3.1

  • 🟩 throughput [+10564.943op/s; +13226.521op/s] or [+8.416%; +10.537%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody net6.0

  • 🟩 throughput [+445972.325op/s; +461594.978op/s] or [+14.871%; +15.392%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody netcoreapp3.1

  • 🟩 execution_time [-19.463ms; -15.111ms] or [-8.972%; -6.965%]
  • 🟩 throughput [+180957.816op/s; +235073.350op/s] or [+7.183%; +9.331%]

scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeArgs net472

  • 🟥 execution_time [+300.041ms; +300.832ms] or [+149.920%; +150.316%]

scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeArgs net6.0

  • 🟥 execution_time [+298.864ms; +302.009ms] or [+150.718%; +152.304%]

scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeArgs netcoreapp3.1

  • 🟥 execution_time [+299.541ms; +302.377ms] or [+150.885%; +152.314%]

scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeLegacyArgs net472

  • 🟥 execution_time [+297.546ms; +298.491ms] or [+146.143%; +146.607%]

scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeLegacyArgs net6.0

  • 🟥 execution_time [+292.389ms; +295.043ms] or [+142.938%; +144.235%]

scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeLegacyArgs netcoreapp3.1

  • 🟥 execution_time [+300.944ms; +303.232ms] or [+150.411%; +151.555%]

scenario:Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmarkWithAttack net6.0

  • 🟥 execution_time [+22.549µs; +46.212µs] or [+7.199%; +14.753%]
  • 🟥 throughput [-430.185op/s; -231.114op/s] or [-13.410%; -7.205%]

scenario:Benchmarks.Trace.AspNetCoreBenchmark.SendRequest net472

  • 🟥 execution_time [+299.671ms; +300.466ms] or [+149.566%; +149.964%]

scenario:Benchmarks.Trace.AspNetCoreBenchmark.SendRequest net6.0

  • unstable execution_time [+409.424ms; +419.011ms] or [+444.856%; +455.272%]
  • 🟩 throughput [+847.217op/s; +1075.427op/s] or [+6.962%; +8.837%]

scenario:Benchmarks.Trace.AspNetCoreBenchmark.SendRequest netcoreapp3.1

  • unstable execution_time [+201.810ms; +265.544ms] or [+153.232%; +201.625%]

scenario:Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces net472

  • unstable execution_time [+313.071ms; +370.618ms] or [+143.947%; +170.407%]
  • 🟥 throughput [-589.685op/s; -539.469op/s] or [-53.431%; -48.881%]

scenario:Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces net6.0

  • unstable execution_time [+145.810ms; +291.879ms] or [+62.138%; +124.387%]
  • 🟥 throughput [-673.152op/s; -589.701op/s] or [-44.899%; -39.333%]

scenario:Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces netcoreapp3.1

  • 🟥 execution_time [+329.551ms; +341.385ms] or [+197.110%; +204.188%]
  • 🟥 throughput [-403.712op/s; -365.394op/s] or [-28.110%; -25.442%]

scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSlice netcoreapp3.1

  • unstable throughput [+19.635op/s; +67.277op/s] or [+5.644%; +19.339%]

scenario:Benchmarks.Trace.CharSliceBenchmark.OriginalCharSlice net6.0

  • 🟩 execution_time [-179.572µs; -129.270µs] or [-9.096%; -6.548%]
  • 🟩 throughput [+37.408op/s; +50.938op/s] or [+7.385%; +10.056%]

scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearch net472

  • 🟥 execution_time [+304.618ms; +307.085ms] or [+153.400%; +154.642%]

scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearch net6.0

  • 🟥 execution_time [+302.645ms; +303.743ms] or [+151.656%; +152.206%]

scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearch netcoreapp3.1

  • 🟥 execution_time [+300.867ms; +304.749ms] or [+151.143%; +153.093%]

scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearchAsync net472

  • 🟥 execution_time [+302.321ms; +304.577ms] or [+151.815%; +152.948%]

scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearchAsync net6.0

  • 🟥 execution_time [+295.933ms; +299.522ms] or [+146.325%; +148.100%]

scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearchAsync netcoreapp3.1

  • 🟥 execution_time [+304.585ms; +308.785ms] or [+154.377%; +156.506%]

scenario:Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync net472

  • 🟥 execution_time [+301.264ms; +303.283ms] or [+151.207%; +152.221%]

scenario:Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync net6.0

  • 🟥 execution_time [+299.074ms; +301.096ms] or [+149.061%; +150.069%]
  • 🟩 throughput [+35702.634op/s; +44261.856op/s] or [+7.089%; +8.789%]

scenario:Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync netcoreapp3.1

  • 🟥 execution_time [+300.370ms; +303.932ms] or [+149.431%; +151.203%]

scenario:Benchmarks.Trace.ILoggerBenchmark.EnrichedLog netcoreapp3.1

  • unstable execution_time [-31.691ms; -7.688ms] or [-15.897%; -3.857%]

scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark net472

  • unstable execution_time [+21.022µs; +66.625µs] or [+5.193%; +16.457%]

scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark net6.0

  • 🟩 allocated_mem [-25.034KB; -25.010KB] or [-9.132%; -9.123%]
  • unstable execution_time [-38.740µs; +21.051µs] or [-7.657%; +4.161%]
  • unstable throughput [-66.080op/s; +142.837op/s] or [-3.297%; +7.128%]

scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark netcoreapp3.1

  • 🟩 allocated_mem [-15.958KB; -15.941KB] or [-5.817%; -5.811%]
  • unstable execution_time [-68.076µs; -8.009µs] or [-11.797%; -1.388%]

scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark net6.0

  • unstable execution_time [+5.910µs; +10.208µs] or [+13.970%; +24.129%]
  • 🟥 throughput [-4675.388op/s; -2853.138op/s] or [-19.682%; -12.011%]

scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark netcoreapp3.1

  • unstable execution_time [-15.097µs; -7.705µs] or [-23.422%; -11.954%]
  • 🟩 throughput [+1955.268op/s; +3569.444op/s] or [+11.996%; +21.900%]

scenario:Benchmarks.Trace.Log4netBenchmark.EnrichedLog net472

  • 🟥 execution_time [+301.004ms; +302.217ms] or [+152.144%; +152.757%]

scenario:Benchmarks.Trace.Log4netBenchmark.EnrichedLog net6.0

  • 🟥 execution_time [+301.692ms; +304.559ms] or [+153.560%; +155.020%]

scenario:Benchmarks.Trace.Log4netBenchmark.EnrichedLog netcoreapp3.1

  • 🟥 execution_time [+299.975ms; +302.239ms] or [+150.175%; +151.308%]

scenario:Benchmarks.Trace.RedisBenchmark.SendReceive net6.0

  • 🟩 throughput [+33811.478op/s; +40334.571op/s] or [+6.400%; +7.634%]

scenario:Benchmarks.Trace.SerilogBenchmark.EnrichedLog net472

  • 🟥 execution_time [+298.230ms; +300.214ms] or [+148.641%; +149.630%]

scenario:Benchmarks.Trace.SerilogBenchmark.EnrichedLog net6.0

  • 🟥 execution_time [+302.271ms; +303.841ms] or [+151.786%; +152.574%]

scenario:Benchmarks.Trace.SerilogBenchmark.EnrichedLog netcoreapp3.1

  • 🟥 execution_time [+301.875ms; +304.603ms] or [+153.091%; +154.475%]

scenario:Benchmarks.Trace.SingleSpanAspNetCoreBenchmark.SingleSpanAspNetCore net472

  • 🟥 execution_time [+300.417ms; +301.221ms] or [+149.850%; +150.250%]
  • 🟩 throughput [+66006384.944op/s; +66330888.797op/s] or [+48.070%; +48.306%]

scenario:Benchmarks.Trace.SingleSpanAspNetCoreBenchmark.SingleSpanAspNetCore net6.0

  • unstable execution_time [+336.823ms; +395.983ms] or [+418.899%; +492.475%]

scenario:Benchmarks.Trace.SingleSpanAspNetCoreBenchmark.SingleSpanAspNetCore netcoreapp3.1

  • 🟥 execution_time [+299.172ms; +300.382ms] or [+149.220%; +149.824%]
  • 🟩 throughput [+18009768.139op/s; +19009543.107op/s] or [+7.977%; +8.420%]

scenario:Benchmarks.Trace.SpanBenchmark.StartFinishScope net6.0

  • 🟩 throughput [+110695.864op/s; +117361.228op/s] or [+10.335%; +10.958%]

scenario:Benchmarks.Trace.SpanBenchmark.StartFinishSpan net6.0

  • 🟩 throughput [+104345.585op/s; +134320.475op/s] or [+8.077%; +10.397%]

scenario:Benchmarks.Trace.SpanBenchmark.StartFinishSpan netcoreapp3.1

  • 🟩 throughput [+76820.230op/s; +90206.689op/s] or [+7.629%; +8.959%]

scenario:Benchmarks.Trace.SpanBenchmark.StartFinishTwoScopes net6.0

  • 🟩 throughput [+55001.596op/s; +59474.414op/s] or [+9.987%; +10.799%]

scenario:Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin net6.0

  • 🟩 throughput [+72402.118op/s; +92579.008op/s] or [+8.089%; +10.343%]

Known flaky benchmarks without significant changes:

  • scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan net472
  • scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan net6.0
  • scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan netcoreapp3.1
  • scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan_AddEvent_Sampled net472
  • scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan_AddEvent_Sampled net6.0
  • scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan_AddEvent_Sampled netcoreapp3.1
  • scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan_GetContext_Sampled net472
  • scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan_GetContext_Sampled net6.0
  • scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan_GetContext_Sampled netcoreapp3.1
  • scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan_SetAttributes_Sampled net472
  • scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan_SetAttributes_Sampled net6.0
  • scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan_SetAttributes_Sampled netcoreapp3.1
  • scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan_SetStatus_Sampled net472
  • scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan_SetStatus_Sampled net6.0
  • scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan_SetStatus_Sampled netcoreapp3.1
  • scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan_UpdateName_Sampled net472
  • scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan_UpdateName_Sampled net6.0
  • scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan_UpdateName_Sampled netcoreapp3.1
  • scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan net472
  • scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan net6.0
  • scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan netcoreapp3.1
  • scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan_AddEvent_Sampled net472
  • scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan_AddEvent_Sampled net6.0
  • scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan_AddEvent_Sampled netcoreapp3.1
  • scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan_GetContext_Sampled net472
  • scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan_GetContext_Sampled net6.0
  • scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan_GetContext_Sampled netcoreapp3.1
  • scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan_RecordException_Sampled net472
  • scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan_RecordException_Sampled net6.0
  • scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan_RecordException_Sampled netcoreapp3.1
  • scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan_SetAttributes_Sampled net472
  • scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan_SetAttributes_Sampled net6.0
  • scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan_SetAttributes_Sampled netcoreapp3.1
  • scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan_SetStatus_Sampled net472
  • scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan_SetStatus_Sampled net6.0
  • scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan_SetStatus_Sampled netcoreapp3.1
  • scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan_UpdateName_Sampled net472
  • scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan_UpdateName_Sampled net6.0
  • scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan_UpdateName_Sampled netcoreapp3.1
  • scenario:Benchmarks.Trace.ActivityBenchmark.StartStopWithChild net6.0
  • scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorMoreComplexBody net472
  • scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody net472
  • scenario:Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmark net472
  • scenario:Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmark net6.0
  • scenario:Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmark netcoreapp3.1
  • scenario:Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmarkWithAttack net472
  • scenario:Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmarkWithAttack netcoreapp3.1
  • scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSlice net472
  • scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSlice net6.0
  • scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSliceWithPool net472
  • scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSliceWithPool net6.0
  • scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSliceWithPool netcoreapp3.1
  • scenario:Benchmarks.Trace.CharSliceBenchmark.OriginalCharSlice net472
  • scenario:Benchmarks.Trace.CharSliceBenchmark.OriginalCharSlice netcoreapp3.1
  • scenario:Benchmarks.Trace.ILoggerBenchmark.EnrichedLog net472
  • scenario:Benchmarks.Trace.ILoggerBenchmark.EnrichedLog net6.0
  • scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark net472
  • scenario:Benchmarks.Trace.RedisBenchmark.SendReceive net472
  • scenario:Benchmarks.Trace.RedisBenchmark.SendReceive netcoreapp3.1
  • scenario:Benchmarks.Trace.SpanBenchmark.StartFinishScope net472
  • scenario:Benchmarks.Trace.SpanBenchmark.StartFinishScope netcoreapp3.1
  • scenario:Benchmarks.Trace.SpanBenchmark.StartFinishSpan net472
  • scenario:Benchmarks.Trace.SpanBenchmark.StartFinishTwoScopes net472
  • scenario:Benchmarks.Trace.SpanBenchmark.StartFinishTwoScopes netcoreapp3.1
  • scenario:Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin net472
  • scenario:Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin netcoreapp3.1

@dd-trace-dotnet-ci-bot

dd-trace-dotnet-ci-bot Bot commented Jun 24, 2026 •
edited
Loading

Copy link
Copy Markdown

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing This PR (8824) and master.

✅ No regressions detected - check the details below

Full Metrics Comparison

FakeDbCommand

Metric Master (Mean ± 95% CI) Current (Mean ± 95% CI) Change Status
.NET Framework 4.8 - Baseline
duration72.75 ± (72.62 - 73.03) ms69.23 ± (69.21 - 69.49) ms-4.8%✅
.NET Framework 4.8 - Bailout
duration76.28 ± (76.14 - 76.63) ms74.54 ± (74.67 - 75.23) ms-2.3%✅
.NET Framework 4.8 - CallTarget+Inlining+NGEN
duration1081.53 ± (1079.61 - 1086.39) ms1082.81 ± (1082.26 - 1089.34) ms+0.1%✅⬆️
.NET Core 3.1 - Baseline
process.internal_duration_ms22.60 ± (22.54 - 22.66) ms21.85 ± (21.83 - 21.88) ms-3.3%✅
process.time_to_main_ms83.87 ± (83.58 - 84.17) ms79.65 ± (79.50 - 79.81) ms-5.0%✅
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed10.92 ± (10.92 - 10.93) MB10.90 ± (10.90 - 10.91) MB-0.2%✅
runtime.dotnet.threads.count12 ± (12 - 12)12 ± (12 - 12)+0.0%✅
.NET Core 3.1 - Bailout
process.internal_duration_ms22.11 ± (22.08 - 22.14) ms22.26 ± (22.21 - 22.32) ms+0.7%✅⬆️
process.time_to_main_ms82.17 ± (82.01 - 82.33) ms84.54 ± (84.23 - 84.85) ms+2.9%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed10.96 ± (10.96 - 10.96) MB10.93 ± (10.93 - 10.93) MB-0.3%✅
runtime.dotnet.threads.count13 ± (13 - 13)13 ± (13 - 13)+0.0%✅
.NET Core 3.1 - CallTarget+Inlining+NGEN
process.internal_duration_ms208.77 ± (207.90 - 209.63) ms209.45 ± (208.42 - 210.48) ms+0.3%✅⬆️
process.time_to_main_ms530.22 ± (529.06 - 531.39) ms530.27 ± (528.95 - 531.58) ms+0.0%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed49.11 ± (49.07 - 49.14) MB49.23 ± (49.19 - 49.27) MB+0.3%✅⬆️
runtime.dotnet.threads.count28 ± (28 - 28)28 ± (28 - 28)-0.1%✅
.NET 6 - Baseline
process.internal_duration_ms20.91 ± (20.87 - 20.95) ms21.06 ± (21.01 - 21.11) ms+0.7%✅⬆️
process.time_to_main_ms70.84 ± (70.61 - 71.06) ms71.15 ± (70.89 - 71.42) ms+0.4%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed10.64 ± (10.64 - 10.65) MB10.62 ± (10.62 - 10.62) MB-0.2%✅
runtime.dotnet.threads.count10 ± (10 - 10)10 ± (10 - 10)+0.0%✅
.NET 6 - Bailout
process.internal_duration_ms21.01 ± (20.96 - 21.05) ms20.93 ± (20.89 - 20.97) ms-0.3%✅
process.time_to_main_ms72.51 ± (72.27 - 72.74) ms72.38 ± (72.13 - 72.62) ms-0.2%✅
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed10.75 ± (10.75 - 10.75) MB10.75 ± (10.75 - 10.75) MB-0.0%✅
runtime.dotnet.threads.count11 ± (11 - 11)11 ± (11 - 11)+0.0%✅
.NET 6 - CallTarget+Inlining+NGEN
process.internal_duration_ms370.35 ± (368.22 - 372.48) ms372.22 ± (369.87 - 374.58) ms+0.5%✅⬆️
process.time_to_main_ms535.96 ± (534.84 - 537.09) ms535.24 ± (533.95 - 536.52) ms-0.1%✅
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed50.13 ± (50.11 - 50.15) MB50.24 ± (50.21 - 50.26) MB+0.2%✅⬆️
runtime.dotnet.threads.count28 ± (28 - 28)28 ± (28 - 28)+0.1%✅⬆️
.NET 8 - Baseline
process.internal_duration_ms19.04 ± (19.01 - 19.06) ms19.07 ± (19.04 - 19.10) ms+0.2%✅⬆️
process.time_to_main_ms68.73 ± (68.61 - 68.86) ms68.63 ± (68.52 - 68.74) ms-0.2%✅
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed7.71 ± (7.71 - 7.72) MB7.68 ± (7.67 - 7.69) MB-0.5%✅
runtime.dotnet.threads.count10 ± (10 - 10)10 ± (10 - 10)+0.0%✅
.NET 8 - Bailout
process.internal_duration_ms19.10 ± (19.07 - 19.13) ms18.99 ± (18.95 - 19.03) ms-0.6%✅
process.time_to_main_ms69.71 ± (69.58 - 69.83) ms69.65 ± (69.55 - 69.76) ms-0.1%✅
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed7.75 ± (7.75 - 7.76) MB7.71 ± (7.70 - 7.71) MB-0.6%✅
runtime.dotnet.threads.count11 ± (11 - 11)11 ± (11 - 11)+0.0%✅
.NET 8 - CallTarget+Inlining+NGEN
process.internal_duration_ms296.04 ± (293.60 - 298.47) ms301.13 ± (298.78 - 303.49) ms+1.7%✅⬆️
process.time_to_main_ms486.96 ± (486.02 - 487.91) ms486.21 ± (485.24 - 487.17) ms-0.2%✅
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed37.55 ± (37.52 - 37.58) MB37.68 ± (37.66 - 37.71) MB+0.4%✅⬆️
runtime.dotnet.threads.count27 ± (27 - 27)27 ± (27 - 27)-0.1%✅

HttpMessageHandler

Metric Master (Mean ± 95% CI) Current (Mean ± 95% CI) Change Status
.NET Framework 4.8 - Baseline
duration201.38 ± (201.33 - 202.23) ms204.84 ± (204.75 - 205.62) ms+1.7%✅⬆️
.NET Framework 4.8 - Bailout
duration208.80 ± (208.51 - 209.44) ms207.93 ± (207.46 - 207.97) ms-0.4%✅
.NET Framework 4.8 - CallTarget+Inlining+NGEN
duration1210.40 ± (1213.11 - 1218.58) ms1211.21 ± (1211.83 - 1217.94) ms+0.1%✅⬆️
.NET Core 3.1 - Baseline
process.internal_duration_ms196.53 ± (196.09 - 196.98) ms196.53 ± (196.02 - 197.04) ms-0.0%✅
process.time_to_main_ms85.27 ± (84.95 - 85.59) ms85.70 ± (85.39 - 86.01) ms+0.5%✅⬆️
runtime.dotnet.exceptions.count3 ± (3 - 3)3 ± (3 - 3)+0.0%✅
runtime.dotnet.mem.committed16.08 ± (16.05 - 16.10) MB16.05 ± (16.03 - 16.08) MB-0.1%✅
runtime.dotnet.threads.count20 ± (20 - 20)20 ± (20 - 20)+0.7%✅⬆️
.NET Core 3.1 - Bailout
process.internal_duration_ms195.33 ± (194.88 - 195.78) ms196.76 ± (196.30 - 197.22) ms+0.7%✅⬆️
process.time_to_main_ms86.10 ± (85.86 - 86.33) ms87.55 ± (87.31 - 87.78) ms+1.7%✅⬆️
runtime.dotnet.exceptions.count3 ± (3 - 3)3 ± (3 - 3)+0.0%✅
runtime.dotnet.mem.committed16.08 ± (16.05 - 16.11) MB16.04 ± (16.02 - 16.06) MB-0.3%✅
runtime.dotnet.threads.count21 ± (21 - 21)21 ± (20 - 21)-1.0%✅
.NET Core 3.1 - CallTarget+Inlining+NGEN
process.internal_duration_ms388.18 ± (386.80 - 389.55) ms388.51 ± (387.15 - 389.87) ms+0.1%✅⬆️
process.time_to_main_ms541.47 ± (540.37 - 542.56) ms547.10 ± (545.76 - 548.44) ms+1.0%✅⬆️
runtime.dotnet.exceptions.count3 ± (3 - 3)3 ± (3 - 3)+0.0%✅
runtime.dotnet.mem.committed58.30 ± (58.06 - 58.53) MB58.51 ± (58.30 - 58.72) MB+0.4%✅⬆️
runtime.dotnet.threads.count30 ± (30 - 30)30 ± (30 - 30)+0.3%✅⬆️
.NET 6 - Baseline
process.internal_duration_ms202.58 ± (202.14 - 203.02) ms202.15 ± (201.64 - 202.65) ms-0.2%✅
process.time_to_main_ms74.68 ± (74.45 - 74.91) ms75.20 ± (74.96 - 75.44) ms+0.7%✅⬆️
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%✅
runtime.dotnet.mem.committed16.34 ± (16.31 - 16.37) MB16.36 ± (16.33 - 16.38) MB+0.1%✅⬆️
runtime.dotnet.threads.count19 ± (19 - 19)19 ± (19 - 19)-0.5%✅
.NET 6 - Bailout
process.internal_duration_ms199.84 ± (199.38 - 200.30) ms201.30 ± (200.96 - 201.63) ms+0.7%✅⬆️
process.time_to_main_ms75.39 ± (75.14 - 75.65) ms75.98 ± (75.77 - 76.19) ms+0.8%✅⬆️
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%✅
runtime.dotnet.mem.committed16.46 ± (16.43 - 16.49) MB16.45 ± (16.43 - 16.47) MB-0.1%✅
runtime.dotnet.threads.count20 ± (20 - 20)20 ± (20 - 20)-0.2%✅
.NET 6 - CallTarget+Inlining+NGEN
process.internal_duration_ms581.88 ± (579.56 - 584.21) ms584.18 ± (581.71 - 586.66) ms+0.4%✅⬆️
process.time_to_main_ms552.78 ± (551.72 - 553.85) ms559.94 ± (558.56 - 561.33) ms+1.3%✅⬆️
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%✅
runtime.dotnet.mem.committed61.25 ± (61.18 - 61.33) MB61.27 ± (61.15 - 61.38) MB+0.0%✅⬆️
runtime.dotnet.threads.count31 ± (31 - 31)31 ± (31 - 31)-0.2%✅
.NET 8 - Baseline
process.internal_duration_ms196.63 ± (196.18 - 197.09) ms199.64 ± (199.21 - 200.06) ms+1.5%✅⬆️
process.time_to_main_ms72.87 ± (72.60 - 73.14) ms73.77 ± (73.51 - 74.03) ms+1.2%✅⬆️
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%✅
runtime.dotnet.mem.committed11.76 ± (11.73 - 11.78) MB11.74 ± (11.72 - 11.76) MB-0.1%✅
runtime.dotnet.threads.count18 ± (18 - 19)19 ± (18 - 19)+0.4%✅⬆️
.NET 8 - Bailout
process.internal_duration_ms195.79 ± (195.39 - 196.19) ms199.23 ± (198.73 - 199.73) ms+1.8%✅⬆️
process.time_to_main_ms73.47 ± (73.24 - 73.71) ms75.03 ± (74.81 - 75.25) ms+2.1%✅⬆️
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%✅
runtime.dotnet.mem.committed11.73 ± (11.71 - 11.75) MB11.77 ± (11.75 - 11.79) MB+0.3%✅⬆️
runtime.dotnet.threads.count19 ± (19 - 19)19 ± (19 - 19)+0.9%✅⬆️
.NET 8 - CallTarget+Inlining+NGEN
process.internal_duration_ms511.84 ± (509.14 - 514.53) ms509.95 ± (507.04 - 512.87) ms-0.4%✅
process.time_to_main_ms499.47 ± (498.57 - 500.36) ms505.92 ± (505.11 - 506.73) ms+1.3%✅⬆️
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%✅
runtime.dotnet.mem.committed51.09 ± (51.06 - 51.12) MB51.11 ± (51.07 - 51.14) MB+0.0%✅⬆️
runtime.dotnet.threads.count30 ± (29 - 30)30 ± (30 - 30)+0.5%✅⬆️
Comparison explanation

Execution-time benchmarks measure the whole time it takes to execute a program, and are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are highlighted in **red**. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

Duration charts
FakeDbCommand (.NET Framework 4.8) 
gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8824) - mean (69ms)  : 67, 71
    master - mean (73ms)  : 70, 76

    section Bailout
    This PR (8824) - mean (75ms)  : 71, 79
    master - mean (76ms)  : 73, 80

    section CallTarget+Inlining+NGEN
    This PR (8824) - mean (1,086ms)  : 1035, 1137
    master - mean (1,083ms)  : 1035, 1131
Loading
FakeDbCommand (.NET Core 3.1) 
gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8824) - mean (108ms)  : 105, 110
    master - mean (114ms)  : 108, 120

    section Bailout
    This PR (8824) - mean (114ms)  : 106, 121
    master - mean (111ms)  : 107, 115

    section CallTarget+Inlining+NGEN
    This PR (8824) - mean (776ms)  : 757, 795
    master - mean (776ms)  : 746, 805
Loading
FakeDbCommand (.NET 6) 
gantt
    title Execution time (ms) FakeDbCommand (.NET 6)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8824) - mean (98ms)  : 93, 104
    master - mean (98ms)  : 93, 103

    section Bailout
    This PR (8824) - mean (100ms)  : 94, 105
    master - mean (100ms)  : 95, 106

    section CallTarget+Inlining+NGEN
    This PR (8824) - mean (937ms)  : 896, 978
    master - mean (938ms)  : 897, 979
Loading
FakeDbCommand (.NET 8) 
gantt
    title Execution time (ms) FakeDbCommand (.NET 8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8824) - mean (94ms)  : 92, 97
    master - mean (94ms)  : 92, 97

    section Bailout
    This PR (8824) - mean (95ms)  : 93, 97
    master - mean (95ms)  : 93, 97

    section CallTarget+Inlining+NGEN
    This PR (8824) - mean (818ms)  : 775, 860
    master - mean (811ms)  : 772, 851
Loading
HttpMessageHandler (.NET Framework 4.8) 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8824) - mean (205ms)  : 201, 210
    master - mean (202ms)  : 197, 206

    section Bailout
    This PR (8824) - mean (208ms)  : 205, 210
    master - mean (209ms)  : 204, 214

    section CallTarget+Inlining+NGEN
    This PR (8824) - mean (1,215ms)  : 1171, 1259
    master - mean (1,216ms)  : 1177, 1255
Loading
HttpMessageHandler (.NET Core 3.1) 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8824) - mean (292ms)  : 283, 301
    master - mean (292ms)  : 286, 298

    section Bailout
    This PR (8824) - mean (294ms)  : 286, 302
    master - mean (291ms)  : 286, 297

    section CallTarget+Inlining+NGEN
    This PR (8824) - mean (978ms)  : 957, 1000
    master - mean (971ms)  : 950, 992
Loading
HttpMessageHandler (.NET 6) 
gantt
    title Execution time (ms) HttpMessageHandler (.NET 6)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8824) - mean (287ms)  : 280, 294
    master - mean (287ms)  : 280, 294

    section Bailout
    This PR (8824) - mean (287ms)  : 280, 293
    master - mean (285ms)  : 279, 291

    section CallTarget+Inlining+NGEN
    This PR (8824) - mean (1,179ms)  : 1147, 1212
    master - mean (1,167ms)  : 1129, 1204
Loading
HttpMessageHandler (.NET 8) 
gantt
    title Execution time (ms) HttpMessageHandler (.NET 8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8824) - mean (284ms)  : 276, 292
    master - mean (280ms)  : 271, 288

    section Bailout
    This PR (8824) - mean (285ms)  : 278, 291
    master - mean (280ms)  : 274, 285

    section CallTarget+Inlining+NGEN
    This PR (8824) - mean (1,047ms)  : 994, 1099
    master - mean (1,045ms)  : 992, 1097
Loading

@andrewlock
andrewlock force-pushed the andrew/css-ddtags-support branch from c4004df to 8fe4238 Compare June 26, 2026 08:37
@andrewlock
andrewlock force-pushed the andrew/css-cardinality-limits branch from 9e8abd4 to 90bb61c Compare June 26, 2026 08:37
@andrewlock
andrewlock marked this pull request as ready for review June 26, 2026 11:31
@andrewlock
andrewlock requested review from a team as code owners June 26, 2026 11:31
chatgpt-codex-connector[bot]
chatgpt-codex-connector Bot reviewed Jun 26, 2026
View reviewed changes

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 90bb61c0d6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread tracer/src/Datadog.Trace/Agent/StatsBuffer.cs
Comment thread tracer/src/Datadog.Trace/Agent/StatsAggregator.cs Outdated
Comment thread tracer/src/Datadog.Trace/Agent/StatsAggregator.cs
Comment on lines +768 to +770
var encodedAdditionalTags = (key.CardinalityLimitedFields & StatsCardinalityLimitedFields.AdditionalMetricTags) == StatsCardinalityLimitedFields.AdditionalMetricTags
? BlockedByTracerSentinelList
: GetEncodedAdditionalTags(span, in additionalTagResults);

@chatgpt-codex-connector chatgpt-codex-connector Bot Jun 26, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve keys when masking additional metric tags

When DD_TRACE_STATS_ADDITIONAL_TAGS is configured and the additional-tag cardinality cap is hit, this replaces the entire AdditionalMetricTags list with a single bare tracer_blocked_value entry. These entries are otherwise serialized as key:value pairs, and oversized values still use region:tracer_blocked_value; dropping the configured key means the stats payload no longer identifies which primary-tag dimension was collapsed, so those overflowed spans cannot be aggregated as the masked region/tenant tag. Build masked key:tracer_blocked_value entries for the present configured tags instead of the bare sentinel.

Useful? React with 👍 / 👎.

@andrewlock andrewlock Jun 26, 2026 •
edited
Loading

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After discussion offline, the current implementation is the correct approach, for consistency

@andrewlock
andrewlock force-pushed the andrew/css-ddtags-support branch from 8fe4238 to a146d8f Compare June 26, 2026 12:40
@andrewlock
andrewlock force-pushed the andrew/css-cardinality-limits branch from 90bb61c to a8f8150 Compare June 26, 2026 12:40
andrewlock added a commit that referenced this pull request Jun 30, 2026
@andrewlock
Minor and performance updates for client-side-stats (#8822)
8b1935f 
## Summary of changes

- Mark testing only methods with `[TestingAndPrivateOnly]`
- Pre-encode header tags for static UTF-8 messagepack keys
- Avoid re-encoding the peer_tags, seeing as we already have that data

## Reason for change

Minor tweaks (mostly perf) before adding the additional
client-side-stats features

## Implementation details

Mostly self-evident, written as distinct commits. The most interesting
one is avoiding the additional interpolated string when encoding
peer_tags. We also avoid re-encoding the tags seeing as we already do
that work up front

## Test coverage

Covered by existing tests, this is all refactoring

## Other details

Part of a client-side stats stack of PRs.

- #8822 👈
- #8766
- #8823
- #8824

https://datadoghq.atlassian.net/browse/APMLP-1446
@andrewlock
andrewlock force-pushed the andrew/css-ddtags-support branch from a146d8f to be245bd Compare June 30, 2026 07:47
@andrewlock
andrewlock force-pushed the andrew/css-cardinality-limits branch from a8f8150 to 7311734 Compare June 30, 2026 07:47
NachoEchevarria
NachoEchevarria approved these changes Jul 1, 2026
View reviewed changes

@NachoEchevarria NachoEchevarria left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While not familiar with some parts of the code, LGTM. Nice!

bouwkast
bouwkast reviewed Jul 1, 2026
View reviewed changes

@bouwkast bouwkast left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will pick up the rest of the review tomorrow

Comment thread tracer/src/Datadog.Trace/Agent/StatsCardinalityReporter.cs Outdated
Comment thread tracer/src/Datadog.Trace/Telemetry/Metrics/MetricTags.cs Outdated
@lucaspimentel
lucaspimentel requested a review from Copilot July 1, 2026 21:56
Copilot AI reviewed Jul 1, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR implements configurable cardinality controls for client-side stats aggregation to prevent unbounded memory growth, and adds telemetry + health metrics to observe when dimensions are collapsed into a sentinel/overflow bucket.

Changes:

  • Added per-field and whole-key cardinality caps for client-side stats buckets, plus an overflow bucket and per-flush reset behavior.
  • Introduced a new telemetry count metric (stats_collapsed_spans) with dimensions for which fields were collapsed and whether tags were oversized.
  • Added/updated unit tests and generated telemetry/configuration artifacts to cover the new behavior.

Reviewed changes

Copilot reviewed 24 out of 52 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
tracer/test/Datadog.Trace.Tests/Telemetry/Metrics/MetricTests.cs Updates expected metric tag dimensions for stats_collapsed_spans.
tracer/test/Datadog.Trace.Tests/Telemetry/Metrics/common_metrics.json Adds stats_collapsed_spans definition to the common metrics contract.
tracer/test/Datadog.Trace.Tests/Configuration/TracerSettingsTests.cs Adds coverage for default/validated cardinality limit settings.
tracer/test/Datadog.Trace.Tests/Agent/StatsBufferTests.cs Updates StatsBuffer construction and adds trilean IsTraceRoot serialization test.
tracer/test/Datadog.Trace.Tests/Agent/ApiTests.cs Updates StatsBuffer construction for API stats tests.
tracer/src/Datadog.Trace/TracerManagerFactory.cs Threads statsd into StatsAggregator.Create(...).
tracer/src/Datadog.Trace/Telemetry/Metrics/MetricTags.cs Adds collapsed/oversized tag enums for new telemetry metric.
tracer/src/Datadog.Trace/Telemetry/Metrics/Count.cs Declares the stats_collapsed_spans telemetry metric.
tracer/src/Datadog.Trace/Generated/netstandard2.0/Datadog.Trace.SourceGenerators/TelemetryMetricGenerator/NullMetricsTelemetryCollector_Count.g.cs Generated: adds no-op recorder for new count metric.
tracer/src/Datadog.Trace/Generated/netstandard2.0/Datadog.Trace.SourceGenerators/TelemetryMetricGenerator/MetricsTelemetryCollector_Count.g.cs Generated: expands count buffer and indices to include new metric.
tracer/src/Datadog.Trace/Generated/netstandard2.0/Datadog.Trace.SourceGenerators/TelemetryMetricGenerator/IMetricsTelemetryCollector_Count.g.cs Generated: adds interface method for new metric.
tracer/src/Datadog.Trace/Generated/netstandard2.0/Datadog.Trace.SourceGenerators/TelemetryMetricGenerator/CountExtensions.g.cs Generated: adds name mapping + length update.
tracer/src/Datadog.Trace/Generated/netstandard2.0/Datadog.Trace.SourceGenerators/TelemetryMetricGenerator/CiVisibilityMetricsTelemetryCollector_Count.g.cs Generated: adds CI Visibility no-op recorder for new metric.
tracer/src/Datadog.Trace/Generated/netstandard2.0/Datadog.Trace.SourceGenerators/EnumExtensionsGenerator/CollapsedStatsFieldsExtensions_EnumExtensions.g.cs Generated: enum fast-string helpers for collapsed-fields tag.
tracer/src/Datadog.Trace/Generated/netstandard2.0/Datadog.Trace.SourceGenerators/ConfigurationKeysGenerator/ConfigurationKeys.g.cs Generated: adds new stats cardinality configuration keys.
tracer/src/Datadog.Trace/Generated/netcoreapp3.1/Datadog.Trace.SourceGenerators/TelemetryMetricGenerator/NullMetricsTelemetryCollector_Count.g.cs Generated: adds no-op recorder for new count metric.
tracer/src/Datadog.Trace/Generated/netcoreapp3.1/Datadog.Trace.SourceGenerators/TelemetryMetricGenerator/MetricsTelemetryCollector_Count.g.cs Generated: expands count buffer and indices to include new metric.
tracer/src/Datadog.Trace/Generated/netcoreapp3.1/Datadog.Trace.SourceGenerators/TelemetryMetricGenerator/IMetricsTelemetryCollector_Count.g.cs Generated: adds interface method for new metric.
tracer/src/Datadog.Trace/Generated/netcoreapp3.1/Datadog.Trace.SourceGenerators/TelemetryMetricGenerator/CountExtensions.g.cs Generated: adds name mapping + length update.
tracer/src/Datadog.Trace/Generated/netcoreapp3.1/Datadog.Trace.SourceGenerators/TelemetryMetricGenerator/CiVisibilityMetricsTelemetryCollector_Count.g.cs Generated: adds CI Visibility no-op recorder for new metric.
tracer/src/Datadog.Trace/Generated/netcoreapp3.1/Datadog.Trace.SourceGenerators/EnumExtensionsGenerator/CollapsedStatsFieldsExtensions_EnumExtensions.g.cs Generated: enum fast-string helpers for collapsed-fields tag.
tracer/src/Datadog.Trace/Generated/netcoreapp3.1/Datadog.Trace.SourceGenerators/ConfigurationKeysGenerator/ConfigurationKeys.g.cs Generated: adds new stats cardinality configuration keys.
tracer/src/Datadog.Trace/Generated/net6.0/Datadog.Trace.SourceGenerators/TelemetryMetricGenerator/NullMetricsTelemetryCollector_Count.g.cs Generated: adds no-op recorder for new count metric.
tracer/src/Datadog.Trace/Generated/net6.0/Datadog.Trace.SourceGenerators/TelemetryMetricGenerator/MetricsTelemetryCollector_Count.g.cs Generated: expands count buffer and indices to include new metric.
tracer/src/Datadog.Trace/Generated/net6.0/Datadog.Trace.SourceGenerators/TelemetryMetricGenerator/IMetricsTelemetryCollector_Count.g.cs Generated: adds interface method for new metric.
tracer/src/Datadog.Trace/Generated/net6.0/Datadog.Trace.SourceGenerators/TelemetryMetricGenerator/CountExtensions.g.cs Generated: adds name mapping + length update.
tracer/src/Datadog.Trace/Generated/net6.0/Datadog.Trace.SourceGenerators/TelemetryMetricGenerator/CiVisibilityMetricsTelemetryCollector_Count.g.cs Generated: adds CI Visibility no-op recorder for new metric.
tracer/src/Datadog.Trace/Generated/net6.0/Datadog.Trace.SourceGenerators/EnumExtensionsGenerator/CollapsedStatsFieldsExtensions_EnumExtensions.g.cs Generated: enum fast-string helpers for collapsed-fields tag.
tracer/src/Datadog.Trace/Generated/net6.0/Datadog.Trace.SourceGenerators/ConfigurationKeysGenerator/ConfigurationKeys.g.cs Generated: adds new stats cardinality configuration keys.
tracer/src/Datadog.Trace/Generated/net461/Datadog.Trace.SourceGenerators/TelemetryMetricGenerator/NullMetricsTelemetryCollector_Count.g.cs Generated: adds no-op recorder for new count metric.
tracer/src/Datadog.Trace/Generated/net461/Datadog.Trace.SourceGenerators/TelemetryMetricGenerator/MetricsTelemetryCollector_Count.g.cs Generated: expands count buffer and indices to include new metric.
tracer/src/Datadog.Trace/Generated/net461/Datadog.Trace.SourceGenerators/TelemetryMetricGenerator/IMetricsTelemetryCollector_Count.g.cs Generated: adds interface method for new metric.
tracer/src/Datadog.Trace/Generated/net461/Datadog.Trace.SourceGenerators/TelemetryMetricGenerator/CountExtensions.g.cs Generated: adds name mapping + length update.
tracer/src/Datadog.Trace/Generated/net461/Datadog.Trace.SourceGenerators/TelemetryMetricGenerator/CiVisibilityMetricsTelemetryCollector_Count.g.cs Generated: adds CI Visibility no-op recorder for new metric.
tracer/src/Datadog.Trace/Generated/net461/Datadog.Trace.SourceGenerators/EnumExtensionsGenerator/CollapsedStatsFieldsExtensions_EnumExtensions.g.cs Generated: enum fast-string helpers for collapsed-fields tag.
tracer/src/Datadog.Trace/Generated/net461/Datadog.Trace.SourceGenerators/ConfigurationKeysGenerator/ConfigurationKeys.g.cs Generated: adds new stats cardinality configuration keys.
tracer/src/Datadog.Trace/DogStatsd/TracerMetricNames.cs Adds tracer-internal StatsD metric name for collapsed spans.
tracer/src/Datadog.Trace/DogStatsd/StatsdConsumer.cs Adds a StatsD consumer flag for StatsAggregator.
tracer/src/Datadog.Trace/Configuration/TracerSettings.cs Adds new cardinality limit settings and docs.
tracer/src/Datadog.Trace/Configuration/supported-configurations.yaml Documents new configuration keys and defaults.
tracer/src/Datadog.Trace/Ci/Agent/ApmAgentWriter.cs Updates stats aggregator creation to pass statsdManager.
tracer/src/Datadog.Trace/Agent/StatsCardinalityTruncatedFields.cs New flags enum to track which fields were oversized-truncated.
tracer/src/Datadog.Trace/Agent/StatsCardinalityReporter.cs New telemetry/health-metrics/logging reporter for collapses.
tracer/src/Datadog.Trace/Agent/StatsCardinalityLimiter.cs New per-field cardinality limiter for stats keys.
tracer/src/Datadog.Trace/Agent/StatsCardinalityLimitedFields.cs New flags enum for which key fields were cardinality-collapsed.
tracer/src/Datadog.Trace/Agent/StatsBuffer.cs Adds limiter/reporter plumbing, active-bucket counting, trilean encoding.
tracer/src/Datadog.Trace/Agent/StatsAggregator.cs Implements whole-key overflow bucket, per-field caps, resource truncation, and statsd emission.
tracer/src/Datadog.Trace/Agent/StatsAggregationKey.cs Converts key to readonly record struct, adds trilean root + collapse/truncation fields.
tracer/src/Datadog.Trace/Agent/NullStatsAggregator.cs Updates key construction to include new truncatedFields argument.
tracer/src/Datadog.Trace/Agent/DiscoveryService/DiscoveryService.cs Parses feature_flags from agent discovery response.
tracer/src/Datadog.Trace/Agent/DiscoveryService/AgentConfiguration.cs Stores agent FeatureFlags in configuration model.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread tracer/src/Datadog.Trace/Agent/StatsCardinalityLimiter.cs Outdated
return limitsApplied;
}

private readonly struct Limiter<T>

@andrewlock andrewlock Jul 2, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's funny, this is the second time an AI has said this, but I was pretty sure its wrong, and when you follow up, it admits it:

The review is not correct for this code.

Limiter<T> being a readonly struct means its instance fields cannot be reassigned, but it does not make the referenced HashSet<T> immutable.

So this line:

_seen.Add(value);

mutates the HashSet<T> object referenced by _seen. That mutation persists.

Even if a defensive copy of the Limiter<T> struct were made, the copy would still contain the same _seen reference pointing to the same HashSet<T> instance. So Add() and Clear() would still affect the real set.

Also, because Limiter<T> itself is a readonly struct, the compiler knows its instance members cannot mutate the struct fields, so the usual “readonly field + mutable struct method = defensive copy” problem does not apply in the same way it would for a non-readonly mutable struct.

So this claim is wrong:

updates to _seen won't persist and cardinality limiting will effectively never trigger

They will persist.

That said, if the AI is constantly confused, we may as well just change it to a sealed class anyway I guess. These are long lived objects, so there's not really any allocation difference. And if it shuts up the robots, so be it

@andrewlock andrewlock Jul 2, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Switched it to sealed class just to keep the clankers happy

Comment thread tracer/src/Datadog.Trace/Agent/StatsCardinalityReporter.cs Outdated
Comment thread tracer/src/Datadog.Trace/Agent/StatsCardinalityReporter.cs
Comment thread tracer/src/Datadog.Trace/Agent/StatsCardinalityReporter.cs
Comment thread tracer/src/Datadog.Trace/Configuration/supported-configurations.yaml Outdated
Comment thread tracer/src/Datadog.Trace/Configuration/TracerSettings.cs
andrewlock added a commit that referenced this pull request Jul 2, 2026
@andrewlock @claude
Implement span-derived primary tags (AKA Additional tags) on Client-S…
7fc15f3 
…ide-Stats computation (#8766)

## Summary of changes

Implements span-derived primary tags (also called "Additional metric
tags")

## Reason for change

This is a feature requested by customers, and defined in [the
RFC](https://datadoghq.atlassian.net/wiki/spaces/APM/history/6482919540/PENDING+Span-Derived+Primary+Tags+-+V1)

## Implementation details

- Add `DD_TRACE_STATS_ADDITIONAL_TAGS` for controlling which span-tags
to include as metric dimensions in client-side-stats
- This is an experimental feature, so is gated on
`DD_TRACE_EXPERIMENTAL_FEATURES_ENABLED` currently
- Include the defined tags as additional dimensions in the metric key
(similar to peer tags)
- Apply cardinality limits to avoid memory leaks
- NOTE: the cardinality limits implemented here are somewhat superseded
by [a subsequent
RFC](https://datadoghq.atlassian.net/wiki/spaces/APM/pages/6821151019/PENDING+Cardinality+Limits),
and so are only partially implemented here.

## Test coverage

- Added unit tests for all the behaviour
- System tests TBC - once they're implemented, I'll enable them and
confirm they're worked

## Other details

Some aspects of the RFC are missing/modified based on other discussions
- The telemetry metric in the span tags RFC is not defined, we will add
the cardinality limit version instead.
- The required logging will be added as part of the cardinality RFC
implementation
- Cardinality protection doesn't entirely match the RFC, as it follows
the later RFC

Part of a stack

- #8822
- #8766 👈
- #8823
- #8824

https://datadoghq.atlassian.net/browse/APMLP-1446

---------

Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
@andrewlock
andrewlock force-pushed the andrew/css-ddtags-support branch from be245bd to d673813 Compare July 2, 2026 08:56
Base automatically changed from andrew/css-ddtags-support to master July 2, 2026 10:20
andrewlock added a commit that referenced this pull request Jul 2, 2026
@andrewlock
Implement support for DD_TAGS in Client-Side stats (#8823)
dcedc21 
## Summary of changes

Adds support for adding `DD_TAGS` as span-derived primary tags (AKA
Additional tags) in Client Side Stats

## Reason for change

Implements [the
RFC](https://datadoghq.atlassian.net/wiki/spaces/APM/pages/6607667207/PENDING+DD_TAGS+Derived+Primary+Tags)
for adding `DD_TAGS` to the client-side stats payload.

## Implementation details

Relatively simple: we just thread any `DD_TAGS` value through to the
payload, and the backend/agent then ensures they're added as
span-derived primary tags. There's no cardinality issues at play here,
which simplifies things, and we can just include them in every payload

## Test coverage

Added some unit tests, system tests to follow...

## Other details

Part of a stack

- #8822
- #8766
- https://github.com/DataDog/dd-trace-dotnet/pull/8823👈
- #8824

https://datadoghq.atlassian.net/browse/APMLP-1446
andrewlock added 11 commits July 2, 2026 11:39
@andrewlock
Convert StatsAggregationKey to readonly record struct
f5be2d8 
Should be a no-op change
@andrewlock
Model IsTraceRoot as a bool? trilean
7bf8232 
The wire field is a trilean (NOT_SET=0, TRUE=1, FALSE=2), and we will need to send NOT_SET for cardinality limited cases
@andrewlock
Add resource byte-length cap with big_resource feature flag
4b2d316 
Parse the agent's feature_flags from /info and expose them on AgentConfiguration. In the stats pipeline, truncate each span's resource to 5000 UTF-8 bytes (15000 when the agent advertises big_resource) after normalization/obfuscation, reusing TraceUtil.TruncateUTF8.
@andrewlock
Add per-field and whole-key stats cardinality limit config
6133006 
Add four DD_TRACE_STATS_*_CARDINALITY_LIMIT keys (resource=1024, http_endpoint=512, peer_tags=1024, computation_buckets=2048) mirroring the existing additional-tags limit plumbing: yaml + regenerated ConfigurationKeys, TracerSettings properties with x>0 validation, config_norm_rules, and TracerSettings tests.
@andrewlock
Add encapsulated per-field and whole-key stats cardinality collapse
db99b14 
Replace the bespoke additional-tags bucket-count cap with a uniform per-field admission-stop.
@andrewlock
Add telemetry metric for stats_collapsed_spans
324a5e1
@andrewlock
Inject StatsdManager into StatsAggregator, and toggle enabled
2990f23
@andrewlock
Record telemetry about collapsed spans
c99312a
@andrewlock
Fix tests and add some more
ab2f89e
@andrewlock
Fix naming collapsed:whole-key -> collapsed:whole_key
9f8ef4b
@andrewlock
Fix copilot feedback
e146b42
@andrewlock
andrewlock force-pushed the andrew/css-cardinality-limits branch from 7311734 to e146b42 Compare July 2, 2026 10:40
bouwkast
bouwkast approved these changes Jul 2, 2026
View reviewed changes

@bouwkast bouwkast left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@andrewlock
andrewlock merged commit c14a299 into master Jul 2, 2026
141 checks passed
@andrewlock
andrewlock deleted the andrew/css-cardinality-limits branch July 2, 2026 12:30
@github-actions github-actions Bot added this to the vNext-v3 milestone Jul 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments
@chatgpt-codex-connector chatgpt-codex-connector[bot] chatgpt-codex-connector[bot] left review comments
@lucaspimentel lucaspimentel lucaspimentel approved these changes
@bouwkast bouwkast bouwkast approved these changes
@NachoEchevarria NachoEchevarria NachoEchevarria approved these changes

Assignees

No one assigned

Labels

area:client-side-stats area:tracer The core tracer library (Datadog.Trace, does not include OpenTracing, native code, or integrations) type:new-feature

Projects

None yet

Milestone

3.49.0

Development

Successfully merging this pull request may close these issues.

5 participants

@andrewlock @lucaspimentel @bouwkast @NachoEchevarria