Implement span-derived primary tags (AKA Additional tags) on Client-Side-Stats computation#8766
Merged
Merged
Conversation
This comment has been minimized.
This comment has been minimized.
andrewlock
force-pushed
the
andrew/span-derived-primary-tags
branch
from
June 8, 2026 15:08
b535a3d to
529937d
Compare
andrewlock
marked this pull request as draft
June 8, 2026 15:12
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b535a3dc2a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
BenchmarksBenchmark execution time: 2026-06-30 08:36:24 Comparing candidate commit 889f756 in PR branch Found 0 performance improvements and 3 performance regressions! Performance is the same for 69 metrics, 0 unstable metrics, 63 known flaky benchmarks, 63 flaky benchmarks without significant changes.
|
Execution-Time Benchmarks Report ⏱️Execution-time results for samples comparing This PR (8766) and master. ✅ No regressions detected - check the details below Full Metrics ComparisonFakeDbCommand
HttpMessageHandler
Comparison explanationExecution-time benchmarks measure the whole time it takes to execute a program, and are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are highlighted in **red**. The following thresholds were used for comparing the execution times:
Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard. Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph). Duration chartsFakeDbCommand (.NET Framework 4.8)gantt
title Execution time (ms) FakeDbCommand (.NET Framework 4.8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8766) - mean (71ms) : 68, 74
master - mean (71ms) : 67, 75
section Bailout
This PR (8766) - mean (77ms) : 74, 80
master - mean (75ms) : 73, 76
section CallTarget+Inlining+NGEN
This PR (8766) - mean (1,080ms) : 1030, 1130
master - mean (1,080ms) : 1034, 1126
FakeDbCommand (.NET Core 3.1)gantt
title Execution time (ms) FakeDbCommand (.NET Core 3.1)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8766) - mean (112ms) : 105, 119
master - mean (111ms) : 106, 116
section Bailout
This PR (8766) - mean (112ms) : 106, 118
master - mean (110ms) : 109, 112
section CallTarget+Inlining+NGEN
This PR (8766) - mean (774ms) : 757, 792
master - mean (770ms) : 751, 789
FakeDbCommand (.NET 6)gantt
title Execution time (ms) FakeDbCommand (.NET 6)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8766) - mean (100ms) : 93, 107
master - mean (99ms) : 94, 105
section Bailout
This PR (8766) - mean (98ms) : 95, 101
master - mean (97ms) : 95, 100
section CallTarget+Inlining+NGEN
This PR (8766) - mean (935ms) : 900, 971
master - mean (936ms) : 890, 982
FakeDbCommand (.NET 8)gantt
title Execution time (ms) FakeDbCommand (.NET 8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8766) - mean (94ms) : 90, 98
master - mean (94ms) : 91, 97
section Bailout
This PR (8766) - mean (100ms) : 92, 109
master - mean (98ms) : 93, 104
section CallTarget+Inlining+NGEN
This PR (8766) - mean (813ms) : 779, 847
master - mean (812ms) : 776, 848
HttpMessageHandler (.NET Framework 4.8)gantt
title Execution time (ms) HttpMessageHandler (.NET Framework 4.8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8766) - mean (204ms) : 199, 208
master - mean (204ms) : 198, 210
section Bailout
This PR (8766) - mean (208ms) : 204, 212
master - mean (208ms) : 203, 212
section CallTarget+Inlining+NGEN
This PR (8766) - mean (1,216ms) : 1182, 1250
master - mean (1,217ms) : 1174, 1259
HttpMessageHandler (.NET Core 3.1)gantt
title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8766) - mean (293ms) : 288, 299
master - mean (292ms) : 286, 298
section Bailout
This PR (8766) - mean (293ms) : 288, 298
master - mean (292ms) : 286, 297
section CallTarget+Inlining+NGEN
This PR (8766) - mean (979ms) : 959, 1000
master - mean (983ms) : 950, 1017
HttpMessageHandler (.NET 6)gantt
title Execution time (ms) HttpMessageHandler (.NET 6)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8766) - mean (287ms) : 280, 294
master - mean (288ms) : 282, 293
section Bailout
This PR (8766) - mean (286ms) : 282, 291
master - mean (287ms) : 282, 293
section CallTarget+Inlining+NGEN
This PR (8766) - mean (1,173ms) : 1138, 1208
master - mean (1,175ms) : 1139, 1211
HttpMessageHandler (.NET 8)gantt
title Execution time (ms) HttpMessageHandler (.NET 8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8766) - mean (284ms) : 278, 289
master - mean (286ms) : 282, 291
section Bailout
This PR (8766) - mean (283ms) : 279, 287
master - mean (287ms) : 282, 292
section CallTarget+Inlining+NGEN
This PR (8766) - mean (1,049ms) : 1009, 1090
master - mean (1,047ms) : 991, 1103
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
andrewlock
force-pushed
the
andrew/span-derived-primary-tags
branch
from
June 11, 2026 16:23
529937d to
cff7ab2
Compare
andrewlock
force-pushed
the
andrew/span-derived-primary-tags
branch
from
June 24, 2026 15:09
cff7ab2 to
81ee269
Compare
andrewlock
changed the title
andrewlock
added
the
area:tracer
andrewlock
changed the base branch from
master
to
andrew/css-encoding-improvements-2
June 24, 2026 15:50
This was referenced Jun 24, 2026
andrewlock
force-pushed
the
andrew/css-encoding-improvements-2
branch
from
June 26, 2026 08:37
ba84f2d to
ab84d18
Compare
andrewlock
force-pushed
the
andrew/span-derived-primary-tags
branch
from
June 26, 2026 08:37
d0a473c to
c6898d1
Compare
andrewlock
marked this pull request as ready for review
June 26, 2026 11:30
andrewlock
force-pushed
the
andrew/span-derived-primary-tags
branch
from
June 26, 2026 12:39
c6898d1 to
f99f6dd
Compare
andrewlock
added a commit
that referenced
this pull request
Jun 30, 2026
## Summary of changes - Mark testing only methods with `[TestingAndPrivateOnly]` - Pre-encode header tags for static UTF-8 messagepack keys - Avoid re-encoding the peer_tags, seeing as we already have that data ## Reason for change Minor tweaks (mostly perf) before adding the additional client-side-stats features ## Implementation details Mostly self-evident, written as distinct commits. The most interesting one is avoiding the additional interpolated string when encoding peer_tags. We also avoid re-encoding the tags seeing as we already do that work up front ## Test coverage Covered by existing tests, this is all refactoring ## Other details Part of a client-side stats stack of PRs. - #8822 👈 - #8766 - #8823 - #8824 https://datadoghq.atlassian.net/browse/APMLP-1446
Adds two tracer-side config keys for span-derived primary tags (client-side stats additional dimensions): - DD_TRACE_STATS_ADDITIONAL_TAGS: comma-separated span tag keys to extract as extra aggregation dimensions. Gated behind DD_TRACE_EXPERIMENTAL_FEATURES_ENABLED including DD_TRACE_STATS_ADDITIONAL_TAGS. Deduplicated, sorted, capped at 6 keys (excess dropped with a warn log). - DD_TRACE_STATS_ADDITIONAL_TAGS_CARDINALITY_LIMIT: per-flush-bucket cap on distinct stat entries with additional tags. Defaults to 100; invalid values (<=0) fall back to the default with a warn log. Wires these through TracerSettings and the telemetry normalization rules. No aggregation behaviour yet. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Adds the schema for span-derived primary tags without any extraction behaviour yet: - Add `AdditionalMetricTagsHash` to `StatsAggregationKey`. - Add `AdditionalMetricTags` to `StatsBucket`
Wires up extraction of the configured span-derived additional-tags: - Pre-encode the configured tag keys into `AdditionalTagKey` structs at construction. - `BuildKey` computes a hash of the tag values, similar to peer tags, and stores it in the aggregation key - On the cold path, the encoded "key:value" pairs are stored on the bucket. we could potentially optimize this in a follow up to avoid the interpolation -Values longer than 200 chars are replaced with "blocked_by_tracer" (consistently in both hash and encoding).
Implements the per-flush-bucket cardinality protection for span-derived additional tags: - A per-bucket counter tracks distinct stat entries admitted with additional tags. Once DD_TRACE_STATS_ADDITIONAL_TAGS_CARDINALITY_LIMIT is reached, all additional tags are replaced with "blocked_by_tracer". - The span's base stats are never blocked, just the additional spans - Adds a log (once per flush cycle) when something is blocked - The cardinality counter and log guard reset on each flush - Deferred the telemetry metric for now
Writes the span-derived primary tags as `AdditionalMetricTags` in the ClientGroupedStats msgpack payload
…he RFC's cross-SDK tracer_blocked_value.
andrewlock
force-pushed
the
andrew/span-derived-primary-tags
branch
from
June 30, 2026 07:47
f99f6dd to
889f756
Compare
lucaspimentel
approved these changes
Jul 1, 2026
andrewlock
added a commit
that referenced
this pull request
Jul 2, 2026
## Summary of changes Adds support for adding `DD_TAGS` as span-derived primary tags (AKA Additional tags) in Client Side Stats ## Reason for change Implements [the RFC](https://datadoghq.atlassian.net/wiki/spaces/APM/pages/6607667207/PENDING+DD_TAGS+Derived+Primary+Tags) for adding `DD_TAGS` to the client-side stats payload. ## Implementation details Relatively simple: we just thread any `DD_TAGS` value through to the payload, and the backend/agent then ensures they're added as span-derived primary tags. There's no cardinality issues at play here, which simplifies things, and we can just include them in every payload ## Test coverage Added some unit tests, system tests to follow... ## Other details Part of a stack - #8822 - #8766 - https://github.com/DataDog/dd-trace-dotnet/pull/8823👈 - #8824 https://datadoghq.atlassian.net/browse/APMLP-1446
andrewlock
added a commit
that referenced
this pull request
Jul 2, 2026
## Summary of changes Implements the Client-Side Stats [cardinality control RFC](https://datadoghq.atlassian.net/wiki/spaces/APM/pages/6821151019/PENDING+Cardinality+Limits) ## Reason for change Currently, we have a risk of unbounded memory growth caused by cardinality explosion. This has not been a big issue so far as CSS is off by default for .NET. However, resource name explosion (in particular), as well as span-derived primary tags and peer tags have the potential to cause OOM issues. This PR adds strict (configurable) caps to prevent issues from runaway memory growth. ## Implementation details Largely broken into separate distinct commits - Convert `StatsAggregationKey` to `readonly record struct`, seeing as that's basically what it was anyway - Make `IsTraceRoot` nullable (required for "overflow bucket") - Make resource length limit configurable (to support "big_resource" feature from agent) - Add per-field cardinality limitation to resource, httpendoint, peertags, additional tags - Per-field limits stop a single very high cardinality field rendering everything else useless - Only added to these 4 fields for now, but we can expand if necessary - Add whole-key cardinality limit - Essentially a cap on the maximum number of buckets we'll allocate - All additional stats flow into an "overflow" bucket - Add telemetry/heath-metrics/logging for tracking overflow occurences - Add some tests ## Test coverage Added a bunch of unit tests. Waiting on system tests to confirm the implementation. ## Other details Part of a stack - #8822 - #8766 - #8823 - #8824 👈 https://datadoghq.atlassian.net/browse/APMLP-1446
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary of changes
Implements span-derived primary tags (also called "Additional metric tags")
Reason for change
This is a feature requested by customers, and defined in the RFC
Implementation details
DD_TRACE_STATS_ADDITIONAL_TAGSfor controlling which span-tags to include as metric dimensions in client-side-statsDD_TRACE_EXPERIMENTAL_FEATURES_ENABLEDcurrentlyTest coverage
Other details
Some aspects of the RFC are missing/modified based on other discussions
Part of a stack
DD_TAGSin Client-Side stats #8823https://datadoghq.atlassian.net/browse/APMLP-1446