Implement Client-Side Stats (CSS) 1.2.0#8420
Merged
andrewlock merged 48 commits intoApr 20, 2026
Merged
Conversation
andrewlock
added
the
area:tracer
andrewlock
added
the
AI Generated
lucaspimentel
changed the title
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e52cc41a00
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+463
to
+466
| { | ||
| if (span.Context.ParentId is null or 0) | ||
| { | ||
| return !filter.ShouldKeepTrace(span); |
There was a problem hiding this comment.
Trace filtering only runs when a span has ParentId null/0, so chunks where every span has a non-zero parent (common in distributed traces with propagated context, or some partial chunks) bypass filter_tags/ignore_resources entirely. In those cases IsTraceFiltered returns false and filtered traces are kept and counted in stats, which defeats agent-configured filtering for a real production traffic pattern.
Useful? React with 👍 / 👎.
Member
There was a problem hiding this comment.
Interesting. We should probably do whatever the Go agent is doing.
Member
Author
There was a problem hiding this comment.
So. This is where things get very interesting 😅 Will discuss this one offline I think 😄
Member
Author
There was a problem hiding this comment.
Mostly fixed this in #8436, though not strictly the same as the Go agent does. Plus this is going to bork with partial flushing (both sides)
BenchmarksBenchmark execution time: 2026-04-16 09:05:29 Comparing candidate commit e9ee48d in PR branch Found 0 performance improvements and 1 performance regressions! Performance is the same for 26 metrics, 0 unstable metrics, 87 known flaky benchmarks.
|
Execution-Time Benchmarks Report ⏱️Execution-time results for samples comparing This PR (8420) and master. ✅ No regressions detected - check the details below Full Metrics ComparisonFakeDbCommand
HttpMessageHandler
Comparison explanationExecution-time benchmarks measure the whole time it takes to execute a program, and are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are highlighted in **red**. The following thresholds were used for comparing the execution times:
Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard. Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph). Duration chartsFakeDbCommand (.NET Framework 4.8)gantt
title Execution time (ms) FakeDbCommand (.NET Framework 4.8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8420) - mean (72ms) : 70, 74
master - mean (72ms) : 70, 74
section Bailout
This PR (8420) - mean (75ms) : 74, 77
master - mean (76ms) : 74, 78
section CallTarget+Inlining+NGEN
This PR (8420) - mean (1,065ms) : 1024, 1107
master - mean (1,076ms) : 1010, 1141
FakeDbCommand (.NET Core 3.1)gantt
title Execution time (ms) FakeDbCommand (.NET Core 3.1)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8420) - mean (112ms) : 109, 115
master - mean (114ms) : 110, 117
section Bailout
This PR (8420) - mean (114ms) : 111, 116
master - mean (115ms) : 112, 118
section CallTarget+Inlining+NGEN
This PR (8420) - mean (786ms) : 763, 809
master - mean (784ms) : 764, 805
FakeDbCommand (.NET 6)gantt
title Execution time (ms) FakeDbCommand (.NET 6)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8420) - mean (100ms) : 96, 103
master - mean (100ms) : 96, 103
section Bailout
This PR (8420) - mean (100ms) : 98, 102
master - mean (101ms) : 98, 104
section CallTarget+Inlining+NGEN
This PR (8420) - mean (931ms) : 893, 969
master - mean (932ms) : 899, 964
FakeDbCommand (.NET 8)gantt
title Execution time (ms) FakeDbCommand (.NET 8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8420) - mean (98ms) : 95, 100
master - mean (98ms) : 94, 103
section Bailout
This PR (8420) - mean (99ms) : 97, 101
master - mean (99ms) : 97, 101
section CallTarget+Inlining+NGEN
This PR (8420) - mean (813ms) : 779, 847
master - mean (814ms) : 781, 848
HttpMessageHandler (.NET Framework 4.8)gantt
title Execution time (ms) HttpMessageHandler (.NET Framework 4.8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8420) - mean (191ms) : 187, 195
master - mean (191ms) : 187, 194
section Bailout
This PR (8420) - mean (194ms) : 192, 196
master - mean (194ms) : 192, 196
section CallTarget+Inlining+NGEN
This PR (8420) - mean (1,144ms) : 1096, 1192
master - mean (1,142ms) : 1100, 1184
HttpMessageHandler (.NET Core 3.1)gantt
title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8420) - mean (273ms) : 268, 277
master - mean (273ms) : 269, 277
section Bailout
This PR (8420) - mean (273ms) : 271, 276
master - mean (273ms) : 270, 276
section CallTarget+Inlining+NGEN
This PR (8420) - mean (926ms) : 906, 946
master - mean (924ms) : 895, 953
HttpMessageHandler (.NET 6)gantt
title Execution time (ms) HttpMessageHandler (.NET 6)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8420) - mean (268ms) : 262, 275
master - mean (267ms) : 262, 272
section Bailout
This PR (8420) - mean (272ms) : 267, 277
master - mean (266ms) : 264, 268
section CallTarget+Inlining+NGEN
This PR (8420) - mean (1,146ms) : 1115, 1177
master - mean (1,133ms) : 1090, 1176
HttpMessageHandler (.NET 8)gantt
title Execution time (ms) HttpMessageHandler (.NET 8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8420) - mean (281ms) : 274, 289
master - mean (264ms) : 261, 267
section Bailout
This PR (8420) - mean (280ms) : crit, 273, 288
master - mean (265ms) : 261, 268
section CallTarget+Inlining+NGEN
This PR (8420) - mean (1,045ms) : 996, 1094
master - mean (1,019ms) : 980, 1058
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| spanMetaStructs: spanMetaStructs, | ||
| spanEvents: spanEvents); | ||
| spanEvents: spanEvents, | ||
| peerTags: peerTags!, |
Member
There was a problem hiding this comment.
peerTags is nullable so we shouldn't need the ! nullability suppression here.
Member
Author
There was a problem hiding this comment.
Yeah, except the ! is suppressing the inner non-nullability, because the compiler isn't clever enough to know that it's a List<string> not List<string?>, even though we filter string.IsNullOrEmpty (note that StringUtil also doesn't help here)
var peerTags = (jObject["peer_tags"] as JArray)?.Values<string>().Where(x => !string.IsNullOrEmpty(x)).Distinct().OrderBy(x => x).ToList();Removing the dammit gives errors 🙁
| { | ||
| if (!string.IsNullOrEmpty(pattern)) | ||
| { | ||
| compiled.Add(new Regex(pattern, RegexOptions.Compiled, matchTimeout: TimeSpan.FromSeconds(1))); |
Member
There was a problem hiding this comment.
Remember these aren't actually compiled until the first time they are used. Not sure if it matters here, but somewhere else we do a dummy Match() to force the compilation early.
Member
Author
There was a problem hiding this comment.
Yeah, it's a good point, hadn't thought about that. I think this is likely actually the correct behaviour, as it means that compilation is happening at serialization time instead of blocking the discovery service updates, but I could see an argument both ways 🤔 Any preference?
| } | ||
|
|
||
| // Find the root span (ParentId == null or 0) and apply the filter | ||
| foreach (var span in spans) |
Member
There was a problem hiding this comment.
If this span collection follows our usual ordering, the root span will usually be the last span, so it might be faster to search in reverse (from last span to first span).
Member
Author
There was a problem hiding this comment.
Heh, that's actually harder than it seems, because it's a SpanCollection which doesn't expose the root array, but it's a good point. I'll address that when looking into the partial flush issue...
lucaspimentel
approved these changes
Apr 8, 2026
Contributor
There was a problem hiding this comment.
Pull request overview
Updates the tracer’s client-side stats pipeline to better match the “client-side-stats” 1.2.0 wire format/behavior by expanding aggregation dimensions, adjusting serialization semantics, and incorporating agent-driven discovery/filtering signals.
Changes:
- Extend stats aggregation key + msgpack payload to include additional dimensions (span kind, trace-root flag, HTTP method/endpoint, gRPC status, service source, peer tags) and add stochastic rounding/weighted accumulation.
- Add agent discovery support for peer tags, span kinds eligibility, obfuscation negotiation, and trace-level filtering (exact/regex/resource-based).
- Update SQL obfuscation tokenization/normalization and adjust stats sending API surface (obfuscation header, no-retry for agent stats).
Reviewed changes
Copilot reviewed 32 out of 32 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| tracer/src/Datadog.Trace/Agent/StatsAggregator.cs | Computes stats with new dimensions (peer tags, span kinds), filtering, weighting, and obfuscation negotiation. |
| tracer/src/Datadog.Trace/Agent/StatsBuffer.cs | Serializes updated msgpack wire format and suppresses empty buckets. |
| tracer/src/Datadog.Trace/Agent/StatsBucket.cs | Switch counters to double for weighted accumulation; store peer tags. |
| tracer/src/Datadog.Trace/Agent/StatsAggregationKey.cs | Adds new dimensions to key equality/hash. |
| tracer/src/Datadog.Trace/Agent/DiscoveryService/DiscoveryService.cs | Parses /info additions (peer_tags, span_kinds_stats_computed, obfuscation_version, trace filters). |
| tracer/src/Datadog.Trace/Agent/DiscoveryService/AgentConfiguration.cs | Carries newly discovered config fields through the system. |
| tracer/src/Datadog.Trace/Agent/TraceSamplers/TraceFilter.cs | Implements agent-driven trace-level filtering on root spans. |
| tracer/src/Datadog.Trace/Processors/ObfuscatorTraceProcessor.cs | Expands SQL token splitters and normalizes spaces around operators near placeholders. |
| tracer/src/Datadog.Trace/Agent/Api.cs | Sends stats with obfuscation header and disables retries for agent stats. |
| tracer/src/Datadog.Trace/Agent/ApiOtlp.cs | Updates stats API signature/state for OTLP path. |
| tracer/src/Datadog.Trace/Agent/TraceSamplers/RareSampler.cs | Uses stats aggregator to build keys (signature change). |
| tracer/test/** | Updates/adds tests for new dimensions, filtering, obfuscation behavior, and API signature changes. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+416
to
421
| var key = BuildKey(span, out var peerTags); | ||
|
|
||
| var buffer = CurrentBuffer; | ||
|
|
||
| if (!buffer.Buckets.TryGetValue(key, out var bucket)) | ||
| { |
There was a problem hiding this comment.
AddToBuffer() calls BuildKey(span, out var peerTags) for every span before TryGetValue. For client/producer/consumer spans this allocates UTF-8 bytes (and intermediate strings via $"{tagKey}:{tagValue}") even when the bucket already exists and the computed peerTags list is discarded. Consider restructuring so peer-tag materialization only happens when inserting a new bucket (e.g., compute hash first, or defer building utf8PeerTags until after TryGetValue indicates the key is new) to avoid per-span allocation overhead.
Suggested change
| var key = BuildKey(span, out var peerTags); | |
| var buffer = CurrentBuffer; | |
| if (!buffer.Buckets.TryGetValue(key, out var bucket)) | |
| { | |
| var buffer = CurrentBuffer; | |
| var key = BuildKey(span, out _); | |
| if (!buffer.Buckets.TryGetValue(key, out var bucket)) | |
| { | |
| key = BuildKey(span, out var peerTags); |
Member
Author
There was a problem hiding this comment.
Yeah, this one is a pain, I don't have a great answer to it currently 🙁 Will address in a follow up, when I work out how 😅
Member
Author
There was a problem hiding this comment.
To Clarify, the suggested approach is much worse, as it allocates twice 😉
andrewlock
force-pushed
the
andrew/client-side-stats/client-side-stats-1-2
branch
from
April 8, 2026 10:40
e52cc41 to
562d9e7
Compare
andrewlock
force-pushed
the
andrew/client-side-stats/ip-obfuscation
branch
from
April 8, 2026 10:41
f00d1b6 to
77f3510
Compare
andrewlock
force-pushed
the
andrew/client-side-stats/client-side-stats-1-2
branch
from
April 8, 2026 10:57
562d9e7 to
0b3eff8
Compare
andrewlock
force-pushed
the
andrew/client-side-stats/ip-obfuscation
branch
from
April 8, 2026 10:57
77f3510 to
536d246
Compare
Co-authored-by: Lucas Pimentel <[email protected]>
…ritten as "" if it's not there)
andrewlock
force-pushed
the
andrew/client-side-stats/client-side-stats-1-2
branch
from
April 16, 2026 07:59
5573211 to
e9ee48d
Compare
andrewlock
added a commit
that referenced
this pull request
Apr 20, 2026
## Summary of changes Extracts `ContainsSpanId` from `TraceChunkModel` to `SpanCollection` ## Reason for change I need to find a given span in a `SpanCollection` for client-side-stats. The algorithm used by `TraceChunkModel` does it "properly" (by stating from the last span when looking for a root span), so it seems to make sense to move this method to `SpanCollection`. Additionally, this essentially hides some of the internals about the `SpanCollection` type, so seems like a better place for it anyway. ## Implementation details - Move the `IndexOf()` method from `TraceChunkModel` to `SpanCollection` - Switch it to `Contains()` instead, seeing as the index is never actually used AFAICT. It's easy to switch it back later if we _do_ need it. ## Test coverage Added some unit tests for the method ## Other details Part of a stack - #8417 - #8418 - #8420
andrewlock
added a commit
that referenced
this pull request
Apr 21, 2026
## Summary of changes Updates the processing pipeline for client side stats ## Reason for change The bulk of the CSS 1.2.0 work was done in #8420, but the span-processing pipeline wasn't the same as the Go agent: | Step | Go Agent | .NET Tracer | | ---- | --------- | -------------------------------------- | | 1 | Normalize | Normalize + Obfuscate (`ProcessTrace`) | | 2 | Filter | Filter + Sample (`ShouldKeepTrace`) | | 3 | Obfuscate | | | 4 | Sample | | | 5 | Stats | Stats (`AddRange`) | ## Implementation details The difference was mostly due to the way we had separate `ProcessTrace()` and `ShouldKeepTrace()` methods. By merging them, we can encapsulate the pipeline, and ensure that we mirror the way the Go agent processes traces. Additionally, there are two different mechanisms for dropping spans now - p0drops and "trace filtering", so added a new metric tag `reason:trace_filter` for it. Additionally, made some small optimizations: - No need for an array of `ITraceProcessor` when it only contains one - Updated `TraceFilter` algorithm to correctly look for local trace root, starting from last span - Made `_statsAggregator` non-null in `AgentWriter` to eliminate a bunch of null checks ## Test coverage Covered by existing tests. ## Other details Part of a stack: - #8417 - #8418 - #8420 - #8435
andrewlock
added a commit
that referenced
this pull request
Apr 21, 2026
## Summary of changes Improves the performance of peer-tag hash calculation for "fast path" cases (where there's an existing bucket) ## Reason for change We have to calculate the hash of all peer tags as part of client-side-stats bucketing calculations, which involves encoding them as utf-8. Additionally, when we _send_ the buckets, we have to send the tags as utf-8. In the initial CSS 1.2.0 implementation, we encoded the tags every-time we ran a calculation, which would allocate for every span that had peer tags, generally quite expensive. In this PR, we switch to doing the encoding twice: once with a zero-allocation implementation (amortized 0 on .NET Framework) to calculate the hash, and then, _if_ we need the "real" encoded tags, then we do that encoding again. ## Implementation details - Split the peer tags work in two, once to calculate the peer tags hash, once to get the actual tags as `key:value` - In the hash-calculation stage, we can use `stackalloc` for .NET Core, and array pool implementation for .NET Framework etc - As additional optimizations, we also - Pre-encode the peer tag _keys_ to utf-8, so that we only need to do those once for the hash calculation. - Pass key details from `BuildKey` to `GetEncodedPeerTags` (i.e. is this a "base service" only tag, if so, what's the tag value, otherwise how big does the peer tag list need to be) - If client-side stats is disabled, don't bother doing all the pre-calculation (or for trace filters) There are some possible future optimizations, _not_ implemented in this PR: - Track which tags require IP quantization to avoid re-doing it. (Maybe we should allow-list the quantization anyway, so that it only applies to specific tags? Or alternatively, block list?) - Instead of doing the encoding of `tag:value`, allow writing the pre-computed `byte[]` to `MessagePackBinary` and appending the `value`. This is doable, but requires updating the `MessagePackBinary` implementation to support it, so I considered it out of scope for now ## Test coverage Mostly covered by existing tests, but added some additional unit tests that directly compare the hashing to values used in Go agent tests. Additionally did some benchmarking. The key thing is that the `ClientSpanWithPeerTags` path is zero-allocation (and it's nice that the slow-path is still lower allocation than before, even if it's slower over all) | Method | Runtime | Mean | Error | Allocated | | -------------------------------------------------- | ------------------ | ----------: | ---------: | --------: | | BuildKey_SimpleSpan_Before | .NET 10.0 | 44.34 ns | 0.906 ns | - | | BuildKey_SimpleSpan_After | .NET 10.0 | 39.17 ns | 0.445 ns | - | | BuildKey_SimpleSpan_Before | .NET 6.0 | 93.13 ns | 1.837 ns | - | | BuildKey_SimpleSpan_After | .NET 6.0 | 87.56 ns | 0.839 ns | - | | BuildKey_SimpleSpan_Before | .NET Core 3.1 | 192.53 ns | 2.572 ns | - | | BuildKey_SimpleSpan_After | .NET Core 3.1 | 217.32 ns | 4.362 ns | - | | BuildKey_SimpleSpan_Before | .NET Framework 4.8 | 131.98 ns | 2.621 ns | - | | BuildKey_SimpleSpan_After | .NET Framework 4.8 | 175.20 ns | 7.647 ns | - | | | | | | | | BuildKey_ClientSpanNoPeerTags_Before | .NET 10.0 | 160.62 ns | 2.622 ns | - | | BuildKey_ClientSpanNoPeerTags_After | .NET 10.0 | 151.27 ns | 1.995 ns | - | | BuildKey_ClientSpanNoPeerTags_Before | .NET 6.0 | 243.62 ns | 3.199 ns | - | | BuildKey_ClientSpanNoPeerTags_After | .NET 6.0 | 232.55 ns | 2.975 ns | - | | BuildKey_ClientSpanNoPeerTags_Before | .NET Core 3.1 | 550.47 ns | 9.998 ns | - | | BuildKey_ClientSpanNoPeerTags_After | .NET Core 3.1 | 680.42 ns | 10.022 ns | - | | BuildKey_ClientSpanNoPeerTags_Before | .NET Framework 4.8 | 442.88 ns | 8.854 ns | - | | BuildKey_ClientSpanNoPeerTags_After | .NET Framework 4.8 | 682.12 ns | 13.468 ns | - | | | | | | | | BuildKey_ClientSpanWithPeerTags_Before | .NET 10.0 | 800.82 ns | 15.062 ns | 840 B | | BuildKey_ClientSpanWithPeerTags_After | .NET 10.0 | 575.04 ns | 5.604 ns | - | | BuildKey_ClientSpanWithPeerTags_Before | .NET 6.0 | 1,076.07 ns | 13.124 ns | 840 B | | BuildKey_ClientSpanWithPeerTags_After | .NET 6.0 | 1,440.26 ns | 28.755 ns | - | | BuildKey_ClientSpanWithPeerTags_Before | .NET Core 3.1 | 1,531.27 ns | 13.509 ns | 840 B | | BuildKey_ClientSpanWithPeerTags_After | .NET Core 3.1 | 1,774.35 ns | 89.601 ns | - | | BuildKey_ClientSpanWithPeerTags_Before | .NET Framework 4.8 | 1,682.15 ns | 17.082 ns | 859 B | | BuildKey_ClientSpanWithPeerTags_After | .NET Framework 4.8 | 2,040.90 ns | 162.895 ns | - | | | | | | | | BuildKey_ClientSpanWithPeerTags_GetEncodedPeerTags | .NET 10.0 | 1,141.85 ns | 9.980 ns | 760 B | | BuildKey_ClientSpanWithPeerTags_GetEncodedPeerTags | .NET 6.0 | 2,720.55 ns | 53.097 ns | 760 B | | BuildKey_ClientSpanWithPeerTags_GetEncodedPeerTags | .NET Core 3.1 | 3,534.28 ns | 141.234 ns | 760 B | | BuildKey_ClientSpanWithPeerTags_GetEncodedPeerTags | .NET Framework 4.8 | 3,438.80 ns | 84.352 ns | 778 B | <details><summary>Details</summary> <p> ```csharp // <copyright file="StatsAggregatorBenchmark.cs" company="Datadog"> // Unless explicitly stated otherwise all files in this repository are licensed under the Apache 2 License. // This product includes software developed at Datadog (https://www.datadoghq.com/). Copyright 2017 Datadog, Inc. // </copyright> using System; using System.Collections.Generic; using System.Threading.Tasks; using BenchmarkDotNet.Attributes; using Datadog.Trace; using Datadog.Trace.Agent; using Datadog.Trace.Agent.DiscoveryService; using Datadog.Trace.Configuration; using Datadog.Trace.Tagging; namespace Benchmarks.Trace; /// <summary> /// StatsAggregator.BuildKey benchmarks /// </summary> [MemoryDiagnoser] [BenchmarkCategory(Constants.TracerCategory)] public class StatsAggregatorBenchmark { private static readonly List<StatsAggregator.PeerTagKey> PeerTagKeys = [ new("_dd.base_service"), new("aws.queue.name"), new("aws.queue.url"), new("aws.s3.bucket"), new("aws.stream.name"), new("bucketname"), new("db.couchbase.seed.nodes"), new("db.hostname"), new("db.instance"), new("db.system"), new("messaging.destination"), new("messaging.kafka.bootstrap.servers"), new("messaging.rabbitmq.exchange"), new("messaging.system"), new("network.destination.name"), new("peer.hostname"), new("peer.service"), new("server.address"), new("topic"), ]; private StatsAggregator _aggregator; private Span _simpleSpan; private Span _clientSpanNoPeerTags; private Span _clientSpanWithPeerTags; private StatsAggregationKey _key; private List<byte[]> _encoded; [GlobalSetup] public void GlobalSetup() { _aggregator = new StatsAggregator( new NoOpApi(), new TracerSettings(), NullDiscoveryService.Instance, isOtlp: false); var now = DateTimeOffset.UtcNow; // Simple span: no span kind, no peer tags — exercises the "internal" fast path _simpleSpan = CreateSpan(now, "web-service", "web.request", "GET /api/users", "web"); // Client span with SpanKind but no matching peer tags — iterates peer tag keys but finds nothing var clientTags = new HttpTags { HttpMethod = "GET", HttpStatusCode = "200" }; _clientSpanNoPeerTags = CreateSpan(now, "web-service", "http.request", "GET /api/orders", "http", clientTags); // Client span with several peer tags set — the most expensive path var peerTags = new HttpTags { HttpMethod = "POST", HttpStatusCode = "200" }; _clientSpanWithPeerTags = CreateSpan(now, "web-service", "http.request", "POST /api/data", "http", peerTags); _clientSpanWithPeerTags.Tags.SetTag("peer.service", "remote-service"); _clientSpanWithPeerTags.Tags.SetTag("db.instance", "i-1234"); _clientSpanWithPeerTags.Tags.SetTag("db.system", "postgres"); _clientSpanWithPeerTags.Tags.SetTag("server.address", "db.example.com"); _clientSpanWithPeerTags.Tags.SetTag("network.destination.name", "db.example.com"); } [GlobalCleanup] public void GlobalCleanup() { _aggregator.DisposeAsync().GetAwaiter().GetResult(); var value = _key; var encoded = _encoded; } /// <summary> /// BuildKey for a simple span with no span kind (internal fast-path, no peer tag iteration). /// </summary> [Benchmark] public void BuildKey_SimpleSpan() { _key = _aggregator.BuildKey(_simpleSpan, PeerTagKeys, out _); } /// <summary> /// BuildKey for a client span that has no matching peer tags (iterates all peer tag keys, finds none). /// </summary> [Benchmark] public void BuildKey_ClientSpanNoPeerTags() { _key = _aggregator.BuildKey(_clientSpanNoPeerTags, PeerTagKeys, out _); } /// <summary> /// BuildKey for a client span with several peer tags set (UTF-8 encoding + FNV hashing). /// </summary> [Benchmark] public void BuildKey_ClientSpanWithPeerTags() { _key = _aggregator.BuildKey(_clientSpanWithPeerTags, PeerTagKeys, out var results); } /// <summary> /// BuildKey for a client span with several peer tags set (UTF-8 encoding + FNV hashing). /// </summary> [Benchmark] public void BuildKey_ClientSpanWithPeerTags_GetEncodedPeerTags() { _key = _aggregator.BuildKey(_clientSpanWithPeerTags, PeerTagKeys, out var results); _encoded = StatsAggregator.GetEncodedPeerTags(_clientSpanWithPeerTags, PeerTagKeys, in results); } private static Span CreateSpan(DateTimeOffset start, string serviceName, string operationName, string resourceName, string type, ITags tags = null) { var tracer = Benchmarks.Trace.Asm.EmptyDatadogTracer.Instance; var traceContext = new TraceContext(tracer); var context = new SpanContext(null, traceContext, serviceName); var span = new Span(context, start, tags); span.OperationName = operationName; span.ResourceName = resourceName; span.Type = type; return span; } private sealed class NoOpApi : IApi { public TracesEncoding TracesEncoding => TracesEncoding.DatadogV0_4; public Task<bool> Ping() => Task.FromResult(true); public Task<bool> SendTracesAsync(ArraySegment<byte> traces, int numberOfTraces, bool statsComputationEnabled, long numberOfDroppedP0Traces, long numberOfDroppedP0Spans, bool apmTracingEnabled = true) => Task.FromResult(true); public Task<bool> SendStatsAsync(StatsBuffer stats, long bucketDuration, int tracerObfuscationVersion) => Task.FromResult(true); } } ``` </p> </details> ## Other details Part of a stack: - #8417 - #8418 - #8420 - #8435 - #8436 - #8444
andrewlock
added a commit
that referenced
this pull request
Apr 23, 2026
## Summary of changes Removes the minimum-agent version check from the client-side-stats check ## Reason for change We didn't have it before, which means people who previously had stats, suddenly won't have them. Also other agents (like bottlecap etc) don't have the version. And we haven't figured out how to handle that yet. While CSS is off by default, this seems like the best option to take. ## Implementation details Remove the code that is checking for CSS support. ## Test coverage Covered by existing (and removed a test for the behaviour) ## Other details Follows on from - #8417 - #8418 - #8420 - #8435 - #8436 - #8444 - #8445
andrewlock
added a commit
that referenced
this pull request
Jun 25, 2026
## Summary of changes Removes the incorrect sampling-based weighting from Client-Side stats ## Reason for change When implementing the updated CSS implementation in #8420, we compared with the canonicial Go implementation, and [added a `GetWeight()`](https://github.com/DataDog/datadog-agent/blob/ce22e11ee71e55be717b9d9a3f8f3d7721a9c6d7/pkg/trace/stats/weight.go) method based on this. However, the sampling rate that this is based on is based on the deprecated `_sample_rate` tag (which .NET never implemented), _not_ the normal sampling rate. Therefore this weighting causes overcounting. ## Implementation details - Rip out GetWeight() and associated usages - Revert various doubles back to `long`s - Remove the unnecessary stochastic rounding ## Test coverage - Removed one test for `GetWeight()` behavior - TBC a system test that would catch this - serverless e2e tests (thanks @apiarian-datadog!) caught this one
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary of changes
Updates the existing client-side-stats implementation to match version 1.2.0 as defined in the RFC and as implemented in the agent.
Reason for change
Our implementation is severely lagging the latest implementation in the agent. This hasn't been a big deal, as it's not documented and not enabled by default, but we'd like to fix the implementation to make it usable.
Implementation details
This was driven almost entirely by 🤖, by comparing our existing implementation to the RFC, and also taking the agent/go implementation as the canonical implementation.
At a high level, the PR contains the following changes:
SpanKind,IsTraceRoot(as Trilean),HTTPMethod,HTTPEndpoint,GRPCStatusCode,ServiceSource,PeerTagsto both the aggregation key and the msgpack wire formatGRPCStatusCodetype: Changed frominttostringto match Go agent's wire format (agent was returning 400 Bad Request in system tests)rpc.grpc.status_code,grpc.code,rpc.grpc.status.code,grpc.status.code)Hits,Errors,Duration,TopLevelHitsaccumulated asdouble(weighted by sampling rate) then rounded probabilistically toint64for serialization1/rate), matching Go agent behavior"unknown-env"when environment is not configuredgit_commit_sha: Added as optional field in stats payloadService: Added as top-level field in stats payloadHasHits()check prevents sending payloads with zero-hit buckets (stale keys retained for sketch reuse)Starttimestamps aligned to 10-second boundaries (ts - ts % 10_000_000_000) matching Go tracer'salignTsStartTimeproperty (onlyStartas aligned nanoseconds)/infoEndpoint)peer_tags: Parsed, sorted, deduplicated; used for peer tag extraction on client/producer/consumer spansspan_kinds_stats_computed: Parsed to override eligible span kindsobfuscation_version: Parsed for obfuscation negotiationfilter_tags,filter_tags_regex,ignore_resourcesfrom/infoTraceFilterimplementation: Evaluates agent-configured filters (exact tags, regex tags, resource patterns) against root spans before stats computation* / = < > ! & ^ % ~ ? @ : #as token splitters (matching Go agent'sgo-sqllexerisOperator()) so queries likeWHERE id='1'are properly obfuscated=,<,>,!) adjacent to?placeholders, matching Go agent's normalizer output (e.g.,id='1'→id = ?)obfuscation_versionis negotiated with agent; sendsDatadog-Obfuscation-VersionheaderIpAddressObfuscationUtil.QuantizePeerIpAddresses()replacing non-allowed IPs with"blocked-ip-address"_dd.base_service:{serviceName}as sole peer tagStartsWith("synthetics")prefix matching (not exact match)obfuscation_versionin/inforesponseTest coverage
There's a lot going on in this PR, because we were so far behind. I could split this into implementing individual features, but there would be a lot of duplication between PRs, and it didn't seem like it would be that easy to track. At least with this big bang we can compare directly against the system tests etc.
The existing system tests for stats computation were checked, and made to pass (which identified a number of hidden expectations which will be added to the RFC). I'll create a PR to enable these in the system-tests repo
Other details
Part of a stack
charhelpers likeIsAsciiHexDigit#8417IpAddressObfuscationUtilfor use with client-side-stats #8418There are still some theoretical gaps between the go implementation and the .NET implementation, but I think these are non-issues in most cases:
"CANCELLED"→"1"); .NET passes raw tag valuehttp.status_codeandhttp.response.status_code(OTel convention); .NET only checkshttp.status_code0in .NETHTTP_method/HTTP_endpoint_top_levelmetric_top_leveland_dd.top_level; .NET only checks_dd.top_levelspan_derived_primary_tagsThere is another big elephant in the room, which is perf. The peer tags, in particular, currently requires a bunch of allocation. I'd rather defer trying to fight against that to another PR if possible, unless anyone has some clear ideas 😄
Another aspect I'm not sure about is how this interacts with @zacharycmontoya's recent work to publish OTLP stats. I took a random guess and fought the refactoring, but need to verify it.