Fix processing pipeline for client-side-stats#8436
Merged
andrewlock merged 13 commits intoApr 21, 2026
Conversation
andrewlock
added
area:tracer
andrewlock
force-pushed
the
andrew/client-side-stats/update-processing-pipeline
branch
from
April 9, 2026 22:41
0b9d17e to
833ad62
Compare
andrewlock
force-pushed
the
andrew/client-side-stats/refactor-trace-chunk-model
branch
from
April 9, 2026 22:41
d367d99 to
efd496f
Compare
Execution-Time Benchmarks Report ⏱️Execution-time results for samples comparing This PR (8436) and master. ✅ No regressions detected - check the details below Full Metrics ComparisonFakeDbCommand
HttpMessageHandler
Comparison explanationExecution-time benchmarks measure the whole time it takes to execute a program, and are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are highlighted in **red**. The following thresholds were used for comparing the execution times:
Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard. Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph). Duration chartsFakeDbCommand (.NET Framework 4.8)gantt
title Execution time (ms) FakeDbCommand (.NET Framework 4.8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8436) - mean (74ms) : 70, 78
master - mean (73ms) : 70, 76
section Bailout
This PR (8436) - mean (77ms) : 74, 79
master - mean (77ms) : 74, 80
section CallTarget+Inlining+NGEN
This PR (8436) - mean (1,084ms) : 1020, 1147
master - mean (1,082ms) : 1041, 1122
FakeDbCommand (.NET Core 3.1)gantt
title Execution time (ms) FakeDbCommand (.NET Core 3.1)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8436) - mean (114ms) : 110, 119
master - mean (114ms) : 108, 120
section Bailout
This PR (8436) - mean (115ms) : 112, 117
master - mean (116ms) : 111, 120
section CallTarget+Inlining+NGEN
This PR (8436) - mean (799ms) : 769, 829
master - mean (797ms) : 776, 817
FakeDbCommand (.NET 6)gantt
title Execution time (ms) FakeDbCommand (.NET 6)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8436) - mean (103ms) : 97, 109
master - mean (102ms) : 97, 107
section Bailout
This PR (8436) - mean (104ms) : 99, 108
master - mean (105ms) : 100, 111
section CallTarget+Inlining+NGEN
This PR (8436) - mean (946ms) : 910, 982
master - mean (944ms) : 905, 983
FakeDbCommand (.NET 8)gantt
title Execution time (ms) FakeDbCommand (.NET 8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8436) - mean (101ms) : 98, 104
master - mean (101ms) : 97, 105
section Bailout
This PR (8436) - mean (105ms) : 99, 111
master - mean (101ms) : 99, 104
section CallTarget+Inlining+NGEN
This PR (8436) - mean (826ms) : 780, 872
master - mean (833ms) : 790, 875
HttpMessageHandler (.NET Framework 4.8)gantt
title Execution time (ms) HttpMessageHandler (.NET Framework 4.8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8436) - mean (191ms) : 188, 194
master - mean (192ms) : 187, 196
section Bailout
This PR (8436) - mean (195ms) : 193, 197
master - mean (194ms) : 193, 196
section CallTarget+Inlining+NGEN
This PR (8436) - mean (1,152ms) : 1106, 1199
master - mean (1,151ms) : 1109, 1192
HttpMessageHandler (.NET Core 3.1)gantt
title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8436) - mean (274ms) : 269, 278
master - mean (274ms) : 268, 281
section Bailout
This PR (8436) - mean (274ms) : 272, 277
master - mean (274ms) : 270, 278
section CallTarget+Inlining+NGEN
This PR (8436) - mean (927ms) : 906, 947
master - mean (927ms) : 907, 947
HttpMessageHandler (.NET 6)gantt
title Execution time (ms) HttpMessageHandler (.NET 6)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8436) - mean (268ms) : 265, 272
master - mean (268ms) : 262, 274
section Bailout
This PR (8436) - mean (269ms) : 266, 272
master - mean (268ms) : 265, 271
section CallTarget+Inlining+NGEN
This PR (8436) - mean (1,140ms) : 1101, 1179
master - mean (1,133ms) : 1080, 1187
HttpMessageHandler (.NET 8)gantt
title Execution time (ms) HttpMessageHandler (.NET 8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8436) - mean (266ms) : 262, 270
master - mean (266ms) : 263, 269
section Bailout
This PR (8436) - mean (266ms) : 263, 269
master - mean (265ms) : 263, 268
section CallTarget+Inlining+NGEN
This PR (8436) - mean (1,020ms) : 975, 1065
master - mean (1,019ms) : 985, 1052
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
andrewlock
force-pushed
the
andrew/client-side-stats/update-processing-pipeline
branch
2 times, most recently
from
April 10, 2026 07:20
a99ae1b to
bf1d86f
Compare
BenchmarksBenchmark execution time: 2026-04-20 22:14:51 Comparing candidate commit 5ed9a0b in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 27 metrics, 0 unstable metrics, 87 known flaky benchmarks.
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3179a17d78
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
andrewlock
marked this pull request as draft
April 10, 2026 09:20
andrewlock
force-pushed
the
andrew/client-side-stats/update-processing-pipeline
branch
from
April 10, 2026 12:29
3179a17 to
2a9f4b3
Compare
andrewlock
force-pushed
the
andrew/client-side-stats/refactor-trace-chunk-model
branch
from
April 10, 2026 12:29
efd496f to
bda5853
Compare
andrewlock
marked this pull request as ready for review
April 10, 2026 12:30
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2a9f4b3577
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
This was referenced Apr 13, 2026
lucaspimentel
approved these changes
Apr 13, 2026
andrewlock
force-pushed
the
andrew/client-side-stats/update-processing-pipeline
branch
from
April 14, 2026 10:08
2a9f4b3 to
2106339
Compare
andrewlock
force-pushed
the
andrew/client-side-stats/refactor-trace-chunk-model
branch
from
April 14, 2026 10:08
bda5853 to
682df90
Compare
zacharycmontoya
approved these changes
Apr 15, 2026
zacharycmontoya
left a comment
zacharycmontoya
left a comment
Contributor
There was a problem hiding this comment.
LGTM with some small comments!
andrewlock
force-pushed
the
andrew/client-side-stats/update-processing-pipeline
branch
from
April 16, 2026 07:59
2106339 to
1783517
Compare
andrewlock
force-pushed
the
andrew/client-side-stats/refactor-trace-chunk-model
branch
2 times, most recently
from
April 20, 2026 16:12
bcf6c7a to
ce99c4b
Compare
Base automatically changed from
andrew/client-side-stats/refactor-trace-chunk-model
to
master
April 20, 2026 20:14
andrewlock
force-pushed
the
andrew/client-side-stats/update-processing-pipeline
branch
from
April 20, 2026 21:05
1783517 to
5ed9a0b
Compare
andrewlock
deleted the
andrew/client-side-stats/update-processing-pipeline
branch
April 21, 2026 06:01
andrewlock
added a commit
that referenced
this pull request
Apr 21, 2026
## Summary of changes Improves the performance of peer-tag hash calculation for "fast path" cases (where there's an existing bucket) ## Reason for change We have to calculate the hash of all peer tags as part of client-side-stats bucketing calculations, which involves encoding them as utf-8. Additionally, when we _send_ the buckets, we have to send the tags as utf-8. In the initial CSS 1.2.0 implementation, we encoded the tags every-time we ran a calculation, which would allocate for every span that had peer tags, generally quite expensive. In this PR, we switch to doing the encoding twice: once with a zero-allocation implementation (amortized 0 on .NET Framework) to calculate the hash, and then, _if_ we need the "real" encoded tags, then we do that encoding again. ## Implementation details - Split the peer tags work in two, once to calculate the peer tags hash, once to get the actual tags as `key:value` - In the hash-calculation stage, we can use `stackalloc` for .NET Core, and array pool implementation for .NET Framework etc - As additional optimizations, we also - Pre-encode the peer tag _keys_ to utf-8, so that we only need to do those once for the hash calculation. - Pass key details from `BuildKey` to `GetEncodedPeerTags` (i.e. is this a "base service" only tag, if so, what's the tag value, otherwise how big does the peer tag list need to be) - If client-side stats is disabled, don't bother doing all the pre-calculation (or for trace filters) There are some possible future optimizations, _not_ implemented in this PR: - Track which tags require IP quantization to avoid re-doing it. (Maybe we should allow-list the quantization anyway, so that it only applies to specific tags? Or alternatively, block list?) - Instead of doing the encoding of `tag:value`, allow writing the pre-computed `byte[]` to `MessagePackBinary` and appending the `value`. This is doable, but requires updating the `MessagePackBinary` implementation to support it, so I considered it out of scope for now ## Test coverage Mostly covered by existing tests, but added some additional unit tests that directly compare the hashing to values used in Go agent tests. Additionally did some benchmarking. The key thing is that the `ClientSpanWithPeerTags` path is zero-allocation (and it's nice that the slow-path is still lower allocation than before, even if it's slower over all) | Method | Runtime | Mean | Error | Allocated | | -------------------------------------------------- | ------------------ | ----------: | ---------: | --------: | | BuildKey_SimpleSpan_Before | .NET 10.0 | 44.34 ns | 0.906 ns | - | | BuildKey_SimpleSpan_After | .NET 10.0 | 39.17 ns | 0.445 ns | - | | BuildKey_SimpleSpan_Before | .NET 6.0 | 93.13 ns | 1.837 ns | - | | BuildKey_SimpleSpan_After | .NET 6.0 | 87.56 ns | 0.839 ns | - | | BuildKey_SimpleSpan_Before | .NET Core 3.1 | 192.53 ns | 2.572 ns | - | | BuildKey_SimpleSpan_After | .NET Core 3.1 | 217.32 ns | 4.362 ns | - | | BuildKey_SimpleSpan_Before | .NET Framework 4.8 | 131.98 ns | 2.621 ns | - | | BuildKey_SimpleSpan_After | .NET Framework 4.8 | 175.20 ns | 7.647 ns | - | | | | | | | | BuildKey_ClientSpanNoPeerTags_Before | .NET 10.0 | 160.62 ns | 2.622 ns | - | | BuildKey_ClientSpanNoPeerTags_After | .NET 10.0 | 151.27 ns | 1.995 ns | - | | BuildKey_ClientSpanNoPeerTags_Before | .NET 6.0 | 243.62 ns | 3.199 ns | - | | BuildKey_ClientSpanNoPeerTags_After | .NET 6.0 | 232.55 ns | 2.975 ns | - | | BuildKey_ClientSpanNoPeerTags_Before | .NET Core 3.1 | 550.47 ns | 9.998 ns | - | | BuildKey_ClientSpanNoPeerTags_After | .NET Core 3.1 | 680.42 ns | 10.022 ns | - | | BuildKey_ClientSpanNoPeerTags_Before | .NET Framework 4.8 | 442.88 ns | 8.854 ns | - | | BuildKey_ClientSpanNoPeerTags_After | .NET Framework 4.8 | 682.12 ns | 13.468 ns | - | | | | | | | | BuildKey_ClientSpanWithPeerTags_Before | .NET 10.0 | 800.82 ns | 15.062 ns | 840 B | | BuildKey_ClientSpanWithPeerTags_After | .NET 10.0 | 575.04 ns | 5.604 ns | - | | BuildKey_ClientSpanWithPeerTags_Before | .NET 6.0 | 1,076.07 ns | 13.124 ns | 840 B | | BuildKey_ClientSpanWithPeerTags_After | .NET 6.0 | 1,440.26 ns | 28.755 ns | - | | BuildKey_ClientSpanWithPeerTags_Before | .NET Core 3.1 | 1,531.27 ns | 13.509 ns | 840 B | | BuildKey_ClientSpanWithPeerTags_After | .NET Core 3.1 | 1,774.35 ns | 89.601 ns | - | | BuildKey_ClientSpanWithPeerTags_Before | .NET Framework 4.8 | 1,682.15 ns | 17.082 ns | 859 B | | BuildKey_ClientSpanWithPeerTags_After | .NET Framework 4.8 | 2,040.90 ns | 162.895 ns | - | | | | | | | | BuildKey_ClientSpanWithPeerTags_GetEncodedPeerTags | .NET 10.0 | 1,141.85 ns | 9.980 ns | 760 B | | BuildKey_ClientSpanWithPeerTags_GetEncodedPeerTags | .NET 6.0 | 2,720.55 ns | 53.097 ns | 760 B | | BuildKey_ClientSpanWithPeerTags_GetEncodedPeerTags | .NET Core 3.1 | 3,534.28 ns | 141.234 ns | 760 B | | BuildKey_ClientSpanWithPeerTags_GetEncodedPeerTags | .NET Framework 4.8 | 3,438.80 ns | 84.352 ns | 778 B | <details><summary>Details</summary> <p> ```csharp // <copyright file="StatsAggregatorBenchmark.cs" company="Datadog"> // Unless explicitly stated otherwise all files in this repository are licensed under the Apache 2 License. // This product includes software developed at Datadog (https://www.datadoghq.com/). Copyright 2017 Datadog, Inc. // </copyright> using System; using System.Collections.Generic; using System.Threading.Tasks; using BenchmarkDotNet.Attributes; using Datadog.Trace; using Datadog.Trace.Agent; using Datadog.Trace.Agent.DiscoveryService; using Datadog.Trace.Configuration; using Datadog.Trace.Tagging; namespace Benchmarks.Trace; /// <summary> /// StatsAggregator.BuildKey benchmarks /// </summary> [MemoryDiagnoser] [BenchmarkCategory(Constants.TracerCategory)] public class StatsAggregatorBenchmark { private static readonly List<StatsAggregator.PeerTagKey> PeerTagKeys = [ new("_dd.base_service"), new("aws.queue.name"), new("aws.queue.url"), new("aws.s3.bucket"), new("aws.stream.name"), new("bucketname"), new("db.couchbase.seed.nodes"), new("db.hostname"), new("db.instance"), new("db.system"), new("messaging.destination"), new("messaging.kafka.bootstrap.servers"), new("messaging.rabbitmq.exchange"), new("messaging.system"), new("network.destination.name"), new("peer.hostname"), new("peer.service"), new("server.address"), new("topic"), ]; private StatsAggregator _aggregator; private Span _simpleSpan; private Span _clientSpanNoPeerTags; private Span _clientSpanWithPeerTags; private StatsAggregationKey _key; private List<byte[]> _encoded; [GlobalSetup] public void GlobalSetup() { _aggregator = new StatsAggregator( new NoOpApi(), new TracerSettings(), NullDiscoveryService.Instance, isOtlp: false); var now = DateTimeOffset.UtcNow; // Simple span: no span kind, no peer tags — exercises the "internal" fast path _simpleSpan = CreateSpan(now, "web-service", "web.request", "GET /api/users", "web"); // Client span with SpanKind but no matching peer tags — iterates peer tag keys but finds nothing var clientTags = new HttpTags { HttpMethod = "GET", HttpStatusCode = "200" }; _clientSpanNoPeerTags = CreateSpan(now, "web-service", "http.request", "GET /api/orders", "http", clientTags); // Client span with several peer tags set — the most expensive path var peerTags = new HttpTags { HttpMethod = "POST", HttpStatusCode = "200" }; _clientSpanWithPeerTags = CreateSpan(now, "web-service", "http.request", "POST /api/data", "http", peerTags); _clientSpanWithPeerTags.Tags.SetTag("peer.service", "remote-service"); _clientSpanWithPeerTags.Tags.SetTag("db.instance", "i-1234"); _clientSpanWithPeerTags.Tags.SetTag("db.system", "postgres"); _clientSpanWithPeerTags.Tags.SetTag("server.address", "db.example.com"); _clientSpanWithPeerTags.Tags.SetTag("network.destination.name", "db.example.com"); } [GlobalCleanup] public void GlobalCleanup() { _aggregator.DisposeAsync().GetAwaiter().GetResult(); var value = _key; var encoded = _encoded; } /// <summary> /// BuildKey for a simple span with no span kind (internal fast-path, no peer tag iteration). /// </summary> [Benchmark] public void BuildKey_SimpleSpan() { _key = _aggregator.BuildKey(_simpleSpan, PeerTagKeys, out _); } /// <summary> /// BuildKey for a client span that has no matching peer tags (iterates all peer tag keys, finds none). /// </summary> [Benchmark] public void BuildKey_ClientSpanNoPeerTags() { _key = _aggregator.BuildKey(_clientSpanNoPeerTags, PeerTagKeys, out _); } /// <summary> /// BuildKey for a client span with several peer tags set (UTF-8 encoding + FNV hashing). /// </summary> [Benchmark] public void BuildKey_ClientSpanWithPeerTags() { _key = _aggregator.BuildKey(_clientSpanWithPeerTags, PeerTagKeys, out var results); } /// <summary> /// BuildKey for a client span with several peer tags set (UTF-8 encoding + FNV hashing). /// </summary> [Benchmark] public void BuildKey_ClientSpanWithPeerTags_GetEncodedPeerTags() { _key = _aggregator.BuildKey(_clientSpanWithPeerTags, PeerTagKeys, out var results); _encoded = StatsAggregator.GetEncodedPeerTags(_clientSpanWithPeerTags, PeerTagKeys, in results); } private static Span CreateSpan(DateTimeOffset start, string serviceName, string operationName, string resourceName, string type, ITags tags = null) { var tracer = Benchmarks.Trace.Asm.EmptyDatadogTracer.Instance; var traceContext = new TraceContext(tracer); var context = new SpanContext(null, traceContext, serviceName); var span = new Span(context, start, tags); span.OperationName = operationName; span.ResourceName = resourceName; span.Type = type; return span; } private sealed class NoOpApi : IApi { public TracesEncoding TracesEncoding => TracesEncoding.DatadogV0_4; public Task<bool> Ping() => Task.FromResult(true); public Task<bool> SendTracesAsync(ArraySegment<byte> traces, int numberOfTraces, bool statsComputationEnabled, long numberOfDroppedP0Traces, long numberOfDroppedP0Spans, bool apmTracingEnabled = true) => Task.FromResult(true); public Task<bool> SendStatsAsync(StatsBuffer stats, long bucketDuration, int tracerObfuscationVersion) => Task.FromResult(true); } } ``` </p> </details> ## Other details Part of a stack: - #8417 - #8418 - #8420 - #8435 - #8436 - #8444
andrewlock
added a commit
that referenced
this pull request
Apr 23, 2026
## Summary of changes Removes the minimum-agent version check from the client-side-stats check ## Reason for change We didn't have it before, which means people who previously had stats, suddenly won't have them. Also other agents (like bottlecap etc) don't have the version. And we haven't figured out how to handle that yet. While CSS is off by default, this seems like the best option to take. ## Implementation details Remove the code that is checking for CSS support. ## Test coverage Covered by existing (and removed a test for the behaviour) ## Other details Follows on from - #8417 - #8418 - #8420 - #8435 - #8436 - #8444 - #8445
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary of changes
Updates the processing pipeline for client side stats
Reason for change
The bulk of the CSS 1.2.0 work was done in #8420, but the span-processing pipeline wasn't the same as the Go agent:
ProcessTrace)ShouldKeepTrace)AddRange)Implementation details
The difference was mostly due to the way we had separate
ProcessTrace()andShouldKeepTrace()methods. By merging them, we can encapsulate the pipeline, and ensure that we mirror the way the Go agent processes traces.Additionally, there are two different mechanisms for dropping spans now - p0drops and "trace filtering", so added a new metric tag
reason:trace_filterfor it.Additionally, made some small optimizations:
ITraceProcessorwhen it only contains oneTraceFilteralgorithm to correctly look for local trace root, starting from last span_statsAggregatornon-null inAgentWriterto eliminate a bunch of null checksTest coverage
Covered by existing tests.
Other details
Part of a stack:
charhelpers likeIsAsciiHexDigit#8417IpAddressObfuscationUtilfor use with client-side-stats #8418ContainsSpanIdtoSpanCollection#8435