Allow passing an offset and count to FnvHash64.GenerateHash()#8444
Closed
andrewlock wants to merge 1 commit into
Closed
Allow passing an offset and count to FnvHash64.GenerateHash()#8444andrewlock wants to merge 1 commit into
FnvHash64.GenerateHash()#8444andrewlock wants to merge 1 commit into
Conversation
andrewlock
added
the
area:tracer
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e94c3c9c6e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+145
to
+146
| var max = offset + count; | ||
| for (var i = offset; i < max; i++) |
There was a problem hiding this comment.
The new GenerateHash(byte[], int offset, int count, ...) path never validates range arguments before iterating, and max = offset + count can lead to silent wrong results for invalid inputs (for example, a negative count makes the loop a no-op and returns initialHash instead of failing). This regression only occurs when callers provide out-of-range values, but in that case it masks slicing bugs and can produce incorrect hashes without any exception; add explicit checks for offset >= 0, count >= 0, and offset + count <= data.Length before hashing.
Useful? React with 👍 / 👎.
Execution-Time Benchmarks Report ⏱️Execution-time results for samples comparing This PR (8444) and master. ✅ No regressions detected - check the details below Full Metrics ComparisonFakeDbCommand
HttpMessageHandler
Comparison explanationExecution-time benchmarks measure the whole time it takes to execute a program, and are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are highlighted in **red**. The following thresholds were used for comparing the execution times:
Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard. Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph). Duration chartsFakeDbCommand (.NET Framework 4.8)gantt
title Execution time (ms) FakeDbCommand (.NET Framework 4.8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8444) - mean (71ms) : 68, 74
master - mean (72ms) : 69, 74
section Bailout
This PR (8444) - mean (75ms) : 73, 77
master - mean (76ms) : 75, 78
section CallTarget+Inlining+NGEN
This PR (8444) - mean (1,067ms) : 1026, 1107
master - mean (1,066ms) : 1023, 1110
FakeDbCommand (.NET Core 3.1)gantt
title Execution time (ms) FakeDbCommand (.NET Core 3.1)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8444) - mean (111ms) : 108, 115
master - mean (111ms) : 108, 115
section Bailout
This PR (8444) - mean (112ms) : 110, 114
master - mean (113ms) : 110, 116
section CallTarget+Inlining+NGEN
This PR (8444) - mean (780ms) : 753, 807
master - mean (783ms) : 763, 803
FakeDbCommand (.NET 6)gantt
title Execution time (ms) FakeDbCommand (.NET 6)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8444) - mean (99ms) : 95, 103
master - mean (99ms) : 95, 103
section Bailout
This PR (8444) - mean (100ms) : 96, 103
master - mean (99ms) : 97, 102
section CallTarget+Inlining+NGEN
This PR (8444) - mean (932ms) : 893, 972
master - mean (931ms) : 903, 960
FakeDbCommand (.NET 8)gantt
title Execution time (ms) FakeDbCommand (.NET 8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8444) - mean (98ms) : 95, 101
master - mean (98ms) : 95, 101
section Bailout
This PR (8444) - mean (99ms) : 96, 101
master - mean (98ms) : 96, 100
section CallTarget+Inlining+NGEN
This PR (8444) - mean (816ms) : 782, 850
master - mean (811ms) : 778, 844
HttpMessageHandler (.NET Framework 4.8)gantt
title Execution time (ms) HttpMessageHandler (.NET Framework 4.8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8444) - mean (198ms) : 193, 203
master - mean (191ms) : 188, 195
section Bailout
This PR (8444) - mean (201ms) : 199, 204
master - mean (195ms) : 193, 196
section CallTarget+Inlining+NGEN
This PR (8444) - mean (1,174ms) : 1121, 1228
master - mean (1,141ms) : 1097, 1185
HttpMessageHandler (.NET Core 3.1)gantt
title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8444) - mean (285ms) : 279, 290
master - mean (273ms) : 269, 277
section Bailout
This PR (8444) - mean (285ms) : 281, 290
master - mean (274ms) : 271, 277
section CallTarget+Inlining+NGEN
This PR (8444) - mean (953ms) : 916, 990
master - mean (925ms) : 905, 944
HttpMessageHandler (.NET 6)gantt
title Execution time (ms) HttpMessageHandler (.NET 6)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8444) - mean (279ms) : 272, 286
master - mean (268ms) : 265, 272
section Bailout
This PR (8444) - mean (279ms) : 275, 284
master - mean (268ms) : 265, 270
section CallTarget+Inlining+NGEN
This PR (8444) - mean (1,153ms) : 1116, 1189
master - mean (1,136ms) : 1093, 1179
HttpMessageHandler (.NET 8)gantt
title Execution time (ms) HttpMessageHandler (.NET 8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8444) - mean (277ms) : 272, 283
master - mean (267ms) : 262, 271
section Bailout
This PR (8444) - mean (276ms) : 273, 279
master - mean (266ms) : 263, 270
section CallTarget+Inlining+NGEN
This PR (8444) - mean (1,036ms) : 985, 1087
master - mean (1,018ms) : 979, 1058
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
BenchmarksBenchmark execution time: 2026-04-16 09:05:49 Comparing candidate commit 25dabec in PR branch Found 0 performance improvements and 1 performance regressions! Performance is the same for 26 metrics, 0 unstable metrics, 87 known flaky benchmarks.
|
andrewlock
force-pushed
the
andrew/client-side-stats/fnvhash
branch
from
April 14, 2026 10:08
e94c3c9 to
9b41576
Compare
andrewlock
force-pushed
the
andrew/client-side-stats/update-processing-pipeline
branch
from
April 14, 2026 10:08
2a9f4b3 to
2106339
Compare
andrewlock
force-pushed
the
andrew/client-side-stats/fnvhash
branch
from
April 16, 2026 07:59
9b41576 to
25dabec
Compare
andrewlock
force-pushed
the
andrew/client-side-stats/update-processing-pipeline
branch
from
April 16, 2026 07:59
2106339 to
1783517
Compare
andrewlock
force-pushed
the
andrew/client-side-stats/update-processing-pipeline
branch
from
April 20, 2026 21:05
1783517 to
5ed9a0b
Compare
Base automatically changed from
andrew/client-side-stats/update-processing-pipeline
to
master
April 21, 2026 06:01
andrewlock
force-pushed
the
andrew/client-side-stats/fnvhash
branch
from
April 21, 2026 06:02
25dabec to
00ff8c1
Compare
andrewlock
force-pushed
the
andrew/client-side-stats/fnvhash
branch
from
April 21, 2026 06:04
00ff8c1 to
399a768
Compare
andrewlock
added a commit
that referenced
this pull request
Apr 21, 2026
## Summary of changes Improves the performance of peer-tag hash calculation for "fast path" cases (where there's an existing bucket) ## Reason for change We have to calculate the hash of all peer tags as part of client-side-stats bucketing calculations, which involves encoding them as utf-8. Additionally, when we _send_ the buckets, we have to send the tags as utf-8. In the initial CSS 1.2.0 implementation, we encoded the tags every-time we ran a calculation, which would allocate for every span that had peer tags, generally quite expensive. In this PR, we switch to doing the encoding twice: once with a zero-allocation implementation (amortized 0 on .NET Framework) to calculate the hash, and then, _if_ we need the "real" encoded tags, then we do that encoding again. ## Implementation details - Split the peer tags work in two, once to calculate the peer tags hash, once to get the actual tags as `key:value` - In the hash-calculation stage, we can use `stackalloc` for .NET Core, and array pool implementation for .NET Framework etc - As additional optimizations, we also - Pre-encode the peer tag _keys_ to utf-8, so that we only need to do those once for the hash calculation. - Pass key details from `BuildKey` to `GetEncodedPeerTags` (i.e. is this a "base service" only tag, if so, what's the tag value, otherwise how big does the peer tag list need to be) - If client-side stats is disabled, don't bother doing all the pre-calculation (or for trace filters) There are some possible future optimizations, _not_ implemented in this PR: - Track which tags require IP quantization to avoid re-doing it. (Maybe we should allow-list the quantization anyway, so that it only applies to specific tags? Or alternatively, block list?) - Instead of doing the encoding of `tag:value`, allow writing the pre-computed `byte[]` to `MessagePackBinary` and appending the `value`. This is doable, but requires updating the `MessagePackBinary` implementation to support it, so I considered it out of scope for now ## Test coverage Mostly covered by existing tests, but added some additional unit tests that directly compare the hashing to values used in Go agent tests. Additionally did some benchmarking. The key thing is that the `ClientSpanWithPeerTags` path is zero-allocation (and it's nice that the slow-path is still lower allocation than before, even if it's slower over all) | Method | Runtime | Mean | Error | Allocated | | -------------------------------------------------- | ------------------ | ----------: | ---------: | --------: | | BuildKey_SimpleSpan_Before | .NET 10.0 | 44.34 ns | 0.906 ns | - | | BuildKey_SimpleSpan_After | .NET 10.0 | 39.17 ns | 0.445 ns | - | | BuildKey_SimpleSpan_Before | .NET 6.0 | 93.13 ns | 1.837 ns | - | | BuildKey_SimpleSpan_After | .NET 6.0 | 87.56 ns | 0.839 ns | - | | BuildKey_SimpleSpan_Before | .NET Core 3.1 | 192.53 ns | 2.572 ns | - | | BuildKey_SimpleSpan_After | .NET Core 3.1 | 217.32 ns | 4.362 ns | - | | BuildKey_SimpleSpan_Before | .NET Framework 4.8 | 131.98 ns | 2.621 ns | - | | BuildKey_SimpleSpan_After | .NET Framework 4.8 | 175.20 ns | 7.647 ns | - | | | | | | | | BuildKey_ClientSpanNoPeerTags_Before | .NET 10.0 | 160.62 ns | 2.622 ns | - | | BuildKey_ClientSpanNoPeerTags_After | .NET 10.0 | 151.27 ns | 1.995 ns | - | | BuildKey_ClientSpanNoPeerTags_Before | .NET 6.0 | 243.62 ns | 3.199 ns | - | | BuildKey_ClientSpanNoPeerTags_After | .NET 6.0 | 232.55 ns | 2.975 ns | - | | BuildKey_ClientSpanNoPeerTags_Before | .NET Core 3.1 | 550.47 ns | 9.998 ns | - | | BuildKey_ClientSpanNoPeerTags_After | .NET Core 3.1 | 680.42 ns | 10.022 ns | - | | BuildKey_ClientSpanNoPeerTags_Before | .NET Framework 4.8 | 442.88 ns | 8.854 ns | - | | BuildKey_ClientSpanNoPeerTags_After | .NET Framework 4.8 | 682.12 ns | 13.468 ns | - | | | | | | | | BuildKey_ClientSpanWithPeerTags_Before | .NET 10.0 | 800.82 ns | 15.062 ns | 840 B | | BuildKey_ClientSpanWithPeerTags_After | .NET 10.0 | 575.04 ns | 5.604 ns | - | | BuildKey_ClientSpanWithPeerTags_Before | .NET 6.0 | 1,076.07 ns | 13.124 ns | 840 B | | BuildKey_ClientSpanWithPeerTags_After | .NET 6.0 | 1,440.26 ns | 28.755 ns | - | | BuildKey_ClientSpanWithPeerTags_Before | .NET Core 3.1 | 1,531.27 ns | 13.509 ns | 840 B | | BuildKey_ClientSpanWithPeerTags_After | .NET Core 3.1 | 1,774.35 ns | 89.601 ns | - | | BuildKey_ClientSpanWithPeerTags_Before | .NET Framework 4.8 | 1,682.15 ns | 17.082 ns | 859 B | | BuildKey_ClientSpanWithPeerTags_After | .NET Framework 4.8 | 2,040.90 ns | 162.895 ns | - | | | | | | | | BuildKey_ClientSpanWithPeerTags_GetEncodedPeerTags | .NET 10.0 | 1,141.85 ns | 9.980 ns | 760 B | | BuildKey_ClientSpanWithPeerTags_GetEncodedPeerTags | .NET 6.0 | 2,720.55 ns | 53.097 ns | 760 B | | BuildKey_ClientSpanWithPeerTags_GetEncodedPeerTags | .NET Core 3.1 | 3,534.28 ns | 141.234 ns | 760 B | | BuildKey_ClientSpanWithPeerTags_GetEncodedPeerTags | .NET Framework 4.8 | 3,438.80 ns | 84.352 ns | 778 B | <details><summary>Details</summary> <p> ```csharp // <copyright file="StatsAggregatorBenchmark.cs" company="Datadog"> // Unless explicitly stated otherwise all files in this repository are licensed under the Apache 2 License. // This product includes software developed at Datadog (https://www.datadoghq.com/). Copyright 2017 Datadog, Inc. // </copyright> using System; using System.Collections.Generic; using System.Threading.Tasks; using BenchmarkDotNet.Attributes; using Datadog.Trace; using Datadog.Trace.Agent; using Datadog.Trace.Agent.DiscoveryService; using Datadog.Trace.Configuration; using Datadog.Trace.Tagging; namespace Benchmarks.Trace; /// <summary> /// StatsAggregator.BuildKey benchmarks /// </summary> [MemoryDiagnoser] [BenchmarkCategory(Constants.TracerCategory)] public class StatsAggregatorBenchmark { private static readonly List<StatsAggregator.PeerTagKey> PeerTagKeys = [ new("_dd.base_service"), new("aws.queue.name"), new("aws.queue.url"), new("aws.s3.bucket"), new("aws.stream.name"), new("bucketname"), new("db.couchbase.seed.nodes"), new("db.hostname"), new("db.instance"), new("db.system"), new("messaging.destination"), new("messaging.kafka.bootstrap.servers"), new("messaging.rabbitmq.exchange"), new("messaging.system"), new("network.destination.name"), new("peer.hostname"), new("peer.service"), new("server.address"), new("topic"), ]; private StatsAggregator _aggregator; private Span _simpleSpan; private Span _clientSpanNoPeerTags; private Span _clientSpanWithPeerTags; private StatsAggregationKey _key; private List<byte[]> _encoded; [GlobalSetup] public void GlobalSetup() { _aggregator = new StatsAggregator( new NoOpApi(), new TracerSettings(), NullDiscoveryService.Instance, isOtlp: false); var now = DateTimeOffset.UtcNow; // Simple span: no span kind, no peer tags — exercises the "internal" fast path _simpleSpan = CreateSpan(now, "web-service", "web.request", "GET /api/users", "web"); // Client span with SpanKind but no matching peer tags — iterates peer tag keys but finds nothing var clientTags = new HttpTags { HttpMethod = "GET", HttpStatusCode = "200" }; _clientSpanNoPeerTags = CreateSpan(now, "web-service", "http.request", "GET /api/orders", "http", clientTags); // Client span with several peer tags set — the most expensive path var peerTags = new HttpTags { HttpMethod = "POST", HttpStatusCode = "200" }; _clientSpanWithPeerTags = CreateSpan(now, "web-service", "http.request", "POST /api/data", "http", peerTags); _clientSpanWithPeerTags.Tags.SetTag("peer.service", "remote-service"); _clientSpanWithPeerTags.Tags.SetTag("db.instance", "i-1234"); _clientSpanWithPeerTags.Tags.SetTag("db.system", "postgres"); _clientSpanWithPeerTags.Tags.SetTag("server.address", "db.example.com"); _clientSpanWithPeerTags.Tags.SetTag("network.destination.name", "db.example.com"); } [GlobalCleanup] public void GlobalCleanup() { _aggregator.DisposeAsync().GetAwaiter().GetResult(); var value = _key; var encoded = _encoded; } /// <summary> /// BuildKey for a simple span with no span kind (internal fast-path, no peer tag iteration). /// </summary> [Benchmark] public void BuildKey_SimpleSpan() { _key = _aggregator.BuildKey(_simpleSpan, PeerTagKeys, out _); } /// <summary> /// BuildKey for a client span that has no matching peer tags (iterates all peer tag keys, finds none). /// </summary> [Benchmark] public void BuildKey_ClientSpanNoPeerTags() { _key = _aggregator.BuildKey(_clientSpanNoPeerTags, PeerTagKeys, out _); } /// <summary> /// BuildKey for a client span with several peer tags set (UTF-8 encoding + FNV hashing). /// </summary> [Benchmark] public void BuildKey_ClientSpanWithPeerTags() { _key = _aggregator.BuildKey(_clientSpanWithPeerTags, PeerTagKeys, out var results); } /// <summary> /// BuildKey for a client span with several peer tags set (UTF-8 encoding + FNV hashing). /// </summary> [Benchmark] public void BuildKey_ClientSpanWithPeerTags_GetEncodedPeerTags() { _key = _aggregator.BuildKey(_clientSpanWithPeerTags, PeerTagKeys, out var results); _encoded = StatsAggregator.GetEncodedPeerTags(_clientSpanWithPeerTags, PeerTagKeys, in results); } private static Span CreateSpan(DateTimeOffset start, string serviceName, string operationName, string resourceName, string type, ITags tags = null) { var tracer = Benchmarks.Trace.Asm.EmptyDatadogTracer.Instance; var traceContext = new TraceContext(tracer); var context = new SpanContext(null, traceContext, serviceName); var span = new Span(context, start, tags); span.OperationName = operationName; span.ResourceName = resourceName; span.Type = type; return span; } private sealed class NoOpApi : IApi { public TracesEncoding TracesEncoding => TracesEncoding.DatadogV0_4; public Task<bool> Ping() => Task.FromResult(true); public Task<bool> SendTracesAsync(ArraySegment<byte> traces, int numberOfTraces, bool statsComputationEnabled, long numberOfDroppedP0Traces, long numberOfDroppedP0Spans, bool apmTracingEnabled = true) => Task.FromResult(true); public Task<bool> SendStatsAsync(StatsBuffer stats, long bucketDuration, int tracerObfuscationVersion) => Task.FromResult(true); } } ``` </p> </details> ## Other details Part of a stack: - #8417 - #8418 - #8420 - #8435 - #8436 - #8444
Member
Author
|
Incorporated into #8445 instead |
andrewlock
added a commit
that referenced
this pull request
Apr 23, 2026
## Summary of changes Removes the minimum-agent version check from the client-side-stats check ## Reason for change We didn't have it before, which means people who previously had stats, suddenly won't have them. Also other agents (like bottlecap etc) don't have the version. And we haven't figured out how to handle that yet. While CSS is off by default, this seems like the best option to take. ## Implementation details Remove the code that is checking for CSS support. ## Test coverage Covered by existing (and removed a test for the behaviour) ## Other details Follows on from - #8417 - #8418 - #8420 - #8435 - #8436 - #8444 - #8445
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary of changes
Allow passing an offset and count to
FnvHash64.GenerateHash()Reason for change
Let's us hash values stored in a
byte[]that comes from an array pool, instead of requiring hashing the wholebyte[]. Allows for various optimizations in places.Implementation details
int offset, int countvaluesbyte[]to use the overloadTest coverage
Added unit tests to cover the new behaviour
Other details
Part of a stack:
charhelpers likeIsAsciiHexDigit#8417IpAddressObfuscationUtilfor use with client-side-stats #8418ContainsSpanIdtoSpanCollection#8435