## Summary of changes

Adds the `IpAddressObfuscationUtil` helper for use by client-side stats 

## Reason for change

It's not part of the RFC, but the agent does quantization of
IPaddresses, which could appear in peer tags (which are used for stats
calculations). Without this quantization, we would end up with an
explosion of buckets

## Implementation details

Used 🤖 to port the implementation from the Go agent, including the
tests. Then iterated on it repeatedly to try to bring the allocations
down as low as possible.

⚠️ There's still one code path which is kinda nasty where we try to
parse if the tag is a valid IP v6, as it causes an allocation on .NET
Framework, but I couldn't find a way to avoid that. Given all the
allocations we're going to get _anyway_, I think we will just have to
accept it for practical reasons, unless other people feel strongly

## Test coverage

Grabbed the quantization unit tests from the agent

## Other details

Pre-requisite for client-side-stats improvements. Part of a stack:

- #8417

## Summary of changes

Updates the existing client-side-stats implementation to match version
1.2.0 [as defined in the
RFC](https://datadoghq.atlassian.net/wiki/spaces/APM/pages/6378947571/Client-Side+Stats+v1.2.0)
and as [implemented in the
agent](https://github.com/DataDog/datadog-agent/blob/c1d67a906f4c594654600760da1eea4c8037471a/pkg/proto/datadog/trace/stats.proto#L83).

## Reason for change

Our implementation is severely lagging the latest implementation in the
agent. This hasn't been a big deal, as it's not documented and not
enabled by default, but we'd like to fix the implementation to make it
usable.

## Implementation details

This was driven almost entirely by 🤖, by comparing our existing
implementation to the RFC, and also taking the agent/go implementation
as the canonical implementation.

> Implementing this in .NET highlighted a number of missing aspects in
the RFC, which I've raised elsewhere, and aim to get incorporated into
the RFC.

At a high level, the PR contains the following changes:

- Stats Wire Format & Serialization
- **Added new aggregation dimensions**: `SpanKind`, `IsTraceRoot` (as
Trilean), `HTTPMethod`, `HTTPEndpoint`, `GRPCStatusCode`,
`ServiceSource`, `PeerTags` to both the aggregation key and the msgpack
wire format
- **Fixed `GRPCStatusCode` type**: Changed from `int` to `string` to
match Go agent's wire format (agent was returning 400 Bad Request in
system tests)
- **gRPC status code extraction**: Checks 4 tag names in priority order
(`rpc.grpc.status_code`, `grpc.code`, `rpc.grpc.status.code`,
`grpc.status.code`)
- **Stochastic rounding**: `Hits`, `Errors`, `Duration`, `TopLevelHits`
accumulated as `double` (weighted by sampling rate) then rounded
probabilistically to `int64` for serialization
- **Duration weighting**: Durations are now multiplied by sampling
weight (`1/rate`), matching Go agent behavior
- **Default env**: Serializes `"unknown-env"` when environment is not
configured
  - **`git_commit_sha`**: Added as optional field in stats payload
  - **`Service`**: Added as top-level field in stats payload
- **Empty bucket suppression**: `HasHits()` check prevents sending
payloads with zero-hit buckets (stale keys retained for sketch reuse)
- Bucket Timing
- **10-second alignment**: Bucket `Start` timestamps aligned to
10-second boundaries (`ts - ts % 10_000_000_000`) matching Go tracer's
`alignTs`
- **Removed unused `StartTime`** property (only `Start` as aligned
nanoseconds)
- Agent Discovery (`/info` Endpoint)
- **`peer_tags`**: Parsed, sorted, deduplicated; used for peer tag
extraction on client/producer/consumer spans
- **`span_kinds_stats_computed`**: Parsed to override eligible span
kinds
  - **`obfuscation_version`**: Parsed for obfuscation negotiation
- **Trace filters**: Parsed `filter_tags`, `filter_tags_regex`,
`ignore_resources` from `/info`
- Trace Filtering
- **`TraceFilter` implementation**: Evaluates agent-configured filters
(exact tags, regex tags, resource patterns) against root spans before
stats computation
- **Tag-only filters**: Handles filter entries that match on tag key
presence without a specific value
- SQL Obfuscation
- **Operator splitters**: Added `* / = < > ! & ^ % ~ ? @ : #` as token
splitters (matching Go agent's `go-sqllexer` `isOperator()`) so queries
like `WHERE id='1'` are properly obfuscated
- **Whitespace normalization**: Post-processing pass adds spaces around
comparison operators (`=`, `<`, `>`, `!`) adjacent to `?` placeholders,
matching Go agent's normalizer output (e.g., `id='1'` → `id = ?`)
- **Obfuscation gating**: Only runs when `obfuscation_version` is
negotiated with agent; sends `Datadog-Obfuscation-Version` header
- Peer Tags
- **IP quantization**: Peer tag values run through
`IpAddressObfuscationUtil.QuantizePeerIpAddresses()` replacing
non-allowed IPs with `"blocked-ip-address"`
- **Base service handling**: Internal/missing-kind spans with
non-default service name use `_dd.base_service:{serviceName}` as sole
peer tag
- **FNV-1a hashing**: Peer tags hashed with null-byte separators for
aggregation key
- Other
- **No retries for stats**: Stats sends are fire-and-forget (retry limit
= 0)
- **Synthetics detection**: Uses `StartsWith("synthetics")` prefix
matching (not exact match)
- **Mock agent updates**: Test mock agent returns `obfuscation_version`
in `/info` response

## Test coverage

There's a _lot_ going on in this PR, because we were so far behind. I
_could_ split this into implementing individual features, but there
would be a lot of duplication between PRs, and it didn't seem like it
would be that easy to track. At least with this big bang we can compare
directly against the system tests etc.

The existing system tests for stats computation were checked, and made
to pass (which identified a number of hidden expectations which will be
added to the RFC). I'll create a PR to enable these in the system-tests
repo

## Other details

Part of a stack

- #8417
- #8418

There are still some _theoretical_ gaps between the go implementation
and the .NET implementation, but I _think_ these are non-issues in
_most_ cases:


  | Area | Gap | Impact |
  |------|-----|--------|
| gRPC status code normalization | Go agent normalizes string statuses
(e.g., `"CANCELLED"` → `"1"`); .NET passes raw tag value | Stats
mismatch if gRPC library uses string-form status codes |
| HTTP status code tags | Go agent checks both `http.status_code` and
`http.response.status_code` (OTel convention); .NET only checks
`http.status_code` | OTel spans using newer convention would get `0` in
.NET |
| Duration precision truncation | Go agent uses float bit masking; .NET
uses integer shifting — both target ~10 bits but may produce slightly
different values | Minor histogram differences |
| SQL obfuscation | Go agent uses full tokenizer + normalizer; .NET uses
character-level splitter with targeted normalization around comparison
operators only | Complex SQL with unusual formatting may produce
different resource strings |
| `HTTP_method` / `HTTP_endpoint` | .NET always populates from span
tags; Go agent only populates from newer OTel pipeline paths | Creates
different aggregation keys — Go groups all methods/routes together in
default path |
| `_top_level` metric | Go checks both `_top_level` and `_dd.top_level`;
.NET only checks `_dd.top_level` | Spans using older metric name would
be missed by .NET |
| `span_derived_primary_tags` | Still in Go agent code but implemented
in v1.3.0 RFC, which was reverted; removed from .NET | No current
impact; may need to re-add if spec reverts back |

There is another big elephant in the room, which is perf. The peer tags,
in particular, currently requires a _bunch_ of allocation. I'd rather
defer trying to fight against that to another PR if possible, unless
anyone has some clear ideas 😄

Another aspect I'm not sure about is how this interacts with
@zacharycmontoya's recent work to publish OTLP stats. I took a random
guess and fought the refactoring, but need to verify it.

---------

Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>
Co-authored-by: Lucas Pimentel <[email protected]>

## Summary of changes

Extracts `ContainsSpanId` from `TraceChunkModel` to `SpanCollection`

## Reason for change

I need to find a given span in a `SpanCollection` for client-side-stats.
The algorithm used by `TraceChunkModel` does it "properly" (by stating
from the last span when looking for a root span), so it seems to make
sense to move this method to `SpanCollection`. Additionally, this
essentially hides some of the internals about the `SpanCollection` type,
so seems like a better place for it anyway.

## Implementation details

- Move the `IndexOf()` method from `TraceChunkModel` to `SpanCollection`
- Switch it to `Contains()` instead, seeing as the index is never
actually used AFAICT. It's easy to switch it back later if we _do_ need
it.

## Test coverage

Added some unit tests for the method

## Other details


Part of a stack

- #8417
- #8418
- #8420

## Summary of changes

Updates the processing pipeline for client side stats

## Reason for change

The bulk of the CSS 1.2.0 work was done in #8420, but the
span-processing pipeline wasn't the same as the Go agent:

| Step | Go Agent  | .NET Tracer                            |
| ---- | --------- | -------------------------------------- |
| 1    | Normalize | Normalize + Obfuscate (`ProcessTrace`) |
| 2    | Filter    | Filter + Sample (`ShouldKeepTrace`)    |
| 3    | Obfuscate |                                        |
| 4    | Sample    |                                        |
| 5    | Stats     | Stats (`AddRange`)                     |


## Implementation details

The difference was mostly due to the way we had separate
`ProcessTrace()` and `ShouldKeepTrace()` methods. By merging them, we
can encapsulate the pipeline, and ensure that we mirror the way the Go
agent processes traces.

Additionally, there are two different mechanisms for dropping spans now
- p0drops and "trace filtering", so added a new metric tag
`reason:trace_filter` for it.

Additionally, made some small optimizations:
- No need for an array of `ITraceProcessor` when it only contains one
- Updated `TraceFilter` algorithm to correctly look for local trace
root, starting from last span
- Made `_statsAggregator` non-null in `AgentWriter` to eliminate a bunch
of null checks

## Test coverage

Covered by existing tests.

## Other details

Part of a stack:
- #8417
- #8418
- #8420
- #8435

## Summary of changes

Improves the performance of peer-tag hash calculation for "fast path"
cases (where there's an existing bucket)

## Reason for change

We have to calculate the hash of all peer tags as part of
client-side-stats bucketing calculations, which involves encoding them
as utf-8. Additionally, when we _send_ the buckets, we have to send the
tags as utf-8.

In the initial CSS 1.2.0 implementation, we encoded the tags every-time
we ran a calculation, which would allocate for every span that had peer
tags, generally quite expensive. In this PR, we switch to doing the
encoding twice: once with a zero-allocation implementation (amortized 0
on .NET Framework) to calculate the hash, and then, _if_ we need the
"real" encoded tags, then we do that encoding again.

## Implementation details

- Split the peer tags work in two, once to calculate the peer tags hash,
once to get the actual tags as `key:value`
- In the hash-calculation stage, we can use `stackalloc` for .NET Core,
and array pool implementation for .NET Framework etc
- As additional optimizations, we also 
- Pre-encode the peer tag _keys_ to utf-8, so that we only need to do
those once for the hash calculation.
- Pass key details from `BuildKey` to `GetEncodedPeerTags` (i.e. is this
a "base service" only tag, if so, what's the tag value, otherwise how
big does the peer tag list need to be)
- If client-side stats is disabled, don't bother doing all the
pre-calculation (or for trace filters)

There are some possible future optimizations, _not_ implemented in this
PR:
- Track which tags require IP quantization to avoid re-doing it. (Maybe
we should allow-list the quantization anyway, so that it only applies to
specific tags? Or alternatively, block list?)
- Instead of doing the encoding of `tag:value`, allow writing the
pre-computed `byte[]` to `MessagePackBinary` and appending the `value`.
This is doable, but requires updating the `MessagePackBinary`
implementation to support it, so I considered it out of scope for now

## Test coverage

Mostly covered by existing tests, but added some additional unit tests
that directly compare the hashing to values used in Go agent tests.

Additionally did some benchmarking. The key thing is that the
`ClientSpanWithPeerTags` path is zero-allocation (and it's nice that the
slow-path is still lower allocation than before, even if it's slower
over all)

| Method | Runtime | Mean | Error | Allocated |
| -------------------------------------------------- |
------------------ | ----------: | ---------: | --------: |
| BuildKey_SimpleSpan_Before | .NET 10.0 | 44.34 ns | 0.906 ns | - |
| BuildKey_SimpleSpan_After | .NET 10.0 | 39.17 ns | 0.445 ns | - |
| BuildKey_SimpleSpan_Before | .NET 6.0 | 93.13 ns | 1.837 ns | - |
| BuildKey_SimpleSpan_After | .NET 6.0 | 87.56 ns | 0.839 ns | - |
| BuildKey_SimpleSpan_Before | .NET Core 3.1 | 192.53 ns | 2.572 ns | -
|
| BuildKey_SimpleSpan_After | .NET Core 3.1 | 217.32 ns | 4.362 ns | - |
| BuildKey_SimpleSpan_Before | .NET Framework 4.8 | 131.98 ns | 2.621 ns
| - |
| BuildKey_SimpleSpan_After | .NET Framework 4.8 | 175.20 ns | 7.647 ns
| - |
| | | | | |
| BuildKey_ClientSpanNoPeerTags_Before | .NET 10.0 | 160.62 ns | 2.622
ns | - |
| BuildKey_ClientSpanNoPeerTags_After | .NET 10.0 | 151.27 ns | 1.995 ns
| - |
| BuildKey_ClientSpanNoPeerTags_Before | .NET 6.0 | 243.62 ns | 3.199 ns
| - |
| BuildKey_ClientSpanNoPeerTags_After | .NET 6.0 | 232.55 ns | 2.975 ns
| - |
| BuildKey_ClientSpanNoPeerTags_Before | .NET Core 3.1 | 550.47 ns |
9.998 ns | - |
| BuildKey_ClientSpanNoPeerTags_After | .NET Core 3.1 | 680.42 ns |
10.022 ns | - |
| BuildKey_ClientSpanNoPeerTags_Before | .NET Framework 4.8 | 442.88 ns
| 8.854 ns | - |
| BuildKey_ClientSpanNoPeerTags_After | .NET Framework 4.8 | 682.12 ns |
13.468 ns | - |
| | | | | |
| BuildKey_ClientSpanWithPeerTags_Before | .NET 10.0 | 800.82 ns |
15.062 ns | 840 B |
| BuildKey_ClientSpanWithPeerTags_After | .NET 10.0 | 575.04 ns | 5.604
ns | - |
| BuildKey_ClientSpanWithPeerTags_Before | .NET 6.0 | 1,076.07 ns |
13.124 ns | 840 B |
| BuildKey_ClientSpanWithPeerTags_After | .NET 6.0 | 1,440.26 ns |
28.755 ns | - |
| BuildKey_ClientSpanWithPeerTags_Before | .NET Core 3.1 | 1,531.27 ns |
13.509 ns | 840 B |
| BuildKey_ClientSpanWithPeerTags_After | .NET Core 3.1 | 1,774.35 ns |
89.601 ns | - |
| BuildKey_ClientSpanWithPeerTags_Before | .NET Framework 4.8 | 1,682.15
ns | 17.082 ns | 859 B |
| BuildKey_ClientSpanWithPeerTags_After | .NET Framework 4.8 | 2,040.90
ns | 162.895 ns | - |
| | | | | |
| BuildKey_ClientSpanWithPeerTags_GetEncodedPeerTags | .NET 10.0 |
1,141.85 ns | 9.980 ns | 760 B |
| BuildKey_ClientSpanWithPeerTags_GetEncodedPeerTags | .NET 6.0 |
2,720.55 ns | 53.097 ns | 760 B |
| BuildKey_ClientSpanWithPeerTags_GetEncodedPeerTags | .NET Core 3.1 |
3,534.28 ns | 141.234 ns | 760 B |
| BuildKey_ClientSpanWithPeerTags_GetEncodedPeerTags | .NET Framework
4.8 | 3,438.80 ns | 84.352 ns | 778 B |


<details><summary>Details</summary>
<p>

```csharp
// <copyright file="StatsAggregatorBenchmark.cs" company="Datadog">
// Unless explicitly stated otherwise all files in this repository are licensed under the Apache 2 License.
// This product includes software developed at Datadog (https://www.datadoghq.com/). Copyright 2017 Datadog, Inc.
// </copyright>

using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using BenchmarkDotNet.Attributes;
using Datadog.Trace;
using Datadog.Trace.Agent;
using Datadog.Trace.Agent.DiscoveryService;
using Datadog.Trace.Configuration;
using Datadog.Trace.Tagging;

namespace Benchmarks.Trace;

/// <summary>
/// StatsAggregator.BuildKey benchmarks
/// </summary>
[MemoryDiagnoser]
[BenchmarkCategory(Constants.TracerCategory)]
public class StatsAggregatorBenchmark
{
    private static readonly List<StatsAggregator.PeerTagKey> PeerTagKeys =
    [
        new("_dd.base_service"),
        new("aws.queue.name"),
        new("aws.queue.url"),
        new("aws.s3.bucket"),
        new("aws.stream.name"),
        new("bucketname"),
        new("db.couchbase.seed.nodes"),
        new("db.hostname"),
        new("db.instance"),
        new("db.system"),
        new("messaging.destination"),
        new("messaging.kafka.bootstrap.servers"),
        new("messaging.rabbitmq.exchange"),
        new("messaging.system"),
        new("network.destination.name"),
        new("peer.hostname"),
        new("peer.service"),
        new("server.address"),
        new("topic"),
    ];

    private StatsAggregator _aggregator;
    private Span _simpleSpan;
    private Span _clientSpanNoPeerTags;
    private Span _clientSpanWithPeerTags;
    private StatsAggregationKey _key;
    private List<byte[]> _encoded;

    [GlobalSetup]
    public void GlobalSetup()
    {
        _aggregator = new StatsAggregator(
            new NoOpApi(),
            new TracerSettings(),
            NullDiscoveryService.Instance,
            isOtlp: false);

        var now = DateTimeOffset.UtcNow;

        // Simple span: no span kind, no peer tags — exercises the "internal" fast path
        _simpleSpan = CreateSpan(now, "web-service", "web.request", "GET /api/users", "web");

        // Client span with SpanKind but no matching peer tags — iterates peer tag keys but finds nothing
        var clientTags = new HttpTags { HttpMethod = "GET", HttpStatusCode = "200" };
        _clientSpanNoPeerTags = CreateSpan(now, "web-service", "http.request", "GET /api/orders", "http", clientTags);

        // Client span with several peer tags set — the most expensive path
        var peerTags = new HttpTags { HttpMethod = "POST", HttpStatusCode = "200" };
        _clientSpanWithPeerTags = CreateSpan(now, "web-service", "http.request", "POST /api/data", "http", peerTags);
        _clientSpanWithPeerTags.Tags.SetTag("peer.service", "remote-service");
        _clientSpanWithPeerTags.Tags.SetTag("db.instance", "i-1234");
        _clientSpanWithPeerTags.Tags.SetTag("db.system", "postgres");
        _clientSpanWithPeerTags.Tags.SetTag("server.address", "db.example.com");
        _clientSpanWithPeerTags.Tags.SetTag("network.destination.name", "db.example.com");

    }

    [GlobalCleanup]
    public void GlobalCleanup()
    {
        _aggregator.DisposeAsync().GetAwaiter().GetResult();
        var value = _key;
        var encoded = _encoded;
    }

    /// <summary>
    /// BuildKey for a simple span with no span kind (internal fast-path, no peer tag iteration).
    /// </summary>
    [Benchmark]
    public void BuildKey_SimpleSpan()
    {
        _key = _aggregator.BuildKey(_simpleSpan, PeerTagKeys, out _);
    }

    /// <summary>
    /// BuildKey for a client span that has no matching peer tags (iterates all peer tag keys, finds none).
    /// </summary>
    [Benchmark]
    public void BuildKey_ClientSpanNoPeerTags()
    {
        _key = _aggregator.BuildKey(_clientSpanNoPeerTags, PeerTagKeys, out _);
    }

    /// <summary>
    /// BuildKey for a client span with several peer tags set (UTF-8 encoding + FNV hashing).
    /// </summary>
    [Benchmark]
    public void BuildKey_ClientSpanWithPeerTags()
    {
        _key = _aggregator.BuildKey(_clientSpanWithPeerTags, PeerTagKeys, out var results);
    }

    /// <summary>
    /// BuildKey for a client span with several peer tags set (UTF-8 encoding + FNV hashing).
    /// </summary>
    [Benchmark]
    public void BuildKey_ClientSpanWithPeerTags_GetEncodedPeerTags()
    {
        _key = _aggregator.BuildKey(_clientSpanWithPeerTags, PeerTagKeys, out var results);
        _encoded = StatsAggregator.GetEncodedPeerTags(_clientSpanWithPeerTags, PeerTagKeys, in results);
    }

    private static Span CreateSpan(DateTimeOffset start, string serviceName, string operationName, string resourceName, string type, ITags tags = null)
    {
        var tracer = Benchmarks.Trace.Asm.EmptyDatadogTracer.Instance;
        var traceContext = new TraceContext(tracer);
        var context = new SpanContext(null, traceContext, serviceName);
        var span = new Span(context, start, tags);
        span.OperationName = operationName;
        span.ResourceName = resourceName;
        span.Type = type;
        return span;
    }

    private sealed class NoOpApi : IApi
    {
        public TracesEncoding TracesEncoding => TracesEncoding.DatadogV0_4;

        public Task<bool> Ping() => Task.FromResult(true);

        public Task<bool> SendTracesAsync(ArraySegment<byte> traces, int numberOfTraces, bool statsComputationEnabled, long numberOfDroppedP0Traces, long numberOfDroppedP0Spans, bool apmTracingEnabled = true)
            => Task.FromResult(true);

        public Task<bool> SendStatsAsync(StatsBuffer stats, long bucketDuration, int tracerObfuscationVersion)
            => Task.FromResult(true);
    }
}
```

</p>
</details> 


## Other details


Part of a stack:
- #8417
- #8418
- #8420
- #8435
- #8436
- #8444

## Summary of changes

Removes the minimum-agent version check from the client-side-stats check

## Reason for change

We didn't have it before, which means people who previously had stats,
suddenly won't have them. Also other agents (like bottlecap etc) don't
have the version. And we haven't figured out how to handle that yet.
While CSS is off by default, this seems like the best option to take.

## Implementation details

Remove the code that is checking for CSS support.

## Test coverage

Covered by existing (and removed a test for the behaviour)

## Other details

Follows on from 
- #8417
- #8418
- #8420
- #8435
- #8436
- #8444
- #8445