Summary of changes

Adds two new capabilities to the stats computation feature:

1. When stats computation is enabled through configuration, the tracer sends a request to the trace agent's /info endpoint to ensure that this feature is supported by the trace agent. If not, the feature is disabled.
2. The tracer also checks whether it can drop P0 traces (user-rejected traces or sampler-rejected traces). When this feature is supported by the trace agent, P0 traces are no longer serialized to the agent and the "Datadog-Client-Dropped-P0-Traces" and "Datadog-Client-Dropped-P0-Spans" request headers are set on the next trace payload.

Additionally, adds unit tests and integration tests that test all of the agreed upon cross-tracer tests for the stats computation feature.

Reason for change

Computing stats in the tracer reduces a large amount of span serialization (from the tracer) and span deserialization (from the trace agent), which reduces CPU overhead for users with significant amounts of trace agent traffic. The increased testing also adds confidence that our implementation is on par with the implementations within other tracers (since this is moving a trace agent feature to the individual language tracers).

Implementation details

StatsAggregator initialization now executes Endpoint Discovery

Background: When stats computation is enabled, the StatsAggregator object is responsible for buffering stats and periodically sending them to the trace agent's stats endpoint. Stats are computed by passing in an array of spans into the AddRange API, which is called by the AgentWriter when a local trace is finished and is buffered/serialized.

Much like the LiveDebugger initialization, the constructor of StatsAggregator will invoke a task on the threadpool to ping the agent URL at the /info endpoint to check what endpoints and features it supports. We specifically check for two properties:

• endpoints[] contains "/v0.6/stats". If true, stats computation is enabled via the new property CanComputeStats.
• client_drop_p0s property is set to true. If so, dropping P0 spans is enabled via the new property CanDropP0s.

Note: Since spans and traces can be finishing while the service discovery is ongoing, we let the StatsAggregator store stats points while the operation is ongoing. If initialization succeeds, the stats aggregrator can start sending stats to the trace agent. If initialization fails, the stats aggregator does not store new stats points and refuses to send stats to the trace agent.

TraceContext changes

Background: The TraceContext object is responsible for flushing finished spans to its Tracer object reference when either all spans in the local trace are finished or the number of finished spans in the ongoing trace exceeds the partial flush limit. After this point, the finished spans are sent to the AgentWriter to be buffered/serialized, so we must make the decision here whether to drop P0 spans.

If Tracer.CanDropP0s == true, then the following algorithm is run on each of the finished spans to determine if the span should be kept. If at least one span is kept, then the entire trace is kept. If not, then the entire trace is dropped.

• Is the sampling priority greater than 0?
• Does the span have an error?
• Does the span have an app analytics sample rate? If so, does the sampling decision determine it should be sampled?

We then pass the keep/drop decision along with the finished span array to the AgentWriter.WriteTrace API.

Note: There will need to be some special handling if a trace is supposed to be sampled out but the new single span ingestion controls determine that single spans should be kept. This is documented in the RFC and will need to be handled when the feature is implemented in the .NET Tracer.

AgentWriter / IApi changes

Background: The AgentWriter puts finished spans into buffers, and when a buffer is full those spans are sent to the trace agent by calling IApi.SendTracesAsync.

When the AgentWriter receives spans, it now accumulates the number of dropped P0 traces and dropped P0 spans. Then, when a span buffer needs to be serialized and IApi.SendTracesAsync is invoked, it passes along these counts and they are added as headers to the trace agent request.

Small changes

• The DiscoveryService.Create method was refactored because the IConfigurationSource parameter was redundant.
• The MockTracerAgent test class and all of its subclasses have been updated to accept an AgentConfiguration object which allows us to mock different agent capabilities. This was used to test scenarios where stats computation was allowed but not the dropping of P0 spans.

Test coverage

New tests cover those documented by internal RFCs.

Other details

N/A