[Tracing] Add experimental support for exporting traces as OTLP (.NET 6+)#8211
Conversation
54892c0 to
771b40f
Compare
BenchmarksBenchmark execution time: 2026-03-24 04:06:06 Comparing candidate commit 58fd979 in PR branch Found 11 performance improvements and 15 performance regressions! Performance is the same for 245 metrics, 17 unstable metrics.
|
771b40f to
e729239
Compare
Execution-Time Benchmarks Report ⏱️Execution-time results for samples comparing This PR (8211) and master. ✅ No regressions detected - check the details below Full Metrics ComparisonFakeDbCommand
HttpMessageHandler
Comparison explanationExecution-time benchmarks measure the whole time it takes to execute a program, and are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are highlighted in **red**. The following thresholds were used for comparing the execution times:
Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard. Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph). Duration chartsFakeDbCommand (.NET Framework 4.8)gantt
title Execution time (ms) FakeDbCommand (.NET Framework 4.8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8211) - mean (69ms) : 67, 71
master - mean (69ms) : 67, 71
section Bailout
This PR (8211) - mean (72ms) : 71, 74
master - mean (73ms) : 71, 74
section CallTarget+Inlining+NGEN
This PR (8211) - mean (1,054ms) : 998, 1110
master - mean (1,051ms) : 978, 1123
FakeDbCommand (.NET Core 3.1)gantt
title Execution time (ms) FakeDbCommand (.NET Core 3.1)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8211) - mean (107ms) : 104, 110
master - mean (107ms) : 104, 110
section Bailout
This PR (8211) - mean (108ms) : 106, 110
master - mean (108ms) : 105, 110
section CallTarget+Inlining+NGEN
This PR (8211) - mean (774ms) : 755, 793
master - mean (769ms) : 753, 785
FakeDbCommand (.NET 6)gantt
title Execution time (ms) FakeDbCommand (.NET 6)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8211) - mean (95ms) : 92, 97
master - mean (95ms) : 92, 97
section Bailout
This PR (8211) - mean (96ms) : 94, 97
master - mean (95ms) : 94, 97
section CallTarget+Inlining+NGEN
This PR (8211) - mean (762ms) : 743, 781
master - mean (753ms) : 730, 776
FakeDbCommand (.NET 8)gantt
title Execution time (ms) FakeDbCommand (.NET 8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8211) - mean (93ms) : 91, 96
master - mean (94ms) : 91, 97
section Bailout
This PR (8211) - mean (95ms) : 93, 96
master - mean (95ms) : 93, 97
section CallTarget+Inlining+NGEN
This PR (8211) - mean (661ms) : 643, 680
master - mean (658ms) : 641, 675
HttpMessageHandler (.NET Framework 4.8)gantt
title Execution time (ms) HttpMessageHandler (.NET Framework 4.8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8211) - mean (192ms) : 189, 196
master - mean (205ms) : 199, 211
section Bailout
This PR (8211) - mean (196ms) : 193, 199
master - mean (211ms) : 205, 217
section CallTarget+Inlining+NGEN
This PR (8211) - mean (1,160ms) : 1097, 1222
master - mean (1,204ms) : 1147, 1260
HttpMessageHandler (.NET Core 3.1)gantt
title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8211) - mean (277ms) : 270, 283
master - mean (297ms) : 287, 307
section Bailout
This PR (8211) - mean (277ms) : 271, 282
master - mean (297ms) : 286, 308
section CallTarget+Inlining+NGEN
This PR (8211) - mean (942ms) : 916, 968
master - mean (987ms) : 960, 1015
HttpMessageHandler (.NET 6)gantt
title Execution time (ms) HttpMessageHandler (.NET 6)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8211) - mean (270ms) : 264, 276
master - mean (290ms) : 281, 299
section Bailout
This PR (8211) - mean (270ms) : 267, 273
master - mean (289ms) : 281, 296
section CallTarget+Inlining+NGEN
This PR (8211) - mean (969ms) : 946, 992
master - mean (1,012ms) : 973, 1051
HttpMessageHandler (.NET 8)gantt
title Execution time (ms) HttpMessageHandler (.NET 8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8211) - mean (269ms) : 263, 275
master - mean (289ms) : 278, 299
section Bailout
This PR (8211) - mean (268ms) : 264, 272
master - mean (289ms) : 281, 296
section CallTarget+Inlining+NGEN
This PR (8211) - mean (854ms) : 823, 886
master - mean (915ms) : 823, 1007
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e729239afd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e729239afd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…m actually sending traces
…all ISpanBufferSerializer/SpanBufferMessagePackSerializer class to enable the future OTLP serialization
…e SpanBuffer serialization.
…Buffer. This does not yet handle issuing the outbound HTTP requests through the Api classes, nor does it handle configuration to enable the feature"
…udes: - Making the explicit histogram bounds configurable - Make property setters available internally so we can construct the final snapshots from our DD Sketches (rather than building up the snapshots over time and reading them at export time)
…ay we must distinguish the histogram timeseries when submitting trace stats using OTLP metrics.
…g APM trace stats) configurations to dd-trace-dotnet and read them in ExporterSettings
…submission, coming in a following commit
…and traces via OTLP traces, and plug them in to TracerManagerFactory.GetAgentWriter so we can use the new feature end-to-end
…itsOtlpTraces. Currently only http/json for Traces and http/protobuf for Metrics is tested.
| using var jsonWriter = new JsonTextWriter(streamWriter) | ||
| { | ||
| CloseOutput = false | ||
| }; |
There was a problem hiding this comment.
Just adding a Note for later: Found this one thanks to the AI help TBH, not blocking:
This creates a new MemoryStream + StreamWriter + JsonTextWriter per trace chunk on the serialization thread. The MessagePack path avoids this by writing directly into the reused _temporaryBuffer. Since the serializer is only called from one thread at a time, these could be instance fields that get reset between calls instead of recreated.
I Was curious to see what I did before and noticed I do the same pattern on OtlpMetricsSerializer and also is done in Utf8CountingPooledTextWriter, so I thinking of creating a follow up to clean up there and update it here as well later on.
There was a problem hiding this comment.
Good insight here. This sounds like an improvement we can make for a future PR 👍🏼
bouwkast
left a comment
There was a problem hiding this comment.
👍
Just a couple of questions throughout, but looks good!
I did get a bit confused with the OtelMapper GetBoundaries/GetCounts "bin" and gap comments as don't think I've done much with that ever
- Remove unused code - Refactor a small amount of JSON OTLP serialization code - Add comments to describe deviations in OTLP behavior from our previous Datadog trace export behaviors
… that only http/json is supported for OTLP trace export. Any other value falls back to Datadog encoding and emits a warning in the startup log indicating as such
… between offsets parameters
…ose metrics at this time. They'll be introduced alongside new unit tests in a separate PR
…tion tests, since we rely on updated behavior for testing OTLP traces
… the OTLP default host value of 'localhost'. Update OpenTelemetrySdkTests to assert this behavior (instead of just relying on system-tests for testing this behavior)
0bdf3bd to
70c4df0
Compare
…preventing the following resource-level attributes from being set as OTLP span attributes: - telemetry.sdk.name - telemetry.sdk.language - telemetry.sdk.version
…g when we're specifically testing the OTel SDK
…Traces to reduce test flakiness. We can run and regenerate OTEL SDK snapshots as needed
…actually needed for traces to picked up by the Datadog SDK for the OTel Tracing API
…he OpenTelemetrySdkTests sends traces to the wrong test-agent port
…rt (#8645) ## Summary of changes Adds `http/protobuf` support to the OTLP traces exporter introduced in #8211, which introduced the exporter and `http/json` support. This feature is enabled by setting `OTEL_TRACES_EXPORTER=otlp` and `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf` (or the parent `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf`). ## Reason for change When #8211 landed, `http/json` was the only functional OTLP traces protocol — setting any other protocol would fall back (with a log message) to the Datadog MessagePack encoding. `http/protobuf` is the OTLP default protocol per the OpenTelemetry specification and is what most OTel collectors, Datadog Agent included, expect to receive, so we need to support it to be useful out-of-the-box for users adopting the OTLP traces export. ## Implementation details ### Serializer Adds `Datadog.Trace.OpenTelemetry.Traces.OtlpTracesProtobufSerializer`, a new `ISpanBufferSerializer` that emits the [OTLP `ExportTraceServiceRequest`](https://github.com/open-telemetry/opentelemetry-proto/blob/v1.2.0/opentelemetry/proto/trace/v1/trace.proto) protobuf payload directly into `SpanBuffer`'s output buffer: - The `resource_spans` and `scope_spans` envelopes are opened on the first `SerializeSpans` call, span entries are appended per chunk, and `FinishBody` patches the two length-prefix placeholders using absolute positions into the destination buffer. - The serializer respects the same per-span / per-event / per-link attribute count limits (`128`) as `OtlpTracesJsonSerializer`, and re-uses the `OtlpMapper` logic to map resource attributes and tags onto the serialized spans. - `OtlpTracesProtobufSerializer` is **stateful** (it tracks the two length-placeholder positions), so `AgentWriter` now creates a fresh instance for the front and back buffers rather than sharing one — the previous shared `OtlpTracesJsonSerializer` was stateless and didn't need this. ### Vendored OpenTelemetry SDK Builds on the protobuf primitives vendored from the OpenTelemetry .NET SDK in #7653 (`ProtobufSerializer`, `ProtobufWireType`): - Adds new field-number constant files: `ProtobufOtlpTraceFieldNumberConstants` and `ProtobufOtlpResourceFieldNumberConstants` (split out so the OTLP trace serializer only depends on what it needs). - Removes the `#if NETCOREAPP3_1_OR_GREATER` guard from the vendored `ProtobufSerializer` — now that we vendor a `Span<>` polyfill the serializer compiles and runs on all supported runtimes (.NET Framework 4.6.1+). - Drops the matching `#if NET6_0_OR_GREATER` guard around the `SubmitsOtlpTraces` integration test (`http/protobuf` and `http/json` both run on .NET Framework now too). ### Wiring - `ExporterSettings`: `OtlpProtocol.HttpProtobuf` now resolves to `TracesEncoding.OtlpProtobuf` instead of falling back to `DatadogV0_4`. - `AgentWriter.GetSpanSerializer`: selects `OtlpTracesProtobufSerializer` when `TracesEncoding == OtlpProtobuf`. - `ApiOtlp.SendTracesAsync`: chooses `application/x-protobuf` vs `application/json` for the request `Content-Type` based on `_tracesEncoding`. Adds the `application/x-protobuf` MIME type constant to `Datadog.Trace.Agent.Transports.MimeTypes`. - `supported-configurations.yaml`: documents `http/protobuf` as a valid value for `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL` and regenerates `ConfigurationKeys.OpenTelemetry.g.cs` for all TFMs. ## Test coverage - **Unit tests**: New `OtlpTracesProtobufSerializerTests` exercises the serializer end-to-end using `Google.Protobuf` and a little custom parsing. Test cases include: - Empty buffer (`FinishBody` returns 0) - Multiple traces reported in a single-chunk - Serializing span events/links/status - Exceeding buffer `maxSize` honoring behavior - Exceeding span limits - **Integration tests**: `OpenTelemetrySdkTests.SubmitsOtlpTraces` now runs both `http/json` and `http/protobuf` cases for the Datadog SDK on all TFMs. - **Snapshot unification**: To keep one Datadog-SDK snapshot regardless of OTLP protocol, snapshots for `http/protobuf` are normalized to the `http/json` shape via scrubbers that translate the test agent's protobuf rendering (snake_case field names, string-form enum values like `SPAN_KIND_SERVER`) into the JSON serializer's shape (camelCase field names, integer enum values). - **Snapshot rename**: `SubmitsOtlpTraces_DD_http_json.verified.txt` renamed to `SubmitsOtlpTraces_DD.verified.txt` (one snapshot now covers both protocols). ## Other details With this PR, `http/json` and `http/protobuf` are both functional; `grpc` still falls back to Datadog MessagePack encoding with a startup warning. ### Follow-up work - Implement the `grpc` OTLP traces protocol - Add OTLP traces telemetry counts (the `TODO: Telemetry - Record OTLP Traces API submissions` in `ApiOtlp.SendTracesAsync` still applies) - Honor `OTEL_EXPORTER_OTLP_TRACES_TIMEOUT` in `ApiOtlp` (still unused) --------- Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
Summary of changes
Adds experimental support for exporting traces using OTLP rather than the Datadog MessagePack protocols. This allows the DD SDK to send traces to an OTel collector rather than a Datadog Trace Agent, with limited support for non-APM products.
This feature is enabled by setting
OTEL_TRACES_EXPORTER=otlp.Note: This feature is currently only supported for .NET 6+, and only the
http/jsonOTLP protocol is supported at this time. Setting any other protocol value falls back to Datadog encoding with a startup warning.Configuration
OTEL_TRACES_EXPORTER=otlpOTEL_EXPORTER_OTLP_TRACES_ENDPOINTOTEL_EXPORTER_OTLP_TRACES_HEADERSOTEL_EXPORTER_OTLP_TRACES_PROTOCOLOTEL_EXPORTER_OTLP_TRACES_TIMEOUTReason for change
We are seeing an increasing number of scenarios where users have applications instrumented with the OTel SDK sending data to OTel collectors, and they would like to get additional features offered by the DD SDK without needing to update their OTel collector deployments. Although there will be follow-up work, this provides the ability for users to write vendor-neutral API instrumentation and emit vendor-neutral telemetry data so DD SDK users don't have to feel locked in when setting up Datadog APM.
Implementation details
Configuration
All OTLP exporter configuration (Traces and Metrics) is read in
ExporterSettingsand exposed as properties (OtlpTracesEndpoint,OtlpTracesProtocol,OtlpTracesHeaders,OtlpTracesTimeoutMs, and the correspondingOtlpMetrics*properties). TheTracesEncodingproperty (of typeDatadog.Trace.Agent.TracesEncoding) determines the serialization format:DatadogV0_4,OtlpProtobuf, orOtlpJson. OnlyOtlpJsonis currently functional;OtlpProtobufis defined but not yet implemented.OTLP endpoint resolution is handled separately from Datadog trace transport settings to avoid comingling the two URL calculation paths. The OTLP endpoint logic considers
DD_AGENT_HOSTbefore falling back to the OTLP default host oflocalhost.Serialization
The
SpanBufferclass was refactored to use a pluggableISpanBufferSerializerinterface (methods:HeaderSize,SerializeSpans,WriteHeader,FinishBody). Two implementations exist:SpanBufferMessagePackSerializer: The existing Datadog MessagePack serialization, extracted into its own class.OtlpTracesJsonSerializer: SerializesTraceChunkModelandSpanobjects into the OTLPExportTraceServiceRequestJSON structure. Key behaviors:OtlpMapper.EmitResourceAttributesFromTraceChunk()Datadog-to-OTLP Mapping
The
OtlpMapperstatic class (underOpenTelemetry/) handles the translation between Datadog span semantics and OTLP attributes. It determines which tags are resource-level vs. span-level, manages attribute limits, and processes string/double/byte[] tag types via an innerTagWriterstruct.Sampling & Stats Aggregation
When OTLP export is enabled:
PrioritySampleris used inShouldKeepTrace(), aligning with the OpenTelemetry SDK behavior of exporting based solely on the sampling decision.AgentSamplingRuleis omitted since there is no Datadog Agent to communicate sampling rules.StatsAggregationKey: Extended withIsErrorandIsTopLevelfields for OTLP mode, allowing distinct histogram timeseries. The Datadog mode constructor continues to set these to false for backwards compatibility.Export
ApiOtlp(NET6_0_OR_GREATER only): ImplementsIApifor OTLP endpoints with retry logic (up to 5 attempts with exponential backoff starting at 100ms).ManagedApiOtlp: Thread-safe wrapper that enables atomic swapping ofApiOtlpinstances (for configuration refresh).OtlpTracesJsonSerializerintoSpanBuffer, then sent byApiOtlp.SendTracesAsync()to the OTLP traces endpoint with custom headers.SendStatsAsyncis defined but currently returns success immediately — DDSketch-to-OTLP histogram conversion was removed from this PR and will be introduced in a follow-up PR alongside new unit tests.Wiring
In
TracerManagerFactory.GetAgentWriter(), whenTracesEncodingisOtlpProtobuforOtlpJson, the method creates aManagedApiOtlpand aStatsAggregatorwithisOtlp: true, then passes both toAgentWriter.Test coverage
OpenTelemetrySdkTests.SubmitsOtlpTraces— A parameterized test exercising different package versions, protocols, and configurations. Sends OTLP traces to the dd-apm-test-agent, retrieves payloads, validates trace/span ID formats (32-char/16-char hex), checks resource attribute consistency across requests, normalizes dynamic values, and performs snapshot testing.Follow-up work
SendStatsAsynccurrently no-ops)SubmitsOtlpTracesintegration testTracerSettingsnow that they are being read inExporterSettingshttp/protobufOTLP protocol support (enum value exists but not yet functional)