Skip to content

[Tracing] Add experimental support for exporting traces as OTLP (.NET 6+)#8211

Merged
zacharycmontoya merged 50 commits into
masterfrom
zach.montoya/otlp-export-refined
Mar 24, 2026
Merged

[Tracing] Add experimental support for exporting traces as OTLP (.NET 6+)#8211
zacharycmontoya merged 50 commits into
masterfrom
zach.montoya/otlp-export-refined

Conversation

@zacharycmontoya

@zacharycmontoya zacharycmontoya commented Feb 17, 2026

Copy link
Copy Markdown
Contributor

Summary of changes

Adds experimental support for exporting traces using OTLP rather than the Datadog MessagePack protocols. This allows the DD SDK to send traces to an OTel collector rather than a Datadog Trace Agent, with limited support for non-APM products.

This feature is enabled by setting OTEL_TRACES_EXPORTER=otlp.

Note: This feature is currently only supported for .NET 6+, and only the http/json OTLP protocol is supported at this time. Setting any other protocol value falls back to Datadog encoding with a startup warning.

Configuration

Configuration Details
OTEL_TRACES_EXPORTER=otlp Enables the OTLP traces export
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT See the OTLP Exporter Configuration docs
OTEL_EXPORTER_OTLP_TRACES_HEADERS See the OTLP Exporter Configuration docs
OTEL_EXPORTER_OTLP_TRACES_PROTOCOL See the OTLP Exporter Configuration docs
OTEL_EXPORTER_OTLP_TRACES_TIMEOUT See the OTLP Exporter Configuration docs

Reason for change

We are seeing an increasing number of scenarios where users have applications instrumented with the OTel SDK sending data to OTel collectors, and they would like to get additional features offered by the DD SDK without needing to update their OTel collector deployments. Although there will be follow-up work, this provides the ability for users to write vendor-neutral API instrumentation and emit vendor-neutral telemetry data so DD SDK users don't have to feel locked in when setting up Datadog APM.

Implementation details

Configuration

All OTLP exporter configuration (Traces and Metrics) is read in ExporterSettings and exposed as properties (OtlpTracesEndpoint, OtlpTracesProtocol, OtlpTracesHeaders, OtlpTracesTimeoutMs, and the corresponding OtlpMetrics* properties). The TracesEncoding property (of type Datadog.Trace.Agent.TracesEncoding) determines the serialization format: DatadogV0_4, OtlpProtobuf, or OtlpJson. Only OtlpJson is currently functional; OtlpProtobuf is defined but not yet implemented.

OTLP endpoint resolution is handled separately from Datadog trace transport settings to avoid comingling the two URL calculation paths. The OTLP endpoint logic considers DD_AGENT_HOST before falling back to the OTLP default host of localhost.

Serialization

The SpanBuffer class was refactored to use a pluggable ISpanBufferSerializer interface (methods: HeaderSize, SerializeSpans, WriteHeader, FinishBody). Two implementations exist:

  • SpanBufferMessagePackSerializer: The existing Datadog MessagePack serialization, extracted into its own class.
  • OtlpTracesJsonSerializer: Serializes TraceChunkModel and Span objects into the OTLP ExportTraceServiceRequest JSON structure. Key behaviors:
    • Resource attributes (service.name, service.version, deployment.environment.name, telemetry SDK info, git metadata, runtime ID) are emitted via OtlpMapper.EmitResourceAttributesFromTraceChunk()
    • Span attributes are capped at 128 per span
    • Span events (128 limit), span links (128 limit), span status, and span kind are all mapped
    • Field names use lowerCamelCase and enums use integer values per the OTLP http/json encoding spec

Datadog-to-OTLP Mapping

The OtlpMapper static class (under OpenTelemetry/) handles the translation between Datadog span semantics and OTLP attributes. It determines which tags are resource-level vs. span-level, manages attribute limits, and processes string/double/byte[] tag types via an inner TagWriter struct.

Sampling & Stats Aggregation

When OTLP export is enabled:

  • Sampling: Only the PrioritySampler is used in ShouldKeepTrace(), aligning with the OpenTelemetry SDK behavior of exporting based solely on the sampling decision.
  • Default sampling rate: A global sampling rate of 1.0 is set by default (can be overridden by user configuration), and the AgentSamplingRule is omitted since there is no Datadog Agent to communicate sampling rules.
  • StatsAggregationKey: Extended with IsError and IsTopLevel fields for OTLP mode, allowing distinct histogram timeseries. The Datadog mode constructor continues to set these to false for backwards compatibility.

Export

  • ApiOtlp (NET6_0_OR_GREATER only): Implements IApi for OTLP endpoints with retry logic (up to 5 attempts with exponential backoff starting at 100ms).
  • ManagedApiOtlp: Thread-safe wrapper that enables atomic swapping of ApiOtlp instances (for configuration refresh).
  • Traces: Serialized by OtlpTracesJsonSerializer into SpanBuffer, then sent by ApiOtlp.SendTracesAsync() to the OTLP traces endpoint with custom headers.
  • Metrics (trace stats): SendStatsAsync is defined but currently returns success immediately — DDSketch-to-OTLP histogram conversion was removed from this PR and will be introduced in a follow-up PR alongside new unit tests.

Wiring

In TracerManagerFactory.GetAgentWriter(), when TracesEncoding is OtlpProtobuf or OtlpJson, the method creates a ManagedApiOtlp and a StatsAggregator with isOtlp: true, then passes both to AgentWriter.

Test coverage

  • Integration test: OpenTelemetrySdkTests.SubmitsOtlpTraces — A parameterized test exercising different package versions, protocols, and configurations. Sends OTLP traces to the dd-apm-test-agent, retrieves payloads, validates trace/span ID formats (32-char/16-char hex), checks resource attribute consistency across requests, normalizes dynamic values, and performs snapshot testing.
  • Unit tests: Tests for OTLP-specific behaviors including: stats aggregator enablement when exporting OTLP, exporting only sampled spans, and omission of the AgentSamplingRule.
  • Docker: The docker-compose configuration was updated to wait for test-agent ports 4317 (gRPC) and 4318 (HTTP) for OTLP traffic.

Follow-up work

  • Export trace metrics (APM stats) via OTLP metrics with DDSketch-to-OTLP histogram conversion (removed from this PR — SendStatsAsync currently no-ops)
  • Assert against trace metrics in the SubmitsOtlpTraces integration test
  • Stop reading OTLP metrics settings in TracerSettings now that they are being read in ExporterSettings
  • Implement http/protobuf OTLP protocol support (enum value exists but not yet functional)

@zacharycmontoya
zacharycmontoya force-pushed the zach.montoya/otlp-export-refined branch 4 times, most recently from 54892c0 to 771b40f Compare February 18, 2026 05:27
@pr-commenter

pr-commenter Bot commented Feb 18, 2026

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-03-24 04:06:06

Comparing candidate commit 58fd979 in PR branch zach.montoya/otlp-export-refined with baseline commit 75d765c in branch master.

Found 11 performance improvements and 15 performance regressions! Performance is the same for 245 metrics, 17 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleMoreComplexBody net6.0

  • 🟩 execution_time [-22.980ms; -16.947ms] or [-10.633%; -7.841%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorMoreComplexBody net6.0

  • 🟥 execution_time [+9.969ms; +15.247ms] or [+5.052%; +7.727%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorMoreComplexBody netcoreapp3.1

  • 🟩 execution_time [-15.375ms; -11.008ms] or [-7.258%; -5.197%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody netcoreapp3.1

  • 🟩 execution_time [-19.583ms; -13.500ms] or [-9.113%; -6.282%]

scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeLegacyArgs net6.0

  • 🟥 throughput [-399.037op/s; -380.627op/s] or [-5.647%; -5.386%]

scenario:Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces net472

  • 🟥 execution_time [+20.947ms; +25.174ms] or [+10.430%; +12.535%]

scenario:Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces net6.0

  • 🟥 execution_time [+41.687ms; +45.839ms] or [+23.234%; +25.548%]

scenario:Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces netcoreapp3.1

  • 🟥 execution_time [+31.250ms; +36.692ms] or [+21.125%; +24.804%]
  • 🟥 throughput [-147.267op/s; -104.600op/s] or [-9.077%; -6.447%]

scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSlice net6.0

  • 🟩 execution_time [-273.314µs; -265.566µs] or [-16.431%; -15.965%]
  • 🟩 throughput [+114.474op/s; +117.941op/s] or [+19.041%; +19.618%]

scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSliceWithPool net6.0

  • 🟥 execution_time [+137.857µs; +142.277µs] or [+13.595%; +14.031%]
  • 🟥 throughput [-121.592op/s; -117.779op/s] or [-12.330%; -11.943%]

scenario:Benchmarks.Trace.CharSliceBenchmark.OriginalCharSlice net6.0

  • 🟩 execution_time [-165.469µs; -111.291µs] or [-8.076%; -5.432%]
  • 🟩 throughput [+29.102op/s; +41.051op/s] or [+5.959%; +8.406%]

scenario:Benchmarks.Trace.DbCommandBenchmark.ExecuteNonQuery net472

  • 🟩 throughput [+26693.446op/s; +28137.996op/s] or [+7.971%; +8.402%]

scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearch net6.0

  • 🟥 execution_time [+10.548ms; +10.813ms] or [+5.272%; +5.405%]
  • 🟥 throughput [-53135.132op/s; -52309.245op/s] or [-7.962%; -7.838%]

scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearchAsync net6.0

  • 🟥 execution_time [+13.519ms; +16.653ms] or [+6.913%; +8.516%]
  • 🟥 throughput [-52088.656op/s; -41963.635op/s] or [-8.211%; -6.615%]

scenario:Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync netcoreapp3.1

  • 🟩 throughput [+22789.512op/s; +31509.424op/s] or [+5.630%; +7.785%]

scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark net6.0

  • 🟥 allocated_mem [+19.336KB; +19.373KB] or [+7.518%; +7.533%]

scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark netcoreapp3.1

  • 🟥 allocated_mem [+13.501KB; +13.523KB] or [+5.234%; +5.243%]
  • 🟥 throughput [-424.635op/s; -238.193op/s] or [-19.551%; -10.967%]

scenario:Benchmarks.Trace.SingleSpanAspNetCoreBenchmark.SingleSpanAspNetCore netcoreapp3.1

  • 🟩 throughput [+15124694.928op/s; +16071604.124op/s] or [+6.688%; +7.106%]

scenario:Benchmarks.Trace.SpanBenchmark.StartFinishSpan net6.0

  • 🟩 execution_time [-17.302ms; -11.810ms] or [-8.160%; -5.570%]

@zacharycmontoya
zacharycmontoya force-pushed the zach.montoya/otlp-export-refined branch from 771b40f to e729239 Compare February 18, 2026 20:50
@dd-trace-dotnet-ci-bot

dd-trace-dotnet-ci-bot Bot commented Feb 18, 2026

Copy link
Copy Markdown

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing This PR (8211) and master.

✅ No regressions detected - check the details below

Full Metrics Comparison

FakeDbCommand

Metric Master (Mean ± 95% CI) Current (Mean ± 95% CI) Change Status
.NET Framework 4.8 - Baseline
duration68.44 ± (68.43 - 68.71) ms68.57 ± (68.62 - 68.89) ms+0.2%✅⬆️
.NET Framework 4.8 - Bailout
duration72.45 ± (72.42 - 72.71) ms72.34 ± (72.18 - 72.47) ms-0.1%
.NET Framework 4.8 - CallTarget+Inlining+NGEN
duration1039.69 ± (1045.57 - 1055.62) ms1048.17 ± (1050.04 - 1057.89) ms+0.8%✅⬆️
.NET Core 3.1 - Baseline
process.internal_duration_ms21.92 ± (21.89 - 21.95) ms21.84 ± (21.81 - 21.88) ms-0.4%
process.time_to_main_ms79.41 ± (79.25 - 79.56) ms79.72 ± (79.56 - 79.88) ms+0.4%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%
runtime.dotnet.mem.committed10.90 ± (10.90 - 10.90) MB10.91 ± (10.90 - 10.91) MB+0.1%✅⬆️
runtime.dotnet.threads.count12 ± (12 - 12)12 ± (12 - 12)+0.0%
.NET Core 3.1 - Bailout
process.internal_duration_ms21.80 ± (21.78 - 21.83) ms21.71 ± (21.68 - 21.73) ms-0.4%
process.time_to_main_ms80.52 ± (80.38 - 80.67) ms80.81 ± (80.65 - 80.97) ms+0.4%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%
runtime.dotnet.mem.committed10.94 ± (10.93 - 10.94) MB10.95 ± (10.95 - 10.96) MB+0.1%✅⬆️
runtime.dotnet.threads.count13 ± (13 - 13)13 ± (13 - 13)+0.0%
.NET Core 3.1 - CallTarget+Inlining+NGEN
process.internal_duration_ms216.64 ± (215.61 - 217.66) ms219.68 ± (218.33 - 221.04) ms+1.4%✅⬆️
process.time_to_main_ms513.92 ± (512.70 - 515.14) ms517.87 ± (516.90 - 518.84) ms+0.8%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%
runtime.dotnet.mem.committed48.17 ± (48.14 - 48.20) MB48.27 ± (48.24 - 48.30) MB+0.2%✅⬆️
runtime.dotnet.threads.count28 ± (28 - 28)28 ± (28 - 28)+0.2%✅⬆️
.NET 6 - Baseline
process.internal_duration_ms20.58 ± (20.56 - 20.60) ms20.56 ± (20.54 - 20.59) ms-0.1%
process.time_to_main_ms68.79 ± (68.66 - 68.93) ms69.01 ± (68.89 - 69.13) ms+0.3%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%
runtime.dotnet.mem.committed10.62 ± (10.61 - 10.62) MB10.64 ± (10.64 - 10.65) MB+0.3%✅⬆️
runtime.dotnet.threads.count10 ± (10 - 10)10 ± (10 - 10)+0.0%
.NET 6 - Bailout
process.internal_duration_ms20.46 ± (20.43 - 20.48) ms20.50 ± (20.47 - 20.52) ms+0.2%✅⬆️
process.time_to_main_ms69.66 ± (69.55 - 69.77) ms69.96 ± (69.86 - 70.07) ms+0.4%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%
runtime.dotnet.mem.committed10.74 ± (10.73 - 10.74) MB10.76 ± (10.75 - 10.76) MB+0.2%✅⬆️
runtime.dotnet.threads.count11 ± (11 - 11)11 ± (11 - 11)+0.0%
.NET 6 - CallTarget+Inlining+NGEN
process.internal_duration_ms206.28 ± (205.05 - 207.50) ms207.25 ± (205.85 - 208.64) ms+0.5%✅⬆️
process.time_to_main_ms514.89 ± (513.71 - 516.07) ms520.19 ± (519.05 - 521.34) ms+1.0%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%
runtime.dotnet.mem.committed49.99 ± (49.95 - 50.04) MB50.09 ± (50.04 - 50.13) MB+0.2%✅⬆️
runtime.dotnet.threads.count29 ± (29 - 29)29 ± (29 - 29)+0.1%✅⬆️
.NET 8 - Baseline
process.internal_duration_ms18.80 ± (18.77 - 18.82) ms18.78 ± (18.75 - 18.80) ms-0.1%
process.time_to_main_ms68.26 ± (68.14 - 68.38) ms67.97 ± (67.84 - 68.09) ms-0.4%
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%
runtime.dotnet.mem.committed7.66 ± (7.65 - 7.67) MB7.69 ± (7.69 - 7.70) MB+0.4%✅⬆️
runtime.dotnet.threads.count10 ± (10 - 10)10 ± (10 - 10)+0.0%
.NET 8 - Bailout
process.internal_duration_ms18.82 ± (18.79 - 18.84) ms18.89 ± (18.86 - 18.91) ms+0.4%✅⬆️
process.time_to_main_ms69.48 ± (69.33 - 69.62) ms69.24 ± (69.13 - 69.34) ms-0.3%
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%
runtime.dotnet.mem.committed7.74 ± (7.73 - 7.75) MB7.74 ± (7.73 - 7.74) MB-0.1%
runtime.dotnet.threads.count11 ± (11 - 11)11 ± (11 - 11)+0.0%
.NET 8 - CallTarget+Inlining+NGEN
process.internal_duration_ms152.80 ± (152.05 - 153.55) ms153.33 ± (152.43 - 154.23) ms+0.3%✅⬆️
process.time_to_main_ms474.97 ± (474.06 - 475.88) ms475.88 ± (475.01 - 476.75) ms+0.2%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%
runtime.dotnet.mem.committed36.94 ± (36.92 - 36.96) MB36.87 ± (36.85 - 36.90) MB-0.2%
runtime.dotnet.threads.count28 ± (28 - 28)28 ± (28 - 28)-0.0%

HttpMessageHandler

Metric Master (Mean ± 95% CI) Current (Mean ± 95% CI) Change Status
.NET Framework 4.8 - Baseline
duration204.71 ± (204.74 - 205.70) ms192.03 ± (191.94 - 192.72) ms-6.2%
.NET Framework 4.8 - Bailout
duration210.61 ± (210.43 - 211.35) ms196.12 ± (195.93 - 196.63) ms-6.9%
.NET Framework 4.8 - CallTarget+Inlining+NGEN
duration1197.10 ± (1199.78 - 1207.77) ms1155.35 ± (1155.70 - 1163.97) ms-3.5%
.NET Core 3.1 - Baseline
process.internal_duration_ms200.85 ± (200.38 - 201.33) ms187.20 ± (186.80 - 187.61) ms-6.8%
process.time_to_main_ms86.36 ± (86.10 - 86.62) ms80.78 ± (80.58 - 80.98) ms-6.5%
runtime.dotnet.exceptions.count3 ± (3 - 3)3 ± (3 - 3)+0.0%
runtime.dotnet.mem.committed15.97 ± (15.96 - 15.99) MB16.12 ± (16.09 - 16.14) MB+0.9%✅⬆️
runtime.dotnet.threads.count20 ± (19 - 20)20 ± (19 - 20)+0.0%✅⬆️
.NET Core 3.1 - Bailout
process.internal_duration_ms199.77 ± (199.26 - 200.27) ms186.65 ± (186.22 - 187.08) ms-6.6%
process.time_to_main_ms87.88 ± (87.67 - 88.10) ms81.96 ± (81.82 - 82.11) ms-6.7%
runtime.dotnet.exceptions.count3 ± (3 - 3)3 ± (3 - 3)+0.0%
runtime.dotnet.mem.committed15.95 ± (15.93 - 15.97) MB16.16 ± (16.13 - 16.18) MB+1.3%✅⬆️
runtime.dotnet.threads.count21 ± (21 - 21)21 ± (20 - 21)-1.4%
.NET Core 3.1 - CallTarget+Inlining+NGEN
process.internal_duration_ms407.85 ± (406.51 - 409.19) ms391.87 ± (390.49 - 393.25) ms-3.9%
process.time_to_main_ms542.90 ± (541.53 - 544.27) ms519.72 ± (518.67 - 520.77) ms-4.3%
runtime.dotnet.exceptions.count3 ± (3 - 3)3 ± (3 - 3)+0.0%
runtime.dotnet.mem.committed58.98 ± (58.93 - 59.03) MB58.67 ± (58.49 - 58.84) MB-0.5%
runtime.dotnet.threads.count30 ± (30 - 30)30 ± (30 - 30)+0.5%✅⬆️
.NET 6 - Baseline
process.internal_duration_ms206.35 ± (205.87 - 206.82) ms191.63 ± (191.29 - 191.97) ms-7.1%
process.time_to_main_ms75.03 ± (74.82 - 75.23) ms69.98 ± (69.82 - 70.14) ms-6.7%
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%
runtime.dotnet.mem.committed16.23 ± (16.21 - 16.25) MB16.26 ± (16.14 - 16.38) MB+0.2%✅⬆️
runtime.dotnet.threads.count19 ± (19 - 19)18 ± (18 - 19)-5.5%
.NET 6 - Bailout
process.internal_duration_ms204.91 ± (204.42 - 205.41) ms190.83 ± (190.58 - 191.08) ms-6.9%
process.time_to_main_ms75.68 ± (75.50 - 75.86) ms71.11 ± (71.00 - 71.22) ms-6.0%
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%
runtime.dotnet.mem.committed16.25 ± (16.23 - 16.28) MB16.29 ± (16.15 - 16.42) MB+0.2%✅⬆️
runtime.dotnet.threads.count20 ± (20 - 20)19 ± (19 - 19)-5.1%
.NET 6 - CallTarget+Inlining+NGEN
process.internal_duration_ms430.59 ± (429.34 - 431.84) ms412.96 ± (411.65 - 414.27) ms-4.1%
process.time_to_main_ms549.03 ± (547.63 - 550.43) ms526.32 ± (524.81 - 527.84) ms-4.1%
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%
runtime.dotnet.mem.committed60.85 ± (60.81 - 60.88) MB61.01 ± (60.95 - 61.06) MB+0.3%✅⬆️
runtime.dotnet.threads.count31 ± (31 - 31)30 ± (30 - 31)-0.8%
.NET 8 - Baseline
process.internal_duration_ms203.54 ± (203.01 - 204.06) ms189.86 ± (189.45 - 190.26) ms-6.7%
process.time_to_main_ms74.24 ± (74.03 - 74.44) ms69.90 ± (69.70 - 70.10) ms-5.8%
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%
runtime.dotnet.mem.committed11.61 ± (11.59 - 11.62) MB11.75 ± (11.73 - 11.78) MB+1.2%✅⬆️
runtime.dotnet.threads.count19 ± (19 - 19)18 ± (18 - 18)-2.9%
.NET 8 - Bailout
process.internal_duration_ms202.85 ± (202.33 - 203.37) ms188.27 ± (187.97 - 188.57) ms-7.2%
process.time_to_main_ms75.73 ± (75.53 - 75.94) ms70.66 ± (70.53 - 70.79) ms-6.7%
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%
runtime.dotnet.mem.committed11.69 ± (11.67 - 11.71) MB11.83 ± (11.80 - 11.85) MB+1.2%✅⬆️
runtime.dotnet.threads.count20 ± (20 - 20)19 ± (19 - 19)-2.5%
.NET 8 - CallTarget+Inlining+NGEN
process.internal_duration_ms373.86 ± (367.70 - 380.03) ms341.85 ± (340.61 - 343.10) ms-8.6%
process.time_to_main_ms506.84 ± (505.76 - 507.92) ms483.57 ± (482.61 - 484.52) ms-4.6%
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%
runtime.dotnet.mem.committed49.20 ± (49.07 - 49.34) MB48.85 ± (48.82 - 48.88) MB-0.7%
runtime.dotnet.threads.count30 ± (30 - 30)30 ± (30 - 30)-0.5%
Comparison explanation

Execution-time benchmarks measure the whole time it takes to execute a program, and are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are highlighted in **red**. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

Duration charts
FakeDbCommand (.NET Framework 4.8)
gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8211) - mean (69ms)  : 67, 71
    master - mean (69ms)  : 67, 71

    section Bailout
    This PR (8211) - mean (72ms)  : 71, 74
    master - mean (73ms)  : 71, 74

    section CallTarget+Inlining+NGEN
    This PR (8211) - mean (1,054ms)  : 998, 1110
    master - mean (1,051ms)  : 978, 1123

Loading
FakeDbCommand (.NET Core 3.1)
gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8211) - mean (107ms)  : 104, 110
    master - mean (107ms)  : 104, 110

    section Bailout
    This PR (8211) - mean (108ms)  : 106, 110
    master - mean (108ms)  : 105, 110

    section CallTarget+Inlining+NGEN
    This PR (8211) - mean (774ms)  : 755, 793
    master - mean (769ms)  : 753, 785

Loading
FakeDbCommand (.NET 6)
gantt
    title Execution time (ms) FakeDbCommand (.NET 6)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8211) - mean (95ms)  : 92, 97
    master - mean (95ms)  : 92, 97

    section Bailout
    This PR (8211) - mean (96ms)  : 94, 97
    master - mean (95ms)  : 94, 97

    section CallTarget+Inlining+NGEN
    This PR (8211) - mean (762ms)  : 743, 781
    master - mean (753ms)  : 730, 776

Loading
FakeDbCommand (.NET 8)
gantt
    title Execution time (ms) FakeDbCommand (.NET 8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8211) - mean (93ms)  : 91, 96
    master - mean (94ms)  : 91, 97

    section Bailout
    This PR (8211) - mean (95ms)  : 93, 96
    master - mean (95ms)  : 93, 97

    section CallTarget+Inlining+NGEN
    This PR (8211) - mean (661ms)  : 643, 680
    master - mean (658ms)  : 641, 675

Loading
HttpMessageHandler (.NET Framework 4.8)
gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8211) - mean (192ms)  : 189, 196
    master - mean (205ms)  : 199, 211

    section Bailout
    This PR (8211) - mean (196ms)  : 193, 199
    master - mean (211ms)  : 205, 217

    section CallTarget+Inlining+NGEN
    This PR (8211) - mean (1,160ms)  : 1097, 1222
    master - mean (1,204ms)  : 1147, 1260

Loading
HttpMessageHandler (.NET Core 3.1)
gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8211) - mean (277ms)  : 270, 283
    master - mean (297ms)  : 287, 307

    section Bailout
    This PR (8211) - mean (277ms)  : 271, 282
    master - mean (297ms)  : 286, 308

    section CallTarget+Inlining+NGEN
    This PR (8211) - mean (942ms)  : 916, 968
    master - mean (987ms)  : 960, 1015

Loading
HttpMessageHandler (.NET 6)
gantt
    title Execution time (ms) HttpMessageHandler (.NET 6)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8211) - mean (270ms)  : 264, 276
    master - mean (290ms)  : 281, 299

    section Bailout
    This PR (8211) - mean (270ms)  : 267, 273
    master - mean (289ms)  : 281, 296

    section CallTarget+Inlining+NGEN
    This PR (8211) - mean (969ms)  : 946, 992
    master - mean (1,012ms)  : 973, 1051

Loading
HttpMessageHandler (.NET 8)
gantt
    title Execution time (ms) HttpMessageHandler (.NET 8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8211) - mean (269ms)  : 263, 275
    master - mean (289ms)  : 278, 299

    section Bailout
    This PR (8211) - mean (268ms)  : 264, 272
    master - mean (289ms)  : 281, 296

    section CallTarget+Inlining+NGEN
    This PR (8211) - mean (854ms)  : 823, 886
    master - mean (915ms)  : 823, 1007

Loading

@zacharycmontoya zacharycmontoya changed the title [Tracing] Add support for exporting traces as OTLP [Tracing] Add support for exporting traces as OTLP (.NET 6+) Feb 18, 2026
@zacharycmontoya zacharycmontoya changed the title [Tracing] Add support for exporting traces as OTLP (.NET 6+) [Tracing] Add experimental support for exporting traces as OTLP (.NET 6+) Feb 19, 2026
@zacharycmontoya
zacharycmontoya marked this pull request as ready for review February 19, 2026 01:13
@zacharycmontoya
zacharycmontoya requested review from a team as code owners February 19, 2026 01:13

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e729239afd

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread tracer/src/Datadog.Trace/Configuration/ExporterSettings.cs Outdated
Comment thread tracer/src/Datadog.Trace/Configuration/ExporterSettings.Shared.cs Outdated
Comment thread tracer/src/Datadog.Trace/Agent/AgentWriter.cs
Comment thread tracer/src/Datadog.Trace/Agent/ApiOtlp.cs

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e729239afd

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread tracer/src/Datadog.Trace/Configuration/ExporterSettings.cs Outdated
Comment thread tracer/src/Datadog.Trace/Configuration/ExporterSettings.Shared.cs Outdated
Comment thread tracer/src/Datadog.Trace/OpenTelemetry/Metrics/OtlpMetricsSerializer.cs Outdated
Comment thread tracer/src/Datadog.Trace/Configuration/supported-configurations-docs.yaml Outdated
…all ISpanBufferSerializer/SpanBufferMessagePackSerializer class to enable the future OTLP serialization
…Buffer. This does not yet handle issuing the outbound HTTP requests through the Api classes, nor does it handle configuration to enable the feature"
…udes:

- Making the explicit histogram bounds configurable
- Make property setters available internally so we can construct the final snapshots from our DD Sketches (rather than building up the snapshots over time and reading them at export time)
…ay we must distinguish the histogram timeseries when submitting trace stats using OTLP metrics.
…g APM trace stats) configurations to dd-trace-dotnet and read them in ExporterSettings
…and traces via OTLP traces, and plug them in to TracerManagerFactory.GetAgentWriter so we can use the new feature end-to-end
…itsOtlpTraces. Currently only http/json for Traces and http/protobuf for Metrics is tested.
Comment thread tracer/src/Datadog.Trace/Agent/ApiOtlp.cs Outdated
Comment thread tracer/src/Datadog.Trace/OpenTelemetry/Traces/OtlpTracesJsonSerializer.cs Outdated
Comment thread tracer/src/Datadog.Trace/Agent/SpanBufferMessagePackSerializer.cs Outdated
Comment thread tracer/src/Datadog.Trace/OpenTelemetry/Traces/OtlpTracesJsonSerializer.cs Outdated
using var jsonWriter = new JsonTextWriter(streamWriter)
{
CloseOutput = false
};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just adding a Note for later: Found this one thanks to the AI help TBH, not blocking:

This creates a new MemoryStream + StreamWriter + JsonTextWriter per trace chunk on the serialization thread. The MessagePack path avoids this by writing directly into the reused _temporaryBuffer. Since the serializer is only called from one thread at a time, these could be instance fields that get reset between calls instead of recreated.

I Was curious to see what I did before and noticed I do the same pattern on OtlpMetricsSerializer and also is done in Utf8CountingPooledTextWriter, so I thinking of creating a follow up to clean up there and update it here as well later on.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good insight here. This sounds like an improvement we can make for a future PR 👍🏼

@bouwkast bouwkast left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

Just a couple of questions throughout, but looks good!

I did get a bit confused with the OtelMapper GetBoundaries/GetCounts "bin" and gap comments as don't think I've done much with that ever

Comment thread tracer/src/Datadog.Trace/Agent/ApiOtlp.cs
Comment thread tracer/src/Datadog.Trace/Agent/ApiOtlp.cs Outdated
Comment thread tracer/src/Datadog.Trace/Configuration/ExporterSettings.cs
Comment thread tracer/src/Datadog.Trace/OpenTelemetry/Traces/OtlpTracesJsonSerializer.cs Outdated
Comment thread tracer/src/Datadog.Trace/OpenTelemetry/OtlpMapper.cs Outdated
Comment thread tracer/src/Datadog.Trace/OpenTelemetry/OtlpMapper.cs Outdated
- Remove unused code
- Refactor a small amount of JSON OTLP serialization code
- Add comments to describe deviations in OTLP behavior from our previous Datadog trace export behaviors
… that only http/json is supported for OTLP trace export. Any other value falls back to Datadog encoding and emits a warning in the startup log indicating as such
…ose metrics at this time. They'll be introduced alongside new unit tests in a separate PR
…tion tests, since we rely on updated behavior for testing OTLP traces
… the OTLP default host value of 'localhost'.

Update OpenTelemetrySdkTests to assert this behavior (instead of just relying on system-tests for testing this behavior)
@zacharycmontoya
zacharycmontoya force-pushed the zach.montoya/otlp-export-refined branch from 0bdf3bd to 70c4df0 Compare March 20, 2026 01:57
@zacharycmontoya zacharycmontoya added area:tracer The core tracer library (Datadog.Trace, does not include OpenTracing, native code, or integrations) area:opentelemetry OpenTelemetry support labels Mar 20, 2026
…preventing the following resource-level attributes from being set as OTLP span attributes:

- telemetry.sdk.name
- telemetry.sdk.language
- telemetry.sdk.version
…g when we're specifically testing the OTel SDK
…Traces to reduce test flakiness. We can run and regenerate OTEL SDK snapshots as needed
…actually needed for traces to picked up by the Datadog SDK for the OTel Tracing API
…he OpenTelemetrySdkTests sends traces to the wrong test-agent port
@zacharycmontoya
zacharycmontoya merged commit b70e6f2 into master Mar 24, 2026
140 checks passed
@zacharycmontoya
zacharycmontoya deleted the zach.montoya/otlp-export-refined branch March 24, 2026 15:54
zacharycmontoya added a commit that referenced this pull request May 22, 2026
…rt (#8645)

## Summary of changes
Adds `http/protobuf` support to the OTLP traces exporter introduced in
#8211, which introduced the exporter and `http/json` support.

This feature is enabled by setting `OTEL_TRACES_EXPORTER=otlp` and
`OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf` (or the parent
`OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf`).

## Reason for change
When #8211 landed, `http/json` was the only functional OTLP traces
protocol — setting any other protocol would fall back (with a log
message) to the Datadog MessagePack encoding.

`http/protobuf` is the OTLP default protocol per the OpenTelemetry
specification and is what most OTel collectors, Datadog Agent included,
expect to receive, so we need to support it to be useful out-of-the-box
for users adopting the OTLP traces export.

## Implementation details

### Serializer
Adds `Datadog.Trace.OpenTelemetry.Traces.OtlpTracesProtobufSerializer`,
a new `ISpanBufferSerializer` that emits the [OTLP
`ExportTraceServiceRequest`](https://github.com/open-telemetry/opentelemetry-proto/blob/v1.2.0/opentelemetry/proto/trace/v1/trace.proto)
protobuf payload directly into `SpanBuffer`'s output buffer:
- The `resource_spans` and `scope_spans` envelopes are opened on the
first `SerializeSpans` call, span entries are appended per chunk, and
`FinishBody` patches the two length-prefix placeholders using absolute
positions into the destination buffer.
- The serializer respects the same per-span / per-event / per-link
attribute count limits (`128`) as `OtlpTracesJsonSerializer`, and
re-uses the `OtlpMapper` logic to map resource attributes and tags onto
the serialized spans.
- `OtlpTracesProtobufSerializer` is **stateful** (it tracks the two
length-placeholder positions), so `AgentWriter` now creates a fresh
instance for the front and back buffers rather than sharing one — the
previous shared `OtlpTracesJsonSerializer` was stateless and didn't need
this.

### Vendored OpenTelemetry SDK
Builds on the protobuf primitives vendored from the OpenTelemetry .NET
SDK in #7653 (`ProtobufSerializer`, `ProtobufWireType`):
- Adds new field-number constant files:
`ProtobufOtlpTraceFieldNumberConstants` and
`ProtobufOtlpResourceFieldNumberConstants` (split out so the OTLP trace
serializer only depends on what it needs).
- Removes the `#if NETCOREAPP3_1_OR_GREATER` guard from the vendored
`ProtobufSerializer` — now that we vendor a `Span<>` polyfill the
serializer compiles and runs on all supported runtimes (.NET Framework
4.6.1+).
- Drops the matching `#if NET6_0_OR_GREATER` guard around the
`SubmitsOtlpTraces` integration test (`http/protobuf` and `http/json`
both run on .NET Framework now too).

### Wiring
- `ExporterSettings`: `OtlpProtocol.HttpProtobuf` now resolves to
`TracesEncoding.OtlpProtobuf` instead of falling back to `DatadogV0_4`.
- `AgentWriter.GetSpanSerializer`: selects
`OtlpTracesProtobufSerializer` when `TracesEncoding == OtlpProtobuf`.
- `ApiOtlp.SendTracesAsync`: chooses `application/x-protobuf` vs
`application/json` for the request `Content-Type` based on
`_tracesEncoding`. Adds the `application/x-protobuf` MIME type constant
to `Datadog.Trace.Agent.Transports.MimeTypes`.
- `supported-configurations.yaml`: documents `http/protobuf` as a valid
value for `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL` and regenerates
`ConfigurationKeys.OpenTelemetry.g.cs` for all TFMs.

## Test coverage
- **Unit tests**: New `OtlpTracesProtobufSerializerTests` exercises the
serializer end-to-end using `Google.Protobuf` and a little custom
parsing. Test cases include:
  - Empty buffer (`FinishBody` returns 0)
  - Multiple traces reported in a single-chunk
  - Serializing span events/links/status
  - Exceeding buffer `maxSize` honoring behavior
  - Exceeding span limits
- **Integration tests**: `OpenTelemetrySdkTests.SubmitsOtlpTraces` now
runs both `http/json` and `http/protobuf` cases for the Datadog SDK on
all TFMs.
- **Snapshot unification**: To keep one Datadog-SDK snapshot regardless
of OTLP protocol, snapshots for `http/protobuf` are normalized to the
`http/json` shape via scrubbers that translate the test agent's protobuf
rendering (snake_case field names, string-form enum values like
`SPAN_KIND_SERVER`) into the JSON serializer's shape (camelCase field
names, integer enum values).
- **Snapshot rename**: `SubmitsOtlpTraces_DD_http_json.verified.txt`
renamed to `SubmitsOtlpTraces_DD.verified.txt` (one snapshot now covers
both protocols).

## Other details
With this PR, `http/json` and `http/protobuf` are both functional;
`grpc` still falls back to Datadog MessagePack encoding with a startup
warning.

### Follow-up work
- Implement the `grpc` OTLP traces protocol
- Add OTLP traces telemetry counts (the `TODO: Telemetry - Record OTLP
Traces API submissions` in `ApiOtlp.SendTracesAsync` still applies)
- Honor `OTEL_EXPORTER_OTLP_TRACES_TIMEOUT` in `ApiOtlp` (still unused)

---------

Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:opentelemetry OpenTelemetry support area:tracer The core tracer library (Datadog.Trace, does not include OpenTracing, native code, or integrations)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants