feat(tracer): Implement OTLP Traces Export by mtoffl01 · Pull Request #4600 · DataDog/dd-trace-go

mtoffl01 · 2026-03-25T16:45:33Z

What does this PR do?

Implements the OTLP trace export pipeline (See: RFC) for dd-trace-go: when OTEL_TRACES_EXPORTER=otlp is set, the tracer converts Datadog spans into OTLP protobuf format and sends them directly to an OpenTelemetry Collector instead of the Datadog agent.

This adds three new components:

span_to_otlp.go — converts internal Datadog spans to OTLP TracesData protobuf, including resource attributes, span kind/status, events, links, 128-bit trace IDs, and sampling filtering (only sampled spans are exported).
otlp_writer.go — a traceWriter implementation that buffers converted OTLP spans, marshals them to protobuf, and flushes with retry logic.
otlp_transport.go — a lightweight HTTP transport for sending protobuf payloads to an OTLP endpoint, separate from the existing Datadog-protocol transport.

The PR also renames the existing transport interface and config.transport field to ddTransport to make it clear that it handles Datadog-specific traffic (msgpack traces, stats), and ensures ddTransport always points at the Datadog agent even when OTLP mode is active.

Motivation

We are adding native OTLP export support to dd-trace-go so users can send traces directly to any OTLP-compatible collector. A prior PR added the configuration layer that resolves OTEL_TRACES_EXPORTER, OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, and OTEL_EXPORTER_OTLP_TRACES_HEADERS. This PR builds the runtime pipeline on top of that configuration.

IMPORTANT NOTE!

The existing agentTraceWriter and ddTransport was too tightly coupled to Datadog conventions — msgpack encoding, agent sampling rate feedback, stats collection — to extend it for the otlp path, so a separate writer and transport were introduced instead. The otlpTraceWriter owns its own otlpTransport directly (rather than reaching through config.ddTransport), keeping OTLP concerns isolated from the Datadog agent path. A follow-up PR may remove ddTransport from the config struct entirely, instead passing the transport directly to each writer and consumer at construction time. This would make all writer implementations structurally consistent — each owning its transport as a field — and eliminate the implicit coupling through config.

An additional PR to disable stats collection in otlp mode is in progress here.

Reviewer's Checklist

Changed code has unit tests for its functionality at or near 100% coverage.
System-Tests covering this feature have been added and enabled with the va.b.c-dev version tag.
There is a benchmark for any new code, or changes to existing code.
If this interacts with the agent in a new way, a system test has been added.
New code is free of linting errors. You can check this by running make lint locally.
New code doesn't break existing tests. You can check this by running make test locally.
Add an appropriate team label so this PR gets put in the right place for the release notes.
All generated files are up to date. You can check this by running make generate locally.
Non-trivial go.mod changes, e.g. adding new modules, are reviewed by @DataDog/dd-trace-go-guild. Make sure all nested modules are up to date by running make fix-modules locally.

Unsure? Have a question? Request a review!

…port

… to determine otlp mode

Made-with: Cursor

…traceURL

…nfig registry to reflect this

…ut ddspan -> otlpspan conversion

… span_to_otlp_test.go

… service name differs from global service name

…me (#4576) ### What does this PR do? Moves the `strconv.ParseInt` call for `_dd.p.dm` (decision maker) out of the v1 payload flush path and into the write path. A `dm uint32` field is added to the `trace` struct. It is kept in sync with `propagatingTags[keyDecisionMaker]` at all mutation sites (`setPropagatingTagLocked`, `unsetPropagatingTagLocked`, `replacePropagatingTags`). A shared `unsetPropagatingTagLocked` helper is introduced so the delete-and-clear logic is not duplicated between `unsetPropagatingTag` and `setSamplingPriorityLockedWithForce`. `payloadV1.push` now reads the pre-computed value via `trace.decisionMaker()` instead of fetching the string from the propagating tags map and parsing it on every flush. ### Motivation `payloadV1.push` is called once per finished trace chunk. On every call it was acquiring a read lock, doing a map lookup, and then running `strconv.ParseInt` — all to read a value that was set at most once or twice during the lifetime of the trace. Moving the parse to write-time eliminates the per-flush map lookup and parse, replacing them with a single scalar read behind an `RLock`. ### Reviewer's Checklist - [x] Changed code has unit tests for its functionality at or near 100% coverage. - [x] New code is free of linting errors. You can check this by running `make lint` locally. - [x] New code doesn't break existing tests. You can check this by running `make test` locally. - [x] Add an appropriate team label so this PR gets put in the right place for the release notes. - [x] All generated files are up to date. You can check this by running `make generate` locally. Co-authored-by: dario.castane <[email protected]>

…ED` once, use a global `bool` (#4548) Stop checking environment variable `DD_TRACE_128_BIT_TRACEID_GENERATION_ENABLED` on span start. Checks it once, use a global `bool`. It'd be better to load it on `newConfig` and make the value available, but this one happens in `newSpanContext`, which modification will cause a lot of changes in multiple files, including tests. Complete #1905 work using the current benchmarking platform. - [x] Changed code has unit tests for its functionality at or near 100% coverage. - [x] There is a benchmark for any new code, or changes to existing code. - [x] New code is free of linting errors. You can check this by running `make lint` locally. - [x] New code doesn't break existing tests. You can check this by running `make test` locally. - [x] Add an appropriate team label so this PR gets put in the right place for the release notes. Unsure? Have a question? Request a review! Co-authored-by: kakkoyun <[email protected]> Co-authored-by: kemal.akkoyun <[email protected]>

Signed-off-by: Moe Zein <[email protected]>

Co-authored-by: Benjamin De Bernardi <[email protected]>

kakkoyun · 2026-03-27T10:26:26Z

-// resolveTraceTransport returns the trace URL and headers for the transport
-// based on whether OTLP export mode is active.
+// resolveTraceTransport returns the trace URL and headers for the Datadog
+// agent transport. In OTLP export mode the ddTransport is not used for trace


darccio · 2026-04-14T12:57:59Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8d6156e0db

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

darccio · 2026-04-14T12:56:44Z

+	return &otlptrace.Span{
+		TraceId:           convertTraceID(s.context.traceID.Upper(), s.context.traceID.Lower()),
+		SpanId:            convertSpanID(s.spanID),
+		ParentSpanId:      convertParentSpanID(s.parentID),
+		Name:              s.resource,
+		Kind:              convertSpanKind(getSpanKind(s)),
+		StartTimeUnixNano: uint64(s.start),
+		EndTimeUnixNano:   uint64(s.start + s.duration),
+		Attributes:        convertSpanAttributes(s, defaultServiceName),
+		Events:            convertEvents(s),
+		Links:             convertSpanLinks(s.spanLinks),
+		Status:            convertSpanStatus(s),
+		TraceState:        convertTraceState(s.context),
+	}


I think we could make this more efficient by wrapping *tracer.Span with otlptrace.Span and having getters returning each expected field in the converted format.

This can be done in the future.

Every field converted here is a required field in the protobuf output; there are no unused conversions. So as I see it, deferring the conversion would just change when the latency is introduced, not how much latency we introduce. Unless I am missing something?

I was thinking on the heap allocation that happens here. If we had a wrapper type that stores a pointer to the tracer.Span we could probably make this function inlineable, as we wouldn't need to run all these conversion functions here.

I'm also not sure if these conversion functions are inlining here.

In any case, looking how this is used, I see it's not possible to optimize that much.

hannahkm

Thanks! 🎉

kakkoyun

LGTM.

Thanks for enduring all the reviews. It's an critical piece and a bi g PR. I hope we haven't missed anything. We can always iterate.

kakkoyun · 2026-04-15T12:37:38Z

+	scope     *otlpcommon.InstrumentationScope
+	spans     []*otlptrace.Span // +checklocks:mu
+	buffSize  int               // +checklocks:mu
+	baseSize  int


nit: Lack of this annotation should not be green on the CI, I need to check this.

mtoffl01 and others added 30 commits March 23, 2026 09:30

Allow Config to resolve traceURL; pass this as a field into httptrans…

8f7f60f

…port

Fix otlp protocol resolution and add tests for all new logic

3e3fec8

Implement OTEL_EXPORTER_OTLP_TRACES_HEADERS

c559f8f

Fix superfluous referecne to unsupport OTLP header config

cb6908e

Introduce otlpExportMode on Config; use this instead of traceProtocol…

a85fedf

… to determine otlp mode

fix relationship between otlpexportmode and protocol

33a9a37

replace tracer.config.otlpExportMode with internalconfig otlpExportMode

3e51fdd

Made-with: Cursor

Clean up resolveOTLPHeaders

5e252be

cleanup

51f72a2

more cleanup and remove OTEL_EXPORTER_OTLP_ENDPOINT

b71eec7

Store otlpTraceURL instead of traceURL on Config; let tracer resolve …

f48d44f

…traceURL

Make DD_TRACE_AGENT_PROTOCOL_VERSION override OTEL_TRACES_EXPORTER

f4360b4

Merge branch 'main' into mtoff/otlp-export-config

49d0937

Use GetMap to resolve OTEL_EXPORTER_OTLP_TRACES_HEADERS and change co…

a88b7b2

…nfig registry to reflect this

give OTEL_EXPORTER_OTLP_TRACES_ENDPOINT default value of null

d475920

Merge branch 'main' into mtoff/otlp-export-config

8188bbe

flesh out traceWriterOtlp

af909c3

add new otlpSender for transporting otlp traces specifically; build o…

9246084

…ut ddspan -> otlpspan conversion

rename file payload_otlp_convert.go -> span_to_otlp.go; and introduce…

a934fdd

… span_to_otlp_test.go

introduce otlp_writer_test.go: basic otlp writer tests

2cfa217

Fix: only export sampled spans, and correct telemetry.sdk.version value

7bba29d

Test buildResource

89cb7e6

implement edge case: add 'service.name' attribute to span when span's…

da1efd4

… service name differs from global service name

add comment about s.metaStruct

49053c1

chore: disable automated dependency updater config [incident-51602]

ffbea4e

Signed-off-by: Moe Zein <[email protected]>

merge conflicts

9882338

merge conflicts

5072b2a

Apply suggestion from @genesor

78e160f

Co-authored-by: Benjamin De Bernardi <[email protected]>

Merge branch 'main' into mtoff/otlp-export-traces

f9ae432

ddyurchenko approved these changes Apr 8, 2026

View reviewed changes

kakkoyun reviewed Apr 9, 2026

View reviewed changes

mtoffl01 added 5 commits April 9, 2026 14:06

drain otlpTransport response body after reading status code

91fac2a

Cache base TracesData size

7c12988

Use Span_SPAN_KIND_UNSPECIFIED for unknown span types

f577ca2

wrap error in otlpTransport.send

ceea3dd

enforce Content-Type header at the otlpTransport.send level

b3a521d

mtoffl01 requested review from hannahkm and kakkoyun April 13, 2026 17:57

Merge branch 'main' into mtoff/otlp-export-traces

8d6156e

chatgpt-codex-connector Bot reviewed Apr 14, 2026

View reviewed changes

Comment thread ddtrace/tracer/span_to_otlp.go

Comment thread ddtrace/tracer/span_to_otlp.go Outdated

darccio reviewed Apr 14, 2026

View reviewed changes

mtoffl01 added 4 commits April 14, 2026 13:19

Address codex feedback on span_to_otlp

6ec6405

Fix nil check on otlpArrayValue converter

df091b3

Document maxAttributesCount

92de62c

Restore default traceWriter test

016cb71

mtoffl01 requested a review from darccio April 14, 2026 17:56

hannahkm approved these changes Apr 14, 2026

View reviewed changes

darccio approved these changes Apr 15, 2026

View reviewed changes

kakkoyun approved these changes Apr 15, 2026

View reviewed changes

gh-worker-dd-devflow-36fce6 Bot added mergequeue-status: queued mergequeue-status: in_progress and removed mergequeue-status: queued labels Apr 15, 2026

gh-worker-dd-mergequeue-cf854d Bot merged commit 73db441 into main Apr 15, 2026
310 of 313 checks passed

gh-worker-dd-devflow-36fce6 Bot added mergequeue-status: done and removed mergequeue-status: in_progress labels Apr 15, 2026

gh-worker-dd-mergequeue-cf854d Bot deleted the mtoff/otlp-export-traces branch April 15, 2026 12:56

mtoffl01 mentioned this pull request Apr 28, 2026

[Golang] enable Test_Otel_Tracing_OTLP for v2.8.0-dev DataDog/system-tests#6825

Merged

5 tasks

Uh oh!

Conversation

mtoffl01 commented Mar 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Motivation

IMPORTANT NOTE!

Reviewer's Checklist

Uh oh!

kakkoyun Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

darccio commented Apr 14, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

darccio Apr 14, 2026

Choose a reason for hiding this comment

Uh oh!

mtoffl01 Apr 14, 2026

Choose a reason for hiding this comment

Uh oh!

darccio Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hannahkm left a comment

Choose a reason for hiding this comment

Uh oh!

kakkoyun left a comment

Choose a reason for hiding this comment

Uh oh!

kakkoyun Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

mtoffl01 commented Mar 25, 2026 •

edited

Loading