Skip to content

feat(otlp): support http/protobuf OTLP trace export#18609

Open
bm1549 wants to merge 1 commit into
mainfrom
brian.marks/otlp-http-protobuf-export
Open

feat(otlp): support http/protobuf OTLP trace export#18609
bm1549 wants to merge 1 commit into
mainfrom
brian.marks/otlp-http-protobuf-export

Conversation

@bm1549

@bm1549 bm1549 commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

Description

Adds http/protobuf as a second OTLP trace-export encoding alongside the existing http/json path, and makes it the default. The encoding is selected from the OTel-standard OTEL_EXPORTER_OTLP_TRACES_PROTOCOL value and passed to libdatadog via the new set_otlp_protocol builder setter.

  • http/protobuf (the default) and http/json are supported over HTTP.
  • Any other value (e.g. the OTel-default grpc) is coerced to http/protobuf.

libdatadog is not bumped here. set_otlp_protocol is available in the v37.0.0 release already on main, so this branch is rebased onto it and carries only the feature code.

Testing

  • contrib::opentelemetry OTLP trace suite passes on Python 3.13 (test_otlp_traces_sent_via_http, test_otlp_traces_sent_via_http_protobuf_default).
  • cargo check/clippy/fmt clean on the native extension against v37.0.0.
  • Backend verification (local DD Agent to datadoghq.com): protobuf OTLP traces ingested with application/x-protobuf wire (HTTP 200), correct service/resource names, and a preserved 128-bit trace id. http/json regression confirmed (application/json, HTTP 200).

Risks

  • OTLP trace export now defaults to http/protobuf; set OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/json for the previous JSON wire format. Only the encoding changes; the endpoint (/v1/traces on port 4318) serves both.
  • The change is gated on _otlp_endpoint being set; non-OTLP export is unaffected.

Additional Notes

The protocol coercion happens on the Python side (writer.py), so the native set_otlp_protocol only ever receives http/json or http/protobuf. This matches libdatadog, which rejects grpc at the parse boundary.

@bm1549 bm1549 added the AI Generated Largely based on code generated by an AI or LLM. This label is the same across all dd-trace-* repos label Jun 12, 2026
@datadog-official

datadog-official Bot commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: c870bbf | Docs | Datadog PR Page | Give us feedback!

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Jun 12, 2026

Copy link
Copy Markdown

Codeowners resolved as

ddtrace/internal/native/_native.pyi                                     @DataDog/apm-core-python
ddtrace/internal/settings/_opentelemetry.py                             @DataDog/apm-sdk-capabilities-python
ddtrace/internal/writer/writer.py                                       @DataDog/apm-core-python
releasenotes/notes/otlp-http-protobuf-trace-export-531f4df90b51b9ff.yaml  @DataDog/apm-python
src/native/data_pipeline/mod.rs                                         @DataDog/apm-core-python
tests/opentelemetry/test_otlp_trace.py                                  @DataDog/apm-sdk-capabilities-python @DataDog/apm-core-python

@pr-commenter

pr-commenter Bot commented Jun 12, 2026

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-07-16 02:14:47

Comparing candidate commit c870bbf in PR branch brian.marks/otlp-http-protobuf-export with baseline commit 620eca9 in branch main.

Found 0 performance improvements and 5 performance regressions! Performance is the same for 613 metrics, 10 unstable metrics.

scenario:iastaspects-lstrip_aspect

  • 🟥 execution_time [+45.795µs; +53.795µs] or [+14.349%; +16.856%]

scenario:iastaspects-upper_aspect

  • 🟥 execution_time [+39.201µs; +46.470µs] or [+12.610%; +14.949%]

scenario:iastaspectsospath-ospathbasename_aspect

  • 🟥 execution_time [+80.587µs; +88.457µs] or [+18.816%; +20.654%]

scenario:span-start

  • 🟥 execution_time [+1.619ms; +1.825ms] or [+10.367%; +11.683%]

scenario:tracer-small

  • 🟥 execution_time [+26.130µs; +28.631µs] or [+7.330%; +8.032%]

gh-worker-dd-mergequeue-cf854d Bot pushed a commit to DataDog/libdatadog that referenced this pull request Jun 23, 2026
# What does this PR do?

Adds OTLP HTTP/protobuf as a trace-export encoding alongside HTTP/JSON, selectable via the OTel-standard `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL` (`http/json` default, `http/protobuf`). The generated `prost` OTLP types are the single intermediate representation: the mapper builds them directly from native spans, protobuf is `prost::encode_to_vec`, and JSON is a serde serializer over the same types.

# Motivation

libdatadog's OTLP trace export only spoke HTTP/JSON. SDKs that honor `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL` need `http/protobuf` to match the OTel default and to talk to collectors that expect protobuf.

# Additional Notes

- Prost is the single IR: no parallel hand-rolled JSON model, no string round-trip. `encode_otlp_protobuf` is `encode_to_vec`; `encode_otlp_json` is a serde serializer (`json_serializer.rs`) emitting OTLP-spec JSON (hex ids, base64 bytes, int64-as-string, lowerCamelCase, proto3 defaults omitted).
- `OtlpProtocol` is `{HttpJson, HttpProtobuf}`; `grpc` is rejected at the parse boundary (`FromStr`) rather than carried as an unsupported variant. `content_type()`/`encode()` live on the type.
- Integrates with the OTLP metrics exporter already on main: the shared low-level sender takes the content-type per request, so traces use the configured protocol while metrics stay JSON.
- Carries through `otel_trace_semantics_enabled` (OTel attribute compatibility, #2091): when set, the prost mapper omits the DD-specific span attributes and promotes the error message to the OTLP `Status`.
- Span-link W3C trace flags are carried through to OTLP `Link.flags`.
- Benchmarks for the encoder hot paths, plus allocation tuning (pre-sized Vecs, allocation-free id/timestamp/int serialization).

# How to test the change?

- `cargo test -p libdd-trace-utils -p libdd-data-pipeline -p libdd-data-pipeline-ffi`, `cargo test --doc`, clippy, fmt, and `cargo ffi-test` pass. A parity test asserts the JSON and protobuf encodings carry the same span from the one prost IR; a protobuf round-trip test asserts the encoding is lossless.
- End-to-end through dd-trace-py (DataDog/dd-trace-py#18609): emitted protobuf-only OTLP traces through a local Agent to the backend. Wire was `application/x-protobuf` (HTTP 200), spans ingested with correct service/resource and a preserved 128-bit trace id.
- Benchmarks: `cargo bench -p libdd-trace-utils --bench main -- otlp/`.

BREAKING CHANGE: removes the previously public `libdd_trace_utils::otlp_encoder::json_types` module (the hand-rolled OTLP JSON model). OTLP encoding now builds prost-generated types as the single IR. libdatadog consumers pin by version, so they pick this up on the next release.


Co-authored-by: brian.marks <[email protected]>
@bm1549
bm1549 force-pushed the brian.marks/otlp-http-protobuf-export branch from d485fb5 to a53e451 Compare June 24, 2026 14:42
taegyunkim pushed a commit to DataDog/libdatadog that referenced this pull request Jul 8, 2026
# What does this PR do?

Adds OTLP HTTP/protobuf as a trace-export encoding alongside HTTP/JSON, selectable via the OTel-standard `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL` (`http/json` default, `http/protobuf`). The generated `prost` OTLP types are the single intermediate representation: the mapper builds them directly from native spans, protobuf is `prost::encode_to_vec`, and JSON is a serde serializer over the same types.

# Motivation

libdatadog's OTLP trace export only spoke HTTP/JSON. SDKs that honor `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL` need `http/protobuf` to match the OTel default and to talk to collectors that expect protobuf.

# Additional Notes

- Prost is the single IR: no parallel hand-rolled JSON model, no string round-trip. `encode_otlp_protobuf` is `encode_to_vec`; `encode_otlp_json` is a serde serializer (`json_serializer.rs`) emitting OTLP-spec JSON (hex ids, base64 bytes, int64-as-string, lowerCamelCase, proto3 defaults omitted).
- `OtlpProtocol` is `{HttpJson, HttpProtobuf}`; `grpc` is rejected at the parse boundary (`FromStr`) rather than carried as an unsupported variant. `content_type()`/`encode()` live on the type.
- Integrates with the OTLP metrics exporter already on main: the shared low-level sender takes the content-type per request, so traces use the configured protocol while metrics stay JSON.
- Carries through `otel_trace_semantics_enabled` (OTel attribute compatibility, #2091): when set, the prost mapper omits the DD-specific span attributes and promotes the error message to the OTLP `Status`.
- Span-link W3C trace flags are carried through to OTLP `Link.flags`.
- Benchmarks for the encoder hot paths, plus allocation tuning (pre-sized Vecs, allocation-free id/timestamp/int serialization).

# How to test the change?

- `cargo test -p libdd-trace-utils -p libdd-data-pipeline -p libdd-data-pipeline-ffi`, `cargo test --doc`, clippy, fmt, and `cargo ffi-test` pass. A parity test asserts the JSON and protobuf encodings carry the same span from the one prost IR; a protobuf round-trip test asserts the encoding is lossless.
- End-to-end through dd-trace-py (DataDog/dd-trace-py#18609): emitted protobuf-only OTLP traces through a local Agent to the backend. Wire was `application/x-protobuf` (HTTP 200), spans ingested with correct service/resource and a preserved 128-bit trace id.
- Benchmarks: `cargo bench -p libdd-trace-utils --bench main -- otlp/`.

BREAKING CHANGE: removes the previously public `libdd_trace_utils::otlp_encoder::json_types` module (the hand-rolled OTLP JSON model). OTLP encoding now builds prost-generated types as the single IR. libdatadog consumers pin by version, so they pick this up on the next release.

Co-authored-by: brian.marks <[email protected]>
Signed-off-by: Taegyun Kim <[email protected]>
@bm1549
bm1549 force-pushed the brian.marks/otlp-http-protobuf-export branch from 132f024 to a68d1ae Compare July 16, 2026 01:36
@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Jul 16, 2026

Copy link
Copy Markdown

Circular import analysis

⚠️ Existing circular imports

There are 61 circular imports that already exist on the base branch and have not been changed by this PR.

Show existing cycles (showing 5 of 61 shortest)
ddtrace.internal.datastreams -> ddtrace.internal.datastreams.kombu -> ddtrace.internal.datastreams
ddtrace.internal.datastreams -> ddtrace.internal.datastreams.kafka -> ddtrace.internal.datastreams
ddtrace.internal.datastreams -> ddtrace.internal.datastreams.aiokafka -> ddtrace.internal.datastreams
ddtrace.internal.core -> ddtrace._trace.span -> ddtrace.internal.core
ddtrace.internal.datastreams -> ddtrace.internal.datastreams.google_cloud_pubsub -> ddtrace.internal.datastreams

To see all cycles, download the cycles-base.json and cycles-pr.json artifacts from this CI job and run:

uv run --script scripts/import-analysis/cycles.py compare cycles-base.json cycles-pr.json

Adds http/protobuf as the default OTLP trace export protocol, exposing
set_otlp_protocol on the native TraceExporterBuilder and wiring the
writer/settings to select it.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
@bm1549
bm1549 force-pushed the brian.marks/otlp-http-protobuf-export branch from a68d1ae to c870bbf Compare July 16, 2026 01:44
@bm1549
bm1549 marked this pull request as ready for review July 22, 2026 19:52
@bm1549
bm1549 requested review from a team as code owners July 22, 2026 19:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AI Generated Largely based on code generated by an AI or LLM. This label is the same across all dd-trace-* repos

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant