feat(otlp): support http/protobuf OTLP trace export#18609
Open
bm1549 wants to merge 1 commit into
Open
Conversation
3 tasks
Contributor
🎉 All green!🧪 All tests passed 🔗 Commit SHA: c870bbf | Docs | Datadog PR Page | Give us feedback! |
Codeowners resolved as |
BenchmarksBenchmark execution time: 2026-07-16 02:14:47 Comparing candidate commit c870bbf in PR branch Found 0 performance improvements and 5 performance regressions! Performance is the same for 613 metrics, 10 unstable metrics. scenario:iastaspects-lstrip_aspect
scenario:iastaspects-upper_aspect
scenario:iastaspectsospath-ospathbasename_aspect
scenario:span-start
scenario:tracer-small
|
gh-worker-dd-mergequeue-cf854d Bot
pushed a commit
to DataDog/libdatadog
that referenced
this pull request
Jun 23, 2026
# What does this PR do?
Adds OTLP HTTP/protobuf as a trace-export encoding alongside HTTP/JSON, selectable via the OTel-standard `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL` (`http/json` default, `http/protobuf`). The generated `prost` OTLP types are the single intermediate representation: the mapper builds them directly from native spans, protobuf is `prost::encode_to_vec`, and JSON is a serde serializer over the same types.
# Motivation
libdatadog's OTLP trace export only spoke HTTP/JSON. SDKs that honor `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL` need `http/protobuf` to match the OTel default and to talk to collectors that expect protobuf.
# Additional Notes
- Prost is the single IR: no parallel hand-rolled JSON model, no string round-trip. `encode_otlp_protobuf` is `encode_to_vec`; `encode_otlp_json` is a serde serializer (`json_serializer.rs`) emitting OTLP-spec JSON (hex ids, base64 bytes, int64-as-string, lowerCamelCase, proto3 defaults omitted).
- `OtlpProtocol` is `{HttpJson, HttpProtobuf}`; `grpc` is rejected at the parse boundary (`FromStr`) rather than carried as an unsupported variant. `content_type()`/`encode()` live on the type.
- Integrates with the OTLP metrics exporter already on main: the shared low-level sender takes the content-type per request, so traces use the configured protocol while metrics stay JSON.
- Carries through `otel_trace_semantics_enabled` (OTel attribute compatibility, #2091): when set, the prost mapper omits the DD-specific span attributes and promotes the error message to the OTLP `Status`.
- Span-link W3C trace flags are carried through to OTLP `Link.flags`.
- Benchmarks for the encoder hot paths, plus allocation tuning (pre-sized Vecs, allocation-free id/timestamp/int serialization).
# How to test the change?
- `cargo test -p libdd-trace-utils -p libdd-data-pipeline -p libdd-data-pipeline-ffi`, `cargo test --doc`, clippy, fmt, and `cargo ffi-test` pass. A parity test asserts the JSON and protobuf encodings carry the same span from the one prost IR; a protobuf round-trip test asserts the encoding is lossless.
- End-to-end through dd-trace-py (DataDog/dd-trace-py#18609): emitted protobuf-only OTLP traces through a local Agent to the backend. Wire was `application/x-protobuf` (HTTP 200), spans ingested with correct service/resource and a preserved 128-bit trace id.
- Benchmarks: `cargo bench -p libdd-trace-utils --bench main -- otlp/`.
BREAKING CHANGE: removes the previously public `libdd_trace_utils::otlp_encoder::json_types` module (the hand-rolled OTLP JSON model). OTLP encoding now builds prost-generated types as the single IR. libdatadog consumers pin by version, so they pick this up on the next release.
Co-authored-by: brian.marks <[email protected]>
bm1549
force-pushed
the
brian.marks/otlp-http-protobuf-export
branch
from
June 24, 2026 14:42
d485fb5 to
a53e451
Compare
taegyunkim
pushed a commit
to DataDog/libdatadog
that referenced
this pull request
Jul 8, 2026
# What does this PR do?
Adds OTLP HTTP/protobuf as a trace-export encoding alongside HTTP/JSON, selectable via the OTel-standard `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL` (`http/json` default, `http/protobuf`). The generated `prost` OTLP types are the single intermediate representation: the mapper builds them directly from native spans, protobuf is `prost::encode_to_vec`, and JSON is a serde serializer over the same types.
# Motivation
libdatadog's OTLP trace export only spoke HTTP/JSON. SDKs that honor `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL` need `http/protobuf` to match the OTel default and to talk to collectors that expect protobuf.
# Additional Notes
- Prost is the single IR: no parallel hand-rolled JSON model, no string round-trip. `encode_otlp_protobuf` is `encode_to_vec`; `encode_otlp_json` is a serde serializer (`json_serializer.rs`) emitting OTLP-spec JSON (hex ids, base64 bytes, int64-as-string, lowerCamelCase, proto3 defaults omitted).
- `OtlpProtocol` is `{HttpJson, HttpProtobuf}`; `grpc` is rejected at the parse boundary (`FromStr`) rather than carried as an unsupported variant. `content_type()`/`encode()` live on the type.
- Integrates with the OTLP metrics exporter already on main: the shared low-level sender takes the content-type per request, so traces use the configured protocol while metrics stay JSON.
- Carries through `otel_trace_semantics_enabled` (OTel attribute compatibility, #2091): when set, the prost mapper omits the DD-specific span attributes and promotes the error message to the OTLP `Status`.
- Span-link W3C trace flags are carried through to OTLP `Link.flags`.
- Benchmarks for the encoder hot paths, plus allocation tuning (pre-sized Vecs, allocation-free id/timestamp/int serialization).
# How to test the change?
- `cargo test -p libdd-trace-utils -p libdd-data-pipeline -p libdd-data-pipeline-ffi`, `cargo test --doc`, clippy, fmt, and `cargo ffi-test` pass. A parity test asserts the JSON and protobuf encodings carry the same span from the one prost IR; a protobuf round-trip test asserts the encoding is lossless.
- End-to-end through dd-trace-py (DataDog/dd-trace-py#18609): emitted protobuf-only OTLP traces through a local Agent to the backend. Wire was `application/x-protobuf` (HTTP 200), spans ingested with correct service/resource and a preserved 128-bit trace id.
- Benchmarks: `cargo bench -p libdd-trace-utils --bench main -- otlp/`.
BREAKING CHANGE: removes the previously public `libdd_trace_utils::otlp_encoder::json_types` module (the hand-rolled OTLP JSON model). OTLP encoding now builds prost-generated types as the single IR. libdatadog consumers pin by version, so they pick this up on the next release.
Co-authored-by: brian.marks <[email protected]>
Signed-off-by: Taegyun Kim <[email protected]>
bm1549
force-pushed
the
brian.marks/otlp-http-protobuf-export
branch
from
July 16, 2026 01:36
132f024 to
a68d1ae
Compare
Circular import analysis
|
Adds http/protobuf as the default OTLP trace export protocol, exposing set_otlp_protocol on the native TraceExporterBuilder and wiring the writer/settings to select it. Co-Authored-By: Claude Sonnet 5 <[email protected]>
bm1549
force-pushed
the
brian.marks/otlp-http-protobuf-export
branch
from
July 16, 2026 01:44
a68d1ae to
c870bbf
Compare
bm1549
marked this pull request as ready for review
July 22, 2026 19:52
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds
http/protobufas a second OTLP trace-export encoding alongside the existinghttp/jsonpath, and makes it the default. The encoding is selected from the OTel-standardOTEL_EXPORTER_OTLP_TRACES_PROTOCOLvalue and passed to libdatadog via the newset_otlp_protocolbuilder setter.http/protobuf(the default) andhttp/jsonare supported over HTTP.grpc) is coerced tohttp/protobuf.libdatadog is not bumped here.
set_otlp_protocolis available in the v37.0.0 release already onmain, so this branch is rebased onto it and carries only the feature code.Testing
contrib::opentelemetryOTLP trace suite passes on Python 3.13 (test_otlp_traces_sent_via_http,test_otlp_traces_sent_via_http_protobuf_default).cargo check/clippy/fmtclean on the native extension against v37.0.0.application/x-protobufwire (HTTP 200), correct service/resource names, and a preserved 128-bit trace id.http/jsonregression confirmed (application/json, HTTP 200).Risks
http/protobuf; setOTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/jsonfor the previous JSON wire format. Only the encoding changes; the endpoint (/v1/traceson port 4318) serves both._otlp_endpointbeing set; non-OTLP export is unaffected.Additional Notes
The protocol coercion happens on the Python side (
writer.py), so the nativeset_otlp_protocolonly ever receiveshttp/jsonorhttp/protobuf. This matches libdatadog, which rejectsgrpcat the parse boundary.