feat(otel): enable improved OTel trace compatibility via opt-in configuration#18495
Conversation
🎉 All green!🧪 All tests passed 🔗 Commit SHA: 3e8bf7a | Docs | Datadog PR Page | Give us feedback! |
Codeowners resolved as |
BenchmarksBenchmark execution time: 2026-07-14 20:34:15 Comparing candidate commit 3e8bf7a in PR branch Found 0 performance improvements and 4 performance regressions! Performance is the same for 610 metrics, 10 unstable metrics. scenario:iastaspects-stringio_noaspect
scenario:iastaspectsospath-ospathbasename_aspect
scenario:span-start
scenario:telemetryaddmetric-1-count-metric-1-times
|
10a9453 to
3869cc9
Compare
This comment has been minimized.
This comment has been minimized.
- Rename datadog-remote-config → libdd-remote-config (package renamed in v36). - Remove libdd-http-client and stats-obfuscation feature (folded into defaults in v36). - Remove bytes/url deps that were only needed by http-client. - SharedRuntime is now a trait in v37; ForkSafeRuntime is the concrete type. Thread ForkSafeRuntime through SharedRuntimePy, TraceExporterBuilderPy, and TraceExporterPy. - Update tracer_flare.rs to the new libdd-remote-config config::agent_config/agent_task API. - Handle AssignmentReason::Default added to datadog-ffe in v37. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
libdatadog removed the libdd-http-client crate in v36; it no longer ships in v37. Remove the native http_client module and all its dependents: - Delete src/native/http_client/ (client, response, errors, mod) - Remove mod/register_http_client from lib.rs - Remove enable_client_side_stats_obfuscation from data_pipeline/mod.rs (stats-obfuscation feature merged into libdd-data-pipeline defaults) - Remove HTTPClient, HttpResponse, HttpClientError and subclasses from _native.pyi and ddtrace/internal/native/__init__.py - Delete ddtrace/internal/http_client.py and its tests The HTTPClient had no production callers outside its own wrapper module. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
stats-obfuscation is now on by default in libdatadog v37; the enable_client_side_stats_obfuscation() builder method no longer exists. Remove the call from TraceExporterWriter. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
…bfuscation feature The method still exists in libdatadog v37 but requires the stats-obfuscation feature on libdd-data-pipeline. Re-add the feature flag and restore the PyO3 wrapper, Python stub, and writer call that were incorrectly removed. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
libdd-http-client still exists in v37 — it was redesigned, not removed. Restore the http_client Rust module rewritten for the v37 API: - HttpClient::new() no longer takes a runtime; BlockingRuntime trait (not SharedRuntime) provides block_on — update imports accordingly - Arc<ForkSafeRuntime> replaces Arc<SharedRuntime> (SharedRuntime is now a trait in v37; ForkSafeRuntime is the concrete type) - Re-add libdd-http-client, bytes, url deps to Cargo.toml - Restore mod http_client and register_http_client in lib.rs - Restore ddtrace/internal/http_client.py and its test suite - Restore HTTPClient, HttpResponse, HttpClientError and subclasses in _native.pyi and ddtrace/internal/native/__init__.py - Add missing Iterable import to _native.pyi Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
The merged libdatadog FFE remaps per-flag config parse failures (bad regex, variant type mismatch) from ConfigurationParseError to the new FlagConfigurationInvalid variant. dd-trace-py's From<EvaluationError> match didn't handle it, so it fell through to ErrorCode::General and the feature_flag.evaluations metric reported error.type=general instead of parse_error, failing the FEATURE_FLAGGING_AND_EXPERIMENTATION system tests. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Co-authored-by: Munir Abdinur <[email protected]>
Restore the bytes 1.10+ comment and the original ordering of url and libdd-http-client that were cosmetically changed during the v37 bump. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
… arm Both map to PARSE_ERROR; combine them into a single match arm. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
…client The code review suggestion incorrectly reverted the import back to SharedRuntime (a v37 trait), but the struct holds Arc<ForkSafeRuntime> and block_on requires BlockingRuntime in scope. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
…ew() ForkSafeRuntime::new() is a trait method from SharedRuntime; without the trait in scope the compiler cannot resolve it. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
With collect_all_threads enabled (default in v37), the receiver uses
PTRACE_SEIZE + PTRACE_INTERRUPT on the crashing thread, which interrupts
poll() in wait_for_pollhup with EINTR. When crashing from a dynamically-
loaded C++ extension on Python 3.14, repeated EINTR prevents the 5-second
timeout from expiring, causing an indefinite hang.
Consistent with the existing pattern of Python 3.14 crashtracker skips
("Stack v2 not supported on 3.14") in 9d9bd80.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
…d from ptrace The receiver in v37 began ptracing the crashing thread (crashing_tid) as part of collect_all_threads. That thread is blocked in poll() inside its signal handler on the alternate signal stack; the resulting PTRACE_INTERRUPT issues EINTR in a loop, causing the receiver to hang indefinitely on CPython 3.14. Pin all libdatadog deps to commit 504ea9d7 on the fix branch, which restores the v35 behaviour of excluding crashing_tid from stream_thread_contexts. With the fix in place the 3.14 skip is no longer needed. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
…ng thread from ptrace" This reverts commit 52fa5a9.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 921244515e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
KowalskiThomas
left a comment
There was a problem hiding this comment.
Looks good for apm-python.
Co-authored-by: Thomas Kowalski <[email protected]>
… 'return True' path is only hit when OTLP trace export is enabled AND there's no user-configured DD Trace protocol version.
link04
left a comment
There was a problem hiding this comment.
LGTM, aside from that not sure how Python is about the nits I added.
…ation variable to '_otel_trace_semantics_enabled'
…re serialization.
…request" for client spans
…bling trace native span events when OTLP export is enabled
emmettbutler
left a comment
There was a problem hiding this comment.
all of the changes to _trace and internal look reasonable
Description
This PR adds
DD_TRACE_OTEL_SEMANTICS_ENABLEDto opt in to OpenTelemetry trace semantics for spans. The result is that using OpenTelemetry Tracing instrumentations allows DD SDKs to produce OTLP spans with nearly identical span fields.When enabled, this updates the dd-trace-py behavior in the following ways:
General tracing behaviors
version,language, andenvtags as span attributesOpenTelemetry Tracing APIs
OtelSpan.set_attributeno longer remaps span tags such ashttp.response.status_codeorservice.nameso they will maintain their intended key and value typeOtelSpan.record_exceptionno longer sets span attributeserror.message,error.type, orerror.stack. These exist on theexceptionspan event and are updated in this PR so that theexception.stacktraceandexception.typeattributes on the span event match what is emitted by the OpenTelemetry SDKOtelSpan.set_statusupdates theerror.messagespan attribute each time a new status is registered, including recording a newer status that lacks a messageOTLP traces export
enable_otel_trace_semantics()on the libdatadog native trace exporter to remove Datadog-specific span attributes from being emitted on OTLP spans (see the related libdatadog PR for specific attribute changes)Also makes the following fixes:
eventsfield.Relies on DataDog/libdatadog#2091 which implements the OTLP trace exporter changes and on #18821 which updates the libdatadog dependency.
Testing
Unit tests are included for the updated OTel test span behavior and the native span events configuration
Risks
Besides the native span events change, the remaining changes are opt-in so the risk is low.
Additional Notes
Currently the libdatadog dependency is set to a development branch which contains the updated
TraceExporterBuilder::enable_otel_trace_semantics()API