Skip to content

feat(otel): enable improved OTel trace compatibility via opt-in configuration#18495

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 60 commits into
mainfrom
zach.montoya/otel-trace-compatibility
Jul 21, 2026
Merged

feat(otel): enable improved OTel trace compatibility via opt-in configuration#18495
gh-worker-dd-mergequeue-cf854d[bot] merged 60 commits into
mainfrom
zach.montoya/otel-trace-compatibility

Conversation

@zacharycmontoya

@zacharycmontoya zacharycmontoya commented Jun 6, 2026

Copy link
Copy Markdown
Contributor

Description

This PR adds DD_TRACE_OTEL_SEMANTICS_ENABLED to opt in to OpenTelemetry trace semantics for spans. The result is that using OpenTelemetry Tracing instrumentations allows DD SDKs to produce OTLP spans with nearly identical span fields.

When enabled, this updates the dd-trace-py behavior in the following ways:

  • General tracing behaviors

    • Stops the library from adding the version, language, and env tags as span attributes
  • OpenTelemetry Tracing APIs

    • OtelSpan.set_attribute no longer remaps span tags such as http.response.status_code or service.name so they will maintain their intended key and value type
    • OtelSpan.record_exception no longer sets span attributes error.message, error.type, or error.stack. These exist on the exception span event and are updated in this PR so that the exception.stacktrace and exception.type attributes on the span event match what is emitted by the OpenTelemetry SDK
    • OtelSpan.set_status updates the error.message span attribute each time a new status is registered, including recording a newer status that lacks a message
  • OTLP traces export

    • Calls enable_otel_trace_semantics() on the libdatadog native trace exporter to remove Datadog-specific span attributes from being emitted on OTLP spans (see the related libdatadog PR for specific attribute changes)

Also makes the following fixes:

  • When OTLP traces export is enabled, native span events are also enabled because OTLP spans have a first-class events field.

Relies on DataDog/libdatadog#2091 which implements the OTLP trace exporter changes and on #18821 which updates the libdatadog dependency.

Testing

Unit tests are included for the updated OTel test span behavior and the native span events configuration

Risks

Besides the native span events change, the remaining changes are opt-in so the risk is low.

Additional Notes

Currently the libdatadog dependency is set to a development branch which contains the updated TraceExporterBuilder::enable_otel_trace_semantics() API

@datadog-datadog-prod-us1-2

datadog-datadog-prod-us1-2 Bot commented Jun 6, 2026

Copy link
Copy Markdown
Contributor

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 3e8bf7a | Docs | Datadog PR Page | Give us feedback!

Comment thread ddtrace/internal/settings/_agent.py Outdated
Comment thread ddtrace/internal/opentelemetry/span.py Outdated
@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Jun 8, 2026

Copy link
Copy Markdown

Codeowners resolved as

ddtrace/_trace/processor/__init__.py                                    @DataDog/apm-sdk-capabilities-python
ddtrace/_trace/tracer.py                                                @DataDog/apm-sdk-capabilities-python
ddtrace/internal/native/_native.pyi                                     @DataDog/apm-core-python
ddtrace/internal/opentelemetry/span.py                                  @DataDog/apm-sdk-capabilities-python
ddtrace/internal/settings/_agent.py                                     @DataDog/apm-core-python
ddtrace/internal/settings/_agent.pyi                                    @DataDog/apm-core-python
ddtrace/internal/settings/_config.py                                    @DataDog/python-guild @DataDog/apm-sdk-capabilities-python
ddtrace/internal/writer/writer.py                                       @DataDog/apm-core-python
releasenotes/notes/otel-trace-semantics-f655af74e566c719.yaml           @DataDog/apm-python
tests/internal/test_env.py                                              @DataDog/apm-sdk-capabilities-python
tests/opentelemetry/test_span.py                                        @DataDog/apm-sdk-capabilities-python @DataDog/apm-core-python
tests/telemetry/test_writer.py                                          @DataDog/apm-python
tests/tracer/test_agent.py                                              @DataDog/apm-sdk-capabilities-python @DataDog/apm-core-python
tests/tracer/test_processors.py                                         @DataDog/apm-sdk-capabilities-python @DataDog/apm-core-python
tests/tracer/test_tracer.py                                             @DataDog/apm-sdk-capabilities-python @DataDog/apm-core-python

@pr-commenter

pr-commenter Bot commented Jun 8, 2026

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-07-14 20:34:15

Comparing candidate commit 3e8bf7a in PR branch zach.montoya/otel-trace-compatibility with baseline commit 9e268d1 in branch main.

Found 0 performance improvements and 4 performance regressions! Performance is the same for 610 metrics, 10 unstable metrics.

scenario:iastaspects-stringio_noaspect

  • 🟥 execution_time [+41.754µs; +47.270µs] or [+11.107%; +12.575%]

scenario:iastaspectsospath-ospathbasename_aspect

  • 🟥 execution_time [+101.146µs; +107.571µs] or [+23.933%; +25.453%]

scenario:span-start

  • 🟥 execution_time [+1.250ms; +1.427ms] or [+7.870%; +8.983%]

scenario:telemetryaddmetric-1-count-metric-1-times

  • 🟥 execution_time [+158.788ns; +187.909ns] or [+7.588%; +8.980%]

Comment thread ddtrace/internal/opentelemetry/span.py Outdated
Comment thread ddtrace/internal/native/_native.pyi Outdated
Comment thread ddtrace/internal/opentelemetry/span.py Outdated
Comment thread docs/superpowers/specs/2026-06-04-otel-record-exception-compat-design.md Outdated
@datadog-dd-trace-py-rkomorn

This comment has been minimized.

Comment thread src/native/Cargo.lock
mabdinur and others added 18 commits June 30, 2026 11:38
- Rename datadog-remote-config → libdd-remote-config (package renamed in v36).
- Remove libdd-http-client and stats-obfuscation feature (folded into defaults in v36).
- Remove bytes/url deps that were only needed by http-client.
- SharedRuntime is now a trait in v37; ForkSafeRuntime is the concrete type.
  Thread ForkSafeRuntime through SharedRuntimePy, TraceExporterBuilderPy, and TraceExporterPy.
- Update tracer_flare.rs to the new libdd-remote-config config::agent_config/agent_task API.
- Handle AssignmentReason::Default added to datadog-ffe in v37.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
libdatadog removed the libdd-http-client crate in v36; it no longer ships
in v37. Remove the native http_client module and all its dependents:

- Delete src/native/http_client/ (client, response, errors, mod)
- Remove mod/register_http_client from lib.rs
- Remove enable_client_side_stats_obfuscation from data_pipeline/mod.rs
  (stats-obfuscation feature merged into libdd-data-pipeline defaults)
- Remove HTTPClient, HttpResponse, HttpClientError and subclasses from
  _native.pyi and ddtrace/internal/native/__init__.py
- Delete ddtrace/internal/http_client.py and its tests

The HTTPClient had no production callers outside its own wrapper module.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
stats-obfuscation is now on by default in libdatadog v37; the
enable_client_side_stats_obfuscation() builder method no longer exists.
Remove the call from TraceExporterWriter.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
…bfuscation feature

The method still exists in libdatadog v37 but requires the stats-obfuscation
feature on libdd-data-pipeline. Re-add the feature flag and restore the
PyO3 wrapper, Python stub, and writer call that were incorrectly removed.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
libdd-http-client still exists in v37 — it was redesigned, not removed.
Restore the http_client Rust module rewritten for the v37 API:

- HttpClient::new() no longer takes a runtime; BlockingRuntime trait
  (not SharedRuntime) provides block_on — update imports accordingly
- Arc<ForkSafeRuntime> replaces Arc<SharedRuntime> (SharedRuntime is
  now a trait in v37; ForkSafeRuntime is the concrete type)
- Re-add libdd-http-client, bytes, url deps to Cargo.toml
- Restore mod http_client and register_http_client in lib.rs
- Restore ddtrace/internal/http_client.py and its test suite
- Restore HTTPClient, HttpResponse, HttpClientError and subclasses in
  _native.pyi and ddtrace/internal/native/__init__.py
- Add missing Iterable import to _native.pyi

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
The merged libdatadog FFE remaps per-flag config parse failures (bad regex,
variant type mismatch) from ConfigurationParseError to the new
FlagConfigurationInvalid variant. dd-trace-py's From<EvaluationError> match
didn't handle it, so it fell through to ErrorCode::General and the
feature_flag.evaluations metric reported error.type=general instead of
parse_error, failing the FEATURE_FLAGGING_AND_EXPERIMENTATION system tests.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Restore the bytes 1.10+ comment and the original ordering of url and
libdd-http-client that were cosmetically changed during the v37 bump.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
… arm

Both map to PARSE_ERROR; combine them into a single match arm.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
…client

The code review suggestion incorrectly reverted the import back to
SharedRuntime (a v37 trait), but the struct holds Arc<ForkSafeRuntime>
and block_on requires BlockingRuntime in scope.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
…ew()

ForkSafeRuntime::new() is a trait method from SharedRuntime; without the
trait in scope the compiler cannot resolve it.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
With collect_all_threads enabled (default in v37), the receiver uses
PTRACE_SEIZE + PTRACE_INTERRUPT on the crashing thread, which interrupts
poll() in wait_for_pollhup with EINTR. When crashing from a dynamically-
loaded C++ extension on Python 3.14, repeated EINTR prevents the 5-second
timeout from expiring, causing an indefinite hang.

Consistent with the existing pattern of Python 3.14 crashtracker skips
("Stack v2 not supported on 3.14") in 9d9bd80.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
…d from ptrace

The receiver in v37 began ptracing the crashing thread
(crashing_tid) as part of collect_all_threads. That thread is blocked
in poll() inside its signal handler on the alternate signal stack; the
resulting PTRACE_INTERRUPT issues EINTR in a loop, causing the receiver
to hang indefinitely on CPython 3.14.

Pin all libdatadog deps to commit 504ea9d7 on the fix branch, which
restores the v35 behaviour of excluding crashing_tid from
stream_thread_contexts. With the fix in place the 3.14 skip is no
longer needed.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
@zacharycmontoya
zacharycmontoya requested review from emmettbutler and removed request for a team July 10, 2026 00:45

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 921244515e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread ddtrace/internal/settings/_agent.py Outdated
Comment thread ddtrace/_trace/tracer.py
Comment thread ddtrace/internal/opentelemetry/span.py

@KowalskiThomas KowalskiThomas left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good for apm-python.

Base automatically changed from munir/bump-libdatadog-v37 to main July 10, 2026 20:38
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot requested a review from a team as a code owner July 10, 2026 20:38
Comment thread releasenotes/notes/otel-trace-semantics-f655af74e566c719.yaml Outdated
Comment thread src/native/Cargo.toml
Comment thread ddtrace/_trace/tracer.py
… 'return True' path is only hit when OTLP trace export is enabled AND there's no user-configured DD Trace protocol version.

@link04 link04 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, aside from that not sure how Python is about the nits I added.

Comment thread supported-configurations.json Outdated
Comment thread ddtrace/internal/settings/_agent.pyi Outdated

@emmettbutler emmettbutler left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

all of the changes to _trace and internal look reasonable

@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot merged commit 6a31064 into main Jul 21, 2026
1245 checks passed
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot deleted the zach.montoya/otel-trace-compatibility branch July 21, 2026 19:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants