OTLP trace metrics export by ida613 · Pull Request #6834 · DataDog/system-tests

ida613 · 2026-05-01T17:47:46Z

Motivation

Datadog tracing libraries can export client-computed span aggregate metrics in OTLP format as a histogram (traces.span.sdk.metrics.duration), complementing OTLP trace export. This is the basis for RED metrics (Requests / Errors / Duration) derived from the histogram:

Hits = count{*}
Errors = count{error:true}
TopLevelHits = count{datadog.span.top_level:true}

This PR adds black-box parametric coverage for that feature: parametric apps run with various configurations, spans are generated, and the resulting OTLP metrics (and native v0.6 stats, where relevant) are captured and asserted on. It lands the test cases called out in the RFC (SEMCON-1093).

What the tests validate

tests/parametric/test_otlp_trace_metrics.py — 42 tests grouped by functional requirement, under the PARAMETRIC scenario (@features.client_side_stats_supported):

Req	Area	Tests
FR01	Enablement & configuration (explicit/default on/off, gated by metrics and OTLP trace export)	5
FR02	Metric identity (single named histogram, no native/SMC names) & mutual exclusion with native v0.6 stats (incl. `Datadog-Client-Computed-Stats` header)	4
FR03	Metric shape — delta-temporality histogram, unit `s`	2
FR04	Span selection — only top-level or measured spans produce data points	2
FR05	Sampling independence — metrics computed before head-based sampling, from 100% of spans	1
FR06	OpenTelemetry span attributes (http/rpc/status/kind) and resource attributes — service identity on the resource, `service.name` on a data point when a span's service differs from the default, hostname, telemetry SDK	14
FR07	OTel-semantics mode (`DD_TRACE_OTEL_SEMANTICS_ENABLED`) — only OpenTelemetry attributes, no `datadog.`/`dd.`	4
FR08	Datadog attributes (default mode) — operation name, span type, top-level, origin, process tags	8
FR09	RED metric derivation — single-point consistency and error count	2

End-state focus: each test exercises one behaviour and asserts on the final exported payload (not on transitional state). Assertions are shape-tolerant where a value is library-dependent — FR03 checks unit, delta temporality and 0 < sum < 60; FR09 checks count, min == max and sum(bucketCounts) for a single span. Flush cadence is fixed at 10s and shortened in tests via the internal _DD_TRACE_METRICS_OTEL_FLUSH_INTERVAL (ms). Tests pin HTTP/JSON export, the transport common to all libraries.

Cross-tracer parity (documented, not asserted): sum/min/max are exact in dd-trace-js (sketch scalars) but sketch-approximated in libdatadog/dd-trace-py; host.name source differs (libdatadog honors DD_HOSTNAME, dd-trace-js uses os.hostname()), so the hostname test asserts presence only.

Activation

The SDK implementations ship in future library releases, so these tests cannot pass against currently released libraries in CI. They are therefore skipped for all languages via a module-level missing_feature entry in every manifest. python and nodejs additionally keep their per-test skips (library-specific gaps such as nodejs native v0.6 stats); leaving those in place means re-enabling the suite once the feature ships is simply removing the single module-level line.

Supporting changes

utils/build/docker/nodejs/parametric/server.js, utils/build/docker/nodejs/install_ddtrace.sh — parametric test-client support for the OTLP trace-metrics path.
utils/docker_fixtures/_test_agent.py — polling helpers (wait_for_num_otlp_metrics, wait_for_num_v06_stats) to accommodate OTLP/native export latency.

What to review

The manifest activation strategy (module-level skip + retained per-test skips) and that re-enabling is a one-line change per language.
That the assertions are shape-tolerant for library-dependent values yet still fail loudly with the actual payload captured in the failure message.
The OTel semantic-convention attribute mapping and the default vs OTel-semantics mode boundaries (FR06/FR07/FR08).

Reviewer checklist

Anything but `tests/` or `manifests/` is modified ? I have the approval from R&P team
A docker base image is modified?
- the relevant `build-XXX-image` label is present
A scenario is added, removed or renamed?
- Get a review from R&P team

…te injection Phase 2 — OTLP trace metrics export: tests/otel/test_otlp_trace_metrics.py validates that all four dd.trace.span.* metrics appear on /v1/metrics when DD_TRACE_OTEL_METRICS_ENABLED=true (new APM_TRACING_OTLP_METRICS scenario). Phase 3 — W3C TraceState ot.th injection: tests/parametric/test_tracestate_ot_th.py validates that an ot=th:<hex> vendor entry is present in the outgoing tracestate for sampled spans. Supporting changes: - Add apm_tracing_otlp_metrics scenario (scenarios/__init__.py) - Add get_otel_metrics() to OpenTelemetryInterfaceValidator (interfaces) - Add otlp_trace_metrics and w3c_tracestate_ot_probability_threshold features - Mark both tests as missing_feature in manifests/nodejs.yml Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

github-actions · 2026-05-01T17:48:09Z

CODEOWNERS have been resolved as:

tests/parametric/test_otlp_trace_metrics.py                             @DataDog/system-tests-core @DataDog/apm-sdk-capabilities
manifests/cpp.yml                                                       @DataDog/dd-trace-cpp
manifests/dotnet.yml                                                    @DataDog/apm-dotnet @DataDog/asm-dotnet
manifests/golang.yml                                                    @DataDog/dd-trace-go-guild
manifests/java.yml                                                      @DataDog/asm-java @DataDog/apm-java
manifests/nodejs.yml                                                    @DataDog/dd-trace-js
manifests/php.yml                                                       @DataDog/apm-php @DataDog/asm-php
manifests/python.yml                                                    @DataDog/apm-python @DataDog/asm-python
manifests/ruby.yml                                                      @DataDog/ruby-guild @DataDog/asm-ruby
manifests/rust.yml                                                      @DataDog/apm-rust
utils/build/docker/nodejs/install_ddtrace.sh                            @DataDog/dd-trace-js @DataDog/system-tests-core
utils/build/docker/nodejs/parametric/server.js                          @DataDog/dd-trace-js @DataDog/system-tests-core
utils/docker_fixtures/_test_agent.py                                    @DataDog/system-tests-core

ot.th injection adds an extra vendor entry to the tracestate, which changes the expected member count in several parametric tests. Mark those tests as bugs until the 32-member eviction logic is implemented. Also remove the missing_feature marker for test_tracestate_ot_th since ot.th is now implemented in the nodejs tracer. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

Single dd.trace.span.duration histogram replaces the previous four metrics. Data points are split by (error, dd.top_level) matrix; error attr only added when error=true. Rename DD_TRACE_OTEL_METRICS_ENABLED to DD_TRACE_OTEL_STATS_COMPUTATION_ENABLED in scenario env. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

…RFC" This reverts commit 5f1f866. Revert "chore(nodejs): mark tests affected by ot.th tracestate injection" This reverts commit 21842f2. Revert "test(nodejs): add system tests for OTLP span stats and ot.th tracestate injection" This reverts commit 4d9c1cb.

Adds a parametric test that enables client-computed span stats and verifies the tracer exports the dd.trace.span.duration histogram (delta temporality) to the test agent's OTLP HTTP receiver with the expected resource attributes. Co-authored-by: Cursor <[email protected]>

datadog-datadog-prod-us1-2 · 2026-06-01T16:40:40Z

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: c35abb2 | Docs | Datadog PR Page | Give us feedback!}

Co-authored-by: Munir Abdinur <[email protected]>

- Restructure tests around traces.span.sdk.metrics.duration and the final FR -> test-class mapping; drive export off OTel config (OTEL_METRIC_EXPORT_INTERVAL). - Assert telemetry.sdk.name/telemetry.sdk.language resource attributes and relax the process-tag check to any one known dd.<key> tag. - Use test_agent.get_v06_stats_requests() (tracestats convention) for the native-stats XOR check; drop the bespoke poller. - Fix install_ddtrace.sh to keep only the .tgz path from `bun pm pack` output. - Drop stale per-test nodejs manifest skips that referenced renamed classes. Co-authored-by: Cursor <[email protected]>

Enable tests/parametric/test_otlp_trace_metrics.py for nodejs now that dd-trace-js implements the trace-metrics contract, skipping the three cases nodejs cannot satisfy: sampling independence, DD_HOSTNAME-based host.name, and different-service top-level tagging. Co-authored-by: Cursor <[email protected]>

…dejs test_fr02_4_native_stats_no_otlp exercises the native /v0.6/stats path (OTLP disabled, native enabled), which nodejs has not implemented (consistent with test_library_tracestats). Co-authored-by: Cursor <[email protected]>

… cadence Move service identity assertions to InstrumentationScope attributes (service.name, service.version, deployment.environment.name), add a multi-service partitioning case, drive timing via the internal _DD_TRACE_METRICS_OTEL_FLUSH_INTERVAL, and enable the suite for python. Co-authored-by: Cursor <[email protected]>

Update the OTLP trace-metrics parametric tests to the corrected attribute model: service.name/service.version/deployment.environment.name are resource attributes (configured default service), all data points share a single InstrumentationScope, and a span whose service differs from the default carries service.name on its data point. Replaces the per-service scope partitioning assertions. Co-authored-by: Cursor <[email protected]>

The OTLP trace-metrics payload no longer carries a `dd-trace` InstrumentationScope (redundant with the telemetry.sdk.* resource attributes). Update the spec docstring and FR06 tests to assert no scope is emitted. Co-authored-by: Cursor <[email protected]>

…RICS_ENABLED Update OTLP trace-metrics assertions to the datadog.*/_datadog.* attribute prefix and rename the enablement env var to OTEL_TRACES_SPAN_METRICS_ENABLED. Co-authored-by: Cursor <[email protected]>

datadog.* is now the required Datadog attribute prefix, so the guard asserts no short dd.*-prefixed attributes leak instead. Co-authored-by: Cursor <[email protected]>

…DK release; simplify end-state assertions Add a module-level missing_feature skip for test_otlp_trace_metrics.py in python.yml and nodejs.yml (existing per-test skips retained so re-enabling is a one-line change once the feature ships). Remove the silently-passing fixed-bucket-layout test (its guard no-oped when a library emitted no bounds) and its dead constant, condense docstrings, and add failure messages that capture the actual payload state for easier debugging. Co-authored-by: Cursor <[email protected]>

Revert the cosmetic docstring/comment rewordings from the previous commit so the change stays focused on its purpose. The functional improvements are retained: the silently-passing fixed-bucket-layout test stays removed and the assertion failure messages that capture the actual payload are kept. Co-authored-by: Cursor <[email protected]>

Re-add test_fr09_3_fixed_bucket_layout (and EXPLICIT_BOUNDS_SECONDS) so it now requires explicitBounds to be present and match the fixed spanmetrics layout instead of no-opping when bounds are absent. Trim obvious narrating comments and drop docstring references to the removed InstrumentationScope checks. Co-authored-by: Cursor <[email protected]>

…ource attribute Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

zacharycmontoya · 2026-07-07T22:51:59Z

+        [
+            {
+                **_BASE_ENVVARS,
+                "OTEL_TRACES_SPAN_METRICS_ENABLED": "false",


What happens if both are true? Which one wins? This could use a test case

Or maybe it's worth setting the env variable OTEL_TRACES_SPAN_METRICS_ENABLED=true in the test_fr02_3_otlp_suppresses_native_stats test case if OTLP wins

zacharycmontoya · 2026-07-07T23:03:19Z

+        "library_env",
+        [{**_BASE_ENVVARS, "OTEL_TRACES_EXPORTER": "none", "DD_METRICS_OTEL_ENABLED": "true"}],
+    )
+    def test_fr01_5_disabled_when_traces_exporter_not_otlp(


The test name describes the scenario as "traces_exporter_not_otlp" but that is already tested in test_fr01_1_enabled_explicit. I suggest renaming this because what you're actually doing is disabling traces altogether with OTEL_TRACES_EXPORTER=none. You could also add a test that disables tracing via the DD environment variable, so DD_APM_TRACING_ENABLED=false

Suggested change

def test_fr01_5_disabled_when_traces_exporter_not_otlp(

def test_fr01_5_disabled_when_tracing_is_disabled(

zacharycmontoya · 2026-07-07T23:07:07Z

+
+        metrics = test_agent.wait_for_num_otlp_metrics(num=1)
+        attrs = _data_point_attrs(_duration_data_points(metrics)[0])
+        assert attrs.get("span.kind") == "server", f"Expected span.kind=server, got attrs: {attrs}"


Should the value of the span.kind attribute match the casing of span metrics connector (in this case SERVER)? See the README here

zacharycmontoya · 2026-07-07T23:13:11Z

+        metrics = test_agent.wait_for_num_otlp_metrics(num=1)
+        attrs = _data_point_attrs(_duration_data_points(metrics)[0])
+        assert attrs.get("http.response.status_code") == 200, (
+            f"Expected http.response.status_code == 200 (typed int), got attrs: {attrs}"


Are we mixing and matching attribute types? Or should all attributes on the time-series have string values?

FWIW, I asked Claude to take a look at the Span Metrics Connector to understand how the 'dimensions' configuration copies span attributes to the resulting time series. If an attribute with the matching name exists on the span, then it is copied as-is to the timeseries, so an example of ip.port with an IntValue would be copied as an IntValue onto the resulting metrics timeseries

zacharycmontoya · 2026-07-07T23:25:13Z

+        ), f"Expected deployment environment=prod, got: {resource_attrs}"
+
+        # The span uses the configured default service, so its data point omits service.name.
+        assert SERVICE not in _data_point_services(metrics), (


I think we should always emit service.name on the time series as a metric attribute, since that follows behavior of SMC. I think OpenTelemetry users would expect the metric itself to have all the metric attributes to fully describe the spans, which includes the service name

zacharycmontoya · 2026-07-07T23:31:51Z

+
+        metrics = test_agent.wait_for_num_otlp_metrics(num=1)
+        attrs = _data_point_attrs(_duration_data_points(metrics)[0])
+        assert attrs.get("span.name") == "/users", f"Expected span.name=/users, got attrs: {attrs}"


If we want to add assertions for datadog.resource.name, we should also assert that it's not included here. Otherwise, we can remove all assertions on this name since it's no longer mentioned in the RFC

Suggested change

assert attrs.get("span.name") == "/users", f"Expected span.name=/users, got attrs: {attrs}"

assert attrs.get("span.name") == "/users", f"Expected span.name=/users, got attrs: {attrs}"

assert "datadog.resource.name" not in attrs, f"datadog.resource.name must be absent: {attrs}"

zacharycmontoya · 2026-07-07T23:38:22Z

+
+        metrics = test_agent.wait_for_num_otlp_metrics(num=1)
+        attrs = _data_point_attrs(_duration_data_points(metrics)[0])
+        assert attrs.get("http.request.method") == "GET", f"Expected http.request.method=GET, got attrs: {attrs}"


By default, the span metrics connector only emits the following attributes:

service.name

span.name

span.kind

status.code

(feature-gate) collector.instance.id

(feature-gate) otel.status_code

In our DD_TRACE_OTEL_SEMANTICS_ENABLED mode we are emitting additional fields like env and these HTTP ones. Is the intention that even in OTel semantics mode we emit the full set of attributes our own stats computation needs? If so, can we consider a configuration flag to not emit a user-specified set of attribute keys?

…-dotnet#8921 Replaces the module-level missing_feature skip with v3.50.0 (the version that first shipped OTLP trace-metrics export via #8826) so CI runs the parametric suite added in #6834 against a specific dd-trace-dotnet branch via the PR title branch override, to check that the PR removing the OtelTracesSpanMetricsEnabled mixed-mode path in TracerManagerFactory still satisfies the FR01 enablement tests. Co-Authored-By: Claude Sonnet 5 <[email protected]>

ida613 and others added 2 commits May 1, 2026 14:18

mabdinur changed the title ~~[WIP] OTLP trace metrics export~~ OTLP trace metrics export Jun 1, 2026

mabdinur and others added 2 commits June 1, 2026 12:38

mabdinur added 4 commits June 1, 2026 12:44

add skips and lint

c538768

extend tests

0758720

address gaps in tests

505a784

enable py tests and clean up broken child spans test

f66920f