[OTLP] Add trace metrics via OTLP span metrics export#8826
Merged
Conversation
github-actions
Bot
added
the
area:tracer
Execution-Time Benchmarks Report ⏱️Execution-time results for samples comparing This PR (8826) and master. ✅ No regressions detected - check the details below Full Metrics ComparisonFakeDbCommand
HttpMessageHandler
Comparison explanationExecution-time benchmarks measure the whole time it takes to execute a program, and are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are highlighted in **red**. The following thresholds were used for comparing the execution times:
Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard. Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph). Duration chartsFakeDbCommand (.NET Framework 4.8)gantt
title Execution time (ms) FakeDbCommand (.NET Framework 4.8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8826) - mean (72ms) : 68, 76
master - mean (71ms) : 68, 73
section Bailout
This PR (8826) - mean (75ms) : 71, 78
master - mean (78ms) : 74, 83
section CallTarget+Inlining+NGEN
This PR (8826) - mean (1,081ms) : 1035, 1127
master - mean (1,085ms) : 1039, 1131
FakeDbCommand (.NET Core 3.1)gantt
title Execution time (ms) FakeDbCommand (.NET Core 3.1)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8826) - mean (112ms) : 105, 119
master - mean (112ms) : 105, 119
section Bailout
This PR (8826) - mean (110ms) : 108, 112
master - mean (113ms) : 108, 117
section CallTarget+Inlining+NGEN
This PR (8826) - mean (774ms) : 752, 795
master - mean (774ms) : 751, 797
FakeDbCommand (.NET 6)gantt
title Execution time (ms) FakeDbCommand (.NET 6)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8826) - mean (100ms) : 95, 105
master - mean (101ms) : 95, 106
section Bailout
This PR (8826) - mean (99ms) : 94, 103
master - mean (99ms) : 94, 103
section CallTarget+Inlining+NGEN
This PR (8826) - mean (941ms) : 901, 980
master - mean (939ms) : 894, 985
FakeDbCommand (.NET 8)gantt
title Execution time (ms) FakeDbCommand (.NET 8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8826) - mean (94ms) : 92, 97
master - mean (95ms) : 92, 97
section Bailout
This PR (8826) - mean (99ms) : 94, 104
master - mean (99ms) : 94, 104
section CallTarget+Inlining+NGEN
This PR (8826) - mean (814ms) : 780, 848
master - mean (814ms) : 775, 853
HttpMessageHandler (.NET Framework 4.8)gantt
title Execution time (ms) HttpMessageHandler (.NET Framework 4.8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8826) - mean (202ms) : 197, 207
master - mean (203ms) : 196, 210
section Bailout
This PR (8826) - mean (205ms) : 201, 209
master - mean (207ms) : 202, 212
section CallTarget+Inlining+NGEN
This PR (8826) - mean (1,214ms) : 1160, 1268
master - mean (1,222ms) : 1177, 1268
HttpMessageHandler (.NET Core 3.1)gantt
title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8826) - mean (289ms) : 282, 296
master - mean (291ms) : 284, 299
section Bailout
This PR (8826) - mean (290ms) : 286, 295
master - mean (292ms) : 283, 301
section CallTarget+Inlining+NGEN
This PR (8826) - mean (972ms) : 947, 997
master - mean (977ms) : 947, 1007
HttpMessageHandler (.NET 6)gantt
title Execution time (ms) HttpMessageHandler (.NET 6)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8826) - mean (286ms) : 280, 291
master - mean (285ms) : 278, 291
section Bailout
This PR (8826) - mean (284ms) : 277, 291
master - mean (284ms) : 278, 290
section CallTarget+Inlining+NGEN
This PR (8826) - mean (1,166ms) : 1134, 1199
master - mean (1,172ms) : 1134, 1209
HttpMessageHandler (.NET 8)gantt
title Execution time (ms) HttpMessageHandler (.NET 8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8826) - mean (282ms) : 276, 287
master - mean (283ms) : 276, 290
section Bailout
This PR (8826) - mean (281ms) : 276, 285
master - mean (283ms) : 276, 290
section CallTarget+Inlining+NGEN
This PR (8826) - mean (1,044ms) : 1006, 1081
master - mean (1,051ms) : 1002, 1101
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
BenchmarksBenchmark execution time: 2026-07-09 14:57:11 Comparing candidate commit ebd7efd in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 72 metrics, 0 unstable metrics, 60 known flaky benchmarks, 66 flaky benchmarks without significant changes.
|
mabdinur
force-pushed
the
munir/otlp-span-metrics
branch
from
June 25, 2026 19:01
0876bf1 to
5f901eb
Compare
2 tasks
mabdinur
marked this pull request as ready for review
June 25, 2026 21:59
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 00915b660f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
mabdinur
force-pushed
the
munir/otlp-span-metrics
branch
from
June 26, 2026 14:55
00915b6 to
74cd1de
Compare
link04
reviewed
Jun 26, 2026
Add OTEL_TRACES_SPAN_METRICS_ENABLED (tri-state gate) and DD_TRACE_OTEL_SEMANTICS_ENABLED config keys. Extend StatsBucket with exact MinDuration/MaxDuration for OTLP histogram min/max fields (DDSketch approximation is insufficient). Wire TracerManagerFactory to create an OTLP-backed StatsAggregator when span metrics are enabled. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
…duration) Introduces OtlpSpanStatsSerializer with SerializeJson: converts a StatsBuffer into an OTLP ExportMetricsServiceRequest (proto3 JSON). Projects DDSketch bins onto 16 fixed explicit bounds (seconds). Uint64 fields (timestamps, counts) are encoded as quoted strings per the proto3 JSON spec. Attribute set per data point: span.name, span.kind, http.request.method, http.response.status_code, http.route, rpc.response.status_code, status.code (errors only), service.name (when non-default). Datadog-specific attributes (datadog.operation.name, datadog.span.type, etc.) suppressed when DD_TRACE_OTEL_SEMANTICS_ENABLED. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Extends OtlpSpanStatsSerializer with a binary-protobuf path (Serialize): hand-rolls varint/fixed64/length-delimited encoding against the OTLP proto schema — no proto codegen dependency. Adds attribute helpers (WriteAttribute, WriteIntAttribute, WriteBoolAttribute) and FieldNumbers/AnyValueFieldNumbers constants. Wires ApiOtlp.SendStatsAsyncImpl to serialize and POST to the OTLP metrics endpoint. Protocol selected by OTEL_EXPORTER_OTLP_METRICS_PROTOCOL (http/protobuf default, http/json for JSON). Custom headers from OTEL_EXPORTER_OTLP_METRICS_HEADERS applied per request. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Mirror the Datadog-Client-Computed-Stats header (AgentWriter.CanComputeStats) on the OTLP trace path so a downstream Datadog Agent does not recompute stats. Co-authored-by: Cursor <[email protected]>
…ve stale comments Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
- Add #nullable enable (CS8632) - Move const before readonly field (SA1203) - Move private CreateBuffer/CreateKey methods after public [Fact] methods (SA1202) - Put each StatsAggregationKey constructor arg on its own line (SA1117) Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Add OTEL_TRACES_SPAN_METRICS_ENABLED and DD_TRACE_OTEL_SEMANTICS_ENABLED to config_norm_rules.json so AllConfigurationValuesAreRegisteredWithIntake passes. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
optional uint64 count and repeated uint64 bucket_counts are varint fields (wire type 0) in the OTLP HistogramDataPoint proto — not fixed64 (wire type 1). Using WireTypeFixed64 produced tag bytes 33/49 instead of 32/48, causing conforming decoders to treat them as unknown fields and drop them. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Change 1: rpc.response.status_code emitted as canonical string name
- Add GrpcStatusNames[0..16] lookup table
- Add NormalizeGrpcStatusName: handles numeric codes, named codes,
StatusCode.* prefix, CANCELED alias, NOTFOUND alias
- Omit attribute when empty/unparseable/out-of-range
Change 2: rpc.method explicitly not emitted (remove TODO comment)
- gRPC method is already the span resource/span.name; separate dimension
would inflate cardinality for no benefit
Change 3: confirm true top-level counts (already implemented)
- OTLP aggregation key already uses span.IsTopLevel, so top-level and
non-top-level spans with identical other attributes bucket separately
- Remove misleading "derived as all-spans-top-level" comment
Add tests covering all three changes.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
mabdinur
force-pushed
the
munir/otlp-span-metrics
branch
from
July 1, 2026 23:15
a183e87 to
bb159a5
Compare
…ndant comments Decorative ── section headers were unique to OtlpSpanStatsSerializer — not used elsewhere in the codebase. Multi-line XML docstrings on public methods and the EncodeKeyValue helper explain what the code does, which is already clear from names and signatures. Per-attribute comments in SerializeDataPoint duplicated the string literals they sat above. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
The metrics request factory was previously built from OtlpTransportStrategy.GetTraces, which uses traces transport config (endpoint, protocol, headers, timeout). Add GetMetrics() so the metrics factory respects OTEL_EXPORTER_OTLP_METRICS_* settings, and stop manually adding metrics headers per-request since they're now DefaultHeaders on the HttpClient. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
…ctory Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
StatsBucket, StatsBuffer, and StatsAggregationKey constructors gained new required parameters in master; update test call sites to match. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
| } | ||
| } | ||
|
|
||
| public static byte[]? Serialize(StatsBuffer buffer, long bucketDurationNs, bool otelSemanticsEnabled) |
Contributor
There was a problem hiding this comment.
TODO for us dd-trace-dotnet maintainers: We should pass a Stream into here, rather than this method creating its own stream and allocating a byte[] as the return value
Contributor
There was a problem hiding this comment.
This pattern should also be updated in OtlpMetricsSerializer.cs where this pattern was introduced
Contributor
There was a problem hiding this comment.
Also, as a follow-up, the protobuf serialization should use the same logic as the JSON serialization so we can easily keep the two in-sync
zacharycmontoya
left a comment
zacharycmontoya
left a comment
Contributor
There was a problem hiding this comment.
Once the sub-second precision is restored (and assuming the system-tests for this feature pass), I think this can be approved. Please ping me when that's ready
…ds; MinDuration/MaxDuration as long - NanosecondsToSeconds now divides by 1_000_000_000.0 (double) to preserve sub-second precision instead of truncating via integer division - MinDuration/MaxDuration changed from double to long in StatsBucket, matching Duration; sentinel changed from double.MaxValue to long.MaxValue - Remove now-unnecessary casts in OtlpSpanStatsSerializer - Update TimeUtilsTests to assert sub-second precision - Update SerializeJson_MinMaxWritten_WhenSet to use 1ms/50ms values Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Contributor
Author
|
Sub-second precision restored in 28a3656. Ready for another look @zacharycmontoya! |
|
|
||
| var api = new ManagedApi(settings.Manager, statsd, updateSampleRates, updateConfigHash, settings.PartialFlushEnabled); | ||
|
|
||
| if (settings.OtelTracesSpanMetricsEnabled) |
Contributor
There was a problem hiding this comment.
This block supports sending traces via MsgPack and also sending OTLP span metrics. Is this a supported scenario? If we're only supporting OTLP span metrics when exporting traces as OTLP then we don't need this block as it's covered in the first block
Contributor
Author
There was a problem hiding this comment.
This block handles OTEL_TRACES_SPAN_METRICS_ENABLED=true with MsgPack traces. We kept it because it simplifies testing and reproducibility — setting the env var to true always does what you'd expect regardless of trace encoding. Happy to remove it if you'd prefer to tie span metrics strictly to OTLP traces.
Contributor
There was a problem hiding this comment.
in this case, should we merge it with the first block with a
if ((settings.Manager.InitialExporterSettings.TracesEncoding is TracesEncoding.OtlpProtobuf or TracesEncoding.OtlpJson)
|| settings.OtelTracesSpanMetricsEnabled)
Contributor
There was a problem hiding this comment.
Actually the proposed approach is correct:
var api = new ManagedApi()ensures that we send DD tracesvar otlpSpanMetricsAggregator = StatsAggregator.Create(otlpMetricsApi, ...)ensures we generate OTLP trace metrics
Merging with the first block would unconditionally send OTLP traces, so we wouldn't be able to send DD traces
zacharycmontoya
approved these changes
Jul 8, 2026
zacharycmontoya
left a comment
zacharycmontoya
left a comment
Contributor
There was a problem hiding this comment.
LGTM! I have a couple of comments so please take a look at those before merging. Also please wait for a 2nd approval before merging, thanks!
anna-git
reviewed
Jul 8, 2026
|
|
||
| for (int i = 1; i < BoundsNs.Length; i++) | ||
| { | ||
| if (valueNs < BoundsNs[i]) |
Contributor
There was a problem hiding this comment.
NIT: shouldn't it be <= here and l.617 as otel histogram upper bounds are inclusive? Also https://github.com/open-telemetry/opentelemetry-dotnet/blob/e60be99c3769fa1ff5c2dbca8cc841b6cb632cf4/src/OpenTelemetry/Metrics/HistogramExplicitBounds.cs#L46-L81
Contributor
Author
There was a problem hiding this comment.
Fixed in a127a0c — both comparisons in FindBucketIndex now use <=.
| When unset, auto-enables iff OTEL_TRACES_EXPORTER=otlp and DD_METRICS_OTEL_ENABLED=true. | ||
| When true, always enables span metrics export regardless of other settings. | ||
| When false, always disables span metrics export. | ||
| DD_TRACE_OTEL_SEMANTICS_ENABLED: |
Contributor
There was a problem hiding this comment.
NIT: could we keep the alphabetical order here
Contributor
Author
There was a problem hiding this comment.
Fixed in a127a0c — moved OTEL_TRACES_SPAN_METRICS_ENABLED to after OTEL_TRACES_SAMPLER_ARG and DD_TRACE_OTEL_SEMANTICS_ENABLED to its correct position near DD_TRACE_OTEL_ENABLED.
| counts[0] += (ulong)Math.Round(sketch.ZeroCount); | ||
| } | ||
|
|
||
| foreach (var bin in sketch.PositiveValueStore.EnumerateAscending()) |
Contributor
There was a problem hiding this comment.
NegativeValueStore is never read, maybe it will never happen, but maybe we could have some clock issue edge cases that we could log somehow?
Contributor
Author
There was a problem hiding this comment.
Added a debug log in ProjectSketch when NegativeValueStore is non-empty (a127a0c). Clock skew edge cases will now surface in logs.
…s; sort YAML; log negative durations - Restore GetWeight and weighted Hits/Errors/Duration/TopLevelHits in StatsAggregator; only MinDuration/MaxDuration tracking is new - Revert StatsBucket Hits/Errors/Duration/TopLevelHits back to double (accumulate fractional sampling weights); MinDuration/MaxDuration stay long - Fix FindBucketIndex to use <= per OTel inclusive upper-bound spec - Add debug log in ProjectSketch when NegativeValueStore is non-empty - Move OTEL_TRACES_SPAN_METRICS_ENABLED and DD_TRACE_OTEL_SEMANTICS_ENABLED to correct alphabetical positions in supported-configurations.yaml Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
- Move GetWeight above ConvertTimestamp XML doc (SA1612) - Cast double Hits/Errors/Duration/TopLevelHits to long for WriteInt64 in StatsBuffer - Use GetTotalCount() instead of TotalCount on vendored Store Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Contributor
Author
|
Fixed build errors in 68a5950:
|
link04
reviewed
Jul 8, 2026
link04
left a comment
link04
left a comment
Contributor
There was a problem hiding this comment.
One last comment I got from the AI after having it look into related stats PRs and recent OTLP metrics changes was performance, for the OtlpSpanStatsSerializer.cs class serialize methods(general, e.g. lines 374-602), it found that:
Nit: lots of small buffer allocations in the serializer
This file allocates a new MemoryStream + BinaryWriter for nearly every field it writes — protobuf needs a field's byte length before writing it, so the pattern here is: write to a temp buffer, measure it, then copy into the real output. Functionally fine, but it's a lot of small allocations for code that runs on every stats-flush interval.
So I would say take a look and see if anything can be improved based on the existing serializers or nah.
Not blocking — if flush intervals are short or there are many distinct stat buckets, a pooled/reusable buffer would reduce GC pressure.
Aside from the above if you think the first comment is relevant to apply spec wise or not, let me know then we could merge if the change is worth it since the AI flag it as request changes worth.
zacharycmontoya
requested changes
Jul 8, 2026
zacharycmontoya
left a comment
zacharycmontoya
left a comment
Contributor
There was a problem hiding this comment.
Updating my review to 'Request changes' so we don't accidentally merge the sampling-based weighting in the most recent changes. Once that's reverted I'll update my review back to 'Approve'
… sentinel - Remove GetWeight and revert to unweighted Hits/Errors/Duration/TopLevelHits, aligning with the intentional removal in PR #8828 - Revert StatsBucket fields back to long (no fractional weights needed) - Remove (long) casts in StatsBuffer now that fields are long again - Fix MaxDuration sentinel: use long.MinValue instead of 0 so zero-duration spans are correctly recorded and emitted - Remove (long) cast on bucket.Duration in serializer (field is long again) Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Covers the long.MinValue sentinel fix: a span with duration 0 must emit min=0 and max=0, distinct from the "never observed" sentinel. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Reverts the GlobalTags/DD_TAGS resource-attribute emission added to the OTLP span metrics serializer (commit 1dd837e). Removes both the JSON and protobuf tag loops, the TryParseTag helper, and the now-unused OpenTelemetry using. The shared DdTags infrastructure (ClientStatsPayload, StatsBuffer MsgPack path) is untouched. This will be reintroduced in a future PR. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
zacharycmontoya
approved these changes
Jul 9, 2026
zacharycmontoya
left a comment
zacharycmontoya
left a comment
Contributor
There was a problem hiding this comment.
LGTM, thanks for the updates!
link04
approved these changes
Jul 9, 2026
zacharycmontoya
added a commit
that referenced
this pull request
Jul 14, 2026
…race compatibility (#8796) ## Summary of changes Adds an opt-in OpenTelemetry semantics mode for OTLP trace export. This feature is enabled by setting `DD_TRACE_OTEL_SEMANTICS_ENABLED=true`, which was previously introduced in #8826. When enabled, spans produced from OpenTelemetry instrumentation are serialized closer to native OpenTelemetry SDK output by preserving OTel attribute keys and value types, and by suppressing Datadog-specific attributes from OTLP trace payloads. ## Reason for change We should support scenarios where customers use the Datadog SDK but want their emitted OTLP data to look as close as possible to data produced by the OpenTelemetry SDK. This helps users route Datadog-instrumented applications through OTel collectors or other OTLP consumers without the payload including Datadog-specific span attributes that are not part of the OpenTelemetry semantic conventions. ## Implementation details Refactors - Refactors the existing `SetExceptionTags` flow into `SetException(exception, markAsError)` so callers can add exception metadata without forcing the span error flag. Instrumentation changes: - Propagates the setting to `OtlpHelpers` so, when enabled, the Activity-to-span conversion does not do any attribute remapping or special handling for Datadog-specific attributes, preserving the original attribute keys and values on the OTLP span. - This also stops us from adding deprecated attributes such as `otel.trace_id`, `otel.library.name`, `otel.library.version`, and `otel.status_code` to match newer OTel SDK behaviors. Span changes: - Propagates the setting into a Span field so, when enabled, exceptions are recorded as the OpenTelemetry `exception` span event with `exception.type`, `exception.message`, and `exception.stacktrace` attributes instead of Datadog error tags. OTLP Exporter changes - Updates `OtlpMapper.EmitAttributesFromSpan` so OTel semantics mode suppresses Datadog-specific or compatibility-only span attributes, including `service.name`, `operation.name`, `resource.name`, `span.type`, `component`, `span.kind`, `service.version`, `service.instance.id`, `otel.status_code`, and `http-client-handler-type`. Test changes - Stabilizes the OTLP trace integration test by polling the test agent for trace data after sample shutdown. ## Test coverage - Adds unit coverage for `OtlpMapper` to assert Datadog attributes are emitted by default and suppressed when OTel semantics mode is enabled. - Adds unit coverage for preserving `http.status_code` and `http.response.status_code` as OTel attributes when OTel semantics mode is enabled. - Adds unit coverage for the new `Span.SetException` behavior, including exception span event creation under OTel semantics mode. - Updates OTLP serializer tests for the new serializer configuration path. - Expands `OpenTelemetrySdkTests.SubmitsOtlpTraces` to run with `DD_TRACE_OTEL_SEMANTICS_ENABLED` enabled and disabled for both `http/json` and `http/protobuf`, adding a new `OpenTelemetrySdkTests.SubmitsOtlpTraces_DD_OtelSemantics.verified.txt` snapshot file. - Miscellaneous: Updates out-of-date OpenTelemetrySdkTests snapshot files with the JSON/Protobuf normalizations ## Other details N/A
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
RFC: https://docs.google.com/document/d/1cDB_NVkyY_DXzb_EfQS6BsGu5Rx3gnf-tbpgJCYbIkc/edit?pli=1&tab=t.0
Overview
Adds the ability to export existing client-side computed stats via OTLP. No changes to cardinality, aggregation keys, or the values being exported — the same
StatsBuffer/DDSketchdata the tracer already computes is serialized as atraces.span.sdk.metrics.durationOTLP histogram (units, DELTA temporality) and POSTed to an OTLP metrics endpoint. This lets downstream collectors consume span metrics using OTel semantics without recomputing them from raw spans.The PR is split into four commits — reviewing commit-by-commit is recommended.
Reviewer guide
Commit 1 — config keys (
OTEL_TRACES_SPAN_METRICS_ENABLED,DD_TRACE_OTEL_SEMANTICS_ENABLED),StatsBucketmin/max fields,TracerManagerFactorywiring. No serialization logic.Commit 2 —
OtlpSpanStatsSerializer.SerializeJson: projects DDSketch bins onto 16 fixed bounds viaPositiveValueStore.EnumerateAscending(); exact min/max fromStatsBucketsince sketch values are approximate.Commit 3 — same projection, hand-rolled protobuf encoding (no codegen).
ApiOtlp.SendStatsAsyncImpldispatches JSON vs protobuf viaOTEL_EXPORTER_OTLP_METRICS_PROTOCOL.Commit 4 — emits
_dd.stats_computedOTLP resource attribute to prevent downstream Agent from recomputing stats (mirrorsDatadog-Client-Computed-Statsheader). Adds unit tests forOtlpSpanStatsSerializer.Test coverage
System tests: DataDog/system-tests#6834
Notes
rpc.methodattribute omitted (high cardinality, follow-up)