Fix OTLP traces export routed through APM Unix domain socket instead of OTLP endpoint#8460
Closed
duncanhewett wants to merge 2 commits into
Closed
Fix OTLP traces export routed through APM Unix domain socket instead of OTLP endpoint#8460duncanhewett wants to merge 2 commits into
duncanhewett wants to merge 2 commits into
Conversation
Co-Authored-By: Claude Opus 4.6 <[email protected]>
duncanhewett
added
the
AI Generated
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4c035d619e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| var otlpEndpoint = exporterSettings.OtlpTracesEndpoint; | ||
| var baseEndpoint = new Uri($"{otlpEndpoint.Scheme}://{otlpEndpoint.Authority}"); | ||
| Log.Debug("Using " + nameof(HttpClientRequestFactory) + " for OTLP traces transport to {Endpoint}", baseEndpoint); | ||
| return new HttpClientRequestFactory(baseEndpoint, defaultHeaders: []); |
There was a problem hiding this comment.
Constructing HttpClientRequestFactory with defaultHeaders: [] drops the x-datadog-tracing-enabled: false header that was previously applied through TracesTransportStrategy/AgentHttpHeaderNames.DefaultHeaders. In OTLP mode this means the tracer can instrument its own exporter HTTP calls (see HttpMessageHandlerCommon.IsTracingEnabled), creating self-generated spans and avoidable export traffic on every flush. This regression appears whenever OTLP traces are sent through HttpClient with HTTP client instrumentation enabled (default behavior).
Useful? React with 👍 / 👎.
BenchmarksBenchmark execution time: 2026-04-15 14:28:06 Comparing candidate commit 4c035d6 in PR branch Found 0 performance improvements and 1 performance regressions! Performance is the same for 26 metrics, 0 unstable metrics, 87 known flaky benchmarks.
|
Execution-Time Benchmarks Report ⏱️Execution-time results for samples comparing This PR (8460) and master. ✅ No regressions detected - check the details below Full Metrics ComparisonFakeDbCommand
HttpMessageHandler
Comparison explanationExecution-time benchmarks measure the whole time it takes to execute a program, and are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are highlighted in **red**. The following thresholds were used for comparing the execution times:
Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard. Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph). Duration chartsFakeDbCommand (.NET Framework 4.8)gantt
title Execution time (ms) FakeDbCommand (.NET Framework 4.8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8460) - mean (74ms) : 71, 77
master - mean (72ms) : 70, 75
section Bailout
This PR (8460) - mean (78ms) : 75, 80
master - mean (76ms) : 74, 78
section CallTarget+Inlining+NGEN
This PR (8460) - mean (1,080ms) : 1033, 1126
master - mean (1,066ms) : 1018, 1115
FakeDbCommand (.NET Core 3.1)gantt
title Execution time (ms) FakeDbCommand (.NET Core 3.1)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8460) - mean (113ms) : 109, 118
master - mean (114ms) : 110, 117
section Bailout
This PR (8460) - mean (114ms) : 111, 116
master - mean (115ms) : 112, 118
section CallTarget+Inlining+NGEN
This PR (8460) - mean (797ms) : 773, 820
master - mean (786ms) : 767, 806
FakeDbCommand (.NET 6)gantt
title Execution time (ms) FakeDbCommand (.NET 6)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8460) - mean (102ms) : 97, 107
master - mean (100ms) : 97, 102
section Bailout
This PR (8460) - mean (103ms) : 99, 107
master - mean (99ms) : 97, 101
section CallTarget+Inlining+NGEN
This PR (8460) - mean (939ms) : 907, 972
master - mean (931ms) : 901, 962
FakeDbCommand (.NET 8)gantt
title Execution time (ms) FakeDbCommand (.NET 8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8460) - mean (98ms) : 96, 101
master - mean (99ms) : 95, 102
section Bailout
This PR (8460) - mean (100ms) : 98, 102
master - mean (99ms) : 97, 102
section CallTarget+Inlining+NGEN
This PR (8460) - mean (822ms) : 780, 863
master - mean (815ms) : 780, 849
HttpMessageHandler (.NET Framework 4.8)gantt
title Execution time (ms) HttpMessageHandler (.NET Framework 4.8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8460) - mean (208ms) : 203, 213
master - mean (212ms) : 206, 218
section Bailout
This PR (8460) - mean (210ms) : 207, 214
master - mean (215ms) : 211, 219
section CallTarget+Inlining+NGEN
This PR (8460) - mean (1,210ms) : 1162, 1259
master - mean (1,231ms) : 1180, 1281
HttpMessageHandler (.NET Core 3.1)gantt
title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8460) - mean (297ms) : 291, 303
master - mean (306ms) : 299, 314
section Bailout
This PR (8460) - mean (299ms) : 293, 305
master - mean (305ms) : 297, 313
section CallTarget+Inlining+NGEN
This PR (8460) - mean (992ms) : 961, 1023
master - mean (1,018ms) : 990, 1047
HttpMessageHandler (.NET 6)gantt
title Execution time (ms) HttpMessageHandler (.NET 6)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8460) - mean (291ms) : 283, 299
master - mean (299ms) : 292, 306
section Bailout
This PR (8460) - mean (291ms) : 287, 296
master - mean (299ms) : 293, 304
section CallTarget+Inlining+NGEN
This PR (8460) - mean (1,171ms) : 1129, 1212
master - mean (1,182ms) : 1152, 1213
HttpMessageHandler (.NET 8)gantt
title Execution time (ms) HttpMessageHandler (.NET 8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8460) - mean (288ms) : 282, 294
master - mean (297ms) : 292, 303
section Bailout
This PR (8460) - mean (290ms) : 285, 295
master - mean (298ms) : 292, 303
section CallTarget+Inlining+NGEN
This PR (8460) - mean (1,058ms) : 998, 1117
master - mean (1,080ms) : 983, 1177
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
andrewlock
reviewed
Apr 15, 2026
andrewlock
left a comment
andrewlock
left a comment
Member
There was a problem hiding this comment.
Presumably this failed if the app was set to UDS or named pipes, so we should probably update the Otlp export integration tests to run a test using each of those transports too
Comment on lines
+63
to
+64
| var otlpEndpoint = exporterSettings.OtlpTracesEndpoint; | ||
| var baseEndpoint = new Uri($"{otlpEndpoint.Scheme}://{otlpEndpoint.Authority}"); |
Member
There was a problem hiding this comment.
Can OTLP not be sent over UDS then? 🤔
Contributor
There was a problem hiding this comment.
We should be able to send OTLP traces over UDS. However, in the currently tested case we are not setting up an OTLP receiver with a given socket file, so the communication is failing. A workaround such as this is needed so we can set the OTLP endpoint independently of an APM socket, and in the future we'll need to ensure that this OTLP endpoint can also be configured to write send to a socket
Co-Authored-By: Claude Opus 4.6 <[email protected]>
zacharycmontoya
added a commit
that referenced
this pull request
Jun 2, 2026
… is set (#8668) ## Summary of changes Fixes a bug where setting the APM UDS socket would inadvertently set the HttpClient used by OTLP traces export, which led to trace submission issues. This is accomplished by having the OTLP traces exporter use a new code path to configure the underlying HTTP client. ## Reason for change The primary motivation was to resolve the following OTLP traces export issue: When OTLP traces export is enabled through the configuration OTEL_TRACES_EXPORTER=otlp, the endpoint defaults to TCP/IP communication with endpoint http://localhost:4317 or http://localhost:4318. However, when DD_APM_RECEIVER_SOCKET is configured and its socket file is present, the underlying HTTP client for the OTLP traces exporter is erroneously being configured for UDS communication rather than TCP/IP. ## Implementation details - Adds a new `OtlpTransportStrategy` that is a factory for OTLP traces HTTP clients. This supports TCP/IP endpoints and UDS endpoints (though UDS is not currently tested) - `ManagedApiOtlp` gets an HTTP client from `OtlpTransportStrategy.GetTraces` rather than `TracesTransportStrategy.Get` - Creates and uses a new `OtlpHeaderHelper` instance to set default headers for all OTLP traces HTTP requests ## Test coverage Adds unit tests in `ManagedApiOtlpTests`. Note: UDS is supported but currently untested. ## Other details Replaces #8614 and #8460
Contributor
|
Closing as fixed by #8668 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary of changes
Fix OTLP traces transport to use the OTLP endpoint directly instead of the APM trace transport strategy.
Reason for change
On Linux, when the default APM Unix domain socket (
/var/run/datadog/apm.socket) exists, OTLP trace requests were incorrectly routed through it instead of the configured OTLP HTTP endpoint, resulting in 404 errors.Implementation details
ManagedApiOtlpnow creates anHttpClientRequestFactoryfromExporterSettings.OtlpTracesEndpointinstead of delegating toTracesTransportStrategy/AgentTransportStrategy, which selects transport based on the configuration.Test coverage
Unit tests in ManagedApiOtlpTests verifying the factory type and endpoint correctness when APM UDS is present.
Other details