Skip to content

Adjust Log levels in OltpExporter#8133

Merged
bouwkast merged 1 commit into
masterfrom
steven/mute-otlp
Mar 16, 2026
Merged

Adjust Log levels in OltpExporter#8133
bouwkast merged 1 commit into
masterfrom
steven/mute-otlp

Conversation

@bouwkast

@bouwkast bouwkast commented Feb 2, 2026

Copy link
Copy Markdown
Collaborator

Summary of changes

Changes the log levels in OtlpExporter to skip telemetry for some errors (while making them go from Debug to Error) and swapping some Debug

Reason for change

Noticed a lot of new errors but they just seem to be transient networking errors.
https://app.datadoghq.com/error-tracking/issue/443da854-fe18-11f0-ae87-da7ad0900002

Some of the current logs were either the wrong level or absent.

Implementation details

  • Bad Request (400) - not retrying - swapped to Error from Warning
  • Add two Log.ErrorSkipTelemetry("An error occurred while sending OTLP request to {AgentEndpoint}. for when we get transient network issue
  • Log.Error(ex, "Error sending OTLP request (attempt {Attempt})", (attempt + 1).ToString()); -> Log.Debug<int>(ex, "Error sending OTLP request (attempt {Attempt})", attempt + 1); along with removing the string allocation

Test coverage

Other details

@bouwkast
bouwkast requested a review from a team as a code owner February 2, 2026 21:12
@pr-commenter

pr-commenter Bot commented Feb 2, 2026

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-03-16 19:09:36

Comparing candidate commit 62a162e in PR branch steven/mute-otlp with baseline commit 8803967 in branch master.

Found 2 performance improvements and 11 performance regressions! Performance is the same for 158 metrics, 21 unstable metrics.

scenario:Benchmarks.Trace.ActivityBenchmark.StartStopWithChild net6.0

  • 🟥 throughput [-8818.093op/s; -6465.964op/s] or [-6.877%; -5.043%]

scenario:Benchmarks.Trace.AgentWriterBenchmark.WriteAndFlushEnrichedTraces net6.0

  • 🟥 execution_time [+82.501ms; +82.682ms] or [+68.026%; +68.175%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody net472

  • 🟥 throughput [-75106.669op/s; -68312.599op/s] or [-7.394%; -6.725%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorMoreComplexBody netcoreapp3.1

  • 🟥 execution_time [+14.622ms; +19.553ms] or [+7.454%; +9.969%]

scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSlice net6.0

  • 🟥 execution_time [+540.029µs; +554.224µs] or [+38.534%; +39.547%]
  • 🟥 throughput [-202.279op/s; -198.384op/s] or [-28.348%; -27.802%]

scenario:Benchmarks.Trace.DbCommandBenchmark.ExecuteNonQuery net6.0

  • 🟥 execution_time [+10.621ms; +14.523ms] or [+5.319%; +7.274%]

scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearchAsync net472

  • 🟥 throughput [-17861.012op/s; -16878.418op/s] or [-5.732%; -5.416%]

scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearchAsync net6.0

  • 🟩 throughput [+32388.598op/s; +34295.828op/s] or [+5.460%; +5.781%]

scenario:Benchmarks.Trace.HttpClientBenchmark.SendAsync netcoreapp3.1

  • 🟥 throughput [-7998.433op/s; -6559.186op/s] or [-6.118%; -5.017%]

scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark netcoreapp3.1

  • 🟥 throughput [-1320.057op/s; -1162.941op/s] or [-67.448%; -59.420%]

scenario:Benchmarks.Trace.Log4netBenchmark.EnrichedLog netcoreapp3.1

  • 🟩 execution_time [-34.574ms; -32.139ms] or [-17.197%; -15.986%]

scenario:Benchmarks.Trace.SingleSpanAspNetCoreBenchmark.SingleSpanAspNetCore netcoreapp3.1

  • 🟥 throughput [-24582383.479op/s; -23086095.011op/s] or [-10.206%; -9.585%]

@andrewlock andrewlock left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I think most of these Should stay as errors, but mostly because AFAICT, they never occur. And we're missing some logs in some places we should have them I think

Comment on lines +364 to +365
// Seeing network connectivity errors so skipping telemetry
Log.ErrorSkipTelemetry(ex, "Error sending OTLP request (attempt {Attempt})", (attempt + 1).ToString());

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Arguably, this shouldn't be an error at all IMO. Transient errors are expected, so this almost certainly shouldn't be logging an error at all. In Api for example, these "intermediate" attempts are Debug. Also nit, AI did the ToString() I assume 🙄

Suggested change
// Seeing network connectivity errors so skipping telemetry
Log.ErrorSkipTelemetry(ex, "Error sending OTLP request (attempt {Attempt})", (attempt + 1).ToString());
Log.Debug<int>(ex, "Error sending OTLP request (attempt {Attempt})", attempt + 1);```

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah yes, I ran my /analyze-error on it, forgot to add the label

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll swap these over thanks!

Comment thread tracer/src/Datadog.Trace/OpenTelemetry/Metrics/OtlpExporter.cs Outdated
Comment thread tracer/src/Datadog.Trace/OpenTelemetry/Metrics/OtlpExporter.cs Outdated
Comment thread tracer/src/Datadog.Trace/OpenTelemetry/Metrics/OtlpExporter.cs Outdated
Comment thread tracer/src/Datadog.Trace/OpenTelemetry/Metrics/OtlpExporter.cs Outdated
Comment thread tracer/src/Datadog.Trace/OpenTelemetry/Metrics/OtlpExporter.cs
Comment thread tracer/src/Datadog.Trace/OpenTelemetry/Metrics/OtlpExporter.cs
@bouwkast bouwkast added the AI Generated Largely based on code generated by an AI or LLM. This label is the same across all dd-trace-* repos label Feb 3, 2026
@github-actions github-actions Bot added the area:tracer The core tracer library (Datadog.Trace, does not include OpenTracing, native code, or integrations) label Feb 3, 2026
@dd-trace-dotnet-ci-bot

dd-trace-dotnet-ci-bot Bot commented Feb 3, 2026

Copy link
Copy Markdown

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing This PR (8133) and master.

✅ No regressions detected - check the details below

Full Metrics Comparison

FakeDbCommand

Metric Master (Mean ± 95% CI) Current (Mean ± 95% CI) Change Status
.NET Framework 4.8 - Baseline
duration79.10 ± (78.89 - 79.30) ms76.60 ± (76.56 - 76.86) ms-3.2%
.NET Framework 4.8 - Bailout
duration83.78 ± (83.71 - 84.14) ms81.36 ± (81.17 - 81.62) ms-2.9%
.NET Framework 4.8 - CallTarget+Inlining+NGEN
duration1117.44 ± (1118.27 - 1124.97) ms1108.27 ± (1111.50 - 1119.11) ms-0.8%
.NET Core 3.1 - Baseline
process.internal_duration_ms23.63 ± (23.57 - 23.68) ms23.54 ± (23.48 - 23.60) ms-0.4%
process.time_to_main_ms89.96 ± (89.76 - 90.17) ms89.46 ± (89.26 - 89.65) ms-0.6%
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%
runtime.dotnet.mem.committed10.89 ± (10.89 - 10.90) MB10.90 ± (10.90 - 10.91) MB+0.1%✅⬆️
runtime.dotnet.threads.count12 ± (12 - 12)12 ± (12 - 12)+0.0%
.NET Core 3.1 - Bailout
process.internal_duration_ms23.57 ± (23.51 - 23.64) ms23.54 ± (23.48 - 23.59) ms-0.2%
process.time_to_main_ms91.41 ± (91.22 - 91.60) ms91.81 ± (91.56 - 92.06) ms+0.4%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%
runtime.dotnet.mem.committed10.93 ± (10.93 - 10.94) MB10.94 ± (10.93 - 10.94) MB+0.0%✅⬆️
runtime.dotnet.threads.count13 ± (13 - 13)13 ± (13 - 13)+0.0%
.NET Core 3.1 - CallTarget+Inlining+NGEN
process.internal_duration_ms227.51 ± (226.00 - 229.03) ms229.35 ± (228.15 - 230.55) ms+0.8%✅⬆️
process.time_to_main_ms507.35 ± (506.20 - 508.50) ms511.68 ± (510.78 - 512.57) ms+0.9%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%
runtime.dotnet.mem.committed47.35 ± (47.32 - 47.38) MB47.14 ± (47.12 - 47.17) MB-0.4%
runtime.dotnet.threads.count28 ± (28 - 28)28 ± (28 - 28)-0.0%
.NET 6 - Baseline
process.internal_duration_ms22.09 ± (22.04 - 22.14) ms22.24 ± (22.20 - 22.29) ms+0.7%✅⬆️
process.time_to_main_ms77.21 ± (77.03 - 77.39) ms77.75 ± (77.58 - 77.92) ms+0.7%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%
runtime.dotnet.mem.committed10.62 ± (10.62 - 10.63) MB10.63 ± (10.63 - 10.63) MB+0.1%✅⬆️
runtime.dotnet.threads.count10 ± (10 - 10)10 ± (10 - 10)+0.0%
.NET 6 - Bailout
process.internal_duration_ms21.95 ± (21.90 - 21.99) ms22.06 ± (22.02 - 22.11) ms+0.5%✅⬆️
process.time_to_main_ms78.15 ± (77.98 - 78.32) ms78.20 ± (77.99 - 78.40) ms+0.1%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%
runtime.dotnet.mem.committed10.72 ± (10.71 - 10.72) MB10.74 ± (10.74 - 10.74) MB+0.2%✅⬆️
runtime.dotnet.threads.count11 ± (11 - 11)11 ± (11 - 11)+0.0%
.NET 6 - CallTarget+Inlining+NGEN
process.internal_duration_ms213.95 ± (211.94 - 215.96) ms212.27 ± (210.45 - 214.10) ms-0.8%
process.time_to_main_ms507.23 ± (506.41 - 508.06) ms504.54 ± (503.77 - 505.32) ms-0.5%
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%
runtime.dotnet.mem.committed49.19 ± (49.15 - 49.23) MB49.25 ± (49.21 - 49.28) MB+0.1%✅⬆️
runtime.dotnet.threads.count29 ± (29 - 29)29 ± (29 - 29)-0.0%
.NET 8 - Baseline
process.internal_duration_ms20.50 ± (20.45 - 20.54) ms20.32 ± (20.27 - 20.36) ms-0.9%
process.time_to_main_ms77.10 ± (76.93 - 77.26) ms76.89 ± (76.74 - 77.04) ms-0.3%
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%
runtime.dotnet.mem.committed7.67 ± (7.66 - 7.68) MB7.67 ± (7.67 - 7.68) MB+0.0%✅⬆️
runtime.dotnet.threads.count10 ± (10 - 10)10 ± (10 - 10)+0.0%
.NET 8 - Bailout
process.internal_duration_ms20.50 ± (20.44 - 20.55) ms20.53 ± (20.48 - 20.59) ms+0.2%✅⬆️
process.time_to_main_ms78.67 ± (78.49 - 78.85) ms78.77 ± (78.61 - 78.94) ms+0.1%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%
runtime.dotnet.mem.committed7.70 ± (7.70 - 7.71) MB7.73 ± (7.72 - 7.74) MB+0.4%✅⬆️
runtime.dotnet.threads.count11 ± (11 - 11)11 ± (11 - 11)+0.0%
.NET 8 - CallTarget+Inlining+NGEN
process.internal_duration_ms168.00 ± (166.99 - 169.01) ms169.30 ± (168.34 - 170.26) ms+0.8%✅⬆️
process.time_to_main_ms484.72 ± (484.00 - 485.45) ms488.23 ± (487.45 - 489.02) ms+0.7%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%
runtime.dotnet.mem.committed36.50 ± (36.48 - 36.52) MB36.55 ± (36.53 - 36.56) MB+0.1%✅⬆️
runtime.dotnet.threads.count28 ± (28 - 28)28 ± (28 - 28)-0.4%

HttpMessageHandler

Metric Master (Mean ± 95% CI) Current (Mean ± 95% CI) Change Status
.NET Framework 4.8 - Baseline
duration194.13 ± (194.00 - 194.83) ms193.70 ± (193.69 - 194.43) ms-0.2%
.NET Framework 4.8 - Bailout
duration197.50 ± (197.28 - 197.75) ms197.50 ± (197.27 - 198.02) ms-0.0%
.NET Framework 4.8 - CallTarget+Inlining+NGEN
duration1150.31 ± (1152.70 - 1161.06) ms1159.34 ± (1163.42 - 1173.60) ms+0.8%✅⬆️
.NET Core 3.1 - Baseline
process.internal_duration_ms188.44 ± (188.01 - 188.87) ms187.98 ± (187.54 - 188.43) ms-0.2%
process.time_to_main_ms80.92 ± (80.72 - 81.12) ms81.31 ± (81.07 - 81.55) ms+0.5%✅⬆️
runtime.dotnet.exceptions.count3 ± (3 - 3)3 ± (3 - 3)+0.0%
runtime.dotnet.mem.committed16.13 ± (16.10 - 16.16) MB16.05 ± (16.02 - 16.07) MB-0.5%
runtime.dotnet.threads.count20 ± (20 - 20)20 ± (19 - 20)-0.1%
.NET Core 3.1 - Bailout
process.internal_duration_ms188.07 ± (187.82 - 188.32) ms188.01 ± (187.70 - 188.32) ms-0.0%
process.time_to_main_ms82.41 ± (82.30 - 82.53) ms82.44 ± (82.31 - 82.57) ms+0.0%✅⬆️
runtime.dotnet.exceptions.count3 ± (3 - 3)3 ± (3 - 3)+0.0%
runtime.dotnet.mem.committed16.17 ± (16.14 - 16.20) MB16.21 ± (16.18 - 16.24) MB+0.3%✅⬆️
runtime.dotnet.threads.count21 ± (21 - 21)21 ± (21 - 21)+0.1%✅⬆️
.NET Core 3.1 - CallTarget+Inlining+NGEN
process.internal_duration_ms393.81 ± (392.19 - 395.42) ms395.99 ± (394.17 - 397.81) ms+0.6%✅⬆️
process.time_to_main_ms474.71 ± (474.12 - 475.30) ms473.68 ± (473.08 - 474.28) ms-0.2%
runtime.dotnet.exceptions.count3 ± (3 - 3)3 ± (3 - 3)+0.0%
runtime.dotnet.mem.committed57.91 ± (57.77 - 58.04) MB57.91 ± (57.76 - 58.06) MB+0.0%✅⬆️
runtime.dotnet.threads.count30 ± (30 - 30)30 ± (30 - 30)+0.4%✅⬆️
.NET 6 - Baseline
process.internal_duration_ms194.04 ± (193.57 - 194.51) ms192.51 ± (192.16 - 192.86) ms-0.8%
process.time_to_main_ms71.05 ± (70.88 - 71.23) ms70.22 ± (70.07 - 70.37) ms-1.2%
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%
runtime.dotnet.mem.committed16.41 ± (16.38 - 16.44) MB16.43 ± (16.37 - 16.50) MB+0.1%✅⬆️
runtime.dotnet.threads.count19 ± (19 - 19)19 ± (19 - 19)-0.8%
.NET 6 - Bailout
process.internal_duration_ms192.31 ± (192.01 - 192.61) ms191.82 ± (191.52 - 192.12) ms-0.3%
process.time_to_main_ms71.50 ± (71.36 - 71.64) ms71.12 ± (71.02 - 71.23) ms-0.5%
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%
runtime.dotnet.mem.committed16.36 ± (16.26 - 16.46) MB16.16 ± (16.02 - 16.31) MB-1.2%
runtime.dotnet.threads.count20 ± (19 - 20)20 ± (19 - 20)+0.2%✅⬆️
.NET 6 - CallTarget+Inlining+NGEN
process.internal_duration_ms428.62 ± (426.81 - 430.44) ms425.58 ± (423.68 - 427.48) ms-0.7%
process.time_to_main_ms477.55 ± (476.62 - 478.48) ms475.42 ± (474.52 - 476.32) ms-0.4%
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%
runtime.dotnet.mem.committed60.28 ± (60.20 - 60.36) MB60.45 ± (60.39 - 60.51) MB+0.3%✅⬆️
runtime.dotnet.threads.count30 ± (30 - 30)30 ± (30 - 30)+0.0%✅⬆️
.NET 8 - Baseline
process.internal_duration_ms190.45 ± (190.09 - 190.82) ms190.81 ± (190.40 - 191.22) ms+0.2%✅⬆️
process.time_to_main_ms70.03 ± (69.87 - 70.19) ms70.39 ± (70.19 - 70.59) ms+0.5%✅⬆️
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%
runtime.dotnet.mem.committed11.72 ± (11.69 - 11.75) MB11.73 ± (11.72 - 11.75) MB+0.1%✅⬆️
runtime.dotnet.threads.count18 ± (18 - 18)18 ± (18 - 18)+0.3%✅⬆️
.NET 8 - Bailout
process.internal_duration_ms190.38 ± (190.08 - 190.69) ms189.50 ± (189.23 - 189.77) ms-0.5%
process.time_to_main_ms71.29 ± (71.16 - 71.42) ms71.32 ± (71.20 - 71.44) ms+0.0%✅⬆️
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%
runtime.dotnet.mem.committed11.78 ± (11.75 - 11.81) MB11.80 ± (11.77 - 11.82) MB+0.1%✅⬆️
runtime.dotnet.threads.count19 ± (19 - 19)19 ± (19 - 19)+0.2%✅⬆️
.NET 8 - CallTarget+Inlining+NGEN
process.internal_duration_ms353.96 ± (352.69 - 355.24) ms354.51 ± (353.26 - 355.75) ms+0.2%✅⬆️
process.time_to_main_ms454.93 ± (454.30 - 455.57) ms454.90 ± (454.20 - 455.61) ms-0.0%
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%
runtime.dotnet.mem.committed48.39 ± (48.35 - 48.42) MB48.58 ± (48.54 - 48.61) MB+0.4%✅⬆️
runtime.dotnet.threads.count30 ± (30 - 30)30 ± (30 - 30)-0.1%
Comparison explanation

Execution-time benchmarks measure the whole time it takes to execute a program, and are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are highlighted in **red**. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

Duration charts
FakeDbCommand (.NET Framework 4.8)
gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8133) - mean (77ms)  : 74, 79
    master - mean (79ms)  : 76, 82

    section Bailout
    This PR (8133) - mean (81ms)  : 79, 84
    master - mean (84ms)  : 81, 87

    section CallTarget+Inlining+NGEN
    This PR (8133) - mean (1,115ms)  : 1058, 1172
    master - mean (1,122ms)  : 1073, 1171

Loading
FakeDbCommand (.NET Core 3.1)
gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8133) - mean (120ms)  : 116, 125
    master - mean (121ms)  : 116, 125

    section Bailout
    This PR (8133) - mean (123ms)  : 119, 126
    master - mean (122ms)  : 120, 125

    section CallTarget+Inlining+NGEN
    This PR (8133) - mean (777ms)  : 743, 810
    master - mean (772ms)  : 743, 800

Loading
FakeDbCommand (.NET 6)
gantt
    title Execution time (ms) FakeDbCommand (.NET 6)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8133) - mean (107ms)  : 103, 110
    master - mean (106ms)  : 103, 109

    section Bailout
    This PR (8133) - mean (107ms)  : 105, 109
    master - mean (107ms)  : 105, 109

    section CallTarget+Inlining+NGEN
    This PR (8133) - mean (755ms)  : 715, 795
    master - mean (761ms)  : 719, 804

Loading
FakeDbCommand (.NET 8)
gantt
    title Execution time (ms) FakeDbCommand (.NET 8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8133) - mean (105ms)  : 103, 108
    master - mean (106ms)  : 102, 109

    section Bailout
    This PR (8133) - mean (107ms)  : 105, 110
    master - mean (107ms)  : 105, 110

    section CallTarget+Inlining+NGEN
    This PR (8133) - mean (698ms)  : 671, 725
    master - mean (692ms)  : 664, 720

Loading
HttpMessageHandler (.NET Framework 4.8)
gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8133) - mean (194ms)  : 190, 198
    master - mean (194ms)  : 190, 199

    section Bailout
    This PR (8133) - mean (198ms)  : 194, 202
    master - mean (198ms)  : 195, 200

    section CallTarget+Inlining+NGEN
    This PR (8133) - mean (1,169ms)  : 1091, 1246
    master - mean (1,157ms)  : 1094, 1220

Loading
HttpMessageHandler (.NET Core 3.1)
gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8133) - mean (278ms)  : 271, 284
    master - mean (278ms)  : 271, 285

    section Bailout
    This PR (8133) - mean (279ms)  : 275, 283
    master - mean (279ms)  : 276, 282

    section CallTarget+Inlining+NGEN
    This PR (8133) - mean (902ms)  : 873, 930
    master - mean (902ms)  : 874, 930

Loading
HttpMessageHandler (.NET 6)
gantt
    title Execution time (ms) HttpMessageHandler (.NET 6)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8133) - mean (271ms)  : 267, 276
    master - mean (274ms)  : 268, 279

    section Bailout
    This PR (8133) - mean (271ms)  : 267, 275
    master - mean (272ms)  : 268, 275

    section CallTarget+Inlining+NGEN
    This PR (8133) - mean (936ms)  : 908, 963
    master - mean (940ms)  : 916, 963

Loading
HttpMessageHandler (.NET 8)
gantt
    title Execution time (ms) HttpMessageHandler (.NET 8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8133) - mean (271ms)  : 265, 277
    master - mean (270ms)  : 266, 275

    section Bailout
    This PR (8133) - mean (270ms)  : 267, 274
    master - mean (271ms)  : 267, 275

    section CallTarget+Inlining+NGEN
    This PR (8133) - mean (840ms)  : 807, 873
    master - mean (842ms)  : 820, 865

Loading

@andrewlock andrewlock left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

bouwkast added a commit that referenced this pull request Feb 4, 2026
## Summary of changes

Updates `/analyze-error` to not update all `Log.Error` calls in the to
`Log.ErrorSkipTelemetry` as it did in
#8133

Added some more generic logging considerations to `AGENTS.md`

## Reason for change

The `/analyze-error` command was instructing to update all instances of
`Log.Error` instead of just the correct / expected one.

AGENTS.md had some logging information but I think it lacked some
specificity, so updating very basically with some "best practices"

## Implementation details

## Test coverage

## Other details
<!-- Fixes #{issue} -->

I reran the `/analyze-error` command with these changes again on the
same error I did with
#8133 and with both of
these it changed to this following diff. It isn't _perfect_ but at least
it isn't super wrong, it also changed the `ToString` used to a generic
call. Note that the initial change was
5ab5b10


  ```diff
diff --git
a/tracer/src/Datadog.Trace/OpenTelemetry/Metrics/OtlpExporter.cs
  b/tracer/src/Datadog.Trace/OpenTelemetry/Metrics/OtlpExporter.cs
  index e90bee8..8c2acd3c2 100644
  --- a/tracer/src/Datadog.Trace/OpenTelemetry/Metrics/OtlpExporter.cs
  +++ b/tracer/src/Datadog.Trace/OpenTelemetry/Metrics/OtlpExporter.cs
  @@ -353,21 +353,27 @@ namespace Datadog.Trace.OpenTelemetry.Metrics
                   }
catch (TaskCanceledException) when (attempt < maxRetries)
                   {
+ Log.Debug<int>("OTLP request timed out (attempt {Attempt}),
retrying...", attempt + 1);
+ retryDelay =
TimeSpan.FromMilliseconds((long)(retryDelay.TotalMilliseconds * 2));
+ await Task.Delay(retryDelay).ConfigureAwait(false);
  +                }
  +                catch (TaskCanceledException ex)
  +                {
  +                    // Final timeout after all retries exhausted
+ Log.ErrorSkipTelemetry<Uri, int>(ex, "OTLP request to {Endpoint} timed
out after {Attempt}
attempts. See https://docs.datadoghq.com/tracing/troubleshooting/ for
troubleshooting.", _endpoint, attempt + 1);
  +                    return false;
  +                }
  +                catch (Exception ex) when (attempt < maxRetries)
  +                {
+ Log.Debug<int>(ex, "Error sending OTLP request (attempt {Attempt}),
retrying...", attempt + 1);
retryDelay =
TimeSpan.FromMilliseconds((long)(retryDelay.TotalMilliseconds * 2));
await Task.Delay(retryDelay).ConfigureAwait(false);
                   }
                   catch (Exception ex)
                   {
- Log.Error(ex, "Error sending OTLP request (attempt {Attempt})",
(attempt + 1).ToString());
  -                    if (attempt < maxRetries)
  -                    {
- retryDelay =
TimeSpan.FromMilliseconds((long)(retryDelay.TotalMilliseconds * 2));
- await Task.Delay(retryDelay).ConfigureAwait(false);
  -                    }
  -                    else
  -                    {
  -                        return false;
  -                    }
  +                    // Final failure after all retries exhausted
+ Log.ErrorSkipTelemetry<Uri, int>(ex, "Failed to send OTLP request to
{Endpoint} after {Attempt}
attempts. See https://docs.datadoghq.com/tracing/troubleshooting/ for
troubleshooting.", _endpoint, attempt + 1);
  +                    return false;
                   }
               }
  ```

<!--  ⚠️ Note:

Where possible, please obtain 2 approvals prior to merging. Unless
CODEOWNERS specifies otherwise, for external teams it is typically best
to have one review from a team member, and one review from apm-dotnet.
Trivial changes do not require 2 reviews.

MergeQueue is NOT enabled in this repository. If you have write access
to the repo, the PR has 1-2 approvals (see above), and all of the
required checks have passed, you can use the Squash and Merge button to
merge the PR. If you don't have write access, or you need help, reach
out in the #apm-dotnet channel in Slack.
-->

---------

Co-authored-by: Andrew Lock <[email protected]>

@link04 link04 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TY

- Use Log.Error for 400 Bad Request (indicates a bug in our payload)
- Use Log.Debug for intermediate retry attempts (transient/expected)
- Add ErrorSkipTelemetry with endpoint and troubleshooting link on final failure
- Fix ToString() allocation in log call by using generic method

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
@bouwkast bouwkast changed the title Skip OTLP exporter errors from being sent to telemetry Adjust Log levels in OltpExporter Mar 16, 2026
@bouwkast
bouwkast enabled auto-merge (squash) March 16, 2026 18:28
@bouwkast
bouwkast merged commit 48a1fe5 into master Mar 16, 2026
139 checks passed
@bouwkast
bouwkast deleted the steven/mute-otlp branch March 16, 2026 20:03
@github-actions github-actions Bot added this to the vNext-v3 milestone Mar 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AI Generated Largely based on code generated by an AI or LLM. This label is the same across all dd-trace-* repos area:tracer The core tracer library (Datadog.Trace, does not include OpenTracing, native code, or integrations) identified-by:telemetry

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants