DEBUG-3700 Replace telemetry transport with a core-based transport that supports UDS by p-datadog · Pull Request #4630 · DataDog/dd-trace-rb

p-datadog · 2025-05-07T21:49:48Z

What does this PR do?

Replaces telemetry's simple transport by the proper "enterprise" transport that implements Unix domain sockets besides HTTP.

Motivation:

Shopist requires usage of UDS for everything.

We also had support escalations asking for UDS support in telemetry.

Change log entry
None

Additional Notes:
I used DI transport as a starting point. There is a fair amount of boilerplate in transport code (still) which I hope to reduce one day in the future.

Telemetry initialization logic that was in Component.build was moved to Component#initialize to centralize processing.

http_transport attributes within telemetry were renamed to transport for consistency with other transports and because UDS is now supported.

Telemetry worker will now not be created at all if telemetry is disabled. This was the easiest route to get some of the tests passing and I believe this is the correct behavior.

I feel like the agent_settings should be a cached attribute of settings rather than having consumers deal with AgentSettingsResolver (again maybe in the bright future).

How to test the change?

Existing CI including integration tests added in #4588 that verify payloads.

Integration tests were expanded to add a UDS configuration.

github-actions · 2025-05-07T21:50:06Z

Thank you for updating Change log entry section 👏

^{Visited at: 2025-05-13 11:55:39 UTC}

datadog-datadog-prod-us1 · 2025-05-07T22:05:56Z

Datadog Report

Branch report: telemetry-transport-2
Commit report: 5b467e7
Test service: dd-trace-rb

❌ 15 Failed (0 Known Flaky), 20469 Passed, 1368 Skipped, 3m 34.93s Total Time

❌ Failed Tests (15)

This report shows up to 5 failed tests.

Datadog::Core::Telemetry::Component.build given settings with :agentless_enabled true and no api key does not enable telemetry when agentless mode requested but api key is not present - rspec - Details

Expand for error

 #<Datadog::Core::Telemetry::Component (class)> received :new with unexpected arguments
   expected: ({:dependency_collection => true, :enabled => false, :heartbeat_interval_seconds => 60, :log_collectio...ansport => #<InstanceDouble(Datadog::Core::Telemetry::Transport::Telemetry::Transport) (anonymous)>}) (options hash)
        got: ({:agent_settings => #<Datadog::Core::Configuration::AgentSettingsResolver::AgentSettings:0x00007fc15c...e_set=#<struct Datadog::Core::Configuration::Option::Precedence::Value numeric=0, name=:default>>}>}) (keyword arguments)
 Diff:
 
 
 @@ -1,11 +1,7 @@
 -[{:dependency_collection => true,
 +[{:agent_settings =>
 +   #<Datadog::Core::Configuration::AgentSettingsResolver::AgentSettings:0x00007fc15c4164f8 @adapter=:net_http, @ssl=false, @hostname="agent", @port=9126, @uds_path=nil, @timeout_seconds=30>,
 ...

Datadog::Core::Telemetry::Component.build given settings with :agentless_enabled true is expected to receive agentless_telemetry({:agent_settings=>an instance of Datadog::Core::Configuration::AgentSettingsResolver::AgentSettings, :api_key=>"api_key", :logger=>#<InstanceDouble(Logger) (anonymous)>}) 1 time - rspec - Details

Expand for error

 #<Datadog::Core::Telemetry::Component (class)> received :new with unexpected arguments
   expected: ({:dependency_collection=>true, :enabled=>true, :heartbeat_interval_seconds=>60, :log_collection_enabl...transport=>#<InstanceDouble(Datadog::Core::Telemetry::Transport::Telemetry::Transport) (anonymous)>}) (options hash)
        got: ({:agent_settings=>#<Datadog::Core::Configuration::AgentSettingsResolver::AgentSettings:0x00007f4c1fa7...e_set=#<struct Datadog::Core::Configuration::Option::Precedence::Value numeric=0, name=:default>>}>}) (keyword arguments)
 Diff:
 
 
 @@ -1,11 +1,7 @@
 -[{:dependency_collection=>true,
 +[{:agent_settings=>
 +   #<Datadog::Core::Configuration::AgentSettingsResolver::AgentSettings:0x00007f4c1fa7bfd0 @adapter=:net_http, @ssl=false, @hostname="agent", @port=9126, @uds_path=nil, @timeout_seconds=30>,
 ...

Datadog::Core::Telemetry::Component.build given settings with :agentless_enabled true is expected to receive agentless_telemetry({:agent_settings => an instance of Datadog::Core::Configuration::AgentSettingsResolver::AgentSettings, :api_key => "api_key", :logger => #<InstanceDouble(Logger) (anonymous)>}) 1 time - rspec - Details

Expand for error

 #<Datadog::Core::Telemetry::Component (class)> received :new with unexpected arguments
   expected: ({:dependency_collection => true, :enabled => true, :heartbeat_interval_seconds => 60, :log_collection...ansport => #<InstanceDouble(Datadog::Core::Telemetry::Transport::Telemetry::Transport) (anonymous)>}) (options hash)
        got: ({:agent_settings => #<Datadog::Core::Configuration::AgentSettingsResolver::AgentSettings:0x00007fc15c...e_set=#<struct Datadog::Core::Configuration::Option::Precedence::Value numeric=0, name=:default>>}>}) (keyword arguments)
 Diff:
 
 
 @@ -1,11 +1,7 @@
 -[{:dependency_collection => true,
 +[{:agent_settings =>
 +   #<Datadog::Core::Configuration::AgentSettingsResolver::AgentSettings:0x00007fc15c3c6b38 @adapter=:net_http, @ssl=false, @hostname="agent", @port=9126, @uds_path=nil, @timeout_seconds=30>,
 ...

Datadog::Core::Telemetry::Component.build given settings with :enabled true - rspec - Details

Expand for error

 The setting \`telemetry.enabled\` inside your app's \`Datadog.configure\` block expects a bool, but a \`RSpec::Mocks::Double\` was provided (#<Double "enabled">). Please update your \`configure\` block.
 
 Failure/Error: raise ArgumentError, error_msg
 
 ArgumentError:
   The setting \`telemetry.enabled\` inside your app's \`Datadog.configure\` block expects a bool, but a \`RSpec::Mocks::Double\` was provided (#<Double "enabled">). Please update your \`configure\` block. 
 ./lib/datadog/core/configuration/option.rb:238:in 'Datadog::Core::Configuration::Option#validate_type'
 ./lib/datadog/core/configuration/option.rb:273:in 'Datadog::Core::Configuration::Option#internal_set'
 ./lib/datadog/core/configuration/option.rb:90:in 'Datadog::Core::Configuration::Option#set'
 ./lib/datadog/core/configuration/options.rb:71:in 'Datadog::Core::Configuration::Options::InstanceMethods#set_option'
 ...

Datadog::Core::Telemetry::Component.build given settings with :enabled true and :unix agent adapter does not enable telemetry for unsupported non-http transport - rspec - Details

Expand for error

 The setting \`telemetry.enabled\` inside your app's \`Datadog.configure\` block expects a bool, but a \`RSpec::Mocks::Double\` was provided (#<Double "enabled">). Please update your \`configure\` block.
 
 Failure/Error: raise ArgumentError, error_msg
 
 ArgumentError:
   The setting \`telemetry.enabled\` inside your app's \`Datadog.configure\` block expects a bool, but a \`RSpec::Mocks::Double\` was provided (#<Double "enabled">). Please update your \`configure\` block. 
 ./lib/datadog/core/configuration/option.rb:238:in 'Datadog::Core::Configuration::Option#validate_type'
 ./lib/datadog/core/configuration/option.rb:273:in 'Datadog::Core::Configuration::Option#internal_set'
 ./lib/datadog/core/configuration/option.rb:90:in 'Datadog::Core::Configuration::Option#set'
 ./lib/datadog/core/configuration/options.rb:71:in 'Datadog::Core::Configuration::Options::InstanceMethods#set_option'
 ...

pr-commenter · 2025-05-08T17:33:21Z

Benchmarks

Benchmark execution time: 2025-05-13 14:16:25

Comparing candidate commit 4bdab94 in PR branch telemetry-transport-2 with baseline commit 7ce8685 in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 31 metrics, 2 unstable metrics.

datadog-datadog-prod-us1 · 2025-05-12T12:41:26Z

Datadog Summary

✅ Dependencies

Test Optimization Report

Branch report: telemetry-transport-2
Commit report: 4e9cd75
Test service: dd-trace-rb

❌ 1 Failed (0 Known Flaky), 21144 Passed, 1374 Skipped, 3m 52.97s Total Time

❌ Failed Tests (1)

Datadog::Core::Telemetry::Emitter#request when event is app-started when call is not successful and debug logging is enabled logs the request correctly - rspec - Details

Expand for error

 expected nil to match "Telemetry sent for event"
 
 Failure/Error: expect(log_message).to match('Telemetry sent for event')
   expected nil to match "Telemetry sent for event"
 ./spec/datadog/core/telemetry/emitter_spec.rb:83:in 'block (6 levels) in <top (required)>'
 ./spec/spec_helper.rb:263:in 'block (2 levels) in <top (required)>'
 ./spec/spec_helper.rb:148:in 'block (2 levels) in <top (required)>'
 /usr/local/bundle/gems/webmock-3.23.1/lib/webmock/rspec.rb:39:in 'block (2 levels) in <top (required)>'
 /usr/local/bundle/gems/rspec-wait-0.0.10/lib/rspec/wait.rb:47:in 'block (2 levels) in <top (required)>'

Was this helpful? Give us feedback!

codecov-commenter · 2025-05-12T20:30:55Z

Codecov Report

❌ Patch coverage is 97.50000% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 97.76%. Comparing base (7ce8685) to head (55f4673).
⚠️ Report is 2738 commits behind head on master.

Files with missing lines	Patch %	Lines
spec/support/log_helpers.rb	71.42%	4 Missing ⚠️
...ib/datadog/core/telemetry/transport/http/client.rb	89.47%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4630      +/-   ##
==========================================
- Coverage   97.77%   97.76%   -0.02%     
==========================================
  Files        1415     1418       +3     
  Lines       86561    86598      +37     
  Branches     4386     4395       +9     
==========================================
+ Hits        84636    84663      +27     
- Misses       1925     1935      +10

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Strech

I have a suggestion to redo the enabled flag internals into 2 separate enabled and disabled telemetry components (or some sub-classes).

In addition I think few interfaces could be slightly changed (like other classes in this PR)

anmarchenko

I have no objections and agentless telemetry tests passed: great job @p-datadog

y9v

great job!

Strech · 2025-05-13T13:52:01Z

Awesome 👏🏼

github-actions Bot added core Involves Datadog core libraries tracing labels May 7, 2025

p-datadog mentioned this pull request May 8, 2025

DEBUG-3700 agentless settings resolver #4590

Merged

p-datadog force-pushed the telemetry-transport-2 branch 2 times, most recently from 5b467e7 to e9b3353 Compare May 12, 2025 12:39

p-datadog mentioned this pull request May 12, 2025

DEBUG-3700 UDS HTTP server helper for test suite #4636

Merged

p-datadog force-pushed the telemetry-transport-2 branch from 80eaa8a to b43ad13 Compare May 12, 2025 16:26

p added 19 commits May 12, 2025 13:52

new telemetry transport mostly copied from DI

f97c80c

telemetry transport

c93002e

telemetry transport

4e92937

telemetry transport

906cae2

fixing

56858d0

fixing maybe

623bf50

use agentless

4b4bb58

use agentless

e7cbfeb

api key

bd781d0

remove

b2ffdf0

remove

4c5e579

convert

590476d

fixing

e63897e

fix

16d6b59

remove api_key from emitter

fde4e21

rename transport attrs

c443519

take api_version as argument

63866b8

rubocop

53677fb

rubocop go away

db4cc9d

p added 6 commits May 12, 2025 13:52

types

6b3a29d

types

b25d5d6

types

a970295

types

b663fdd

types

d520f98

remove unused assignment

4e9cd75

p-datadog force-pushed the telemetry-transport-2 branch from 22102ea to 4e9cd75 Compare May 12, 2025 17:52

p added 4 commits May 12, 2025 16:05

inject logger

572ed80

types

729d206

remove

41cb04c

rubocop

84ff9b2

p-datadog changed the title ~~Telemetry transport 2~~ DEBUG-3700 Replace telemetry transport with a core-based transport that supports UDS May 13, 2025

p-datadog marked this pull request as ready for review May 13, 2025 11:55

p-datadog requested a review from a team as a code owner May 13, 2025 11:55

Merge branch 'master' into telemetry-transport-2

533d545

Strech reviewed May 13, 2025

View reviewed changes

Comment thread lib/datadog/core/telemetry/component.rb Outdated

Comment thread lib/datadog/core/telemetry/emitter.rb Outdated

Comment thread lib/datadog/core/telemetry/request.rb

Comment thread lib/datadog/core/telemetry/transport/telemetry.rb Outdated

anmarchenko approved these changes May 13, 2025

View reviewed changes

p added 2 commits May 13, 2025 09:28

convert to positional

eab7c2c

refactor

a9b0518

y9v approved these changes May 13, 2025

View reviewed changes

Strech approved these changes May 13, 2025

View reviewed changes

p added 3 commits May 13, 2025 09:52

move

0a14272

rubocop

4bdab94

fix tests

55f4673

p-datadog merged commit 03eefec into master May 13, 2025

p-datadog deleted the telemetry-transport-2 branch May 13, 2025 16:16

github-actions Bot added this to the 2.16.0 milestone May 13, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

DEBUG-3700 Replace telemetry transport with a core-based transport that supports UDS#4630

DEBUG-3700 Replace telemetry transport with a core-based transport that supports UDS#4630
p-datadog merged 55 commits into
masterfrom
telemetry-transport-2

p-datadog commented May 7, 2025 •

edited

Loading

Uh oh!

github-actions Bot commented May 7, 2025 •

edited

Loading

Uh oh!

datadog-datadog-prod-us1 Bot commented May 7, 2025 •

edited

Loading

Uh oh!

pr-commenter Bot commented May 8, 2025 •

edited

Loading

Uh oh!

datadog-datadog-prod-us1 Bot commented May 12, 2025 •

edited

Loading

Uh oh!

codecov-commenter commented May 12, 2025 •

edited by codecov Bot

Loading

Uh oh!

Strech left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

anmarchenko left a comment

Uh oh!

y9v left a comment

Uh oh!

Strech commented May 13, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Uh oh!

Conversation

p-datadog commented May 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented May 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

datadog-datadog-prod-us1 Bot commented May 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Datadog Report

❌ Failed Tests (15)

Uh oh!

pr-commenter Bot commented May 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Uh oh!

datadog-datadog-prod-us1 Bot commented May 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Datadog Summary

Test Optimization Report

❌ Failed Tests (1)

Uh oh!

codecov-commenter commented May 12, 2025 • edited by codecov Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Strech left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

anmarchenko left a comment

Choose a reason for hiding this comment

Uh oh!

y9v left a comment

Choose a reason for hiding this comment

Uh oh!

Strech commented May 13, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

p-datadog commented May 7, 2025 •

edited

Loading

github-actions Bot commented May 7, 2025 •

edited

Loading

datadog-datadog-prod-us1 Bot commented May 7, 2025 •

edited

Loading

pr-commenter Bot commented May 8, 2025 •

edited

Loading

datadog-datadog-prod-us1 Bot commented May 12, 2025 •

edited

Loading

codecov-commenter commented May 12, 2025 •

edited by codecov Bot

Loading