fix(telemetry): skip agent proxy URL when agent doesn't expose /telemetry/proxy/ by eliottness · Pull Request #4454 · DataDog/dd-trace-go

eliottness · 2026-02-20T09:04:58Z

What does this PR do?

When the Datadog Lambda extension is running at localhost:8126, it serves traces but does not expose the /telemetry/proxy/ endpoint. Previously the tracer always set AgentURL in the telemetry client config unconditionally, causing a WARN log on every flush:

Datadog Tracer v2.6.0 WARN: appsec: error while flushing SCA Security Data: Post "http://localhost:8126/telemetry/proxy/api/v2/apmtelemetry": dial tcp 127.0.0.1:8126: connect: connection refused

This PR fixes the root cause by using the /info endpoint response (already fetched at tracer startup) to determine whether the agent actually supports the telemetry proxy, rather than assuming it always does.

Changes

Add hasTelemetryProxy bool to agentFeatures, set when /telemetry/proxy/ appears in the endpoints list returned by /info (the agent only advertises this when TelemetryConfig.Enabled is true).
Add reachable bool to agentFeatures, set only when /info returns a valid parseable response. This distinguishes "agent is reachable but has no proxy" (Lambda extension case) from "agent is unreachable" (transient startup failure).
In startTelemetry, only set AgentURL when !reachable || hasTelemetryProxy:
- Lambda extension (reachable=true, hasTelemetryProxy=false): AgentURL not set → NewClient fails at debug level → no WARN logs ✓
- Normal agent (reachable=true, hasTelemetryProxy=true): AgentURL set → telemetry flows through proxy ✓
- Agent unreachable at startup (reachable=false): AgentURL still set → preserves previous behaviour, still attempts ✓

Why this is better than #4421

PR #4421 uses IsLambdaFunction() (checking AWS_LAMBDA_FUNCTION_NAME) as a heuristic. The actual problem is not Lambda-specific: any environment where the agent doesn't expose the telemetry proxy would benefit from this fix. This approach derives the behaviour from what the agent actually advertises rather than from environment detection.

Fixes

Fixes #3808. Supersedes #4421.

Reviewer's Checklist

Changed code has unit tests for its functionality at or near 100% coverage.
System-Tests covering this feature have been added and enabled with the va.b.c-dev version tag.
There is a benchmark for any new code, or changes to existing code.
If this interacts with the agent in a new way, a system test has been added.
New code is free of linting errors.
New code doesn't break existing tests.
Add an appropriate team label so this PR gets put in the right place for the release notes.
All generated files are up to date.

…etry/proxy/ When the Datadog Lambda extension is running at localhost:8126, it serves traces but does not expose the /telemetry/proxy/ endpoint. Previously the tracer always set AgentURL in the telemetry client config, causing a WARN log on every flush: appsec: error while flushing SCA Security Data: Post "http://localhost:8126/telemetry/proxy/api/v2/apmtelemetry": dial tcp 127.0.0.1:8126: connect: connection refused This fix uses the /info endpoint response (already fetched at startup) to determine whether the agent actually exposes the telemetry proxy: - Add hasTelemetryProxy bool to agentFeatures, set when /telemetry/proxy/ appears in the endpoints list returned by /info. - Add reachable bool to agentFeatures, set only when /info returns a valid parseable response. This distinguishes "agent is reachable but has no proxy" (Lambda extension) from "agent is unreachable" (transient issue). - In startTelemetry, only set AgentURL when the agent is reachable with the proxy, or when the agent was unreachable at startup (preserving previous behaviour of still attempting to send telemetry). Fixes #3808. Supersedes #4421. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

codecov · 2026-02-20T09:08:16Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 55.66%. Comparing base (e42d8e1) to head (e26fd6b).
⚠️ Report is 4 commits behind head on main.

Additional details and impacted files

Files with missing lines	Coverage Δ
ddtrace/tracer/option.go	`83.89% <100.00%> (+0.07%)`	⬆️
ddtrace/tracer/telemetry.go	`89.02% <100.00%> (+3.83%)`	⬆️

... and 4 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

darccio

LGTM

pr-commenter · 2026-02-20T09:25:48Z

Benchmarks

Benchmark execution time: 2026-02-20 09:25:45

Comparing candidate commit e26fd6b in PR branch eliottness/make-sure-telemetry-agent-endpoint-is-available with baseline commit feef6ce in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 156 metrics, 8 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

🟩 = significantly better candidate vs. baseline
🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

genesor

LGTM

eliottness requested review from a team as code owners February 20, 2026 09:04

darccio approved these changes Feb 20, 2026

View reviewed changes

genesor approved these changes Feb 20, 2026

View reviewed changes

eliottness merged commit 6af22e4 into main Feb 20, 2026
209 checks passed

eliottness deleted the eliottness/make-sure-telemetry-agent-endpoint-is-available branch February 20, 2026 14:38

magnetikonline mentioned this pull request Feb 20, 2026

fix(telemetry): Disable starting tracer telemetry client if running as Lambda function #4421

Closed

9 tasks

kakkoyun mentioned this pull request Feb 24, 2026

feat(tracer): periodically poll agent /info endpoint for dynamic capability updates #4451

Merged

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(telemetry): skip agent proxy URL when agent doesn't expose /telemetry/proxy/#4454

fix(telemetry): skip agent proxy URL when agent doesn't expose /telemetry/proxy/#4454
eliottness merged 1 commit into
mainfrom
eliottness/make-sure-telemetry-agent-endpoint-is-available

eliottness commented Feb 20, 2026

Uh oh!

codecov Bot commented Feb 20, 2026 •

edited

Loading

Uh oh!

darccio left a comment

Uh oh!

pr-commenter Bot commented Feb 20, 2026

Explanation

More details about the CI and significant changes

Uh oh!

genesor left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

eliottness commented Feb 20, 2026

What does this PR do?

Changes

Why this is better than #4421

Fixes

Reviewer's Checklist

Uh oh!

codecov Bot commented Feb 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

darccio left a comment

Choose a reason for hiding this comment

Uh oh!

pr-commenter Bot commented Feb 20, 2026

Benchmarks

Explanation

More details about the CI and significant changes

Uh oh!

genesor left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov Bot commented Feb 20, 2026 •

edited

Loading