Skip to content

feat(tracer): periodically poll agent /info endpoint for dynamic capability updates#4451

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 9 commits into
DataDog:mainfrom
kakkoyun:kakkoyun/APMLP-860
Mar 17, 2026
Merged

feat(tracer): periodically poll agent /info endpoint for dynamic capability updates#4451
gh-worker-dd-mergequeue-cf854d[bot] merged 9 commits into
DataDog:mainfrom
kakkoyun:kakkoyun/APMLP-860

Conversation

@kakkoyun

@kakkoyun kakkoyun commented Feb 19, 2026

Copy link
Copy Markdown
Member

What does this PR do?

The tracer currently queries the Datadog Agent's /info endpoint once at startup. Any agent-side capability changes (peer-tag policies, span events support, client_drop_p0s, etc.) require a full tracer restart to take effect.

This PR adds periodic polling of /info every 5 seconds so the tracer picks up agent config changes dynamically, without a restart.

Thread-safety design: agentFeatures is wrapped in atomicAgentFeatures backed by atomic.Pointer[agentFeatures], giving lock-free snapshot reads on the hot path. A CAS-loop update() method prevents lost-update races if two writers ever race. All reads were updated from c.agent.X to c.agent.load().X.

Static vs dynamic field split: Fields baked into components at init (transport URL, statsd address, AppSec config, concentrator aggregation key, obfuscator feature flags) are preserved across every poll. Only fields safe to update at runtime (DropP0s, Stats, peerTags, spanEventsAvailable, obfuscationVersion) are refreshed.

Motivation

Reviewer's Checklist

  • Changed code has unit tests for its functionality at or near 100% coverage.
    • TestRefreshAgentFeaturesPreservesStaticFields — static fields frozen, dynamic fields updated
    • TestPollAgentInfoUpdatesFeaturesDynamically — convergence verified via assert.Eventually
    • TestPollAgentInfoRetainsLastKnownGoodOnError — network failure preserves last known good
    • TestPollAgentInfoRetainsLastKnownGoodOn404 — 404 (agent downgrade) preserves last known good
    • TestPollAgentInfoGoroutineStopsOnTracerStop — no goroutine leak on Stop()
  • System-Tests covering this feature have been added and enabled with the va.b.c-dev version tag.
  • There is a benchmark for any new code, or changes to existing code.
  • If this interacts with the agent in a new way, a system test has been added. (polls existing /info endpoint — no new agent interaction)
  • New code is free of linting errors. You can check this by running make lint locally.
  • New code doesn't break existing tests. You can check this by running make test locally.
  • Add an appropriate team label so this PR gets put in the right place for the release notes.
  • All generated files are up to date. You can check this by running make generate locally.
  • Non-trivial go.mod changes, e.g. adding new modules, are reviewed by @DataDog/dd-trace-go-guild. Make sure all nested modules are up to date by running make fix-modules locally. (no go.mod changes)

@kakkoyun
kakkoyun requested review from a team as code owners February 19, 2026 15:34
@kakkoyun
kakkoyun marked this pull request as draft February 19, 2026 15:38
Comment thread ddtrace/tracer/tracer.go Outdated
@kakkoyun
kakkoyun requested a review from hannahkm February 20, 2026 13:32
@kakkoyun kakkoyun added the AI Assisted AI/LLM assistance used in this PR (partially or fully) label Feb 23, 2026
@kakkoyun

Copy link
Copy Markdown
Member Author

Related #4454

@kakkoyun
kakkoyun marked this pull request as ready for review March 10, 2026 09:59
@codecov

codecov Bot commented Mar 10, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.33028% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 60.07%. Comparing base (0a8b48b) to head (3a1d706).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
ddtrace/tracer/option.go 95.74% 2 Missing ⚠️
ddtrace/tracer/tracer.go 96.15% 2 Missing ⚠️
Additional details and impacted files
Files with missing lines Coverage Δ
ddtrace/tracer/log.go 86.81% <100.00%> (ø)
ddtrace/tracer/stats.go 98.50% <100.00%> (ø)
ddtrace/tracer/telemetry.go 89.15% <100.00%> (ø)
ddtrace/tracer/option.go 83.82% <95.74%> (ø)
ddtrace/tracer/tracer.go 88.07% <96.15%> (ø)

... and 426 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@mtoffl01 mtoffl01 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nothing blocking, but some questions.

Comment thread ddtrace/tracer/option.go Outdated
Comment thread ddtrace/tracer/option.go
return agentFeatures{}
}
features, err := fetchAgentFeatures(agentURL, httpClient)
if err != nil && !errors.Is(err, errAgentFeaturesNotSupported) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So if agent is <7.28, we just don't need/use agentFeatures?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So it seems.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct — agents before 7.28 don't support /info, so fetchAgentFeatures returns errAgentFeaturesNotSupported and the tracer runs with zero-valued features. The polling goroutine silently ignores this error (only logs unexpected failures).

Comment thread ddtrace/tracer/tracer.go
Comment on lines +539 to +542
interval := c.agentInfoPollInterval
if interval <= 0 {
interval = defaultAgentInfoPollInterval
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see where agentInfoPollInterval is configurable; was that your intention? Right now it will always be defaultAgentInfoPollInterval, no matter what.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right now the option to configure it is for testing only, which I think is okay? But will leave if up to our config/SDK lords to decide 😉

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Intentionally internal/testing-only for now. No public env var planned at this stage — we can add one later if there's a use case.

Comment thread ddtrace/tracer/tracer.go
Comment on lines +578 to +588
t.config.agent.update(func(current agentFeatures) agentFeatures {
f := newFeatures
f.v1ProtocolAvailable = current.v1ProtocolAvailable
f.StatsdPort = current.StatsdPort
f.evpProxyV2 = current.evpProxyV2
f.metaStructAvailable = current.metaStructAvailable
f.featureFlags = maps.Clone(current.featureFlags) // defensive copy of map
f.peerTags = slices.Clone(newFeatures.peerTags) // defensive copy of slice
f.defaultEnv = current.defaultEnv
return f
})

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think anything is stopping future developers from adding the static fields in here.

If it's important that those don't change, perhaps there is better way we can enforce it; like by breaking type agentFeatures up into type staticFields and type dynamicFields or something similar.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea for longer-term safety. For this PR, the field-preservation approach with test coverage is the pragmatic choice. A follow-up could introduce staticAgentFeatures / dynamicAgentFeatures types if the field list keeps growing.

if (!c.agent.reachable || c.agent.hasTelemetryProxy) && !c.internalConfig.LogToStdout() {
a := c.agent.load()
if (!a.reachable || a.hasTelemetryProxy) && !c.internalConfig.LogToStdout() {
cfg.AgentURL = c.agentURL.String()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible for the agent configuration to change later on when the proxy endpoint becomes available? If we're loading c.agent here, it makes me think that it can change, and thus cfg.AgentURL could become outdated eventually.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reachable and hasTelemetryProxy are both frozen at their startup values (static fields in refreshAgentFeatures). So cfg.AgentURL is set once during startTelemetry() and the telemetry client config is immutable after creation — no risk of it becoming outdated.

Comment thread ddtrace/tracer/telemetry.go Outdated
Comment on lines +42 to +44
{Name: "agent_feature_drop_p0s", Value: c.agent.load().DropP0s},
{Name: "stats_computation_enabled", Value: c.canComputeStats()},
{Name: "dogstatsd_port", Value: c.agent.StatsdPort},
{Name: "dogstatsd_port", Value: c.agent.load().StatsdPort},

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AFAICT, these values in the telemetryConfig is also being used elsewhere in the code without being re-evaluated. Is it possible for these values to become outdated?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DropP0s and StatsdPort are classified as static fields — they're frozen at startup values in refreshAgentFeatures and never change during polling. Same situation as newConcentrator (darccio confirmed in the stats.go thread).

Comment thread ddtrace/tracer/tracer.go
Comment on lines +539 to +542
interval := c.agentInfoPollInterval
if interval <= 0 {
interval = defaultAgentInfoPollInterval
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right now the option to configure it is for testing only, which I think is okay? But will leave if up to our config/SDK lords to decide 😉

Comment thread ddtrace/tracer/option.go Outdated
Comment thread ddtrace/tracer/option.go Outdated
Comment thread ddtrace/tracer/option.go Outdated
Comment thread ddtrace/tracer/option.go Outdated
Comment thread ddtrace/tracer/option.go Outdated
Comment thread ddtrace/tracer/option.go
return agentFeatures{}
}
features, err := fetchAgentFeatures(agentURL, httpClient)
if err != nil && !errors.Is(err, errAgentFeaturesNotSupported) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So it seems.

Comment thread ddtrace/tracer/stats.go
BucketInterval: defaultStatsBucketSize,
}
env := c.agent.defaultEnv
env := c.agent.load().defaultEnv

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here we are in a similar case than in newConfig but there are a lot of other config dependencies here, so we can't apply the same pattern of just passing the required values.

@kakkoyun kakkoyun Mar 13, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ack. Too many config dependencies in newConcentrator to apply the same local-var pattern. The static field preservation in refreshAgentFeatures keeps these values safe.

Comment thread ddtrace/tracer/telemetry.go Outdated
traceEnabled, traceEnabledOrigin := c.enabled.getCurrentAndOrigin()
telemetryConfigs := []telemetry.Configuration{
{Name: "agent_feature_drop_p0s", Value: c.agent.DropP0s},
{Name: "agent_feature_drop_p0s", Value: c.agent.load().DropP0s},

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same like in newConcentrator.

@kakkoyun kakkoyun Mar 13, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ack. Same situation: startTelemetry() runs once at startup, so the values are frozen. Hoisted to a local var for consistency (same pattern as newConcentrator).

Comment thread ddtrace/tracer/tracer.go Outdated
@darccio

darccio commented Mar 12, 2026

Copy link
Copy Markdown
Member

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 175e4247a0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread ddtrace/tracer/tracer.go Outdated
// components at startup (transport URL, statsd address, obfuscator config, etc.)
// are preserved from the current snapshot so a poll can never change them.
func (t *tracer) refreshAgentFeatures() {
newFeatures, err := fetchAgentFeatures(t.config.agentURL, t.config.httpClient)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Make /info polling request cancelable during tracer shutdown

newTracer adds a long-lived polling goroutine that is part of t.wg, and Stop() waits for that waitgroup to drain before returning; however, refreshAgentFeatures performs httpClient.Get without a stop-aware context, so if /info is slow or hangs when shutdown begins, the goroutine cannot observe t.stop until the HTTP timeout fires, delaying shutdown by the full agent timeout (10s by default, 45s in CI visibility mode). Using NewRequestWithContext tied to tracer shutdown (or equivalent cancellation) would avoid this blocking behavior.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed — fetchAgentFeatures now takes a context.Context parameter. The polling goroutine creates a context derived from t.stop, so in-flight HTTP requests are canceled promptly during shutdown.

@kakkoyun
kakkoyun force-pushed the kakkoyun/APMLP-860 branch from 175e424 to 6605ec4 Compare March 13, 2026 10:27
@kakkoyun
kakkoyun requested a review from darccio March 13, 2026 10:31
@kakkoyun
kakkoyun requested a review from darccio March 13, 2026 11:44
…e runtime updates

Replace the plain agentFeatures value field on config with atomicAgentFeatures,
a thin wrapper around atomic.Pointer[agentFeatures]. This provides lock-free
reads on the hot path (canComputeStats, StartSpan, stats flush) while keeping
all fields consistent — they update as a single snapshot from one /info response.

Extract fetchAgentFeatures() from loadAgentFeatures() so the HTTP+parse logic
can be reused by the periodic polling goroutine added in the next commit.

All callers updated to use .load() / .store(); no behaviour change.
…nges

Add a polling goroutine that refreshes agentFeatures from the Datadog Agent's
/info endpoint every 5 seconds (overridable in tests via config). Dynamic
fields (DropP0s, Stats, peerTags, spanEventsAvailable, obfuscationVersion) are
updated from each successful response. Static fields that are baked into
transport, stats, and appsec components at startup (v1ProtocolAvailable,
StatsdPort, evpProxyV2, metaStructAvailable, featureFlags, defaultEnv) are
always preserved from the startup snapshot.

When a poll fails the last-known-good snapshot is retained. The goroutine exits
cleanly when the tracer is stopped via the existing t.stop channel.

Add table-driven tests covering: static-field preservation, dynamic updates,
error resilience, and goroutine shutdown.
- Add CAS-loop update() to atomicAgentFeatures to prevent potential
  load-modify-store races if refreshAgentFeatures is ever called
  concurrently in the future
- Add errAgentFeaturesNotSupported sentinel for 404 responses so both
  startup and polling callers uniformly treat 404 as "keep last good"
  without zeroing dynamic fields
- Bound HTTP response body with io.LimitReader(1 MiB) before JSON decode
  to prevent DoS from a misbehaving agent
- Use agentURL.JoinPath("info") instead of fmt.Sprintf for safe URL
  construction
- Add agentEnabled() helper on config to consolidate the three-part
  condition for launching the poll goroutine, replacing the inline copy
- Fix comment typo: agentInfoPollInterval → defaultAgentInfoPollInterval
- Use maps.Clone for featureFlags in refreshAgentFeatures for defensive
  isolation between snapshots
- Import errors and maps in tracer.go
- Test: add Stats to dynamic-field assertions in
  TestRefreshAgentFeaturesPreservesStaticFields
- Test: add TestPollAgentInfoRetainsLastKnownGoodOn404
- Test: use any instead of interface{} in handlers; add error handling
  for json.NewEncoder().Encode(); remove time.Sleep from goroutine-stop
  test (unnecessary)
- refreshAgentFeatures: use local copy f := newFeatures in the update()
  closure so CAS retries always start from the original fetched state
  (pure-transform contract required by the CAS loop)
- refreshAgentFeatures: clone peerTags slice alongside featureFlags map to
  prevent the new snapshot from aliasing the old one's backing array
- fetchAgentFeatures: return a clear error for non-200/non-404 HTTP status
  codes instead of falling through to JSON decode on 5xx responses
- TestPollAgentInfoUpdatesFeaturesDynamically: load agent snapshot once
  in the assert.Eventually predicate to avoid a TOCTOU double-load
- TestPollAgentInfoGoroutineStopsOnTracerStop: replace time.After with
  time.NewTimer + defer timer.Stop() to prevent timer resource leak
- TestPollAgentInfoRetainsLastKnownGoodOnError: add defer srv.Close()
  so the server is always reclaimed even on early test failure
- TestPollAgentInfoRetainsLastKnownGoodOn404: use w.WriteHeader instead
  of http.NotFound(w, nil) to avoid passing a nil *http.Request
Place "slices" after rt "runtime/trace" in alphabetical import order.
Remove extra alignment spaces before inline comments to satisfy gofmt.
…tFeatures refactor

Upstream added c.agent.reachable and c.agent.hasTelemetryProxy direct field
accesses in telemetry.go (e26fd6b). After rebasing our atomicAgentFeatures
wrapper onto upstream/main, those accesses must go through agent.load().
…atures

Both fields are consumed only by startTelemetry(), which runs once at
startup and produces an immutable telemetry client. Updating them during
periodic /info polling would have no observable effect, so preserve their
startup values alongside the other static fields.

Also extends TestRefreshAgentFeaturesPreservesStaticFields to include
/telemetry/proxy/ in the startup endpoint list and assert that both
fields remain frozen across a manual refresh call.
- Use local af variable in newConfig to avoid redundant load() calls
- Change resolveDogstatsdAddr to take (addr string, af agentFeatures) instead of *config
- Extract pollAgentInfo method from inline goroutine in newTracer
- Add context.Context to fetchAgentFeatures; use NewRequestWithContext so in-flight /info requests cancel promptly on tracer shutdown
- Fix agentInfoPollInterval comment to drop file reference
- telemetry.go: hoist dual c.agent.load() calls to a single local var
  so DropP0s and StatsdPort come from the same atomic snapshot; removes
  the now-redundant second load() near the proxy check (same pattern
  as startTelemetry's existing usage at line 120).
- tracer.go: document that the bridging goroutine lifetime is bounded by
  defer cancel() so no WaitGroup tracking is needed.
- tracer.go: document that f is a shallow copy of newFeatures and that
  reference-typed fields must be cloned to avoid shared backing storage
  across CAS retries.
- option.go: drain the 404 response body before returning so the
  underlying HTTP connection can be reused (keep-alive).
@kakkoyun
kakkoyun force-pushed the kakkoyun/APMLP-860 branch from 12467b1 to 3a1d706 Compare March 13, 2026 11:47
@kakkoyun
kakkoyun requested review from hannahkm and mtoffl01 and removed request for genesor March 13, 2026 11:47

@hannahkm hannahkm left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! 🎉

@kakkoyun

Copy link
Copy Markdown
Member Author

Waiting for @mtoffl01's approval.

Comment thread ddtrace/tracer/option.go
// agentEnabled reports whether the tracer should communicate with the agent.
// The agent is considered disabled in serverless (LogToStdout), when the
// tracer itself is disabled, or in CI visibility agentless mode.
func (c *config) agentEnabled() bool {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AI Assisted AI/LLM assistance used in this PR (partially or fully) mergequeue-status: done

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants