feat(tracer): periodically poll agent /info endpoint for dynamic capability updates#4451
Conversation
b57e131 to
0164602
Compare
8dab341 to
685a846
Compare
|
Related #4454 |
8f3ed94 to
d76a2f0
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files
🚀 New features to boost your workflow:
|
mtoffl01
left a comment
There was a problem hiding this comment.
nothing blocking, but some questions.
| return agentFeatures{} | ||
| } | ||
| features, err := fetchAgentFeatures(agentURL, httpClient) | ||
| if err != nil && !errors.Is(err, errAgentFeaturesNotSupported) { |
There was a problem hiding this comment.
So if agent is <7.28, we just don't need/use agentFeatures?
There was a problem hiding this comment.
Correct — agents before 7.28 don't support /info, so fetchAgentFeatures returns errAgentFeaturesNotSupported and the tracer runs with zero-valued features. The polling goroutine silently ignores this error (only logs unexpected failures).
| interval := c.agentInfoPollInterval | ||
| if interval <= 0 { | ||
| interval = defaultAgentInfoPollInterval | ||
| } |
There was a problem hiding this comment.
I don't see where agentInfoPollInterval is configurable; was that your intention? Right now it will always be defaultAgentInfoPollInterval, no matter what.
There was a problem hiding this comment.
Right now the option to configure it is for testing only, which I think is okay? But will leave if up to our config/SDK lords to decide 😉
There was a problem hiding this comment.
Intentionally internal/testing-only for now. No public env var planned at this stage — we can add one later if there's a use case.
| t.config.agent.update(func(current agentFeatures) agentFeatures { | ||
| f := newFeatures | ||
| f.v1ProtocolAvailable = current.v1ProtocolAvailable | ||
| f.StatsdPort = current.StatsdPort | ||
| f.evpProxyV2 = current.evpProxyV2 | ||
| f.metaStructAvailable = current.metaStructAvailable | ||
| f.featureFlags = maps.Clone(current.featureFlags) // defensive copy of map | ||
| f.peerTags = slices.Clone(newFeatures.peerTags) // defensive copy of slice | ||
| f.defaultEnv = current.defaultEnv | ||
| return f | ||
| }) |
There was a problem hiding this comment.
I don't think anything is stopping future developers from adding the static fields in here.
If it's important that those don't change, perhaps there is better way we can enforce it; like by breaking type agentFeatures up into type staticFields and type dynamicFields or something similar.
There was a problem hiding this comment.
Good idea for longer-term safety. For this PR, the field-preservation approach with test coverage is the pragmatic choice. A follow-up could introduce staticAgentFeatures / dynamicAgentFeatures types if the field list keeps growing.
| if (!c.agent.reachable || c.agent.hasTelemetryProxy) && !c.internalConfig.LogToStdout() { | ||
| a := c.agent.load() | ||
| if (!a.reachable || a.hasTelemetryProxy) && !c.internalConfig.LogToStdout() { | ||
| cfg.AgentURL = c.agentURL.String() |
There was a problem hiding this comment.
Is it possible for the agent configuration to change later on when the proxy endpoint becomes available? If we're loading c.agent here, it makes me think that it can change, and thus cfg.AgentURL could become outdated eventually.
There was a problem hiding this comment.
reachable and hasTelemetryProxy are both frozen at their startup values (static fields in refreshAgentFeatures). So cfg.AgentURL is set once during startTelemetry() and the telemetry client config is immutable after creation — no risk of it becoming outdated.
| {Name: "agent_feature_drop_p0s", Value: c.agent.load().DropP0s}, | ||
| {Name: "stats_computation_enabled", Value: c.canComputeStats()}, | ||
| {Name: "dogstatsd_port", Value: c.agent.StatsdPort}, | ||
| {Name: "dogstatsd_port", Value: c.agent.load().StatsdPort}, |
There was a problem hiding this comment.
AFAICT, these values in the telemetryConfig is also being used elsewhere in the code without being re-evaluated. Is it possible for these values to become outdated?
There was a problem hiding this comment.
DropP0s and StatsdPort are classified as static fields — they're frozen at startup values in refreshAgentFeatures and never change during polling. Same situation as newConcentrator (darccio confirmed in the stats.go thread).
| interval := c.agentInfoPollInterval | ||
| if interval <= 0 { | ||
| interval = defaultAgentInfoPollInterval | ||
| } |
There was a problem hiding this comment.
Right now the option to configure it is for testing only, which I think is okay? But will leave if up to our config/SDK lords to decide 😉
| return agentFeatures{} | ||
| } | ||
| features, err := fetchAgentFeatures(agentURL, httpClient) | ||
| if err != nil && !errors.Is(err, errAgentFeaturesNotSupported) { |
| BucketInterval: defaultStatsBucketSize, | ||
| } | ||
| env := c.agent.defaultEnv | ||
| env := c.agent.load().defaultEnv |
There was a problem hiding this comment.
Here we are in a similar case than in newConfig but there are a lot of other config dependencies here, so we can't apply the same pattern of just passing the required values.
There was a problem hiding this comment.
Ack. Too many config dependencies in newConcentrator to apply the same local-var pattern. The static field preservation in refreshAgentFeatures keeps these values safe.
| traceEnabled, traceEnabledOrigin := c.enabled.getCurrentAndOrigin() | ||
| telemetryConfigs := []telemetry.Configuration{ | ||
| {Name: "agent_feature_drop_p0s", Value: c.agent.DropP0s}, | ||
| {Name: "agent_feature_drop_p0s", Value: c.agent.load().DropP0s}, |
There was a problem hiding this comment.
Ack. Same situation: startTelemetry() runs once at startup, so the values are frozen. Hoisted to a local var for consistency (same pattern as newConcentrator).
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 175e4247a0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // components at startup (transport URL, statsd address, obfuscator config, etc.) | ||
| // are preserved from the current snapshot so a poll can never change them. | ||
| func (t *tracer) refreshAgentFeatures() { | ||
| newFeatures, err := fetchAgentFeatures(t.config.agentURL, t.config.httpClient) |
There was a problem hiding this comment.
Make /info polling request cancelable during tracer shutdown
newTracer adds a long-lived polling goroutine that is part of t.wg, and Stop() waits for that waitgroup to drain before returning; however, refreshAgentFeatures performs httpClient.Get without a stop-aware context, so if /info is slow or hangs when shutdown begins, the goroutine cannot observe t.stop until the HTTP timeout fires, delaying shutdown by the full agent timeout (10s by default, 45s in CI visibility mode). Using NewRequestWithContext tied to tracer shutdown (or equivalent cancellation) would avoid this blocking behavior.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Addressed — fetchAgentFeatures now takes a context.Context parameter. The polling goroutine creates a context derived from t.stop, so in-flight HTTP requests are canceled promptly during shutdown.
175e424 to
6605ec4
Compare
…e runtime updates Replace the plain agentFeatures value field on config with atomicAgentFeatures, a thin wrapper around atomic.Pointer[agentFeatures]. This provides lock-free reads on the hot path (canComputeStats, StartSpan, stats flush) while keeping all fields consistent — they update as a single snapshot from one /info response. Extract fetchAgentFeatures() from loadAgentFeatures() so the HTTP+parse logic can be reused by the periodic polling goroutine added in the next commit. All callers updated to use .load() / .store(); no behaviour change.
…nges Add a polling goroutine that refreshes agentFeatures from the Datadog Agent's /info endpoint every 5 seconds (overridable in tests via config). Dynamic fields (DropP0s, Stats, peerTags, spanEventsAvailable, obfuscationVersion) are updated from each successful response. Static fields that are baked into transport, stats, and appsec components at startup (v1ProtocolAvailable, StatsdPort, evpProxyV2, metaStructAvailable, featureFlags, defaultEnv) are always preserved from the startup snapshot. When a poll fails the last-known-good snapshot is retained. The goroutine exits cleanly when the tracer is stopped via the existing t.stop channel. Add table-driven tests covering: static-field preservation, dynamic updates, error resilience, and goroutine shutdown.
- Add CAS-loop update() to atomicAgentFeatures to prevent potential
load-modify-store races if refreshAgentFeatures is ever called
concurrently in the future
- Add errAgentFeaturesNotSupported sentinel for 404 responses so both
startup and polling callers uniformly treat 404 as "keep last good"
without zeroing dynamic fields
- Bound HTTP response body with io.LimitReader(1 MiB) before JSON decode
to prevent DoS from a misbehaving agent
- Use agentURL.JoinPath("info") instead of fmt.Sprintf for safe URL
construction
- Add agentEnabled() helper on config to consolidate the three-part
condition for launching the poll goroutine, replacing the inline copy
- Fix comment typo: agentInfoPollInterval → defaultAgentInfoPollInterval
- Use maps.Clone for featureFlags in refreshAgentFeatures for defensive
isolation between snapshots
- Import errors and maps in tracer.go
- Test: add Stats to dynamic-field assertions in
TestRefreshAgentFeaturesPreservesStaticFields
- Test: add TestPollAgentInfoRetainsLastKnownGoodOn404
- Test: use any instead of interface{} in handlers; add error handling
for json.NewEncoder().Encode(); remove time.Sleep from goroutine-stop
test (unnecessary)
- refreshAgentFeatures: use local copy f := newFeatures in the update() closure so CAS retries always start from the original fetched state (pure-transform contract required by the CAS loop) - refreshAgentFeatures: clone peerTags slice alongside featureFlags map to prevent the new snapshot from aliasing the old one's backing array - fetchAgentFeatures: return a clear error for non-200/non-404 HTTP status codes instead of falling through to JSON decode on 5xx responses - TestPollAgentInfoUpdatesFeaturesDynamically: load agent snapshot once in the assert.Eventually predicate to avoid a TOCTOU double-load - TestPollAgentInfoGoroutineStopsOnTracerStop: replace time.After with time.NewTimer + defer timer.Stop() to prevent timer resource leak - TestPollAgentInfoRetainsLastKnownGoodOnError: add defer srv.Close() so the server is always reclaimed even on early test failure - TestPollAgentInfoRetainsLastKnownGoodOn404: use w.WriteHeader instead of http.NotFound(w, nil) to avoid passing a nil *http.Request
Place "slices" after rt "runtime/trace" in alphabetical import order. Remove extra alignment spaces before inline comments to satisfy gofmt.
…tFeatures refactor Upstream added c.agent.reachable and c.agent.hasTelemetryProxy direct field accesses in telemetry.go (e26fd6b). After rebasing our atomicAgentFeatures wrapper onto upstream/main, those accesses must go through agent.load().
…atures Both fields are consumed only by startTelemetry(), which runs once at startup and produces an immutable telemetry client. Updating them during periodic /info polling would have no observable effect, so preserve their startup values alongside the other static fields. Also extends TestRefreshAgentFeaturesPreservesStaticFields to include /telemetry/proxy/ in the startup endpoint list and assert that both fields remain frozen across a manual refresh call.
- Use local af variable in newConfig to avoid redundant load() calls - Change resolveDogstatsdAddr to take (addr string, af agentFeatures) instead of *config - Extract pollAgentInfo method from inline goroutine in newTracer - Add context.Context to fetchAgentFeatures; use NewRequestWithContext so in-flight /info requests cancel promptly on tracer shutdown - Fix agentInfoPollInterval comment to drop file reference
- telemetry.go: hoist dual c.agent.load() calls to a single local var so DropP0s and StatsdPort come from the same atomic snapshot; removes the now-redundant second load() near the proxy check (same pattern as startTelemetry's existing usage at line 120). - tracer.go: document that the bridging goroutine lifetime is bounded by defer cancel() so no WaitGroup tracking is needed. - tracer.go: document that f is a shallow copy of newFeatures and that reference-typed fields must be cloned to avoid shared backing storage across CAS retries. - option.go: drain the 404 response body before returning so the underlying HTTP connection can be reused (keep-alive).
12467b1 to
3a1d706
Compare
|
Waiting for @mtoffl01's approval. |
| // agentEnabled reports whether the tracer should communicate with the agent. | ||
| // The agent is considered disabled in serverless (LogToStdout), when the | ||
| // tracer itself is disabled, or in CI visibility agentless mode. | ||
| func (c *config) agentEnabled() bool { |
6e49f81
into
DataDog:main
What does this PR do?
The tracer currently queries the Datadog Agent's
/infoendpoint once at startup. Any agent-side capability changes (peer-tag policies, span events support,client_drop_p0s, etc.) require a full tracer restart to take effect.This PR adds periodic polling of
/infoevery 5 seconds so the tracer picks up agent config changes dynamically, without a restart.Thread-safety design:
agentFeaturesis wrapped inatomicAgentFeaturesbacked byatomic.Pointer[agentFeatures], giving lock-free snapshot reads on the hot path. A CAS-loopupdate()method prevents lost-update races if two writers ever race. All reads were updated fromc.agent.Xtoc.agent.load().X.Static vs dynamic field split: Fields baked into components at init (transport URL, statsd address, AppSec config, concentrator aggregation key, obfuscator feature flags) are preserved across every poll. Only fields safe to update at runtime (DropP0s, Stats, peerTags, spanEventsAvailable, obfuscationVersion) are refreshed.
Motivation
Reviewer's Checklist
TestRefreshAgentFeaturesPreservesStaticFields— static fields frozen, dynamic fields updatedTestPollAgentInfoUpdatesFeaturesDynamically— convergence verified viaassert.EventuallyTestPollAgentInfoRetainsLastKnownGoodOnError— network failure preserves last known goodTestPollAgentInfoRetainsLastKnownGoodOn404— 404 (agent downgrade) preserves last known goodTestPollAgentInfoGoroutineStopsOnTracerStop— no goroutine leak onStop()/infoendpoint — no new agent interaction)make lintlocally.make testlocally.make generatelocally.make fix-moduleslocally. (no go.mod changes)