Add Dynamic Instrumentation system probe feature#3245
Merged
grantseltzer merged 2 commits intoJul 10, 2026
Conversation
Adds spec.features.dynamicInstrumentation.enabled to the DatadogAgent CRD and a dyninst feature handler that turns it into the pod-spec changes system-probe needs (HostPID, capabilities, procdir/cgroups/ debugfs/socket volumes, and the DD_DYNAMIC_INSTRUMENTATION_ENABLED / DD_SYSTEM_PROBE_ENABLED env vars). Enabling the flag pulls in the system-probe container automatically rather than requiring a separate toggle. This intentionally omits the node-scoped remote-config path and per-node DatadogAgentProfile management from the earlier draft of this work: Dynamic Instrumentation will only ever be enabled cluster-wide, and services move between nodes too often for node-level targeting to make sense. Delivery of the flag itself (remote, controlled rollout with rollback) is handled by Fleet Automation's existing operator experiment mechanism, which patches this same spec field directly, so no bespoke remote-config wiring is needed here.
grantseltzer
force-pushed
the
grantseltzer/dynamic-instrumentation-cluster-wide
branch
from
July 9, 2026 20:02
9118cd2 to
59f53f7
Compare
|
🎯 Code Coverage (details) 🔗 Commit SHA: d05fdeb | Docs | Datadog PR Page | Give us feedback! |
levan-m
approved these changes
Jul 9, 2026
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 59f53f7d18
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
system-probe runs with a read-only root filesystem, and the dyninst feature handler didn't mount anywhere for the Dynamic Instrumentation module's own writable state (the debugger-probes tombstone file and the SymDB upload cache), so the module had nowhere to write. The path matters here: it needs to match run_path/system-probe/ dynamic-instrumentation, not /tmp/datadog-agent/system-probe/ dynamic-instrumentation, since datadog-agent#53286 just moved this state out of /tmp for a security fix (world-writable /tmp was a symlink-attack vector). Mounting at the old /tmp path would satisfy nothing, since the module doesn't write there anymore. Adds a dedicated emptyDir volume for this, mounted only on the system-probe container, at a path derived from the same RunPathVolume Mount constant logcollection already uses for its own run-path mount, so it stays in sync if that base path ever changes. Verified with go build, go vet, and go test across the datadogagent and datadogagentinternal controller packages, including updating dyninst's own feature test to expect the new volume/mount. Confirmed none of the renderer golden-file fixtures needed updating, since none of them enable dynamicInstrumentation (they only render its default off value).
grantseltzer
deleted the
grantseltzer/dynamic-instrumentation-cluster-wide
branch
July 10, 2026 13:09
|
The backport to To backport manually, run these commands in your terminal: # Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-v1.29 v1.29
# Navigate to the new working tree
cd .worktrees/backport-v1.29
# Create a new branch
git switch --create backport-3245-to-v1.29
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 eb0751427e141cb95638457563bd88e968646a4b
# Push it to GitHub
git push --set-upstream origin backport-3245-to-v1.29
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-v1.29Then, create a pull request where the |
levan-m
added a commit
that referenced
this pull request
Jul 13, 2026
* Add Dynamic Instrumentation system probe feature Adds spec.features.dynamicInstrumentation.enabled to the DatadogAgent CRD and a dyninst feature handler that turns it into the pod-spec changes system-probe needs (HostPID, capabilities, procdir/cgroups/ debugfs/socket volumes, and the DD_DYNAMIC_INSTRUMENTATION_ENABLED / DD_SYSTEM_PROBE_ENABLED env vars). Enabling the flag pulls in the system-probe container automatically rather than requiring a separate toggle. This intentionally omits the node-scoped remote-config path and per-node DatadogAgentProfile management from the earlier draft of this work: Dynamic Instrumentation will only ever be enabled cluster-wide, and services move between nodes too often for node-level targeting to make sense. Delivery of the flag itself (remote, controlled rollout with rollback) is handled by Fleet Automation's existing operator experiment mechanism, which patches this same spec field directly, so no bespoke remote-config wiring is needed here. * Mount a writable state dir for the DI system-probe module system-probe runs with a read-only root filesystem, and the dyninst feature handler didn't mount anywhere for the Dynamic Instrumentation module's own writable state (the debugger-probes tombstone file and the SymDB upload cache), so the module had nowhere to write. The path matters here: it needs to match run_path/system-probe/ dynamic-instrumentation, not /tmp/datadog-agent/system-probe/ dynamic-instrumentation, since datadog-agent#53286 just moved this state out of /tmp for a security fix (world-writable /tmp was a symlink-attack vector). Mounting at the old /tmp path would satisfy nothing, since the module doesn't write there anymore. Adds a dedicated emptyDir volume for this, mounted only on the system-probe container, at a path derived from the same RunPathVolume Mount constant logcollection already uses for its own run-path mount, so it stays in sync if that base path ever changes. Verified with go build, go vet, and go test across the datadogagent and datadogagentinternal controller packages, including updating dyninst's own feature test to expect the new volume/mount. Confirmed none of the renderer golden-file fixtures needed updating, since none of them enable dynamicInstrumentation (they only render its default off value). (cherry picked from commit eb07514) Co-authored-by: grantseltzer <[email protected]> Co-authored-by: Timothée Bavelier <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds
spec.features.dynamicInstrumentation.enabledto the DatadogAgent CRD and adyninstfeature handler that turns it on.Motivation
This is the operator-side plumbing for letting users enable the Dynamic Instrumentation system-probe module (used by the Go live debugger) for a whole cluster from the Datadog UI, via Fleet Automation.
Delivery of the flag itself (a remote, controlled rollout with automatic rollback) will go through Fleet Automation's existing operator experiment mechanism, which patches this spec field directly.
Additional Notes
The field is inert until Fleet Automation's schema (dd-source#10080) and a UI to send the config patch are in place, so this is safe to merge ahead of those.
Minimum Agent Versions
Describe your test plan
make generate && make manifests, regenerated deepcopy, openapi, CRD manifests, and docsgo build ./...andgo vet ./..., both cleango test ./internal/controller/datadogagent/... ./internal/controller/datadogagentinternal/..., all passingspec.features.dynamicInstrumentation.enabled: trueto confirm the node agent picksup the system-probe container and the expected volumes/env vars. Worth doing before merge.
Checklist
bug,enhancement,refactoring,documentation,tooling, and/ordependenciesqa/skip-qalabel