[Backport v1.29] Add Dynamic Instrumentation system probe feature (#3245)#3252
Merged
Conversation
* Add Dynamic Instrumentation system probe feature Adds spec.features.dynamicInstrumentation.enabled to the DatadogAgent CRD and a dyninst feature handler that turns it into the pod-spec changes system-probe needs (HostPID, capabilities, procdir/cgroups/ debugfs/socket volumes, and the DD_DYNAMIC_INSTRUMENTATION_ENABLED / DD_SYSTEM_PROBE_ENABLED env vars). Enabling the flag pulls in the system-probe container automatically rather than requiring a separate toggle. This intentionally omits the node-scoped remote-config path and per-node DatadogAgentProfile management from the earlier draft of this work: Dynamic Instrumentation will only ever be enabled cluster-wide, and services move between nodes too often for node-level targeting to make sense. Delivery of the flag itself (remote, controlled rollout with rollback) is handled by Fleet Automation's existing operator experiment mechanism, which patches this same spec field directly, so no bespoke remote-config wiring is needed here. * Mount a writable state dir for the DI system-probe module system-probe runs with a read-only root filesystem, and the dyninst feature handler didn't mount anywhere for the Dynamic Instrumentation module's own writable state (the debugger-probes tombstone file and the SymDB upload cache), so the module had nowhere to write. The path matters here: it needs to match run_path/system-probe/ dynamic-instrumentation, not /tmp/datadog-agent/system-probe/ dynamic-instrumentation, since datadog-agent#53286 just moved this state out of /tmp for a security fix (world-writable /tmp was a symlink-attack vector). Mounting at the old /tmp path would satisfy nothing, since the module doesn't write there anymore. Adds a dedicated emptyDir volume for this, mounted only on the system-probe container, at a path derived from the same RunPathVolume Mount constant logcollection already uses for its own run-path mount, so it stays in sync if that base path ever changes. Verified with go build, go vet, and go test across the datadogagent and datadogagentinternal controller packages, including updating dyninst's own feature test to expect the new volume/mount. Confirmed none of the renderer golden-file fixtures needed updating, since none of them enable dynamicInstrumentation (they only render its default off value). (cherry picked from commit eb07514)
levan-m
added
enhancement
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 256d8c7530
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| // USM (Universal Service Monitoring) configuration. | ||
| USM *USMFeatureConfig `json:"usm,omitempty"` | ||
| // DynamicInstrumentation configuration. | ||
| DynamicInstrumentation *DynamicInstrumentationFeatureConfig `json:"dynamicInstrumentation,omitempty"` |
There was a problem hiding this comment.
Adding this served API field only updates config/crd, but the shipped OLM bundle schemas were not regenerated: rg dynamicInstrumentation bundle/manifests returns no matches for datadoghq.com_datadogagents.yaml, datadogagentinternals.yaml, or datadogagentprofiles.yaml. Clusters installed from the bundle/OperatorHub will therefore not have spec.features.dynamicInstrumentation in their CRD schema, so the API server can drop or reject the field and users cannot enable the new feature through that installation path.
Useful? React with 👍 / 👎.
This comment has been minimized.
This comment has been minimized.
tbavelier
approved these changes
Jul 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds spec.features.dynamicInstrumentation.enabled to the DatadogAgent CRD and a dyninst feature handler that turns it into the pod-spec changes system-probe needs (HostPID, capabilities, procdir/cgroups/ debugfs/socket volumes, and the DD_DYNAMIC_INSTRUMENTATION_ENABLED / DD_SYSTEM_PROBE_ENABLED env vars). Enabling the flag pulls in the system-probe container automatically rather than requiring a separate toggle.
This intentionally omits the node-scoped remote-config path and per-node DatadogAgentProfile management from the earlier draft of this work: Dynamic Instrumentation will only ever be enabled cluster-wide, and services move between nodes too often for node-level targeting to make sense. Delivery of the flag itself (remote, controlled rollout with rollback) is handled by Fleet Automation's existing operator experiment mechanism, which patches this same spec field directly, so no bespoke remote-config wiring is needed here.
system-probe runs with a read-only root filesystem, and the dyninst feature handler didn't mount anywhere for the Dynamic Instrumentation module's own writable state (the debugger-probes tombstone file and the SymDB upload cache), so the module had nowhere to write.
The path matters here: it needs to match run_path/system-probe/ dynamic-instrumentation, not /tmp/datadog-agent/system-probe/ dynamic-instrumentation, since datadog-agent#53286 just moved this state out of /tmp for a security fix (world-writable /tmp was a symlink-attack vector). Mounting at the old /tmp path would satisfy nothing, since the module doesn't write there anymore.
Adds a dedicated emptyDir volume for this, mounted only on the system-probe container, at a path derived from the same RunPathVolume Mount constant logcollection already uses for its own run-path mount, so it stays in sync if that base path ever changes.
Verified with go build, go vet, and go test across the datadogagent and datadogagentinternal controller packages, including updating dyninst's own feature test to expect the new volume/mount. Confirmed none of the renderer golden-file fixtures needed updating, since none of them enable dynamicInstrumentation (they only render its default off value).
(cherry picked from commit eb07514)
What does this PR do?
A brief description of the change being made with this pull request.
Motivation
What inspired you to submit this pull request?
Additional Notes
Anything else we should know when reviewing?
Minimum Agent Versions
Are there minimum versions of the Datadog Agent and/or Cluster Agent required?
Describe your test plan
Write there any instructions and details you may have to test your PR.
Checklist
bug,enhancement,refactoring,documentation,tooling, and/ordependenciesqa/skip-qalabel