Skip to content

Add Dynamic Instrumentation system probe feature#3245

Merged
grantseltzer merged 2 commits into
mainfrom
grantseltzer/dynamic-instrumentation-cluster-wide
Jul 10, 2026
Merged

Add Dynamic Instrumentation system probe feature#3245
grantseltzer merged 2 commits into
mainfrom
grantseltzer/dynamic-instrumentation-cluster-wide

Conversation

@grantseltzer

Copy link
Copy Markdown
Member

What does this PR do?

Adds spec.features.dynamicInstrumentation.enabled to the DatadogAgent CRD and a dyninst feature handler that turns it on.

Motivation

This is the operator-side plumbing for letting users enable the Dynamic Instrumentation system-probe module (used by the Go live debugger) for a whole cluster from the Datadog UI, via Fleet Automation.

Delivery of the flag itself (a remote, controlled rollout with automatic rollback) will go through Fleet Automation's existing operator experiment mechanism, which patches this spec field directly.

Additional Notes

The field is inert until Fleet Automation's schema (dd-source#10080) and a UI to send the config patch are in place, so this is safe to merge ahead of those.

Minimum Agent Versions

  • Agent: 7.73.0
  • Cluster Agent: none

Describe your test plan

  • make generate && make manifests, regenerated deepcopy, openapi, CRD manifests, and docs
  • go build ./... and go vet ./..., both clean
  • go test ./internal/controller/datadogagent/... ./internal/controller/datadogagentinternal/..., all passing
  • Not yet done: an actual cluster rollout with spec.features.dynamicInstrumentation.enabled: true to confirm the node agent picks
    up the system-probe container and the expected volumes/env vars. Worth doing before merge.

Checklist

  • PR has at least one valid label: bug, enhancement, refactoring, documentation, tooling, and/or dependencies
  • PR has a milestone or the qa/skip-qa label
  • All commits are signed (see: signing commits)

Adds spec.features.dynamicInstrumentation.enabled to the DatadogAgent
CRD and a dyninst feature handler that turns it into the pod-spec
changes system-probe needs (HostPID, capabilities, procdir/cgroups/
debugfs/socket volumes, and the DD_DYNAMIC_INSTRUMENTATION_ENABLED /
DD_SYSTEM_PROBE_ENABLED env vars). Enabling the flag pulls in the
system-probe container automatically rather than requiring a separate
toggle.

This intentionally omits the node-scoped remote-config path and
per-node DatadogAgentProfile management from the earlier draft of this
work: Dynamic Instrumentation will only ever be enabled cluster-wide,
and services move between nodes too often for node-level targeting to
make sense. Delivery of the flag itself (remote, controlled rollout
with rollback) is handled by Fleet Automation's existing operator
experiment mechanism, which patches this same spec field directly, so
no bespoke remote-config wiring is needed here.
@grantseltzer
grantseltzer force-pushed the grantseltzer/dynamic-instrumentation-cluster-wide branch from 9118cd2 to 59f53f7 Compare July 9, 2026 20:02
@datadog-datadog-prod-us1

datadog-datadog-prod-us1 Bot commented Jul 9, 2026

Copy link
Copy Markdown

Code Coverage

🎯 Code Coverage (details)
Patch Coverage: 86.89%
Overall Coverage: 46.44% (+0.07%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: d05fdeb | Docs | Datadog PR Page | Give us feedback!

@levan-m levan-m added this to the v1.29.0 milestone Jul 9, 2026
@levan-m levan-m added the enhancement New feature or request label Jul 9, 2026
@levan-m
levan-m marked this pull request as ready for review July 9, 2026 21:02
@levan-m
levan-m requested a review from a team July 9, 2026 21:02
@levan-m
levan-m requested review from a team as code owners July 9, 2026 21:02

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 59f53f7d18

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread internal/controller/datadogagent/feature/dyninst/feature.go
system-probe runs with a read-only root filesystem, and the dyninst
feature handler didn't mount anywhere for the Dynamic Instrumentation
module's own writable state (the debugger-probes tombstone file and
the SymDB upload cache), so the module had nowhere to write.

The path matters here: it needs to match run_path/system-probe/
dynamic-instrumentation, not /tmp/datadog-agent/system-probe/
dynamic-instrumentation, since datadog-agent#53286 just moved this
state out of /tmp for a security fix (world-writable /tmp was a
symlink-attack vector). Mounting at the old /tmp path would satisfy
nothing, since the module doesn't write there anymore.

Adds a dedicated emptyDir volume for this, mounted only on the
system-probe container, at a path derived from the same RunPathVolume
Mount constant logcollection already uses for its own run-path mount,
so it stays in sync if that base path ever changes.

Verified with go build, go vet, and go test across the datadogagent
and datadogagentinternal controller packages, including updating
dyninst's own feature test to expect the new volume/mount. Confirmed
none of the renderer golden-file fixtures needed updating, since none
of them enable dynamicInstrumentation (they only render its default
off value).
@tbavelier tbavelier modified the milestones: v1.29.0, v1.30.0 Jul 10, 2026
@grantseltzer
grantseltzer merged commit eb07514 into main Jul 10, 2026
46 checks passed
@grantseltzer
grantseltzer deleted the grantseltzer/dynamic-instrumentation-cluster-wide branch July 10, 2026 13:09
@dd-octo-sts

dd-octo-sts Bot commented Jul 10, 2026

Copy link
Copy Markdown

The backport to v1.29 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-v1.29 v1.29
# Navigate to the new working tree
cd .worktrees/backport-v1.29
# Create a new branch
git switch --create backport-3245-to-v1.29
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 eb0751427e141cb95638457563bd88e968646a4b
# Push it to GitHub
git push --set-upstream origin backport-3245-to-v1.29
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-v1.29

Then, create a pull request where the base branch is v1.29 and the compare/head branch is backport-3245-to-v1.29.

levan-m added a commit that referenced this pull request Jul 13, 2026
* Add Dynamic Instrumentation system probe feature

Adds spec.features.dynamicInstrumentation.enabled to the DatadogAgent
CRD and a dyninst feature handler that turns it into the pod-spec
changes system-probe needs (HostPID, capabilities, procdir/cgroups/
debugfs/socket volumes, and the DD_DYNAMIC_INSTRUMENTATION_ENABLED /
DD_SYSTEM_PROBE_ENABLED env vars). Enabling the flag pulls in the
system-probe container automatically rather than requiring a separate
toggle.

This intentionally omits the node-scoped remote-config path and
per-node DatadogAgentProfile management from the earlier draft of this
work: Dynamic Instrumentation will only ever be enabled cluster-wide,
and services move between nodes too often for node-level targeting to
make sense. Delivery of the flag itself (remote, controlled rollout
with rollback) is handled by Fleet Automation's existing operator
experiment mechanism, which patches this same spec field directly, so
no bespoke remote-config wiring is needed here.

* Mount a writable state dir for the DI system-probe module

system-probe runs with a read-only root filesystem, and the dyninst
feature handler didn't mount anywhere for the Dynamic Instrumentation
module's own writable state (the debugger-probes tombstone file and
the SymDB upload cache), so the module had nowhere to write.

The path matters here: it needs to match run_path/system-probe/
dynamic-instrumentation, not /tmp/datadog-agent/system-probe/
dynamic-instrumentation, since datadog-agent#53286 just moved this
state out of /tmp for a security fix (world-writable /tmp was a
symlink-attack vector). Mounting at the old /tmp path would satisfy
nothing, since the module doesn't write there anymore.

Adds a dedicated emptyDir volume for this, mounted only on the
system-probe container, at a path derived from the same RunPathVolume
Mount constant logcollection already uses for its own run-path mount,
so it stays in sync if that base path ever changes.

Verified with go build, go vet, and go test across the datadogagent
and datadogagentinternal controller packages, including updating
dyninst's own feature test to expect the new volume/mount. Confirmed
none of the renderer golden-file fixtures needed updating, since none
of them enable dynamicInstrumentation (they only render its default
off value).

(cherry picked from commit eb07514)

Co-authored-by: grantseltzer <[email protected]>
Co-authored-by: Timothée Bavelier <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants