Skip to content

[PROF-14068] Remove privileges for host-profiler#2586

Merged
fanny-jiang merged 11 commits into
mainfrom
theomagellan/host-profiler-unprivileged
May 11, 2026
Merged

[PROF-14068] Remove privileges for host-profiler#2586
fanny-jiang merged 11 commits into
mainfrom
theomagellan/host-profiler-unprivileged

Conversation

@theomagellan

@theomagellan theomagellan commented Apr 21, 2026

Copy link
Copy Markdown
Contributor

What this PR does / why we need it

This PR removes the Host Profiler's needs for privileges: true by only allowing strictly minimal capabilities and restricting available syscalls.

Special notes for your reviewer

  • Capabilities: SYS_ADMIN, SYS_PTRACE, SYS_RESOURCE, DAC_READ_SEARCH, SYSLOG
  • Seccomp: allowlist with SCMP_ACT_ERRNO default. kill restricted to signal 0 via arg filter. setns excluded.
  • AppArmor: wired into generate-security-context (defaults to unconfined, matching system-probe)
  • Egress: added to the FQDN allow-list for symbol uploads and OTLP metrics
    • sourcemap-intake.%s: symbol intake
    • otlp.%s: OTLP metrics intake

Test plan

  • Deployed on EKS 1.35 (kernel 6.17) with seccomp + AppArmor localhost profiles enforced
  • --go-runtime-metrics OTLP self-loop works

Profiles using the chart can be found here for both supported architectures.

Note: self profiles (the profiler profiling its own process) are currently broken since non-privileged containers with hostPID get their own cgroup namespace and container.id extraction fails. The cluster used for testing is using a custom image with a hotfix for this.

Checklist

[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]

  • All commits are signed and show as "Verified" on GitHub (see: signing commits)
  • Chart Version semver bump label has been added (use <chartName>/minor-version, <chartName>/patch-version, or <chartName>/no-version-bump)
  • For datadog or datadog-operator chart or value changes, update the test baselines (run: make update-test-baselines)
  • For datadog chart changes, received ✅ from a member of your team

GitHub CI takes care of the below, but are still required:

  • Documentation has been updated with helm-docs (run: .github/helm-docs.sh)
  • CHANGELOG.md has been updated
  • Variables are documented in the README.md

@theomagellan theomagellan added the datadog/minor-version Minor version bump for datadog chart label Apr 21, 2026
@theomagellan
theomagellan force-pushed the theomagellan/host-profiler-unprivileged branch from fdaa036 to 00fe6fb Compare April 21, 2026 14:02
@github-actions github-actions Bot added the chart/datadog This issue or pull request is related to the datadog chart label Apr 21, 2026
@theomagellan
theomagellan force-pushed the theomagellan/host-profiler-unprivileged branch from 1356c97 to 6c21719 Compare April 22, 2026 08:00
@theomagellan theomagellan changed the title Remove privileges for host-profiler [PROF-14068] Remove privileges for host-profiler Apr 22, 2026
@theomagellan
theomagellan force-pushed the theomagellan/host-profiler-unprivileged branch from 44b7570 to 84eafd6 Compare April 22, 2026 09:05
@theomagellan
theomagellan requested review from a team and mackjmr April 22, 2026 09:33
@theomagellan
theomagellan marked this pull request as ready for review April 22, 2026 09:58
@theomagellan
theomagellan requested a review from a team as a code owner April 22, 2026 09:58

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 84eafd6197

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread charts/datadog/values.yaml

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not read only ?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, I didn't really challenge the existing deployment configuration.
cilium/ebpf seems to rely on writing to tracefs as fallback to old kernel limitations (cf. here).

The 5.10 kernel's alloc_trace_kprobe is not able to return -EINVAL based on symbol name, so we should be safe even if the fallback still exists in current cilium/ebpf versions.
Overall I would lean towards keeping it writeable in case something changes or odd behaviors occur somehow, but no strong opinions. I can make it read only if you feel strongly about this

@theomagellan theomagellan May 6, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, deployments in datadog-operator seem to be mounting it in read only so let's align to that as well. I'll comment previous findings in the yaml so we can keep an eye on it just in case.

@r1viollet

Copy link
Copy Markdown

We discussed a couple of points here.
It might be worth trying out CAP_BPF + CAP_PERFMON instead of SYS_ADMIN, we can follow up on this later.
Endpoints look good.
There is follow up work on the UIDs and child processes, so we might not need some of the UID syscalls (setuid/setgid/setresuid/setresgid/setgroups).

@r1viollet

Copy link
Copy Markdown

The PR needs to be aligned

@r1viollet r1viollet left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, some minor improvement proposals
This is a step in the correct direction (even if we will no longer be able to do things like gdb / ptrace etc 😄 )

@theomagellan
theomagellan force-pushed the theomagellan/host-profiler-unprivileged branch from ee93e41 to 78cd53a Compare May 5, 2026 17:32
@theomagellan
theomagellan requested a review from a team as a code owner May 5, 2026 17:32
@theomagellan
theomagellan requested review from gpalmz and removed request for a team May 5, 2026 17:32
@theomagellan

Copy link
Copy Markdown
Contributor Author

It might be worth trying out CAP_BPF + CAP_PERFMON instead of SYS_ADMIN, we can follow up on this later.

Yes since upstream now has kernel 5.10 as minimum. While testing though I found that we also needed CAP_CHECKPOINT_RESTORE to be able to read the contents in /proc/pid/map_files for other processes
Implemented in 78cd53a78cd53a and profiles can be found here
I'm re-requesting a review from you @r1viollet since I changed the capability set 🙇 sorry for the trouble!

There is follow up work on the UIDs and child processes, so we might not need some of the UID syscalls (setuid/setgid/setresuid/setresgid/setgroups).

I looked into this and it's still pretty unclear to me if they are needed at all. Earlier today the container would fail without these but now it seems to be running fine in my testing environment. The only changes made were the capabilities, but I don't think this should change much 🤔
Caught this in a jira ticket for investigation https://datadoghq.atlassian.net/browse/PROF-14540

@theomagellan
theomagellan requested a review from r1viollet May 5, 2026 17:35
readOnlyRootFilesystem: true
privileged: true
privileged: false
capabilities:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is great, thanks!

@r1viollet r1viollet left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@theomagellan
theomagellan force-pushed the theomagellan/host-profiler-unprivileged branch from 3deacc1 to 67cc2de Compare May 6, 2026 17:46
@theomagellan

Copy link
Copy Markdown
Contributor Author

hey @gpalmz 👋 this is ready for a review from your team! Thank you 🙇

@theomagellan
theomagellan force-pushed the theomagellan/host-profiler-unprivileged branch 2 times, most recently from ca1e730 to 04360ce Compare May 7, 2026 10:03
  - provide necessary capabilities
  - restrict available syscalls
  - support apparmor profiles
  - sourcemap-intake.%site for symbols
  - otlp.%site for metrics
  the resourcedetection processor uses syscalls like `sysinfo` that were
  not present in the original allow-list that was bundled-centric.
@theomagellan
theomagellan force-pushed the theomagellan/host-profiler-unprivileged branch from e8c64cd to bc70da0 Compare May 11, 2026 07:57
- bump version for datadog to 3.209.0 (minor-version)
- update changelog for datadog with version 3.209.0
- update readme for datadog
@fanny-jiang
fanny-jiang merged commit 6549a9f into main May 11, 2026
31 checks passed
@fanny-jiang
fanny-jiang deleted the theomagellan/host-profiler-unprivileged branch May 11, 2026 17:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

chart/datadog This issue or pull request is related to the datadog chart datadog/minor-version Minor version bump for datadog chart mergequeue-status: rejected

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants