Skip to content

feat(appsec): add ingress-nginx injection pattern for AppSec#49318

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 13 commits into
mainfrom
eliott.bouhana/appsec-nginx-injection
Apr 16, 2026
Merged

feat(appsec): add ingress-nginx injection pattern for AppSec#49318
gh-worker-dd-mergequeue-cf854d[bot] merged 13 commits into
mainfrom
eliott.bouhana/appsec-nginx-injection

Conversation

@eliottness

Copy link
Copy Markdown
Contributor

What does this PR do?

Adds a new AppSec injection pattern for ingress-nginx controllers. The pattern:

  • Injects an init container (datadog/ingress-nginx-injection:<version>) that copies the nginx-datadog module .so into a shared emptyDir volume
  • Redirects the controller's --configmap arg to a DD-owned ConfigMap with AppSec directives (load_module, datadog_appsec_enabled on, WAF thread pool) prepended to main-snippet and http-snippet
  • Watches IngressClass resources and cleans up DD-owned ConfigMaps on deletion
  • Registers ProxyTypeIngressNginx in the proxy constructor/detection maps

New files:

  • pkg/clusteragent/appsec/nginx/ — full package with sidecar injection, ConfigMap management, detection, and events
  • Config bindings for admission_controller.appsec.nginx.{init_image,module_mount_path}
  • Updated RBAC testdata to match Helm chart permissions (scoped update/delete via resourceNames)

Motivation

ingress-nginx is one of the most widely deployed ingress controllers. This extends AppSec proxy injection coverage beyond Envoy Gateway and Istio to support ingress-nginx clusters without requiring users to manually configure extraModules.

Describe how you validated your changes

  • All 37 unit tests pass (dda inv test --targets=./pkg/clusteragent/appsec/nginx/...)
  • Tests cover: pod mutation, ConfigMap creation/update/preservation, version parsing (including edge cases like registry:port without tag, digest-only, latest), label matching, IngressClass deletion cleanup, and CEL match conditions

Additional Notes

  • A release note (reno) is expected for this feature change — will add as a follow-up or in this PR if requested.
  • RBAC testdata was updated from the Helm chart baseline. The ingress-nginx ConfigMap rule is split: unscoped get/list/create (needed for reading original ConfigMaps and label-selector cleanup) + scoped update/delete with resourceNames.
  • The injection pattern always uses sidecar (init container) mode — there is no external processing mode for nginx.

Add a new injection pattern that enables Datadog AppSec on ingress-nginx
controllers by injecting an init container with the nginx-datadog module
and redirecting the controller's --configmap arg to a DD-owned ConfigMap
with AppSec directives prepended.

The pattern watches IngressClass resources and mutates ingress-nginx
controller pods via the admission webhook. ConfigMap cleanup is handled
on IngressClass deletion.

Includes config bindings, RBAC testdata, proxy registration, and unit tests.
@eliottness eliottness added qa/done QA done before merge and regressions are covered by tests team/asm-go labels Apr 14, 2026
@dd-octo-sts dd-octo-sts Bot added the internal Identify a non-fork PR label Apr 14, 2026
@github-actions github-actions Bot added the long review PR is complex, plan time to review it label Apr 14, 2026
@dd-octo-sts

dd-octo-sts Bot commented Apr 14, 2026

Copy link
Copy Markdown
Contributor

Go Package Import Differences

Baseline: 3ba93e3
Comparison: 142c117

binaryosarchchange
cluster-agentlinuxamd64
+1, -0
+github.com/DataDog/datadog-agent/pkg/clusteragent/appsec/nginx
cluster-agentlinuxarm64
+1, -0
+github.com/DataDog/datadog-agent/pkg/clusteragent/appsec/nginx

@dd-octo-sts

dd-octo-sts Bot commented Apr 14, 2026

Copy link
Copy Markdown
Contributor

Files inventory check summary

File checks results against ancestor 3ba93e39:

Results for datadog-agent_7.79.0~devel.git.808.142c117.pipeline.108015948-1_amd64.deb:

No change detected

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Apr 14, 2026

Copy link
Copy Markdown

Regression Detector

Regression Detector Results

Metrics dashboard
Target profiles
Run ID: ddeabe8e-10a0-45e8-a3ec-a04938b2d72b

Baseline: 1479bfa
Comparison: 28c7518
Diff

Optimization Goals: ✅ No significant changes detected

Experiments ignored for regressions

Regressions in experiments with settings containing erratic: true are ignored.

perf experiment goal Δ mean % Δ mean % CI trials links
docker_containers_cpu % cpu utilization -0.86 [-3.87, +2.15] 1 Logs

Fine details of change detection per experiment

perf experiment goal Δ mean % Δ mean % CI trials links
quality_gate_metrics_logs memory utilization +2.69 [+2.46, +2.92] 1 Logs bounds checks dashboard
quality_gate_logs % cpu utilization +1.17 [-0.47, +2.80] 1 Logs bounds checks dashboard
ddot_logs memory utilization +1.15 [+1.08, +1.22] 1 Logs
file_tree memory utilization +0.49 [+0.43, +0.55] 1 Logs
otlp_ingest_logs memory utilization +0.40 [+0.28, +0.52] 1 Logs
ddot_metrics_sum_cumulative memory utilization +0.26 [+0.11, +0.41] 1 Logs
quality_gate_idle_all_features memory utilization +0.21 [+0.18, +0.25] 1 Logs bounds checks dashboard
quality_gate_idle memory utilization +0.16 [+0.11, +0.21] 1 Logs bounds checks dashboard
uds_dogstatsd_to_api_v3 ingress throughput +0.06 [-0.15, +0.27] 1 Logs
file_to_blackhole_100ms_latency egress throughput +0.04 [-0.08, +0.15] 1 Logs
uds_dogstatsd_to_api ingress throughput +0.01 [-0.20, +0.21] 1 Logs
tcp_dd_logs_filter_exclude ingress throughput +0.00 [-0.10, +0.11] 1 Logs
file_to_blackhole_0ms_latency egress throughput -0.01 [-0.56, +0.54] 1 Logs
file_to_blackhole_1000ms_latency egress throughput -0.02 [-0.46, +0.41] 1 Logs
ddot_metrics_sum_delta memory utilization -0.04 [-0.22, +0.14] 1 Logs
file_to_blackhole_500ms_latency egress throughput -0.04 [-0.44, +0.36] 1 Logs
uds_dogstatsd_20mb_12k_contexts_20_senders memory utilization -0.09 [-0.16, -0.03] 1 Logs
otlp_ingest_metrics memory utilization -0.13 [-0.28, +0.03] 1 Logs
docker_containers_memory memory utilization -0.14 [-0.22, -0.06] 1 Logs
ddot_metrics memory utilization -0.45 [-0.62, -0.27] 1 Logs
ddot_metrics_sum_cumulativetodelta_exporter memory utilization -0.67 [-0.89, -0.45] 1 Logs
docker_containers_cpu % cpu utilization -0.86 [-3.87, +2.15] 1 Logs
tcp_syslog_to_blackhole ingress throughput -1.95 [-2.16, -1.75] 1 Logs

Bounds Checks: ✅ Passed

perf experiment bounds_check_name replicates_passed observed_value links
docker_containers_cpu simple_check_run 10/10 679 ≥ 26
docker_containers_memory memory_usage 10/10 275.35MiB ≤ 370MiB
docker_containers_memory simple_check_run 10/10 702 ≥ 26
file_to_blackhole_0ms_latency memory_usage 10/10 0.19GiB ≤ 1.20GiB
file_to_blackhole_0ms_latency missed_bytes 10/10 0B = 0B
file_to_blackhole_1000ms_latency memory_usage 10/10 0.23GiB ≤ 1.20GiB
file_to_blackhole_1000ms_latency missed_bytes 10/10 0B = 0B
file_to_blackhole_100ms_latency memory_usage 10/10 0.20GiB ≤ 1.20GiB
file_to_blackhole_100ms_latency missed_bytes 10/10 0B = 0B
file_to_blackhole_500ms_latency memory_usage 10/10 0.22GiB ≤ 1.20GiB
file_to_blackhole_500ms_latency missed_bytes 10/10 0B = 0B
quality_gate_idle intake_connections 10/10 4 = 4 bounds checks dashboard
quality_gate_idle memory_usage 10/10 174.77MiB ≤ 181MiB bounds checks dashboard
quality_gate_idle_all_features intake_connections 10/10 4 = 4 bounds checks dashboard
quality_gate_idle_all_features memory_usage 10/10 496.54MiB ≤ 550MiB bounds checks dashboard
quality_gate_logs intake_connections 10/10 4 ≤ 6 bounds checks dashboard
quality_gate_logs memory_usage 10/10 211.53MiB ≤ 220MiB bounds checks dashboard
quality_gate_logs missed_bytes 10/10 0B = 0B bounds checks dashboard
quality_gate_metrics_logs cpu_usage 10/10 365.24 ≤ 2000 bounds checks dashboard
quality_gate_metrics_logs intake_connections 10/10 4 ≤ 6 bounds checks dashboard
quality_gate_metrics_logs memory_usage 10/10 417.31MiB ≤ 475MiB bounds checks dashboard
quality_gate_metrics_logs missed_bytes 10/10 0B = 0B bounds checks dashboard

Explanation

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

Performance changes are noted in the perf column of each table:

  • ✅ = significantly better comparison variant performance
  • ❌ = significantly worse comparison variant performance
  • ➖ = no significant change in performance

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

  1. Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.

  2. Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.

  3. Its configuration does not mark it "erratic".

CI Pass/Fail Decision

Passed. All Quality Gates passed.

  • quality_gate_idle_all_features, bounds check intake_connections: 10/10 replicas passed. Gate passed.
  • quality_gate_idle_all_features, bounds check memory_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_metrics_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_metrics_logs, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_metrics_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
  • quality_gate_metrics_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
  • quality_gate_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
  • quality_gate_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
  • quality_gate_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_idle, bounds check intake_connections: 10/10 replicas passed. Gate passed.
  • quality_gate_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.

Signed-off-by: Eliott Bouhana <[email protected]>
Use errors.New instead of fmt.Errorf for static strings (perfsprint)
and rename unused parameter to _ (revive).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This RBAC file was from an older version of the agent which I updated so it makes sense there are unrelated changes

Replace fmt.Errorf with errors.New for DD_AGENT_HOST missing error
(perfsprint lint rule).
@eliottness
eliottness marked this pull request as ready for review April 15, 2026 10:55
@eliottness
eliottness requested review from a team as code owners April 15, 2026 10:55
@eliottness
eliottness requested a review from s-alad April 15, 2026 10:55

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4f2cbfce16

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread pkg/clusteragent/appsec/nginx/sidecar.go Outdated
Comment thread pkg/clusteragent/appsec/nginx/nginx.go

@s-alad s-alad left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm for agent config file

The nginx-datadog module reads DD_AGENT_HOST via std::getenv() at init
time (preserved by the `env DD_AGENT_HOST;` main-snippet directive).
The explicit `datadog_agent_url` http-snippet directive is redundant
and the DD_AGENT_HOST gate in MutatePod was incorrect — the admission
controller's config webhook injects DD_AGENT_HOST after the appsec
webhook runs, so the env var is never present at mutation time.

Validated on minikube: identical behavior with and without
datadog_agent_url. RC startup timeouts are an upstream nginx-datadog
issue (DataDog/nginx-datadog#210), not caused by our code.
Before deleting DD-owned ConfigMaps on IngressClass deletion, check
if other ingress-nginx IngressClasses still exist. This prevents
removing ConfigMaps still needed by other controllers in clusters
with multiple ingress-nginx IngressClasses.

Follows the same pattern as istio's cross-pattern coordination
(anyIstioGatewayClassExists in istio.go).
…onfigMap

Set an ownerReference on the DD-owned ConfigMap pointing to the
original ingress-nginx ConfigMap. This lets Kubernetes garbage-collect
the DD ConfigMap automatically if the original is deleted, and makes
the relationship between the two ConfigMaps explicit.

When the original ConfigMap doesn't exist (user hasn't customized
anything), no owner reference is set.

@gabedos gabedos left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nginx appsec config lgtm

@eliottness

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 596dbd4fa3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread pkg/clusteragent/appsec/nginx/nginx.go Outdated
Comment thread pkg/clusteragent/appsec/nginx/sidecar.go Outdated
Comment thread pkg/clusteragent/appsec/nginx/nginx.go Outdated
Comment thread pkg/clusteragent/appsec/nginx/sidecar.go Outdated
Comment thread pkg/clusteragent/appsec/nginx/sidecar.go Outdated
Comment thread pkg/clusteragent/appsec/nginx/sidecar.go Outdated
…ess PR feedback

- Fix unsafe type assertion on unstructured metadata with SetResourceVersion
- Add retry loop for 409 Conflict during concurrent ConfigMap updates
- Check NestedStringMap error instead of silently discarding it
- Wrap bare errors with context in Detect and anyOtherIngressNginxClassExists
- Use strings.CutPrefix/Cut, maps.Copy, and errors.Join idioms
- Replace context.TODO with context.Background (documented interface limitation)
- Use CommonLabels from config instead of hardcoded label map
- Add proxy-type label to cleanup selector to scope deletions to nginx
- Add --configmap arg if missing instead of hard-failing pod mutation
- Remove redundant initImage/moduleMountPath defaults (set in common_settings)
- Use fmt.Sprintf for configmap arg construction
- Rename dc->client, eventRecorderInstance->recorder, ingressClassRef
- Improve test assertions: t.Context, t.Helper, require.NoError, ErrorContains

Confidence: high
Scope-risk: moderate
@eliottness
eliottness force-pushed the eliott.bouhana/appsec-nginx-injection branch from 2f25be3 to 86d9e12 Compare April 16, 2026 09:03
Comment thread pkg/clusteragent/appsec/nginx/sidecar.go
Comment on lines +367 to +373
resources:
- configmaps
resourceNames:
- datadog-appsec-ingress-nginx-controller
verbs:
- update
- delete

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code supports arbitrary --configmap=<ns>/<name> values and derives the DD ConfigMap name dynamically, but this RBAC only grants update/delete on datadog-appsec-ingress-nginx-controller.
That means non-default Helm release names or custom configmap names will create datadog-appsec-<custom-name> objects that the Cluster Agent can create once, but won’t be able to update or clean up later.
The RBAC and naming model need to match here.

Comment thread pkg/clusteragent/appsec/nginx/sidecar.go
…d pod detection

Address PR review feedback from e-n-0 and eliottness:

1. ConfigMap sync regression: Add a ConfigMap reconciler that watches
   original ConfigMaps (via server-side label filtering) and re-merges
   DD directives into the DD copy when the original is updated. This
   fixes a behavioral regression where user edits to the original
   ConfigMap were lost since ingress-nginx dynamically reloads from
   ConfigMap changes.

2. RBAC mismatch: Remove resourceNames scoping from the ConfigMap
   update/delete RBAC rule. ConfigMap names are derived dynamically
   from the controller's --configmap arg and cannot be predicted at
   RBAC definition time. Add watch verb for the reconciler.

3. Label reliability: Add --controller-class=k8s.io/ingress-nginx
   args-based fallback in ShouldMutatePod and MatchCondition CEL for
   non-Helm deployments that may lack standard app.kubernetes.io labels.

Also hardens the admission path: bounded context timeout (10s) in
MutatePod, empty-containers guard, and conflict retry in
unlabelOriginalConfigMap.

Constraint: MutatePod interface does not accept a context parameter
Constraint: Dynamic informers cannot filter by annotation server-side
Rejected: Polling-based sync | 30s staleness unacceptable for live-reload
Rejected: Image-based pod detection | false positive risk with custom images
Confidence: high
Scope-risk: moderate
Not-tested: Cluster with >1000 ConfigMaps (informer memory pressure)
Address review findings: validate moduleMountPath against nginx directive
injection, add SecurityContext to init container (PSA restricted-compliant),
propagate reconciler Start() failure, handle AddFunc events after restart,
emit K8s events on permanent reconciliation failure, aggregate ConfigMap
deletion errors, and add constructor nil guards.

Add 10 new tests covering Deleted() cleanup, ConfigMap update retry loops,
and reconciler reconcile paths. Fix doc.go inaccuracies around marker-based
injection and sidecar terminology.
@eliottness

Copy link
Copy Markdown
Contributor Author

Review Fixes Applied

Pushed 142c1176227 addressing findings from a comprehensive code review (5 specialized agents: code reviewer, silent failure hunter, test analyzer, type design analyzer, comment analyzer).

Critical Fixes (7)

# Issue Fix
1 Nginx directive injection via unsanitized moduleMountPath validateNginxConfig() rejects semicolons, newlines, braces, non-absolute paths
2 RBAC too broad without documentation Added SECURITY CONSIDERATION comment explaining blast radius and mitigations
3 Init container runs as root Added SecurityContext with RunAsNonRoot, AllowPrivilegeEscalation:false, ReadOnlyRootFilesystem, Drop:ALL
4 Reconciler misses AddFunc after restart Registered both AddFunc and UpdateFunc on informer
5 Start() failure silently swallowed Changed to Errorf + return error (stops the pattern)
6 Permanent reconciliation failure invisible Added K8s event ReconciliationFailed on the original ConfigMap
7 No nginx config validation validateNginxConfig() called from FromComponent()

Important Fixes (6)

# Issue Fix
I1 EmptyDir no SizeLimit Added SizeLimit: 50MB (done with critical fixes)
I2 CM deletion errors silently continue Collect errors, return aggregated via errors.Join
I4 No constructor validation panic() guard in New() for nil dependencies
I5-7 Missing test coverage +10 new tests: Deleted() cleanup, CM retry loops, reconciler
I8 doc.go inaccuracies Fixed marker-based injection description, added ConfigMapReconciler, clarified init container terminology

Test Results

  • 168/168 tests pass across the full appsec package tree
  • 10 new tests added (48 total in nginx package, up from 38)
  • All pre-commit hooks pass

@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot merged commit 28c7518 into main Apr 16, 2026
267 of 268 checks passed
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot deleted the eliott.bouhana/appsec-nginx-injection branch April 16, 2026 23:42
@github-actions github-actions Bot added this to the 7.79.0 milestone Apr 16, 2026
gh-worker-dd-mergequeue-cf854d Bot pushed a commit that referenced this pull request Jun 2, 2026
…ation (#51635)

### What does this PR do?

Fixes a confused-deputy vulnerability in the Cluster Agent's AppSec ingress-nginx admission mutator. The webhook previously extracted the namespace from the pod's `--configmap=<ns>/<name>` argument and used it verbatim for ConfigMap `Get`/`Create`/`Update` calls. Combined with the DCA's cluster-wide `configmaps` permissions, a low-privileged tenant with `create pods` rights in one namespace could trigger writes to ConfigMaps in arbitrary namespaces.

Changes:

- **`pkg/clusteragent/appsec/nginx/sidecar.go`** — `findControllerConfigMapArg` now requires the `<ns>` portion to match the pod's own namespace (resolving `$(POD_NAMESPACE)` first) and rejects empty names. On rejection, `MutatePod` returns `(false, nil)` to preserve fail-open admission semantics — the pod is admitted unmodified.
- **`pkg/clusteragent/appsec/nginx/events.go`** — New `CrossNamespaceConfigMapRefused` warning event on the **pod** (not the IngressClass) so the diagnostic lands in the tenant's namespace where their operator can see it.
- **`pkg/clusteragent/appsec/nginx/configmap.go`** — Defense-in-depth `validateConfigMapTarget` (DNS-1123 validation) at the entry of `createOrUpdateDDConfigMap`, covering both webhook and reconciler paths.
- **Tests** — `TestMutatePod_CrossNamespaceConfigMapRefused` (the bisect anchor) asserts no API calls escape and the pod spec is unmodified. `TestFindControllerConfigMapArg` extended from 3 cases to 10 covering same-ns, foreign ns, `kube-system` reference, leading/trailing slash, multi-container priority.
- **Release note** — `releasenotes/notes/fix-appsec-nginx-configmap-confused-deputy-*.yaml` (security section).

The introducing change was PR #49318 (Agent 7.78.0). All releases ≥7.78.0 are affected; backports to `7.78.x`, `7.79.x`, `7.80.x` will follow.

### Motivation

Tracking: [APPSEC-68212](https://datadoghq.atlassian.net/browse/APPSEC-68212). Internal vulnerability report `clusteragent-appsec-nginx-configmap-confused-deputy` (severity High, threat model k8s-tenant). Full mitigation plan: `.sisyphus/plans/clusteragent-appsec-nginx-configmap-confused-deputy-mitigation.md`.

Pre-condition for exploitation: DCA with `cluster_agent.appsec.injector.enabled = true` (helm: `datadog.appsec.injector.enabled: true`), at least one ingress-nginx `IngressClass` (`controller: k8s.io/ingress-nginx`), and a tenant with `create pods` permission in any namespace.

### Describe how you validated your changes

**1. Automated tests (run in CI):**

```bash
dda inv test --targets=./pkg/clusteragent/appsec/nginx        # 57/57 passed
dda inv test --targets=./pkg/clusteragent/admission/mutate/appsec   # 35/35 passed
dda inv linter.go --targets=./pkg/clusteragent/appsec/nginx   # 0 issues
dda inv linter.releasenote                                     # passed
```

`TestMutatePod_CrossNamespaceConfigMapRefused` is the **bisect anchor**: it fails against the unpatched code (which accepted `--configmap=kube-system/coredns` verbatim) and passes against this patch.

**2. Live exploit reproduction (k3s, rancher-desktop, 7.78.0 + this patch):**

Setup the DCA with my patched binary:

```bash
# Overlay the patched binary on the 7.78.0 base image
cat > Dockerfile.overlay <<'DOCKERFILE'
FROM datadog/cluster-agent:7.78.0
COPY bin/datadog-cluster-agent/datadog-cluster-agent /opt/datadog-agent/bin/datadog-cluster-agent
DOCKERFILE
docker build --platform linux/arm64 -t datadog/cluster-agent:7.78.0-fix -f Dockerfile.overlay .

# Install DCA with AppSec ingress-nginx enabled
cat > values.yaml <<'YAML'
datadog:
  apiKey: "0000000000000000000000000000000000000000"
  appKey: "0000000000000000000000000000000000000000"
  clusterName: confused-deputy-test
  appsec:
    injector:
      enabled: true
      autoDetect: false
      proxies: [ingress-nginx]
clusterAgent:
  image:
    repository: datadog/cluster-agent
    tag: 7.78.0-fix
    pullPolicy: IfNotPresent
  admissionController:
    enabled: true
agents: { enabled: false }
clusterChecksRunner: { enabled: false }
YAML
helm install dd datadog/datadog -n datadog --create-namespace -f values.yaml

# Pre-condition: at least one ingress-nginx IngressClass
kubectl apply -f - <<'YAML'
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata: { name: nginx-test }
spec: { controller: k8s.io/ingress-nginx }
YAML
```

Apply the exploit pod from a low-privileged tenant namespace:

```yaml
# attacker-pod.yaml
apiVersion: v1
kind: Namespace
metadata: { name: attacker-ns }
---
apiVersion: v1
kind: Pod
metadata:
  name: confused-deputy-poc
  namespace: attacker-ns
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/component: controller
spec:
  containers:
  - name: c
    image: registry.k8s.io/ingress-nginx/controller:v1.15.1
    args:
    - /nginx-ingress-controller
    - --configmap=kube-system/coredns
    - --election-id=test
```

**Expected (and observed) outcomes:**

| Assertion | Command | Result |
|---|---|---|
| Pod admitted (fail-open) | `kubectl get pod -n attacker-ns confused-deputy-poc` | ✅ admitted, no admission error |
| Args UNMODIFIED | `kubectl get pod -n attacker-ns confused-deputy-poc -o jsonpath='{.spec.containers[0].args}'` | ✅ `--configmap=kube-system/coredns` preserved |
| No init container injected | `kubectl get pod -n attacker-ns confused-deputy-poc -o jsonpath='{.spec.initContainers}'` | ✅ empty |
| No DD ConfigMap in `kube-system` | `kubectl get cm -n kube-system \| grep -i datadog-appsec` | ✅ none |
| Warning event on pod | `kubectl get events -n attacker-ns --field-selector involvedObject.name=confused-deputy-poc` | ✅ `Warning CrossNamespaceConfigMapRefused AppSec nginx mutation skipped: --configmap references a namespace different from the pod's namespace; refusing to mutate to avoid confused-deputy ConfigMap writes: pod attacker-ns/confused-deputy-poc, arg "--configmap=kube-system/coredns"` |
| DCA log line | `kubectl logs -n datadog deploy/dd-datadog-cluster-agent \| grep "AppSec mutation skipped"` | ✅ `WARN \| nginx AppSec mutation skipped for pod attacker-ns/confused-deputy-poc: --configmap references a namespace different from the pod's namespace` |

**3. Regression check — legitimate ingress-nginx deployments still work**

Pods using the upstream Helm default (`--configmap=$(POD_NAMESPACE)/ingress-nginx-controller`) and pods using a literal same-namespace ref (`--configmap=ingress-nginx/my-config` when the pod is in `ingress-nginx`) are accepted and mutated normally. Covered by `TestFindControllerConfigMapArg/standard_$(POD_NAMESPACE)_form_is_accepted` and `.../hardcoded_same_namespace_is_accepted`.

### Additional Notes

- **`qa/rc-required` is required** — admission webhook changes touch cross-component behavior (DCA ↔ kube-apiserver ↔ node agents) per `AGENTS.md` guidance.
- The fix is **fail-open**: rejection results in the pod being admitted unmodified with a warning event and log line — never a failed admission. Legitimate ingress-nginx deployments using `$(POD_NAMESPACE)/...` (the upstream Helm default) are unaffected.
- `createOrUpdateDDConfigMap` gains DNS-1123 validation as defense-in-depth. It is a no-op for the reconciler path (whose namespace/name come from a label-filtered informer watch and are already valid Kubernetes objects) and catches any future code path that bypasses `findControllerConfigMapArg`.
- Follow-ups deferred to separate Jira tickets per the plan:
  - E2E test in `test/new-e2e/tests/clusteragent/appsec/` (§4.5)
  - Owner-reference pre-check on ingress-nginx pods (§5.2)
  - `ValidatingAdmissionPolicy` for ConfigMap creation scope (§6 Option C)

Co-authored-by: eliott.bouhana <[email protected]>
FlorentClarret pushed a commit that referenced this pull request Jun 15, 2026
…ap refs in pod mutation (#51709)

Backport 9ae4dea from #51635.

 ___

### What does this PR do?

Fixes a confused-deputy vulnerability in the Cluster Agent's AppSec
ingress-nginx admission mutator. The webhook previously extracted the
namespace from the pod's `--configmap=<ns>/<name>` argument and used it
verbatim for ConfigMap `Get`/`Create`/`Update` calls. Combined with the
DCA's cluster-wide `configmaps` permissions, a low-privileged tenant
with `create pods` rights in one namespace could trigger writes to
ConfigMaps in arbitrary namespaces.

Changes:

- **`pkg/clusteragent/appsec/nginx/sidecar.go`** —
`findControllerConfigMapArg` now requires the `<ns>` portion to match
the pod's own namespace (resolving `$(POD_NAMESPACE)` first) and rejects
empty names. On rejection, `MutatePod` returns `(false, nil)` to
preserve fail-open admission semantics — the pod is admitted unmodified.
- **`pkg/clusteragent/appsec/nginx/events.go`** — New
`CrossNamespaceConfigMapRefused` warning event on the **pod** (not the
IngressClass) so the diagnostic lands in the tenant's namespace where
their operator can see it.
- **`pkg/clusteragent/appsec/nginx/configmap.go`** — Defense-in-depth
`validateConfigMapTarget` (DNS-1123 validation) at the entry of
`createOrUpdateDDConfigMap`, covering both webhook and reconciler paths.
- **Tests** — `TestMutatePod_CrossNamespaceConfigMapRefused` (the bisect
anchor) asserts no API calls escape and the pod spec is unmodified.
`TestFindControllerConfigMapArg` extended from 3 cases to 10 covering
same-ns, foreign ns, `kube-system` reference, leading/trailing slash,
multi-container priority.
- **Release note** —
`releasenotes/notes/fix-appsec-nginx-configmap-confused-deputy-*.yaml`
(security section).

The introducing change was PR #49318 (Agent 7.78.0). All releases
≥7.78.0 are affected; backports to `7.78.x`, `7.79.x`, `7.80.x` will
follow.

### Motivation

Tracking:
[APPSEC-68212](https://datadoghq.atlassian.net/browse/APPSEC-68212).
Internal vulnerability report
`clusteragent-appsec-nginx-configmap-confused-deputy` (severity High,
threat model k8s-tenant). Full mitigation plan:
`.sisyphus/plans/clusteragent-appsec-nginx-configmap-confused-deputy-mitigation.md`.

Pre-condition for exploitation: DCA with
`cluster_agent.appsec.injector.enabled = true` (helm:
`datadog.appsec.injector.enabled: true`), at least one ingress-nginx
`IngressClass` (`controller: k8s.io/ingress-nginx`), and a tenant with
`create pods` permission in any namespace.

### Describe how you validated your changes

**1. Automated tests (run in CI):**

```bash
dda inv test --targets=./pkg/clusteragent/appsec/nginx        # 57/57 passed
dda inv test --targets=./pkg/clusteragent/admission/mutate/appsec   # 35/35 passed
dda inv linter.go --targets=./pkg/clusteragent/appsec/nginx   # 0 issues
dda inv linter.releasenote                                     # passed
```

`TestMutatePod_CrossNamespaceConfigMapRefused` is the **bisect anchor**:
it fails against the unpatched code (which accepted
`--configmap=kube-system/coredns` verbatim) and passes against this
patch.

**2. Live exploit reproduction (k3s, rancher-desktop, 7.78.0 + this
patch):**

Setup the DCA with my patched binary:

```bash
# Overlay the patched binary on the 7.78.0 base image
cat > Dockerfile.overlay <<'DOCKERFILE'
FROM datadog/cluster-agent:7.78.0
COPY bin/datadog-cluster-agent/datadog-cluster-agent /opt/datadog-agent/bin/datadog-cluster-agent
DOCKERFILE
docker build --platform linux/arm64 -t datadog/cluster-agent:7.78.0-fix -f Dockerfile.overlay .

# Install DCA with AppSec ingress-nginx enabled
cat > values.yaml <<'YAML'
datadog:
  apiKey: "0000000000000000000000000000000000000000"
  appKey: "0000000000000000000000000000000000000000"
  clusterName: confused-deputy-test
  appsec:
    injector:
      enabled: true
      autoDetect: false
      proxies: [ingress-nginx]
clusterAgent:
  image:
    repository: datadog/cluster-agent
    tag: 7.78.0-fix
    pullPolicy: IfNotPresent
  admissionController:
    enabled: true
agents: { enabled: false }
clusterChecksRunner: { enabled: false }
YAML
helm install dd datadog/datadog -n datadog --create-namespace -f values.yaml

# Pre-condition: at least one ingress-nginx IngressClass
kubectl apply -f - <<'YAML'
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata: { name: nginx-test }
spec: { controller: k8s.io/ingress-nginx }
YAML
```

Apply the exploit pod from a low-privileged tenant namespace:

```yaml
# attacker-pod.yaml
apiVersion: v1
kind: Namespace
metadata: { name: attacker-ns }
---
apiVersion: v1
kind: Pod
metadata:
  name: confused-deputy-poc
  namespace: attacker-ns
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/component: controller
spec:
  containers:
  - name: c
    image: registry.k8s.io/ingress-nginx/controller:v1.15.1
    args:
    - /nginx-ingress-controller
    - --configmap=kube-system/coredns
    - --election-id=test
```

**Expected (and observed) outcomes:**

| Assertion | Command | Result |
|---|---|---|
| Pod admitted (fail-open) | `kubectl get pod -n attacker-ns
confused-deputy-poc` | ✅ admitted, no admission error |
| Args UNMODIFIED | `kubectl get pod -n attacker-ns confused-deputy-poc
-o jsonpath='{.spec.containers[0].args}'` | ✅
`--configmap=kube-system/coredns` preserved |
| No init container injected | `kubectl get pod -n attacker-ns
confused-deputy-poc -o jsonpath='{.spec.initContainers}'` | ✅ empty |
| No DD ConfigMap in `kube-system` | `kubectl get cm -n kube-system \|
grep -i datadog-appsec` | ✅ none |
| Warning event on pod | `kubectl get events -n attacker-ns
--field-selector involvedObject.name=confused-deputy-poc` | ✅ `Warning
CrossNamespaceConfigMapRefused AppSec nginx mutation skipped:
--configmap references a namespace different from the pod's namespace;
refusing to mutate to avoid confused-deputy ConfigMap writes: pod
attacker-ns/confused-deputy-poc, arg "--configmap=kube-system/coredns"`
|
| DCA log line | `kubectl logs -n datadog
deploy/dd-datadog-cluster-agent \| grep "AppSec mutation skipped"` | ✅
`WARN \| nginx AppSec mutation skipped for pod
attacker-ns/confused-deputy-poc: --configmap references a namespace
different from the pod's namespace` |

**3. Regression check — legitimate ingress-nginx deployments still
work**

Pods using the upstream Helm default
(`--configmap=$(POD_NAMESPACE)/ingress-nginx-controller`) and pods using
a literal same-namespace ref (`--configmap=ingress-nginx/my-config` when
the pod is in `ingress-nginx`) are accepted and mutated normally.
Covered by
`TestFindControllerConfigMapArg/standard_$(POD_NAMESPACE)_form_is_accepted`
and `.../hardcoded_same_namespace_is_accepted`.

### Additional Notes

- **`qa/rc-required` is required** — admission webhook changes touch
cross-component behavior (DCA ↔ kube-apiserver ↔ node agents) per
`AGENTS.md` guidance.
- The fix is **fail-open**: rejection results in the pod being admitted
unmodified with a warning event and log line — never a failed admission.
Legitimate ingress-nginx deployments using `$(POD_NAMESPACE)/...` (the
upstream Helm default) are unaffected.
- `createOrUpdateDDConfigMap` gains DNS-1123 validation as
defense-in-depth. It is a no-op for the reconciler path (whose
namespace/name come from a label-filtered informer watch and are already
valid Kubernetes objects) and catches any future code path that bypasses
`findControllerConfigMapArg`.
- Follow-ups deferred to separate Jira tickets per the plan:
  - E2E test in `test/new-e2e/tests/clusteragent/appsec/` (§4.5)
  - Owner-reference pre-check on ingress-nginx pods (§5.2)
- `ValidatingAdmissionPolicy` for ConfigMap creation scope (§6 Option C)

[APPSEC-68212]:
https://datadoghq.atlassian.net/browse/APPSEC-68212?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

Co-authored-by: dd-octo-sts[bot] <200755185+dd-octo-sts[bot]@users.noreply.github.com>
Co-authored-by: Eliott B <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

internal Identify a non-fork PR long review PR is complex, plan time to review it qa/done QA done before merge and regressions are covered by tests team/agent-configuration team/asm-go team/container-platform The Container Platform Team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants