feat(appsec): add ingress-nginx injection pattern for AppSec#49318
feat(appsec): add ingress-nginx injection pattern for AppSec#49318gh-worker-dd-mergequeue-cf854d[bot] merged 13 commits into
Conversation
Add a new injection pattern that enables Datadog AppSec on ingress-nginx controllers by injecting an init container with the nginx-datadog module and redirecting the controller's --configmap arg to a DD-owned ConfigMap with AppSec directives prepended. The pattern watches IngressClass resources and mutates ingress-nginx controller pods via the admission webhook. ConfigMap cleanup is handled on IngressClass deletion. Includes config bindings, RBAC testdata, proxy registration, and unit tests.
Files inventory check summaryFile checks results against ancestor 3ba93e39: Results for datadog-agent_7.79.0~devel.git.808.142c117.pipeline.108015948-1_amd64.deb:No change detected |
Regression DetectorRegression Detector ResultsMetrics dashboard Baseline: 1479bfa Optimization Goals: ✅ No significant changes detected
|
| perf | experiment | goal | Δ mean % | Δ mean % CI | trials | links |
|---|---|---|---|---|---|---|
| ➖ | docker_containers_cpu | % cpu utilization | -0.86 | [-3.87, +2.15] | 1 | Logs |
Fine details of change detection per experiment
| perf | experiment | goal | Δ mean % | Δ mean % CI | trials | links |
|---|---|---|---|---|---|---|
| ➖ | quality_gate_metrics_logs | memory utilization | +2.69 | [+2.46, +2.92] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_logs | % cpu utilization | +1.17 | [-0.47, +2.80] | 1 | Logs bounds checks dashboard |
| ➖ | ddot_logs | memory utilization | +1.15 | [+1.08, +1.22] | 1 | Logs |
| ➖ | file_tree | memory utilization | +0.49 | [+0.43, +0.55] | 1 | Logs |
| ➖ | otlp_ingest_logs | memory utilization | +0.40 | [+0.28, +0.52] | 1 | Logs |
| ➖ | ddot_metrics_sum_cumulative | memory utilization | +0.26 | [+0.11, +0.41] | 1 | Logs |
| ➖ | quality_gate_idle_all_features | memory utilization | +0.21 | [+0.18, +0.25] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_idle | memory utilization | +0.16 | [+0.11, +0.21] | 1 | Logs bounds checks dashboard |
| ➖ | uds_dogstatsd_to_api_v3 | ingress throughput | +0.06 | [-0.15, +0.27] | 1 | Logs |
| ➖ | file_to_blackhole_100ms_latency | egress throughput | +0.04 | [-0.08, +0.15] | 1 | Logs |
| ➖ | uds_dogstatsd_to_api | ingress throughput | +0.01 | [-0.20, +0.21] | 1 | Logs |
| ➖ | tcp_dd_logs_filter_exclude | ingress throughput | +0.00 | [-0.10, +0.11] | 1 | Logs |
| ➖ | file_to_blackhole_0ms_latency | egress throughput | -0.01 | [-0.56, +0.54] | 1 | Logs |
| ➖ | file_to_blackhole_1000ms_latency | egress throughput | -0.02 | [-0.46, +0.41] | 1 | Logs |
| ➖ | ddot_metrics_sum_delta | memory utilization | -0.04 | [-0.22, +0.14] | 1 | Logs |
| ➖ | file_to_blackhole_500ms_latency | egress throughput | -0.04 | [-0.44, +0.36] | 1 | Logs |
| ➖ | uds_dogstatsd_20mb_12k_contexts_20_senders | memory utilization | -0.09 | [-0.16, -0.03] | 1 | Logs |
| ➖ | otlp_ingest_metrics | memory utilization | -0.13 | [-0.28, +0.03] | 1 | Logs |
| ➖ | docker_containers_memory | memory utilization | -0.14 | [-0.22, -0.06] | 1 | Logs |
| ➖ | ddot_metrics | memory utilization | -0.45 | [-0.62, -0.27] | 1 | Logs |
| ➖ | ddot_metrics_sum_cumulativetodelta_exporter | memory utilization | -0.67 | [-0.89, -0.45] | 1 | Logs |
| ➖ | docker_containers_cpu | % cpu utilization | -0.86 | [-3.87, +2.15] | 1 | Logs |
| ➖ | tcp_syslog_to_blackhole | ingress throughput | -1.95 | [-2.16, -1.75] | 1 | Logs |
Bounds Checks: ✅ Passed
| perf | experiment | bounds_check_name | replicates_passed | observed_value | links |
|---|---|---|---|---|---|
| ✅ | docker_containers_cpu | simple_check_run | 10/10 | 679 ≥ 26 | |
| ✅ | docker_containers_memory | memory_usage | 10/10 | 275.35MiB ≤ 370MiB | |
| ✅ | docker_containers_memory | simple_check_run | 10/10 | 702 ≥ 26 | |
| ✅ | file_to_blackhole_0ms_latency | memory_usage | 10/10 | 0.19GiB ≤ 1.20GiB | |
| ✅ | file_to_blackhole_0ms_latency | missed_bytes | 10/10 | 0B = 0B | |
| ✅ | file_to_blackhole_1000ms_latency | memory_usage | 10/10 | 0.23GiB ≤ 1.20GiB | |
| ✅ | file_to_blackhole_1000ms_latency | missed_bytes | 10/10 | 0B = 0B | |
| ✅ | file_to_blackhole_100ms_latency | memory_usage | 10/10 | 0.20GiB ≤ 1.20GiB | |
| ✅ | file_to_blackhole_100ms_latency | missed_bytes | 10/10 | 0B = 0B | |
| ✅ | file_to_blackhole_500ms_latency | memory_usage | 10/10 | 0.22GiB ≤ 1.20GiB | |
| ✅ | file_to_blackhole_500ms_latency | missed_bytes | 10/10 | 0B = 0B | |
| ✅ | quality_gate_idle | intake_connections | 10/10 | 4 = 4 | bounds checks dashboard |
| ✅ | quality_gate_idle | memory_usage | 10/10 | 174.77MiB ≤ 181MiB | bounds checks dashboard |
| ✅ | quality_gate_idle_all_features | intake_connections | 10/10 | 4 = 4 | bounds checks dashboard |
| ✅ | quality_gate_idle_all_features | memory_usage | 10/10 | 496.54MiB ≤ 550MiB | bounds checks dashboard |
| ✅ | quality_gate_logs | intake_connections | 10/10 | 4 ≤ 6 | bounds checks dashboard |
| ✅ | quality_gate_logs | memory_usage | 10/10 | 211.53MiB ≤ 220MiB | bounds checks dashboard |
| ✅ | quality_gate_logs | missed_bytes | 10/10 | 0B = 0B | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | cpu_usage | 10/10 | 365.24 ≤ 2000 | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | intake_connections | 10/10 | 4 ≤ 6 | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | memory_usage | 10/10 | 417.31MiB ≤ 475MiB | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | missed_bytes | 10/10 | 0B = 0B | bounds checks dashboard |
Explanation
Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%
Performance changes are noted in the perf column of each table:
- ✅ = significantly better comparison variant performance
- ❌ = significantly worse comparison variant performance
- ➖ = no significant change in performance
A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".
For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:
-
Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
-
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
-
Its configuration does not mark it "erratic".
CI Pass/Fail Decision
✅ Passed. All Quality Gates passed.
- quality_gate_idle_all_features, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_idle_all_features, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_idle, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
Signed-off-by: Eliott Bouhana <[email protected]>
Use errors.New instead of fmt.Errorf for static strings (perfsprint) and rename unused parameter to _ (revive).
There was a problem hiding this comment.
This RBAC file was from an older version of the agent which I updated so it makes sense there are unrelated changes
Replace fmt.Errorf with errors.New for DD_AGENT_HOST missing error (perfsprint lint rule).
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4f2cbfce16
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The nginx-datadog module reads DD_AGENT_HOST via std::getenv() at init time (preserved by the `env DD_AGENT_HOST;` main-snippet directive). The explicit `datadog_agent_url` http-snippet directive is redundant and the DD_AGENT_HOST gate in MutatePod was incorrect — the admission controller's config webhook injects DD_AGENT_HOST after the appsec webhook runs, so the env var is never present at mutation time. Validated on minikube: identical behavior with and without datadog_agent_url. RC startup timeouts are an upstream nginx-datadog issue (DataDog/nginx-datadog#210), not caused by our code.
Before deleting DD-owned ConfigMaps on IngressClass deletion, check if other ingress-nginx IngressClasses still exist. This prevents removing ConfigMaps still needed by other controllers in clusters with multiple ingress-nginx IngressClasses. Follows the same pattern as istio's cross-pattern coordination (anyIstioGatewayClassExists in istio.go).
…onfigMap Set an ownerReference on the DD-owned ConfigMap pointing to the original ingress-nginx ConfigMap. This lets Kubernetes garbage-collect the DD ConfigMap automatically if the original is deleted, and makes the relationship between the two ConfigMaps explicit. When the original ConfigMap doesn't exist (user hasn't customized anything), no owner reference is set.
gabedos
left a comment
There was a problem hiding this comment.
nginx appsec config lgtm
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 596dbd4fa3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…ess PR feedback - Fix unsafe type assertion on unstructured metadata with SetResourceVersion - Add retry loop for 409 Conflict during concurrent ConfigMap updates - Check NestedStringMap error instead of silently discarding it - Wrap bare errors with context in Detect and anyOtherIngressNginxClassExists - Use strings.CutPrefix/Cut, maps.Copy, and errors.Join idioms - Replace context.TODO with context.Background (documented interface limitation) - Use CommonLabels from config instead of hardcoded label map - Add proxy-type label to cleanup selector to scope deletions to nginx - Add --configmap arg if missing instead of hard-failing pod mutation - Remove redundant initImage/moduleMountPath defaults (set in common_settings) - Use fmt.Sprintf for configmap arg construction - Rename dc->client, eventRecorderInstance->recorder, ingressClassRef - Improve test assertions: t.Context, t.Helper, require.NoError, ErrorContains Confidence: high Scope-risk: moderate
2f25be3 to
86d9e12
Compare
| resources: | ||
| - configmaps | ||
| resourceNames: | ||
| - datadog-appsec-ingress-nginx-controller | ||
| verbs: | ||
| - update | ||
| - delete |
There was a problem hiding this comment.
The code supports arbitrary --configmap=<ns>/<name> values and derives the DD ConfigMap name dynamically, but this RBAC only grants update/delete on datadog-appsec-ingress-nginx-controller.
That means non-default Helm release names or custom configmap names will create datadog-appsec-<custom-name> objects that the Cluster Agent can create once, but won’t be able to update or clean up later.
The RBAC and naming model need to match here.
…d pod detection Address PR review feedback from e-n-0 and eliottness: 1. ConfigMap sync regression: Add a ConfigMap reconciler that watches original ConfigMaps (via server-side label filtering) and re-merges DD directives into the DD copy when the original is updated. This fixes a behavioral regression where user edits to the original ConfigMap were lost since ingress-nginx dynamically reloads from ConfigMap changes. 2. RBAC mismatch: Remove resourceNames scoping from the ConfigMap update/delete RBAC rule. ConfigMap names are derived dynamically from the controller's --configmap arg and cannot be predicted at RBAC definition time. Add watch verb for the reconciler. 3. Label reliability: Add --controller-class=k8s.io/ingress-nginx args-based fallback in ShouldMutatePod and MatchCondition CEL for non-Helm deployments that may lack standard app.kubernetes.io labels. Also hardens the admission path: bounded context timeout (10s) in MutatePod, empty-containers guard, and conflict retry in unlabelOriginalConfigMap. Constraint: MutatePod interface does not accept a context parameter Constraint: Dynamic informers cannot filter by annotation server-side Rejected: Polling-based sync | 30s staleness unacceptable for live-reload Rejected: Image-based pod detection | false positive risk with custom images Confidence: high Scope-risk: moderate Not-tested: Cluster with >1000 ConfigMaps (informer memory pressure)
Address review findings: validate moduleMountPath against nginx directive injection, add SecurityContext to init container (PSA restricted-compliant), propagate reconciler Start() failure, handle AddFunc events after restart, emit K8s events on permanent reconciliation failure, aggregate ConfigMap deletion errors, and add constructor nil guards. Add 10 new tests covering Deleted() cleanup, ConfigMap update retry loops, and reconciler reconcile paths. Fix doc.go inaccuracies around marker-based injection and sidecar terminology.
Review Fixes AppliedPushed Critical Fixes (7)
Important Fixes (6)
Test Results
|
28c7518
into
main
…ation (#51635) ### What does this PR do? Fixes a confused-deputy vulnerability in the Cluster Agent's AppSec ingress-nginx admission mutator. The webhook previously extracted the namespace from the pod's `--configmap=<ns>/<name>` argument and used it verbatim for ConfigMap `Get`/`Create`/`Update` calls. Combined with the DCA's cluster-wide `configmaps` permissions, a low-privileged tenant with `create pods` rights in one namespace could trigger writes to ConfigMaps in arbitrary namespaces. Changes: - **`pkg/clusteragent/appsec/nginx/sidecar.go`** — `findControllerConfigMapArg` now requires the `<ns>` portion to match the pod's own namespace (resolving `$(POD_NAMESPACE)` first) and rejects empty names. On rejection, `MutatePod` returns `(false, nil)` to preserve fail-open admission semantics — the pod is admitted unmodified. - **`pkg/clusteragent/appsec/nginx/events.go`** — New `CrossNamespaceConfigMapRefused` warning event on the **pod** (not the IngressClass) so the diagnostic lands in the tenant's namespace where their operator can see it. - **`pkg/clusteragent/appsec/nginx/configmap.go`** — Defense-in-depth `validateConfigMapTarget` (DNS-1123 validation) at the entry of `createOrUpdateDDConfigMap`, covering both webhook and reconciler paths. - **Tests** — `TestMutatePod_CrossNamespaceConfigMapRefused` (the bisect anchor) asserts no API calls escape and the pod spec is unmodified. `TestFindControllerConfigMapArg` extended from 3 cases to 10 covering same-ns, foreign ns, `kube-system` reference, leading/trailing slash, multi-container priority. - **Release note** — `releasenotes/notes/fix-appsec-nginx-configmap-confused-deputy-*.yaml` (security section). The introducing change was PR #49318 (Agent 7.78.0). All releases ≥7.78.0 are affected; backports to `7.78.x`, `7.79.x`, `7.80.x` will follow. ### Motivation Tracking: [APPSEC-68212](https://datadoghq.atlassian.net/browse/APPSEC-68212). Internal vulnerability report `clusteragent-appsec-nginx-configmap-confused-deputy` (severity High, threat model k8s-tenant). Full mitigation plan: `.sisyphus/plans/clusteragent-appsec-nginx-configmap-confused-deputy-mitigation.md`. Pre-condition for exploitation: DCA with `cluster_agent.appsec.injector.enabled = true` (helm: `datadog.appsec.injector.enabled: true`), at least one ingress-nginx `IngressClass` (`controller: k8s.io/ingress-nginx`), and a tenant with `create pods` permission in any namespace. ### Describe how you validated your changes **1. Automated tests (run in CI):** ```bash dda inv test --targets=./pkg/clusteragent/appsec/nginx # 57/57 passed dda inv test --targets=./pkg/clusteragent/admission/mutate/appsec # 35/35 passed dda inv linter.go --targets=./pkg/clusteragent/appsec/nginx # 0 issues dda inv linter.releasenote # passed ``` `TestMutatePod_CrossNamespaceConfigMapRefused` is the **bisect anchor**: it fails against the unpatched code (which accepted `--configmap=kube-system/coredns` verbatim) and passes against this patch. **2. Live exploit reproduction (k3s, rancher-desktop, 7.78.0 + this patch):** Setup the DCA with my patched binary: ```bash # Overlay the patched binary on the 7.78.0 base image cat > Dockerfile.overlay <<'DOCKERFILE' FROM datadog/cluster-agent:7.78.0 COPY bin/datadog-cluster-agent/datadog-cluster-agent /opt/datadog-agent/bin/datadog-cluster-agent DOCKERFILE docker build --platform linux/arm64 -t datadog/cluster-agent:7.78.0-fix -f Dockerfile.overlay . # Install DCA with AppSec ingress-nginx enabled cat > values.yaml <<'YAML' datadog: apiKey: "0000000000000000000000000000000000000000" appKey: "0000000000000000000000000000000000000000" clusterName: confused-deputy-test appsec: injector: enabled: true autoDetect: false proxies: [ingress-nginx] clusterAgent: image: repository: datadog/cluster-agent tag: 7.78.0-fix pullPolicy: IfNotPresent admissionController: enabled: true agents: { enabled: false } clusterChecksRunner: { enabled: false } YAML helm install dd datadog/datadog -n datadog --create-namespace -f values.yaml # Pre-condition: at least one ingress-nginx IngressClass kubectl apply -f - <<'YAML' apiVersion: networking.k8s.io/v1 kind: IngressClass metadata: { name: nginx-test } spec: { controller: k8s.io/ingress-nginx } YAML ``` Apply the exploit pod from a low-privileged tenant namespace: ```yaml # attacker-pod.yaml apiVersion: v1 kind: Namespace metadata: { name: attacker-ns } --- apiVersion: v1 kind: Pod metadata: name: confused-deputy-poc namespace: attacker-ns labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/component: controller spec: containers: - name: c image: registry.k8s.io/ingress-nginx/controller:v1.15.1 args: - /nginx-ingress-controller - --configmap=kube-system/coredns - --election-id=test ``` **Expected (and observed) outcomes:** | Assertion | Command | Result | |---|---|---| | Pod admitted (fail-open) | `kubectl get pod -n attacker-ns confused-deputy-poc` | ✅ admitted, no admission error | | Args UNMODIFIED | `kubectl get pod -n attacker-ns confused-deputy-poc -o jsonpath='{.spec.containers[0].args}'` | ✅ `--configmap=kube-system/coredns` preserved | | No init container injected | `kubectl get pod -n attacker-ns confused-deputy-poc -o jsonpath='{.spec.initContainers}'` | ✅ empty | | No DD ConfigMap in `kube-system` | `kubectl get cm -n kube-system \| grep -i datadog-appsec` | ✅ none | | Warning event on pod | `kubectl get events -n attacker-ns --field-selector involvedObject.name=confused-deputy-poc` | ✅ `Warning CrossNamespaceConfigMapRefused AppSec nginx mutation skipped: --configmap references a namespace different from the pod's namespace; refusing to mutate to avoid confused-deputy ConfigMap writes: pod attacker-ns/confused-deputy-poc, arg "--configmap=kube-system/coredns"` | | DCA log line | `kubectl logs -n datadog deploy/dd-datadog-cluster-agent \| grep "AppSec mutation skipped"` | ✅ `WARN \| nginx AppSec mutation skipped for pod attacker-ns/confused-deputy-poc: --configmap references a namespace different from the pod's namespace` | **3. Regression check — legitimate ingress-nginx deployments still work** Pods using the upstream Helm default (`--configmap=$(POD_NAMESPACE)/ingress-nginx-controller`) and pods using a literal same-namespace ref (`--configmap=ingress-nginx/my-config` when the pod is in `ingress-nginx`) are accepted and mutated normally. Covered by `TestFindControllerConfigMapArg/standard_$(POD_NAMESPACE)_form_is_accepted` and `.../hardcoded_same_namespace_is_accepted`. ### Additional Notes - **`qa/rc-required` is required** — admission webhook changes touch cross-component behavior (DCA ↔ kube-apiserver ↔ node agents) per `AGENTS.md` guidance. - The fix is **fail-open**: rejection results in the pod being admitted unmodified with a warning event and log line — never a failed admission. Legitimate ingress-nginx deployments using `$(POD_NAMESPACE)/...` (the upstream Helm default) are unaffected. - `createOrUpdateDDConfigMap` gains DNS-1123 validation as defense-in-depth. It is a no-op for the reconciler path (whose namespace/name come from a label-filtered informer watch and are already valid Kubernetes objects) and catches any future code path that bypasses `findControllerConfigMapArg`. - Follow-ups deferred to separate Jira tickets per the plan: - E2E test in `test/new-e2e/tests/clusteragent/appsec/` (§4.5) - Owner-reference pre-check on ingress-nginx pods (§5.2) - `ValidatingAdmissionPolicy` for ConfigMap creation scope (§6 Option C) Co-authored-by: eliott.bouhana <[email protected]>
…ap refs in pod mutation (#51709) Backport 9ae4dea from #51635. ___ ### What does this PR do? Fixes a confused-deputy vulnerability in the Cluster Agent's AppSec ingress-nginx admission mutator. The webhook previously extracted the namespace from the pod's `--configmap=<ns>/<name>` argument and used it verbatim for ConfigMap `Get`/`Create`/`Update` calls. Combined with the DCA's cluster-wide `configmaps` permissions, a low-privileged tenant with `create pods` rights in one namespace could trigger writes to ConfigMaps in arbitrary namespaces. Changes: - **`pkg/clusteragent/appsec/nginx/sidecar.go`** — `findControllerConfigMapArg` now requires the `<ns>` portion to match the pod's own namespace (resolving `$(POD_NAMESPACE)` first) and rejects empty names. On rejection, `MutatePod` returns `(false, nil)` to preserve fail-open admission semantics — the pod is admitted unmodified. - **`pkg/clusteragent/appsec/nginx/events.go`** — New `CrossNamespaceConfigMapRefused` warning event on the **pod** (not the IngressClass) so the diagnostic lands in the tenant's namespace where their operator can see it. - **`pkg/clusteragent/appsec/nginx/configmap.go`** — Defense-in-depth `validateConfigMapTarget` (DNS-1123 validation) at the entry of `createOrUpdateDDConfigMap`, covering both webhook and reconciler paths. - **Tests** — `TestMutatePod_CrossNamespaceConfigMapRefused` (the bisect anchor) asserts no API calls escape and the pod spec is unmodified. `TestFindControllerConfigMapArg` extended from 3 cases to 10 covering same-ns, foreign ns, `kube-system` reference, leading/trailing slash, multi-container priority. - **Release note** — `releasenotes/notes/fix-appsec-nginx-configmap-confused-deputy-*.yaml` (security section). The introducing change was PR #49318 (Agent 7.78.0). All releases ≥7.78.0 are affected; backports to `7.78.x`, `7.79.x`, `7.80.x` will follow. ### Motivation Tracking: [APPSEC-68212](https://datadoghq.atlassian.net/browse/APPSEC-68212). Internal vulnerability report `clusteragent-appsec-nginx-configmap-confused-deputy` (severity High, threat model k8s-tenant). Full mitigation plan: `.sisyphus/plans/clusteragent-appsec-nginx-configmap-confused-deputy-mitigation.md`. Pre-condition for exploitation: DCA with `cluster_agent.appsec.injector.enabled = true` (helm: `datadog.appsec.injector.enabled: true`), at least one ingress-nginx `IngressClass` (`controller: k8s.io/ingress-nginx`), and a tenant with `create pods` permission in any namespace. ### Describe how you validated your changes **1. Automated tests (run in CI):** ```bash dda inv test --targets=./pkg/clusteragent/appsec/nginx # 57/57 passed dda inv test --targets=./pkg/clusteragent/admission/mutate/appsec # 35/35 passed dda inv linter.go --targets=./pkg/clusteragent/appsec/nginx # 0 issues dda inv linter.releasenote # passed ``` `TestMutatePod_CrossNamespaceConfigMapRefused` is the **bisect anchor**: it fails against the unpatched code (which accepted `--configmap=kube-system/coredns` verbatim) and passes against this patch. **2. Live exploit reproduction (k3s, rancher-desktop, 7.78.0 + this patch):** Setup the DCA with my patched binary: ```bash # Overlay the patched binary on the 7.78.0 base image cat > Dockerfile.overlay <<'DOCKERFILE' FROM datadog/cluster-agent:7.78.0 COPY bin/datadog-cluster-agent/datadog-cluster-agent /opt/datadog-agent/bin/datadog-cluster-agent DOCKERFILE docker build --platform linux/arm64 -t datadog/cluster-agent:7.78.0-fix -f Dockerfile.overlay . # Install DCA with AppSec ingress-nginx enabled cat > values.yaml <<'YAML' datadog: apiKey: "0000000000000000000000000000000000000000" appKey: "0000000000000000000000000000000000000000" clusterName: confused-deputy-test appsec: injector: enabled: true autoDetect: false proxies: [ingress-nginx] clusterAgent: image: repository: datadog/cluster-agent tag: 7.78.0-fix pullPolicy: IfNotPresent admissionController: enabled: true agents: { enabled: false } clusterChecksRunner: { enabled: false } YAML helm install dd datadog/datadog -n datadog --create-namespace -f values.yaml # Pre-condition: at least one ingress-nginx IngressClass kubectl apply -f - <<'YAML' apiVersion: networking.k8s.io/v1 kind: IngressClass metadata: { name: nginx-test } spec: { controller: k8s.io/ingress-nginx } YAML ``` Apply the exploit pod from a low-privileged tenant namespace: ```yaml # attacker-pod.yaml apiVersion: v1 kind: Namespace metadata: { name: attacker-ns } --- apiVersion: v1 kind: Pod metadata: name: confused-deputy-poc namespace: attacker-ns labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/component: controller spec: containers: - name: c image: registry.k8s.io/ingress-nginx/controller:v1.15.1 args: - /nginx-ingress-controller - --configmap=kube-system/coredns - --election-id=test ``` **Expected (and observed) outcomes:** | Assertion | Command | Result | |---|---|---| | Pod admitted (fail-open) | `kubectl get pod -n attacker-ns confused-deputy-poc` | ✅ admitted, no admission error | | Args UNMODIFIED | `kubectl get pod -n attacker-ns confused-deputy-poc -o jsonpath='{.spec.containers[0].args}'` | ✅ `--configmap=kube-system/coredns` preserved | | No init container injected | `kubectl get pod -n attacker-ns confused-deputy-poc -o jsonpath='{.spec.initContainers}'` | ✅ empty | | No DD ConfigMap in `kube-system` | `kubectl get cm -n kube-system \| grep -i datadog-appsec` | ✅ none | | Warning event on pod | `kubectl get events -n attacker-ns --field-selector involvedObject.name=confused-deputy-poc` | ✅ `Warning CrossNamespaceConfigMapRefused AppSec nginx mutation skipped: --configmap references a namespace different from the pod's namespace; refusing to mutate to avoid confused-deputy ConfigMap writes: pod attacker-ns/confused-deputy-poc, arg "--configmap=kube-system/coredns"` | | DCA log line | `kubectl logs -n datadog deploy/dd-datadog-cluster-agent \| grep "AppSec mutation skipped"` | ✅ `WARN \| nginx AppSec mutation skipped for pod attacker-ns/confused-deputy-poc: --configmap references a namespace different from the pod's namespace` | **3. Regression check — legitimate ingress-nginx deployments still work** Pods using the upstream Helm default (`--configmap=$(POD_NAMESPACE)/ingress-nginx-controller`) and pods using a literal same-namespace ref (`--configmap=ingress-nginx/my-config` when the pod is in `ingress-nginx`) are accepted and mutated normally. Covered by `TestFindControllerConfigMapArg/standard_$(POD_NAMESPACE)_form_is_accepted` and `.../hardcoded_same_namespace_is_accepted`. ### Additional Notes - **`qa/rc-required` is required** — admission webhook changes touch cross-component behavior (DCA ↔ kube-apiserver ↔ node agents) per `AGENTS.md` guidance. - The fix is **fail-open**: rejection results in the pod being admitted unmodified with a warning event and log line — never a failed admission. Legitimate ingress-nginx deployments using `$(POD_NAMESPACE)/...` (the upstream Helm default) are unaffected. - `createOrUpdateDDConfigMap` gains DNS-1123 validation as defense-in-depth. It is a no-op for the reconciler path (whose namespace/name come from a label-filtered informer watch and are already valid Kubernetes objects) and catches any future code path that bypasses `findControllerConfigMapArg`. - Follow-ups deferred to separate Jira tickets per the plan: - E2E test in `test/new-e2e/tests/clusteragent/appsec/` (§4.5) - Owner-reference pre-check on ingress-nginx pods (§5.2) - `ValidatingAdmissionPolicy` for ConfigMap creation scope (§6 Option C) [APPSEC-68212]: https://datadoghq.atlassian.net/browse/APPSEC-68212?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ Co-authored-by: dd-octo-sts[bot] <200755185+dd-octo-sts[bot]@users.noreply.github.com> Co-authored-by: Eliott B <[email protected]>
What does this PR do?
Adds a new AppSec injection pattern for ingress-nginx controllers. The pattern:
datadog/ingress-nginx-injection:<version>) that copies thenginx-datadogmodule.sointo a shared emptyDir volume--configmaparg to a DD-owned ConfigMap with AppSec directives (load_module,datadog_appsec_enabled on, WAF thread pool) prepended tomain-snippetandhttp-snippetIngressClassresources and cleans up DD-owned ConfigMaps on deletionProxyTypeIngressNginxin the proxy constructor/detection mapsNew files:
pkg/clusteragent/appsec/nginx/— full package with sidecar injection, ConfigMap management, detection, and eventsadmission_controller.appsec.nginx.{init_image,module_mount_path}update/deleteviaresourceNames)Motivation
ingress-nginx is one of the most widely deployed ingress controllers. This extends AppSec proxy injection coverage beyond Envoy Gateway and Istio to support ingress-nginx clusters without requiring users to manually configure
extraModules.Describe how you validated your changes
dda inv test --targets=./pkg/clusteragent/appsec/nginx/...)Additional Notes
reno) is expected for this feature change — will add as a follow-up or in this PR if requested.get/list/create(needed for reading original ConfigMaps and label-selector cleanup) + scopedupdate/deletewithresourceNames.