Backport e3f69bf from #2857.

What does this PR do?

When spec.global.csi.enabled=true, the DatadogAgent controller can now create and manage a DatadogCSIDriver custom resource on behalf of the DDA. The DatadogCSIDriver controller then reconciles it into the CSI DaemonSet and the cluster-scoped k8s.csi.datadoghq.com CSIDriver.

Controller-level opt-in. Management is gated on the existing operator flag --datadogCSIDriverEnabled (default false). With the flag off, csi.enabled=true keeps its long-standing semantics — the Agent uses a CSI driver that's installed by some other means (Helm chart, manual apply, GitOps) — and the operator doesn't touch anything. This preserves backward compatibility for existing clusters.

Defer to external installs. On the first-time create path (no DDA-owned CR yet), if a cluster-scoped k8s.csi.datadoghq.com CSIDriver is already present, the operator defers and skips creation. Avoids collisions with Helm/manual installs. Once the operator owns a CR, subsequent reconciles stay on the update path regardless of external CSIDriver presence.

Per-DDA opt-out for migrations. New field spec.global.csi.autoManage (*bool, default true). Flipping to false triggers cleanup of the operator-owned CR so users can hand ownership over to a DatadogCSIDriver they maintain themselves (useful when they need customizations not exposed via DDA spec) without having to toggle csi.enabled off/on.

Spec propagation from DDA. The created DatadogCSIDriver is built from:

• Spec.APMSocketPath ← dda.Spec.Features.APM.UnixDomainSocketConfig.Path
• Spec.DSDSocketPath ← dda.Spec.Features.Dogstatsd.UnixDomainSocketConfig.Path
• Spec.Override.Tolerations ← dda.Spec.Global.CSI.Tolerations
• Spec.Override.NodeSelector ← dda.Spec.Global.CSI.NodeSelector
• Spec.Override.Affinity.NodeAffinity ← dda.Spec.Global.CSI.NodeAffinity
• Default operator-managed labels/annotations (app.kubernetes.io/managed-by=datadog-operator, etc.)
• controllerutil.SetControllerReference → ownerRef for cascade delete

Scheduling fields (tolerations, nodeSelector, nodeAffinity) are set explicitly on spec.global.csi rather than being implicitly inherited from the node-agent component override. The Spec.Override struct is only populated when at least one scheduling field is set.

Update semantics. Spec-drift updates use a client.MergeFrom JSON merge patch (instead of UpdateFromObject), so only the fields we own (.spec, labels, annotations) appear in the wire payload. Finalizers added by the sibling DatadogCSIDriver controller (finalizer.datadoghq.com/csi-driver) and .status are not touched — prevents a class of bug where a spec-drift update would wipe the finalizer and a subsequent delete would leak the cluster-scoped CSIDriver.

Event-driven reconciles. The DDA controller now declares Owns(&v1alpha1.DatadogCSIDriver{}) (gated on DatadogCSIDriverEnabled to avoid startup crashes when the CRD is absent, matching the DatadogAgentInternal/ExtendedDaemonSet pattern). Drift on the operator-owned CR now self-heals within milliseconds via watch events rather than waiting for the 15s periodic requeue.

Motivation

Simplify CSI driver setup. Users can configure CSI on their DatadogAgent (and for cluster admins that want the operator to install the driver, flip --datadogCSIDriverEnabled at deployment time) without maintaining a separate DatadogCSIDriver by hand. For advanced users who need finer-grained control, autoManage=false offers a seamless migration path off "automatic" operator management based on DDA spec.

Additional Notes

• The DatadogCSIDriver lifecycle is managed via direct API calls (get/patch/delete) rather than the dependency store, because the store only supports built-in Kubernetes types. Documented in code comments on reconcileDatadogCSIDriver.
• RBAC is unchanged — the existing csidrivers rules (list/watch/create cluster-wide, get/update/patch/delete scoped to k8s.csi.datadoghq.com) cover the new external-check Get.

Minimum Agent Versions

N/A — this is an operator-level feature.

Describe your test plan

Unit tests in ddcsi_test.go cover:

• Disabled — csi.enabled=false, no CR created
• EnabledAndCreated — csi.enabled=true, flag on, CR created with owner ref and operator-managed labels
• SpecFromDDA — APMSocketPath, DSDSocketPath, and tolerations (via spec.global.csi.tolerations) propagated from DDA
• NodeSelectorPropagated — spec.global.csi.nodeSelector forwarded to Spec.Override.NodeSelector
• NodeAffinityPropagated — spec.global.csi.nodeAffinity forwarded to Spec.Override.Affinity.NodeAffinity
• NoOverrideWhenNoTolerations — override not set when no scheduling fields are configured
• AutoManageDisabledCleansUpAndDoesNotRecreate — opt-out deletes the CR and stays out of the way
• AutoManageDisabledSkipsForeignCR — opt-out honors IsControlledBy guard
• ControllerDisabled — flag off is a no-op (no error, no CR)
• DefersToExternalCSIDriver — pre-existing cluster-scoped CSIDriver blocks first-time create
• ExternalAppearsAfterCRCreated — once owning a CR, external appearance doesn't abandon it
• Idempotent — repeat reconciles are no-ops
• CleanupOnDisable — owned resource deleted on toggle
• CleanupSkipsNotOwned — manually-created CRs left alone
• UpdateOnSpecDrift — drift reconciled back via merge patch
• UpdatePreservesFinalizers — spec-drift update preserves externally-added finalizers (finalizer-wipe bug regression test)
• CleanupCRDNotAvailable — no error when CRD isn't present

Checklist

• PR has at least one valid label: bug, enhancement, refactoring, documentation, tooling, and/or dependencies
• PR has a milestone or the qa/skip-qa label
• All commits are signed (see: signing commits)