What does this PR do?

Improves DatadogGenericResource reconciliation behavior at high resource counts:

• adds configurable DatadogGenericResource controller concurrency with --datadogGenericResourceMaxConcurrentReconciles / DD_GENERIC_RESOURCE_MAX_CONCURRENT_RECONCILES
• adds configurable DDGR status polling cadence with --datadogGenericResourceRequeuePeriod / DD_GENERIC_RESOURCE_REQUEUE_PERIOD
• keeps option precedence aligned with other operator startup options: compiled default < environment variable < CLI flag
• marks DDGR live-state polling requeues as low priority so create/update/delete work can run ahead of status refresh backlog
• documents the new DDGR tuning options

DD_GENERIC_RESOURCE_REQUEUE_PERIOD uses the shared duration env parser, so values must be valid Go duration strings such as 30s, 5m, or 1h. Bare integers are invalid and leave the default 60s in effect.

Motivation

At high CR counts, periodic status polling can dominate the controller queue and delay user-facing create/update/delete operations. This keeps DDGR status polling tunable and lower priority while leaving normal reconciliation and backend error retries at regular priority.

Should help in situations like #2816

Additional Notes

The low-priority path is limited to DDGR refreshState reconciles, which currently applies to resource types that fetch backend state such as monitors and SLOs. It intentionally does not lower the priority of create/update/delete retries.

Validated manually that workqueue_depth{controller="DatadogGenericResource", priority="-100"} exposes low-priority queue depth for status polling requeues. Created 1k DDGR with 10 concurrent reconciles, then changed to 1 concurrent reconcile to accumulate backlog in low-priority queue for status updates. Created an additional DDGR and verified it was treated in priority over the backlog and synced with the backend.

Shows the low-priority queue backlog while the regular priority is staying healthy.

Minimum Agent Versions

No minimum Agent or Cluster Agent version changes.

• Agent: N/A
• Cluster Agent: N/A

Describe your test plan

Automated tests:

go test ./cmd/...
go test ./internal/controller/datadoggenericresource
go test ./internal/controller

Manual scale and priority check:

1. Deploy the operator with DDGR enabled and --datadogGenericResourceMaxConcurrentReconciles=10 or DD_GENERIC_RESOURCE_MAX_CONCURRENT_RECONCILES=10.
2. Create 1k DDGRs at once (see helper in DDGR monitor load helper block below). The operator should process the queue in a few minutes and each DDGR should get an ID.
3. Change datadogGenericResourceMaxConcurrentReconciles back to 1, or remove it since 1 is the default, and wait for the new operator pod to acquire the lease.
4. Confirm backlog accumulates in the low-priority queue for status updates.
5. Create a new DDGR, for example kubectl apply -f examples/datadoggenericresource/dashboard-sample.yaml.
6. Confirm it syncs and gets an ID ahead of the status update backlog.
7. Delete it with kubectl delete ddgr ddgr-dashboard-sample and confirm it is deleted.
8. Change back to --datadogGenericResourceMaxConcurrentReconciles=10 or DD_GENERIC_RESOURCE_MAX_CONCURRENT_RECONCILES=10 and clean up the load-test resources.

#!/usr/bin/env bash
set -euo pipefail

ACTION="${1:-create}" # create or delete
COUNT="${COUNT:-100}"
NAMESPACE="${NAMESPACE:-default}"
PREFIX="${PREFIX:-ddgr-load-test-monitor}"
MANIFEST="$(mktemp)"
trap 'rm -f "$MANIFEST"' EXIT

render() {
  for i in $(seq 1 "$COUNT"); do
    name="${PREFIX}-${i}"
    cat <<EOF
---
apiVersion: datadoghq.com/v1alpha1
kind: DatadogGenericResource
metadata:
  name: ${name}
spec:
  type: monitor
  jsonSpec: |-
    {
      "name": "${name}",
      "type": "metric alert",
      "query": "avg(last_5m):avg:system.load.1{*} > 100000",
      "message": "load test monitor",
      "tags": ["test:ddgr-load", "source:datadog-operator-pr-3143"]
    }
EOF
  done
}

render > "$MANIFEST"

case "$ACTION" in
  create)
    kubectl -n "$NAMESPACE" apply -f "$MANIFEST"
    ;;
  delete)
    kubectl -n "$NAMESPACE" delete -f "$MANIFEST" --ignore-not-found
    ;;
  *)
    echo "usage: COUNT=100 NAMESPACE=default $0 [create|delete]" >&2
    exit 1
    ;;
esac

COUNT=1000 NAMESPACE=default ./ddgr-load.sh create
COUNT=1000 NAMESPACE=default ./ddgr-load.sh delete

Requeue period check:

1. Deploy the operator with DDGR enabled and DD_GENERIC_RESOURCE_REQUEUE_PERIOD=5m, or use --datadogGenericResourceRequeuePeriod=5m.
2. Create a few DDGR monitors and verify state sync happens after roughly 5 minutes.
3. Set DD_GENERIC_RESOURCE_REQUEUE_PERIOD to an invalid duration such as foo or 120.
4. Verify the parser logs an ignoring invalid env var DD_GENERIC_RESOURCE_REQUEUE_PERIOD=... message and resources fall back to the default 60s sync cadence.
5. Set both DD_GENERIC_RESOURCE_REQUEUE_PERIOD=5m and --datadogGenericResourceRequeuePeriod=2m, then verify the CLI flag wins.

Checklist

• PR has at least one valid label: bug, enhancement, refactoring, documentation, tooling, and/or dependencies
• PR has a milestone or the qa/skip-qa label
• All commits are signed (see: signing commits)