Pre-submission Checklist

• I have searched existing issues and this is not a duplicate
• This is a Datadog Operator issue (CRDs, reconciliation, etc.), not a Datadog Agent or Datadog service problem (dashboards, monitors, etc.)

Operator version

1.23.1

Operator Helm chart version

2.18.1

Bug Report

TL;DR: At high CR counts, the 60-second status-polling RequeueAfter saturates the single-threaded reconcile queue, starving new and changed CRs indefinitely. We propose making MaxConcurrentReconciles and defaultRequeuePeriod configurable, and using controller-runtime's priority queue to deprioritize status-polling requeues.

What problem are you trying to solve?

When a cluster has many DatadogMonitor CRs, creating a new monitor takes an unreasonably long time to reconcile — or never reconciles at all. In our load testing, after scaling to 6,000 DatadogMonitor CRs, a brand-new CR still had no status.id after 40+ minutes of waiting. The operator was healthy the entire time; the new CR was simply starved by the reconcile queue.

The root cause is that the DatadogMonitor controller's reconcile queue is saturated with periodic status-polling requeues for existing monitors, and new CRs have no priority over these routine refreshes.

Why this matters

At scale, developers creating or editing DatadogMonitor CRs will experience long, unpredictable delays before their changes take effect in Datadog. A new monitor may not go live for 30-40 minutes. Worse, edits to existing monitors (e.g., adjusting a threshold during an incident) face the same queue starvation — the spec change sits behind thousands of status-refresh requeues. This is confusing and erodes trust in the operator as the source of truth for monitoring configuration. In severe cases, the queue backlog can cause liveness probe failures or OOM kills, leading to a CrashLoopBackOff cycle where each restart resets all progress and re-enqueues every CR simultaneously — compounding the problem further (see #2344).

How the queue saturates

Three factors compound to create this problem:

1. Every CR self-requeues every 60 seconds via RequeueAfter (defaultRequeuePeriod). With N monitors, this produces N queue entries per minute regardless of whether anything changed.

2. MaxConcurrentReconciles defaults to 1 and is not configurable. All reconciliation is single-threaded. Each status-polling reconcile makes up to two external calls (Datadog API GET + Kubernetes status update if state changed; the K8s write is skipped when the monitor state is unchanged via a DeepEqual guard), with an observed average of ~300-400ms per reconcile under load (from controller_runtime_reconcile_time_seconds, controller=datadogmonitor).

3. The queue is FIFO and never drains. A new CR from a watch event joins the back of the queue behind all the status-refresh requeues. At 6,000 monitors, the queue reaches a steady-state depth of ~5,850 items — items re-enter via RequeueAfter faster than the single worker can process them. A new CR at the back of this queue waits 30-40 minutes before being processed.

The controller does use GenerationChangedPredicate as an event filter, but this only applies to informer watch events. It has no effect on RequeueAfter-driven self-requeues, which are the dominant source of queue entries.

Operator restarts amplify this further. On restart, the informer's initial List enqueues all N CRs simultaneously as Create events (GenerationChangedPredicate does not filter Create events), eliminating the natural stagger that exists in steady-state. Every CR is also more expensive to reconcile after a restart: MonitorStateLastUpdateTime is stale for all CRs, forcing a Datadog API GET on every one, and CRs where MonitorLastForceSyncTime is also stale hit the heavier force-sync path (GET + VALIDATE + UPDATE + K8s status write — 4 external calls vs. up to 2 for a status poll). The result is a burst of maximum-cost reconciles with no spread.

What we've observed

We confirmed via datadog.operator.leader_election_master_status metrics and pod logs that the operator was healthy during these delays — the new CR was simply waiting in the queue. The steady-state queue math from the section above explains why.

Proposed solutions

We explored running multiple operator replicas to increase throughput, but leader election restricts the controller to a single active instance. The solutions below address the bottleneck within the single leader.

Any combination of the following would significantly improve the situation. For maximum effect at 6,000+ CRs, we recommend implementing all three — the quantitative analysis below shows why no single solution is sufficient on its own. Our goal: a new DatadogMonitor CR should reconcile within 30 seconds regardless of total CR count.

1. Make MaxConcurrentReconciles configurable (highest impact, lowest risk)

Expose a configuration option (environment variable, Helm value, CLI flag — whatever fits the operator's config pattern) to set MaxConcurrentReconciles for the DatadogMonitor controller. The current hardcoded default of 1 means all reconciliation is single-threaded.

ctrl.NewControllerManagedBy(mgr).
    WithOptions(controller.Options{
        MaxConcurrentReconciles: configuredValue, // default 1, user-configurable
    }).
    // ...

The Datadog API supports concurrent requests, and the work queue's key-based deduplication prevents the same CR from being reconciled concurrently. The main concern is API rate limiting, which operators at scale are already managing. For reference, other operators in the ecosystem expose this as configurable: Crossplane defaults to 10, Flux to 4, and cert-manager to 5.

Important: Increasing MaxConcurrentReconciles alone does not fully solve the problem at 6,000 CRs with a 60-second requeue. The minimum workers needed to drain the queue is (N × AvgReconcileTime) / RequeuePeriod:

At MaxConcurrentReconciles=10 with a 60s requeue, wait time improves ~10x (from ~30 min to ~3 min) but the queue still cannot drain. The combination of MaxConcurrentReconciles=10 + defaultRequeuePeriod=300s is what achieves seconds-level responsiveness for new CRs (7 workers needed, 10 available).

2. Use the controller-runtime priority queue to deprioritize status-polling requeues

The operator recently enabled UsePriorityQueue on main (PR #2772, merged 2026-03-18, not yet in any release — including v1.24.0 and v1.25.0-rc.1). This provides partial help during operator restarts: controller-runtime automatically assigns LowPriority (-100) to initial List events via the IsInInitialList flag (wired through all three built-in event handlers), so any CRs created during a restart storm would be processed ahead of the initial list backlog. However, this only affects ordering during startup — it does not reduce total volume, and it has no effect on steady-state operation.

The dominant queue pressure comes from RequeueAfter self-requeues, not informer events. The IsInInitialList deprioritization does not apply to self-requeues. To address the core problem, the controller would need to assign lower priority to status-polling requeues so that create/update/delete watch events are always processed first. Controller-runtime's WithLowPriorityWhenUnchanged handler wrapper targets informer resyncs (where resource version has not changed), not RequeueAfter, so custom priority assignment in the reconcile return path would be needed.

The good news is that the infrastructure for this is already in place. Controller-runtime v0.22.4 (which the operator uses) supports returning priority directly on the reconcile result via reconcile.Result.Priority. Separately, the priority retention bug (controller-runtime #3157, fixed in PR #3167) ensures that handler-assigned priorities (such as the IsInInitialList low priority from startup) are preserved across requeues — complementing the explicit Result.Priority approach below. The implementation would be straightforward:

// Status-polling requeue — low priority, processed after create/update/delete events
return reconcile.Result{
    RequeueAfter: defaultRequeuePeriod,
    Priority:     ptr.To(-100),
}, nil

3. Make defaultRequeuePeriod configurable

The 60-second requeue is aggressive for status polling. Exposing this as a user-configurable option (environment variable, Helm value, CLI flag) would let operators tune it based on their scale. At 6,000 monitors, even increasing to 120 seconds halves the queue pressure. This was also requested in #1171.

Longer-term: batch status polling

As an architectural note, the most effective long-term solution would be to decouple status polling from the reconcile queue entirely. The Datadog List Monitors API (GET /api/v1/monitor) can return all monitors with their current state in a single call. A background goroutine could periodically call this API (e.g., every 60-120s), compare returned states with the operator's last-known state, and only enqueue CRs whose state has actually changed. This would reduce queue entries from N/minute to only those with real state drift — likely tens per minute rather than thousands. We recognize this is a larger architectural change and not the scope of this request, but mention it as context for the design direction.

Related issues

• #1754 — "Many Clusters - Many Monitors - Impossible to Use"
• #2344 — "Datadog Operator times out while reconciling - CrashLoopBackOff" (restart thundering herd can trigger this feedback loop)
• #2682 — "Allow for configurable forceSyncPeriod for resources"
• #1171 — "Feature Request: customizable requeue time per monitor"
• controller-runtime #2374 — "Feature: Priority Queue"
• controller-runtime #3157 — Priority retention fix (resolved via PR #3167)

Steps to Reproduce

1. Deploy Datadog Operator v1.23.1 (Helm chart 2.18.1) to an EKS cluster
2. Create 6,000 DatadogMonitor CRs
3. Wait for steady state (~30 min for all CRs to get status.id)
4. Create a new DatadogMonitor CR
5. Observe that the new CR does not receive status.id for 40+ minutes

Environment

Kubernetes version: 1.34 (EKS)
Helm version: 3.x
Cloud provider: AWS

Additional Context

• CR count: up to 6,000 DatadogMonitors (load test), planning for 12+ clusters