Extend introspection and support DDAI code path by levan-m · Pull Request #3108 · DataDog/datadog-operator

levan-m · 2026-06-08T19:04:22Z

What does this PR do?

CONTP-1577

Change extends introspection detection mechanism to allow OOTB support for control plane monitoring on EKS (non-auto) and Openshift.

Change summary:

Moves detection logic outside reconciler.
Adds detection based on node operator is running as primary source, with an optional fallback (iff operator is watching nodes) to listing all nodes in the cluster.
Persists detection result in DDA status.
Implements precedence order
1. User provided datadoghq.com/provider annotation.
2. Non-empty detection result.
3. Provider persisted in DDA status.
Adds detection grace period in reconciler (when provider is empty) to prevent unnecessary resource churn.

Motivation

Since 1.24 users had to disable DDAI code path to use OOTB control plane monitoring, since 1.27 this option isn't available. In addition users needed to set introspectionEnabled flag for caching nodes for List operation. This change resolves former and make latter optional.

Additional Notes

Anything else we should know when reviewing?

Minimum Agent Versions

Are there minimum versions of the Datadog Agent and/or Cluster Agent required?

Agent: vX.Y.Z
Cluster Agent: vX.Y.Z

Describe your test plan

Provision regular EKS cluster
Install Operator
Apply minimal DDA
Verify control plane metrics like kube_apiserver.*, kube_scheduler.* appear in the app.

Checklist

PR has at least one valid label: bug, enhancement, refactoring, documentation, tooling, and/or dependencies
PR has a milestone or the qa/skip-qa label
All commits are signed (see: signing commits)

datadog-prod-us1-5 · 2026-06-08T19:07:05Z

✨ Fix all issues with BitsAI

🛑 Gate Violations

🎯 1 Code Coverage issue detected

A Patch coverage percentage gate may be blocking this PR.

• Patch coverage: 71.76% (threshold: 80.00%)

ℹ️ Info

🎯 Code Coverage (details)
• Patch Coverage: 71.76%
• Overall Coverage: 44.65% (+0.38%)

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: eb26997 | Docs | Datadog PR Page | Give us feedback!}

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 74e241e65d

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

codecov-commenter · 2026-06-08T20:35:28Z

Codecov Report

❌ Patch coverage is 64.03941% with 73 lines in your changes missing coverage. Please review.
✅ Project coverage is 44.18%. Comparing base (92788bf) to head (c326289).

Files with missing lines	Patch %	Lines
pkg/introspection/detector.go	63.51%	25 Missing and 2 partials ⚠️
...dogagent/feature/controlplanemonitoring/feature.go	61.70%	13 Missing and 5 partials ⚠️
cmd/main.go	0.00%	10 Missing ⚠️
pkg/kubernetes/provider.go	50.00%	8 Missing ⚠️
...rnal/controller/datadogagentinternal/controller.go	0.00%	5 Missing ⚠️
...controller/datadogagent/controller_reconcile_v2.go	92.68%	2 Missing and 1 partial ⚠️
internal/controller/datadogagent/ddai.go	85.71%	0 Missing and 1 partial ⚠️
...er/datadogagentinternal/controller_reconcile_v2.go	0.00%	1 Missing ⚠️

❌ Your patch status has failed because the patch coverage (64.03%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3108      +/-   ##
==========================================
+ Coverage   44.03%   44.18%   +0.14%     
==========================================
  Files         377      378       +1     
  Lines       30713    30903     +190     
==========================================
+ Hits        13525    13654     +129     
- Misses      16300    16352      +52     
- Partials      888      897       +9

Flag	Coverage Δ
unittests	`44.18% <64.03%> (+0.14%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
api/datadoghq/v2alpha1/datadogagent_types.go	`0.00% <ø> (ø)`
internal/controller/datadogagent/controller.go	`48.00% <ø> (ø)`
internal/controller/datadogagent/feature/types.go	`22.10% <ø> (ø)`
internal/controller/setup.go	`74.00% <100.00%> (+0.35%)`	⬆️
internal/controller/datadogagent/ddai.go	`68.25% <85.71%> (+2.15%)`	⬆️
...er/datadogagentinternal/controller_reconcile_v2.go	`0.00% <0.00%> (ø)`
...controller/datadogagent/controller_reconcile_v2.go	`64.28% <92.68%> (+11.28%)`	⬆️
...rnal/controller/datadogagentinternal/controller.go	`0.00% <0.00%> (ø)`
pkg/kubernetes/provider.go	`82.25% <50.00%> (-4.11%)`	⬇️
cmd/main.go	`6.49% <0.00%> (-0.19%)`	⬇️
... and 2 more

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 92788bf...c326289. Read the comment docs.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Mathew-Estafanous · 2026-06-09T12:48:57Z

+	case kubernetes.IsSpecificProvider(provider):
+		reason = clusterProviderReasonDetected


Why isn't source == clusterProviderSourceDetected used here to determine the reason?

First case is clusterProviderSourceUser, which effectively leaves only clusterProviderSourceDetected. Added the conditional in a77b92b regardless to make it more obvious.

Mathew-Estafanous · 2026-06-09T12:59:31Z

+		if wasProviderDetected(instance) && kubernetes.IsSpecificProvider(statusProvider) && !kubernetes.IsSpecificProvider(live) {
+			return statusProvider, clusterProviderSourceDetected, false
+		}


❓ Is it possible for the new live detection to be persistent and not a 'blip?' Say a GKE cluster going from entirely GKE COS nodes to regular ubuntu. If so, wouldn't this gate always choose the status provider over live?

It was possible as you pointed out with COS example, however this change should only handle cluster/global provider (EKS, Openshift, etc.) user cases. Intention here is correct, once cluster level provider detected, it can't shouldn't become empty, if we detect we are on EKS node label changes shouldn't revert that. Only way to override this is user provided annotation.
Made fix in a77b92b provider.go to separate cluster and node level provider detection.

Mathew-Estafanous · 2026-06-09T13:07:49Z

+	// Stage 1 — opportunistic keyed read of the operator's own node.
+	if d.nodeName != "" {
+		node := &corev1.Node{}
+		if err := d.apiReader.Get(ctx, types.NamespacedName{Name: d.nodeName}, node); err != nil {
+			d.logger.V(1).Info("operator-node provider read failed; trying cluster-node-list fallback", "node", d.nodeName, "error", err)


❓ Looking at this detection logic, is it possible the node that the operator is running on doesn't tell the full picture? Say on a GKE cluster that has both COS and ubuntu nodes. Depending on where the operator lands, it will 'detect' something different.

this should be answered in the above comment; in this PR we only care global/cluster level changes for use cases like control plane monitoring for EKS and Openshift, autopilot.

Mathew-Estafanous

provider introspection look good

tbavelier

LGTM overall, but two points:

This turns introspection "on" by default while it was previously opt-in: this is what we want, but should be called out. It is not immediately clear that introspection can be disabled using datadoghq.com/provider: "" annotation on DDA
As a follow-up to the previous point and that one is why I'm "requesting changes", this breaks OpenShift users that did not have introspection before as we enable control-plane monitoring which mounts a secret in the Agent daemonset, however, this secret needs to be manually copied over (we log the command to run).

With the second point in mind, considering the operator has cluster-wide secrets access, I'm proposing as part of the control-plane monitoring feature for OpenShift to do this operation for users: #3131 so that way we don't break them and they start benefiting from it automatically. My approach is not perfect, it has tradeoffs (direct api read (not a write) every reconciliation to avoid the error below), I'm open to alternatives (e.g. add this namespace in bundles that we deploy on OpenShift / do something else)

{"level":"INFO","ts":"2026-06-12T09:46:10.796Z","logger":"controllers.DatadogAgentInternal","msg":"Unable to copy OpenShift etcd metric client secret automatically","namespace":"system","name":"datadog","reconcileID":"dd20b327-ef67-45a2-88f4-91e849090cf0","kind":"DatadogAgentInternal","namespace":"openshift-etcd-operator","name":"etcd-metric-client","error":"unable to get: openshift-etcd-operator/etcd-metric-client because of unknown namespace for the cache"}

* copy OpenShift etcd metric secret * Add 5s timeout * Keep existing secret if source read fails

tbavelier

Conflict with main to resolve, but LGTM!

* Avoid agent getting stuck in container creating * Do not use direct apireader client and instead make f.etcdSecretPresent as part of existing calls (#3180) --------- Co-authored-by: Timothée Bavelier <[email protected]>

Extend introspection and support DDAI code path

74e241e

levan-m added this to the v1.29.0 milestone Jun 8, 2026

levan-m requested a review from a team June 8, 2026 19:04

levan-m requested a review from a team as a code owner June 8, 2026 19:04

levan-m added the enhancement New feature or request label Jun 8, 2026

github-actions Bot added team/container-platform team/container-autoscaling labels Jun 8, 2026

chatgpt-codex-connector Bot reviewed Jun 8, 2026

View reviewed changes

Comment thread api/datadoghq/v2alpha1/datadogagent_types.go

handle edge cases when user unsets annotation

9539308

Mathew-Estafanous reviewed Jun 9, 2026

View reviewed changes

levan-m added 2 commits June 9, 2026 16:34

pr review fixes

a77b92b

Merge branch 'main' into levan-m/introspection-ddai

8dc65bc

Mathew-Estafanous approved these changes Jun 9, 2026

View reviewed changes

levan-m added 2 commits June 9, 2026 21:05

skip adding default provider to ddai

0fce08d

remove logging

c5de88a

tbavelier requested changes Jun 12, 2026

View reviewed changes

Copy OpenShift etcd metric secret (#3131)

59dbb2c

* copy OpenShift etcd metric secret * Add 5s timeout * Keep existing secret if source read fails

tbavelier requested a review from a team as a code owner June 15, 2026 07:56

github-actions Bot added the team/container-integrations label Jun 15, 2026

tbavelier approved these changes Jun 15, 2026

View reviewed changes

khewonc approved these changes Jun 16, 2026

View reviewed changes

levan-m and others added 3 commits June 17, 2026 16:29

Merge branch 'main' into levan-m/introspection-ddai

c326289

fix podstartup (#3179)

e36d53e

* Avoid agent getting stuck in container creating * Do not use direct apireader client and instead make f.etcdSecretPresent as part of existing calls (#3180) --------- Co-authored-by: Timothée Bavelier <[email protected]>

Merge branch 'main' into levan-m/introspection-ddai

eb26997

tbavelier approved these changes Jun 24, 2026

View reviewed changes

levan-m merged commit d0a7240 into main Jun 24, 2026
37 of 38 checks passed

levan-m deleted the levan-m/introspection-ddai branch June 24, 2026 13:37

		case kubernetes.IsSpecificProvider(provider):
		reason = clusterProviderReasonDetected

Uh oh!

Conversation

levan-m commented Jun 8, 2026

What does this PR do?

Motivation

Additional Notes

Minimum Agent Versions

Describe your test plan

Checklist

Uh oh!

datadog-prod-us1-5 Bot commented Jun 8, 2026 • edited by datadog-official Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🛑 Gate Violations

🎯 1 Code Coverage issue detected

ℹ️ Info

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

codecov-commenter commented Jun 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Mathew-Estafanous Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

levan-m Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

Mathew-Estafanous Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

levan-m Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

Mathew-Estafanous Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

levan-m Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

Mathew-Estafanous left a comment

Choose a reason for hiding this comment

Uh oh!

tbavelier left a comment

Choose a reason for hiding this comment

Uh oh!

tbavelier left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

datadog-prod-us1-5 Bot commented Jun 8, 2026 •

edited by datadog-official Bot

Loading

codecov-commenter commented Jun 8, 2026 •

edited

Loading