Refactor experiment signals by khewonc · Pull Request #2944 · DataDog/datadog-operator

khewonc · 2026-04-27T14:20:19Z

What does this PR do?

Refactor experiment signals for more robustness. Instead of having both daemon and controller write to DDA status, have only controller write to DDA status and daemon modify DDA annotations

Motivation

What inspired you to submit this pull request?

Additional Notes

Anything else we should know when reviewing?

Minimum Agent Versions

Are there minimum versions of the Datadog Agent and/or Cluster Agent required?

Agent: vX.Y.Z
Cluster Agent: vX.Y.Z

Describe your test plan

Write there any instructions and details you may have to test your PR.

Checklist

PR has at least one valid label: bug, enhancement, refactoring, documentation, tooling, and/or dependencies
PR has a milestone or the qa/skip-qa label
All commits are signed (see: signing commits)

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 56971366d8

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

codecov-commenter · 2026-04-27T21:12:11Z

Codecov Report

❌ Patch coverage is 79.56081% with 121 lines in your changes missing coverage. Please review.
✅ Project coverage is 41.50%. Comparing base (7f86fa0) to head (f269455).

Files with missing lines	Patch %	Lines
internal/controller/datadogagent/experiment.go	72.18%	30 Missing and 12 partials ⚠️
pkg/fleet/daemon_worker.go	76.22%	30 Missing and 4 partials ⚠️
pkg/fleet/daemon_operations.go	86.85%	14 Missing and 14 partials ⚠️
pkg/fleet/daemon.go	64.86%	12 Missing and 1 partial ⚠️
pkg/remoteconfig/updater.go	80.00%	3 Missing ⚠️
cmd/main.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2944      +/-   ##
==========================================
+ Coverage   40.90%   41.50%   +0.60%     
==========================================
  Files         333      335       +2     
  Lines       28327    28714     +387     
==========================================
+ Hits        11588    11919     +331     
- Misses      15960    16001      +41     
- Partials      779      794      +15

Flag	Coverage Δ
unittests	`41.50% <79.56%> (+0.60%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
internal/controller/datadogagent/revision.go	`78.46% <100.00%> (+0.33%)`	⬆️
pkg/fleet/experiment.go	`79.48% <100.00%> (-4.19%)`	⬇️
pkg/fleet/remote_config.go	`100.00% <100.00%> (ø)`
cmd/main.go	`6.88% <0.00%> (ø)`
pkg/remoteconfig/updater.go	`4.02% <80.00%> (+4.02%)`	⬆️
pkg/fleet/daemon.go	`69.94% <64.86%> (+4.48%)`	⬆️
pkg/fleet/daemon_operations.go	`86.85% <86.85%> (ø)`
pkg/fleet/daemon_worker.go	`76.22% <76.22%> (ø)`
internal/controller/datadogagent/experiment.go	`77.60% <72.18%> (-7.56%)`	⬇️

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7f86fa0...f269455. Read the comment docs.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

datadog-official · 2026-04-27T21:12:59Z

✨ Fix all issues with BitsAI

🛑 Gate Violations

🎯 1 Code Coverage issue detected

A Patch coverage percentage gate may be blocking this PR.

• Patch coverage: 78.99% (threshold: 80.00%)

ℹ️ Info

🎯 Code Coverage (details)
• Patch Coverage: 78.99%
• Overall Coverage: 41.82% (+0.61%)

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: f269455 | Docs | Datadog PR Page | Give us feedback!}

khewonc · 2026-04-28T14:41:08Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f0e5eb7b83

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-04-28T14:47:01Z

+
+	// Process annotation-based signals first — they take priority over
+	// automatic timeout since they represent explicit human/RC intent.
+	pendingClearID, err := r.processExperimentSignal(ctx, instance, newStatus, revList)


Clear no-op experiment signals before returning

manageExperiment now processes annotation signals first, but when status.experiment is nil the function returns immediately and never reaches the annotation-clearing block. That leaves no-op signals (for example rollback/promote when no experiment is active) stuck on the resource, causing repeated reprocessing and making daemon phase waits fail to converge. Ensure processed signals are cleared even when there is no current experiment status.

Useful? React with 👍 / 👎.

* refactor * review fixes * Review suggestions * add cluster uid tag * Fix version check * Update default versions * separate goroutine for acks * Review suggestions * Simplify refactor * fix go.mod * skip checking experiment ID on promote signal * exclude fleet.datadoghq.com annotation from controller revision --------- (cherry picked from commit 3df38d6) Co-authored-by: khewonc <[email protected]> Co-authored-by: Paul Coignet <[email protected]> Co-authored-by: levan-m <[email protected]> Co-authored-by: Levan Machablishvili <[email protected]>

* refactor * review fixes * Review suggestions * add cluster uid tag * Fix version check * Update default versions * separate goroutine for acks * Review suggestions * Simplify refactor * fix go.mod * skip checking experiment ID on promote signal * exclude fleet.datadoghq.com annotation from controller revision --------- Co-authored-by: Paul Coignet <[email protected]> Co-authored-by: levan-m <[email protected]> Co-authored-by: Levan Machablishvili <[email protected]>

khewonc added this to the v1.27.0 milestone Apr 27, 2026

khewonc added the refactoring label Apr 27, 2026

github-actions Bot added team/container-platform team/container-autoscaling team/fleet labels Apr 27, 2026

khewonc marked this pull request as ready for review April 27, 2026 20:50

khewonc requested a review from a team April 27, 2026 20:50

khewonc requested review from a team as code owners April 27, 2026 20:50

refactor

4dac8a7

chatgpt-codex-connector Bot reviewed Apr 27, 2026

View reviewed changes

Comment thread internal/controller/datadogagent/experiment.go Outdated

Comment thread pkg/fleet/daemon.go Outdated

Comment thread pkg/fleet/experiment.go Outdated

khewonc force-pushed the khewonc/refactor-experiment-signals branch from 5697136 to 4dac8a7 Compare April 27, 2026 21:00

review fixes

f0e5eb7

chatgpt-codex-connector Bot reviewed Apr 28, 2026

View reviewed changes

khewonc and others added 3 commits April 29, 2026 12:11

Review suggestions

5be8f3c

add cluster uid tag

656fd85

Fix version check

0ccb0ad

AlexanderYastrebov reviewed May 5, 2026

View reviewed changes

Comment thread api/datadoghq/v2alpha1/datadogagent_types.go

coignetp and others added 2 commits May 5, 2026 17:44

Update default versions

4ed3169

separate goroutine for acks

b441e06

This was referenced May 6, 2026

[FA] Serialize UPDATER_TASK execution and fix no-op config version clearing #2956

Closed

[FA] Read name from UPDATER_TASK #2952

Closed

coignetp approved these changes May 7, 2026

View reviewed changes

Comment thread pkg/fleet/daemon_operations.go Outdated

khewonc and others added 4 commits May 7, 2026 16:16

Review suggestions

7b9a544

Simplify refactor

fa556ba

Merge branch 'main' into khewonc/refactor-experiment-signals

bbf7624

fix go.mod

c4caf3a

levan-m and others added 3 commits May 12, 2026 15:29

skip checking experiment ID on promote signal

a57d95b

exclude fleet.datadoghq.com annotation from controller revision

ebccc0f

Merge branch 'main' into khewonc/refactor-experiment-signals

f269455

levan-m approved these changes May 14, 2026

View reviewed changes

levan-m merged commit 3df38d6 into main May 14, 2026
38 of 39 checks passed

levan-m deleted the khewonc/refactor-experiment-signals branch May 14, 2026 18:35

levan-m added the backport/v1.27 label May 14, 2026

dd-octo-sts Bot mentioned this pull request May 14, 2026

[Backport v1.27] Refactor experiment signals #3016

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Refactor experiment signals#2944

Refactor experiment signals#2944
levan-m merged 14 commits into
mainfrom
khewonc/refactor-experiment-signals

khewonc commented Apr 27, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov-commenter commented Apr 27, 2026 •

edited

Loading

Uh oh!

datadog-official Bot commented Apr 27, 2026 •

edited

Loading

🎯 1 Code Coverage issue detected

Uh oh!

khewonc commented Apr 28, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot Apr 28, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

khewonc commented Apr 27, 2026

What does this PR do?

Motivation

Additional Notes

Minimum Agent Versions

Describe your test plan

Checklist

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov-commenter commented Apr 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

datadog-official Bot commented Apr 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🛑 Gate Violations

🎯 1 Code Coverage issue detected

ℹ️ Info

Uh oh!

khewonc commented Apr 28, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot Apr 28, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

codecov-commenter commented Apr 27, 2026 •

edited

Loading

datadog-official Bot commented Apr 27, 2026 •

edited

Loading