Skip to content

[Backport v1.27] Emit Kubernetes events for FA experiment transitions#3051

Merged
levan-m merged 1 commit into
v1.27from
backport-3037-to-v1.27
May 27, 2026
Merged

[Backport v1.27] Emit Kubernetes events for FA experiment transitions#3051
levan-m merged 1 commit into
v1.27from
backport-3037-to-v1.27

Conversation

@dd-octo-sts

@dd-octo-sts dd-octo-sts Bot commented May 27, 2026

Copy link
Copy Markdown

Backport 1e1aeff from #3037.

What does this PR do?

Produce k8s events on experiment signals and transition. Option enabled by

        - name: DD_FLEET_MANAGEMENT_EVENTS_ENABLED
          value: "true"

and collected in DD by adding

spec:
  features:
    eventCollection:
      collectKubernetesEvents: true
      unbundleEvents: true
      collectedEventTypes:
        - kind: DatadogAgent
          reasons:
            - ExperimentStartProcessed
            - ExperimentPromoted
            - ExperimentAborted
            - ExperimentTimedOut
            - ExperimentRolledBack
            - RemoteTaskReceived
            - RemoteTaskRejected
            - RemoteTaskCompleted
            - LocalTerminationPublishedToRC
            - InstallerStateRehydrated
experiment transitions in reconciler

    - ExperimentStartProcessed         Normal   (nil → Running)
    - ExperimentPromoted                Normal   (Running → Promoted)
    - ExperimentAborted                 Warning  (Running → Aborted)
    - ExperimentTimedOut                Warning  (Running → Terminated/timed_out)
    - ExperimentRolledBack              Normal   (Running → Terminated/stopped)

fleet daemon incoming/outgoing signals

    - RemoteTaskReceived               Normal   (handleTask entry)
    - RemoteTaskRejected               Warning  (after setTaskState INVALID_STATE/ERROR)
    - RemoteTaskCompleted              Normal   (after setTaskState DONE in finishPendingOperation)
    - LocalTerminationPublishedToRC    Normal   (timeout or abort published to RC)
    - InstallerStateRehydrated         Normal   (state seeded from DDA on daemon startup)

Motivation

What inspired you to submit this pull request?

Additional Notes

Anything else we should know when reviewing?

Minimum Agent Versions

Are there minimum versions of the Datadog Agent and/or Cluster Agent required?

  • Agent: vX.Y.Z
  • Cluster Agent: vX.Y.Z

Describe your test plan

Enabled events, run experiment and describe DDA to see events

Events:
  Type    Reason                    Age    From          Message
  ----    ------                    ----   ----          -------
  Normal  RemoteTaskReceived        5m56s  fleet-daemon  Received start task "c03e6686-0b61-41c5-9f2c-ce7fc91a42ce" for experiment "p2wg-tvmn-g14r"
  Normal  RemoteTaskCompleted       5m56s  fleet-daemon  Task "c03e6686-0b61-41c5-9f2c-ce7fc91a42ce" (start) reported to Fleet Automation as DONE
  Normal  ExperimentStartProcessed  5m56s  DatadogAgent  Experiment "p2wg-tvmn-g14r" started (task "c03e6686-0b61-41c5-9f2c-ce7fc91a42ce")
  Normal  RemoteTaskReceived        5m25s  fleet-daemon  Received promote task "5b5ba1f1-df31-4aa2-849b-d17f845f6ec9" for experiment ""
  Normal  ExperimentPromoted        5m25s  DatadogAgent  Experiment "p2wg-tvmn-g14r" promoted
  Normal  RemoteTaskCompleted       5m25s  fleet-daemon  Task "5b5ba1f1-df31-4aa2-849b-d17f845f6ec9" (promote) reported to Fleet Automation as DONE
  Normal  RemoteTaskReceived        68s    fleet-daemon  Received start task "c3985b4a-7f1d-4bae-b1ab-058ff978e795" for experiment "djsg-awhw-oxg7"
  Normal  RemoteTaskCompleted       68s    fleet-daemon  Task "c3985b4a-7f1d-4bae-b1ab-058ff978e795" (start) reported to Fleet Automation as DONE
  Normal  RemoteTaskReceived        47s    fleet-daemon  Received promote task "9de4334d-b049-4955-aca6-956454c236f6" for experiment ""
  Normal  ExperimentPromoted        47s    DatadogAgent  Experiment "djsg-awhw-oxg7" promoted
  Normal  RemoteTaskCompleted       47s    fleet-daemon  Task "9de4334d-b049-4955-aca6-956454c236f6" (promote) reported to Fleet Automation as DONE

Checklist

  • PR has at least one valid label: bug, enhancement, refactoring, documentation, tooling, and/or dependencies
  • PR has a milestone or the qa/skip-qa label
  • All commits are signed (see: signing commits)

@levan-m @github-actions
Emit Kubernetes events for FA experiment transitions (#3037)
3bbba75 
* fix(experiment): anchor timeout on Status.Experiment.StartedAt

* fix(fleet): rehydrate installer state from DatadogAgent on daemon startup

* fix(experiment): report TaskState_ERROR for the original start task on timeout

* refactor(experiment): single experiment-state annotation on revisions

* fix(fleet): publish abort to Fleet Automation the same way as timeout

* feat(experiment): emit Kubernetes events for state transitions and Fleet tasks

* fix linter

* fix(experiment): cover all phase transitions and emit completed event for idempotent tasks

* fix(fleet): bypass cache when emitting rehydrate event

* fix linter

(cherry picked from commit 1e1aeff)
@dd-octo-sts dd-octo-sts Bot added enhancement New feature or request backport label added by backport action labels May 27, 2026
@dd-octo-sts
dd-octo-sts Bot requested a review from a team as a code owner May 27, 2026 18:46
@dd-octo-sts dd-octo-sts Bot added bot label added by backport bot team/container-platform team/fleet labels May 27, 2026
@dd-octo-sts
dd-octo-sts Bot requested a review from a team as a code owner May 27, 2026 18:46
@dd-octo-sts dd-octo-sts Bot added this to the v1.27.0 milestone May 27, 2026
@datadog-official

datadog-official Bot commented May 27, 2026 •
edited
Loading

Copy link
Copy Markdown

Code Coverage

🎯 Code Coverage (details)
• Patch Coverage: 82.95%
• Overall Coverage: 41.91% (+0.15%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 3bbba75 | Docs | Datadog PR Page | Give us feedback!

@codecov-commenter

codecov-commenter commented May 27, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 85.00000% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 41.61%. Comparing base (68f6e13) to head (3bbba75).

Files with missing lines Patch % Lines
pkg/fleet/events.go 82.22% 7 Missing and 1 partial ⚠️
pkg/fleet/daemon_worker.go 64.70% 6 Missing ⚠️
pkg/fleet/daemon.go 88.88% 1 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##            v1.27    #3051      +/-   ##
==========================================
+ Coverage   41.44%   41.61%   +0.16%     
==========================================
  Files         334      336       +2     
  Lines       28637    28737     +100     
==========================================
+ Hits        11869    11958      +89     
- Misses      15973    15984      +11     
  Partials      795      795
Flag Coverage Δ
unittests 41.61% <85.00%> (+0.16%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
...controller/datadogagent/controller_reconcile_v2.go 52.52% <100.00%> (+0.48%) ⬆️
...ernal/controller/datadogagent/experiment_events.go 100.00% <100.00%> (ø)
pkg/fleet/daemon.go 72.11% <88.88%> (+2.76%) ⬆️
pkg/fleet/daemon_worker.go 76.92% <64.70%> (-1.26%) ⬇️
pkg/fleet/events.go 82.22% <82.22%> (ø)

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 68f6e13...3bbba75. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@levan-m
levan-m merged commit d67ed03 into v1.27 May 27, 2026
64 checks passed
@levan-m
levan-m deleted the backport-3037-to-v1.27 branch May 27, 2026 20:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@levan-m levan-m levan-m approved these changes

Assignees

No one assigned

Labels

backport label added by backport action bot label added by backport bot enhancement New feature or request team/container-platform team/fleet

Projects

None yet

Milestone

v1.27.0

Development

Successfully merging this pull request may close these issues.

2 participants

@codecov-commenter @levan-m