Backport c85321f from #3035.

What does this PR do?

Addressing several bugs/issues:

• When changing same config back and forth to rollout, two controller revisions basically get version bump reusing same name/hash. This leads to accumulation of both marker annotations if one config update is rolled back.

      operator.datadoghq.com/experiment-promoted: "true"
      operator.datadoghq.com/experiment-rollback: "true"

• Again when flipping same config, revisions of two controllerrevs increase while their creationTimestamp which is used as proxy for experiment start time, doesn't change. Once these revisions are old enough controller will timeout and rollback.
• If experiment times out, subsequent experiments will fail unless different config change to force new controller rev creation.
• Operator doesn't send task abort or timeout errors to FA (regardless how FA processes those, which now are ignored).

Easiest to review Commit by commit

b6ef9e5 Decouples experiment timing from ControllerRevision metadata by adding explicit StartedAt field to ExperimentStatus. rev.CreationTimestamp might be stale if new spec has is same as earlier experiment. This could lead to immediate timeout if spec matched pre-existing controller rev spec.

d202cf7 rehydrate installer state from DatadogAgent on daemon startup to report state with correct task ID instead of empty one forcing FA to send start requests.

6cf24ad report TaskState_ERROR for the original start task on timeout.

8509c37 replaces two annotations (*experiment-promoted, *experiment-rollback) on controller rev with a single annotation to avoid accidentally accumulating two mutually exclusive annotations and to simplify logic. Bug reproducible by flipping same config back and forth and intentionally failing experiment to force rollback.

5aa1b51 similar to 6cf24ad, report abort error for the experiment task when DDA is changed during experiment.

Motivation

What inspired you to submit this pull request?

Additional Notes

Anything else we should know when reviewing?

Minimum Agent Versions

Are there minimum versions of the Datadog Agent and/or Cluster Agent required?

• Agent: vX.Y.Z
• Cluster Agent: vX.Y.Z

Describe your test plan

Below args needs to be set on Operator (debug level logging recommended)

        - --createControllerRevisions=true
        - --remoteUpdatesEnabled=true
        - --remoteConfigEnabled=true

and env vars to configure cluster name and endpoint

        - name: DD_CLUSTER_NAME
          value: my-test-cluster
        - name: DD_SITE
          value: datad0g.com          # endpoint for internal testing and api/app key in the same org

Happy path

1. Apply DDA
2. Go to FA -> find cluster -> Configuration -> Edit -> Deploy
3. Switch to Deployments view in FA see it succeeds.
4. Check the change was applied in cluster.

Failures
timeout
2. Watch dda yaml | yq .status.experiment, once experiment is running scale Operator to 0 replicas.
3. Wait 15min
4. Scale back to 1
5. Operator marks experiment as timed_out and is rolled back. FA retries same experiment eventually failing deployment.
6. try happy path

abort
2. Watch dda yaml | yq .status.experiment, once experiment is running scale Operator to 0 replicas.
3. Apply updated DDA in cluster
4. Operator marks experiment as aborted, updated DDA should be reconciled in cluster. FA retries same experiment eventually failing deployment.
5. Try happy path.

Checklist

• PR has at least one valid label: bug, enhancement, refactoring, documentation, tooling, and/or dependencies
• PR has a milestone or the qa/skip-qa label
• All commits are signed (see: signing commits)