Skip to content

Remove early exit in credential-dependent controllers when credentials are missing#2530

Merged
fanny-jiang merged 3 commits into
mainfrom
fanny/AGENTONB-2857/revert-2387
Jan 28, 2026
Merged

Remove early exit in credential-dependent controllers when credentials are missing#2530
fanny-jiang merged 3 commits into
mainfrom
fanny/AGENTONB-2857/revert-2387

Conversation

@fanny-jiang

Copy link
Copy Markdown
Contributor

What does this PR do?

Motivation

  • Align behavior among the credentials-dependent controllers so that the operator pod does not crash upon initial install and credentials are unavailable.
  • Provide better user experience for first-time datadog chart users who might otherwise see the operator pod in CrashLoopBackoff due to credentials not being immediately available.

Additional Notes

Anything else we should know when reviewing?

Minimum Agent Versions

Are there minimum versions of the Datadog Agent and/or Cluster Agent required?

  • Agent: vX.Y.Z
  • Cluster Agent: vX.Y.Z

Describe your test plan

Reproducing the bug:

  1. First, uninstall any existing helm releases of the datadog chart from your local kind cluster. Then, install the datadog chart v3.163.1 with minimal values:
datadog:
  apiKeyExistingSecret: datadog-secret
  appKeyExistingSecret: datadog-secret
  kubelet:
    tlsVerify: false
  1. Observe that the operator pod immediately crashes with CrashLoopBackOff. In the Operator logs, you'll see
    errors about missing credentials for the datadogSLO and datadogGenericResource controllers:
{"level":"ERROR","ts":"2026-01-27T21:18:23.133Z","logger":"setup","msg":"Couldn't start controller","controller":"DatadogSLO","error":"error obtaining API key and/or app key","stacktrace":"github.com/DataDog/datadog-operator/internal/controller.SetupControllers\n\t/workspace/internal/controller/setup.go:100\nmain.run\n\t/workspace/cmd/main.go:387\nmain.main\n\t/workspace/cmd/main.go:205\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:285"}
{"level":"ERROR","ts":"2026-01-27T21:18:23.134Z","logger":"setup","msg":"Couldn't start controller","controller":"DatadogGenericResource","error":"error obtaining API key and/or app key","stacktrace":"github.com/DataDog/datadog-operator/internal/controller.SetupControllers\n\t/workspace/internal/controller/setup.go:100\nmain.run\n\t/workspace/cmd/main.go:387\nmain.main\n\t/workspace/cmd/main.go:205\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:285"}
  1. Check the operator deployment YAML. It'll be expected that the DD_API_KEY and DD_APP_KEY env vars are not set. Perform a helm upgrade, and check the operator deployment again. The env vars should now be present and the operator pod should stop crashing.

QA'ing the changes:

  1. Build custom operator image based off of this branch.
  2. Uninstall datadog chart and install the latest datadog chart version (3.164.0+) with these operator.* overrides:
operator:
  image:
    repository: <your_dockerhub>
    tag: <custom_tag>
    doNotCheckTag: true

  datadogMonitor:
    enabled: true
  datadogDashboard:
    enabled: true
  datadogGenericResource:
    enabled: true
  datadogSLO:
    enabled: true

  datadogCRDs:
    crds:
      datadogMonitors: true
      datadogSLOs: true
      datadogDashboards: true
      datadogGenericResources: true
  1. The operator pod should become ready and not crash. Check the operator pod logs and there should be errors informing you of required credentials:
{"level":"ERROR","ts":"2026-01-27T21:46:02.805Z","logger":"setup","msg":"Couldn't start controller","controller":"DatadogMonitor","error":"error obtaining API key and/or app key","stacktrace":"github.com/DataDog/datadog-operator/internal/controller.SetupControllers\n\t/workspace/internal/controller/setup.go:100\nmain.run\n\t/workspace/cmd/main.go:387\nmain.main\n\t/workspace/cmd/main.go:205\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:285"}
  1. Perform a helm upgrade. Those errors should not be present in the new operator pod and the operator deployment YAML should have the API/APP key env vars present.
  2. Deploy example manifests from the examples folder and they should work as normal.

Checklist

  • PR has at least one valid label: bug, enhancement, refactoring, documentation, tooling, and/or dependencies
  • PR has a milestone or the qa/skip-qa label
  • All commits are signed (see: signing commits)

@fanny-jiang fanny-jiang added this to the v1.23.0 milestone Jan 27, 2026
@fanny-jiang fanny-jiang added the bug Something isn't working label Jan 27, 2026
@fanny-jiang
fanny-jiang marked this pull request as ready for review January 27, 2026 22:01
@fanny-jiang
fanny-jiang requested a review from a team as a code owner January 27, 2026 22:01
@codecov-commenter

codecov-commenter commented Jan 27, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 37.84%. Comparing base (938bb25) to head (f5ac70d).

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #2530      +/-   ##
==========================================
- Coverage   37.87%   37.84%   -0.03%     
==========================================
  Files         303      303              
  Lines       26003    25990      -13     
==========================================
- Hits         9849     9837      -12     
+ Misses      15411    15410       -1     
  Partials      743      743              
Flag Coverage Δ
unittests 37.84% <ø> (-0.03%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
cmd/main.go 7.33% <ø> (-3.53%) ⬇️

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 938bb25...f5ac70d. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@fanny-jiang
fanny-jiang merged commit 5a04b5d into main Jan 28, 2026
32 checks passed
@fanny-jiang
fanny-jiang deleted the fanny/AGENTONB-2857/revert-2387 branch January 28, 2026 15:08
@dd-octo-sts

dd-octo-sts Bot commented Jan 28, 2026

Copy link
Copy Markdown

The backport to v1.23 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-v1.23 v1.23
# Navigate to the new working tree
cd .worktrees/backport-v1.23
# Create a new branch
git switch --create backport-2530-to-v1.23
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 5a04b5dc214d6d289595b6598bf705d30319c9f0
# Push it to GitHub
git push --set-upstream origin backport-2530-to-v1.23
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-v1.23

Then, create a pull request where the base branch is v1.23 and the compare/head branch is backport-2530-to-v1.23.

@dd-octo-sts

dd-octo-sts Bot commented Jan 28, 2026

Copy link
Copy Markdown

The backport to v1.23 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-v1.23 v1.23
# Navigate to the new working tree
cd .worktrees/backport-v1.23
# Create a new branch
git switch --create backport-2530-to-v1.23
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 5a04b5dc214d6d289595b6598bf705d30319c9f0
# Push it to GitHub
git push --set-upstream origin backport-2530-to-v1.23
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-v1.23

Then, create a pull request where the base branch is v1.23 and the compare/head branch is backport-2530-to-v1.23.

fanny-jiang added a commit that referenced this pull request Jan 28, 2026
…s are missing (#2530)

* Revert "[AGENTONB-1702] Early exit if no credentials despite controllers requiring them (#2387)"

This reverts commit 39bd16a.

* remove fail-fast for datadogMonitor controller
fanny-jiang added a commit that referenced this pull request Jan 29, 2026
…s are missing (#2530) (#2549)

* Revert "[AGENTONB-1702] Early exit if no credentials despite controllers requiring them (#2387)"

This reverts commit 39bd16a.

* remove fail-fast for datadogMonitor controller
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport/v1.23 bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants