Skip to content

Copy OpenShift etcd metric secret#3131

Merged
tbavelier merged 3 commits into
levan-m/introspection-ddaifrom
tbavelier/openshift-etcd-secret-copy
Jun 15, 2026
Merged

Copy OpenShift etcd metric secret#3131
tbavelier merged 3 commits into
levan-m/introspection-ddaifrom
tbavelier/openshift-etcd-secret-copy

Conversation

@tbavelier

@tbavelier tbavelier commented Jun 12, 2026

Copy link
Copy Markdown
Member

What

PoC for automatically copying the OpenShift etcd metric client Secret when OpenShift control plane monitoring is enabled.

  • Pass an uncached API reader into feature construction from the DatadogAgentInternal reconciler.
  • Read openshift-etcd-operator/etcd-metric-client directly from the API server.
  • Add a managed etcd-metric-client Secret in the DatadogAgent namespace through the existing dependency store.
  • Keep the manual-copy log fallback when the source Secret cannot be read.

Why

OpenShift etcd monitoring mounts etcd-metric-client in the DatadogAgent namespace. Previously the operator only logged an oc get ... | oc apply ... command for users to copy it manually. This secret is necessary otherwise the pod fails to initialize. Now that introspection is enabled by default through target PR, it would "break" users that do not expect/use control-plane monitoring.

The operator ClusterRole already allows core Secret get/list/watch/create/update/patch/delete, so it has the required RBAC to perform this copy.

Alternatives considered

  • Watch openshift-etcd-operator by default: rejected for this PoC because it broadens the controller cache to a sensitive platform namespace, keeps platform Secrets in memory, and ties this behavior to bundle/install packaging.
  • Use the cached client: rejected because default installs do not watch openshift-etcd-operator, producing unknown namespace for the cache.
  • Use a direct API server read: chosen for the source Secret because this is a narrow one-off read outside the watched namespace set. The copied target Secret still uses the normal dependency store for ownership, apply, and cleanup.

Follow-ups before productionizing

This PoC performs the direct source Secret read during each relevant DDAI reconciliation. Better production options include:

  • Read only when the target Secret is missing or differs.
  • Add a small TTL/cache around the source Secret content.
  • Move copying into a dedicated helper/reconciler with status/error reporting.
  • Consider a narrow watch only if it can target the exact source object without broadly caching the namespace.

Validation

  • Tested on my OpenShift cluster

@datadog-datadog-prod-us1

This comment has been minimized.

@tbavelier tbavelier added this to the v1.29.0 milestone Jun 12, 2026
@tbavelier tbavelier added the enhancement New feature or request label Jun 12, 2026
@codecov-commenter

codecov-commenter commented Jun 12, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 55.55556% with 24 lines in your changes missing coverage. Please review.
✅ Project coverage is 43.64%. Comparing base (c5de88a) to head (7c146b0).
⚠️ Report is 1 commits behind head on levan-m/introspection-ddai.

Files with missing lines Patch % Lines
...dogagent/feature/controlplanemonitoring/feature.go 61.70% 13 Missing and 5 partials ⚠️
...rnal/controller/datadogagentinternal/controller.go 0.00% 5 Missing ⚠️
...er/datadogagentinternal/controller_reconcile_v2.go 0.00% 1 Missing ⚠️

❌ Your patch status has failed because the patch coverage (55.55%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

Impacted file tree graph

@@                      Coverage Diff                       @@
##           levan-m/introspection-ddai    #3131      +/-   ##
==============================================================
+ Coverage                       43.60%   43.64%   +0.04%     
==============================================================
  Files                             353      353              
  Lines                           30168    30215      +47     
==============================================================
+ Hits                            13154    13187      +33     
- Misses                          16139    16147       +8     
- Partials                          875      881       +6     
Flag Coverage Δ
unittests 43.64% <55.55%> (+0.04%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
internal/controller/datadogagent/feature/types.go 22.10% <ø> (ø)
internal/controller/setup.go 69.38% <100.00%> (+0.20%) ⬆️
...er/datadogagentinternal/controller_reconcile_v2.go 0.00% <0.00%> (ø)
...rnal/controller/datadogagentinternal/controller.go 0.00% <0.00%> (ø)
...dogagent/feature/controlplanemonitoring/feature.go 24.26% <61.70%> (+11.06%) ⬆️

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c5de88a...7c146b0. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Comment thread internal/controller/datadogagent/feature/controlplanemonitoring/feature.go Outdated
@tbavelier
tbavelier marked this pull request as ready for review June 15, 2026 07:27
@tbavelier
tbavelier requested review from a team as code owners June 15, 2026 07:27

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e0d2cbde9b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread internal/controller/datadogagent/feature/controlplanemonitoring/feature.go Outdated
@tbavelier
tbavelier merged commit 59dbb2c into levan-m/introspection-ddai Jun 15, 2026
14 of 15 checks passed
@tbavelier
tbavelier deleted the tbavelier/openshift-etcd-secret-copy branch June 15, 2026 07:56
levan-m added a commit that referenced this pull request Jun 24, 2026
* Extend introspection and support DDAI code path

* handle edge cases when user unsets annotation

* pr review fixes

* skip adding default provider to ddai

* remove logging

* Copy OpenShift etcd metric secret (#3131)

* copy OpenShift etcd metric secret

* Add 5s timeout

* Keep existing secret if source read fails

* fix podstartup (#3179)

* Avoid agent getting stuck in container creating

* Do not use direct apireader client and instead make f.etcdSecretPresent as part of existing calls (#3180)

---------

Co-authored-by: Timothée Bavelier <[email protected]>

---------

Co-authored-by: Timothée Bavelier <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants