[CONTP-1608] enrich untaint controller metrics with tags by adel121 · Pull Request #3184 · DataDog/datadog-operator

adel121 · 2026-06-23T10:35:14Z

What does this PR do?

The current TaintRemovalsTotal and TaintRemovalLatency metrics are untagged scalars, making them impossible to attribute to specific nodes or distinguish between normal removals and timeout-forced ones. This task promotes both to labeled variants.

This PR:

Promotes TaintRemovalsTotal to CounterVec with node and reason labels (agent_ready or timeout)

Promotes TaintRemovalLatency to HistogramVec with node label

Updates all call sites in the controller
Updates unit tests for new label signatures

Additional Information

To avoid keeping series of metrics in memory for every node even after the node is removed from the cluster (due to autoscaling for example), we added cleanup mechanism to delete metrics series locally for removed nodes.

This ensure scrape payload doesn't grow indefinitely, and memory doesn't keep holding series for removed nodes that accumulate over time.

Describe your test plan

Added unit tests.

Manual testing:

Create kind cluster with several nodes:

kind create cluster --config ./kind.yaml

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:

role: control-plane
role: worker
role: worker

Deploy the datadog operator with the following settings:

- --untaintControllerEnabled=true

Taint the nodes. For example:

kubectl taint node kind-worker2  agent.datadoghq.com/not-ready=presence:NoSchedule

Create datadog-agent.yml. Example:

apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
  namespace: system
spec:
  features:
    autoscaling:
      workload:
        enabled: true
      cluster:
        enabled: true
  global:
    clusterName: adel-csi-operator
    kubelet:
      tlsVerify: false
    csi:
      enabled: true
    credentials:
      apiSecret:
        secretName: datadog-secret
        keyName: api-key
      appSecret:
        secretName: datadog-secret
        keyName: app-key

After some time, check the emitted metrics and make sure they are promoted to tagged metrics:

kubectl -n system exec deploy/datadog-operator-manager -- \
  bash -c 'exec 3<>/dev/tcp/127.0.0.1/8080; printf "GET /metrics HTTP/1.0\r\n\r\n" >&3; cat <&3' \
  | grep untaint
# HELP untaint_taint_removal_errors_total Total number of errors encountered while attempting to remove the agent-not-ready taint from a node
# TYPE untaint_taint_removal_errors_total counter
untaint_taint_removal_errors_total 0
# HELP untaint_taint_removal_latency_seconds Time between agent pod becoming Ready and taint removal from the node, by node
# TYPE untaint_taint_removal_latency_seconds histogram
untaint_taint_removal_latency_seconds_bucket{node="kind-worker",le="0.005"} 0
untaint_taint_removal_latency_seconds_bucket{node="kind-worker",le="0.01"} 0
untaint_taint_removal_latency_seconds_bucket{node="kind-worker",le="0.025"} 0
untaint_taint_removal_latency_seconds_bucket{node="kind-worker",le="0.05"} 0
untaint_taint_removal_latency_seconds_bucket{node="kind-worker",le="0.1"} 0
untaint_taint_removal_latency_seconds_bucket{node="kind-worker",le="0.25"} 0
untaint_taint_removal_latency_seconds_bucket{node="kind-worker",le="0.5"} 1
untaint_taint_removal_latency_seconds_bucket{node="kind-worker",le="1"} 1
untaint_taint_removal_latency_seconds_bucket{node="kind-worker",le="2.5"} 1
untaint_taint_removal_latency_seconds_bucket{node="kind-worker",le="5"} 1
untaint_taint_removal_latency_seconds_bucket{node="kind-worker",le="10"} 1
untaint_taint_removal_latency_seconds_bucket{node="kind-worker",le="+Inf"} 1
untaint_taint_removal_latency_seconds_sum{node="kind-worker"} 0.41908018
untaint_taint_removal_latency_seconds_count{node="kind-worker"} 1
untaint_taint_removal_latency_seconds_bucket{node="kind-worker2",le="0.005"} 0
untaint_taint_removal_latency_seconds_bucket{node="kind-worker2",le="0.01"} 0
untaint_taint_removal_latency_seconds_bucket{node="kind-worker2",le="0.025"} 0
untaint_taint_removal_latency_seconds_bucket{node="kind-worker2",le="0.05"} 0
untaint_taint_removal_latency_seconds_bucket{node="kind-worker2",le="0.1"} 0
untaint_taint_removal_latency_seconds_bucket{node="kind-worker2",le="0.25"} 0
untaint_taint_removal_latency_seconds_bucket{node="kind-worker2",le="0.5"} 0
untaint_taint_removal_latency_seconds_bucket{node="kind-worker2",le="1"} 1
untaint_taint_removal_latency_seconds_bucket{node="kind-worker2",le="2.5"} 1
untaint_taint_removal_latency_seconds_bucket{node="kind-worker2",le="5"} 1
untaint_taint_removal_latency_seconds_bucket{node="kind-worker2",le="10"} 1
untaint_taint_removal_latency_seconds_bucket{node="kind-worker2",le="+Inf"} 1
untaint_taint_removal_latency_seconds_sum{node="kind-worker2"} 0.897452666
untaint_taint_removal_latency_seconds_count{node="kind-worker2"} 1
# HELP untaint_taint_removals_total Total number of taints removed from nodes, by node and reason
# TYPE untaint_taint_removals_total counter
untaint_taint_removals_total{node="kind-worker",reason="agent_ready"} 1
untaint_taint_removals_total{node="kind-worker2",reason="agent_ready"} 1

Checklist

PR has at least one valid label: bug, enhancement, refactoring, documentation, tooling, and/or dependencies
PR has a milestone or the qa/skip-qa label
All commits are signed (see: signing commits)

datadog-datadog-prod-us1-2 · 2026-06-23T10:35:28Z

✨ Fix all issues with BitsAI

🛑 Gate Violations

🎯 1 Code Coverage issue detected

A Patch coverage percentage gate may be blocking this PR.

• Patch coverage: 72.73% (threshold: 80.00%)

ℹ️ Info

🎯 Code Coverage (details)
• Patch Coverage: 72.73%
• Overall Coverage: 44.28% (+0.00%)

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 5beaa14 | Docs | Datadog PR Page | Give us feedback!}

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0baea7ebf9

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

levan-m · 2026-06-23T21:25:31Z

+			// Node is gone (e.g. autoscaler scale-down). Drop its per-node
+			// metric series so node names don't accumulate in the exporter for
+			// the operator's lifetime. No-op if the node never had series.
+			metrics.DeleteNodeSeries(req.Name)


this is nice, wouldn't have thought about it!

enrich untaint controller metrics with tags

0baea7e

github-actions Bot added the team/container-platform label Jun 23, 2026

github-actions Bot added the team/documentation label Jun 23, 2026

adel121 added the enhancement New feature or request label Jun 23, 2026

adel121 added this to the v1.29.0 milestone Jun 23, 2026

adel121 changed the title ~~enrich untaint controller metrics with tags~~ [CONTP-1608] enrich untaint controller metrics with tags Jun 23, 2026

adel121 marked this pull request as ready for review June 23, 2026 12:48

adel121 requested a review from a team June 23, 2026 12:48

adel121 requested a review from a team as a code owner June 23, 2026 12:48

chatgpt-codex-connector Bot reviewed Jun 23, 2026

View reviewed changes

Comment thread internal/controller/metrics/untaint.go Outdated

adel121 added 2 commits June 23, 2026 14:53

go mod tidy

779bed7

delete series when node is deleted

5beaa14

joepeeples approved these changes Jun 23, 2026

View reviewed changes

levan-m reviewed Jun 23, 2026

View reviewed changes

levan-m approved these changes Jun 23, 2026

View reviewed changes

adel121 merged commit 270fde8 into main Jun 24, 2026
37 of 38 checks passed

adel121 deleted the adelhajhassan/enrich-untaint-controller-metrics-with-tags branch June 24, 2026 10:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[CONTP-1608] enrich untaint controller metrics with tags#3184

[CONTP-1608] enrich untaint controller metrics with tags#3184
adel121 merged 3 commits into
mainfrom
adelhajhassan/enrich-untaint-controller-metrics-with-tags

adel121 commented Jun 23, 2026 •

edited

Loading

Uh oh!

datadog-datadog-prod-us1-2 Bot commented Jun 23, 2026 •

edited by datadog-prod-us1-3 Bot

Loading

🎯 1 Code Coverage issue detected

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

levan-m Jun 23, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

adel121 commented Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Additional Information

Describe your test plan

Checklist

Uh oh!

datadog-datadog-prod-us1-2 Bot commented Jun 23, 2026 • edited by datadog-prod-us1-3 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🛑 Gate Violations

🎯 1 Code Coverage issue detected

ℹ️ Info

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

levan-m Jun 23, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

adel121 commented Jun 23, 2026 •

edited

Loading

datadog-datadog-prod-us1-2 Bot commented Jun 23, 2026 •

edited by datadog-prod-us1-3 Bot

Loading