go.d/k8s_state: remove only the deleted object's charts by stelfrag · Pull Request #24402 · netdata/netdata

stelfrag · 2026-10-22T07:24:39Z

Summary

When k8s_state saw a node, pod, deployment or cronjob deleted, it marked obsolete every chart whose ID merely started with the deleted object's ID. Deleting pod web-1 also removed all charts of the running pods web-10..web-19 (same for worker-1/worker-10 nodes, backup/backup-db cronjobs, etc.). Charts are only added for new objects, so the live object's charts never came back until it was recreated or the agent restarted. Alerts on them (e.g. k8s_state_deployment_condition_available, k8s_state_cronjob_last_execution_failed) silently stopped.

The pod node-label update had the same open prefix: scheduling web-1 rewrote the node label on web-10's charts.

Fix

One helper, isObjectChart, matches only the object's own chart IDs: <kind>_<id>. and, for pods, pod_<id>_container_. Kubernetes names cannot contain _, so this boundary is exact. Used by all four removal paths and the node-label update.

No chart ID, context, dimension, label or alert definition changes. Visible effect: charts and alerts of such live objects keep running, so users may now get notifications they were silently missing.

Not changed: replaceDots still maps web.1 and web-1 to the same chart ID; fixing that would change chart IDs.

Tests

New: deleting x-1 keeps every chart of live x-10 and obsoletes all of x-1's (node, pod incl. container charts, deployment, cronjob); expected sets come from collectors that saw only one object.
New: scheduling pending pod x-1 changes the node label only on its own charts.
All new cases fail on master and pass with the fix; package go test -race, go vet, gofmt clean.

Summary by CodeRabbit

Bug Fixes
- Removing a Kubernetes node, pod, deployment, or CronJob no longer incorrectly marks charts for similarly named resources as obsolete.
- Pod scheduling updates now preserve the node labels of other similarly named pods. This keeps chart data aligned with the correct resource when names share a common prefix.

Chart removal and the pod node-label update matched chart IDs by an open prefix, so deleting pod "x-1" (or node, deployment, cronjob) also obsoleted every chart of a live "x-10", which never came back. Match the object's own IDs only: "<kind>_<id>." and, for pods, "pod_<id>_container_". Chart IDs, contexts and labels are unchanged.

coderabbitai · 2026-10-22T07:24:47Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Source mode: Auto
Repository: Unavailable
Branch: Unavailable
File: No configuration file from this repository applied
Commit: Unavailable
Layers: defaults
Review profile: CHILL
Plan: Advanced
Run ID: 11804476-656e-4544-80cb-5f93d5aa2b65

📥 Commits

Reviewing files that changed from the base of the PR and between 513d4b4 and 3aae3b7.

📒 Files selected for processing: 2

src/go/plugin/go.d/collector/k8s_state/charts.go
src/go/plugin/go.d/collector/k8s_state/collector_test.go

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 6 remain after this review.

📝 Walkthrough

Chart removal and pod node-label updates now match charts by object kind and normalized ID. Regression tests verify that deleting or scheduling an object does not affect charts for another object whose ID shares its prefix.

Changes: Kubernetes chart ID matching

Layer / File(s)	Summary
Object chart matching `src/go/plugin/go.d/collector/k8s_state/charts.go`	Node, pod, deployment, and CronJob chart removal now uses a shared kind-and-ID matching helper. Pod matching includes container charts. Pod node-label updates use object-kind and ID matching.
Prefix-sharing ID regression tests `src/go/plugin/go.d/collector/k8s_state/collector_test.go`	Tests verify that deleting `x-1` leaves charts for `x-10` active, and that scheduling `x-1` updates only its node label. Helpers initialize multiple collectors and read chart labels.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~15 minutes

Change: Bug fix · Severity of issue fixed: Low

Merge Risk: ⚪ Minimal · up to 3aae3

No actionable risk from this change remains; it is ready to merge after normal checks.

Pre-merge checks: 4 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 22.22% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 9 functions across 2 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 4 passed

Check name	Status	Explanation
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title clearly and concisely describes the main change: removing charts only for the deleted object.
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create a new PR

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

cubic-dev-ai

All reported issues were addressed across 2 files

_{You’re at about 91% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact [email protected].}

Architecture diagram

sequenceDiagram
    participant K8s as K8s API Server
    participant Collector as k8s_state Collector
    participant Charts as Chart Store
    participant Discoverers as Discoverers (nodes, pods, deployments, cronjobs)

    Note over K8s,Discoverers: Runtime Object Lifecycle

    loop Discover resources
        Discoverers->>K8s: Watch/informers for kinds
        K8s-->>Discoverers: Events (create/update/delete)
    end

    Note over Collector,Charts: Object deletion handling

    Discoverers->>Collector: Object deleted (node/pod/deployment/cronjob)
    Collector->>Charts: Iterate all charts

    loop For each chart in store
        Collector->>Collector: isObjectChart(chartID, kind, id)
        alt Chart matches exact object ownership
            Collector->>Charts: MarkObsolete + MarkNotCreated
        else Chart belongs to sibling object (e.g., "x-10")
            Collector->>Charts: Leave chart active
        end
    end

    Note over Collector,Charts: Pod node-label update

    Discoverers->>Collector: Pod scheduled (node assignment changed)
    Collector->>Charts: Iterate pod charts
    loop For each chart
        Collector->>Collector: isObjectChart(chartID, "pod", id)
        alt Only charts of the scheduled pod match
            Collector->>Charts: Update node label dimension
            Collector->>Charts: MarkNotCreated
        else Charts of other pods (e.g., "x-10")
            Collector->>Charts: No modification
        end
    end

    Note over Collector,Charts: Chart ID matching logic
    Note over Collector: isObjectChart checks prefix + "." boundary,<br/>or "pod_" + container suffix for pods
    Note over Collector: Ensures "x-1" does not match "x-10" charts

_{Reply with feedback, questions, or to request a fix.

View guided diff | Re-trigger cubic}

sonarqubecloud · 2026-10-22T08:30:20Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

(cherry picked from commit 877d2a7)

github-actions Bot added area/collectors Everything related to data collection collectors/go.d area/go labels Oct 22, 2026

cubic-dev-ai Bot reviewed Oct 22, 2026

View reviewed changes

Comment thread src/go/plugin/go.d/collector/k8s_state/collector_test.go Resolved

stelfrag added 2 commits October 22, 2026 11:08

go.d/k8s_state: note why the live-only expected loop runs second

eff0408

go.d/k8s_state: wait for informer events in the chart ownership tests

3aae3b7

stelfrag marked this pull request as ready for review October 22, 2026 08:47

stelfrag requested a review from ilyam8 as a code owner October 22, 2026 08:47

ilyam8 approved these changes Oct 23, 2026

View reviewed changes

ilyam8 merged commit 877d2a7 into netdata:master Oct 23, 2026
157 checks passed

stelfrag added a commit to stelfrag/netdata that referenced this pull request Oct 23, 2026

go.d/k8s_state: remove only the deleted object's charts (netdata#24402)

3607483

(cherry picked from commit 877d2a7)

stelfrag mentioned this pull request Oct 23, 2026

Patch release 2.12.2 #24334

Merged

stelfrag deleted the fix/k8s-state-chart-prefix-removal branch October 23, 2026 08:08

Ferroin pushed a commit that referenced this pull request Oct 23, 2026

go.d/k8s_state: remove only the deleted object's charts (#24402)

311dcb6

(cherry picked from commit 877d2a7)

github-actions Bot mentioned this pull request Nov 2, 2026

Update Helm release netdata to v3.7.176 claytono/infra#3329

Merged

1 task

This was referenced Nov 2, 2026

fix(go.d/docker): remove only the charts of the removed container #24539

Closed

fix(go.d): remove only the stale peer's/device's charts in ntpd and nvme #24542

Closed

fix(go.d/wireguard): remove only the removed device's charts #24568

Closed

github-actions Bot mentioned this pull request Nov 9, 2026

Update Helm release netdata to v3.7.177 claytono/infra#3420

Merged

1 task

Navigation Menu

Repository navigation

go.d/k8s_state: remove only the deleted object's charts - #24402

go.d/k8s_state: remove only the deleted object's charts#24402
ilyam8 merged 3 commits into
netdata:masterfrom
stelfrag:fix/k8s-state-chart-prefix-removal

stelfrag commented Oct 22, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Oct 22, 2026 •

edited

Loading

Uh oh!

cubic-dev-ai Bot left a comment •

edited

Loading

Uh oh!

Uh oh!

sonarqubecloud Bot commented Oct 22, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

stelfrag commented Oct 22, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Fix

Tests

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sonarqubecloud Bot commented Oct 22, 2026

Quality Gate passed

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

stelfrag commented Oct 22, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Oct 22, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading