Skip to content

[Bug]: Operator deletes PodMonitor that it didn't create due to lack of owner reference check #6109

Closed

Description

@ari-becker

Is there an existing issue already for this bug?

  • I have searched for an existing issue, and could not find anything. I believe this is a new bug.

I have read the troubleshooting guide

  • I have read the troubleshooting guide and I think this is a new bug.

I am running a supported version of CloudNativePG

  • I have read the troubleshooting guide and I think this is a new bug.

Contact Details

No response

Version

1.24.0

What version of Kubernetes are you using?

1.30

What is your Kubernetes environment?

Cloud: Google GKE

How did you install the operator?

YAML manifest

What happened?

  1. [Bug]: Prometheus v2.52+ fails to scrape metrics: Error on ingesting samples with different value but same timestamp #6108 , decide to set monitoring.enablePodMonitor: false. In the same commit, create a PodMonitor to be applied manually, with the same name (i.e. the cluster name) and namespace.
  2. Apply both changes simultaneously
  3. Wonder why I don't have a PodMonitor
  4. Go spelunking in the code:
    client.ObjectKeyFromObject(expectedPodMonitor),
    gets a reference to the PodMonitor based on namespace and name only, not looking at the owner reference. Therefore
    case !manager.IsPodMonitorEnabled() && podMonitor != nil:
    indeed finds the PodMonitor (that it doesn't have an owner reference on) and deletes it.

I worked around the issue by adding a short suffix to the PodMonitor name so that it wouldn't be deleted, but this isn't ideal because it changes the job label that Prometheus attaches to it.

Cluster resource

N/A

Relevant log output

N/A

Code of Conduct

  • I agree to follow this project's Code of Conduct

Activity

  1. Jean-Daniel commented on Feb 27, 2025

    @Jean-Daniel

    I'm also impacted by this issue.

    Note that if you need to create your own PodMonitor but want to keep the same job label, you can override it using relabelings:

    apiVersion: monitoring.coreos.com/v1
    kind: PodMonitor
    metadata:
      name: clustername-suffix
    spec:
      podMetricsEndpoints:
      - relabelings:
        - replacement: clustername
          targetLabel: job
        port: metrics
      selector:
        matchLabels:
          cnpg.io/cluster: clustername
    
  2. github-actions commented on May 18, 2025

    @github-actions
    Contributor

    This issue is stale because it has been open for 60 days with no activity.

  3. sorrison commented on May 18, 2025

    @sorrison

    bump to keep open

  4. SeaRoll commented on Jul 8, 2025

    @SeaRoll

    Have the same problem

  5. gbartolini commented on Jul 21, 2025

    @gbartolini
    Contributor

    As project maintainers, we have decided to deprecate the native integration with PodMonitor resources starting from version 1.28. We will delegate all functionality to the official upstream PodMonitor implementation. For more details, please take a look at issue #8075.

  6. runningman84 commented on Nov 22, 2025

    @runningman84

    I get that you want to deprecate it but right now it is deleting my custom resources even if the operator did not create them in the first place.

  7. veshant commented on Nov 24, 2025

    @veshant

    This is happening for me too

  8. christiaangoossens commented on Dec 21, 2025

    @christiaangoossens

    Issue is still happening, hopefully #9340 gets merged soon.

  9. ardentperf commented on Jan 15, 2026

    @ardentperf
    Contributor

    we should probably update the cnpg docs so they don't have explicit examples showing configs where the podmonitors will get deleted. will try to put together a PR. personally dont think we need the relabeling for doc examples, i didnt think the CNPG official dashboard had any references to it

  10. added a commit that references this issue on Jan 15, 2026
    694b3b9
  11. added 3 commits that reference this issue on Jan 15, 2026
    871c4f9
    266dcfa
    fab73f5
  12. added a commit that references this issue on Jan 20, 2026
    f1f96d9
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

triagePending triage

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions