Skip to content

monitoring (tenant module): storage PVCs and TLS secrets orphaned after the module is disabled #3091

Closed

Description

@scooby87

Describe the bug

After enabling and then disabling the monitoring tenant module, the VictoriaMetrics/VictoriaLogs storage PVCs and the alerta/grafana TLS secrets are left orphaned in the tenant namespace.

Environment

  • Cozystack: dev9 (config version 45, charts @ fe13212a10a6)
  • Provider: on-prem (bare-metal)

To Reproduce

  1. Create a tenant with monitoring: true; wait until Ready.
  2. Set monitoring: false; wait ~120s (the monitoring/monitoring-system HelmReleases are removed cleanly).
  3. kubectl -n <tenant-ns> get pvc,secret — storage PVCs and TLS secrets remain.

Expected behaviour
No orphaned PVCs/secrets remain after the monitoring module is disabled.

Actual behaviour

pvc/vlstorage-db-vlstorage-generic-0
pvc/vlstorage-db-vlstorage-generic-1
pvc/vmselect-cachedir-vmselect-longterm-0
pvc/vmselect-cachedir-vmselect-longterm-1
pvc/vmselect-cachedir-vmselect-shortterm-0
pvc/vmselect-cachedir-vmselect-shortterm-1
pvc/vmstorage-db-vmstorage-longterm-0
pvc/vmstorage-db-vmstorage-longterm-1
pvc/vmstorage-db-vmstorage-shortterm-0
pvc/vmstorage-db-vmstorage-shortterm-1
secret/alerta-tls
secret/grafana-ingress-tls

Additional context

  • Root cause: The monitoring/monitoring-system HelmReleases delete cleanly, but the VictoriaMetrics/VictoriaLogs StatefulSet volumeClaimTemplates (vmstorage/vmselect/vlstorage) are never garbage-collected on StatefulSet deletion (no persistentVolumeClaimRetentionPolicy/cleanup). The alerta-tls / grafana-ingress-tls cert-manager secrets are not owner-referenced and orphan.
  • Proposed fix: Add a release-scoped post-delete PVC cleanup hook for the monitoring storage PVCs, plus cleanup (or ownerReferences) for the orphaned TLS secrets.

Found during a create→delete leak sweep of packages/extra tenant modules on Cozystack dev9 (toggle module on → off; the tenant namespace persists). Companion to the apps sweep (#3056–#3062).

Activity

  1. added
    kind/bugCategorizes issue or PR as related to a bug
    priority/important-soonMust be staffed and worked on either currently, or very soon, ideally in time for the next release
    area/monitoringIssues or PRs related to the monitoring stack (vlogs, vmstack, grafana, workloadmonitor)
    on Jun 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/monitoringIssues or PRs related to the monitoring stack (vlogs, vmstack, grafana, workloadmonitor)kind/bugCategorizes issue or PR as related to a bugpriority/important-soonMust be staffed and worked on either currently, or very soon, ideally in time for the next releasetriage/acceptedIndicates an issue is ready to be actively worked on

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions