Repository navigation
Closed
Labels
area/monitoringIssues or PRs related to the monitoring stack (vlogs, vmstack, grafana, workloadmonitor)Issues or PRs related to the monitoring stack (vlogs, vmstack, grafana, workloadmonitor)kind/bugCategorizes issue or PR as related to a bugCategorizes issue or PR as related to a bugpriority/important-soonMust be staffed and worked on either currently, or very soon, ideally in time for the next releaseMust be staffed and worked on either currently, or very soon, ideally in time for the next releasetriage/acceptedIndicates an issue is ready to be actively worked onIndicates an issue is ready to be actively worked on
Description
Describe the bug
After enabling and then disabling the monitoring tenant module, the VictoriaMetrics/VictoriaLogs storage PVCs and the alerta/grafana TLS secrets are left orphaned in the tenant namespace.
Environment
- Cozystack: dev9 (config version 45, charts @
fe13212a10a6) - Provider: on-prem (bare-metal)
To Reproduce
- Create a tenant with
monitoring: true; wait until Ready. - Set
monitoring: false; wait ~120s (themonitoring/monitoring-systemHelmReleases are removed cleanly). kubectl -n <tenant-ns> get pvc,secret— storage PVCs and TLS secrets remain.
Expected behaviour
No orphaned PVCs/secrets remain after the monitoring module is disabled.
Actual behaviour
pvc/vlstorage-db-vlstorage-generic-0
pvc/vlstorage-db-vlstorage-generic-1
pvc/vmselect-cachedir-vmselect-longterm-0
pvc/vmselect-cachedir-vmselect-longterm-1
pvc/vmselect-cachedir-vmselect-shortterm-0
pvc/vmselect-cachedir-vmselect-shortterm-1
pvc/vmstorage-db-vmstorage-longterm-0
pvc/vmstorage-db-vmstorage-longterm-1
pvc/vmstorage-db-vmstorage-shortterm-0
pvc/vmstorage-db-vmstorage-shortterm-1
secret/alerta-tls
secret/grafana-ingress-tls
Additional context
- Root cause: The
monitoring/monitoring-systemHelmReleases delete cleanly, but the VictoriaMetrics/VictoriaLogs StatefulSetvolumeClaimTemplates(vmstorage/vmselect/vlstorage) are never garbage-collected on StatefulSet deletion (nopersistentVolumeClaimRetentionPolicy/cleanup). Thealerta-tls/grafana-ingress-tlscert-manager secrets are not owner-referenced and orphan. - Proposed fix: Add a release-scoped
post-deletePVC cleanup hook for the monitoring storage PVCs, plus cleanup (orownerReferences) for the orphaned TLS secrets.
Found during a create→delete leak sweep of packages/extra tenant modules on Cozystack dev9 (toggle module on → off; the tenant namespace persists). Companion to the apps sweep (#3056–#3062).
Reacted by dosubot
Metadata
Metadata
Assignees
Labels
area/monitoringIssues or PRs related to the monitoring stack (vlogs, vmstack, grafana, workloadmonitor)Issues or PRs related to the monitoring stack (vlogs, vmstack, grafana, workloadmonitor)kind/bugCategorizes issue or PR as related to a bugCategorizes issue or PR as related to a bugpriority/important-soonMust be staffed and worked on either currently, or very soon, ideally in time for the next releaseMust be staffed and worked on either currently, or very soon, ideally in time for the next releasetriage/acceptedIndicates an issue is ready to be actively worked onIndicates an issue is ready to be actively worked on
Activity
fix(monitoring): clean up orphaned storage PVCs and TLS secrets on te…
fix(monitoring): clean up orphaned storage PVCs and TLS secrets on te…
fix(etcd): clean up orphaned data PVCs and TLS secrets on tenant modu…
fix(monitoring): stamp release-scoping label on VM/VL storage PVCs so…
fix(monitoring): backfill release label onto pre-existing VM/VL stora…
fix(monitoring): clean up orphaned storage PVCs and TLS secrets on te…
fix(monitoring): stamp release-scoping label on VM/VL storage PVCs so…
fix(monitoring): backfill release label onto pre-existing VM/VL stora…
fix(monitoring): clean up orphaned storage PVCs and TLS secrets on te…
fix(monitoring): stamp release-scoping label on VM/VL storage PVCs so…
fix(monitoring): backfill release label onto pre-existing VM/VL stora…
fix(monitoring): clean up orphaned storage PVCs and TLS secrets on te…
fix(monitoring): stamp release-scoping label on VM/VL storage PVCs so…
fix(monitoring): backfill release label onto pre-existing VM/VL stora…
fix(monitoring)!: clean up orphaned storage PVCs and TLS secrets on t…