Monitoring failed volume snapshot backups #9161

Hi,

thanks for the reply. You mentioned that "the deprecated metrics—cnpg_collector_last_failed_backup_timestamp and cnpg_collector_last_available_backup_timestamp—still work for volume snapshot backups if you are not using plugin-based backups". But we are using both volume snapshot backups (using kubernetes volume snapshots, see https://cloudnative-pg.io/documentation/current/appendixes/backup_volumesnapshot/) and plugin based backup using plugin-barman-cloud. Can we still monitor failed volume snapshots in this case using the deprecated metrics?

No, the deprecated metrics (cnpg_collector_last_failed_backup_timestamp and cnpg_collector_last_available_backup_timestamp) do not reliably monitor failed volume snapshot backups when both volume snapshot and plugin-based backups are used together. In this mixed setup, these metrics often remain stale or do not update for volume snapshot failures reference reference.

Currently, there are no new Prometheus metrics for volume snapshot backup failures in this scenario. The recommended approach is to monitor the status of Backup and ScheduledBackup resources directly in Kubernetes. For alerting or metrics, you can use kube-state-metrics to expose custom metrics based on these resource statuses reference.

Let me know if you want details or examples for setting up kube-state-metrics for this purpose.

To reply, just mention @dosu.

Docs are dead. Just use Dosu.

Hello,

please tell me some details and examples for setting up kube-state-metrics for this purpose.

To monitor failed volume snapshot backups in CloudNativePG using kube-state-metrics, you need to expose the status of Backup resources as Prometheus metrics, since the built-in/deprecated metrics do not reliably reflect failures in mixed backup setups.

Here's how you can approach this:

1. Expose Backup CRDs to kube-state-metrics:
   By default, kube-state-metrics does not generate metrics for custom resources like CloudNativePG's Backup or ScheduledBackup. To get metrics, you can use the custom resource support in kube-state-metrics v2.10+ (or use a community fork or custom exporter if your version does not support this).

2. Configure kube-state-metrics for Backups:
   Create a CustomResourceState configuration for the Backup resource. Focus on the .status.phase, .status.error, and .spec.method fields. For example, a failed backup will have .status.phase: failed and .spec.method: volumeSnapshot.

   Example (YAML snippet for kube-state-metrics v2.10+):

   apiVersion: monitoring.coreos.com/v1alpha1
   kind: CustomResourceState
   metadata:
     name: cnpg-backup
   spec:
     groupVersionKind:
       group: postgresql.cnpg.io
       version: v1
       kind: Backup
     metrics:
       - name: cnpg_backup_status_phase
         help: "Phase of the CloudNativePG Backup"
         each:
           type: Gauge
           gauge:
             valueFrom: 'status.phase == "failed" ? 1 : 0'
         labelsFromPath:
           name: metadata.name
           namespace: metadata.namespace
           method: spec.method
           error: status.error

   This will create a metric cnpg_backup_status_phase with value 1 for failed backups.

3. Prometheus Alert Example:
   Once the metric is available, you can create a Prometheus alert rule like:

   - alert: CNPGBackupFailed
     expr: cnpg_backup_status_phase{method="volumeSnapshot"} == 1
     for: 5m
     labels:
       severity: critical
     annotations:
       summary: "CloudNativePG Backup failed (volumeSnapshot)"
       description: "Backup {{ $labels.name }} in namespace {{ $labels.namespace }} failed: {{ $labels.error }}"

4. Alternative: Direct API Scraping
   If you can't use kube-state-metrics custom resource support, you can use the kube-prometheus-stack to scrape the Kubernetes API directly with a custom exporter or use a tool like kube-custom-metrics to expose these fields.

Key fields to monitor:

• .status.phase (should be failed for failed backups)
• .status.error (contains the error message)
• .spec.method (should be volumeSnapshot for your use case)

ScheduledBackup resources do not directly indicate failure; instead, monitor the Backup resources they create.

There is no official CloudNativePG documentation or example for this setup, but this approach follows standard Kubernetes monitoring practices for custom resources. For more on the Backup CRD fields, see the CRD definition and community discussions.

Let me know if you want a more detailed example or help with a specific monitoring stack!

To reply, just mention @dosu.

Share context across your team and agents. Try Dosu.