Repository navigation
cnpg_collector_last_available_backup_timestamp metric does NOT get updated when using barman-cloud plugin #380
Description
As per #353, I've followed the migration guide and I'm now able to create manual backups using kubectl cnpg;
However, I've just noticed that the metric cnpg_collector_last_available_backup_timestamp does not seem to get updated and thus all backup related fields ((1) Last Base Backup field in the Backups section and (2) Backupsin the Health section) show red in the Grafana dashboard:
The (scheduled and/or manual) backups are created though:
Details
Name: scheduledbackup-harbor-20250602040000
Namespace: harbor
Labels: cnpg.io/cluster=harbor-db
cnpg.io/immediateBackup=false
cnpg.io/scheduled-backup=scheduledbackup-harbor
Annotations: <none>
API Version: postgresql.cnpg.io/v1
Kind: Backup
Metadata:
Creation Timestamp: 2025-06-02T04:00:00Z
Generation: 1
Owner References:
API Version: postgresql.cnpg.io/v1
Controller: true
Kind: ScheduledBackup
Name: scheduledbackup-harbor
UID: 09e764db-5b63-406c-bf4d-bbd22d1a3bc0
Resource Version: 170252222
UID: 80d55ffb-176b-47ee-9cde-8bdfbc973985
Spec:
Cluster:
Name: harbor-db
Method: plugin
Plugin Configuration:
Name: barman-cloud.cloudnative-pg.io
Status:
Backup Id: 20250602T040002
Backup Name: backup-20250602040000
Begin LSN: 0/B302A900
Begin Wal: 0000000B00000000000000B3
End LSN: 0/B5000000
End Wal: 0000000B00000000000000B5
Instance ID:
Container ID: containerd://d4d305adf34c5a019f49e05d0d82c66796f81b3594d69cd829f132408f1c16f6
Pod Name: harbor-db-1
Method: plugin
Online: true
Phase: completed
Plugin Metadata:
Cluster UID: 6cc84a1b-6f55-40e7-a922-15a4799fcc06
Display Name: BarmanCloudInstance
Name: barman-cloud.cloudnative-pg.io
Plugin Name: barman-cloud.cloudnative-pg.io
Timeline: 11
Version: 0.4.0
Started At: 2025-06-02T04:00:02Z
Stopped At: 2025-06-02T04:00:06Z
Events: <none>However, the metric itself does not get updated:
-> this is the 2025-05-26 04:00:04: on said date I've updated all my test clusters to use the barman-cloud plugin
Reacted by Lázár György, Tyler Holinka, LeoKitabdjian, Utwo, nikki, Pieter, Henrik Eidnes, Sascha Gärtner, kuyziss, Omar El Oubari and 3 more
Metadata
Metadata
Assignees
Labels
No labels
Activity
lgyurci commentedon Jun 2, 2025
I'm facing the same issue currently
hwo-wd commentedon Jun 2, 2025
@lgyurci are you also seeing increased memory usage on the primary instance since the migration or is it just me?
Meanwhile I've (1) updated the barman-cloud plugin to
v0.4.1and (2) PG tov17.5, which caused the cluster to be restarted and the memory of the primary instance being "normal" againlgyurci commentedon Jun 2, 2025
I don't really have a reference unfortunately, since I started using CNPG since the plugin is available, and went with it from the start
OOub commentedon Jun 23, 2025
same problem here. I cannot monitor the state of backups anymore
Pionerd commentedon Jun 24, 2025
We can all complain about this, or think about a solution. I’m performing a test with using kube-state-metrics to alert on fields in the Backup and ScheduledBackup resources. Unfortunately my time to work on this in the coming weeks is limited. I’ll update here when I made progress, but maybe someone else is quicker than I am.
gbartolini commentedon Aug 7, 2025
As highlighted in the CloudNativePG 1.26.1 release notes, the project continues its transition towards a plugin-based, backup-agnostic and recovery-agnostic architecture — originally introduced in version 1.26.0 with the Barman Cloud plugin.
As part of this evolution, we have deprecated the following metrics (which will continue to work with the in-core Barman Cloud support):
cnpg_collector_first_recoverability_pointcnpg_collector_last_available_backup_timestampcnpg_collector_last_failed_backup_timestampStarting from CloudNativePG 1.27.0, if you are using the Barman Cloud plugin, metrics related to backup and recovery will instead be provided directly by the plugin and made available to the instance manager (see CNPG#8033).
The first recoverability point and last available backup metrics will still be exposed through the same Prometheus exporter, but with a new naming convention: they will now use the
barman_cloud_cloudnative_pg_io_prefix, replacing the previouscnpg_prefix.Information about the last failed backup is not at the moment possible to obtain.
See #459
feat: add upstream backup and recovery metrics (#459)
gbartolini commentedon Aug 10, 2025
The last failed backup information will be added to the objectstore status and then as a metric, as part of ticket #470
fix(prometheus-stack): use plugin-native metrics for CNPGBackupFailing
feat(cloudnative-pg): add barman-cloud S3 backup dead-man's-switch alert
monitoring: fix broken queries in the CNPG/TimescaleDB dashboard