[Bug]: Missing prometheus metrics cnpg_collector_last_available_backup_timestamp and cnpg_collector_first_recoverability_point #7812

Activity

1. sbv-csis

   assigned

   gbartolini

   on Jun 12, 2025
2. sbv-csis

   added

   triagePending triagePending triage

   on Jun 12, 2025
3. dosubot commented on Jun 12, 2025

   

   dosubotBot

   on Jun 12, 2025 – with DosuBot

   Last edited by dosubot

   More actions

   This is a known issue when using the barman plugin: the Prometheus metrics cnpg_collector_last_available_backup_timestamp and cnpg_collector_first_recoverability_point are not updated after switching from the deprecated backup method, even if backups and WAL archiving are working. This has been discussed in issue #7658, and the behavior is confirmed in the code and docs.

   The metrics will remain at zero until the first backup to the object store is completed (this is separate from WAL archiving) as described in the official documentation. However, with the plugin, even after successful backups, the metrics may still not update due to the way the plugin integration is currently handled.

   A suggested workaround is to add an empty backup: {} section to your manifest, but this does not resolve the metrics issue—it only circumvents a nil check elsewhere. The root cause appears to be that the plugin backup method is not fully integrated with the metrics update logic yet, as noted in the code and discussion.

   For now, monitoring the repository for updates or fixes is the best course of action.

   To reply, just mention @dosu.

   ---

   Share context across your team and agents. Try Dosu.

   
4. sbv-csis commented on Jun 12, 2025

   

   sbv-csis

   on Jun 12, 2025

   Author

   More actions

   The ObjectStore shows the right values:

   Name:         google-store
   API Version:  barmancloud.cnpg.io/v1
   Kind:         ObjectStore
   ...
   Status:
     Server Recovery Window:
       Clustername:
         First Recoverability Point:   2025-06-11T15:35:02Z
         Last Succussful Backup Time:  2025-06-12T12:48:12Z
   

   I guess its not necessarily up to cnpg to expose these metrics in the age of plugins?
   As far as I can see in the code it would have to use the Backup CRs to generate/update this data with if it should be backported when using plugin for backup.
5. gbartolini

   added

   bug 🐛Something isn't workingSomething isn't working

   enhancement 🪄New feature or requestNew feature or request

   on Jun 19, 2025
6. gbartolini

   assigned

   armru

   and

   leonardoce

   on Jun 19, 2025
7. gbartolini

   added the

   Defect

   issue type on Jun 19, 2025
8. gbartolini

   added this to CloudNativePG operator developmenton Jun 19, 2025
9. gbartolini

   moved this to Backlog in CloudNativePG operator developmenton Jun 19, 2025
10. gbartolini

    added

    documentation 📖Improvements or additions to documentationImprovements or additions to documentation

    and removed

    triagePending triagePending triage

    on Jun 19, 2025
11. gbartolini commented on Jun 19, 2025

    

    gbartolini

    on Jun 19, 2025

    Contributor

    More actions

    This is at least a documentation bug. Our direction is for CloudNativePG to delegate backup responsibilities to plugins. As you noticed, the object store status contains the relevant information. This is also because there could be several ways to perform backups (although there is only one WAL archiver).

    Regarding metrics, we are missing a way for the instance manager (currently exporting the metrics with the legacy code) to get this information from the actual plugin(s).

    I need help from other maintainers, though - @cloudnative-pg/maintainers please.

    👍React with 👍1Reacted by joaocosmej
12. daanvinken commented on Jul 16, 2025

    

    daanvinken

    on Jul 16, 2025

    Contributor

    More actions

    Just a small note, as an intermediate solution, one can leverage kube-state-metrics to create metrics from the field of any (custom) resources:
    CustomResourceStateMetrics
13. 5 remaining items

    Load more
14. gbartolini

    modified the milestones: 1.26.1, 1.27.0 on Jul 24, 2025
15. gbartolini commented on Jul 24, 2025

    

    gbartolini

    on Jul 24, 2025

    Contributor

    More actions

    I am moving this ticket to 1.27.0 as we only fixed part of the problem with #8033 and #8091. We now need to complete this work with changes in Barman Cloud.
16. usernamedt commented on Jul 28, 2025

    

    usernamedt

    on Jul 28, 2025

    Contributor

    More actions

    For anyone interested, here is a quick and dirty workaround for 1.26.0 operator version to update last backup time:

    From d853dc929de587f05bc96da3434ced7432c8a801 Mon Sep 17 00:00:00 2001
    From: Daniil Zakhlystov <[email protected]>
    Date: Mon, 28 Jul 2025 12:55:35 +0200
    Subject: [PATCH 1/1] Add workaround to report last successfull
     backup and first recoverability point
    
    ---
     internal/controller/backup_controller.go | 73 ++++++++++++++++++++++++
     1 file changed, 73 insertions(+)
    
    diff --git a/internal/controller/backup_controller.go b/internal/controller/backup_controller.go
    index d22809492..67054f3ee 100644
    --- a/internal/controller/backup_controller.go
    +++ b/internal/controller/backup_controller.go
    @@ -275,6 +275,14 @@ func (r *BackupReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctr
     			tryFlagBackupAsFailed(ctx, r.Client, &backup, fmt.Errorf("encountered an error while taking the backup: %w", err))
     			return ctrl.Result{}, nil
     		}
    +
    +		if backup.Spec.Method == apiv1.BackupMethodPlugin {
    +			// Reconcile existing plugin backups info
    +			// TODO: Remove this hinge after https://github.com/cloudnative-pg/cloudnative-pg/issues/7812
    +			if err := updateClusterWithPluginsBackupTimes(ctx, r.Client, cluster.Namespace, cluster.Name); err != nil {
    +				contextLogger.Error(err, "could not update cluster's backups metadata")
    +			}
    +		}
     	case apiv1.BackupMethodVolumeSnapshot:
     		if cluster.Spec.Backup == nil || cluster.Spec.Backup.VolumeSnapshot == nil {
     			tryFlagBackupAsFailed(ctx, r.Client, &backup,
    @@ -577,6 +585,71 @@ func updateClusterWithSnapshotsBackupTimes(
     	return nil
     }
    
    +// updateClusterWithPluginsBackupTimes updates a cluster's FirstRecoverabilityPoint
    +// and LastSuccessfulBackup based on the available plugin backup CRs
    +func updateClusterWithPluginsBackupTimes(
    +	ctx context.Context,
    +	cli client.Client,
    +	namespace string,
    +	name string,
    +) error {
    +	wrapErr := func(msg string, err error) error {
    +		return fmt.Errorf("in updateFirstRecoverabilityPont, %s: %w", msg, err)
    +	}
    +
    +	// refresh the cluster, as this function will get called after the backup
    +	// has finished, potentially a long time
    +	var cluster apiv1.Cluster
    +	if err := cli.Get(ctx, client.ObjectKey{
    +		Namespace: namespace,
    +		Name:      name,
    +	}, &cluster); err != nil {
    +		return wrapErr("could not refresh cluster", err)
    +	}
    +
    +	var clusterBackups apiv1.BackupList
    +	if err := cli.List(
    +		ctx,
    +		&clusterBackups,
    +		client.InNamespace(namespace),
    +		client.MatchingFields{
    +			clusterName: cluster.Name,
    +			backupPhase: apiv1.BackupPhaseCompleted,
    +		},
    +	); err != nil {
    +		return wrapErr("could not get backup list", err)
    +	}
    +
    +	var oldestBackupTime *time.Time
    +	var newestBackupTime *time.Time
    +
    +	for _, backup := range clusterBackups.Items {
    +		if backup.GetStatus().StoppedAt == nil {
    +			continue
    +		}
    +
    +		stoppedAt := backup.GetStatus().StoppedAt.Time
    +
    +		if oldestBackupTime == nil || oldestBackupTime.After(stoppedAt) {
    +			oldestBackupTime = &stoppedAt
    +		}
    +		if newestBackupTime == nil || newestBackupTime.Before(stoppedAt) {
    +			newestBackupTime = &stoppedAt
    +		}
    +	}
    +
    +	origCluster := cluster.DeepCopy()
    +	cluster.UpdateBackupTimes(apiv1.BackupMethodPlugin, oldestBackupTime, newestBackupTime)
    +
    +	if !reflect.DeepEqual(origCluster.Status, cluster.Status) {
    +		err := cli.Status().Patch(ctx, &cluster, client.MergeFrom(origCluster))
    +		if err != nil {
    +			return wrapErr("could not patch cluster status", err)
    +		}
    +	}
    +	return nil
    +}
    +
     // getBackupTargetPod returns the pod that should run the backup according to the current
     // cluster's target policy
     func (r *BackupReconciler) getBackupTargetPod(ctx context.Context,
    --
    2.46.1
    

    last-backup-workaround.patch
17. shiavm006 commented on Jul 29, 2025

    

    shiavm006

    on Jul 29, 2025

    More actions

    @usernamedt

    but Is this safe to apply on 1.26.1 in production, or should we wait for the official 1.27.0 fix?
18. oujonny commented on Aug 7, 2025

    

    oujonny

    on Aug 7, 2025

    More actions

    hey @gbartolini
    I'm a bit confused. After the release of version 1.27.0, is the issue directly fixed, or is there sill some work needed on the barman-cloud-plugin to address it?
19. gbartolini commented on Aug 8, 2025

    

    gbartolini

    on Aug 8, 2025

    Contributor

    More actions

    This will be fixed with version 1.27.0 of the operator and the next release of Barman Cloud plugin.

    See cloudnative-pg/plugin-barman-cloud#459
20. oujonny commented on Aug 20, 2025

    

    oujonny

    on Aug 20, 2025

    More actions

    I upgraded both cnpg and barman cloud plugin. Still I can't see the metrics. Is there a flag, config which needs to get set in order to profit from the feature?
21. andruccho commented on Aug 31, 2025

    

    andruccho

    on Aug 31, 2025

    More actions

    I upgraded both cnpg and barman cloud plugin. Still I can't see the metrics. Is there a flag, config which needs to get set in order to profit from the feature?

    Try new metrics:
    barman_cloud_cloudnative_pg_io_first_recoverability_point
    barman_cloud_cloudnative_pg_io_last_available_backup_timestamp

    Works for me with CNPG 0.26.0 and barman cloud plugin 0.6.0

    👍React with 👍1Reacted by Brad Ash
22. data-dude commented on Sep 29, 2025

    

    data-dude

    on Sep 29, 2025

    Last edited by data-dude

    More actions

    Do these new barman_cloud metrics come from scraping barman? Like oujonny, I don't see the metrics
23. gbartolini commented on Oct 14, 2025

    

    gbartolini

    on Oct 14, 2025

    Contributor

    More actions

    I have deployed CloudNativePG 1.27.0 and the Barman Cloud plugin 0.7.0 using the cnpg-playground and confirmed that the metrics are present. I have been following the instructions in the monitoring section:

    POD_IP=$(kubectl get pod pg-eu-1 --template '{{.status.podIP}}')
    kubectl exec -ti curl -- curl -s ${POD_IP}:9187/metrics | grep barman
    

    Here is the output:

    # HELP barman_cloud_cloudnative_pg_io_first_recoverability_point The first point of recoverability for the cluster as a unix timestamp
    # TYPE barman_cloud_cloudnative_pg_io_first_recoverability_point gauge
    barman_cloud_cloudnative_pg_io_first_recoverability_point 1.759219063e+09
    # HELP barman_cloud_cloudnative_pg_io_last_available_backup_timestamp The last available backup as a unix timestamp
    # TYPE barman_cloud_cloudnative_pg_io_last_available_backup_timestamp gauge
    barman_cloud_cloudnative_pg_io_last_available_backup_timestamp 1.76037037e+09
    # HELP barman_cloud_cloudnative_pg_io_last_failed_backup_timestamp The last failed backup as a unix timestamp
    # TYPE barman_cloud_cloudnative_pg_io_last_failed_backup_timestamp gauge
    barman_cloud_cloudnative_pg_io_last_failed_backup_timestamp 0

    I am closing this ticket.
24. gbartolini

    closed this as completedon Oct 14, 2025
25. github-project-automation

    moved this from Waiting for First Review to Done in CloudNativePG operator developmenton Oct 14, 2025
26. ringerc commented on Oct 22, 2025

    

    ringerc

    on Oct 22, 2025

    Contributor

    More actions

    So the metrics are renamed, but the documentation doesn't mention the new ones?

    And the old ones are still exported, mentioned as deprecated, but may carry stale and incorrect values?

    👀React with 👀1Reacted by Felix Scheinost
27. Yornik

    added a commit that references this issue on May 23, 2026

    feat(monitoring): use upstream-maintained CNPG dashboard, drop workar…

    ...

    1095d68