New observability metrics are not available

According to the docs, I'm expecting to see 3 new backup metrics, instead of the old ones. However despite the fact that backup through plugin is configured and working, the metrics are not popping up. And the old metrics are no longer working of course. Now I'm a bit lost, is there a problem with my config?

Plugin is installed via Helm:
https://github.com/maxim-mityutko/home-infra/blob/efc76d0398d9f46d65d8e2a72143bd5d4b0e742c/kubernetes/cluster/default/cloudnative-pg/kustomization.yaml#L40-L56

Plugin Configuration in CNPG Cluster

  plugins:
    - name: barman-cloud.cloudnative-pg.io
      enabled: true
      isWALArchiver: false
      parameters:
        serverName: cnpg-default
        barmanObjectName: cnpg-default-barman

Objectstore

apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
  name: cnpg-default-barman
spec:
  retentionPolicy: 7d
  instanceSidecarConfiguration:
    retentionPolicyIntervalSeconds: 21600 # 6 hours
    resources:
      requests:
        memory: 100Mi
        cpu: 100m
      limits:
        memory: 500Mi
        cpu: 2
  configuration:
    destinationPath: s3://cnpg-backup/
    endpointURL: <url>
    s3Credentials:
      accessKeyId:
        name: cnpg-default-secret
        key: username
      secretAccessKey:
        name: cnpg-default-secret
        key: password
    data:
      compression: bzip2
    wal:
      compression: bzip2

Backup

apiVersion: postgresql.cnpg.io/v1
kind: ScheduledBackup
metadata:
  name: cnpg-default-backup
spec:
  immediate: true
  schedule: "0 0 2 * * *"
  backupOwnerReference: self
  cluster:
    name: cnpg-default
  method: plugin
  pluginConfiguration:
    name: barman-cloud.cloudnative-pg.io

Environment

plugin-barman-cloud:v0.9.0
postgresql:17.4
cloudnative-pg:1.28.0

@maxim-mityutko

Hi @maxim-mityutko! I'm Dosu and I’m helping the plugin-barman-cloud team.

The new backup metrics with the barman_cloud_cloudnative_pg_io_ prefix are implemented and should be exposed by the barman-cloud plugin, but there are a few common reasons they might not appear even if backups are working:

The metrics are exposed by the controller-manager pod (the plugin operator) via the /metrics endpoint (usually port 8443) in the system or cnpg-system namespace, not by the PostgreSQL pods or sidecars. Make sure your Prometheus is scraping the controller-manager-metrics-service in the correct namespace and on the right port and path /metrics.
If the ObjectStore status does not contain the expected serverName key (matching your plugin config), the metrics will return zero values. Double-check that the serverName in your plugin config matches the key in the ObjectStore's status.ServerRecoveryWindow field metrics.go.
After deploying or upgrading the plugin, you must restart the CloudNativePG operator (controller-manager) so it can detect and register the plugin. If you haven't done this, metrics may not appear even if everything else looks correct issue #655.
For S3-compatible storage, you may need to set AWS_REQUEST_CHECKSUM_CALCULATION and AWS_RESPONSE_CHECKSUM_VALIDATION to when_required in your ObjectStore's instanceSidecarConfiguration.env to avoid silent failures issue #655.
If all of the above are correct and you still don't see the metrics, check the logs for the controller-manager and plugin sidecar for errors related to metrics, registration, or ObjectStore access.

The metrics you should see are:

barman_cloud_cloudnative_pg_io_last_failed_backup_timestamp
barman_cloud_cloudnative_pg_io_last_available_backup_timestamp
barman_cloud_cloudnative_pg_io_first_recoverability_point

These replace the old cnpg_collector_* metrics, which are deprecated and no longer updated after migration to the plugin-based backup architecture observability.md.

If you check these areas and still have issues, sharing the output of kubectl get pods, kubectl get objectstore, and kubectl get service in the relevant namespace, along with relevant logs, will help pinpoint the problem.

To reply, just mention @dosu.

Share context across your team and agents. Try Dosu.

I've got the same issue. The metrics are exposed neither by the cloudnative-pg operator pod nor by the individual cluster pods.

@dosu

@dosu Are you following the installation instructions on the Barman CNPG-I page? I have a feeling that something from the default kustomization.yaml is missing from the release manifest.yaml, but you're using some other instructions to install the plugin.

From looking at the release manifest.yaml I see no way that controller-manager-metrics-service will be created using the default installation instructions. In the kustomization.yaml I see mention of the metrics parts, but that's not in the release manifest.yaml.

As a data point, I'm using theplugin-barman-cloud Helm chart v0.3.1.

@kruton

Upgraded to the latest plugin version v0.10.0 via the Helm chart v0.4.0.
Restarted both CNPG operator pods and even the database pod for good measure. Still no metrics.

@kruton I've manually exposed the metrics endpoint that you are referring to in kustomization.yaml, and they are not related to the actual backups, but rather to the Go processes, garbage collection, etc.

@kruton

@kruton I've manually exposed the metrics endpoint that you are referring to in kustomization.yaml, and they are not related to the actual backups, but rather to the Go processes, garbage collection, etc.

Yes, I think there is a bug here. I don't know what dosu was referring to. Dosu is a bot so it might just be an LLM hallucination.

I looked at this for a while, but my thought without knowing this code base is there is some bug in the detection of plugins for monitoring.

@kruton

Hey,

Since the enablePodMonitor is now deprecrated, are you sure that the resources is actually created? Previously the PodMonitor was created automatically but now it is no longer the case.

I also lost some hours on this issue before I understood correctly the documentation.

The metrics @kruton and @maxim-mityutko are talking about (Go processes and garbage collection) are the ones exposed directly by the barman-cloud plugin deployment. But the three barman_cloud_cloudnative* metrics are actually retrieved by the postgres Pods and exposeed through the Postgres Pods.

Can you check if you have the PodMonitor created and working correctly?

I'm using VictoriaMetrics, so it's VMPodScrape for me, but the monitoring is working and it had been working prior to Barman plugin migration as well. All the other metrics from the Postgres pods are coming in, with the exception of the 3 new metrics related specifically to Barman plugin.

Hi, for me is exactly the same barman_cloud_cloudnative_pg_io.* metrics are not shown. This is how i tested it to avoid PodMonitor issues.

Got the Primary DB Pod and created a port-forward.
kubectl port-forward $(kubectl get po -o name -l cnpg.io/cluster=test-db,cnpg.io/instanceRole=primary) 9187
look for backup_timestamp: or barman_cloud_cloudnative_pg_io

$ curl -s http://localhost:9187/metrics | grep backup_timestamp
# HELP cnpg_collector_last_available_backup_timestamp The last available backup as a unix timestamp (Deprecated)
# TYPE cnpg_collector_last_available_backup_timestamp gauge
cnpg_collector_last_available_backup_timestamp 0
# HELP cnpg_collector_last_failed_backup_timestamp The last failed backup as a unix timestamp (Deprecated)
# TYPE cnpg_collector_last_failed_backup_timestamp gauge
cnpg_collector_last_failed_backup_timestamp 1.767964969e+09

$ curl -s http://localhost:9187/metrics | grep barman_cloud_cloudnative_pg_io
nothing

Im using already: plugin-barman-cloud:v0.10.0

It seems the plugin is not really exposing the metrics correctly. Additionally at least for me the Documentation is confusing. The Barman Cloud Plugin exposes the following metrics through the native Prometheus exporter of the instance manager for me this means, the metrics are being exposed on the barman-cloud-plugin operator/pod on the cnpg-system. I tried to look there also for metrics and did not found anything at least on port 9090.

Hello, is there any progress?

@maxim-mityutko

I think I found the bug. This plugin lacks TYPE_INSTANCE_SIDECAR_INJECTION in its plugin capabilities which causes it not to be considered enabled. This is needed to have the metrics queried from the sidecar.

Why is it working for others? There is backward compatibility code that masks this problem if isWALArchiver is set to true. As the @maxim-mityutko config shows in the original report, that isWALArchiver: false is in the config. That's the same with my configuration. This may be why @jeremyalbrecht could see the metrics with the new versions of these packages.

Brilliant! Looks like you are on point! I've changed the setting for isWALArchiver = True. The metrics popped up immidiately, no need to even reboot the instance.

That's great. I think there's another bug where isWALArchiver is ignored and plugins are always used to archive WAL files if they have the capability to do it.

So you might as well enable isWALArchiver now since it has no effect to set it to false other than to break metrics.

Interesting, good find! The cnpg.io/skipWalArchiving: enabled annotation on the Cluster is still respected, which is nice.

fix(metrics): announce sidecar injection capability (#776)

Navigation Menu

Repository navigation

New observability metrics are not available #682

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

New observability metrics are not available #682

Description

Activity

dosubot commented on Dec 21, 2025

Kernald commented on Dec 25, 2025

kruton commented on Dec 30, 2025

Kernald commented on Dec 30, 2025

maxim-mityutko commented on Jan 5, 2026

kruton commented on Jan 6, 2026

jeremyalbrecht commented on Jan 20, 2026

maxim-mityutko commented on Jan 20, 2026

dcardozoo commented on Jan 21, 2026

Selim-Oeksuez commented on Mar 5, 2026

kruton commented on Mar 5, 2026

maxim-mityutko commented on Mar 6, 2026

kruton commented on Mar 6, 2026

maxim-mityutko commented on Mar 6, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions