Repository navigation
backups: a BackupJob for an app without backups enabled hangs in Running instead of failing #4599
Description
A backup job for a ClickHouse instance with backups turned off never fails. It stays Running for good (the instance had backup.enabled: false). The strategy pod starts, but its ch-backup-client container sits in CreateContainerConfigError:
bj-chtest-backup-rp44g 0/1 CreateContainerConfigError
ch-backup-client: secret "clickhouse-chtest-backup-api-auth" not found
The chart creates that secret only when backups are on, in packages/apps/clickhouse/templates/backup-api-auth-secret.yaml ({{- if .Values.backup.enabled }}). The BackupJob status gives the user no hint about this. After 10 minutes it was still Running with no message.
Once I set backup.enabled: true and useSystemBucket: true on the same instance, the same BackupJob succeeded in 35s and a restore from it succeeded in 18s. So the backup path works. The only problem is how the job behaves when the app has not been set up for it.
I'd expect the BackupJob to fail fast with a message like "backups are not enabled on ClickHouse/chtest". The check could happen before the strategy pod is created, or the controller could mark the job Failed when its pod cannot start. Other apps whose strategy depends on the app's own backup settings may hang the same way. I only tried ClickHouse.
Activity
lexfrei commentedon Sep 30, 2026
Postgres has the same hang with a different trigger. I created the BackupJob right after turning on
backup.enabled, before the pod rollout that adds the barman-cloud sidecar had finished. The CNPG Backup failed withrequested plugin is not available: barman-cloud.cloudnative-pg.io, and the BackupJob stayedRunningwithCNPGBackupTransientFailurefor 20 minutes. It neither failed nor retried. A new BackupJob created after the rollout succeeded in 18s.