ClickHouse system-bucket backup needs pod restart after first credential projection #3986

Activity

1. myasnikovdaniil

   added

   kind/bugCategorizes issue or PR as related to a bugCategorizes issue or PR as related to a bug

   priority/backlogGeneral backlog priority. Lower than priority/important-longtermGeneral backlog priority. Lower than priority/important-longterm

   area/databaseIssues or PRs related to managed databases (postgres, mariadb, redis, etcd, kafka, clickhouse)Issues or PRs related to managed databases (postgres, mariadb, redis, etcd, kafka, clickhouse)

   triage/acceptedIndicates an issue is ready to be actively worked onIndicates an issue is ready to be actively worked on

   on Aug 28, 2026
2. myasnikovdaniil commented on Nov 4, 2026

   

   myasnikovdaniil

   on Nov 4, 2026

   ContributorAuthor

   More actions

   I reproduced this on v1.7.0 on a clean tenant and measured how it fails, maybe it helps with the acceptance criteria.

   Sidecar of a release with backup.useSystemBucket: true starts with only S3_PATH and a literal, unexpanded S3_ENDPOINT=https://$(S3_ENDPOINT_HOST) in its environment, because every secretKeyRef is optional and the Secret does not exist yet. The first BackupJob creates cozy-backups-creds within a second, before the first strategy pod runs, so the Secret is complete when the sidecar is called, but the sidecar never sees it.

   BackupJob ends Failed after 2m53s: three strategy pods, each calling POST /backup/create_remote and getting can't upload <archive>/metadata.json: S3 PutFileAbsolute Upload: operation error S3: PutObject, get identity: get credentials: ... no EC2 IMDS role found. BackupJob status says only Job has reached the specified backoff limit, so the cause is visible in the pod and sidecar logs only. It is terminal, a second BackupJob four minutes later fails the same way and the first is still Failed 30 minutes later. After kubectl delete pod of the ClickHouse pod a new BackupJob succeeds in 20 seconds.

   RestoreJob on a fresh pod without the Secret fails the same way and is more confusing: GET /backup/list/remote answers HTTP 200 with an empty body when S3 authentication fails, so the strategy pod prints no remote backup found although the archives are in the bucket.

   Rotation has the same shape. With dead keys in the sidecar a BackupJob fails with 403 InvalidAccessKeyId while the projected Secret already holds the right keys, because the projector rewrites it at every BackupJob reconcile and the sidecar keeps what it read at start. A changed region does not fail on SeaweedFS (it accepted a mismatching signing region in my test), so there the divergence is silent.

   Whatever the fix is (roll the pod after the first projection, mount the Secret as files, make the sidecar re-read), I would also make the strategy fail with a message that names the missing sidecar credentials, otherwise a ClickHouse pod started before the Secret and never restarted keeps failing.

   Failed attempts of this also leave archives behind that the sidecar uploads at the next restart, I filed that separately.
3. scooby87

   self-assigned this

   on Nov 4, 2026
4. scooby87 commented on Nov 12, 2026

   

   scooby87

   on Nov 12, 2026

   Contributor

   More actions

   Fixed by #5232.
5. scooby87

   closed this as completedon Nov 12, 2026