clickhouse backup: failed BackupJob attempts leave archives that the sidecar uploads at the next restart #5200

Describe the bug

Each strategy pod attempt starts POST /backup/create_remote with a new timestamped name. clickhouse-backup creates the backup locally and uploads it after, so when the upload fails the local backup stays on the data volume. At start the sidecar resumes unfinished uploads (ResumeOperationsAfterRestart) and uploads those backups. Result is archives in the bucket under <namespace>/<release>/ that no Backup refers to, one per failed attempt (three per failed BackupJob), and local backups are never removed (no BACKUPS_TO_KEEP_LOCAL).

The resumed upload can also hold the API lock. With dead keys it retries about 35 seconds, and the strategy pod got HTTP 423 another operation is currently running on 2 of 3 attempts, which burns the Job backoffLimit: 2 in seconds.

Environment

Cozystack v1.7.0, clickhouse-backup 2.7.4.

To reproduce

1. Start a ClickHouse app with backup.useSystemBucket: true in a namespace where cozy-backups-creds does not exist yet, so the sidecar has no S3 settings (this is #3986).
2. Run a BackupJob. It fails after three attempts and GET /backup/list/local on the sidecar shows three local backups.
3. Restart the ClickHouse pod and run a BackupJob that succeeds.
4. List the bucket prefix. The three archives of the failed attempts are there next to the real one, and GET /backup/list/remote lists all of them.

Actual behaviour

After my test the bucket held 11 archive prefixes for 4 Backup objects, so 7 orphans. One of them was a full 36 KiB copy of the table data from a failed attempt.

Expected behaviour

A failed attempt leaves nothing behind, or the strategy removes the local backup it created before it exits.

The orphans matter for restore: the strategy takes the newest archive in the prefix (see the issue about ClickHouse restore picking the newest archive), so an orphan can be what gets restored.