fix(clickhouse): delete the archive when its Backup is deleted - #5224

Deleting a Backup taken with the Altinity strategy released its
finalizer at once and left the archive in the bucket. On the system
bucket nothing else prunes it, so every ClickHouse backup stayed until
someone called the sidecar by hand.

Deleting such a Backup now runs a one-shot Job that asks the
application's sidecar to delete the archive the Backup records, and
holds the finalizer until the Job completes. An archive that is
already gone counts as deleted. The sidecar refuses a delete while
another operation runs, so the Job waits that out within its deadline;
any other refusal fails the Job, which is retried and reported as an
ArtifactCleanupFailed Warning Event carrying the reason the Job failed,
since the Job and its Pod log go with the retry.

The Job is built by the controller, not rendered from the strategy
template, because the shipped and the example templates run a restore
for every mode other than backup; only the image and how to pull it
come from the template. When the delete cannot run (the namespace is
terminating, the strategy, the application or its API Secret is gone,
the application has left the system bucket, or another Job holds the
cleanup Job name), the Backup is released with an ArtifactNotDeleted
Warning Event naming the archive left behind, as it is when the
application leaves the system bucket while the delete runs. The
backups.cozystack.io/skip-artifact-cleanup annotation releases it
without deleting anything. An application on its own bucket keeps the
archive's lifecycle, as with MongoDB.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <[email protected]>

Two runs of one application that a controller naming archives without
the BackupJob's UID started within the same second share one archive.
When their Jobs complete after the upgrade, both Backups record it
together with the bucket setting, so deleting one of them would delete
the archive the other still points at, leaving it unrestorable.
Backups recorded before the upgrade carry no bucket setting and never
delete their archive.

A Backup whose archive another live Backup of the namespace records is
released without deleting it, with an ArtifactNotDeleted Warning Event
naming that Backup; the last of them deletes the archive. A Backup that
is being deleted as well does not hold the archive back, or two Backups
deleted together would each leave it to the other.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <[email protected]>

…erves

The chart comments and the restore scenario told tenants to delete an
archive with DELETE /backup/<name>/remote, which clickhouse-backup 2.7
does not serve and answers with 404; the route is
POST /backup/delete/remote/<name>. The scenario also said the archive
always outlives its Backup, which no longer holds on the system bucket.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <[email protected]>

…rchive delete

The cleanup script waited out a busy sidecar but ended at once when it
could not reach one, as while the ClickHouse Pod restarts or its node
is gone. Each such Job failed within seconds, the controller replaced
it, and an application that stayed down kept spinning a new Job, two
Pods and a Warning Event every few seconds per deleted Backup.

A connection that cannot be made is now retried within the same bounded
wait as a busy answer, so a sidecar that stays down costs one failed Job
per deadline instead.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <[email protected]>

The strategy only supplies the image of a cleanup Job not created yet,
but it was checked first, so a strategy deleted or emptied while the
Job ran released the Backup with a Warning that the archive was left
behind, while the Job went on to delete it. The strategy is now read
only when a Job has to be created.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <[email protected]>

…not run

A Job standing under a BackupJob's Job name was taken as its own. A
BackupJob re-created under a used name, as a Plan does when the
BackupJob of a schedule slot is deleted within the slot's starting
deadline, met the completed Job of the earlier one before the garbage
collector removed it, was marked Succeeded without uploading anything,
and recorded a Backup for an archive nobody uploaded for it.

A Job controlled by an earlier BackupJob of the same name is now
waited out, with the Ready condition naming it; any other Job holding
the name fails the BackupJob, since nothing will remove it.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <[email protected]>

…d not run

The restore path took any Job standing under a RestoreJob's Job name as
its own, so a RestoreJob re-created under a used name met the completed
Job of the earlier one before the garbage collector removed it and was
marked Succeeded although nothing was restored for it.

The restore path now refuses such a Job the way the backup path does:
the Job of an earlier RestoreJob of the same name is waited out, with
the Ready condition naming it, and any other Job holding the name
fails the RestoreJob.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <[email protected]>

The Backup reconciler attached the cleanup finalizer only on its first
pass over a new Backup. A Backup deleted before that pass had no
finalizer, so the API server removed it at once and its archive stayed
in the bucket with nothing left to delete it.

The driver now creates the Backup with the finalizer already set.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <[email protected]>

An archive name without the BackupJob's UID ends in the run's start
second, and a controller that did not append the UID gave two runs of
one application started in that second the same name. A run still in
flight when the controller is upgraded records that name later, so
checking other Backups for the same archive could not see it, and
deleting one Backup could take the archive another run is about to
record.

Cleanup now releases a Backup whose archive has that name, with an
ArtifactNotDeleted Warning, instead of deleting it. Such a name reaches
this check only from a run that crossed the upgrade, because a Backup
recorded before it carries no bucket setting and is never deleted, so
the archives left are few and named in the Event. The check of other
Backups is dropped: per-run names cannot be shared.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <[email protected]>

The cleanup Pod took only the image and its pull settings from the
strategy, so on a cluster that keeps backup Pods off tainted nodes, or
pins them to nodes that reach the sidecar, it could not be scheduled:
each Job sat Pending until its deadline and the Backup stayed in
deletion.

The cleanup Pod now carries the node selector, tolerations, affinity
and priority class of the strategy's Pod. The service account stays
out, as the cleanup Pod only calls the sidecar's HTTP API.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <[email protected]>

… records

The example strategy minted its own archive name from the clock and
restored the newest archive of the source application. The controller
then found no archive name in the backup Job and recorded none, so a
restore of an older Backup silently restored the latest archive and
deleting a Backup left its archive in the bucket.

The example now uploads under the name the controller chose and
restores the archive the Backup records, as the shipped strategy does;
the clock-made name and the newest archive remain the fallback for a
run the controller gave no name. The backup scenario no longer says the
archive is expired by the upstream tool's retention, which the chart
does not configure.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <[email protected]>

A cleanup Job that failed was deleted and a new one created a few
seconds later, so a delete the sidecar refuses for good, such as one
made with a wrong password, cost a Job, its Pod and a Warning Event
every few seconds for as long as the Backup stayed.

Each failure now doubles the wait before the next cleanup Job, from 30
seconds up to 30 minutes. The count of failures and the time the next
Job may start are kept on the Backup, because deleting the failed Job
is what starts the next attempt and the Backup controller does not
watch Jobs. The finalizer stays: the delete is retried until it
succeeds, or until the skip annotation releases the Backup.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <[email protected]>

A Backup recorded which archive it took but not which bucket it went
to, and the delete goes through the application's current sidecar,
which treats "not found on remote storage" as already deleted. After
the platform's backup storage moved to another bucket, deleting an
older Backup asked the new bucket, was told the archive was not there,
and released the Backup as if the archive had been deleted, while it
stayed in the previous bucket with nothing recording it.

The bucket the projected credentials name is now stamped on the backup
Job when it is created and recorded on the Backup. Before the delete,
and again when the delete Job completes, it is compared with the bucket
the projected credentials name now; when they differ, or the current
one cannot be read, the Backup is released with an ArtifactNotDeleted
Warning naming the bucket the archive is in. A Backup taken before
buckets were recorded is deleted as before.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <[email protected]>

A BackupJob or RestoreJob that waited for the Job of an earlier owner
of its name kept its Ready=False PredecessorJobPending condition after
that Job was gone and its own Job had started, so for the whole run it
still said it was waiting for the garbage collector.

The condition is now dropped when the run's own Job is running, the
way the credentials wait is.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <[email protected]>

…edentials

The sidecar takes up the projected credentials only some time after the
Secret changes, which is why the backup and restore Jobs already wait
that time out. The cleanup Job did not: right after the platform's
bucket moved back to the one a Backup records, the bucket check passed
while the sidecar still served the other bucket, answered "not found"
for the archive, and the delete was counted as done.

The cleanup Job now waits for the credentials to reach the sidecar
before it starts, as the other Jobs do.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <[email protected]>

The cleanup Pod took its image and scheduling settings from the
strategy template as written, while the backup and restore Pods take
them rendered. A strategy that writes one of them as a template, a node
selector naming the release for one, produced a cleanup Job the API
server refused, so the Backup was released and its archive left in the
system bucket.

The cleanup Pod now takes these settings from the template rendered
against the application and the Backup, as the strategy's own Pods are.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <[email protected]>

The time the next cleanup Job may start is read from an annotation on
the Backup, and a time far ahead, from a skewed clock or a hand edit,
held the Backup until then with nothing saying why.

A time further ahead than the longest wait a failure sets is now logged
and ignored, as the credentials stamp is.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <[email protected]>

A deleted Backup whose archive delete waited for the projected
credentials to reach the sidecar stayed for up to two minutes with
nothing saying why; a debug line now says what it waits for.

The test explaining where the cleanup Pod is scheduled had ended up
above the test of rendering, and the reconciler's CredentialsConfig
still named only the Rabbitmq path although the Altinity cleanup reads
the same Secret; both comments now describe their subject.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <[email protected]>