Skip to content

Commit 09df69e

Browse files
committed
fix(backups): bound Kafka metadata run at the Job layer, not a controller wall clock
The r5 running-Job deadline was enforced in reconcileKafka / reconcileKafkaRestore: at kafkaDefaultBackupDeadline from StartedAt the BackupJob/RestoreJob was marked Failed. But the batch Job carried no activeDeadlineSeconds and markBackupJobFailed only writes status, so the Job kept running, finished, and its final `s3 PUT` uploaded the object — while the terminal phase meant no Backup was ever created and cleanup (keyed on Backup deletion) never ran. Every timed-out run thus orphaned a uniquely-keyed object, and a cluster whose export legitimately exceeds the deadline could never produce a usable backup. The shipped docs also claimed the run "keeps the BackupJob Running ... rather than failing on a wall clock", which the deadline made false. Move the bound onto the Job as activeDeadlineSeconds (setKafkaJobDeadline): Kubernetes fails an unschedulable or wedged Job as DeadlineExceeded and kills its pod before the upload, so a timed-out run lands in the JobFailed branch cleanly and leaves no object behind. Make it operator-tunable with the `backupTimeout` BackupClass parameter (a Go duration), which round-trips through the Backup so a restore is bounded by the same value; the readiness-precondition wait honours it too. Correct the docs. Also requeue on a transient broker-image List error instead of failing the whole run: resolveKafkaBrokerImage now wraps a sentinel (errNoKafkaBroker) so only a Ready cluster with no listable broker is terminal. Tests: pin the deadline on the built Job, the backupTimeout override, a JobFailed run marking the owner Failed, and the transient-List requeue. Signed-off-by: Andrey Kolkov <[email protected]> Assisted-by: LLM
1 parent 0ad0462 commit 09df69e

3 files changed

Lines changed: 238 additions & 52 deletions

File tree

‎docs/operations/backup-classes.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -361,4 +361,4 @@ This is the "reconstruct a cluster's topic topology" flow: an in-place restore r
361361

362362
One consequence of the Admin-API approach: if a topic is declared in the Kafka app's `spec.topics`, the Strimzi Topic Operator owns it and reconciles its config from the `KafkaTopic` CR, reverting any dynamic config an in-place restore sets that the CR does not list. So this strategy is durable for **out-of-band** topics (created directly, no CR); for CR-declared topics the `KafkaTopic` CR / GitOps is the source of truth. A to-copy restore onto a fresh empty cluster is unaffected. For message-data durability, use a volume-snapshot strategy instead. See `examples/backups/kafka-metadata/` for the end-to-end flow.
363363

364-
Restore is not transactional: it applies topics one at a time as it reads the object, so a failure part-way through (an RF mismatch, an unreachable broker) leaves the topics already created behind — re-running the restore is safe (existing topics are reconciled, not recreated). Like the Rabbitmq driver, this one owns its object outright and deletes it from the bucket when its `Backup` is deleted — via a one-shot Job the Backup's removal waits on — so a retention-pruned `Plan` does not accumulate objects; the delete is **best-effort** with the same give-up conditions and `backups.cozystack.io/skip-artifact-cleanup` escape hatch documented for Rabbitmq above. One caveat specific to this driver: its image ships an older `curl` (7.76.1) whose `--aws-sigv4` omits a non-default port from the SigV4 canonical host, which every S3 backend then rejects with `403 SignatureDoesNotMatch` on a ported endpoint (fixed in curl 7.86.0). The strategy script works around it with `--connect-to` — it signs and sends a portless URL and redirects the connection to the real port — so a ported endpoint (the `backupStorage.endpoint` fallback `…:8333`, `provisionBucket: false`, or SeaweedFS with `s3.ingress.enabled: false`) works too; the sibling drivers run curl 8.x and need no such workaround. Export time scales with topic count — the backup spends two Kafka CLI invocations (each a fresh JVM, ~1.5–3s) per topic — so a cluster with many hundreds of topics takes tens of minutes; the Job inherits the shared Job-strategy pod (no `activeDeadlineSeconds`, like every Job-shaped driver), so a wedged export keeps the BackupJob `Running` until the operator intervenes rather than failing on a wall clock.
364+
Restore is not transactional: it applies topics one at a time as it reads the object, so a failure part-way through (an RF mismatch, an unreachable broker) leaves the topics already created behind — re-running the restore is safe (existing topics are reconciled, not recreated). Like the Rabbitmq driver, this one owns its object outright and deletes it from the bucket when its `Backup` is deleted — via a one-shot Job the Backup's removal waits on — so a retention-pruned `Plan` does not accumulate objects; the delete is **best-effort** with the same give-up conditions and `backups.cozystack.io/skip-artifact-cleanup` escape hatch documented for Rabbitmq above. One caveat specific to this driver: its image ships an older `curl` (7.76.1) whose `--aws-sigv4` omits a non-default port from the SigV4 canonical host, which every S3 backend then rejects with `403 SignatureDoesNotMatch` on a ported endpoint (fixed in curl 7.86.0). The strategy script works around it with `--connect-to` — it signs and sends a portless URL and redirects the connection to the real port — so a ported endpoint (the `backupStorage.endpoint` fallback `…:8333`, `provisionBucket: false`, or SeaweedFS with `s3.ingress.enabled: false`) works too; the sibling drivers run curl 8.x and need no such workaround. Export time scales with topic count — the backup spends two Kafka CLI invocations (each a fresh JVM, ~1.5–3s) per topic — so a cluster with many hundreds of topics takes tens of minutes; the run is bounded at the Job layer by `activeDeadlineSeconds` (default 30 minutes, from the same wait the readiness precondition uses), so a wedged or unschedulable run is failed by Kubernetes as `DeadlineExceeded` — the pod is killed before the script's final upload, so no object is left orphaned — rather than requeuing forever. A cluster whose export legitimately runs longer raises the bound with the `backupTimeout` BackupClass parameter (a Go duration, e.g. `2h`), which round-trips through the `Backup` so a restore is bounded by the same value.

‎internal/backupcontroller/kafkastrategy_controller.go‎

Lines changed: 91 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@ package backupcontroller
33

44
import (
55
"context"
6+
"errors"
67
"fmt"
78
"strings"
89
"time"
@@ -55,11 +56,23 @@ const (
5556

5657
kafkaStrategyPollInterval = 5 * time.Second
5758

58-
// Cap on the Ready-precondition wait so a cluster that never comes up fails
59-
// the BackupJob/RestoreJob with a legible message instead of requeuing
60-
// forever.
59+
// Default wall-clock bound, used for two things: the Ready-precondition wait
60+
// (controller-timed, before any Job exists) and the run itself (the Job's
61+
// activeDeadlineSeconds). Export time scales with topic count, so operators
62+
// whose cluster legitimately runs past this raise it per BackupClass with the
63+
// kafkaBackupTimeoutParam parameter rather than being capped at a fixed value.
6164
kafkaDefaultBackupDeadline = 30 * time.Minute
6265

66+
// Floor for an operator-supplied backupTimeout: a value below this (or an
67+
// unparseable one) falls back to kafkaDefaultBackupDeadline, so a typo cannot
68+
// wedge every run at a near-zero deadline.
69+
kafkaMinBackupDeadline = time.Minute
70+
71+
// BackupClass parameter overriding the wall-clock deadline (a Go duration
72+
// string, e.g. "2h"). It round-trips through the Backup's DriverMetadata like
73+
// every other parameter, so a restore is bounded by the same value.
74+
kafkaBackupTimeoutParam = "backupTimeout"
75+
6376
// The Cozystack chart names the Strimzi Kafka cluster "kafka-<app>".
6477
kafkaClusterNamePrefix = "kafka-"
6578

@@ -132,6 +145,18 @@ func kafkaStrategyParameters(b *backupsv1alpha1.Backup) map[string]string {
132145
return out
133146
}
134147

148+
// kafkaRunDeadline resolves the wall-clock bound for a run from its parameters,
149+
// falling back to kafkaDefaultBackupDeadline when unset, unparseable, or below
150+
// the floor.
151+
func kafkaRunDeadline(parameters map[string]string) time.Duration {
152+
if v := parameters[kafkaBackupTimeoutParam]; v != "" {
153+
if d, err := time.ParseDuration(v); err == nil && d >= kafkaMinBackupDeadline {
154+
return d
155+
}
156+
}
157+
return kafkaDefaultBackupDeadline
158+
}
159+
135160
// kafkaRenderContext builds the template context. Unlike the Job strategy it
136161
// carries .BackupName (the per-run identity) so the strategy can scope the S3
137162
// object key and a restore reads the exact object its Backup wrote.
@@ -195,6 +220,11 @@ func renderKafkaArtifactURI(tmplStr string, ctxMap map[string]any) (string, erro
195220
return strings.TrimSpace(uri), nil
196221
}
197222

223+
// errNoKafkaBroker distinguishes "the cluster is Ready but no broker pod is
224+
// listable" (terminal - the callers gate on Ready first) from a transient List
225+
// error, which the caller requeues instead of failing the whole run.
226+
var errNoKafkaBroker = errors.New("no Kafka broker pod to resolve the client image from")
227+
198228
// resolveKafkaBrokerImage returns the container image of a running broker of the
199229
// Strimzi Kafka cluster, so the backup/restore Job runs the exact Admin-API CLI
200230
// the broker ships rather than a separately pinned guess that drifts from it.
@@ -213,7 +243,7 @@ func resolveKafkaBrokerImage(ctx context.Context, c client.Client, namespace, cl
213243
}
214244
}
215245
}
216-
return "", fmt.Errorf("no broker pod with a %q container found for Kafka cluster %s/%s", kafkaBrokerContainerName, namespace, clusterName)
246+
return "", fmt.Errorf("%w (looked for a %q container in cluster %s/%s)", errNoKafkaBroker, kafkaBrokerContainerName, namespace, clusterName)
217247
}
218248

219249
// ---------------------------------------------------------------------------
@@ -233,6 +263,8 @@ func (r *BackupJobReconciler) reconcileKafka(ctx context.Context, j *backupsv1al
233263
return r.markBackupJobFailed(ctx, j, err.Error())
234264
}
235265

266+
deadline := kafkaRunDeadline(resolved.Parameters)
267+
236268
// First-reconcile bookkeeping (refetch guards against a stale informer
237269
// sliding StartedAt forward). Mirrors reconcileJob.
238270
if j.Status.StartedAt == nil {
@@ -281,19 +313,23 @@ func (r *BackupJobReconciler) reconcileKafka(ctx context.Context, j *backupsv1al
281313
if !apierrors.IsNotFound(clusterErr) {
282314
return ctrl.Result{}, clusterErr
283315
}
284-
return r.kafkaBackupNotReady(ctx, j, "KafkaClusterNotReady",
316+
return r.kafkaBackupNotReady(ctx, j, deadline, "KafkaClusterNotReady",
285317
fmt.Sprintf("Kafka cluster %s not found yet", kafkaClusterName(appName)))
286318
}
287319
if msg := kafkaNotReadyMessage(cluster); msg != "" {
288-
return r.kafkaBackupNotReady(ctx, j, "KafkaClusterNotReady", msg)
320+
return r.kafkaBackupNotReady(ctx, j, deadline, "KafkaClusterNotReady", msg)
289321
}
290322

291323
// Run the exact CLI the broker ships by resolving its image now that the
292324
// cluster is Ready (so a broker pod exists), rather than from a chart pin that
293-
// drifts from the broker version.
325+
// drifts from the broker version. A transient List error requeues; only a
326+
// Ready cluster with no listable broker is terminal.
294327
clientImage, err := resolveKafkaBrokerImage(ctx, r.Client, j.Namespace, kafkaClusterName(appName))
295328
if err != nil {
296-
return r.markBackupJobFailed(ctx, j, fmt.Sprintf("cannot resolve Kafka client image: %v", err))
329+
if errors.Is(err, errNoKafkaBroker) {
330+
return r.markBackupJobFailed(ctx, j, fmt.Sprintf("cannot resolve Kafka client image: %v", err))
331+
}
332+
return ctrl.Result{}, err
297333
}
298334

299335
// The artifact URI is the single source of truth for where the export is
@@ -315,7 +351,7 @@ func (r *BackupJobReconciler) reconcileKafka(ctx context.Context, j *backupsv1al
315351
}
316352

317353
batchJob, err := r.ensureKafkaJob(ctx, j, j.Namespace, jobNameForBackupJob(j),
318-
kafkaStrategyModeBackup,
354+
kafkaStrategyModeBackup, deadline,
319355
map[string]string{
320356
backupsv1alpha1.OwningJobNameLabel: j.Name,
321357
backupsv1alpha1.OwningJobNamespaceLabel: j.Namespace,
@@ -358,21 +394,19 @@ func (r *BackupJobReconciler) reconcileKafka(ctx context.Context, j *backupsv1al
358394
return r.markBackupJobFailed(ctx, j, message)
359395

360396
default:
361-
// buildJobStrategyBatchJob sets no activeDeadlineSeconds, so a Job whose
362-
// pod never schedules (no capacity) would requeue in Running forever.
363-
// Bound the whole run by the same deadline the readiness wait uses (from
364-
// StartedAt) and fail the BackupJob legibly instead.
365-
if j.Status.StartedAt != nil && time.Since(j.Status.StartedAt.Time) > kafkaDefaultBackupDeadline {
366-
return r.markBackupJobFailed(ctx, j, fmt.Sprintf("Kafka metadata backup Job did not complete within %s (its pod may be unschedulable)", kafkaDefaultBackupDeadline))
367-
}
397+
// The Job carries activeDeadlineSeconds (set in ensureKafkaJob), so a run
398+
// that never completes - an unschedulable pod or a wedged export - is
399+
// failed by Kubernetes as DeadlineExceeded and lands in the JobFailed
400+
// branch above; the pod is killed before the script's final upload, so no
401+
// object is left orphaned. Here we only poll while it makes progress.
368402
return ctrl.Result{RequeueAfter: kafkaStrategyPollInterval}, nil
369403
}
370404
}
371405

372406
// kafkaBackupNotReady surfaces a precise Ready=False on the BackupJob and
373407
// requeues, or fails once the deadline since StartedAt is exceeded.
374-
func (r *BackupJobReconciler) kafkaBackupNotReady(ctx context.Context, j *backupsv1alpha1.BackupJob, reason, message string) (ctrl.Result, error) {
375-
if j.Status.StartedAt != nil && time.Since(j.Status.StartedAt.Time) > kafkaDefaultBackupDeadline {
408+
func (r *BackupJobReconciler) kafkaBackupNotReady(ctx context.Context, j *backupsv1alpha1.BackupJob, deadline time.Duration, reason, message string) (ctrl.Result, error) {
409+
if j.Status.StartedAt != nil && time.Since(j.Status.StartedAt.Time) > deadline {
376410
return r.markBackupJobFailed(ctx, j, fmt.Sprintf("timed out waiting for Kafka cluster to become Ready: %s", message))
377411
}
378412
apimeta.SetStatusCondition(&j.Status.Conditions, metav1.Condition{
@@ -391,6 +425,7 @@ func (r *BackupJobReconciler) ensureKafkaJob(
391425
ctx context.Context,
392426
owner client.Object,
393427
namespace, name, mode string,
428+
deadline time.Duration,
394429
ownerLabels map[string]string,
395430
rendered *corev1.PodTemplateSpec,
396431
) (*batchv1.Job, error) {
@@ -409,6 +444,11 @@ func (r *BackupJobReconciler) ensureKafkaJob(
409444
}
410445

411446
desired := buildJobStrategyBatchJob(namespace, name, labels, rendered)
447+
// Bound the run at the Job layer: Kubernetes fails an unschedulable or wedged
448+
// Job as DeadlineExceeded and kills its pod before the script's final upload,
449+
// so a timed-out run cannot both leave the BackupJob terminal and go on to
450+
// write an object no Backup ever references.
451+
setKafkaJobDeadline(desired, deadline)
412452
if err := controllerutil.SetControllerReference(owner, desired, r.Scheme); err != nil {
413453
return nil, fmt.Errorf("set controller reference on backup Job: %w", err)
414454
}
@@ -503,6 +543,10 @@ func (r *RestoreJobReconciler) reconcileKafkaRestore(ctx context.Context, restor
503543
return r.markRestoreJobFailed(ctx, restoreJob, err.Error())
504544
}
505545

546+
// The deadline round-trips from backup time via the Backup's parameters, so a
547+
// restore is bounded by the same value the operator set for the backup.
548+
deadline := kafkaRunDeadline(kafkaStrategyParameters(backup))
549+
506550
// Resolve the effective target: source app by default, overridden per-field
507551
// by targetApplicationRef for a to-copy restore.
508552
targetNamespace := restoreJob.Namespace
@@ -577,19 +621,23 @@ func (r *RestoreJobReconciler) reconcileKafkaRestore(ctx context.Context, restor
577621
if !apierrors.IsNotFound(clusterErr) {
578622
return ctrl.Result{}, clusterErr
579623
}
580-
return r.kafkaRestoreNotReady(ctx, restoreJob, "KafkaClusterNotReady",
624+
return r.kafkaRestoreNotReady(ctx, restoreJob, deadline, "KafkaClusterNotReady",
581625
fmt.Sprintf("target Kafka cluster %s not found yet", kafkaClusterName(targetAppName)))
582626
}
583627
if msg := kafkaNotReadyMessage(cluster); msg != "" {
584-
return r.kafkaRestoreNotReady(ctx, restoreJob, "KafkaClusterNotReady", msg)
628+
return r.kafkaRestoreNotReady(ctx, restoreJob, deadline, "KafkaClusterNotReady", msg)
585629
}
586630

587631
// Resolve the client image from the TARGET broker (Ready above, so a broker
588632
// pod exists), so a to-copy restore into a differently-versioned cluster runs
589-
// that cluster's own CLI rather than the source's.
633+
// that cluster's own CLI rather than the source's. Transient List errors
634+
// requeue; only a Ready cluster with no listable broker is terminal.
590635
clientImage, err := resolveKafkaBrokerImage(ctx, r.Client, targetNamespace, kafkaClusterName(targetAppName))
591636
if err != nil {
592-
return r.markRestoreJobFailed(ctx, restoreJob, fmt.Sprintf("cannot resolve Kafka client image: %v", err))
637+
if errors.Is(err, errNoKafkaBroker) {
638+
return r.markRestoreJobFailed(ctx, restoreJob, fmt.Sprintf("cannot resolve Kafka client image: %v", err))
639+
}
640+
return ctrl.Result{}, err
593641
}
594642

595643
// Restore reads the object the Backup recorded, not a reconstruction, so a
@@ -605,7 +653,7 @@ func (r *RestoreJobReconciler) reconcileKafkaRestore(ctx context.Context, restor
605653
}
606654

607655
batchJob, err := r.ensureKafkaRestoreJob(ctx, restoreJob, targetNamespace, jobNameForRestoreJob(restoreJob),
608-
kafkaStrategyModeRestore,
656+
kafkaStrategyModeRestore, deadline,
609657
map[string]string{
610658
backupsv1alpha1.OwningJobNameLabel: restoreJob.Name,
611659
backupsv1alpha1.OwningJobNamespaceLabel: restoreJob.Namespace,
@@ -640,17 +688,16 @@ func (r *RestoreJobReconciler) reconcileKafkaRestore(ctx context.Context, restor
640688
return r.markRestoreJobFailed(ctx, restoreJob, message)
641689

642690
default:
643-
// Same bound as the backup path: fail a restore whose Job never completes
644-
// (e.g. an unschedulable pod) instead of requeuing in Running forever.
645-
if restoreJob.Status.StartedAt != nil && time.Since(restoreJob.Status.StartedAt.Time) > kafkaDefaultBackupDeadline {
646-
return r.markRestoreJobFailed(ctx, restoreJob, fmt.Sprintf("Kafka metadata restore Job did not complete within %s (its pod may be unschedulable)", kafkaDefaultBackupDeadline))
647-
}
691+
// Same bound as the backup path, at the Job layer: the restore Job carries
692+
// activeDeadlineSeconds, so an unschedulable or wedged pod is failed by
693+
// Kubernetes and lands in the JobFailed branch above rather than requeuing
694+
// in Running forever.
648695
return ctrl.Result{RequeueAfter: kafkaStrategyPollInterval}, nil
649696
}
650697
}
651698

652-
func (r *RestoreJobReconciler) kafkaRestoreNotReady(ctx context.Context, rj *backupsv1alpha1.RestoreJob, reason, message string) (ctrl.Result, error) {
653-
if rj.Status.StartedAt != nil && time.Since(rj.Status.StartedAt.Time) > kafkaDefaultBackupDeadline {
699+
func (r *RestoreJobReconciler) kafkaRestoreNotReady(ctx context.Context, rj *backupsv1alpha1.RestoreJob, deadline time.Duration, reason, message string) (ctrl.Result, error) {
700+
if rj.Status.StartedAt != nil && time.Since(rj.Status.StartedAt.Time) > deadline {
654701
return r.markRestoreJobFailed(ctx, rj, fmt.Sprintf("timed out waiting for Kafka cluster to become Ready: %s", message))
655702
}
656703
apimeta.SetStatusCondition(&rj.Status.Conditions, metav1.Condition{
@@ -669,6 +716,7 @@ func (r *RestoreJobReconciler) ensureKafkaRestoreJob(
669716
ctx context.Context,
670717
owner client.Object,
671718
namespace, name, mode string,
719+
deadline time.Duration,
672720
ownerLabels map[string]string,
673721
rendered *corev1.PodTemplateSpec,
674722
) (*batchv1.Job, error) {
@@ -687,6 +735,7 @@ func (r *RestoreJobReconciler) ensureKafkaRestoreJob(
687735
}
688736

689737
desired := buildJobStrategyBatchJob(namespace, name, labels, rendered)
738+
setKafkaJobDeadline(desired, deadline)
690739
if err := controllerutil.SetControllerReference(owner, desired, r.Scheme); err != nil {
691740
return nil, fmt.Errorf("set controller reference on restore Job: %w", err)
692741
}
@@ -837,6 +886,18 @@ func (r *BackupReconciler) releaseKafkaCleanup(ctx context.Context, backup *back
837886
return ctrl.Result{}
838887
}
839888

889+
// setKafkaJobDeadline stamps activeDeadlineSeconds on a backup/restore Job. No
890+
// TTLSecondsAfterFinished: the controller must observe the Job's terminal
891+
// condition (to create the Backup, or mark the owner Failed) before it is
892+
// collected, and the owning BackupJob/RestoreJob garbage-collects it.
893+
func setKafkaJobDeadline(job *batchv1.Job, deadline time.Duration) {
894+
secs := int64(deadline / time.Second)
895+
if secs < 1 {
896+
secs = 1
897+
}
898+
job.Spec.ActiveDeadlineSeconds = &secs
899+
}
900+
840901
// buildKafkaCleanupJob wraps the cleanup-mode pod in a one-shot, ownerless Job.
841902
// Ownerless because cleanupKafkaBackup manages its lifecycle explicitly and it
842903
// must outlive the Backup being deleted; activeDeadlineSeconds + TTL are

0 commit comments

Comments
 (0)