Skip to content

Commit 0ad0462

Browse files
committed
fix(backups): pin the restore-side guards and bound a running Kafka Job
Address review r5 on #4000: - delete_topic waited with `grep -qx` (BRE), so deleting `audit.events` matched the live `audit-events` and spun to its timeout — the e2e failed at step 25 on every run that reached it. Use `grep -qxF`. - The two -k/--connect-to guards were written as `! grep`, which cannot fail under hack/cozytest.sh (it wraps each @test in `set -e` + a trailing `return 0`, and bash exempts a `!`-negated command from errexit). Rewrite them as positive `if grep ...; then return 1; fi`; verified the -k reinstatement now reddens the suite under cozytest. - Cover the restore-side literal-topic guard that was unpinned at every level: a bats case runs the existing-topic branch with a colliding sibling and asserts the `--alter` argv carries `\Q…\E`, and run-all.sh now leaves `audit.events` live so the e2e exercises that branch instead of only `--create`. Add bats cases for the partition-shrink refusal, the `-f` upload guard, and `--exclude-internal`. - Bound a running Job: reconcileKafka / reconcileKafkaRestore now fail the BackupJob/RestoreJob once kafkaDefaultBackupDeadline elapses in the Job-running branch (buildJobStrategyBatchJob sets no activeDeadlineSeconds), so an unschedulable pod no longer pins it in Running forever. Pinned by TestReconcileKafka_FailsWhenJobNeverCompletes. - Fail closed on an unknown record kind and an unknown MODE, and validate the live partition/RF parsed on restore (an empty value silently skipped reconciliation). Correct the stale "image pinned in the strategy" comment in the example helpers and the chainsaw test (the controller now resolves the target broker's image as .ClientImage). Signed-off-by: Andrey Kolkov <[email protected]> Assisted-by: LLM
1 parent 511d4be commit 0ad0462

7 files changed

Lines changed: 160 additions & 14 deletions

File tree

‎examples/backups/kafka-metadata/00-helpers.sh‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -58,9 +58,11 @@ export S3_CA_KEY="${S3_CA_KEY:-ca.crt}"
5858
export CA_MOUNT_DIR="${CA_MOUNT_DIR:-/etc/ssl/kafka-backup-ca}"
5959
# The Cozystack chart names the Strimzi cluster kafka-<app>; its plaintext
6060
# bootstrap Service is kafka-<app>-kafka-bootstrap:9092.
61-
# KAFKA_IMAGE only drives the host-side throwaway CLI pods here (seed/verify);
62-
# the backup/restore Jobs use the image pinned in the strategy. Override to
63-
# match your operator's Kafka image if it differs.
61+
# KAFKA_IMAGE only drives the host-side throwaway CLI pods here (seed/verify).
62+
# The backup/restore Jobs no longer use it: the controller resolves the target
63+
# broker's own image at reconcile time and renders it as the strategy's
64+
# .ClientImage. Override this only to match your operator's image for the
65+
# seed/verify pods if it differs.
6466
export KAFKA_IMAGE="${KAFKA_IMAGE:-quay.io/strimzi/kafka:0.45.1-rc1-kafka-3.9.1@sha256:ba52ed046b1dccdbd96f4e68057ce014d862a7c9c1fc670760c023b9aa09f23f}"
6567
export KAFKA_BIN="${KAFKA_BIN:-/opt/kafka/bin}"
6668

@@ -245,7 +247,7 @@ delete_topic() {
245247
"$BIN"/kafka-topics.sh --bootstrap-server "$BOOT" --delete --topic "\Q$TOPIC\E" || true
246248
for _ in $(seq 1 60); do
247249
list=$("$BIN"/kafka-topics.sh --bootstrap-server "$BOOT" --list) || { sleep 2; continue; }
248-
printf "%s\n" "$list" | grep -qx "$TOPIC" || { echo "topic $TOPIC deleted"; exit 0; }
250+
printf "%s\n" "$list" | grep -qxF -- "$TOPIC" || { echo "topic $TOPIC deleted"; exit 0; }
249251
sleep 2
250252
done
251253
echo "topic $TOPIC still present after wait" >&2; exit 1

‎examples/backups/kafka-metadata/run-all.sh‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -83,9 +83,13 @@ if [[ "${SKIP_RESTORE:-0}" == "1" ]]; then
8383
exit 0
8484
fi
8585

86-
print_header "Step 25: In-place restore — drop the topics, then restore them"
86+
print_header "Step 25: In-place restore — drop two topics, keep '${COLLIDE_DOT}' live, then restore"
87+
# Drop orders + audit-events but LEAVE audit.events live, so the restore runs
88+
# the existing-topic branch (grep -qxF + --describe/--alter under \Q) for a name
89+
# whose regex would collide with audit-events. A stripped \Q on the restore
90+
# --describe would match the sibling, read the wrong partition count and fail the
91+
# restore here — so this path, not just --create, is exercised end to end.
8792
delete_topic "$KAFKA_SRC_NAME" "$TOPIC"
88-
delete_topic "$KAFKA_SRC_NAME" "$COLLIDE_DOT"
8993
delete_topic "$KAFKA_SRC_NAME" "$COLLIDE_DASH"
9094
kubectl apply -f "$SCRIPT_DIR/25-restorejob-in-place.yaml"
9195
wait_for_field restorejobs.backups.cozystack.io "$RESTOREJOB_INPLACE_NAME" \

‎hack/e2e-chainsaw/kafka-metadata/chainsaw-test.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,8 +22,8 @@
2222
# is an unroutable placeholder in CI, and the .svc name is the one the seaweedfs
2323
# serving cert's SAN covers (curl verifies TLS against the copied CA).
2424
# KAFKA_IMAGE for the host-side seed/verify pods is resolved from the running
25-
# operator so it reuses the cached broker image (the backup/restore Jobs use the
26-
# image pinned in the strategy regardless).
25+
# operator so it reuses the cached broker image; the backup/restore Jobs resolve
26+
# their own image from the target broker (the strategy's .ClientImage).
2727
apiVersion: chainsaw.kyverno.io/v1alpha1
2828
kind: Test
2929
metadata:

‎hack/kafka-backup-script.bats‎

Lines changed: 73 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -38,21 +38,26 @@ init_stubs() {
3838
cat > "$BINDIR/kafka-topics.sh" <<STUB
3939
#!/usr/bin/env bash
4040
set -eu
41-
mode=""; topic=""; parts=""
41+
mode=""; topic=""; parts=""; excl=""
4242
while [ \$# -gt 0 ]; do
4343
case "\$1" in
4444
--list) mode=list ;;
4545
--describe) mode=describe ;;
4646
--create) mode=create ;;
4747
--alter) mode=alter ;;
48+
--exclude-internal) excl=1 ;;
4849
--topic) topic="\$2"; shift ;;
4950
--partitions) parts="\$2"; shift ;;
5051
esac
5152
shift
5253
done
5354
if [ "\$mode" = list ]; then
5455
[ -n "\${LIST_FAIL:-}" ] && exit 3
55-
cat "$STATE/topics" 2>/dev/null || true
56+
if [ -n "\$excl" ]; then
57+
grep -v '^__' "$STATE/topics" 2>/dev/null || true
58+
else
59+
cat "$STATE/topics" 2>/dev/null || true
60+
fi
5661
exit 0
5762
fi
5863
if [ "\$mode" = describe ]; then
@@ -103,13 +108,22 @@ STUB
103108
#!/usr/bin/env bash
104109
set -eu
105110
printf '%s\n' "\$*" >> "$STATE/curl_args"
106-
up=""; out=""; wfmt=""; prev=""
111+
up=""; out=""; wfmt=""; prev=""; hasf=""
107112
for a in "\$@"; do
108113
case "\$prev" in --upload-file) up="\$a" ;; -o) out="\$a" ;; -w) wfmt="\$a" ;; esac
114+
case "\$a" in -fsS|-f) hasf=1 ;; esac
109115
prev="\$a"
110116
done
111117
if [ -n "\$wfmt" ]; then echo "\${DELETE_CODE:-204}"; exit 0; fi
112-
if [ -n "\$up" ]; then cp "\$up" "$STATE/s3_object"; exit 0; fi
118+
if [ -n "\$up" ]; then
119+
# PUT_FAIL simulates a 403 on upload: with -f curl errors (exit 22) and stores
120+
# nothing; without -f it would exit 0 and a bad backup would report success.
121+
if [ -n "\${PUT_FAIL:-}" ]; then
122+
[ -n "\$hasf" ] && exit 22
123+
exit 0
124+
fi
125+
cp "\$up" "$STATE/s3_object"; exit 0
126+
fi
113127
if [ -n "\$out" ]; then
114128
[ -f "$STATE/s3_object" ] || exit 22
115129
cp "$STATE/s3_object" "\$out"; exit 0
@@ -180,13 +194,21 @@ expect_fail() {
180194
init_stubs; seed_three
181195
MODE=backup S3_ENDPOINT="https://s3.example.org:8333" bash "$SCRIPT"
182196
grep -qF -- '--connect-to s3.example.org:443:s3.example.org:8333' "$STATE/curl_args"
183-
! grep -qE -- '(^| )-k( |$)' "$STATE/curl_args"
197+
# Positive form, not `! grep`: hack/cozytest.sh wraps each @test in `set -e`
198+
# + a trailing `return 0`, and bash exempts a `!`-negated command from errexit,
199+
# so a `! grep` guard can never fail under the runner CI uses (only real bats).
200+
if grep -qE -- '(^| )-k( |$)' "$STATE/curl_args"; then
201+
echo "regression: -k (insecure TLS) in curl argv: $(cat "$STATE/curl_args")" >&2; return 1
202+
fi
184203
}
185204

186205
@test "default-port endpoint needs no --connect-to" {
187206
init_stubs; seed_three
188207
MODE=backup S3_ENDPOINT="https://s3.example.org" bash "$SCRIPT"
189-
! grep -qF -- '--connect-to' "$STATE/curl_args"
208+
# Positive form (see the -k note above): a `! grep` is vacuous under cozytest.
209+
if grep -qF -- '--connect-to' "$STATE/curl_args"; then
210+
echo "unexpected --connect-to on a default-port endpoint: $(cat "$STATE/curl_args")" >&2; return 1
211+
fi
190212
}
191213

192214
@test "virtual-hosted ported endpoint keys --connect-to off the bucket host" {
@@ -232,6 +254,51 @@ write_backup_object() {
232254
[ ! -f "$STATE/actions" ]
233255
}
234256

257+
@test "restore of an existing under-partitioned topic alters it with the literal-Q wrapper" {
258+
init_stubs
259+
printf 'T\tpay.events\t7\t1\n' > "$STATE/s3_object" # backup wants 7 partitions
260+
printf 'pay.events\npay-events\n' > "$STATE/topics" # both live -> the collision
261+
printf 'Topic: pay.events\tPartitionCount: 2\tReplicationFactor: 1\n' > "$STATE/desc.pay.events"
262+
printf 'Topic: pay-events\tPartitionCount: 2\tReplicationFactor: 1\n' > "$STATE/desc.pay-events"
263+
MODE=restore S3_ENDPOINT="https://s3.example.org" bash "$SCRIPT"
264+
# The alter must target the literal topic (\Q...\E). A stripped wrapper sends a
265+
# regex that re-partitions the colliding sibling permanently — the destructive
266+
# bug this guards. Dropping \Q on the restore --alter reddens here.
267+
grep -qxF 'alter \Qpay.events\E 7' "$STATE/actions"
268+
}
269+
270+
@test "restore refuses to shrink partitions" {
271+
init_stubs
272+
printf 'T\torders\t2\t1\n' > "$STATE/s3_object" # backup wants 2
273+
printf 'orders\n' > "$STATE/topics"
274+
printf 'Topic: orders\tPartitionCount: 5\tReplicationFactor: 1\n' > "$STATE/desc.orders" # live 5 > 2
275+
out=""
276+
if out=$(MODE=restore S3_ENDPOINT="https://s3.example.org" bash "$SCRIPT" 2>&1); then
277+
echo "restore unexpectedly succeeded: $out" >&2; return 1
278+
fi
279+
printf '%s\n' "$out" | grep -q 'Kafka cannot decrease partitions'
280+
}
281+
282+
@test "backup fails closed when the S3 upload is rejected (curl -f)" {
283+
init_stubs; seed_three
284+
# PUT_FAIL simulates a 403; -fsS must turn that into a non-zero exit so the
285+
# backup does not report success against an object that never landed.
286+
expect_fail env MODE=backup PUT_FAIL=1 S3_ENDPOINT="https://s3.example.org" bash "$SCRIPT"
287+
[ ! -f "$STATE/s3_object" ]
288+
}
289+
290+
@test "backup excludes Kafka internal topics" {
291+
init_stubs
292+
printf 'orders\n__consumer_offsets\n' > "$STATE/topics"
293+
printf 'Topic: orders\tPartitionCount: 3\tReplicationFactor: 1\n' > "$STATE/desc.orders"
294+
# No desc.__consumer_offsets on purpose: with --exclude-internal it is filtered
295+
# from --list; drop the flag and the backup instead tries to describe it and
296+
# fails — either way it must never land in the object.
297+
MODE=backup S3_ENDPOINT="https://s3.example.org" bash "$SCRIPT"
298+
grep -qxF "$(printf 'T\torders\t3\t1')" "$STATE/s3_object"
299+
if grep -qF '__consumer_offsets' "$STATE/s3_object"; then echo "internal topic captured" >&2; return 1; fi
300+
}
301+
235302
# ---- cleanup --------------------------------------------------------------
236303

237304
@test "cleanup treats 204 as success" {

‎internal/backupcontroller/kafkastrategy_controller.go‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -358,6 +358,13 @@ func (r *BackupJobReconciler) reconcileKafka(ctx context.Context, j *backupsv1al
358358
return r.markBackupJobFailed(ctx, j, message)
359359

360360
default:
361+
// buildJobStrategyBatchJob sets no activeDeadlineSeconds, so a Job whose
362+
// pod never schedules (no capacity) would requeue in Running forever.
363+
// Bound the whole run by the same deadline the readiness wait uses (from
364+
// StartedAt) and fail the BackupJob legibly instead.
365+
if j.Status.StartedAt != nil && time.Since(j.Status.StartedAt.Time) > kafkaDefaultBackupDeadline {
366+
return r.markBackupJobFailed(ctx, j, fmt.Sprintf("Kafka metadata backup Job did not complete within %s (its pod may be unschedulable)", kafkaDefaultBackupDeadline))
367+
}
361368
return ctrl.Result{RequeueAfter: kafkaStrategyPollInterval}, nil
362369
}
363370
}
@@ -633,6 +640,11 @@ func (r *RestoreJobReconciler) reconcileKafkaRestore(ctx context.Context, restor
633640
return r.markRestoreJobFailed(ctx, restoreJob, message)
634641

635642
default:
643+
// Same bound as the backup path: fail a restore whose Job never completes
644+
// (e.g. an unschedulable pod) instead of requeuing in Running forever.
645+
if restoreJob.Status.StartedAt != nil && time.Since(restoreJob.Status.StartedAt.Time) > kafkaDefaultBackupDeadline {
646+
return r.markRestoreJobFailed(ctx, restoreJob, fmt.Sprintf("Kafka metadata restore Job did not complete within %s (its pod may be unschedulable)", kafkaDefaultBackupDeadline))
647+
}
636648
return ctrl.Result{RequeueAfter: kafkaStrategyPollInterval}, nil
637649
}
638650
}

‎internal/backupcontroller/kafkastrategy_controller_test.go‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -406,6 +406,48 @@ func TestReconcileKafka_FailsAfterReadinessDeadline(t *testing.T) {
406406
}
407407
}
408408

409+
// TestReconcileKafka_FailsWhenJobNeverCompletes pins the running-Job deadline: a
410+
// Job whose pod never completes (e.g. unschedulable) must terminate the
411+
// BackupJob as Failed once the deadline elapses, not requeue in Running forever
412+
// (buildJobStrategyBatchJob sets no activeDeadlineSeconds). Neutralising the
413+
// default-branch deadline check turns this red.
414+
func TestReconcileKafka_FailsWhenJobNeverCompletes(t *testing.T) {
415+
old := metav1.NewTime(time.Now().Add(-(kafkaDefaultBackupDeadline + time.Minute)))
416+
bj := &backupsv1alpha1.BackupJob{
417+
ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "bj"},
418+
Spec: backupsv1alpha1.BackupJobSpec{ApplicationRef: kafkaAppRef("src"), BackupClassName: "cozy-default"},
419+
Status: backupsv1alpha1.BackupJobStatus{StartedAt: &old, Phase: backupsv1alpha1.BackupJobPhaseRunning},
420+
}
421+
cluster := &kafkatypes.Kafka{
422+
ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "kafka-src"},
423+
Status: kafkatypes.KafkaStatus{Conditions: []metav1.Condition{
424+
{Type: kafkatypes.ConditionTypeReady, Status: metav1.ConditionTrue, Reason: "Ready"},
425+
}},
426+
}
427+
// A broker pod so the client-image resolves, and a Job already running
428+
// (no terminal condition) so reconcile reaches the default branch.
429+
broker := &corev1.Pod{
430+
ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "kafka-src-kafka-0", Labels: map[string]string{"strimzi.io/cluster": "kafka-src"}},
431+
Spec: corev1.PodSpec{Containers: []corev1.Container{{Name: "kafka", Image: "quay.io/strimzi/kafka:test"}}},
432+
}
433+
runningJob := &batchv1.Job{ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "bj-backup"}}
434+
r, _ := newKafkaTestEnv(t, newKafkaApp("src", "tenant"),
435+
clientfake.NewClientBuilder().WithObjects(bj, newKafkaStrategy("cozy-default-kafka"), cluster, broker, runningJob))
436+
437+
if _, err := r.reconcileKafka(context.Background(), bj, &ResolvedBackupConfig{
438+
StrategyRef: corev1.TypedLocalObjectReference{Kind: strategyv1alpha1.KafkaStrategyKind, Name: "cozy-default-kafka"},
439+
}); err != nil {
440+
t.Fatalf("reconcileKafka: %v", err)
441+
}
442+
got := &backupsv1alpha1.BackupJob{}
443+
if err := r.Get(context.Background(), client.ObjectKeyFromObject(bj), got); err != nil {
444+
t.Fatalf("get bj: %v", err)
445+
}
446+
if got.Status.Phase != backupsv1alpha1.BackupJobPhaseFailed {
447+
t.Fatalf("expected Failed once the running Job exceeded the deadline, got %q", got.Status.Phase)
448+
}
449+
}
450+
409451
// kafkaBackup builds a Backup fixture that has already recorded its metadata
410452
// object, for the cleanup path (whose delete is keyed on status.artifact.uri).
411453
func kafkaBackup(name, namespace string) *backupsv1alpha1.Backup {

‎packages/system/backupstrategy-controller/files/kafka-backup.sh‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,15 @@ BIN="${BIN:-/opt/kafka/bin}"
66
TAB=$(printf '\t')
77
file=/tmp/kafka-metadata.txt
88

9+
# Validate MODE up front and fail closed on anything unexpected. internal/
10+
# template renders the input unchanged when it fails, so a broken env render
11+
# would ship the literal "{{ .Mode }}"; without this guard the bare `else`
12+
# below would run the destructive restore path under the guise of a backup.
13+
case "${MODE}" in
14+
backup|restore|cleanup) ;;
15+
*) echo "unknown MODE '${MODE}' (expected backup|restore|cleanup)" >&2; exit 1 ;;
16+
esac
17+
918
case "${S3_ENDPOINT}" in
1019
http://*|https://*) base="${S3_ENDPOINT}" ;;
1120
*) base="https://${S3_ENDPOINT}" ;;
@@ -151,6 +160,10 @@ else
151160
line=${desc%%$'\n'*}
152161
lp=$(describe_field "${line}" PartitionCount)
153162
lrf=$(describe_field "${line}" ReplicationFactor)
163+
# Validate the live shape parsed (as the backup path does for parts/rf):
164+
# an empty lp/lrf makes the numeric comparisons below exit 2, which the
165+
# if/elif would swallow and report the topic restored without altering.
166+
{ [ -n "${lp}" ] && [ -n "${lrf}" ]; } || { echo "unparsable live describe for ${t}: ${line}" >&2; exit 1; }
154167
# A restore that cannot reach the recorded shape must fail loudly, not
155168
# report success against a divergent topic.
156169
if [ "${lrf}" != "${rf}" ]; then
@@ -177,6 +190,12 @@ else
177190
"${BIN}"/kafka-configs.sh --bootstrap-server "${BOOTSTRAP}" --alter --entity-type topics --entity-name "${t}" --add-config "${add}"
178191
echo " config ${t}: ${k}"
179192
;;
193+
*)
194+
# Fail closed on an unrecognised record: a well-formed but wrong-content
195+
# object (curl -f catches transport errors, not a wrong-content 200) must
196+
# not restore a subset of topics and still report success.
197+
echo "unrecognised metadata record kind '${kind}' (corrupt or wrong object?)" >&2; exit 1
198+
;;
180199
esac
181200
done < "${file}"
182201
echo "restore complete"

0 commit comments

Comments
 (0)