Skip to content

Commit 98486b5

Browse files
committed
fix(backups): restore Kafka S3 port workaround and hold the script with a test
Address review r4 on #4000: - Regression fix: r3 deleted the curl --connect-to port workaround along with -k, but curl 7.76.1 (the image's) 403s on ANY ported endpoint, which is the shape backupStorage.endpoint ships (:8333) and what provisionBucket:false and s3.ingress.enabled:false produce. Restore --connect-to (sign a portless URL, redirect to the real port), keeping -k off so TLS is still verified. IPv6-safe host/port split, path prefix preserved. The dev7 happy path passed only because its ingress is port-less; the ported case is the one that regressed. - Correct the S3 envelope claim in values.yaml and backup-classes.md: the 403 is decided by the port, not a body-hash the backend recomputes, and --connect-to handles it on SeaweedFS and MinIO alike. - Hold the script fixes at the layer the contract lives at. Externalise the driver to files/kafka-backup.sh (embedded verbatim via .Files.Get, render byte-identical) and add hack/kafka-backup-script.bats, which EXECUTES it against stubbed kafka CLIs + curl and asserts behaviour: \Q literal match on the colliding pair, --connect-to with no -k on a ported endpoint, fail-closed --list, comma-value bracket grouping, RF-divergence abort, and the cleanup delete's HTTP handling. Not a template grep. - Pin the Backup-delete dispatch and the readiness deadline in Go: TestCleanupOnDelete_Kafka_RoutesToKafkaCleanup and TestReconcileKafka_FailsAfterReadinessDeadline. - Docs: config restore is additive (no --delete-config); export time scales with topic count and the Job has no wall clock (as its siblings). Signed-off-by: Andrey Kolkov <[email protected]> Assisted-by: LLM
1 parent de399d3 commit 98486b5

6 files changed

Lines changed: 487 additions & 172 deletions

File tree

‎docs/operations/backup-classes.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@ Different operators expect different endpoint shapes; the strategy templates ren
4141
| FoundationDB | `blobStoreConfiguration.accountName` + `urlParameters.secure_connection` | bare host:port + derived secure flag |
4242
| Velero | `BackupStorageLocation.spec.config.s3Url` | full URL (scheme preserved) |
4343
| ClickHouse sidecar | `S3_ENDPOINT` env | bare host:port (from projected Secret) |
44-
| Kafka | `S3_ENDPOINT` env on the strategy Job | full URL (scheme preserved); the Job's `curl --aws-sigv4` prepends `https://` if the endpoint carries no scheme, keeps the whole base URL and honours `backupStorage.forcePathStyle` when building the object URL |
44+
| Kafka | `S3_ENDPOINT` env on the strategy Job | full URL (scheme preserved); the Job's `curl --aws-sigv4` prepends `https://` if the endpoint carries no scheme, honours `backupStorage.forcePathStyle`, and (for the image's curl 7.76.1) signs a portless URL with `--connect-to` to the real port so a ported endpoint still verifies |
4545

4646
The projected `cozy-backups-creds.endpoint` key is **stripped of scheme** so chart-emitted sidecars (ClickHouse) consume it directly. Drivers that need the full URL receive the resolved endpoint described above — derived from the COSI system Secret (forced `https://`) for a provisioned bucket, or the `backupStorage.endpoint` fallback for external S3.
4747

@@ -357,8 +357,8 @@ An in-progress restore is safe to reconcile. The driver purges the target `Clust
357357

358358
The `cozy-default-kafka` strategy backs up **topic metadata**, not message data. Its Job talks to the Kafka Admin API (`kafka-topics --describe` on backup, `kafka-topics --create` + `kafka-configs --alter` on restore) and stores one small object per run at `s3://<bucket>/<namespace>/<application>/<backup-name>/kafka-metadata.txt`. It captures every non-internal topic's partition count, replication factor and non-default configs; message payloads, consumer-group offsets, ACLs, quotas and `KafkaUser`s are out of scope, and there is no point-in-time recovery. The driver gates each run on the Strimzi `Kafka` cluster reporting `Ready` and never mutates it.
359359

360-
This is the "reconstruct a cluster's topic topology" flow: an in-place restore recreates dropped topics into the source, and a to-copy restore applies the source's topics onto a freshly-bootstrapped empty `Kafka` (partitions and configs preserved). Restore is additive at the topic-set level — it never deletes topics that exist live but not in the backup — but it does not silently accept a divergent existing topic: it grows the partition count when the backup asks for more, and fails loudly (naming the topic and both values) when the backup asks for fewer partitions or a different replication factor, since Kafka cannot shrink partitions or change RF in place and a restore that cannot reach the recorded state must not report `Succeeded`. Restored topics are created directly through the Admin API, so on the target they are **unmanaged** (no `KafkaTopic` CR); that is the accepted tradeoff of an Admin-API-only driver. Restoring a topic whose replication factor exceeds the target cluster's broker count fails at `--create`.
360+
This is the "reconstruct a cluster's topic topology" flow: an in-place restore recreates dropped topics into the source, and a to-copy restore applies the source's topics onto a freshly-bootstrapped empty `Kafka` (partitions and configs preserved). Restore is additive at the topic-set level — it never deletes topics that exist live but not in the backup — but it does not silently accept a divergent existing topic: it grows the partition count when the backup asks for more, and fails loudly (naming the topic and both values) when the backup asks for fewer partitions or a different replication factor, since Kafka cannot shrink partitions or change RF in place and a restore that cannot reach the recorded state must not report `Succeeded`. Restored topics are created directly through the Admin API, so on the target they are **unmanaged** (no `KafkaTopic` CR); that is the accepted tradeoff of an Admin-API-only driver. Restoring a topic whose replication factor exceeds the target cluster's broker count fails at `--create`. Topic configs are restored **additively** (`kafka-configs --add-config`): a config the backup recorded is re-applied, but a config added to a live topic after the backup is not removed, so an in-place restore does not return a topic's config set to its exact backup-time state (partition count and replication factor are the only shape the restore reconciles or fails loudly on).
361361

362362
One consequence of the Admin-API approach: if a topic is declared in the Kafka app's `spec.topics`, the Strimzi Topic Operator owns it and reconciles its config from the `KafkaTopic` CR, reverting any dynamic config an in-place restore sets that the CR does not list. So this strategy is durable for **out-of-band** topics (created directly, no CR); for CR-declared topics the `KafkaTopic` CR / GitOps is the source of truth. A to-copy restore onto a fresh empty cluster is unaffected. For message-data durability, use a volume-snapshot strategy instead. See `examples/backups/kafka-metadata/` for the end-to-end flow.
363363

364-
Restore is not transactional: it applies topics one at a time as it reads the object, so a failure part-way through (an RF mismatch, an unreachable broker) leaves the topics already created behind — re-running the restore is safe (existing topics are reconciled, not recreated). Like the Rabbitmq driver, this one owns its object outright and deletes it from the bucket when its `Backup` is deleted — via a one-shot Job the Backup's removal waits on — so a retention-pruned `Plan` does not accumulate objects; the delete is **best-effort** with the same give-up conditions and `backups.cozystack.io/skip-artifact-cleanup` escape hatch documented for Rabbitmq above. One caveat specific to this driver: its image ships an older `curl` (7.76.1) whose `--aws-sigv4` signs an empty-body payload hash on upload and omits `x-amz-content-sha256`, so its S3 envelope is narrower than the other drivers' — SeaweedFS and MinIO accept it, but a backend that recomputes the body hash may reject it (relevant only for `provisionBucket: false` external S3 pointing at AWS S3 or Ceph RGW).
364+
Restore is not transactional: it applies topics one at a time as it reads the object, so a failure part-way through (an RF mismatch, an unreachable broker) leaves the topics already created behind — re-running the restore is safe (existing topics are reconciled, not recreated). Like the Rabbitmq driver, this one owns its object outright and deletes it from the bucket when its `Backup` is deleted — via a one-shot Job the Backup's removal waits on — so a retention-pruned `Plan` does not accumulate objects; the delete is **best-effort** with the same give-up conditions and `backups.cozystack.io/skip-artifact-cleanup` escape hatch documented for Rabbitmq above. One caveat specific to this driver: its image ships an older `curl` (7.76.1) whose `--aws-sigv4` omits a non-default port from the SigV4 canonical host, which every S3 backend then rejects with `403 SignatureDoesNotMatch` on a ported endpoint (fixed in curl 7.86.0). The strategy script works around it with `--connect-to` — it signs and sends a portless URL and redirects the connection to the real port — so a ported endpoint (the `backupStorage.endpoint` fallback `…:8333`, `provisionBucket: false`, or SeaweedFS with `s3.ingress.enabled: false`) works too; the sibling drivers run curl 8.x and need no such workaround. Export time scales with topic count — the backup spends two Kafka CLI invocations (each a fresh JVM, ~1.5–3s) per topic — so a cluster with many hundreds of topics takes tens of minutes; the Job inherits the shared Job-strategy pod (no `activeDeadlineSeconds`, like every Job-shaped driver), so a wedged export keeps the BackupJob `Running` until the operator intervenes rather than failing on a wall clock.

‎hack/kafka-backup-script.bats‎

Lines changed: 232 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,232 @@
1+
#!/usr/bin/env bats
2+
# Behavioural unit test for the Kafka topic-metadata strategy driver script
3+
# (packages/system/backupstrategy-controller/files/kafka-backup.sh), the shell
4+
# embedded verbatim into strategy-kafka-default.yaml.
5+
#
6+
# The chart's helm-unittest suite can only assert the rendered Strategy CR's
7+
# shape (kind, name, artifactURITemplate); the 160-line driver script that does
8+
# the actual backup/restore/cleanup work is otherwise unpinned, so a regression
9+
# in it (a stripped \Q, a reinstated -k, a fail-open --list) passes the chart
10+
# suite untouched. This test closes that gap the way the contract is really
11+
# defined: it EXECUTES the script against stubbed kafka-topics.sh /
12+
# kafka-configs.sh (BIN override) and a stubbed curl (on PATH), and asserts what
13+
# the script does. It is not a template grep — a comment cannot satisfy it.
14+
#
15+
# Each named guard below corresponds to a mutation the script must not survive:
16+
# \Q literal-match on describe and alter, the --connect-to port workaround with
17+
# no -k, the fail-closed --list, the comma-value bracket grouping, the
18+
# replication-factor divergence guard, and the cleanup delete's HTTP handling.
19+
20+
SCRIPT="$(cd "$(dirname "${BATS_TEST_FILENAME:-$0}")/.." && pwd)/packages/system/backupstrategy-controller/files/kafka-backup.sh"
21+
22+
setup() {
23+
STATE="$BATS_TEST_TMPDIR/state"
24+
BINDIR="$BATS_TEST_TMPDIR/bin" # stub kafka CLIs (script's $BIN)
25+
PATHDIR="$BATS_TEST_TMPDIR/path" # stub curl (first on PATH)
26+
mkdir -p "$STATE" "$BINDIR" "$PATHDIR"
27+
28+
cat > "$BINDIR/kafka-topics.sh" <<STUB
29+
#!/usr/bin/env bash
30+
set -eu
31+
mode=""; topic=""; parts=""
32+
while [ \$# -gt 0 ]; do
33+
case "\$1" in
34+
--list) mode=list ;;
35+
--describe) mode=describe ;;
36+
--create) mode=create ;;
37+
--alter) mode=alter ;;
38+
--topic) topic="\$2"; shift ;;
39+
--partitions) parts="\$2"; shift ;;
40+
esac
41+
shift
42+
done
43+
if [ "\$mode" = list ]; then
44+
[ -n "\${LIST_FAIL:-}" ] && exit 3
45+
cat "$STATE/topics" 2>/dev/null || true
46+
exit 0
47+
fi
48+
if [ "\$mode" = describe ]; then
49+
echo "\$topic" >> "$STATE/describe_args"
50+
name="\${topic#'\\Q'}"; name="\${name%'\\E'}"
51+
if [ "\$name" = "\$topic" ]; then
52+
# No \\Q...\\E wrapping: --topic is a Java regex, so emit every stored
53+
# topic whose name matches it as an ERE (this is the collision a stripped
54+
# \\Q reintroduces).
55+
hit=0
56+
for f in "$STATE"/desc.*; do
57+
[ -e "\$f" ] || continue
58+
n="\${f##*/desc.}"
59+
if printf '%s\n' "\$n" | grep -qE "^\${name}\$"; then cat "\$f"; hit=1; fi
60+
done
61+
[ "\$hit" = 1 ] || exit 1
62+
exit 0
63+
fi
64+
[ -f "$STATE/desc.\$name" ] || exit 1
65+
cat "$STATE/desc.\$name"
66+
exit 0
67+
fi
68+
if [ "\$mode" = create ]; then echo "create \$topic \$parts" >> "$STATE/actions"; exit 0; fi
69+
if [ "\$mode" = alter ]; then echo "alter \$topic \$parts" >> "$STATE/actions"; exit 0; fi
70+
exit 0
71+
STUB
72+
73+
cat > "$BINDIR/kafka-configs.sh" <<STUB
74+
#!/usr/bin/env bash
75+
set -eu
76+
mode=""; name=""; add=""
77+
while [ \$# -gt 0 ]; do
78+
case "\$1" in
79+
--describe) mode=describe ;;
80+
--alter) mode=alter ;;
81+
--entity-name) name="\$2"; shift ;;
82+
--add-config) add="\$2"; shift ;;
83+
esac
84+
shift
85+
done
86+
if [ "\$mode" = describe ]; then cat "$STATE/cfg.\$name" 2>/dev/null || true; exit 0; fi
87+
if [ "\$mode" = alter ]; then echo "\$name \$add" >> "$STATE/configs_applied"; exit 0; fi
88+
exit 0
89+
STUB
90+
91+
# curl stub: records argv, simulates a tiny S3 (PUT stores the body, GET
92+
# serves it back), and prints an HTTP code for the -w cleanup DELETE.
93+
cat > "$PATHDIR/curl" <<STUB
94+
#!/usr/bin/env bash
95+
set -eu
96+
printf '%s\n' "\$*" >> "$STATE/curl_args"
97+
up=""; out=""; wfmt=""; method=""
98+
prev=""
99+
for a in "\$@"; do
100+
case "\$prev" in --upload-file) up="\$a" ;; -o) out="\$a" ;; -w) wfmt="\$a" ;; -X) method="\$a" ;; esac
101+
prev="\$a"
102+
done
103+
if [ -n "\$wfmt" ]; then echo "\${DELETE_CODE:-204}"; exit 0; fi
104+
if [ -n "\$up" ]; then cp "\$up" "$STATE/s3_object"; exit 0; fi
105+
if [ -n "\$out" ]; then
106+
[ -f "$STATE/s3_object" ] || exit 22
107+
cp "$STATE/s3_object" "\$out"; exit 0
108+
fi
109+
exit 0
110+
STUB
111+
chmod +x "$BINDIR"/*.sh "$PATHDIR/curl"
112+
113+
export BIN="$BINDIR"
114+
export PATH="$PATHDIR:$PATH"
115+
export BOOTSTRAP="kafka-x-kafka-bootstrap.tenant-root.svc:9092"
116+
export S3_REGION="us-east-1"
117+
export S3_FORCE_PATH_STYLE="true"
118+
export AWS_ACCESS_KEY_ID="k"
119+
export AWS_SECRET_ACCESS_KEY="s"
120+
}
121+
122+
# ---- backup ---------------------------------------------------------------
123+
124+
seed_three() {
125+
printf 'orders\naudit.events\naudit-events\n' > "$STATE/topics"
126+
echo "Topic: orders PartitionCount: 3 ReplicationFactor: 1" > "$STATE/desc.orders"
127+
echo "Topic: audit.events PartitionCount: 2 ReplicationFactor: 1" > "$STATE/desc.audit.events"
128+
echo "Topic: audit-events PartitionCount: 5 ReplicationFactor: 1" > "$STATE/desc.audit-events"
129+
printf ' retention.ms=1234567890000 sensitive=false\n' > "$STATE/cfg.orders"
130+
}
131+
132+
@test "backup records each colliding topic's own partition count (\\Q literal match)" {
133+
seed_three
134+
MODE=backup ARTIFACT_URI="s3://bkt/ns/app/run/kafka-metadata.txt" \
135+
S3_ENDPOINT="https://s3.example.org" run bash "$SCRIPT"
136+
[ "$status" -eq 0 ]
137+
# The uploaded object must carry each topic's true partition count. A stripped
138+
# \Q makes the describe of audit.events match audit-events too, taking the
139+
# wrong first line — this assertion goes red then.
140+
grep -qx "T orders 3 1" "$STATE/s3_object"
141+
grep -qx "T audit.events 2 1" "$STATE/s3_object"
142+
grep -qx "T audit-events 5 1" "$STATE/s3_object"
143+
# And the describe was invoked with the literal \Q...\E wrapper.
144+
grep -qxF '\Qaudit.events\E' "$STATE/describe_args"
145+
}
146+
147+
@test "backup fails closed when --list errors (no empty backup reported as success)" {
148+
seed_three
149+
MODE=backup LIST_FAIL=1 ARTIFACT_URI="s3://bkt/ns/app/run/kafka-metadata.txt" \
150+
S3_ENDPOINT="https://s3.example.org" run bash "$SCRIPT"
151+
[ "$status" -ne 0 ]
152+
[ ! -f "$STATE/s3_object" ]
153+
}
154+
155+
@test "backup stores a comma-valued config in its own tab field" {
156+
printf 'orders\n' > "$STATE/topics"
157+
echo "Topic: orders PartitionCount: 1 ReplicationFactor: 1" > "$STATE/desc.orders"
158+
printf ' cleanup.policy=compact,delete sensitive=false\n' > "$STATE/cfg.orders"
159+
MODE=backup ARTIFACT_URI="s3://bkt/ns/app/run/kafka-metadata.txt" \
160+
S3_ENDPOINT="https://s3.example.org" run bash "$SCRIPT"
161+
[ "$status" -eq 0 ]
162+
grep -qx "C orders cleanup.policy compact,delete" "$STATE/s3_object"
163+
}
164+
165+
# ---- S3 signing (curl 7.76.1 port workaround, no -k) -----------------------
166+
167+
@test "ported endpoint signs with --connect-to and never -k" {
168+
seed_three
169+
MODE=backup ARTIFACT_URI="s3://bkt/ns/app/run/kafka-metadata.txt" \
170+
S3_ENDPOINT="https://s3.example.org:8333" run bash "$SCRIPT"
171+
[ "$status" -eq 0 ]
172+
grep -qF -- '--connect-to s3.example.org:443:s3.example.org:8333' "$STATE/curl_args"
173+
! grep -qE -- '(^| )-k( |$)' "$STATE/curl_args"
174+
}
175+
176+
@test "default-port endpoint needs no --connect-to" {
177+
seed_three
178+
MODE=backup ARTIFACT_URI="s3://bkt/ns/app/run/kafka-metadata.txt" \
179+
S3_ENDPOINT="https://s3.example.org" run bash "$SCRIPT"
180+
[ "$status" -eq 0 ]
181+
! grep -qF -- '--connect-to' "$STATE/curl_args"
182+
}
183+
184+
# ---- restore --------------------------------------------------------------
185+
186+
write_backup_object() {
187+
printf 'T\torders\t3\t1\nC\torders\tcleanup.policy\tcompact,delete\n' > "$STATE/s3_object"
188+
}
189+
190+
@test "restore of an absent topic creates it and brackets a comma-valued config" {
191+
write_backup_object
192+
: > "$STATE/topics" # nothing live -> create path
193+
MODE=restore ARTIFACT_URI="s3://bkt/ns/app/run/kafka-metadata.txt" \
194+
S3_ENDPOINT="https://s3.example.org" run bash "$SCRIPT"
195+
[ "$status" -eq 0 ]
196+
grep -qxF "create orders 3" "$STATE/actions"
197+
# comma value must be bracket-grouped so kafka-configs does not split it.
198+
grep -qxF "orders cleanup.policy=[compact,delete]" "$STATE/configs_applied"
199+
}
200+
201+
@test "restore fails loudly on a replication-factor mismatch" {
202+
printf 'T\torders\t3\t2\n' > "$STATE/s3_object" # backup wants RF 2
203+
printf 'orders\n' > "$STATE/topics" # live exists
204+
echo "Topic: orders PartitionCount: 3 ReplicationFactor: 1" > "$STATE/desc.orders"
205+
MODE=restore ARTIFACT_URI="s3://bkt/ns/app/run/kafka-metadata.txt" \
206+
S3_ENDPOINT="https://s3.example.org" run bash "$SCRIPT"
207+
[ "$status" -ne 0 ]
208+
[[ "$output" == *"replication factor differs"* ]]
209+
}
210+
211+
@test "restore aborts when the broker is unreachable (fail-closed --list)" {
212+
write_backup_object
213+
MODE=restore LIST_FAIL=1 ARTIFACT_URI="s3://bkt/ns/app/run/kafka-metadata.txt" \
214+
S3_ENDPOINT="https://s3.example.org" run bash "$SCRIPT"
215+
[ "$status" -ne 0 ]
216+
# Must not have routed the unreachable broker into --create.
217+
[ ! -f "$STATE/actions" ]
218+
}
219+
220+
# ---- cleanup --------------------------------------------------------------
221+
222+
@test "cleanup treats 204 as success" {
223+
MODE=cleanup DELETE_CODE=204 ARTIFACT_URI="s3://bkt/ns/app/run/kafka-metadata.txt" \
224+
S3_ENDPOINT="https://s3.example.org" run bash "$SCRIPT"
225+
[ "$status" -eq 0 ]
226+
}
227+
228+
@test "cleanup treats a 500 as a retryable failure" {
229+
MODE=cleanup DELETE_CODE=500 ARTIFACT_URI="s3://bkt/ns/app/run/kafka-metadata.txt" \
230+
S3_ENDPOINT="https://s3.example.org" run bash "$SCRIPT"
231+
[ "$status" -ne 0 ]
232+
}

0 commit comments

Comments
 (0)