Skip to content

Commit b89a18b

Browse files
androndoclaude
andcommitted
test(e2e): sharpen round-trip diagnostics and make teardown settle
Recommendations from the third review pass (which returned LGTM; none of these blocked, all three are cheap and make the failure paths honest). 1. The round-trip's catch named nothing. An unnamed `describe HelmRelease` in tenant-root dumps every release on the root tenant of a live cluster and buries the two that matter, so name them (mariadb-2-backup-roundtrip names its release for the same reason). Raised podLogs tail 100 -> 200, since the sidecar logs every action and an S3 error sits well back from the tail by the time a step fails, and added the Keeper Pods: chkeeper.yaml labels them `app: <release>-keeper`, so the existing app.kubernetes.io/instance selectors missed them entirely — and a source ClickHouse that never reaches a ready replica is usually Keeper failing to form a quorum, which explains itself only there. 2. SSL_CERT_DIR is now `/etc/ssl/cozy-s3-ca:/etc/ssl/certs`. The single-directory form works — Go reads the default bundle FILE regardless, which is what the earlier measurement showed — but that left the public-CA half of the contract resting on a file this image happens to ship. Naming the system directory costs nothing and survives a base-image bump that moves the bundle; verified against public AWS S3 that both forms still verify (403 auth, no x509). The mount path stays the bare directory, and the unit test now pins both halves separately — it caught the over-broad edit that had briefly made mountPath a colon list. 3. cleanup.sh returned while the uninstalls it triggered were still running, so the next suite (etcd) could start with helm-controller still draining this one, and the pre-clean could hand step 04 an application whose predecessor was still going away. It now waits for both ClickHouse releases and the Bucket to actually disappear, and — since the script deliberately runs without `set -e` so one absent resource cannot abandon the rest — records those failures and exits non-zero, rather than reporting a teardown that did not settle as success. wait_deleted polls `get` instead of using `kubectl wait --for=delete`, which errors on an already-absent resource: the normal case here, and indistinguishable from a real failure. Sized the two cleanup step budgets to 12m for the same reason the run step is 60m: cleanup.sh's waits now sum to 540s, which the previous 8m did not clear. The Keeper-sized-from-the-server-resources bug the review found outside this diff is filed as #3487. Assisted-By: Claude <[email protected]> Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Signed-off-by: Andrey Kolkov <[email protected]>
1 parent 7331ecb commit b89a18b

5 files changed

Lines changed: 97 additions & 7 deletions

File tree

‎examples/backups/clickhouse/00-helpers.sh‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -185,6 +185,35 @@ wait_sts_ready() {
185185
done
186186
}
187187

188+
# Wait until a namespaced resource is really gone.
189+
#
190+
# `kubectl wait --for=delete` is not used: it errors out when the resource is
191+
# already absent, which is the normal case on a clean namespace (cleanup.sh is
192+
# idempotent and runs as a pre-clean too), and swallowing that error would also
193+
# swallow a genuine failure. Polling `get` treats "already gone" and "gone now"
194+
# as the same success.
195+
wait_deleted() {
196+
local resource_type="$1"
197+
local resource_name="$2"
198+
local timeout="${3:-300}"
199+
local elapsed=0
200+
201+
while true; do
202+
if ! kubectl -n "$NAMESPACE" get "$resource_type" "$resource_name" >/dev/null 2>&1; then
203+
[[ $elapsed -gt 0 ]] && log_success "$resource_type/$resource_name is gone"
204+
return 0
205+
fi
206+
if [[ $elapsed -ge $timeout ]]; then
207+
log_error "Timeout waiting for $resource_type/$resource_name to be deleted; it is still present after ${timeout}s:"
208+
kubectl -n "$NAMESPACE" get "$resource_type" "$resource_name" -o wide >&2 || true
209+
return 1
210+
fi
211+
[[ $elapsed -eq 0 ]] && log_substep "Waiting for $resource_type/$resource_name to be deleted..."
212+
sleep 5
213+
elapsed=$((elapsed + 5))
214+
done
215+
}
216+
188217
# Run a SQL statement against the ClickHouse cluster of the given app instance.
189218
# Args: <release-name> <sql>
190219
# Note: the Cozystack ClickHouse RD prefixes Helm release names with

‎examples/backups/clickhouse/cleanup.sh‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,11 +15,32 @@ kubectl -n "$NAMESPACE" delete backup "$BACKUPJOB_NAME" --ignore-not-found
1515
kubectl -n "$NAMESPACE" delete clickhouse "$CLICKHOUSE_RESTORE_NAME" --ignore-not-found
1616
kubectl -n "$NAMESPACE" delete clickhouse "$CLICKHOUSE_NAME" --ignore-not-found
1717
kubectl -n "$NAMESPACE" delete bucket "$BUCKET_NAME" --ignore-not-found
18+
19+
# `kubectl delete` on these returns as soon as deletionTimestamp is set, while
20+
# the HelmRelease uninstall behind each one (a ClickHouseInstallation, a Keeper
21+
# installation and their PVCs) is still running. Returning here has two costs:
22+
# the next suite starts while helm-controller workers are still draining this
23+
# one, and on the pre-clean path step 04 can re-apply an application whose
24+
# predecessor has not finished uninstalling. So wait for the objects to actually
25+
# go, with teeth — a teardown that silently does not settle is what leaks into
26+
# later tests.
27+
# This script deliberately runs without `set -e` so one absent resource cannot
28+
# abandon the rest of the teardown, which means the waits have to record their
29+
# own failure and surface it at the end.
30+
teardown_rc=0
31+
wait_deleted clickhouse "$CLICKHOUSE_RESTORE_NAME" 180 || teardown_rc=1
32+
wait_deleted clickhouse "$CLICKHOUSE_NAME" 180 || teardown_rc=1
33+
wait_deleted bucket "$BUCKET_NAME" 180 || teardown_rc=1
1834
# The S3 endpoint CA copied out of the seaweedfs CA Secret by step 03. The
1935
# source Secret in S3_CA_NAMESPACE is the platform's and is never touched.
2036
kubectl -n "$NAMESPACE" delete secret "$CH_CA_SECRET_NAME" --ignore-not-found
2137
rm -f "$SCRIPT_DIR/.bucket-info.env"
2238
kubectl delete backupclass "$BACKUPCLASS_NAME" --ignore-not-found
2339
kubectl delete altinity.strategy.backups.cozystack.io "$STRATEGY_NAME" --ignore-not-found
2440

41+
if [[ $teardown_rc -ne 0 ]]; then
42+
log_error "Cleanup did not settle: see the resources reported above. Failing rather than leaving a half-uninstalled release for the next test to trip over."
43+
exit 1
44+
fi
45+
2546
log_success "Cleanup complete."

‎hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml‎

Lines changed: 32 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -263,9 +263,19 @@ spec:
263263
skipDelete: true
264264
catch:
265265
- events: {}
266+
# Name the HelmReleases. tenant-root is the root tenant of a live e2e cluster,
267+
# so an unnamed describe here dumps every HelmRelease on it and buries the two
268+
# that matter (mariadb-2-backup-roundtrip names its release for the same
269+
# reason). BackupJob/RestoreJob/Altinity stay unnamed — there are only ever a
270+
# couple, and the names are the harness's to choose.
266271
- describe:
267272
apiVersion: helm.toolkit.fluxcd.io/v2
268273
kind: HelmRelease
274+
name: clickhouse-clickhouse-test
275+
- describe:
276+
apiVersion: helm.toolkit.fluxcd.io/v2
277+
kind: HelmRelease
278+
name: clickhouse-clickhouse-restore
269279
- describe:
270280
apiVersion: backups.cozystack.io/v1alpha1
271281
kind: BackupJob
@@ -275,13 +285,26 @@ spec:
275285
- describe:
276286
apiVersion: strategy.backups.cozystack.io/v1alpha1
277287
kind: Altinity
288+
# tail 200, not 100: the sidecar logs each backup/restore action, so an S3
289+
# error can sit well back from the tail by the time the step fails.
278290
- podLogs:
279291
selector: app.kubernetes.io/instance=clickhouse-clickhouse-test
280-
tail: 100
292+
tail: 200
281293
# The to-copy leg (step 07) restores into a second release; capture its Pod
282294
# logs too so a failure isolated to that leg keeps its diagnostics.
283295
- podLogs:
284296
selector: app.kubernetes.io/instance=clickhouse-clickhouse-restore
297+
tail: 200
298+
# Keeper Pods carry a different label shape (chkeeper.yaml labels them
299+
# app: <release>-keeper, not app.kubernetes.io/instance), so the selectors
300+
# above miss them entirely. A source ClickHouse that never reaches a ready
301+
# replica is usually Keeper failing to form a quorum, and that explains itself
302+
# only here.
303+
- podLogs:
304+
selector: app=clickhouse-clickhouse-test-keeper
305+
tail: 100
306+
- podLogs:
307+
selector: app=clickhouse-clickhouse-restore-keeper
285308
tail: 100
286309
steps:
287310
- name: run
@@ -293,7 +316,11 @@ spec:
293316
# pass without a real backup ever running. cleanup.sh is idempotent
294317
# (--ignore-not-found), so this is a no-op on a clean namespace.
295318
- script:
296-
timeout: 8m
319+
# cleanup.sh now waits for the two ClickHouse releases and the Bucket to
320+
# actually go (3 x 180s = 540s), so the step budget has to clear that sum
321+
# for the same reason the run step does: an inner wait names the resource
322+
# that would not delete, a SIGKILL at the op timeout names nothing.
323+
timeout: 12m
297324
content: |
298325
set -eu
299326
cd ../../..
@@ -338,7 +365,9 @@ spec:
338365
# and the cluster-scoped Altinity strategy + BackupClass.
339366
- description: backup-flow cleanup
340367
script:
341-
timeout: 8m
368+
# 12m for the same arithmetic as the pre-clean above (540s of deletion
369+
# waits inside cleanup.sh).
370+
timeout: 12m
342371
content: |
343372
set -eu
344373
cd ../../..

‎packages/apps/clickhouse/templates/clickhouse.yaml‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -328,9 +328,18 @@ spec:
328328
S3_DISABLE_CERT_VERIFICATION is the third option and is
329329
rejected on purpose: it drops verification entirely rather
330330
than trusting one more CA.
331+
332+
/etc/ssl/certs is listed explicitly alongside our directory.
333+
Go would still read the default bundle FILE
334+
(/etc/ssl/certs/ca-certificates.crt) even if SSL_CERT_DIR
335+
named only our dir — that is what the measurement above
336+
showed — but that leaves the public-CA half of the contract
337+
resting on a file this image happens to ship. Naming the
338+
system directory costs nothing and survives a base-image
339+
bump that moves the bundle.
331340
*/}}
332341
- name: SSL_CERT_DIR
333-
value: /etc/ssl/cozy-s3-ca
342+
value: /etc/ssl/cozy-s3-ca:/etc/ssl/certs
334343
{{- end }}
335344
resources:
336345
requests:

‎packages/apps/clickhouse/tests/backup_test.yaml‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -151,10 +151,12 @@ tests:
151151
# altinity/clickhouse-backup:2.7.4).
152152
- equal:
153153
path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="SSL_CERT_DIR")].value
154-
value: /etc/ssl/cozy-s3-ca
154+
value: /etc/ssl/cozy-s3-ca:/etc/ssl/certs
155155
- notExists:
156156
path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="AWS_CA_BUNDLE")]
157-
# The mount path must equal SSL_CERT_DIR or the tool reads an empty dir.
157+
# The mount path must be SSL_CERT_DIR's FIRST entry (a mountPath is a
158+
# single directory, never a colon list) or the tool scans an empty dir.
159+
# The second entry, /etc/ssl/certs, is the image's own system dir.
158160
- equal:
159161
path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].volumeMounts[?(@.name=="backup-s3-ca")].mountPath
160162
value: /etc/ssl/cozy-s3-ca
@@ -209,7 +211,7 @@ tests:
209211
asserts:
210212
- equal:
211213
path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="SSL_CERT_DIR")].value
212-
value: /etc/ssl/cozy-s3-ca
214+
value: /etc/ssl/cozy-s3-ca:/etc/ssl/certs
213215
- equal:
214216
path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_BUCKET")].valueFrom.secretKeyRef.name
215217
value: cozy-backups-creds

0 commit comments

Comments
 (0)