Skip to content

Commit 89c7ea0

Browse files
authored
fix(e2e): hold replicated-lane LINSTOR cleanup to a pre-suite baseline and delete kafka suite PVCs (cozystack#4651)
## What this PR does This fixes two cleanup failures in the kubernetes suites. The nightly runs failed `kubernetes-previous` and `kubernetes-latest` in the replicated storage lane. PR runs fail `kubernetes-latest` with `LINSTOR pools did not return to their pre-suite baseline within 300s`. The replicated lane asked for too much free space at cleanup. It wanted 90 GiB free on every LINSTOR node, and volumes from earlier suites kept srv2 at 83 GiB, so both suites failed without leaking anything (nightly run 36659663201). A replicated suite still needs those 90 GiB for its workers and CDI scratch volumes, so the floor is now checked before the suite starts. If a node is short there, the suite fails with the per-node table and an error that blames the sandbox, not the suite. Cleanup in both lanes now compares against a baseline recorded before the suite creates anything, and that baseline is no longer free bytes. On a node whose pools are all thick ZFS, `linstor sp l` does not ask the satellite for free space. The controller fetches it live only for nodes with thin pools and otherwise returns the value cached at the satellite's last report. In LINSTOR v1.33.3 that is `FreeCapacityFetcherProto.assembleRequests` filtering on `hasThinPools`, with `CtrlStorPoolListApiCallHandler` falling back to `getFreeCapacityLastUpdated`. The last report can be taken while `zfs destroy` is still freeing blocks in the background. A cleanup that had removed everything then kept polling a frozen figure 0.4 to 0.9 GiB short and failed after 300s. A byte tolerance does not fix that, because the residue is already close to the smallest volume a suite is known to have leaked (1 GiB). The baseline is now the set of ZFS datasets under each node's LINSTOR pool. It is read with `zfs list` inside that node's `linstor-satellite` container. The zpool name comes from the pool's `StorDriver/StorPoolName` property, because it is `data` on QEMU nodes and `data-srvN` in container mode. Cleanup waits until no dataset outside the baseline remains. A destroyed dataset leaves the namespace when `zfs destroy` returns, whether or not its blocks are still being freed, so background freeing no longer fails the check. A real leak, like the 21 GiB scratch volume left on srv3 in an earlier run, is reported by node and dataset name instead of a KiB difference. A node whose pool cannot be read fails the check. A run that fails before it records a baseline no longer compares against the previous suite's file. Every read the new probe makes runs under `timeout -k 5 30`, so a hung controller or satellite fails one poll and the wait still ends with its failure message. The kafka suite deleted its apps but never their PVCs. The chart keeps them on purpose with `deleteClaim: false`. One run left 35 GiB in `tenant-test`, 20 GiB of it on srv2, and the backup round trip left its two clusters' volumes in `tenant-root` the same way. Each test now deletes its clusters' PVCs by `strimzi.io/cluster` after the apps are gone and waits for them. The `kafka` test does it from a step cleanup, so it also runs on a failed run. The tests cover both fixes. `hack/linstor-storage-mode_test.bats` has cases for a clean cleanup, a volume still listed on the first poll and gone on the next while the cached free capacity stays short, a leaked volume, a node that cannot be read, a hung satellite read, a missing or stale baseline, and the replicated-lane floor. Its fake runs the real controller-side `jq` filter against a LINSTOR-shaped listing. `hack/kafka-suite-pvc-cleanup_test.bats` checks that each cluster's PVC delete is in the suite and runs after the app cleanup. One stub line goes into `hack/run-kubernetes-talos-diagnostics_test.bats`, because that test calls `cozy_cleanup` with no baseline and only checks the order of the deletes. The new guards were mutation-tested. Some limits stay as they are. The local lane waits for dataset names to disappear, not for the freed space, and has no free-space floor, so a background destroy can still be freeing blocks when the next suite starts. The 90 GiB precondition in the replicated lane still reads the cached free capacity, and its comment says so. The two `kubectl exec` reads in `cozy_wait_linstor_pool_free` were there before this PR and still have no time limit of their own. Two related problems stay out of this PR. The `kafka-metadata` suite leaks PVCs the same way, tracked in cozystack#4641. The stale comment in `examples/backups/kafka/cleanup.sh` and the missing PVC pre-clean in the round trip are in cozystack#4649. Closes cozystack#4617, closes cozystack#4616, closes cozystack#4653 ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(e2e): the kubernetes suites' LINSTOR cleanup compares each node's ZFS datasets against a pre-suite baseline instead of cached free capacity, the replicated lane checks its 90 GiB floor before the suite starts, and the kafka suite deletes the PVCs its clusters keep. ```
2 parents bc400df + 200a7ff commit 89c7ea0

5 files changed

Lines changed: 612 additions & 120 deletions

File tree

‎hack/e2e-chainsaw/_lib/run-kubernetes.sh‎

Lines changed: 113 additions & 90 deletions
Original file line numberDiff line numberDiff line change
@@ -448,9 +448,13 @@ cozy_wait_schedulable_node() {
448448
# autoPlace=3 places one replica per node) plus two 21 GiB CDI scratch
449449
# PVCs (worst case both landing on the same node via the local
450450
# storageClass) yields a ~82 GiB per-satellite peak footprint; 90 GiB
451-
# default threshold covers that with margin. The wait is bounded and
452-
# authoritative: cleanup propagates its timeout so the next suite cannot start
453-
# on storage whose reclamation was never observed.
451+
# default threshold covers that with margin. That makes it a precondition of
452+
# the replicated suite rather than a cleanup criterion: volumes earlier suites
453+
# legitimately keep can hold a node below it after this suite reclaimed
454+
# everything it used, which only the pre-suite baseline can tell apart. It reads
455+
# the free capacity LINSTOR caches for thick pools (see
456+
# _cozy_linstor_pool_datasets), so it can understate a pool whose last destroy
457+
# was still freeing blocks when the satellite reported.
454458
cozy_wait_linstor_pool_free() {
455459
_min_free_gib="${1:-90}"
456460
_timeout="${2:-300}"
@@ -496,113 +500,124 @@ cozy_wait_linstor_pool_free() {
496500
done
497501
}
498502

499-
# Capture the local lane's per-node free-capacity baseline after stale tenant
500-
# cleanup and before creating this suite's workers. Chainsaw executes `try` and
501-
# `finally` in separate shells, so the small state file is the hand-off between
502-
# them. An absolute threshold cannot describe this lane: persistent platform
503-
# PVCs legitimately leave one pool far below 90 GiB, while all three sparse
504-
# pools also share one runner filesystem beneath ZFS.
503+
_cozy_linstor_pool_baseline_file() {
504+
printf '%s\n' "${COZY_LINSTOR_POOL_BASELINE_FILE:-_out/e2e-kubernetes-linstor-pool-baseline}"
505+
}
506+
507+
# Print every ZFS dataset under each node's LINSTOR storage pool as
508+
# `node dataset` rows, the pool's own root included, so a node whose pool holds
509+
# no volume still shows up as read. The names come from `zfs list` on the
510+
# satellite rather than from LINSTOR's free capacity: on a node with only thick
511+
# pools `linstor sp l` serves the figure cached at the satellite's last report,
512+
# which can be taken while `zfs destroy` is still freeing blocks in the
513+
# background, so a pool can read hundreds of MiB short of its baseline until
514+
# some later report with nothing left on it. A destroyed dataset's name is gone
515+
# by the time `zfs destroy` returns. The zpool is read off the pool's
516+
# `StorDriver/StorPoolName` property, under which LINSTOR stores the zpool of
517+
# both ZFS drivers, because it differs by lane: `data` on QEMU nodes,
518+
# `data-srvN` in container mode, where every node's pool is imported into one
519+
# shared kernel. Every read carries its own ceiling, so a wedged controller or
520+
# satellite fails the poll within 35s instead of holding the caller for good.
521+
_cozy_linstor_pool_datasets() {
522+
_pools=$(timeout -k 5 30 kubectl -n cozy-linstor exec deploy/linstor-controller -- sh -c '
523+
linstor --machine-readable sp l 2>/dev/null |
524+
jq -r "first | .[] | select(.provider_kind | test(\"^ZFS\")) | \"\(.node_name) \(.props[\"StorDriver/StorPoolName\"] // \"\")\""
525+
' 2>/dev/null) || return 1
526+
[ -n "$_pools" ] || return 1
527+
_satellites=$(timeout -k 5 30 kubectl -n cozy-linstor get pods -l app.kubernetes.io/component=linstor-satellite \
528+
-o jsonpath='{range .items[*]}{.spec.nodeName} {.metadata.name}{"\n"}{end}' 2>/dev/null) || return 1
529+
while read -r _node _zpool; do
530+
[ -n "$_node" ] && [ -n "$_zpool" ] || return 1
531+
_pod=$(printf '%s\n' "$_satellites" | awk -v node="$_node" '$1 == node { print $2; exit }')
532+
[ -n "$_pod" ] || return 1
533+
_datasets=$(timeout -k 5 30 kubectl -n cozy-linstor exec "$_pod" --container=linstor-satellite -- \
534+
zfs list -H -o name -t filesystem,volume,snapshot -r "$_zpool" </dev/null 2>/dev/null) || return 1
535+
[ -n "$_datasets" ] || return 1
536+
printf '%s\n' "$_datasets" | awk -v node="$_node" '{ print node " " $0 }'
537+
done <<EOF
538+
$_pools
539+
EOF
540+
}
541+
542+
# Capture the per-node dataset baseline after stale tenant cleanup and before
543+
# creating this suite's workers. Chainsaw executes `try` and `finally` in
544+
# separate shells, so the small state file is the hand-off between them.
505545
cozy_capture_linstor_pool_baseline() {
506-
_baseline_file="${COZY_LINSTOR_POOL_BASELINE_FILE:-_out/e2e-kubernetes-linstor-pool-baseline}"
546+
_baseline_file=$(_cozy_linstor_pool_baseline_file)
507547
_baseline_dir=${_baseline_file%/*}
508548
[ "$_baseline_dir" != "$_baseline_file" ] || _baseline_dir=.
509549
mkdir -p "$_baseline_dir"
510550
: >"$_baseline_file"
511551

512-
_baseline=$(kubectl -n cozy-linstor exec deploy/linstor-controller -- sh -c '
513-
linstor --machine-readable sp l 2>/dev/null |
514-
jq -r "first | .[] | select((.provider_kind | test(\"^ZFS\")) and .free_capacity != null) | \"\(.node_name):\(.free_capacity)\"" |
515-
sort
516-
' 2>/dev/null) || _baseline=""
552+
_baseline=$(_cozy_linstor_pool_datasets) || _baseline=""
517553
if [ -z "$_baseline" ]; then
518-
echo "» ERROR: could not record the local LINSTOR pool baseline; physical ZFS reclamation would be unverifiable" >&2
554+
echo "» ERROR: could not record the LINSTOR pool baseline; reclamation of this suite's volumes would be unverifiable" >&2
519555
return 1
520556
fi
521-
_baseline_nodes=$(printf '%s\n' "$_baseline" | awk -F: 'NF == 2 { print $1 }' | sort | tr '\n' ' ')
557+
_baseline_nodes=$(printf '%s\n' "$_baseline" | awk '{ print $1 }' | sort -u | tr '\n' ' ')
522558
if [ "$_baseline_nodes" != "srv1 srv2 srv3 " ] \
523-
|| ! cozy_linstor_pools_at_baseline "$_baseline" "$_baseline" 0; then
524-
echo "» ERROR: local LINSTOR pool baseline is incomplete or malformed (expected numeric rows for srv1, srv2 and srv3): ${_baseline:-<empty>}" >&2
559+
|| printf '%s\n' "$_baseline" | awk 'NF != 2 { bad = 1 } END { exit !bad }'; then
560+
echo "» ERROR: LINSTOR pool baseline is incomplete or malformed (expected 'node dataset' rows for srv1, srv2 and srv3): ${_baseline}" >&2
525561
return 1
526562
fi
527563
printf '%s\n' "$_baseline" >"$_baseline_file"
528-
echo "» local LINSTOR pool baseline recorded:"
529-
printf '%s\n' "$_baseline" | sed 's/^/ baseline-free-kib: /'
564+
echo "» LINSTOR pool baseline recorded:"
565+
printf '%s\n' "$_baseline" | sed 's/^/ baseline-dataset: /'
530566
}
531567

532-
# Pure comparison used by the local-pool reclamation loop. Both captures contain
533-
# `node:free_capacity_kib` rows. Every node present in the baseline must still be
534-
# present and may fall short only by the small caller-provided metadata tolerance.
535-
# The 512 MiB default covers the ~0.27 GiB free-capacity drift measured between
536-
# two otherwise-clean container suites, while still rejecting the smallest known
537-
# leaked test volume (the 1 GiB ClickHouse keeper PVC).
538-
cozy_linstor_pools_at_baseline() {
539-
_baseline_rows="$1"
540-
_current_rows="$2"
541-
_tolerance_kib="${3:-524288}"
542-
[ -n "$_baseline_rows" ] && [ -n "$_current_rows" ] || return 1
543-
case "$_tolerance_kib" in
544-
'' | *[!0-9]*) return 1 ;;
545-
esac
546-
547-
while IFS=: read -r _baseline_node _baseline_kib; do
548-
[ -n "$_baseline_node" ] && [ -n "$_baseline_kib" ] || return 1
549-
_current_kib=$(printf '%s\n' "$_current_rows" | awk -F: -v node="$_baseline_node" '$1 == node { print $2; exit }')
550-
[ -n "$_current_kib" ] || return 1
551-
case "$_baseline_kib" in '' | *[!0-9]*) return 1 ;; esac
552-
case "$_current_kib" in '' | *[!0-9]*) return 1 ;; esac
553-
if [ "$(( _current_kib + _tolerance_kib ))" -lt "$_baseline_kib" ]; then
554-
return 1
555-
fi
556-
done <<EOF
557-
$_baseline_rows
558-
EOF
559-
return 0
568+
# Pure comparison of two `node dataset` listings. Prints `new <node> <dataset>`
569+
# for each dataset the current listing has and the baseline does not, and
570+
# `unread <node>` for each baseline node the current listing lacks, and fails
571+
# when it printed anything. A baseline dataset that is gone is not a finding:
572+
# the suite may have reclaimed something an earlier one left.
573+
cozy_linstor_pool_new_datasets() {
574+
[ -n "$1" ] && [ -n "$2" ] || return 1
575+
{
576+
printf '%s\n' "$1"
577+
printf '%s\n' '--'
578+
printf '%s\n' "$2"
579+
} | awk '
580+
!current && $0 == "--" { current = 1; next }
581+
!current { seen[$0] = 1; base[$1] = 1; next }
582+
{ read_nodes[$1] = 1; if (!($0 in seen)) { print "new " $0; bad = 1 } }
583+
END {
584+
for (node in base) if (!(node in read_nodes)) { print "unread " node; bad = 1 }
585+
exit bad
586+
}'
560587
}
561588

562589
cozy_wait_linstor_pool_baseline() {
563590
_timeout="${1:-300}"
564-
_tolerance_kib="${2:-524288}"
565-
_baseline_file="${COZY_LINSTOR_POOL_BASELINE_FILE:-_out/e2e-kubernetes-linstor-pool-baseline}"
591+
_baseline_file=$(_cozy_linstor_pool_baseline_file)
592+
# Returns 2 here and 1 on a timeout, so the caller can tell an unchecked
593+
# reclamation from a failed one.
566594
if [ ! -s "$_baseline_file" ]; then
567-
echo "» ERROR: no local LINSTOR pool baseline was recorded; physical ZFS reclamation is unknown" >&2
568-
return 1
595+
echo "» ERROR: no LINSTOR pool baseline was recorded for this suite, so it stopped before creating its tenant; reclamation was not checked" >&2
596+
return 2
569597
fi
570598
_baseline=$(cat "$_baseline_file")
571599
_deadline=$(( $(date +%s) + _timeout ))
572600
_current=""
601+
_residue=""
573602
while :; do
574-
_current=$(kubectl -n cozy-linstor exec deploy/linstor-controller -- sh -c '
575-
linstor --machine-readable sp l 2>/dev/null |
576-
jq -r "first | .[] | select((.provider_kind | test(\"^ZFS\")) and .free_capacity != null) | \"\(.node_name):\(.free_capacity)\"" |
577-
sort
578-
' 2>/dev/null) || _current=""
579-
if cozy_linstor_pools_at_baseline "$_baseline" "$_current" "$_tolerance_kib"; then
580-
echo "» local LINSTOR pools returned to their pre-suite baseline"
603+
_current=$(_cozy_linstor_pool_datasets) || _current=""
604+
if [ -n "$_current" ] && _residue=$(cozy_linstor_pool_new_datasets "$_baseline" "$_current"); then
605+
echo "» LINSTOR pools hold no dataset beyond their pre-suite baseline"
581606
return 0
582607
fi
583608
if [ "$(date +%s)" -ge "$_deadline" ]; then
584-
echo "» ERROR: local LINSTOR pools did not return to their pre-suite baseline within ${_timeout}s; a worker or CDI scratch volume may still occupy ZFS space" >&2
585-
printf '%s\n' "$_baseline" | sed 's/^/ baseline-free-kib: /' >&2
586-
printf '%s\n' "${_current:-<the LINSTOR pool probe returned nothing>}" | sed 's/^/ current-free-kib: /' >&2
609+
echo "» ERROR: LINSTOR pools did not return to their pre-suite baseline within ${_timeout}s; a worker or CDI scratch volume may still occupy ZFS space" >&2
610+
if [ -z "$_current" ]; then
611+
echo " linstor-pool: the dataset probe returned nothing" >&2
612+
else
613+
printf '%s\n' "$_residue" | sed 's/^/ linstor-pool: /' >&2
614+
fi
587615
return 1
588616
fi
589617
sleep 5
590618
done
591619
}
592620

593-
# The absolute 90 GiB threshold describes only the replicated lane's DRBD
594-
# teardown. The local lane instead waits for the exact capacity it had before
595-
# this Kubernetes suite, which detects its own leaked worker/scratch volumes
596-
# without requiring persistent platform volumes to disappear.
597-
cozy_wait_linstor_pool_reclaimed() {
598-
_storage_class=$(cozy_e2e_storage_class) || return 1
599-
if [ "$_storage_class" = local ]; then
600-
cozy_wait_linstor_pool_baseline "${2:-300}" 524288
601-
return $?
602-
fi
603-
cozy_wait_linstor_pool_free "${1:-90}" "${2:-300}"
604-
}
605-
606621
# Unconditional cleanup hook, invoked from the kubernetes-* tests' Chainsaw
607622
# `finally` block (which always runs, after any crust-gather `catch`). The tenant
608623
# Kubernetes CR is applied imperatively (kubectl) inside run_kubernetes_test, so
@@ -666,13 +681,15 @@ cozy_cleanup() {
666681
echo "» ERROR: cleanup has no test name; scoped tenant drain cannot be verified" >&2
667682
cleanup_failed=1
668683
fi
669-
# In the replicated lane, wait for the DRBD-safe absolute threshold. In the
670-
# local lane, wait for this suite's saved per-node baseline instead: that still
671-
# catches delayed physical teardown without requiring platform PVCs to vanish.
672-
if ! cozy_wait_linstor_pool_reclaimed 90 300; then
673-
echo "» ERROR: LINSTOR capacity did not return to the pre-suite level" >&2
674-
cleanup_failed=1
684+
# Wait for this suite's saved per-node baseline: that catches a volume LINSTOR
685+
# has not destroyed yet, or never will, without requiring volumes the suite
686+
# did not create to vanish. Blocks a destroy is still freeing do not count.
687+
local pool_rc=0
688+
cozy_wait_linstor_pool_baseline 300 || pool_rc=$?
689+
if [ "$pool_rc" -eq 1 ]; then
690+
echo "» ERROR: LINSTOR pools did not return to the pre-suite level" >&2
675691
fi
692+
[ "$pool_rc" -eq 0 ] || cleanup_failed=1
676693
return "$cleanup_failed"
677694
}
678695

@@ -5042,6 +5059,9 @@ run_kubernetes_test() {
50425059
# The previous-version suite leaves it empty because OIDC is feature
50435060
# coverage, not a compatibility matrix that justifies another upgrade.
50445061
local enable_oidc="${5:-}"
5062+
# The previous suite's baseline must not survive into this suite's cleanup
5063+
# when this run fails before recording its own.
5064+
rm -f -- "$(_cozy_linstor_pool_baseline_file)" || return 1
50455065
local storage_class
50465066
storage_class=$(cozy_e2e_storage_class) || return 1
50475067
local k8s_version
@@ -5075,15 +5095,18 @@ run_kubernetes_test() {
50755095
return 1
50765096
fi
50775097

5078-
# The local lane cannot use the replicated lane's absolute free-capacity
5079-
# threshold because persistent platform volumes already put one pool below it.
5098+
# The replicated suite needs the absolute floor free before it starts (see
5099+
# cozy_wait_linstor_pool_free); the local lane's platform volumes already put
5100+
# one pool below it, so it has no such precondition.
5101+
if [ "$storage_class" = replicated ] && ! cozy_wait_linstor_pool_free 90 300; then
5102+
echo "» ERROR: ${test_name} has created nothing yet and its LINSTOR pool precondition failed: either a pool is below the 90 GiB this suite needs, held by earlier suites or the platform, or the pools could not be read. Either way it is an environment failure rather than this suite's" >&2
5103+
return 1
5104+
fi
50805105
# Save the post-stale-cleanup state for the separate Chainsaw `finally` shell;
5081-
# cleanup later verifies that this suite returned every node to these figures.
5082-
if [ "$storage_class" = local ]; then
5083-
if ! cozy_capture_linstor_pool_baseline; then
5084-
echo "cannot verify local LINSTOR reclamation without a complete baseline" >&2
5085-
return 1
5086-
fi
5106+
# cleanup later verifies that this suite left no dataset outside this listing.
5107+
if ! cozy_capture_linstor_pool_baseline; then
5108+
echo "cannot verify LINSTOR reclamation without a complete baseline" >&2
5109+
return 1
50875110
fi
50885111

50895112
# Compose the optional ouroboros addon block. Indentation matches the

‎hack/e2e-chainsaw/kafka/chainsaw-test.yaml‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,18 @@ spec:
3131
try:
3232
- apply:
3333
file: kafka.yaml
34+
# Both node pools set deleteClaim: false, so the volumes outlive the app
35+
# and would hold their LINSTOR space for every later suite. Chainsaw runs
36+
# a step's cleanup ops after this step's cleaner has deleted the app, on a
37+
# failed run as well. A volume a terminating broker still mounts finishes
38+
# deleting once that pod is gone, and the delete waits for that.
39+
cleanup:
40+
- delete:
41+
ref:
42+
apiVersion: v1
43+
kind: PersistentVolumeClaim
44+
labels:
45+
strimzi.io/cluster: kafka-test
3446

3547
- name: wait-ready
3648
try:
@@ -299,6 +311,20 @@ spec:
299311
set -eu
300312
cd ../../..
301313
NAMESPACE="$NAMESPACE" examples/backups/kafka/cleanup.sh
314+
# cleanup.sh leaves the volumes, which deleteClaim: false keeps past each
315+
# app's uninstall, so they go here, after both apps are gone.
316+
- delete:
317+
ref:
318+
apiVersion: v1
319+
kind: PersistentVolumeClaim
320+
labels:
321+
strimzi.io/cluster: kafka-kafka-test
322+
- delete:
323+
ref:
324+
apiVersion: v1
325+
kind: PersistentVolumeClaim
326+
labels:
327+
strimzi.io/cluster: kafka-kafka-restore
302328

303329
# cleanup.sh above deleted both Kafka apps and waited for them to go, so their
304330
# uninstalls have run, and the node pools the migration hook created outside
Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
#!/usr/bin/env bats
2+
# The kafka chart sets deleteClaim: false on both node pools, so deleting a
3+
# Kafka app leaves its broker and controller PVCs behind on purpose. The e2e
4+
# suite has to delete them itself, after the app is gone (a mounted PVC cannot
5+
# finish deleting), or every later suite in the sandbox runs short of LINSTOR
6+
# space. These tests read hack/e2e-chainsaw/kafka/chainsaw-test.yaml, because
7+
# running it needs a cluster.
8+
9+
SUITE=hack/e2e-chainsaw/kafka/chainsaw-test.yaml
10+
11+
@test "kafka test deletes its PVCs in the cleanup of the step that applies the app" {
12+
# Chainsaw runs a step's cleanup ops after that step's own cleaner has
13+
# deleted what the step applied, on a passing and a failing run alike.
14+
clusters=$(yq 'select(.metadata.name == "kafka") | .spec.steps[]
15+
| select([.try[]? | select(.apply.file == "kafka.yaml")] | length > 0)
16+
| .cleanup[]?.delete.ref | select(.apiVersion == "v1" and .kind == "PersistentVolumeClaim")
17+
| .labels."strimzi.io/cluster"' "$SUITE")
18+
19+
if ! printf '%s\n' "$clusters" | grep -qx kafka-test; then
20+
echo "kafka test does not delete the PVCs of cluster kafka-test after deleting the app" >&2
21+
return 1
22+
fi
23+
}
24+
25+
@test "kafka backup round-trip deletes each cluster's PVCs after its app cleanup" {
26+
# examples/backups/kafka creates two apps, kafka-test and kafka-restore; the
27+
# chart names their Strimzi clusters with a kafka- prefix.
28+
# One line per finally op, in source order, which is the order they run in.
29+
# yq emits the facts and awk names the op, because if/then/else only reached
30+
# yq in v4.54.1 and older ones reject the expression. The test document is
31+
# cut out first: in a multi-document file yq also evaluates the array literal
32+
# against the documents the select drops, and prints a line for each.
33+
sequence=$(yq 'select(.metadata.name == "kafka-2-backup-roundtrip")' "$SUITE" \
34+
| yq '.spec.steps[].finally[]?
35+
| [((.script.content // "") | test("examples/backups/kafka/cleanup.sh") | tostring),
36+
((.delete.ref.apiVersion // "") + "/" + (.delete.ref.kind // "")),
37+
(.delete.ref.labels."strimzi.io/cluster" // "")] | join(" ")' \
38+
| awk '$1 == "true" { print "app-cleanup"; next }
39+
$2 == "v1/PersistentVolumeClaim" { print "pvc:" $3; next }
40+
{ print "other" }')
41+
42+
for cluster in kafka-kafka-test kafka-kafka-restore; do
43+
if ! printf '%s\n' "$sequence" | awk 'seen; /^app-cleanup$/ { seen = 1 }' | grep -qx "pvc:${cluster}"; then
44+
echo "kafka-2-backup-roundtrip does not delete the PVCs of cluster ${cluster} after cleanup.sh: ${sequence}" >&2
45+
return 1
46+
fi
47+
done
48+
}

0 commit comments

Comments
 (0)