Skip to content

Commit a66907f

Browse files
authored
fix(backups): share the walkthrough wait helpers and fix the S3 preflight cleanup (cozystack#4635)
## What this PR does This fixes the S3 access preflight and a few backup walkthroughs, and moves the walkthrough wait helpers into one shared file. The access preflight now keeps its evidence and cleans up after itself. When `mc alias set` failed, it dropped the port-forward log, and now it prints it like the other exits do. The exit trap used to kill the port-forward before `mc rm`, so the delete always hit a closed port and the warning gave no reason. The delete now runs first, and the warning carries mc's own error, or says the delete timed out when mc was killed before writing one. A PUT that timed out (rc 124) may have landed, so it counts as uploaded. A PUT the server refused does not, so access that never went live gets no delete and no warning. The wait helpers were copied into every walkthrough and had drifted. `wait_for_field`, `wait_hr_ready` and `wait_deleted` had 12, 8 and 2 copies with 5, 3 and 2 different bodies. They now live in `examples/backups/_lib/wait-helpers.sh`, and each `00-helpers.sh` sources it. Call sites and wait budgets are unchanged. Where copies differed, the shared body keeps every diagnostic any copy had. What changes per walkthrough: - `wait_for_field` in every walkthrough but redis now shows kubectl's stderr, so a Forbidden or transport error is visible instead of reading as an empty field. `--ignore-not-found` keeps a resource that does not exist yet from printing an error on every poll. Redis already kept stderr. - `wait_for_field` in foundationdb, nats and vmi gains the optional 7th terminal-value argument. None of their callers pass it, so nothing changes at runtime. - vmi loses its progress dots. - `wait_hr_ready` in redis and kafka-metadata now prints every condition and the release history on Stalled or timeout. It also says whether the release never appeared, was deleted, or could not be looked up. - The kafka `wait_deleted` timeout message reads "it is still present" instead of "still present". `hack/backup-examples-shared-helpers.bats` goes red if a walkthrough defines one of these helpers again or stops providing them, and it pins both sides of the `wait_for_field` read: kubectl's reason for a failed read is shown, a missing resource prints nothing. The byte-identity pin between copies in `hack/mariadb-first-boot-waits.bats` is gone, and its errexit test runs through the mariadb walkthrough's helpers. The kafka topic-data demo had the empty-read race from cozystack#4400. `kafka_run` read topic state through a throwaway `kubectl run -i` Pod. It now execs into a long-lived CLI Pod that keeps the restricted security context, and `cleanup.sh` removes it by label. The Pod name comes from its own `KAFKA_TOPIC_CLI_POD`, so setting the kafka-metadata demo's `KAFKA_CLI_POD` does not move it. cozystack#4400 itself was already fixed on main for nats, kafka-metadata and rabbitmq. The nats and kafka-metadata demos could exec into a Pod they had just refused. They read CLI output inside `$(...)`, where bash does not apply errexit, so the ownership refusal was ignored. A failed read of the Pod's phase was treated as no Pod, and the demo went on to delete and replace it. Every setup step now returns on failure. Their "exec fails with kubectl's own error" tests could never fail: the runner's trace wrote the expected message into the same stderr the test grepped. The stub now prints the message itself. Each fix has a test that fails on main. For each fix I also reverted it and checked that only its own test goes red. One gap stays open outside this PR. `hack/select-e2e.sh` maps `examples/backups/<app>/` paths by app name, so a later PR that changes only `examples/backups/_lib/wait-helpers.sh` would select no suite. This PR is fine because it touches every walkthrough. The rule belongs to the select-e2e owners and is tracked in cozystack#4626. While reviewing I also found that the kafka README puts steps that create and exec into Pods under the tenant persona, and the tenant role can do neither. That is cozystack#4625, left for its own change. Left out on purpose, because they are outside what this diff changes: - The kafka topic demo and the kafka-metadata demo each carry a CLI Pod helper, and the kafka-metadata one runs without the restricted security context. - The walkthrough READMEs do not say that a directory now needs `../_lib` next to it. - The errexit test in `hack/mariadb-first-boot-waits.bats` loops over one file. I kept it so the rebase against cozystack#4471 stays mechanical. Closes cozystack#3757, closes cozystack#4243, closes cozystack#4244 ### Screenshots No UI changes. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(backups): the backup walkthroughs share one set of wait helpers that keep kubectl errors visible, the kafka topic-data demo no longer reads empty replies from a throwaway Pod, the nats and kafka-metadata demos no longer exec into a CLI Pod they refused, and the e2e S3 access preflight says why client setup or cleanup failed and deletes its test object reliably. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Kafka topic backup walkthroughs now reuse a named CLI Pod, with restricted security settings and checks to prevent using a Pod owned by another demo. * Backup walkthroughs can wait for resources to reach a desired state, Helm releases to become ready, or resources to be deleted. * **Bug Fixes** * Kafka and NATS walkthroughs now stop and report errors when CLI Pod lookups, creation, deletion, or command execution fail. * S3 preflight cleanup now attempts removal after a timed-out upload and reports cleanup failures, including timeouts. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents a9dc106 + 873d9f4 commit a66907f

22 files changed

Lines changed: 872 additions & 1142 deletions
Lines changed: 165 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,165 @@
1+
#!/bin/bash
2+
# Wait helpers shared by every backup walkthrough under examples/backups/.
3+
# Each walkthrough's 00-helpers.sh sources this file after defining the log_*
4+
# functions these call; wait_hr_ready and wait_deleted read $NAMESPACE.
5+
# hack/backup-examples-shared-helpers.bats fails if a walkthrough grows its own
6+
# copy again.
7+
8+
# Wait until a JSONPath value on a resource matches the desired string.
9+
#
10+
# The optional 7th argument is a TERMINAL value to bail on: a BackupJob or
11+
# RestoreJob that reaches Failed never reaches Succeeded, so polling out the
12+
# remaining budget only delays the report, lets the TTL reaper take the Job's
13+
# Pod log, and under Chainsaw risks the step being SIGKILLed at its op timeout
14+
# instead of failing cleanly.
15+
#
16+
# kubectl's stderr is left attached: a read that fails for RBAC or transport
17+
# reasons says so instead of reading as an empty value. --ignore-not-found
18+
# keeps a resource that does not exist yet, the usual case at the start of a
19+
# wait, from printing an error on every poll.
20+
wait_for_field() {
21+
local resource_type="$1"
22+
local resource_name="$2"
23+
local jsonpath="$3"
24+
local desired="$4"
25+
local namespace="${5:-}"
26+
local timeout="${6:-300}"
27+
local fail_value="${7:-}"
28+
29+
log_substep "Waiting for $resource_type/$resource_name $jsonpath to become '$desired'..."
30+
local elapsed=0
31+
local ns_flag=()
32+
[[ -n "$namespace" ]] && ns_flag=(-n "$namespace")
33+
34+
while true; do
35+
local current
36+
current=$(kubectl get "$resource_type" "$resource_name" "${ns_flag[@]}" --ignore-not-found -o jsonpath="$jsonpath" || true)
37+
if [[ "$current" == "$desired" ]]; then
38+
log_success "$resource_type/$resource_name reached '$desired'"
39+
return 0
40+
fi
41+
if [[ -n "$fail_value" && "$current" == "$fail_value" ]]; then
42+
log_error "$resource_type/$resource_name reached terminal '$current' (expected '$desired')"
43+
return 1
44+
fi
45+
if [[ $elapsed -ge $timeout ]]; then
46+
log_error "Timeout waiting for $resource_type/$resource_name (current: '$current', expected: '$desired')"
47+
return 1
48+
fi
49+
sleep 5
50+
elapsed=$((elapsed + 5))
51+
done
52+
}
53+
54+
# Wait for a HelmRelease to become Ready, with an existence backstop (the apps
55+
# controller creates the HR asynchronously, so a bare `kubectl wait` right after
56+
# `kubectl apply` races it) and a fail-fast on Stalled=True: a stalled HR has
57+
# exhausted its remediation retries and will never turn Ready, so polling to the
58+
# timeout only hides the real error.
59+
wait_hr_ready() {
60+
local name="$1"
61+
local timeout="${2:-300}"
62+
local elapsed=0
63+
local lookup state seen=0
64+
65+
log_substep "Waiting for HelmRelease/$name to become Ready..."
66+
while true; do
67+
# --ignore-not-found makes an absent release exit 0, so a non-zero exit
68+
# is kubectl failing to answer. Presence is read off the name line,
69+
# because a warning on stderr lands in the same string.
70+
if ! lookup=$(kubectl -n "$NAMESPACE" get hr "$name" --ignore-not-found -o name 2>&1); then
71+
state=error
72+
elif [[ $'\n'"$lookup"$'\n' != *"/$name"$'\n'* ]]; then
73+
state=absent
74+
else
75+
state=present
76+
seen=1
77+
local ready stalled
78+
ready=$(kubectl -n "$NAMESPACE" get hr "$name" \
79+
-o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' || true)
80+
if [[ "$ready" == "True" ]]; then
81+
log_success "HelmRelease/$name is Ready"
82+
return 0
83+
fi
84+
stalled=$(kubectl -n "$NAMESPACE" get hr "$name" \
85+
-o jsonpath='{.status.conditions[?(@.type=="Stalled")].status}' || true)
86+
if [[ "$stalled" == "True" ]]; then
87+
log_error "HelmRelease/$name is Stalled (terminal):"
88+
break
89+
fi
90+
fi
91+
if [[ $elapsed -ge $timeout ]]; then
92+
log_error "Timeout waiting for HelmRelease/$name to become Ready:"
93+
case "$state" in
94+
error)
95+
echo " could not look up HelmRelease/$name: $lookup" >&2
96+
# Seen earlier: the reads below may answer where this lookup
97+
# did not.
98+
[[ $seen -eq 1 ]] || return 1
99+
;;
100+
absent)
101+
if [[ $seen -eq 1 ]]; then
102+
echo " HelmRelease/$name was deleted from $NAMESPACE while waiting" >&2
103+
else
104+
echo " HelmRelease/$name never appeared in $NAMESPACE" >&2
105+
fi
106+
return 1
107+
;;
108+
esac
109+
break
110+
fi
111+
sleep 5
112+
elapsed=$((elapsed + 5))
113+
done
114+
# Reached on Stalled or on timeout. Conditions and history rather than the
115+
# Ready message alone: a failed install is retried on an interval, so Ready
116+
# may describe the retry rather than the failure behind it.
117+
kubectl -n "$NAMESPACE" get hr "$name" \
118+
-o jsonpath='{range .status.conditions[*]} {.type}={.status} ({.reason}): {.message}{"\n"}{end}' >&2 || true
119+
local hist
120+
if ! hist=$(kubectl -n "$NAMESPACE" get hr "$name" \
121+
-o jsonpath='{range .status.history[*]} history: {.status} {.chartVersion} {.lastDeployed}{"\n"}{end}'); then
122+
echo " history: kubectl could not read it" >&2
123+
elif [[ -n "${hist//[[:space:]]/}" ]]; then
124+
printf '%s\n' "$hist" >&2
125+
else
126+
# A release with no history prints nothing and exits 0; say so,
127+
# because a blank in a failure dump reads as a dump that broke.
128+
echo " history: (none recorded)" >&2
129+
fi
130+
return 1
131+
}
132+
133+
# Wait until a namespaced resource is really gone.
134+
#
135+
# `kubectl wait --for=delete` is not used: it errors out when the resource is
136+
# already absent, which is the normal case on a clean namespace (cleanup.sh is
137+
# idempotent and runs as a pre-clean too), and swallowing that error would also
138+
# swallow a genuine failure. Polling `get --ignore-not-found` treats "already
139+
# gone" and "gone now" as the same success (empty output, exit 0) while a
140+
# retrieval failure — RBAC, API-server, transport — is a non-zero exit that
141+
# must NOT be read as deletion, or cleanup settles on a false success and leaves
142+
# half-uninstalled resources for the next run. Such an error just keeps us
143+
# polling until the resource is confirmed gone or the timeout fires.
144+
wait_deleted() {
145+
local resource_type="$1"
146+
local resource_name="$2"
147+
local timeout="${3:-300}"
148+
local elapsed=0
149+
local got
150+
151+
while true; do
152+
if got=$(kubectl -n "$NAMESPACE" get "$resource_type" "$resource_name" --ignore-not-found 2>/dev/null) && [[ -z "$got" ]]; then
153+
[[ $elapsed -gt 0 ]] && log_success "$resource_type/$resource_name is gone"
154+
return 0
155+
fi
156+
if [[ $elapsed -ge $timeout ]]; then
157+
log_error "Timeout waiting for $resource_type/$resource_name to be deleted; it is still present after ${timeout}s:"
158+
kubectl -n "$NAMESPACE" get "$resource_type" "$resource_name" -o wide >&2 || true
159+
return 1
160+
fi
161+
[[ $elapsed -eq 0 ]] && log_substep "Waiting for $resource_type/$resource_name to be deleted..."
162+
sleep 5
163+
elapsed=$((elapsed + 5))
164+
done
165+
}

‎examples/backups/clickhouse/00-helpers.sh‎

Lines changed: 4 additions & 156 deletions
Original file line numberDiff line numberDiff line change
@@ -79,128 +79,10 @@ print_header() {
7979
echo -e "\n${MAGENTA}${BOLD}== $title ==${NC}\n" >&2
8080
}
8181

82-
# Wait until a JSONPath value on a resource matches the desired string.
83-
#
84-
# The optional 7th argument is a TERMINAL value to bail on: a BackupJob or
85-
# RestoreJob that reaches Failed will never reach Succeeded, so polling out the
86-
# remaining budget only delays the report and — when this runs under Chainsaw —
87-
# risks the step being SIGKILLed at its op timeout with nothing collected
88-
# instead of failing cleanly. Same contract as the postgres/mariadb helpers.
89-
wait_for_field() {
90-
local resource_type="$1"
91-
local resource_name="$2"
92-
local jsonpath="$3"
93-
local desired="$4"
94-
local namespace="${5:-}"
95-
local timeout="${6:-300}"
96-
local fail_value="${7:-}"
97-
98-
log_substep "Waiting for $resource_type/$resource_name $jsonpath to become '$desired'..."
99-
local elapsed=0
100-
local ns_flag=()
101-
[[ -n "$namespace" ]] && ns_flag=(-n "$namespace")
102-
103-
while true; do
104-
local current
105-
current=$(kubectl get "$resource_type" "$resource_name" "${ns_flag[@]}" -o jsonpath="$jsonpath" 2>/dev/null || true)
106-
if [[ "$current" == "$desired" ]]; then
107-
log_success "$resource_type/$resource_name reached '$desired'"
108-
return 0
109-
fi
110-
if [[ -n "$fail_value" && "$current" == "$fail_value" ]]; then
111-
log_error "$resource_type/$resource_name reached terminal '$current' (expected '$desired')"
112-
return 1
113-
fi
114-
if [[ $elapsed -ge $timeout ]]; then
115-
log_error "Timeout waiting for $resource_type/$resource_name (current: '$current', expected: '$desired')"
116-
return 1
117-
fi
118-
sleep 5
119-
elapsed=$((elapsed + 5))
120-
done
121-
}
122-
123-
# Wait for a HelmRelease to become Ready, failing fast on Stalled=True.
124-
#
125-
# `kubectl wait hr ... --for=condition=ready` cannot do either half of this: it
126-
# errors out if the HelmRelease does not exist yet (the app CR is reconciled
127-
# asynchronously, so the HR appears a moment after `kubectl apply` returns), and
128-
# it keeps waiting through a terminal Stalled — a chart that will never install
129-
# burns the whole timeout instead of reporting the reason. Same contract as the
130-
# postgres/mariadb helpers.
131-
wait_hr_ready() {
132-
local name="$1"
133-
local timeout="${2:-300}"
134-
local elapsed=0
135-
local lookup state seen=0
136-
137-
log_substep "Waiting for HelmRelease/$name to become Ready..."
138-
while true; do
139-
# --ignore-not-found makes an absent release exit 0, so a non-zero exit
140-
# is kubectl failing to answer. Presence is read off the name line,
141-
# because a warning on stderr lands in the same string.
142-
if ! lookup=$(kubectl -n "$NAMESPACE" get hr "$name" --ignore-not-found -o name 2>&1); then
143-
state=error
144-
elif [[ $'\n'"$lookup"$'\n' != *"/$name"$'\n'* ]]; then
145-
state=absent
146-
else
147-
state=present
148-
seen=1
149-
local ready stalled
150-
ready=$(kubectl -n "$NAMESPACE" get hr "$name" \
151-
-o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' || true)
152-
if [[ "$ready" == "True" ]]; then
153-
log_success "HelmRelease/$name is Ready"
154-
return 0
155-
fi
156-
stalled=$(kubectl -n "$NAMESPACE" get hr "$name" \
157-
-o jsonpath='{.status.conditions[?(@.type=="Stalled")].status}' || true)
158-
if [[ "$stalled" == "True" ]]; then
159-
log_error "HelmRelease/$name is Stalled (terminal):"
160-
break
161-
fi
162-
fi
163-
if [[ $elapsed -ge $timeout ]]; then
164-
log_error "Timeout waiting for HelmRelease/$name to become Ready:"
165-
case "$state" in
166-
error)
167-
echo " could not look up HelmRelease/$name: $lookup" >&2
168-
# Seen earlier: the reads below may answer where this lookup
169-
# did not.
170-
[[ $seen -eq 1 ]] || return 1
171-
;;
172-
absent)
173-
if [[ $seen -eq 1 ]]; then
174-
echo " HelmRelease/$name was deleted from $NAMESPACE while waiting" >&2
175-
else
176-
echo " HelmRelease/$name never appeared in $NAMESPACE" >&2
177-
fi
178-
return 1
179-
;;
180-
esac
181-
break
182-
fi
183-
sleep 5
184-
elapsed=$((elapsed + 5))
185-
done
186-
# Reached on Stalled or on timeout. Conditions and history rather than the
187-
# Ready message alone: a failed install is retried on an interval, so Ready
188-
# may describe the retry rather than the failure behind it.
189-
kubectl -n "$NAMESPACE" get hr "$name" \
190-
-o jsonpath='{range .status.conditions[*]} {.type}={.status} ({.reason}): {.message}{"\n"}{end}' >&2 || true
191-
local hist
192-
if ! hist=$(kubectl -n "$NAMESPACE" get hr "$name" \
193-
-o jsonpath='{range .status.history[*]} history: {.status} {.chartVersion} {.lastDeployed}{"\n"}{end}'); then
194-
echo " history: kubectl could not read it" >&2
195-
elif [[ -n "${hist//[[:space:]]/}" ]]; then
196-
printf '%s\n' "$hist" >&2
197-
else
198-
# A release with no history prints nothing and exits 0; say so,
199-
# because a blank in a failure dump reads as a dump that broke.
200-
echo " history: (none recorded)" >&2
201-
fi
202-
return 1
203-
}
82+
# wait_for_field, wait_hr_ready and wait_deleted live in one file shared by
83+
# every backup walkthrough, so a fix to one reaches all of them.
84+
# shellcheck source-path=SCRIPTDIR source=../_lib/wait-helpers.sh
85+
source "$(dirname "${BASH_SOURCE[0]}")/../_lib/wait-helpers.sh"
20486

20587
# Wait for a StatefulSet to report a ready replica, tolerating its asynchronous
20688
# creation (the ClickHouse operator creates it after the CHI is reconciled).
@@ -237,40 +119,6 @@ wait_sts_ready() {
237119
done
238120
}
239121

240-
# Wait until a namespaced resource is really gone.
241-
#
242-
# `kubectl wait --for=delete` is not used: it errors out when the resource is
243-
# already absent, which is the normal case on a clean namespace (cleanup.sh is
244-
# idempotent and runs as a pre-clean too), and swallowing that error would also
245-
# swallow a genuine failure. Polling `get --ignore-not-found` treats "already
246-
# gone" and "gone now" as the same success (empty output, exit 0) while a
247-
# retrieval failure — RBAC, API-server, transport — is a non-zero exit that
248-
# must NOT be read as deletion, or cleanup settles on a false success and leaves
249-
# half-uninstalled resources for the next run. Such an error just keeps us
250-
# polling until the resource is confirmed gone or the timeout fires.
251-
wait_deleted() {
252-
local resource_type="$1"
253-
local resource_name="$2"
254-
local timeout="${3:-300}"
255-
local elapsed=0
256-
local got
257-
258-
while true; do
259-
if got=$(kubectl -n "$NAMESPACE" get "$resource_type" "$resource_name" --ignore-not-found 2>/dev/null) && [[ -z "$got" ]]; then
260-
[[ $elapsed -gt 0 ]] && log_success "$resource_type/$resource_name is gone"
261-
return 0
262-
fi
263-
if [[ $elapsed -ge $timeout ]]; then
264-
log_error "Timeout waiting for $resource_type/$resource_name to be deleted; it is still present after ${timeout}s:"
265-
kubectl -n "$NAMESPACE" get "$resource_type" "$resource_name" -o wide >&2 || true
266-
return 1
267-
fi
268-
[[ $elapsed -eq 0 ]] && log_substep "Waiting for $resource_type/$resource_name to be deleted..."
269-
sleep 5
270-
elapsed=$((elapsed + 5))
271-
done
272-
}
273-
274122
# Run a SQL statement against the ClickHouse cluster of the given app instance.
275123
# Args: <release-name> <sql>
276124
# Note: the Cozystack ClickHouse RD prefixes Helm release names with

‎examples/backups/etcd/00-helpers.sh‎

Lines changed: 4 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -43,46 +43,10 @@ print_header() {
4343
echo -e "\n${MAGENTA}${BOLD}== $title ==${NC}\n" >&2
4444
}
4545

46-
# Wait until a JSONPath value on a resource matches the desired string.
47-
# Optional 7th arg is a TERMINAL failure value: once the field reaches it the
48-
# wait returns 1 immediately instead of polling to the timeout. BackupJob and
49-
# RestoreJob settle on a terminal phase=Failed that never becomes Succeeded, so
50-
# waiting the full timeout only wastes wall-clock AND risks the snapshot Job's
51-
# Pod being TTL-reaped (TTLSecondsAfterFinished=600s) before diagnostics can read
52-
# its log. Failing fast on the terminal phase keeps that log in reach.
53-
wait_for_field() {
54-
local resource_type="$1"
55-
local resource_name="$2"
56-
local jsonpath="$3"
57-
local desired="$4"
58-
local namespace="${5:-}"
59-
local timeout="${6:-300}"
60-
local fail_value="${7:-}"
61-
62-
log_substep "Waiting for $resource_type/$resource_name $jsonpath to become '$desired'..."
63-
local elapsed=0
64-
local ns_flag=()
65-
[[ -n "$namespace" ]] && ns_flag=(-n "$namespace")
66-
67-
while true; do
68-
local current
69-
current=$(kubectl get "$resource_type" "$resource_name" "${ns_flag[@]}" -o jsonpath="$jsonpath" 2>/dev/null || true)
70-
if [[ "$current" == "$desired" ]]; then
71-
log_success "$resource_type/$resource_name reached '$desired'"
72-
return 0
73-
fi
74-
if [[ -n "$fail_value" && "$current" == "$fail_value" ]]; then
75-
log_error "$resource_type/$resource_name reached terminal '$current' (expected '$desired')"
76-
return 1
77-
fi
78-
if [[ $elapsed -ge $timeout ]]; then
79-
log_error "Timeout waiting for $resource_type/$resource_name (current: '$current', expected: '$desired')"
80-
return 1
81-
fi
82-
sleep 5
83-
elapsed=$((elapsed + 5))
84-
done
85-
}
46+
# wait_for_field, wait_hr_ready and wait_deleted live in one file shared by
47+
# every backup walkthrough, so a fix to one reaches all of them.
48+
# shellcheck source-path=SCRIPTDIR source=../_lib/wait-helpers.sh
49+
source "$(dirname "${BASH_SOURCE[0]}")/../_lib/wait-helpers.sh"
8650

8751
# Ephemeral etcdctl pod name. Created on first etcdctl_exec call; reused
8852
# for subsequent calls in the same script run. cleanup.sh removes it.

0 commit comments

Comments
 (0)