Skip to content

Commit 9e3ed1f

Browse files
authored
fix(backups): stop the redis backup demo from reading empty values (#4399)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does The redis backup round-trip suite went red on PRs that do not touch redis, each time on an empty read (#4395). The report snapshots of those runs point to two separate causes, and I fix both here. The empty marker reads came from the demo helper that ran redis-cli in a throwaway `kubectl run -i --rm` Pod. kubectl gets that Pod's stdout over an attach, and containerd's attach only carries output written after the attach is registered. kubectl falls back to the Pod log only when the attach fails. A redis-cli that finishes in that gap comes back as an empty string with exit 0. In the failed runs, the master's log shows the key present, and sentinel logged no failover, yet the GET read ''. `redis_cmd` now runs redis-cli by `kubectl exec` in a sentinel Pod, which already has it. An exec'd process gets its own pipes, so its output can't be lost that way, and its exit code comes back. The password now goes over stdin rather than a Pod env var. The missing-strategy failure was real: `cozy-default-redis` did not exist yet. It gets rendered behind a lookup of the platform bucket, so it appears only after a forced Helm upgrade. helm-controller holds that upgrade while `cozystack-basics` is not Ready, and the e2e install's late platform patches keep it not Ready for several minutes. The demo read the object once. It now waits for the object with `kubectl wait --for=create` and a 15 minute bound. That wait fails right away on anything other than NotFound. The demo's reads also stop sending stderr to `/dev/null`, so the next failure names its cause. `hack/redis-backup-demo-helpers.bats` covers `redis_cmd` with stubbed kubectl and redis-cli: it reads through exec, keeps the password out of argv, sends no AUTH when there is no Secret, and fails loudly when the exec fails or sentinel has no master. Closes #4395 ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> No UI changes. ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: The diff touches only `examples/backups/redis/` and a new unit test under `hack/`. No entry in the trigger map covers either. ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(backups): the Redis backup demo reads its keys through `kubectl exec`, so a short-lived Pod can no longer drop its output, and it waits for the platform's `cozy-default-redis` strategy to appear instead of failing on its first read. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Redis backup and restore commands now report Kubernetes and Secret read errors instead of hiding them, while still treating a missing password Secret as empty. * Redis commands pass authentication securely and fail clearly when Sentinel cannot identify a master. * Backup demos wait for the Redis backup strategy to appear and provide clearer error messages. * **Tests** * Added coverage for Redis authentication handling, command failures, and missing Sentinel masters. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 339cffe + 01feb36 commit 9e3ed1f

4 files changed

Lines changed: 152 additions & 34 deletions

File tree

‎examples/backups/redis/00-helpers.sh‎

Lines changed: 34 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -47,13 +47,10 @@ export S3_CA_SECRET="${S3_CA_SECRET:-seaweedfs-ca-cert}"
4747
export S3_CA_NAMESPACE="${S3_CA_NAMESPACE:-tenant-root}"
4848
export S3_CA_KEY="${S3_CA_KEY:-ca.crt}"
4949
export MARKER_KEY="${MARKER_KEY:-sentinel:marker}"
50-
# Single-token value: the redis_cmd helper word-splits REDIS_ARGS, so a value
51-
# with spaces would break. A UUID-shaped token is enough to prove the exact
52-
# bytes round-tripped through object storage.
50+
# Single-token value: redis_cmd strips all whitespace from the reply, so a
51+
# value with spaces would not compare equal. A UUID-shaped token is enough to
52+
# prove the exact bytes round-tripped through object storage.
5353
export MARKER_VALUE="${MARKER_VALUE:-roundtrip-4f1c9a2b}"
54-
# redis:*-alpine ships redis-cli (SENTINEL discovery + data ops); the seed /
55-
# verify helpers run it as a throwaway Pod via `kubectl run`.
56-
export REDIS_CLI_IMAGE="${REDIS_CLI_IMAGE:-redis:7.4-alpine}"
5754

5855
log_info() { echo -e "${BLUE}i${NC} $*" >&2; }
5956
log_success() { echo -e "${GREEN}OK${NC} $*" >&2; }
@@ -80,7 +77,7 @@ wait_for_field() {
8077
[[ -n "$namespace" ]] && ns_flag=(-n "$namespace")
8178
while true; do
8279
local current
83-
current=$(kubectl get "$resource_type" "$resource_name" "${ns_flag[@]}" -o jsonpath="$jsonpath" 2>/dev/null || true)
80+
current=$(kubectl get "$resource_type" "$resource_name" "${ns_flag[@]}" -o jsonpath="$jsonpath" || true)
8481
[[ "$current" == "$desired" ]] && { log_success "$resource_type/$resource_name reached '$desired'"; return 0; }
8582
[[ -n "$fail_value" && "$current" == "$fail_value" ]] && { log_error "$resource_type/$resource_name reached terminal '$current' (expected '$desired')"; return 1; }
8683
(( elapsed >= timeout )) && { log_error "Timeout waiting for $resource_type/$resource_name (current: '$current', expected: '$desired')"; return 1; }
@@ -97,9 +94,9 @@ wait_hr_ready() {
9794
while true; do
9895
if kubectl -n "$NAMESPACE" get hr "$name" >/dev/null 2>&1; then
9996
local ready stalled
100-
ready=$(kubectl -n "$NAMESPACE" get hr "$name" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || true)
97+
ready=$(kubectl -n "$NAMESPACE" get hr "$name" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' || true)
10198
[[ "$ready" == "True" ]] && { log_success "HelmRelease/$name is Ready"; return 0; }
102-
stalled=$(kubectl -n "$NAMESPACE" get hr "$name" -o jsonpath='{.status.conditions[?(@.type=="Stalled")].status}' 2>/dev/null || true)
99+
stalled=$(kubectl -n "$NAMESPACE" get hr "$name" -o jsonpath='{.status.conditions[?(@.type=="Stalled")].status}' || true)
103100
[[ "$stalled" == "True" ]] && { log_error "HelmRelease/$name is Stalled (terminal)"; return 1; }
104101
fi
105102
(( elapsed >= timeout )) && { log_error "Timeout waiting for HelmRelease/$name to become Ready"; return 1; }
@@ -111,30 +108,40 @@ wait_hr_ready() {
111108
# The cozystack redis chart names its RedisFailover (hence the operator's rfr-/
112109
# rfs- Services and the -auth Secret) redis-<app>, mirroring the strategy driver
113110
# that derives the same base from .Release.Name. So app X's password lives in
114-
# redis-X-auth. Empty when auth is disabled.
111+
# redis-X-auth. The chart renders that Secret only with authEnabled, so its
112+
# absence reads as no password; any other error still fails the read.
115113
redis_pw() {
116-
kubectl -n "$NAMESPACE" get secret "redis-$1-auth" -o jsonpath='{.data.password}' 2>/dev/null | base64 -d
114+
kubectl -n "$NAMESPACE" get secret "redis-$1-auth" --ignore-not-found -o jsonpath='{.data.password}' | base64 -d
117115
}
118116

119117
# Run a redis-cli command against an application's CURRENT master. Master
120-
# discovery goes through the operator's sentinel Service (rfs-redis-<app>, master
121-
# name "mymaster"), so a write always lands on the writable node even after a
122-
# failover. Args after the app name are passed verbatim to redis-cli.
118+
# discovery asks a sentinel of the operator's rfs-redis-<app> Deployment
119+
# (master name "mymaster"), so a write always lands on the writable node even
120+
# after a failover. Args after the app name are passed verbatim to redis-cli.
123121
# redis_cmd redis-test SET sentinel:marker hello
122+
#
123+
# The command runs by `kubectl exec` in a sentinel Pod, whose image carries
124+
# redis-cli. A throwaway `kubectl run -i` Pod is not usable here: its stdout
125+
# comes back over an attach, which only carries what the container writes after
126+
# the attach is established, and kubectl falls back to the Pod log only when
127+
# the attach fails. A redis-cli that finishes in between comes back as an
128+
# empty reply with exit 0. An exec'd process writes into pipes of its own, so
129+
# its output cannot be missed that way.
130+
#
131+
# The password travels on stdin rather than in argv, which the exec request
132+
# URL carries into the API server's audit log. kubectl's and redis-cli's
133+
# stderr stay attached, so a failed read says why instead of reading as ''.
124134
redis_cmd() {
125-
local app="$1"; shift
126-
kubectl -n "$NAMESPACE" run "redis-cli-$RANDOM" \
127-
--image="$REDIS_CLI_IMAGE" --restart=Never --rm -i --quiet \
128-
--env="REDIS_PW=$(redis_pw "$app")" \
129-
--env="APP=redis-$app" \
130-
--env="REDIS_ARGS=$*" \
131-
--command -- sh -c '
132-
addr=$(redis-cli -h "rfs-$APP" -p 26379 sentinel get-master-addr-by-name mymaster)
135+
local app="$1" pw; shift
136+
pw=$(redis_pw "$app")
137+
printf '%s\n' "$pw" | kubectl -n "$NAMESPACE" exec -i "deploy/rfs-redis-$app" -c sentinel -- sh -c '
138+
IFS= read -r pw || true
139+
addr=$(redis-cli -p 26379 sentinel get-master-addr-by-name mymaster) || exit 1
133140
h=$(echo "$addr" | sed -n 1p); p=$(echo "$addr" | sed -n 2p)
134-
[ -n "$h" ] && [ -n "$p" ] || { echo "no master from sentinel rfs-$APP" >&2; exit 1; }
135-
auth=""; [ -n "$REDIS_PW" ] && auth="-a $REDIS_PW --no-auth-warning"
136-
exec redis-cli -h "$h" -p "$p" $auth $REDIS_ARGS
137-
' 2>/dev/null | tr -d '[:space:]'
141+
[ -n "$h" ] && [ -n "$p" ] || { echo "sentinel knows no master for mymaster" >&2; exit 1; }
142+
[ -z "$pw" ] || export REDISCLI_AUTH="$pw"
143+
exec redis-cli -h "$h" -p "$p" "$@"
144+
' sh "$@" | tr -d '[:space:]'
138145
}
139146

140147
# Block until the RedisFailover has an elected master reachable through sentinel.
@@ -143,7 +150,7 @@ wait_redis_master() {
143150
log_substep "Waiting for '$app' master election via sentinel..."
144151
while true; do
145152
local h
146-
h=$(redis_cmd "$app" PING 2>/dev/null || true)
153+
h=$(redis_cmd "$app" PING || true)
147154
[[ "$h" == "PONG" ]] && { log_success "'$app' master is reachable"; return 0; }
148155
(( elapsed >= timeout )) && { log_error "Timeout waiting for '$app' master"; return 1; }
149156
sleep 5

‎examples/backups/redis/03-restore-in-place.sh‎

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -12,10 +12,9 @@ print_header "Step 03: In-place restore from '${BACKUPJOB_NAME}'"
1212

1313
log_substep "Corrupting the marker to simulate data loss..."
1414
redis_cmd "$REDIS_NAME" SET "$MARKER_KEY" "corrupted-do-not-keep" >/dev/null
15-
# Read the corruption back and assert it: redis_cmd swallows stderr, so a write
16-
# that silently failed (wrong password, sentinel not answering) would leave the
17-
# marker at its backed-up value and the final check would pass without a restore
18-
# having done anything.
15+
# Read the corruption back and assert it: a SET that did not land would leave
16+
# the marker at its backed-up value, and the final check would then pass
17+
# without the restore having done anything.
1918
corrupted=$(redis_cmd "$REDIS_NAME" GET "$MARKER_KEY")
2019
[[ "$corrupted" == "corrupted-do-not-keep" ]] || { log_error "corruption step did not take (got '${corrupted}'); aborting so the restore assertion cannot false-pass"; exit 1; }
2120

‎examples/backups/redis/run-all.sh‎

Lines changed: 14 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -72,7 +72,7 @@ if [[ -n "$S3_CA_SECRET" ]] \
7272
log_warning "S3 CA secret ${S3_CA_NAMESPACE}/${S3_CA_SECRET} not found; discovering the seaweedfs CA Certificate..."
7373
DISCOVERED_CA=$(kubectl -n "$S3_CA_NAMESPACE" get certificates.cert-manager.io \
7474
-l app.kubernetes.io/name=seaweedfs \
75-
-o jsonpath='{range .items[*]}{.spec.isCA}{" "}{.spec.secretName}{"\n"}{end}' 2>/dev/null \
75+
-o jsonpath='{range .items[*]}{.spec.isCA}{" "}{.spec.secretName}{"\n"}{end}' \
7676
| awk '$1=="true"{print $2; exit}' || true)
7777
if [[ -n "$DISCOVERED_CA" ]]; then
7878
log_success "Discovered seaweedfs CA secret ${S3_CA_NAMESPACE}/${DISCOVERED_CA}"
@@ -113,9 +113,20 @@ print_header "Step 00c: Create the Redis strategy + BackupClass"
113113
# Pod needs no package install at run time. Its presence also confirms the
114114
# platform default backups stack (backupstrategy-controller + CRDs) is
115115
# installed, without which the BackupJob below cannot reconcile at all.
116+
#
117+
# Wait for it rather than read it once. The strategy is rendered behind a
118+
# lookup of the platform bucket, so it exists only once a Helm upgrade has run
119+
# after that bucket was provisioned. The controller forces that upgrade, but
120+
# helm-controller holds it while any release in its dependsOn is not Ready, so
121+
# a platform change made just before this runs can keep it absent for minutes.
122+
# kubectl wait keeps polling through NotFound and fails on any other error.
123+
log_substep "Waiting for the platform's cozy-default-redis strategy..."
124+
kubectl wait --for=create redis.strategy.backups.cozystack.io/cozy-default-redis \
125+
--timeout=15m >/dev/null \
126+
|| { log_error "cozy-default-redis strategy did not appear: enable the platform default backups (backupstrategy-controller) before running this demo"; exit 1; }
116127
CLIENT_IMAGE=$(kubectl get redis.strategy.backups.cozystack.io cozy-default-redis \
117-
-o jsonpath='{.spec.template.spec.containers[?(@.name=="redis-backup")].image}' 2>/dev/null || true)
118-
[[ -n "$CLIENT_IMAGE" ]] || { log_error "cozy-default-redis strategy not found: enable the platform default backups (backupstrategy-controller) before running this demo"; exit 1; }
128+
-o jsonpath='{.spec.template.spec.containers[?(@.name=="redis-backup")].image}')
129+
[[ -n "$CLIENT_IMAGE" ]] || { log_error "cozy-default-redis strategy has no redis-backup container image"; exit 1; }
119130
log_substep "Reusing the platform strategy's client image: ${CLIENT_IMAGE}"
120131
subst redis-strategy.yaml | kubectl apply -f -
121132
kubectl apply -f "$SCRIPT_DIR/backupclass.yaml"
Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
#!/usr/bin/env bats
2+
# Unit tests for redis_cmd in examples/backups/redis/00-helpers.sh, the helper
3+
# the redis-2-backup-roundtrip suite reads and writes its marker key through.
4+
#
5+
# kubectl and redis-cli are stub executables on PATH. The kubectl stub accepts
6+
# exactly the calls the helper is expected to make and runs an exec'd script
7+
# locally with the redis-cli stub, so the in-Pod shell is exercised too. Any
8+
# other kubectl call, a throwaway `kubectl run` Pod among them, fails the test.
9+
#
10+
# The helpers are Bash and this runner is POSIX sh, so each case drives them
11+
# through `bash -c` under the same `set -euo pipefail` the demo scripts use.
12+
# The two helper functions close on an indented brace because the runner
13+
# rewrites a column-0 `}` into `return 0`, which would mask run_redis_cmd's
14+
# exit status.
15+
16+
make_stubs() {
17+
stub=$(mktemp -d)
18+
cat > "$stub/kubectl" <<'EOF'
19+
#!/bin/bash
20+
printf '%s\n' "$*" >> "$STUB_DIR/kubectl.argv"
21+
if [ "$*" = "-n tenant-root get secret redis-demo-auth --ignore-not-found -o jsonpath={.data.password}" ]; then
22+
printf '%s' "${STUB_PW_B64:-}"
23+
exit 0
24+
fi
25+
if [ "$1 $2 $3 $4 $5 $6 $7 $8 $9 ${10}" != "-n tenant-root exec -i deploy/rfs-redis-demo -c sentinel -- sh -c" ]; then
26+
echo "unexpected kubectl call: $*" >&2
27+
exit 97
28+
fi
29+
if [ -n "${STUB_EXEC_ERROR:-}" ]; then
30+
echo "$STUB_EXEC_ERROR" >&2
31+
exit 1
32+
fi
33+
shift 10
34+
script=$1
35+
shift
36+
exec sh -c "$script" "$@"
37+
EOF
38+
cat > "$stub/redis-cli" <<'EOF'
39+
#!/bin/sh
40+
if [ "$*" = "-p 26379 sentinel get-master-addr-by-name mymaster" ]; then
41+
[ -n "${STUB_NO_MASTER:-}" ] || printf '10.0.0.5\n6379\n'
42+
exit 0
43+
fi
44+
printf '%s|%s\n' "${REDISCLI_AUTH-unset}" "$*" >> "$STUB_DIR/redis-cli.calls"
45+
printf 'reply-value\n'
46+
EOF
47+
chmod +x "$stub/kubectl" "$stub/redis-cli"
48+
: > "$stub/kubectl.argv"
49+
: > "$stub/redis-cli.calls"
50+
}
51+
52+
run_redis_cmd() {
53+
STUB_DIR=$stub PATH="$stub:$PATH" NAMESPACE=tenant-root bash -c '
54+
set -euo pipefail
55+
. examples/backups/redis/00-helpers.sh
56+
redis_cmd demo "$@"
57+
' bash "$@"
58+
}
59+
60+
@test "redis_cmd reads the reply through exec and passes the password on stdin" {
61+
make_stubs
62+
export STUB_PW_B64
63+
STUB_PW_B64=$(printf 's3cret' | base64)
64+
out=$(run_redis_cmd GET sentinel:marker)
65+
[ "$out" = "reply-value" ]
66+
[ "$(cat "$stub/redis-cli.calls")" = "s3cret|-h 10.0.0.5 -p 6379 GET sentinel:marker" ]
67+
if grep -q s3cret "$stub/kubectl.argv"; then
68+
echo "the password reached kubectl's argv" >&2
69+
exit 1
70+
fi
71+
}
72+
73+
@test "redis_cmd sends no AUTH when the app has no password Secret" {
74+
make_stubs
75+
out=$(run_redis_cmd SET sentinel:marker v1)
76+
[ "$out" = "reply-value" ]
77+
[ "$(cat "$stub/redis-cli.calls")" = "unset|-h 10.0.0.5 -p 6379 SET sentinel:marker v1" ]
78+
}
79+
80+
@test "redis_cmd fails with kubectl's own error when the exec fails" {
81+
make_stubs
82+
err=$(mktemp)
83+
export STUB_EXEC_ERROR="error: unable to upgrade connection: container not found"
84+
if run_redis_cmd GET sentinel:marker 2>"$err"; then
85+
echo "a failed exec read as a successful empty reply" >&2
86+
exit 1
87+
fi
88+
grep -q 'unable to upgrade connection: container not found' "$err"
89+
}
90+
91+
@test "redis_cmd fails when the sentinel names no master" {
92+
make_stubs
93+
err=$(mktemp)
94+
export STUB_NO_MASTER=1
95+
if run_redis_cmd GET sentinel:marker 2>"$err"; then
96+
echo "a sentinel with no master read as a successful reply" >&2
97+
exit 1
98+
fi
99+
grep -q 'sentinel knows no master for mymaster' "$err"
100+
[ ! -s "$stub/redis-cli.calls" ]
101+
}

0 commit comments

Comments
 (0)