Skip to content

Commit 4878250

Browse files
committed
fix(e2e): wait for the redis replica link instead of sampling it once
The redis-tls topology check read INFO replication from both pods once and failed the suite when a replica reported role:slave without master_link_status:up. That value is legitimately in motion there. Redis reports the link up only in REPL_STATE_CONNECTED, while the operator's readiness script admits a replica on two different predicates -- master_host off 127.0.0.1, and sync not in the transfer phase -- so every handshake state between the two is ready and unlinked, and a dropped link re-enters that window. Ready replicas carry no evidence about the link at all, which is more than the step's own comment allowed for and makes its claim of determinism wrong. Poll the topology to a 60s deadline instead, and on expiry report each pod's observed state along with how long it waited. Nothing models the link declaratively -- the RedisFailover CRD defines no status, so there is no condition to assert on and the poll carries the marker the e2e conventions require. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent 79602e9 commit 4878250

1 file changed

Lines changed: 54 additions & 30 deletions

File tree

‎hack/e2e-chainsaw/redis/chainsaw-test.yaml‎

Lines changed: 54 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -264,7 +264,17 @@ spec:
264264
metadata:
265265
name: redis-secure.ca-cert
266266
(metadata.labels."internal.cozystack.io/tenantresource" != 'true'): true
267-
# Deterministic once the asserts above have passed, so no retry.
267+
# No retry around this script. The replica link it reads is settled by
268+
# none of the asserts above and carries its own bounded wait below. The
269+
# whole step measured 3.0s on run 34711755303, so link_wait is wide
270+
# against what it costs today. That wait draws link_wait plus one poll
271+
# interval plus a read pair at the k() cap worst case, since the deadline
272+
# is tested before the sleep and never after, and the op ceiling covers
273+
# that on top of reads ahead of it at the cost that measurement shows,
274+
# not at their caps, whose sum exceeds any ceiling this step could carry.
275+
# Sized generously on purpose: unused margin costs a passing run nothing,
276+
# while a ceiling that expires first kills the shell and takes the wait's
277+
# own failure message with it.
268278
#
269279
# Every redis-cli below runs under the container's own `timeout`. That
270280
# inner cap, not the op timeout, is what keeps a dial that never answers
@@ -276,7 +286,7 @@ spec:
276286
# on the sentinel's busybox and 124 on the data pods' coreutils, and
277287
# neither is zero, which is all these checks read.
278288
- script:
279-
timeout: 3m
289+
timeout: 5m
280290
content: |
281291
set -eu
282292
@@ -425,37 +435,51 @@ spec:
425435
exit 1 ;;
426436
esac
427437
428-
# readyReplicas above witnesses the bootstrap and stops there. A pod
429-
# still replicating from the 127.0.0.1 the operator's initial config
430-
# names never goes ready, so two ready replicas mean the operator
431-
# promoted a master and linked the other over its own TLS dials --
432-
# but the readiness script exits 0 for anything reporting
433-
# role:master, so a link that breaks afterwards is invisible to it.
434-
# Assert the topology directly, and assert the sentinels moved off
435-
# the bootstrap address, which nothing above covers at all.
436-
masters=0
437-
linked=0
438-
for i in 0 1; do
439-
if ! info=$(k -n "$NAMESPACE" exec "rfr-redis-secure-$i" -c redis -- \
440-
sh -c 'REDISCLI_AUTH="$REDIS_PASSWORD" timeout 15 redis-cli --tls --cacert /tls/ca.crt -h localhost -p 6379 info replication'); then
441-
echo "rfr-redis-secure-$i did not answer INFO replication within the 15s cap" >&2
438+
# readyReplicas above witnesses nothing about the link. Redis reports
439+
# master_link_status:up only in REPL_STATE_CONNECTED, while the
440+
# operator's readiness script passes a replica on two other
441+
# predicates -- master_host off 127.0.0.1, and sync not in the
442+
# transfer phase -- so the whole handshake between them is ready and
443+
# unlinked, and a dropped link re-enters it. The topology is thus in
444+
# motion here and one sampling would decide on it. Wait for it to
445+
# settle, then assert the sentinels moved off the bootstrap address,
446+
# which nothing above covers at all.
447+
link_wait=60
448+
started=$(date +%s)
449+
deadline=$(( started + link_wait ))
450+
while :; do
451+
masters=0
452+
linked=0
453+
seen=
454+
for i in 0 1; do
455+
if ! info=$(k -n "$NAMESPACE" exec "rfr-redis-secure-$i" -c redis -- \
456+
sh -c 'REDISCLI_AUTH="$REDIS_PASSWORD" timeout 15 redis-cli --tls --cacert /tls/ca.crt -h localhost -p 6379 info replication'); then
457+
echo "rfr-redis-secure-$i did not answer INFO replication within the 15s cap" >&2
458+
exit 1
459+
fi
460+
case "$info" in
461+
*role:master*) masters=$((masters + 1)); state=master ;;
462+
*role:slave*)
463+
case "$info" in
464+
*master_link_status:up*) linked=$((linked + 1)); state=linked ;;
465+
*) state=replicating-no-link ;;
466+
esac
467+
;;
468+
*) state=no-replication-role ;;
469+
esac
470+
seen="$seen rfr-redis-secure-$i=$state"
471+
done
472+
if [ "$masters" = "1" ] && [ "$linked" = "1" ]; then break; fi
473+
if [ "$(date +%s)" -ge "$deadline" ]; then
474+
echo "replication did not settle on one master and one linked replica after $(( $(date +%s) - started ))s:$seen" >&2
442475
exit 1
443476
fi
444-
case "$info" in
445-
*role:master*) masters=$((masters + 1)) ;;
446-
*role:slave*)
447-
case "$info" in
448-
*master_link_status:up*) linked=$((linked + 1)) ;;
449-
*) echo "rfr-redis-secure-$i replicates with no link up" >&2; exit 1 ;;
450-
esac
451-
;;
452-
*) echo "rfr-redis-secure-$i reports no replication role" >&2; exit 1 ;;
453-
esac
477+
# TODO(e2e-replace-fixed-timeouts): nothing models the link -- the
478+
# readiness probe does not read master_link_status, and the
479+
# RedisFailover CRD defines no status at all, so there is no
480+
# condition to assert on instead of this poll.
481+
sleep 5
454482
done
455-
if [ "$masters" != "1" ] || [ "$linked" != "1" ]; then
456-
echo "expected one master and one linked replica, got $masters and $linked" >&2
457-
exit 1
458-
fi
459483
460484
sentinel=$(k -n "$NAMESPACE" get pod \
461485
-l app.kubernetes.io/component=sentinel,app.kubernetes.io/name=redis-secure \

0 commit comments

Comments
 (0)