Skip to content

Commit a79f800

Browse files
lexfreilexfrei
committed
test(foundationdb): widen late-converging status wait budgets
The reconciled, hasListenIPsForAllPods, fullReplication and healthy gates polled status fields with a 60s ceiling, while the strictly-earlier health.available gate had 300s. The operator stamps generations.reconciled only after every process holds its role for minimumUptimeSecondsForBounce (60s), making reconciled and the health fields that depend on it the last status fields to converge. The short ceiling closed the gate seconds before the field landed on a healthy, fully-reconciled cluster, causing intermittent jsonpath poll timeouts. Raise those four gates to 300s to match the health.available budget. The waits stay event-driven; only the ceilings widen. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent 29bf56f commit a79f800

1 file changed

Lines changed: 21 additions & 9 deletions

File tree

‎hack/e2e-apps/foundationdb.bats‎

Lines changed: 21 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -91,15 +91,27 @@ EOF
9191
# Validate status.desiredProcessGroups field
9292
timeout 60 sh -ec "until kubectl -n tenant-test get foundationdbclusters.apps.foundationdb.org foundationdb-$name -o jsonpath='{.status.desiredProcessGroups}' | grep -q '^[0-9][0-9]*$'; do sleep 10; done"
9393

94-
# Validate status.generations.reconciled field
95-
timeout 60 sh -ec "until kubectl -n tenant-test get foundationdbclusters.apps.foundationdb.org foundationdb-$name -o jsonpath='{.status.generations.reconciled}' | grep -q '^[0-9][0-9]*$'; do sleep 10; done"
96-
97-
# Validate status.hasListenIPsForAllPods field
98-
timeout 60 sh -ec "until kubectl -n tenant-test get foundationdbclusters.apps.foundationdb.org foundationdb-$name -o jsonpath='{.status.hasListenIPsForAllPods}' | grep -q 'true'; do sleep 10; done"
99-
100-
# Validate comprehensive status.health fields
101-
timeout 60 sh -ec "until kubectl -n tenant-test get foundationdbclusters.apps.foundationdb.org foundationdb-$name -o jsonpath='{.status.health.fullReplication}' | grep -q 'true'; do sleep 10; done"
102-
timeout 60 sh -ec "until kubectl -n tenant-test get foundationdbclusters.apps.foundationdb.org foundationdb-$name -o jsonpath='{.status.health.healthy}' | grep -q 'true'; do sleep 10; done"
94+
# Validate status.generations.reconciled field.
95+
# The operator only stamps generations.reconciled after every process has
96+
# held its role for minimumUptimeSecondsForBounce (60s, see
97+
# packages/apps/foundationdb/templates/cluster.yaml), so this is structurally
98+
# the last status field to converge. It needs at least the same budget as the
99+
# earlier health.available gate (300s), not the short 60s ceiling.
100+
timeout 300 sh -ec "until kubectl -n tenant-test get foundationdbclusters.apps.foundationdb.org foundationdb-$name -o jsonpath='{.status.generations.reconciled}' | grep -q '^[0-9][0-9]*$'; do sleep 10; done"
101+
102+
# Validate status.hasListenIPsForAllPods field. A bounce restarts fdbserver
103+
# in-place, so pods and their listen IPs do not churn and this usually settles
104+
# before the reconciled gate. Widened to 300s for a conservative ceiling
105+
# consistent with the surrounding gates; the 300s is headroom, not because
106+
# this field is as late-converging as generations.reconciled.
107+
timeout 300 sh -ec "until kubectl -n tenant-test get foundationdbclusters.apps.foundationdb.org foundationdb-$name -o jsonpath='{.status.hasListenIPsForAllPods}' | grep -q 'true'; do sleep 10; done"
108+
109+
# Validate comprehensive status.health fields. fullReplication and healthy
110+
# settle only once the cluster is fully reconciled (after the post-bounce
111+
# uptime gate), so they share the same 300s budget rather than the short 60s
112+
# ceiling.
113+
timeout 300 sh -ec "until kubectl -n tenant-test get foundationdbclusters.apps.foundationdb.org foundationdb-$name -o jsonpath='{.status.health.fullReplication}' | grep -q 'true'; do sleep 10; done"
114+
timeout 300 sh -ec "until kubectl -n tenant-test get foundationdbclusters.apps.foundationdb.org foundationdb-$name -o jsonpath='{.status.health.healthy}' | grep -q 'true'; do sleep 10; done"
103115

104116
# Verify security context is applied correctly (non-root user)
105117
storage_pod=$(kubectl -n tenant-test get pods -l foundationdb.org/fdb-cluster-name=foundationdb-$name,foundationdb.org/fdb-process-class=storage --no-headers | head -n1 | awk '{print $1}')

0 commit comments

Comments
 (0)