Skip to content

Commit 0dae3ca

Browse files
authored
fix(e2e): budget tenant scheduling separately from workload readiness (#3579)
## What this PR does The tenant backend Deployment in the kubernetes suites had one 300s budget to reach `condition=Available`, and two unrelated variable costs shared it. The first is scheduling. What the suite establishes before that point is two tenant nodes Ready, which is weaker than schedulable: a Ready node still carries `node.cilium.io/agent-not-ready` until the tenant cilium agent claims it, and a node the bringup has not finished with is `SchedulingDisabled`. In the run recorded in #3577 scheduling took 2m18s and 1m57s in the two suites, which left the image pull to finish inside what remained. It did not, and both suites reported the same `timed out waiting for the condition` for what were two different shortfalls. So this waits for a node that actually accepts a Pod before creating the workload, on its own budget and its own failure message. The gate encodes the scheduler's rule for a Pod that tolerates nothing (Ready, not unschedulable, no `NoSchedule` or `NoExecute` taint) over a `custom-columns` probe, prints the node table on both outcomes so a timeout names the taint that held it, and treats a failed probe as not-schedulable so an API blip cannot release it. On the happy path it adds no wall time: it spends the seconds the Pod would otherwise spend Pending, plus at most one 5s poll interval. On a failing run the two budgets stack, so a run that exhausts both now gives up at ~600s where it used to give up at 300s, inside the 40m Chainsaw script op the suites document as a ~25m bringup. The readiness wait keeps its 300s, now starting from a schedulable node, and dumps deployment, pod and event state when it runs out. The workload image is also pinned by digest. The tenant workers reach no Docker Hub mirror (`hack/e2e-talos-image-cache.yaml` serves the Talos worker OS disk image over HTTP and is not a registry mirror), so nginx is pulled from Docker Hub on every run either way. The digest does not take the pull off the critical path; it fixes what that pull returns instead of leaving a floating tag free to change size and content under a fixed deadline. Preloading the image would need infrastructure this tree does not have, and is not attempted here. Nothing will bump the pin: no renovate manager reads `hack/`, and the comment says so, because for a throwaway test workload the freeze is the point. What this does not claim: it removes two measured consumers from a fixed budget, both taken from the events of a single run. Whether that budget was the only thing making those suites red is not established from one run, so this is not offered as the fix for a red pipeline. `hack/run-kubernetes-schedulable_test.bats` covers the new logic: every branch of the predicate, the multi-node scan, the poll-again path, the deadline, and a probe that fails. Each test was verified by mutating the helper and checking that the intended test, and no other, went red. Note for whoever reviews alongside #3575: that branch mirrors `ghcr.io` for tenant worker pulls, not `docker.io`, so it does not change the Docker Hub pull described here. `git merge-tree` reports no conflict between the two, nor with #3548. Observed in #3577. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: The trigger map was walked against the diff. The change is confined to `hack/e2e-chainsaw/_lib/run-kubernetes.sh` and one new `hack/*.bats` unit test. It moves and renames nothing under `hack/`, changes no make target, and does not touch `hack/e2e-prepare-cluster.bats`, any package, any CRD or any namespace name, so none of the listed repositories are reached. ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved Kubernetes deployment readiness by waiting for a suitable, schedulable node before timing backend startup. * Added clearer diagnostics when nodes are unavailable or backend readiness fails. * Replaced the floating backend container image tag with a fixed, verified version. * **Tests** * Added coverage for node readiness, cordoning, taints, polling, timeouts, probe failures, and deployment sequencing. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 879d0f6 + 5542ae5 commit 0dae3ca

2 files changed

Lines changed: 432 additions & 3 deletions

File tree

‎hack/e2e-chainsaw/_lib/run-kubernetes.sh‎

Lines changed: 139 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -104,6 +104,94 @@ cozy_wait_tenant_drained() {
104104
done
105105
}
106106

107+
# Pure predicate for ONE node row of the capture cozy_has_schedulable_node
108+
# scans. Encodes the scheduler's own admission rule for a Pod that tolerates
109+
# nothing: the NodeUnschedulable plugin rejects a node whose
110+
# .spec.unschedulable is set (what `kubectl get nodes` renders as
111+
# SchedulingDisabled), and the TaintToleration plugin rejects one carrying any
112+
# taint with effect NoSchedule or NoExecute. PreferNoSchedule only lowers the
113+
# node's score, so it is deliberately not treated as blocking. Ready is checked
114+
# explicitly rather than left to the not-ready taint, so the gate does not
115+
# depend on how promptly the node-lifecycle controller applies that taint.
116+
#
117+
# The backend Pod is not literally toleration-free: DefaultTolerationSeconds
118+
# admission gives every Pod a 300s NoExecute toleration for
119+
# node.kubernetes.io/not-ready and node.kubernetes.io/unreachable. So for those
120+
# two taints this predicate is stricter than the scheduler and would keep
121+
# waiting where the Pod could in fact be placed. That errs toward waiting,
122+
# never toward releasing the gate early, and requiring Ready makes the case
123+
# nearly unreachable anyway.
124+
cozy_node_accepts_pods() {
125+
# $1 Ready condition status, $2 .spec.unschedulable, $3 taint effects
126+
if [ "$1" != True ]; then
127+
return 1
128+
fi
129+
case "$2" in
130+
true | True) return 1 ;;
131+
esac
132+
case ",$3," in
133+
*,NoSchedule,* | *,NoExecute,*) return 1 ;;
134+
esac
135+
return 0
136+
}
137+
138+
# Pure exit-condition for the tenant scheduling gate
139+
# (cozy_wait_schedulable_node). The single argument is the capture of a
140+
# `kubectl get nodes --no-headers -o custom-columns=NAME,READY,UNSCHEDULABLE,TAINTS`,
141+
# one whitespace-separated row per node. custom-columns renders an absent field
142+
# as the literal "<none>" and joins several taint effects with a comma, so both
143+
# are matched as text. Returns 0 as soon as one row describes a node that would
144+
# accept a Pod that tolerates nothing. A failed probe leaves the capture empty,
145+
# which reports not-schedulable rather than schedulable, so an API blip can
146+
# never release the gate (same guard as cozy_tenant_drained). The scan runs in
147+
# the subshell on the right of the pipeline, so its `exit` ends that subshell
148+
# and becomes the function's status -- it never leaves the caller. Pure text
149+
# logic, unit-tested in hack/run-kubernetes-schedulable_test.bats.
150+
cozy_has_schedulable_node() {
151+
printf '%s\n' "$1" | {
152+
while read -r _name _ready _unschedulable _taints _rest; do
153+
if [ -z "$_name" ]; then
154+
continue
155+
fi
156+
if cozy_node_accepts_pods "$_ready" "$_unschedulable" "$_taints"; then
157+
exit 0
158+
fi
159+
done
160+
exit 1
161+
}
162+
}
163+
164+
# Block until the tenant cluster has at least one node that actually accepts a
165+
# Pod, then print the node table it decided on (the same table the failure path
166+
# prints, so the two outcomes are read the same way). The node-join gate in
167+
# run_kubernetes_test establishes that two nodes are Ready, which is a weaker
168+
# guarantee: a Ready node still carries `node.cilium.io/agent-not-ready` until
169+
# the tenant's cilium agent claims it, and a node the bringup has not finished
170+
# with is Ready,SchedulingDisabled.
171+
# Scheduling against such a set is not stuck, only slow, and the time it takes
172+
# is variable -- so it belongs in a budget of its own rather than inside the
173+
# workload's readiness budget, where it is indistinguishable from a slow image
174+
# pull or a failing probe.
175+
cozy_wait_schedulable_node() {
176+
_kc="$1"
177+
_timeout="${2:-300}"
178+
_deadline=$(( $(date +%s) + _timeout ))
179+
while :; do
180+
_nodes=$(kubectl --kubeconfig "$_kc" get nodes --no-headers -o custom-columns='NAME:.metadata.name,READY:.status.conditions[?(@.type=="Ready")].status,UNSCHEDULABLE:.spec.unschedulable,TAINTS:.spec.taints[*].effect' 2>/dev/null) || _nodes=""
181+
if cozy_has_schedulable_node "$_nodes"; then
182+
echo "» tenant has a node that accepts Pods:"
183+
printf '%s\n' "$_nodes" | sed 's/^/ node: /'
184+
return 0
185+
fi
186+
if [ "$(date +%s)" -ge "$_deadline" ]; then
187+
echo "» no tenant node became schedulable (Ready, not cordoned, no NoSchedule/NoExecute taint) within ${_timeout}s" >&2
188+
printf '%s\n' "${_nodes:-<the node probe returned nothing>}" | sed 's/^/ node: /' >&2
189+
return 1
190+
fi
191+
sleep 5
192+
done
193+
}
194+
107195
# Block until every ZFS storage pool on every LINSTOR satellite reports at
108196
# least _min_free_gib of FreeCapacity. Motivation is proven from a
109197
# cozyreport artefact captured by hack/cozyreport.sh (see PR #3044 run
@@ -575,7 +663,43 @@ EOF
575663
kubectl delete service --kubeconfig "tenantkubeconfig-${test_name}" "${test_name}-backend" \
576664
-n tenant-test --ignore-not-found --timeout=60s || true
577665

666+
# Start the workload's clock from a node that accepts Pods, not from a node
667+
# that is merely Ready. Both instances of run 31020254620 spent 2m18s and
668+
# 1m57s of the 300s readiness budget below on FailedScheduling, against nodes
669+
# that were Ready but carried `node.cilium.io/agent-not-ready` or were
670+
# SchedulingDisabled, and then ran out while the image was still being
671+
# pulled. This gate is not extra waiting on the happy path: it spends the
672+
# seconds the Pod would otherwise spend Pending (plus at most one 5s poll
673+
# interval) and moves them out of a budget that has a different job. 300s is
674+
# more than twice the longest scheduling delay observed, and the gate prints
675+
# the node table on both outcomes so a timeout names the taint that held it.
676+
# The two budgets do stack on a failing run: one that spends the full 300s
677+
# here and then overruns the readiness wait gives up at ~600s where it used
678+
# to give up at 300s. That sits inside the enclosing 40m Chainsaw script op,
679+
# which the kubernetes-* suites document as a ~25m bringup.
680+
if ! cozy_wait_schedulable_node "tenantkubeconfig-${test_name}" 300; then
681+
echo "=== tenant scheduling gate failed: no node became schedulable within 300s — diagnostics follow ==="
682+
kubectl --kubeconfig "tenantkubeconfig-${test_name}" describe nodes || true
683+
kubectl -n tenant-test get hr || true
684+
exit 1
685+
fi
686+
578687
# Backend 1
688+
#
689+
# nginx is pinned by digest. The tenant workers reach no registry mirror --
690+
# hack/e2e-talos-image-cache.yaml serves the Talos worker OS disk image over
691+
# HTTP and is not one, and nothing else in the tree mirrors container images
692+
# for a tenant -- so this is pulled from Docker Hub on every run either way.
693+
# The digest does not remove that pull, it fixes what the pull returns: a
694+
# floating `nginx:alpine` silently changes size and layer count under the
695+
# readiness budget below, and supplies whatever content the tag points at on
696+
# the day. The digest is the OCI index, not a per-architecture manifest, so
697+
# the kubelet still selects the image for the worker's own architecture.
698+
# Nothing will bump it: renovate's enabledManagers are gomod, dockerfile,
699+
# github-actions and custom.regex, and both custom managers match packages/
700+
# paths only, so no manager reads this file. That is the intent rather than an
701+
# oversight -- the point of the pin is that the bytes stay the same run to
702+
# run, and this is a throwaway test workload, not an image the platform ships.
579703
kubectl apply --kubeconfig "tenantkubeconfig-${test_name}" -f- <<EOF
580704
apiVersion: apps/v1
581705
kind: Deployment
@@ -596,7 +720,7 @@ spec:
596720
spec:
597721
containers:
598722
- name: nginx
599-
image: nginx:alpine
723+
image: nginx:1.31.3-alpine@sha256:4a73073bd557c65b759505da037898b61f1be6cbcc3c2c3aeac22d2a470c1752
600724
ports:
601725
- containerPort: 80
602726
readinessProbe:
@@ -624,8 +748,20 @@ spec:
624748
targetPort: 80
625749
EOF
626750

627-
# Wait for pods readiness
628-
kubectl wait deployment --kubeconfig "tenantkubeconfig-${test_name}" "${test_name}-backend" -n tenant-test --for=condition=Available --timeout=300s
751+
# Wait for pods readiness. With scheduling gated above, these 300s start from
752+
# a node that accepts Pods, so they cover placement, sandbox setup, the image
753+
# pull and the probe -- and no longer the wait for a node to stop rejecting
754+
# the Pod, which now has a budget and a message of its own. The events in the
755+
# diagnostics below tell the remaining consumers apart. The number is
756+
# unchanged from when it also had to absorb scheduling; the pull alone
757+
# measured 1m42s in run 31020254620.
758+
if ! kubectl wait deployment --kubeconfig "tenantkubeconfig-${test_name}" "${test_name}-backend" -n tenant-test --for=condition=Available --timeout=300s; then
759+
echo "=== backend readiness failed: the Pod was schedulable but did not become Available within 300s — diagnostics follow ==="
760+
kubectl --kubeconfig "tenantkubeconfig-${test_name}" -n tenant-test describe deployment "${test_name}-backend" || true
761+
kubectl --kubeconfig "tenantkubeconfig-${test_name}" -n tenant-test describe pods -l "backend=${test_name}-backend" || true
762+
kubectl --kubeconfig "tenantkubeconfig-${test_name}" -n tenant-test get events --sort-by=.lastTimestamp || true
763+
exit 1
764+
fi
629765

630766
# Wait for LoadBalancer to be provisioned (IP or hostname)
631767
timeout 90 sh -ec "

0 commit comments

Comments
 (0)