Repository navigation
Commit 0dae3ca
authored
fix(e2e): budget tenant scheduling separately from workload readiness (#3579)
## What this PR does
The tenant backend Deployment in the kubernetes suites had one 300s
budget to reach `condition=Available`, and two unrelated variable costs
shared it.
The first is scheduling. What the suite establishes before that point is
two tenant nodes Ready, which is weaker than schedulable: a Ready node
still carries `node.cilium.io/agent-not-ready` until the tenant cilium
agent claims it, and a node the bringup has not finished with is
`SchedulingDisabled`. In the run recorded in #3577 scheduling took 2m18s
and 1m57s in the two suites, which left the image pull to finish inside
what remained. It did not, and both suites reported the same `timed out
waiting for the condition` for what were two different shortfalls.
So this waits for a node that actually accepts a Pod before creating the
workload, on its own budget and its own failure message. The gate
encodes the scheduler's rule for a Pod that tolerates nothing (Ready,
not unschedulable, no `NoSchedule` or `NoExecute` taint) over a
`custom-columns` probe, prints the node table on both outcomes so a
timeout names the taint that held it, and treats a failed probe as
not-schedulable so an API blip cannot release it. On the happy path it
adds no wall time: it spends the seconds the Pod would otherwise spend
Pending, plus at most one 5s poll interval. On a failing run the two
budgets stack, so a run that exhausts both now gives up at ~600s where
it used to give up at 300s, inside the 40m Chainsaw script op the suites
document as a ~25m bringup. The readiness wait keeps its 300s, now
starting from a schedulable node, and dumps deployment, pod and event
state when it runs out.
The workload image is also pinned by digest. The tenant workers reach no
Docker Hub mirror (`hack/e2e-talos-image-cache.yaml` serves the Talos
worker OS disk image over HTTP and is not a registry mirror), so nginx
is pulled from Docker Hub on every run either way. The digest does not
take the pull off the critical path; it fixes what that pull returns
instead of leaving a floating tag free to change size and content under
a fixed deadline. Preloading the image would need infrastructure this
tree does not have, and is not attempted here. Nothing will bump the
pin: no renovate manager reads `hack/`, and the comment says so, because
for a throwaway test workload the freeze is the point.
What this does not claim: it removes two measured consumers from a fixed
budget, both taken from the events of a single run. Whether that budget
was the only thing making those suites red is not established from one
run, so this is not offered as the fix for a red pipeline.
`hack/run-kubernetes-schedulable_test.bats` covers the new logic: every
branch of the predicate, the multi-node scan, the poll-again path, the
deadline, and a probe that fails. Each test was verified by mutating the
helper and checking that the intended test, and no other, went red.
Note for whoever reviews alongside #3575: that branch mirrors `ghcr.io`
for tenant worker pulls, not `docker.io`, so it does not change the
Docker Hub pull described here. `git merge-tree` reports no conflict
between the two, nor with #3548.
Observed in #3577.
### Screenshots
Not a UI change.
### Downstream repositories
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
The trigger map was walked against the diff. The change is confined to
`hack/e2e-chainsaw/_lib/run-kubernetes.sh` and one new `hack/*.bats`
unit test. It moves and renames nothing under `hack/`, changes no make
target, and does not touch `hack/e2e-prepare-cluster.bats`, any package,
any CRD or any namespace name, so none of the listed repositories are
reached.
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved Kubernetes deployment readiness by waiting for a suitable,
schedulable node before timing backend startup.
* Added clearer diagnostics when nodes are unavailable or backend
readiness fails.
* Replaced the floating backend container image tag with a fixed,
verified version.
* **Tests**
* Added coverage for node readiness, cordoning, taints, polling,
timeouts, probe failures, and deployment sequencing.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->2 files changed
Lines changed: 432 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
104 | 104 | | |
105 | 105 | | |
106 | 106 | | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
107 | 195 | | |
108 | 196 | | |
109 | 197 | | |
| |||
575 | 663 | | |
576 | 664 | | |
577 | 665 | | |
| 666 | + | |
| 667 | + | |
| 668 | + | |
| 669 | + | |
| 670 | + | |
| 671 | + | |
| 672 | + | |
| 673 | + | |
| 674 | + | |
| 675 | + | |
| 676 | + | |
| 677 | + | |
| 678 | + | |
| 679 | + | |
| 680 | + | |
| 681 | + | |
| 682 | + | |
| 683 | + | |
| 684 | + | |
| 685 | + | |
| 686 | + | |
578 | 687 | | |
| 688 | + | |
| 689 | + | |
| 690 | + | |
| 691 | + | |
| 692 | + | |
| 693 | + | |
| 694 | + | |
| 695 | + | |
| 696 | + | |
| 697 | + | |
| 698 | + | |
| 699 | + | |
| 700 | + | |
| 701 | + | |
| 702 | + | |
579 | 703 | | |
580 | 704 | | |
581 | 705 | | |
| |||
596 | 720 | | |
597 | 721 | | |
598 | 722 | | |
599 | | - | |
| 723 | + | |
600 | 724 | | |
601 | 725 | | |
602 | 726 | | |
| |||
624 | 748 | | |
625 | 749 | | |
626 | 750 | | |
627 | | - | |
628 | | - | |
| 751 | + | |
| 752 | + | |
| 753 | + | |
| 754 | + | |
| 755 | + | |
| 756 | + | |
| 757 | + | |
| 758 | + | |
| 759 | + | |
| 760 | + | |
| 761 | + | |
| 762 | + | |
| 763 | + | |
| 764 | + | |
629 | 765 | | |
630 | 766 | | |
631 | 767 | | |
| |||
0 commit comments