Repository navigation
Commit 1b4b6ff
authored
test(e2e): warm the ghcr.io mirror before the tenant suites need it (#3768)
## What this PR does
The most expensive e2e flake, #3513, is a tenant
worker missing the 18m node-Ready budget with zero Nodes registered. The
in-guest capture in #3548 diagnosed one variant: the
worker's kubelet image pull crawls to ghcr.io at 40-165 KB/s over the
runner egress and does not finish inside the budget, Talos holds the
kubelet service until its image arrives, so the node never registers and
the tenant cilium HelmRelease then fails for want of a schedulable node
rather than for any fault of its own.
The sandbox already runs an in-cluster pull-through mirror for ghcr.io,
but a pull-through cache fetches a blob only when a client first asks
for it, so the first tenant worker pays the cold fetch inside its
node-Ready budget. Measured with the same fetch script against a local
instance of the same registry image: 32m47s cold, 1.1s warm, for the
same tags the failing runs stalled on.
This PR fills the cache from a Job started by the same install step that
deploys the mirror, tens of minutes before the kubernetes suites run. It
warms the two kubelet tags the suites actually select, resolved from the
chart's version map in the same order the suites resolve it, and one
platform per tag (the one the workers run, ~61 MiB each) - warming what
nobody pulls would be a true addition to the same throttled egress the
install itself is using.
The warm-up is best-effort by construction: a Job that fails, times out
or never starts does not fail the install, and the suites behave exactly
as they did before it existed. The Job itself exits non-zero on any
missed tag, so its status stays honest.
The node-join failure diagnostics now answer the questions a red run
used to leave open: whether a worker pulled through the mirror at all
(counting GET and HEAD access lines per request, excluding the warm-up's
own requests, which name themselves via user-agent), how long the mirror
took to answer (server-side `http.response.duration` is the only
after-the-fact signal separating a warm cache from a cold one, since the
proxy stores blobs in the background and the Job reports success either
way), and whether the warm-up itself got anywhere. The diagnostic
section grows from five to seven bounded reads, moving its worst case
from roughly 100s to roughly 150s inside the existing phase budget gate;
unbudgeted diagnostic residuals stay tracked in
#3666.
This addresses one variant of #3513 and does not
close it: the zero-CSR class remains untouched, and a green suite after
this change is not evidence the flake is gone. The 18m node-Ready budget
is not raised.
Covered by 65 unit tests in `hack/ghcr-mirror_test.bats`: tag selection
and ordering against the suites' own expressions, rejection of malformed
and injected tag values, Job shape and restricted security context, the
readiness wait as a wall clock, every failure path of the new
diagnostics including an over-1-MiB log fixture that pins the duration
filter against real log sizes, and the exclusion of the warm-up's own
traffic from both the request count and the duration report.
### Screenshots
No UI changes.
### Downstream repositories
The diff is confined to e2e infrastructure under `hack/`: the mirror
manifest, the warm-up library and its tests, the install step, and
diagnostic labels. Walked the trigger map file by file - no chart
values, no APIs, no documentation content, nothing any downstream
repository consumes.
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added best-effort GHCR mirror warm-up during installation.
* Preloads selected Kubernetes image manifests and blobs to improve
cache readiness.
* Warm-up failures safely fall back to direct image pulls.
* **Diagnostics**
* Improved request classification, timing, repository identification,
and reporting of warm-up status and logs.
* **Tests**
* Expanded coverage for warm-up behavior, validation, retries, security
settings, caching, logging, installation, and diagnostics.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->8 files changed
Lines changed: 1547 additions & 19 deletions
File tree
- hack
- e2e-chainsaw
- _lib
- kubernetes-latest
- kubernetes-previous
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1908 | 1908 | | |
1909 | 1909 | | |
1910 | 1910 | | |
1911 | | - | |
1912 | | - | |
| 1911 | + | |
| 1912 | + | |
1913 | 1913 | | |
1914 | 1914 | | |
1915 | 1915 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
35 | 35 | | |
36 | 36 | | |
37 | 37 | | |
38 | | - | |
| 38 | + | |
39 | 39 | | |
40 | 40 | | |
41 | 41 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
29 | 29 | | |
30 | 30 | | |
31 | 31 | | |
32 | | - | |
| 32 | + | |
33 | 33 | | |
34 | 34 | | |
35 | 35 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
50 | 50 | | |
51 | 51 | | |
52 | 52 | | |
53 | | - | |
54 | | - | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
55 | 57 | | |
56 | 58 | | |
57 | 59 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
93 | 93 | | |
94 | 94 | | |
95 | 95 | | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
96 | 115 | | |
97 | 116 | | |
98 | 117 | | |
| |||
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
516 | 516 | | |
517 | 517 | | |
518 | 518 | | |
519 | | - | |
| 519 | + | |
520 | 520 | | |
521 | 521 | | |
522 | 522 | | |
| |||
0 commit comments