Skip to content

Commit 5f2bea1

Browse files
ci: warm the build cache on release-1.5 pushes (#3469)
## What this PR does Nothing has ever warmed `release-1.5`'s build cache, so every PR build on this branch runs cold end to end and exceeds its 30-minute timeout. This branch's copy of `build-main.yaml` is dead code. It triggers on `push: branches: [main]`, which never happens here, and it targets `[self-hosted]` — the decommissioned cozy-runner-1 (#2937) — so it could not have run even if it fired. This PR points it at this branch instead. Both halves of the cold build trace back to 39cdc88 (`feat(kubernetes): bootstrap tenant workers with Talos via CABPT`) landing on `main` on 2026-06-16, six days before `release-1.5` was cut on 2026-06-22: - That commit dropped `ubuntu-container-disk` from `make image`, so `main` stopped writing its six `:<minor>-buildcache` refs. `release-1.5` is now the only branch that still builds the image, and the CI log shows all six importers failing with `not found`. Serially that was ~28min of the 30min budget; #3457 parallelises the target down to ~10min. - Every other cache ref carries `main`'s sources, which have diverged from this branch, so most of the remaining 28 packages miss too. Measured on the #3457 run: ~38min across the full build, and no single hot spot left to attack — `apps/kubernetes` at 15m07s and `system/linstor` at 5m39s, everything else 20-60s each. ### Changes | | before | after | | --- | --- | --- | | trigger | `push: [main]` | `push: [release-1.5]` | | runner | `[self-hosted]` (decommissioned) | `oracle-vm-24cpu-96gb-x86-64` | | cache namespace | shared `cozystack/` | `cozystack-cache-release-1.5/` | | `IMAGE_TAG` | `main` | `release-1.5` | | warmer timeout | 90 | 120 | `pull-requests.yaml` gets the same `CACHE_REGISTRY` so PR builds read what the warmer writes, and its Build timeout goes to 75 minutes. The cache namespace is branch-scoped on purpose. Writing the shared refs from here would replace `main`'s layers with this branch's, so `main`'s next PR build would miss on everything and blow its own 30-minute timeout until the next `main` push re-warmed it. `CACHE_REGISTRY` already exists as a knob in `hack/common-envs.mk`, so this is one env var per job rather than new machinery. Image tags are unaffected — they still resolve to `cozystack/<image>`, only cache refs move. The timeout raise is not optional. The new namespace starts empty, so this PR and the next one still build cold at ~45min, and anything that later invalidates the cache broadly — a Dockerfile edit, a `go.mod` bump — costs the same again. 30 minutes only ever fit a warm build, and when it did not fit, the job reported a timeout rather than whatever actually went wrong. ### Verification - Both workflows are asserted to agree on the cache namespace, rather than eyeballed. - Confirmed the env split renders as intended: `--tag iad.ocir.io/…/cozystack/ubuntu-container-disk:v1.35-release-1.5` alongside `--cache-from`/`--cache-to` on `…/cozystack-cache-release-1.5/ubuntu-container-disk:v1.35-buildcache`. - `WRITE_CACHE` stays unset in PR builds, so they remain read-only against the cache and never race on a manifest. ### Notes for the reviewer - **Stacked on #3457.** Both PRs edit adjacent lines of `pull-requests.yaml`, so keeping them independent guaranteed a conflict. GitHub retargets this to `release-1.5` automatically once #3457 merges. - **The filename stays `build-main.yaml`** even though it no longer warms `main`. Renaming it would make any later backport touching `main`'s copy arrive as a whole new file rather than a modification, which is the failure mode the backport bot already hits on this branch. - **One assumption I could not test locally:** the registry credentials need repo-create permission for the new `cozystack-cache-release-1.5/*` path. It is the same tenancy namespace that already auto-creates repos as packages get added, so it should hold — but if it does not, `--cache-to` fails the warmer, and the fallback is a cache-tag suffix in `hack/common-envs.mk` instead of a separate namespace. - **Sequencing:** the merge commit is itself a push to `release-1.5`, so the warmer fires on merge and takes ~50-60min cold. PR builds after that should be ~15min. The warmer cannot be triggered manually — `workflow_dispatch` is only available for workflows present on the default branch, and this one exists only here. - **Out of scope, but adjacent:** `tags.yaml`, `release-e2e.yaml` and `pull-requests-release.yaml` on this branch all still say `runs-on: [self-hosted]`, so the release and release-e2e lanes are broken the same way this warmer was. ### Release note ```release-note NONE ```
2 parents 2b2869f + 55d25aa commit 5f2bea1

3 files changed

Lines changed: 79 additions & 29 deletions

File tree

‎.github/workflows/build-main.yaml‎

Lines changed: 33 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,23 @@
1-
name: Build cache (main)
1+
name: Build cache (release-1.5)
22

3-
# Warms the shared mode=max build cache so PR builds start hot.
3+
# Warms the mode=max build cache for THIS branch so its PR builds start hot.
4+
#
5+
# On main this workflow fires on main pushes; on a release branch it has to fire
6+
# on that branch's own pushes, because the cache main writes is main's content.
7+
# Nothing had been warming release-1.5: main dropped ubuntu-container-disk from
8+
# `make image` in 39cdc8866, six days before this branch was cut, so its six
9+
# :*-buildcache refs were never written at all, and every other ref carries
10+
# main's sources rather than this branch's. The result was a PR Build that ran
11+
# cold end to end and timed out -- ~28min of it in ubuntu-container-disk alone
12+
# before #3457 parallelised that target, and ~38min across all 28 packages after.
13+
#
14+
# The cache is written to its OWN namespace (CACHE_REGISTRY below), not the
15+
# shared one. Writing the shared refs from here would replace main's layers with
16+
# this branch's, so main's next PR build would miss on everything and blow its
17+
# own 30-min timeout until the next main push re-warmed it.
418
#
519
# PR builds (pull-requests.yaml) read the cache but never write it
6-
# (WRITE_CACHE unset -> 0). Only this serialized main build writes
20+
# (WRITE_CACHE unset -> 0). Only this serialized branch build writes
721
# CACHE_REGISTRY/<img>:buildcache, so concurrent PR builds never race on the
822
# cache manifest -- the 409 collisions PR #2711 fixed for image tags would
923
# otherwise reappear on the cache ref. Cache is mode=max so all multistage
@@ -19,23 +33,32 @@ name: Build cache (main)
1933
env:
2034
# Same per-CI registry as pull-requests.yaml; the cache is co-located here.
2135
REGISTRY: iad.ocir.io/idyksih5sir9/cozystack
36+
# Branch-scoped cache namespace, kept in sync with the Build job in
37+
# pull-requests.yaml. Both must point at the same place or PR builds read a
38+
# cache nobody writes.
39+
CACHE_REGISTRY: iad.ocir.io/idyksih5sir9/cozystack-cache-release-1.5
2240

2341
on:
2442
push:
25-
branches: [main]
43+
branches: [release-1.5]
2644
paths-ignore:
2745
- 'docs/**'
2846

29-
# Only the newest main commit's cache matters; cancel older in-flight warmers.
47+
# Only the newest commit's cache matters; cancel older in-flight warmers.
3048
concurrency:
31-
group: build-main-cache
49+
group: build-release-1.5-cache
3250
cancel-in-progress: true
3351

3452
jobs:
3553
warm-cache:
3654
name: Warm build cache
37-
runs-on: [self-hosted]
38-
timeout-minutes: 90
55+
# Was [self-hosted], which is the decommissioned cozy-runner-1 (#2937) -- this
56+
# workflow could not have run on this branch even once. Same ephemeral shape
57+
# the main warmer moved to; a cold serial `make build` here is CPU-bound.
58+
runs-on: oracle-vm-24cpu-96gb-x86-64
59+
# The first run is fully cold and builds six ubuntu-container-disk images on
60+
# top of the other 28 packages; later runs mostly re-export a warm cache.
61+
timeout-minutes: 120
3962
permissions:
4063
contents: read
4164
packages: write
@@ -92,7 +115,7 @@ jobs:
92115
BUILDER: ${{ steps.buildx.outputs.name }}
93116
# WRITE_CACHE=1 turns on --cache-to (mode=max) for every image.
94117
WRITE_CACHE: '1'
95-
# Publish a floating :main handle; do not move :latest (releases do).
96-
IMAGE_TAG: main
118+
# Floating handle for this branch; do not move :latest (releases do).
119+
IMAGE_TAG: release-1.5
97120
PUBLISH_VERSIONED: '0'
98121
PUBLISH_FLOATING: '0'

‎.github/workflows/pull-requests.yaml‎

Lines changed: 23 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,11 @@ name: Pull Request
33
env:
44
# TODO: unhardcode this
55
REGISTRY: iad.ocir.io/idyksih5sir9/cozystack
6+
# Read the cache build-main.yaml writes for THIS branch, not the shared refs
7+
# main writes -- those carry main's sources and mostly miss here. Must stay in
8+
# sync with CACHE_REGISTRY in build-main.yaml. Builds only read it
9+
# (WRITE_CACHE defaults to 0), so a miss is a cold build, never an error.
10+
CACHE_REGISTRY: iad.ocir.io/idyksih5sir9/cozystack-cache-release-1.5
611
on:
712
pull_request:
813
types: [opened, synchronize, reopened]
@@ -33,13 +38,24 @@ jobs:
3338
# buildkit; concurrent build jobs serialize on buildkit's single-writer bbolt
3439
# cache lock + exporter mutex and stall to the 30-min timeout (proven via a
3540
# live SIGUSR1 goroutine dump). One VM per job => one buildkit per job => that
36-
# contention is severed by construction. `make build` is serial (one image at
37-
# a time), so a small shape suffices -- 4cpu/16gb, not the 24cpu the e2e job
38-
# uses -- and it leans on the warm mode=max cache (#2938) to stay under the
39-
# 30-min wall. The `debug` label still routes to self-hosted for the
40-
# breakpoint path. Phase 0.5 of #2937.
41-
runs-on: ${{ contains(github.event.pull_request.labels.*.name, 'debug') && 'self-hosted' || 'oracle-vm-4cpu-16gb-x86-64' }}
42-
timeout-minutes: 30
41+
# contention is severed by construction. Phase 0.5 of #2937.
42+
#
43+
# 24cpu/96gb rather than the 4cpu/16gb this used to run on: `make build` is
44+
# still serial across images, but this branch is the only one that still
45+
# builds ubuntu-container-disk, and it does so once per Kubernetes minor with
46+
# no shared build cache to warm it (main dropped the image in 39cdc8866, six
47+
# days before release-1.5 was cut, so nothing writes its :*-buildcache refs).
48+
# Those six builds now run concurrently, and each drives its own libguestfs
49+
# appliance -- ~9gb of RAM and a core apiece, which the small shape cannot
50+
# give. The `debug` label still routes to self-hosted for the breakpoint path.
51+
runs-on: ${{ contains(github.event.pull_request.labels.*.name, 'debug') && 'self-hosted' || 'oracle-vm-24cpu-96gb-x86-64' }}
52+
# A warm cache puts this job around 15min. 30 was not enough for a cold one:
53+
# the branch-scoped cache starts empty, and anything that invalidates it
54+
# (a Dockerfile edit, a go.mod bump) drops a later build back to ~45min --
55+
# six container disks plus 28 packages built from scratch. Sized for that
56+
# case rather than for the warm one, so a cold build reports its real
57+
# failure instead of a timeout that says nothing about why.
58+
timeout-minutes: 75
4359
permissions:
4460
contents: read
4561
packages: write

‎packages/apps/kubernetes/Makefile‎

Lines changed: 23 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -20,19 +20,30 @@ update:
2020

2121
image: image-ubuntu-container-disk image-kubevirt-cloud-provider image-kubevirt-csi-driver image-cluster-autoscaler
2222

23+
# One container disk per Kubernetes minor. Each build spends ~4-5min inside a
24+
# libguestfs appliance installing that minor's kubelet/kubeadm into its own copy
25+
# of the cloud image, and the versions share no per-version work, so build them
26+
# concurrently: the serial loop this replaced spent ~28min of the Build job's
27+
# 30min budget on this target alone. The sub-make carries its own -j so the root
28+
# `make build` stays serial, and buildkit dedups the shared guestfish and
29+
# cloud-image stages across the concurrent solves. --output-sync=target groups
30+
# each version's log under its target name -- six interleaved buildx progress
31+
# streams are unreadable otherwise.
2332
image-ubuntu-container-disk:
24-
$(foreach ver,$(KUBERNETES_VERSIONS), \
25-
docker buildx build images/ubuntu-container-disk \
26-
--build-arg KUBERNETES_VERSION=$(ver) \
27-
--tag $(REGISTRY)/ubuntu-container-disk:$(ver)-$(IMAGE_TAG) \
28-
$(if $(filter 1,$(PUBLISH_VERSIONED)),--tag $(REGISTRY)/ubuntu-container-disk:$(ver)) \
29-
$(call cache-args,ubuntu-container-disk,$(ver)-buildcache) \
30-
--metadata-file images/ubuntu-container-disk-$(ver).json \
31-
$(BUILDX_ARGS) && \
32-
echo "$(REGISTRY)/ubuntu-container-disk:$(ver)-$(IMAGE_TAG)@$$(yq e '."containerimage.digest"' images/ubuntu-container-disk-$(ver).json -o json -r)" \
33-
> images/ubuntu-container-disk-$(ver).tag && \
34-
rm -f images/ubuntu-container-disk-$(ver).json; \
35-
)
33+
$(MAKE) -j$(words $(KUBERNETES_VERSIONS)) --output-sync=target \
34+
$(addprefix image-ubuntu-container-disk-,$(KUBERNETES_VERSIONS))
35+
36+
image-ubuntu-container-disk-%:
37+
docker buildx build images/ubuntu-container-disk \
38+
--build-arg KUBERNETES_VERSION=$* \
39+
--tag $(REGISTRY)/ubuntu-container-disk:$*-$(IMAGE_TAG) \
40+
$(if $(filter 1,$(PUBLISH_VERSIONED)),--tag $(REGISTRY)/ubuntu-container-disk:$*) \
41+
$(call cache-args,ubuntu-container-disk,$*-buildcache) \
42+
--metadata-file images/ubuntu-container-disk-$*.json \
43+
$(BUILDX_ARGS)
44+
echo "$(REGISTRY)/ubuntu-container-disk:$*-$(IMAGE_TAG)@$$(yq e '."containerimage.digest"' images/ubuntu-container-disk-$*.json -o json -r)" \
45+
> images/ubuntu-container-disk-$*.tag
46+
rm -f images/ubuntu-container-disk-$*.json
3647

3748
image-kubevirt-cloud-provider:
3849
docker buildx build images/kubevirt-cloud-provider \

0 commit comments

Comments
 (0)