Skip to content

Commit 2cb16d5

Browse files
refactor(build): standardize image tagging to fix concurrent PR push conflicts (#2711)
## What this PR does Concurrent PR builds were racing in the OCIR registry. Every package Makefile tagged its images via `settag`, which on any non-release commit collapsed to `:latest` — so every image across every PR was pushed to the same tag, and parallel builds got 409s on the manifest PUT. Two outliers (`metallb`, `redis-operator`) used hardcoded version tags and collided across PRs unconditionally. This PR replaces `settag` and the `TAG=latest` fallback in `hack/common-envs.mk` with a single `image-tags` macro driven by three env vars: | Env | Meaning | |---|---| | `IMAGE_TAG` | Build-unique tag, always pushed | | `PUBLISH_VERSIONED` | Also push `:<component-version>` (when provided) | | `PUBLISH_FLOATING` | Also push `:latest` | Wired into the two workflows: - **`pull-requests.yaml`** — `IMAGE_TAG=pr-<N>-<sha>`, both `PUBLISH_*` off. Every PR build pushes a unique tag set; no collisions, no `:latest` move. - **`tags.yaml`** — `IMAGE_TAG=<ref_name>`, both `PUBLISH_*` on. Release output preserved (component versions + `:latest`) plus a uniform `:<git-tag>` handle on every image for traceability. - **Local** — defaults to `IMAGE_TAG=dev`, no flags, `PUSH=0`. All 30 package Makefiles converted to `$(call image-tags,<repo>,<versioned-tag>)`. Image consumers reference digests pinned into `values.yaml` by each Makefile, so the tag is decorative for cluster-side pulls — the build-unique tag is sufficient for E2E. Also drops the redundant `:<component-version>-<git-tag>` form (cilium, linstor, matchbox, ubuntu-container-disk, kubevirt-*, cluster-autoscaler). Nothing in the repo or downstream values.yaml consumed it; the same information is preserved via the new `:<git-tag>` plus the existing `:<component-version>` tags on release. `ubuntu-container-disk` builds a separate image per Kubernetes version in a loop, so its primary tag became `:<k8s-ver>-<IMAGE_TAG>` to prevent loop iterations within a single build from pushing to the same tag. Verified by dry-running `make -n image` across representative packages (simple, dual-tag, skopeo, flux-push, loop) under all four contexts (PR / release / main / dev) — tag sets are unique and behave as expected. ### Release note ```release-note refactor(build): PR CI now pushes images under a unique `pr-<N>-<sha>` tag per build instead of `:latest`. Concurrent PR builds no longer race in the registry. Release builds keep the existing `:<component-version>` and `:latest` tags and additionally publish a uniform `:<git-tag>` handle across every image. The redundant `:<component-version>-<git-tag>` tag (cilium, linstor, matchbox, ubuntu-container-disk, kubevirt-*, cluster-autoscaler) is no longer published — image references in values.yaml are digest-pinned and were the only legitimate consumers. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Centralized and simplified container image tagging across build workflows. * PR builds now use unique image tags derived from PR metadata for better isolation. * Added configurable flags to control publishing of versioned and floating tags, making release tagging behavior more predictable. <!-- review_stack_entry_start --> [![Review Change Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2711?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack) <!-- review_stack_entry_end --> <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents f36240a + e42a582 commit 2cb16d5

34 files changed

Lines changed: 123 additions & 95 deletions

File tree

‎.github/workflows/pull-requests.yaml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -70,6 +70,12 @@ jobs:
7070
run: make build
7171
env:
7272
DOCKER_CONFIG: ${{ runner.temp }}/.docker
73+
# Unique per (PR, commit) so concurrent PR builds never push the same
74+
# tag. PUBLISH_VERSIONED/PUBLISH_FLOATING stay off — PR images are
75+
# consumed by digest, and :latest belongs to release builds.
76+
IMAGE_TAG: pr-${{ github.event.pull_request.number }}-${{ github.event.pull_request.head.sha }}
77+
PUBLISH_VERSIONED: '0'
78+
PUBLISH_FLOATING: '0'
7379

7480
- name: Build Talos image
7581
run: make -C packages/core/talos talos-nocloud

‎.github/workflows/tags.yaml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -118,6 +118,11 @@ jobs:
118118
run: make build
119119
env:
120120
DOCKER_CONFIG: ${{ runner.temp }}/.docker
121+
# Release semantics: push the release tag as :IMAGE_TAG (vX.Y.Z),
122+
# the component versions as :<component-ver>, and move :latest.
123+
IMAGE_TAG: ${{ github.ref_name }}
124+
PUBLISH_VERSIONED: '1'
125+
PUBLISH_FLOATING: '1'
121126

122127
# Commit built artifacts
123128
- name: Commit release artifacts

‎hack/common-envs.mk‎

Lines changed: 27 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -6,11 +6,25 @@ else
66
endif
77

88
REGISTRY ?= ghcr.io/cozystack/cozystack
9-
TAG = $(shell git describe --tags --exact-match --match 'v*' 2>/dev/null || echo latest)
9+
10+
# IMAGE_TAG: build-unique tag pushed for every image. Set by CI to a value
11+
# that does not collide between concurrent builds:
12+
# * pull-requests.yaml -> pr-<N>-<sha>
13+
# * tags.yaml -> <ref_name> (e.g. v1.5.0)
14+
# * local -> dev (with PUSH=0 by default)
15+
IMAGE_TAG ?= dev
16+
17+
# Opt-in extra tags. Workflows set these explicitly; defaults are off so a
18+
# local `make build` never races with CI or accidentally moves :latest.
19+
# PUBLISH_VERSIONED=1 -> also push :<component-version> (release semantics)
20+
# PUBLISH_FLOATING=1 -> also push :latest (release/main only)
21+
PUBLISH_VERSIONED ?= 0
22+
PUBLISH_FLOATING ?= 0
23+
1024
PUSH := 1
1125
LOAD := 0
1226
BUILDER ?=
13-
PLATFORM ?=
27+
PLATFORM ?=
1428
BUILDX_EXTRA_ARGS ?=
1529
COZYSTACK_VERSION = $(patsubst v%,%,$(shell git describe --tags --match 'v*'))
1630

@@ -20,14 +34,21 @@ BUILDX_ARGS := --provenance=false --push=$(PUSH) --load=$(LOAD) \
2034
$(if $(strip $(PLATFORM)),--platform=$(PLATFORM)) \
2135
$(BUILDX_EXTRA_ARGS)
2236

23-
# Returns 'latest' if the git tag is not assigned, otherwise returns the provided value
24-
define settag
25-
$(if $(filter $(TAG),latest),latest,$(1))
37+
# image-tags <repo> <versioned-tag>
38+
# Expands to one or more `--tag` flags for `docker buildx build`:
39+
# - always: :$(IMAGE_TAG) (build-unique handle)
40+
# - if PUBLISH_VERSIONED=1: :<versioned-tag> (skipped when arg2 is empty
41+
# or equals IMAGE_TAG)
42+
# - if PUBLISH_FLOATING=1: :latest
43+
# Consumers reference images by digest, so the build-unique tag is enough
44+
# for cluster-side pulls; the versioned and floating tags exist for human
45+
# discoverability and downstream tooling on releases.
46+
define image-tags
47+
--tag $(REGISTRY)/$(1):$(IMAGE_TAG)$(if $(filter 1,$(PUBLISH_VERSIONED)),$(if $(filter-out $(IMAGE_TAG),$(strip $(2))), --tag $(REGISTRY)/$(1):$(strip $(2))))$(if $(filter 1,$(PUBLISH_FLOATING)), --tag $(REGISTRY)/$(1):latest)
2648
endef
2749

2850
ifeq ($(COZYSTACK_VERSION),)
2951
$(shell git remote add upstream https://github.com/cozystack/cozystack.git || true)
3052
$(shell git fetch upstream --tags)
3153
COZYSTACK_VERSION = $(patsubst v%,%,$(shell git describe --tags --match 'v*'))
3254
endif
33-

‎packages/apps/clickhouse/Makefile‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -12,20 +12,20 @@ test:
1212

1313
image:
1414
docker buildx build images/clickhouse-backup \
15-
--tag $(REGISTRY)/clickhouse-backup:$(call settag,$(CLICKHOUSE_BACKUP_TAG)) \
15+
$(call image-tags,clickhouse-backup,$(CLICKHOUSE_BACKUP_TAG)) \
1616
--cache-from type=registry,ref=$(REGISTRY)/clickhouse-backup:latest \
1717
--cache-to type=inline \
1818
--metadata-file images/clickhouse-backup.json \
1919
$(BUILDX_ARGS)
20-
echo "$(REGISTRY)/clickhouse-backup:$(call settag,$(CLICKHOUSE_BACKUP_TAG))@$$(yq e '."containerimage.digest"' images/clickhouse-backup.json -o json -r)" \
20+
echo "$(REGISTRY)/clickhouse-backup:$(IMAGE_TAG)@$$(yq e '."containerimage.digest"' images/clickhouse-backup.json -o json -r)" \
2121
> images/clickhouse-backup.tag
2222
rm -f images/clickhouse-backup.json
2323
docker buildx build images/altinity-clickhouse-backup \
24-
--tag $(REGISTRY)/altinity-clickhouse-backup:$(call settag,$(CLICKHOUSE_BACKUP_TAG)) \
24+
$(call image-tags,altinity-clickhouse-backup,$(CLICKHOUSE_BACKUP_TAG)) \
2525
--cache-from type=registry,ref=$(REGISTRY)/altinity-clickhouse-backup:latest \
2626
--cache-to type=inline \
2727
--metadata-file images/altinity-clickhouse-backup.json \
2828
$(BUILDX_ARGS)
29-
echo "$(REGISTRY)/altinity-clickhouse-backup:$(call settag,$(CLICKHOUSE_BACKUP_TAG))@$$(yq e '."containerimage.digest"' images/altinity-clickhouse-backup.json -o json -r)" \
29+
echo "$(REGISTRY)/altinity-clickhouse-backup:$(IMAGE_TAG)@$$(yq e '."containerimage.digest"' images/altinity-clickhouse-backup.json -o json -r)" \
3030
> images/altinity-clickhouse-backup.tag
3131
rm -f images/altinity-clickhouse-backup.json

‎packages/apps/http-cache/Makefile‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,12 +7,12 @@ image: image-nginx
77

88
image-nginx:
99
docker buildx build images/nginx-cache \
10-
--tag $(REGISTRY)/nginx-cache:$(call settag,$(NGINX_CACHE_TAG)) \
10+
$(call image-tags,nginx-cache,$(NGINX_CACHE_TAG)) \
1111
--cache-from type=registry,ref=$(REGISTRY)/nginx-cache:latest \
1212
--cache-to type=inline \
1313
--metadata-file images/nginx-cache.json \
1414
$(BUILDX_ARGS)
15-
echo "$(REGISTRY)/nginx-cache:$(call settag,$(NGINX_CACHE_TAG))@$$(yq e '."containerimage.digest"' images/nginx-cache.json -o json -r)" \
15+
echo "$(REGISTRY)/nginx-cache:$(IMAGE_TAG)@$$(yq e '."containerimage.digest"' images/nginx-cache.json -o json -r)" \
1616
> images/nginx-cache.tag
1717
rm -f images/nginx-cache.json
1818

‎packages/apps/kubernetes/Makefile‎

Lines changed: 10 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -23,38 +23,36 @@ image-ubuntu-container-disk:
2323
$(foreach ver,$(KUBERNETES_VERSIONS), \
2424
docker buildx build images/ubuntu-container-disk \
2525
--build-arg KUBERNETES_VERSION=$(ver) \
26-
--tag $(REGISTRY)/ubuntu-container-disk:$(call settag,$(ver)) \
27-
--tag $(REGISTRY)/ubuntu-container-disk:$(call settag,$(ver)-$(TAG)) \
28-
--cache-from type=registry,ref=$(REGISTRY)/ubuntu-container-disk:$(call settag,$(ver)) \
26+
--tag $(REGISTRY)/ubuntu-container-disk:$(ver)-$(IMAGE_TAG) \
27+
$(if $(filter 1,$(PUBLISH_VERSIONED)),--tag $(REGISTRY)/ubuntu-container-disk:$(ver)) \
28+
--cache-from type=registry,ref=$(REGISTRY)/ubuntu-container-disk:$(ver) \
2929
--cache-to type=inline \
3030
--metadata-file images/ubuntu-container-disk-$(ver).json \
3131
$(BUILDX_ARGS) && \
32-
echo "$(REGISTRY)/ubuntu-container-disk:$(call settag,$(ver))@$$(yq e '."containerimage.digest"' images/ubuntu-container-disk-$(ver).json -o json -r)" \
32+
echo "$(REGISTRY)/ubuntu-container-disk:$(ver)-$(IMAGE_TAG)@$$(yq e '."containerimage.digest"' images/ubuntu-container-disk-$(ver).json -o json -r)" \
3333
> images/ubuntu-container-disk-$(ver).tag && \
3434
rm -f images/ubuntu-container-disk-$(ver).json; \
3535
)
3636

3737
image-kubevirt-cloud-provider:
3838
docker buildx build images/kubevirt-cloud-provider \
39-
--tag $(REGISTRY)/kubevirt-cloud-provider:$(call settag,$(KUBERNETES_PKG_TAG)) \
40-
--tag $(REGISTRY)/kubevirt-cloud-provider:$(call settag,$(KUBERNETES_PKG_TAG)-$(TAG)) \
39+
$(call image-tags,kubevirt-cloud-provider,$(KUBERNETES_PKG_TAG)) \
4140
--cache-from type=registry,ref=$(REGISTRY)/kubevirt-cloud-provider:latest \
4241
--cache-to type=inline \
4342
--metadata-file images/kubevirt-cloud-provider.json \
4443
$(BUILDX_ARGS)
45-
echo "$(REGISTRY)/kubevirt-cloud-provider:$(call settag,$(KUBERNETES_PKG_TAG))@$$(yq e '."containerimage.digest"' images/kubevirt-cloud-provider.json -o json -r)" \
44+
echo "$(REGISTRY)/kubevirt-cloud-provider:$(IMAGE_TAG)@$$(yq e '."containerimage.digest"' images/kubevirt-cloud-provider.json -o json -r)" \
4645
> images/kubevirt-cloud-provider.tag
4746
rm -f images/kubevirt-cloud-provider.json
4847

4948
image-kubevirt-csi-driver:
5049
docker buildx build images/kubevirt-csi-driver \
51-
--tag $(REGISTRY)/kubevirt-csi-driver:$(call settag,$(KUBERNETES_PKG_TAG)) \
52-
--tag $(REGISTRY)/kubevirt-csi-driver:$(call settag,$(KUBERNETES_PKG_TAG)-$(TAG)) \
50+
$(call image-tags,kubevirt-csi-driver,$(KUBERNETES_PKG_TAG)) \
5351
--cache-from type=registry,ref=$(REGISTRY)/kubevirt-csi-driver:latest \
5452
--cache-to type=inline \
5553
--metadata-file images/kubevirt-csi-driver.json \
5654
$(BUILDX_ARGS)
57-
echo "$(REGISTRY)/kubevirt-csi-driver:$(call settag,$(KUBERNETES_PKG_TAG))@$$(yq e '."containerimage.digest"' images/kubevirt-csi-driver.json -o json -r)" \
55+
echo "$(REGISTRY)/kubevirt-csi-driver:$(IMAGE_TAG)@$$(yq e '."containerimage.digest"' images/kubevirt-csi-driver.json -o json -r)" \
5856
> images/kubevirt-csi-driver.tag
5957
IMAGE=$$(cat images/kubevirt-csi-driver.tag) \
6058
yq -i '.csiDriver.image = strenv(IMAGE)' ../../system/kubevirt-csi-node/values.yaml
@@ -63,13 +61,12 @@ image-kubevirt-csi-driver:
6361

6462
image-cluster-autoscaler:
6563
docker buildx build images/cluster-autoscaler \
66-
--tag $(REGISTRY)/cluster-autoscaler:$(call settag,$(KUBERNETES_PKG_TAG)) \
67-
--tag $(REGISTRY)/cluster-autoscaler:$(call settag,$(KUBERNETES_PKG_TAG)-$(TAG)) \
64+
$(call image-tags,cluster-autoscaler,$(KUBERNETES_PKG_TAG)) \
6865
--cache-from type=registry,ref=$(REGISTRY)/cluster-autoscaler:latest \
6966
--cache-to type=inline \
7067
--metadata-file images/cluster-autoscaler.json \
7168
$(BUILDX_ARGS)
72-
echo "$(REGISTRY)/cluster-autoscaler:$(call settag,$(KUBERNETES_PKG_TAG))@$$(yq e '."containerimage.digest"' images/cluster-autoscaler.json -o json -r)" \
69+
echo "$(REGISTRY)/cluster-autoscaler:$(IMAGE_TAG)@$$(yq e '."containerimage.digest"' images/cluster-autoscaler.json -o json -r)" \
7370
> images/cluster-autoscaler.tag
7471
rm -f images/cluster-autoscaler.json
7572

‎packages/apps/mariadb/Makefile‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -13,11 +13,11 @@ update:
1313

1414
image:
1515
docker buildx build images/mariadb-backup \
16-
--tag $(REGISTRY)/mariadb-backup:$(call settag,$(MARIADB_BACKUP_TAG)) \
16+
$(call image-tags,mariadb-backup,$(MARIADB_BACKUP_TAG)) \
1717
--cache-from type=registry,ref=$(REGISTRY)/mariadb-backup:latest \
1818
--cache-to type=inline \
1919
--metadata-file images/mariadb-backup.json \
2020
$(BUILDX_ARGS)
21-
echo "$(REGISTRY)/mariadb-backup:$(call settag,$(MARIADB_BACKUP_TAG))@$$(yq e '."containerimage.digest"' images/mariadb-backup.json -o json -r)" \
21+
echo "$(REGISTRY)/mariadb-backup:$(IMAGE_TAG)@$$(yq e '."containerimage.digest"' images/mariadb-backup.json -o json -r)" \
2222
> images/mariadb-backup.tag
2323
rm -f images/mariadb-backup.json

‎packages/core/installer/Makefile‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -19,21 +19,21 @@ image: pre-checks image-operator image-packages chart
1919

2020
image-operator:
2121
docker buildx build -f images/cozystack-operator/Dockerfile ../../.. \
22-
--tag $(REGISTRY)/cozystack-operator:$(call settag,$(TAG)) \
23-
--build-arg VERSION=$(call settag,$(TAG)) \
22+
$(call image-tags,cozystack-operator,v$(COZYSTACK_VERSION)) \
23+
--build-arg VERSION=v$(COZYSTACK_VERSION) \
2424
--cache-from type=registry,ref=$(REGISTRY)/cozystack-operator:latest \
2525
--cache-to type=inline \
2626
--metadata-file images/cozystack-operator.json \
2727
$(BUILDX_ARGS)
28-
IMAGE="$(REGISTRY)/cozystack-operator:$(call settag,$(TAG))@$$(yq -e '.["containerimage.digest"]' images/cozystack-operator.json -o json -r)" \
28+
IMAGE="$(REGISTRY)/cozystack-operator:$(IMAGE_TAG)@$$(yq -e '.["containerimage.digest"]' images/cozystack-operator.json -o json -r)" \
2929
yq -i '.cozystackOperator.image = strenv(IMAGE)' values.yaml
3030
rm -f images/cozystack-operator.json
3131

3232
image-packages:
3333
mkdir -p ../../../_out/assets images
3434
echo $$(git rev-parse HEAD; git diff HEAD; git ls-files --others --exclude-standard) | shasum -a 256 > ../platform/.build-revision
3535
flux push artifact \
36-
oci://$(REGISTRY)/cozystack-packages:$(call settag,$(TAG)) \
36+
oci://$(REGISTRY)/cozystack-packages:$(IMAGE_TAG) \
3737
--path=../../../packages \
3838
--source=https://github.com/cozystack/cozystack \
3939
--revision="$$(git describe --tags):$$(git rev-parse HEAD)" \

‎packages/core/platform/Makefile‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,11 +18,11 @@ image: image-migrations
1818

1919
image-migrations:
2020
docker buildx build images/migrations \
21-
--tag $(REGISTRY)/platform-migrations:$(call settag,$(TAG)) \
21+
$(call image-tags,platform-migrations,v$(COZYSTACK_VERSION)) \
2222
--cache-from type=registry,ref=$(REGISTRY)/platform-migrations:latest \
2323
--cache-to type=inline \
2424
--metadata-file images/migrations.json \
2525
$(BUILDX_ARGS)
26-
IMAGE="$(REGISTRY)/platform-migrations:$(call settag,$(TAG))@$$(yq -e -o json -r '.["containerimage.digest"]' images/migrations.json)" \
26+
IMAGE="$(REGISTRY)/platform-migrations:$(IMAGE_TAG)@$$(yq -e -o json -r '.["containerimage.digest"]' images/migrations.json)" \
2727
yq --inplace '.migrations.image = strenv(IMAGE)' values.yaml
2828
rm -f images/migrations.json

‎packages/core/talos/Makefile‎

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -12,19 +12,22 @@ image: image-matchbox image-talos
1212

1313
image-talos:
1414
test -f ../../../_out/assets/installer-amd64.tar || make talos-installer
15-
skopeo copy docker-archive:../../../_out/assets/installer-amd64.tar docker://$(REGISTRY)/talos:$(call settag,$(TALOS_VERSION))
15+
set -e; \
16+
SRC=docker-archive:../../../_out/assets/installer-amd64.tar; \
17+
skopeo copy "$$SRC" docker://$(REGISTRY)/talos:$(IMAGE_TAG); \
18+
if [ "$(PUBLISH_VERSIONED)" = "1" ]; then skopeo copy "$$SRC" docker://$(REGISTRY)/talos:$(TALOS_VERSION); fi; \
19+
if [ "$(PUBLISH_FLOATING)" = "1" ]; then skopeo copy "$$SRC" docker://$(REGISTRY)/talos:latest; fi
1620

1721
image-matchbox:
1822
test -f ../../../_out/assets/kernel-amd64 || make talos-kernel
1923
test -f ../../../_out/assets/initramfs-metal-amd64.xz || make talos-initramfs
2024
docker buildx build -f images/matchbox/Dockerfile ../../.. \
21-
--tag $(REGISTRY)/matchbox:$(call settag,$(TAG)) \
22-
--tag $(REGISTRY)/matchbox:$(call settag,$(TALOS_VERSION)-$(TAG)) \
25+
$(call image-tags,matchbox,$(TALOS_VERSION)) \
2326
--cache-from type=registry,ref=$(REGISTRY)/matchbox:latest \
2427
--cache-to type=inline \
2528
--metadata-file images/matchbox.json \
2629
$(BUILDX_ARGS)
27-
echo "$(REGISTRY)/matchbox:$(call settag,$(TAG))@$$(yq e '."containerimage.digest"' images/matchbox.json -o json -r)" \
30+
echo "$(REGISTRY)/matchbox:$(IMAGE_TAG)@$$(yq e '."containerimage.digest"' images/matchbox.json -o json -r)" \
2831
> ../../extra/bootbox/images/matchbox.tag
2932
rm -f images/matchbox.json
3033

0 commit comments

Comments
 (0)