Skip to content

Commit 52f7d05

Browse files
myasnikovdaniilclaude
authored andcommitted
ci: overlay current-main images for packages a PR didn't rebuild
Conditional PR builds (the per-package matrix in pull-requests.yaml) only rebuild packages whose dir changed. Every other package falls back to the image refs committed in the tree — the last release's digests (currently v1.5.0), which go stale as main advances. So e2e and the installer's packages artifact ran current-main configs against last-release images for everything outside the PR's build matrix, and a PR installed via the cozy-installer chart was an incoherent mix. This is the systemic gap in issue #3143 (its concrete bite: the merged objectstorage-controller requeue fix #3034 never ran in e2e, flaking the harbor BucketClaim check). In finalize, pull the cozystack-packages:main OCI artifact that build-main already publishes (the whole current-main packages tree, refs digest-pinned to the images build-main pushed) and walk every ref-bearing file in it (values.yaml + images/*.tag), overlaying the repo's copy. This is driven by the artifact, not the build matrix, so it covers the ENTIRE first-party tree — apps/system/core AND extra/, library/ — and a merged fix to any first-party image (e.g. the objectstorage-sidecar ref under extra/seaweedfs) takes effect. For a file build-main did not rebuild the artifact copy is byte-identical and is skipped. Skipped explicitly: units the PR itself rebuilt (their pr-<N>-<sha> refs win), packages/core/talos and packages/core/installer (owned by their dedicated build-talos / finalize-installer jobs), and vendored charts/ subtrees. The copy is self-validating: a file is overlaid only when every differing line is image-reference-bearing (a full @sha256: ref, or a split image/repository/registry/tag/digest key, or a --…-image= arg); otherwise it keeps its committed ref. A missing artifact is a no-op (prior behaviour) and the step is best-effort so it never blocks a PR. No build-main change is needed: cozystack-packages:main is already published by `make build`. Runs before the installer build so the flux-pushed packages artifact and the pr.patch e2e applies both reflect the overlay. Result: a PR is tested and installed as current-main + its own changes, coherent across config and image, instead of current-main configs running on last-release images. Covered by hack/overlay-main-images_test.bats (run by `make unit-tests` in the checks job): single-line and split-form ref overlay, extra/* coverage, skip-rebuilt, talos/installer skip, vendored-charts prune, drift self-validation, and missing-artifact no-op. Assisted-By: Claude <[email protected]> Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Signed-off-by: Myasnikov Daniil <[email protected]>
1 parent eab9b03 commit 52f7d05

3 files changed

Lines changed: 246 additions & 0 deletions

File tree

‎.github/workflows/pull-requests.yaml‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -335,6 +335,40 @@ jobs:
335335
registry: iad.ocir.io
336336
env:
337337
DOCKER_CONFIG: ${{ runner.temp }}/.docker
338+
# Overlay current-main image refs onto every package this PR did NOT
339+
# rebuild. Without it, unbuilt packages fall back to the last-release refs
340+
# committed in the tree (e.g. v1.5.0) even when main is hundreds of commits
341+
# ahead — so e2e and the installer's packages artifact would ship stale
342+
# images for everything outside this PR's build matrix. Source of truth is
343+
# the cozystack-packages:main OCI artifact build-main.yaml already
344+
# publishes (the whole current-main packages tree, refs digest-pinned to
345+
# main's images). Runs after login (creds available) and before the
346+
# installer build below so the flux-pushed packages tree — and the
347+
# pr.patch e2e applies — already reflect the overlay. A missing artifact
348+
# (build-main not yet green) is a no-op: packages keep their committed
349+
# refs (prior behaviour). See hack/overlay-main-images.sh.
350+
- name: Pull current-main packages tree
351+
run: |
352+
rm -rf _out/mainpkgs
353+
flux pull artifact "oci://${REGISTRY}/cozystack-packages:main" \
354+
--output _out/mainpkgs \
355+
|| { echo "No cozystack-packages:main yet — keeping committed refs"; rm -rf _out/mainpkgs; }
356+
env:
357+
DOCKER_CONFIG: ${{ runner.temp }}/.docker
358+
- name: Overlay current-main refs for unbuilt packages
359+
# Best-effort: this only improves WHICH images e2e and the installer use,
360+
# so an unexpected failure must degrade to the committed refs (prior
361+
# behaviour), never block the PR. overlay-main-images.sh already handles
362+
# the expected paths (missing artifact, drift, absent unit) internally.
363+
run: |
364+
hack/overlay-main-images.sh _out/mainpkgs "$BUILD_MATRIX" \
365+
|| echo "::warning::overlay-main-images failed; unbuilt packages keep committed refs"
366+
env:
367+
# The set of packages THIS PR rebuilt; their pr-<N>-<sha> refs are
368+
# authoritative, so overlay-main-images.sh skips them. Computed by the
369+
# plan job from the Makefile (a JSON array of package dirs), not from
370+
# untrusted PR input — passed via env per workflow-injection hygiene.
371+
BUILD_MATRIX: ${{ needs.plan.outputs.matrix }}
338372
# buildkitd-config routes docker.io base-image pulls through mirror.gcr.io
339373
# (hack/buildkitd.toml) so the installer build dodges Docker Hub's
340374
# anonymous-pull rate limit; see the build job for the full rationale.

‎hack/overlay-main-images.sh‎

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
#!/bin/sh
2+
# Overlay current-main image references onto packages a PR did NOT rebuild.
3+
#
4+
# Source of truth: the cozystack-packages:main OCI artifact build-main.yaml
5+
# already publishes (`make build` -> packages/core/installer image-packages does
6+
# `flux push artifact oci://$REGISTRY/cozystack-packages:main --path=packages`).
7+
# That artifact is the ENTIRE current-main packages tree with every image
8+
# reference digest-pinned to the images build-main just pushed.
9+
#
10+
# We walk EVERY ref-bearing file (values.yaml + images/*.tag) in that tree and
11+
# overlay the repo's copy when it differs only in image-reference lines. This is
12+
# deliberately driven by the artifact, NOT the per-package build matrix, so it
13+
# covers the WHOLE first-party tree — apps/, system/, core/, AND extra/,
14+
# library/, ... For any file build-main did not rebuild, the artifact copy is
15+
# byte-identical to the committed copy and is skipped; only files carrying a
16+
# rebuilt current-main digest differ and get overlaid. So a fix merged to ANY
17+
# first-party image (e.g. the objectstorage-sidecar ref under extra/seaweedfs)
18+
# takes effect in e2e and the installer artifact, instead of the frozen
19+
# last-release refs committed in the repo (e.g. v1.5.0).
20+
#
21+
# Skipped:
22+
# - units the PR itself rebuilt (BUILT_JSON, the plan job's matrix): their
23+
# pr-<N>-<sha> refs, already applied in finalize, are authoritative.
24+
# - packages/core/talos and packages/core/installer: rebuilt unconditionally
25+
# by their dedicated jobs (build-talos / the finalize installer build),
26+
# which own those files — never overlay them.
27+
# - vendored charts/ subtrees: upstream chart values we do not build.
28+
#
29+
# Surgical + self-validating: a file is overlaid only when every line that
30+
# differs from the current-main version is image-reference-bearing. The repo's
31+
# committed refs use the release registry (ghcr) while the artifact uses the CI
32+
# registry (OCIR), so refs differ in registry host / tag / digest — and charts
33+
# split those across separate lines (`repository:`, `tag:`, `digest:`), not all
34+
# of which carry '@sha256:'. So a changed line counts as image-related if it
35+
# carries '@sha256:' OR its key is image/repository/registry/tag/digest (or it
36+
# is a `--…-image=` arg). If any OTHER line differs — the PR branched from a
37+
# main whose config for that file differs from the artifact's base — the file is
38+
# left on its committed ref (safe degradation, logged, never fatal). CI checks
39+
# out the pull_request merge commit (current main + PR), so for an unbuilt file
40+
# the config already matches current main and only ref lines differ.
41+
#
42+
# Usage: hack/overlay-main-images.sh <mainpkgs-dir> <built-matrix-json>
43+
# <mainpkgs-dir> = extracted cozystack-packages:main tree; its root holds
44+
# apps/ system/ core/ extra/ ... (the CONTENTS of
45+
# packages/).
46+
# (run from the repo root)
47+
set -eu
48+
49+
MAINPKGS="${1:?usage: overlay-main-images.sh <mainpkgs-dir> <built-matrix-json>}"
50+
MAINPKGS="${MAINPKGS%/}"
51+
BUILT_JSON="${2:-[]}"
52+
53+
if [ ! -d "$MAINPKGS" ]; then
54+
echo "No current-main packages tree at $MAINPKGS — unbuilt packages keep committed refs"
55+
exit 0
56+
fi
57+
58+
# Dirs whose files must never be overlaid: the two units owned by dedicated
59+
# jobs, plus every unit the PR rebuilt. Space-wrapped for whole-token matching.
60+
skip=" packages/core/talos packages/core/installer $(echo "$BUILT_JSON" | tr -d '[]"' | tr ',' ' ') "
61+
62+
# A changed line is image-reference-bearing if it carries a full ref (@sha256:),
63+
# is a split ref key (image/repository/registry/tag/digest), or a `--…-image=` arg.
64+
img_line='(@sha256:|^[[:space:]]*(- )?(image|repository|registry|tag|digest):|--[A-Za-z-]*image=)'
65+
66+
overlaid=0
67+
same=0
68+
skipped=0
69+
drift=0
70+
failed=0
71+
# Walk the artifact's ref-bearing files, pruning vendored charts/ subtrees.
72+
# `for … in $(find)` (not a pipe) keeps the counters in this shell; package
73+
# paths and filenames carry no spaces (same assumption as hack/build-matrix.sh).
74+
for new in $(find "$MAINPKGS" -type d -name charts -prune -o \
75+
\( -name values.yaml -o -name '*.tag' \) -type f -print 2>/dev/null); do
76+
# Artifact root == contents of packages/, so map back by re-adding the prefix.
77+
cur="packages/${new#"$MAINPKGS"/}"
78+
79+
in_skip=0
80+
for d in $skip; do
81+
case "$cur" in "$d"/*) in_skip=1; break ;; esac
82+
done
83+
if [ "$in_skip" -eq 1 ]; then
84+
skipped=$((skipped + 1))
85+
continue
86+
fi
87+
88+
[ -f "$cur" ] || continue # not in the PR tree -> don't introduce it
89+
cmp -s "$cur" "$new" && { same=$((same + 1)); continue; }
90+
91+
if diff "$cur" "$new" | sed -n 's/^[<>] //p' | grep -qvE "$img_line"; then
92+
echo "drift (non-ref change) in $cur -> keeping committed ref"
93+
drift=$((drift + 1))
94+
continue
95+
fi
96+
97+
if cp "$new" "$cur"; then
98+
echo "overlay: $cur -> current-main"
99+
overlaid=$((overlaid + 1))
100+
else
101+
echo "WARN: cp failed for $cur (keeping committed ref)"
102+
failed=$((failed + 1))
103+
fi
104+
done
105+
106+
echo "Overlay current-main images: overlaid=$overlaid same=$same skipped(rebuilt/owned)=$skipped drift=$drift failed=$failed"

‎hack/overlay-main-images_test.bats‎

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
#!/usr/bin/env bats
2+
# Unit tests for hack/overlay-main-images.sh — the PR-finalize step that points
3+
# packages a PR did NOT rebuild at the current-main images from the
4+
# cozystack-packages:main artifact.
5+
#
6+
# Run from the repo root: bats hack/overlay-main-images_test.bats
7+
# (CI runs it via hack/cozytest.sh through `make unit-tests`.)
8+
#
9+
# Each test builds a throwaway tree: a `packages/` tree on release (ghcr/v1.5.0)
10+
# refs, and a `main/` dir standing in for the extracted cozystack-packages:main
11+
# artifact (root = contents of packages/) on current-main (OCIR/:main) refs.
12+
# $root is the real repo, captured before cd.
13+
14+
@test "overlays an unbuilt unit (.tag) to current-main and reports it" {
15+
root=$(pwd)
16+
w=$(mktemp -d); trap 'rm -rf "$w"' EXIT
17+
mkdir -p "$w/packages/apps/foo/images" "$w/main/apps/foo/images"
18+
echo 'ghcr.io/cozystack/cozystack/foo:v1.5.0@sha256:aaaa' > "$w/packages/apps/foo/images/foo.tag"
19+
echo 'iad.ocir.io/x/cozystack/foo:main@sha256:bbbb' > "$w/main/apps/foo/images/foo.tag"
20+
cd "$w"
21+
out=$("$root/hack/overlay-main-images.sh" main '[]')
22+
grep -q 'foo:main@sha256:bbbb' packages/apps/foo/images/foo.tag
23+
echo "$out" | grep -q 'overlaid=1'
24+
}
25+
26+
@test "overlays a split-form ref (repository/tag/digest, no @sha256 on those lines)" {
27+
root=$(pwd)
28+
w=$(mktemp -d); trap 'rm -rf "$w"' EXIT
29+
mkdir -p "$w/packages/system/split" "$w/main/system/split"
30+
printf 'image:\n repository: ghcr.io/cozystack/cozystack/split\n tag: v1.5.0\n digest: "sha256:aaaa"\n' > "$w/packages/system/split/values.yaml"
31+
printf 'image:\n repository: iad.ocir.io/x/cozystack/split\n tag: main\n digest: "sha256:bbbb"\n' > "$w/main/system/split/values.yaml"
32+
cd "$w"
33+
"$root/hack/overlay-main-images.sh" main '[]'
34+
grep -q 'repository: iad.ocir.io/x/cozystack/split' packages/system/split/values.yaml
35+
grep -q 'tag: main' packages/system/split/values.yaml
36+
grep -q 'sha256:bbbb' packages/system/split/values.yaml
37+
}
38+
39+
@test "overlays an extra/* unit (outside the per-package build matrix)" {
40+
root=$(pwd)
41+
w=$(mktemp -d); trap 'rm -rf "$w"' EXIT
42+
mkdir -p "$w/packages/extra/seaweedfs/images" "$w/main/extra/seaweedfs/images"
43+
echo 'ghcr.io/cozystack/cozystack/objectstorage-sidecar:v1.5.0@sha256:aaaa' > "$w/packages/extra/seaweedfs/images/objectstorage-sidecar.tag"
44+
echo 'iad.ocir.io/x/cozystack/objectstorage-sidecar:main@sha256:bbbb' > "$w/main/extra/seaweedfs/images/objectstorage-sidecar.tag"
45+
cd "$w"
46+
"$root/hack/overlay-main-images.sh" main '[]'
47+
grep -q 'objectstorage-sidecar:main@sha256:bbbb' packages/extra/seaweedfs/images/objectstorage-sidecar.tag
48+
}
49+
50+
@test "skips a unit the PR rebuilt (its pr-<N>-<sha> ref wins)" {
51+
root=$(pwd)
52+
w=$(mktemp -d); trap 'rm -rf "$w"' EXIT
53+
mkdir -p "$w/packages/apps/foo" "$w/main/apps/foo"
54+
echo 'image: ghcr.io/cozystack/cozystack/foo:v1.5.0@sha256:aaaa' > "$w/packages/apps/foo/values.yaml"
55+
echo 'image: iad.ocir.io/x/cozystack/foo:main@sha256:bbbb' > "$w/main/apps/foo/values.yaml"
56+
cd "$w"
57+
"$root/hack/overlay-main-images.sh" main '["packages/apps/foo"]'
58+
grep -q 'foo:v1.5.0@sha256:aaaa' packages/apps/foo/values.yaml
59+
}
60+
61+
@test "never overlays core/talos or core/installer (owned by dedicated jobs)" {
62+
root=$(pwd)
63+
w=$(mktemp -d); trap 'rm -rf "$w"' EXIT
64+
mkdir -p "$w/packages/core/installer" "$w/main/core/installer" "$w/packages/core/talos" "$w/main/core/talos"
65+
echo 'image: ghcr.io/cozystack/cozystack/cozystack-operator:v1.5.0@sha256:aaaa' > "$w/packages/core/installer/values.yaml"
66+
echo 'image: iad.ocir.io/x/cozystack/cozystack-operator:main@sha256:bbbb' > "$w/main/core/installer/values.yaml"
67+
echo 'image: ghcr.io/cozystack/cozystack/talos:v1.5.0@sha256:cccc' > "$w/packages/core/talos/values.yaml"
68+
echo 'image: iad.ocir.io/x/cozystack/talos:main@sha256:dddd' > "$w/main/core/talos/values.yaml"
69+
cd "$w"
70+
"$root/hack/overlay-main-images.sh" main '[]'
71+
grep -q 'cozystack-operator:v1.5.0@sha256:aaaa' packages/core/installer/values.yaml
72+
grep -q 'talos:v1.5.0@sha256:cccc' packages/core/talos/values.yaml
73+
}
74+
75+
@test "does not descend into vendored charts/ subtrees" {
76+
root=$(pwd)
77+
w=$(mktemp -d); trap 'rm -rf "$w"' EXIT
78+
mkdir -p "$w/packages/system/bar/charts/sub" "$w/main/system/bar/charts/sub"
79+
echo 'image: ghcr.io/cozystack/cozystack/sub:v1.5.0@sha256:aaaa' > "$w/packages/system/bar/charts/sub/values.yaml"
80+
echo 'image: iad.ocir.io/x/cozystack/sub:main@sha256:bbbb' > "$w/main/system/bar/charts/sub/values.yaml"
81+
cd "$w"
82+
"$root/hack/overlay-main-images.sh" main '[]'
83+
grep -q 'sub:v1.5.0@sha256:aaaa' packages/system/bar/charts/sub/values.yaml
84+
}
85+
86+
@test "keeps the committed ref when a non-ref line differs (drift)" {
87+
root=$(pwd)
88+
w=$(mktemp -d); trap 'rm -rf "$w"' EXIT
89+
mkdir -p "$w/packages/system/drift" "$w/main/system/drift"
90+
printf 'tuning: old\nimage: ghcr.io/cozystack/cozystack/drift:v1.5.0@sha256:aaaa\n' > "$w/packages/system/drift/values.yaml"
91+
printf 'tuning: new\nimage: iad.ocir.io/x/cozystack/drift:main@sha256:bbbb\n' > "$w/main/system/drift/values.yaml"
92+
cd "$w"
93+
"$root/hack/overlay-main-images.sh" main '[]'
94+
grep -q 'drift:v1.5.0@sha256:aaaa' packages/system/drift/values.yaml
95+
grep -q 'tuning: old' packages/system/drift/values.yaml
96+
}
97+
98+
@test "missing artifact directory is a no-op and exits 0" {
99+
root=$(pwd)
100+
w=$(mktemp -d); trap 'rm -rf "$w"' EXIT
101+
mkdir -p "$w/packages/apps/foo"
102+
echo 'image: ghcr.io/cozystack/cozystack/foo:v1.5.0@sha256:aaaa' > "$w/packages/apps/foo/values.yaml"
103+
cd "$w"
104+
"$root/hack/overlay-main-images.sh" does-not-exist '[]'
105+
grep -q 'foo:v1.5.0@sha256:aaaa' packages/apps/foo/values.yaml
106+
}

0 commit comments

Comments
 (0)