Skip to content

Commit 41ef4a6

Browse files
fix(ci): teach the overlay to repair an unstamped images/*.tag ref (#4362)
## What `hack/overlay-main-images.sh` walks `images/*.tag`, but every branch of its line classifier needs either a YAML key or an `@sha256:`. A `.tag` file holds one bare reference and nothing else, so until a build stamps it the line has neither, the drift guard reads it as a non-ref change, and the committed reference survives. For a package onboarded with the `:v0.0.0` placeholder this repo already uses for a new image, that surviving reference is one no registry serves. The overlay is the only thing that would replace it for a PR that does not rebuild the package, so once such a package is on main every full-suite run that skips its build keeps the placeholder. The same placeholder inside a `values.yaml` repairs itself, because there it sits under an `image:` key the classifier recognises. The difference is the file shape, not the placeholder. ## Reproduction A fixture holding a committed bare tag against an artifact copy carrying a stamped one: ``` $ cd fixture/fakerepo && sh ../../hack/overlay-main-images.sh ../mainpkgs '[]' '' overlay: packages/system/x/values.yaml -> current-main drift (non-ref change) in packages/apps/x/images/x.tag -> keeping committed ref ``` Both files differ only in their reference. The `values.yaml` one is repaired; the `.tag` one is not. Adding any digest to the same placeholder makes it pass, which is what narrows this to the classifier rather than the file. ## Why it has not bitten yet 17 of the 19 `.tag` files in the tree carry `@sha256:` and pass on that branch alone. The only other digest-less one is `packages/extra/seaweedfs/images/seaweedfs-cosi-driver.tag`, and it survives only because it is third-party: `build-main` never rebuilds it, the artifact copy is byte-identical, and the classifier is never reached. The first digest-less first-party `.tag` is what turns the gap live, and one is coming in #3426. ## The fix `hack/promote-rewrite-tags.sh` already carries a second match branch for exactly this shape, with a comment saying why ("the second branch is what catches a bare `.tag` file, whose content has no YAML key at all"). This gives the overlay the same one, scoped to `*.tag` so an unkeyed line in a `values.yaml` still counts as drift, because only in a `.tag` file is a whole line known to be a reference and nothing else. Two tests in `hack/overlay-main-images_test.bats`: the unstamped placeholder is overlaid, and a `.tag` whose non-reference line differs still counts as drift, so the rule cannot quietly widen into "every line in a `.tag` file is a reference". Removing the fix fails the first one. ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Bug Fixes** - Fixed image overlay handling for `.tag` files containing bare image references, allowing current references to be applied correctly. - Preserved drift detection when `.tag` files also contain unrelated, non-reference changes. - **Documentation** - Clarified how image-reference tooling handles bare references and package-tree processing. - **Tests** - Added coverage for bare-reference overlays and mixed reference/comment changes. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents c0ff783 + 1118b34 commit 41ef4a6

3 files changed

Lines changed: 58 additions & 2 deletions

File tree

‎docs/agents/image-refs.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -59,7 +59,7 @@ If a package stamps a reference somewhere the two globs do not cover, add that f
5959
| `hack/nightly-mirror.sh` | all three shapes | copy each digest to the public registry, then rewrite hosts in the same file set |
6060
| `hack/overlay-main-images.sh` | `values.yaml` + `*.tag` | overlay current-main references onto packages a PR did not rebuild |
6161

62-
Two rows do not source the shared library, for opposite reasons. `hack/overlay-main-images.sh` reads references and walks the tree itself, so it is a real exception to the invariant above: a file newly declared in `IMAGE_REF_EXTRA_FILES` does not automatically reach it. `hack/promote-packages-artifact.sh` is not a consumer of references at all — it hands the whole packages tree to `flux push artifact` and touches exactly one reference, the installer's own `platformSourceUrl`/`platformSourceRef` pair, by key with `yq`. Every shape travels inside the artifact because every file does, so there is nothing there for the enumeration to widen and no gap for a newly declared file to fall into.
62+
Two rows do not source the shared library, for opposite reasons. `hack/overlay-main-images.sh` reads references and walks the tree itself, so it is a real exception to the invariant above: a file newly declared in `IMAGE_REF_EXTRA_FILES` does not automatically reach it. Its line classifier also has to know the bare-reference shape by itself: a `*.tag` file holds one reference with no YAML key around it, and until a build stamps it there is no `@sha256:` either, so a changed line there is recognised by shape rather than by key, scoped to `*.tag` because only there is a whole line known to be a reference and nothing else. `hack/promote-rewrite-tags.sh` carries a second match branch for that same shape and for the same reason. `hack/promote-packages-artifact.sh` is not a consumer of references at all — it hands the whole packages tree to `flux push artifact` and touches exactly one reference, the installer's own `platformSourceUrl`/`platformSourceRef` pair, by key with `yq`. Every shape travels inside the artifact because every file does, so there is nothing there for the enumeration to widen and no gap for a newly declared file to fall into.
6363

6464
`nightly-mirror.sh` is the one where a missed file is worst. The set it rewrites hosts in must equal the set it mirrored: a file rewritten but not mirrored leaves a dangling reference to an image never pushed, and a file mirrored but not rewritten leaves the published tree pointing at the private build registry.
6565

‎hack/overlay-main-images.sh‎

Lines changed: 22 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -105,6 +105,24 @@ skip=" packages/core/talos packages/core/installer $(echo "$BUILT_JSON" | tr -d
105105
# `tag:` and `digest:` legitimately hold values with no slash in them.
106106
img_line='(@sha256:|^[[:space:]]*(- )?(image|repository|registry|tag|digest):|^[[:space:]]*(- )?[A-Za-z]+[Ii]mage:[[:space:]]*"?[^"[:space:]]*/|--[A-Za-z-]*image=)'
107107

108+
# ...and, in an images/*.tag file only, a bare reference: a whole line that is
109+
# one ref with no YAML key around it, which is the entire format of that file
110+
# (hack/lib/image-refs.sh: "a plain file holding one ref"). Every branch above
111+
# needs either a key or an `@sha256:`, and a .tag file carries neither until a
112+
# build stamps it, so an unstamped one is classified as a non-ref change, the
113+
# drift guard keeps the committed ref, and the overlay never replaces the
114+
# placeholder it exists to replace. 17 of the 19 .tag files in the tree pass on
115+
# `@sha256:` alone, which is why this went unnoticed; the exception that proves
116+
# it is packages/extra/seaweedfs/images/seaweedfs-cosi-driver.tag, digest-less
117+
# but third-party, so build-main never rebuilds it, the artifact copy is
118+
# byte-identical and this test is never reached.
119+
#
120+
# Scoped to *.tag because only there is a whole line known to be a reference and
121+
# nothing else. An unkeyed line in a values.yaml is genuinely unclassifiable and
122+
# must keep counting as drift. hack/promote-rewrite-tags.sh carries a second
123+
# match branch for this same shape and for this same reason.
124+
bare_ref_line='^[^[:space:]#][^[:space:]]*/[^[:space:]]+$'
125+
108126
overlaid=0
109127
same=0
110128
skipped=0
@@ -139,8 +157,11 @@ for new in $(find "$MAINPKGS" -type d -name charts -prune -o \
139157
# behind #4265 it was the entire remaining drift list: core/platform,
140158
# cozystack-api, dashboard, kubeovn-webhook and linstor-gui each differed
141159
# from the artifact by one blank line and their own ref, nothing else.
160+
line_pat="$img_line"
161+
case "$cur" in *.tag) line_pat="($img_line)|($bare_ref_line)" ;; esac
162+
142163
if diff "$cur" "$new" | sed -n 's/^[<>] //p' \
143-
| grep -vE '^[[:space:]]*$' | grep -qvE "$img_line"; then
164+
| grep -vE '^[[:space:]]*$' | grep -qvE "$line_pat"; then
144165
echo "drift (non-ref change) in $cur -> keeping committed ref"
145166
drift=$((drift + 1))
146167
# Name the PACKAGE, not the directory the file happens to sit in: an

‎hack/overlay-main-images_test.bats‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,41 @@
2424
echo "$out" | grep -q 'overlaid=1'
2525
}
2626

27+
@test "overlays an UNSTAMPED .tag placeholder (bare ref, no key and no digest)" {
28+
# The shape a newly onboarded first-party image is committed in, before any
29+
# build has stamped it. Its one line carries no YAML key and no @sha256, so the
30+
# generic classifier reads it as a non-ref change and the drift guard keeps it,
31+
# leaving in the tree a reference no registry serves, for every later PR that
32+
# does not rebuild that package. No EXIT trap here: see the debt note at the
33+
# top of this file; on failure the scratch dir survives for inspection.
34+
root=$(pwd)
35+
w=$(mktemp -d)
36+
mkdir -p "$w/packages/apps/newpkg/images" "$w/main/apps/newpkg/images"
37+
echo 'ghcr.io/cozystack/cozystack/newpkg:v0.0.0' > "$w/packages/apps/newpkg/images/newpkg.tag"
38+
echo 'iad.ocir.io/x/cozystack/newpkg:main@sha256:bbbb' > "$w/main/apps/newpkg/images/newpkg.tag"
39+
cd "$w"
40+
out=$("$root/hack/overlay-main-images.sh" main '[]')
41+
grep -q 'newpkg:main@sha256:bbbb' packages/apps/newpkg/images/newpkg.tag
42+
echo "$out" | grep -q 'drift=0'
43+
cd "$root"; rm -rf "$w"
44+
}
45+
46+
@test "keeps a .tag whose NON-ref line differs (the bare-ref rule is scoped, not blanket)" {
47+
# The bare-ref rule says a whole line that is one ref counts as a ref line in a
48+
# .tag file. It must not become "every line in a .tag file is a ref": a line
49+
# the classifier cannot read is still drift, there as anywhere else.
50+
root=$(pwd)
51+
w=$(mktemp -d)
52+
mkdir -p "$w/packages/apps/noted/images" "$w/main/apps/noted/images"
53+
printf '# pinned by hand, see #1234\nghcr.io/cozystack/cozystack/noted:v1.5.0@sha256:aaaa\n' > "$w/packages/apps/noted/images/noted.tag"
54+
printf '# pinned by hand, see #5678\niad.ocir.io/x/cozystack/noted:main@sha256:bbbb\n' > "$w/main/apps/noted/images/noted.tag"
55+
cd "$w"
56+
out=$("$root/hack/overlay-main-images.sh" main '[]')
57+
grep -q 'noted:v1.5.0@sha256:aaaa' packages/apps/noted/images/noted.tag
58+
echo "$out" | grep -q 'drift=1'
59+
cd "$root"; rm -rf "$w"
60+
}
61+
2762
@test "overlays a split-form ref (repository/tag/digest, no @sha256 on those lines)" {
2863
root=$(pwd)
2964
w=$(mktemp -d); trap 'rm -rf "$w"' EXIT

0 commit comments

Comments
 (0)