Skip to content

Commit 25091a7

Browse files
committed
fix(cdi): stop make update from deleting first-party templates
The update recipes in kubevirt-cdi and kubevirt-cdi-operator opened with `rm -rf templates` and then refetched one upstream manifest each. kubevirt-cdi keeps two more files in templates/ -- the upload-proxy Ingress and TLSRoute -- that are written by hand and never refetched, so the next CDI bump would drop upload-proxy exposure from the chart while `make update` still exited 0. The clean slate bought nothing either way: wget -O truncates the file it writes. Both recipes now create the directory with `mkdir -p` and leave the rest of it alone, which is what kubevirt-operator moved to when it was last bumped. kubevirt-cdi-operator has nothing local to lose today; it changes so the three recipes read the same way, and its one template is rewritten on every run either way. What it gives up is the wipe's incidental cleanup, which costs nothing yet: that recipe's patch call runs at a strip level resolving to no file, so it exits 1 without rejecting a hunk at all. Once that is fixed a .rej stays put. The bats case scans the package Makefiles and fails when a recipe that clears its own templates/ leaves behind a tracked file it never names. The search skips the boilerplate `export` lines every package Makefile opens with, or a package called foo-bar would satisfy the check for its own templates/bar.yaml through its name alone. No package in the tree needs that exclusion today, since each also names its templates on a recipe line, so it closes a channel rather than a live hole. A removal reaching into another package is outside the scan's range, and the header records the one instance of that in the tree. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent d3559d4 commit 25091a7

3 files changed

Lines changed: 291 additions & 4 deletions

File tree

‎hack/package-update-templates.bats‎

Lines changed: 285 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,285 @@
1+
#!/usr/bin/env bats
2+
# -----------------------------------------------------------------------------
3+
# Invariant for the package `update` recipes that clear templates/ wholesale.
4+
#
5+
# `update` refetches a package's vendored upstream manifests, and several
6+
# recipes open by deleting templates/ to guarantee a clean slate. That is safe
7+
# only while every file under templates/ comes back from the recipe. templates/
8+
# is an ordinary chart directory, so a package can also keep first-party
9+
# manifests there -- kubevirt-cdi keeps the CDI upload-proxy Ingress and
10+
# TLSRoute -- and the recipe knows nothing about those. The wipe takes them,
11+
# `make update` exits 0, and the loss reaches a cluster as an object that
12+
# silently stopped being rendered at the next version bump, not as a failed
13+
# command.
14+
#
15+
# The invariant: a recipe that clears templates/ must NAME every file tracked
16+
# under it. Both halves are approximations of the shell the recipe would run,
17+
# and each is argued at its helper below: the wipe is a text match over the
18+
# recipe's lines rather than a parse, and "names" means the file's basename
19+
# minus its extension appears somewhere in the Makefile -- a mention, not proof
20+
# of a write. What that catches is the shape the CDI packages had, a file the
21+
# Makefile does not refer to at all: every recipe in the tree writes each path
22+
# it produces, so a name appearing nowhere in the file is a name nothing
23+
# refetches. A recipe that unpacked an archive or split a stream into
24+
# templates/ would produce files it never names and break that reading; none
25+
# does today.
26+
#
27+
# The tree is scanned rather than a list of packages pinned, so a package that
28+
# grows its first first-party manifest under a wiping recipe of its own is
29+
# caught by the commit that adds it. The packages that clear their own
30+
# templates/ today refetch all of it, which is what keeps this green; the two
31+
# CDI packages stopped clearing it, which is what this file was written for.
32+
#
33+
# What that leaves uncovered is a recipe clearing SOMEONE ELSE'S templates/, and
34+
# the tree has one: packages/system/cert-manager/Makefile runs
35+
# `rm -rf ../cert-manager-crds/templates/*` and moves the CRDs it pulled into
36+
# that directory, while cert-manager-crds declares no update target at all. A
37+
# first-party manifest added there is deleted by the next `make update` in
38+
# cert-manager and nothing below reports it. Covering that means resolving the
39+
# removal's target directory rather than reading the wiping package's own, which
40+
# is a second helper; the pattern below is deliberately not widened to accept a
41+
# path prefix, because that alone would fire on cert-manager and then list the
42+
# templates of a directory it does not have -- the removal names the sibling's
43+
# path, so the lookup has to follow it.
44+
#
45+
# The scan alone would also be green against a detector that finds nothing, so
46+
# the control below runs the same two helpers over a fixture that does hold an
47+
# unrefetched file, and over that fixture with the removal taken out.
48+
#
49+
# Constraints from cozytest.sh: it rewrites the closing brace of every function
50+
# in this file into `return 0`, so a helper cannot report through its exit
51+
# status -- both report through stdout -- and it injects `set -e`, so statuses
52+
# are captured explicitly.
53+
#
54+
# Run with: hack/cozytest.sh hack/package-update-templates.bats
55+
# -----------------------------------------------------------------------------
56+
57+
# Prints the recipe line that clears templates/ wholesale, for a package
58+
# Makefile whose `update` target has one. Silent otherwise, including for a
59+
# Makefile with no `update` target at all.
60+
#
61+
# The recipe is read as the run of tab-indented lines after `update:`. make also
62+
# ignores blank lines inside a recipe and this stops at the first one, so such a
63+
# recipe would be read short; none in the tree is written that way.
64+
#
65+
# The match accepts `rm -rf`, `rm -fr` and `rm -r` against `templates` and
66+
# `./templates`, bare, with a trailing slash, or with a `/*` glob -- the three
67+
# spellings that empty the directory. It deliberately does not accept
68+
# `rm -rf templates/one-file.yml`, which takes a single file the recipe goes on
69+
# to refetch (rabbitmq-operator), nor a narrower glob such as
70+
# `rm -rf templates/*.yaml`, which takes a subset the same way.
71+
#
72+
# It is a text match and not a shell parse, so a quoted `rm -rf "templates"`, a
73+
# removal reached through a variable, a `find -delete` and a `cd` into another
74+
# directory first are all invisible to it, as is a path-prefixed target -- the
75+
# cert-manager case in the header is one, and the prefix is the only thing
76+
# keeping it out: the `/*` it ends with is accepted. The enumeration is only as
77+
# complete as the last reader made it, so extend it in the same edit as the
78+
# pattern.
79+
wipe_command() {
80+
awk '/^update:/ { r = 1; next } r && /^\t/ { print; next } r { exit }' "$1" |
81+
grep -E 'rm[[:space:]]+-(rf|fr|r)[[:space:]]+(\./)?templates(/\*?)?([[:space:]]|;|&|$)' || true
82+
}
83+
84+
# Prints every file tracked under a package's templates/ that its Makefile
85+
# never mentions, one path per line. Whether that matters is the caller's call:
86+
# it matters for a package whose recipe clears the directory, and for no other.
87+
#
88+
# The listing is the index, so only a tracked file is at stake -- it is the one
89+
# the next `make update` deletes out from under a reviewer who has no reason to
90+
# look. Reading the working tree instead would fail a package over the
91+
# developer's own litter: an untracked .orig from a hand-applied patch, a
92+
# half-written template, a .DS_Store. A guard that goes red on scratch files is
93+
# a guard that gets switched off.
94+
#
95+
# The extension is stripped before matching, because a recipe may assemble the
96+
# filename from a variable holding the bare stem -- etcd-operator-crds writes
97+
# templates/$$c.yaml from a CRDS list of names without one. A basename whose
98+
# only dot is its first character keeps the whole name instead: stripping there
99+
# leaves the empty string, which grep matches against every Makefile, and the
100+
# file would pass without being looked at. The search covers the whole Makefile
101+
# rather than the recipe alone for the same reason as the stem: the variable is
102+
# assigned above the target. Both make the match generous, so a stem that
103+
# appears for an unrelated reason satisfies it; this reports the files nothing in
104+
# the Makefile refers to, not every file the recipe fails to write.
105+
#
106+
# Comment lines are dropped before the search, so a filename written into one is
107+
# not a mention: a rationale comment above a recipe, naming the very file it is
108+
# about, would otherwise exempt that package from its own guard. Precautionary
109+
# rather than a fix -- no package in the tree is exempted that way today, and
110+
# the one recipe comment that does name a template, in kubevirt-instancetypes,
111+
# names it on a `cp` line as well.
112+
#
113+
# The top-level `export` lines go the same way, and for the same reason. Every
114+
# package Makefile opens with `export NAME=` and `export NAMESPACE=`, so a
115+
# package called foo-bar whose recipe writes templates/bar.yaml has that file's
116+
# stem inside its own name and satisfies the search on that alone. Four
117+
# packages match their own name that way -- kubevirt-cdi-operator
118+
# (cdi-operator), forklift-operator, objectstorage-controller (controller) and
119+
# kubevirt-instancetypes (instancetypes) -- but each of them also names every
120+
# template on a real recipe line, so dropping the exports reports nothing that
121+
# was not reported before and exempts no package today. Precautionary like the
122+
# comment strip above: the channel is open and nothing in the tree is going
123+
# through it. Only NAME, NAMESPACE and the four chart-identity variables are
124+
# exported at top level and none of them is a way to name a template; the
125+
# anchor is deliberate, so the tab-indented `export VERSION=` inside a recipe
126+
# still counts as a mention.
127+
unnamed_templates() {
128+
_mk=$1
129+
_dir=$(dirname "$_mk")
130+
_code=$(grep -vE '^([[:space:]]*@?#|export [A-Z_]+=)' "$_mk") || true
131+
git -C "$_dir" ls-files -- templates | while IFS= read -r _f; do
132+
_base=${_f##*/}
133+
_stem=${_base%.*}
134+
[ -n "$_stem" ] || _stem=$_base
135+
printf '%s\n' "$_code" | grep -Fq -- "$_stem" || printf '%s\n' "$_dir/$_f"
136+
done
137+
}
138+
139+
@test "no package update recipe clears templates/ over a file it never refetches" {
140+
examined=0
141+
offenders=""
142+
143+
for mk in packages/*/*/Makefile; do
144+
[ -f "$mk" ] || continue
145+
examined=$((examined + 1))
146+
[ -n "$(wipe_command "$mk")" ] || continue
147+
found=$(unnamed_templates "$mk")
148+
if [ -n "$found" ]; then
149+
offenders="$offenders
150+
$mk clears templates/ but names none of:
151+
$found"
152+
fi
153+
done
154+
155+
# A glob that matched nothing -- packages/ moved, or the suite run from
156+
# somewhere other than the repository root -- would otherwise report success
157+
# having looked at no Makefile at all.
158+
if [ "$examined" -eq 0 ]; then
159+
echo "no packages/*/*/Makefile found: run this from the repository root" >&2
160+
exit 1
161+
fi
162+
163+
# The index is where the scan reads a package's templates from, so an answer
164+
# of nothing here means it could not have reported anything either.
165+
if [ -z "$(git ls-files -- packages | head -1)" ]; then
166+
echo "git ls-files reported no tracked file under packages/: this test" >&2
167+
echo "cannot see a template directory, so it measured nothing" >&2
168+
exit 1
169+
fi
170+
171+
if [ -n "$offenders" ]; then
172+
echo "a package update recipe deletes templates/ over a file it does not refetch:" >&2
173+
echo "$offenders" >&2
174+
echo "" >&2
175+
echo "templates/ holds first-party manifests alongside the vendored ones and" >&2
176+
echo "'rm -rf templates' takes both, silently: the next 'make update' drops the" >&2
177+
echo "first-party ones and still exits 0. Either keep 'mkdir -p templates'" >&2
178+
echo "without the removal -- wget -O and kustomize > overwrite what they fetch" >&2
179+
echo "anyway -- or write the local file from the recipe too, where the clean" >&2
180+
echo "slate is load-bearing (etcd-operator-crds stages into a tmpdir first)." >&2
181+
exit 1
182+
fi
183+
}
184+
185+
@test "the scan reports a template file that a wiping recipe does not name" {
186+
# Negative control. The test above passes on a tree with no offender and on
187+
# a detector that cannot see one; this separates the two by handing the same
188+
# helpers a package that is broken in exactly the way kubevirt-cdi was.
189+
tmp=$(mktemp -d)
190+
mkdir -p "$tmp/pkg/templates"
191+
: > "$tmp/pkg/templates/fetched.yaml"
192+
: > "$tmp/pkg/templates/local-extra.yaml"
193+
# A tracked dotfile. Stripping its extension leaves nothing and an empty
194+
# pattern matches every Makefile, so without the guard in the helper this
195+
# file is skipped in silence rather than reported.
196+
: > "$tmp/pkg/templates/.gitkeep"
197+
# A file whose stem sits inside the package's own name, the shape every
198+
# `foo-bar` package writing templates/bar.yaml has. `export NAME=` is in
199+
# every package Makefile, so before that line was dropped from the search
200+
# this file matched without anything writing it. No real package reaches
201+
# the guard that way today, because each also names its templates on a
202+
# recipe line; this fixture is what keeps that exclusion from rotting.
203+
: > "$tmp/pkg/templates/name-echo.yaml"
204+
# The comment names the file the recipe does not write, which is what a
205+
# rationale comment above a real recipe looks like. It must not count as the
206+
# recipe naming it. The exports are the boilerplate every package opens with.
207+
printf 'export NAME=demo-name-echo\nexport NAMESPACE=cozy-$(NAME)\n# local-extra.yaml is written by hand\nupdate:\n\trm -rf templates\n\tmkdir templates\n\twget -O templates/fetched.yaml https://example.invalid/f.yaml\n' \
208+
> "$tmp/pkg/Makefile"
209+
# An index is what the helper reads, so the fixture needs one. Staged rather
210+
# than committed: nothing here needs a commit object, and a commit would want
211+
# an identity and a signature from whoever runs the suite.
212+
git -C "$tmp" init --quiet
213+
git -C "$tmp" add --all
214+
215+
# Untracked litter, written after the staging so it stays untracked. Neither
216+
# file is at stake and neither may be reported.
217+
: > "$tmp/pkg/templates/scratch.yaml"
218+
: > "$tmp/pkg/templates/.DS_Store"
219+
220+
if [ -z "$(wipe_command "$tmp/pkg/Makefile")" ]; then
221+
echo "the matcher did not see 'rm -rf templates' in the fixture recipe" >&2
222+
rm -rf "$tmp"
223+
exit 1
224+
fi
225+
226+
# Exactly the tracked files the recipe does not name. Reporting the fetched
227+
# one too would fail every wiping package in the tree, reporting none of them
228+
# would pass all of them, and reporting the litter would fail a package over
229+
# a file nobody is going to miss.
230+
expected=$(printf '%s\n%s\n%s' "$tmp/pkg/templates/.gitkeep" "$tmp/pkg/templates/local-extra.yaml" "$tmp/pkg/templates/name-echo.yaml")
231+
found=$(unnamed_templates "$tmp/pkg/Makefile")
232+
if [ "$found" != "$expected" ]; then
233+
echo "expected only the tracked unrefetched files to be reported:" >&2
234+
printf '%s\n' "$expected" >&2
235+
echo "got:" >&2
236+
printf '%s\n' "$found" >&2
237+
rm -rf "$tmp"
238+
exit 1
239+
fi
240+
241+
# The trailing-slash spelling removes exactly as much, so it has to be seen
242+
# as well. Spelled with the leading ./ at the same time, which is the other
243+
# optional half of the pattern; the fixture above carries neither.
244+
printf 'update:\n\trm -rf ./templates/\n\twget -O templates/fetched.yaml https://example.invalid/f.yaml\n' \
245+
> "$tmp/pkg/Makefile"
246+
if [ -z "$(wipe_command "$tmp/pkg/Makefile")" ]; then
247+
echo "the matcher missed 'rm -rf ./templates/', which clears the directory" >&2
248+
rm -rf "$tmp"
249+
exit 1
250+
fi
251+
252+
# And the glob spelling, which empties the directory without naming it as
253+
# the argument. It is what cert-manager's recipe writes, so it is the line a
254+
# package copies when it wants to clear its own templates.
255+
printf 'update:\n\trm -rf templates/*\n\twget -O templates/fetched.yaml https://example.invalid/f.yaml\n' \
256+
> "$tmp/pkg/Makefile"
257+
if [ -z "$(wipe_command "$tmp/pkg/Makefile")" ]; then
258+
echo "the matcher missed 'rm -rf templates/*', which empties the directory" >&2
259+
rm -rf "$tmp"
260+
exit 1
261+
fi
262+
263+
# The other side of that boundary: removing one file inside templates/ is
264+
# rabbitmq-operator's shape, it takes only what the recipe refetches, and a
265+
# pattern loose enough to call it a wipe would report that package forever.
266+
printf 'update:\n\trm -rf templates/fetched.yaml\n\twget -O templates/fetched.yaml https://example.invalid/f.yaml\n' \
267+
> "$tmp/pkg/Makefile"
268+
if [ -n "$(wipe_command "$tmp/pkg/Makefile")" ]; then
269+
echo "the matcher read a single-file removal as clearing the directory" >&2
270+
rm -rf "$tmp"
271+
exit 1
272+
fi
273+
274+
# And the shape the CDI packages moved to switches the check off, so the
275+
# scan is measuring the removal rather than the package.
276+
printf 'update:\n\tmkdir -p templates\n\twget -O templates/fetched.yaml https://example.invalid/f.yaml\n' \
277+
> "$tmp/pkg/Makefile"
278+
if [ -n "$(wipe_command "$tmp/pkg/Makefile")" ]; then
279+
echo "the matcher fired on a recipe that only creates the directory" >&2
280+
rm -rf "$tmp"
281+
exit 1
282+
fi
283+
284+
rm -rf "$tmp"
285+
}

‎packages/system/kubevirt-cdi-operator/Makefile‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,8 +4,7 @@ export NAMESPACE=cozy-kubevirt-cdi
44
include ../../../hack/package.mk
55

66
update:
7-
rm -rf templates
8-
mkdir templates
7+
mkdir -p templates
98
export VERSION=$$(basename $$(curl -s -w %{redirect_url} https://github.com/kubevirt/containerized-data-importer/releases/latest)) && \
109
wget https://github.com/kubevirt/containerized-data-importer/releases/download/$$VERSION/cdi-operator.yaml -O templates/cdi-operator.yaml
1110
sed -i 's/namespace: cdi/namespace: cozy-kubevirt-cdi/g' templates/cdi-operator.yaml

‎packages/system/kubevirt-cdi/Makefile‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,8 +9,11 @@ include ../../../hack/package.mk
99
test:
1010
helm unittest .
1111

12+
# templates/ holds first-party manifests as well as the vendored CR: the
13+
# upload-proxy Ingress and TLSRoute, which nothing below refetches. So this
14+
# recipe must never clear the directory -- wget -O truncates the file it
15+
# fetches and leaves the rest alone.
1216
update:
13-
rm -rf templates
14-
mkdir templates
17+
mkdir -p templates
1518
export VERSION=$$(basename $$(curl -s -w %{redirect_url} https://github.com/kubevirt/containerized-data-importer/releases/latest)) && \
1619
wget https://github.com/kubevirt/containerized-data-importer/releases/download/$$VERSION/cdi-cr.yaml -O templates/cdi-cr.yaml

0 commit comments

Comments
 (0)