|
| 1 | +#!/usr/bin/env bats |
| 2 | +# ----------------------------------------------------------------------------- |
| 3 | +# Invariant for the package `update` recipes that clear templates/ wholesale. |
| 4 | +# |
| 5 | +# `update` refetches a package's vendored upstream manifests, and several |
| 6 | +# recipes open by deleting templates/ to guarantee a clean slate. That is safe |
| 7 | +# only while every file under templates/ comes back from the recipe. templates/ |
| 8 | +# is an ordinary chart directory, so a package can also keep first-party |
| 9 | +# manifests there -- kubevirt-cdi keeps the CDI upload-proxy Ingress and |
| 10 | +# TLSRoute -- and the recipe knows nothing about those. The wipe takes them and |
| 11 | +# `make update` exits 0, so the deletion lands in the diff with nothing marking |
| 12 | +# it. Whether anything downstream notices depends on the package carrying a |
| 13 | +# helm-unittest suite that names the file, and where none does the loss is |
| 14 | +# silent all the way to a cluster. That is a property of the package, not of |
| 15 | +# the tree, so it is worth re-deriving rather than reading: at the time of |
| 16 | +# writing kubevirt-cdi has such a suite and a deletion there fails `make test` |
| 17 | +# as well, while three of the packages this scans declare no `test` target at |
| 18 | +# all. Adding one to any of them changes that sentence and not this one. |
| 19 | +# |
| 20 | +# The invariant: a recipe that clears templates/ must NAME every file tracked |
| 21 | +# under it. Both halves are approximations of the shell the recipe would run, |
| 22 | +# and each is argued at its helper below: the wipe is a text match over the |
| 23 | +# recipe's lines rather than a parse, and "names" means the file's basename |
| 24 | +# minus its extension appears somewhere in the Makefile -- a mention, not proof |
| 25 | +# of a write. What that catches is the shape kubevirt-cdi had, a file the |
| 26 | +# Makefile does not refer to at all: every recipe that clears its own |
| 27 | +# templates/ writes each path it puts back there, so a name appearing nowhere |
| 28 | +# in the file is a name nothing refetches. A recipe that unpacked an archive or |
| 29 | +# globbed a stream into templates/ would produce files it never names and break |
| 30 | +# that reading. cert-manager's `mv charts/.../crd-*.yaml` is that shape, and it |
| 31 | +# is out of range for the separate reason below: the directory it fills belongs |
| 32 | +# to another package. |
| 33 | +# |
| 34 | +# The tree is scanned rather than a list of packages pinned, so a package that |
| 35 | +# grows its first first-party manifest under a wiping recipe of its own is |
| 36 | +# caught by the commit that adds it. The packages that clear their own |
| 37 | +# templates/ today refetch all of it, which is what keeps this green; |
| 38 | +# kubevirt-cdi stopped clearing it, which is what this file was written for. |
| 39 | +# kubevirt-cdi-operator still clears its own, and is scanned like the rest. |
| 40 | +# |
| 41 | +# What that leaves uncovered is a recipe clearing SOMEONE ELSE'S templates/, and |
| 42 | +# the tree has one: packages/system/cert-manager/Makefile runs |
| 43 | +# `rm -rf ../cert-manager-crds/templates/*` and moves the CRDs it pulled into |
| 44 | +# that directory, while cert-manager-crds declares no update target at all. A |
| 45 | +# first-party manifest added there is deleted by the next `make update` in |
| 46 | +# cert-manager and nothing below reports it. Covering that means resolving the |
| 47 | +# removal's target directory rather than reading the wiping package's own, which |
| 48 | +# is a second helper; the pattern below is deliberately not widened to accept a |
| 49 | +# path prefix, because that alone would fire on cert-manager and then list the |
| 50 | +# templates of a directory it does not have -- the removal names the sibling's |
| 51 | +# path, so the lookup has to follow it. |
| 52 | +# |
| 53 | +# The scan alone would also be green against a detector that finds nothing, so |
| 54 | +# the control below runs the same two helpers over a fixture that does hold an |
| 55 | +# unrefetched file, and over that fixture with the removal taken out. |
| 56 | +# |
| 57 | +# Constraints from cozytest.sh: it rewrites the closing brace of every function |
| 58 | +# in this file into `return 0`, so a helper cannot report through its exit |
| 59 | +# status -- both report through stdout -- and it injects `set -e`, so statuses |
| 60 | +# are captured explicitly. |
| 61 | +# |
| 62 | +# Run with: hack/cozytest.sh hack/package-update-templates.bats |
| 63 | +# ----------------------------------------------------------------------------- |
| 64 | + |
| 65 | +# Prints the recipe line that clears templates/ wholesale, for a package |
| 66 | +# Makefile whose `update` target has one. Silent otherwise, including for a |
| 67 | +# Makefile with no `update` target at all. |
| 68 | +# |
| 69 | +# The recipe is read as the run of tab-indented lines after `update:`. make also |
| 70 | +# ignores blank lines inside a recipe and this stops at the first one, so such a |
| 71 | +# recipe would be read short; none in the tree is written that way. |
| 72 | +# |
| 73 | +# The match accepts `rm -rf`, `rm -fr` and `rm -r` against `templates` and |
| 74 | +# `./templates`, bare, with a trailing slash, or with a `/*` glob -- the three |
| 75 | +# spellings that empty the directory. It deliberately does not accept |
| 76 | +# `rm -rf templates/one-file.yml`, which takes a single file the recipe goes on |
| 77 | +# to refetch (rabbitmq-operator), nor a narrower glob such as |
| 78 | +# `rm -rf templates/*.yaml`, which takes a subset the same way. |
| 79 | +# |
| 80 | +# It is a text match and not a shell parse, so a quoted `rm -rf "templates"`, a |
| 81 | +# removal reached through a variable, a `find -delete` and a `cd` into another |
| 82 | +# directory first are all invisible to it, as is a path-prefixed target -- the |
| 83 | +# cert-manager case in the header is one, and the prefix is the only thing |
| 84 | +# keeping it out: the `/*` it ends with is accepted. The enumeration is only as |
| 85 | +# complete as the last reader made it, so extend it in the same edit as the |
| 86 | +# pattern. |
| 87 | +wipe_command() { |
| 88 | + awk '/^update:/ { r = 1; next } r && /^\t/ { print; next } r { exit }' "$1" | |
| 89 | + grep -E 'rm[[:space:]]+-(rf|fr|r)[[:space:]]+(\./)?templates(/\*?)?([[:space:]]|;|&|$)' || true |
| 90 | +} |
| 91 | + |
| 92 | +# Prints every file tracked under a package's templates/ that its Makefile |
| 93 | +# never mentions, one path per line. Whether that matters is the caller's call: |
| 94 | +# it matters for a package whose recipe clears the directory, and for no other. |
| 95 | +# |
| 96 | +# The listing is the index, so only a tracked file is at stake -- it is the one |
| 97 | +# the next `make update` deletes out from under a reviewer who has no reason to |
| 98 | +# look. Reading the working tree instead would fail a package over the |
| 99 | +# developer's own litter: an untracked .orig from a hand-applied patch, a |
| 100 | +# half-written template, a .DS_Store. A guard that goes red on scratch files is |
| 101 | +# a guard that gets switched off. |
| 102 | +# |
| 103 | +# The extension is stripped before matching, because a recipe may assemble the |
| 104 | +# filename from a variable holding the bare stem -- etcd-operator-crds writes |
| 105 | +# templates/$$c.yaml from a CRDS list of names without one. A basename whose |
| 106 | +# only dot is its first character keeps the whole name instead: stripping there |
| 107 | +# leaves the empty string, which grep matches against every Makefile, and the |
| 108 | +# file would pass without being looked at. The search covers the whole Makefile |
| 109 | +# rather than the recipe alone for the same reason as the stem: the variable is |
| 110 | +# assigned above the target. Both make the match generous, so a stem that |
| 111 | +# appears for an unrelated reason satisfies it; this reports the files nothing in |
| 112 | +# the Makefile refers to, not every file the recipe fails to write. |
| 113 | +# |
| 114 | +# Comment lines are dropped before the search, so a filename written into one is |
| 115 | +# not a mention: a rationale comment above a recipe, naming the very file it is |
| 116 | +# about, would otherwise exempt that package from its own guard. Comments |
| 117 | +# naming a template are not rare in the tree, so this is the difference between |
| 118 | +# a package being read and a package being taken at its word. |
| 119 | +# |
| 120 | +# The NAME and NAMESPACE assignments go the same way. They are the package's |
| 121 | +# identity rather than a reference to any file, and a package called foo-bar |
| 122 | +# whose recipe writes templates/bar.yaml has that file's stem inside its own |
| 123 | +# name: left in, they let a package vouch for its own templates without any |
| 124 | +# recipe touching them. The match is on the variable name rather than on one |
| 125 | +# way of writing the assignment, because the hole does not care how it was |
| 126 | +# spelled: `export` is optional, whitespace either side of the operator is |
| 127 | +# optional, and make's assignment forms (`=`, `:=`, `::=`, `?=`, `+=`) are all |
| 128 | +# accepted. The tree uses at least three of these -- `export NAME=`, the bare |
| 129 | +# `NAME=`, and `NAME := ` in packages/core/testing -- and pinning the list of |
| 130 | +# spellings is what this deliberately avoids. Two names are named rather than a |
| 131 | +# `[A-Z_]+=` shape for the opposite reason: a variable holding content is what |
| 132 | +# the whole-file search is there to find, and etcd-operator-crds names its |
| 133 | +# templates through nothing but its CRDS list, so a shape that broad would be |
| 134 | +# one unspaced assignment away from blinding the scan to five CRDs. That list |
| 135 | +# is written spaced, which is why the broader pattern would not reach it today |
| 136 | +# and why no case below can pin this choice; two names cannot reach it at any |
| 137 | +# spacing. Precautionary like the comment strip above: no package the scan |
| 138 | +# reaches is exempted this way today, and the line is what keeps the first one |
| 139 | +# that tries from being. |
| 140 | +unnamed_templates() { |
| 141 | + _mk=$1 |
| 142 | + _dir=$(dirname "$_mk") |
| 143 | + _code=$(grep -vE '^([[:space:]]*@?#|(export[[:space:]]+)?(NAME|NAMESPACE)[[:space:]]*[:?+]*=)' "$_mk") || true |
| 144 | + git -C "$_dir" ls-files -- templates | while IFS= read -r _f; do |
| 145 | + _base=${_f##*/} |
| 146 | + _stem=${_base%.*} |
| 147 | + [ -n "$_stem" ] || _stem=$_base |
| 148 | + printf '%s\n' "$_code" | grep -Fq -- "$_stem" || printf '%s\n' "$_dir/$_f" |
| 149 | + done |
| 150 | +} |
| 151 | + |
| 152 | +@test "no package update recipe clears templates/ over a file it never refetches" { |
| 153 | + examined=0 |
| 154 | + offenders="" |
| 155 | + |
| 156 | + for mk in packages/*/*/Makefile; do |
| 157 | + [ -f "$mk" ] || continue |
| 158 | + examined=$((examined + 1)) |
| 159 | + [ -n "$(wipe_command "$mk")" ] || continue |
| 160 | + found=$(unnamed_templates "$mk") |
| 161 | + if [ -n "$found" ]; then |
| 162 | + offenders="$offenders |
| 163 | +$mk clears templates/ but names none of: |
| 164 | +$found" |
| 165 | + fi |
| 166 | + done |
| 167 | + |
| 168 | + # A glob that matched nothing -- packages/ moved, or the suite run from |
| 169 | + # somewhere other than the repository root -- would otherwise report success |
| 170 | + # having looked at no Makefile at all. |
| 171 | + if [ "$examined" -eq 0 ]; then |
| 172 | + echo "no packages/*/*/Makefile found: run this from the repository root" >&2 |
| 173 | + exit 1 |
| 174 | + fi |
| 175 | + |
| 176 | + # The index is where the scan reads a package's templates from, so an answer |
| 177 | + # of nothing here means it could not have reported anything either. |
| 178 | + if [ -z "$(git ls-files -- packages | head -1)" ]; then |
| 179 | + echo "git ls-files reported no tracked file under packages/: this test" >&2 |
| 180 | + echo "cannot see a template directory, so it measured nothing" >&2 |
| 181 | + exit 1 |
| 182 | + fi |
| 183 | + |
| 184 | + if [ -n "$offenders" ]; then |
| 185 | + echo "a package update recipe deletes templates/ over a file it does not refetch:" >&2 |
| 186 | + echo "$offenders" >&2 |
| 187 | + echo "" >&2 |
| 188 | + echo "templates/ holds first-party manifests alongside the vendored ones and" >&2 |
| 189 | + echo "'rm -rf templates' takes both, silently: the next 'make update' drops the" >&2 |
| 190 | + echo "first-party ones and still exits 0. Either keep 'mkdir -p templates'" >&2 |
| 191 | + echo "without the removal -- wget -O and kustomize > overwrite what they fetch" >&2 |
| 192 | + echo "anyway -- or write the local file from the recipe too, where the clean" >&2 |
| 193 | + echo "slate is load-bearing (etcd-operator-crds stages into a tmpdir first)." >&2 |
| 194 | + exit 1 |
| 195 | + fi |
| 196 | +} |
| 197 | + |
| 198 | +@test "the scan reports a template file that a wiping recipe does not name" { |
| 199 | + # Negative control. The test above passes on a tree with no offender and on |
| 200 | + # a detector that cannot see one; this separates the two by handing the same |
| 201 | + # helpers a package that is broken in exactly the way kubevirt-cdi was. |
| 202 | + tmp=$(mktemp -d) |
| 203 | + mkdir -p "$tmp/pkg/templates" |
| 204 | + : > "$tmp/pkg/templates/fetched.yaml" |
| 205 | + : > "$tmp/pkg/templates/local-extra.yaml" |
| 206 | + # A tracked dotfile. Stripping its extension leaves nothing and an empty |
| 207 | + # pattern matches every Makefile, so without the guard in the helper this |
| 208 | + # file is skipped in silence rather than reported. |
| 209 | + : > "$tmp/pkg/templates/.gitkeep" |
| 210 | + # Two files whose stems sit inside the package's own identity, the shape |
| 211 | + # every `foo-bar` package writing templates/bar.yaml has. One is covered by |
| 212 | + # NAME and one by NAMESPACE, and the Makefile below writes the first as |
| 213 | + # `export NAME=` and the second as `NAMESPACE := `, so between them the |
| 214 | + # export prefix, the bare form, spacing around the operator and the `:=` |
| 215 | + # spelling are all exercised. Left in the search, each of these matches |
| 216 | + # with nothing writing it. |
| 217 | + : > "$tmp/pkg/templates/name-echo.yaml" |
| 218 | + : > "$tmp/pkg/templates/bare-echo.yaml" |
| 219 | + # The comment names the file the recipe does not write, which is what a |
| 220 | + # rationale comment above a real recipe looks like. It must not count as the |
| 221 | + # recipe naming it. |
| 222 | + printf 'export NAME=demo-name-echo\nNAMESPACE := cozy-bare-echo\n# local-extra.yaml is written by hand\nupdate:\n\trm -rf templates\n\tmkdir templates\n\twget -O templates/fetched.yaml https://example.invalid/f.yaml\n' \ |
| 223 | + > "$tmp/pkg/Makefile" |
| 224 | + # An index is what the helper reads, so the fixture needs one. Staged rather |
| 225 | + # than committed: nothing here needs a commit object, and a commit would want |
| 226 | + # an identity and a signature from whoever runs the suite. |
| 227 | + git -C "$tmp" init --quiet |
| 228 | + git -C "$tmp" add --all |
| 229 | + |
| 230 | + # Untracked litter, written after the staging so it stays untracked. Neither |
| 231 | + # file is at stake and neither may be reported. |
| 232 | + : > "$tmp/pkg/templates/scratch.yaml" |
| 233 | + : > "$tmp/pkg/templates/.DS_Store" |
| 234 | + |
| 235 | + if [ -z "$(wipe_command "$tmp/pkg/Makefile")" ]; then |
| 236 | + echo "the matcher did not see 'rm -rf templates' in the fixture recipe" >&2 |
| 237 | + rm -rf "$tmp" |
| 238 | + exit 1 |
| 239 | + fi |
| 240 | + |
| 241 | + # Exactly the tracked files the recipe does not name. Reporting the fetched |
| 242 | + # one too would fail every wiping package in the tree, reporting none of them |
| 243 | + # would pass all of them, and reporting the litter would fail a package over |
| 244 | + # a file nobody is going to miss. |
| 245 | + expected=$(printf '%s\n%s\n%s\n%s' "$tmp/pkg/templates/.gitkeep" "$tmp/pkg/templates/bare-echo.yaml" "$tmp/pkg/templates/local-extra.yaml" "$tmp/pkg/templates/name-echo.yaml") |
| 246 | + found=$(unnamed_templates "$tmp/pkg/Makefile") |
| 247 | + if [ "$found" != "$expected" ]; then |
| 248 | + echo "expected only the tracked unrefetched files to be reported:" >&2 |
| 249 | + printf '%s\n' "$expected" >&2 |
| 250 | + echo "got:" >&2 |
| 251 | + printf '%s\n' "$found" >&2 |
| 252 | + rm -rf "$tmp" |
| 253 | + exit 1 |
| 254 | + fi |
| 255 | + |
| 256 | + # The trailing-slash spelling removes exactly as much, so it has to be seen |
| 257 | + # as well. Spelled with the leading ./ at the same time, which is the other |
| 258 | + # optional half of the pattern; the fixture above carries neither. |
| 259 | + printf 'update:\n\trm -rf ./templates/\n\twget -O templates/fetched.yaml https://example.invalid/f.yaml\n' \ |
| 260 | + > "$tmp/pkg/Makefile" |
| 261 | + if [ -z "$(wipe_command "$tmp/pkg/Makefile")" ]; then |
| 262 | + echo "the matcher missed 'rm -rf ./templates/', which clears the directory" >&2 |
| 263 | + rm -rf "$tmp" |
| 264 | + exit 1 |
| 265 | + fi |
| 266 | + |
| 267 | + # And the glob spelling, which empties the directory without naming it as |
| 268 | + # the argument. It is what cert-manager's recipe writes, so it is the line a |
| 269 | + # package copies when it wants to clear its own templates. |
| 270 | + printf 'update:\n\trm -rf templates/*\n\twget -O templates/fetched.yaml https://example.invalid/f.yaml\n' \ |
| 271 | + > "$tmp/pkg/Makefile" |
| 272 | + if [ -z "$(wipe_command "$tmp/pkg/Makefile")" ]; then |
| 273 | + echo "the matcher missed 'rm -rf templates/*', which empties the directory" >&2 |
| 274 | + rm -rf "$tmp" |
| 275 | + exit 1 |
| 276 | + fi |
| 277 | + |
| 278 | + # The two flag spellings the header claims to accept. Nothing in the tree |
| 279 | + # writes either today, so they are precautionary in the same way the |
| 280 | + # identity strip is -- and the file's standard is that a precautionary |
| 281 | + # branch gets pinned, or the claim in the header is prose nothing checks. |
| 282 | + for flags in -fr -r; do |
| 283 | + printf 'update:\n\trm %s templates\n\twget -O templates/fetched.yaml https://example.invalid/f.yaml\n' \ |
| 284 | + "$flags" > "$tmp/pkg/Makefile" |
| 285 | + if [ -z "$(wipe_command "$tmp/pkg/Makefile")" ]; then |
| 286 | + echo "the matcher missed 'rm $flags templates', which the header says it accepts" >&2 |
| 287 | + rm -rf "$tmp" |
| 288 | + exit 1 |
| 289 | + fi |
| 290 | + done |
| 291 | + |
| 292 | + # Scoping. The helper claims to read only the `update` recipe, and to stay |
| 293 | + # silent for a Makefile that has no `update` target at all. The tree cannot |
| 294 | + # test either, because every wipe in it already sits inside `update`, so |
| 295 | + # without these two a matcher that searched the whole file would look |
| 296 | + # identical. The consequence of losing this is a false positive that never |
| 297 | + # clears: a package growing `clean: rm -rf templates` reported forever. |
| 298 | + printf 'clean:\n\trm -rf templates\nupdate:\n\tmkdir -p templates\n\twget -O templates/fetched.yaml https://example.invalid/f.yaml\n' \ |
| 299 | + > "$tmp/pkg/Makefile" |
| 300 | + if [ -n "$(wipe_command "$tmp/pkg/Makefile")" ]; then |
| 301 | + echo "the matcher read a removal from a target other than update" >&2 |
| 302 | + rm -rf "$tmp" |
| 303 | + exit 1 |
| 304 | + fi |
| 305 | + |
| 306 | + printf 'clean:\n\trm -rf templates\n' > "$tmp/pkg/Makefile" |
| 307 | + if [ -n "$(wipe_command "$tmp/pkg/Makefile")" ]; then |
| 308 | + echo "the matcher reported on a Makefile with no update target" >&2 |
| 309 | + rm -rf "$tmp" |
| 310 | + exit 1 |
| 311 | + fi |
| 312 | + |
| 313 | + # The other side of that boundary: removing one file inside templates/ is |
| 314 | + # rabbitmq-operator's shape, it takes only what the recipe refetches, and a |
| 315 | + # pattern loose enough to call it a wipe would report that package forever. |
| 316 | + printf 'update:\n\trm -rf templates/fetched.yaml\n\twget -O templates/fetched.yaml https://example.invalid/f.yaml\n' \ |
| 317 | + > "$tmp/pkg/Makefile" |
| 318 | + if [ -n "$(wipe_command "$tmp/pkg/Makefile")" ]; then |
| 319 | + echo "the matcher read a single-file removal as clearing the directory" >&2 |
| 320 | + rm -rf "$tmp" |
| 321 | + exit 1 |
| 322 | + fi |
| 323 | + |
| 324 | + # The narrower glob is the other rejection the header claims and the last |
| 325 | + # of that class without a case. It takes a subset the recipe refetches, so |
| 326 | + # calling it a wipe would report the package forever -- and it |
| 327 | + # discriminates: widening the tail to accept any glob, the obvious "also |
| 328 | + # match globs" edit, slips past the single-file case above but not this one. |
| 329 | + printf 'update:\n\trm -rf templates/*.yaml\n\twget -O templates/fetched.yaml https://example.invalid/f.yaml\n' \ |
| 330 | + > "$tmp/pkg/Makefile" |
| 331 | + if [ -n "$(wipe_command "$tmp/pkg/Makefile")" ]; then |
| 332 | + echo "the matcher read a narrower glob as clearing the directory" >&2 |
| 333 | + rm -rf "$tmp" |
| 334 | + exit 1 |
| 335 | + fi |
| 336 | + |
| 337 | + # And the shape kubevirt-cdi moved to switches the check off, so the scan |
| 338 | + # is measuring the removal rather than the package. |
| 339 | + printf 'update:\n\tmkdir -p templates\n\twget -O templates/fetched.yaml https://example.invalid/f.yaml\n' \ |
| 340 | + > "$tmp/pkg/Makefile" |
| 341 | + if [ -n "$(wipe_command "$tmp/pkg/Makefile")" ]; then |
| 342 | + echo "the matcher fired on a recipe that only creates the directory" >&2 |
| 343 | + rm -rf "$tmp" |
| 344 | + exit 1 |
| 345 | + fi |
| 346 | + |
| 347 | + rm -rf "$tmp" |
| 348 | +} |
0 commit comments