ci: guard against pinned base-image digests disappearing upstream #3305

Problem

Dockerfile base images are pinned to immutable digests for reproducibility and for OpenSSF Scorecard Pinned-Dependencies (.github/renovate.json sets pinDigests: true for the dockerfile and github-actions managers). Some registries retire the superseded manifest as soon as a tag moves, so a pinned digest can stop resolving with no warning and no diff, and the failure lands inside whatever unrelated PR happens to rebuild that package next.

This has already happened twice, on the same image, six days apart:

• 2026-07-15: quay.io/centos/centos:stream9@sha256:0c9db821… returned MANIFEST_UNKNOWN and blocked unrelated PR builds at packages/apps/kubernetes (failing run). Fixed by hand in fix(kubernetes): re-pin deleted centos:stream9 digest for kubevirt-csi-driver #3303.
• 2026-07-21: the digest fix(kubernetes): re-pin deleted centos:stream9 digest for kubevirt-csi-driver #3303 had just pinned was deleted too. Fixed by hand again in fix(kubernetes): re-pin deleted centos:stream9 digest for kubevirt-csi-driver #3386, then fix(kubernetes): move kubevirt-csi-driver base off quay.io/centos to almalinux #3400 moved the image off quay.io to docker.io/library/almalinux.

Neither fix added a check, so the exposure is unchanged for every other pin in the tree.

Why a guard rather than a mirror

The original version of this issue proposed mirroring volatile bases into a project-controlled registry and pinning the mirrored digest. That is more machinery than the measured failure rate supports: two incidents in six days on one image, zero in the 23 days since #3400 removed the last rolling-tag pin on a retiring registry. A mirror means a registry to keep, a sync job to maintain, and a rewrite of every FROM. What #3400 did instead, move to a registry that retains, cost one line and solved the real case.

So the durable part of this issue is detection, not indirection. Prefer a retaining registry when picking a base, and have CI tell us when a pin is gone instead of letting a PR author discover it.

What is exposed today

38 Dockerfile files plus 1 Containerfile under packages/*/*/images/ pin 17 unique base refs. Verified with crane manifest on 2026-08-13: all 17 resolve. Nothing is broken right now.

By registry:

The two quay.io pins:

• quay.io/poseidon/matchbox:v0.11.0@sha256:06bcdae8…, at packages/core/talos/images/matchbox/Dockerfile:1
• quay.io/cilium/cilium:v1.19.5@sha256:20fbbc14…, at packages/system/cilium/images/cilium/Dockerfile:1

Both sit on immutable-style release tags rather than a rolling tag like stream9, so upstream would have to re-push a released version for them to break. That is lower likelihood, not a different failure mode.

Also in scope, from packages/*/*/images/*.tag (13 of the 18 files hold first-party refs the build stamps, 5 hold third-party ones, carrying 3 unique refs):

• docker.io/alpine/k8s:1.33.4@sha256:b0523f0a… in kubernetes-nodes/images/kubectl.tag, kubernetes/images/kubectl.tag, monitoring/images/kubectl.tag
• docker.io/library/busybox:1.37.0@sha256:1487d0af… in kubernetes/images/busybox.tag
• ghcr.io/seaweedfs/seaweedfs-cosi-driver:v0.3.1 in extra/seaweedfs/images/seaweedfs-cosi-driver.tag, not digest-pinned, so Renovate's pinDigests rule cannot reach it (it covers the dockerfile and github-actions managers only) and a digest liveness check has nothing to test

Unpinned bases, which cannot 404 but can change content with no diff, listed for completeness rather than as work:

• gcr.io/distroless/static:nonroot-${TARGETARCH}, the ARG BASEIMAGE default at packages/apps/kubernetes/images/cluster-autoscaler/Dockerfile:3. packages/apps/kubernetes/Makefile passes no --build-arg BASEIMAGE, so this default is what builds.
• quay.io/metallb/controller:${VERSION} and :speaker, at packages/system/metallb/images/metallb/Dockerfile:5,7, with VERSION from the vendored chart's appVersion.
• kubeovn/kube-ovn-base:$BASE_TAG and golang:1.26-bookworm, at packages/system/kubeovn/images/kubeovn/Dockerfile:5,19,39.

Proposed guard

What it checks

A script, hack/check-base-digests.sh, POSIX sh, that:

1. Enumerates every digest-pinned third-party base ref reachable from the build: FROM …@sha256:… and ARG …=…@sha256:… lines in packages/*/*/images/**/Dockerfile and **/Containerfile, plus refs in packages/*/*/images/*.tag whose repository is not ghcr.io/cozystack/cozystack/*. Deduplicates by ref: today that is 19 digest-pinned refs to resolve (17 Dockerfile bases plus alpine/k8s and busybox from the .tag files), spread across many more occurrence sites, plus one tag-only ref to flag as unpinned.
2. Resolves each ref anonymously with a manifest HEAD (crane manifest or skopeo inspect --raw, whichever the lane already has, and nightly.yaml already installs skopeo).
3. Prints one line per ref with its verdict and its file:line, and exits non-zero when any ref is definitively gone.

Fail-open discipline, stated because getting it wrong here is worse than not having the guard. A MANIFEST_UNKNOWN or a 404 is a failure. A timeout, a 5xx, a rate limit or an auth error is not a failure, but it must be reported as ::warning:: and the summary must say how many refs went unchecked. The guard must never print a clean result when a lookup did not complete, because a check that fails open silently reads exactly like a check that passed.

Where it runs

Two places, both existing lanes. No new workflow file: this repo prefers extending a lane and asks for a measured failure frequency first, and two incidents in six days does not justify standing up a workflow of its own.

1. Scheduled detection, a base-digests job in .github/workflows/nightly.yaml. That workflow already runs cron: '0 2 * * *' and already installs skopeo in its mirror job. The new job needs no secrets (every base is anonymously pullable), no build, and runs on ubuntu-latest in seconds. It gates nothing, it fails so the nightly goes red and names the ref and its file:line in a ::error:: annotation. Detection window drops from "whenever something next rebuilds that package" to under 24 hours.

2. PR-time signal, invoke the same script from the existing checks job in .github/workflows/pull-requests.yaml, conditioned on the PR having touched a Dockerfile, Containerfile or *.tag. This catches a bad digest at review time, including one a Renovate digest PR is proposing. Because checks is a required status, this invocation fails only on a definitive MANIFEST_UNKNOWN/404 and treats every transport error as a pass with a warning.

An offline unit test, hack/check-base-digests.bats under make unit-tests, covers the enumerator against fixtures: that it finds FROM and ARG forms, that it skips first-party ghcr.io/cozystack/cozystack/* refs, that it deduplicates, and that a tag-only ref is reported as unpinned rather than silently dropped. Follow the harness constraints documented in hack/image-pin-consistency.bats (hack/cozytest.sh is not real bats: no run, no $status, no skip, repo-root-relative paths, and env -u REGISTRY where the ambient CI REGISTRY would skew a tree scan). The network resolve stays out of unit-tests so no PR-gating unit test acquires a network dependency.

What it does not do

• No auto-repin. Adopting a base nobody reviewed is how a supply chain gets a surprise.
• No mirror registry. Revisit if the guard fires more than once a quarter.
• No new workflow file.

Also worth deciding

packages/extra/seaweedfs/images/seaweedfs-cosi-driver.tag holds ghcr.io/seaweedfs/seaweedfs-cosi-driver:v0.3.1 with no digest, so it is outside Renovate's pinDigests rule and outside anything a digest check can verify. Either pin it, or record it here as a deliberate exception. The original version of this issue asked for exactly that ("document any deliberate exceptions where a rolling tag must remain unpinned") and this is the only case in the tree.

Definition of done

• hack/check-base-digests.sh exists, is covered offline by hack/check-base-digests.bats under make unit-tests, and reports unchecked refs rather than swallowing them.
• It runs daily as a job in .github/workflows/nightly.yaml and on Dockerfile-touching PRs from the existing checks job.
• The seaweedfs cosi-driver ref is either digest-pinned or recorded here as an exception.
• No new workflow file, no mirror registry.