You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit dcff043
Browse filesBrowse the repository at this point in the historyBrowse files
feat(release): sign the release images and attach their SBOMs
A release has signed its assets since #4861, but not the container
images a cluster installed from it pulls, and the SBOM each image carries
since #4997 sat in the registry only, readable with buildx and signed by
nothing. A user could verify the ISO they booted and not the platform it
installed.
hack/sign-release-images.sh signs every first-party digest the repinned
tree holds with cosign, keylessly, in prepare-release after the stitch.
It skips a digest that already verifies, so a re-run stacks no second
signature, and verifies each one against the identity docs/release.md
names before hack/upload-assets.sh runs, so nothing is uploaded over an
unsigned image. The signature over an index covers the attestation
manifest the index lists, so the SBOM needs none of its own, and cosign
stores it as a referrer of the digest in the repository
hack/promote-retag.sh copies the digest within, so promote-rc.yaml only
verifies, with the rc tree's script, before it stages the stable
release.
hack/collect-image-sboms.sh reads the SBOM attestation of every such
image back into cozystack-images-sbom.tar.gz, one SPDX document per
platform, and fails on an image that lacks one for any platform its
index serves. With cozystack-images.txt, the digests the signatures are
bound to, both join the asset list hack/upload-assets.sh hands to
hack/sign-release-assets.sh, so the existing checksums signature covers
them. docs/release.md gains the verification recipe for a whole release;
the bats files pin the scripts, the recipe, the upload list and the two
workflow steps.
Refs #2937, #3278
Assisted-by: LLM
Signed-off-by: Marian Koreniuk <[email protected]>
Copy file name to clipboardExpand all lines: docs/agents/image-refs.md
+2Lines changed: 2 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -62,6 +62,8 @@ If a package stamps a reference somewhere the two globs do not cover, add that f
62
62
|`hack/stitch-multiarch.sh`| all three shapes | join each amd64 image and its `-arm64` twin into one index, each side an image or, built with `SBOM=1`, an index of the image and its attestation that the result keeps, repin the tree on it, then fail if any old digest survives |
63
63
|`hack/retention-packages.sh`| all three shapes, in the published nightly tree | name the GHCR packages the retention workflow prunes (those linked to another repository are then skipped); a ref it cannot see is a package whose nightlies are never pruned |
64
64
|`hack/verify-multiarch.sh`| all three shapes, first- and third-party | fail a release unless every pinned image is an index with `linux/amd64` and `linux/arm64`, attestation entries not counted, bar the repositories in `hack/multiarch-allowlist`|
65
+
|`hack/sign-release-images.sh`| all three shapes, first-party | sign every pinned digest keylessly after the stitch, skipping one that already verifies, or with `--verify-only` fail a promotion whose rc images are unsigned; `--list` writes the `cozystack-images.txt` release asset |
66
+
|`hack/collect-image-sboms.sh`| all three shapes, first-party | read the SBOM attestation of every pinned index into the `cozystack-images-sbom.tar.gz` release asset, one document per platform; an image missing one for any platform its index serves fails the release |
65
67
66
68
`hack/multiarch-allowlist` is the only bypass for that gate, and it is for images that cannot exist for arm64, not for images nobody has rebuilt yet. An entry is a repository with no tag or digest, so bumping an allowed image needs no edit, and it must carry a reason after `#` on the same line: the script refuses an entry without one. Allowed refs are still printed, under their own "allowed single-arch" heading, and in tree mode an entry that matches no ref that failed the check is reported as stale without changing the exit status, so it gets removed once the image is multi-arch or gone. `--refs-file` reports no stale entries, since a list of pulled images omits everything the run did not pull.
Copy file name to clipboardExpand all lines: docs/release.md
+35-3Lines changed: 35 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -327,10 +327,12 @@ So a commit on a supported `release-X.Y` line ships when a maintainer promotes t
327
327
328
328
## Release signatures and SBOM
329
329
330
-
A release cut from a tree that has [`hack/sign-release-assets.sh`](../hack/sign-release-assets.sh) carries three files beside its other assets, which [`hack/upload-assets.sh`](../hack/upload-assets.sh) has that script write before it uploads anything. Releases cut before it have none of them, and neither does a maintenance line until the script is backported to it (see below).
330
+
A release cut from a tree that has [`hack/sign-release-assets.sh`](../hack/sign-release-assets.sh) carries three files beside its other assets, which [`hack/upload-assets.sh`](../hack/upload-assets.sh) has that script write before it uploads anything, and one cut from a tree that also has [`hack/sign-release-images.sh`](../hack/sign-release-images.sh) and [`hack/collect-image-sboms.sh`](../hack/collect-image-sboms.sh) carries two more, which `tags.yaml::prepare-release` writes before the upload. Releases cut before them have none of these files, and neither does a maintenance line until the scripts are backported to it (see below).
331
331
332
332
-`cozystack-source-sbom.spdx.json` is an SPDX SBOM, made with syft, of the committed source tree (the Go modules, npm lockfiles, GitHub Actions references and other manifests syft reads in it) and of the `cozypkg` binaries in the `cozypkg-*.tar.gz` assets. It does not describe what runs in the cluster: the container images carry their own SBOMs, see [SBOM of the container images](#sbom-of-the-container-images), and the Talos installer image, ISO, disk, kernel and initramfs images have none. Its name says what it covers, so that `cozystack-sbom` stays free for an SBOM of the release as a whole.
333
-
-`cozystack-checksums.txt` holds the SHA-256 of every other asset and of the SBOM.
333
+
-`cozystack-images.txt` lists every first-party container image the release pins, one `<repository>@<digest>` per line: the multi-arch indexes [`hack/stitch-multiarch.sh`](../hack/stitch-multiarch.sh) repinned the tree on, which is what a cluster installed from this release pulls. The packages artifact and the installer chart are not in it.
334
+
-`cozystack-images-sbom.tar.gz` holds the SPDX SBOM of each image in that list, as `<image>.<os>-<arch>.spdx.json`, one document per platform, read from the attestation BuildKit attached to the image at build time (`SBOM=1` in `tags.yaml`, the scanner pinned in [`hack/common-envs.mk`](../hack/common-envs.mk)). These documents come from the syft inside that scanner, the source SBOM from the syft the workflow installs, so the two need not agree on a version. An image without a document for every platform its index serves fails the release instead of being left out.
335
+
-`cozystack-checksums.txt` holds the SHA-256 of every other asset, the SBOMs and the image list among them.
334
336
-`cozystack-checksums.txt.sigstore.json` is a Sigstore bundle with a keyless cosign signature over the checksums. Its certificate names the workflow that published the release: `tags.yaml` for a pre-release, `promote-rc.yaml` for a stable one.
335
337
336
338
One signature covers every asset through the checksums. To check a download, verify the checksums, then the file against them, with cosign v3.1.3 or later (v2.6.5 or later on the v2 line):
@@ -353,7 +355,36 @@ The script runs the same `cosign verify-blob` before it returns, so a signature
353
355
354
356
### SBOM of the container images
355
357
356
-
The container images a release pins by digest each carry an SPDX SBOM of the packages in them, which BuildKit attaches at build time as an attestation (`SBOM=1` in `tags.yaml`, scanner and flags in [`hack/common-envs.mk`](../hack/common-envs.mk)). The attestation is an entry of the image's index, so the digest the release pins names both, and promotion copies it with the image. Read it with `docker buildx imagetools inspect <image>@<digest> --format '{{ json .SBOM }}'`, which prints one document per platform. The attestations are not signed. A release cut before `SBOM=1` was set has none, and neither has the Talos installer image, which the imager builds and `skopeo` pushes, nor the third-party images `packages/` vendors by digest.
358
+
The container images a release pins by digest each carry an SPDX SBOM of the packages in them, which BuildKit attaches at build time as an attestation (`SBOM=1` in `tags.yaml`, scanner and flags in [`hack/common-envs.mk`](../hack/common-envs.mk)). The attestation is an entry of the image's index, so the digest the release pins names both, and promotion copies it with the image. Read it with `docker buildx imagetools inspect <image>@<digest> --format '{{ json .SBOM }}'`, which prints one document per platform. They carry no signature of their own: the signature over the index covers them, see [Image signatures](#image-signatures). A release cut before `SBOM=1` was set has none, and neither has the Talos installer image, which the imager builds and `skopeo` pushes, nor the third-party images `packages/` vendors by digest.
359
+
360
+
### Image signatures
361
+
362
+
The images in `cozystack-images.txt` are signed too. After the stitch, `hack/sign-release-images.sh` signs each digest with cosign, keylessly, as the `tags.yaml` run that prepared the pre-release, and verifies every signature against the same identity as above before `hack/upload-assets.sh` runs, so a release is never published with unsigned images. A signature over an index covers every manifest the index lists by digest, the attestation manifest that holds the SBOM included, which is why the SBOM carries no signature of its own. cosign stores each signature as a Sigstore bundle, an OCI referrer of the digest it signs, in the same repository. GHCR serves no referrers API, so cosign lists the referrer under the fallback tag `sha256-<hex>` of that digest, where `cosign verify` finds it with no extra flags; the tag `sha256-<hex>.sig` is the legacy format, written only with `--new-bundle-format=false`, and a release has none. Every digest gets its own short-lived key, Fulcio certificate and Rekor entry. Promotion copies each digest to its stable tag inside that repository, so a stable release's images carry the signatures made for its rc, and their certificates name `tags.yaml` at the rc tag. `promote-rc.yaml::promote` checks every signature with the rc tree's `hack/sign-release-images.sh --verify-only` before it stages the stable release, and skips the check with a notice for an rc cut from a tree that has no such script. A re-run of `prepare-release` signs only the digests that do not verify yet, so it never stacks a second signature on an image. The packages artifact and the installer chart are not signed: promotion republishes both under a new digest.
363
+
364
+
### How to verify a release
365
+
366
+
With cosign v3.1.3 or later and the assets of the release in the current directory, this checks the signature over the checksums, every downloaded asset against them, the signature of every image the release pins, and unpacks the image SBOMs:
mkdir -p image-sboms && tar -xzf cozystack-images-sbom.tar.gz -C image-sboms
383
+
```
384
+
385
+
The loop stops at the first image that fails, names it and fails the chain, so the SBOMs are unpacked only after every image verified; the list is read from fd 3 so that cosign cannot read it from stdin. `cosign verify` prints the certificate of each signature; its `Subject` names the workflow and the tag, and the image is pulled only as far as its manifests. The same SBOM can be read from the registry without the archive, with `docker buildx imagetools inspect <image> --format '{{ json .SBOM }}'`, which prints the documents keyed by platform. Checking an image by the tag a release page shows rather than by the digest in `cozystack-images.txt` proves less: a tag can be moved, a digest cannot, and the signature is bound to the digest.
386
+
387
+
What fails, and what it means: `cosign verify-blob` rejecting the bundle, or `cosign verify` finding no matching signature, means the file or the image is not what the release workflow published, or cosign is older than the version above; `sha256sum` reporting a mismatch means the download is corrupt or replaced; an image in `cozystack-images.txt` that `cosign verify` cannot find at all means the registry no longer serves that digest. An image missing from the archive cannot happen for a release that has one, since `hack/collect-image-sboms.sh` fails the release instead.
357
388
358
389
## Nightly builds
359
390
@@ -649,6 +680,7 @@ For RCs and final releases, run this before merging the release PR. Each item is
649
680
| Changelog Copilot job returns 402 | Token quota — `COPILOT_GITHUB_TOKEN` premium-request quota empty | Both changelog entry points hit it: the optional rc-time `changelog-rc` dispatch and the promote-time `changelog` job (both call the reusable `changelog-generate.yaml`), plus the `tags.yaml` backstop | No — `continue-on-error: true` on the AI step, generate changelog by hand following [`agents/changelog.md`](./agents/changelog.md)|
650
681
|`Prepare Release` skipped and the run red, `Build arm64 images` failed | The rc's arm64 build failed | Read the `build-arm64` log; `build-arm64-nightly.yaml` usually failed the same way first. Fix on the line, cut the next rc | Yes — an rc is not published amd64 only |
651
682
|`Stitch multi-arch indexes and republish` fails listing refs |`verify-multiarch.sh` found refs that are not amd64+arm64 indexes, or that it could not check | Each listed ref names its reason. Single-platform manifest or index without one architecture: make the image multi-arch, or allowlist its repository with a reason if it cannot be. Manifest unreadable: correct the ref or digest; the allowlist does not excuse it. Rate-limited or unavailable after 3 tries: re-run once the throttle clears | Yes |
683
+
|`Sign the release images and collect their SBOMs` fails |`hack/sign-release-images.sh` could not sign a pinned digest or its signature does not verify against the release identity, or `hack/collect-image-sboms.sh` found a pinned image with no SBOM attestation | A signing failure names the ref: Fulcio or Rekor unreachable, or the OIDC token missing, re-run once they answer; a signature already there is kept, so a re-run signs only what is missing. An image without an SBOM was built without `SBOM=1` on the docker-container builder, or its index lost the attestation: fix the build, re-cut the rc | Yes |
652
684
| E2E `kubernetes-test` fails with `CSINode does not contain driver csi.kubevirt.io`| Test ordering bug | Tenant's `kubernetes-${test_name}-csi` HelmRelease still installing when NFS PVC is created. Fix: wait for `csinode/<node>` to advertise the driver before creating the PVC | Yes — flaky and masks real CSI bugs |
653
685
|`tenant-root` HR in perpetual upgrade-rollback loop after OIDC patch | Real bug at the intersection of #2602's readiness change and the cozystack reconcile cadence | Look for `rate: Wait(n=1) would exceed context deadline` in operator logs; multiple tenant child HRs stuck `InProgress`| Yes |
654
686
|`cozy-dashboard` ImagePullBackOff from `cozystack-ui:latest`| Registry flake | Known noisy on some OCI mirrors; not currently in the prepull set | No — retry |
0 commit comments