Skip to content

Commit c7a7a2a

Browse files
committed
feat(release): sign the release images and attach their SBOMs
A release has signed its assets since #4861, but not the container images a cluster installed from it pulls, and the SBOM each image carries since #4997 sat in the registry only, readable with buildx and signed by nothing. A user could verify the ISO they booted and not the platform it installed. hack/sign-release-images.sh signs every first-party digest the repinned tree holds with cosign, keylessly, in prepare-release after the stitch. It skips a digest that already verifies, so a re-run stacks no second signature, and verifies each one against the identity docs/release.md names before hack/upload-assets.sh runs, so nothing is uploaded over an unsigned image. The signature over an index covers the attestation manifest the index lists, so the SBOM needs none of its own, and cosign keeps it in the repository hack/promote-retag.sh copies the digest within, so promote-rc.yaml only verifies, with the rc tree's script, before it stages the stable release. hack/collect-image-sboms.sh reads the SBOM attestation of every such image back into cozystack-images-sbom.tar.gz, one SPDX document per platform, and fails on an image that has none. With cozystack-images.txt, the digests the signatures are bound to, both join the asset list hack/upload-assets.sh hands to hack/sign-release-assets.sh, so the existing checksums signature covers them. docs/release.md gains the verification recipe for a whole release; the bats files pin the scripts, the upload list and the two workflow steps. Refs #2937, #3278 Assisted-by: LLM Signed-off-by: Marian Koreniuk <[email protected]>
1 parent d78897f commit c7a7a2a

11 files changed

Lines changed: 649 additions & 5 deletions

‎.github/workflows/promote-rc.yaml‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -612,6 +612,25 @@ jobs:
612612
with:
613613
cosign-release: v3.1.3
614614

615+
# The stable release ships the rc's image digests, so their signatures
616+
# are the rc's too: tags.yaml signed each index in the repository that
617+
# finalize's retag copies the digest within, where cosign keeps the
618+
# signature. Checked here, before any stable asset is uploaded, with the
619+
# rc tree's script, so an rc cut from a tree without it is promoted as it
620+
# always was, and one whose signatures are missing never reaches the
621+
# promote PR.
622+
- name: Verify the rc image signatures
623+
env:
624+
DOCKER_CONFIG: ${{ runner.temp }}/.docker
625+
REGISTRY: ghcr.io/cozystack/cozystack
626+
RC_TAG: ${{ needs.parse.outputs.rc_tag }}
627+
run: |
628+
if [ -f hack/sign-release-images.sh ]; then
629+
hack/sign-release-images.sh --verify-only packages "$REGISTRY"
630+
else
631+
echo "::notice::${RC_TAG} was cut from a tree without hack/sign-release-images.sh, so its images carry no signature to check"
632+
fi
633+
615634
- name: Create draft stable release and upload assets
616635
env:
617636
GH_TOKEN: ${{ steps.app-token.outputs.token }}

‎.github/workflows/tags.yaml‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -561,6 +561,24 @@ jobs:
561561
with:
562562
cosign-release: v3.1.3
563563

564+
# Every index the stitch repinned the tree on is signed by digest with
565+
# this job's OIDC identity, and the SBOM attestation each carries is read
566+
# back into one archive, before anything is uploaded: a signature that
567+
# does not verify, or an image that lost its SBOM, fails the job with no
568+
# asset uploaded. The checksums hack/upload-assets.sh signs next cover
569+
# the archive and the image list. The signature over an index covers its
570+
# attestation manifest, and promotion copies the digest within the same
571+
# repository, so the stable release needs no signing step of its own.
572+
- name: Sign the release images and collect their SBOMs
573+
if: needs.build-amd64.outputs.release_exists == 'false'
574+
run: |
575+
set -eu
576+
hack/sign-release-images.sh --list _out/assets/cozystack-images.txt packages "$REGISTRY"
577+
hack/collect-image-sboms.sh packages "$REGISTRY" _out/assets/cozystack-images-sbom.tar.gz
578+
env:
579+
DOCKER_CONFIG: ${{ runner.temp }}/.docker
580+
REGISTRY: ghcr.io/cozystack/cozystack
581+
564582
# The assets were built by build-amd64; the operator manifests are
565583
# rendered here, from the tree pinned on the index digests.
566584
- name: Upload assets

‎docs/agents/image-refs.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,8 @@ If a package stamps a reference somewhere the two globs do not cover, add that f
6161
| `hack/stitch-multiarch.sh` | all three shapes | join each amd64 image and its `-arm64` twin into one index, repin the tree on it, then fail if any old digest survives |
6262
| `hack/retention-packages.sh` | all three shapes, in the published nightly tree | name the GHCR packages the retention workflow prunes (those linked to another repository are then skipped); a ref it cannot see is a package whose nightlies are never pruned |
6363
| `hack/verify-multiarch.sh` | all three shapes, first- and third-party | fail a release unless every pinned image is an index with `linux/amd64` and `linux/arm64`, bar the repositories in `hack/multiarch-allowlist` |
64+
| `hack/sign-release-images.sh` | all three shapes, first-party | sign every pinned digest keylessly after the stitch, skipping one that already verifies, or with `--verify-only` fail a promotion whose rc images are unsigned; `--list` writes the `cozystack-images.txt` release asset |
65+
| `hack/collect-image-sboms.sh` | all three shapes, first-party | read the SBOM attestation of every pinned index into the `cozystack-images-sbom.tar.gz` release asset, one document per platform; an image without one fails the release |
6466

6567
`hack/multiarch-allowlist` is the only bypass for that gate, and it is for images that cannot exist for arm64, not for images nobody has rebuilt yet. An entry is a repository with no tag or digest, so bumping an allowed image needs no edit, and it must carry a reason after `#` on the same line: the script refuses an entry without one. Allowed refs are still printed, under their own "allowed single-arch" heading, and in tree mode an entry that matches no ref that failed the check is reported as stale without changing the exit status, so it gets removed once the image is multi-arch or gone. `--refs-file` reports no stale entries, since a list of pulled images omits everything the run did not pull.
6668

‎docs/release.md‎

Lines changed: 31 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -327,10 +327,12 @@ So a commit on a supported `release-X.Y` line ships when a maintainer promotes t
327327

328328
## Release signatures and SBOM
329329

330-
A release cut from a tree that has [`hack/sign-release-assets.sh`](../hack/sign-release-assets.sh) carries three files beside its other assets, which [`hack/upload-assets.sh`](../hack/upload-assets.sh) has that script write before it uploads anything. Releases cut before it have none of them, and neither does a maintenance line until the script is backported to it (see below).
330+
A release cut from a tree that has [`hack/sign-release-assets.sh`](../hack/sign-release-assets.sh) carries three files beside its other assets, which [`hack/upload-assets.sh`](../hack/upload-assets.sh) has that script write before it uploads anything, and one cut from a tree that also has [`hack/sign-release-images.sh`](../hack/sign-release-images.sh) and [`hack/collect-image-sboms.sh`](../hack/collect-image-sboms.sh) carries two more, which `tags.yaml::prepare-release` writes before the upload. Releases cut before them have none of these files, and neither does a maintenance line until the scripts are backported to it (see below).
331331

332332
- `cozystack-source-sbom.spdx.json` is an SPDX SBOM, made with syft, of the committed source tree (the Go modules, npm lockfiles, GitHub Actions references and other manifests syft reads in it) and of the `cozypkg` binaries in the `cozypkg-*.tar.gz` assets. It does not describe what runs in the cluster: neither the container images that `packages/` pins by digest, whose SBOMs are [#2937](https://github.com/cozystack/cozystack/issues/2937) (item A2 in [#3278](https://github.com/cozystack/cozystack/issues/3278)), nor the Talos ISO, disk, kernel and initramfs images. Its name says what it covers, so that `cozystack-sbom` stays free for an SBOM of the release as a whole.
333-
- `cozystack-checksums.txt` holds the SHA-256 of every other asset and of the SBOM.
333+
- `cozystack-images.txt` lists every first-party container image the release pins, one `<repository>@<digest>` per line: the multi-arch indexes [`hack/stitch-multiarch.sh`](../hack/stitch-multiarch.sh) repinned the tree on, which is what a cluster installed from this release pulls. The packages artifact and the installer chart are not in it.
334+
- `cozystack-images-sbom.tar.gz` holds the SPDX SBOM of each image in that list, as `<image>.<os>-<arch>.spdx.json`, one document per platform, read from the attestation BuildKit attached to the image at build time (`SBOM=1` in `tags.yaml`, the scanner pinned in [`hack/common-envs.mk`](../hack/common-envs.mk)). These documents come from the syft inside that scanner, the source SBOM from the syft the workflow installs, so the two need not agree on a version. An image without an attestation fails the release instead of being left out.
335+
- `cozystack-checksums.txt` holds the SHA-256 of every other asset, the SBOMs and the image list among them.
334336
- `cozystack-checksums.txt.sigstore.json` is a Sigstore bundle with a keyless cosign signature over the checksums. Its certificate names the workflow that published the release: `tags.yaml` for a pre-release, `promote-rc.yaml` for a stable one.
335337

336338
One signature covers every asset through the checksums. To check a download, verify the checksums, then the file against them, with cosign v3.1.3 or later (v2.6.5 or later on the v2 line):
@@ -351,6 +353,32 @@ The script runs the same `cosign verify-blob` before it returns, so a signature
351353

352354
`promote-rc.yaml` runs the `hack/upload-assets.sh` of the rc tree, not the dispatch ref's. A stable release is therefore signed only when its rc was tagged from a tree that has `hack/sign-release-assets.sh`, and a maintenance line starts carrying signatures once that script is backported to its `release-X.Y` branch.
353355

356+
### Image signatures
357+
358+
The images in `cozystack-images.txt` are signed too. After the stitch, `hack/sign-release-images.sh` signs each digest with cosign, keylessly, as the `tags.yaml` run that prepared the pre-release, and verifies every signature against the same identity as above before `hack/upload-assets.sh` runs, so a release is never published with unsigned images. A signature over an index covers every manifest the index lists by digest, the attestation manifest that holds the SBOM included, which is why the SBOM carries no signature of its own. cosign keeps a signature beside the image, as the tag `sha256-<digest>.sig` of the same repository: promotion copies each digest to its stable tag inside that repository, so a stable release's images carry the signatures made for its rc, and their certificates name `tags.yaml` at the rc tag. `promote-rc.yaml::promote` checks every signature with the rc tree's `hack/sign-release-images.sh --verify-only` before it stages the stable release, and skips the check with a notice for an rc cut from a tree that has no such script. A re-run of `prepare-release` signs only the digests that do not verify yet, so it never stacks a second signature on an image. The packages artifact and the installer chart are not signed: promotion republishes both under a new digest.
359+
360+
### How to verify a release
361+
362+
With cosign v3.1.3 or later and the assets of the release in the current directory, this checks the signature over the checksums, every downloaded asset against them, the signature of every image the release pins, and unpacks the image SBOMs:
363+
364+
```sh
365+
cosign verify-blob cozystack-checksums.txt \
366+
--bundle cozystack-checksums.txt.sigstore.json \
367+
--certificate-identity-regexp '^https://github\.com/cozystack/cozystack/\.github/workflows/(tags\.yaml@refs/tags/v|promote-rc\.yaml@refs/heads/main$)' \
368+
--certificate-oidc-issuer https://token.actions.githubusercontent.com &&
369+
sha256sum --check --ignore-missing cozystack-checksums.txt &&
370+
while read -r image; do
371+
cosign verify "$image" \
372+
--certificate-identity-regexp '^https://github\.com/cozystack/cozystack/\.github/workflows/(tags\.yaml@refs/tags/v|promote-rc\.yaml@refs/heads/main$)' \
373+
--certificate-oidc-issuer https://token.actions.githubusercontent.com >/dev/null
374+
done <cozystack-images.txt &&
375+
mkdir -p image-sboms && tar -xzf cozystack-images-sbom.tar.gz -C image-sboms
376+
```
377+
378+
`cosign verify` prints the certificate of each signature; its `Subject` names the workflow and the tag, and the image is pulled only as far as its manifests. The same SBOM can be read from the registry without the archive, with `docker buildx imagetools inspect <image> --format '{{ json .SBOM }}'`, which prints the documents keyed by platform. Checking an image by the tag a release page shows rather than by the digest in `cozystack-images.txt` proves less: a tag can be moved, a digest cannot, and the signature is bound to the digest.
379+
380+
What fails, and what it means: `cosign verify-blob` rejecting the bundle, or `cosign verify` finding no matching signature, means the file or the image is not what the release workflow published, or cosign is older than the version above; `sha256sum` reporting a mismatch means the download is corrupt or replaced; an image in `cozystack-images.txt` that `cosign verify` cannot find at all means the registry no longer serves that digest. An image missing from the archive cannot happen for a release that has one, since `hack/collect-image-sboms.sh` fails the release instead.
381+
354382
## Nightly builds
355383

356384
A nightly is an **installable copy of `main` on GHCR — not a rebuild and not a release**. [`build-main.yaml`](../.github/workflows/build-main.yaml) already builds every push to `main` into the CI registry (OCIR); the nightly promotes that build to the public release registry (GHCR) and proves it installs.
@@ -645,6 +673,7 @@ For RCs and final releases, run this before merging the release PR. Each item is
645673
| Changelog Copilot job returns 402 | Token quota — `COPILOT_GITHUB_TOKEN` premium-request quota empty | Both changelog entry points hit it: the optional rc-time `changelog-rc` dispatch and the promote-time `changelog` job (both call the reusable `changelog-generate.yaml`), plus the `tags.yaml` backstop | No — `continue-on-error: true` on the AI step, generate changelog by hand following [`agents/changelog.md`](./agents/changelog.md) |
646674
| `Prepare Release` skipped and the run red, `Build arm64 images` failed | The rc's arm64 build failed | Read the `build-arm64` log; `build-arm64-nightly.yaml` usually failed the same way first. Fix on the line, cut the next rc | Yes — an rc is not published amd64 only |
647675
| `Stitch multi-arch indexes and republish` fails listing refs | `verify-multiarch.sh` found refs that are not amd64+arm64 indexes, or that it could not check | Each listed ref names its reason. Single-platform manifest or index without one architecture: make the image multi-arch, or allowlist its repository with a reason if it cannot be. Manifest unreadable: correct the ref or digest; the allowlist does not excuse it. Rate-limited or unavailable after 3 tries: re-run once the throttle clears | Yes |
676+
| `Sign the release images and collect their SBOMs` fails | `hack/sign-release-images.sh` could not sign a pinned digest or its signature does not verify against the release identity, or `hack/collect-image-sboms.sh` found a pinned image with no SBOM attestation | A signing failure names the ref: Fulcio or Rekor unreachable, or the OIDC token missing, re-run once they answer; a signature already there is kept, so a re-run signs only what is missing. An image without an SBOM was built without `SBOM=1` on the docker-container builder, or its index lost the attestation: fix the build, re-cut the rc | Yes |
648677
| E2E `kubernetes-test` fails with `CSINode does not contain driver csi.kubevirt.io` | Test ordering bug | Tenant's `kubernetes-${test_name}-csi` HelmRelease still installing when NFS PVC is created. Fix: wait for `csinode/<node>` to advertise the driver before creating the PVC | Yes — flaky and masks real CSI bugs |
649678
| `tenant-root` HR in perpetual upgrade-rollback loop after OIDC patch | Real bug at the intersection of #2602's readiness change and the cozystack reconcile cadence | Look for `rate: Wait(n=1) would exceed context deadline` in operator logs; multiple tenant child HRs stuck `InProgress` | Yes |
650679
| `cozy-dashboard` ImagePullBackOff from `cozystack-ui:latest` | Registry flake | Known noisy on some OCI mirrors; not currently in the prepull set | No — retry |

‎docs/security/self-assessment.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ This document is a security self-assessment for [Cozystack](https://github.com/c
1010
| Software | <https://github.com/cozystack/cozystack> |
1111
| Security Provider | No. Cozystack is a cloud platform, not a security product; security is a supporting property, not its primary function. |
1212
| Languages | Go, Helm, shell |
13-
| SBOM | Not published at this time. Image builds can emit a CycloneDX SBOM attestation (`SBOM=1` in `hack/common-envs.mk`), but this is disabled by default; SBOM generation on release is a planned improvement. |
13+
| SBOM | Published with each release as signed assets, for releases cut from a tree that has the scripts: `cozystack-source-sbom.spdx.json` (SPDX, made with syft from the committed source tree and the cozypkg binaries) and `cozystack-images-sbom.tar.gz` (SPDX, one document per platform of every first-party container image the release pins, read from the attestation BuildKit attaches at build time with `SBOM=1` in `hack/common-envs.mk`, which is off for other builds). Both are listed in `cozystack-checksums.txt`, whose keyless cosign signature a release carries; see [docs/release.md](../release.md#release-signatures-and-sbom). |
1414

1515
### Security links
1616

0 commit comments

Comments
 (0)