You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit c7a7a2a
Browse filesBrowse the repository at this point in the historyBrowse files
feat(release): sign the release images and attach their SBOMs
A release has signed its assets since #4861, but not the container
images a cluster installed from it pulls, and the SBOM each image carries
since #4997 sat in the registry only, readable with buildx and signed by
nothing. A user could verify the ISO they booted and not the platform it
installed.
hack/sign-release-images.sh signs every first-party digest the repinned
tree holds with cosign, keylessly, in prepare-release after the stitch.
It skips a digest that already verifies, so a re-run stacks no second
signature, and verifies each one against the identity docs/release.md
names before hack/upload-assets.sh runs, so nothing is uploaded over an
unsigned image. The signature over an index covers the attestation
manifest the index lists, so the SBOM needs none of its own, and cosign
keeps it in the repository hack/promote-retag.sh copies the digest
within, so promote-rc.yaml only verifies, with the rc tree's script,
before it stages the stable release.
hack/collect-image-sboms.sh reads the SBOM attestation of every such
image back into cozystack-images-sbom.tar.gz, one SPDX document per
platform, and fails on an image that has none. With
cozystack-images.txt, the digests the signatures are bound to, both join
the asset list hack/upload-assets.sh hands to hack/sign-release-assets.sh,
so the existing checksums signature covers them. docs/release.md gains
the verification recipe for a whole release; the bats files pin the
scripts, the upload list and the two workflow steps.
Refs #2937, #3278
Assisted-by: LLM
Signed-off-by: Marian Koreniuk <[email protected]>
Copy file name to clipboardExpand all lines: docs/agents/image-refs.md
+2Lines changed: 2 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -61,6 +61,8 @@ If a package stamps a reference somewhere the two globs do not cover, add that f
61
61
|`hack/stitch-multiarch.sh`| all three shapes | join each amd64 image and its `-arm64` twin into one index, repin the tree on it, then fail if any old digest survives |
62
62
|`hack/retention-packages.sh`| all three shapes, in the published nightly tree | name the GHCR packages the retention workflow prunes (those linked to another repository are then skipped); a ref it cannot see is a package whose nightlies are never pruned |
63
63
|`hack/verify-multiarch.sh`| all three shapes, first- and third-party | fail a release unless every pinned image is an index with `linux/amd64` and `linux/arm64`, bar the repositories in `hack/multiarch-allowlist`|
64
+
|`hack/sign-release-images.sh`| all three shapes, first-party | sign every pinned digest keylessly after the stitch, skipping one that already verifies, or with `--verify-only` fail a promotion whose rc images are unsigned; `--list` writes the `cozystack-images.txt` release asset |
65
+
|`hack/collect-image-sboms.sh`| all three shapes, first-party | read the SBOM attestation of every pinned index into the `cozystack-images-sbom.tar.gz` release asset, one document per platform; an image without one fails the release |
64
66
65
67
`hack/multiarch-allowlist` is the only bypass for that gate, and it is for images that cannot exist for arm64, not for images nobody has rebuilt yet. An entry is a repository with no tag or digest, so bumping an allowed image needs no edit, and it must carry a reason after `#` on the same line: the script refuses an entry without one. Allowed refs are still printed, under their own "allowed single-arch" heading, and in tree mode an entry that matches no ref that failed the check is reported as stale without changing the exit status, so it gets removed once the image is multi-arch or gone. `--refs-file` reports no stale entries, since a list of pulled images omits everything the run did not pull.
Copy file name to clipboardExpand all lines: docs/release.md
+31-2Lines changed: 31 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -327,10 +327,12 @@ So a commit on a supported `release-X.Y` line ships when a maintainer promotes t
327
327
328
328
## Release signatures and SBOM
329
329
330
-
A release cut from a tree that has [`hack/sign-release-assets.sh`](../hack/sign-release-assets.sh) carries three files beside its other assets, which [`hack/upload-assets.sh`](../hack/upload-assets.sh) has that script write before it uploads anything. Releases cut before it have none of them, and neither does a maintenance line until the script is backported to it (see below).
330
+
A release cut from a tree that has [`hack/sign-release-assets.sh`](../hack/sign-release-assets.sh) carries three files beside its other assets, which [`hack/upload-assets.sh`](../hack/upload-assets.sh) has that script write before it uploads anything, and one cut from a tree that also has [`hack/sign-release-images.sh`](../hack/sign-release-images.sh) and [`hack/collect-image-sboms.sh`](../hack/collect-image-sboms.sh) carries two more, which `tags.yaml::prepare-release` writes before the upload. Releases cut before them have none of these files, and neither does a maintenance line until the scripts are backported to it (see below).
331
331
332
332
-`cozystack-source-sbom.spdx.json` is an SPDX SBOM, made with syft, of the committed source tree (the Go modules, npm lockfiles, GitHub Actions references and other manifests syft reads in it) and of the `cozypkg` binaries in the `cozypkg-*.tar.gz` assets. It does not describe what runs in the cluster: neither the container images that `packages/` pins by digest, whose SBOMs are [#2937](https://github.com/cozystack/cozystack/issues/2937) (item A2 in [#3278](https://github.com/cozystack/cozystack/issues/3278)), nor the Talos ISO, disk, kernel and initramfs images. Its name says what it covers, so that `cozystack-sbom` stays free for an SBOM of the release as a whole.
333
-
-`cozystack-checksums.txt` holds the SHA-256 of every other asset and of the SBOM.
333
+
-`cozystack-images.txt` lists every first-party container image the release pins, one `<repository>@<digest>` per line: the multi-arch indexes [`hack/stitch-multiarch.sh`](../hack/stitch-multiarch.sh) repinned the tree on, which is what a cluster installed from this release pulls. The packages artifact and the installer chart are not in it.
334
+
-`cozystack-images-sbom.tar.gz` holds the SPDX SBOM of each image in that list, as `<image>.<os>-<arch>.spdx.json`, one document per platform, read from the attestation BuildKit attached to the image at build time (`SBOM=1` in `tags.yaml`, the scanner pinned in [`hack/common-envs.mk`](../hack/common-envs.mk)). These documents come from the syft inside that scanner, the source SBOM from the syft the workflow installs, so the two need not agree on a version. An image without an attestation fails the release instead of being left out.
335
+
-`cozystack-checksums.txt` holds the SHA-256 of every other asset, the SBOMs and the image list among them.
334
336
-`cozystack-checksums.txt.sigstore.json` is a Sigstore bundle with a keyless cosign signature over the checksums. Its certificate names the workflow that published the release: `tags.yaml` for a pre-release, `promote-rc.yaml` for a stable one.
335
337
336
338
One signature covers every asset through the checksums. To check a download, verify the checksums, then the file against them, with cosign v3.1.3 or later (v2.6.5 or later on the v2 line):
@@ -351,6 +353,32 @@ The script runs the same `cosign verify-blob` before it returns, so a signature
351
353
352
354
`promote-rc.yaml` runs the `hack/upload-assets.sh` of the rc tree, not the dispatch ref's. A stable release is therefore signed only when its rc was tagged from a tree that has `hack/sign-release-assets.sh`, and a maintenance line starts carrying signatures once that script is backported to its `release-X.Y` branch.
353
355
356
+
### Image signatures
357
+
358
+
The images in `cozystack-images.txt` are signed too. After the stitch, `hack/sign-release-images.sh` signs each digest with cosign, keylessly, as the `tags.yaml` run that prepared the pre-release, and verifies every signature against the same identity as above before `hack/upload-assets.sh` runs, so a release is never published with unsigned images. A signature over an index covers every manifest the index lists by digest, the attestation manifest that holds the SBOM included, which is why the SBOM carries no signature of its own. cosign keeps a signature beside the image, as the tag `sha256-<digest>.sig` of the same repository: promotion copies each digest to its stable tag inside that repository, so a stable release's images carry the signatures made for its rc, and their certificates name `tags.yaml` at the rc tag. `promote-rc.yaml::promote` checks every signature with the rc tree's `hack/sign-release-images.sh --verify-only` before it stages the stable release, and skips the check with a notice for an rc cut from a tree that has no such script. A re-run of `prepare-release` signs only the digests that do not verify yet, so it never stacks a second signature on an image. The packages artifact and the installer chart are not signed: promotion republishes both under a new digest.
359
+
360
+
### How to verify a release
361
+
362
+
With cosign v3.1.3 or later and the assets of the release in the current directory, this checks the signature over the checksums, every downloaded asset against them, the signature of every image the release pins, and unpacks the image SBOMs:
mkdir -p image-sboms && tar -xzf cozystack-images-sbom.tar.gz -C image-sboms
376
+
```
377
+
378
+
`cosign verify` prints the certificate of each signature; its `Subject` names the workflow and the tag, and the image is pulled only as far as its manifests. The same SBOM can be read from the registry without the archive, with `docker buildx imagetools inspect <image> --format '{{ json .SBOM }}'`, which prints the documents keyed by platform. Checking an image by the tag a release page shows rather than by the digest in `cozystack-images.txt` proves less: a tag can be moved, a digest cannot, and the signature is bound to the digest.
379
+
380
+
What fails, and what it means: `cosign verify-blob` rejecting the bundle, or `cosign verify` finding no matching signature, means the file or the image is not what the release workflow published, or cosign is older than the version above; `sha256sum` reporting a mismatch means the download is corrupt or replaced; an image in `cozystack-images.txt` that `cosign verify` cannot find at all means the registry no longer serves that digest. An image missing from the archive cannot happen for a release that has one, since `hack/collect-image-sboms.sh` fails the release instead.
381
+
354
382
## Nightly builds
355
383
356
384
A nightly is an **installable copy of `main` on GHCR — not a rebuild and not a release**. [`build-main.yaml`](../.github/workflows/build-main.yaml) already builds every push to `main` into the CI registry (OCIR); the nightly promotes that build to the public release registry (GHCR) and proves it installs.
@@ -645,6 +673,7 @@ For RCs and final releases, run this before merging the release PR. Each item is
645
673
| Changelog Copilot job returns 402 | Token quota — `COPILOT_GITHUB_TOKEN` premium-request quota empty | Both changelog entry points hit it: the optional rc-time `changelog-rc` dispatch and the promote-time `changelog` job (both call the reusable `changelog-generate.yaml`), plus the `tags.yaml` backstop | No — `continue-on-error: true` on the AI step, generate changelog by hand following [`agents/changelog.md`](./agents/changelog.md)|
646
674
|`Prepare Release` skipped and the run red, `Build arm64 images` failed | The rc's arm64 build failed | Read the `build-arm64` log; `build-arm64-nightly.yaml` usually failed the same way first. Fix on the line, cut the next rc | Yes — an rc is not published amd64 only |
647
675
|`Stitch multi-arch indexes and republish` fails listing refs |`verify-multiarch.sh` found refs that are not amd64+arm64 indexes, or that it could not check | Each listed ref names its reason. Single-platform manifest or index without one architecture: make the image multi-arch, or allowlist its repository with a reason if it cannot be. Manifest unreadable: correct the ref or digest; the allowlist does not excuse it. Rate-limited or unavailable after 3 tries: re-run once the throttle clears | Yes |
676
+
|`Sign the release images and collect their SBOMs` fails |`hack/sign-release-images.sh` could not sign a pinned digest or its signature does not verify against the release identity, or `hack/collect-image-sboms.sh` found a pinned image with no SBOM attestation | A signing failure names the ref: Fulcio or Rekor unreachable, or the OIDC token missing, re-run once they answer; a signature already there is kept, so a re-run signs only what is missing. An image without an SBOM was built without `SBOM=1` on the docker-container builder, or its index lost the attestation: fix the build, re-cut the rc | Yes |
648
677
| E2E `kubernetes-test` fails with `CSINode does not contain driver csi.kubevirt.io`| Test ordering bug | Tenant's `kubernetes-${test_name}-csi` HelmRelease still installing when NFS PVC is created. Fix: wait for `csinode/<node>` to advertise the driver before creating the PVC | Yes — flaky and masks real CSI bugs |
649
678
|`tenant-root` HR in perpetual upgrade-rollback loop after OIDC patch | Real bug at the intersection of #2602's readiness change and the cozystack reconcile cadence | Look for `rate: Wait(n=1) would exceed context deadline` in operator logs; multiple tenant child HRs stuck `InProgress`| Yes |
650
679
|`cozy-dashboard` ImagePullBackOff from `cozystack-ui:latest`| Registry flake | Known noisy on some OCI mirrors; not currently in the prepull set | No — retry |
Copy file name to clipboardExpand all lines: docs/security/self-assessment.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -10,7 +10,7 @@ This document is a security self-assessment for [Cozystack](https://github.com/c
| Security Provider | No. Cozystack is a cloud platform, not a security product; security is a supporting property, not its primary function. |
12
12
| Languages | Go, Helm, shell |
13
-
| SBOM |Not published at this time. Image builds can emit a CycloneDX SBOM attestation (`SBOM=1` in `hack/common-envs.mk`), but this is disabled by default; SBOM generation on release is a planned improvement. |
13
+
| SBOM |Published with each release as signed assets, for releases cut from a tree that has the scripts: `cozystack-source-sbom.spdx.json` (SPDX, made with syft from the committed source tree and the cozypkg binaries) and `cozystack-images-sbom.tar.gz` (SPDX, one document per platform of every first-party container image the release pins, read from the attestation BuildKit attaches at build time with `SBOM=1` in `hack/common-envs.mk`, which is off for other builds). Both are listed in `cozystack-checksums.txt`, whose keyless cosign signature a release carries; see [docs/release.md](../release.md#release-signatures-and-sbom). |
0 commit comments