Skip to content

[Tracking] Remaining CI / build / release / E2E automation work #3278

New issue
New issue
Open
Open
[Tracking] Remaining CI / build / release / E2E automation work#3278
Assignees
myasnikovdaniil
Labels
area/buildIssues or PRs related to image build infrastructure, multi-arch supportarea/ciIssues or PRs related to CI workflows, GitHub Actions, automationarea/releaseIssues or PRs related to release tooling (changelog, backport, release pipeline)area/testingIssues or PRs related to testing (e2e, bats, unit tests)epicA large development increment that brings definite value to Cozystack userstriage/acceptedIndicates an issue is ready to be actively worked on

Description

@myasnikovdaniil
myasnikovdaniil
opened on Jul 13, 2026

Summary

Umbrella tracking issue for the remaining CI / build / release / E2E-testing automation work. It is a companion to the individual epics (#2937, #2677, #2499, #2847, #2653, #2585, #3238, #4642) and gives one place to see what is done, what is ready, and what is externally blocked. Action IDs (A1-A20) are stable references. A2 was split into A2a and A2b on 2026-10-20.

Update 2026-10-20

#4861 merged on 2026-10-20 (d1d4e92). A release cut from a tree that has hack/sign-release-assets.sh now carries cozystack-source-sbom.spdx.json (SPDX 2.3 of the source tree and the cozypkg binaries), cozystack-checksums.txt and a keyless Sigstore bundle over the checksums, and docs/release.md shows how to verify a download. That is the release-asset half of A2, now A2a. It is not an SBOM of what runs, the container images and the Talos images are not in it. Those are A2b, still open, and what it needs changed (see below).

A2a has not run for real yet. The keyless signature needs the OIDC token of a release job, so the first rc or alpha cut after the merge is its test, and the first signed stable is the promotion of an rc cut after the merge (release-1.6 only after a backport). #4759 stays open for two follow-ups that wait for a signed release: project.documentation.signature-verification in security-insights.yml and the SBOM row in docs/security/self-assessment.md, which keeps the image caveat.

The identity check in docs/release.md now pins tags.yaml to v* tag pushes and promote-rc.yaml to main. Until A3 exists anyone with write access can still push a v* tag whose tree has an edited tags.yaml, and its signature passes, so A3 also decides what the signature proves. New A20: nothing refuses a promote-rc.yaml dispatch from another ref up front. On 2026-10-16 a dispatch from release-1.6 ran that branch's older copy of the workflow (one job instead of five, https://github.com/cozystack/cozystack/actions/runs/38882938797) and was re-run from main. With signing in place such a dispatch now fails at its upload step, after the staging branch, the promotion-* candidate and the draft already exist.

Update 2026-10-01

The E2E lane items were re-checked first. The remaining actions were checked against main at bc400dff1 on 2026-10-02 and the bullets below carry the corrections, the 2026-08-13 text further down is the older snapshot.

A14 and A15 are done via #4020 (2026-09-07, backported in #4437): srv1-srv3 of both merge-gating lanes run as Talos containers, and #3238 is closed. What the container lane cannot run (DRBD, the replicated StorageClass, live migration, the cozystack talos image) is the new action A18, tracked in #4642. nightly.yaml and e2e-tag.yaml still boot QEMU, the rc e2e job failed on v1.6.2-rc.1, v1.6.3-rc.1, v1.6.3-rc.2 and v1.6.4-rc.1, and v1.6.2, v1.6.3 and v1.6.4 were promoted with skip_e2e_gate: true. The nightly mirror job failure since 2026-09-15 is #4423 and is not part of A18.

Nightly is worse than the 2026-08-13 text says. The last scheduled green was 2026-09-03, before that 2026-08-17, and Artifact Retention has been red on every scheduled run since 2026-08-18 (new action A19). Promote RC skipped the e2e gate on all five runs since 2026-08-05 (v1.6.1, v1.6.2, v1.5.4, v1.6.3 and v1.6.4), each logged Promoting <rc> without verified green full E2E evidence.

Status as of 2026-08-13

Re-verified against main at 09dc7f47f plus live PR, run and registry state. Two actions moved to done since the 2026-07-20 snapshot: A5, because promote-rc.yaml has now cut two releases live (v1.6.0 on 2026-07-22 and v1.6.1 on 2026-08-05), and A16, closed by #3441 on 2026-08-10, which removes the sequencing constraint it placed on A9. One diagnosis is obsolete rather than resolved. A4's "nightly has never completed a run" was fixed by #3332 and nightly published for ten days, but it has failed every scheduled run since 2026-08-02 on its E2E (full suite) job. That failure is known and is being worked, so what changes here is the recorded cause, not the ownership. A17 is now partial, the specific retired-digest blocker is gone and the systemic preflight is not. The other twelve actions are unchanged, and none has been overtaken by the Chainsaw migration.

One sequencing change: A9 is gated on measurement rather than merely unblocked. Its payoff rests on a "~60% of PRs touch one app" figure that has never been measured, while docs/agents/e2e-testing.md §7 records that most PRs legitimately escalate to the full suite. The install-telemetry task in #2847 Track A is A9's prerequisite. The escalation surface that measurement would run against moved three times in the six days to 2026-08-10 and settled when #3434 closed on 2026-08-13, so a number taken now describes post-#3441 behaviour and should record the commit it was taken at.

Status snapshot

Epic Area State Remaining
#2937 build, ci ~90% SBOM of the container images (A2b), physical runner retire + rehome the breakpoint rendezvous server (A1 code done)
#2677 release closed 2026-08-14 tag-protection ruleset (A3, admin), tracked here rather than there. Nightly publishes again, and its E2E job is red and being worked (A4). Promote flow proven live (A5, done)
#2499 ci, testing gates exist go build ./... runs in codeql.yml on every PR and go-module-tests runs go test in the nested modules, A6 is a decision now
#2847 ci, testing ~40% of its checklist Track A: A8 and A16 are done, A9 moved to select-install.sh --disabled in #4022 (open) and the measurement was read by hand, install takes about 28 min and 73% of PR runs are full-suite. Track B is #3276 (A11), which needs a rework on the container lane
#2653 testing audit redone redone on the Chainsaw suites on 2026-10-01, no cleanup PR yet (A12)
#2585 ci, release closed 2026-08-27 namespaced backport labels (A13, done)
#3238 testing closed 2026-10-01 done via #4020 (A14 and A15). What the container lane cannot run is A18 (#4642). The opt-in OpenEBS LocalPV package #3266 is open on its own, changes requested

Done since the 2026-07-20 snapshot

Carried forward from earlier snapshots, still true:

Corrections to the previous snapshot

Ready now, no blockers

  • A2b, SBOM of the container images, on the release lane. The old A2 text said the blocker was an OCIR referrers probe, and that was the wrong place to look: release images are pushed to GHCR (REGISTRY ?= ghcr.io/cozystack/cozystack, OCIR is only CACHE_REGISTRY there), and the PR and main builds that do push to OCIR are not what users install. What blocks it, checked on a local registry with docker 29.8 and buildx 0.37.1 and by reading the scripts:

    • --sbom=true turns the pushed single-platform image into an OCI index of the image and an attestation manifest. image_arch in hack/stitch-multiarch.sh prints nothing for an index, so with SBOM on it would skip every image ("not one amd64 and one arm64 image") and hack/verify-multiarch.sh would then fail the release with "index without linux/arm64". The rerun branch, which looks for the amd64 digest among the children of the existing index, would not match either.
    • the release make build in tags.yaml runs on the default docker driver (use: false), which cannot attach attestations. The arm64 leg already uses docker-container with hack/buildkitd.toml (the docker.io mirror), the amd64 leg needs the same.
    • SBOM=1 passes --sbom=true, which pulls docker.io/docker/buildkit-syft-scanner:stable-1, a floating tag. It should be pinned by digest like the other build inputs.
    • the comment at hack/common-envs.mk:69 says CycloneDX, BuildKit writes SPDX 2.3 (predicate https://spdx.dev/Document) with the image's own packages.

    What works: an index stitched from two such images keeps both attestations and passes the verify-multiarch.sh check, and skopeo copy --multi-arch all (what promote-retag.sh and nightly-mirror.sh run) keeps the index digest and its children. So the delivery decision is attestation: the SBOM travels in the index through retag and mirror, and images need no workflow artifact or release attachment. Not checked: that GHCR accepts the attestations (the first alpha shows it), what retention does to the child manifests (retention-packages.sh), the Talos and matchbox builds in tags.yaml, and the fork lane's OCI_EXPORT_DIR output. Ties into the OpenSSF supply-chain work (ci: harden GitHub Actions workflows for OpenSSF Scorecard (pin, permissions, zizmor gate) #3223) and is cited as a gap in the CNCF GTR questionnaire (docs: add CNCF General Technical Review (GTR) questionnaire #3572).

  • A12, drop redundant post-HR-Ready waits across hack/e2e-chainsaw/*/chainsaw-test.yaml, find the assert/wait steps that duplicate HR Ready, keep the CR-operator-async waits. The audit in Audit e2e Chainsaw post-HR-Ready asserts for redundancy under Flux v2.8 kstatus #2653 was redone on the Chainsaw suites on 2026-10-01, the candidate list, the check of the nested -system HR on a dev cluster and the scope of the first PR are in its comments. Coordinate with [Epic] Adopt the kstatus readiness model platform-wide (Flux 2.8 / helm-controller v1.5.0) #2642: feat(api): typed waitStrategy/healthCheckExprs for kstatus readiness (#2642) #3273 (merged 2026-07-21) landed the typed waitStrategy/healthCheckExprs API, adopted so far only in packages/extra/monitoring and packages/extra/seaweedfs, and feat(flux): gate releases on operator status kstatus cannot read #3264 is still a draft with changes requested.

  • A20 (new), refuse a promote-rc.yaml dispatch from any ref but main in the parse job, before the first write. One github.ref check and a test, and the author of feat(release): sign the release assets and attach a source SBOM #4861 offered to send it. The update above has the dispatch from release-1.6 on 2026-10-16.

Blocked / sequential

Effort and closeability

Sizes recalculated 2026-08-13 against current state, not carried over. The A6, A9, A11 and A13 rows were updated on 2026-10-02, A2 was split into A2a and A2b on 2026-10-20. S = hours, M = a day or two, L = multi-day.

Action Issue Size Note
A6 #2499 - Done, CodeQL is a required check.
A13 #2585 - Done 2026-08-27. full-e2e is still undeclared in labels.yml.
A3 #2677 - Done 2026-11-04.
A4 #2677 S One RETENTION_APPLY flip. A3 and A19 are done.
A1 (residual) #2937 S Ops, outside the repo, physical host retire + rehome the breakpoint rendezvous server.
A2a #4759 - Done 2026-10-20 (#4861). Keyless signing is unproven until the first rc or alpha after the merge.
A2b #2937 M Release lane only: a docker-container builder for make build in tags.yaml, the scanner pinned by digest, stitch-multiarch.sh and verify-multiarch.sh taught about indexes with the stub-registry tests extended, docs. No OCIR probe.
A20 - S One github.ref check in parse and a test.
A11 #2847 - Rework on the container lane, not a rebase. Size not re-estimated.
A12 #2653 M Read-through of 21 runnable Chainsaw suites, lands as one cleanup PR. A14, the audit that shared this pass, is done via #4020.
A17 (systemic) #3305 M-L M for the inventory + scheduled preflight, L if the base-image mirror lands with it.
A9 #2847 - Done, #4022 merged on 2026-10-07 and the lanes are wired. The soak is in #2847.
A18 #4642 L The temporary-cluster half needs a design first: provisioning from CI in OCI, quota, cost, when it runs. The interim swap of nightly and rc e2e onto the container lane is not sized.

Issue → closeability

Recommended sequence

  1. The RETENTION_APPLY flip in A4.
  2. Close ci: add CI gates for codegen drift, Go build, and Go unit tests #2499, A6 is done.
  3. One Chainsaw read-through for A12.
  4. A2b SBOM of the images: builder, scanner pin, stitch and verify for indexes, then one alpha as the end-to-end check for A2a and A2b together.
  5. A11 once A18 says which substrate the upgrade lane runs on, then A17's preflight.
  6. The nightly lane's red runs are ci(nightly): Mirror images to GHCR fails on ghcr write_package, so the nightly e2e has not run since 2026-09-15 #4423 (mirror job, closed on 2026-10-02) and A18 (e2e: retire the QEMU lane, run what only it covered on a temporary cluster #4642, E2E job).

Activity

  1. myasnikovdaniil
    added
    epicA large development increment that brings definite value to Cozystack users
    area/ciIssues or PRs related to CI workflows, GitHub Actions, automation
    area/testingIssues or PRs related to testing (e2e, bats, unit tests)
    area/buildIssues or PRs related to image build infrastructure, multi-arch support
    area/releaseIssues or PRs related to release tooling (changelog, backport, release pipeline)
    on Jul 13, 2026
  2. myasnikovdaniil
    added 2 commits that reference this issue on Jul 13, 2026

    test(e2e): diff-driven install selector select-install.sh (#2847)

    103b8e1

    test(e2e): diff-driven install selector select-install.sh (#3279)

    215bd43

  3. myasnikovdaniil commented on Jul 20, 2026

    @myasnikovdaniil
    myasnikovdaniil
    on Jul 20, 2026
    ContributorAuthor

    State refresh — 2026-07-20

    Body updated to the 2026-07-20 snapshot, verified against main + live PR/issue state. What moved since the 07-13 snapshot:

    Done

    Two corrections to what the previous snapshot claimed

    • A4 was overstated. It recorded NIGHTLY_ENABLED as a partial win. The Nightly workflow has in fact never completed a run — every scheduled run since the variable was set fails ~80s in at "Resolve source commit and version", because cozystack-packages is a Flux OCI artifact (application/vnd.cncf.flux.config.v1+json) and skopeo inspect --format '{{.Digest}}' is an image-specific operation on it. Nothing has been published: cozy-installer:nightly and cozystack-packages:nightly both 404 on GHCR. fix(ci): resolve nightly packages digest without image-specific skopeo ops #3332 fixes that plus two flux pull artifact --output failures queued behind it. A4 is enabled but non-functional, not partially done.
    • A11 changed shape. No longer "unstall the [ci] Add upgrade test job to release pipeline #2401 review". [ci] Add upgrade test job to release pipeline #2401 predates the BATS→Chainsaw migration and is now a single conflicting 895-line BATS file; test(e2e): add chainsaw-native release upgrade testing lane #3276 supersedes it with a Chainsaw-native upgrade lane (install-previous → seed canary data → upgrade → assert survival/data-integrity/HR-reconciliation/migration-stamp) as a parallel non-blocking upgrade-e2e job. Remaining is review + a dev-cluster run for the canary access paths and the env-gated tenant-Kubernetes suites.

    Two new action IDs

    • A16 — select-e2e.sh: a per-suite test edit silently defeats the full-suite escalation #3330: select-e2e.sh's full-suite escalation is silently defeated when a PR also edits one unrelated suite's tests. The empty-final_apps safety net keys off emptiness alone, so any other contributed suite name swallows the escalation — the same system-package change selects 22 suites alone and exactly 1 alongside a suite edit. This is in the selection foundation A8 mirrors and A9 wires into CI, so it is sequenced ahead of A9.
    • A17 — ci: guard against pinned base-image digests disappearing upstream #3305: retired base-image digests block unrelated builds (quay.io/centos/centos:stream9 retired its pinned manifest on 07-15, five days after the last Renovate digest bump). Direction: mirror volatile bases into a project-controlled registry + a scheduled preflight that verifies every pinned base still resolves.

    Verified unchanged — A6 (no go build ./... gate in pull-requests.yaml), A13 (backport / backport-previous still top-level in .github/labels.yml), A12, A14 not started; A3 and A5 still admin / event-driven. A15's #3266 is still an open draft.

  4. Timur Tukaev (tym83)
    added a commit that references this issue on Aug 10, 2026

    docs(security): make the disclosure line and the support test checkable

    a627788
  5. github-actions
    added
    triage/acceptedIndicates an issue is ready to be actively worked on
    on Aug 12, 2026

  6. myasnikovdaniil commented on Aug 14, 2026

    @myasnikovdaniil
    myasnikovdaniil
    on Aug 14, 2026
    ContributorAuthor

    State refresh, 2026-08-13

    Body re-verified against main at 09dc7f47f plus live PR, workflow-run and registry state. A5 is done, promote-rc.yaml has cut two releases live, v1.6.0 (2026-07-22) and v1.6.1 (2026-08-05), so the orchestration has been proven twice rather than never. A16 is done, #3441 (2026-08-10) closed #3330 and #3665 by deciding coverage per changed path, and #3670 closed #3392 before it, which removes the correctness prerequisite that was sequenced ahead of A9. A9 is re-sequenced rather than started: it is now gated on the install-telemetry measurement in #2847 Track A, because the "~60% of PRs touch one app" payoff has never been measured and §7 of docs/agents/e2e-testing.md records that most PRs legitimately escalate to the full suite. The escalation surface moved three times in the six days to 2026-08-10 and settled when #3434 closed on 2026-08-13, so a measurement should record the commit it was taken at. A4's recorded diagnosis is obsolete. #3332 worked, cozy-installer:nightly and cozystack-packages:nightly both resolve on GHCR now where both previously 404'd, and the lane published for ten days, but every scheduled run since 2026-08-02 fails on its E2E (full suite) job. That failure is known and being worked, so A4's remaining repo-side content is the RETENTION_APPLY flip and the action ID stays whole. A17 is partial: #3386 and #3400 fixed the retired-digest trigger and moved kubevirt-csi-driver to docker.io/library/almalinux:9, but the inventory, mirror and scheduled preflight #3305 asks for do not exist. Two smaller corrections: A11 has gone backwards (#3276 is now conflicting as well as CHANGES_REQUESTED, and #3449 moved the release-validation jobs it sequences against), and A10 should not be read as closing #2979, which still has two BATS suites left. Everything else on the list is unchanged, and #3572 is a citation of this issue in the CNCF GTR questionnaire rather than work in scope.

  7. myasnikovdaniil commented on Sep 1, 2026

    @myasnikovdaniil
    myasnikovdaniil
    on Sep 1, 2026
    ContributorAuthor

    Status as of 2026-09-01

    Re-verified against main at e84a44689 plus live run, PR and label state.

    A13 is done, #2585 closed 2026-08-27. kind/backport and kind/backport-previous are in .github/labels.yml with bare names kept as aliases:, and backport.yaml uses the namespaced literals. Adjacent fix noted alongside it did not land: full-e2e is still undeclared (zero occurrences in labels.yml) and survives only because sync runs with delete-other-labels: false. Worth folding into next labels change.

    Everything else was re-checked individually rather than carried forward, and is unchanged: A6 (no go build ./... gate in pull-requests.yaml), A9 (select-install.sh invoked by no workflow), A12 (.chainsaw.yaml still parallel: 1), A14/A15 (hack/e2e-chainsaw/backup/chainsaw-test.yaml.disabled still parked), A2 (SBOM ?= 0 in hack/common-envs.mk, no workflow sets it), A17 systemic half (no scheduled base-digest preflight among the 27 workflows), A3 (one ruleset) and A4's RETENTION_APPLY (still false). A1 code half remains done, every runs-on is ubuntu-* or oracle-vm-*.

    A4: nightly is the steady state, not an incident

    The 2026-08-13 snapshot recorded nightly's E2E (full suite) job as a known failure being worked. It has failed every scheduled run since 2026-08-02, thirty days in a row, always at Run E2E tests (full suite). Latest: https://github.com/cozystack/cozystack/actions/runs/33462145699

    "Being worked" doesn't describe that anymore. Platform has had no daily full-suite signal for a month, so whatever TIA narrows away on the PR lane is unobserved until a release cut re-runs the full suite. It also blocks work nobody has proposed yet: any plan that moves an expensive suite off the PR lane and into nightly needs somewhere green to move it to, and there isn't one right now. I suggest promoting this from a note under A4 into its own action, size is unknown and other things queue behind it.

    A9: the measurement it waits on is partly answered, and points elsewhere

    A9 is gated on measuring the "~60% of PRs touch one app" figure. A related number is already available from workflow-run history, and it reframes the action instead of unblocking it. Over 2026-06-23..2026-08-23, of 2,118 E2E job executions only ~12% selected light suites, 1,260 selected kubernetes-*, and 692 died before the selector ran. Most runs escalate, which matches what docs/agents/e2e-testing.md already says about escalation being usually correct.

    So wiring select-install.sh as it stands would act on roughly one run in eight, because escalation decides install scope before the closure ever gets to narrow it. That doesn't make A9 wrong, but its payoff is bounded by escalation rate rather than by closure precision, so narrowing what an escalation escalates to should come first.

    Not currently tracked

    Three items with no action ID, all acting on the PR lane without touching what any suite asserts.

    1. Fourth selector outcome. select-e2e.sh has three now: full suite, a scoped set, nothing. Fail-closed is right and stays, but the only thing to fail closed to is the full suite, which includes the two most expensive tests in the tree. kubernetes-latest and kubernetes-previous are 68 of the 131 minutes of chainsaw time on a green run, measured on run 33372552453 via its chainsaw-report artifact. A broad-but-not-heaviest tier, every app suite without kubernetes-*, keeps every path classified and every escalation mandatory, only the destination changes. This is the prerequisite for A9 above.

    2. Suite parallelism. .chainsaw.yaml runs parallel: 1 and the stated reason is one suite's fixed port-forward port. The 17 suites that are neither kubernetes-* nor a backup round-trip cost 26 minutes sequentially. Needs scoping rather than a global flip, the heavy suites are already CPU starved on the current runner class, so per-Test concurrency control is likelier to be the right shape.

    3. Expensive lane once per PR instead of once per push. Currently 4.5 runs per PR on average, and one renovate branch got 41. Trigger it on pull_request_review (approved) rather than on synchronize. This costs nothing in coverage and that is checkable rather than asserted: renovate runs with automerge: false and platformAutomerge: false, the ruleset requires one approving review, so every PR that reaches main is approved by definition and gets the lane. Organization admins can still bypass, exactly as they can today. dismiss_stale_reviews_on_push is already on, so a push after approval invalidates it and the next approval re-runs the lane.

      Worth separating two axes that the full-e2e label currently carries together: when the lane runs, and what scope it runs at. The gating sites read github.event.label.name (the concurrency key, plan, and the two job guards), the scope sites read contains(labels.*.name, 'full-e2e') (the rc check, the TIA skip, and the FULL_E2E env). They are already in separate places. A second label for "run the lane now at the scope this diff selects" fits the gating sites only, and leaves full-e2e owning scope alone. It should be one-shot, removed when the run starts, while full-e2e stays sticky because promote PRs rely on that. Naming: e2e/run and e2e/full, with aliases: ['full-e2e'] on the second, same shape A13 just gave the backport labels. full-e2e has to be declared in labels.yml anyway, so declaring the pair in the right form now is cheaper than renaming it later.

      One caution for whoever writes it: the concurrency key and plan's guard are exact complements of each other by design, and the comment above them records what happened on live CI when they disagreed. A third label name has to be added to both.

    A15 has moved

    Tracker points A15 at #3266, which is still a draft with changes requested and hasn't moved since 2026-07-20. Live work is #3961 now, container lane plus three storage fixes it exposed. Worth repointing.

  8. Andrei Kvapil (kvaps)
    added a commit that references this issue on Sep 7, 2026

    docs(security): make the disclosure line and the support test checkable

    9cc8c6c
  9. myasnikovdaniil
    self-assigned this
    on Oct 1, 2026

  10. myasnikovdaniil commented on Oct 2, 2026

    @myasnikovdaniil
    myasnikovdaniil
    on Oct 2, 2026
    ContributorAuthor

    State refresh, 2026-10-02

    Added the rest of the re-check to the body, against main at bc400dff1. A18 stays as written (#4642).

    Two things I wrote on 2026-09-01 are wrong. Nightly was not red 30 days in a row, there was a green scheduled run on 2026-09-03 (and one on 2026-08-17). And A15 should not point at #3961, it was closed unmerged on 2026-09-30 after #4020 landed the container lane.

    New is A19: the GHCR retention job fails on its package listing (HTTP 400, red on every scheduled run since 2026-08-18), so flipping RETENTION_APPLY in A4 would delete nothing. A6 is partial, go build ./... already runs in codeql.yml, it is just not a required check. A9 changed design, #4022 uses a --disabled list, so test-minimal is obsolete. A13 is done. A17 got a third incident (#4486) outside the scan scope of #3305.

    The PR-cost series was in no tracker before. #4005, #4008, #4022 and #4023 are open, #4019, #4020 and #4021 are merged.

  11. lexfrei commented on Oct 9, 2026

    @lexfrei
    Aleksei Sviridkin (lexfrei)
    on Oct 9, 2026
    Contributor

    A19 has a cause: GitHub refuses to list an org's container packages to a GitHub App installation token. GET /orgs/{org}/packages?package_type=container returns 400 Invalid argument for that token type, other package types work with the same token, and retention.yaml calls it with the cozystack-ci app token. A classic PAT with read:packages lists all 97 org packages with the same request. This is undocumented but known, see actions/github-script#449 and cli/cli#9606.

    The listing never worked in this workflow. Every scheduled run from 2026-07-15 to 2026-08-17 logged the same 400 and still exited green with "No cozystack/* container packages found", so retention has never pruned anything. ffae003 only made the old failure visible, from 2026-08-18 on.

    Two ways out. One is a classic PAT with read:packages and delete:packages as a secret for this job only. The other is to drop the org-wide listing and loop over a known list of package names, because /orgs/{org}/packages/container/{name}/versions accepts an installation token. I didn't check whether the version DELETE accepts the app token either, that needs a test before RETENTION_APPLY is flipped.

  12. lexfrei commented on Oct 9, 2026

    @lexfrei
    Aleksei Sviridkin (lexfrei)
    on Oct 9, 2026
    Contributor

    A4 follow-up from #4826: the retention workflow will not prune cozystack/cozystack-ui. It has 68 dated nightly versions today, 54 over the keep count of 14, and the number grows with every nightly.

    The reason is the token. #4826 deletes with this repository's GITHUB_TOKEN, because the app token gets 404 on DELETE. That token deleted a version of a package linked to cozystack/cozystack, but cozystack/cozystack-ui is linked to the cozystack/cozystack-ui repository. I did not test the token there and the workflow does not try: it skips the package and names it in a warning on every run.

    Two ways to cover it:

    • a retention job in cozystack/cozystack-ui itself, deleting with that repository's own GITHUB_TOKEN
    • an explicit Actions access role for cozystack/cozystack in the package settings (needs a package admin, and it is untested whether that is enough for DELETE)
  13. myasnikovdaniil
    added a commit that references this issue on Oct 12, 2026

    ci(retention): sweep packages named by the published nightly tree (#4826

    ebceef4
  14. myasnikovdaniil
    added a commit that references this issue on Oct 20, 2026

    feat(release): sign the release assets and attach a source SBOM (#4861)

    d1d4e92
  15. Marian Koreniuk (themoriarti)
    added 2 commits that reference this issue on Oct 26, 2026

    feat(build): attach a provenance attestation beside the SBOM

    0e3b4a4

    feat(release): sign the release images and attach their SBOMs

    dcff043
  16. Aleksei Sviridkin (lexfrei)
    added a commit that references this issue on Nov 1, 2026

    feat(release): attach an SBOM to every release image (#4997)

    d32092a
  17. Marian Koreniuk (themoriarti)
    added 2 commits that reference this issue on Nov 1, 2026

    feat(release): sign the release images and attach their SBOMs

    57346c1

    feat(build): attach a provenance attestation beside the SBOM

    57db352

  18. themoriarti commented on Nov 2, 2026

    @themoriarti
    Marian Koreniuk (themoriarti)
    on Nov 2, 2026
    Collaborator

    Status of A2a, A2b, A20 and two re-checks, 2026-11-02

    Checked against main, release-1.7 and the releases page. The body still says A2a has not run for real, and that is still true, but three releases have been cut since it merged, so here is why.

    A2a is still unproven. v1.7.0-rc.2 (2026-10-21), v1.7.0-rc.3 (2026-10-27) and v1.7.0 (2026-10-29) all came out of release-1.7, which does not have hack/sign-release-assets.sh, so each of them carries cozypkg-checksums.txt and nothing else: no cozystack-source-sbom.spdx.json, no cozystack-checksums.txt, no Sigstore bundle. The backport is #5172, approved and mergeable, held only by the red Unit & controller tests check, which is #5178 (the release prep rewrote two image refs the tests read) and not the signing change. #5179 fixes it on release-1.7 and #5180 on main, both green, both waiting for review. Order: #5179, then #5172 rebased on it, then the first v1.7.1-rc.* is the test of the keyless signature. main has had no tag since 2026-10-20, so there is no alpha to look at either.

    A2b moved. #4997 (merged 2026-11-01) attaches an SPDX SBOM to every release image, which is the part the body lists as "ready now": a docker-container builder for make build in tags.yaml, the scanner pinned by digest, stitch-multiarch.sh and verify-multiarch.sh taught about indexes. What is left is the image signature, #5042, and the provenance attestation beside the SBOM, #5043. Both are rebased onto #4997 and ready for review; they touch the same block of docs/release.md, so whichever lands second needs a rebase. Like A2a, none of it has run on a real tag yet.

    A20 is mine and not sent yet. I offered it on 2026-10-20 and have not opened the PR. Nothing in promote-rc.yaml on main checks github.ref today, the two ref: lines in it only pass github.ref_name through to checkouts. #5169 reorders the same workflow's publish step, so I will open A20 after it lands or against it, not beside it.

    Two re-checks against the body.

    • A9: feat(platform, tests): select E2E packages and declare controller dependencies #4022 merged on 2026-10-07 and the wiring the body lists as follow-up work is on main: pull-requests.yaml:1108 and e2e-fork.yaml:681 both call ./hack/select-install.sh --disabled "${apps}". What remains for A9 is whether it saves anything, which the 73% full-suite figure says is bounded, not whether it is wired.
    • A19: the cause is in lexfrei's comment above, ci(retention): sweep packages named by the published nightly tree #4826 (merged 2026-10-12) sweeps a known package list instead of the org listing, so the scheduled run is no longer red for that reason. RETENTION_APPLY still resolves to false at retention.yaml:89, and cozystack/cozystack-ui is skipped with a warning on every run, as described in the follow-up. A4 is therefore unblocked on the listing and blocked on the two token questions.

  19. themoriarti commented on Nov 2, 2026

    @themoriarti
    Marian Koreniuk (themoriarti)
    on Nov 2, 2026
    Collaborator

    A20 is #5181: the first step of parse in promote-rc.yaml fails when github.ref is not refs/heads/main, with a bats test over the step's shell. It does not overlap with #5169's hunks, so the two can land in either order.

  20. Marian Koreniuk (themoriarti)
    added 2 commits that reference this issue on Nov 2, 2026

    feat(release): sign the release images and attach their SBOMs

    777dbe7

    feat(build): attach a provenance attestation beside the SBOM

    662fcf8

  21. myasnikovdaniil commented on Nov 10, 2026

    @myasnikovdaniil
    myasnikovdaniil
    on Nov 10, 2026
    ContributorAuthor

    State refresh, 2026-11-10

    A3, A6, A9 and A19 are done and moved to the done list. Tag rulesets exist since 2026-11-04 (creation only by the CI app, tags immutable), CodeQL and API Review Gate are required on main, #4022 merged on 2026-10-07 and #4826 on 2026-10-12, retention runs are green since 2026-10-13. A4 is only the RETENTION_APPLY flip now, it is still a dry-run.

    A8, A10 and A11 lost text that #2847 and #3453 already carry, IDs are the same. A2b and A20 are not done, #5042 and #5181 are open for them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area/buildIssues or PRs related to image build infrastructure, multi-arch supportarea/ciIssues or PRs related to CI workflows, GitHub Actions, automationarea/releaseIssues or PRs related to release tooling (changelog, backport, release pipeline)area/testingIssues or PRs related to testing (e2e, bats, unit tests)epicA large development increment that brings definite value to Cozystack userstriage/acceptedIndicates an issue is ready to be actively worked on

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions