Skip to content

ci: publish keyless release attestations#232

Merged
lockwobr merged 7 commits into
NVIDIA:mainfrom
AnouarMohamed:fix/issue-224-release-attestations
May 19, 2026
Merged

ci: publish keyless release attestations#232
lockwobr merged 7 commits into
NVIDIA:mainfrom
AnouarMohamed:fix/issue-224-release-attestations

Conversation

@AnouarMohamed

@AnouarMohamed AnouarMohamed commented May 13, 2026

Copy link
Copy Markdown
Contributor

Description

Adds keyless release signatures and attestations for GHCR image release artifacts.

The workflows now resolve top-level GHCR OCI digests before signing, sign GHCR agent and operator image releases with GitHub Actions OIDC, attach CycloneDX SBOM attestations, publish SLSA v1 provenance through GitHub artifact attestations, and verify the signature, SBOM, and provenance before the release jobs pass.

Addresses #224.

Validation:

  • git diff --check
  • go run github.com/rhysd/actionlint/cmd/actionlint@latest -color -shellcheck= .github/workflows/agent-ci.yaml .github/workflows/operator-ci.yaml
  • yamllint -c ci/yamllint.yaml .github/actions/resolve-oci-digest/action.yml .github/actions/cosign-sign-sbom/action.yml .github/actions/cosign-verify-release/action.yml .github/workflows/agent-ci.yaml .github/workflows/operator-ci.yaml
  • npx --yes markdownlint-cli2 --config ci/.markdownlint-cli2.yaml docs/release-process.md

Checklist

  • I am familiar with the Contributing Guidelines.
  • My commits are signed off (git commit -s) per the DCO.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@AnouarMohamed AnouarMohamed force-pushed the fix/issue-224-release-attestations branch from 6dbcdf4 to 5c13922 Compare May 13, 2026 00:38
@AnouarMohamed AnouarMohamed marked this pull request as ready for review May 13, 2026 14:04
@coderabbitai

coderabbitai Bot commented May 13, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

This PR adds three composite GitHub Actions: resolve-oci-digest (resolves image:tag to sha256 digest), cosign-sign-sbom (installs cosign and Syft, generates CycloneDX SBOM, signs the OCI subject, and attaches the SBOM attestation), and cosign-verify-release (verifies cosign signature and CycloneDX/SLSA attestations). agent-ci and operator-ci workflows now run when .github/actions/** change, expose manifest digest/subject outputs, and integrate digest resolution, SBOM signing, provenance attestation, and verification. docs/release-process.md documents the digest-first verification steps and expected OIDC/certificate identity constraints.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Suggested reviewers

  • lockwobr
  • mskalka
  • rice-riley
  • ayuskauskas
🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately summarizes the main change: adding keyless release attestations (signatures and attestations) to CI workflows for GHCR image artifacts.
Description check ✅ Passed The description is clearly related to the changeset, explaining the implementation of keyless release signatures and attestations, providing context about the workflows and validation steps performed.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/release-process.md`:
- Around line 187-188: The IDENTITY regexp in the release-process doc is missing
start/end anchors which weakens strict matching; update the IDENTITY value to
include anchors (e.g. ^...$) so it becomes
'^https://github.com/NVIDIA/nodewright/.*$' and do the same for ISSUER if you
want identical strictness (e.g.
'^https://token.actions.githubusercontent.com$'), ensuring the IDENTITY and
ISSUER symbols now use anchored regexes for exact verification.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 78499a44-334b-49f5-92bf-0b8aef0fcf25

📥 Commits

Reviewing files that changed from the base of the PR and between 13294d3 and 5c13922.

📒 Files selected for processing (7)
  • .github/actions/cosign-sign-sbom/action.yml
  • .github/actions/cosign-verify-release/action.yml
  • .github/actions/resolve-oci-digest/action.yml
  • .github/workflows/agent-ci.yaml
  • .github/workflows/operator-ci.yaml
  • .github/workflows/release.yml
  • docs/release-process.md

Comment thread docs/release-process.md Outdated
@lockwobr

Copy link
Copy Markdown
Collaborator

The nvcr.io part of the ticket is not going to work for a few reasons,so I think we need to drop support for nvcr part of this PR.

@AnouarMohamed

Copy link
Copy Markdown
Contributor Author

Dropped the NVCR/NGC attestation path from this PR. GHCR image signing/attestation remains, and the release docs now cover GHCR-only verification with anchored identity matching.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/release-process.md`:
- Around line 174-178: The current certificate identity regex
'^https://github.com/NVIDIA/nodewright/.*$' is too broad; replace it wherever
used (including the IDENTITY variable and the documented regex examples) with a
scoped pattern that only allows the release workflow files and tag refs, e.g. a
pattern matching '.github/workflows/(agent-ci|operator-ci)\.yaml@refs/tags/.*'
so attestations are accepted only from the release jobs; update both occurrences
in the document accordingly.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 7db70f17-5919-429d-bc85-ddcc2e6ecb59

📥 Commits

Reviewing files that changed from the base of the PR and between 5c13922 and 5c8c569.

📒 Files selected for processing (3)
  • .github/workflows/agent-ci.yaml
  • .github/workflows/operator-ci.yaml
  • docs/release-process.md

Comment thread docs/release-process.md Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/agent-ci.yaml:
- Around line 275-296: The three steps "Sign GHCR agent image and attach SBOM",
"Attest GHCR agent provenance", and "Verify GHCR agent signature and
attestations" currently gate only on env.PUSH_TO_REGISTRY == 'true' but the
verify step only accepts identities from refs/tags/agent/*; update the steps' if
conditions to also require the run is an agent tag by changing each if to:
env.PUSH_TO_REGISTRY == 'true' && startsWith(github.ref, 'refs/tags/agent/'),
e.g., replace the existing if for the steps named above with this combined
expression so only agent tag refs run signing, attestation, and verification.

In @.github/workflows/operator-ci.yaml:
- Around line 411-432: The verify step fails on non-tag workflows because
env.PUSH_TO_REGISTRY is true for main/PRs too while certificate-identity-regexp
expects refs/tags/operator/*; update the three steps named "Sign GHCR operator
image and attach SBOM", "Attest GHCR operator provenance", and "Verify GHCR
operator signature and attestations" to gate on the operator tag ref (e.g.,
replace or augment the existing if: env.PUSH_TO_REGISTRY == 'true' with a check
like startsWith(github.ref, 'refs/tags/operator/') or equivalent), and keep the
certificate-identity-regexp as-is so verification only runs for operator tag
builds.

In `@docs/release-process.md`:
- Around line 174-178: Change the combined IDENTITY regex that allows both agent
and operator workflow certificates into two image-specific regexes so each image
only accepts its own workflow: replace the single pattern containing
"(agent-ci\.yaml@refs/tags/agent/|operator-ci\.yaml@refs/tags/operator/)" with a
dedicated operator pattern
"^https://github.com/NVIDIA/nodewright/\.github/workflows/operator-ci\.yaml@refs/tags/operator/.*$"
and add a separate agent pattern
"^https://github.com/NVIDIA/nodewright/\.github/workflows/agent-ci\.yaml@refs/tags/agent/.*$"
in the documentation examples (update the intro text around the current combined
regex and the corresponding example sections) and make the same split for the
other occurrence referenced in the comment (the block around lines 182-204).
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: dc07779f-1f40-487c-911c-05b56ef6599d

📥 Commits

Reviewing files that changed from the base of the PR and between 5c8c569 and 82524d5.

📒 Files selected for processing (3)
  • .github/workflows/agent-ci.yaml
  • .github/workflows/operator-ci.yaml
  • docs/release-process.md

Comment thread .github/workflows/agent-ci.yaml
Comment thread .github/workflows/operator-ci.yaml
Comment thread docs/release-process.md Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/release-process.md`:
- Around line 244-247: The table under "OCI subject" lists image repositories
but must show immutable digest format; update the GHCR entries
(`ghcr.io/nvidia/nodewright/operator` and `ghcr.io/nvidia/nodewright/agent`) to
include their corresponding immutable digests (e.g.
`ghcr.io/nvidia/nodewright/operator@sha256:<digest>` and
`ghcr.io/nvidia/nodewright/agent@sha256:<digest>`) so the table matches the
digest-first verification flow and prevents copy/paste errors.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: c3ebcffa-a430-4389-9562-480f341b08ff

📥 Commits

Reviewing files that changed from the base of the PR and between 82524d5 and 8913486.

📒 Files selected for processing (3)
  • .github/workflows/agent-ci.yaml
  • .github/workflows/operator-ci.yaml
  • docs/release-process.md

Comment thread docs/release-process.md Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/release-process.md`:
- Around line 166-172: Add a short "Prerequisites" bullet list under the "Verify
release signatures and attestations" section so readers know they must install
external tools before running verification; specifically mention that users need
cosign, jq, and docker buildx (and any minimum version if relevant), and place
this directly above or below the paragraph containing the expected OIDC issuer
string "https://token.actions.githubusercontent.com" to mirror the style used in
the digest pinning section.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: f7dc672f-8e0e-496f-8fbb-7ab290978e0a

📥 Commits

Reviewing files that changed from the base of the PR and between 8913486 and 424ff7f.

📒 Files selected for processing (1)
  • docs/release-process.md

Comment thread docs/release-process.md
@lockwobr

lockwobr commented May 18, 2026

Copy link
Copy Markdown
Collaborator

This is missing one requirement, and might because we merged this since you opened this, but the helm chart also needs the to signed and attested just like the images. Once we have that I think this is good to merge. Thanks again for working on this.

@lockwobr lockwobr merged commit 2af03eb into NVIDIA:main May 19, 2026
67 of 71 checks passed
lockwobr added a commit that referenced this pull request May 19, 2026
* ci: publish keyless release attestations



* ci: drop nvcr release attestations



* ci: narrow release attestation identity



* ci: gate attestations to release tags



* docs: show immutable release subjects



* ci: attest helm chart releases



---------

Signed-off-by: AnouarMohamed <[email protected]>
Co-authored-by: Anouar Mohamed <[email protected]>
lockwobr added a commit that referenced this pull request May 22, 2026
* ci: publish keyless release attestations (#232) (#240)

* ci: publish keyless release attestations



* ci: drop nvcr release attestations



* ci: narrow release attestation identity



* ci: gate attestations to release tags



* docs: show immutable release subjects



* ci: attest helm chart releases



---------

Signed-off-by: AnouarMohamed <[email protected]>
Co-authored-by: Anouar Mohamed <[email protected]>

* chore(docs): update docs around release and location of helm chart (#237) (#239)

* fix(test): uninstall downgraded package before cleanup in downgrade-after-uninstall (#241)

The test left [email protected] freshly installed at chainsaw's automatic
CLEANUP, where the new delete-time uninstall finalizer (from #200) must
run uninstall pods on every node before releasing the CR. That exceeds
chainsaw's default cleanup window, causing "context deadline exceeded".

Add a final uninstall-v1 step that flips uninstall.apply=true on the
downgraded version and waits for nodeState to empty, so the CR has no
installed packages when cleanup deletes it.

Signed-off-by: Alex Yuskauskas <[email protected]>

* fix: deadlock in webhook controller when upgrading from old versions (#243)

* fix: close StageInterrupt trap + harden core e2e pool (#242)

* fix(test): harden core e2e pool against finalizer-cleanup races and improve diagnosability

Two changes across the six core-pool chainsaw tests:

1. Add `timeouts.cleanup: 120s` to every core-pool test. Chainsaw's
   default cleanup window (~30s) is too tight for the project's CR
   finalizer, which must uncordon nodes, remove labels/annotations,
   and GC package pods before releasing. Same failure mode that hit
   downgrade-after-uninstall (PR #241).

2. Add `catch:` blocks to `depends-on` and `simple-skyhook`, mirroring
   the pattern used by the other four core tests. When these tests
   flake, CI now captures the Node, Skyhook CR, and package Pods for
   diagnosis instead of failing without artifacts.

No assert logic changes — this is a foundation pass to remove a known
flake class and make future flakes diagnosable.

Signed-off-by: Alex Yuskauskas <[email protected]>

* fix(operator): close StageInterrupt trap when configUpdates signal decays

A package whose only interrupt is a `configInterrupts` entry (no top-level
`interrupt` block) could get stuck at `StageInterrupt/StateSkipped`
permanently when `Status.ConfigUpdates` cleared or never persisted (e.g.
due to a 409 on the spec patch). The state machine queried the dynamic
`HasInterrupt(config)` signal at four points, and once that signal
decayed to false, the package was untouchable: `ProgressSkipped`
wouldn't promote it, `NextStage` wouldn't advance it past Interrupt,
and `GetComplete`/`IsPackageComplete` wouldn't count it complete.

Decouple progression-past-Interrupt from the dynamic signal:

- `NextStage`: add `StageInterrupt → StagePostInterrupt` to the
  no-interrupt default map. The with-interrupt full-replacement map is
  preserved as-is — PR #200 deliberately omits `StageUninstall →
  StageApply` from it so with-interrupt uninstalls route via
  `StageUninstallInterrupt`; collapsing the maps would silently
  re-enable that transition.
- `ProgressSkipped`: drop the `HasInterrupt` gate. The only writer of
  `StateSkipped` at `StageInterrupt` is `ProcessInterrupt`'s
  budget-contention branch, which already decided the package needed
  an interrupt to schedule. Stage alone is sufficient.
- `GetComplete`, `IsPackageComplete`: treat `StagePostInterrupt` as
  unconditionally terminal. The only way to reach it is via the
  interrupt cycle; gating the terminal check on a signal that can
  decay is redundant with the entry gate and only becomes load-bearing
  when the trap fires.

Reproducer in `skyhook_types_test.go` exercises all three sites.

Surfaced by the `chainsaw/config-skyhook` "update while running" step,
which patches the spec mid-flight and concurrently triggers the
`HandleMigrations` and `processSkyhooksPerNode` 409 conflict paths —
those stretch convergence enough that ConfigUpdates can be lost
between the package reaching StageInterrupt and ProgressSkipped firing.

Signed-off-by: Alex Yuskauskas <[email protected]>

* refactor(operator): GetComplete delegates to IsPackageComplete

GetComplete and IsPackageComplete encoded the same terminal-state logic
twice. Have GetComplete iterate the spec packages and call
IsPackageComplete for each, and move the explanatory comment to
IsPackageComplete where the predicate lives.

Behavior preserved: the prior implementation iterated node state and
filtered to spec packages by name+version match; the new one iterates
spec packages and looks them up in node state by unique name (name|version).
Same set of packages either way.

Signed-off-by: Alex Yuskauskas <[email protected]>

---------

Signed-off-by: Alex Yuskauskas <[email protected]>

---------

Signed-off-by: AnouarMohamed <[email protected]>
Signed-off-by: Alex Yuskauskas <[email protected]>
Co-authored-by: Anouar Mohamed <[email protected]>
Co-authored-by: ayuskauskas <[email protected]>
rice-riley pushed a commit that referenced this pull request Jun 9, 2026
* ci: publish keyless release attestations (#232) (#240)

* ci: publish keyless release attestations

* ci: drop nvcr release attestations

* ci: narrow release attestation identity

* ci: gate attestations to release tags

* docs: show immutable release subjects

* ci: attest helm chart releases

---------

Signed-off-by: AnouarMohamed <[email protected]>
Co-authored-by: Anouar Mohamed <[email protected]>

* chore(docs): update docs around release and location of helm chart (#237) (#239)

* fix(test): uninstall downgraded package before cleanup in downgrade-after-uninstall (#241)

The test left [email protected] freshly installed at chainsaw's automatic
CLEANUP, where the new delete-time uninstall finalizer (from #200) must
run uninstall pods on every node before releasing the CR. That exceeds
chainsaw's default cleanup window, causing "context deadline exceeded".

Add a final uninstall-v1 step that flips uninstall.apply=true on the
downgraded version and waits for nodeState to empty, so the CR has no
installed packages when cleanup deletes it.

Signed-off-by: Alex Yuskauskas <[email protected]>

* fix: deadlock in webhook controller when upgrading from old versions (#243)

* fix: close StageInterrupt trap + harden core e2e pool (#242)

* fix(test): harden core e2e pool against finalizer-cleanup races and improve diagnosability

Two changes across the six core-pool chainsaw tests:

1. Add `timeouts.cleanup: 120s` to every core-pool test. Chainsaw's
   default cleanup window (~30s) is too tight for the project's CR
   finalizer, which must uncordon nodes, remove labels/annotations,
   and GC package pods before releasing. Same failure mode that hit
   downgrade-after-uninstall (PR #241).

2. Add `catch:` blocks to `depends-on` and `simple-skyhook`, mirroring
   the pattern used by the other four core tests. When these tests
   flake, CI now captures the Node, Skyhook CR, and package Pods for
   diagnosis instead of failing without artifacts.

No assert logic changes — this is a foundation pass to remove a known
flake class and make future flakes diagnosable.

Signed-off-by: Alex Yuskauskas <[email protected]>

* fix(operator): close StageInterrupt trap when configUpdates signal decays

A package whose only interrupt is a `configInterrupts` entry (no top-level
`interrupt` block) could get stuck at `StageInterrupt/StateSkipped`
permanently when `Status.ConfigUpdates` cleared or never persisted (e.g.
due to a 409 on the spec patch). The state machine queried the dynamic
`HasInterrupt(config)` signal at four points, and once that signal
decayed to false, the package was untouchable: `ProgressSkipped`
wouldn't promote it, `NextStage` wouldn't advance it past Interrupt,
and `GetComplete`/`IsPackageComplete` wouldn't count it complete.

Decouple progression-past-Interrupt from the dynamic signal:

- `NextStage`: add `StageInterrupt → StagePostInterrupt` to the
  no-interrupt default map. The with-interrupt full-replacement map is
  preserved as-is — PR #200 deliberately omits `StageUninstall →
  StageApply` from it so with-interrupt uninstalls route via
  `StageUninstallInterrupt`; collapsing the maps would silently
  re-enable that transition.
- `ProgressSkipped`: drop the `HasInterrupt` gate. The only writer of
  `StateSkipped` at `StageInterrupt` is `ProcessInterrupt`'s
  budget-contention branch, which already decided the package needed
  an interrupt to schedule. Stage alone is sufficient.
- `GetComplete`, `IsPackageComplete`: treat `StagePostInterrupt` as
  unconditionally terminal. The only way to reach it is via the
  interrupt cycle; gating the terminal check on a signal that can
  decay is redundant with the entry gate and only becomes load-bearing
  when the trap fires.

Reproducer in `skyhook_types_test.go` exercises all three sites.

Surfaced by the `chainsaw/config-skyhook` "update while running" step,
which patches the spec mid-flight and concurrently triggers the
`HandleMigrations` and `processSkyhooksPerNode` 409 conflict paths —
those stretch convergence enough that ConfigUpdates can be lost
between the package reaching StageInterrupt and ProgressSkipped firing.

Signed-off-by: Alex Yuskauskas <[email protected]>

* refactor(operator): GetComplete delegates to IsPackageComplete

GetComplete and IsPackageComplete encoded the same terminal-state logic
twice. Have GetComplete iterate the spec packages and call
IsPackageComplete for each, and move the explanatory comment to
IsPackageComplete where the predicate lives.

Behavior preserved: the prior implementation iterated node state and
filtered to spec packages by name+version match; the new one iterates
spec packages and looks them up in node state by unique name (name|version).
Same set of packages either way.

Signed-off-by: Alex Yuskauskas <[email protected]>

---------

Signed-off-by: Alex Yuskauskas <[email protected]>

---------

Signed-off-by: AnouarMohamed <[email protected]>
Signed-off-by: Alex Yuskauskas <[email protected]>
Co-authored-by: Anouar Mohamed <[email protected]>
Co-authored-by: ayuskauskas <[email protected]>
rice-riley pushed a commit that referenced this pull request Jun 9, 2026
* ci: publish keyless release attestations (#232) (#240)

* ci: publish keyless release attestations

* ci: drop nvcr release attestations

* ci: narrow release attestation identity

* ci: gate attestations to release tags

* docs: show immutable release subjects

* ci: attest helm chart releases

---------

Signed-off-by: AnouarMohamed <[email protected]>
Co-authored-by: Anouar Mohamed <[email protected]>

* chore(docs): update docs around release and location of helm chart (#237) (#239)

* fix(test): uninstall downgraded package before cleanup in downgrade-after-uninstall (#241)

The test left [email protected] freshly installed at chainsaw's automatic
CLEANUP, where the new delete-time uninstall finalizer (from #200) must
run uninstall pods on every node before releasing the CR. That exceeds
chainsaw's default cleanup window, causing "context deadline exceeded".

Add a final uninstall-v1 step that flips uninstall.apply=true on the
downgraded version and waits for nodeState to empty, so the CR has no
installed packages when cleanup deletes it.

Signed-off-by: Alex Yuskauskas <[email protected]>

* fix: deadlock in webhook controller when upgrading from old versions (#243)

* fix: close StageInterrupt trap + harden core e2e pool (#242)

* fix(test): harden core e2e pool against finalizer-cleanup races and improve diagnosability

Two changes across the six core-pool chainsaw tests:

1. Add `timeouts.cleanup: 120s` to every core-pool test. Chainsaw's
   default cleanup window (~30s) is too tight for the project's CR
   finalizer, which must uncordon nodes, remove labels/annotations,
   and GC package pods before releasing. Same failure mode that hit
   downgrade-after-uninstall (PR #241).

2. Add `catch:` blocks to `depends-on` and `simple-skyhook`, mirroring
   the pattern used by the other four core tests. When these tests
   flake, CI now captures the Node, Skyhook CR, and package Pods for
   diagnosis instead of failing without artifacts.

No assert logic changes — this is a foundation pass to remove a known
flake class and make future flakes diagnosable.

Signed-off-by: Alex Yuskauskas <[email protected]>

* fix(operator): close StageInterrupt trap when configUpdates signal decays

A package whose only interrupt is a `configInterrupts` entry (no top-level
`interrupt` block) could get stuck at `StageInterrupt/StateSkipped`
permanently when `Status.ConfigUpdates` cleared or never persisted (e.g.
due to a 409 on the spec patch). The state machine queried the dynamic
`HasInterrupt(config)` signal at four points, and once that signal
decayed to false, the package was untouchable: `ProgressSkipped`
wouldn't promote it, `NextStage` wouldn't advance it past Interrupt,
and `GetComplete`/`IsPackageComplete` wouldn't count it complete.

Decouple progression-past-Interrupt from the dynamic signal:

- `NextStage`: add `StageInterrupt → StagePostInterrupt` to the
  no-interrupt default map. The with-interrupt full-replacement map is
  preserved as-is — PR #200 deliberately omits `StageUninstall →
  StageApply` from it so with-interrupt uninstalls route via
  `StageUninstallInterrupt`; collapsing the maps would silently
  re-enable that transition.
- `ProgressSkipped`: drop the `HasInterrupt` gate. The only writer of
  `StateSkipped` at `StageInterrupt` is `ProcessInterrupt`'s
  budget-contention branch, which already decided the package needed
  an interrupt to schedule. Stage alone is sufficient.
- `GetComplete`, `IsPackageComplete`: treat `StagePostInterrupt` as
  unconditionally terminal. The only way to reach it is via the
  interrupt cycle; gating the terminal check on a signal that can
  decay is redundant with the entry gate and only becomes load-bearing
  when the trap fires.

Reproducer in `skyhook_types_test.go` exercises all three sites.

Surfaced by the `chainsaw/config-skyhook` "update while running" step,
which patches the spec mid-flight and concurrently triggers the
`HandleMigrations` and `processSkyhooksPerNode` 409 conflict paths —
those stretch convergence enough that ConfigUpdates can be lost
between the package reaching StageInterrupt and ProgressSkipped firing.

Signed-off-by: Alex Yuskauskas <[email protected]>

* refactor(operator): GetComplete delegates to IsPackageComplete

GetComplete and IsPackageComplete encoded the same terminal-state logic
twice. Have GetComplete iterate the spec packages and call
IsPackageComplete for each, and move the explanatory comment to
IsPackageComplete where the predicate lives.

Behavior preserved: the prior implementation iterated node state and
filtered to spec packages by name+version match; the new one iterates
spec packages and looks them up in node state by unique name (name|version).
Same set of packages either way.

Signed-off-by: Alex Yuskauskas <[email protected]>

---------

Signed-off-by: Alex Yuskauskas <[email protected]>

---------

Signed-off-by: AnouarMohamed <[email protected]>
Signed-off-by: Alex Yuskauskas <[email protected]>
Co-authored-by: Anouar Mohamed <[email protected]>
Co-authored-by: ayuskauskas <[email protected]>
rice-riley pushed a commit that referenced this pull request Jun 9, 2026
* ci: publish keyless release attestations (#232) (#240)

* ci: publish keyless release attestations

* ci: drop nvcr release attestations

* ci: narrow release attestation identity

* ci: gate attestations to release tags

* docs: show immutable release subjects

* ci: attest helm chart releases

---------

Signed-off-by: AnouarMohamed <[email protected]>
Co-authored-by: Anouar Mohamed <[email protected]>

* chore(docs): update docs around release and location of helm chart (#237) (#239)

* fix(test): uninstall downgraded package before cleanup in downgrade-after-uninstall (#241)

The test left [email protected] freshly installed at chainsaw's automatic
CLEANUP, where the new delete-time uninstall finalizer (from #200) must
run uninstall pods on every node before releasing the CR. That exceeds
chainsaw's default cleanup window, causing "context deadline exceeded".

Add a final uninstall-v1 step that flips uninstall.apply=true on the
downgraded version and waits for nodeState to empty, so the CR has no
installed packages when cleanup deletes it.

Signed-off-by: Alex Yuskauskas <[email protected]>

* fix: deadlock in webhook controller when upgrading from old versions (#243)

* fix: close StageInterrupt trap + harden core e2e pool (#242)

* fix(test): harden core e2e pool against finalizer-cleanup races and improve diagnosability

Two changes across the six core-pool chainsaw tests:

1. Add `timeouts.cleanup: 120s` to every core-pool test. Chainsaw's
   default cleanup window (~30s) is too tight for the project's CR
   finalizer, which must uncordon nodes, remove labels/annotations,
   and GC package pods before releasing. Same failure mode that hit
   downgrade-after-uninstall (PR #241).

2. Add `catch:` blocks to `depends-on` and `simple-skyhook`, mirroring
   the pattern used by the other four core tests. When these tests
   flake, CI now captures the Node, Skyhook CR, and package Pods for
   diagnosis instead of failing without artifacts.

No assert logic changes — this is a foundation pass to remove a known
flake class and make future flakes diagnosable.

Signed-off-by: Alex Yuskauskas <[email protected]>

* fix(operator): close StageInterrupt trap when configUpdates signal decays

A package whose only interrupt is a `configInterrupts` entry (no top-level
`interrupt` block) could get stuck at `StageInterrupt/StateSkipped`
permanently when `Status.ConfigUpdates` cleared or never persisted (e.g.
due to a 409 on the spec patch). The state machine queried the dynamic
`HasInterrupt(config)` signal at four points, and once that signal
decayed to false, the package was untouchable: `ProgressSkipped`
wouldn't promote it, `NextStage` wouldn't advance it past Interrupt,
and `GetComplete`/`IsPackageComplete` wouldn't count it complete.

Decouple progression-past-Interrupt from the dynamic signal:

- `NextStage`: add `StageInterrupt → StagePostInterrupt` to the
  no-interrupt default map. The with-interrupt full-replacement map is
  preserved as-is — PR #200 deliberately omits `StageUninstall →
  StageApply` from it so with-interrupt uninstalls route via
  `StageUninstallInterrupt`; collapsing the maps would silently
  re-enable that transition.
- `ProgressSkipped`: drop the `HasInterrupt` gate. The only writer of
  `StateSkipped` at `StageInterrupt` is `ProcessInterrupt`'s
  budget-contention branch, which already decided the package needed
  an interrupt to schedule. Stage alone is sufficient.
- `GetComplete`, `IsPackageComplete`: treat `StagePostInterrupt` as
  unconditionally terminal. The only way to reach it is via the
  interrupt cycle; gating the terminal check on a signal that can
  decay is redundant with the entry gate and only becomes load-bearing
  when the trap fires.

Reproducer in `skyhook_types_test.go` exercises all three sites.

Surfaced by the `chainsaw/config-skyhook` "update while running" step,
which patches the spec mid-flight and concurrently triggers the
`HandleMigrations` and `processSkyhooksPerNode` 409 conflict paths —
those stretch convergence enough that ConfigUpdates can be lost
between the package reaching StageInterrupt and ProgressSkipped firing.

Signed-off-by: Alex Yuskauskas <[email protected]>

* refactor(operator): GetComplete delegates to IsPackageComplete

GetComplete and IsPackageComplete encoded the same terminal-state logic
twice. Have GetComplete iterate the spec packages and call
IsPackageComplete for each, and move the explanatory comment to
IsPackageComplete where the predicate lives.

Behavior preserved: the prior implementation iterated node state and
filtered to spec packages by name+version match; the new one iterates
spec packages and looks them up in node state by unique name (name|version).
Same set of packages either way.

Signed-off-by: Alex Yuskauskas <[email protected]>

---------

Signed-off-by: Alex Yuskauskas <[email protected]>

---------

Signed-off-by: AnouarMohamed <[email protected]>
Signed-off-by: Alex Yuskauskas <[email protected]>
Co-authored-by: Anouar Mohamed <[email protected]>
Co-authored-by: ayuskauskas <[email protected]>
rice-riley pushed a commit that referenced this pull request Jun 9, 2026
* ci: publish keyless release attestations (#232) (#240)

* ci: publish keyless release attestations

* ci: drop nvcr release attestations

* ci: narrow release attestation identity

* ci: gate attestations to release tags

* docs: show immutable release subjects

* ci: attest helm chart releases

---------

Signed-off-by: AnouarMohamed <[email protected]>
Co-authored-by: Anouar Mohamed <[email protected]>

* chore(docs): update docs around release and location of helm chart (#237) (#239)

* fix(test): uninstall downgraded package before cleanup in downgrade-after-uninstall (#241)

The test left [email protected] freshly installed at chainsaw's automatic
CLEANUP, where the new delete-time uninstall finalizer (from #200) must
run uninstall pods on every node before releasing the CR. That exceeds
chainsaw's default cleanup window, causing "context deadline exceeded".

Add a final uninstall-v1 step that flips uninstall.apply=true on the
downgraded version and waits for nodeState to empty, so the CR has no
installed packages when cleanup deletes it.

Signed-off-by: Alex Yuskauskas <[email protected]>

* fix: deadlock in webhook controller when upgrading from old versions (#243)

* fix: close StageInterrupt trap + harden core e2e pool (#242)

* fix(test): harden core e2e pool against finalizer-cleanup races and improve diagnosability

Two changes across the six core-pool chainsaw tests:

1. Add `timeouts.cleanup: 120s` to every core-pool test. Chainsaw's
   default cleanup window (~30s) is too tight for the project's CR
   finalizer, which must uncordon nodes, remove labels/annotations,
   and GC package pods before releasing. Same failure mode that hit
   downgrade-after-uninstall (PR #241).

2. Add `catch:` blocks to `depends-on` and `simple-skyhook`, mirroring
   the pattern used by the other four core tests. When these tests
   flake, CI now captures the Node, Skyhook CR, and package Pods for
   diagnosis instead of failing without artifacts.

No assert logic changes — this is a foundation pass to remove a known
flake class and make future flakes diagnosable.

Signed-off-by: Alex Yuskauskas <[email protected]>

* fix(operator): close StageInterrupt trap when configUpdates signal decays

A package whose only interrupt is a `configInterrupts` entry (no top-level
`interrupt` block) could get stuck at `StageInterrupt/StateSkipped`
permanently when `Status.ConfigUpdates` cleared or never persisted (e.g.
due to a 409 on the spec patch). The state machine queried the dynamic
`HasInterrupt(config)` signal at four points, and once that signal
decayed to false, the package was untouchable: `ProgressSkipped`
wouldn't promote it, `NextStage` wouldn't advance it past Interrupt,
and `GetComplete`/`IsPackageComplete` wouldn't count it complete.

Decouple progression-past-Interrupt from the dynamic signal:

- `NextStage`: add `StageInterrupt → StagePostInterrupt` to the
  no-interrupt default map. The with-interrupt full-replacement map is
  preserved as-is — PR #200 deliberately omits `StageUninstall →
  StageApply` from it so with-interrupt uninstalls route via
  `StageUninstallInterrupt`; collapsing the maps would silently
  re-enable that transition.
- `ProgressSkipped`: drop the `HasInterrupt` gate. The only writer of
  `StateSkipped` at `StageInterrupt` is `ProcessInterrupt`'s
  budget-contention branch, which already decided the package needed
  an interrupt to schedule. Stage alone is sufficient.
- `GetComplete`, `IsPackageComplete`: treat `StagePostInterrupt` as
  unconditionally terminal. The only way to reach it is via the
  interrupt cycle; gating the terminal check on a signal that can
  decay is redundant with the entry gate and only becomes load-bearing
  when the trap fires.

Reproducer in `skyhook_types_test.go` exercises all three sites.

Surfaced by the `chainsaw/config-skyhook` "update while running" step,
which patches the spec mid-flight and concurrently triggers the
`HandleMigrations` and `processSkyhooksPerNode` 409 conflict paths —
those stretch convergence enough that ConfigUpdates can be lost
between the package reaching StageInterrupt and ProgressSkipped firing.

Signed-off-by: Alex Yuskauskas <[email protected]>

* refactor(operator): GetComplete delegates to IsPackageComplete

GetComplete and IsPackageComplete encoded the same terminal-state logic
twice. Have GetComplete iterate the spec packages and call
IsPackageComplete for each, and move the explanatory comment to
IsPackageComplete where the predicate lives.

Behavior preserved: the prior implementation iterated node state and
filtered to spec packages by name+version match; the new one iterates
spec packages and looks them up in node state by unique name (name|version).
Same set of packages either way.

Signed-off-by: Alex Yuskauskas <[email protected]>

---------

Signed-off-by: Alex Yuskauskas <[email protected]>

---------

Signed-off-by: AnouarMohamed <[email protected]>
Signed-off-by: Alex Yuskauskas <[email protected]>
Co-authored-by: Anouar Mohamed <[email protected]>
Co-authored-by: ayuskauskas <[email protected]>
rice-riley added a commit that referenced this pull request Jun 9, 2026
* feat(agent): port enums and steps for agent go rewrite

Signed-off-by: Riley Rice <[email protected]>

* chore: Cherry pick (#244)

* ci: publish keyless release attestations (#232) (#240)

* ci: publish keyless release attestations

* ci: drop nvcr release attestations

* ci: narrow release attestation identity

* ci: gate attestations to release tags

* docs: show immutable release subjects

* ci: attest helm chart releases

---------

Signed-off-by: AnouarMohamed <[email protected]>
Co-authored-by: Anouar Mohamed <[email protected]>

* chore(docs): update docs around release and location of helm chart (#237) (#239)

* fix(test): uninstall downgraded package before cleanup in downgrade-after-uninstall (#241)

The test left [email protected] freshly installed at chainsaw's automatic
CLEANUP, where the new delete-time uninstall finalizer (from #200) must
run uninstall pods on every node before releasing the CR. That exceeds
chainsaw's default cleanup window, causing "context deadline exceeded".

Add a final uninstall-v1 step that flips uninstall.apply=true on the
downgraded version and waits for nodeState to empty, so the CR has no
installed packages when cleanup deletes it.

Signed-off-by: Alex Yuskauskas <[email protected]>

* fix: deadlock in webhook controller when upgrading from old versions (#243)

* fix: close StageInterrupt trap + harden core e2e pool (#242)

* fix(test): harden core e2e pool against finalizer-cleanup races and improve diagnosability

Two changes across the six core-pool chainsaw tests:

1. Add `timeouts.cleanup: 120s` to every core-pool test. Chainsaw's
   default cleanup window (~30s) is too tight for the project's CR
   finalizer, which must uncordon nodes, remove labels/annotations,
   and GC package pods before releasing. Same failure mode that hit
   downgrade-after-uninstall (PR #241).

2. Add `catch:` blocks to `depends-on` and `simple-skyhook`, mirroring
   the pattern used by the other four core tests. When these tests
   flake, CI now captures the Node, Skyhook CR, and package Pods for
   diagnosis instead of failing without artifacts.

No assert logic changes — this is a foundation pass to remove a known
flake class and make future flakes diagnosable.

Signed-off-by: Alex Yuskauskas <[email protected]>

* fix(operator): close StageInterrupt trap when configUpdates signal decays

A package whose only interrupt is a `configInterrupts` entry (no top-level
`interrupt` block) could get stuck at `StageInterrupt/StateSkipped`
permanently when `Status.ConfigUpdates` cleared or never persisted (e.g.
due to a 409 on the spec patch). The state machine queried the dynamic
`HasInterrupt(config)` signal at four points, and once that signal
decayed to false, the package was untouchable: `ProgressSkipped`
wouldn't promote it, `NextStage` wouldn't advance it past Interrupt,
and `GetComplete`/`IsPackageComplete` wouldn't count it complete.

Decouple progression-past-Interrupt from the dynamic signal:

- `NextStage`: add `StageInterrupt → StagePostInterrupt` to the
  no-interrupt default map. The with-interrupt full-replacement map is
  preserved as-is — PR #200 deliberately omits `StageUninstall →
  StageApply` from it so with-interrupt uninstalls route via
  `StageUninstallInterrupt`; collapsing the maps would silently
  re-enable that transition.
- `ProgressSkipped`: drop the `HasInterrupt` gate. The only writer of
  `StateSkipped` at `StageInterrupt` is `ProcessInterrupt`'s
  budget-contention branch, which already decided the package needed
  an interrupt to schedule. Stage alone is sufficient.
- `GetComplete`, `IsPackageComplete`: treat `StagePostInterrupt` as
  unconditionally terminal. The only way to reach it is via the
  interrupt cycle; gating the terminal check on a signal that can
  decay is redundant with the entry gate and only becomes load-bearing
  when the trap fires.

Reproducer in `skyhook_types_test.go` exercises all three sites.

Surfaced by the `chainsaw/config-skyhook` "update while running" step,
which patches the spec mid-flight and concurrently triggers the
`HandleMigrations` and `processSkyhooksPerNode` 409 conflict paths —
those stretch convergence enough that ConfigUpdates can be lost
between the package reaching StageInterrupt and ProgressSkipped firing.

Signed-off-by: Alex Yuskauskas <[email protected]>

* refactor(operator): GetComplete delegates to IsPackageComplete

GetComplete and IsPackageComplete encoded the same terminal-state logic
twice. Have GetComplete iterate the spec packages and call
IsPackageComplete for each, and move the explanatory comment to
IsPackageComplete where the predicate lives.

Behavior preserved: the prior implementation iterated node state and
filtered to spec packages by name+version match; the new one iterates
spec packages and looks them up in node state by unique name (name|version).
Same set of packages either way.

Signed-off-by: Alex Yuskauskas <[email protected]>

---------

Signed-off-by: Alex Yuskauskas <[email protected]>

---------

Signed-off-by: AnouarMohamed <[email protected]>
Signed-off-by: Alex Yuskauskas <[email protected]>
Co-authored-by: Anouar Mohamed <[email protected]>
Co-authored-by: ayuskauskas <[email protected]>

---------

Signed-off-by: Riley Rice <[email protected]>
Signed-off-by: AnouarMohamed <[email protected]>
Signed-off-by: Alex Yuskauskas <[email protected]>
Co-authored-by: Brian Lockwood <[email protected]>
Co-authored-by: Anouar Mohamed <[email protected]>
Co-authored-by: ayuskauskas <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Publish keyless cosign signatures, SLSA provenance, and SBOM attestations for releases

2 participants