Skip to content

feat(tests): KMS bundle-signing e2e against MiniStack over TLS#1298

Merged
lockwobr merged 2 commits into
mainfrom
feat/kms-ministack-e2e
Jun 11, 2026
Merged

feat(tests): KMS bundle-signing e2e against MiniStack over TLS#1298
lockwobr merged 2 commits into
mainfrom
feat/kms-ministack-e2e

Conversation

@lockwobr

Copy link
Copy Markdown
Contributor

Summary

Adds an end-to-end test that exercises the real awskms:// provider path for
aicr bundle --attest --signing-key and aicr verify, against an emulated AWS
KMS (MiniStack) so the full provider-resolution, sign, and verify round-trip
runs in CI with no real AWS credentials and no paid license.

Motivation / Context

PR #1205 added --signing-key <kms-uri> to aicr bundle --attest, but its unit
tests stub the KMS at the kmsSigner seam, so the real provider path
(kms.Get("awskms://...") -> SignerVerifier.PublicKey -> SignMessage) was
never exercised automatically. This closes that gap.

Fixes: N/A
Related: #1205
Closes: #1214

Type of Change

  • New feature (non-breaking change that adds functionality)
  • Build/CI/tooling

Component(s) Affected

  • Other: e2e test (tests/chainsaw/cli/bundle-attestation-kms-ministack), CI workflow, tooling

Implementation Notes

  • MiniStack instead of LocalStack. LocalStack's latest/stable images now
    refuse to boot without a paid license token (exit 55). MiniStack
    (https://ministack.org) is MIT-licensed and token-free; its KMS supports the
    asymmetric ECDSA P-256 SIGN_VERIFY keys cosign's awskms:// signer needs.
    Pinned in .settings.yaml (never :latest).
  • TLS via mkcert. sigstore's awskms client hardcodes https://, so
    MiniStack runs with USE_SSL=1 behind a mkcert localhost cert. Trust spans
    three planes: Go/aicr via the system store (mkcert CA), the aws CLI via
    AWS_CA_BUNDLE, and binary-attestation verification against the untouched
    public Sigstore root. mkcert is added as a pinned tool.
  • Binary attestation identity. The bundle and verify steps pass
    --certificate-identity-regexp so a binary attested by an e2e/build-attested
    workflow (not on-tag) is accepted.
  • The CI workflow builds and SLSA-attests its own binary, then runs the suite;
    run.sh reproduces the same flow locally.

Testing

# Local end-to-end run against MiniStack over TLS with a build-attested binary
AICR_BIN=<attested> AICR_ATTESTED=true \
AICR_IDENTITY_REGEXP='https://github.com/NVIDIA/aicr/.github/workflows/build-attested\.yaml@.*' \
  ./tests/chainsaw/cli/bundle-attestation-kms-ministack/run.sh
  • Full chainsaw suite passes: Passed 1, Failed 0 (KMS MiniStack E2E: PASSED),
    including tamper-detection and wrong-key-rejection negative steps.
  • yamllint -c .yamllint.yaml clean; bash -n clean on run.sh,
    tools/setup-tools, tools/check-tools.
  • No Go source changed, so the Go lint and coverage gates do not apply.

Risk Assessment

  • Low - New test plus tooling; no production code paths changed; CI-gated.

Rollout notes: N/A. The new workflow triggers on push to main and
workflow_dispatch.

Checklist

  • Linter passes (yamllint)
  • I did not skip/disable tests to make CI green
  • I added tests for new functionality (this PR is the test)
  • I updated docs (README for the test directory)
  • Changes follow existing patterns in the codebase
  • Commits are cryptographically signed (git commit -S)

@lockwobr lockwobr requested review from a team as code owners June 10, 2026 21:13
@lockwobr lockwobr self-assigned this Jun 10, 2026
@lockwobr lockwobr added the theme/ci-dx CI pipelines, developer experience, and build tooling label Jun 10, 2026
@lockwobr lockwobr force-pushed the feat/kms-ministack-e2e branch from 78526cc to 65d44fa Compare June 10, 2026 21:21
@coderabbitai

coderabbitai Bot commented Jun 10, 2026

Copy link
Copy Markdown

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

This PR introduces end-to-end testing infrastructure for KMS-backed bundle signing using MiniStack (AWS emulator). It pins awscli, ministack_image, and mkcert (with checksums) in .settings.yaml, extends the load-versions action to export those values, adds mkcert detection and installation to tooling scripts, updates cli-e2e to include signing tests with selectors, adds a kms-ministack-e2e CI workflow that provisions MiniStack over TLS, creates an ECDSA P-256 KMS key, builds/selects aicr, exports the KMS public key, and runs Chainsaw tests. It also adds a run.sh harness, a Chainsaw test spec for signing/verification/tamper/wrong-key cases, and a README documenting local/CI usage and trust handling.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title 'feat(tests): KMS bundle-signing e2e against MiniStack over TLS' clearly and specifically summarizes the main change: adding an end-to-end test for KMS-backed bundle signing using MiniStack.
Description check ✅ Passed The description comprehensively explains the PR's purpose (e2e KMS test via MiniStack), motivation (gap in PR #1205 coverage), implementation notes (MiniStack vs LocalStack, TLS via mkcert, binary attestation), and testing results.
Linked Issues check ✅ Passed The PR fully addresses issue #1214's phase-1 AWS emulator acceptance criteria: uses MiniStack (an AWS KMS emulator), provisions ECDSA P-256 signing keys, runs bundle signing and verification against the emulator endpoint with custom TLS setup, asserts output bundles contain public-key verification material without Fulcio certs, handles verification with the emulator's public key, includes tamper detection and wrong-key rejection tests, runs deterministically without cloud credentials, and documents local run instructions.
Out of Scope Changes check ✅ Passed All changes are in-scope: new e2e test (bundle-attestation-kms-ministack), CI workflow (kms-ministack-e2e.yaml), tool configuration (.settings.yaml), tooling updates (setup-tools, check-tools), and documentation. No production code or unrelated changes are present.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/kms-ministack-e2e

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/kms-ministack-e2e.yaml:
- Around line 41-43: Add an inline explanatory comment next to the permissions
block explaining why id-token: write is required: state that id-token: write
enables GitHub Actions OIDC tokens used for Sigstore keyless OIDC signing which
the workflow uses to produce SLSA provenance, and reference the
provenance-generating step or job by name so reviewers can easily correlate the
permission to the Sigstore signing step.

In `@tools/setup-tools`:
- Around line 499-519: Add SHA256 checksum verification for the mkcert binary
download similar to helmfile/chainsaw: add entries under
testing_tools.mkcert_checksums in .settings.yaml (keys like linux_amd64 and
linux_arm64), compute expected checksum lookup using MKCERT_ARCH mapping,
download the MKCERT_URL to a temp file (instead of streaming to /usr/local/bin),
verify the file's sha256sum against the expected checksum, only then move it to
/usr/local/bin and chmod +x; update the install block that constructs
MKCERT_URL, the download step and the final install step to perform these
verification checks and fail with a clear log message if the checksum does not
match.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: b87e50ea-b099-44c0-8c79-ea71c7de526b

📥 Commits

Reviewing files that changed from the base of the PR and between 4e8f778 and 78526cc.

📒 Files selected for processing (8)
  • .github/actions/load-versions/action.yml
  • .github/workflows/kms-ministack-e2e.yaml
  • .settings.yaml
  • tests/chainsaw/cli/bundle-attestation-kms-ministack/README.md
  • tests/chainsaw/cli/bundle-attestation-kms-ministack/chainsaw-test.yaml
  • tests/chainsaw/cli/bundle-attestation-kms-ministack/run.sh
  • tools/check-tools
  • tools/setup-tools

Comment thread .github/workflows/kms-ministack-e2e.yaml Outdated
Comment thread tools/setup-tools
@lockwobr lockwobr force-pushed the feat/kms-ministack-e2e branch from 65d44fa to 4c522f8 Compare June 10, 2026 21:25
@github-actions

github-actions Bot commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

Coverage Report ✅

Metric Value
Coverage 77.0%
Threshold 75%
Status Pass
Coverage Badge
![Coverage](https://img.shields.io/badge/coverage-77.0%25-green)

No Go source files changed in this PR.

@lockwobr lockwobr force-pushed the feat/kms-ministack-e2e branch from 4c522f8 to 8f1d1ab Compare June 10, 2026 21:44

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 7

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/kms-ministack-e2e.yaml:
- Around line 126-281: The workflow embeds heavy implementation (Install AWS
CLI, Install mkcert and provision TLS cert, Start MiniStack, Provision KMS key
in MiniStack, Export public key to PEM, Run KMS MiniStack chainsaw tests)
directly in inline steps instead of a reusable composite action; extract these
steps into a Layer-2 composite action that encapsulates AWS
CLI/mkcert/MiniStack/KMS setup and teardown and replace the inline blocks with
calls to that composite action, exposing inputs for MINISTACK_IMAGE,
MINISTACK_CERT_DIR, MINISTACK_ENDPOINT and outputs like
KMS_KEY_ARN/KMS_URI/KMS_PEM_FILE and an option to run the chainsaw test so the
workflow only orchestrates the composite action.

In @.settings.yaml:
- Around line 86-88: The pinned awscli entry in .settings.yaml is not being
wired into the local tool installers and validators; update tools/setup-tools to
install testing_tools.awscli (use the pinned version from .settings.yaml) and
update tools/check-tools to verify the aws binary and its version matches
testing_tools.awscli, so
tests/chainsaw/cli/bundle-attestation-kms-ministack/run.sh can rely on the
installed tool; reference the awscli key in .settings.yaml, the setup logic in
tools/setup-tools, and the validation logic in tools/check-tools when making the
changes.

In `@tests/chainsaw/cli/bundle-attestation-kms-ministack/chainsaw-test.yaml`:
- Around line 24-30: Update the default MiniStack endpoint scheme from http to
https by replacing the literal "http://localhost:4566" with
"https://localhost:4566" wherever it appears in this YAML (notably in the
MINISTACK_ENDPOINT default in the description and the other occurrences at lines
referenced); search for the string "http://localhost:4566" and change it to
"https://localhost:4566" so the harness, CI, and run.sh TLS usage matches the
documented default and any CLI fallbacks.
- Around line 140-144: The test currently only asserts that the keys
"checksumsPassed" and "bundleAttested" appear in stdout, which lets false values
pass; update the assertions to verify the actual boolean values emitted by the
verifier (ChecksumsPassed and BundleAttested) are true. Locate the CLI test
checks around the blocks referencing 'checksumsPassed' and 'bundleAttested'
(also the similar block at 185-188) and change them to assert the JSON value is
true (e.g., check for '"checksumsPassed": true' and '"bundleAttested": true' or
parse stdout as JSON and assert .ChecksumsPassed == true and .BundleAttested ==
true) so failed verification no longer falsely passes. Ensure you reference the
verifier fields ChecksumsPassed and BundleAttested from
pkg/bundler/verifier/verifier.go when updating the expectations.

In `@tests/chainsaw/cli/bundle-attestation-kms-ministack/README.md`:
- Around line 68-70: The fenced code block containing the example URI
awskms://<host:port>/<arn> is unlabeled and triggers MD040; add a language tag
(e.g., text) to the opening fence so it becomes ```text to satisfy linting and
ensure consistent rendering in docs (update the fenced block around the
awskms://<host:port>/<arn> example in README.md).
- Around line 139-149: Update the "Triggers on push to `main` and
`workflow_dispatch`" paragraph to also list the `pull_request` trigger (and
which branch patterns it matches) so the README accurately reflects the
workflow's current triggers; edit the sentence that begins "Triggers on push to
`main` and `workflow_dispatch`." to read the full set (e.g., push to main,
pull_request matching the same branches, and workflow_dispatch), and add a brief
clarifying note that this describes current behavior (not future intent) so
contributors know when CI will run.

In `@tests/chainsaw/cli/bundle-attestation-kms-ministack/run.sh`:
- Around line 237-278: In build_binary(), when using a pre-built executable (the
branch that checks [ -n "${AICR_BIN}" ] && [ -x "${AICR_BIN}" ]), export
AICR_ATTESTED=true so downstream tests (e.g., verify-min-trust-verified) run for
CI-attested artifacts; conversely, in the snapshot build path (the goreleaser
branch that builds an unattested binary) explicitly set/export
AICR_ATTESTED=false to ensure attestation state is clear. Reference: function
build_binary, variables AICR_BIN and AICR_ATTESTED.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: f526079a-d596-4026-a922-559887357f32

📥 Commits

Reviewing files that changed from the base of the PR and between 4c522f8 and 8f1d1ab.

📒 Files selected for processing (8)
  • .github/actions/load-versions/action.yml
  • .github/workflows/kms-ministack-e2e.yaml
  • .settings.yaml
  • tests/chainsaw/cli/bundle-attestation-kms-ministack/README.md
  • tests/chainsaw/cli/bundle-attestation-kms-ministack/chainsaw-test.yaml
  • tests/chainsaw/cli/bundle-attestation-kms-ministack/run.sh
  • tools/check-tools
  • tools/setup-tools

Comment thread .github/workflows/kms-ministack-e2e.yaml
Comment thread .settings.yaml
Comment thread tests/chainsaw/cli/bundle-attestation-kms-ministack/README.md Outdated
Comment thread tests/chainsaw/cli/bundle-attestation-kms-ministack/README.md Outdated
Comment thread tests/chainsaw/signing/bundle-attestation-kms-ministack/run.sh
@lockwobr lockwobr force-pushed the feat/kms-ministack-e2e branch from 8f1d1ab to aab418d Compare June 10, 2026 22:49

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 5

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
tools/e2e (1)

10-10: 🧹 Nitpick | 🔵 Trivial | 💤 Low value

Update comments to reflect all test directories.

The script now runs tests from three directories (cli/, signing/, bundle-templates/) but the comments on lines 10 and 99 still reference only two.

📝 Suggested fix
 # WHAT THIS SCRIPT DOES:
 # 1. Builds the aicr binary via make build
 # 2. Locates the platform-specific binary
-# 3. Runs chainsaw test --no-cluster on tests/chainsaw/cli/ and tests/chainsaw/bundle-templates/
+# 3. Runs chainsaw test --no-cluster on tests/chainsaw/cli/, signing/, and bundle-templates/
-msg "Running chainsaw tests (CLI + bundle template tests)"
+msg "Running chainsaw tests (CLI, signing, and bundle template tests)"

Also applies to: 99-99

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tools/e2e` at line 10, Update the top-of-file comment that currently lists
"cli/ and bundle-templates/" so it accurately reflects all three test
directories by including "signing/" as well; similarly update the other matching
comment later in the file (the comment around line 99) to list "cli/, signing/,
and bundle-templates/" so both comments match the actual test directories the
script runs.
♻️ Duplicate comments (1)
tests/chainsaw/signing/bundle-attestation-kms-ministack/chainsaw-test.yaml (1)

190-193: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Strengthen PEM-verify assertions to validate successful outcomes.

Line 192 currently asserts only field presence, so the step can pass even when verification reports failure values. Assert true values (and include bundleAttested) to avoid false-green E2E results.

Suggested patch
     - name: verify-with-pem-public-key
@@
-              "${AICR_BIN}" verify "${WORK}/bundle" \
+              OUTPUT=$("${AICR_BIN}" verify "${WORK}/bundle" \
                 --key "${PEM_FILE}" \
                 --certificate-identity-regexp "${AICR_IDENTITY_REGEXP:-}" \
-                --format json
+                --format json)
+              echo "${OUTPUT}"
+              echo "${OUTPUT}" | grep -q '"checksumsPassed": true'
+              echo "${OUTPUT}" | grep -q '"bundleAttested": true'
             check:
               ($error == null): true
-              (contains($stdout, '"checksumsPassed"')): true
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/chainsaw/signing/bundle-attestation-kms-ministack/chainsaw-test.yaml`
around lines 190 - 193, The test's PEM-verify step currently only checks
presence of fields; update the check block to assert successful boolean results
and include bundleAttested. Replace or augment the existing checks (($error ==
null) and contains($stdout, '"checksumsPassed"')) with explicit assertions that
checksumsPassed is true and verificationSucceeded is true (e.g., check
contains('"checksumsPassed": true') and contains('"verificationSucceeded":
true')) and also assert bundleAttested is present/true (e.g.,
contains('"bundleAttested": true') or contains('"bundleAttested"')). Ensure
these checks are in the same check mapping as the original to prevent
false-positive passes.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/kms-ministack-e2e.yaml:
- Around line 46-55: The pull_request.paths array in the workflow is missing the
local action path '.github/actions/setup-build-tools/**' which this workflow
uses (the local action referenced as setup-build-tools); add
'.github/actions/setup-build-tools/**' to the pull_request.paths list so changes
to that action will trigger the workflow, ensuring the workflow's
pull_request.paths entry and the referenced local action (setup-build-tools)
stay in sync.
- Around line 56-64: Move the id-token: write permission out of the
workflow-level permissions block and add it under the specific job's permissions
(the kms-ministack-e2e job) so OIDC issuance is scoped to that job; update the
permissions block named permissions (remove id-token: write from the top-level
permissions) and add id-token: write under jobs.kms-ministack-e2e.permissions to
ensure only the signing job can mint OIDC tokens.

In `@tests/chainsaw/signing/bundle-attestation-kms-ministack/README.md`:
- Around line 1-4: The README uses the acronym "E2E" without defining it; update
the first mention of "End-to-end test suite" to "End-to-end (E2E) test suite" so
the acronym is expanded on first use (e.g., in the line containing "End-to-end
test suite for KMS-backed bundle signing and verification (`aicr bundle --attest
--signing-key awskms://...`)"). Ensure any subsequent uses of "E2E" remain
unchanged.

In `@tests/chainsaw/signing/bundle-attestation-kms-ministack/run.sh`:
- Around line 67-76: The MINISTACK_IMAGE default can drift from .settings.yaml;
update the run.sh logic for setting MINISTACK_IMAGE so that if MINISTACK_IMAGE
is empty it parses .settings.yaml to read testing_tools.ministack_image and uses
that value (falling back to the current hardcoded
"ministackorg/ministack:1.3.61" only if the YAML key is missing or parsing
fails); implement this in the same script near the existing MINISTACK_IMAGE
assignment, using a lightweight YAML extraction (e.g., grep/sed or yq) and
ensure MINISTACK_IMAGE remains exported and compatible with existing
MINISTACK_ENDPOINT and setup_tls usage.

In `@tools/setup-tools`:
- Around line 528-543: Update the comment to state that aws will be reinstalled
when UPGRADE=true (so it’s not strictly "only installed when absent"), and after
the pip install in the aws installation block (which uses python3 -m pip install
--user "awscli==${AWSCLI_VERSION}"), add a PATH verification step: use
command_exists aws (or check for ${HOME}/.local/bin/aws and $(python3 -m site
--user-base)/bin) to confirm the installed aws is on PATH and if not, call
log_warning with instructions to add the user's local bin to PATH (or advise
manual install), and keep using existing helpers like command_exists,
prompt_continue, and variables AWSCLI_VERSION and UPGRADE to locate the change.

---

Outside diff comments:
In `@tools/e2e`:
- Line 10: Update the top-of-file comment that currently lists "cli/ and
bundle-templates/" so it accurately reflects all three test directories by
including "signing/" as well; similarly update the other matching comment later
in the file (the comment around line 99) to list "cli/, signing/, and
bundle-templates/" so both comments match the actual test directories the script
runs.

---

Duplicate comments:
In `@tests/chainsaw/signing/bundle-attestation-kms-ministack/chainsaw-test.yaml`:
- Around line 190-193: The test's PEM-verify step currently only checks presence
of fields; update the check block to assert successful boolean results and
include bundleAttested. Replace or augment the existing checks (($error == null)
and contains($stdout, '"checksumsPassed"')) with explicit assertions that
checksumsPassed is true and verificationSucceeded is true (e.g., check
contains('"checksumsPassed": true') and contains('"verificationSucceeded":
true')) and also assert bundleAttested is present/true (e.g.,
contains('"bundleAttested": true') or contains('"bundleAttested"')). Ensure
these checks are in the same check mapping as the original to prevent
false-positive passes.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: bb2ff1e2-0ac9-4882-bc8d-8054765beaa6

📥 Commits

Reviewing files that changed from the base of the PR and between 8f1d1ab and aab418d.

📒 Files selected for processing (13)
  • .github/actions/cli-e2e/action.yml
  • .github/actions/load-versions/action.yml
  • .github/workflows/kms-ministack-e2e.yaml
  • .settings.yaml
  • tests/chainsaw/signing/bundle-attestation-ci/chainsaw-test.yaml
  • tests/chainsaw/signing/bundle-attestation-kms-ministack/README.md
  • tests/chainsaw/signing/bundle-attestation-kms-ministack/chainsaw-test.yaml
  • tests/chainsaw/signing/bundle-attestation-kms-ministack/run.sh
  • tests/chainsaw/signing/bundle-attestation/chainsaw-test.yaml
  • tests/chainsaw/signing/bundle-headless-oidc-ci/chainsaw-test.yaml
  • tools/check-tools
  • tools/e2e
  • tools/setup-tools

Comment thread .github/workflows/kms-ministack-e2e.yaml
Comment thread .github/workflows/kms-ministack-e2e.yaml
Comment thread tests/chainsaw/signing/bundle-attestation-kms-ministack/README.md
Comment thread tests/chainsaw/signing/bundle-attestation-kms-ministack/run.sh
Comment thread tools/setup-tools
@lockwobr lockwobr force-pushed the feat/kms-ministack-e2e branch from aab418d to cdec678 Compare June 11, 2026 00:42

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/chainsaw/signing/bundle-attestation-kms-ministack/chainsaw-test.yaml`:
- Around line 185-192: The PEM verification assertion is only checking for the
existence of "checksumsPassed" rather than its value and omits "bundleAttested";
update the check block for the "${AICR_BIN} verify \"${WORK}/bundle\" --key
\"${PEM_FILE}\"" invocation so it asserts (contains($stdout, '"checksumsPassed":
true')): true and (contains($stdout, '"bundleAttested": true')): true (matching
the KMS verification pattern), ensuring both values are actually true.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 38790a21-b656-4988-89d0-1d5456d135a3

📥 Commits

Reviewing files that changed from the base of the PR and between aab418d and cdec678.

📒 Files selected for processing (13)
  • .github/actions/cli-e2e/action.yml
  • .github/actions/load-versions/action.yml
  • .github/workflows/kms-ministack-e2e.yaml
  • .settings.yaml
  • tests/chainsaw/signing/bundle-attestation-ci/chainsaw-test.yaml
  • tests/chainsaw/signing/bundle-attestation-kms-ministack/README.md
  • tests/chainsaw/signing/bundle-attestation-kms-ministack/chainsaw-test.yaml
  • tests/chainsaw/signing/bundle-attestation-kms-ministack/run.sh
  • tests/chainsaw/signing/bundle-attestation/chainsaw-test.yaml
  • tests/chainsaw/signing/bundle-headless-oidc-ci/chainsaw-test.yaml
  • tools/check-tools
  • tools/e2e
  • tools/setup-tools

Comment thread tests/chainsaw/signing/bundle-attestation-kms-ministack/chainsaw-test.yaml Outdated
@lockwobr lockwobr force-pushed the feat/kms-ministack-e2e branch from cdec678 to 5f0aae1 Compare June 11, 2026 16:14
@lockwobr lockwobr enabled auto-merge (squash) June 11, 2026 16:32

@njhensley njhensley left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Multi-persona council review, with a meta-review pass over the candidate findings. Requesting changes because the wrong-key negative test can currently pass even if the verifier accepts the wrong key.

General note: tests/chainsaw/README.md still documents only tests/chainsaw/cli/ and omits the new tests/chainsaw/signing/ layout. Since this PR moves the signing suites and updates runners to include that directory, please refresh the manual command/tree there too.

Hardening notes that are not blockers: consider hash-locking the AWS CLI install in the new id-token-enabled workflow, and consider digest-pinning the third-party MiniStack image if you want the reproducibility comment to cover image mutability as well.

Comment thread .github/workflows/kms-ministack-e2e.yaml
Comment thread tests/chainsaw/signing/bundle-attestation-kms-ministack/run.sh Outdated
Comment thread tests/chainsaw/signing/bundle-attestation-kms-ministack/chainsaw-test.yaml Outdated
Comment thread tests/chainsaw/signing/bundle-attestation-kms-ministack/run.sh
Adds an end-to-end test that exercises the real awskms:// provider path for
`aicr bundle --attest --signing-key` and `aicr verify` against an emulated AWS
KMS, closing the automated-test gap for PR #1205's KMS signing feature.

Key points:
- MiniStack (https://ministack.org), an MIT-licensed token-free AWS emulator,
  replaces LocalStack, whose latest/stable images now refuse to boot without a
  paid license token. The image is pinned in .settings.yaml (never :latest).
- sigstore's awskms client hardcodes https://, so MiniStack runs with USE_SSL=1
  behind a mkcert localhost cert. Trust is handled across three planes: Go/aicr
  via the system store (mkcert CA), the aws CLI via AWS_CA_BUNDLE, and binary
  attestation against the untouched public Sigstore root.
- The bundle and verify steps pass --certificate-identity-regexp so a binary
  attested by an e2e/build-attested workflow (not on-tag) is accepted.
- mkcert is added as a pinned tool (.settings.yaml, load-versions, setup-tools,
  check-tools).
- run.sh reproduces the CI flow locally; the chainsaw suite is gated by
  --selector 'requires=ministack'.

Closes #1214

@njhensley njhensley left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at 1773ae2f. All five points from my earlier review are addressed — verified against the current file contents, not just the reply claims:

  • [blocker] verify-wrong-key-rejected now captures the exit code (|| rc=$?), runs with --format json, and fails explicitly when rc == 0. The non-zero exit is the primary assertion; the bundleAttested: true grep is retained as a defense-in-depth guard. ✅
  • [major] PR path filter now includes pkg/cli/**, go.mod, and go.sum, so CLI and sigstore-dependency changes exercise this lane. ✅
  • [major] run.sh cleanup no longer rm -rfs the user-overridable MINISTACK_CERT_DIR; it removes only the generated cert.pem/key.pem and rmdirs the dir only if empty. ✅
  • [minor] Step description reworded — public-key signing, no Fulcio cert (not keyless), still logs to the public Sigstore Rekor by default. ✅
  • [minor] run.sh adds an explicit smoke mode (smoke_test/run_tests) that validates the MiniStack/KMS plumbing and exits 0 when no attested binary is present, instead of failing the full --attest suite. ✅

Resolving all five threads. LGTM — approving.

@lockwobr lockwobr merged commit e848e1f into main Jun 11, 2026
33 checks passed
@lockwobr lockwobr deleted the feat/kms-ministack-e2e branch June 11, 2026 22:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/ci area/tests size/XL theme/ci-dx CI pipelines, developer experience, and build tooling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Integration-test KMS-backed bundle signing via LocalStack (AWS)

2 participants