feat(tests): private-Sigstore bundle-signing e2e via Helm scaffold#1321
Conversation
16cef8f to
90dd801
Compare
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds a private-Sigstore end-to-end test: pins and exposes a scaffold Helm chart version via the load-versions composite action and .settings.yaml, a new GitHub Actions workflow that builds/artifact-checks and invokes a shared E2E harness, Chainsaw test definition and README, Helm override values and RBAC manifest for OIDC discovery, a Bash orchestration script to provision Kind and run the test, and a small TLS-terminating reverse proxy (Go) plus .gitignore entry. Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Suggested labels
Suggested reviewers
🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/sigstore-scaffolding-e2e.yaml:
- Around line 133-137: The workflow step consumes outputs
steps.versions.outputs.mkcert and
steps.versions.outputs.mkcert_sha256_linux_amd64 that are not defined; add those
outputs to the version-loading action and ensure the source settings include
corresponding keys (e.g., mkcert and mkcert_sha256_linux_amd64) so they
populate, or alternatively hardcode/derive MKCERT_VERSION and MKCERT_SHA256 in
the workflow; update the load-versions action (declare outputs with those names)
and the settings data source (add mkcert and mkcert_sha256_linux_amd64 entries)
so the Install mkcert step receives non-empty values.
In
`@tests/chainsaw/signing/bundle-attestation-private-sigstore/chainsaw-test.yaml`:
- Around line 128-135: Replace brittle grep string checks in the test that
inspect "${BUNDLE}" with semantic JSON assertions using jq: instead of grep for
"verificationMaterial", "certificate", and "rawBytes" validate that
.verificationMaterial.certificate.rawBytes exists and is a non-empty string,
assert that .verificationMaterial.publicKey is absent for the keyless path, and
for checksum results assert .checksumsPassed is the boolean true (not a
whitespace-sensitive string match). Locate usages of the BUNDLE variable and the
checks around verificationMaterial.certificate.rawBytes and checksumsPassed in
the chainsaw-test.yaml and change them to jq-based queries that fail the test if
the field is missing, empty, or not the expected type/value.
In `@tests/chainsaw/signing/bundle-attestation-private-sigstore/run.sh`:
- Around line 158-167: The wait_url function can hang because curl calls lack
per-request timeouts; update the curl invocations inside wait_url to include
connection and total time limits (e.g., --connect-timeout 5 and --max-time 10)
so each poll attempt is bounded, and use the same timeout flags when capturing
the final failing output (the last="$(curl ... || true)") so the error message
reflects a timed-out attempt rather than a stuck TCP connect; modify the curl
usages within wait_url (the loop check and the final diagnostic curl)
accordingly.
In `@tests/chainsaw/signing/bundle-attestation-private-sigstore/tlsproxy/main.go`:
- Around line 100-107: The server is currently bound to all interfaces via Addr:
":" + listen in the srv := &http.Server {...} block; change Addr to bind to
loopback only (e.g. "127.0.0.1:"+listen or use net.JoinHostPort("127.0.0.1",
listen)) so the TLS proxy only listens on localhost. Update the same pattern for
the other server instance(s) referenced in the file (the similar Addr
assignments around lines 112-113) to ensure both proxies are loopback-only;
leave all other fields (Handler, TLSConfig, ReadHeaderTimeout) unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 29084d79-93ce-4548-84f6-5a3b93bf4378
📒 Files selected for processing (10)
.github/actions/load-versions/action.yml.github/workflows/sigstore-scaffolding-e2e.yaml.gitignore.settings.yamltests/chainsaw/signing/bundle-attestation-private-sigstore/README.mdtests/chainsaw/signing/bundle-attestation-private-sigstore/chainsaw-test.yamltests/chainsaw/signing/bundle-attestation-private-sigstore/oidc-discovery-rbac.yamltests/chainsaw/signing/bundle-attestation-private-sigstore/run.shtests/chainsaw/signing/bundle-attestation-private-sigstore/scaffold-values.yamltests/chainsaw/signing/bundle-attestation-private-sigstore/tlsproxy/main.go
Coverage Report ✅
Coverage BadgeCoverage unchanged by this PR. |
There was a problem hiding this comment.
Actionable comments posted: 2
♻️ Duplicate comments (2)
tests/chainsaw/signing/bundle-attestation-private-sigstore/tlsproxy/main.go (1)
100-113:⚠️ Potential issue | 🟠 Major | ⚡ Quick winBind the TLS proxy to loopback only.
Line 101 currently listens on all interfaces (
:port), while this helper is explicitly localhost-only. That unnecessarily exposes the proxy outside the host.Suggested fix
- Addr: ":" + listen, + Addr: "127.0.0.1:" + listen,🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/chainsaw/signing/bundle-attestation-private-sigstore/tlsproxy/main.go` around lines 100 - 113, The server is currently bound to all interfaces via Addr ":" + listen in the srv http.Server, which exposes the test TLS proxy externally; change the binding to loopback-only (e.g. "127.0.0.1:"+listen or "localhost:"+listen) in the srv initialization where Addr is set (and update the log message using the same listen binding if needed) so the proxy only listens on the local host; update references to listen/targetURL in that block (srv, listen, targetURL, and the slog.Info call) to reflect the loopback binding.tests/chainsaw/signing/bundle-attestation-private-sigstore/run.sh (1)
156-167:⚠️ Potential issue | 🟠 Major | ⚡ Quick win
wait_urlis not time-bounded per curl attempt.Line 161 and Line 166 call
curlwithout--connect-timeout/--max-time; one hung request can exceed the intended 90s window and stall the harness.Suggested fix
- if curl -fsS -o /dev/null "${url}" 2>/dev/null; then + if curl --connect-timeout 2 --max-time 4 -fsS -o /dev/null "${url}" 2>/dev/null; then return 0 fi @@ - last="$(curl -sS -o /dev/null "${url}" 2>&1 || true)" + last="$(curl --connect-timeout 2 --max-time 4 -sS -o /dev/null "${url}" 2>&1 || true)"🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/chainsaw/signing/bundle-attestation-private-sigstore/run.sh` around lines 156 - 167, The wait_url function uses curl without per-attempt timeouts, so a single hung request can exceed the intended ~90s polling window; update the curl invocations inside wait_url (both the success-check curl and the final diagnostic curl stored in last) to include sensible timeouts (e.g., --connect-timeout and --max-time) so each attempt is bounded, and ensure the diagnostic curl uses the same timeouts when capturing the last error; adjust the timeout values to keep total polling duration ≈90s given 45 attempts with 2s sleeps.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/sigstore-scaffolding-e2e.yaml:
- Around line 133-145: The inline shell step "Install mkcert" (and similarly the
steps "Build aicr binary", "Locate binary and detect attestation", and "Run
private-sigstore e2e") must either be moved into a Layer-2 composite action or
be annotated with a one-line rationale above the run: block explaining why the
inline implementation is intentional (e.g., "inline by design / single-use: this
step contains short, non-reusable orchestration specific to this workflow and
extracting a composite would add indirection without reuse"). Update each
retained inline run to include that exact brief justification comment or
refactor the logic into a composite action to comply with the three-layer
composite actions architecture.
In `@tests/chainsaw/signing/bundle-attestation-private-sigstore/README.md`:
- Around line 3-52: The README uses the acronyms "OIDC" and "SA" without
defining them; update README.md to define these on first use (replace "OIDC"
with "OpenID Connect (OIDC)" and "SA" with "ServiceAccount (SA)" where they
first appear) and/or add a short "Terminology" section near the top defining
OIDC and SA (and optionally Fulcio and Rekor) so readers see full terms before
acronyms are reused.
---
Duplicate comments:
In `@tests/chainsaw/signing/bundle-attestation-private-sigstore/run.sh`:
- Around line 156-167: The wait_url function uses curl without per-attempt
timeouts, so a single hung request can exceed the intended ~90s polling window;
update the curl invocations inside wait_url (both the success-check curl and the
final diagnostic curl stored in last) to include sensible timeouts (e.g.,
--connect-timeout and --max-time) so each attempt is bounded, and ensure the
diagnostic curl uses the same timeouts when capturing the last error; adjust the
timeout values to keep total polling duration ≈90s given 45 attempts with 2s
sleeps.
In `@tests/chainsaw/signing/bundle-attestation-private-sigstore/tlsproxy/main.go`:
- Around line 100-113: The server is currently bound to all interfaces via Addr
":" + listen in the srv http.Server, which exposes the test TLS proxy
externally; change the binding to loopback-only (e.g. "127.0.0.1:"+listen or
"localhost:"+listen) in the srv initialization where Addr is set (and update the
log message using the same listen binding if needed) so the proxy only listens
on the local host; update references to listen/targetURL in that block (srv,
listen, targetURL, and the slog.Info call) to reflect the loopback binding.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 40b48d1e-163f-476d-aedb-78892157f049
📒 Files selected for processing (10)
.github/actions/load-versions/action.yml.github/workflows/sigstore-scaffolding-e2e.yaml.gitignore.settings.yamltests/chainsaw/signing/bundle-attestation-private-sigstore/README.mdtests/chainsaw/signing/bundle-attestation-private-sigstore/chainsaw-test.yamltests/chainsaw/signing/bundle-attestation-private-sigstore/oidc-discovery-rbac.yamltests/chainsaw/signing/bundle-attestation-private-sigstore/run.shtests/chainsaw/signing/bundle-attestation-private-sigstore/scaffold-values.yamltests/chainsaw/signing/bundle-attestation-private-sigstore/tlsproxy/main.go
90dd801 to
e979436
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@tests/chainsaw/signing/bundle-attestation-private-sigstore/chainsaw-test.yaml`:
- Around line 183-185: Update the stale comment that references "grep" to
reflect the current `yq` JSON evaluation: replace the phrase "The grep above
fails (non-zero) on a checksum mismatch" with something like "The yq assertion
above fails (non-zero) on a checksum mismatch" so the comment next to the
checksum check (the comment describing why the checksum check is independent of
verify's exit code and the expected private-Rekor attestation failure)
accurately describes the `yq` assertion being used.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 94a08af0-be34-4600-866a-a795218be714
📒 Files selected for processing (10)
.github/actions/load-versions/action.yml.github/workflows/sigstore-scaffolding-e2e.yaml.gitignore.settings.yamltests/chainsaw/signing/bundle-attestation-private-sigstore/README.mdtests/chainsaw/signing/bundle-attestation-private-sigstore/chainsaw-test.yamltests/chainsaw/signing/bundle-attestation-private-sigstore/oidc-discovery-rbac.yamltests/chainsaw/signing/bundle-attestation-private-sigstore/run.shtests/chainsaw/signing/bundle-attestation-private-sigstore/scaffold-values.yamltests/chainsaw/signing/bundle-attestation-private-sigstore/tlsproxy/main.go
8999c66 to
ed8d0b5
Compare
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/actions/load-versions/action.yml:
- Around line 94-105: The composite action duplicates outputs named mkcert,
mkcert_sha256_linux_amd64, and mkcert_sha256_linux_arm64 (and their
corresponding echo assignments), which breaks action parsing; remove the
duplicated output entries and the duplicate echo/set-output assignments so each
output (scaffold_chart, mkcert, mkcert_sha256_linux_amd64,
mkcert_sha256_linux_arm64) is defined only once and populated only once from
steps.versions.outputs. Locate the repeated definitions of the mkcert* outputs
in the outputs block and the repeated echo lines (the second set around the
195-206/204-206 region) and delete the redundant entries to restore a single
canonical output assignment for each mkcert* symbol.
In `@tests/chainsaw/signing/bundle-attestation-private-sigstore/run.sh`:
- Around line 53-55: The script currently assumes goreleaser is present when
AICR_BIN is unset; update the build fallback to match the stated prerequisites
by checking for goreleaser and falling back to go build when goreleaser is
missing (or alternatively add a preflight check that ensures goreleaser is
installed before using it). Concretely, in the AICR_BIN unset branch (the block
that invokes goreleaser), replace the hard dependency with a conditional: if
command -v goreleaser >/dev/null then run goreleaser, else run go build (or set
an error with a clear message prompting installation), and also add a
corresponding preflight check for goreleaser where environment/tools are
verified so the script behavior and the “goreleaser or go” prereq list stay
aligned.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 8d003088-e487-4e32-af99-fc88ac79fac8
📒 Files selected for processing (10)
.github/actions/load-versions/action.yml.github/workflows/sigstore-scaffolding-e2e.yaml.gitignore.settings.yamltests/chainsaw/signing/bundle-attestation-private-sigstore/README.mdtests/chainsaw/signing/bundle-attestation-private-sigstore/chainsaw-test.yamltests/chainsaw/signing/bundle-attestation-private-sigstore/oidc-discovery-rbac.yamltests/chainsaw/signing/bundle-attestation-private-sigstore/run.shtests/chainsaw/signing/bundle-attestation-private-sigstore/scaffold-values.yamltests/chainsaw/signing/bundle-attestation-private-sigstore/tlsproxy/main.go
ed8d0b5 to
a3756ac
Compare
Add an automated keyless-signing e2e for `aicr bundle --attest --fulcio-url --rekor-url` against a self-hosted Sigstore stack, the second integration test for the supply-chain epic (#1149). Resolves #1215. Deploys the stack from the sigstore `scaffold` Helm chart (Fulcio + Rekor + CTLog + Trillian) instead of sigstore/scaffolding's Knative + MetalLB + sslip.io scripts, which are unreachable from a macOS/Colima host. The chart's plain Deployment + ClusterIP services are reached via `kubectl port-forward`, so the suite runs identically on macOS arm64 and Linux CI. - run.sh: shared local+CI harness (kind, helm install, Fulcio OIDC trust, port-forward, TLS proxy, SA token mint, chainsaw, teardown). - scaffold-values.yaml: trust the in-cluster Kubernetes ServiceAccount OIDC issuer; replace Trillian's EOL amd64-only MySQL 5.7 with the multi-arch official mysql image so the stack runs arm64-native (no OOM under emulation). - oidc-discovery-rbac.yaml: grant anonymous access to the OIDC discovery endpoints Fulcio fetches. - tlsproxy/: localhost TLS termination (aicr requires https:// endpoints); run.sh `go build`s it at runtime, the binary is not committed. - sigstore-scaffolding-e2e.yaml: build + attest the binary, then invoke run.sh (id-token scoped to the job). - Pin the scaffold chart in .settings.yaml and wire it through load-versions. aicr verify cannot yet target the private Rekor (deferred to Phase 2, #1215), so the verify step asserts only bundle checksums.
a3756ac to
10d8add
Compare
Summary
Add an automated keyless-signing e2e for
aicr bundle --attest --fulcio-url --rekor-urlagainst a self-hosted Sigstore stack, deployed from the sigstorescaffoldHelm chart so it runs identically on macOS (Colima) and Linux CI.Motivation / Context
PR #408 added
--fulcio-url/--rekor-urlfor private Sigstore; its unit tests stub at theKeylessIdentityseam, so nothing actually issues a Fulcio cert and writes a Rekor proof against a real instance. This is the second integration test for the supply-chain epic (#1149), alongside the KMS MiniStack e2e (#1298).The earlier
sigstore/scaffoldingsetup-kind.shapproach deploys Knative Services behind MetalLB + Kourier + sslip.io, which are unreachable from a macOS/Colima host (Kind runs in a VM). This PR uses thescaffoldHelm chart instead: plainDeployment+ClusterIP, reached viakubectl port-forwardto localhost. No MetalLB, Knative, sslip.io, or sudo.Fixes: #1215
Related: #1149, #408, #1298
Type of Change
Component(s) Affected
tests/chainsaw/signing/) + CI workflow + tool-version pinsNo product/source code (
pkg/,cmd/) is changed.Implementation Notes
run.sh— shared local + CI harness: kind cluster,helm install scaffold, Fulcio OIDC trust, port-forward, TLS proxy, SA token mint, chainsaw, teardown. The CI workflow builds + attests the binary then invokesrun.sh, so local == CI.scaffold-values.yaml— (1) Fulcio trusts the in-cluster Kubernetes ServiceAccount OIDC issuer (token minted viakubectl create token --audience sigstore); (2) Trillian's default MySQL image is amd64-only EOL 5.7 and is OOM-killed under emulation on Apple Silicon, so it is overridden to the multi-arch officialmysqlimage (arm64-native), pinned by digest.oidc-discovery-rbac.yaml— grants anonymous access to the cluster OIDC discovery endpoints Fulcio fetches;run.shalso setsSSL_CERT_FILEon the Fulcio deployment so it trusts the apiserver cert during discovery. Neither is expressible as a chart value.tlsproxy/—aicrrequireshttps://signing endpoints; this stdlib TLS-termination proxy fronts the plain-HTTP port-forwards with an mkcert cert.run.shgo builds it at runtime; the compiled binary is not committed.id-token: writeis scoped to the job;${{ github.repository }}passed via env (no template injection); actions pinned by SHA.aicr verifycannot yet target the private Rekor, so the verify step asserts only bundle checksums (Phase 2, Integration-test private Sigstore signing via sigstore/scaffolding #1215).Depends on #1298 for the shared
mkcertwiring in.settings.yaml/load-versions; this branch should rebase onto it before merge. Until then,actionlintreportssteps.versions.outputs.mkcert*as undefined (expected).Testing
The full suite was run end-to-end against a fresh Kind cluster on macOS arm64 (Colima):
Static checks:
shellcheck(run.sh),yamllint,golangci-lint(tlsproxy, 0 issues),actionlint(clean except the expectedmkcert*pending the #1298 rebase),chainsawparse-check.Risk Assessment
Rollout notes: Dedicated workflow gated to same-repo events (fork PRs skip; they lack OIDC). N/A for runtime.
Checklist
make lintequivalents: shellcheck/yamllint/golangci-lint/actionlint)git commit -S)