feat(verify): private Sigstore trust root (aicr verify --trust-root)#1449
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Enterprise Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughThis PR adds Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Suggested reviewers
🚥 Pre-merge checks | ✅ 3 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
There was a problem hiding this comment.
Actionable comments posted: 6
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
pkg/cli/bundle_verify.go (1)
141-147: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winDo not reclassify structured verifier errors as internal.
With
--trust-root,verifier.Verifymay return a structured invalid-request error for a bad trusted root; this wrapper currently converts it toErrCodeInternal.Proposed fix
result, err := verifier.Verify(ctx, absDir, verifyOpts) if err != nil { - return errors.Wrap(errors.ErrCodeInternal, "bundle verification failed", err) + return errors.PropagateOrWrap(err, errors.ErrCodeInternal, "bundle verification failed") }Based on learnings, use
errors.PropagateOrWrap(err, errCode, message)so already-coded structured errors propagate unchanged.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pkg/cli/bundle_verify.go` around lines 141 - 147, The bundle verification path is reclassifying structured errors from verifier.Verify as internal, which breaks bad trust-root handling. Update the error handling in the verify command to use errors.PropagateOrWrap(err, errCode, message) instead of always wrapping with ErrCodeInternal, so existing structured verifier errors like invalid-request pass through unchanged while unexpected errors are still wrapped. Use the verifyOpts/ verifier.Verify flow in bundle_verify.go to locate the fix.Source: Learnings
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@pkg/bundler/attestation/verifying.go`:
- Around line 42-46: PublicGoodTrustedRoot is collapsing the structured error
code returned by trust.GetTrustedMaterial() by always wrapping failures as
ErrCodeInternal. Update the error handling in PublicGoodTrustedRoot to preserve
any existing coded error from trust.GetTrustedMaterial() and only wrap when the
error is uncoded, using errors.PropagateOrWrap or equivalent so
VerifyBinaryAttestation and default bundle verification keep the original
Unavailable, Unauthorized, or Internal classification.
In `@pkg/bundler/verifier/verifier_test.go`:
- Around line 563-588: The test currently only checks generic attestation
failure outcomes, so it can still pass even if Verify() never reaches the
TrustRoot loader path. Update TestVerify_TrustRootOption_LoaderFailure to assert
a loader-specific message in result.Errors from the
newUnionTrustedRoot/TrustedMaterial path, such as the missing trusted_root.json
path or file-not-found detail. Keep the existing TrustLevel, Errors, and
BundleAttested checks, but add a concrete expectation that proves the
opts.TrustRoot branch actually ran.
In `@pkg/bundler/verifier/verifier.go`:
- Around line 100-116: Preserve the coded error from --trust-root by moving the
private trusted root load out of newUnionTrustedRoot and into option resolution,
so LoadTrustedMaterialFromFile can fail early and return ErrCodeInvalidRequest
as-is instead of being wrapped by the attestation verifier. Update the flow
around newUnionTrustedRoot and the related option-building path to pass a
preloaded private root into the verifier, keeping verification errors for actual
bundle failures only.
In `@pkg/cli/bundle_verify_test.go`:
- Around line 195-259: The trust-root plumbing test currently asserts only the
generic “bundle attestation verification failed” banner, which is too broad. In
TestBundleVerifyCmd_TrustRootFlagPlumbing, tighten the trust-root-only case to
assert the missing trust-root loader detail from the union loader path, so the
test proves --trust-root actually reaches trust-root resolution. Keep the second
subtest focused on verifying --trust-root can coexist with --key and
--certificate-identity-regexp without a flag-parse or mutual-exclusion error,
and use the existing bundleVerifyCmd and writeParseableTrustRootFixture setup to
preserve the real command execution.
In `@pkg/trust/trust.go`:
- Around line 232-245: Reject non-regular trust-root paths in
LoadTrustedMaterialFromFile before opening the file. Add a path check using the
existing os-based flow to ensure --trust-root points to a regular file (not
FIFO, device, or other special file), and return ErrCodeInvalidRequest with a
clear message when it doesn’t. Keep the existing open/read logic and size cap
after the new validation.
In
`@tests/chainsaw/signing/bundle-attestation-private-sigstore/chainsaw-test.yaml`:
- Line 321: The negative control in the bundle verification test is not
identical to the positive path because the verify command omits the same
certificate identity regexp. Update the command in the chainsaw test so the
verification call includes the `--certificate-identity-regexp` argument used
elsewhere, while keeping the rest of the flags unchanged except for the intended
trust-root difference, and continue asserting that `bundleAttested` remains
false.
---
Outside diff comments:
In `@pkg/cli/bundle_verify.go`:
- Around line 141-147: The bundle verification path is reclassifying structured
errors from verifier.Verify as internal, which breaks bad trust-root handling.
Update the error handling in the verify command to use
errors.PropagateOrWrap(err, errCode, message) instead of always wrapping with
ErrCodeInternal, so existing structured verifier errors like invalid-request
pass through unchanged while unexpected errors are still wrapped. Use the
verifyOpts/ verifier.Verify flow in bundle_verify.go to locate the fix.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: a62e0644-a8bc-4a7b-93f7-42b0275b6983
📒 Files selected for processing (20)
.github/workflows/sigstore-scaffolding-e2e.yamldocs/user/artifact-verification.mddocs/user/cli-reference.mdpkg/bundler/attestation/keylessverifyidentity.gopkg/bundler/attestation/keyverifyidentity.gopkg/bundler/attestation/keyverifyidentity_test.gopkg/bundler/attestation/trustsource_test.gopkg/bundler/attestation/verifying.gopkg/bundler/attestation/verifying_test.gopkg/bundler/verifier/verifier.gopkg/bundler/verifier/verifier_test.gopkg/cli/bundle_verify.gopkg/cli/bundle_verify_test.gopkg/defaults/timeouts.gopkg/trust/testdata/trusted_root.jsonpkg/trust/trust.gopkg/trust/trust_test.gotests/chainsaw/signing/bundle-attestation-private-sigstore/README.mdtests/chainsaw/signing/bundle-attestation-private-sigstore/chainsaw-test.yamltests/chainsaw/signing/bundle-attestation-private-sigstore/run.sh
Coverage Report ✅
Coverage BadgeMerging this branch will increase overall coverage
Coverage by fileChanged files (no unit tests)
Please note that the "Total", "Covered", and "Missed" counts above refer to code statements instead of lines of code. The value in brackets refers to the test coverage of that file in the old version of the code. |
a035866 to
2bdbe62
Compare
Recipe evidence checkProtected recipesRecipes with committed evidence (
How to refresh evidenceRun on a cluster matching the recipe's aicr snapshot -o snapshot.yaml
aicr validate \
-r recipes/overlays/<slug>.yaml \
-s snapshot.yaml \
--emit-attestation ./out \
--push ghcr.io/<your-fork>/aicr-evidence
cp ./out/pointer.yaml recipes/evidence/<slug>.yamlThis gate is warning-only and never blocks merge. See ADR-007 for the trust model. |
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@pkg/trust/trust_test.go`:
- Around line 99-153: Move the non-regular-file validation in
LoadTrustedMaterialFromFile ahead of os.Open by checking the path with
stat/lstat first, then only opening regular files so FIFOs never get a chance to
block. Add a regression subtest in TestLoadTrustedMaterialFromFile that creates
a named pipe and verifies the function returns ErrCodeInvalidRequest for it,
alongside the existing valid/missing/malformed/empty/oversized cases.
In `@tests/chainsaw/signing/bundle-attestation-private-sigstore/run.sh`:
- Around line 319-325: The CTLog public key extraction in extract_ctlog_pubkey()
uses a GNU-only base64 decode flag, so update that decode step to be portable
across macOS and Linux by selecting the correct flag per platform or by routing
through a small cross-platform decode helper; keep the existing validation
checks for CTLOG_PUBKEY after the decode.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: d2084c61-8606-489e-91cd-8e02994d9cf4
📒 Files selected for processing (22)
.github/workflows/sigstore-scaffolding-e2e.yamldocs/user/artifact-verification.mddocs/user/cli-reference.mdpkg/bundler/attestation/keylessverifyidentity.gopkg/bundler/attestation/keyverifyidentity.gopkg/bundler/attestation/keyverifyidentity_test.gopkg/bundler/attestation/trustsource_test.gopkg/bundler/attestation/verifying.gopkg/bundler/attestation/verifying_test.gopkg/bundler/verifier/verifier.gopkg/bundler/verifier/verifier_test.gopkg/cli/bundle_verify.gopkg/cli/bundle_verify_test.gopkg/defaults/timeouts.gopkg/trust/testdata/trusted_root.jsonpkg/trust/trust.gopkg/trust/trust_test.gorecipes/evidence/gb200-eks-ubuntu-training.yamlrecipes/evidence/h100-gke-cos-training.yamltests/chainsaw/signing/bundle-attestation-private-sigstore/README.mdtests/chainsaw/signing/bundle-attestation-private-sigstore/chainsaw-test.yamltests/chainsaw/signing/bundle-attestation-private-sigstore/run.sh
41531b5 to
31404d3
Compare
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@pkg/trust/trust_fifo_test.go`:
- Around line 38-40: The FIFO test is silently skipping on `syscall.Mkfifo`
failure, which hides regressions in the Unix-only path. In `TestTrustFIFO` (or
the test helper that creates `fifo`), replace the `t.Skipf` fallback with a hard
test failure so broken environments surface immediately instead of bypassing the
pre-open FIFO block-path coverage. Keep the behavior focused on the existing
`syscall.Mkfifo` setup and the FIFO regression check in this test file.
In `@pkg/trust/trust.go`:
- Around line 235-250: The trust-root loading path still has a TOCTOU gap
because os.Stat(path) and os.Open(path) check different path resolutions; remove
the pre-open pathname check and validate the opened file descriptor instead,
using the trust root loader logic around os.Open and the subsequent file
handling. After opening, inspect the descriptor with file info from the open
handle and reject non-regular files there, so the regular-file guard cannot be
bypassed by swapping the pathname between calls.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 3f8962dc-bc45-466d-8946-3c2b5b0916d1
📒 Files selected for processing (23)
.github/workflows/sigstore-scaffolding-e2e.yamldocs/user/artifact-verification.mddocs/user/cli-reference.mdpkg/bundler/attestation/keylessverifyidentity.gopkg/bundler/attestation/keyverifyidentity.gopkg/bundler/attestation/keyverifyidentity_test.gopkg/bundler/attestation/trustsource_test.gopkg/bundler/attestation/verifying.gopkg/bundler/attestation/verifying_test.gopkg/bundler/verifier/verifier.gopkg/bundler/verifier/verifier_test.gopkg/cli/bundle_verify.gopkg/cli/bundle_verify_test.gopkg/defaults/timeouts.gopkg/trust/testdata/trusted_root.jsonpkg/trust/trust.gopkg/trust/trust_fifo_test.gopkg/trust/trust_test.gorecipes/evidence/gb200-eks-ubuntu-training.yamlrecipes/evidence/h100-gke-cos-training.yamltests/chainsaw/signing/bundle-attestation-private-sigstore/README.mdtests/chainsaw/signing/bundle-attestation-private-sigstore/chainsaw-test.yamltests/chainsaw/signing/bundle-attestation-private-sigstore/run.sh
Add `aicr verify --trust-root <path>` to verify a bundle's bundle attestation against a private Sigstore trusted_root.json, the verify counterpart to `bundle --fulcio-url`/`--rekor-url`. The supplied root is additively unioned with AICR's built-in public-good root, so both NVIDIA-signed and privately-signed bundles verify with one command; the binary attestation stays pinned to the public-good root. Implemented as an injectable attestation.TrustedRootSource threaded into both verification identities (so it composes with --key); the verifier builds the union for the bundle attestation only. Adds trust.LoadTrustedMaterialFromFile (rejects non-regular files; user-file errors classified ErrCodeInvalidRequest, distinct from the TUF-cache Internal path, preserved through PublicGoodTrustedRoot and the CLI via PropagateOrWrap), defaults.MaxTrustedRootBytes, CLI plumbing, docs, and extends the private-Sigstore chainsaw e2e with a --trust-root verify round trip plus a negative control. A bad --trust-root file is resolved up front so it fails fast with its coded ErrCodeInvalidRequest rather than being folded into a verification-failure result. Also adds missing Apache-2.0 license headers to two recipes/evidence fixtures (via addlicense), surfaced by `make lint`. Fixes #1153
31404d3 to
a2c2011
Compare
mchmarny
left a comment
There was a problem hiding this comment.
Clean, well-scoped supply-chain feature — aicr verify --trust-root as the verify counterpart to bundle --fulcio-url/--rekor-url, in-scope per the architecture boundaries (provenance for generated artifacts). Strong on the details that matter here: the loader bounds the read, closes the stat/open TOCTOU window with O_NONBLOCK + descriptor fstat, rejects non-regular files, and classifies user-file vs. TUF-cache failures correctly; the union is additive so the binary attestation stays pinned to public-good (structurally unreachable by the flag); and a bad --trust-root fails fast with its real ErrCodeInvalidRequest instead of being folded into a generic verification result. Test coverage is thorough — the injected-source seam, base+key retention, fail-fast loader path, CLI plumbing/coexistence with --key, and a real e2e round trip with a negative control proving the flag is the differentiator.
Two inline notes are informational only (confirming the additive-trust semantics are intentional, and the bare error return is correct). Nothing blocks merge. CI is still running — no failures so far; mergeable_state: blocked is just the pending-checks gate.
Approving.
| // both verify. Counterpart to `bundle --fulcio-url`/`--rekor-url`. | ||
| // Composable with Key. Does NOT affect the binary attestation, which is | ||
| // always NVIDIA-public-CI-signed and stays pinned to the public-good root. | ||
| TrustRoot string |
There was a problem hiding this comment.
Confirming the security semantics, since this is a supply-chain trust surface: because the root is a union, --trust-root alone admits any bundle that chains to either root — including a publicly-signed one — so bundleAttested without --require-creator/identity pinning still only proves "someone signed it." This doesn't weaken the existing public-good default (which has the same property), and the additive design + deferred strict "must chain to the private root" mode are clearly documented. Noting it's understood and intentional, not a gap. No change requested.
There was a problem hiding this comment.
The bare return nil, err from k.src(ctx) here is correct, not an oversight: the source (PublicGoodTrustedRoot or the union closure) already returns a coded pkg/errors error, so re-wrapping would clobber the classification — matches the "don't double-wrap" rule. The O_NONBLOCK + descriptor-fstat loader and the //go:build unix FIFO regression guard are nicely done.
Summary
Adds
aicr verify --trust-root <path>to verify a bundle's bundle attestation against a private Sigstoretrusted_root.json, additively unioned with AICR's built-in public-good root. It is the verify counterpart tobundle --fulcio-url/--rekor-url(#408) and the next leg of the closed-supply-chain verification story afterverify --key(#1152).Motivation / Context
Private Sigstore signing (#408) lets an org sign bundles against a self-hosted Fulcio/Rekor, but
aicr verifycould only validate against the public-good trusted root, so privately-signed bundles failed verification: the chain broke at the consumer. This closes that asymmetry for the private-trust-root mode.Fixes: #1153
Related: #1149 (epic), #408, #1152
Type of Change
Component(s) Affected
cmd/aicr,pkg/cli)pkg/bundler,pkg/component/*)pkg/errors,pkg/k8s)docs/,examples/)pkg/trust,pkg/defaults, chainsaw e2eImplementation Notes
--trust-rootis additive: the supplied root is combined with the public-good root via sigstore-go'sroot.TrustedMaterialCollection, so NVIDIA-signed and org-signed bundles both verify with one command. The public-good root is never dropped.nilsource there, structurally unreachable by--trust-root).attestation.TrustedRootSource(defaultPublicGoodTrustedRoot) threaded into both the keyless and key verification identities, so it composes with--keyfor free.verifier.newUnionTrustedRootbuilds the union;trust.LoadTrustedMaterialFromFileloads/parses the file with a bounded read (defaults.MaxTrustedRootBytes) and classifies user-file errors asErrCodeInvalidRequest(distinct from the TUF-cacheInternalpath).--trust-rootonly supplies trust anchors; pin the signer with the existing--require-creator/--certificate-identity-regexp. A strict "must chain to the private root" mode is intentionally deferred.Testing
TrustedRootSourceseam on both identities, the union retains both base and key material (hermetic), theVerify-level union branch, and CLI flag plumbing/coexistence with--key.LoadTrustedMaterialFromFile91.7%,parseTrustedRoot85.7%,PublicGoodTrustedRoot75%).tests/chainsaw/signing/bundle-attestation-private-sigstorewith a real sign-then---trust-root-verify round trip (assembles atrusted_root.jsonfrom the local stack viacosign trusted-root create) plus a negative control proving the flag is what makes the difference. This runs only in thesigstore-scaffolding-e2e.yamlworkflow (needs Kind + the scaffolding stack + a CI-attested binary); it cannot run locally.make qualify(e2e + grype scan) was not runnable in the dev sandbox; it runs in CI.Risk Assessment
Rollout notes: Behavior is byte-identical when
--trust-rootis unset (default public-good path unchanged). The flag is additive and the binary-attestation trust guarantee is preserved. Backwards compatible; no migration.Checklist
make testwith-race)make lint)git commit -S)