Summary
The release/e2e build's aicr recipe sign-catalog goreleaser post-hook can fail the whole build on a transient OIDC-token network blip, because the ambient OIDC token fetch is not wrapped in any retry. The sibling cosign attest-blob post-hook is retry-wrapped, so it survives the same class of blip — leaving an inconsistent resilience gap.
Evidence
Observed in CI on PR #1350 (tests / CLI E2E), during the goreleaser build post-hook:
[cli] command failed: error=[SERVICE_UNAVAILABLE] OIDC token request failed:
Get ".../idtoken/...?api-version=2.0&audience=sigstore": net/http: TLS handshake timeout
exitCode=6
build failed after 24s error=post hook failed: exit status 6
Run: https://github.com/NVIDIA/aicr/actions/runs/27550655342/job/81435823531
The change under test touched only pkg/fingerprint (GPU SKU token matching) — the failure is purely infra (TLS handshake timeout to GitHub's idtoken endpoint), i.e. a flake that a retry would absorb.
Root Cause
FetchAmbientOIDCToken (pkg/bundler/attestation/oidc.go:38) issues a single client.Do(req) and wraps any error as ErrCodeUnavailable with no retry. A transient TLS handshake timeout therefore fails immediately.
The existing Sigstore retry budget does not cover this step. Per pkg/bundler/attestation/doc.go ("Retry Contract"), SignStatementWith wraps only the Fulcio/Rekor sign.Bundle call in exponential-backoff retry; the "OIDC Token Acquisition" flows are documented as a separate, un-retried step.
By contrast, the cosign attest-blob post-hook in .goreleaser.yaml (lines 51–63) wraps its entire invocation — including cosign's own OIDC fetch — in a 3-attempt bash retry loop with backoff, so the binary-attestation step is resilient to the same blip. The recipe sign-catalog hook (.goreleaser.yaml lines 74–78) has no such wrapper, and FetchAmbientOIDCToken provides none internally.
Proposed Fix
Add bounded exponential-backoff retry to the ambient OIDC token acquisition, consistent with the existing Sigstore retry contract:
- Preferred: wrap the
client.Do in FetchAmbientOIDCToken (and likely the other ambient/identity-token fetch paths) in a retry using the existing defaults.Sigstore* knobs, so all keyless OIDC callers (CLI recipe sign-catalog, API, future consumers) benefit — not just the goreleaser hook. Retry only on transient transport errors / 5xx; fail fast on 4xx (bad/empty token).
- At minimum: wrap the
recipe sign-catalog post-hook in the same bash retry loop already used for cosign attest-blob, to remove the immediate CI-flake exposure.
The Go-level fix is preferable since the bash-loop approach re-runs the full signing each attempt (including the already-succeeded portions) and only protects the goreleaser path.
References
Summary
The release/e2e build's
aicr recipe sign-cataloggoreleaser post-hook can fail the whole build on a transient OIDC-token network blip, because the ambient OIDC token fetch is not wrapped in any retry. The siblingcosign attest-blobpost-hook is retry-wrapped, so it survives the same class of blip — leaving an inconsistent resilience gap.Evidence
Observed in CI on PR #1350 (
tests / CLI E2E), during thegoreleaser buildpost-hook:Run: https://github.com/NVIDIA/aicr/actions/runs/27550655342/job/81435823531
The change under test touched only
pkg/fingerprint(GPU SKU token matching) — the failure is purely infra (TLS handshake timeout to GitHub'sidtokenendpoint), i.e. a flake that a retry would absorb.Root Cause
FetchAmbientOIDCToken(pkg/bundler/attestation/oidc.go:38) issues a singleclient.Do(req)and wraps any error asErrCodeUnavailablewith no retry. A transient TLS handshake timeout therefore fails immediately.The existing Sigstore retry budget does not cover this step. Per
pkg/bundler/attestation/doc.go("Retry Contract"),SignStatementWithwraps only the Fulcio/Rekorsign.Bundlecall in exponential-backoff retry; the "OIDC Token Acquisition" flows are documented as a separate, un-retried step.By contrast, the
cosign attest-blobpost-hook in.goreleaser.yaml(lines 51–63) wraps its entire invocation — including cosign's own OIDC fetch — in a 3-attempt bash retry loop with backoff, so the binary-attestation step is resilient to the same blip. Therecipe sign-cataloghook (.goreleaser.yamllines 74–78) has no such wrapper, andFetchAmbientOIDCTokenprovides none internally.Proposed Fix
Add bounded exponential-backoff retry to the ambient OIDC token acquisition, consistent with the existing Sigstore retry contract:
client.DoinFetchAmbientOIDCToken(and likely the other ambient/identity-token fetch paths) in a retry using the existingdefaults.Sigstore*knobs, so all keyless OIDC callers (CLIrecipe sign-catalog, API, future consumers) benefit — not just the goreleaser hook. Retry only on transient transport errors / 5xx; fail fast on 4xx (bad/empty token).recipe sign-catalogpost-hook in the same bash retry loop already used forcosign attest-blob, to remove the immediate CI-flake exposure.The Go-level fix is preferable since the bash-loop approach re-runs the full signing each attempt (including the already-succeeded portions) and only protects the goreleaser path.
References