Skip to content

ate-setup: retry each ko release image with backoff - #3446

Merged
Benjamin Elder (BenTheElder) merged 2 commits into
agent-substrate:mainfrom
annapendleton:release-images-retry
Oct 29, 2026
Merged

ate-setup: retry each ko release image with backoff#3446
Benjamin Elder (BenTheElder) merged 2 commits into
agent-substrate:mainfrom
annapendleton:release-images-retry

Conversation

@annapendleton

@annapendleton Anna Pendleton (annapendleton) commented Oct 29, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

The v1.0.0-rc.1 release run (https://github.com/agent-substrate/substrate/actions/runs/40925903340) failed in Publish images with DENIED: permission_denied. ghcr.io can briefly deny pushes to a package it has just created, and this is the first release that creates them.

ate-setup publish release-images now builds each ko image in its own ko build. If an image fails, it logs the error and retries that image, for up to 3 attempts, waiting 10s then 20s. The workflow is unchanged.

LiorLieberman
Lior Lieberman (LiorLieberman) reviewed Oct 29, 2026
View reviewed changes

@LiorLieberman Lior Lieberman (LiorLieberman) left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we do the retry within the "publish release-images" script? will be nicer than running the workflow again and more targeted to amend/remove later

@annapendleton Anna Pendleton (annapendleton) changed the title [release-1.0] ci: retry publishing release images when ghcr.io denies a new package [release-1.0] ate-setup: retry the ko release images when ghcr.io denies a push Oct 29, 2026
LiorLieberman
Lior Lieberman (LiorLieberman) previously approved these changes Oct 29, 2026
View reviewed changes

@LiorLieberman Lior Lieberman (LiorLieberman) left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

approving to unblock first relrease - will revisit before it we ship something like this at main

dberkov
Dmitry Berkovich (dberkov) reviewed Oct 29, 2026
View reviewed changes
Comment thread cmd/ate-setup/internal/ko/ko.go Resolved
Comment thread cmd/ate-setup/internal/ko/ko.go Resolved
@annapendleton Anna Pendleton (annapendleton) changed the title [release-1.0] ate-setup: retry the ko release images when ghcr.io denies a push [release-1.0] ate-setup: retry each ko release image with backoff Oct 29, 2026
@annapendleton Anna Pendleton (annapendleton) changed the title [release-1.0] ate-setup: retry each ko release image with backoff ate-setup: retry each ko release image with backoff Oct 29, 2026
@annapendleton
Anna Pendleton (annapendleton) changed the base branch from release-1.0 to main October 29, 2026 19:31
@annapendleton
ate-setup: retry each ko release image with backoff
3bc51a7 
Build each ko image in its own ko run and retry only that image if it
fails: up to 3 attempts, waiting 10s then 20s, logging each error.
ghcr.io can briefly deny pushes to a package it has just created.

Signed-off-by: Anna Pendleton <[email protected]>
@annapendleton Anna Pendleton (annapendleton) added this to the M3 milestone Oct 29, 2026
MushuEE
Grant McCloskey (MushuEE) approved these changes Oct 29, 2026
View reviewed changes

@MushuEE Grant McCloskey (MushuEE) left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[This is not blocking for this review - bias to merge for rc1]

  1. The test swaps a package global. Tests replace var wait. AGENTS.md says not to add package globals just so tests can swap behavior. The wait could live on Runner or come from Retry instead.
  2. It retries every error, not just the denial. A compile error or a bad label would also be retried twice, adding 30 seconds of waiting per image before the job fails. AGENTS.md says to retry only the errors you expect. Checking ko's stderr for DENIED: permission_denied would keep other failures fast. My branch does that. 3. Release images now build one at a time. Before, one ko invocation built all 11 images in parallel in about 51 seconds. Eleven separate invocations will be slower. The Go build cache is shared, so I'd expect a few minutes, well within the
    job's 30-minute timeout, but I haven't measured it.
  3. Minor:
    • The test's denial message is a write_package blob PATCH, but the real failure was read_package on an SBOM manifest PUT. It doesn't affect the test, which retries any error.
    • The exported Retry type and the pushRetry package variable are configuration with only one user. A constant inside BuildEach would do.

BenTheElder
Benjamin Elder (BenTheElder) reviewed Oct 29, 2026
View reviewed changes
Comment thread cmd/ate-setup/internal/ko/ko_test.go
t.Helper()
dir = t.TempDir()
binary = filepath.Join(dir, "ko")
script := "#!/bin/sh\n" +

@BenTheElder Benjamin Elder (BenTheElder) Oct 29, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO (non-blocking): we should put scripts like this into a testdata folder under the package instead of writing them to disk at runtime. This is less complex, less flaky, faster, and allows us to shellcheck the scripts.

@annapendleton
ate-setup: use retry.OnError for the ko release image retries
b1e1fcd 
Replace the exported Retry type and hand-rolled backoff with
retry.OnError and a wait.Backoff, as kube/apply.go does.

Signed-off-by: Anna Pendleton <[email protected]>
BenTheElder
Benjamin Elder (BenTheElder) reviewed Oct 29, 2026
View reviewed changes
Comment thread cmd/ate-setup/internal/ko/ko.go
// has just created.
var buildBackoff = wait.Backoff{Steps: 3, Duration: 10 * time.Second, Factor: 2}

// BuildEach runs BuildTagged once per import path, retrying each failure per

@BenTheElder Benjamin Elder (BenTheElder) Oct 29, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO:

  • BuildEach doesn't imply we're pushing, this API feels like a footgun. however it is an internal package
  • It seems like we should be able to have ko build them all in one pass to dedupe some costs?

This was referenced Oct 29, 2026
Merged
Open
Open
@BenTheElder
Benjamin Elder (BenTheElder) added this pull request to the merge queue Oct 29, 2026
Lior Lieberman (LiorLieberman) pushed a commit that referenced this pull request Oct 29, 2026
@annapendleton
[release-1.0] Cherry-pick #3446: ate-setup: retry each ko release ima…
f86fde9 
…ge with backoff (#3459)

Cherry-pick of #3446 onto `release-1.0`, with no changes (`git
cherry-pick -x` of both commits).

Needed for the 1.0 release: the v1.0.0-rc.1 release run
(https://github.com/agent-substrate/substrate/actions/runs/40925903340)
failed in **Publish images** with `DENIED: permission_denied`. ghcr.io
can briefly deny pushes to a package it has just created, and this is
the first release that creates them.

`ate-setup publish release-images` now builds each ko image in its own
`ko build`. If an image fails, it logs the error and retries that image,
for up to 3 attempts, waiting 10s then 20s (`retry.OnError` with a
`wait.Backoff`).

AI use: AI-assisted.

- [x] Tests pass

---------

Signed-off-by: Anna Pendleton <[email protected]>
Co-authored-by: Anna Pendleton <[email protected]>
Merged via the queue into agent-substrate:main with commit 3aa3eec Oct 29, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@BenTheElder Benjamin Elder (BenTheElder) BenTheElder approved these changes
@dberkov Dmitry Berkovich (dberkov) dberkov left review comments
@LiorLieberman Lior Lieberman (LiorLieberman) LiorLieberman left review comments
+1 more reviewer
@MushuEE Grant McCloskey (MushuEE) MushuEE approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

@BenTheElder Benjamin Elder (BenTheElder)

@LiorLieberman Lior Lieberman (LiorLieberman)

Labels

area/dev-infra area/reliability milestone-ack

Projects

None yet

Milestone

M3

Development

Successfully merging this pull request may close these issues.

5 participants

@annapendleton @BenTheElder @MushuEE @dberkov @LiorLieberman