Skip to content

ci: automate release artifact build and publish - #2112

Merged
Benjamin Elder (BenTheElder) merged 12 commits into
agent-substrate:mainfrom
stshrive:dev/publish-artifacts
Oct 23, 2026
Merged

Benjamin Elder (BenTheElder) merged 12 commits into
agent-substrate:mainfrom
stshrive:dev/publish-artifacts

Conversation

@stshrive

@stshrive Steven Shriver (stshrive) commented Oct 2, 2026 •

Copy link
Copy Markdown
Collaborator

Amends the release workflow to include a build and publish step for the system artifacts required for consumers.

Part of #1871

  • Tests pass
  • Appropriate changes to documentation are included in the PR

Comment thread cmd/ate-setup/README.md Resolved
Comment thread hack/release/publish-images.sh Resolved
@ahmedtd

Copy link
Copy Markdown
Collaborator

Nit: I wouldn't block submission on this, but we've been pushing hard to minimize the amount of shell-scripting in the repo. Do you think it could be possible to have this logic in Go?

@BenTheElder Benjamin Elder (BenTheElder) left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should consider decoupling "automate a workflow to publish release notes + tagged release" from "start pushing pre-built content"

The former requires nailing down things like "who can trigger it and how do they specify a version" and de-bottlenecks "publish a tag with release notes"

"Releases contain pre-built artifacts" feels like a follow-up step with more to review and consideration that could be incremental. We need it, but I don't think we need it as badly for literally tomorrow and it makes the diff a lot more involved.

@BenTheElder

Copy link
Copy Markdown
Collaborator

What is the minimum we need to leverage the workflow tomorrow without the artifact bit?

Are https://github.com/agent-substrate/substrate/blob/main/docs/dev/release-notes.md#cut-a-release the plan? (we should probably split release cutting out of the docs for how to tag PRs for release notes)

@stshrive

Copy link
Copy Markdown
Collaborator Author

What is the minimum we need to leverage the workflow tomorrow without the artifact bit?

Are https://github.com/agent-substrate/substrate/blob/main/docs/dev/release-notes.md#cut-a-release the plan? (we should probably split release cutting out of the docs for how to tag PRs for release notes)

So the change in #2018 does the minimum steps. The docs have been split to make that distinction better. The Cut a release part of the docs appear to have been used already for v0.4.0 unless that was done manually with a agent again.

I think we should consider decoupling "automate a workflow to publish release notes + tagged release" from "start pushing pre-built content"

This is additive to the notes/releases and adds the image artifacts.

@stshrive

Copy link
Copy Markdown
Collaborator Author

Nit: I wouldn't block submission on this, but we've been pushing hard to minimize the amount of shell-scripting in the repo. Do you think it could be possible to have this logic in Go?

I made the changes for this, though now there are additions to ate-setup since there were already some publish steps. PTAL.

@LiorLieberman

Lior Lieberman (LiorLieberman) commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

deferring to ben's review on this

@BenTheElder Benjamin Elder (BenTheElder) left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I gave some extra instructions to my review helper given the security sensitivity here.
I audited the findings and read through it myself (as usual).

Thanks for working on this.

One other question we should double check: are the right set of images included, for things we plan to move out of the repo, maybe we specially exclude them temporarily.

Also: binaries? We can attach them to the draft (kubectl-ate at least?), probably cross-compiled + .sha256 sidecar for validating with shasum like tools

Comment thread cmd/ate-setup/internal/steps/publish.go Resolved
Comment thread hack/release/draft-release.sh Resolved
Comment thread .github/workflows/release.yaml Resolved
Comment thread .github/workflows/release.yaml Resolved
Comment thread cmd/ate-setup/internal/steps/publish.go Resolved
Comment thread cmd/ate-setup/internal/steps/publish.go Resolved
Comment thread .github/workflows/release.yaml Resolved
Comment thread .github/workflows/release.yaml Resolved
@bowei Bowei Du (bowei) added area/dev-infra kind/feature An enhancement / feature request or implementation labels Oct 19, 2026
Amends the release workflow to include a build and publish step
for the system artifacts required for consumers.
Publish releases implemented in golang: No Make, no bash additions. Use
the official substrate name in examples from README.md. Splits the
release cutting and labeling docs.
- Fixes the module path to pin to github repo source.
- Updates the list of images to release.
- Fixes the overwrite naming issue and doc formatting.
- Pins the registry:3 to a SHA.
- Updates the documentation on the credential provider to match
  across all docs.
- fixes ref matching on publish and release
Adds build and publish for kubectl-ate and align publish with
`kustomize` and `helm` plan by removing the `ate-setup --image-dir`
installation path for installing from pre-built images.
Removes the AGENTS.md entry directing agents not to use inline bash
@stshrive

Copy link
Copy Markdown
Collaborator Author

Addressed comments - rebased onto upstream/main

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI protect-tags has not been in enforcing mode, it is now, but that means 0.6.0 didn't go through it.

}
ref, err := p.Pin(ctx, image)
if err != nil {
// The registry may hold an image releases do not publish, so it is

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 nit 🟢 – Pin's doc comment at line 79 still says a release publishes envoy-dataplane under the same repository and tag as the rest. Releases leave it out now, as dockerfileImage in steps/env.go and docs/dev/releasing.md say, so that part of the comment should go.

// of releases until it is added here in review. A pre-built install still
// looks an unlisted image up in its registry; if the lookup fails, the error
// says releases do not publish it.
var released = []string{

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks

Comment thread AGENTS.md
- **Binaries**: `go build -o bin/kubectl-ate ./cmd/kubectl-ate`, `go build -o bin/ate-setup ./cmd/ate-setup`
- **Release images**: `VERSION=TAG go run ./cmd/ate-setup publish release-images --ko-docker-repo REPO` (every image a pre-built install needs, all tagged `TAG`, including the docker-built `envoy-dataplane`)
- **Release images**: `VERSION=TAG go run ./cmd/ate-setup publish release-images --ko-docker-repo REPO` (the release set, `images.ReleaseComponents`, all tagged `TAG`; see `docs/dev/releasing.md`)
- **Release binaries**: `VERSION=TAG go run ./cmd/ate-setup publish release-binaries --out DIR` (`kubectl-ate` for linux and darwin, amd64 and arm64, with `.sha256` files)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we ... might also need windows for kubectl-ate. that can be a follow-up, cc Eitan Yarmush (@EItanya)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree - adding it now would have been a bunch more stuff in the review that I thought would be best to keep separate.

@BenTheElder
Benjamin Elder (BenTheElder) added this pull request to the merge queue Oct 23, 2026
@BenTheElder Benjamin Elder (BenTheElder) added this to the M3 milestone Oct 23, 2026
Merged via the queue into agent-substrate:main with commit 57e385b Oct 23, 2026
15 checks passed
igooch pushed a commit to igooch/substrate that referenced this pull request Oct 27, 2026
Part of agent-substrate#3217. The default atenet dataplane runs envoy-dataplane in the
router and egress gateways, but releases do not publish it, so a
pre-built install has to use `--atenet-dataplane=agentgateway` or build
from source. agent-substrate#2112 dropped it because building Rust under QEMU was slow.
agent-substrate#3152 made the Dockerfile cross-compile, so the release job now builds
it with buildx and no QEMU.

Released Dockerfile images are an allowlist (`releasedDockerfiles`) next
to the ko allowlist. The release jobs pin BuildKit by digest. Release
builds take each Dockerfile image's digest from buildx's
`--metadata-file`, build Dockerfile images before the ko images, ignore
`DOCKER_BUILD_FLAGS`, and label every image with its source, revision
and version. With every image a default install deploys now published,
the release notes give the install command.

After the first release that includes it, an org owner has to make the
new `ghcr.io/agent-substrate/substrate/envoy-dataplane` package public.

### Validation

`release.yaml` from this branch, run in my fork at 619f0d9 with a
throwaway tag `v0.0.0-envoytest.1`:

-
[`images-dry-run`](https://github.com/MushuEE/substrate/actions/runs/40562367990),
dispatched with `dry_run_images`, built all eleven release images cold
on `ubuntu-latest` and passed the platform and digest checks in 11m35s,
well inside the 30-minute timeout.
- The [tag push
run](https://github.com/MushuEE/substrate/actions/runs/40561810523)
created the draft release. The images job skips itself outside
agent-substrate/substrate, as designed.

Earlier, at bbd8614, I ran the same publish locally against a throwaway
registry, as the `images-dry-run` job does:

```sh
docker run -d --rm --name ate-release-test -p 5000:5000 \
  registry:3@sha256:ddf754342cfc8acc51a56d5d0ab6af06826461864460636d8bd5c546dab2a7b8
printf '[registry."localhost:5000"]\n  http = true\n' > /tmp/buildkitd.toml
docker buildx create --name ate-release --driver docker-container \
  --driver-opt network=host --buildkitd-config /tmp/buildkitd.toml --use
VERSION=v0.0.0-local.1 KO_DEFAULTPLATFORMS=linux/amd64,linux/arm64 \
  go run ./cmd/ate-setup publish release-images --ko-docker-repo localhost:5000/substrate
docker buildx imagetools inspect localhost:5000/substrate/envoy-dataplane:v0.0.0-local.1
docker buildx imagetools inspect localhost:5000/substrate/envoy-dataplane:v0.0.0-local.1 \
  --format '{{json (index .Image "linux/amd64").Config.Labels}}'
```

- buildx cross-compiled the Rust module (`linux/amd64->arm64`), with no
QEMU.
- The envoy-dataplane digest from `--metadata-file` matched what the tag
resolved to.
- envoy-dataplane had `linux/amd64` and `linux/arm64`, and the `source`
and `revision` labels.
- A rerun built nothing.

That run showed envoy-dataplane inheriting
`org.opencontainers.image.version: "22.04"` from the Ubuntu base of
`envoyproxy/envoy`. Release images now set that label to the release
tag.

AI use: I used Claude Code to trace the code agent-substrate#2112 removed, draft the
change, address review comments, and script the local and fork
validation. I reviewed the diff and ran the ate-setup tests and the
validation above.

- [x] Tests pass
- [x] Docs updated

---------

Signed-off-by: Grant McCloskey <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/dev-infra kind/feature An enhancement / feature request or implementation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants