Repository navigation
ci: automate release artifact build and publish - #2112
Benjamin Elder (BenTheElder) merged 12 commits into
Conversation
c51c9a6 to
0f1ea22
Compare
|
Nit: I wouldn't block submission on this, but we've been pushing hard to minimize the amount of shell-scripting in the repo. Do you think it could be possible to have this logic in Go? |
There was a problem hiding this comment.
I think we should consider decoupling "automate a workflow to publish release notes + tagged release" from "start pushing pre-built content"
The former requires nailing down things like "who can trigger it and how do they specify a version" and de-bottlenecks "publish a tag with release notes"
"Releases contain pre-built artifacts" feels like a follow-up step with more to review and consideration that could be incremental. We need it, but I don't think we need it as badly for literally tomorrow and it makes the diff a lot more involved.
|
What is the minimum we need to leverage the workflow tomorrow without the artifact bit? Are https://github.com/agent-substrate/substrate/blob/main/docs/dev/release-notes.md#cut-a-release the plan? (we should probably split release cutting out of the docs for how to tag PRs for release notes) |
0f1ea22 to
3683e55
Compare
So the change in #2018 does the minimum steps. The docs have been split to make that distinction better. The Cut a release part of the docs appear to have been used already for v0.4.0 unless that was done manually with a agent again.
This is additive to the notes/releases and adds the image artifacts. |
I made the changes for this, though now there are additions to |
3683e55 to
30db292
Compare
f3c835c to
6c75331
Compare
|
deferring to ben's review on this |
There was a problem hiding this comment.
I gave some extra instructions to my review helper given the security sensitivity here.
I audited the findings and read through it myself (as usual).
Thanks for working on this.
One other question we should double check: are the right set of images included, for things we plan to move out of the repo, maybe we specially exclude them temporarily.
Also: binaries? We can attach them to the draft (kubectl-ate at least?), probably cross-compiled + .sha256 sidecar for validating with shasum like tools
Amends the release workflow to include a build and publish step for the system artifacts required for consumers.
Publish releases implemented in golang: No Make, no bash additions. Use the official substrate name in examples from README.md. Splits the release cutting and labeling docs.
- Fixes the module path to pin to github repo source. - Updates the list of images to release. - Fixes the overwrite naming issue and doc formatting. - Pins the registry:3 to a SHA. - Updates the documentation on the credential provider to match across all docs. - fixes ref matching on publish and release
Adds build and publish for kubectl-ate and align publish with `kustomize` and `helm` plan by removing the `ate-setup --image-dir` installation path for installing from pre-built images.
Removes the AGENTS.md entry directing agents not to use inline bash
5c86ddc to
ebfc92d
Compare
|
Addressed comments - rebased onto upstream/main |
Benjamin Elder (BenTheElder)
left a comment
There was a problem hiding this comment.
FYI protect-tags has not been in enforcing mode, it is now, but that means 0.6.0 didn't go through it.
| } | ||
| ref, err := p.Pin(ctx, image) | ||
| if err != nil { | ||
| // The registry may hold an image releases do not publish, so it is |
There was a problem hiding this comment.
🤖 nit 🟢 – Pin's doc comment at line 79 still says a release publishes envoy-dataplane under the same repository and tag as the rest. Releases leave it out now, as dockerfileImage in steps/env.go and docs/dev/releasing.md say, so that part of the comment should go.
| // of releases until it is added here in review. A pre-built install still | ||
| // looks an unlisted image up in its registry; if the lookup fails, the error | ||
| // says releases do not publish it. | ||
| var released = []string{ |
There was a problem hiding this comment.
thanks
| - **Binaries**: `go build -o bin/kubectl-ate ./cmd/kubectl-ate`, `go build -o bin/ate-setup ./cmd/ate-setup` | ||
| - **Release images**: `VERSION=TAG go run ./cmd/ate-setup publish release-images --ko-docker-repo REPO` (every image a pre-built install needs, all tagged `TAG`, including the docker-built `envoy-dataplane`) | ||
| - **Release images**: `VERSION=TAG go run ./cmd/ate-setup publish release-images --ko-docker-repo REPO` (the release set, `images.ReleaseComponents`, all tagged `TAG`; see `docs/dev/releasing.md`) | ||
| - **Release binaries**: `VERSION=TAG go run ./cmd/ate-setup publish release-binaries --out DIR` (`kubectl-ate` for linux and darwin, amd64 and arm64, with `.sha256` files) |
There was a problem hiding this comment.
we ... might also need windows for kubectl-ate. that can be a follow-up, cc Eitan Yarmush (@EItanya)
There was a problem hiding this comment.
I agree - adding it now would have been a bunch more stuff in the review that I thought would be best to keep separate.
Part of agent-substrate#3217. The default atenet dataplane runs envoy-dataplane in the router and egress gateways, but releases do not publish it, so a pre-built install has to use `--atenet-dataplane=agentgateway` or build from source. agent-substrate#2112 dropped it because building Rust under QEMU was slow. agent-substrate#3152 made the Dockerfile cross-compile, so the release job now builds it with buildx and no QEMU. Released Dockerfile images are an allowlist (`releasedDockerfiles`) next to the ko allowlist. The release jobs pin BuildKit by digest. Release builds take each Dockerfile image's digest from buildx's `--metadata-file`, build Dockerfile images before the ko images, ignore `DOCKER_BUILD_FLAGS`, and label every image with its source, revision and version. With every image a default install deploys now published, the release notes give the install command. After the first release that includes it, an org owner has to make the new `ghcr.io/agent-substrate/substrate/envoy-dataplane` package public. ### Validation `release.yaml` from this branch, run in my fork at 619f0d9 with a throwaway tag `v0.0.0-envoytest.1`: - [`images-dry-run`](https://github.com/MushuEE/substrate/actions/runs/40562367990), dispatched with `dry_run_images`, built all eleven release images cold on `ubuntu-latest` and passed the platform and digest checks in 11m35s, well inside the 30-minute timeout. - The [tag push run](https://github.com/MushuEE/substrate/actions/runs/40561810523) created the draft release. The images job skips itself outside agent-substrate/substrate, as designed. Earlier, at bbd8614, I ran the same publish locally against a throwaway registry, as the `images-dry-run` job does: ```sh docker run -d --rm --name ate-release-test -p 5000:5000 \ registry:3@sha256:ddf754342cfc8acc51a56d5d0ab6af06826461864460636d8bd5c546dab2a7b8 printf '[registry."localhost:5000"]\n http = true\n' > /tmp/buildkitd.toml docker buildx create --name ate-release --driver docker-container \ --driver-opt network=host --buildkitd-config /tmp/buildkitd.toml --use VERSION=v0.0.0-local.1 KO_DEFAULTPLATFORMS=linux/amd64,linux/arm64 \ go run ./cmd/ate-setup publish release-images --ko-docker-repo localhost:5000/substrate docker buildx imagetools inspect localhost:5000/substrate/envoy-dataplane:v0.0.0-local.1 docker buildx imagetools inspect localhost:5000/substrate/envoy-dataplane:v0.0.0-local.1 \ --format '{{json (index .Image "linux/amd64").Config.Labels}}' ``` - buildx cross-compiled the Rust module (`linux/amd64->arm64`), with no QEMU. - The envoy-dataplane digest from `--metadata-file` matched what the tag resolved to. - envoy-dataplane had `linux/amd64` and `linux/arm64`, and the `source` and `revision` labels. - A rerun built nothing. That run showed envoy-dataplane inheriting `org.opencontainers.image.version: "22.04"` from the Ubuntu base of `envoyproxy/envoy`. Release images now set that label to the release tag. AI use: I used Claude Code to trace the code agent-substrate#2112 removed, draft the change, address review comments, and script the local and fork validation. I reviewed the diff and ran the ate-setup tests and the validation above. - [x] Tests pass - [x] Docs updated --------- Signed-off-by: Grant McCloskey <[email protected]>
Amends the release workflow to include a build and publish step for the system artifacts required for consumers.
Part of #1871