Skip to content

Releases publish the envoy-dataplane image - #3252

Merged
Tim Hockin (thockin) merged 7 commits into
agent-substrate:mainfrom
MushuEE:build/release-envoy-dataplane
Oct 27, 2026
Merged

Tim Hockin (thockin) merged 7 commits into
agent-substrate:mainfrom
MushuEE:build/release-envoy-dataplane

Conversation

@MushuEE

@MushuEE Grant McCloskey (MushuEE) commented Oct 26, 2026 •

Copy link
Copy Markdown
Collaborator

Part of #3217. The default atenet dataplane runs envoy-dataplane in the router and egress gateways, but releases do not publish it, so a pre-built install has to use --atenet-dataplane=agentgateway or build from source. #2112 dropped it because building Rust under QEMU was slow. #3152 made the Dockerfile cross-compile, so the release job now builds it with buildx and no QEMU.

Released Dockerfile images are an allowlist (releasedDockerfiles) next to the ko allowlist. The release jobs pin BuildKit by digest. Release builds take each Dockerfile image's digest from buildx's --metadata-file, build Dockerfile images before the ko images, ignore DOCKER_BUILD_FLAGS, and label every image with its source, revision and version. With every image a default install deploys now published, the release notes give the install command.

After the first release that includes it, an org owner has to make the new ghcr.io/agent-substrate/substrate/envoy-dataplane package public.

Validation

release.yaml from this branch, run in my fork at 619f0d9 with a throwaway tag v0.0.0-envoytest.1:

  • images-dry-run, dispatched with dry_run_images, built all eleven release images cold on ubuntu-latest and passed the platform and digest checks in 11m35s, well inside the 30-minute timeout.
  • The tag push run created the draft release. The images job skips itself outside agent-substrate/substrate, as designed.

Earlier, at bbd8614, I ran the same publish locally against a throwaway registry, as the images-dry-run job does:

docker run -d --rm --name ate-release-test -p 5000:5000 \
  registry:3@sha256:ddf754342cfc8acc51a56d5d0ab6af06826461864460636d8bd5c546dab2a7b8
printf '[registry."localhost:5000"]\n  http = true\n' > /tmp/buildkitd.toml
docker buildx create --name ate-release --driver docker-container \
  --driver-opt network=host --buildkitd-config /tmp/buildkitd.toml --use
VERSION=v0.0.0-local.1 KO_DEFAULTPLATFORMS=linux/amd64,linux/arm64 \
  go run ./cmd/ate-setup publish release-images --ko-docker-repo localhost:5000/substrate
docker buildx imagetools inspect localhost:5000/substrate/envoy-dataplane:v0.0.0-local.1
docker buildx imagetools inspect localhost:5000/substrate/envoy-dataplane:v0.0.0-local.1 \
  --format '{{json (index .Image "linux/amd64").Config.Labels}}'
  • buildx cross-compiled the Rust module (linux/amd64->arm64), with no QEMU.
  • The envoy-dataplane digest from --metadata-file matched what the tag resolved to.
  • envoy-dataplane had linux/amd64 and linux/arm64, and the source and revision labels.
  • A rerun built nothing.

That run showed envoy-dataplane inheriting org.opencontainers.image.version: "22.04" from the Ubuntu base of envoyproxy/envoy. Release images now set that label to the release tag.

AI use: I used Claude Code to trace the code #2112 removed, draft the change, address review comments, and script the local and fork validation. I reviewed the diff and ran the ate-setup tests and the validation above.

  • Tests pass
  • Docs updated

@MushuEE
Grant McCloskey (MushuEE) marked this pull request as ready for review October 27, 2026 02:37
The router and egress gateways run envoy-dataplane by default, so an
install from a release could not use the default dataplane. The
Dockerfile now cross-compiles the Rust module, so the release job builds
it with buildx and no QEMU.

Signed-off-by: Grant McCloskey <[email protected]>
releasedDockerfiles in images.go names the Dockerfile images a release
publishes, so adding one is a one-line change like adding a ko image.
The installer and the release build share images.EnvoyDataplane.

Signed-off-by: Grant McCloskey <[email protected]>
PublishDockerfileImage read the digest back from the tag after pushing,
so the release check that each tag names the pushed digest compared the
tag with itself. It now reads containerimage.digest from the file
--metadata-file writes.

Also update comments and help text that still said releases leave out
envoy-dataplane, or that only ko reads the ghcr.io login.

Signed-off-by: Grant McCloskey <[email protected]>
With envoy-dataplane and the Kubernetes Secrets credential provider
published, releases leave out only the demos and the benchmarking
fakes, which an install deploys only when asked to.

Signed-off-by: Grant McCloskey <[email protected]>
@annapendleton Anna Pendleton (annapendleton) added this to the M3 milestone Oct 27, 2026
@annapendleton Anna Pendleton (annapendleton) added kind/bug Something isn't working / bugfixes prio/P0 Highest priority / required for next milestone area/dev-infra labels Oct 27, 2026
@thockin Tim Hockin (thockin) self-assigned this Oct 27, 2026

@stshrive Steven Shriver (stshrive) left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Agent-assisted adversarial review focused on CI failures and release-pipeline security. Severity: 🔴 Blocker, 🟡 Should Fix, 🟢 Nit. Checked without findings: merges cleanly onto main, the setup-buildx-action SHA matches the existing pin in pr-workflow, the Dockerfile base images are digest-pinned, CheckPublished ignores buildx attestation entries, and package visibility is already covered in docs/dev/releasing.md.

Comment thread .github/workflows/release.yaml Resolved
Comment thread .github/workflows/release.yaml Resolved
Comment thread cmd/ate-setup/internal/steps/publish.go Resolved
Comment thread cmd/ate-setup/internal/steps/publish.go Resolved
Comment thread cmd/ate-setup/internal/steps/env.go Resolved
Comment thread docs/dev/releasing.md Resolved
The images job builds envoy-dataplane with the ghcr.io login, so its
BuildKit image is now pinned by digest, and the dry run uses the same
one. Release builds ignore DOCKER_BUILD_FLAGS, so they restore no cache,
and build the Dockerfile images before the ko images, so a failed envoy
build publishes nothing under the tag.

The error for a missing envoy-dataplane again suggests
--atenet-dataplane=agentgateway, since older releases, mirrors, and a
package that is still private have no readable copy.

Signed-off-by: Grant McCloskey <[email protected]>

@stshrive Steven Shriver (stshrive) left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Agent-assisted functional review: can a user install a release on a cluster and run a demo? With envoy-dataplane published, every image a default install deploys is either a release image or a public, digest-pinned third-party image, so ate-setup deploy ate-system --image-repo ghcr.io/agent-substrate/substrate --image-tag <tag> from a checkout at the tag should work, --kind included.

🟡 Should Fix: cmd/ate-setup/README.md:29-31, under "Installing pre-built images", still says: "The images a release publishes to ghcr.io/agent-substrate/substrate are not enough on their own". This PR updated the matching text in commands.md, releasing.md, substrate-install.yaml and draft-release.sh, but this file isn't in the diff, and it is the main doc for pre-built installs. It should say what commands.md:85-88 now says.

Comment thread hack/release/draft-release.sh Resolved
Comment thread docs/dev/releasing.md
Both lists are allowlists in `images.go`: `released` for the ko images and `releasedDockerfiles` for the images built from a Dockerfile. A new image is not published until a pull request adds it there, so publishing an image is always an explicit, reviewed change.

It leaves out `envoy-dataplane`, the demos, and the benchmarking fakes. The release images are therefore not a complete install of the default configuration, and the notes give no install command: an install has to take the images releases leave out from another registry, or build them from source. `ate-setup --image-repo` looks those images up like any other; if one is missing, the error says releases do not publish it.
It leaves out the demos and the benchmarking fakes, which an install deploys only when asked to. An install that needs them has to take them from another registry, or build them from source. `ate-setup --image-repo` looks those images up like any other; if one is missing, the error says releases do not publish it.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 🟡 Should Fix: "An install that needs them has to take them from another registry, or build them from source" doesn't tell a user who installed a release how to run a demo. Six of the seven demos need an image releases don't publish, and the error they hit (prebuilt.go:136) says to publish it to ghcr.io/agent-substrate/substrate themselves, which they can't, or to drop --image-repo, without saying the system install can keep it. The tool already supports the mixed flow: deploy demo resolves only the demo's WorkerPool and ActorTemplates, so the control plane can come from the release while the demo is built from source. Please document it next to the release-install command, e.g.:

hack/create-kind-cluster.sh
go run ./cmd/ate-setup deploy ate-system --kind --credential-provider='{"name":"k8s.io"}' \
  --image-repo ghcr.io/agent-substrate/substrate --image-tag vX.Y.Z
# Demo images are not released; build them from this checkout (at vX.Y.Z) into the kind registry.
go run ./cmd/ate-setup deploy demo demo-counter --kind

Also say that jupyter needs no build, that the demo step needs Go, ko and docker and must not inherit images.repo or ATE_IMAGE_REPO, and that outside kind it needs KO_DOCKER_REPO set to a registry the cluster can pull from. Updating the prebuilt.go:136 hint to point at this would close the loop. Fine as a follow-up under #3217 if you'd rather keep this PR focused.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would like to do the release-plus-demo flow and the prebuilt.go hint as a follow-up under #3217, so this PR stays on publishing envoy-dataplane.

@MushuEE

Copy link
Copy Markdown
Collaborator Author

Steven Shriver (@stshrive) thanks for review, coving the comments, some things I defered because I wasn't sure they were needed but no harm in covering it all now.

I will do a mock release in my own repo to validate the workflow

envoyproxy/envoy sets org.opencontainers.image.version to its Ubuntu
release, which envoy-dataplane inherited. Every release image now
carries the release tag in that label.

Also fix the ate-setup README, which still said release images are not
enough for an install.

Signed-off-by: Grant McCloskey <[email protected]>
With every image a default install deploys now published, the notes
show how to install a release from a checkout of its tag.

The pre-built install examples in the ate-setup docs now pass
--ingress-allowed-spiffe-ids, which the envoy router requires, and say
that running ate-setup still needs Go.

Signed-off-by: Grant McCloskey <[email protected]>
@MushuEE

Copy link
Copy Markdown
Collaborator Author

Steven Shriver (@stshrive) the README text from your functional review ("not enough on their own") is fixed in 619f0d9.

@MushuEE

Copy link
Copy Markdown
Collaborator Author

Tim Hockin (@thockin) This should be in a valid state right now. I ran a mock release off my fork with fake tags.

  localhost:5000/substrate/envoy-dataplane@sha256:de213e60…
  localhost:5000/substrate/kubernetes-secrets:v0.0.0-envoytest.1@sha256:30283eb0…
  localhost:5000/substrate/ateapi:v0.0.0-envoytest.1@sha256:b4dd166b…
  … (eleven in total)

@thockin
Tim Hockin (thockin) added this pull request to the merge queue Oct 27, 2026
Merged via the queue into agent-substrate:main with commit a54dc8d Oct 27, 2026
5 checks passed
@MushuEE
Grant McCloskey (MushuEE) deleted the build/release-envoy-dataplane branch November 1, 2026 21:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/dev-infra kind/bug Something isn't working / bugfixes milestone-ack prio/P0 Highest priority / required for next milestone

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants