Skip to content

envoy: cross compile instead of compiling under slow emulation - #3152

Merged
Eitan Yarmush (EItanya) merged 1 commit into
agent-substrate:mainfrom
BenTheElder:envoy-dataplane-cross-compile
Oct 23, 2026
Merged

Eitan Yarmush (EItanya) merged 1 commit into
agent-substrate:mainfrom
BenTheElder:envoy-dataplane-cross-compile

Conversation

@BenTheElder

Copy link
Copy Markdown
Collaborator

Why is this needed? (Or link the issue. Open one first for features, API, install or CI changes.)

Upgrading to current Ubuntu in limactl (#3149) on macOS led to emulation errors with rosetta.
Using qemu is super slow.

We should just cross-compile, pretty much always, that's a key benefit of buildx and multi-stage builds.
It's WAY faster. Compiling under emulation is major kludge.

AI use: Yes, I let it draft this. Then hand edited, bash and dockerfiles I'm quite familiar with :-)

  • Tests pass
  • Docs updated

@BenTheElder

Copy link
Copy Markdown
Collaborator Author

Eitan Yarmush (@EItanya) I don't know if this is in scope or punted to post-GA, but these builds were wasting a ton of time when e2e testing. Cross-compile is much faster. It's not really security or reliability etc though so ... perhaps we have to wait.

@dims Davanum Srinivas (dims) left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

tested on an amd64 host ... the cross-built arm64 module loads in the arm64 Envoy (validate OK, control fails) and exports the same 172 symbols as the native build. Nits: a README sentence, a commit body, CC_ next to the linker var.

@EItanya

Eitan Yarmush (EItanya) commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

Eitan Yarmush (Eitan Yarmush (@EItanya)) I don't know if this is in scope or punted to post-GA, but these builds were wasting a ton of time when e2e testing. Cross-compile is much faster. It's not really security or reliability etc though so ... perhaps we have to wait.

No I definitely think this is worth it, speeding up build times always feels good

@EItanya Eitan Yarmush (EItanya) left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW, in other projects I've ended up using separate machines for each architecture and then combining them post-facto

@EItanya
Eitan Yarmush (EItanya) added this pull request to the merge queue Oct 23, 2026
Merged via the queue into agent-substrate:main with commit e6a974e Oct 23, 2026
15 checks passed
@BenTheElder

Copy link
Copy Markdown
Collaborator Author

FWIW, in other projects I've ended up using separate machines for each architecture and then combining them post-facto

that makes sense, but this was also costing me to compile from arm mac => amd64 cloud test cluster

I think there's a lot of value in making cross builds faster without multi-host infra, though releases and CI might still consider using multi arch infra.

@BenTheElder
Benjamin Elder (BenTheElder) deleted the envoy-dataplane-cross-compile branch October 24, 2026 08:27
igooch pushed a commit to igooch/substrate that referenced this pull request Oct 27, 2026
Part of agent-substrate#3217. The default atenet dataplane runs envoy-dataplane in the
router and egress gateways, but releases do not publish it, so a
pre-built install has to use `--atenet-dataplane=agentgateway` or build
from source. agent-substrate#2112 dropped it because building Rust under QEMU was slow.
agent-substrate#3152 made the Dockerfile cross-compile, so the release job now builds
it with buildx and no QEMU.

Released Dockerfile images are an allowlist (`releasedDockerfiles`) next
to the ko allowlist. The release jobs pin BuildKit by digest. Release
builds take each Dockerfile image's digest from buildx's
`--metadata-file`, build Dockerfile images before the ko images, ignore
`DOCKER_BUILD_FLAGS`, and label every image with its source, revision
and version. With every image a default install deploys now published,
the release notes give the install command.

After the first release that includes it, an org owner has to make the
new `ghcr.io/agent-substrate/substrate/envoy-dataplane` package public.

### Validation

`release.yaml` from this branch, run in my fork at 619f0d9 with a
throwaway tag `v0.0.0-envoytest.1`:

-
[`images-dry-run`](https://github.com/MushuEE/substrate/actions/runs/40562367990),
dispatched with `dry_run_images`, built all eleven release images cold
on `ubuntu-latest` and passed the platform and digest checks in 11m35s,
well inside the 30-minute timeout.
- The [tag push
run](https://github.com/MushuEE/substrate/actions/runs/40561810523)
created the draft release. The images job skips itself outside
agent-substrate/substrate, as designed.

Earlier, at bbd8614, I ran the same publish locally against a throwaway
registry, as the `images-dry-run` job does:

```sh
docker run -d --rm --name ate-release-test -p 5000:5000 \
  registry:3@sha256:ddf754342cfc8acc51a56d5d0ab6af06826461864460636d8bd5c546dab2a7b8
printf '[registry."localhost:5000"]\n  http = true\n' > /tmp/buildkitd.toml
docker buildx create --name ate-release --driver docker-container \
  --driver-opt network=host --buildkitd-config /tmp/buildkitd.toml --use
VERSION=v0.0.0-local.1 KO_DEFAULTPLATFORMS=linux/amd64,linux/arm64 \
  go run ./cmd/ate-setup publish release-images --ko-docker-repo localhost:5000/substrate
docker buildx imagetools inspect localhost:5000/substrate/envoy-dataplane:v0.0.0-local.1
docker buildx imagetools inspect localhost:5000/substrate/envoy-dataplane:v0.0.0-local.1 \
  --format '{{json (index .Image "linux/amd64").Config.Labels}}'
```

- buildx cross-compiled the Rust module (`linux/amd64->arm64`), with no
QEMU.
- The envoy-dataplane digest from `--metadata-file` matched what the tag
resolved to.
- envoy-dataplane had `linux/amd64` and `linux/arm64`, and the `source`
and `revision` labels.
- A rerun built nothing.

That run showed envoy-dataplane inheriting
`org.opencontainers.image.version: "22.04"` from the Ubuntu base of
`envoyproxy/envoy`. Release images now set that label to the release
tag.

AI use: I used Claude Code to trace the code agent-substrate#2112 removed, draft the
change, address review comments, and script the local and fork
validation. I reviewed the diff and ran the ate-setup tests and the
validation above.

- [x] Tests pass
- [x] Docs updated

---------

Signed-off-by: Grant McCloskey <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants