Repository navigation
envoy: cross compile instead of compiling under slow emulation - #3152
Eitan Yarmush (EItanya) merged 1 commit into
Conversation
|
Eitan Yarmush (@EItanya) I don't know if this is in scope or punted to post-GA, but these builds were wasting a ton of time when e2e testing. Cross-compile is much faster. It's not really security or reliability etc though so ... perhaps we have to wait. |
Davanum Srinivas (dims)
left a comment
There was a problem hiding this comment.
LGTM
tested on an amd64 host ... the cross-built arm64 module loads in the arm64 Envoy (validate OK, control fails) and exports the same 172 symbols as the native build. Nits: a README sentence, a commit body, CC_ next to the linker var.
No I definitely think this is worth it, speeding up build times always feels good |
that makes sense, but this was also costing me to compile from arm mac => amd64 cloud test cluster I think there's a lot of value in making cross builds faster without multi-host infra, though releases and CI might still consider using multi arch infra. |
Part of agent-substrate#3217. The default atenet dataplane runs envoy-dataplane in the router and egress gateways, but releases do not publish it, so a pre-built install has to use `--atenet-dataplane=agentgateway` or build from source. agent-substrate#2112 dropped it because building Rust under QEMU was slow. agent-substrate#3152 made the Dockerfile cross-compile, so the release job now builds it with buildx and no QEMU. Released Dockerfile images are an allowlist (`releasedDockerfiles`) next to the ko allowlist. The release jobs pin BuildKit by digest. Release builds take each Dockerfile image's digest from buildx's `--metadata-file`, build Dockerfile images before the ko images, ignore `DOCKER_BUILD_FLAGS`, and label every image with its source, revision and version. With every image a default install deploys now published, the release notes give the install command. After the first release that includes it, an org owner has to make the new `ghcr.io/agent-substrate/substrate/envoy-dataplane` package public. ### Validation `release.yaml` from this branch, run in my fork at 619f0d9 with a throwaway tag `v0.0.0-envoytest.1`: - [`images-dry-run`](https://github.com/MushuEE/substrate/actions/runs/40562367990), dispatched with `dry_run_images`, built all eleven release images cold on `ubuntu-latest` and passed the platform and digest checks in 11m35s, well inside the 30-minute timeout. - The [tag push run](https://github.com/MushuEE/substrate/actions/runs/40561810523) created the draft release. The images job skips itself outside agent-substrate/substrate, as designed. Earlier, at bbd8614, I ran the same publish locally against a throwaway registry, as the `images-dry-run` job does: ```sh docker run -d --rm --name ate-release-test -p 5000:5000 \ registry:3@sha256:ddf754342cfc8acc51a56d5d0ab6af06826461864460636d8bd5c546dab2a7b8 printf '[registry."localhost:5000"]\n http = true\n' > /tmp/buildkitd.toml docker buildx create --name ate-release --driver docker-container \ --driver-opt network=host --buildkitd-config /tmp/buildkitd.toml --use VERSION=v0.0.0-local.1 KO_DEFAULTPLATFORMS=linux/amd64,linux/arm64 \ go run ./cmd/ate-setup publish release-images --ko-docker-repo localhost:5000/substrate docker buildx imagetools inspect localhost:5000/substrate/envoy-dataplane:v0.0.0-local.1 docker buildx imagetools inspect localhost:5000/substrate/envoy-dataplane:v0.0.0-local.1 \ --format '{{json (index .Image "linux/amd64").Config.Labels}}' ``` - buildx cross-compiled the Rust module (`linux/amd64->arm64`), with no QEMU. - The envoy-dataplane digest from `--metadata-file` matched what the tag resolved to. - envoy-dataplane had `linux/amd64` and `linux/arm64`, and the `source` and `revision` labels. - A rerun built nothing. That run showed envoy-dataplane inheriting `org.opencontainers.image.version: "22.04"` from the Ubuntu base of `envoyproxy/envoy`. Release images now set that label to the release tag. AI use: I used Claude Code to trace the code agent-substrate#2112 removed, draft the change, address review comments, and script the local and fork validation. I reviewed the diff and ran the ate-setup tests and the validation above. - [x] Tests pass - [x] Docs updated --------- Signed-off-by: Grant McCloskey <[email protected]>
Why is this needed? (Or link the issue. Open one first for features, API, install or CI changes.)
Upgrading to current Ubuntu in limactl (#3149) on macOS led to emulation errors with rosetta.
Using qemu is super slow.
We should just cross-compile, pretty much always, that's a key benefit of buildx and multi-stage builds.
It's WAY faster. Compiling under emulation is major kludge.
AI use: Yes, I let it draft this. Then hand edited, bash and dockerfiles I'm quite familiar with :-)