When building artifacts for another architecture, it is sometimes useful to use qemu-user emulation. Although this isn't very performant for compilation steps where cross-compilation is preferred, it seems to work reasonably well for tasks like installing packages.

Qemu is usually invoked through binfmt_misc linux kernel feature, where you need to register emulator binary with the kernel. Docker Desktop ships with these registrations already made and usually these registrations can be done by just invoking docker run command.

That being said, we still get a fair amount of feedback where users are in trouble with this step. As we know (usually) what arch a build needs, if the arch is already supported by the system (https://github.com/moby/buildkit/tree/master/util/binfmt_misc) , and how to install it, we could make this experience better and allow automatic setup (given user allows that).

An obvious alternative would be to say that a wrapper layer(for example buildx) should do this instead, and BuildKit is too low level. Unfortunately, the wrapper doesn't know when emulation support is needed. Just starting a build with --platform=arm64 doesn't mean that qemu was required as the build might be just doing cross-compilation. Instead, we would need to fail the build with a typed error, and wrapper would need to restart it after qemu registration. This looks complicated and inefficient.

If we want to do qemu registration directly in BuildKit there are two possibilities.

One of them is to take a registration image (eg. https://github.com/tonistiigi/binfmt/blob/master/binfmt/Dockerfile ) in the BuildKit configuration. When BuildKit detects that emulation support is needed, it will execute the image, registering a specific architecture. This needs to be opt-in as registration in kernel is privileged and shouldn't come as a surprise. In a wrapper like buildx we can make the opt-in step simpler.

Another way would be to avoid depending on binfmt feature at all, and instead when execution through an emulator is needed, mount qemu-user binary into the container and modify the entrypoint to go through qemu.

Unfortunately, it looks that this does not work atm for exec(2) and qemu-user expects that binfmt will relaunch the emulator.

root@do:~# docker create -t arm64v8/alpine /usr/bin/qemu-aarch64 /bin/ls
1366a8a8b9a60a559c0d43eae26a595740a27af622060ee96476e9ba3835316a
root@do:~# docker cp qemu-aarch64 13:/usr/bin/
root@do:~# docker start -a 13
bin    etc    lib    mnt    proc   run    srv    tmp    var
dev    home   media  opt    root   sbin   sys    usr
root@do:~# docker create -t arm64v8/alpine /usr/bin/qemu-aarch64 /bin/ash -c "ls"
487004f5bd761977adf898b4f5bcbd660d06313ff6655b30dc01b3ce78930526
root@do:~# docker cp qemu-aarch64 48:/usr/bin/
root@do:~# docker start -a 48
/bin/ash: ls: Exec format error

I believe this might be fixable, though, with some modification to qemu. https://github.com/qemu/qemu/blob/06539ebc76b8625587aa78d646a9d8d5fddf84f3/linux-user/main.c#L800 . The benefit of this method is that we don't need to mess with kernel and therefore ask the user to opt-in. Note that they are not quite equivalent though, as binfmt allows mixing architectures in a single container, but that doesn't look like an important use-case.