qemu-aarch64 nondeterministic crashes

Contributing guidelines

I've read the contributing guidelines and wholeheartedly agree

I've found a bug, and:

The documentation does not mention anything about my problem
There are no open or closed issues that are related to my problem

Description

wanted to report that i recently switched a build pipeline running on ubuntu to the upstream ubuntu qemu-user-static. the upstream ubuntu qemu is further along on version number than this action btw, so that this saves people time.

we were witnessing a problem where qemu-aarch64 installed by this action would crash nondeterministically on github's x86-64 runners as well as on self-hosted runners using 12th gen and 13th gen intel systems. we dont have time to triage and diagnose the issue, because it's non deterministic its not clear what the underlying cause is.

Expected behaviour

qemu-aarch64 doesnt crash at random

Actual behaviour

qemu-aarch64 crashed at random

Repository URL

No response

Workflow run URL

No response

YAML workflow

n/a

Workflow logs

No response

BuildKit logs

Additional info

No response

We have a Ruby app depends on quite a bunch of "native" extensions packages - that are packages having to be built through gcc

Targeting linux/amd64,linux/arm64

we have consistent Gem::Ext::BuildError: ERROR: Failed to build gem native extension. caused by gcc: internal compiler error: Segmentation fault signal terminated program cc1
but that will affect anything built by gcc or similar (nodejs packages, etc.).

Similar thing happened with a Go project needed some stuff built with gcc.

I was suspecting this issue, but it turned out another action int128/[email protected] is rogue, I am still puzzled how it could brake stuff in such way int128/create-ecr-repository-action@v1.342.0...v1.348.0

UPDATE: Still investigating, no clarity ... Issue is happening sporadically, combined with caching it is really difficult to get a good grip on what is the cause and what is a fix.

the qemu that is shipped by this action seemed broken the last time i used it. i recommend trying to use your base images qemu instead.

Indeed there are some very strange and sporadic qemu@v7 issues in combination with some other unidentified changes / inconsistencies (buildx, gh-actions, runners or whatever).
I have managed to reproduce locally.

Qemu v7 used to work well for quite some time but not anymore, since ~1-2 weeks it started resulting in sporadic but stubborn segmentation faults in gcc and other executables.

Having build cache lead this to be very difficult to identify and fix, so I am still not sure how stable the fix is, but
what seems to fix it so far is using qemu@v8 through:

      uses: docker/[email protected]
      with:
        image: tonistiigi/binfmt:qemu-v8.1.5
        platforms: ${{ inputs.platforms }}

As this is qemu related can you open an issue on https://github.com/tonistiigi/binfmt? Thanks

there is related ticket already tonistiigi/binfmt#215

@crazy-max

This does not only affect aarch64 running on x86_64 but also at least s390x & ppc64le.
All repos I have seen using setup-qemu-action with ubuntu-latest/ubuntu-24.04 started failing with the update of the runner this week.

@crazy-max, I understand the issue is upstream so not asking to re-open the issue but could it be at least pinned in order for users of the action to easily find the issue and workaround while waiting for a fix upstream ?

For aarch64, another solution is to switch from emulated to native with the availability of ubuntu-24.04-arm runners: https://github.blog/changelog/2025-01-16-linux-arm64-hosted-runners-now-available-for-free-in-public-repositories-public-preview/

ci(fix): use `tonistiigi/binfmt:qemu-v8.1.5` image for qemu

@mayeut

@mayeut or anyone else would you have an example for how to use a runner to handle an architecture specific part of a docker bake?

Update main.yml

Update qemu to fix aarch64 wheel action (#1056)

publish: explicitly use qemu 8

ci: setup more recent qemu before executing the build

ci: remove dependency on run-on-arch-action

Work around random build failures when using QEMU 7.x

CI: aarch64 build workaround

fixing-raspberry-64bit-segfault-build (#3084)

fix .github/workflows/docker-build.yml

I'm sorry for this comment on a closed issue. QEMU v8.1.5 seem to not address the problem with arm64v8, at least for the run I did. It keeps segfaulting when running some commands.
At the end reverting to ubuntu 22.04 fixed the issue, but it doesn't sound like a long term solution.

Merge pull request #1530 from bettio/run-qemu-on-ubuntu-22.04

Build: Use older Ubuntu image for legacy builds to see if they fix a …

ci: remove dependency on run-on-arch-action

Run on any architecture: prebuilt images, an early guard, and two rea…

Navigation Menu

Repository navigation

qemu-aarch64 nondeterministic crashes #188

Contributing guidelines

I've found a bug, and:

Description

Expected behaviour

Actual behaviour

Repository URL

Workflow run URL

YAML workflow

Workflow logs

BuildKit logs

Additional info

3 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

qemu-aarch64 nondeterministic crashes #188

Description

Contributing guidelines

I've found a bug, and:

Description

Expected behaviour

Actual behaviour

Repository URL

Workflow run URL

YAML workflow

Workflow logs

BuildKit logs

Additional info

Activity

smoke commented on Jan 20, 2025

lts-rad commented on Jan 20, 2025

smoke commented on Jan 21, 2025

crazy-max commented on Jan 21, 2025

smoke commented on Jan 21, 2025

mayeut commented on Jan 25, 2025

lts-rad commented on Jan 25, 2025

3 remaining items

bettio commented on Feb 10, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions