Skip to content

runtime: build fails when run via QEMU for linux/amd64 running on linux/arm64 #69255

Closed

Description

@myitcv

Go version

go version go1.23.0 linux/arm64

Output of go env in your module/workspace:

$ go env
GO111MODULE=''
GOARCH='arm64'
GOBIN=''
GOCACHE='/home/myitcv/.cache/go-build'
GOENV='/home/myitcv/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/myitcv/gostuff/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/myitcv/gostuff'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/home/myitcv/gos'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='local'
GOTOOLDIR='/home/myitcv/gos/pkg/tool/linux_arm64'
GOVCS=''
GOVERSION='go1.23.0'
GODEBUG=''
GOTELEMETRY='on'
GOTELEMETRYDIR='/home/myitcv/.config/go/telemetry'
GCCGO='gccgo'
GOARM64='v8.0'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/home/myitcv/tmp/dockertests/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build810191502=/tmp/go-build -gno-record-gcc-switches'

What did you do?

Given:

-- Dockerfile --
FROM golang:1.23.0

WORKDIR /app
COPY . ./

RUN go build -o asdf ./blah

-- blah/main.go --
package main

func main() {

}
-- go.mod --
module mod.example

go 1.23.0

Running:

docker buildx build --platform linux/amd64 .

What did you see happen?

[+] Building 0.8s (8/8) FINISHED                                                                                                                                 docker-container:container-builder
 => [internal] load build definition from Dockerfile                                                                                                                                           0.0s
 => => transferring dockerfile: 110B                                                                                                                                                           0.0s
 => [internal] load metadata for docker.io/library/golang:1.23.0                                                                                                                               0.4s
 => [internal] load .dockerignore                                                                                                                                                              0.0s
 => => transferring context: 2B                                                                                                                                                                0.0s
 => [internal] load build context                                                                                                                                                              0.0s
 => => transferring context: 271B                                                                                                                                                              0.0s
 => CACHED [1/4] FROM docker.io/library/golang:1.23.0@sha256:613a108a4a4b1dfb6923305db791a19d088f77632317cfc3446825c54fb862cd                                                                  0.0s
 => => resolve docker.io/library/golang:1.23.0@sha256:613a108a4a4b1dfb6923305db791a19d088f77632317cfc3446825c54fb862cd                                                                         0.0s
 => [2/4] WORKDIR /app                                                                                                                                                                         0.0s
 => [3/4] COPY . ./                                                                                                                                                                            0.0s
 => ERROR [4/4] RUN go build -o asdf ./blah                                                                                                                                                    0.3s
------
 > [4/4] RUN go build -o asdf ./blah:
0.268 runtime: lfstack.push invalid packing: node=0xffffa45142c0 cnt=0x1 packed=0xffffa45142c00001 -> node=0xffffffffa45142c0
0.268 fatal error: lfstack.push
0.270
0.270 runtime stack:
0.270 runtime.throw({0xaf644d?, 0x0?})
0.271   runtime/panic.go:1067 +0x48 fp=0xc000231f08 sp=0xc000231ed8 pc=0x471228
0.271 runtime.(*lfstack).push(0xffffa45040b8?, 0xc0005841c0?)
0.271   runtime/lfstack.go:29 +0x125 fp=0xc000231f48 sp=0xc000231f08 pc=0x40ef65
0.271 runtime.(*spanSetBlockAlloc).free(...)
0.271   runtime/mspanset.go:322
0.271 runtime.(*spanSet).reset(0xfe7680)
0.271   runtime/mspanset.go:264 +0x79 fp=0xc000231f78 sp=0xc000231f48 pc=0x433559
0.271 runtime.finishsweep_m()
0.272   runtime/mgcsweep.go:257 +0x8d fp=0xc000231fb8 sp=0xc000231f78 pc=0x4263ad
0.272 runtime.gcStart.func2()
0.272   runtime/mgc.go:702 +0xf fp=0xc000231fc8 sp=0xc000231fb8 pc=0x46996f
0.272 runtime.systemstack(0x0)
0.272   runtime/asm_amd64.s:514 +0x4a fp=0xc000231fd8 sp=0xc000231fc8 pc=0x4773ca
...

My setup here is my host machine is linux/arm64, Qemu installed, following the approach described at https://docs.docker.com/build/building/multi-platform/#qemu, to build for linux/amd64.

This has definitely worked in the past which leads me to suggest that something other than Go has changed/been broken here. However I note the virtually identical call stack reported in #54104 hence raising here in the first instance.

What did you expect to see?

Successful run of docker build.

Activity

  1. dmitshur commented on Sep 4, 2024

    @dmitshur
    Member

    Do you think this is this similar or related to issue #68976? (It wasn't listed in the comment above, but it feels similar from a quick initial look.)

    CC @prattmic, @matloob.

  2. added
    NeedsInvestigationSomeone must examine and confirm this is a valid issue and not a duplicate of an existing one.
    compiler/runtimeIssues related to the Go compiler and/or runtime.
    on Sep 4, 2024
  3. added this to the Backlog milestone on Sep 4, 2024
  4. changed the title [-]cmd/go: build fails when run via QEMU for linux/amd64 running on linux/arm64[/-] [+]cmd/go, runtime: build fails when run via QEMU for linux/amd64 running on linux/arm64[/+] on Sep 4, 2024
  5. myitcv commented on Sep 4, 2024

    @myitcv
    MemberAuthor

    Do you think this is this similar or related to issue #68976?

    I don't know I'm afraid. That said the stack trace and symptoms seem quite different. I will however defer to @prattmic

  6. prattmic commented on Sep 4, 2024

    @prattmic
    Member

    I agree, it looks quite different. #68976 is very specific to pidfd use in os/syscall. This looks like some form of corruption.

    Do you know if this build is running a full Linux kernel in a VM, or using QEMU user mode Linux emulation?

  7. prattmic commented on Sep 4, 2024

    @prattmic
    Member
    0.268 runtime: lfstack.push invalid packing: node=0xffffa45142c0 cnt=0x1 packed=0xffffa45142c00001 -> node=0xffffffffa45142c0
    

    Notice

    node=0xffffa45142c0       # before
    node=0xffffffffa45142c0   # after
    

    This seems like a sign extension issue when right shifting the packed value (See https://cs.opensource.google/go/go/+/master:src/runtime/lfstack.go;l=26-30, specifically lfstackUnpack).

    I could imagine this being a code generation issue, or an issue in QEMU instruction emulation.

    cc @golang/compiler

  8. changed the title [-]cmd/go, runtime: build fails when run via QEMU for linux/amd64 running on linux/arm64[/-] [+]runtime: build fails when run via QEMU for linux/amd64 running on linux/arm64[/+] on Sep 4, 2024
  9. prattmic commented on Sep 4, 2024

    @prattmic
    Member

    Does the same issue occur on Go 1.22?

  10. myitcv commented on Sep 4, 2024

    @myitcv
    MemberAuthor

    Does the same issue occur on Go 1.22?

    Yes. Indeed similar looking stacks for 1.21.13, 1.22.6, 1.23.0. Confirmed via:

    cat <<EOD > template.txtar
    -- Dockerfile --
    FROM golang:$GOVERSION
    
    WORKDIR /app
    COPY . ./
    
    RUN go build -o asdf ./blah
    
    -- blah/main.go --
    package main
    
    func main() {
    
    }
    -- go.mod --
    module mod.example
    
    go $GOVERSION
    EOD
    for i in 1.23.0 1.22.6 1.21.13
    do
            mkdir $i
            pushd $i > /dev/null
    cat ../template.txtar | GOVERSION=$i envsubst | txtar-x
    docker buildx build --platform linux/amd64 . > output 2>&1
    popd > /dev/null
    done
    cat */output
    
  11. myitcv commented on Sep 4, 2024

    @myitcv
    MemberAuthor

    I'm miles out of my depth here, but in case this is useful:

    $ qemu-amd64-static --version
    qemu-x86_64 version 9.0.2 (Debian 1:9.0.2+ds-2+b1)
    Copyright (c) 2003-2024 Fabrice Bellard and the QEMU Project developers
    
  12. myitcv commented on Sep 4, 2024

    @myitcv
    MemberAuthor

    ... but just to be super clear, I'm doing this via Docker:

    https://docs.docker.com/build/building/multi-platform/#qemu

    (so I'm actually unsure whether the host system qemu is used or not)

  13. 63 remaining items

  14. ianlancetaylor commented on Oct 22, 2025

    @ianlancetaylor
    Contributor

    @lczyk Are you saying that Go 1.4 for linux/amd64 does not run correctly under QEMU when running on linux/arm64?

    Please open a new issue for that. We can add a patch to the Go 1.4 release branch. Thanks.

  15. cherrymui commented on Oct 22, 2025

    @cherrymui
    Member

    This also means VMA addresses are handled by the arm64 MMU. (I.e., it's not emulating a full amd64 kernel or an amd64 MMU.)

    If QEMU user space emulation can provide an address that is not possible on any physical hardware, it sounds like a bug in QEMU. From the comment above, this looks like the case. They should fix it.

    We can work around it on the Go side to some degree (like on tip and in 1.25), but going back all the way to Go 1.4 seems too much.

  16. pwillis-oi commented on Oct 23, 2025

    @pwillis-oi

    Thanks very much @mdempsky for the fixes provided here. I can confirm that on an M3 w/Sequoia v15.7.1, and Podman Desktop v1.22.1, with default machine (with Rosetta enabled, and Apple Hypervisor selected), the crashes still happen; but rebuilding the machine with the 5.4 OS image fixes the crashes. (In my case it was Terraform crashing.) (Podman command to make the "fixed" machine: podman machine init --image docker://quay.io/podman/machine-os:5.4 --disk-size 100 --cpus 4 --memory 4096 --now)

  17. locked and limited conversation to collaborators on Oct 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

FrozenDueToAgeNeedsFixThe path to resolution is known, but the work has not been done.OS-Linuxcompiler/runtimeIssues related to the Go compiler and/or runtime.

Type

No type

Projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions