Repository navigation
Closed
Labels
FrozenDueToAgeNeedsFixThe path to resolution is known, but the work has not been done.The path to resolution is known, but the work has not been done.OS-Linuxcompiler/runtimeIssues related to the Go compiler and/or runtime.Issues related to the Go compiler and/or runtime.
Milestone
Description
Go version
go version go1.23.0 linux/arm64
Output of go env in your module/workspace:
$ go env
GO111MODULE=''
GOARCH='arm64'
GOBIN=''
GOCACHE='/home/myitcv/.cache/go-build'
GOENV='/home/myitcv/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/myitcv/gostuff/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/myitcv/gostuff'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/home/myitcv/gos'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='local'
GOTOOLDIR='/home/myitcv/gos/pkg/tool/linux_arm64'
GOVCS=''
GOVERSION='go1.23.0'
GODEBUG=''
GOTELEMETRY='on'
GOTELEMETRYDIR='/home/myitcv/.config/go/telemetry'
GCCGO='gccgo'
GOARM64='v8.0'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/home/myitcv/tmp/dockertests/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build810191502=/tmp/go-build -gno-record-gcc-switches'What did you do?
Given:
-- Dockerfile --
FROM golang:1.23.0
WORKDIR /app
COPY . ./
RUN go build -o asdf ./blah
-- blah/main.go --
package main
func main() {
}
-- go.mod --
module mod.example
go 1.23.0
Running:
docker buildx build --platform linux/amd64 .
What did you see happen?
[+] Building 0.8s (8/8) FINISHED docker-container:container-builder
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 110B 0.0s
=> [internal] load metadata for docker.io/library/golang:1.23.0 0.4s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 271B 0.0s
=> CACHED [1/4] FROM docker.io/library/golang:1.23.0@sha256:613a108a4a4b1dfb6923305db791a19d088f77632317cfc3446825c54fb862cd 0.0s
=> => resolve docker.io/library/golang:1.23.0@sha256:613a108a4a4b1dfb6923305db791a19d088f77632317cfc3446825c54fb862cd 0.0s
=> [2/4] WORKDIR /app 0.0s
=> [3/4] COPY . ./ 0.0s
=> ERROR [4/4] RUN go build -o asdf ./blah 0.3s
------
> [4/4] RUN go build -o asdf ./blah:
0.268 runtime: lfstack.push invalid packing: node=0xffffa45142c0 cnt=0x1 packed=0xffffa45142c00001 -> node=0xffffffffa45142c0
0.268 fatal error: lfstack.push
0.270
0.270 runtime stack:
0.270 runtime.throw({0xaf644d?, 0x0?})
0.271 runtime/panic.go:1067 +0x48 fp=0xc000231f08 sp=0xc000231ed8 pc=0x471228
0.271 runtime.(*lfstack).push(0xffffa45040b8?, 0xc0005841c0?)
0.271 runtime/lfstack.go:29 +0x125 fp=0xc000231f48 sp=0xc000231f08 pc=0x40ef65
0.271 runtime.(*spanSetBlockAlloc).free(...)
0.271 runtime/mspanset.go:322
0.271 runtime.(*spanSet).reset(0xfe7680)
0.271 runtime/mspanset.go:264 +0x79 fp=0xc000231f78 sp=0xc000231f48 pc=0x433559
0.271 runtime.finishsweep_m()
0.272 runtime/mgcsweep.go:257 +0x8d fp=0xc000231fb8 sp=0xc000231f78 pc=0x4263ad
0.272 runtime.gcStart.func2()
0.272 runtime/mgc.go:702 +0xf fp=0xc000231fc8 sp=0xc000231fb8 pc=0x46996f
0.272 runtime.systemstack(0x0)
0.272 runtime/asm_amd64.s:514 +0x4a fp=0xc000231fd8 sp=0xc000231fc8 pc=0x4773ca
...
My setup here is my host machine is linux/arm64, Qemu installed, following the approach described at https://docs.docker.com/build/building/multi-platform/#qemu, to build for linux/amd64.
This has definitely worked in the past which leads me to suggest that something other than Go has changed/been broken here. However I note the virtually identical call stack reported in #54104 hence raising here in the first instance.
What did you expect to see?
Successful run of docker build.
Reacted by Peter Waller, Dmytro Kostiuchenko, Mauri de Souza Meneguzzo, Takuma Kobayashi, Flip Hess, Jannik Hollenbach, Christoph Mewes, Baptiste Roux, Ryan, Kevin Rudde and 18 more
Metadata
Metadata
Assignees
Labels
FrozenDueToAgeNeedsFixThe path to resolution is known, but the work has not been done.The path to resolution is known, but the work has not been done.OS-Linuxcompiler/runtimeIssues related to the Go compiler and/or runtime.Issues related to the Go compiler and/or runtime.
Type
Projects
- StatusShow more project fieldsDone
Activity
gabyhelp commentedon Sep 4, 2024
Related Issues and Documentation
(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)
dmitshur commentedon Sep 4, 2024
Do you think this is this similar or related to issue #68976? (It wasn't listed in the comment above, but it feels similar from a quick initial look.)
CC @prattmic, @matloob.
[-]cmd/go: build fails when run via QEMU for linux/amd64 running on linux/arm64[/-][+]cmd/go, runtime: build fails when run via QEMU for linux/amd64 running on linux/arm64[/+]myitcv commentedon Sep 4, 2024
I don't know I'm afraid. That said the stack trace and symptoms seem quite different. I will however defer to @prattmic
prattmic commentedon Sep 4, 2024
I agree, it looks quite different. #68976 is very specific to pidfd use in os/syscall. This looks like some form of corruption.
Do you know if this build is running a full Linux kernel in a VM, or using QEMU user mode Linux emulation?
prattmic commentedon Sep 4, 2024
Notice
This seems like a sign extension issue when right shifting the packed value (See https://cs.opensource.google/go/go/+/master:src/runtime/lfstack.go;l=26-30, specifically
lfstackUnpack).I could imagine this being a code generation issue, or an issue in QEMU instruction emulation.
cc @golang/compiler
[-]cmd/go, runtime: build fails when run via QEMU for linux/amd64 running on linux/arm64[/-][+]runtime: build fails when run via QEMU for linux/amd64 running on linux/arm64[/+]prattmic commentedon Sep 4, 2024
Does the same issue occur on Go 1.22?
myitcv commentedon Sep 4, 2024
Yes. Indeed similar looking stacks for 1.21.13, 1.22.6, 1.23.0. Confirmed via:
myitcv commentedon Sep 4, 2024
I'm miles out of my depth here, but in case this is useful:
myitcv commentedon Sep 4, 2024
... but just to be super clear, I'm doing this via Docker:
https://docs.docker.com/build/building/multi-platform/#qemu
(so I'm actually unsure whether the host system
qemuis used or not)63 remaining items
ianlancetaylor commentedon Oct 22, 2025
@lczyk Are you saying that Go 1.4 for linux/amd64 does not run correctly under QEMU when running on linux/arm64?
Please open a new issue for that. We can add a patch to the Go 1.4 release branch. Thanks.
cherrymui commentedon Oct 22, 2025
If QEMU user space emulation can provide an address that is not possible on any physical hardware, it sounds like a bug in QEMU. From the comment above, this looks like the case. They should fix it.
We can work around it on the Go side to some degree (like on tip and in 1.25), but going back all the way to Go 1.4 seems too much.
pwillis-oi commentedon Oct 23, 2025
Thanks very much @mdempsky for the fixes provided here. I can confirm that on an M3 w/Sequoia v15.7.1, and Podman Desktop v1.22.1, with default machine (with Rosetta enabled, and Apple Hypervisor selected), the crashes still happen; but rebuilding the machine with the 5.4 OS image fixes the crashes. (In my case it was Terraform crashing.) (Podman command to make the "fixed" machine:
podman machine init --image docker://quay.io/podman/machine-os:5.4 --disk-size 100 --cpus 4 --memory 4096 --now)[release-branch.go1.25] runtime: use one more address bit for tagged …
Add ARM64 architecture support to integration tests (#7068)
[release-branch.go1.25] runtime: use one more address bit for tagged …
[release-branch.go1.25] runtime: use one more address bit for tagged …
fix(examples): cross-compile chrome-sandbox entrypoint to avoid QEMU …
fix: build releases with a Go that survives qemu-user emulation (#14)
Implement docker-socket runtime requirement, onboard fact_extractor
ci: revert devenv-ci arm64 runners back to amd64-only
build-image: document the QEMU/Go GC crash on cross-built amd64 legs
build-image: correct the QEMU/Go GC crash doc - GOGC=off, not asyncpr…