ci: buildkit CI is broken; "cgo: malformed DWARF TagVariable entry" #44570
Description
Description
Since late yesterday evening, BuildKit CI is broken; as far as I can see, it's breaking in a step that does not use the code in this repository (but please double-check if I'm right), so either
- The base image used
- The Dockerfile frontend (it looks to be using
dockerfile-upstream:master) - Perhaps some obscure issue, similar to including git attributes in "vendor" makes archive checksum change containerd/containerd#6387, where the Git "short commit" is used in the source code?
- Corrupted remote build-cache? (does it use that?)
- something changes on GitHub action hosts ?
- GitHub actions builds of docker / containerd / buildx ?
Failure looks like this: (https://github.com/moby/moby/actions/runs/3600129177/jobs/6064524682)
> [containerd-alt-15 1/1] RUN --mount=from=containerd-src,src=/usr/src/containerd,readwrite --mount=target=/root/.cache,type=cache git fetch origin && git checkout -q "v1.5.11" && make bin/containerd && make bin/containerd-shim-runc-v2 && mv bin /out:
#0 4.935 + bin/containerd
#47 268.1 # github.com/containerd/containerd/vendor/github.com/miekg/pkcs11
#47 268.1 cgo: malformed DWARF TagVariable entry
#47 279.5 make: *** [Makefile:213: bin/containerd] Error 2
Which is similar to reports in go1.17 some time ago;
- looks reported in cgo: malformed DWARF TagVariable entry golang/go#53000
- and a fix in https://go-review.googlesource.com/c/go/+/406816 ?
Reproduce
See #44567 and other PRs
Expected behavior
No response
docker version
sudo rm /etc/docker/daemon.json
sudo service docker restart
docker version
docker info
shell: /usr/bin/bash -e {0}
env:
BUNDLES_OUTPUT: ./bundles
BUILDKIT_REF: 4febae4f874bd8ef52dec30e988c8fe0bc96b3b9
Client:
Version: 20.10.21+azure-1
API version: 1.41
Go version: go1.18.7
Git commit: baeda1f82a10204ec5708d5fbba130ad76cfee49
Built: Tue Oct 25 17:53:02 UTC 2022
OS/Arch: linux/amd64
Context: default
Experimental: truedocker info
Server:
Engine:
Version: 20.10.21+azure-1
API version: 1.41 (minimum version 1.12)
Go version: go1.18.7
Git commit: 3056208812eb5e792fa99736c9167d1e10f4ab49
Built: Tue Oct 25 11:44:15 2022
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.5.14+azure-1
GitCommit: b84d0b151c2395a5917996d602b192ce1e0fa461
runc:
Version: 1.1.4
GitCommit: 5fd4c4d144137e991c4acebb2146ab1483a97925
docker-init:
Version: 0.19.0
GitCommit:
Client:
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc., 0.9.1+azure-2)
compose: Docker Compose (Docker Inc., 2.12.2+azure-1)
Server:
Containers: 1
Running: 0
Paused: 0
Stopped: 1
Images: 21
Server Version: 20.10.21+azure-1
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: false
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: b84d0b151c2395a5917996d602b192ce1e0fa461
runc version: 5fd4c4d144137e991c4acebb2146ab1483a97925
init version:
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 5.15.0-1023-azure
Operating System: Ubuntu 20.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 6.781GiB
Name: fv-az77-14
ID: 4IWW:VLI2:7YTM:AFEZ:4NJ5:VCDD:VCJU:3HXK:DSAL:C6S2:OKSN:N3A5
Docker Root Dir: /var/lib/docker
Debug Mode: false
Username: githubactions
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: falseAdditional Info
The step where things break:
shell: /usr/bin/bash -e {0}
env:
BUNDLES_OUTPUT: ./bundles
BUILDKIT_REF: 4febae4f874bd8ef52dec30e988c8fe0bc96b3b9
CONTEXT: .
TEST_DOCKERD: 1
TEST_DOCKERD_BINARY: ./build/moby/binary-daemon/dockerd
TESTPKGS: ./client
TESTFLAGS: -v --parallel=1 --timeout=30m --run=//worker=dockerd$
+ docker buildx build --target integration-tests --output type=docker,name=buildkit-tests . --progress=plain
Failure with more detail:
> [containerd-alt-15 1/1] RUN --mount=from=containerd-src,src=/usr/src/containerd,readwrite --mount=target=/root/.cache,type=cache git fetch origin && git checkout -q "v1.5.11" && make bin/containerd && make bin/containerd-shim-runc-v2 && mv bin /out:
#0 5.824 + bin/containerd
#49 311.7 # github.com/containerd/containerd/vendor/github.com/miekg/pkcs11
#49 311.7 cgo: malformed DWARF TagVariable entry
#49 324.2 make: *** [Makefile:213: bin/containerd] Error 2
------
Dockerfile:143
--------------------
142 | ARG GO111MODULE=off
143 | >>> RUN --mount=from=containerd-src,src=/usr/src/containerd,readwrite --mount=target=/root/.cache,type=cache \
144 | >>> git fetch origin \
145 | >>> && git checkout -q "$CONTAINERD_ALT_VERSION_15" \
146 | >>> && make bin/containerd \
147 | >>> && make bin/containerd-shim-runc-v2 \
148 | >>> && mv bin /out
149 |
--------------------
ERROR: failed to solve: process "/bin/sh -c git fetch origin && git checkout -q \"$CONTAINERD_ALT_VERSION_15\" && make bin/containerd && make bin/containerd-shim-runc-v2 && mv bin /out" did not complete successfully: exit code: 2
Error: Process completed with exit code 1.
- a container builder
- the
docker.io/docker/dockerfile-upstream:masterfrontend (sha256:0e39c80bd858c49913c85dd3636aae0f6c8266ff1dcc24e5aeabf98e93579843)
#1 [internal] booting buildkit
#1 starting container buildx_buildkit_builder-9473aa25-e043-4654-934c-be4271ea8e3d0
#1 starting container buildx_buildkit_builder-9473aa25-e043-4654-934c-be4271ea8e3d0 0.6s done
#1 DONE 0.6s
#2 [internal] load .dockerignore
#2 transferring context: 56B done
#2 DONE 0.0s
#3 [internal] load build definition from Dockerfile
#3 transferring dockerfile: 11.57kB done
#3 DONE 0.0s
#4 resolve image config for docker.io/docker/dockerfile-upstream:master
#4 ...
#5 [auth] docker/dockerfile-upstream:pull token for registry-1.docker.io
#5 DONE 0.0s
#4 resolve image config for docker.io/docker/dockerfile-upstream:master
#4 DONE 0.8s
#6 docker-image://docker.io/docker/dockerfile-upstream:master@sha256:0e39c80bd858c49913c85dd3636aae0f6c8266ff1dcc24e5aeabf98e93579843
#6 resolve docker.io/docker/dockerfile-upstream:master@sha256:0e39c80bd858c49913c85dd3636aae0f6c8266ff1dcc24e5aeabf98e93579843 done
#6 sha256:0268775466e82aa89ffd3bd52916812c04ce4f8fb96af0e504edd8f997ffbcfc 1.05MB / 11.47MB 0.2s
#6 sha256:0268775466e82aa89ffd3bd52916812c04ce4f8fb96af0e504edd8f997ffbcfc 3.15MB / 11.47MB 0.3s
#6 sha256:0268775466e82aa89ffd3bd52916812c04ce4f8fb96af0e504edd8f997ffbcfc 11.47MB / 11.47MB 0.5s done
#6 extracting sha256:0268775466e82aa89ffd3bd52916812c04ce4f8fb96af0e504edd8f997ffbcfc
#6 extracting sha256:0268775466e82aa89ffd3bd52916812c04ce4f8fb96af0e504edd8f997ffbcfc 0.2s done
#6 DONE 0.8s