update to go 1.16 by thaJeztah · Pull Request #2144 · moby/buildkit

thaJeztah · 2021-06-01T14:03:08Z

depends on #2146

relates to moby/moby#40353 moby/moby#40353 (comment)

This updates all occurrences of Go 1.13 to Go 1.16; also updated the code that's used to redact credentials in URLs to use the Go implementation (based on golang/go@e3323f5, it's available in Go 1.15 and up).

thaJeztah · 2021-06-01T14:34:50Z

Runc Compile is failing;

#30 [buildkitd 1/1] RUN --mount=target=. --mount=target=/root/.cache,type=cache   --mount=target=/go/pkg/mod,type=cache   --mount=source=/tmp/.ldflags,target=/tmp/.ldflags,from=buildkit-version   CGO_ENABLED=0 xx-go build -ldflags "$(cat /tmp/.ldflags) -extldflags '-static'" -tags "osusergo netgo static_build seccomp ${BUILDKITD_TAGS}" -o /usr/bin/buildkitd ./cmd/buildkitd &&   xx-verify --static /usr/bin/buildkitd
#30 sha256:85c4cf958570ea07260c8042fd2d26e1fbeb9b8059a9fba73bd826a388619fcd
#30 CANCELED
------
 > [runc 3/3] RUN --mount=from=runc-src,src=/usr/src/runc,target=. --mount=target=/root/.cache,type=cache   CGO_ENABLED=1 xx-go build -mod=vendor -ldflags '-extldflags -static' -tags 'apparmor seccomp netgo cgo static_build osusergo' -o /usr/bin/runc ./ &&   xx-verify --static /usr/bin/runc:
------
Dockerfile:56
--------------------
  55 |       [ "$(xx-info arch)" != "ppc64le" ] || XX_CC_PREFER_LINKER=ld xx-clang --setup-target-triple
  56 | >>> RUN --mount=from=runc-src,src=/usr/src/runc,target=. --mount=target=/root/.cache,type=cache \
  57 | >>>   CGO_ENABLED=1 xx-go build -mod=vendor -ldflags '-extldflags -static' -tags 'apparmor seccomp netgo cgo static_build osusergo' -o /usr/bin/runc ./ && \
  58 | >>>   xx-verify --static /usr/bin/runc
  59 |     
--------------------
error: failed to solve: rpc error: code = Unknown desc = executor failed running [/bin/sh -c CGO_ENABLED=1 xx-go build -mod=vendor -ldflags '-extldflags -static' -tags 'apparmor seccomp netgo cgo static_build osusergo' -o /usr/bin/runc ./ &&   xx-verify --static /usr/bin/runc]: exit code: 2
Error: Process completed with exit code 1.

thaJeztah · 2021-06-01T14:43:04Z

Looks like I get a more detailed error when running locally;

make TARGETPLATFORM=linux/amd64 binaries
hack/binaries
+ docker buildx build --platform linux/amd64 --target binaries --output type=local,dest=./bin/ .
[+] Building 65.3s (24/28)
...
 => ERROR [runc 3/3] RUN --mount=from=runc-src,src=/usr/src/runc,target=. --mount=target=/root/.cache,type=cache   CGO_ENABLED=1 xx-go build -mod=vendor -ldflags '-extldflags -static' -tags 'appar  46.3s
------
 > [runc 3/3] RUN --mount=from=runc-src,src=/usr/src/runc,target=. --mount=target=/root/.cache,type=cache   CGO_ENABLED=1 xx-go build -mod=vendor -ldflags '-extldflags -static' -tags 'apparmor seccomp netgo cgo static_build osusergo' -o /usr/bin/runc ./ &&   xx-verify --static /usr/bin/runc:
#18 45.08 # github.com/opencontainers/runc
#18 45.08 /usr/local/go/pkg/tool/linux_amd64/link: running x86_64-alpine-linux-musl-clang failed: exit status 1
#18 45.08 ld.lld: error: unable to find library -lseccomp
#18 45.08 ld.lld: error: unable to find library -lseccomp
#18 45.08 clang-10: error: linker command failed with exit code 1 (use -v to see invocation)
#18 45.08
------
error: failed to solve: rpc error: code = Unknown desc = executor failed running [/bin/sh -c CGO_ENABLED=1 xx-go build -mod=vendor -ldflags '-extldflags -static' -tags 'apparmor seccomp netgo cgo static_build osusergo' -o /usr/bin/runc ./ &&   xx-verify --static /usr/bin/runc]: exit code: 2
make: *** [binaries] Error 1

crazy-max · 2021-06-01T15:18:26Z

@thaJeztah

Looks like I get a more detailed error when running locally;

More details above: https://github.com/moby/buildkit/pull/2144/checks?check_run_id=2719843639#step:7:287

thaJeztah · 2021-06-01T15:24:45Z

Ah! Looks like I didn't scroll back far enough. Slightly confusing that that error is not included in the final error that buildkit returns (perhaps it should be?)

tonistiigi · 2021-06-01T21:34:48Z

Need https://pkgs.alpinelinux.org/package/v3.13/main/aarch64/libseccomp-static as alpine 3.13 does not contain static library in https://pkgs.alpinelinux.org/contents?branch=v3.13&name=libseccomp-dev&arch=aarch64&repo=main anymore

thaJeztah · 2021-06-01T21:55:41Z

Ugh.. more fun with modules;

#30 [containerd-alt 1/1] RUN --mount=from=containerd-src,src=/usr/src/containerd,readwrite --mount=target=/root/.cache,type=cache   git fetch origin   && git checkout -q "v1.4.6"   && make bin/containerd   && make bin/containerd-shim-runc-v2   && mv bin /out
#30 sha256:449b41208d4462fcc0693e02e9743ec7fffa53a71c7c2028a30ab89c6d3622b5
#30 3.283 + bin/containerd
#30 3.297 go: cannot find main module, but found vendor.conf in /go/src/github.com/containerd/containerd
#30 3.297 	to create a module there, run:
#30 3.297 	go mod init
#30 3.298 make: *** [Makefile:193: bin/containerd] Error 1
#30 ERROR: executor failed running [/bin/sh -c git fetch origin   && git checkout -q "$CONTAINERD_ALT_VERSION"   && make bin/containerd   && make bin/containerd-shim-runc-v2   && mv bin /out]: exit code: 2

thaJeztah · 2021-06-01T22:29:53Z

All green now 🎉 ✅

I'll rebase, and move this out of draft once #2146 is merged

This updates all occurrences of Go 1.13 to Go 1.16; also updated the code that's used to redact credentials in URLs to use the Go implementation. Signed-off-by: Sebastiaan van Stijn <[email protected]>

thaJeztah · 2021-06-01T22:36:15Z

 # containerd v1.4 for integration tests
 FROM containerd-base as containerd-alt
 ARG CONTAINERD_ALT_VERSION
+ARG GO111MODULE=off


TBH, not sure why go1.16 didn't detect vendor here, and complains about no go.mod being present; guess it still requires a go.mod even if a vendor is present 🤷‍♂️

I'lll have a look at adding this to the containerd makefile in the 1.4 branch

Or would it be better to build this version of containerd with Go 1.15? https://github.com/containerd/containerd/blob/v1.4.6/.github/workflows/ci.yml#L29

It doesn't matter until the tests pass. These binaries are never released.

AkihiroSuda

We will probably have to mess around with the EINTR stuff, but anyway building is flaky by nature mostly due to network flakiness, so possible extra flakiness added by EINTR is not a huge deal 😛

So LGTM.

thaJeztah · 2021-06-02T09:23:34Z

We will probably have to mess around with the EINTR stuff, but anyway building is flaky by nature mostly due to network flakiness, so possible extra flakiness added by EINTR is not a huge deal 😛

From https://docs.google.com/spreadsheets/d/1U6HUWokcRnQtWCtym6mRNmF9f_OQdxFbjx6F3ItHxKY/edit#gid=1727584665

Looks like the "vendor" directory of BuildKit in moby is ok, but perhaps it'd be good to run Tibor's script on the BuildKit codebase itself (script is here: https://gist.github.com/tiborvass/786c66dfa614db4a60805c05bdf8a923#file-syscalls-go)

Here's an Excel export of that file (for others to view) - I think the conditional formatting got lost in the export from Google docs though);
Syscall_Interruption_Analysis_of_Docker.xlsx

crazy-max · 2021-06-02T16:12:27Z

@thaJeztah In a follow-up, wonder if we should start produce riscv64 arch bin which is available since 1.14.

tonistiigi · 2021-06-02T19:27:06Z

@crazy-max There are no alpine packages (for runc and image)

AkihiroSuda · 2021-06-07T06:37:24Z

@tonistiigi LGTY?

tonistiigi

Merging. But please do run the script above and create follow-up issues for any discoveries.

tonistiigi · 2021-06-09T15:51:23Z

I've seen "flightcontrol: exceeded retry timeout" errors in ci 2 times in the past day that I've never seen before. Wondering if it can be related somehow.

thaJeztah force-pushed the bump_golang branch from b891e31 to 04c49e3 Compare June 1, 2021 14:56

thaJeztah force-pushed the bump_golang branch from 04c49e3 to 690fd85 Compare June 1, 2021 20:13

thaJeztah force-pushed the bump_golang branch from 690fd85 to ca1d2b0 Compare June 1, 2021 21:38

thaJeztah force-pushed the bump_golang branch 2 times, most recently from 1ddca36 to 4045235 Compare June 1, 2021 22:00

update to go 1.16

22dd74a

This updates all occurrences of Go 1.13 to Go 1.16; also updated the code that's used to redact credentials in URLs to use the Go implementation. Signed-off-by: Sebastiaan van Stijn <[email protected]>

thaJeztah force-pushed the bump_golang branch from 4045235 to 22dd74a Compare June 1, 2021 22:32

thaJeztah marked this pull request as ready for review June 1, 2021 22:32

thaJeztah commented Jun 1, 2021

View reviewed changes

AkihiroSuda approved these changes Jun 2, 2021

View reviewed changes

AkihiroSuda mentioned this pull request Jun 2, 2021

update to go1.15 #1640

Closed

AkihiroSuda added the needs-cherry-pick/v0.8 label Jun 2, 2021

tonistiigi approved these changes Jun 8, 2021

View reviewed changes

tonistiigi merged commit 03af083 into moby:master Jun 8, 2021

thaJeztah deleted the bump_golang branch June 8, 2021 18:00

tonistiigi mentioned this pull request Jun 8, 2021

vendor: fix broken gosum #2160

Merged

thaJeztah mentioned this pull request Sep 4, 2022

[v0.8] update to go1.18, debian bullseye #3079

Merged

Uh oh!

Conversation

thaJeztah commented Jun 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thaJeztah commented Jun 1, 2021

Uh oh!

thaJeztah commented Jun 1, 2021

Uh oh!

crazy-max commented Jun 1, 2021

Uh oh!

thaJeztah commented Jun 1, 2021

Uh oh!

tonistiigi commented Jun 1, 2021

Uh oh!

thaJeztah commented Jun 1, 2021

Uh oh!

thaJeztah commented Jun 1, 2021

Uh oh!

thaJeztah Jun 1, 2021

Choose a reason for hiding this comment

Uh oh!

tonistiigi Jun 2, 2021

Choose a reason for hiding this comment

Uh oh!

AkihiroSuda left a comment

Choose a reason for hiding this comment

Uh oh!

thaJeztah commented Jun 2, 2021

Uh oh!

crazy-max commented Jun 2, 2021

Uh oh!

tonistiigi commented Jun 2, 2021

Uh oh!

AkihiroSuda commented Jun 7, 2021

Uh oh!

tonistiigi left a comment

Choose a reason for hiding this comment

Uh oh!

tonistiigi commented Jun 9, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

thaJeztah commented Jun 1, 2021 •

edited

Loading