vendor: BuildKit v0.8.0-rc2, containerd, and dependencies #41688

thaJeztah · 2020-11-18T10:36:13Z

vendor: golang.org/x/crypto c1f2f97bffc9c53fc40a1a28a5b460094c0050d9

(included from #41690)

full diff: golang/crypto@75b2880...c1f2f97

relevant changes:

pkcs12: document that we use the wrong PEM type
pkcs12: drop PKCS#12 attributes with unknown OIDs
ocsp: Improve documentation for ParseResponse and ParseResponseForCert
ssh: improve error message for KeyboardInteractiveChallenge
ssh: remove slow unnecessary diffie-hellman-group-exchange primality check

other changes (not in vendor);

ssh/terminal: replace with a golang.org/x/term wrapper
- Deprecates ssh/terminal in favor of golang.org/x/term
ssh/terminal: add support for zos
ssh/terminal: bump x/term dependency to fix js/nacl
nacl/auth: use Size instead of KeySize for Sum output
sha3: remove go:nocheckptr annotation

vendor github.com/tonistiigi/fsutil 0834f99b7b85462efb69b4f571a4fa3ca7da5ac9

tonistiigi/fsutil@c3ed55f...0834f99

copy: use Clonefileat from golang.org/x/sys/unix on macOS
go.mod: update opencontainers/go-digest v1.0.0
Add github action CI
github: test go1.15

vendor: github.com/containerd/ttrpc v1.0.2

full diff: containerd/ttrpc@v1.0.1...v1.0.2

fix bug, failed to assert net error due to error wrap
- fixes: ttrpc client receive "read: connection reset by peer: unknown"
client: add UserOnCloseWait function
travis: add go 1.15

vendor: github.com/containerd/fifo 0724c46b320cf96bb172a0550c19a4b1fca4dacb

full diff: containerd/fifo@f15a329...0724c46

Add OpenFifoDup2
use golang.org/x/sys/unix for dup2 (fixes build on arm64)

vendor: github.com/containerd/cgroups 0b889c03f102012f1d93a97ddd3ef71cd6f4f510

full diff: containerd/cgroups@318312a...0b889c0

v1/stats: add all fields of memory.oom_control
memory: remove wrong memory.kmem.limit_in_bytes check
CI: test against Go 1.15

vendor: github.com/containerd/containerd 0edc412565dcc6e3d6125ff9e4b009ad4b89c638

full diff: containerd/containerd@d4e7820...0edc412

integrates containerd/cri into main containerd repository
seccomp: add pidfd_open and pidfd_send_signal
seccomp: add pidfd_getfd syscall (gated by CAP_SYS_PTRACE)
docker: don’t hide pusher response error

vendor: github.com/moby/buildkit v0.8.0-rc2

full diff: moby/buildkit@6861f17...v0.8.0-rc2

dockerfile: rename experimental channel to labs
dockerfile build: fix not exit when meet error in load config metadata
copy containerd.UnknownExitStatus to local const to reduce dependency graph in client
executor: switch to docker seccomp profile
add retry handlers to push/pull
SSH-based auth for llb.Git operations
Allow gateway exec-ing into a failed solve with an exec op
Fix parsing ssh-based git sources
Fix sshkeyscan to work with ipv6
fix assumption that ssh port must be 2 digits
vendor: github.com/Microsoft/go-winio v0.4.15
vendor: github.com/tonistiigi/fsutil v0.0.0-20201103201449-0834f99b7b85
vendor: containerd v1.4.1-0.20201117152358-0edc412565dc
vendor: golang.org/x/crypto v0.0.0-20201117144127-c1f2f97bffc9

thaJeztah · 2020-11-18T10:37:52Z

vendor/github.com/moby/buildkit/go.mod

I'm unsure about what to do with this dependency; I didn't update because we currently vendor the special "moby" branch, but with both containerd, buildkit, and docker using the dependency, I wonder if we should move back to these tags

https://github.com/Microsoft/hcsshim/compare/9dcb42f100215f8d375b4a9265e5bba009217a85..v0.8.10

Opened #41689 for discussion

thaJeztah · 2020-11-18T10:56:29Z

Looks like some changes are needed;


[2020-11-18T10:52:02.906Z] === Errors
[2020-11-18T10:52:02.906Z] builder/builder-next/worker/worker.go:432:28: not enough arguments in call to contentutil.Copy
[2020-11-18T10:52:02.906Z] 	have (context.Context, content.Store, content.Provider, v1.Descriptor)
[2020-11-18T10:52:02.906Z] 	want (context.Context, content.Ingester, content.Provider, v1.Descriptor, func([]byte))

thaJeztah · 2020-11-18T11:14:47Z

builder/builder-next/worker/worker.go

@tonistiigi @AkihiroSuda Could use some help with this part; should this write output to progressOutput? Or use something similar to loggerFromContext()? see moby/buildkit@aa29e77 (moby/buildkit#1791)

We don't need this. Downloadmanager is smart enough to do retries on its own.

thaJeztah · 2020-11-18T12:34:20Z

Looks like a flaky test; #23626

=== RUN   TestDockerSwarmSuite/TestAPISwarmServicesMultipleAgents
    --- FAIL: TestDockerSwarmSuite/TestAPISwarmServicesMultipleAgents (55.18s)
        docker_api_swarm_service_test.go:99: [ddfe3d25d57a3] joining swarm manager [d034f1702ea9f]@0.0.0.0:2477, swarm listen addr 0.0.0.0:2478
        docker_api_swarm_service_test.go:100: [d5b56e426e832] joining swarm manager [d034f1702ea9f]@0.0.0.0:2477, swarm listen addr 0.0.0.0:2479
        docker_api_swarm_service_test.go:121: timeout hit after 30s: output: "8bee4f1ca802\n4983b04092b1\n310a4c4192b2\n", output: "add4032c4018\n74365e630934\n196944e32520\n973d2e9fe91a\n"
        poll.go:121: assertion failed: error is not nil: exit status 1
        check_test.go:380: [ddfe3d25d57a3] daemon is not started

thaJeztah · 2020-11-19T09:13:27Z

Updated to v0.8.0-rc2

full diff: golang/crypto@75b2880...c1f2f97 relevant changes: - pkcs12: document that we use the wrong PEM type - pkcs12: drop PKCS#12 attributes with unknown OIDs - ocsp: Improve documentation for ParseResponse and ParseResponseForCert other changes (not in vendor); - ssh: improve error message for KeyboardInteractiveChallenge - ssh: remove slow unnecessary diffie-hellman-group-exchange primality check - ssh/terminal: replace with a golang.org/x/term wrapper - Deprecates ssh/terminal in favor of golang.org/x/term - ssh/terminal: add support for zos - ssh/terminal: bump x/term dependency to fix js/nacl - nacl/auth: use Size instead of KeySize for Sum output - sha3: remove go:nocheckptr annotation Signed-off-by: Sebastiaan van Stijn <[email protected]>

…7da5ac9 tonistiigi/fsutil@c3ed55f...0834f99 - copy: use Clonefileat from golang.org/x/sys/unix on macOS - go.mod: update opencontainers/go-digest v1.0.0 - Add github action CI - github: test go1.15 Signed-off-by: Sebastiaan van Stijn <[email protected]>

full diff: containerd/ttrpc@v1.0.1...v1.0.2 - fix bug, failed to assert net error due to error wrap - fixes: ttrpc client receive "read: connection reset by peer: unknown" - client: add UserOnCloseWait function - travis: add go 1.15 Signed-off-by: Sebastiaan van Stijn <[email protected]>

…a4dacb full diff: containerd/fifo@f15a329...0724c46 - Add OpenFifoDup2 - use golang.org/x/sys/unix for dup2 (fixes build on arm64) Signed-off-by: Sebastiaan van Stijn <[email protected]>

…cd6f4f510 full diff: containerd/cgroups@318312a...0b889c0 - v1/stats: add all fields of memory.oom_control - memory: remove wrong memory.kmem.limit_in_bytes check - CI: test against Go 1.15 Signed-off-by: Sebastiaan van Stijn <[email protected]>

…09ad4b89c638 full diff: containerd/containerd@d4e7820...0edc412 - integrates containerd/cri into main containerd repository - seccomp: add `pidfd_open` and `pidfd_send_signal` - seccomp: add `pidfd_getfd` syscall (gated by `CAP_SYS_PTRACE`) - docker: don’t hide pusher response error Signed-off-by: Sebastiaan van Stijn <[email protected]>

full diff: moby/buildkit@6861f17...v0.8.0-rc2 - dockerfile: rename experimental channel to labs - dockerfile build: fix not exit when meet error in load config metadata - copy containerd.UnknownExitStatus to local const to reduce dependency graph in client - executor: switch to docker seccomp profile - add retry handlers to push/pull - SSH-based auth for llb.Git operations - Allow gateway exec-ing into a failed solve with an exec op - Fix parsing ssh-based git sources - Fix sshkeyscan to work with ipv6 - fix assumption that ssh port must be 2 digits - vendor: github.com/Microsoft/go-winio v0.4.15 - vendor: github.com/tonistiigi/fsutil v0.0.0-20201103201449-0834f99b7b85 - vendor: containerd v1.4.1-0.20201117152358-0edc412565dc - vendor: golang.org/x/crypto v0.0.0-20201117144127-c1f2f97bffc9 Signed-off-by: Sebastiaan van Stijn <[email protected]>

thaJeztah · 2020-11-19T09:35:20Z

included #41690, in this PR because buildkit uses some new packages from this dependency, and otherwise we may get incorrect vendoring if the PR's are merged out of order

ShemTovYosef · 2020-12-03T00:33:20Z

ping @tonistiigi

AkihiroSuda · 2020-12-03T04:48:05Z

vendor.conf

 # containerd
-github.com/containerd/containerd                    d4e78200d6da62480c85bf6f26b7221ea938f396
-github.com/containerd/fifo                          f15a3290365b9d2627d189e619ab4008e0069caf
+github.com/containerd/containerd                    0edc412565dcc6e3d6125ff9e4b009ad4b89c638 # master (v1.5.0-dev)


Could you update this to be containerd/containerd@7126310 or newer?
I believe the older commit is safe for just using containerd as a library, but the older daemon binary was known not to work with Docker, so I'm not a fan of vendoring the older commit.

Can be a follow-up PR though.

AkihiroSuda · 2020-12-03T04:49:11Z

vendor.conf

 # buildkit
-github.com/moby/buildkit                            6861f17f15364de0fe1fd1e6e8da07598a485123
-github.com/tonistiigi/fsutil                        c3ed55f3b48161fd3dc42c17ba09e12ac52d57dc
+github.com/moby/buildkit                            fcb87e6b8ccf3631a65799cc56caa76f9117816e # v0.8.0-rc2


v0.8.0 is now GA

thaJeztah · 2020-12-03T10:18:04Z

v0.8.0 is now GA

@AkihiroSuda did you want that bump in this PR, or ok to do a follow-up after this one?

AkihiroSuda · 2020-12-03T10:19:11Z

Can be a follow-up

thaJeztah · 2020-12-03T10:24:03Z

I'll have a look at updating this afternoon; let me know if this is good to merge 👍

…

On 3 Dec 2020, at 11:19, Akihiro Suda ***@***.***> wrote: Can be a follow-up — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

AkihiroSuda · 2020-12-03T10:37:53Z

opened #41745 as a follow-up

thaJeztah · 2020-12-23T11:50:20Z

1f88736 also brings in containerd/containerd#4530 ("Add RPi1/RPi0 workaround"), related to:

On the very popular Raspberry Pi 1 and Zero devices, the CPU is actually ARMv6, but the chip happens to support the feature bit the kernel uses to differentiate v6/v7, so it gets reported as "CPU architecture: 7" and thus fails to run many of the images that get pulled.

To account for this very popular edge case, this also checks "model name" which on these chips will begin with "ARMv6-compatible" -- we could also check uname, but getCPUInfo is already handy, low overhead, and mirrors the code before this.

To give a small taste of how common this issue is and why (in my opinion) it warrants a special case:

moby/moby#41017

moby/moby#37647

moby/moby#34875

biarms/mysql#4

(I proposed this over in moby/moby#41017 (comment), which prompted the following query:)

@estesp: is there any variant of the ARMv7 CPU family that might be adversely affected by this; e.g. might get "downgraded" to a v6 variant inadvertently?

I'm not aware of any that report model name to be ARMv6-compatible like these RPi devices do (and my Google search for "model name : armv6-compatible" comes up with tons of hits for RPi-related discussion, usually around this very kernel quirk, but not much else I can see), so I think the regression potential here is really low.

I personally tested on a Raspberry Pi 1 (ARMv6) and a NanoPi NEO (ARMv7), and ctr i pull docker.io/library/alpine:3.12 does the right thing in both instances (on the RPi1, I get unpacking linux/arm/v6 sha256:... and on the NanoPi, I get unpacking linux/arm/v7 sha256:..., whereas before this change the RPi1 would choose the linux/arm/v7 image which then fails to run).

Possibly more dependencies need to be updated, and instead of this we should cherry-pick. This is just a quick check "what would it look like if we bumped the version in this branch"; Updating to containerd 1.5 Last containerd update in 20.10 is moby@1f88736 (moby#41688) - moby@ab1dd80 moby#42274 - moby@5761fca moby#42274 - moby@42ef2c5 moby#42276 - moby@6202322 moby#42254 - moby@7c1c123 moby#42249 - moby@84df737 moby#42636 - moby@4fc2d4d moby#42656 - moby@3d58d13 moby#42697 - moby@582ef29 moby#42994 Signed-off-by: Sebastiaan van Stijn <[email protected]>

Possibly more dependencies need to be updated, and instead of this we should cherry-pick. This is just a quick check "what would it look like if we bumped the version in this branch"; Updating to containerd 1.5 Last containerd update in 20.10 is moby@1f88736 (moby#41688) - moby@ab1dd80 moby#42274 - moby@5761fca moby#42274 - moby@42ef2c5 moby#42276 - moby@6202322 moby#42254 - moby@7c1c123 moby#42249 - moby@84df737 moby#42636 - moby@4fc2d4d moby#42656 - moby@3d58d13 moby#42697 - moby@582ef29 moby#42994 - moby@458b4aa moby#43025 Signed-off-by: Sebastiaan van Stijn <[email protected]>

thaJeztah commented Nov 18, 2020

View reviewed changes

thaJeztah force-pushed the bump_buildkit branch from 4b4683e to e67f9b9 Compare November 18, 2020 10:47

thaJeztah force-pushed the bump_buildkit branch from e67f9b9 to 50aadb6 Compare November 18, 2020 11:13

thaJeztah requested a review from tonistiigi as a code owner November 18, 2020 11:13

thaJeztah commented Nov 18, 2020

View reviewed changes

thaJeztah changed the title ~~vendor: BuildKit, containerd, and dependencies~~ vendor: BuildKit v0.8.0-rc2, containerd, and dependencies Nov 19, 2020

thaJeztah force-pushed the bump_buildkit branch from 50aadb6 to 3b3e2fc Compare November 19, 2020 09:12

thaJeztah added area/builder Build status/2-code-review labels Nov 19, 2020

thaJeztah added this to the 20.10.0 milestone Nov 19, 2020

thaJeztah added 7 commits November 19, 2020 10:30

vendor: github.com/containerd/fifo 0724c46b320cf96bb172a0550c19a4b1fc…

ba475d4

…a4dacb full diff: containerd/fifo@f15a329...0724c46 - Add OpenFifoDup2 - use golang.org/x/sys/unix for dup2 (fixes build on arm64) Signed-off-by: Sebastiaan van Stijn <[email protected]>

thaJeztah mentioned this pull request Nov 19, 2020

vendor: golang.org/x/crypto c1f2f97bffc9c53fc40a1a28a5b460094c0050d9 #41690

Closed

thaJeztah force-pushed the bump_buildkit branch from 3b3e2fc to 73571e4 Compare November 19, 2020 09:45

AkihiroSuda approved these changes Nov 19, 2020

View reviewed changes

tiborvass approved these changes Dec 2, 2020

View reviewed changes

tonistiigi approved these changes Dec 3, 2020

View reviewed changes

AkihiroSuda reviewed Dec 3, 2020

View reviewed changes

AkihiroSuda approved these changes Dec 3, 2020

View reviewed changes

AkihiroSuda merged commit ab373df into moby:master Dec 3, 2020

AkihiroSuda mentioned this pull request Dec 3, 2020

vendor: BuildKit 950603da215ae03b843f3f66fbe86c4876a6f5a1 #41745

Merged

thaJeztah deleted the bump_buildkit branch December 3, 2020 11:16

thaJeztah mentioned this pull request Dec 23, 2020

manifest list (multiarch) picks wrong arch on ARMv6 #41017

Closed

This was referenced Jul 9, 2021

ARMv6 machine pulls v7 image from manifest list #37647

Closed

manifest list (multiarch) picks wrong arch on ARMv7 #34875

Closed

docker pull via manifest on a rpi 1 device (armv6l) will not pull the arm/v6 image, but the arm/v7 image if any biarms/mysql#4

Open

thaJeztah mentioned this pull request Nov 9, 2021

[20.10] vendor update containerd client code to v1.5.10 #43006

Closed

vendor: BuildKit v0.8.0-rc2, containerd, and dependencies #41688

vendor: BuildKit v0.8.0-rc2, containerd, and dependencies #41688

Uh oh!

Conversation

thaJeztah commented Nov 18, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

vendor: golang.org/x/crypto c1f2f97bffc9c53fc40a1a28a5b460094c0050d9

vendor github.com/tonistiigi/fsutil 0834f99b7b85462efb69b4f571a4fa3ca7da5ac9

vendor: github.com/containerd/ttrpc v1.0.2

vendor: github.com/containerd/fifo 0724c46b320cf96bb172a0550c19a4b1fca4dacb

vendor: github.com/containerd/cgroups 0b889c03f102012f1d93a97ddd3ef71cd6f4f510

vendor: github.com/containerd/containerd 0edc412565dcc6e3d6125ff9e4b009ad4b89c638

vendor: github.com/moby/buildkit v0.8.0-rc2

Uh oh!

thaJeztah Nov 18, 2020

Choose a reason for hiding this comment

Uh oh!

thaJeztah Nov 18, 2020

Choose a reason for hiding this comment

Uh oh!

thaJeztah commented Nov 18, 2020

Uh oh!

thaJeztah Nov 18, 2020

Choose a reason for hiding this comment

Uh oh!

tonistiigi Dec 2, 2020

Choose a reason for hiding this comment

Uh oh!

thaJeztah commented Nov 18, 2020

Uh oh!

thaJeztah commented Nov 19, 2020

Uh oh!

thaJeztah commented Nov 19, 2020

Uh oh!

ShemTovYosef commented Dec 3, 2020

Uh oh!

AkihiroSuda Dec 3, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AkihiroSuda Dec 3, 2020

Choose a reason for hiding this comment

Uh oh!

thaJeztah commented Dec 3, 2020

Uh oh!

AkihiroSuda commented Dec 3, 2020

Uh oh!

thaJeztah commented Dec 3, 2020 via email

Uh oh!

AkihiroSuda commented Dec 3, 2020

Uh oh!

thaJeztah commented Dec 23, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

thaJeztah commented Nov 18, 2020 •

edited

Loading

AkihiroSuda Dec 3, 2020 •

edited

Loading