c8d/push: Push lazy blobs with distribution source labels #74

vvoland · 2022-08-24T11:59:30Z

This makes it possible to push multi-platform images if some contents are missing from the local content store, but can be fetched from a source repository.

The trick here is wrapping the ContentStore to return a fake content.Info with a containerd.io/distribution.source. label which specifies the source registry and repository for the blob. It's then used by the repository to cross-repo mount the blobs.

Caveats:

Doesn't work with other registries than the one which provided the missing blob.
Works only as long as we have a source manifest (it's deleted by buildx purge -a or restarting the daemon)

Test:

printf 'FROM alpine\nCMD ["echo", "hello"]' | docker buildx build . \
    -t docker.io/repo/test:latest --platform linux/amd64,linux/arm64
docker push docker.io/repo/test:latest

Test result with a fresh repo

root # docker buildx imagetools inspect docker.io/pawelgronowski465/pushworks:220824 # Repo doesn't exist1
ERROR: docker.io/pawelgronowski465/pushnotworks:220824: not found
root # cat Dockerfile
FROM alpine
CMD ["echo", "test"]
root # docker build . -t pawelgronowski465/pushworks:220824 --platform linux/arm64,linux/amd64,linux/386
[+] Building 4.4s (9/9) FINISHED
 => [internal] load .dockerignore                                                                                                         0.0s
 => => transferring context: 2B                                                                                                           0.0s
 => [internal] load build definition from Dockerfile                                                                                      0.0s
 => => transferring dockerfile: 70B                                                                                                       0.0s
 => [linux/arm64 internal] load metadata for docker.io/library/alpine:latest                                                              3.9s
 => [linux/amd64 internal] load metadata for docker.io/library/alpine:latest                                                              3.9s
 => [linux/386 internal] load metadata for docker.io/library/alpine:latest                                                                3.9s
 => [linux/amd64 1/1] FROM docker.io/library/alpine@sha256:bc41182d7ef5ffc53a40b044e725193bc10142a1243f395ee852a8d9730fc2ad               0.0s
 => => resolve docker.io/library/alpine@sha256:bc41182d7ef5ffc53a40b044e725193bc10142a1243f395ee852a8d9730fc2ad                           0.0s
 => [linux/386 1/1] FROM docker.io/library/alpine@sha256:bc41182d7ef5ffc53a40b044e725193bc10142a1243f395ee852a8d9730fc2ad                 0.0s
 => => resolve docker.io/library/alpine@sha256:bc41182d7ef5ffc53a40b044e725193bc10142a1243f395ee852a8d9730fc2ad                           0.0s
 => [linux/arm64 1/1] FROM docker.io/library/alpine@sha256:bc41182d7ef5ffc53a40b044e725193bc10142a1243f395ee852a8d9730fc2ad               0.4s
 => => resolve docker.io/library/alpine@sha256:bc41182d7ef5ffc53a40b044e725193bc10142a1243f395ee852a8d9730fc2ad                           0.0s
 => => sha256:9b18e9b68314027565b90ff6189d65942c0f7986da80df008b8431276885218e 0B / 2.71MB                                                0.4s
 => exporting to image                                                                                                                    0.4s
 => => exporting layers                                                                                                                   0.0s
 => => exporting manifest sha256:1e3431ac147d5ab9aad7408f9e9cf77191b909b8d5eeee4c3074d29e012a7fcf                                         0.0s
 => => exporting config sha256:03a88cfd93db02177bb2b47b7fc73504ebf75b7cc3eb4432edc01caae5535098                                           0.0s
 => => exporting manifest sha256:89aebd05109a4dba92ff76ea73b52ded8286b2c5a30467c9810eb3c6b92db91a                                         0.0s
 => => exporting config sha256:9faa6ecf5f1b46d07c961b9a490a16391a9b1f7fb30cea316dbe10992595207a                                           0.0s
 => => exporting manifest sha256:c8ed6a8c58f8e50c90be744426398f3b2fcbea51d45f0e27a266b1a69cf99d5c                                         0.0s
 => => exporting config sha256:9cff1119afd2277751867756a94fe0f20495b47374cc84d8fe7096006cdf272e                                           0.0s
 => => exporting manifest list sha256:610e37756255c286b3e3706d3239e339f4b378487c4aafe03c0941cae65ff462                                    0.0s
 => => naming to docker.io/pawelgronowski465/pushworks:220824                                                                             0.0s
 => => unpacking to docker.io/pawelgronowski465/pushworks:220824                                                                          0.4s
root # docker push docker.io/pawelgronowski465/pushworks:220824
03a88cfd93db: Pushed
9b18e9b68314: Pushed
89aebd05109a: Pushed
c8ed6a8c58f8: Pushed
610e37756255: Pushed
9cff1119afd2: Pushed
6c0d3b419d84: Pushed
213ec9aee27d: Pushed
1e3431ac147d: Pushed
9faa6ecf5f1b: Pushed
root # docker buildx imagetools inspect docker.io/pawelgronowski465/pushworks:220824
Name:      docker.io/pawelgronowski465/pushworks:220824
MediaType: application/vnd.docker.distribution.manifest.list.v2+json
Digest:    sha256:610e37756255c286b3e3706d3239e339f4b378487c4aafe03c0941cae65ff462

Manifests:
  Name:      docker.io/pawelgronowski465/pushworks:220824@sha256:1e3431ac147d5ab9aad7408f9e9cf77191b909b8d5eeee4c3074d29e012a7fcf
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/arm64

  Name:      docker.io/pawelgronowski465/pushworks:220824@sha256:89aebd05109a4dba92ff76ea73b52ded8286b2c5a30467c9810eb3c6b92db91a
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/amd64

  Name:      docker.io/pawelgronowski465/pushworks:220824@sha256:c8ed6a8c58f8e50c90be744426398f3b2fcbea51d45f0e27a266b1a69cf99d5c
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/386

daemon/containerd/image_push.go

vvoland · 2022-08-25T08:32:32Z

Undrafting this as it allows the build+push case to work.
Will work on the pull+tag+push case separately.

daemon/containerd/image_push.go

vvoland · 2022-08-25T10:26:56Z

Added some comments and cleaned up the source label handling a bit.
Will squash the commits later when merging.

thaJeztah

ah, have a call, so need to continue later; submitted some nits (no blockers so far)

daemon/containerd/image_push.go

thaJeztah · 2022-08-25T13:28:54Z

daemon/containerd/image_push.go

+// Do a small peek of the content to check if content is definitely not a json.
+// Returns true when content is definitely not a json.
+// If it returns false, then the content may still not be a json.


Suggested change

// Do a small peek of the content to check if content is definitely not a json.

// Returns true when content is definitely not a json.

// If it returns false, then the content may still not be a json.

// peekNotJson does a small peek of the content to check if content is definitely not JSON.

// It returns true if content is definitely not JSON, or false if it was unable to detect if it's

// JSON or not.

The error doesn't not appear to be described (on how it should be used in combination with the bool)

Thanks. I didn't bother much describing the error case, as it's not really something that should happen under normal circumstances. This is called only when traversing the content store, so we expect the content to be there.
The only case of error here would be if the content is silently deleted in background between the beginning of the Walk and running handler for this particular content.
BTW, I wonder if there is some way to "lock" the store, so we can be 100% sure that this won't happen.

I think you can use a leased context maybe?

But how do I put a lease on everything in the Content store? :P
And if I do - will lease be enough to pause something like ctr content remove <hash> until the Walk is finished?

Yeah nah it doesn't work, the lease you add to your context is added to objects you create during that context.

thaJeztah

Left some comments, but mostly looks sane 😄

daemon/containerd/image_push.go

daemon/containerd/store.go

This makes it possible to push multi-platform images which contents are missing from the local content store, but can be fetched from a source repository. Co-authored-by: Djordje Lukic <[email protected]> Signed-off-by: Paweł Gronowski <[email protected]>

thaJeztah

LGTM, thanks!

Let's make sure we have a tracking ticket to improve and upstream a more official approach, and discuss with the containerd maintainers.

thaJeztah · 2022-09-01T10:56:54Z

When upstreaming:

depends on / needs to go after containerd integration: produce progress events polling ctrd's content.Store moby/moby#43819
but, we may want to squash all fixes related to "common" progress (push/pull) code into a single commit

combine with one or more of;

vvoland force-pushed the c8d-push-lazy-blobs branch from 9142d60 to 81bd076 Compare August 24, 2022 12:28

thaJeztah mentioned this pull request Aug 24, 2022

c8d/push: Push multi-platform images #56

Closed

rumpl reviewed Aug 24, 2022

View reviewed changes

daemon/containerd/image_push.go Outdated Show resolved Hide resolved

vvoland force-pushed the c8d-push-lazy-blobs branch 2 times, most recently from 9541dc3 to 6282eaa Compare August 25, 2022 08:31

vvoland marked this pull request as ready for review August 25, 2022 08:31

vvoland requested review from rumpl and thaJeztah August 25, 2022 08:32

rumpl reviewed Aug 25, 2022

View reviewed changes

daemon/containerd/image_push.go Show resolved Hide resolved

rumpl reviewed Aug 25, 2022

View reviewed changes

daemon/containerd/image_push.go Outdated Show resolved Hide resolved

rumpl reviewed Aug 25, 2022

View reviewed changes

daemon/containerd/image_push.go Show resolved Hide resolved

rumpl approved these changes Aug 25, 2022

View reviewed changes

thaJeztah reviewed Aug 25, 2022

View reviewed changes

vvoland force-pushed the c8d-push-lazy-blobs branch from 0720a48 to cf0b3d6 Compare August 25, 2022 16:24

thaJeztah approved these changes Aug 25, 2022

View reviewed changes

vvoland merged commit 39c445a into rumpl:c8d Aug 25, 2022

vvoland mentioned this pull request Aug 30, 2022

c8d/push: Fetch missing resources that can't be mounted #80

Merged

rumpl added the to-be-upstreamed label Aug 30, 2022

thaJeztah mentioned this pull request Sep 1, 2022

produce progress events polling ctrd's content.Store #12

Merged

vvoland added upstreamed to-be-upstreamed and removed to-be-upstreamed upstreamed labels Feb 9, 2023

vvoland mentioned this pull request Feb 9, 2023

c8d: Implement push moby/moby#44963

Merged

c8d/push: Push lazy blobs with distribution source labels #74

c8d/push: Push lazy blobs with distribution source labels #74

Uh oh!

Conversation

vvoland commented Aug 24, 2022

Uh oh!

Uh oh!

vvoland commented Aug 25, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vvoland commented Aug 25, 2022

Uh oh!

thaJeztah left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

thaJeztah Aug 25, 2022

Choose a reason for hiding this comment

Uh oh!

vvoland Aug 25, 2022

Choose a reason for hiding this comment

Uh oh!

rumpl Aug 25, 2022

Choose a reason for hiding this comment

Uh oh!

vvoland Aug 25, 2022

Choose a reason for hiding this comment

Uh oh!

rumpl Aug 25, 2022

Choose a reason for hiding this comment

Uh oh!

thaJeztah left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

thaJeztah left a comment

Choose a reason for hiding this comment

Uh oh!

thaJeztah commented Sep 1, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants