contrib: Docker v1 image importer#1602
Conversation
Codecov Report
@@ Coverage Diff @@
## master #1602 +/- ##
==========================================
+ Coverage 41.06% 42.43% +1.36%
==========================================
Files 66 73 +7
Lines 7748 8430 +682
==========================================
+ Hits 3182 3577 +395
- Misses 4064 4331 +267
- Partials 502 522 +20
Continue to review full report at Codecov.
|
|
|
||
| The default selector string is <ref>, but it is not guaranteed to be a valid selector string, depending on the importer implementation. | ||
| `, | ||
| Flags: []cli.Flag{ |
There was a problem hiding this comment.
This selector interface might be too much overloaded. RFC.
There was a problem hiding this comment.
Can you use the filter package?
|
@AkihiroSuda Thanks for working on this! |
ee82ae4 to
410f4a5
Compare
|
updated PR |
| // OCI format is assumed by default. | ||
| // Filter strings are format-specific, but needs to match exactly one item. | ||
| // | ||
| // TODO(AkihiroSuda): support importing multiple images from a single archive at once? (map[ref]filter?) |
There was a problem hiding this comment.
For us, we only need loading one image from tarball for now. We only build one image into each tar in the Kubernetes building process.
| func (c *Client) Import(ctx context.Context, ref string, reader io.Reader, opts ...ImportOpt) (Image, error) { | ||
| iopts, err := resolveImportOpt(ref, opts...) | ||
| // Note that unreferenced blobs may be imported to the content store as well. | ||
| func (c *Client) Import(ctx context.Context, ref string, reader io.Reader, filter string, opts ...ImportOpt) (Image, error) { |
There was a problem hiding this comment.
On second thought, should I remove filters, and change this function definition to Import (ctx context.Context, reader io.Reader, opts ...ImportOpt)(map[string]Image, error)?
The map key for map[string]Image is reference string, which is determined by the importer implementation.
(However, it may override existing refs... should we let importer add some prefix to refs?)
There was a problem hiding this comment.
What would the prefix represent? Various importers?
There was a problem hiding this comment.
Suppose we are importing Docker v1 Image archive which has RepoTags: [docker.io/library/busybox:latest, busybox:latest] and RepoTags: [docker.io/library/nginx:latest]. (Note that a single archive contain multiple different images)
prefix := "tmp-42424242.local/"
importer := &docker1.Importer{Prefix: prefix}
res, err := Import (ctx, tarReader, WithImporter(importer))Then the result map keys will be tmp-424242.local/docker.io/library/busybox:latest, tmp-424242.local/busybox:latest, and tmp-424242.local/docker.io/library/nginx:latest.
The caller can rename these images later by itself.
For OCI images, which has tags (e.g. latest) but no repotags (e.g. busybox:latest), prefix might be like tmp-424242.local/busybox:.
cc @stevvooe @dmcgowan @Random-Liu RFC
There was a problem hiding this comment.
@AkihiroSuda What is this filter used for? Selecting an image from the tarball when there are multiple images inside the tar? I didn't quite get it.
There was a problem hiding this comment.
I'm suggesting remove filters here to support loading multiple images from a single archive at once. #1602 (comment)
There was a problem hiding this comment.
Hm, the prefix part seems confusing to me. :p
Are you trying to avoid conflict refs inside the tarball? Or conflict ref with existing ref in containerd?
If it's the former, is that going to happen?
If it's the later, how does Docker solve it today?
There was a problem hiding this comment.
Latter and it is not resolved in Docker (just override).
For importing v1 image, you could omit prefix with risk of overriding.
But for importing OCI image, we somehow need to pass extra prefix (and default tag?), because OCI images lack RepoTags-equivalent.
410f4a5 to
c9ff059
Compare
|
Hi @AkihiroSuda! From just reading the description I'm not sure what |
| if c.format != "" { | ||
| return errors.New("format already set") | ||
| if c.importer != nil { | ||
| return errors.New("impoter already set") |
There was a problem hiding this comment.
nit: s/impoter/importer
| return c.importFromOCITar(ctx, ref, reader, iopts) | ||
| default: | ||
| return nil, errors.Errorf("unsupported format: %s", iopts.format) | ||
| imgrec := images.Image{ |
There was a problem hiding this comment.
what does rec mean here?
There was a problem hiding this comment.
Probably "record" (copy-pasted from Pull())
|
@stevvooe I've tried this, and this does work. Can we get this in anytime soon? This is blocking our work to integrate cri-containerd with Thanks! |
|
@Random-Liu I think this PR still needs discussion about how to import multiple images in a single archive at once. Since we are not going to stabilize the client library in v1.0, IIUC it is not impossible to merge this PR to the upstream right now, but could you consider copy-pasting the code of this PR to CRI-containerd side? |
|
@AkihiroSuda The problem is that we need a command line tool to load image during Kubernetes bootstrap. We do have our own command line tool The only command line tool we could use for One option is to cherypick this commit, and maintain our own temporary containerd branch for us to use for now. I'm fine with that if we think this PR may take longer to merge. |
|
@Random-Liu How about adding some CRI-containerd-specific utility command to CRI-containerd repo, which CRI-containerd maintainers can keep compatibility as they want? |
|
@AkihiroSuda I just checked, we do use Since we control containerd version in our production environment today, as long as the implementation is stable, I'm fine with If we do find that |
|
@stevvooe @Random-Liu shall we discuss this in person tomorrow? |
|
As discussed in Moby Summit:
I'll try to update this PR next week (if WiFi at the hotel is fine😅) |
|
s/ctr-containerd/cri-containerd/ |
I'll use
I'll make change 1) based on this PR, and maintain our own fork for now. |
c9ff059 to
5469ff1
Compare
|
Updated PR. Please review the CLI design. Fixes for client API, minor bugs, and typos will be follow-up PR if needed. |
5469ff1 to
b4e2a49
Compare
8fec634 to
30d0b3b
Compare
|
@Random-Liu @dmcgowan @stevvooe PTAL? I think the GC&interface fix for OCI importer should be merged before v1.0 GA. For Docker v1 importer, no need to hurry, but it is placed on |
There was a problem hiding this comment.
This doesn't look right: the normalization should ignore the tag.
There was a problem hiding this comment.
You meant it should NOT ignore the tag?
This code is copypasted from CRI-containerd.
Does CRI-containerd need to be modified?
There was a problem hiding this comment.
Perhaps. It shouldn't be dropping information from the image reference.
There was a problem hiding this comment.
This is specific to cri-containerd I think. The reason is that in CRI there is only repoDigest and repoTag. I remember I tried docker, and if we pull a image which has both tag and digest, it only keep the digest and ignore the tag. Let me try again.
In containerd, we should do whatever is right. :)
There was a problem hiding this comment.
$ docker pull busybox:latest@sha256:2107a35b58593c58ec5f4e8f2c4a70d195321078aebfadfbfb223a2ff4a4ed21
sha256:2107a35b58593c58ec5f4e8f2c4a70d195321078aebfadfbfb223a2ff4a4ed21: Pulling from library/busybox
d070b8ef96fc: Pull complete
Digest: sha256:2107a35b58593c58ec5f4e8f2c4a70d195321078aebfadfbfb223a2ff4a4ed21
Status: Downloaded newer image for busybox@sha256:2107a35b58593c58ec5f4e8f2c4a70d195321078aebfadfbfb223a2ff4a4ed21
$ docker inspect sha256:f6e427c148a766d2d6c117d67359a0aa7d133b5bc05830a7ff6e8b64ff6b1d1d
[
{
"Id": "sha256:f6e427c148a766d2d6c117d67359a0aa7d133b5bc05830a7ff6e8b64ff6b1d1d",
"RepoTags": [],
"RepoDigests": [
"busybox@sha256:2107a35b58593c58ec5f4e8f2c4a70d195321078aebfadfbfb223a2ff4a4ed21"
],
"Parent": "",
"Comment": "",
"Created": "2018-02-28T22:14:49.023807051Z",
"Container": "8d2c840a1a9b2544fe713c2e24b6757d52328f09bdfc9c2ef6219afbf7ae6b59",
"ContainerConfig": {
"Hostname": "8d2c840a1a9b",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": [
"/bin/sh",
"-c",
"#(nop) ",
"CMD [\"sh\"]"
],
"ArgsEscaped": true,
"Image": "sha256:8cae5980d887cc55ba2f978ae99c662007ee06d79881678d57f33f0473fe0736",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": null,
"OnBuild": null,
"Labels": {}
},
"DockerVersion": "17.06.2-ce",
"Author": "",
"Config": {
"Hostname": "",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": [
"sh"
],
"ArgsEscaped": true,
"Image": "sha256:8cae5980d887cc55ba2f978ae99c662007ee06d79881678d57f33f0473fe0736",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": null,
"OnBuild": null,
"Labels": null
},
"Architecture": "amd64",
"Os": "linux",
"Size": 1146369,
"VirtualSize": 1146369,
"GraphDriver": {
"Name": "aufs",
"Data": null
},
"RootFS": {
"Type": "layers",
"Layers": [
"sha256:c5183829c43c4698634093dc38f9bee26d1b931dedeba71dbee984f42fe1270d"
]
}
}
]There was a problem hiding this comment.
added Conservative bool for switching the behavior.
If true, it removes tag as in CRI implementation.
30d0b3b to
b0dca29
Compare
b0dca29 to
170370a
Compare
|
Where does this PR sit against #2175? |
|
@stevvooe For CRI, we do still need a load command, until we remove the internal in-memory cache for images, and always list images from containerd. Since we've removed grpc, it makes sense to always list images from containerd. We can:
|
170370a to
50bba91
Compare
|
Updated, but feel free to postpone this to v1.2 if there is a problem. |
For usage, please refer to `ctr images import --help`. Signed-off-by: Akihiro Suda <[email protected]>
50bba91 to
b163f65
Compare
| /* | ||
| Copyright The docker/distribution Authors. | ||
|
|
||
| Licensed under the Apache License, Version 2.0 (the "License"); |
There was a problem hiding this comment.
This redundancy is for making ltag happy.
I think ideally we should just list a single Apache boilerplate with Copyright The docker/distribution Authors and containerd Authors.
|
@AkihiroSuda The |
|
temporarily closing for now |
|
containerd/cri#865 is merged. We should be able to see images loaded/pulled into containerd with If we can get this in containerd, we can deprecate the |
|
I am working on that now |
|
@dmcgowan Cool! :D |
EDIT (Apr 10, 2018): TODO: cherry-pick containerd/cri#727
EDIT (Dec 5, 2017): Now OCI fixes were merged separately via #1880, and almost whole this PR consists of just copy-pasted code from containerd/cri#360 . So now this PR should be easy to review.
This PR adds an importer for Docker v1 images.
ctr help
Importing Docker images
You can import multiple images on a single archive at once.
RepoTags are normalized in
docker.io/library/foo:barform:GC labels:
Importing OCI images
In this example,
skopeois used for preparing an OCI image to import, but it is also possible to create an OCI image from the containerd store:ctr images export a.tar docker.io/library/busybox:latestAbout this PR
busybox->docker.io/library/busybox:latest) is based on @Random-Liu 's CRI-containerd implementation Add image load cri#360 , so the design should be now OK.github.com/docker/distribution/referenceasgithub.com/containerd/containerd/contrib/docker1/reference. So please skip reviewing this pkg. This pkg should soon go away.This PR also contains fix for OCI importer, which were not updated for GCOCI fix was merged in importer: refactor and fix GC #1880 )Update #1229
@Random-Liu @stevvooe