Conversation
GordonTheTurtle
added
area/distribution
aaronlehmann
force-pushed
the
content-addressability
branch
3 times, most recently
from
8b59d05 to
1d793b1
Compare
|
@swernli - Stefan can you review from the Windows side please? |
aaronlehmann
force-pushed
the
content-addressability
branch
4 times, most recently
from
49ec06e to
44e6989
Compare
Curious; will repeated upgrades / downgrades work? (e.g., Used 1.10 for testing, then continue on 1.9 and run 1.10 again) |
New images built or pulled in 1.10 wouldn't be visible to 1.9. But the original images would still be there, unless you manually deleted some of them. |
daemon/daemon.go
Outdated
There was a problem hiding this comment.
should this be versioned? or can't it anymore?
There was a problem hiding this comment.
VirtualSize as a concept has been removed as all images freely share all of their layers and no layer is unique to an image. So the value Size is now same that was previously VirtualSize.
I changed the PR so that VirtualSize isn't cleared any more but shows same data. That should provide correct data to older clients. New clients don't use this field any more. As this field isn't useful for new clients we can figure out a deprecation path and this can be done through API versions, but I think we can do that after this PR is merged.
There was a problem hiding this comment.
cool thanks @tonistiigi just making sure we won't forget this
|
Reviewing for Windows... |
aaronlehmann
force-pushed
the
content-addressability
branch
from
59afa95 to
9c6133a
Compare
|
ping @docker/maintainers please review and test this one. It's a huge change, but really exciting stuff. also: |
|
\o/ On Thu, Nov 12, 2015 at 12:56 PM, Sebastiaan van Stijn <
|
|
A difference I see with this one (probably expected, but just to verify) Dockerfile FROM ubuntu:14.04
ENV foo=bar
RUN echo hello > /foodocker build -t foo .Docker 1.9: docker history foobar
IMAGE CREATED CREATED BY SIZE COMMENT
408581b4301b 7 seconds ago /bin/sh -c echo hello > /foo 6 B
012645df8321 7 seconds ago /bin/sh -c #(nop) ENV foo=bar 0 B
e9ae3c220b23 2 days ago /bin/sh -c #(nop) CMD ["/bin/bash"] 0 B
a6785352b25c 2 days ago /bin/sh -c sed -i 's/^#\s*\(deb.*universe\)$/ 1.895 kB
0998bf8fb9e9 2 days ago /bin/sh -c echo '#!/bin/sh' > /usr/sbin/polic 194.5 kB
0a85502c06c9 2 days ago /bin/sh -c #(nop) ADD file:531ac3e55db4293b8f 187.7 MBWith this PR: docker history foobar
IMAGE CREATED CREATED BY SIZE COMMENT
a12448c314a7 6 minutes ago /bin/sh -c echo hello > /foo 2.048 kB
e39f86d96664 6 minutes ago /bin/sh -c #(nop) ENV foo=bar 0 B
4b1e42b414f6 2 days ago /bin/sh -c #(nop) CMD ["/bin/bash"] 1.024 kB
<missing> 2 days ago /bin/sh -c sed -i 's/^#\s*\(deb.*universe\)$/ 4.608 kB
<missing> 2 days ago /bin/sh -c echo '#!/bin/sh' > /usr/sbin/polic 208.9 kB
<missing> 2 days ago /bin/sh -c #(nop) ADD file:531ac3e55db4293b8f 196.8 MBDifferences:
|
|
@thaJeztah The sizes are currently from tars so they have the added padding. We are exploring ways to revert this if possible. |
|
@tonistiigi correct, it was a fresh pull |
|
Currently, these changes break the use of alternate ID files in layers necessary for Windows containers to work. I'm looking for a fix that can be applied to these changes to unblock us. |
|
Not in doc review but this could have documentation impact that is pretty big depending on the implementation. So, engineering hours could be required in terms of reviewing at the least. I'd like to include this information early in the PR lifecycle.
|
|
|
@aaronlehmann @tonistiigi Hello, Is there any tools that I can used to calculate the content-hash of a tar file? |
|
@dbdd4us The content hash of a layer is the hash of the compressed content, as sent to the registry. |
|
@stevvooe layer hashes in image config are the hashes of uncompressed content. |
When content addressablity was introduced in moby#17924, a compatibility layer for registry v1 pushes was added. When the engine is asked to push an image to a v1 registry it needs to create v1 IDs for the images. The strategy so far has been to use the full imageID for the first v1 layer and the ChainID for all other layers, effectively creating as many v1 layers as there are in the image. Only the top most layer contained the image configuration and the other layers had a dummy json containing only a parent reference. This becomes problematic when the first layer of the image is big. Consinder the following two Dockerfiles: FROM busybox RUN create_very_big_file CMD /foo FROM busybox RUN create_very_big_file CMD /bar Both of these images will have the exact same layers, with the layer created by `RUN create_very_big_file` being the topmost one, but their imageIDs will differ since they have a different CMD and therefore different image configs. When pushing to a v1 registry, the `RUN create_very_big_file` layer will be pushed twice, once with the v1 ID set to foo's imageID and once with the v1 ID set to bar's imageID. Also, any clients wanting to pull those images won't realise it's the same layer and will proceed to download it twice. This commit solves this problem by separating the layers from the image configuration information when pushing to a v1 registry. To do this, all layers of an image are pushed with their ChainIDs and a synthetic top level layer is created with its contents set to the EmptyLayer, it's config set to the image config, and its v1 ID set to the imageID. This will have the side-effect of adding one layer. To prevent new layers being piled on top of each other forever, the code checks if the topmost layer is already an empty layer and in that case it uses that for the image configuration. Signed-off-by: Petros Angelatos <[email protected]>
When content addressablity was introduced in moby#17924, a compatibility layer for registry v1 pushes was added. When the engine is asked to push an image to a v1 registry it needs to create v1 IDs for the images. The strategy so far has been to use the full imageID for the first v1 layer and the ChainID for all other layers, effectively creating as many v1 layers as there are in the image. Only the top most layer contained the image configuration and the other layers had a dummy json containing only a parent reference. This becomes problematic when the first layer of the image is big. Consinder the following two Dockerfiles: FROM busybox RUN create_very_big_file CMD /foo FROM busybox RUN create_very_big_file CMD /bar Both of these images will have the exact same layers, with the layer created by `RUN create_very_big_file` being the topmost one, but their imageIDs will differ since they have a different CMD and therefore different image configs. When pushing to a v1 registry, the `RUN create_very_big_file` layer will be pushed twice, once with the v1 ID set to foo's imageID and once with the v1 ID set to bar's imageID. Also, any clients wanting to pull those images won't realise it's the same layer and will proceed to download it twice. This commit solves this problem by separating the layers from the image configuration information when pushing to a v1 registry. To do this, all layers of an image are pushed with their ChainIDs and a synthetic top level layer is created with its contents set to the EmptyLayer, it's config set to the image config, and its v1 ID set to the imageID. This will have the side-effect of adding one layer. To prevent new layers being piled on top of each other forever, the code checks if the topmost layer is already an empty layer and in that case it uses that for the image configuration. Signed-off-by: Petros Angelatos <[email protected]>
| mountID := name | ||
| if runtime.GOOS != "windows" { | ||
| // windows has issues if container ID doesn't match mount ID | ||
| mountID = stringid.GenerateRandomID() |
There was a problem hiding this comment.
Hi, i'm investigating moby source code, when i read this, i doubt if there is any reason that moby do not always use name regard of windows or not. Is there any consideration not commented here?
The v1.10 layout and the migrator was added in 2015 via moby#17924. Although the migrator is not marked as "deprecated" explicitly in cli/docs/deprecated.md, I suppose people should have already migrated from pre-v1.10 and they no longer need the migrator, because pre-v1.10 version do not support schema2 images (and these versions no longer receives security updates). Signed-off-by: Akihiro Suda <[email protected]>
The v1.10 layout and the migrator was added in 2015 via moby#17924. Although the migrator is not marked as "deprecated" explicitly in cli/docs/deprecated.md, I suppose people should have already migrated from pre-v1.10 and they no longer need the migrator, because pre-v1.10 version do not support schema2 images (and these versions no longer receives security updates). Signed-off-by: Akihiro Suda <[email protected]>
20 participants
This PR changes the Docker engine to use content addressable storage for images and layers. This means that image IDs are no longer arbitrarily assigned - now they are computed based on the filesystem contents and the image configuration. Images can securely share their underlying layers without duplicating data on disk.
This work involves a big refactoring of image-related code in the engine. The
graphpackage is entirely removed, and is replaced by a few new packages:layer: Layer store interface and implementation. This manages filesystem layers without being aware of image configurations. The layer store maintains reference counts for layers and removes unreferenced layers.image: Image store interface and implementation. This manages image configurations. Image configurations include runtime configuration and reference the underlying filesystem data.tag: The tag functionality from thegraphpackage has been separated out and moved here. Tags now use types from thedistribution/referencepackage instead of plain strings.distribution: Push and pull code that originally was part of thegraphpackage has been separated out into its own package.The first time a version of the engine with these commits is started, it will migrate old graph metadata to the new format. This involves calculating content hashes for the existing data, but it does not move underlying graphdriver filesystem data. It doesn’t remove old graph metadata, so the migration process is not destructive.
The new data model does not have a one-to-one relationship between images and layers. A single image can have many layers. Existing versions of Docker create an image for each layer, and use the parent chain to link them together. That means that pulling a specific image requires pulling all the artifacts from the original build process of that image. With this PR, when an image is pulled from a registry, only a single image is created, to match the new data model. The history is preserved through a list of commands, dates, etc.
Summary of UI Changes:
sha256:prefix. This prefix is hidden for truncated image IDs for convenience.Future work:
Unit test coverage for the new packages:
For the full design document, see https://gist.github.com/aaronlehmann/b42a2eaf633fc949f93b