Difference in image extraction between Docker CE 17.05 and 17.06

**Description**

Image extraction between Docker CE 17.05 and 17.06 results in different root filesystems. I am able to reproduce the issue with both aufs and overlay2 storage drivers.

I ran into this issue with an image from a private repository. If the information that follows isn't enough to identify the issue, please let me know and I will attempt to create a public image that reproduces the behaviour described.

**Steps to reproduce the issue:**

Using Docker 17.06 CE

1. `docker-machine create --virtualbox-boot2docker-url "https://github.com/boot2docker/boot2docker/releases/download/v17.06.0-ce/boot2docker.iso" -d virtualbox --engine-insecure-registry internal_repo 1706ce`
2. `eval $(docker-machine env 1706ce)`
3. `docker run --rm -ti internal_repo/image ls /`

**Describe the results you received:**

```
#
0001-gyp-always-install-into-PRODUCT_DIR.patch
0002-gyp-apply-https-codereview.chromium.org-11361103.patch
0003-gyp-don-t-use-links-at-all-just-copy-the-files-inste.patch
2015-01-30.md
@@iterator
AUTHORS
AUTHORS.md
BufferList.js
CHANGELOG.md
CHANGES
CHANGES.md
CONTRIBUTING.md
COPYING.txt
CodecMalformedException.java

<~1000 output lines truncated>
```

**Describe the results you expected:**

Using Docker 17.05 CE

1. `docker-machine create --virtualbox-boot2docker-url "https://github.com/boot2docker/boot2docker/releases/download/v17.05.0-ce/boot2docker.iso" -d virtualbox --engine-insecure-registry internal_repo 1705ce`
2. `eval $(docker-machine env 1705ce)`
3. `docker run --rm -ti internal_repo/image ls /`

result:

```
INFORMIXTMP  artifacts	bin  boot  database  dev  etc  home  lib  lib64  media	mnt  nohup.out	nonexistent  opt  proc	root  run  sbin  selinux  shared  srv  sys  tmp  usr  var
```

**Additional information**

The filesystem in the 17.06 case is not completely flattened. It does contain the directories shown in the 17.05 output, and they do contain files.

An excerpt follows showing the result of comparing file checksums/locations between the 17.06 and 17.05 cases:

```
< 00564742e3c70905fdb4d3318b01c827  /node_modules/es5-ext/.lint
---
> 00564742e3c70905fdb4d3318b01c827  /usr/lib/node_modules/npm/node_modules/node-gyp/node_modules/path-array/node_modules/array-index/node_modules/es6-symbol/node_modules/es5-ext/.lint
145a146,147
> 00564742e3c70905fdb4d3318b01c827  /usr/local/lib/node_modules/npm/node_modules/node-gyp/node_modules/path-array/node_modules/array-index/node_modules/es6-symbol/node_modules/es5-ext/.lint
> 00564742e3c70905fdb4d3318b01c827  /usr/local/n/versions/node/7.0.0/lib/node_modules/npm/node_modules/node-gyp/node_modules/path-array/node_modules/array-index/node_modules/es6-symbol/node_modules/es5-ext/.lint
322a325,327
> 00ac2be339a766e2df37641d0350c63e  /usr/lib/node_modules/npm/node_modules/read-package-json/node_modules/glob/node_modules/path-is-absolute/package.json
> 00ac2be339a766e2df37641d0350c63e  /usr/local/lib/node_modules/npm/node_modules/read-package-json/node_modules/glob/node_modules/path-is-absolute/package.json
> 00ac2be339a766e2df37641d0350c63e  /usr/local/n/versions/node/7.0.0/lib/node_modules/npm/node_modules/read-package-json/node_modules/glob/node_modules/path-is-absolute/package.json
492a498,500
> 01028d57163f4901221a7a457774e202  /usr/lib/node_modules/npm/node_modules/init-package-json/node_modules/glob/node_modules/path-is-absolute/package.json
> 01028d57163f4901221a7a457774e202  /usr/local/lib/node_modules/npm/node_modules/init-package-json/node_modules/glob/node_modules/path-is-absolute/package.json
> 01028d57163f4901221a7a457774e202  /usr/local/n/versions/node/7.0.0/lib/node_modules/npm/node_modules/init-package-json/node_modules/glob/node_modules/path-is-absolute/package.json
493a502
> 0103318a939e4c2198689f6f143e556d  /usr/lib/node_modules/npm/node_modules/request/node_modules/http-signature/node_modules/sshpk/lib/formats/x509.js
495c504,505
< 0103318a939e4c2198689f6f143e556d  /x509.js
---

```
It can be seen that several files are missing.

`/usr/lib/node_modules/npm/node_modules/node-gyp/node_modules/path-array/node_modules/array-index/node_modules/es6-symbol/node_modules/es5-ext/.lint` under 17.05 moved to `/node_modules/es5-ext/.lint` in 17.06 and `/usr/lib/node_modules/npm/node_modules/request/node_modules/http-signature/node_modules/sshpk/lib/formats/x509.js` in 17.05 moved to `/x509.js` in 17.06. File contents are unaffected.

Here is the output of `docker inspect internal_repo/image` for the 17.05 case:

```
[
    {
        "Id": "sha256:a126cdbff382390a94be039424f1db5f6022ecb4444f6c4ea78853cc774dee12",
        "RepoTags": [
            "internal_repo/image:latest"
        ],
        "RepoDigests": [
            "internal_repo/image@sha256:bc287cd9d506ede8ccd4d5c083ba71b60157b85af90188044860e2cffa5b996d"
        ],
        "Parent": "",
        "Comment": "",
        "Created": "2017-07-05T11:06:34.148937884Z",
        "Container": "d802f466bfefc17dce309a7ec63ea22929377e70f3105f03754d01f298aba457",
        "ContainerConfig": {
            "Hostname": "d802f466bfef",
            "Domainname": "",
            "User": "",
            "AttachStdin": true,
            "AttachStdout": true,
            "AttachStderr": true,
            "Tty": true,
            "OpenStdin": true,
            "StdinOnce": true,
            "Env": null,
            "Cmd": [
                "/bin/bash"
            ],
            "Image": "internal_repo/image",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {}
        },
        "DockerVersion": "17.05.0-ce",
        "Author": "",
        "Config": {
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": null,
            "Cmd": [
                "/bin/bash"
            ],
            "Image": "",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {}
        },
        "Architecture": "amd64",
        "Os": "linux",
        "Size": 3265786582,
        "VirtualSize": 3265786582,
        "GraphDriver": {
            "Data": null,
            "Name": "aufs"
        },
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:3cb5cef9e16b6c2fc4c25591b64c3a37ee728e41a9dbb02be96e94ee0784536d",
                "sha256:cf19831f7cead5e80c994c75812f26e8010ad4d422b1693b73cd2b06e8890913",
                "sha256:092bf598ac10f3529cb9225e560c903aaae3ed524e9e793c74b5c1ea71cf26de"
            ]
        }
    }
]
```

The only difference in the output for 17.06 is

```
<         "Size": 3265786582,
<         "VirtualSize": 3265786582,
---
>         "Size": 3258003733,
>         "VirtualSize": 3258003733,
```
I have confirmed that in both the 17.05 and 17.06 cases that the `/var/lib/docker/tmp/GetImageBlob*` files created have identical sha256sums, which means the same layers are being pulled from the repo in both cases.

`/var/log/docker.log` does not show any usual errors or output when extracting in either case. Output from the 17.06 case:

```
time="2017-07-13T09:19:18.115570865Z" level=debug msg="Downloaded 515ece5796b4 to tempfile /mnt/sda1/var/lib/docker/tmp/GetImageBlob669060717" 
time="2017-07-13T09:19:19.312413910Z" level=debug msg="Downloaded c10ec98cceed to tempfile /mnt/sda1/var/lib/docker/tmp/GetImageBlob298385640" 
time="2017-07-13T09:21:35.851669441Z" level=debug msg="Downloaded a697c759d873 to tempfile /mnt/sda1/var/lib/docker/tmp/GetImageBlob421049798" 
time="2017-07-13T09:22:55.763161662Z" level=debug msg="Applied tar sha256:3cb5cef9e16b6c2fc4c25591b64c3a37ee728e41a9dbb02be96e94ee0784536d to b60e75153572e514ebc5fa9f6f4589a3306f42d5bd4b738edba74e5b538e6124
, size: 3250951555" 
time="2017-07-13T09:22:55.841672470Z" level=debug msg="Applied tar sha256:cf19831f7cead5e80c994c75812f26e8010ad4d422b1693b73cd2b06e8890913 to c6fb8d7b5c89bfb515dbc6f7ad1e3fc189319c2fb4e4255370883b186e7af36e
, size: 807" 
time="2017-07-13T09:22:55.978860870Z" level=debug msg="Applied tar sha256:092bf598ac10f3529cb9225e560c903aaae3ed524e9e793c74b5c1ea71cf26de to efb16627adf62001b66723902dcc85b58fc4972ff314120ab18185b187c08e45
, size: 7051371" 
```

When reproducing the issue with overlay2, I can confirm that the `/diff` directory of the `a697c759d873` layer in `/var/lib/docker` contains the incorrectly extracted file paths before starting any containers.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Difference in image extraction between Docker CE 17.05 and 17.06 #34092

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Difference in image extraction between Docker CE 17.05 and 17.06 #34092

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions