Image built from cached layers is missing data

**Description**

I built an image on one machine using 
```
DOCKER_BUILDKIT=1 docker build . --build-arg BUILDKIT_INLINE_CACHE=1 --build-arg platform=generic --tag nubots/nubots:buildkit --no-cache
```

I then pushed this image up to docker hub and pulled it on a different machine. I then re-built the image on this second machine using
```
DOCKER_BUILDKIT=1 docker build . --build-arg BUILDKIT_INLINE_CACHE=1 --build-arg platform=generic --tag nubots:buildkit --cache-from nubots:buildkit,nubots/nubots:buildkit
```

As expected, this second build used the `nubots/nubots:buildkit` image for its cache and the build was completed almost instantaneously.

However, when trying to run commands in a container spun up on the `nubots:buildkit` image it immediately becomes apparent that there are lots of files missing in the image.

**Steps to reproduce the issue:**
1. Machine 1: `DOCKER_BUILDKIT=1 docker build . --build-arg BUILDKIT_INLINE_CACHE=1 --build-arg platform=generic --tag nubots/nubots:buildkit --no-cache`
2. Machine 1: `docker push nubots/nubots:buildkit`
3. Machine 2: `docker pull nubots/nubots:buildkit`
4. Machine 2: `DOCKER_BUILDKIT=1 docker build . --build-arg BUILDKIT_INLINE_CACHE=1 --build-arg platform=generic --tag nubots:buildkit --cache-from nubots:buildkit,nubots/nubots:buildkit`
5. Machine 2: `docker container run --rm --tty --workdir $(pwd) --name nubots_buildkit --attach stdin --attach stdout --attach stderr --hostname docker --network host --interactive nubots/nubots:buildkit pip freeze`
6. Machine 2: `docker container run --rm --tty --workdir $(pwd) --name nubots_buildkit --attach stdin --attach stdout --attach stderr --hostname docker --network host --interactive nubots:buildkit pip freeze`

**Describe the results you received:**

Output from step 4
```
#1 [internal] load build definition from Dockerfile
#1 DONE 0.0s

#2 [internal] load .dockerignore
#2 transferring context: 34B done
#2 DONE 5.0s

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 38B done
#1 DONE 14.9s

#3 [internal] load metadata for docker.io/library/archlinux:20200505
#3 DONE 2.5s

#6 [1/82] FROM docker.io/library/archlinux:20200505@sha256:5dc4a4df8cfe4745...
#6 DONE 0.0s

#10 [internal] load build context
#10 DONE 0.0s

#5 importing cache manifest from nubots/nubots:buildkit
#5 DONE 0.0s

#4 importing cache manifest from nubots:buildkit
#4 ERROR: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

#10 [internal] load build context
#10 transferring context: 1.69kB 0.0s done
#10 DONE 2.2s

#79 [73/82] RUN pip install --upgrade     cmake-format==0.6.9     isort==4.3...
#79 CACHED

#59 [53/82] RUN sudo install-package protobuf
#59 CACHED

#39 [33/82] RUN install-from-source https://github.com/intel/gmmlib/archive/...
#39 CACHED

#34 [28/82] RUN install-meson-from-source https://dri.freedesktop.org/libdrm...
#34 CACHED

#60 [54/82] RUN install-autotools-from-source https://github.com/protocolbuf...
#60 CACHED

#25 [19/82] COPY --chown=nubots:nubots usr/local/bin/install-from-source /us...
#25 CACHED

#62 [56/82] RUN install-from-source https://github.com/libjpeg-turbo/libjpeg...
#62 CACHED

#61 [55/82] RUN install-package yasm
#61 CACHED

#63 [57/82] RUN install-from-source https://github.com/jbeder/yaml-cpp/archi...
#63 CACHED

#81 [75/82] COPY --chown=nubots:nubots home/nubots/.ssh/id_rsa /home/nubots/...
#81 CACHED

#31 [25/82] RUN install-from-source https://www.zlib.net/zlib-1.2.11.tar.gz
#31 CACHED

#46 [40/82] RUN /usr/local/package/opencl-headers.sh https://github.com/Khro...
#46 CACHED

#51 [45/82] COPY --chown=nubots:nubots usr/local/package/openblas/generic.sh...
#51 CACHED

#21 [15/82] COPY --chown=nubots:nubots usr/local/toolchain/generate_toolchai...
#21 CACHED

#84 [78/82] COPY --chown=nubots:nubots home/nubots/.gdbinit /home/nubots/.gd...
#84 CACHED

#53 [47/82] RUN install-cmake-from-source https://downloads.sourceforge.net/...
#53 CACHED

#19 [13/82] RUN chown -R nubots:nubots /usr/local
#19 CACHED

#83 [77/82] COPY --chown=nubots:nubots home/nubots/.ssh/config /home/nubots/...
#83 CACHED

#33 [27/82] RUN install-from-source https://xorg.freedesktop.org/releases/in...
#33 CACHED

#27 [21/82] COPY --chown=nubots:nubots usr/local/toolchain/generate_generic_...
#27 CACHED

#11 [5/82] COPY --chown=root:root etc/sudoers.d/user /etc/sudoers.d/user
#11 CACHED

#86 [80/82] RUN install -d -m 0755 -o nubots -g nubots /home/nubots/NUbots
#86 CACHED

#65 [59/82] RUN install-header-from-source https://github.com/catchorg/Catch...
#65 CACHED

#45 [39/82] COPY --chown=nubots:nubots usr/local/package/opencl-headers.sh /...
#45 CACHED

#12 [6/82] RUN chmod 440 /etc/sudoers.d/user
#12 CACHED

#43 [37/82] COPY --chown=nubots:nubots usr/local/package/intel-compute-runti...
#43 CACHED

#49 [43/82] RUN install-from-source-with-patches https://github.com/OCL-dev/...
#49 CACHED

#40 [34/82] RUN install-from-source https://github.com/intel/libva/releases/...
#40 CACHED

#9 [4/82] RUN useradd --no-log-init -m -U nubots
#9 CACHED

#66 [60/82] RUN install-from-source http://xmlsoft.org/sources/libxml2-2.9.1...
#66 CACHED

#64 [58/82] RUN  install-from-source https://github.com/fmtlib/fmt/archive/6...
#64 CACHED

#24 [18/82] COPY --chown=nubots:nubots usr/local/bin/download-and-extract /u...
#24 CACHED

#57 [51/82] RUN install-from-source-with-patches https://gitlab.com/libeigen...
#57 CACHED

#68 [62/82] RUN install-from-source https://gitlab.gnome.org/GNOME/glib/-/ar...
#68 CACHED

#38 [32/82] RUN install-from-source https://github.com/intel/intel-graphics-...
#38 CACHED

#29 [23/82] COPY --chown=nubots:nubots usr/local/toolchain/generate_generic_...
#29 CACHED

#22 [16/82] COPY --chown=nubots:nubots usr/local/toolchain/generate_generic_...
#22 CACHED

#69 [63/82] RUN install-meson-from-source https://github.com/AravisProject/a...
#69 CACHED

#26 [20/82] RUN ln -s /usr/local/bin/install-from-source /usr/local/bin/inst...
#26 CACHED

#55 [49/82] COPY --chown=nubots:nubots usr/local/package/eigen3/deprecated-c...
#55 CACHED

#58 [52/82] RUN install-from-source     https://github.com/gperftools/gperft...
#58 CACHED

#52 [46/82] RUN /usr/local/package/openblas.sh https://github.com/xianyi/Ope...
#52 CACHED

#75 [69/82] COPY --chown=nubots:nubots usr/local/package/espeak.sh /usr/loca...
#75 CACHED

#8 [3/82] RUN pacman -Syyuu --noconfirm --needed --overwrite *
#8 CACHED

#7 [2/82] RUN echo "Server=https://archive.archlinux.org/repos/2020/05/05/$...
#7 CACHED

#32 [26/82] RUN BUILD_FILE_DIR='llvm' install-cmake-from-source     https://...
#32 CACHED

#77 [71/82] RUN pip install     pylint     termcolor     stringcase     Pill...
#77 CACHED

#17 [11/82] COPY --chown=root:root etc/ld.so.conf.d/usrlocal.conf /etc/ld.so...
#17 CACHED

#73 [67/82] RUN install-from-source ftp://ftp.alsa-project.org/pub/lib/alsa-...
#73 CACHED

#44 [38/82] RUN /usr/local/package/intel-compute-runtime.sh https://github.c...
#44 CACHED

#67 [61/82] RUN install-from-source https://github.com/libffi/libffi/release...
#67 CACHED

#85 [79/82] RUN chmod 600 /home/nubots/.ssh/id_rsa /home/nubots/.ssh/config
#85 CACHED

#20 [14/82] RUN cd /usr/local && ln -sf lib lib64
#20 CACHED

#28 [22/82] RUN sudo python /usr/local/generate_toolchain.py --prefix /usr
#28 CACHED

#18 [12/82] RUN ldconfig
#18 CACHED

#13 [7/82] COPY --chown=nubots:nubots usr/local/bin/install-package /usr/loc...
#13 CACHED

#15 [9/82] RUN echo $(python -c "import site; print(site.getsitepackages()[0...
#15 CACHED

#54 [48/82] COPY --chown=nubots:nubots usr/local/include/armadillo_bits/conf...
#54 CACHED

#56 [50/82] COPY --chown=nubots:nubots usr/local/package/eigen3/cxx17-overal...
#56 CACHED

#41 [35/82] RUN install-from-source https://github.com/intel/media-driver/ar...
#41 CACHED

#47 [41/82] RUN install-from-source https://github.com/KhronosGroup/OpenCL-C...
#47 CACHED

#82 [76/82] COPY --chown=nubots:nubots home/nubots/.ssh/id_rsa.pub /home/nub...
#82 CACHED

#80 [74/82] RUN install -d -m 0755 -o nubots -g nubots /home/nubots/.ssh
#80 CACHED

#76 [70/82] RUN /usr/local/package/espeak.sh https://github.com/espeak-ng/es...
#76 CACHED

#72 [66/82] RUN install-from-source https://github.com/ianlancetaylor/libbac...
#72 CACHED

#35 [29/82] RUN install-cmake-from-source https://github.com/KhronosGroup/SP...
#35 CACHED

#37 [31/82] RUN /usr/local/package/opencl-clang.sh https://github.com/intel/...
#37 CACHED

#74 [68/82] RUN install-from-source https://github.com/espeak-ng/pcaudiolib/...
#74 CACHED

#70 [64/82] RUN install-cmake-from-source https://github.com/libuv/libuv/arc...
#70 CACHED

#42 [36/82] COPY --chown=root:root etc/OpenCL/vendors/intel.icd /etc/OpenCL/...
#42 CACHED

#78 [72/82] RUN install-package     arm-none-eabi-gcc     arm-none-eabi-newl...
#78 CACHED

#36 [30/82] COPY --chown=nubots:nubots usr/local/package/opencl-clang.sh /us...
#36 CACHED

#23 [17/82] RUN sudo python /usr/local/generate_toolchain.py --prefix /usr
#23 CACHED

#14 [8/82] RUN install-package     sudo     wget     python     python-pip  ...
#14 CACHED

#50 [44/82] RUN install-package gcc-fortran
#50 CACHED

#87 [81/82] RUN install -d -m 0755 -o nubots -g nubots /home/nubots/build
#87 CACHED

#48 [42/82] RUN install-package ruby
#48 CACHED

#16 [10/82] COPY --chown=root:root usr/pip.conf /usr/pip.conf
#16 CACHED

#30 [24/82] RUN python /usr/local/generate_toolchain.py --prefix /usr/local
#30 CACHED

#71 [65/82] RUN install-from-source https://github.com/Fastcode/NUClear/arch...
#71 CACHED

#88 [82/82] WORKDIR /home/nubots/NUbots
#88 CACHED

#89 exporting to image
#89 exporting layers done
#89 writing image sha256:38f4d8e5a072983e029d658756d9fe82bfef3cf677e372bc99a4b6c1e9f38d11
#89 writing image sha256:38f4d8e5a072983e029d658756d9fe82bfef3cf677e372bc99a4b6c1e9f38d11 1.7s done
#89 naming to docker.io/library/nubots:buildkit
#89 naming to docker.io/library/nubots:buildkit 4.8s done
#89 DONE 12.9s

#90 exporting cache
#90 preparing build cache for export 0.0s done
#90 DONE 0.0s
------
 > importing cache manifest from nubots:buildkit:
------
```

Output from step 5
```
appdirs==1.4.3
astroid==2.4.2
attrs==19.3.0
black==19.10b0
CacheControl==0.12.6
chardet==3.0.4
click==7.1.2
cmake-format==0.6.9
colorama==0.4.3
contextlib2==0.6.0.post1
distlib==0.3.0
distro==1.5.0
html5lib==1.0.1
idna==2.9
isort==4.3.21
lazy-object-proxy==1.4.3
mccabe==0.6.1
meson==0.54.1
msgpack==1.0.0
ordered-set==3.1.1
packaging==20.3
pathspec==0.8.0
pep517==0.8.2
Pillow==7.2.0
progress==1.5
protobuf==3.11.2
pylint==2.5.3
pyparsing==2.4.7
pytoml==0.1.21
regex==2020.7.14
requests==2.23.0
retrying==1.3.3
six==1.14.0
stringcase==1.2.0
termcolor==1.1.0
toml==0.10.0
tqdm==4.47.0
typed-ast==1.4.1
urllib3==1.25.9
webencodings==0.5.1
wrapt==1.12.1
```

Output from step 6
```
appdirs==1.4.3
CacheControl==0.12.6
chardet==3.0.4
colorama==0.4.3
contextlib2==0.6.0.post1
distlib==0.3.0
distro==1.5.0
html5lib==1.0.1
idna==2.9
meson==0.54.1
msgpack==1.0.0
ordered-set==3.1.1
packaging==20.3
pep517==0.8.2
progress==1.5
pyparsing==2.4.7
pytoml==0.1.21
requests==2.23.0
retrying==1.3.3
six==1.14.0
toml==0.10.0
urllib3==1.25.9
webencodings==0.5.1
```

This is just one of many differences between the two images.


**Describe the results you expected:**

I expected to receive the exact same output from both steps 5 and 6.

**Additional information you deem important (e.g. issue happens only occasionally):**

Dockerfile and associated context can be found [here](https://github.com/NUbots/NUbots/tree/master/docker)

**Output of `docker version`:**

```
$ docker version
Client:
 Version:           19.03.12-ce
 API version:       1.40
 Go version:        go1.14.4
 Git commit:        48a66213fe
 Built:             Wed Jul  1 17:05:50 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          19.03.12-ce
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.14.4
  Git commit:       48a66213fe
  Built:            Wed Jul  1 17:05:26 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.3.4.m
  GitCommit:        d76c121f76a5fc8a462dc64594aea72fe18e1178.m
 runc:
  Version:          1.0.0-rc91
  GitCommit:        24a3cf88a7ae5f4995f6750654c0e2ca61ef4bb2
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683
```

**Output of `docker info`:**

```
$ docker info
Client:
 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 7
 Server Version: 19.03.12-ce
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: d76c121f76a5fc8a462dc64594aea72fe18e1178.m
 runc version: 24a3cf88a7ae5f4995f6750654c0e2ca61ef4bb2
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.7.8-arch1-1
 Operating System: Arch Linux
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 31.37GiB
 Name: bidski-beast
 ID: 7VDS:LNEA:32P4:MFRX:3AQ6:UIPP:GNCD:2M5N:XG6Y:A2XT:NABW:Q2H3
 Docker Root Dir: /media/RAID/docker
 Debug Mode: false
 Username: bidski
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
```

**Additional environment details (AWS, VirtualBox, physical, etc.):**

Docker is being run on a physical Arch Linux machine.

```
$ uname -a
Linux bidski-beast 5.7.8-arch1-1 #1 SMP PREEMPT Thu, 09 Jul 2020 16:34:01 +0000 x86_64 GNU/Linux
```

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Image built from cached layers is missing data #41219

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Image built from cached layers is missing data #41219

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions