Description
As mentioned in the title of the ticket, docker pull under rootless fails with the following error for "certain" images:
failed to register layer: ApplyLayer exit status 1 stdout: stderr: lchown : invalid argument
Steps to reproduce the issue:
Using the docker rootless daemon, perform the following:
$ docker run --rm -it --entrypoint sh gcr.io/kubeflow-images-public/tensorflow-1.14.0-notebook-cpu:v0.7.0 -c "ls -l /usr/local/bin/docker-credential-gcr"
5b7339215d1d: Pull complete
14ca88e9f672: Pull complete
a31c3b1caad4: Pull complete
b054a26005b7: Pull complete
8832e3773578: Pull complete
5e671b828b2a: Pull complete
2b940936f993: Pull complete
016724bbd2c9: Pull complete
5bd1cb597025: Pull complete
68543864d644: Pull complete
7babe47a4c40: Pull complete
dc2840b44171: Pull complete
330a9002e0b4: Pull complete
107cba84ef3d: Pull complete
4b9d9f2fa2a2: Pull complete
d684674aa1a4: Pull complete
21a7832aeb86: Pull complete
5bd2e6f0de43: Pull complete
b5494e32d013: Pull complete
823f4685c03b: Pull complete
777cec03b3e2: Pull complete
01ad04a655b2: Pull complete
35daced67e59: Pull complete
b4ecb6928817: Pull complete
5bac0c144f6e: Extracting [==================================================>] 3.456MB/3.456MB
e3ab47ad84d9: Download complete
9269cef1ab8b: Download complete
21640f54008c: Download complete
daa5c419d33d: Download complete
4e8a6b90828e: Download complete
96685dce34a0: Download complete
92d24c89f5bc: Download complete
f44c204b0402: Download complete
failed to register layer: ApplyLayer exit status 1 stdout: stderr: lchown /usr/local/bin/docker-credential-gcr: invalid argument
Describe the results you received:
failed to register layer: ApplyLayer exit status 1 stdout: stderr: lchown /usr/local/bin/docker-credential-gcr: invalid argument
Describe the results you expected:
Successful pull of the image
Additional information you deem important (e.g. issue happens only occasionally):
This seems to be a unique case that we've run into as almost all other images work. So, the question is - is there an explanation for this and possibly a workaround -- maybe a rebuild of the image with some changes ?
Output of docker version:
# docker version
Client:
Version: 20.10.15
API version: 1.41
Go version: go1.17.9
Git commit: fd82621
Built: Thu May 5 13:10:20 2022
OS/Arch: linux/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.15
API version: 1.41 (minimum version 1.12)
Go version: go1.17.9
Git commit: 4433bf6
Built: Thu May 5 13:10:29 2022
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: v1.6.4
GitCommit: 212e8b6fa2f44b9c21b2798135fc6fb7c53efc16
runc:
Version: 1.1.1
GitCommit: v1.1.1-0-g52de29d7
docker-init:
Version: 0.19.0
GitCommit: de40ad0
Output of docker info:
# docker info
Client:
Context: default
Debug Mode: false
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 20.10.15
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: false
userxattr: true
Logging Driver: json-file
Cgroup Driver: none
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runtime.v1.linux runc io.containerd.runc.v2
Default Runtime: runc
Init Binary: docker-init
containerd version: 212e8b6fa2f44b9c21b2798135fc6fb7c53efc16
runc version: v1.1.1-0-g52de29d7
init version: de40ad0
Security Options:
seccomp
Profile: default
rootless
cgroupns
Kernel Version: 5.13.0-40-generic
Operating System: Alpine Linux v3.15
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.63GiB
Name: 6bf2401643c5
ID: MR3P:UTT7:UTBO:CT3P:ELMW:TDXB:BJFP:SWYL:62BS:MIIK:L5ZH:VBJ4
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine
WARNING: API is accessible on http://0.0.0.0:2375 without encryption.
Access to the remote API is equivalent to root access on the host. Refer
to the 'Docker daemon attack surface' section in the documentation for
more information: https://docs.docker.com/go/attack-surface/
WARNING: Running in rootless-mode without cgroups. Systemd is required to enable cgroups in rootless-mode.Additional environment details (AWS, VirtualBox, physical, etc.):
This is on my Ubuntu 20.04 virtual machine.
$ uname -a
Linux ubuntu 5.13.0-40-generic #45~20.04.1-Ubuntu SMP Mon Apr 4 09:38:31 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.4 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.4 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
Description
As mentioned in the title of the ticket, docker pull under rootless fails with the following error for "certain" images:
failed to register layer: ApplyLayer exit status 1 stdout: stderr: lchown : invalid argument
Steps to reproduce the issue:
Using the docker rootless daemon, perform the following:
Describe the results you received:
Describe the results you expected:
Successful pull of the image
Additional information you deem important (e.g. issue happens only occasionally):
This seems to be a unique case that we've run into as almost all other images work. So, the question is - is there an explanation for this and possibly a workaround -- maybe a rebuild of the image with some changes ?
Output of
docker version:Output of
docker info:Additional environment details (AWS, VirtualBox, physical, etc.):
This is on my Ubuntu 20.04 virtual machine.