Description
After specified multiple registry mirrors for one registry, and they have different username and password, I found that crictl is unable to pull from the second registry mirror or third one.
But if I moved the second registry mirror to first, then I'm able to pull from this new first registry mirror, but not the new second one.
It will be able to pull from second one only if the second one doesn't need authentication.
Steps to reproduce the issue:
I configured three registry mirrors for gcr.io: http://172.18.0.7:5000, https://172.18.0.30:5010, https://172.18.0.20:5007. The 172.18.0.30:5010 one and 172.18.0.20:5007 one need authentication, and I have specified the auth information in the settings.
containerd's config.toml is like below:
[plugins."io.containerd.grpc.v1.cri".registry]
[plugins."io.containerd.grpc.v1.cri".registry.configs]
[plugins."io.containerd.grpc.v1.cri".registry.configs."gcr.io"]
[plugins."io.containerd.grpc.v1.cri".registry.configs."gcr.io".auth]
auth = 'auth'
[plugins."io.containerd.grpc.v1.cri".registry.configs."172.18.0.20:5007".tls]
ca_file = '/etc/containerd/certs.d/172.18.0.20:5007/ca.crt'
[plugins."io.containerd.grpc.v1.cri".registry.configs."172.18.0.20:5007".auth]
auth = 'dGVzdHVzZXI6dGVzdHBhc3N3b3Jk'
[plugins."io.containerd.grpc.v1.cri".registry.configs."172.18.0.30:5010".tls]
ca_file = '/etc/containerd/certs.d/172.18.0.30:5010/ca.crt'
[plugins."io.containerd.grpc.v1.cri".registry.configs."172.18.0.30:5010".auth]
auth = 'dGVzdHVzZXIyOnRlc3RwYXNzd29yZDI='
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."gcr.io"]
endpoint = ["http://172.18.0.7:5000", "https://172.18.0.30:5010", "https://172.18.0.20:5007"]
I have uploaded a image into 172.18.0.7 named second-hello:latest
Also uploaded an image into 172.18.0.30 named hello-world:latest
Uploaded an image into 172.18.0.20 named third-hello:latest
Run crictl pull gcr.io/second-hello
It pulled successfully: Image is up to date for sha256:d1165f2212346b2bab48cb01c1e39ee8ad1be46b87873d9ca7a4e434980a7726
Run crictl pull gcr.io/hello-world
It pulled successfully: Image is up to date for sha256:d1165f2212346b2bab48cb01c1e39ee8ad1be46b87873d9ca7a4e434980a7726
Run crictl pull gcr.io/third-world
Failed: FATA[0000] pulling image: rpc error: code = Unknown desc = failed to pull and unpack image "gcr.io/third-hello:latest": failed to resolve reference "gcr.io/third-hello:latest": unexpected status code [manifests latest]: 401 Unauthorized
Update containerd's config.toml, change registry mirror's sequence only:
[plugins."io.containerd.grpc.v1.cri".registry]
[plugins."io.containerd.grpc.v1.cri".registry.configs]
[plugins."io.containerd.grpc.v1.cri".registry.configs."gcr.io"]
[plugins."io.containerd.grpc.v1.cri".registry.configs."gcr.io".auth]
auth = 'auth'
[plugins."io.containerd.grpc.v1.cri".registry.configs."172.18.0.20:5007".tls]
ca_file = '/etc/containerd/certs.d/172.18.0.20:5007/ca.crt'
[plugins."io.containerd.grpc.v1.cri".registry.configs."172.18.0.20:5007".auth]
auth = 'dGVzdHVzZXI6dGVzdHBhc3N3b3Jk'
[plugins."io.containerd.grpc.v1.cri".registry.configs."172.18.0.30:5010".tls]
ca_file = '/etc/containerd/certs.d/172.18.0.30:5010/ca.crt'
[plugins."io.containerd.grpc.v1.cri".registry.configs."172.18.0.30:5010".auth]
auth = 'dGVzdHVzZXIyOnRlc3RwYXNzd29yZDI='
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."gcr.io"]
endpoint = ["https://172.18.0.20:5007", "http://172.18.0.7:5000", "https://172.18.0.30:5010"]
Run crictl pull gcr.io/second-hello
It pulled successfully: Image is up to date for sha256:d1165f2212346b2bab48cb01c1e39ee8ad1be46b87873d9ca7a4e434980a7726
Run crictl pull gcr.io/hello-world
Failed: FATA[0000] pulling image: rpc error: code = Unknown desc = failed to pull and unpack image "gcr.io/hello-world:latest": failed to resolve reference "gcr.io/hello-world:latest": unexpected status code [manifests latest]: 401 Unauthorized
Run crictl pull gcr.io/third-world
It pulled successfully: Image is up to date for sha256:d1165f2212346b2bab48cb01c1e39ee8ad1be46b87873d9ca7a4e434980a7726
Describe the results you expected:
No matter the registry mirror sequence or they required auth or not, I should be able to pull all these three images successfully.
What version of containerd are you using:
$ containerd --version
containerd github.com/containerd/containerd v1.4.3-gke.0 269548fa27e0089a8b8278fc4fc781d7f65a939b
Any other relevant information (runC version, CRI configuration, OS/Kernel version, etc.):
runC version:
runc version v1.0.0-rc92
commit: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
spec: 1.0.2-dev
Description
After specified multiple registry mirrors for one registry, and they have different username and password, I found that crictl is unable to pull from the second registry mirror or third one.
But if I moved the second registry mirror to first, then I'm able to pull from this new first registry mirror, but not the new second one.
It will be able to pull from second one only if the second one doesn't need authentication.
Steps to reproduce the issue:
I configured three registry mirrors for gcr.io: http://172.18.0.7:5000, https://172.18.0.30:5010, https://172.18.0.20:5007. The 172.18.0.30:5010 one and 172.18.0.20:5007 one need authentication, and I have specified the auth information in the settings.
containerd's config.toml is like below:
I have uploaded a image into 172.18.0.7 named
second-hello:latestAlso uploaded an image into 172.18.0.30 named
hello-world:latestUploaded an image into 172.18.0.20 named
third-hello:latestRun
crictl pull gcr.io/second-helloIt pulled successfully:
Image is up to date for sha256:d1165f2212346b2bab48cb01c1e39ee8ad1be46b87873d9ca7a4e434980a7726Run
crictl pull gcr.io/hello-worldIt pulled successfully:
Image is up to date for sha256:d1165f2212346b2bab48cb01c1e39ee8ad1be46b87873d9ca7a4e434980a7726Run
crictl pull gcr.io/third-worldFailed:
FATA[0000] pulling image: rpc error: code = Unknown desc = failed to pull and unpack image "gcr.io/third-hello:latest": failed to resolve reference "gcr.io/third-hello:latest": unexpected status code [manifests latest]: 401 UnauthorizedUpdate containerd's config.toml, change registry mirror's sequence only:
Run
crictl pull gcr.io/second-helloIt pulled successfully:
Image is up to date for sha256:d1165f2212346b2bab48cb01c1e39ee8ad1be46b87873d9ca7a4e434980a7726Run
crictl pull gcr.io/hello-worldFailed:
FATA[0000] pulling image: rpc error: code = Unknown desc = failed to pull and unpack image "gcr.io/hello-world:latest": failed to resolve reference "gcr.io/hello-world:latest": unexpected status code [manifests latest]: 401 UnauthorizedRun
crictl pull gcr.io/third-worldIt pulled successfully:
Image is up to date for sha256:d1165f2212346b2bab48cb01c1e39ee8ad1be46b87873d9ca7a4e434980a7726Describe the results you expected:
No matter the registry mirror sequence or they required auth or not, I should be able to pull all these three images successfully.
What version of containerd are you using:
Any other relevant information (runC version, CRI configuration, OS/Kernel version, etc.):
runC version: