Description
I have a larger container which I have self-hosted in a Docker registry running registry:2. When I pull a particular container running graphite the deployment failed in k3s with an unexpected commit digest error. This only appears to happen on a private repo.
Steps to reproduce the issue:
- Deploy graphite.yaml below onto Kube running contained (k3s)
kubectl describe pod graphite
apiVersion: v1
kind: Pod
metadata:
name: graphite
labels:
app: graphite
face: external
spec:
containers:
- name: graphite
image: docker-public.fluid.aseit.com.au/graphite:1.1.7-6
imagePullPolicy: Always
ports:
- containerPort: 80
- containerPort: 2003
- containerPort: 2004
Describe the results you received:
The pod always falls into ImageBackoff status
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 17s default-scheduler Successfully assigned default/graphite to alex
Normal Pulling 17s kubelet Pulling image "docker-public.fluid.aseit.com.au/graphite:1.1.7-6"
Warning Failed 0s kubelet Failed to pull image "docker-public.fluid.aseit.com.au/graphite:1.1.7-6": rpc error: code = FailedPrecondition desc = failed to pull and unpack image "docker-public.fluid.aseit.com.au/graphite:1.1.7-6": failed commit on ref "layer-sha256:f9a569415da5aad8ff56d04fdbc1c23a3ed2d4fd577e43f9d9e34e08c4b811da": unexpected commit digest sha256:0e258385e7eaba4e84dcfaf61c2c92b1d345c59645cbfc5d24ca7c90d023bd17, expected sha256:f9a569415da5aad8ff56d04fdbc1c23a3ed2d4fd577e43f9d9e34e08c4b811da: failed precondition
Warning Failed 0s kubelet Error: ErrImagePull
Describe the results you expected:
The image is pulled successfully.
Output of containerd --version:
Running as part of k3s
containerd-shim-runc-v2:
Version: 1.4.3+unknown
Revision:
Go version: go1.15.5
Any other relevant information:
When pulling the image directly from the registry's /v2 endpoint I'm able to decompress it as well as compute the correct SHA256, as expected.
root@alex:/home/alex# wget https://docker-public.fluid.aseit.com.au/v2/graphite/blobs/sha256:f9a569415da5aad8ff56d04fdbc1c23a3ed2d4fd577e43f9d9e34e08c4b811da
--2021-01-06 05:21:27-- https://docker-public.fluid.aseit.com.au/v2/graphite/blobs/sha256:f9a569415da5aad8ff56d04fdbc1c23a3ed2d4fd577e43f9d9e34e08c4b811da
Resolving docker-public.fluid.aseit.com.au (docker-public.fluid.aseit.com.au)... 113.29.244.30
Connecting to docker-public.fluid.aseit.com.au (docker-public.fluid.aseit.com.au)|113.29.244.30|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 134427484 (128M) [application/octet-stream]
Saving to: ‘sha256:f9a569415da5aad8ff56d04fdbc1c23a3ed2d4fd577e43f9d9e34e08c4b811da’
sha256:f9a569415da5aad8ff56d04fdbc1c 100%[====================================================================>] 128.20M 10.3MB/s in 12s
2021-01-06 05:21:39 (10.7 MB/s) - ‘sha256:f9a569415da5aad8ff56d04fdbc1c23a3ed2d4fd577e43f9d9e34e08c4b811da’ saved [134427484/134427484]
root@alex:/home/alex# sha256sum sha256\:f9a569415da5aad8ff56d04fdbc1c23a3ed2d4fd577e43f9d9e34e08c4b811da
f9a569415da5aad8ff56d04fdbc1c23a3ed2d4fd577e43f9d9e34e08c4b811da sha256:f9a569415da5aad8ff56d04fdbc1c23a3ed2d4fd577e43f9d9e34e08c4b811da
root@alex:/home/alex#
This has been mentioned in the past here, here and most recently here - deleting ingest doesn't seem to resolve this, infact - the directory is empty.
root@alex:/home/alex# ls /var/lib/rancher/k3s/agent/containerd/io.containerd.content.v1.content/ingest/
root@alex:/home/alex#
Description
I have a larger container which I have self-hosted in a Docker registry running
registry:2. When I pull a particular container running graphite the deployment failed in k3s with an unexpected commit digest error. This only appears to happen on a private repo.Steps to reproduce the issue:
kubectl describe pod graphiteDescribe the results you received:
The pod always falls into
ImageBackoffstatusDescribe the results you expected:
The image is pulled successfully.
Output of
containerd --version:Running as part of k3s
Any other relevant information:
When pulling the image directly from the registry's /v2 endpoint I'm able to decompress it as well as compute the correct SHA256, as expected.
This has been mentioned in the past here, here and most recently here - deleting ingest doesn't seem to resolve this, infact - the directory is empty.