Description
When containerd tries to pull an image from a registry that does not return a "content-length" header it results in a hard to understand error.
Steps to reproduce the issue:
Reproduction is a little difficult as it requires an arguably broken or non-conformant registry. Perhaps a unit test with manually created HTTP headers would be enough. One way is to use an old build of Trow e.g:
docker run containersol/trow:2021-02-11-109-amd64. You will also need to either configure TLS or use --no-tls and configure containerd to accept this registry as insecure (which is a little tricky).- push an image to the registry
- pull via containerd from the registry
(A complete but slightly complex recreation is in the linked report on the Trow repo)
Describe the results you received:
K3S reports the following, also tested with latest stable containerd:
Failed to pull image "trow.reg.com/image:k3s": rpc error: code = FailedPrecondition desc = failed to pull and unpack image "trow.reg.com/image": failed commit on ref "config-sha256:695e143792a2c7692076f30f705bbfcc8bb89bb312d32bc16837b032938ba37a": unexpected commit digest sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855, expected sha256:695e143792a2c7692076f30f705bbfcc8bb89bb312d32bc16837b032938ba37a: failed precondition
Describe the results you expected:
Either an understandable error message or successfully pull the image. Note that the expected digest sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 is that of an empty file. It took a long time to figure out, but this seems to occur if the registry fails to set the content-length header.
Output of containerd --version:
Tested with
containerd github.com/containerd/containerd v1.4.3 269548fa27e0089a8b8278fc4fc781d7f65a939b
And k3s which returns the following:
k3s ctr --version
ctr github.com/k3s-io/containerd v1.4.3-k3s3
Any other relevant information:
As suggested before this is (arguably) a problem with the registry rather than containerd. It's not clear to me what should happen when the content-length header is missing - interestingly Docker will still pull the image. I have labelled this as a bug, but the fix might be to just output a better error in the case of a missing header.
You can see some more details, including a recreation with k3s here: Trow-Registry/trow#228 (comment)
Description
When containerd tries to pull an image from a registry that does not return a "content-length" header it results in a hard to understand error.
Steps to reproduce the issue:
Reproduction is a little difficult as it requires an arguably broken or non-conformant registry. Perhaps a unit test with manually created HTTP headers would be enough. One way is to use an old build of Trow e.g:
docker run containersol/trow:2021-02-11-109-amd64. You will also need to either configure TLS or use--no-tlsand configure containerd to accept this registry as insecure (which is a little tricky).(A complete but slightly complex recreation is in the linked report on the Trow repo)
Describe the results you received:
K3S reports the following, also tested with latest stable containerd:
Describe the results you expected:
Either an understandable error message or successfully pull the image. Note that the expected digest
sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855is that of an empty file. It took a long time to figure out, but this seems to occur if the registry fails to set the content-length header.Output of
containerd --version:Tested with
And k3s which returns the following:
Any other relevant information:
As suggested before this is (arguably) a problem with the registry rather than containerd. It's not clear to me what should happen when the content-length header is missing - interestingly Docker will still pull the image. I have labelled this as a bug, but the fix might be to just output a better error in the case of a missing header.
You can see some more details, including a recreation with k3s here: Trow-Registry/trow#228 (comment)