Skip to content

distribution: Error when pulling OCI artifacts#44367

Merged
thaJeztah merged 1 commit intomoby:masterfrom
vvoland:oci-artifacts-error
Nov 3, 2022
Merged

distribution: Error when pulling OCI artifacts#44367
thaJeztah merged 1 commit intomoby:masterfrom
vvoland:oci-artifacts-error

Conversation

@vvoland
Copy link
Contributor

@vvoland vvoland commented Oct 27, 2022

Currently an attempt to pull a reference which resolves to an OCI artifact (Helm chart for example), results in a bit unrelated error message invalid rootfs in image configuration.

- What I did
This provides a more meaningful error in case a user attempts to download a media type which isn't image related.

- How to verify it
docker pull docker.io/docker/demo:0.1.0
Should return: unsupported media type application/vnd.cncf.helm.config.v1+json

- Description for the changelog

- A picture of a cute animal (not mandatory but encouraged)
Are there some other media types which we need to support?

@vvoland vvoland force-pushed the oci-artifacts-error branch from a21d409 to d2a9104 Compare October 28, 2022 09:12
@vvoland vvoland marked this pull request as ready for review October 28, 2022 09:13
@vvoland vvoland force-pushed the oci-artifacts-error branch from d2a9104 to 987e1fa Compare October 31, 2022 09:45
@vvoland
Copy link
Contributor Author

vvoland commented Oct 31, 2022

Strange CI failure of docker-py:

2022-10-31T09:55:18.0008654Z =================================== FAILURES ===================================
2022-10-31T09:55:18.0015712Z ___________________ SwarmTest.test_init_swarm_with_ca_config ___________________
2022-10-31T09:55:18.0021705Z docker/api/client.py:268: in _raise_for_status
2022-10-31T09:55:18.0027651Z     response.raise_for_status()
2022-10-31T09:55:18.0038028Z /usr/local/lib/python3.7/site-packages/requests/models.py:953: in raise_for_status
2022-10-31T09:55:18.0038470Z     raise HTTPError(http_error_msg, response=self)
2022-10-31T09:55:18.0038993Z E   requests.exceptions.HTTPError: 503 Server Error: Service Unavailable for url: http+docker://localhost/v1.43/swarm/init
2022-10-31T09:55:18.0039363Z 
2022-10-31T09:55:18.0039562Z During handling of the above exception, another exception occurred:
2022-10-31T09:55:18.0039991Z tests/integration/api_swarm_test.py:89: in test_init_swarm_with_ca_config
2022-10-31T09:55:18.0040368Z     assert self.init_swarm(swarm_spec=spec)
2022-10-31T09:55:18.0040921Z tests/integration/base.py:127: in init_swarm
2022-10-31T09:55:18.0041267Z     return self._init_swarm(self.client, **kwargs)
2022-10-31T09:55:18.0041605Z tests/integration/base.py:94: in _init_swarm
2022-10-31T09:55:18.0042069Z     '127.0.0.1', listen_addr=helpers.swarm_listen_addr(), **kwargs
2022-10-31T09:55:18.0042426Z docker/utils/decorators.py:34: in wrapper
2022-10-31T09:55:18.0042732Z     return f(self, *args, **kwargs)
2022-10-31T09:55:18.0043013Z docker/api/swarm.py:170: in init_swarm
2022-10-31T09:55:18.0043333Z     return self._result(response, json=True)
2022-10-31T09:55:18.0043651Z docker/api/client.py:274: in _result
2022-10-31T09:55:18.0057337Z     self._raise_for_status(response)
2022-10-31T09:55:18.0057674Z docker/api/client.py:270: in _raise_for_status
2022-10-31T09:55:18.0058023Z     raise create_api_error_from_http_exception(e)
2022-10-31T09:55:18.0058399Z docker/errors.py:31: in create_api_error_from_http_exception
2022-10-31T09:55:18.0058779Z     raise cls(e, response=response, explanation=explanation)
2022-10-31T09:55:18.0059481Z E   docker.errors.APIError: 503 Server Error for http+docker://localhost/v1.43/swarm/init: Service Unavailable ("Swarm is encrypted and needs to be unlocked before it can be used. Please use "docker swarm unlock" to unlock it.")
2022-10-31T09:55:18.0060417Z ------- generated xml file: /src/bundles/test-docker-py/junit-report.xml -------

Seems unrelated. I haven't seen this failure in other PRs though. Will re-run this job.

EDIT: Meh, I can't re-run the jobs. Will use this as an opportunity to sign my commit.

@vvoland vvoland force-pushed the oci-artifacts-error branch from 987e1fa to 8130c1d Compare October 31, 2022 16:05
Copy link
Member

@cpuguy83 cpuguy83 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM after @tianon's suggestion.

@thaJeztah thaJeztah added area/distribution Image Distribution status/2-code-review kind/enhancement Enhancements are not bugs or new features but can improve usability or performance. process/cherry-pick labels Nov 1, 2022
@thaJeztah thaJeztah added this to the v-next milestone Nov 1, 2022
Currently an attempt to pull a reference which resolves to an OCI
artifact (Helm chart for example), results in a bit unrelated error
message `invalid rootfs in image configuration`.

This provides a more meaningful error in case a user attempts to
download a media type which isn't image related.

Signed-off-by: Paweł Gronowski <[email protected]>
@vvoland vvoland force-pushed the oci-artifacts-error branch from 8130c1d to 407e3a4 Compare November 2, 2022 12:00
@vvoland
Copy link
Contributor Author

vvoland commented Nov 2, 2022

CI failure unrelated, flaky tests: TestNetworkDBIslands, TestGetContainerStatsStream

@thaJeztah
Copy link
Member

Kicked CI 👍

Copy link
Member

@thaJeztah thaJeztah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@thaJeztah
Copy link
Member

I think this introduced a regression for (very) old images. Seeing this on one of our internal Docker Desktop CI runs;

* error running [docker pull python:3.5.1-alpine]:
Command stdout:
3.5.1-alpine: Pulling from library/python

stderr:
unsupported media type application/octet-stream

On my M1 (with containerd integration);

docker pull python:3.5.1-alpine
f88925c97b97: Download complete
1c7f1bf64dde: Download complete
failed to unpack image on snapshotter stargz: failed to extract layer sha256:4fe15f8d0ae69e169824f25f1d4da3015a48feeeeebb265cd2e328e15c6a869f: failed to get reader from content store: content digest sha256:e110a4a1794126ef308a49f2d65785af2f25538f06700721aad8283b81fdfa58: not found

On docker 23.0.0-beta.1 (x86)

docker pull python:3.5.1-alpine
3.5.1-alpine: Pulling from library/python
unsupported media type application/octet-stream

On Docker 20.10 (x86);

docker pull python:3.5.1-alpine
3.5.1-alpine: Pulling from library/python
e110a4a17941: Pull complete
30dac23631f0: Pull complete
202fc3980a36: Pull complete
Digest: sha256:f88925c97b9709dd6da0cb2f811726da9d724464e9be17a964c70f067d2aa64a
Status: Downloaded newer image for python:3.5.1-alpine
docker.io/library/python:3.5.1-alpine

Trying with containerd (ctr), the output also looks... odd, but it seems to manage to pull;

ctr i pull docker.io/library/python:3.5.1-alpine
docker.io/library/python:3.5.1-alpine:                                            resolved       |++++++++++++++++++++++++++++++++++++++|
docker.io/library/python:3.5.1-alpine:                                            resolved       |++++++++++++++++++++++++++++++++++++++|
docker.io/library/python:3.5.1-alpine:                                            resolved       |++++++++++++++++++++++++++++++++++++++|
docker.io/library/python:3.5.1-alpine:                                            resolved       |++++++++++++++++++++++++++++++++++++++|
docker.io/library/python:3.5.1-alpine:                                            resolved       |++++++++++++++++++++++++++++++++++++++|
docker.io/library/python:3.5.1-alpine:                                            resolved       |++++++++++++++++++++++++++++++++++++++|
docker.io/library/python:3.5.1-alpine:                                            resolved       |++++++++++++++++++++++++++++++++++++++|
docker.io/library/python:3.5.1-alpine:                                            resolved       |++++++++++++++++++++++++++++++++++++++|
docker.io/library/python:3.5.1-alpine:                                            resolved       |++++++++++++++++++++++++++++++++++++++|
docker.io/library/python:3.5.1-alpine:                                            resolved       |++++++++++++++++++++++++++++++++++++++|
docker.io/library/python:3.5.1-alpine:                                            resolved       |++++++++++++++++++++++++++++++++++++++|
docker.io/library/python:3.5.1-alpine:                                            resolved       |++++++++++++++++++++++++++++++++++++++|
docker.io/library/python:3.5.1-alpine:                                            resolved       |++++++++++++++++++++++++++++++++++++++|
docker.io/library/python:3.5.1-alpine:                                            resolved       |++++++++++++++++++++++++++++++++++++++|
docker.io/library/python:3.5.1-alpine:                                            resolved       |++++++++++++++++++++++++++++++++++++++|
docker.io/library/python:3.5.1-alpine:                                            resolved       |++++++++++++++++++++++++++++++++++++++|
docker.io/library/python:3.5.1-alpine:                                            resolved       |++++++++++++++++++++++++++++++++++++++|
docker.io/library/python:3.5.1-alpine:                                            resolved       |++++++++++++++++++++++++++++++++++++++|
docker.io/library/python:3.5.1-alpine:                                            resolved       |++++++++++++++++++++++++++++++++++++++|
docker.io/library/python:3.5.1-alpine:                                            resolved       |++++++++++++++++++++++++++++++++++++++|
docker.io/library/python:3.5.1-alpine:                                            resolved       |++++++++++++++++++++++++++++++++++++++|
docker.io/library/python:3.5.1-alpine:                                            resolved       |++++++++++++++++++++++++++++++++++++++|
docker.io/library/python:3.5.1-alpine:                                            resolved       |++++++++++++++++++++++++++++++++++++++|
docker.io/library/python:3.5.1-alpine:                                            resolved       |++++++++++++++++++++++++++++++++++++++|
docker.io/library/python:3.5.1-alpine:                                            resolved       |++++++++++++++++++++++++++++++++++++++|
docker.io/library/python:3.5.1-alpine:                                            resolved       |++++++++++++++++++++++++++++++++++++++|
docker.io/library/python:3.5.1-alpine:                                            resolved       |++++++++++++++++++++++++++++++++++++++|
docker.io/library/python:3.5.1-alpine:                                            resolved       |++++++++++++++++++++++++++++++++++++++|
docker.io/library/python:3.5.1-alpine:                                            resolved       |++++++++++++++++++++++++++++++++++++++|
docker.io/library/python:3.5.1-alpine:                                            resolved       |++++++++++++++++++++++++++++++++++++++|
docker.io/library/python:3.5.1-alpine:                                            resolved       |++++++++++++++++++++++++++++++++++++++|
docker.io/library/python:3.5.1-alpine:                                            resolved       |++++++++++++++++++++++++++++++++++++++|
docker.io/library/python:3.5.1-alpine:                                            resolved       |++++++++++++++++++++++++++++++++++++++|
docker.io/library/python:3.5.1-alpine:                                            resolved       |++++++++++++++++++++++++++++++++++++++|
docker.io/library/python:3.5.1-alpine:                                            resolved       |++++++++++++++++++++++++++++++++++++++|
docker.io/library/python:3.5.1-alpine:                                            resolved       |++++++++++++++++++++++++++++++++++++++|
docker.io/library/python:3.5.1-alpine:                                            resolved       |++++++++++++++++++++++++++++++++++++++|
docker.io/library/python:3.5.1-alpine:                                            resolved       |++++++++++++++++++++++++++++++++++++++|
docker.io/library/python:3.5.1-alpine:                                            resolved       |++++++++++++++++++++++++++++++++++++++|
docker.io/library/python:3.5.1-alpine:                                            resolved       |++++++++++++++++++++++++++++++++++++++|
docker.io/library/python:3.5.1-alpine:                                            resolved       |++++++++++++++++++++++++++++++++++++++|
docker.io/library/python:3.5.1-alpine:                                            resolved       |++++++++++++++++++++++++++++++++++++++|
docker.io/library/python:3.5.1-alpine:                                            resolved       |++++++++++++++++++++++++++++++++++++++|
docker.io/library/python:3.5.1-alpine:                                            resolved       |++++++++++++++++++++++++++++++++++++++|
docker.io/library/python:3.5.1-alpine:                                            resolved       |++++++++++++++++++++++++++++++++++++++|
docker.io/library/python:3.5.1-alpine:                                            resolved       |++++++++++++++++++++++++++++++++++++++|
docker.io/library/python:3.5.1-alpine:                                            resolved       |++++++++++++++++++++++++++++++++++++++|
docker.io/library/python:3.5.1-alpine:                                            resolved       |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:f88925c97b9709dd6da0cb2f811726da9d724464e9be17a964c70f067d2aa64a: done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:202fc3980a369fecd7f39c054b996e033347e441a500b9bb0dd03db6b9cc125c:    done           |++++++++++++++++++++++++++++++++++++++|
unknown-sha256:1c7f1bf64dde149e36ffe0bcc0b9f1fbaeacc379dd80e4064c3a1e1f0b4b34b5:  done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:e110a4a1794126ef308a49f2d65785af2f25538f06700721aad8283b81fdfa58:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:30dac23631f0219d44d2f43a5dbfb7ed47aa23e09abed01813510d0f64aa16af:    done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 6.1 s                                                                    total:  18.2 M (3.0 MiB/s)
unpacking linux/amd64 sha256:a5e016298b8e3e0f17bb8d8c6204679b706db42b0a0f9afcf9b56e86a1670720...
done: 1.40535854s

The image looks to be very old, and may be one of the "v1 images" (perhaps converted), and it looks like application/octet-stream was used for those.

docker buildx imagetools inspect --raw python:3.5.1-alpine
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/octet-stream",
      "size": 4648,
      "digest": "sha256:1c7f1bf64dde149e36ffe0bcc0b9f1fbaeacc379dd80e4064c3a1e1f0b4b34b5"
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 2310286,
         "digest": "sha256:e110a4a1794126ef308a49f2d65785af2f25538f06700721aad8283b81fdfa58"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 20528388,
         "digest": "sha256:30dac23631f0219d44d2f43a5dbfb7ed47aa23e09abed01813510d0f64aa16af"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 258,
         "digest": "sha256:202fc3980a369fecd7f39c054b996e033347e441a500b9bb0dd03db6b9cc125c"
      }
   ]
}

This is the image config itself;

docker buildx imagetools inspect --raw python:3.5.1-alpine@sha256:1c7f1bf64dde149e36ffe0bcc0b9f1fbaeacc379dd80e4064c3a1e1f0b4b34b5 | jq .
{
  "architecture": "amd64",
  "config": {
    "Hostname": "4c573a7cf4a3",
    "Domainname": "",
    "User": "",
    "AttachStdin": false,
    "AttachStdout": false,
    "AttachStderr": false,
    "Tty": false,
    "OpenStdin": false,
    "StdinOnce": false,
    "Env": [
      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
      "LANG=C.UTF-8",
      "GPG_KEY=97FC712E4C024BBEA48A61ED3A5CA953F73C700D",
      "PYTHON_VERSION=3.5.1",
      "PYTHON_PIP_VERSION=8.1.2"
    ],
    "Cmd": [
      "python3"
    ],
    "ArgsEscaped": true,
    "Image": "sha256:9bdb1a84ef984fe0fde4ed782ace400d6bb3d83139d884ccae66f03aba63618c",
    "Volumes": null,
    "WorkingDir": "",
    "Entrypoint": null,
    "OnBuild": [],
    "Labels": {}
  },
  "container": "623f7eec2de14161c6f59c957c65856314f77dde6f5f0b3ef93ec5c9b773780f",
  "container_config": {
    "Hostname": "4c573a7cf4a3",
    "Domainname": "",
    "User": "",
    "AttachStdin": false,
    "AttachStdout": false,
    "AttachStderr": false,
    "Tty": false,
    "OpenStdin": false,
    "StdinOnce": false,
    "Env": [
      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
      "LANG=C.UTF-8",
      "GPG_KEY=97FC712E4C024BBEA48A61ED3A5CA953F73C700D",
      "PYTHON_VERSION=3.5.1",
      "PYTHON_PIP_VERSION=8.1.2"
    ],
    "Cmd": [
      "/bin/sh",
      "-c",
      "#(nop) CMD [\"python3\"]"
    ],
    "ArgsEscaped": true,
    "Image": "sha256:9bdb1a84ef984fe0fde4ed782ace400d6bb3d83139d884ccae66f03aba63618c",
    "Volumes": null,
    "WorkingDir": "",
    "Entrypoint": null,
    "OnBuild": [],
    "Labels": {}
  },
  "created": "2016-06-23T21:51:47.148716248Z",
  "docker_version": "1.10.3",
  "history": [
    {
      "created": "2016-06-23T19:55:18.048565927Z",
      "created_by": "/bin/sh -c #(nop) ADD file:852e9d0cb9d906535af512a89339fc70b2873a0f94defbcbe41cd44942dd6ac8 in /"
    },
    {
      "created": "2016-06-23T20:34:53.766848547Z",
      "created_by": "/bin/sh -c #(nop) ENV LANG=C.UTF-8",
      "empty_layer": true
    },
    {
      "created": "2016-06-23T21:49:32.019395426Z",
      "created_by": "/bin/sh -c #(nop) ENV GPG_KEY=97FC712E4C024BBEA48A61ED3A5CA953F73C700D",
      "empty_layer": true
    },
    {
      "created": "2016-06-23T21:49:32.441732037Z",
      "created_by": "/bin/sh -c #(nop) ENV PYTHON_VERSION=3.5.1",
      "empty_layer": true
    },
    {
      "created": "2016-06-23T21:49:32.897793606Z",
      "created_by": "/bin/sh -c #(nop) ENV PYTHON_PIP_VERSION=8.1.2",
      "empty_layer": true
    },
    {
      "created": "2016-06-23T21:51:45.127167986Z",
      "created_by": "/bin/sh -c set -ex \t&& apk add --no-cache --virtual .fetch-deps curl gnupg \t&& curl -fSL \"https://www.python.org/ftp/python/${PYTHON_VERSION%%[a-z]*}/Python-$PYTHON_VERSION.tar.xz\" -o python.tar.xz \t&& curl -fSL \"https://www.python.org/ftp/python/${PYTHON_VERSION%%[a-z]*}/Python-$PYTHON_VERSION.tar.xz.asc\" -o python.tar.xz.asc \t&& export GNUPGHOME=\"$(mktemp -d)\" \t&& gpg --keyserver ha.pool.sks-keyservers.net --recv-keys \"$GPG_KEY\" \t&& gpg --batch --verify python.tar.xz.asc python.tar.xz \t&& rm -r \"$GNUPGHOME\" python.tar.xz.asc \t&& mkdir -p /usr/src \t&& tar -xJC /usr/src -f python.tar.xz \t&& mv \"/usr/src/Python-$PYTHON_VERSION\" /usr/src/python \t&& rm python.tar.xz \t&& apk del .fetch-deps \t\t&& apk add --no-cache --virtual .build-deps  \t\tbzip2-dev \t\tgcc \t\tlibc-dev \t\tlinux-headers \t\tmake \t\tncurses-dev \t\topenssl-dev \t\tpax-utils \t\treadline-dev \t\tsqlite-dev \t\txz-dev \t\tzlib-dev \t&& cd /usr/src/python \t&& ./configure --enable-shared --enable-unicode=ucs4 \t&& make -j$(getconf _NPROCESSORS_ONLN) \t&& make install \t&& pip3 install --no-cache-dir --upgrade --ignore-installed pip==$PYTHON_PIP_VERSION \t&& find /usr/local -depth \t\t\\( \t\t    \\( -type d -a -name test -o -name tests \\) \t\t    -o \t\t    \\( -type f -a -name '*.pyc' -o -name '*.pyo' \\) \t\t\\) -exec rm -rf '{}' + \t&& runDeps=\"$( \t\tscanelf --needed --nobanner --recursive /usr/local \t\t\t| awk '{ gsub(/,/, \"\\nso:\", $2); print \"so:\" $2 }' \t\t\t| sort -u \t\t\t| xargs -r apk info --installed \t\t\t| sort -u \t)\" \t&& apk add --virtual .python-rundeps $runDeps \t&& apk del .build-deps \t&& rm -rf /usr/src/python ~/.cache"
    },
    {
      "created": "2016-06-23T21:51:46.663723315Z",
      "created_by": "/bin/sh -c cd /usr/local/bin \t&& ln -s easy_install-3.5 easy_install \t&& ln -s idle3 idle \t&& ln -s pydoc3 pydoc \t&& ln -s python3 python \t&& ln -s python3-config python-config"
    },
    {
      "created": "2016-06-23T21:51:47.148716248Z",
      "created_by": "/bin/sh -c #(nop) CMD [\"python3\"]",
      "empty_layer": true
    }
  ],
  "os": "linux",
  "rootfs": {
    "type": "layers",
    "diff_ids": [
      "sha256:4fe15f8d0ae69e169824f25f1d4da3015a48feeeeebb265cd2e328e15c6a869f",
      "sha256:e138e99eb396c549412b6af549a268822aee5d94d550d5ec874b5af510685611",
      "sha256:f835c614d282e735299ae901275dcda769d01426ad7e670fa2252cd69bf1f1c1"
    ]
  }
}

@thaJeztah
Copy link
Member

Looks like we had something similar at some point (long time ago); see

// defaultImageTypes represents the schema2 config types for images
defaultImageTypes = []string{
schema2.MediaTypeImageConfig,
ocispec.MediaTypeImageConfig,
// Handle unexpected values from https://github.com/docker/distribution/issues/1621
// (see also https://github.com/docker/docker/issues/22378,
// https://github.com/docker/docker/issues/30083)
"application/octet-stream",
"application/json",
"text/html",
// Treat defaulted values as images, newer types cannot be implied
"",
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/distribution Image Distribution kind/enhancement Enhancements are not bugs or new features but can improve usability or performance. process/cherry-picked status/2-code-review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants