integration: Fix TestImageLoad() failure on CI#12903
Conversation
54df5c1 to
390f0af
Compare
If docker in the host is configured to use the containerd image format,
then the docker pull/save commands above download the multi-platform
image. IOW, the index.json has references to the other platforms SHAs.
This references to things that are not present (the artifacts for other
platforms) is what make `ctr` fail. However, if we specify a platform,
then ctr ignores the other platforms and the import works just fine.
To try this locally, you can:
$ docker pull ghcr.io/containerd/busybox:1.36
$ docker save ghcr.io/containerd/busybox:1.36 -o image.tar
$ ctr images import --local=true image.tar
The last command will fail if you are using the containerd image store
in docker. If you specify the platform with --platform, it works fine.
With docker overlayfs2 storage driver, if you untar the image you get only
things relevant for your platform:
```
$ cat index.json | jq
{
"schemaVersion": 2,
"mediaType": "application/vnd.oci.image.index.v1+json",
"manifests": [
{
"mediaType": "application/vnd.oci.image.manifest.v1+json",
"digest": "sha256:4d6b13f2ddbe87da8d9dee3719df1723a6d768e511802e70d42ab15370c6eb24",
"size": 401,
"annotations": {
"io.containerd.image.name": "ghcr.io/containerd/busybox:1.36",
"org.opencontainers.image.ref.name": "1.36"
}
}
]
}
$ cat blobs/sha256/4d6b13f2ddbe87da8d9dee3719df1723a6d768e511802e70d42ab15370c6eb24 | jq
{
"schemaVersion": 2,
"mediaType": "application/vnd.oci.image.manifest.v1+json",
"config": {
"mediaType": "application/vnd.oci.image.config.v1+json",
"digest": "sha256:66ba00ad3de8677a3fa4bc4ea0fc46ebca0f14db46ca365e7f60833068dd0148",
"size": 1457
},
"layers": [
{
"mediaType": "application/vnd.oci.image.layer.v1.tar",
"digest": "sha256:b64792c17e4ad443d16b218afb3a8f5d03ca0f4ec49b11c1a7aebe17f6c3c1d2",
"size": 5096448
}
]
}
```
But with the other you get for lot of other platforms too:
```
$ cat index.json | jq
{
"schemaVersion": 2,
"mediaType": "application/vnd.oci.image.index.v1+json",
"manifests": [
{
"mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
"digest": "sha256:7b3ccabffc97de872a30dfd234fd972a66d247c8cfc69b0550f276481852627c",
"size": 2295,
"annotations": {
"containerd.io/distribution.source.ghcr.io": "containerd/busybox",
"io.containerd.image.name": "ghcr.io/containerd/busybox:1.36",
"org.opencontainers.image.ref.name": "1.36"
}
}
]
}
$ cat blobs/sha256/7b3ccabffc97de872a30dfd234fd972a66d247c8cfc69b0550f276481852627c | jq
{
"manifests": [
{
"digest": "sha256:907ca53d7e2947e849b839b1cd258c98fd3916c60f2e6e70c30edbf741ab6754",
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"platform": {
"architecture": "amd64",
"os": "linux"
},
"size": 528
},
{
"digest": "sha256:dde8e930c7b6a490f728e66292bc9bce42efc9bbb5278bae40e4f30f6e00fe8c",
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"platform": {
"architecture": "arm",
"os": "linux",
"variant": "v5"
},
"size": 528
},
{
...
```
You can switch to the containerd snapshotter in docker by adding this to the daemon.json:
```
{
"features": {
"containerd-snapshotter": true
}
}
```
Signed-off-by: Rodrigo Campos <[email protected]>
390f0af to
fafbfcb
Compare
|
Ah, nice find! Looking at that linked ticket; looks like adding a |
|
/cherrypick release/2.2 |
|
@estesp: new pull request created: #12906 DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/cherry-pick release/2.1 |
|
@estesp: #12903 failed to apply on top of branch "release/1.7": DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
@estesp: new pull request created: #12907 DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
When I opened #12894 I found that CI was failing. It seemed to affect only release/2.1. After I found the problem,
mainstarted to fail. My guess is that the github runners changed the default storage format just now.EDIT: Yes, Github runners started the docker 29.1 rollout on Feb 9 (actions/runner-images#13474), it seems we are getting it consistently here now. This is probably what is causing this to fail now. Because docker 29.1, with fresh installs, uses the containerdimage storage by default. And that is causing this issue (see the commit msg below for the explanation).
This is a fix to make the CI green again. I used GOOS and GOARCH because it seems the test wants to run for Windows (although skipped now).
The main issue (although the title is outdated) seems to be this one, reported here: #11344 (comment).
If docker in the host is configured to use the containerd image format, then the docker pull/save commands above download the multi-platform image. IOW, the index.json has references to the other platforms SHAs.
This references to things that are not present (the artifacts for other platforms) is what make
ctrfail. However, if we specify a platform, then ctr ignores the other platforms and the import works just fine.To try this locally, you can:
$ docker pull ghcr.io/containerd/busybox:1.36
$ docker save ghcr.io/containerd/busybox:1.36 -o image.tar
$ ctr images import --local=true image.tar
The last command will fail if you are using the containerd image store in docker. If you specify the platform with --platform, it works fine.
With docker overlayfs2 storage driver, if you untar the image we just downloaded, you get only things relevant for your platform:
But if you do the docker sqeuence using the other storage driver (more info below on how to change it) you get for lot of other platforms too:
You can switch to the containerd snapshotter in docker by adding this to the daemon.json: