Skip to content

integration: Fix TestImageLoad() failure on CI#12903

Merged
estesp merged 1 commit intocontainerd:mainfrom
rata:fix-TestImageLoad
Feb 16, 2026
Merged

integration: Fix TestImageLoad() failure on CI#12903
estesp merged 1 commit intocontainerd:mainfrom
rata:fix-TestImageLoad

Conversation

@rata
Copy link
Copy Markdown
Contributor

@rata rata commented Feb 16, 2026

When I opened #12894 I found that CI was failing. It seemed to affect only release/2.1. After I found the problem, main started to fail. My guess is that the github runners changed the default storage format just now.

EDIT: Yes, Github runners started the docker 29.1 rollout on Feb 9 (actions/runner-images#13474), it seems we are getting it consistently here now. This is probably what is causing this to fail now. Because docker 29.1, with fresh installs, uses the containerdimage storage by default. And that is causing this issue (see the commit msg below for the explanation).

This is a fix to make the CI green again. I used GOOS and GOARCH because it seems the test wants to run for Windows (although skipped now).

The main issue (although the title is outdated) seems to be this one, reported here: #11344 (comment).


If docker in the host is configured to use the containerd image format, then the docker pull/save commands above download the multi-platform image. IOW, the index.json has references to the other platforms SHAs.

This references to things that are not present (the artifacts for other platforms) is what make ctr fail. However, if we specify a platform, then ctr ignores the other platforms and the import works just fine.

To try this locally, you can:
$ docker pull ghcr.io/containerd/busybox:1.36
$ docker save ghcr.io/containerd/busybox:1.36 -o image.tar
$ ctr images import --local=true image.tar

The last command will fail if you are using the containerd image store in docker. If you specify the platform with --platform, it works fine.

With docker overlayfs2 storage driver, if you untar the image we just downloaded, you get only things relevant for your platform:

$ cat index.json | jq
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.index.v1+json",
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:4d6b13f2ddbe87da8d9dee3719df1723a6d768e511802e70d42ab15370c6eb24",
      "size": 401,
      "annotations": {
        "io.containerd.image.name": "ghcr.io/containerd/busybox:1.36",
        "org.opencontainers.image.ref.name": "1.36"
      }
    }
  ]
}
$ cat blobs/sha256/4d6b13f2ddbe87da8d9dee3719df1723a6d768e511802e70d42ab15370c6eb24 | jq
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:66ba00ad3de8677a3fa4bc4ea0fc46ebca0f14db46ca365e7f60833068dd0148",
    "size": 1457
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar",
      "digest": "sha256:b64792c17e4ad443d16b218afb3a8f5d03ca0f4ec49b11c1a7aebe17f6c3c1d2",
      "size": 5096448
    }
  ]
}

But if you do the docker sqeuence using the other storage driver (more info below on how to change it) you get for lot of other platforms too:


$ cat index.json | jq
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.index.v1+json",
  "manifests": [
    {
      "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
      "digest": "sha256:7b3ccabffc97de872a30dfd234fd972a66d247c8cfc69b0550f276481852627c",
      "size": 2295,
      "annotations": {
        "containerd.io/distribution.source.ghcr.io": "containerd/busybox",
        "io.containerd.image.name": "ghcr.io/containerd/busybox:1.36",
        "org.opencontainers.image.ref.name": "1.36"
      }
    }
  ]
}
$ cat blobs/sha256/7b3ccabffc97de872a30dfd234fd972a66d247c8cfc69b0550f276481852627c | jq
{
  "manifests": [
    {
      "digest": "sha256:907ca53d7e2947e849b839b1cd258c98fd3916c60f2e6e70c30edbf741ab6754",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      },
      "size": 528
    },
    {
      "digest": "sha256:dde8e930c7b6a490f728e66292bc9bce42efc9bbb5278bae40e4f30f6e00fe8c",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "arm",
        "os": "linux",
        "variant": "v5"
      },
      "size": 528
    },
    {
    ...

You can switch to the containerd snapshotter in docker by adding this to the daemon.json:

{
  "features": {
    "containerd-snapshotter": true
  }
}

If docker in the host is configured to use the containerd image format,
then the docker pull/save commands above download the multi-platform
image. IOW, the index.json has references to the other platforms SHAs.

This references to things that are not present (the artifacts for other
platforms) is what make `ctr` fail. However, if we specify a platform,
then ctr ignores the other platforms and the import works just fine.

To try this locally, you can:
	$ docker pull ghcr.io/containerd/busybox:1.36
	$ docker save ghcr.io/containerd/busybox:1.36 -o image.tar
	$ ctr images import --local=true image.tar

The last command will fail if you are using the containerd image store
in docker. If you specify the platform with --platform, it works fine.

With docker overlayfs2 storage driver, if you untar the image you get only
things relevant for your platform:

```
$ cat index.json | jq
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.index.v1+json",
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:4d6b13f2ddbe87da8d9dee3719df1723a6d768e511802e70d42ab15370c6eb24",
      "size": 401,
      "annotations": {
        "io.containerd.image.name": "ghcr.io/containerd/busybox:1.36",
        "org.opencontainers.image.ref.name": "1.36"
      }
    }
  ]
}
$ cat blobs/sha256/4d6b13f2ddbe87da8d9dee3719df1723a6d768e511802e70d42ab15370c6eb24 | jq
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:66ba00ad3de8677a3fa4bc4ea0fc46ebca0f14db46ca365e7f60833068dd0148",
    "size": 1457
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar",
      "digest": "sha256:b64792c17e4ad443d16b218afb3a8f5d03ca0f4ec49b11c1a7aebe17f6c3c1d2",
      "size": 5096448
    }
  ]
}
```

But with the other you get for lot of other platforms too:

```

$ cat index.json | jq
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.index.v1+json",
  "manifests": [
    {
      "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
      "digest": "sha256:7b3ccabffc97de872a30dfd234fd972a66d247c8cfc69b0550f276481852627c",
      "size": 2295,
      "annotations": {
        "containerd.io/distribution.source.ghcr.io": "containerd/busybox",
        "io.containerd.image.name": "ghcr.io/containerd/busybox:1.36",
        "org.opencontainers.image.ref.name": "1.36"
      }
    }
  ]
}
$ cat blobs/sha256/7b3ccabffc97de872a30dfd234fd972a66d247c8cfc69b0550f276481852627c | jq
{
  "manifests": [
    {
      "digest": "sha256:907ca53d7e2947e849b839b1cd258c98fd3916c60f2e6e70c30edbf741ab6754",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      },
      "size": 528
    },
    {
      "digest": "sha256:dde8e930c7b6a490f728e66292bc9bce42efc9bbb5278bae40e4f30f6e00fe8c",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "arm",
        "os": "linux",
        "variant": "v5"
      },
      "size": 528
    },
    {
    ...
```

You can switch to the containerd snapshotter in docker by adding this to the daemon.json:
```
{
  "features": {
    "containerd-snapshotter": true
  }
}

```

Signed-off-by: Rodrigo Campos <[email protected]>
@rata rata force-pushed the fix-TestImageLoad branch from 390f0af to fafbfcb Compare February 16, 2026 15:57
@thaJeztah
Copy link
Copy Markdown
Member

Ah, nice find! Looking at that linked ticket; looks like adding a --skip-missing flag would probably be good to consider?

Copy link
Copy Markdown
Member

@thaJeztah thaJeztah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@thaJeztah thaJeztah added cherry-pick/1.7.x Change to be cherry picked to release/1.7 branch cherry-pick/2.1.x Change to be cherry picked to release/2.1 branch cherry-pick/2.2.x Change to be cherry picked to release/2.2 branch labels Feb 16, 2026
@github-project-automation github-project-automation Bot moved this from Needs Triage to Review In Progress in Pull Request Review Feb 16, 2026
@estesp estesp added this pull request to the merge queue Feb 16, 2026
Merged via the queue into containerd:main with commit 9538c2c Feb 16, 2026
54 checks passed
@github-project-automation github-project-automation Bot moved this from Review In Progress to Done in Pull Request Review Feb 16, 2026
@estesp
Copy link
Copy Markdown
Member

estesp commented Feb 16, 2026

/cherrypick release/2.2

@k8s-infra-cherrypick-robot
Copy link
Copy Markdown

@estesp: new pull request created: #12906

Details

In response to this:

/cherrypick release/2.2

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@estesp
Copy link
Copy Markdown
Member

estesp commented Feb 16, 2026

/cherry-pick release/2.1
/cherry-pick release/1.7

@estesp estesp added cherry-picked/2.2.x PR commits are cherry-picked into release/2.2 branch and removed cherry-pick/2.2.x Change to be cherry picked to release/2.2 branch labels Feb 16, 2026
@k8s-infra-cherrypick-robot
Copy link
Copy Markdown

@estesp: #12903 failed to apply on top of branch "release/1.7":

Applying: integration: Fix TestImageLoad() failure on CI
Using index info to reconstruct a base tree...
M	integration/image_load_test.go
Falling back to patching base and 3-way merge...
Auto-merging integration/image_load_test.go
CONFLICT (content): Merge conflict in integration/image_load_test.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
hint: When you have resolved this problem, run "git am --continue".
hint: If you prefer to skip this patch, run "git am --skip" instead.
hint: To restore the original branch and stop patching, run "git am --abort".
hint: Disable this message with "git config set advice.mergeConflict false"
Patch failed at 0001 integration: Fix TestImageLoad() failure on CI

Details

In response to this:

/cherry-pick release/2.1
/cherry-pick release/1.7

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-infra-cherrypick-robot
Copy link
Copy Markdown

@estesp: new pull request created: #12907

Details

In response to this:

/cherry-pick release/2.1
/cherry-pick release/1.7

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@estesp estesp added cherry-picked/2.1.x PR commits are cherry picked into the release/2.1 branch and removed cherry-pick/2.1.x Change to be cherry picked to release/2.1 branch labels Feb 16, 2026
@estesp estesp added cherry-picked/1.7.x PR commits are cherry-picked into release/1.7 branch and removed cherry-pick/1.7.x Change to be cherry picked to release/1.7 branch labels Feb 16, 2026
@thaJeztah thaJeztah mentioned this pull request Feb 16, 2026
@chrishenzie chrishenzie added the cherry-picked/2.0.x PR commits are cherry picked into the release/2.0 branch label Apr 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/cri Container Runtime Interface (CRI) cherry-picked/1.7.x PR commits are cherry-picked into release/1.7 branch cherry-picked/2.0.x PR commits are cherry picked into the release/2.0 branch cherry-picked/2.1.x PR commits are cherry picked into the release/2.1 branch cherry-picked/2.2.x PR commits are cherry-picked into release/2.2 branch kind/bug size/XS

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

8 participants