Skip to content

ctr.exe images import can fail on Windows #5690

Closed
#5916
Closed
ctr.exe images import can fail on Windows#5690
#5916
Labels
kind/bug
@claudiubelu

Description

@claudiubelu
claudiubelu
opened on Jul 6, 2021

Description

Importing images into containerd through ctr.exe can fail with the following error:

unpacking docker.io/claudiubelu/busybox:1.29 (sha256:435b5572d74b64c9646803376baad711af044e2130e3567bed07b809af4ca8bf)...ctr: content digest sha256:7c3822e9f6e642ac53d1da8e3aa2f08b556d82ee8827dda1c93868cde18799d9: not found

This can happen because the Garbage Collector deletes the layers while it's still being unpacked / diffs are being applied. This can be seen in the server logs:

evel=debug msg="removed content" digest="sha256:77b0672dda4a9a40f4a00b73dc21b446e2bc1b2ffb3e6f37525fc5967a971d46"
level=warning msg="content garbage collection failed" error="remove C:\\Program Files\\Git\\var\\lib\\containerd-test\\io.containerd.content.v1.content\\blobs\\sha256\\7c3822e9f6e642ac53d1da8e3aa2f08b556d82ee8827dda1c93868cde18799d9: The process cannot access the file because it is being used by another process."
level=debug msg="garbage collected" d=126.4602ms
level=debug msg="diff applied" d=501.3442ms digest="sha256:7c3822e9f6e642ac53d1da8e3aa2f08b556d82ee8827dda1c93868cde18799d9" media=application/vnd.docker.image.rootfs.diff.tar size=1376768

That blob couldn't be removed because the diff is still being applied at that moment.

The image still shows up in ctr.exe image list:

REF                                                                                                                  TYPE                                                      DIGEST          SIZE      PLATFORMS                                                                    LABELS
docker.io/claudiubelu/busybox:1.29                                                                                   application/vnd.docker.distribution.manifest.v2+json      sha256:435b5572d74b64c9646803376baad711af044e2130e3567bed07b809af4ca8bf 402.5 MiB -                                                                            io.cri-containerd.image=managed

But that image is still unusable. Trying to spawn containers using that image will generate errors.

Steps to reproduce the issue:

# export image to tar.
docker save docker.io/claudiubelu/busybox:1.29 -o image.tar

# make sure we don't have it in store already.
ctr.exe --address //./pipe//run/containerd-test/containerd --namespace k8s.io image remove docker.io/claudiubelu/busybox:1.29
ctr.exe --address //./pipe//run/containerd-test/containerd --namespace k8s.io images import .\image.tar

Describe the results you received:

Output:

PS C:\containerd> ctr.exe --address //./pipe//run/containerd-test/containerd --namespace k8s.io images import .\image.tar
unpacking docker.io/claudiubelu/busybox:1.29 (sha256:435b5572d74b64c9646803376baad711af044e2130e3567bed07b809af4ca8bf)...ctr: content digest sha256:7c3822e9f6e642ac53d1da8e3aa2f08b556d82ee8827dda1c93868cde18799d9: not found

Server logs: https://paste.ubuntu.com/p/pNdkv6YFSm/

Describe the results you expected:

The image should have been imported properly, and it can be used to start new containers.

At the very least, broken images should not be kept / listed in containerd.

What version of containerd are you using:

$ containerd --version
containerd github.com/containerd/containerd v1.5.0-197-g25d7f907c 25d7f907c03a58b5fa30c6f90c3a36aa308e124c

Any other relevant information (runC version, CRI configuration, OS/Kernel version, etc.):

runc --version
$ runc --version

crictl info
$ crictl info

{
  "status": {
    "conditions": [
      {
        "type": "RuntimeReady",
        "status": true,
        "reason": "",
        "message": ""
      },
      {
        "type": "NetworkReady",
        "status": true,
        "reason": "",
        "message": ""
      }
    ]
  },
  "cniconfig": {
    "PluginDirs": [
      "C:\\Program Files\\containerd\\cni\\bin"
    ],
    "PluginConfDir": "C:\\Program Files\\containerd\\cni\\conf",
    "PluginMaxConfNum": 1,
    "Prefix": "eth",
    "Networks": [
      {
        "Config": {
          "Name": "nat",
          "CNIVersion": "0.2.0",
          "Plugins": [
            {
              "Network": {
                "cniVersion": "0.2.0",
                "name": "nat",
                "type": "nat",
                "capabilities": {
                  "dns": true,
                  "portMappings": true
                },
                "ipam": {},
                "dns": {}
              },
              "Source": "{\"capabilities\":{\"dns\":true,\"portMappings\":true},\"cniVersion\":\"0.2.0\",\"ipam\":{\"routes\":[{\"GW\":\"172.27.240.1\"}],\"subnet\":\"172.27.240.0/12\"},\"master\":\"Ethernet\",\"name\":\"nat\",\"type\":\
"nat\"}"
            }
          ],
          "Source": "{\"cniVersion\":\"0.2.0\",\"name\":\"nat\",\"plugins\":[{\"capabilities\":{\"dns\":true,\"portMappings\":true},\"cniVersion\":\"0.2.0\",\"ipam\":{\"routes\":[{\"GW\":\"172.27.240.1\"}],\"subnet\":\"172.27.240.0/12\"}
,\"master\":\"Ethernet\",\"name\":\"nat\",\"type\":\"nat\"}]}"
        },
        "IFName": "eth0"
      }
    ]
  },
  "config": {
    "containerd": {
      "snapshotter": "windows",
      "defaultRuntimeName": "default",
      "defaultRuntime": {
        "runtimeType": "io.containerd.runhcs.v1",
        "runtimeEngine": "",
        "PodAnnotations": null,
        "ContainerAnnotations": null,
        "runtimeRoot": "",
        "options": {
          "Debug": true,
          "DebugType": 2,
          "SandboxImage": "mcr.microsoft.com/oss/kubernetes/pause:3.4.1-windows-1809-amd64",
          "SandboxIsolation": 0,
          "SandboxPlatform": "windows/amd64"
        },
        "privileged_without_host_devices": false,
        "baseRuntimeSpec": ""
      },
      "untrustedWorkloadRuntime": {
        "runtimeType": "",
        "runtimeEngine": "",
        "PodAnnotations": null,
        "ContainerAnnotations": null,
        "runtimeRoot": "",
        "options": null,
        "privileged_without_host_devices": false,
        "baseRuntimeSpec": ""
      },
      "runtimes": {
        "default": {
          "runtimeType": "io.containerd.runhcs.v1",
          "runtimeEngine": "",
          "PodAnnotations": null,
          "ContainerAnnotations": null,
          "runtimeRoot": "",
          "options": {
            "Debug": true,
            "DebugType": 2,
            "SandboxImage": "mcr.microsoft.com/oss/kubernetes/pause:3.4.1-windows-1809-amd64",
            "SandboxIsolation": 0,
            "SandboxPlatform": "windows/amd64"
          },
          "privileged_without_host_devices": false,
          "baseRuntimeSpec": ""
        },
        "runhcs-wcow-process": {
          "runtimeType": "io.containerd.runhcs.v1",
          "runtimeEngine": "",
          "PodAnnotations": null,
          "ContainerAnnotations": null,
          "runtimeRoot": "",
          "options": {
            "Debug": true,
            "DebugType": 2,
            "SandboxImage": "mcr.microsoft.com/oss/kubernetes/pause:3.4.1-windows-1809-amd64",
            "SandboxPlatform": "windows/amd64"
          },
          "privileged_without_host_devices": false,
          "baseRuntimeSpec": ""
        }
      },
      "noPivot": false,
      "disableSnapshotAnnotations": false,
      "discardUnpackedLayers": true
    },
    "cni": {
      "binDir": "C:\\Program Files\\containerd\\cni\\bin",
      "confDir": "C:\\Program Files\\containerd\\cni\\conf",
      "maxConfNum": 1,
      "confTemplate": ""
    },
    "registry": {
      "configPath": "",
      "mirrors": {
        "docker.io": {
          "endpoint": [
            "https://registry-1.docker.io"
          ]
        }
      },
      "configs": null,
      "auths": null,
      "headers": null
    },
    "imageDecryption": {
      "keyModel": "node"
    },
    "disableTCPService": true,
    "streamServerAddress": "127.0.0.1",
    "streamServerPort": "0",
    "streamIdleTimeout": "4h0m0s",
    "enableSelinux": false,
    "selinuxCategoryRange": 0,
    "sandboxImage": "mcr.microsoft.com/oss/kubernetes/pause:3.4.1-windows-1809-amd64",
    "statsCollectPeriod": 10,
    "systemdCgroup": false,
    "enableTLSStreaming": false,
    "x509KeyPairStreaming": {
      "tlsCertFile": "",
      "tlsKeyFile": ""
    },
    "maxContainerLogSize": 16384,
    "disableCgroup": false,
    "disableApparmor": false,
    "restrictOOMScoreAdj": false,
    "maxConcurrentDownloads": 3,
    "disableProcMount": false,
    "unsetSeccompProfile": "",
    "tolerateMissingHugetlbController": false,
    "disableHugetlbController": false,
    "ignoreImageDefinedVolumes": false,
    "netnsMountsUnderStateDir": false,
    "containerdRootDir": "C:\\Program Files\\Git\\var\\lib\\containerd-test",
    "containerdEndpoint": "//./pipe//run/containerd-test/containerd",
    "rootDir": "C:\\Program Files\\Git\\var\\lib\\containerd-test\\io.containerd.grpc.v1.cri",
    "stateDir": "C:\\Program Files\\Git\\run\\containerd-test\\io.containerd.grpc.v1.cri"
  },
  "golang": "go1.16.5",
  "lastCNILoadStatus": "OK"
}

uname -a
$ uname -a

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bug

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions