Skip to content

task delete --force returns an error on Windows #5607

New issue
New issue
Closed
#5956
Closed
task delete --force returns an error on Windows#5607
#5956
Labels
kind/bugplatform/windowsWindows
@claudiubelu

Description

@claudiubelu
claudiubelu
opened on Jun 15, 2021

Description

If the container in a pod is stopped, the Task associated with the pod cannot be forcefully deleted on Windows. Trying to delete the task will generate the following error:

time="2021-06-15T10:48:23Z" level=error msg="unable to delete b58135235ceb6040324160ba3d08507a4adc6c71e83a8d5d02a62c75855a52ae" error="task must be stopped before deletion: running: failed precondition"
ctr: task must be stopped before deletion: running: failed precondition

Secondly, if the container in a pod is started, the Task associated with the pod cannot be forcefully deleted imediately afterwards on Windows, the following error occurs:

time="2021-06-15T12:12:09Z" level=error msg="unable to delete ce3491b4583585867fb9dbf13f36ff2659a7543ce0fc30e05092d640c1d54cf3" error="ttrpc: closed: unknown"
ctr: ttrpc: closed: unknown

This error does not occur if a few seconds pass between starting the Container and forcefully deleting the Task.

In both scenarios, checking the task list afterwards, we can see that the task is no longer present.

The same operation on Linux only generates a Warning. This error can cause the test TestContainerdRestart to fail on Windows.

Steps to reproduce the issue:

On windows:

crictl --runtime-endpoint=npipe://./pipe/run/containerd-test/containerd pull docker.io/claudiubelu/pause:3.4.1
$POD_ID=(crictl --runtime-endpoint=npipe://./pipe/run/containerd-test/containerd runp .\pod.json)
$CONTAINER_ID=(crictl --runtime-endpoint=npipe://./pipe/run/containerd-test/containerd create $POD_ID .\container.json .\pod.json)
crictl --runtime-endpoint=npipe://./pipe/run/containerd-test/containerd start $CONTAINER_ID
crictl --runtime-endpoint=npipe://./pipe/run/containerd-test/containerd stop $CONTAINER_ID

ctr.exe --address //./pipe//run/containerd-test/containerd --namespace k8s.io task list
ctr.exe --address //./pipe//run/containerd-test/containerd --namespace k8s.io task delete --force $POD_ID
ctr.exe --address //./pipe//run/containerd-test/containerd --namespace k8s.io task list

On Linux:


./crictl --runtime-endpoint=unix://./run/containerd-test/containerd.sock pull docker.io/claudiubelu/pause:3.4.1
POD_ID=$(./crictl --runtime-endpoint=unix://./run/containerd-test/containerd.sock runp ./pod.json)
CONTAINER_ID=$(./crictl --runtime-endpoint=unix://./run/containerd-test/containerd.sock create $POD_ID ./container.json ./pod.json)
./crictl --runtime-endpoint=unix://./run/containerd-test/containerd.sock start $CONTAINER_ID
./crictl --runtime-endpoint=unix://./run/containerd-test/containerd.sock stop $CONTAINER_ID

ctr --address //./run/containerd-test/containerd.sock --namespace k8s.io task list
ctr --address //./run/containerd-test/containerd.sock --namespace k8s.io task delete --force $POD_ID
ctr --address //./run/containerd-test/containerd.sock --namespace k8s.io task list

For the running container case, the stop step should be omitted.

Files:

pod.json:

{
  "metadata": {
    "name": "sandbox",
    "namespace": "default",
    "attempt": 1,
    "uid": "hdishd83djaidwnduwk28bcsb"
  },
  "logDirectory": "/tmp"
}

container.json:

{
  "metadata": {
      "name": "busybox"
  },
  "image":{
      "image": "docker.io/claudiubelu/pause:3.4.1"
  },
  "command": [],
  "args": []
}

Describe the results you received:

Output on Windows (stopped container case):

PS C:\tmp> crictl --runtime-endpoint=npipe://./pipe/run/containerd-test/containerd pull docker.io/claudiubelu/pause:3.4.1
Image is up to date for sha256:a233cb888fa372899fd8cd02bad651820809c11303d8537525e53c159d1a7e1e

PS C:\tmp> $POD_ID=(crictl --runtime-endpoint=npipe://./pipe/run/containerd-test/containerd runp .\pod.json)
PS C:\tmp> $CONTAINER_ID=(crictl --runtime-endpoint=npipe://./pipe/run/containerd-test/containerd create $POD_ID .\container.json .\pod.json)

PS C:\tmp> crictl --runtime-endpoint=npipe://./pipe/run/containerd-test/containerd start $CONTAINER_ID
9b7c95e25713c6c37e11fa21780961c3fbac6cb61ce2083670aca6ccd13059bd

PS C:\tmp> crictl --runtime-endpoint=npipe://./pipe/run/containerd-test/containerd stop $CONTAINER_ID
9b7c95e25713c6c37e11fa21780961c3fbac6cb61ce2083670aca6ccd13059bd

PS C:\tmp> ctr.exe --address //./pipe//run/containerd-test/containerd --namespace k8s.io task list
TASK                                                                PID     STATUS
b58135235ceb6040324160ba3d08507a4adc6c71e83a8d5d02a62c75855a52ae    4652    RUNNING

PS C:\tmp> ctr.exe --address //./pipe//run/containerd-test/containerd --namespace k8s.io task delete --force $POD_ID

time="2021-06-15T10:48:23Z" level=error msg="unable to delete b58135235ceb6040324160ba3d08507a4adc6c71e83a8d5d02a62c75855a52ae" error="task must be stopped before deletion: running: failed precondition"
ctr: task must be stopped before deletion: running: failed precondition

PS C:\tmp> ctr.exe --address //./pipe//run/containerd-test/containerd --namespace k8s.io task list
TASK    PID    STATUS

Output on Windows (running container case):


PS C:\tmp> crictl --runtime-endpoint=npipe://./pipe/run/containerd-test/containerd pull docker.io/claudiubelu/pause:3.4.1
Image is up to date for sha256:a233cb888fa372899fd8cd02bad651820809c11303d8537525e53c159d1a7e1e

PS C:\tmp> $POD_ID=(crictl --runtime-endpoint=npipe://./pipe/run/containerd-test/containerd runp .\pod.json)
PS C:\tmp> $CONTAINER_ID=(crictl --runtime-endpoint=npipe://./pipe/run/containerd-test/containerd create $POD_ID .\container.json .\pod.json)

PS C:\tmp> crictl --runtime-endpoint=npipe://./pipe/run/containerd-test/containerd start $CONTAINER_ID
b9062b70e0d0da9ad175bb7d37b2a4657accc4fa2d2f610dc47ddafb6ed4079e

PS C:\tmp> ctr.exe --address //./pipe//run/containerd-test/containerd --namespace k8s.io task list
TASK                                                                PID     STATUS
ce3491b4583585867fb9dbf13f36ff2659a7543ce0fc30e05092d640c1d54cf3    4256    RUNNING
b9062b70e0d0da9ad175bb7d37b2a4657accc4fa2d2f610dc47ddafb6ed4079e    6840    RUNNING

PS C:\tmp> ctr.exe --address //./pipe//run/containerd-test/containerd --namespace k8s.io task delete --force $POD_ID
time="2021-06-15T12:12:09Z" level=error msg="unable to delete ce3491b4583585867fb9dbf13f36ff2659a7543ce0fc30e05092d640c1d54cf3" error="ttrpc: closed: unknown"
ctr: ttrpc: closed: unknown

PS C:\tmp> ctr.exe --address //./pipe//run/containerd-test/containerd --namespace k8s.io task list
TASK    PID    STATUS

Describe the results you expected:

Output on Linux (stopped container case):

sudo ./crictl --runtime-endpoint=unix://./run/containerd-test/containerd.sock pull docker.io/claudiubelu/pause:3.4.1
Image is up to date for sha256:ac0486fc1856a38cec1b47122a4a0c472d854f185daa975781d671de41a24616

POD_ID=$(sudo ./crictl --runtime-endpoint=unix://./run/containerd-test/containerd.sock runp ./pod.json)
CONTAINER_ID=$(sudo ./crictl --runtime-endpoint=unix://./run/containerd-test/containerd.sock create $POD_ID ./container.json ./pod.json)

sudo ./crictl --runtime-endpoint=unix://./run/containerd-test/containerd.sock start $CONTAINER_ID
4f30bbc97f0bf0777a26fb2c7e102be75d99972e4ca885c75778620baeb41383

sudo ./crictl --runtime-endpoint=unix://./run/containerd-test/containerd.sock stop $CONTAINER_ID
4f30bbc97f0bf0777a26fb2c7e102be75d99972e4ca885c75778620baeb41383

sudo ctr --address //./run/containerd-test/containerd.sock --namespace k8s.io task list
TASK                                                                PID      STATUS
5e3e8ed58e61bd2680f083f66d397179ac35422953911aa742f4702194636e10    25644    RUNNING

sudo ctr --address //./run/containerd-test/containerd.sock --namespace k8s.io task delete --force $POD_ID
WARN[0000] task 5e3e8ed58e61bd2680f083f66d397179ac35422953911aa742f470219466e10 exit with non-zero exit code 137

sudo ctr --address //./run/containerd-test/containerd.sock --namespace k8s.io task list
TASK    PID    STATUS

Output on Linux (stopped container case):


+ sudo ./crictl --runtime-endpoint=unix://./run/containerd-test/containerd.sock pull docker.io/claudiubelu/pause:3.4.1
Image is up to date for sha256:ac0486fc1856a38cec1b47122a4a0c472d854f185daa975781d671de41a24616

++ sudo ./crictl --runtime-endpoint=unix://./run/containerd-test/containerd.sock runp ./pod.json
+ POD_ID=34e40e760f00e15f98e00d0a9e6967b4f21ca0cf4b896f540b816ef435554796
++ sudo ./crictl --runtime-endpoint=unix://./run/containerd-test/containerd.sock create 34e40e760f00e15f98e00d0a9e6967b4f21ca0cf4b896f540b816ef435554796 ./container.json ./pod.json

+ CONTAINER_ID=3b333c8db7e98f14b57fa36130d1b0b4b4debdecc17566b021c5fa6dc217f540
+ sudo ./crictl --runtime-endpoint=unix://./run/containerd-test/containerd.sock start 3b333c8db7e98f14b57fa36130d1b0b4b4debdecc17566b021c5fa6dc217f540
3b333c8db7e98f14b57fa36130d1b0b4b4debdecc17566b021c5fa6dc217f540

+ sudo ctr --address //./run/containerd-test/containerd.sock --namespace k8s.io task list
TASK                                                                PID     STATUS
34e40e760f00e15f98e00d0a9e6967b4f21ca0cf4b896f540b816ef435554796    9712    RUNNING
3b333c8db7e98f14b57fa36130d1b0b4b4debdecc17566b021c5fa6dc217f540    9790    RUNNING

+ sudo ctr --address //./run/containerd-test/containerd.sock --namespace k8s.io task delete --force 34e40e760f00e15f98e00d0a9e6967b4f21ca0cf4b896f540b816ef435554796
WARN[0000] task 34e40e760f00e15f98e00d0a9e6967b4f21ca0cf4b896f540b816ef435554796 exit with non-zero exit code 137

+ sudo ctr --address //./run/containerd-test/containerd.sock --namespace k8s.io task list
TASK    PID    STATUS

What version of containerd are you using:

$ containerd --version
containerd github.com/containerd/containerd v1.5.0-153-g0a3a77bc4 0a3a77bc445379851a5c737955411fc9e630c3ce

Any other relevant information (runC version, CRI configuration, OS/Kernel version, etc.):

runc --version
$ runc --version

crictl info
$ crictl info

{
  "status": {
    "conditions": [
      {
        "type": "RuntimeReady",
        "status": true,
        "reason": "",
        "message": ""
      },
      {
        "type": "NetworkReady",
        "status": true,
        "reason": "",
        "message": ""
      }
    ]
  },
  "cniconfig": {
    "PluginDirs": [
      "c:/k/azurecni/bin"
    ],
    "PluginConfDir": "c:/k/azurecni/netconf",
    "PluginMaxConfNum": 1,
    "Prefix": "eth",
    "Networks": [
      {
        "Config": {
          "Name": "azure",
          "CNIVersion": "0.3.0",
          "Plugins": [
            {
              "Network": {
                "type": "azure-vnet",
                "capabilities": {
                  "dns": true,
                  "portMappings": true
                },
                "ipam": {
                  "type": "azure-vnet-ipam"
                },
                "dns": {
                  "nameservers": [
                    "10.0.0.10",
                    "168.63.129.16"
                  ],
                  "search": [
                    "svc.cluster.local"
                  ]
                }
              },
              "Source": "{\"AdditionalArgs\":[{\"Name\":\"EndpointPolicy\",\"Value\":{\"ExceptionList\":[\"10.0.0.0
/8\",\"10.240.0.0/12\"],\"Type\":\"OutBoundNAT\"}},{\"Name\":\"EndpointPolicy\",\"Value\":{\"DestinationPrefix\":\"
10.0.0.0/16\",\"NeedEncap\":true,\"Type\":\"ROUTE\"}},{\"Name\":\"EndpointPolicy\",\"Value\":{\"Action\":\"Block\",
\"Direction\":\"Out\",\"Priority\":200,\"Protocols\":\"6\",\"RemoteAddresses\":\"168.63.129.16/32\",\"RemotePorts\"
:\"80\",\"RuleType\":\"Switch\",\"Type\":\"ACL\"}},{\"Name\":\"EndpointPolicy\",\"Value\":{\"Action\":\"Allow\",\"D
irection\":\"In\",\"Priority\":65500,\"Type\":\"ACL\"}},{\"Name\":\"EndpointPolicy\",\"Value\":{\"Action\":\"Allow\
",\"Direction\":\"Out\",\"Priority\":65500,\"Type\":\"ACL\"}}],\"bridge\":\"azure0\",\"capabilities\":{\"dns\":true
,\"portMappings\":true},\"dns\":{\"Nameservers\":[\"10.0.0.10\",\"168.63.129.16\"],\"Search\":[\"svc.cluster.local\
"]},\"ipam\":{\"type\":\"azure-vnet-ipam\"},\"mode\":\"bridge\",\"type\":\"azure-vnet\"}"
            }
          ],
          "Source": "{\r\n    \"cniVersion\":  \"0.3.0\",\r\n    \"name\":  \"azure\",\r\n    \"adapterName\":  \"\
",\r\n    \"plugins\":  [\r\n                    {\r\n                        \"type\":  \"azure-vnet\",\r\n
                 \"mode\":  \"bridge\",\r\n                        \"bridge\":  \"azure0\",\r\n
    \"capabilities\":  {\r\n                                             \"portMappings\":  true,\r\n
                               \"dns\":  true\r\n                                         },\r\n
     \"ipam\":  {\r\n                                     \"type\":  \"azure-vnet-ipam\"\r\n
          },\r\n                        \"dns\":  {\r\n                                    \"Nameservers\":  [\r\n
                                                       \"10.0.0.10\",\r\n
              \"168.63.129.16\"\r\n                                                    ],\r\n
              \"Search\":  [\r\n                                                   \"svc.cluster.local\"\r\n
                                        ]\r\n                                },\r\n                        \"Additi
onalArgs\":  [\r\n                                               {\r\n
      \"Name\":  \"EndpointPolicy\",\r\n                                                   \"Value\":  {\r\n
                                                          \"Type\":  \"OutBoundNAT\",\r\n
                                       \"ExceptionList\":  [\r\n
                                    \"10.0.0.0/8\",\r\n
                           \"10.240.0.0/12\"\r\n
                ]\r\n                                                             }\r\n
                   },\r\n                                               {\r\n
             \"Name\":  \"EndpointPolicy\",\r\n                                                   \"Value\":  {\r\n
                                                                 \"Type\":  \"ROUTE\",\r\n
                                        \"DestinationPrefix\":  \"10.0.0.0/16\",\r\n
                                  \"NeedEncap\":  true\r\n
    }\r\n                                               },\r\n                                               {\r\n
                                                  \"Name\":  \"EndpointPolicy\",\r\n
                    \"Value\":  {\r\n                                                                 \"Type\":  \"
ACL\",\r\n                                                                 \"Protocols\":  \"6\",\r\n
                                                   \"Action\":  \"Block\",\r\n
                            \"Direction\":  \"Out\",\r\n
      \"RemoteAddresses\":  \"168.63.129.16/32\",\r\n
   \"RemotePorts\":  \"80\",\r\n                                                                 \"Priority\":  200
,\r\n                                                                 \"RuleType\":  \"Switch\"\r\n
                                             }\r\n                                               },\r\n
                                   {\r\n                                                   \"Name\":  \"EndpointPol
icy\",\r\n                                                   \"Value\":  {\r\n
                            \"Type\":  \"ACL\",\r\n
 \"Action\":  \"Allow\",\r\n                                                                 \"Direction\":  \"In\"
,\r\n                                                                 \"Priority\":  65500\r\n
                                        }\r\n                                               },\r\n
                              {\r\n                                                   \"Name\":  \"EndpointPolicy\"
,\r\n                                                   \"Value\":  {\r\n
                       \"Type\":  \"ACL\",\r\n                                                                 \"Ac
tion\":  \"Allow\",\r\n                                                                 \"Direction\":  \"Out\",\r\
n                                                                 \"Priority\":  65500\r\n
                                    }\r\n                                               }\r\n
                     ]\r\n                    }\r\n                ]\r\n}\r\n"
        },
        "IFName": "eth0"
      }
    ]
  },
  "config": {
    "containerd": {
      "snapshotter": "windows",
      "defaultRuntimeName": "default",
      "defaultRuntime": {
        "runtimeType": "io.containerd.runhcs.v1",
        "runtimeEngine": "",
        "PodAnnotations": null,
        "ContainerAnnotations": null,
        "runtimeRoot": "",
        "options": {
          "Debug": true,
          "DebugType": 2,
          "SandboxImage": "mcr.microsoft.com/oss/kubernetes/pause:3.4.1-windows-1809-amd64",
          "SandboxIsolation": 0,
          "SandboxPlatform": "windows/amd64"
        },
        "privileged_without_host_devices": false,
        "baseRuntimeSpec": ""
      },
      "untrustedWorkloadRuntime": {
        "runtimeType": "",
        "runtimeEngine": "",
        "PodAnnotations": null,
        "ContainerAnnotations": null,
        "runtimeRoot": "",
        "options": null,
        "privileged_without_host_devices": false,
        "baseRuntimeSpec": ""
      },
      "runtimes": {
        "default": {
          "runtimeType": "io.containerd.runhcs.v1",
          "runtimeEngine": "",
          "PodAnnotations": null,
          "ContainerAnnotations": null,
          "runtimeRoot": "",
          "options": {
            "Debug": true,
            "DebugType": 2,
            "SandboxImage": "mcr.microsoft.com/oss/kubernetes/pause:3.4.1-windows-1809-amd64",
            "SandboxIsolation": 0,
            "SandboxPlatform": "windows/amd64"
          },
          "privileged_without_host_devices": false,
          "baseRuntimeSpec": ""
        },
        "runhcs-wcow-process": {
          "runtimeType": "io.containerd.runhcs.v1",
          "runtimeEngine": "",
          "PodAnnotations": null,
          "ContainerAnnotations": null,
          "runtimeRoot": "",
          "options": {
            "Debug": true,
            "DebugType": 2,
            "SandboxImage": "mcr.microsoft.com/oss/kubernetes/pause:3.4.1-windows-1809-amd64",
            "SandboxPlatform": "windows/amd64"
          },
          "privileged_without_host_devices": false,
          "baseRuntimeSpec": ""
        }
      },
      "noPivot": false,
      "disableSnapshotAnnotations": false,
      "discardUnpackedLayers": true
    },
    "cni": {
      "binDir": "c:/k/azurecni/bin",
      "confDir": "c:/k/azurecni/netconf",
      "maxConfNum": 1,
      "confTemplate": ""
    },
    "registry": {
      "configPath": "",
      "mirrors": {
        "docker.io": {
          "endpoint": [
            "https://registry-1.docker.io"
          ]
        }
      },
      "configs": null,
      "auths": null,
      "headers": null
    },
    "imageDecryption": {
      "keyModel": "node"
    },
    "disableTCPService": true,
    "streamServerAddress": "127.0.0.1",
    "streamServerPort": "0",
    "streamIdleTimeout": "4h0m0s",
    "enableSelinux": false,
    "selinuxCategoryRange": 0,
    "sandboxImage": "mcr.microsoft.com/oss/kubernetes/pause:3.4.1-windows-1809-amd64",
    "statsCollectPeriod": 10,
    "systemdCgroup": false,
    "enableTLSStreaming": false,
    "x509KeyPairStreaming": {
      "tlsCertFile": "",
      "tlsKeyFile": ""
    },
    "maxContainerLogSize": 16384,
    "disableCgroup": false,
    "disableApparmor": false,
    "restrictOOMScoreAdj": false,
    "maxConcurrentDownloads": 3,
    "disableProcMount": false,
    "unsetSeccompProfile": "",
    "tolerateMissingHugetlbController": false,
    "disableHugetlbController": false,
    "ignoreImageDefinedVolumes": false,
    "netnsMountsUnderStateDir": false,
    "containerdRootDir": "C:\\Program Files\\Git\\var\\lib\\containerd-test",
    "containerdEndpoint": "//./pipe//run/containerd-test/containerd",
    "rootDir": "C:\\Program Files\\Git\\var\\lib\\containerd-test\\io.containerd.grpc.v1.cri",
    "stateDir": "C:\\Program Files\\Git\\run\\containerd-test\\io.containerd.grpc.v1.cri"
  },
  "golang": "go1.16.4",
  "lastCNILoadStatus": "OK"
}

uname -a
$ uname -a

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugplatform/windowsWindows

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions