[CRI] container IO will be leaky if removing a created container

### Description

After CRI-Test finish, I found that the containerd still hold file descriptors of deleted fifo.

### Steps to reproduce the issue

1. systemctl restart containerd
2. critest -runtime-endpoint /run/containerd/containerd.sock -ginkgo.focus 'runtime should support removing created container'
3. lsof -p $(pidof containerd)


### Describe the results you received and expected

```
➜  testing sudo lsof -p $(pidof containerd)
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs
      Output information may be incomplete.
lsof: WARNING: can't stat() fuse file system /run/user/1000/doc
      Output information may be incomplete.
COMMAND      PID USER   FD      TYPE             DEVICE SIZE/OFF     NODE NAME
container 155110 root  cwd       DIR              259,2     4096        2 /
container 155110 root  rtd       DIR              259,2     4096        2 /
container 155110 root  txt       REG              259,2 47675128  8398013 /usr/bin/containerd
container 155110 root  mem-W     REG              259,2   524288 17566340 /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/metadata.db
container 155110 root  mem-W     REG              259,2  8388608 17575408 /var/lib/containerd/io.containerd.metadata.v1.bolt/meta.db
container 155110 root  mem       REG              259,2  1983576  8391102 /usr/lib/x86_64-linux-gnu/libc-2.33.so
container 155110 root  mem       REG              259,2   150720  8391620 /usr/lib/x86_64-linux-gnu/libpthread-2.33.so
container 155110 root  mem       REG              259,2    22912  8391104 /usr/lib/x86_64-linux-gnu/libdl-2.33.so
container 155110 root  mem       REG              259,2   216192  8391094 /usr/lib/x86_64-linux-gnu/ld-2.33.so
container 155110 root    0r      CHR                1,3      0t0        5 /dev/null
container 155110 root    1u     unix 0xffff9202e6745940      0t0  1957280 type=STREAM
container 155110 root    2u     unix 0xffff9202e6745940      0t0  1957280 type=STREAM
container 155110 root    3u  a_inode               0,14        0    12472 [eventpoll]
container 155110 root    4r     FIFO               0,13      0t0  1955442 pipe
container 155110 root    5w     FIFO               0,13      0t0  1955442 pipe
container 155110 root    6uW     REG              259,2  8388608 17575408 /var/lib/containerd/io.containerd.metadata.v1.bolt/meta.db
container 155110 root    7u  a_inode               0,14        0    12472 [eventpoll]
container 155110 root    8r  a_inode               0,14        0    12472 inotify
container 155110 root    9u  a_inode               0,14        0    12472 [eventpoll]
container 155110 root   10r     FIFO               0,13      0t0  1949683 pipe
container 155110 root   11w     FIFO               0,13      0t0  1949683 pipe
container 155110 root   12u     unix 0xffff920220e92a80      0t0  1949684 /run/containerd/debug.sock type=STREAM
container 155110 root   13u     unix 0xffff920220e96a40      0t0  1949685 /run/containerd/containerd.sock.ttrpc type=STREAM
container 155110 root   14u     unix 0xffff920220e91980      0t0  1949686 /run/containerd/containerd.sock type=STREAM
container 155110 root   15uW     REG              259,2   524288 17566340 /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/metadata.db
container 155110 root   16u     IPv4            1952557      0t0      TCP localhost:41601 (LISTEN)
container 155110 root   23u     FIFO               0,25      0t0    10010 /run/containerd/io.containerd.grpc.v1.cri/containers/36fc8dc0b6e479999877bf7fdafc83b26766c667df0bfe17f292ffe7ee04a885/io/2766993513/36fc8dc0b6e479999877bf7fdafc83b26766c667df0bfe17f292ffe7ee04a885-stdout (deleted)
container 155110 root   24u     FIFO               0,25      0t0    10011 /run/containerd/io.containerd.grpc.v1.cri/containers/36fc8dc0b6e479999877bf7fdafc83b26766c667df0bfe17f292ffe7ee04a885/io/2766993513/36fc8dc0b6e479999877bf7fdafc83b26766c667df0bfe17f292ffe7ee04a885-stderr (deleted)
```

The stdout/stderr fifo are still opened.

> /run/containerd/io.containerd.grpc.v1.cri/containers/36fc8dc0b6e479999877bf7fdafc83b26766c667df0bfe17f292ffe7ee04a885/io/2766993513/36fc8dc0b6e479999877bf7fdafc83b26766c667df0bfe17f292ffe7ee04a885-stderr (deleted)

### What version of containerd are you using?

$ containerd github.com/containerd/containerd v1.5.11-13-gc94cb001d c94cb001d26e3d62d24fc04622a246929e8c87e8

### Any other relevant information

$ runc --version

```
runc version 1.1.0+dev
commit: v1.1.0-65-g2436322f
spec: 1.0.2-dev
go: go1.17.3
libseccomp: 2.5.1
```

$ uname -a 

```
Linux chaofan 5.13.0-39-generic #44~20.04.1-Ubuntu SMP Thu Mar 24 16:43:35 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
```

### Show configuration if it is related to CRI plugin.

<details><summary>cat /etc/containerd/config.toml</summary>

disabled_plugins = []
imports = []
oom_score = 0
plugin_dir = ""
required_plugins = []
root = "/var/lib/containerd"
state = "/run/containerd"
version = 2

[cgroup]
  path = ""

[debug]
  address = "/run/containerd/debug.sock"
  format = ""
  gid = 0
  level = ""
  uid = 0

[grpc]
  address = "/run/containerd/containerd.sock"
  gid = 0
  max_recv_message_size = 16777216
  max_send_message_size = 16777216
  tcp_address = ""
  tcp_tls_cert = ""
  tcp_tls_key = ""
  uid = 0

[metrics]
  address = ""
  grpc_histogram = false

[plugins]

  [plugins."io.containerd.gc.v1.scheduler"]
    deletion_threshold = 0
    mutation_threshold = 100
    pause_threshold = 0.02
    schedule_delay = "0s"
    startup_delay = "100ms"

  [plugins."io.containerd.grpc.v1.cri"]
    disable_apparmor = false
    disable_cgroup = false
    disable_hugetlb_controller = true
    disable_proc_mount = false
    disable_tcp_service = true
    enable_selinux = false
    enable_tls_streaming = false
    ignore_image_defined_volumes = false
    max_concurrent_downloads = 3
    max_container_log_line_size = 16384
    netns_mounts_under_state_dir = false
    restrict_oom_score_adj = false
    sandbox_image = "k8s.gcr.io/pause:3.5"
    selinux_category_range = 1024
    stats_collect_period = 10
    stream_idle_timeout = "4h0m0s"
    stream_server_address = "127.0.0.1"
    stream_server_port = "0"
    systemd_cgroup = false
    tolerate_missing_hugetlb_controller = true
    unset_seccomp_profile = ""

    [plugins."io.containerd.grpc.v1.cri".cni]
      bin_dir = "/opt/cni/bin"
      conf_dir = "/etc/cni/net.d"
      conf_template = ""
      max_conf_num = 1

    [plugins."io.containerd.grpc.v1.cri".containerd]
      default_runtime_name = "runc"
      disable_snapshot_annotations = true
      discard_unpacked_layers = false
      no_pivot = false
      snapshotter = "overlayfs"

      [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime]
        base_runtime_spec = ""
        container_annotations = []
        pod_annotations = []
        privileged_without_host_devices = false
        runtime_engine = ""
        runtime_root = ""
        runtime_type = ""

        [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime.options]

      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]

        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
          base_runtime_spec = ""
          container_annotations = []
          pod_annotations = []
          privileged_without_host_devices = false
          runtime_engine = ""
          runtime_root = ""
          # runtime_type = "io.containerd.runtime.v1.embed"
	  runtime_type = "io.containerd.runc.v2"

          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
            BinaryName = ""
            CriuImagePath = ""
            CriuPath = ""
            CriuWorkPath = ""
            IoGid = 0
            IoUid = 0
            NoNewKeyring = false
            NoPivotRoot = false
            Root = ""
            ShimCgroup = "/shimv2"
            SystemdCgroup = false

      [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime]
        base_runtime_spec = ""
        container_annotations = []
        pod_annotations = []
        privileged_without_host_devices = false
        runtime_engine = ""
        runtime_root = ""
        runtime_type = ""

        [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime.options]

    [plugins."io.containerd.grpc.v1.cri".image_decryption]
      key_model = "node"

    [plugins."io.containerd.grpc.v1.cri".registry]
      config_path = ""

      [plugins."io.containerd.grpc.v1.cri".registry.auths]

      [plugins."io.containerd.grpc.v1.cri".registry.configs]

      [plugins."io.containerd.grpc.v1.cri".registry.headers]

      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]

    [plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming]
      tls_cert_file = ""
      tls_key_file = ""

  [plugins."io.containerd.internal.v1.opt"]
    path = "/opt/containerd"

  [plugins."io.containerd.internal.v1.restart"]
    interval = "10s"

  [plugins."io.containerd.metadata.v1.bolt"]
    content_sharing_policy = "isolated"

  [plugins."io.containerd.monitor.v1.cgroups"]
    no_prometheus = false

  [plugins."io.containerd.runtime.v1.embed"]

  [plugins."io.containerd.runtime.v1.linux"]
    no_shim = false
    runtime = "runc"
    runtime_root = ""
    shim = "containerd-shim"
    shim_debug = false

  [plugins."io.containerd.runtime.v2.task"]
    platforms = ["linux/amd64"]

  [plugins."io.containerd.service.v1.diff-service"]
    default = ["walking"]

  [plugins."io.containerd.snapshotter.v1.overlayfs"]
    root_path = ""

[proxy_plugins]

[stream_processors]

  [stream_processors."io.containerd.ocicrypt.decoder.v1.tar"]
    accepts = ["application/vnd.oci.image.layer.v1.tar+encrypted"]
    args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"]
    env = ["OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"]
    path = "ctd-decoder"
    returns = "application/vnd.oci.image.layer.v1.tar"

  [stream_processors."io.containerd.ocicrypt.decoder.v1.tar.gzip"]
    accepts = ["application/vnd.oci.image.layer.v1.tar+gzip+encrypted"]
    args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"]
    env = ["OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"]
    path = "ctd-decoder"
    returns = "application/vnd.oci.image.layer.v1.tar+gzip"

[timeouts]
  "io.containerd.timeout.shim.cleanup" = "5s"
  "io.containerd.timeout.shim.load" = "5s"
  "io.containerd.timeout.shim.shutdown" = "3s"
  "io.containerd.timeout.task.state" = "2s"

[ttrpc]
  address = ""
  gid = 0
  uid = 0
</details>

From https://github.com/fuweid/embedshim/issues/22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CRI] container IO will be leaky if removing a created container #6834

Description

Steps to reproduce the issue

Describe the results you received and expected

What version of containerd are you using?

Any other relevant information

Show configuration if it is related to CRI plugin.

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[CRI] container IO will be leaky if removing a created container #6834

Description

Description

Steps to reproduce the issue

Describe the results you received and expected

What version of containerd are you using?

Any other relevant information

Show configuration if it is related to CRI plugin.

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions