Skip to content

runtime: leaking shim socket files #5173

Closed
#5195
Closed
runtime: leaking shim socket files#5173
#5195
Labels
kind/bugpriority/P2
@fuweid

Description

@fuweid
fuweid
opened on Mar 12, 2021

Description

With running short-life containers repeatedly, I found that the /run/containerd/s contains some leaking socket files.

Steps to reproduce the issue:

  1. Init your kubernetes - a single control-plane cluster (by kubeadm/minikube etc)

  2. apply the following yaml

cat containerd-test.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: sleep
spec:
  parallelism: 20
  completions: 9999
  template:
    spec:
      restartPolicy: Never
      hostNetwork: true
      containers:
      - name: sleep
        image: busybox:1.25
        command: ["sleep", "3"]

  1. wait for 5 minutes

  2. run the script like

ls /run/containerd/s | xargs -n 1 -I {} bash -c "socat - UNIX-CONNECT:/run/containerd/s/{}"
2021/03/12 11:39:56 socat[15752] E connect(5, AF=1 "/run/containerd/s/160ac805443d5514215c3e87563986c00aa523e76289a84ba8bdfdf69229b352", 84): Connection refused
2021/03/12 11:39:56 socat[15754] E connect(5, AF=1 "/run/containerd/s/4d14d7ab49e84dc262faa6182000f658edbff8e6fa2df14ac82b739b7d1ec854", 84): Connection refused
2021/03/12 11:39:56 socat[15756] E connect(5, AF=1 "/run/containerd/s/522085d46f169f443fca2ce30087fb742ae8995b794dce8018a78556be2a6f26", 84): Connection refused
2021/03/12 11:39:56 socat[15769] E connect(5, AF=1 "/run/containerd/s/6355c50fac0cbad6b762e188bfb99fb8e0ff9fd19d018a6861c95603d3549f3e", 84): Connection refused
2021/03/12 11:39:56 socat[15770] E connect(5, AF=1 "/run/containerd/s/681b055a2f5c78df4441147c68e3eef1e7bc35025df563483704aede86fa5897", 84): Connection refused
2021/03/12 11:39:56 socat[15772] E connect(5, AF=1 "/run/containerd/s/7b5bd86351fec16d6f1e6bbf07fadfec4dab57e39f4f13da91986048dd44dfce", 84): Connection refused
2021/03/12 11:39:56 socat[15773] E connect(5, AF=1 "/run/containerd/s/7ca56c6a9c5ab7a8ba5d535cf598efe9a2182a460b6d5915590f3e396e499fbf", 84): Connection refused
2021/03/12 11:39:56 socat[15777] E connect(5, AF=1 "/run/containerd/s/9d18b6d08162c0a0a6ae98021e0133c7d713e77c46a0ea1126a344014bccdda6", 84): Connection refused
2021/03/12 11:39:56 socat[15779] E connect(5, AF=1 "/run/containerd/s/ab07024ded81af2bee6c8c8140d7e9bd799db9c4d23ee426ad5f877718ae0e2b", 84): Connection refused
2021/03/12 11:39:56 socat[15783] E connect(5, AF=1 "/run/containerd/s/ddfbf5cbbde6797c24498c74f0654c5572d885ee7239fbfd65e815aa6859ae3d", 84): Connection refused
2021/03/12 11:39:56 socat[15785] E connect(5, AF=1 "/run/containerd/s/fa1a1f84d692fd896bf56b8414349fabd380986bbdb311f508c7e13357e83bf0", 84): Connection refused

Describe the results you received:

The containerd uses containerd-runc-shim-v2 with grouping containers. The number of ready pods is less than the socket files.

 ⚡  crictl pods | grep -v NotReady | wc -l
16
 ⚡  ls /run/containerd/s | wc -l
28

Describe the results you expected:

No leaking socket file

What version of containerd are you using:

$ containerd --version
containerd containerd.io 1.4.4 05f951a3781f4f2c1911b05e61c160e9c30eaa8e

Any other relevant information (runC version, CRI configuration, OS/Kernel version, etc.):

runc --version
runc version 1.0.0-rc93
commit: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
spec: 1.0.2-dev
go: go1.13.15
libseccomp: 2.4.3

crictl info
$ crictl info
{
  "status": {
    "conditions": [
      {
        "type": "RuntimeReady",
        "status": true,
        "reason": "",
        "message": ""
      },
      {
        "type": "NetworkReady",
        "status": true,
        "reason": "",
        "message": ""
      }
    ]
  },
  "cniconfig": {
    "PluginDirs": [
      "/opt/cni/bin"
    ],
    "PluginConfDir": "/etc/cni/net.d",
    "PluginMaxConfNum": 1,
    "Prefix": "eth",
    "Networks": [
      {
        "Config": {
          "Name": "cni-loopback",
          "CNIVersion": "0.3.1",
          "Plugins": [
            {
              "Network": {
                "type": "loopback",
                "ipam": {},
                "dns": {}
              },
              "Source": "{\"type\":\"loopback\"}"
            }
          ],
          "Source": "{\n\"cniVersion\": \"0.3.1\",\n\"name\": \"cni-loopback\",\n\"plugins\": [{\n  \"type\": \"loopback\"\n}]\n}"
        },
        "IFName": "lo"
      },
      {
        "Config": {
          "Name": "cbr0",
          "CNIVersion": "0.3.1",
          "Plugins": [
            {
              "Network": {
                "type": "flannel",
                "ipam": {},
                "dns": {}
              },
              "Source": "{\"delegate\":{\"hairpinMode\":true,\"isDefaultGateway\":true},\"type\":\"flannel\"}"
            },
            {
              "Network": {
                "type": "portmap",
                "capabilities": {
                  "portMappings": true
                },
                "ipam": {},
                "dns": {}
              },
              "Source": "{\"capabilities\":{\"portMappings\":true},\"type\":\"portmap\"}"
            }
          ],
          "Source": "{\n  \"name\": \"cbr0\",\n  \"cniVersion\": \"0.3.1\",\n  \"plugins\": [\n    {\n      \"type\": \"flannel\",\n      \"delegate\": {\n        \"hairpinMode\": true,\n        \"isDefaultGateway\": true\n      }\n    },\n    {\n      \"type\": \"portmap\",\n      \"capabilities\": {\n        \"portMappings\": true\n      }\n    }\n  ]\n}\n"
        },
        "IFName": "eth0"
      }
    ]
  },
  "config": {
    "containerd": {
      "snapshotter": "overlayfs",
      "defaultRuntimeName": "runc",
      "defaultRuntime": {
        "runtimeType": "",
        "runtimeEngine": "",
        "PodAnnotations": null,
        "ContainerAnnotations": null,
        "runtimeRoot": "",
        "options": null,
        "privileged_without_host_devices": false,
        "baseRuntimeSpec": ""
      },
      "untrustedWorkloadRuntime": {
        "runtimeType": "",
        "runtimeEngine": "",
        "PodAnnotations": null,
        "ContainerAnnotations": null,
        "runtimeRoot": "",
        "options": null,
        "privileged_without_host_devices": false,
        "baseRuntimeSpec": ""
      },
      "runtimes": {
        "runc": {
          "runtimeType": "io.containerd.runc.v2",
          "runtimeEngine": "",
          "PodAnnotations": null,
          "ContainerAnnotations": null,
          "runtimeRoot": "",
          "options": {},
          "privileged_without_host_devices": false,
          "baseRuntimeSpec": ""
        }
      },
      "noPivot": false,
      "disableSnapshotAnnotations": true,
      "discardUnpackedLayers": false
    },
    "cni": {
      "binDir": "/opt/cni/bin",
      "confDir": "/etc/cni/net.d",
      "maxConfNum": 1,
      "confTemplate": ""
    },
    "registry": {
      "mirrors": {
        "docker.io": {
          "endpoint": [
            "https://registry-1.docker.io"
          ]
        }
      },
      "configs": null,
      "auths": null,
      "headers": null
    },
    "imageDecryption": {
      "keyModel": ""
    },
    "disableTCPService": true,
    "streamServerAddress": "127.0.0.1",
    "streamServerPort": "0",
    "streamIdleTimeout": "4h0m0s",
    "enableSelinux": false,
    "selinuxCategoryRange": 1024,
    "sandboxImage": "k8s.gcr.io/pause:3.2",
    "statsCollectPeriod": 10,
    "systemdCgroup": false,
    "enableTLSStreaming": false,
    "x509KeyPairStreaming": {
      "tlsCertFile": "",
      "tlsKeyFile": ""
    },
    "maxContainerLogSize": 16384,
    "disableCgroup": false,
    "disableApparmor": false,
    "restrictOOMScoreAdj": false,
    "maxConcurrentDownloads": 3,
    "disableProcMount": false,
    "unsetSeccompProfile": "",
    "tolerateMissingHugetlbController": true,
    "disableHugetlbController": true,
    "ignoreImageDefinedVolumes": false,
    "containerdRootDir": "/var/lib/containerd",
    "containerdEndpoint": "/run/containerd/containerd.sock",
    "rootDir": "/var/lib/containerd/io.containerd.grpc.v1.cri",
    "stateDir": "/run/containerd/io.containerd.grpc.v1.cri"
  },
  "golang": "go1.13.15",
  "lastCNILoadStatus": "OK"
}
uname -a
$ uname -a
Linux open-source-ubuntu 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugpriority/P2

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions