Issue

• When containerd collects metrics from shim's Stat API, there is no timeout set:

• If the shim process is not responsive, the request blocks there, Collect.Collect() blocks.

  containerd/metrics/cgroups/v2/metrics.go

  Line 92 in d3aa7ee

  func (c *Collector) Collect(ch chan<- prometheus.Metric) {

• This causes leaked goroutines and memory. Requests to prometheus metrics /v1/metrics to hangs when there is a "bad shim"

Reproducing

• Launch containerd with "metrics" and "debug" enable

cat /etc/containerd/config.toml
[metrics]
    address = ":8080"
[debug]
    address = "/run/containerd/debug.sock"

• I launched a container with a mock shim implementation that hangs on Stats: phanhuyn@a6add78

• Observe request to prometheus endpoint hangs

curl localhost:8080/v1/metrics
# hangs

• Goroutines dump of containerd daemon shows hangs in the Stats call to the shim process

ctr pprof goroutines > /tmp/containerd_goroutines

grep "/metrics.go" /tmp/containerd_goroutines
        /go/src/github.com/containerd/containerd/metrics/cgroups/v1/metrics.go:117 +0x233
        /go/src/github.com/containerd/containerd/metrics/cgroups/v1/metrics.go:126 +0x118
        /go/src/github.com/containerd/containerd/metrics/cgroups/v1/metrics.go:104 +0xb1

Fix

• Send a request to shim with timeout. Default to 2 seconds.
• I repeat the same steps to verify. After the fix, containerd daemon returns Prometheus metrics even when one of the shim hangs, containerd logs show a request timeout to stat:

Apr 06 09:00:04 ubuntu-xenial containerd[25364]: time="2022-04-06T09:00:04.189993886Z" level=error msg="stat task test1" error="context deadline exceeded: unknown"

Timeout setting

The default timeout is set to 2 seconds. Can be configured by:

[timeouts]
    "io.containerd.timeout.metrics.shimstats" = "10s"

Signed-off-by: Nguyen Phan Huy [email protected]