Skip to content

containerd hangs on start in low-entropy situations #2451

@hairyhenderson

Description

@hairyhenderson

Description

Based on discussion in linuxkit/linuxkit#3032 and linuxkit/linuxkit#3096, it seems that a call to crypto/rand's Read function will block when running on Linux kernel versions 4.14.36 and up, due to a fix for CVE-2018-1108.

As I understand it, prior to the fix, /dev/random was usable before enough entropy was gathered to consider it as a crytographically-safe source. The fix effectively causes reads to /dev/random to block until enough entropy was gathered to fully initialize the CRNG. And Go's crypto/rand Read function uses the Linux getrandom(2) syscall (without setting GRND_NONBLOCK), which reads from /dev/random and therefore now blocks.

Anyway, all this boils down to this line:

rand.Read(b[:])

which will now block when not enough entropy is available. In my case it blocks for ~2mins, but it varies.

Based on the PR comment when that code was originally introduced, I believe that using math/rand instead would be "good enough" - especially:

Since there is already a nanosecond granular time component, a failure to get a unique value here is not a huge concern.

(from @dmcgowan)

Steps to reproduce the issue:

  1. start containerd on a system with a recent-enough kernel, and not enough entropy and without a RNG, for example with LinuxKit - the https://github.com/linuxkit/linuxkit/blob/master/examples/getty.yml example displays this every time for me on my PCEngines APU2C4, which has no HWRNG, though QEMU without a virtio-rng-pci device may work too
  2. watch containerd hang for seconds-to-minutes until CRNG is initialized

Describe the results you received:

containerd blocks for ~2mins on start

Describe the results you expected:

containerd starting ~2mins faster 😉

Output of containerd --version:

containerd github.com/containerd/containerd v1.1.0-rc.2 f630d5f0a639d7d73a806f19f1a6157e768756a5

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions