It looks like this patch allows containerd to start "enough" to not block other services in LinuxKit from starting:

diff --git a/services/leases/local.go b/services/leases/local.go
index d3e0c2f2c5..739e58ea9e 100644
--- a/services/leases/local.go
+++ b/services/leases/local.go
@@ -17,9 +17,9 @@
 package leases
 
 import (
-	"crypto/rand"
 	"encoding/base64"
 	"fmt"
+	"math/rand"
 	"time"
 
 	"google.golang.org/grpc"

but containerd itself still hangs, presumably due to the other uses of crypto/rand - I'll continue hacking away at this...

The hanging was not expected, the lease generator is intended to be designed that in low entropy situations, the ID generated is just less random rather than blocking or erroring. Looks like we need to re-address this assumption that rand.Read will error and not block when encountering low entropy.

This assumption was based on previous experience here https://github.com/docker/distribution/blob/master/uuid/uuid.go#L56

The hanging was not expected, the lease generator is intended to be designed that in low entropy situations, the ID generated is just less random rather than blocking or erroring.

Yeah, I think a lot of userspace code that makes the non-blocking assumption is going to start hanging in very hard-to-diagnose ways on early-boot when the newer kernel versions become more prevalent 😞.

Would you accept a PR to simply change crypto/rand to math/rand?

The Read method in both packages is effectively the same, just that math/rand will never block or error, but also isn't guaranteed to return cryptographically-unpredictable random data (which I don't see as a requirement for this).

Would you accept a PR to simply change crypto/rand to math/rand?

Yes, I agree cryptographically random is not necessary. We also add a nanosecond component, we could update this in the future to use UUID logic.

We can also just implement a reader with unix.GetRandom(bytes, unix.GRND_NONBLOCK)

@crosbymichael I agree that it'd be useful to have a GRND_NONBLOCK version available (it's already available internally at https://github.com/golang/go/blob/master/src/internal/syscall/unix/getrandom_linux.go#L26..L49).

One thing to watch for is that GRND_NONBLOCK simply changes the behaviour in low-entropy situations from blocking to erroring. So, in this case, containerd would need to handle the error and retry (thus effectively blocking start-up anyway), or fall back to some other behaviour such as using math/rand.

Probably simpler and less surprising to stick to math/rand in all cases, especially since this isn't a situation that calls for cryptographically random IDs.

@hairyhenderson ok, that makes sense. Thanks!

#2454 got merged, which is great, and that'll solve the main blocking issue. But there appears to be something else blocking - in my LinuxKit setup I get a shell from getty now, but the containerd logs don't start appearing until crng_init is done.

So I'm going to keep this issue open for now - I want to dig a little deeper and see if there's another blocking getrandom call (maybe in a dependency)... 🕵️

We really need to get rid of the docker/docker imports
https://github.com/containerd/containerd/blob/master/vendor/github.com/docker/docker/pkg/stringid/stringid.go#L85

This does raise an important issue which was not part of the PR, random should be seeded, but we should seed it in the binary command and not in the imports.

Yup, that'll do it. 😂

Is there anything else we still need to investigate here or are the 2 PRs already merged sufficient?

👋 @estesp 😁 TBH I'm not sure. I seem to recall that the patched containerd still had some entropy-related hanging. But my memory's fuzzy on exactly what combination of things I tried.

That particular project has been a bit on hold while I've enjoyed some summer vacation, but I'll try to get back to this in the next few days to see if the latest containerd still stalls.