balena depends on random to be initialised before starting #105
Description
Until crng has been initialised and there is enough entropy in the system, balena does not start.
This is introduced by a patch in the kernel which was backported to stable branches as well.
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=43838a23a05fbd13e47d750d3dfd77001536dd33
Reverting this patch does make balena start faster.
tl;dr Kernel used to report crng initialised when it had a small amount of entropy. Now it reports crng initialised correctly when there is good entropy.
Worst case I've seen until now is ~7 minutes
root@2410213:~# dmesg | grep -i random
...
[ 1.182282] random: systemd: uninitialized urandom read (16 bytes read)
[ 444.351446] random: crng init done
[ 444.351453] random: 7 urandom warning(s) missed due to ratelimiting
root@2410213:~#
As a result, balena starts 7 minutes later.
Various packages have fixed this in various ways.
e.g. In util-linux, a flag was passed to getrandom to make the call nonblocking
util-linux/util-linux@a9cf659#diff-6341cb5466770a24a1f3bfdb80803412
Lots of Debian chatter about it. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=897572
From a cursory look, it looks like packages which wanted random (but not necessarily entropy in the security sense e.g. uuid generation) basically called getrandom with a nonblock flag so whatever random numbers came was fine.
Security stuff like ssh key gen basically have to wait. Or install something like rng-tools to improve entropy at boot faster.
Description
Steps to reproduce the issue:
- Boot resinOS v2.19 on a nuc image in a VM
- Check if balena is starting immediately on boot
- Check if balena starts after you see the crng init message comes when you type random stuff on the keyboard
Describe the results you received:
Balena starts after crng init.
Describe the results you expected:
Balena should start at boot
Additional information you deem important (e.g. issue happens only occasionally):
Output of balena version:
root@resin:~# balena version
Client:
Version: unknown-version
API version: 1.35
Go version: go1.9.7
Git commit: unknown-commit
Built: unknown-buildtime
OS/Arch: linux/amd64
Experimental: false
Orchestrator: swarm
Server:
Engine:
Version: 17.12.0-dev
API version: 1.35 (minimum version 1.12)
Go version: go1.9.7
Git commit: 3e2973d26934bd22c46f07764afb1ed8b11bf6a1
Built: Tue Oct 2 12:18:21 2018
OS/Arch: linux/amd64
Experimental: true
root@resin:~#
Output of balena info:
root@resin:~# balena info
Containers: 4
Running: 4
Paused: 0
Stopped: 0
Images: 4
Server Version: 17.12.0-dev
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 41
Dirperm1 Supported: true
Logging Driver: journald
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan null
Log: journald json-file
Swarm:
NodeID:
Is Manager: false
Node Address:
Runtimes: bare runc
Default Runtime: runc
Init Binary: balena-init
containerd version:
runc version: 13e66eedaddfbfeda2a73d23701000e4e63b5471
init version: 949e6fa-dirty (expected: 949e6facb7738)
Kernel Version: 4.14.48-yocto-standard
Operating System: Resin OS 2.15.1+rev2
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 989.3MiB
Name: resin
ID: 4FGZ:T47Z:U7X6:45DJ:4J7P:YFN3:QQM4:WOZ3:TTBL:UMQM:QOCU:7M6V
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 128
Goroutines: 101
System Time: 2018-10-08T17:23:38.887619837Z
EventsListeners: 1
Registry: https://index.docker.io/v1/
Labels:
Experimental: true
Insecure Registries:
192.168.1.166:5000
127.0.0.0/8
Live Restore Enabled: false
root@resin:~#
Additional environment details (AWS, VirtualBox, physical, etc.):