Skip to content

Conversation

@laanwj
Copy link
Member

@laanwj laanwj commented Feb 21, 2017

These are available in sandboxes without access to files or to devices. Also they are considered safer and more straightforward to use than /dev/urandom as reading from a file has quite a few edge cases:

  • Linux: getrandom(buf, buflen, 0). getrandom(2) was introduced in version 3.17 of the Linux kernel.
  • OpenBSD: getentropy(buf, buflen). The getentropy(2) function appeared in OpenBSD 5.6.
  • FreeBSD and NetBSD: sysctl(KERN_ARND). Not sure when this was added but it has existed for quite a while.

Alternatives:

  • Linux has sysctl CTL_KERN / KERN_RANDOM / RANDOM_UUID which gives 16 bytes of randomness. This may be available on older kernels, however sysctl is deprecated on Linux and even removed in some distros so we shouldn't use it.

Add tests for GetOSRand():

  • Test that no error happens (otherwise RandFailure() which aborts)
  • Test that all 32 bytes are overwritten (initialize with zeros, try multiple times)

Discussion:

  • When to use these? Currently they are always used when available. Another option would be to use them only when /dev/urandom is not available. But this would mean these code paths receive less testing, and I'm not sure there is any reason to prefer /dev/urandom.

Closes: #9676. I've tested this on all three OS-es.

These are available in sandboxes without access to files or
devices. Also [they are safer and more straightforward](https://en.wikipedia.org/wiki/Entropy-supplying_system_calls)
to use than `/dev/urandom` as reading from a file has quite a few edge
cases:

- Linux: `getrandom(buf, buflen, 0)`. [getrandom(2)](http://man7.org/linux/man-pages/man2/getrandom.2.html)
  was introduced in version 3.17 of the Linux kernel.
- OpenBSD: `getentropy(buf, buflen)`. The [getentropy(2)](http://man.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man2/getentropy.2)
  function appeared in OpenBSD 5.6.
- FreeBSD and NetBSD: `sysctl(KERN_ARND)`. Not sure when this was added
  but it has existed for quite a while.

Alternatives:

- Linux has sysctl `CTL_KERN` / `KERN_RANDOM` / `RANDOM_UUID`
  which gives 16 bytes of randomness. This may be available
  on older kernels, however [sysctl is deprecated on Linux](https://lwn.net/Articles/605392/)
  and even removed in some distros so we shouldn't use it.

Add tests for `GetOSRand()`:

- Test that no error happens (otherwise `RandFailure()` which aborts)
- Test that all 32 bytes are overwritten (initialize with zeros, try multiple times)

Discussion:

- When to use these? Currently they are always used when available.
  Another option would be to use them only when `/dev/urandom` is not
  available. But this would mean these code paths receive less testing,
  and I'm not sure there is any reason to prefer `/dev/urandom`.

Closes: bitcoin#9676
@gmaxwell
Copy link
Contributor

Concept ACK. I think this should use a runtime test case at init, not just a boost testcase-- code may be built on a system where it works but run elsewhere and this is too important to mess up. :)

@laanwj
Copy link
Member Author

laanwj commented Feb 21, 2017

I'm OK with adding this to the sanity checks, but in all fairness we don't check that for /dev/urandom either, we used to not even have a boost testcase for this code path. Also a failure while reading randomness will already always trigger a fatal abort.

@sipa
Copy link
Member

sipa commented Feb 21, 2017

I think getrandom if available should be preferred over using /dev/urandom. While a local root access can both replace /dev/urandom with something else or intercept getrandom calls, I expect the latter to require more invasive/detectable changes.

@sipa
Copy link
Member

sipa commented Feb 21, 2017

Two questions:

  • What if you're using a 2.25 or later glibc (which has the getrandom function), but your Linux kernel is pre-3.17 (which does not have the syscall)? I have a hard time fnding information about this in the glibc documentation. Either it simulates it by reading /dev/urandom instead, or it fails. If the actual behavior is to fail, we should probably fall back to reading /dev/urandom at runtime rather than at compile time.
  • Given that our release binaries should probably not be required to work only on post-2.25 glibc, with this patch we can't compile release binaries against a post-2.25 glibc...

Nevermind, it seems you're using the syscall directly.

@gmaxwell
Copy link
Contributor

but in all fairness we don't check that for /dev/urandom either,

Yes but we don't have an issue there where there are older kernels that don't have the functionality... (Also, I would be of the view that we should have a runtime test currently too-- but I didn't want to bloat up the original PR that added this.)

@theuni
Copy link
Member

theuni commented Feb 21, 2017

Concept ACK. utACK on the build changes in particular.

src/random.cpp Outdated
* will always return as many bytes as requested and will not be
* interrupted by signals."
*/
if (syscall(SYS_getrandom, ent32, NUM_OS_RANDOM_BYTES, 0) != NUM_OS_RANDOM_BYTES) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this not need the GRND_BLOCK flag? I think we want this blocking rather than returning /dev/random data, no?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to the man page there is no GRND_BLOCK flag, just a GRND_NONBLOCK flag, which is only significant when passing GRND_RANDOM (which selects /dev/random) too. In that mode, it means EAGAIN can be returned if there is not enough entropy.

In /dev/urandom mode it will never block. When requesting <= 256 bytes it will not even be interruptable by a signal.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for clarifying. When I searched for the kernel implementation, I landed on https://lwn.net/Articles/605828/, which I guess was an early draft of getrandom with the flag reversed. Carry on :)

@laanwj
Copy link
Member Author

laanwj commented Feb 22, 2017

Nevermind, it seems you're using the syscall directly.

Well it still makes sense. On Linux, syscalls that are not implemented will predictably return an error w/ errno ENOSYS. We want to support older kernels (3.16 isn't that old - 2014) so need to handle this and fall back to /dev/urandom. Currently it will crash with a "randomness error" on those platforms [but only if compiled with kernel headers that did have the syscall].
Edit: added

Move the OS random test to a sanity check function that is called every
time bitcoind is initialized.

Keep `src/test/random_tests.cpp` for the case that later random tests
are added, and keep a rudimentary test that just calls the sanity check.
If the code was compiled with newer (>=3.17) kernel headers but executed
on a system without the system call, every use of random would crash the
program. Add a fallback for that case.
@sipa
Copy link
Member

sipa commented Feb 23, 2017

@laanwj It seems that Debian Jessie (stable) ships with Linux kernel 3.16, right before 3.17 which introduced getrandom(). That means that a binary compiled on a system with a >=3.17 kernel won't run on Jessie anymore, at all. I think we should prefer trying to call getrandom(), but if it fails, fall back to reading /dev/urandom.

It seems I'm consistently commenting on old states of this PR.

@sipa
Copy link
Member

sipa commented Feb 24, 2017

utACK 7e6dcd9

It seems none of the new methods are available on OSX (or at least in the build environment in Travis), but a new method can be added later.

@laanwj
Copy link
Member Author

laanwj commented Feb 24, 2017

It seems none of the new methods are available on OSX (or at least in the build environment in Travis), but a new method can be added later.

It may well be that OSX has no way to read kernel randomness besides /dev/urandom. We checked and their own frameworks use that, and nothing else. But yes someone with more knowledge about OSX could take a look at that later.

@jameshilliard
Copy link
Contributor

Libressl has getentropy emulation for a number of different OS's that may be useful as a reference.

@laanwj
Copy link
Member Author

laanwj commented Mar 1, 2017

Libressl has getentropy emulation for a number of different OS's that may be useful as a reference.

Thanks. That source confirms that there is no other way besides using /dev/urandom in OSX.

@laanwj laanwj merged commit 7e6dcd9 into bitcoin:master Mar 1, 2017
laanwj added a commit that referenced this pull request Mar 1, 2017
7e6dcd9 random: Add fallback if getrandom syscall not available (Wladimir J. van der Laan)
7cad849 sanity: Move OS random to sanity check function (Wladimir J. van der Laan)
aa09ccb squashme: comment that NUM_OS_RANDOM_BYTES should not be changed lightly (Wladimir J. van der Laan)
224e6eb util: Specific GetOSRandom for Linux/FreeBSD/OpenBSD (Wladimir J. van der Laan)

Tree-SHA512: 9fd408b1316c69de86674f342339b2f89192fd317c8c036b5df4320f828fa263c7966146bfc1904c51137ee4a26e4cb0f560b2cd05e18cde4d808b9b92ad15c4
@laanwj
Copy link
Member Author

laanwj commented Apr 1, 2017

Thanks. That source confirms that there is no other way besides using /dev/urandom in OSX.

While digging around I found that OSX 10.12+ does have getentropy, however it seems to be defined in sys/random.h instead of unistd.h as in OpenBSD:

__OSX_AVAILABLE(10.12) __IOS_AVAILABLE(10.0) __TVOS_AVAILABLE(10.0) __WATCHOS_AVAILABLE(3.0)
int getentropy(void* buffer, size_t size);

Example:

#include <stdlib.h>
#include <unistd.h>
#include <sys/random.h>

int main()
{
    unsigned char buf[32];
    if (getentropy(buf, sizeof(buf)) < 0) {
        return 1;
    }
    write(STDOUT_FILENO, buf, sizeof(buf));
    return 0;
}

I'm not sure how to make autoconf detect something that may be in different headers.

laanwj added a commit that referenced this pull request Aug 7, 2017
ee2d10a Check if sys/random.h is required for getentropy on OSX. (James Hilliard)

Pull request description:

  This should check and include sys/random.h if required for osx as mentioned [here](#9821 (comment)).

Tree-SHA512: e9491f67f2e8b2e6bcdbcbb8063295e844d5627daf5336e3e17b4a8027d888fa65a08e4580a745abdc35ffd8d86b4fc7434daaac172c4a06ab7566a2ed0bfb92
mempko pushed a commit to meritlabs/merit that referenced this pull request Sep 28, 2017
ee2d10a Check if sys/random.h is required for getentropy on OSX. (James Hilliard)

Pull request description:

  This should check and include sys/random.h if required for osx as mentioned [here](bitcoin/bitcoin#9821 (comment)).

Tree-SHA512: e9491f67f2e8b2e6bcdbcbb8063295e844d5627daf5336e3e17b4a8027d888fa65a08e4580a745abdc35ffd8d86b4fc7434daaac172c4a06ab7566a2ed0bfb92
PastaPastaPasta pushed a commit to PastaPastaPasta/dash that referenced this pull request Jan 23, 2019
7e6dcd9 random: Add fallback if getrandom syscall not available (Wladimir J. van der Laan)
7cad849 sanity: Move OS random to sanity check function (Wladimir J. van der Laan)
aa09ccb squashme: comment that NUM_OS_RANDOM_BYTES should not be changed lightly (Wladimir J. van der Laan)
224e6eb util: Specific GetOSRandom for Linux/FreeBSD/OpenBSD (Wladimir J. van der Laan)

Tree-SHA512: 9fd408b1316c69de86674f342339b2f89192fd317c8c036b5df4320f828fa263c7966146bfc1904c51137ee4a26e4cb0f560b2cd05e18cde4d808b9b92ad15c4
PastaPastaPasta added a commit to PastaPastaPasta/dash that referenced this pull request Jan 23, 2019
PastaPastaPasta pushed a commit to PastaPastaPasta/dash that referenced this pull request Aug 2, 2019
ee2d10a Check if sys/random.h is required for getentropy on OSX. (James Hilliard)

Pull request description:

  This should check and include sys/random.h if required for osx as mentioned [here](bitcoin#9821 (comment)).

Tree-SHA512: e9491f67f2e8b2e6bcdbcbb8063295e844d5627daf5336e3e17b4a8027d888fa65a08e4580a745abdc35ffd8d86b4fc7434daaac172c4a06ab7566a2ed0bfb92
charlesrocket pushed a commit to AXErunners/axe that referenced this pull request Nov 29, 2019
ee2d10a Check if sys/random.h is required for getentropy on OSX. (James Hilliard)

Pull request description:

  This should check and include sys/random.h if required for osx as mentioned [here](bitcoin/bitcoin#9821 (comment)).

Tree-SHA512: e9491f67f2e8b2e6bcdbcbb8063295e844d5627daf5336e3e17b4a8027d888fa65a08e4580a745abdc35ffd8d86b4fc7434daaac172c4a06ab7566a2ed0bfb92
@bitcoin bitcoin locked as resolved and limited conversation to collaborators Sep 8, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Use arc4random for entropy on supported systems

5 participants