At this stage, I'm trying to nail down a sensible API for the EVP_RAND interface and the libcrypto to provider interface.

The intention is for libcrypto to create and link up the DRBG chains (master, public & private), however FIPS effectively mandates that the data moving between the DRBGs in the chain must stay within the FIPS boundary.

Mapping existing RAND and DRBG APIs onto the new will be done as a best effort later. Note: only CTR DRBG was present in 1.1.1, so we've some flexibility to not wedge the other DRBGs into the framework.

Mapping existing RAND and DRBG APIs onto the new will be done as a best effort later. Note: only CTR DRBG was present in 1.1.1, so we've some flexibility to not wedge the other DRBGs into the framework.

So...until that is done calls to RAND_bytes etc will use the libcrypto RAND implementation? But applications would have the option of using fetchable RAND instead? That sounds like a workable compromise if we convert our internal usage to the new API (particularly libssl).

If things go badly, yes. I hope that RAND and DRBG are reworked in time for release.

Still, early days.

Suppose you remove the ability to reseed across providers, but instead mandate that each provider has to seed from the environment and do seeding solely inside itself. In other words, follow the current openssl drbg-chaining. Doesn't that make things MUCH simpler?

Also libcrypto could provide call (callback for providers?) to provide the entropy from environment.

But how would be the per-thread and separate drbg for private and non-private data be done within a provider? I think the per-thread instances could be internal implementation detail of the provider but separating private and non-private RNGs would have to be somehow reflected in the provider interface.

@t8m are you asking me or paul? If me, put the master drbg and the per-thread-data keys into OPENSSL_CTX. Then RAND_BYTES_ex just looks things up in the ctx data. Trivial.

Suppose you remove the ability to reseed across providers, but instead mandate that each provider has to seed from the environment and do seeding solely inside itself. In other words, follow the current openssl drbg-chaining. Doesn't that make things MUCH simpler?

Doesn't this mandate adding down calls to the provider which wasn't popular?

Also libcrypto could provide call (callback for providers?) to provide the entropy from environment.

From a FIPS perspective, this isn't workable. The seed material needs to stay within the FIPS boundary or be transferred using a secure channel.

The DRBG callbacks will, of course, need to be supported.

Can you explain why it's not workable if the provider calls to the OS to get seed material itself?

You have to add some downcalls. I envision RAND{priv}bytes_ex calls to the provider to do the rand_generate and the reseeding. So only 3-4 downcalls are needed.

The provider can call the OS to get seed material. That's FIPS friendly.

There might be a nomenclature issue. I'm using down calls meaning calls to the OSSL_PROVIDER object as distinct to calls to the rand algorithm the provider has. I think this is the what @mattcaswell uses the term for too.

At this stage this is just a pair of proposed APIs. One for EVP_RAND and one for the libcrypto/provider boundary. I think they should be workable within the available time but I made a mess of it last try...

Nomenclature-wise, I've borrowed the terms used with SGX, where they talk about downcalls when it's calls from the insecure environment into the secure enclave, and upcalls when it's calls from the secure enclave to the insecure environment.
In OpenSSL terms, the calls from libcrypto to providers have become "downcalls", while calls from providers to libcrypto have become "upcalls".

I suspect that @mattcaswell uses the same nomenclature... at the very least, he seems to understand precisely what I mean when I use those terms.

Side note: I recently discovered gitglossary(7). I've wondered for some time if we should have something like that.

Also libcrypto could provide call (callback for providers?) to provide the entropy from environment.

From a FIPS perspective, this isn't workable. The seed material needs to stay within the FIPS boundary or be transferred using a secure channel.

I am talking here about collecting the entropy from environment - what is the difference from calling getrandom() from glibc or calling a function OSSL_getrandom() from openssl which in turn calls getrandom() from glibc? BTW the buffer for the data can be supplied by the provider so there does not necessarily have to be any data transfer outside the FIPS boundary.
But yes, if this is not acceptable then I suppose we will have to build the rand_platform.c files into the FIPS provider.

Call glibc directly is acceptable in a FIPS context. Calling a wrapper in OpenSSL (outside of the FIPS boundary) requires a secure channel for the entropy. There is a world of difference.

rand_platform.c needs to be inside the FIPS provider.

And that's the time when the non-sensical FIPS rules strike again. Let's just imagine that the "secure channel" would be provided. Where the channel would end on the other side? In kernel? Unless kernel is a FIPS module it could not be. And if there would be no buffer copying involved in the imagined OpenSSL callback that would call the getrandom() the data as collected from the system would not even be transferred through OpenSSL itself so why we need any secure channel if there is no intermediary data transfer. (Not talking about the triviality that there is single process with a single address space involved anyway.) Anyway I am not going to fight FIPS rules here so let's duplicate some code! 😁

Just a side-note, in case this force-push was the result of an interactive rebase to edit commits:

Unless you really need to integrate changes (e.g. bugfixes) from upstream, it would be convenient for the reviewers if you could avoid moving the merge-base, because then GitHub's force-push diff shows exactly your changes and not those from master as well. (Note the link which is displayed by GitHub automatically in the force-push message.)

@paulidale force-pushed the paulidale:evp-rand branch from c3f2640 to 8908b8a ...

You can either do this by specifying the number of commits to rebase (here: git rebase -i HEAD~2), or more generically, by using git rebase -i --keep-base master

See git-rebase(1):

--keep-base
    Set the starting point at which to create the new commits to the merge base of <upstream> <branch>.
    Running git rebase --keep-base <upstream> <branch> is equivalent to
    running git rebase --onto <upstream>... <upstream>.

    This option is useful in the case where one is developing a feature on top of an upstream branch.
    While the feature is being worked on, the upstream branch may advance and
    it may not be the best idea to keep rebasing on top of the upstream but to keep the base commit as-is.

The force push was primarily a rebase to the current master.

Summary of changes:

1. hash DRBG is present in the provider.
2. DRBG common infrastructure is mostly done but doesn't build yet.
3. A test RNG is present but again, doesn't build.
4. The continuous tests are present in the provider -- this is intended to be inserted into the DRBG chain as a filter.
5. EVP level calls are more refined.

24 hours has passed since 'approval: done' was set, but as this PR has been updated in that time the label 'approval: ready to merge' is not being automatically set. Please review the updates and set the label manually.

CI failures aren't related.
Merged to master. Thanks for the marathon efforts from all.