@bbbrumley if you have the extra cycles, you might want to chime in here!

@romen although I applaud your enthusiasm, we need to start locking stuff down for beta. This is feature creep :)
The only thing that I will glean from this - is if the existing kem interface meets your needs for expansion after 3.0.
If it doesn't then submit a PR for that part.

@romen although I applaud your enthusiasm, we need to start locking stuff down for beta. This is feature creep :)

@slontis Yes, I opened this mostly as a follow-up to the discussion in the last weekly meeting, to have an actual user of the KEM API.

I still really really wish we could manage to get this in before beta1, but I know it is tough and I don't expect it.

The only thing that I will glean from this - is if the existing kem interface meets your needs for expansion after 3.0.
If it doesn't then submit a PR for that part.

It seems like the current API is flexible enough to implement this, so I don't think there is a need for a separate PR to change something in the KEM API.

It seems like the current API is flexible enough to implement this, so I don't think there is a need for a separate PR to change something in the KEM API.

I am hoping there is a discussion about what beta is - and how we get there soon..

It seems like the current API is flexible enough to implement this, so I don't think there is a need for a separate PR to change something in the KEM API.

I am hoping there is a discussion about what beta is - and how we get there soon..

Maybe I phrased my answer ambiguously: I meant that, considering this PR just as a test of the usability of the existing KEM API, I believe the work done in #12750 does not require further changes that I can foresee.

Regarding the second part, i.e. my wish to have this merged in 3.0, I have no high expectations, and I understand and appreciate the "no feature creep" angle.
The only reason I am still proposing this is that I believe this PR would have been part of #11914 if the requirement for RSASVE that led to #12750 did not surface quite late and well into the "no feature creep" window, and that it is indeed valuable from a strategic point of view.

Thanks @mattcaswell for the feedback, 51b8d41dc12ae808ea96ed0b844238c10fd969fd should address all the issues.

Regarding the hold, I plan to talk about this on the committer or OTC meeting, as the schedule allows it, so we can collectively clarify if we want to include this feature or not in 3.0.

FWIW, this is the sort of thing I had in mind when I suggested that "potentially destabilizing changes" should not go into master until after 3.0 branches.

We share concerns about merging this PR before 3.0-beta1, because it is coming very late in the development cycle and because we all want to avoid "feature creep".

In favor of inclusion

My argument in favor of inclusion is that from a technical risk profile #13018 is not adding any significant amount of risk to the current 3.0 state and is, on the other hand, fulfilling long-standing requests from our users.

More in detail:

• Fully Pluggable TLSv1.3 KEM #13018 is a relatively small increment, from the public API perspective, compared to Fully Pluggable TLSv1.3 Key Exchange #11914, and is the natural completion of the feature originally implemented in it;
• IMO, the main reason why pluggable KEM wasn't already part of Fully Pluggable TLSv1.3 Key Exchange #11914 is that the latter limited its scope to DH-like KEX because, at the time of its writing, libcrypto had no concept of KEM and we had no plans to add it (as documented in API support for Key Encapsulation Mechanisms #7616);
• although Fully Pluggable TLSv1.3 KEM #13018 was opened this week, its design is not a "late-stage" addition:
  • personally, I have been working with a set of libssl patches affine to the libssl changes included in Fully Pluggable TLSv1.3 KEM #13018 since last December and announced last April;
  • similarly the OQS project has been managing an OpenSSL fork to enable post-quantum KEM in TLS, and this is not the only case of expert cryptographers wishing to use OpenSSL as a toolkit for the development and experimentation of new primitives and new integrations. All these users had so far no choice other than forking the project, diverting energies and expertise in maintaining such a fork rather than spending them improving and testing directly on OpenSSL;
  • API support for Key Encapsulation Mechanisms #7616 shows that our users have been asking for this level of flexibility for years now, and also how our stance on having an EVP API for KEM has changed over time
• "late stage featurization" is more prominent in Add RSASVE from SP800-56Br2 #12750, especially in terms of risk profile for the OpenSSL project, than in Fully Pluggable TLSv1.3 KEM #13018
  • for Add RSASVE from SP800-56Br2 #12750 it was a whole new API, backed by a single new implementation (RSASVE as required for FIPS validation) which has no known internal (except the additional tests) or external consumer
  • for Fully Pluggable TLSv1.3 KEM #13018 we have a relatively small API change (optional support for a single new OSSL_PARAM), it is adding an internal consumer for the API added in Add RSASVE from SP800-56Br2 #12750, and is backed by a long-standing request from our userbase, including many expert practitioners that are very likely going to be consumers of the features introduced in Fully Pluggable TLSv1.3 KEM #13018, Fully Pluggable TLSv1.3 Key Exchange #11914 and Add RSASVE from SP800-56Br2 #12750

Call for comments

To build support for my argument for inclusion, I'd like to seek endorsement by members of the community, especially among the expert practitioners that are using OpenSSL —or forks of OpenSSL— as a toolkit as part of their ongoing efforts in the NIST PQC process.

I encourage you to share your view on the capability added by this PR, suggest improvements if it is lacking in some fundamental regard, and weight in with your expertise in favor or against the inclusion of this change in the 3.0 timeframe, and its impact on the evolution of the KEM software ecosystem.

I'd jumpstart this by pinging, in no particular order, @thomwiggers, @henrydcase, @cryptojedi, @christianpaquin, @baentsch, @xvzcf, @dstebila, @bbbrumley.
There is more people that I'd like to ping, but I don't know their Github handles.
For some of them I am also extending the invite via side-channels (:wink:) outside Github, please feel free to do the same to reach users that you know might want to participate in this discussion.

I do not see this as a mainstream feature. Folks can always fork it themselves and experiment. Or it can go into the 3.1 release. I do not see any pressure to have every downstream user be able to plug in their own key exchange. It's neat, but not worth it to most.

I updated the description adding a summary written targeting users that have not been following the latest developments in 3.0, and adding a link to my argument for inclusion.
The original description is still included for ease of reference.

I fully support the idea of KEM API. I don't see any reason to wait for it until NIST PQC to be finished, as whatever will be chosen it will be a KEM. "Fully pluggable" will help the community to experiment with PQ crypto and will help companies to prepare for migrations.
One important aspect, in my opinion, is to keep in mind that industry is mostly interested in hybrid-KEM/KEX.
Not sure if it should be in 3.0 or 3.1. Would be great to have it rather soon.

Thanks @richsalz for the feedback, I appreciate it!

I don't expect to change your opinion on the matter, which is perfectly fine and agreeable for many, but I want to counterargument it to stimulate debate.

Folks can always fork it themselves and experiment. Or it can go into the 3.1 release.

True, and people have been doing so, but I believe this is also preventing the users that have to maintain their fork to be able to conduct their experiments directly in OpenSSL and devote more energies in improving OpenSSL directly.

Regarding delay to 3.1, what I argue is that now is the time to include this to avoid alienating a valuable part of our community, because now is when the focus of the NIST PQC is shifting more and more on experiments on real-world deployments, now national standardization bodies outside NIST are updating their recommendations to include a selection of PQC KEM.

I do not see any pressure to have every downstream user be able to plug in their own key exchange. It's neat, but not worth it to most.

That is not under discussion, that feature is already in, and I would say required by many users outside of US that have a recommendation or requirement to use different groups than P-256 or X25519, or the FFDHE without having to replace the OpenSSL version provided by the maintainers of their distribution: if someone had a requirement to use FourQ or NationalGroup42 they could do so with a provider, without patching OpenSSL.

On this matter, this PR only adds an extra flag to signal that the group is modeled according to KEM rather than DH-KEX.

I agree that at the moment the latter feature is useful almost exclusively to people experimenting about real-world deployments. The likelihood that some organization today wants/needs to deploy PQC KEM for TLSv1.3 is also quite low, but my expectation is that even this likelihood is bound to increase before we will be ready for a 3.1 release.

And I would still argue that this might not be motivation enough if this was a major new feature potentially causing further delays for 3.0 or with a moderate/high risk of biting us in the foreseeable future, but this is not the case for this PR.

• The code change in libssl is small, maximizing reuse of existing code.
• The change to the public API consists in a single extra OSSL_PARAM, within an already merged query mechanism that is already used by libssl and by our providers.
• The risk profile seems minimal.

Under these conditions I would argue that the pressure of unnecessarily alienating some of our users is the pressure I am seeing to get this in.
And it is the reason why I posted the above Call for Comments, to make the pressure more visible rather than just my personal gut feeling (that can easily be mistaken).

Sorry, I used the wrong term and that led you to go onto a larger rebuttal than necessary. I made most of my point and will not otherwise get into a debate to that others can contribute.

Thanks @henrydcase for your feedback!

I fully support the idea of KEM API. I don't see any reason to wait for it until NIST PQC to be finished, as whatever will be chosen it will be a KEM.

The KEM API is already in (thanks to #12750 work), so that already allows authors of OpenSSL Providers to inject such functionality in libcrypto.

"Fully pluggable" will help the community to experiment with PQ crypto and will help companies to prepare for migrations.
One important aspect, in my opinion, is to keep in mind that industry is mostly interested in hybrid-KEM/KEX.
Not sure if it should be in 3.0 or 3.1. Would be great to have it rather soon.

This PR further improves the mechanism that lets libssl interact with providers, allowing the provider to plug in a group as a KEM-group rather than a KEX-group: this wouldn't prevent hybrid-KEM/KEX scenarios, as KEX can be modeled under a KEM scheme and easily be combined with another KEM scheme, e.g. under this Internet-Draft by @dstebila.

It would be up to the OpenSSL Provider to plug a group PQCfooKEM and a separate group X25519+PQCfooKEM.
In the first case PQCfooKEM would encapsulate/decapsulate according to the reference implementation of the PQCfooKEM primitive, the second would encapsulate/decapsulate combining the X25519 operation and the PQCfooKEM

Excellent write-up, @romen!

yep, that sounds very useful, would be great to have it. makes integration of PQ into TLS much simpler. After looking at a code a bit closer, it seems like that's a relatively small change, comparing to #12750).

I realize only now that answering point by point to feedback as I did above, might come across as off-putting.
It was not my intention to do so, especially after soliciting feedback. It's just my (poor) choice of style in trying to organize my answers to compensate my writing skills in English, that can be sometime lacking.

I can do better and I'll try to, in the meantime I think it is important to extend my apologies if anyone felt annoyed by it, and to remark that I truly do welcome all feedback, not just the answers that are more or less clearly in favor of my argument for inclusion in 3.0!
I will strive to be more considerate!

I support this PR and the reasons are exactly those mentioned by Kris. Having personally hacked libssl so (1) it supports some kind of KEM paradigm; and (2) it knows about OIDs being injected from engines, I can confidently say that is not fun and nobody wants to maintain a fork.

Since others being tagged might feel a bit blindsided, I can summarize quickly -- at least for one project. @dstebila and @christianpaquin for OQS this would mean your OpenSSL fork goes away entirely -- cryptosystems load dynamically in libcrypto by registering through the Provider API (an upcoming OpenSSL 3.0 concept which, among other things, supersedes the Engine concept), and for KEMs it's this PR that bubbles that functionality up to libssl.

I support this PR and the reasons are exactly those mentioned by Kris. Having personally hacked libssl so (1) it supports some kind of KEM paradigm; and (2) it knows about OIDs being injected from engines, I can confidently say that is not fun and nobody wants to maintain a fork.

Since others being tagged might feel a bit blindsided, I can summarize quickly -- at least for one project. @dstebila and @christianpaquin for OQS this would mean your OpenSSL fork goes away entirely -- cryptosystems load dynamically in libcrypto by registering through the Provider API (an upcoming OpenSSL 3.0 concept which, among other things, supersedes the Engine concept), and for KEMs it's this PR that bubbles that functionality up to libssl.

Having a more direct method to add key exchange methods in libssl would be welcome. The goal of our fork is to be able to experiment with and demonstrate post-quantum algorithms prior to widespread adoption. We're able to do so by modifying the libcrypto and libssl code in many places, but would be happy to have a simpler route.

Our fork will probably go away eventually after the end of the NIST standardization and OpenSSL's eventual incorporation of those algorithms, and we're okay with that -- although it will take more than just pluggable KEMs in libssl, as we also do PQ and hybrid signatures in libssl, S/MIME & CMS signing, and a few more things.

Currently all our work has been on 1.1.1, and we haven't touched 3.0 yet, but are looking forward to experimenting with it when the time is right.

I didn't review the details of this PR or the KEM integration strategy, but to echo @dstebila's comments, I also fully agree with the addition of KEM support in OpenSSL. NIST will standardize a post-quantum KEM; doesn't matter which one, OpenSSL can proactively add support for the mechanism. In fact, OpenSSL's integration feedback would be helpful to the NIST competition, and the project would benefit if its preferred (from an integration's perspective) finalist is selected, rather than being forced later to integrate a scheme that's not as well suited for TLS or OpenSSL.

Early integration would also encourage people to experiment with real-world deployments. Hybrid (classical+PQ) is critical, and a good strategy should be implemented (see this paper for example). There is this chicken and egg problem of people wanting the feature but also wanting to use official OpenSSL code to conduct quantum-safe pilots. I, as many of my OQS colleagues I'm sure, would be happy to work with the OpenSSL team to make this happen.

OQS's goal is to provide post-quantum features on top of OpenSSL; if none exist, then we declare victory as the library would be quantum safe!

Just rebased.

• 316e1e5 fixup! [test][tls-provider] Group xor_group properties in a struct
• 47d809f fixup! [test][tls-provider] Add 2nd pluggable tls group for KEM
• 74ac213 fixup! [test][tls-provider] Implement KEM algorithm

Are the ones addressing latest feedback from @mattcaswell (last listed is latest)

The commits from 1574d6f up to 55a6890 (both included) are just the result of an automatic rebase with not conflicts.

This pull request is ready to merge

Merged on master with

• 32fea07 [test][tls-provider] Group xor_group properties in a struct
• c8e3a4c [test][sslapitest] Add test for pluggable KEM group
• ecff43e [test][tls-provider] Add 2nd pluggable tls group for KEM
• c1a74f5 Define OSSL_CAPABILITY_TLS_GROUP_IS_KEM
• a011b58 [ssl] Support ssl_decapsulate on client side
• 8b17fba [ssl] Support ssl_encapsulate on server side
• 5b70206 [test][tls-provider] Implement KEM algorithm

Thanks everyone for the feedback and the review.