This is an alternative to #10636, which implements kind of the same thing, but the other way around. Instead of asking the operation implementations what key types they support, we ask the KEYMGMT for the key what operations it supports and by what name.

This should fit better with the way OpenSSL works with EVP_PKEY_CTX, where the key type governs what is being done.

@levitte do you already have ideas of what could happen when we will have a "SM2:ec-PublicKey", sm2_keymgmt_functions entry?

The SM2 key type is serialized exactly like any other EC key type and differs only by the specific curve OID used inside the ASN1 structure, so both the EC and the SM2 keymgmt should intercept deserialization requests.

Do you think it would be possible to establish some priority order that works within a provider and across providers?

Ideally such a mechanism could allow to try to deserialize in the following scenario:

1. External EC provider for P-256 hw module
2. External EC provider for nation-specific EC (e.g. GOST)
3. Default provider: try SM2
4. Default provider: try EC

This order should be configurable by the user/system administrator and not coded inside the providers or libcrypto

I see two ways to handle EC vs SM2:

1. Treat them as distinct key types. Deserializers will have to reject data that doesn't fit the criteria for the type, even if they can decode the incoming payload. Are SM2 keys identified with a distinct OID when wrapped in PKCS#8? In that case, this seems like the way to go, even if the internal structure used in our providers when operating on it is the same as for ECDSA et al.
   Incidently, this is what we already do in the legacy implementation of RSA-PSS.
2. Treat them as the same, i.e see then as EC keys with some distinct parameters attached. This would be the choice if the PKCS#8 algorithm identifier is id-ecPublicKey for SM2 keys. Internally in the provider, we will still get a parameterised key structure, but the difference is that the operation algorithm will have to be able to support all variants.

The second choice makes me think that we might want to support recursive providers, i.e implementations that do their own fetching. Oh boy...

In the end, we need to look at PKCS#8 when looking at what types we face...

Unfortunately, SM2 keys use the same OID as EC keys.
But the approach in number 2 would mean that any provider capable of handling id-ecPubliKey keys must support every EC (or derived) algorithm.

For example if a user wants to load a provider to enable her shiny hw crypto accelerator (that only supports P-256) we don't want (unless configured otherwise) to lose the ability to handle other curves through the default provider or SM2, etc.

Oh, one thing that I'm forgetting here is that we currently don't have any provided deserializers. They will normally be used via the OSSL_STORE "file:" backend, and my plan is to do exactly what I think you request, to loop through all available deserializers until one successfully returns an unpacked object.

Same problem would arise if we carry on the plans to have national-based crypto in separate providers.

Let's say the brainpool curves are not in the default provider anymore but in a German specific provider, or the binary curves are moved to the legacy provider.

We really don't won't to repeat the mistake of having ENGINE/providers have to hijack the whole EC module for that!

It dawned on me that we're not as dependent on pkcs#8 as I was thinking, so we can choose to make SM2 keys a distinct key type for keymgmt, and teach our deserializers to check for specific parameters to decide if the payload is acceptable or not.
Thank you, your input also made it clearer how to treat RSA-PSS keys.

We really don't won't to repeat the mistake of having ENGINE/providers have to hijack the whole EC module for that!

Agreed. So I think that between the two choices in #10647 (comment), the first choice provides the possibility for distinct and less monolithic providers.

Rebased for better Travis results

Minus the nit mentioned in the previous review, the testing done so far in #10631 (comment) suggests everything seems to work fine in this PR.

Fixed, and rebased

Merged.

e62a45b CORE & EVP: Specify OP_query_operation_name() for KEYMGMT
f23bc0b EVP: Adapt KEYEXCH, SIGNATURE and ASYM_CIPHER to handle key types better
2293032 PROV: Adjust the KEYMGMT name specs to include all names