Skip to content

Comments

OSSL_STORE for providers, take 2#12512

Closed
levitte wants to merge 15 commits intoopenssl:masterfrom
levitte:ossl_store-for-providers-2
Closed

OSSL_STORE for providers, take 2#12512
levitte wants to merge 15 commits intoopenssl:masterfrom
levitte:ossl_store-for-providers-2

Conversation

@levitte
Copy link
Member

@levitte levitte commented Jul 22, 2020

This adapts the OSSL_STORE API to use provider backends.

On the provider side, this adds a new "operation" called storemgmt.

This also takes the special set of OSSL_PARAM keys and values used in #12410 and formalizes it into something called a provider-native object abstraction. It turns out that the STORE backend and the DESERIALIZER backend share quite a few properties through that set of parameters. Documentation in provider-object(7).

@levitte levitte added the branch: master Applies to master branch label Jul 22, 2020
@levitte
Copy link
Member Author

levitte commented Jul 22, 2020

Note that this is built on top of #12410, so there are quite a number of commits that don't really belong to this PR.

@levitte levitte force-pushed the ossl_store-for-providers-2 branch from b4bf78a to 5d833aa Compare July 22, 2020 21:09
@levitte
Copy link
Member Author

levitte commented Jul 22, 2020

Tomorrow, my plan is to move the file: scheme loader to become a provider implementation.

@levitte levitte changed the title [WIP] OSSL_STORE for providers, take 2 [WIP, Pending on #12544] OSSL_STORE for providers, take 2 Jul 28, 2020
@slontis
Copy link
Member

slontis commented Aug 4, 2020

This needs to be rebased now that other dependencies are closed

@levitte
Copy link
Member Author

levitte commented Aug 4, 2020

This needs to be rebased now that other dependencies are closed

There's more coming anyway...

@slontis
Copy link
Member

slontis commented Aug 4, 2020

There's more coming anyway...

Of course there is :)

@levitte levitte force-pushed the ossl_store-for-providers-2 branch from 588a753 to fc0896d Compare August 4, 2020 07:46
@levitte levitte changed the title [WIP, Pending on #12544] OSSL_STORE for providers, take 2 [WIP] OSSL_STORE for providers, take 2 Aug 4, 2020
@levitte levitte changed the title [WIP] OSSL_STORE for providers, take 2 [WIP, pending #12574] OSSL_STORE for providers, take 2 Aug 5, 2020
@levitte levitte force-pushed the ossl_store-for-providers-2 branch from fc0896d to f912be0 Compare August 5, 2020 09:51
@levitte levitte changed the title [WIP, pending #12574] OSSL_STORE for providers, take 2 [WIP] OSSL_STORE for providers, take 2 Aug 7, 2020
@levitte levitte force-pushed the ossl_store-for-providers-2 branch 3 times, most recently from af90249 to e268208 Compare August 7, 2020 03:31
@levitte levitte changed the title [WIP] OSSL_STORE for providers, take 2 OSSL_STORE for providers, take 2 Aug 7, 2020
@levitte
Copy link
Member Author

levitte commented Aug 7, 2020

This is ready for review

@levitte levitte force-pushed the ossl_store-for-providers-2 branch from a9f4777 to 876d4a0 Compare August 7, 2020 15:44
openssl-machine pushed a commit that referenced this pull request Aug 24, 2020
This is placed as CORE because the core of libcrypto is the authority
for what is possible to do and what's required to make these abstract
objects work.

In essence, an abstract object is an OSSL_PARAM array with well
defined parameter keys and values:

-   an object type, which is a number indicating what kind of
    libcrypto structure the object in question can be used with.  The
    currently possible numbers are defined in <openssl/core_object.h>.
-   an object data type, which is a string that indicates more closely
    what the contents of the object are.
-   the object data, an octet string.  The exact encoding used depends
    on the context in which it's used.  For example, the decoder
    sub-system accepts any encoding, as long as there is a decoder
    implementation that takes that as input.  If central code is to
    handle the data directly, DER encoding is assumed. (*)
-   an object reference, also an octet string.  This octet string is
    not the object contents, just a mere reference to a provider-native
    object. (**)
-   an object description, which is a human readable text string that
    can be displayed if some software desires to do so.

The intent is that certain provider-native operations (called X
here) are able to return any sort of object that belong with other
operations, or an object that has no provider support otherwise.

(*) A future extension might be to be able to specify encoding.

(**) The possible mechanisms for dealing with object references are:

-   An object loading function in the target operation.  The exact
    target operation is determined by the object type (for example,
    OSSL_OBJECT_PKEY implies that the target operation is a KEYMGMT)
    and the implementation to be fetched by its object data type (for
    an OSSL_OBJECT_PKEY, that's the KEYMGMT keytype to be fetched).
    This loading function is only useful for this if the implementations
    that are involved (X and KEYMGMT, for example) are from the same
    provider.

-   An object exporter function in the operation X implementation.
    That exporter function can be used to export the object data in
    OSSL_PARAM form that can be imported by a target operation's
    import function.  This can be used when it's not possible to fetch
    the target operation implementation from the same provider.

Reviewed-by: Shane Lontis <[email protected]>
(Merged from #12512)
openssl-machine pushed a commit that referenced this pull request Aug 24, 2020
The pass phrase prompter that's part of OSSL_ENCODER and OSSL_DECODER
is really a passphrase callback bridge between the diverse forms of
prompters that exist within OpenSSL: pem_password_cb, ui_method and
OSSL_PASSPHRASE_CALLBACK.

This can be generalised, to be re-used by other parts of OpenSSL, and
to thereby allow the users to specify whatever form of pass phrase
callback they need, while being able to pass that on to other APIs
that are called internally, in the form that those APIs demand.

Additionally, we throw in the possibility to cache pass phrases during
a "session" (we leave it to each API to define what a "session" is).
This is useful for any API that implements discovery and therefore may
need to get the same password more than once, such as OSSL_DECODER and
OSSL_STORE.

Reviewed-by: Shane Lontis <[email protected]>
(Merged from #12512)
openssl-machine pushed a commit that referenced this pull request Aug 24, 2020
openssl-machine pushed a commit that referenced this pull request Aug 24, 2020
openssl-machine pushed a commit that referenced this pull request Aug 24, 2020
This includes fixing a bug that could only be discovered when no
loaders were registered.

Reviewed-by: Shane Lontis <[email protected]>
(Merged from #12512)
openssl-machine pushed a commit that referenced this pull request Aug 24, 2020
When some function receives an OSSL_PARAM array to pilfer for data,
and there is a string of some sort, and all the code needs is to get
the pointer to the data, rather than a copy, there is currently no
other way than to use |param->data| directly.  This is of course a
valid method, but lacks any safety check (is |param->data_type|
correct, for example?).

OSSL_PARAM_get_utf8_string_ptr() and OSSL_PARAM_get_octet_string_ptr()
helps the programmer with such things, by setting the argument pointer
to |param->data|.
Additionally, the handle the data types OSSL_PARAM_UTF8_PTR and
OSSL_PARAM_OCTET_PTR as well.

Reviewed-by: Shane Lontis <[email protected]>
(Merged from #12512)
openssl-machine pushed a commit that referenced this pull request Aug 24, 2020
This makes it possible to use OSSL_DECODER in functions that are passed
a OSSL_PASSPHRASE_CALLBACK already.

Reviewed-by: Shane Lontis <[email protected]>
(Merged from #12512)
openssl-machine pushed a commit that referenced this pull request Aug 24, 2020
This adds the needed code to make the OSSL_STORE API functions handle
provided STORE implementations.

This also modifies OSSL_STORE_attach() for have the URI, the
library context and the properties in the same order as
OSSL_STORE_open_with_libctx().

The most notable change, though, is how this creates a division of
labor between libcrypto and any storemgmt implementation that wants to
pass X.509, X.509 CRL, etc structures back to libcrypto.  Since those
structures aren't directly supported in the libcrypto <-> provider
interface (asymmetric keys being the only exception so far), we resort
to a libcrypto object callback that can handle passed data in DER form
and does its part of figuring out what the DER content actually is.

This also adds the internal x509_crl_set0_libctx(), which works just
like x509_set0_libctx(), but for X509_CRL.

Reviewed-by: Shane Lontis <[email protected]>
(Merged from #12512)
openssl-machine pushed a commit that referenced this pull request Aug 24, 2020
@levitte
Copy link
Member Author

levitte commented Aug 24, 2020

Merged

14c8a3d CORE: Define provider-native abstract objects
a517ede CORE: Generalise internal pass phrase prompter
faa64bc STORE: Add missing function OSSL_STORE_LOADER_set_open_with_libctx()
8704b6b STORE for providers: define libcrypto <-> provider interface
c4fc564 STORE: Add the base functions to support provider based loaders
ab00ddb OSSL_PARAM: Add string pointer getters
4fd3978 DECODER: Add function to set an OSSL_PASSPHRASE_CALLBACK type callback
34b80d0 STORE: Modify to support loading with provider based loaders
a8b7ea8 STORE: Change all error recording to use ERR_raise() / ERR_raise_data()

@levitte levitte closed this Aug 24, 2020
@levitte levitte deleted the ossl_store-for-providers-2 branch August 24, 2020 08:05
swenkeratmicrosoft pushed a commit to swenkeratmicrosoft/openssl that referenced this pull request Sep 1, 2020
This is placed as CORE because the core of libcrypto is the authority
for what is possible to do and what's required to make these abstract
objects work.

In essence, an abstract object is an OSSL_PARAM array with well
defined parameter keys and values:

-   an object type, which is a number indicating what kind of
    libcrypto structure the object in question can be used with.  The
    currently possible numbers are defined in <openssl/core_object.h>.
-   an object data type, which is a string that indicates more closely
    what the contents of the object are.
-   the object data, an octet string.  The exact encoding used depends
    on the context in which it's used.  For example, the decoder
    sub-system accepts any encoding, as long as there is a decoder
    implementation that takes that as input.  If central code is to
    handle the data directly, DER encoding is assumed. (*)
-   an object reference, also an octet string.  This octet string is
    not the object contents, just a mere reference to a provider-native
    object. (**)
-   an object description, which is a human readable text string that
    can be displayed if some software desires to do so.

The intent is that certain provider-native operations (called X
here) are able to return any sort of object that belong with other
operations, or an object that has no provider support otherwise.

(*) A future extension might be to be able to specify encoding.

(**) The possible mechanisms for dealing with object references are:

-   An object loading function in the target operation.  The exact
    target operation is determined by the object type (for example,
    OSSL_OBJECT_PKEY implies that the target operation is a KEYMGMT)
    and the implementation to be fetched by its object data type (for
    an OSSL_OBJECT_PKEY, that's the KEYMGMT keytype to be fetched).
    This loading function is only useful for this if the implementations
    that are involved (X and KEYMGMT, for example) are from the same
    provider.

-   An object exporter function in the operation X implementation.
    That exporter function can be used to export the object data in
    OSSL_PARAM form that can be imported by a target operation's
    import function.  This can be used when it's not possible to fetch
    the target operation implementation from the same provider.

Reviewed-by: Shane Lontis <[email protected]>
(Merged from openssl#12512)
swenkeratmicrosoft pushed a commit to swenkeratmicrosoft/openssl that referenced this pull request Sep 1, 2020
The pass phrase prompter that's part of OSSL_ENCODER and OSSL_DECODER
is really a passphrase callback bridge between the diverse forms of
prompters that exist within OpenSSL: pem_password_cb, ui_method and
OSSL_PASSPHRASE_CALLBACK.

This can be generalised, to be re-used by other parts of OpenSSL, and
to thereby allow the users to specify whatever form of pass phrase
callback they need, while being able to pass that on to other APIs
that are called internally, in the form that those APIs demand.

Additionally, we throw in the possibility to cache pass phrases during
a "session" (we leave it to each API to define what a "session" is).
This is useful for any API that implements discovery and therefore may
need to get the same password more than once, such as OSSL_DECODER and
OSSL_STORE.

Reviewed-by: Shane Lontis <[email protected]>
(Merged from openssl#12512)
swenkeratmicrosoft pushed a commit to swenkeratmicrosoft/openssl that referenced this pull request Sep 1, 2020
swenkeratmicrosoft pushed a commit to swenkeratmicrosoft/openssl that referenced this pull request Sep 1, 2020
swenkeratmicrosoft pushed a commit to swenkeratmicrosoft/openssl that referenced this pull request Sep 1, 2020
This includes fixing a bug that could only be discovered when no
loaders were registered.

Reviewed-by: Shane Lontis <[email protected]>
(Merged from openssl#12512)
swenkeratmicrosoft pushed a commit to swenkeratmicrosoft/openssl that referenced this pull request Sep 1, 2020
When some function receives an OSSL_PARAM array to pilfer for data,
and there is a string of some sort, and all the code needs is to get
the pointer to the data, rather than a copy, there is currently no
other way than to use |param->data| directly.  This is of course a
valid method, but lacks any safety check (is |param->data_type|
correct, for example?).

OSSL_PARAM_get_utf8_string_ptr() and OSSL_PARAM_get_octet_string_ptr()
helps the programmer with such things, by setting the argument pointer
to |param->data|.
Additionally, the handle the data types OSSL_PARAM_UTF8_PTR and
OSSL_PARAM_OCTET_PTR as well.

Reviewed-by: Shane Lontis <[email protected]>
(Merged from openssl#12512)
swenkeratmicrosoft pushed a commit to swenkeratmicrosoft/openssl that referenced this pull request Sep 1, 2020
This makes it possible to use OSSL_DECODER in functions that are passed
a OSSL_PASSPHRASE_CALLBACK already.

Reviewed-by: Shane Lontis <[email protected]>
(Merged from openssl#12512)
swenkeratmicrosoft pushed a commit to swenkeratmicrosoft/openssl that referenced this pull request Sep 1, 2020
This adds the needed code to make the OSSL_STORE API functions handle
provided STORE implementations.

This also modifies OSSL_STORE_attach() for have the URI, the
library context and the properties in the same order as
OSSL_STORE_open_with_libctx().

The most notable change, though, is how this creates a division of
labor between libcrypto and any storemgmt implementation that wants to
pass X.509, X.509 CRL, etc structures back to libcrypto.  Since those
structures aren't directly supported in the libcrypto <-> provider
interface (asymmetric keys being the only exception so far), we resort
to a libcrypto object callback that can handle passed data in DER form
and does its part of figuring out what the DER content actually is.

This also adds the internal x509_crl_set0_libctx(), which works just
like x509_set0_libctx(), but for X509_CRL.

Reviewed-by: Shane Lontis <[email protected]>
(Merged from openssl#12512)
swenkeratmicrosoft pushed a commit to swenkeratmicrosoft/openssl that referenced this pull request Sep 1, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approval: ready to merge The 24 hour grace period has passed, ready to merge branch: master Applies to master branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants