I marked this as draft because it will probably require some discussion to figure out if the approach is correct.

I ran make check locally and it did not regress the number of failures (my local make check is not clean with the master tree).

@dengert PTAL.

OpenSC can be built using OpenSSL 1.1x and OpenSSL-3.0.x LiberSSL so differet ifdefs are needed.

I have identified in OpenSC, these OpenSSL routines (or the _ex version) can use a libctx:
BIO_new
BN_CTX_new
EC_GROUP_new_by_curve_name
EC_KEY_new_by_curve_name
EC_KEY_new
EVP_default_properties_is_fips_enabled
EVP_MAC_fetch
EVP_PKEY_CTX_new_from_name
EVP_PKEY_new_raw_public_key
OSSL_PROVIDER_try_load
RAND_bytes

Some of these are used in many places. We may not need to worry about BIO_new or BN_CTX_new.

What we have done in the past is to add defines or macros in src/libopensc/sc-ossl-compat.h so if OpenSSL 3 is being used the define or macro would then add point to a libctx and call the appropiate OpenSSL 3 routine.

@dengert please check the commits
I added a separate ossl.c for the wrappers, but the function definitions are in opensc.h
Can easily move them to sc-ossl-compat.h if you think that's a better place, just let me know.

Btw you mised the two most important changes needed
EVP_MD_fetch() and EVP_CIPHER_fetch(0 which are the things that actually allow you to select which provider to use (based on libctx) for digest and encryption operations.
See the patches I already committed, they handle those cases.

EVP_default_properties_is_fips_enabled) is used directly at a high level in the tools, so it probably does not need its own context, applications can be left unchanged.
The libraries/modules are the problem.

What I did was for grep "(OSSL_LIB_CTX " include/openssl/* There were 79 functions osslinc-2.txt found

Then greped for these 79 functions and 24 functions without "_ex" to see what is in OpenSC source.

So there might be others that should be included.

EVP_default_properties_is_fips_enabled) is used directly at a high level in the tools, so it probably does not need its own context, applications can be left unchanged.

EVP_default_properties_is_fips_enabled was the command that caused OSSL_CRYPTO_init to be called. And it looks out of place. ./tools/pkcs11-tool.c line 5079 as FIPS_mode is define as EVP_default_properties_is_fips_enabled(NULL)

Could OSSL_LIB_CTX_get0_global_default() be used inplace of a libctx? What exactly the global_default?

It cause the issue because:
pkcs11-tool -> openssl -> pkcs11-provider -> opensc-pkcs11 -> openssl ->pkcs11-provider -> ...

But once we cut the loop in opensc-pkcs11 we will be fine, there is really no need to modify pkcs11-tool itself, only libraries/modules.

The call OSSL_LIB_CTX_get0_global_default() basically returns the default context, it is the call that tells you what context you get when you pass NULL in functions that can specify a libctx.

From OpenSSL code:

OSSL_LIB_CTX *OSSL_LIB_CTX_get0_global_default(void)
{
    if (!RUN_ONCE(&default_context_init, default_context_do_init))
        return NULL;

    return &default_context_int;
}

Passing the context returned by that call is equal to passing NULL.

So there might be others that should be included.

Be nice if you could take a quick look at the patches I am piling on, to see if I am going in the right direction ... there should be enough meat now to get a sense of what this change entails.

@dengert how stable need libsm interface need to be?

This library has two problems.

1. it is loading the legacy provider arbitrarily in the default context, this is a big no-no no matter what else
2. it expose public intefaces that do not take a sc_context_t pointer so I can't get to the custom openssl libctx unless I change the functions signatures.

Looks to me that libsm is a static library while libssm is a public one, so I will liberally change also libsm functions ...

Unfortunately the same problem is present in libssm ... I will break it and rthen we can figure out how to pick up the pieces

Checked out #2712 bd1fc42
and applied to OpenSC master

/opt/ossl-3.0.8/bin/pkcs11-tool --module /opt/ossl-3.0.8/lib/opensc-pkcs11.so --verify -m SHA384-RSA-PKCS-PSS -y cert -d 02 --signature-file /tmp/derive.132501.text.sig -i /tmp/derive.132501.text

0  futex_wait (private=0, expected=2, futex_word=0x7ffff7f8b420 <c_initialize_m>)
    at ../sysdeps/nptl/futex-internal.h:146
#1  __GI___lll_lock_wait (futex=futex@entry=0x7ffff7f8b420 <c_initialize_m>, private=0)
    at ./nptl/lowlevellock.c:49
#2  0x00007ffff7298082 in lll_mutex_lock_optimized (mutex=0x7ffff7f8b420 <c_initialize_m>)
    at ./nptl/pthread_mutex_lock.c:48
#3  ___pthread_mutex_lock (mutex=0x7ffff7f8b420 <c_initialize_m>)
    at ./nptl/pthread_mutex_lock.c:93
#4  0x00007ffff7f49c36 in C_Initialize (pInitArgs=0x7fffffffd0b0)
    at ../../../src/src/pkcs11/pkcs11-global.c:310
#5  0x00007ffff7ee82d4 in p11prov_Initialize (ctx=0x5555555db8d0, pInitArgs=0x7fffffffd0b0)
    at ../../src/src/interface.gen.c:10
#6  0x00007ffff7ef62eb in p11prov_module_init (ctx=0x5555555db8d0)
    at ../../src/src/provider.c:1011
#7  0x00007ffff7ef6973 in OSSL_provider_init (handle=0x5555555dad30, 
    in=0x7ffff7e66fc0 <core_dispatch_>, out=0x7fffffffd2c8, provctx=0x7fffffffd2d0)
    at ../../src/src/provider.c:1103
#8  0x00007ffff7c289ee in provider_init (prov=0x5555555dad30)
    at ../openssl/crypto/provider_core.c:930
#9  provider_activate (prov=prov@entry=0x5555555dad30, lock=0, lock@entry=1, 
    upcalls=upcalls@entry=1) at ../openssl/crypto/provider_core.c:1128
#10 0x00007ffff7c29b5c in ossl_provider_activate (aschild=0, upcalls=1, prov=0x5555555dad30)
    at ../openssl/crypto/provider_core.c:1257
--Type <RET> for more, q to quit, c to continue without paging--
#11 ossl_provider_activate (prov=prov@entry=0x5555555dad30, upcalls=upcalls@entry=1, 
    aschild=aschild@entry=0) at ../openssl/crypto/provider_core.c:1243
#12 0x00007ffff7c28051 in provider_conf_load (cnf=0x5555555d1560, 
    value=0x5555555d32d0 "pkcs11_sect", name=0x5555555d32b0 "pkcs11", libctx=0x0)
    at ../openssl/crypto/provider_conf.c:222
#13 provider_conf_init (md=<optimized out>, cnf=0x5555555d1560)
    at ../openssl/crypto/provider_conf.c:304
#14 0x00007ffff7b45bf6 in module_init (cnf=0x5555555d1560, 
    value=0x5555555d2fe0 "provider_sect", name=0x5555555d2f90 "providers", 
    pmod=0x5555555d1c20) at ../openssl/crypto/conf/conf_mod.c:421
#15 module_run (flags=<optimized out>, value=<optimized out>, 
    name=0x5555555d2f90 "providers", cnf=0x5555555d1560)
    at ../openssl/crypto/conf/conf_mod.c:266
#16 CONF_modules_load (flags=<optimized out>, appname=<optimized out>, cnf=0x5555555d1560)
    at ../openssl/crypto/conf/conf_mod.c:165
#17 CONF_modules_load (cnf=0x5555555d1560, appname=<optimized out>, flags=<optimized out>)
    at ../openssl/crypto/conf/conf_mod.c:117
#18 0x00007ffff7b466b4 in CONF_modules_load_file_ex (libctx=libctx@entry=0x0, filename=0x0, 
    appname=0x0, flags=50) at ../openssl/crypto/conf/conf_mod.c:208
#19 0x00007ffff7b467f4 in CONF_modules_load_file (filename=<optimized out>, 
    appname=<optimized out>, flags=<optimized out>) at ../openssl/crypto/conf/conf_mod.c:230
#20 0x00007ffff7b46a0a in ossl_config_int (settings=settings@entry=0x0)
    at ../openssl/crypto/conf/conf_sap.c:68
--Type <RET> for more, q to quit, c to continue without paging--
#21 0x00007ffff7c1d46f in ossl_init_config () at ../openssl/crypto/init.c:249
#22 ossl_init_config_ossl_ () at ../openssl/crypto/init.c:247
#23 0x00007ffff7299f68 in __pthread_once_slow (once_control=0x7ffff7e7e1e0 <config>, 
    init_routine=0x7ffff7c1d460 <ossl_init_config_ossl_>) at ./nptl/pthread_once.c:116
#24 0x00007ffff7c2c7ad in CRYPTO_THREAD_run_once (once=once@entry=0x7ffff7e7e1e0 <config>, 
    init=init@entry=0x7ffff7c1d460 <ossl_init_config_ossl_>)
    at ../openssl/crypto/threads_pthread.c:156
#25 0x00007ffff7c1dc49 in OPENSSL_init_crypto (opts=opts@entry=64, 
    settings=settings@entry=0x0) at ../openssl/crypto/init.c:588
#26 0x00007ffff7c5c79a in ossl_method_store_fetch (store=store@entry=0x5555555906c0, 
    nid=nid@entry=24577, prop_query=0x7ffff7d85b74 "", 
    prov_rw=prov_rw@entry=0x7fffffffd710, method=method@entry=0x7fffffffd610)
    at ../openssl/crypto/property/property.c:514
#27 0x00007ffff7bf0e3f in get_evp_method_from_store (store=0x5555555906c0, 
    prov=0x7fffffffd710, data=0x7fffffffd760) at ../openssl/crypto/evp/evp_fetch.c:172
#28 0x00007ffff7c1a2bb in ossl_method_construct (libctx=<optimized out>, 
    operation_id=operation_id@entry=1, provider_rw=provider_rw@entry=0x7fffffffd710, 
    force_store=force_store@entry=0, mcm=mcm@entry=0x7fffffffd720, 
    mcm_data=mcm_data@entry=0x7fffffffd760) at ../openssl/crypto/core_fetch.c:168
#29 0x00007ffff7bf149e in inner_evp_generic_fetch (
    free_method=0x7ffff7bd9860 <evp_md_free>, up_ref_method=0x7ffff7bd90c0 <evp_md_up_ref>, 
    new_method=0x7ffff7bd9870 <evp_md_from_algorithm>, properties=0x0, 
    name=0x7ffff7f78ef0 "sha1", name_id=<optimized out>, operation_id=1, 
--Type <RET> for more, q to quit, c to continue without paging--
    prov=<optimized out>, methdata=0x7fffffffd760) at ../openssl/crypto/evp/evp_fetch.c:344
#30 evp_generic_fetch (libctx=<optimized out>, operation_id=operation_id@entry=1, 
    name=0x7ffff7f78ef0 "sha1", properties=0x0, 
    new_method=new_method@entry=0x7ffff7bd9870 <evp_md_from_algorithm>, 
    up_ref_method=up_ref_method@entry=0x7ffff7bd90c0 <evp_md_up_ref>, 
    free_method=0x7ffff7bd9860 <evp_md_free>) at ../openssl/crypto/evp/evp_fetch.c:396
#31 0x00007ffff7bd907e in EVP_MD_fetch (ctx=<optimized out>, algorithm=<optimized out>, 
    properties=<optimized out>) at ../openssl/crypto/evp/digest.c:1073
#32 0x00007ffff7627b7f in sc_evp_md (context=0x55555558e250, 
    algorithm=0x7ffff7f78ef0 "sha1") at ../../../src/src/libopensc/ossl.c:32
#33 0x00007ffff7f5b416 in sc_pkcs11_register_openssl_mechanisms (p11card=0x555555597310)
    at ../../../src/src/pkcs11/openssl.c:307
#34 0x00007ffff7f5b112 in sc_pkcs11_register_generic_mechanisms (p11card=0x555555597310)
    at ../../../src/src/pkcs11/mechanism.c:1739
#35 0x00007ffff7f7066a in register_mechanisms (p11card=0x555555597310)
    at ../../../src/src/pkcs11/framework-pkcs15.c:6347
#36 0x00007ffff7f5d959 in pkcs15_bind (p11card=0x555555597310, app_info=0x0)
    at ../../../src/src/pkcs11/framework-pkcs15.c:345
#37 0x00007ffff7f541a6 in card_detect (reader=0x555555596550)
    at ../../../src/src/pkcs11/slot.c:323
#38 0x00007ffff7f5473d in card_detect_all () at ../../../src/src/pkcs11/slot.c:429
#39 0x00007ffff7f49d9b in C_Initialize (pInitArgs=0x0)
    at ../../../src/src/pkcs11/pkcs11-global.c:351
--Type <RET> for more, q to quit, c to continue without paging--
#40 0x000055555555fafe in main (argc=14, argv=0x7fffffffddb8)
    at ../../../src/src/tools/pkcs11-tool.c:1169
(gdb) 

We really need to get pkcs11-tool to force the OpenSSL to init before pkcs11-tool calls C_Initialize.
Waiting till the opensc-pkcs11 is loaded it to late.

Since pkcs11-tool loads libopensc but only to access some hex print routines and other utility functions,
I think it could access your OSSL commands to create a libctx in pkcs11-tool.

We really need to get pkcs11-tool to force the OpenSSL to init before pkcs11-tool calls C_Initialize.

Yes, that we have to do, but it should be quite simple, any call into openssl should force that.
Calling OSSL_LIB_CTX_get0_global_default() will force initialization if it didn't happen already.
The result can then be discarded for now.

@dengert see commit: Explicitly init OpenSSL in pkcs11-tool

Note that the pkcs11-tool commit is just a temporary workaround, it is not the correct fix, because it does not help any other application that may load opensc-pkcs11 in simialr situations (ie before openssl is initialized).
I believe the solution in this case will require us to completely forgot the standard openssl configuration file.
It should be fine because there isn't much this code cares about in the main openssl.cnf

An option could be to provide a custom configuration file for opensc-pkcs11, that would have the advantage of letting users customize it if they need, including adding or removing the legacy provider.
That may be a better option than manually loading the providers as the current code does.

Alternatively the configuration file might be optional and could be sourced from opensc.conf only if needed ...

Sadly it seems that the only way to init openssl without reading the config would then interfere with an application expecting a normal config to happen ...

I will leave this here for now, I think this patch is still something that needs to be introduced, if for nothing else because the current code is mucking with the default config by surreptitiously loading the legacy provider.

But this code alone does not, in fact, resolve the potential C_initialize loop ...
Will need more thinking about that.

Using OSSL_LIB_CTX_get0_global_default() does not cause OPENSSL_init_crypto to be run. But EVP_default_properties_is_fips_enabled(NULL); does:
pkcs11-tool-poke-openssl.diff.txt The loop does not occur when but for strange reason.

pkcs11-tool->EVP_default_properties_is_fips_enabled->...->OPENSSL_init_crypto->loads pkcs11-provider->loads-opensc-pkcs11-> pkcs11-provider->C_Initialize ... C_initialize returns OPENSSL_init_returns EVP_default_properties_is_fips_enabled returns pkcs11-tool loads opensc-pkcs11, then calls C_Initialize which returns

Main C_Initialize(NULL) rv:CKR_CRYPTOKI_ALREADY_INITIALIZED
Using slot 0 with a present token (0x0)
Using signature algorithm SHA384-RSA-PKCS-PSS
PSS parameters: hashAlg=SHA384, mgf=MGF1-SHA384, salt_len=48 B

then goes on to says "Invalid signature"

Without this change the path was opensc-pkcs11 while in C_Initialize to cause OpenSSL to init, anp provider, load opensc-pkcs11 and provider calls C_Initialize which is holding a lock.

PKCS11 C_Initialize can only return: CKR_ARGUMENTS_BAD, CKR_CANT_LOCK, CKR_CRYPTOKI_ALREADY_INITIALIZED, CKR_FUNCTION_FAILED, CKR_GENERAL_ERROR, CKR_HOST_MEMORY, CKR_NEED_TO_CREATE_THREADS, CKR_OK.
And it does not expect to be called by the same thread while in C_Initialize which was what was happing before.

And it does make any sense to me, that a PKCS11 application would ever use the pkcs11-provider to load the same PKCS11 module the application has loaded. But there maybe a way around that. If while in C_Initialize, C_Initalize is called by same thread, it should return CKR_FUNCTION_FAILED or maybe some vender specific CKR, which would then cause the provider not run.

I can look at this Thursday afternoon or Friday.

while in C_initialize causes provider to load

I think another way to deal with this could be to just call OPENSSL_init_crypto(0 in C_initialize, before it takes the lock.

It will return CKR_CRYPTOKI_ALREADY_INITIALIZED if nothing else is done.
The question is whether an error should be returned, or if some magic should be done such that both initializations end up returning CKR_OK.

Ideally we have a way to avoid getting OpenSSL to initialize, but the the code that detect cards always will end up calling something that makes openssl initialize apparently.

Sounds like a no-win scenario :-(

@dengert after thinking a bit about this I think it should be relatively easy to change C_Initialize so that we make the second initialization in OpenSSL either fail or return an alredy initialized error which pkcs11-provider currently accept as ok.

What should be the actual behavior?
I am thinking that perhaps we should make the second nested initialization actually always fail with a hard error, as you said if an application is using a pkcs11 module directly, openssl should probably not have access to it.

However we should not cause hard error if two initialization happen and they are not nested IMO (because otherwise you might get a hard error in the application if openssl initializes first through some other library load).

I'll propose a patch that handles just the nest (same process) case.

@dengert PTAL @ 2506431

I didn't use atomics so this may fail to trigger if mutliple threads call C_Initialize concurrently, but then for threads there is the mutex protecting later.
If we think spuriously returning CKR_GENERAL_ERROR with threads is problematic we can change the variable "nesting" to be a Thread Local variable (but I do not know how to do that portably with Windows, so I did not do it in this first draft).

Changed as requested, compiles here, but haven't run tests myself.

Thank you for the patience. Tested locally and I did not notice any issues. Merging.

@Jakuje You my also want to commit #2715