The internal function pkey_set_type seems to mishandle ENGINE references passed to it. Most of the time it consumes the reference on success - but not always:

Although by luck I think the above codepath probably can't be hit at the moment with existing usage of pkey_set_type where the ENGINE is non-null since the pkey is always freshly allocated when it is called.

The fact that pkey_set_type() consumes the ENGINE reference seems quite surprising and means that the public API functions EVP_PKEY_new_raw_private_key_ex(), EVP_PKEY_new_raw_private_key(), EVP_PKEY_new_raw_public_key_ex() and EVP_PKEY_new_raw_public_key() also consume it - even in some failure cases (but there is at least one case where it is not consumed).

Due to the behaviour of pkey_set_type() this also means the function EVP_PKEY_new_CMAC_key() consumes the ENGINE reference in (most) cases in 1.1.1, but no longer does so in 3.0 (because the implementation of EVP_PKEY_new_CMAC_key() has changed considerably and no longer calls pkey_set_type()).

The documentation for these functions makes no reference to the fact that the ENGINE reference is consumed, and this seems quite surprising behaviour because most other similar functions that take an ENGINE reference do not do so.

I do not believe it was the original intent for the reference to be consumed, and I think the behaviour of pkey_set_type is an error.

See also:
https://mta.openssl.org/pipermail/openssl-users/2021-October/014423.html