OpenSSL UBSan Type Mismatch in EVP_DigestUpdate

Dear OpenSSL developers,

I found an UndefinedBehaviorSanitizer error in OpenSSL when using the EVP digest API.

Version

openssl-3.4.0-alpha1-2545-gc8b4a397d0

Description

The crash is a UBSan function type mismatch in EVP_DigestUpdate at crypto/evp/digest.c:299. When calling EVP_Digest() with EVP_sha256(), the library dispatches through a function pointer ctx->digest->dupdate which has type OSSL_FUNC_digest_update_fn*. However, it points to SHA256_Update which has signature int SHA256_Update(SHA256_CTX*, const void*, size_t).

The type mismatch is:

• Expected by function pointer: int (*)(void *, const unsigned char *, unsigned long)
• Actual function signature: int SHA256_Update(SHA256_CTX*, const void*, size_t)

This violates strict aliasing rules and is undefined behavior according to the C standard. The issue appears to be in the bridge between legacy digest implementations and the new provider architecture.

Reproduction Steps

1. Compile the PoC:

clang++ -g -fsanitize=address,undefined \
    -I/path/to/openssl/include \
    poc.cpp -o poc \
    -L/path/to/openssl/lib64 -lcrypto \
    -Wl,-rpath,/path/to/openssl/lib64

2. Run with UBSan:

export UBSAN_OPTIONS="halt_on_error=1:print_stacktrace=1"
./poc

Stack Trace

$ export UBSAN_OPTIONS="halt_on_error=1:print_stacktrace=1"
$ ./poc
crypto/evp/digest.c:299:12: runtime error: call to function SHA256_Update through pointer to incorrect function type 'int (*)(void *, const unsigned char *, unsigned long)'
/root/src/openssl/include/crypto/md32_common.h:158: note: SHA256_Update defined here
    #0 0x7dbd3c4ac305 in EVP_DigestUpdate /root/src/openssl/crypto/evp/digest.c:299:12
    #1 0x7dbd3c4b8614 in EVP_Digest /root/src/openssl/crypto/evp/digest.c:601:12
    #2 0x5d4670f0dcea in main poc.cpp:17:5
    #3 0x7dbd3a02a1c9  (/lib/x86_64-linux-gnu/libc.so.6+0x2a1c9)
    #4 0x7dbd3a02a28a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2a28a)
    #5 0x5d4670e32364 in _start (poc+0x2c364)

SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior crypto/evp/digest.c:299:12

PoC Code

// Minimal PoC: OpenSSL UBSan type mismatch in EVP_DigestUpdate
// Compile: clang++ -g -fsanitize=address,undefined -I<openssl_include> poc.cpp -o poc -L<openssl_lib> -lcrypto

#include <openssl/evp.h>
#include <cstring>

int main() {
    unsigned char data[] = "test data for SHA256 digest";
    unsigned char hash[EVP_MAX_MD_SIZE];
    unsigned int hash_len = 0;
    
    // This triggers the type mismatch in EVP_DigestUpdate -> SHA256_Update
    EVP_Digest(data, strlen((char*)data), hash, &hash_len, EVP_sha256(), NULL);
    
    return 0;
}