The openssl accepts invalid malformed signature in s390x arch in v3.2 and 3.5.

Affected versions: openssl 3.2 and openssl 3.5

How to reproduce:

Download the pyca-cryptography and wycheproof . The pyca-cryptography uses wycheproof vectors for testing.
Create a virtual env if needed beforehand.

git clone [email protected]:pyca/cryptography.git
git clone [email protected]:C2SP/wycheproof.git

Install the needed dependencies

cd pyca-cryptography
pip3 install setuptools
pip3 install .[test]
pip3 install -e vectors
pip3 install .

Run the tests

pytest -n auto tests --wycheproof-root=../wycheproof

Description:

The tests failing are specifically in test_ecdsa_signature[1] with following specifications:

• test_ecdsa_signature[ecdsa_secp521r1_sha3_512_test.json] with the vector with id-157 [2].
  It replaces r by r + 2**585.
• test_ecdsa_signature[ecdsa_secp521r1_sha512_test.json] with the vector with id-159 [3].
  It replaces r by r + 2**585.

It is expected that OpenSSL will trow an exception, but it does not.

The expected result:
Raising <class 'cryptography.exceptions.InvalidSignature'>

The actual result:
Failed: DID NOT RAISE <class 'cryptography.exceptions.InvalidSignature'>

The tested architectures were: x86_64, aarch64, ppc64le, s390x
The failing happens only on s390x architectures.

[1] test_ecdsa_signature
[2] id-157
[3] id-159