With the 1.0.2j/1.1.0b openssl built in subdirectories using "./config; make" I can use the roundtrip script in the attached zipfile to encrypt a file with a public key derived from the included private key, then decrypt using the private and finally compare the output to the plaintext (in raw mode (-r) at least).
If the openssl binary used is openssl 1.0.1t (say) all is well. However the machines with cpuinfo attached in the zipfile as E7450.cpuinfo and PavillionGaming.cpuinfo show errors for versions 1.0.2j & 1.1.0b.:
$ LD_LIBRARY_PATH=$(pwd)/openssl-1.1.0b ./roundtrip rsa.key openssl-1.1.0b/apps/openssl RSA operation error 140193654368000:error:0407109F:rsa routines:RSA_padding_check_PKCS1_type_2:pkcs decoding error:crypto/rsa/rsa_pk1.c:243: 140193654368000:error:04065072:rsa routines:rsa_ossl_private_decrypt:padding check failed:crypto/rsa/rsa_ossl.c:477: cmp: EOF on rsa.output
and $ OPENSSL_CONF=openssl-1.0.2j/apps/openssl.cnf ./roundtrip rsa.key openssl-1.0.2j/apps/openssl RSA operation error 140062524487320:error:0407109F:rsa routines:RSA_padding_check_PKCS1_type_2:pkcs decoding error:rsa_pk1.c:273: 140062524487320:error:04065072:rsa routines:RSA_EAY_PRIVATE_DECRYPT:padding check failed:rsa_eay.c:602: cmp: EOF on rsa.output
Using the machine described by E5-2670v2.cpuinfo results in the same invocations succeeding.
If I replace the file crypto/bn/asm/x86_64-mont.pl with the same file taken from the openssl 1.0.1 release then the problem is solved. By using different openssl executables for the encrypt/decrypt step I was able to see that it is the encryption step that is wrong in the affected version/machines.
Using the -r switch to roundtrip causing "raw" encryption to be used rather than PKCS#1, with the result that the output file does not match the plaintext: i.e. it's not a problem with the padding code.
Not every RSA key causes the failure, however I with different RSA keys I was seeing errors approximately 50% of the time.
rsa_encrypt.zip
With the 1.0.2j/1.1.0b openssl built in subdirectories using "./config; make" I can use the roundtrip script in the attached zipfile to encrypt a file with a public key derived from the included private key, then decrypt using the private and finally compare the output to the plaintext (in raw mode (-r) at least).
If the openssl binary used is openssl 1.0.1t (say) all is well. However the machines with cpuinfo attached in the zipfile as E7450.cpuinfo and PavillionGaming.cpuinfo show errors for versions 1.0.2j & 1.1.0b.:
$ LD_LIBRARY_PATH=$(pwd)/openssl-1.1.0b ./roundtrip rsa.key openssl-1.1.0b/apps/openssl RSA operation error 140193654368000:error:0407109F:rsa routines:RSA_padding_check_PKCS1_type_2:pkcs decoding error:crypto/rsa/rsa_pk1.c:243: 140193654368000:error:04065072:rsa routines:rsa_ossl_private_decrypt:padding check failed:crypto/rsa/rsa_ossl.c:477: cmp: EOF on rsa.outputand
$ OPENSSL_CONF=openssl-1.0.2j/apps/openssl.cnf ./roundtrip rsa.key openssl-1.0.2j/apps/openssl RSA operation error 140062524487320:error:0407109F:rsa routines:RSA_padding_check_PKCS1_type_2:pkcs decoding error:rsa_pk1.c:273: 140062524487320:error:04065072:rsa routines:RSA_EAY_PRIVATE_DECRYPT:padding check failed:rsa_eay.c:602: cmp: EOF on rsa.outputUsing the machine described by E5-2670v2.cpuinfo results in the same invocations succeeding.
If I replace the file crypto/bn/asm/x86_64-mont.pl with the same file taken from the openssl 1.0.1 release then the problem is solved. By using different openssl executables for the encrypt/decrypt step I was able to see that it is the encryption step that is wrong in the affected version/machines.
Using the -r switch to roundtrip causing "raw" encryption to be used rather than PKCS#1, with the result that the output file does not match the plaintext: i.e. it's not a problem with the padding code.
Not every RSA key causes the failure, however I with different RSA keys I was seeing errors approximately 50% of the time.
rsa_encrypt.zip