OCSP callback failure is incorrectly reported #12735
Description
I am implementing OCSP stapling in Ruby OpenSSL extension and I ran into the following error:
test.rb:17: warning: error on stack: error:141BA041:SSL routines:tls_process_initial_server_flight:malloc failure
After investigating this, I arrived at this snippet in statem_clnt.c:
/*
* Call the ocsp status callback if needed. The |ext.ocsp.resp| and
* |ext.ocsp.resp_len| values will be set if we actually received a status
* message, or NULL and -1 otherwise
*/
if (s->ext.status_type != TLSEXT_STATUSTYPE_nothing
&& s->ctx->ext.status_cb != NULL) {
int ret = s->ctx->ext.status_cb(s, s->ctx->ext.status_arg);
if (ret == 0) {
SSLfatal(s, SSL_AD_BAD_CERTIFICATE_STATUS_RESPONSE,
SSL_F_TLS_PROCESS_INITIAL_SERVER_FLIGHT,
SSL_R_INVALID_STATUS_RESPONSE);
return 0;
}
if (ret < 0) {
SSLfatal(s, SSL_AD_INTERNAL_ERROR,
SSL_F_TLS_PROCESS_INITIAL_SERVER_FLIGHT,
ERR_R_MALLOC_FAILURE);
return 0;
}
}
The docs here state:
The callback when used on the client side should return a negative value on error; 0 if the response is not acceptable (in which case the handshake will fail) or a positive value if it is acceptable.
My OCSP callback right now returns -1 to indicate an error. It is my understanding that the quoted OpenSSL code above converts this error into "malloc failure".
The error is not a malloc failure, and has nothing to do with memory allocation at all. In fact there may be no memory allocation performed during OCSP processing. Therefore the reported error of "malloc failure" is extremely confusing.
OpenSSL should provide an error report that reflects the actual problem, such as "OCSP callback failed".