Conversation
|
I guess also:
I also got one request out of band to more clearly document the callback's responsibility to handle SSLv2 ClientHellos more clearly. |
|
I notice this callback is blocking only and does not have a way to pause the state machine like For instance, Node.js's
I too twitch when I see such O(N) calls. :-) But I don't think it actually matters for this. There's no reason to ever write code like: The callback would ultimately only query the three or five specific extension types it cares about, so it wouldn't go quadratic. (Ours just naively walks the serialized extensions block to find the extension.)
One detail to note is the cipher list format is different (2-byte vs 3-byte cipher values), so you'll need to decide whether you convert internally or force the caller to handle both cases. (Our callback acts after V2ClientHellos are converted to SSLv3/TLS ClientHellos, so the callback never sees V2ClientHellos. Though I see OpenSSL 1.1.x uses a different strategy for V2ClientHello now.) The random is also different, but less likely to trip callers up (or even be queried): in SSLv3/TLS it has fixed size, while in a V2ClientHello, the size is variable.
Why? What would that do? |
The idea was to prevent the builtin processing of that extension from taking place, though I realize now that I may be misremembering what that field in the structure is used for. |
|
I'd like to sincerely ask you to consider doing asynchronous version of this. There is just no functionality that can substitute this, and both Node.js and bud need this to some extent. |
That sounds undesirable. Extensions aren't independent sub-protocols. They're how we add new ClientHello fields. Two different extensions may or may not interact. (Of course, we usually make them not interact because it's simpler that way, but not always. The crypto negotiation extensions used in TLS 1.3 are fairly coupled together.) If you have two extensions which do interact, having a low-level "prevent built-in processing" knob means you expose implementation details. Perhaps you happen to implement it such that feature X is keyed on extension A rather than B. It's also not clear to me why you'd want to do that anyway. I can imagine "please disable high-level feature X", but that should be done by having an API to disable said feature, not by shutting off low-level extensions via this callback. |
Yeah, I was pretty sure I didn't want it, but figured I could mention it explicitly so we consciously decide to not do it. |
|
We don't want to prevent the builtin processing. |
#2280 seems relevant |
|
Does anything even send SSLv2-compatible ClientHellos any more? |
|
All session handling is done via the initial_ctx/session_ctx (ugh, two names for the same thing), so, if you want to change how resumed sessions are handled, an API to change the initial_ctx/session_ctx needs to be proposed. Right now, sessions from different virtual servers may share the same resumption database (and ticket crypto). This may be desirable if those virtual servers are on the same machine, but not so much if certificate authentication is being used. |
ssl/statem/extensions.c
Outdated
There was a problem hiding this comment.
If this is a number of extension, not sure it needs to be a size_t; the original int is likely sufficient, given the maximum size of a ClientHello.
There was a problem hiding this comment.
This has theh type match with clienthello.pre_proc_exts_len, which is what we're going to assign it to, but I agree that int is going to be big enough to hold the values that would be seen.
There was a problem hiding this comment.
I prefer not to overload the return type to mean different things, i.e. I'd prefer a simple boolean (int) return type to indicate success or failure, with a separate size_t * to set the length. Incidentally the length should definitely be a size_t and not an int in order to match the types. I have seen far too many subtle bugs introduced by lengths swapping backwards and forwards between int and size_t. They should be consistent throughout. I don't want to have to redo all the size_t work I did in libssl just recently.
There was a problem hiding this comment.
Okay, I'll change to a separate output parameter.
doc/man3/SSL_CTX_set_early_cb.pod
Outdated
There was a problem hiding this comment.
Typically, SSL_EARLY would be the first type in a function named as such, so perhaps not include the SSL structure, and "hide" it inside the SSL_EARLY? Maybe not? But the name might need to be changed to make it consistent with other APIs.
There was a problem hiding this comment.
I thought about that, but the temptation to make SSL_EARLY just be reusing an existing type (CLIENTHELLO_MSG) was too strong, at least for the first crack. The callback is almost certainly going to need the SSL object itself, though, so it seems reasonable to have it be a top-level API parameter. I guess we could go the other way and attach the CLIENTHELLO_MSG to the SSL before invoking the callback, though.
There was a problem hiding this comment.
If the function name is SSL_EARLY_* then the SSL_EARLY argument should be the first IMO
There was a problem hiding this comment.
That's a reasonable argument. But maybe we should confirm that we're okay reusing the clienthello_msg structure as-is for the SSL_EARLY type first, since the other option is probably to define an entirely new type that contains a pointer to clienthello_msg and pointer to SSL, which would clean up these function signatures.
We recently added code to get some data on our end. I would love to lose the V2ClientHello code, but I'm not optimistic. |
|
Just to re-state my inquiry, it would be great if this could work asynchronously. |
Acknowledged. Last week was a bit busy around here... |
I am a little reluctant to add that into this changeset (which is going to be getting bigger once I add the ability for the callback to be async and tests. It seems like it could potentially be done as a follow-up change, if there is even consensus from the team that it should be done. |
|
It's really more of a question, is modifying session resumption part of the intended functionality? If yes, then an API will need to be introduced. |
kaduk
left a comment
There was a problem hiding this comment.
I'm working on plumbing support for early_cb into test/handshake_helper.c, and making the callback resumable/async.
I also have some code to make SSL_bytes_to_cipher_list() a public function (with tests), for use by early callbacks, since SSL_CIPHERs will be more useful to callback implementors than the raw bytes.
doc/man3/SSL_CTX_set_early_cb.pod
Outdated
There was a problem hiding this comment.
I thought about that, but the temptation to make SSL_EARLY just be reusing an existing type (CLIENTHELLO_MSG) was too strong, at least for the first crack. The callback is almost certainly going to need the SSL object itself, though, so it seems reasonable to have it be a top-level API parameter. I guess we could go the other way and attach the CLIENTHELLO_MSG to the SSL before invoking the callback, though.
ssl/statem/extensions.c
Outdated
There was a problem hiding this comment.
This has theh type match with clienthello.pre_proc_exts_len, which is what we're going to assign it to, but I agree that int is going to be big enough to hold the values that would be seen.
|
No async yet (still considering options), but this adds tests and exposes a helper to convert raw cipherlist octets into SSL_CIPHERs, that would probably be useful enough to applications to be worth exposing. |
mattcaswell
left a comment
There was a problem hiding this comment.
I get test failures with enable-tls1_3 (probably the use of SSL_OP_NO_TLSv1_2 instead of SSL_set_max_proto_version() noted below)
doc/man3/SSL_CTX_set_early_cb.pod
Outdated
doc/man3/SSL_CTX_set_early_cb.pod
Outdated
There was a problem hiding this comment.
If the function name is SSL_EARLY_* then the SSL_EARLY argument should be the first IMO
doc/man3/SSL_get_ciphers.pod
Outdated
There was a problem hiding this comment.
Presumably SSL_bytes_to_cipher_list() returns NULL on failure (e.g. a bad length)
There was a problem hiding this comment.
Added "NULL is returned on error" above, since it would be weird to have something other than "See DESCRIPTION" here for just one function.
There was a problem hiding this comment.
Actually, ended up changing the return convention to use an output pointer, since we want to return SCSVs separately so that the state machine can act on them.
doc/man3/SSL_get_ciphers.pod
Outdated
There was a problem hiding this comment.
Yuck. That sounds like a really bad idea. Surely we can fix this?
There was a problem hiding this comment.
We should be able to, yes, it just requires a bit of refactoring that I didn't want to do right while in the midst of things.
include/openssl/ssl.h
Outdated
There was a problem hiding this comment.
Hmmm.....we should perhaps change the name of clienthello_msg_st. That's going to get really confusing.
There was a problem hiding this comment.
Any suggestions? Alternately we can embed clienthello_msg* and SSL* into a new ssl_early_st, as mentioned above. Might be worth the extra type...
There was a problem hiding this comment.
Given that you are now storing the clienthello structure in the SSL object, do you still need the SSL_EARLY type at all? Just use the SSL object.
ssl/ssl_lib.c
Outdated
There was a problem hiding this comment.
Why is it necessary to distinguish whether it is v2 here or not? If it is v2 then we use the variable length challenge as the random. If it is longer than SSL3_RANDOM_SIZE then we just use the first SSL3_RANDOM_SIZE bytes. If it is less than SSL3_RANDOM_SIZE then we pad to the left with 0s until it is SSL3_RANDOM_SIZE in length. Either way the resulting length is SSL3_RANDOM_SIZE in all cases.
There was a problem hiding this comment.
To be sure that someone who actually is familiar with the v2 protocol (i.e., not me) looks at this logic :)
I'm happy to take your word for it; it didn't really seem worth putting the time in to understand the v2-parsing code given that it's (hopefully) on its way out.
ssl/ssl_lib.c
Outdated
There was a problem hiding this comment.
Should we call this legacy_version?
ssl/statem/extensions.c
Outdated
There was a problem hiding this comment.
I prefer not to overload the return type to mean different things, i.e. I'd prefer a simple boolean (int) return type to indicate success or failure, with a separate size_t * to set the length. Incidentally the length should definitely be a size_t and not an int in order to match the types. I have seen far too many subtle bugs introduced by lengths swapping backwards and forwards between int and size_t. They should be consistent throughout. I don't want to have to redo all the size_t work I did in libssl just recently.
ssl/ssl_lib.c
Outdated
There was a problem hiding this comment.
pre_proc_exts_len is the number of built in extensions plus the number of custom extensions that are defined. It is not the number of extensions that were sent. If we don't recognise them then they don't end up in pre_proc_exts. So this is bounded by the server config not by how many extensions we get sent. That said and index -> type lookup might not be a bad idea....or maybe expose the index????
There was a problem hiding this comment.
I'll remove the XXX comment, given this and davidben's comment.
That said and index -> type lookup might not be a bad idea....or maybe expose the index????
Do you mean type->index lookup? I had thought about doing that at some point, but demurred given that (1) there wasn't one already, possibly intentionally, and (2) I didn't want to think about how it would interact with custom extensions right then.
test/handshake_helper.c
Outdated
There was a problem hiding this comment.
Use SSL_set_max_proto_version()
|
The latest push is just the minor fixes and making SSL_bytes_to_cipher_list() "pure" -- async is still WIP. |
150ca45 to
05ea38c
Compare
|
And, landed the async functionality. |
test/handshake_helper.c
Outdated
There was a problem hiding this comment.
I guess we could add some calls to the SSL_EARLY_* accessors in here to stop coveralls from complaining.
(And please confirm that I should make the change to putting the SSL_EARLY object first, or make a new type for it to avoid the need to pass the SSL as well, or ...)
There was a problem hiding this comment.
As mentioned earlier I think you can get rid of SSL_EARLY altogether and just use SSL
ssl/ssl_lib.c
Outdated
There was a problem hiding this comment.
This arguably ought to be part of the SCSV processing, but I didn't want to change the order of processing more than needed, at first.
There was a problem hiding this comment.
Yes - it definitely should - it makes no sense here.
doc/man3/SSL_CTX_set_early_cb.pod
Outdated
There was a problem hiding this comment.
Hmm, this needs to go if we decide to keep the commits that remove this functionality.
ssl/statem/statem_srvr.c
Outdated
There was a problem hiding this comment.
Also need to revert this comment back, if we keep the commits that remove access to the raw ClientHello octets.
ssl/statem/statem_srvr.c
Outdated
There was a problem hiding this comment.
This is related to the test failure I mentioned in a previous comment.
doc/man3/SSL_CTX_set_early_cb.pod
Outdated
There was a problem hiding this comment.
Name needs updating to SSL_EARLY_get0_legacy_version()
ssl/ssl_lib.c
Outdated
There was a problem hiding this comment.
Mem leak here if one is NULL but the other is not. Perhaps just goto err?
ssl/statem/statem_srvr.c
Outdated
There was a problem hiding this comment.
Probably should just ditch the f_err label
There was a problem hiding this comment.
done, even though it adds more churn.
ssl/ssl_lib.c
Outdated
There was a problem hiding this comment.
Yes - it definitely should - it makes no sense here.
test/cipherbytes_test.c
Outdated
There was a problem hiding this comment.
Perhaps this should be !SSL_bytes_to_cipher_list() || sk != NULL
There was a problem hiding this comment.
SSL_bytes_to_cipher_list() is expected to fail when a zero-length byte array is provided.
So it's better as SSL_bytes_to_cipher_list() || sk != NULL, I think. (Later modified to also check the SCSVs.)
test/cipherbytes_test.c
Outdated
There was a problem hiding this comment.
Add a test with SCSV ciphersuites?
There was a problem hiding this comment.
I added the two known SCSVs to test_v3().
test/recipes/80-test_cipherbytes.t
Outdated
include/openssl/ssl.h
Outdated
There was a problem hiding this comment.
Given that you are now storing the clienthello structure in the SSL object, do you still need the SSL_EARLY type at all? Just use the SSL object.
test/handshake_helper.c
Outdated
There was a problem hiding this comment.
As mentioned earlier I think you can get rid of SSL_EARLY altogether and just use SSL
test/ssl-tests/05-sni.conf.in
Outdated
There was a problem hiding this comment.
This test fails with the no-tls1_1 config option
There was a problem hiding this comment.
Hmm, and we don't have skip functionality at this fine-grained of a level (right?). Should I move this to a new 05-early.conf.in so that 05-sni.conf.in can be run always?
There was a problem hiding this comment.
Take a look at 20-cert-select.conf.in for an example of how to do this type of thing. Some of the other ".in" files do similar things.
There was a problem hiding this comment.
Ah, thanks. I got there ... eventually ... I should probably call it quits for the night if I'm making this many brainos.
|
Decided to not push -f for this round of changes; please let me know if/when I should rebase/squash. |
|
Hmm, just a doc-nits issue from travis, apparently. |
|
you must fix all doc nits you know :) |
levitte
left a comment
There was a problem hiding this comment.
This is for some of the doc-nits issues.
doc/man3/SSL_CTX_set_early_cb.pod
Outdated
|
|
||
| =head1 SYNOPSIS | ||
|
|
||
| typedef int (*SSL_early_cb_fn) (SSL *s, int *al, void *arg); |
There was a problem hiding this comment.
I think that space between ) and ( is the cause of one of the doc-nits issues...
doc/man3/SSL_CTX_set_early_cb.pod
Outdated
|
|
||
| =head1 NAME | ||
|
|
||
| SSL_CTX_set_early_cb, SSL_early_isv2, SSL_early_get0_legacy_version, SSL_early_get0_random, SSL_early_get0_session, SSL_early_get0_ciphers, SSL_early_get0_compression_methods, SSL_early_get0_ext - callback functions for early server-side ClientHello processing |
There was a problem hiding this comment.
SSL_early_get0_session should be SSL_early_get0_session_id...
[skip ci]
|
Thanks for the pointer about the space in ') (', @levitte , that would have been rather nonobvious from the given error message. |
|
Who should merge it? |
|
Per standard practice, it will usually be one of the reviewers. It must be someone on the team of course :) |
|
I can do it. |
Modify the API of tls_collect_extensions() to be able to output the number of extensions that are known (i.e., the length of its 'res' output). This number can never be zero on a successful return due to the builtin extensions list, but use a separate output variable so as to not overload the return value semantics. Having this value easily available will give consumers a way to avoid repeating the calculation. Reviewed-by: Matt Caswell <[email protected]> Reviewed-by: Richard Levitte <[email protected]> (Merged from #2279)
Keep track of the length of the pre_proc_exts array. Reviewed-by: Matt Caswell <[email protected]> Reviewed-by: Richard Levitte <[email protected]> (Merged from #2279)
We'll be adding a field of this type to struct ssl_st in a subsequent commit, and need the type definition to be in scope already. Also move up the RAW_EXTENSION definition that it depends on. Reviewed-by: Matt Caswell <[email protected]> Reviewed-by: Richard Levitte <[email protected]> (Merged from #2279)
Just as we have a table of ssl3_ciphers, add a table of ssl3_scsvs, to contain SSL_CIPHER objects for these non-valid ciphers. This will allow for unified handling of such indicators, especially as we are preparing to pass them around between functions. Since the 'valid' field is not set for the SCSVs, they should not be used for anything requiring a cryptographic cipher (as opposed to something being stuck in a cipher-shaped hole in the TLS wire protocol). Reviewed-by: Matt Caswell <[email protected]> Reviewed-by: Richard Levitte <[email protected]> (Merged from #2279)
Now that we have made SCSVs into more of a first-class object, provide a way for the bytes-to-SSL_CIPHER conversion to actually return them. Add a flag 'all' to ssl_get_cipher_by_char to indicate that we want all the known ciphers, not just the ones valid for encryption. This will, in practice, let the caller retrieve the SCSVs. Reviewed-by: Matt Caswell <[email protected]> Reviewed-by: Richard Levitte <[email protected]> (Merged from #2279)
Move ssl_bytes_to_cipher_list() to ssl_lib.c and create a public wrapper around it. This lets application early callbacks easily get SSL_CIPHER objects from the raw ciphers bytes without having to reimplement the parsing code. In particular, they do not need to know the details of the sslv2 format ClientHello's ciphersuite specifications. Document the new public function, including the arguably buggy behavior of modifying the supplied SSL object. On the face of it, such a function should be able to be pure, just a direct translation of wire octets to internal data structures. Reviewed-by: Matt Caswell <[email protected]> Reviewed-by: Richard Levitte <[email protected]> (Merged from #2279)
Split off the portions that mutate the SSL object into a separate function that the state machine calls, so that the public API can be a pure function. (It still needs the SSL parameter in order to determine what SSL_METHOD's get_cipher_by_char() routine to use, though.) Instead of returning the stack of ciphers (functionality that was not used internally), require using the output parameter, and add a separate output parameter for the SCSVs contained in the supplied octets, if desired. This lets us move to the standard return value convention. Also make both output stacks optional parameters. Reviewed-by: Matt Caswell <[email protected]> Reviewed-by: Richard Levitte <[email protected]> (Merged from #2279)
Reviewed-by: Matt Caswell <[email protected]> Reviewed-by: Richard Levitte <[email protected]> (Merged from #2279)
Add the new enum value and case statements as appropriate. Reviewed-by: Matt Caswell <[email protected]> Reviewed-by: Richard Levitte <[email protected]> (Merged from #2279)
Provide a callback interface that gives the application the ability to adjust the nascent SSL object at the earliest stage of ClientHello processing, immediately after extensions have been collected but before they have been processed. This is akin to BoringSSL's "select_certificate_cb" (though it is not API compatible), and as the name indicates, one major use is to examine the supplied server name indication and select what certificate to present to the client. However, it can also be used to make more sweeping configuration changes to the SSL object according to the selected server identity and configuration. That may include adjusting the permitted TLS versions, swapping out the SSL_CTX object (as is traditionally done in a tlsext_servername_callback), changing the server's cipher list, and more. We also wish to allow an early callback to indicate that it needs to perform additional work asynchronously and resume processing later. To that effect, refactor the second half of tls_process_client_hello() into a subroutine to be called at the post-processing stage (including the early callback itself), to allow the callback to result in remaining in the same work stage for a later call to succeed. This requires allocating for and storing the CLIENTHELLO_MSG in the SSL object to be preserved across such calls, but the storage is reclaimed after ClientHello processing finishes. Information about the CliehtHello is available to the callback by means of accessor functions that can only be used from the early callback. This allows extensions to make use of the existing internal parsing machinery without exposing structure internals (e.g., of PACKET), so that applications do not have to write fragile parsing code. Applications are encouraged to utilize an early callback and not use a servername_callback, in order to avoid unexpected behavior that occurs due to the relative order of processing between things like session resumption and the historical servername callback. Also tidy up nearby style by removing unnecessary braces around one-line conditional bodies. Reviewed-by: Matt Caswell <[email protected]> Reviewed-by: Richard Levitte <[email protected]> (Merged from #2279)
Plumb things through in the same place as the SNI callback, since we recommend that the early callback replace (and supplement) the SNI callback, and add a few test cases. Reviewed-by: Matt Caswell <[email protected]> Reviewed-by: Richard Levitte <[email protected]> (Merged from #2279)
Reviewed-by: Matt Caswell <[email protected]> Reviewed-by: Richard Levitte <[email protected]> (Merged from #2279)
create_ssl_connection() prints out the results if SSL_accept() and/or SSL_connect() fail, but was reusing the client return value when printing about SSL_accept() failures. Reviewed-by: Matt Caswell <[email protected]> Reviewed-by: Richard Levitte <[email protected]> (Merged from #2279)
Certain callback APIs allow the callback to request async processing by trickling a particular error value up the stack to the application as an error return from the handshake function. In those cases, SSL_want() returns a code specific to the type of async processing needed. The create_ssl_connection() helper function for the tests is very helpful for several things, including creating API tests. However, it does not currently let us test the async processing functionality of these callback interfaces, because the special SSL error codes are treated as generic errors and the helper continues to loop until it reaches its maximum iteration count. Add a new parameter, 'want', that indicates an expected/desired special SSL error code, so that the helper will terminate when either side reports that error, giving control back to the calling function and allowing the test to proceed. Reviewed-by: Matt Caswell <[email protected]> Reviewed-by: Richard Levitte <[email protected]> (Merged from #2279)
Make sure that we can stop handshake processing and resume it later. Also check that the cipher list and compression methods are sane. Unfortunately, we don't have the client-side APIs needed to force a specific (known) session ID to be sent in the ClientHello, so that accessor cannot be tested here. Reviewed-by: Matt Caswell <[email protected]> Reviewed-by: Richard Levitte <[email protected]> (Merged from #2279)
|
(I did some editorial work... the last commits were obviously fixups, so I squashed them with the appropriate original commits) |
Checklist
Description of change
Implement an early callback similar to BoringSSL's select_certificate_cb, allowing for SNI processing and server certificate selection prior to session resumption and version negotiation.
The chosen API (with opaque structure and accessors, rather than public structure) differs from BoringSSL's, in order to remain consistent with most other OpenSSL APIs, and retain the trend of opaque data structures that facilitate ABI stability. This also should make it easier to backport the API to 1.1.0 if desired.
The WIP label will remain until tests are added, but other interesting questions include:
should the raw clienthello bytes be exposed at all?
Is now the time to implement a TLS extension type to index lookup, to avoid having to loop over the preprocessed extensions?
What level of support should SSLv2 clienthellos get from this API? (I am not much of an expert on the SSLv2 wire protocol, so the current support is likely lacking.)
Should we expose the alert to send to the callback, or even allow the callback to terminate the connection at all?
cc @mattcaswell