Skip to content

TLSv1.3: Refactor ClientHello processing so that extensions get parsed earlier#1807

Closed
mattcaswell wants to merge 15 commits intoopenssl:masterfrom
mattcaswell:tls1_3-client-hello-ext-refactor
Closed

TLSv1.3: Refactor ClientHello processing so that extensions get parsed earlier#1807
mattcaswell wants to merge 15 commits intoopenssl:masterfrom
mattcaswell:tls1_3-client-hello-ext-refactor

Conversation

@mattcaswell
Copy link
Member

The existing server side ClientHello extension parsing is a bit of a mess. Historically the way we have parsed the ClientHello is to read each field in turn and then process it in some way immediately. Extensions are at the end so they get parsed and processed last.

That approach became problematic for session tickets, because we needed to know about them earlier in the ClientHello processing, i.e. when we are dealing with the sessionid field. So then we had to introduce the misleadingly named function tls_check_serverhello_tlsext_early()...which is actually for doing early processing of the ClientHello extensions!!!! This function peeked ahead at the ClientHello extensions to get the session ticket data. Later when we added EMS we had a similar problem, so tls_check_serverhello_tlsext_early() was extended to also deal with that.

For TLSv1.3 we're going to have more extensions like this. For example the supported_versions extension is for doing version negotiation. This is one of the first things we need to know about (i.e. even earlier than the processing that tls_check_serverhello_tlsext_early() currently does)...so maybe we need a tls_check_clienthello_tlsext_really_early() :-)

This doesn't seem to be the right way to go about things, so this PR refactors the whole ClientHello processing to split it into two phases. First we parse the ClientHello, where we just basically collect the data into an easily queryable format. Then subsequently we do the processing. For extensions, actually what we do is parse the extensions block up front and create an array of all the extension types mapped to the raw extension data. We can then process the extension data as and when we need it.

This PR will form the basis for my next one, which is to implement the supported_versions extension.

ssl/ssl_locl.h Outdated
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i don't see it in the style, but we conventionally have a struct name here.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

ssl/ssl_sess.c Outdated
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't need a length check on the incoming session_id?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, we did it already when we first parsed it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove the else after return's

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actually, can't you just do "return e1->type - e2->type" ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

return e1->type - e2->type isn't feasible. type is an unsigned int...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the else.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

blank line after declarations.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think "collect" is better than "parse_raw" or perhaps "parse_all" instead of "parse_raw"

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renamed to tls_collect_extensions()

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

indent to be under the opening parent?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The line is too long to do that...you'd have to put the == 0 on the next line. I thought it was more readable this way.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's not what @richsalz meant, he meant this:

+                if (s->ctx->app_verify_cookie_cb(s, clienthello.dtls_cookie,
+                    clienthello.dtls_cookie_len) == 0) {

and I know you disagree with that style...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no, i meant line it up under the s,

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

curly in wrong spot.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops. Fixed.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why parens around ciphers?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was there in the original code...but fixed.

ssl/t1_lib.c Outdated
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you should introduce a local variable to factor out all the hello->pre_proc_exts[loop] expressions inside this loop.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and why isn't this a big switch statement?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and make the loop count down:

for (loop = hello->num_extensions; loop; --loop) {

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That suggested for loop is just wrong, you would need to start at num_extensions - 1 or use loop - 1 everywhere inside the for loop. Note that starting from num_extensions - 1 will give you a problem because loop is a size_t. I also don't see why this should loop in the reverse order.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no, the loop is right. and if you have a local pointer variable like I also said, then you never need to refer to "loop" inside the body of the loop.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I used a local pointer variable - that makes a lot of sense. I haven't changed the loop to count down...I don't see why it makes any difference?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a micro optimization. On a machine instruction level, comparing with zero is a bit faster than comparing with another value. For a loop with a large span, it actually makes a difference. In this case, the difference is probably deeply marginal.

ssl/t1_lib.c Outdated
Copy link
Contributor

@richsalz richsalz Oct 31, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use a local pointer and make the loop count down:

for (loop = numexts; loop; --loop, ++ext) 
    if (exts->type == type)
       return ext;

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above. Why do you want count down? Is this some premature optimization thing?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's just a habit of mine. sure, if you want to call it premature, fine. the key point is have a local pointer variable and always use that.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure I see the point here? There are only two uses of loop here anyway...it doesn't seem to simplify things at all? Similarly I'm not sure why counting down is better.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In assembler it's usually faster to compare to 0 than to compare to some other number. In real applications you usually can't measure the difference.

Copy link
Member

@kroeckx kroeckx Oct 31, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And I think the compiler is actually smart enough to turn that into a pointer itself. And it might also do some other optimizations depending on in which direction you go. But I really don't see the point.

You could for intance also write it as:

for (ext = exts + numexts; ext != exts; ext--)
    if (ext->type == type)
        return ext;

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and also, notice that there is actually no extra variable here. look at the code i wrote.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You mean the one that always looks at the same first entry?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we just agree to disagree on this?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

well it was a comment :) but yeah, ext++ in the end of the for loop.
i'm advocating for a particular style, but i am not going to minus-one over it.

ssl/ssl_sess.c Outdated
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You forgot to update the documentation above.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

ssl/t1_lib.c Outdated
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please document return values

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like it's a largely based on tls1_check_duplicate_extensions(). Can we get rid of that function and use this new one for both the client and the server?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That was actually my thinking, but we can't do it just yet without also refactoring the ServerHello extension parsing too. That was more than I wanted to do in this PR at this stage. I've added a TODO item to remind us to come back to that and delete tls1_check_duplicate_extensions() later.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we maybe get rid of this client_version variable?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmmm...the hello structure is currently transient and disappears once we've finished the ClientHello processing. We need client_version later though, so I don't think we can delete it just yet.

@mattcaswell
Copy link
Member Author

Pushed new commits to address feedback so far. There's also a couple of commits to fix a memory leak, and to fix a problem in compression method parsing - both issues picked up by Travis.

@mattcaswell
Copy link
Member Author

make update was failing...added a new commit to try and fix it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add some short documentation.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

ssl/t1_lib.c Outdated
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please document this function.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You get the extensions now, not just the types.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm wondering if we still have a test that sends a v2 client hello (with TLS 1.0 for instance, like 0.9.8 used to do).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes we do. See 70-test_sslrecords.t

ssl/t1_lib.c Outdated
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This documentation is outdated.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated.

@kroeckx
Copy link
Member

kroeckx commented Oct 31, 2016

Looks good except for those minor points above.

@mattcaswell
Copy link
Member Author

New commit added to address latest comments from @kroeckx

ssl/t1_lib.c Outdated
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess that is NULL when not found?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes...failure means not found...do you want me to change it to say that?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@mattcaswell mattcaswell force-pushed the tls1_3-client-hello-ext-refactor branch from cefa5d2 to 8d42da3 Compare November 1, 2016 00:48
@mattcaswell
Copy link
Member Author

Updated commit pushed.

@mattcaswell
Copy link
Member Author

Any more comments on this?

Copy link
Contributor

@richsalz richsalz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i'm fine with this, but maybe you want @kroeckx to also approve?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

blank line after the return.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

delete this blank line.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

@mattcaswell mattcaswell force-pushed the tls1_3-client-hello-ext-refactor branch from 8d42da3 to 99df412 Compare November 2, 2016 14:17
@mattcaswell
Copy link
Member Author

Rebased, and a new commit added to address the style feedback comments from @richsalz.

Ping @kroeckx.

/*
* Do SSL/TLS version negotiation if applicable. For DTLS we just check
* versions are potentially compatible. Version negotiation comes later.
*/
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Per the discussion in #1771 should we consider moving SNI processing up here, or at least before ssl_get_prev_session()?
I had started on a WIP but got interrupted; there are some gnarly compatibility issues with respect to how currently the servername_callback can inspect the session (and the tlsext_hostname therein). I am not sure how well those issues can be resolved.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe, but I've not been following #1771, so not as part of this PR.

@kroeckx
Copy link
Member

kroeckx commented Nov 2, 2016

I probably wont have time until the weekend to look at anything.

@t-j-h
Copy link
Member

t-j-h commented Nov 2, 2016

tls_decrypt_ticket is a pretty icky function - magic number return codes which make little sense in how they are set out - and they should be #defines for each of the return values given they are being used in a switch statement ... I know that is existing code - but it stands out as needing addressing when reading through the changes.

@mattcaswell
Copy link
Member Author

Yeah....I know...but I'd really rather stay focused and keep that out of scope.

@mattcaswell mattcaswell force-pushed the tls1_3-client-hello-ext-refactor branch from 99df412 to 45f63e2 Compare November 4, 2016 12:27
@mattcaswell
Copy link
Member Author

Rebased to fix the various merge conflicts due to all the changes in the master branch.

@mattcaswell
Copy link
Member Author

Ping @kroeckx

ssl/ssl_locl.h Outdated

typedef struct {
unsigned int isv2;
unsigned int version;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm wondering if we should call this legacy_version

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea. Done.

/*
* Handle an SSLv2 backwards compatible ClientHello
* Note, this is only for SSLv3+ using the backward compatible format.
* Real SSLv2 is not supported, and is rejected above.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment is at least outdated.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

if (!PACKET_get_sub_packet(pkt, &clienthello.ciphersuites,
ciphersuite_len)
|| !PACKET_get_sub_packet(pkt, &session_id,
clienthello.session_id_len)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason this gets saved in a local PACKET and is not directly written to the clienthello?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I guess not. I'm not sure it makes a lot of difference, but I changed it anyway.

al = SSL_AD_DECODE_ERROR;
goto f_err;
}
clienthello.session_id_len = session_id_len;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Setting the length here just seems wrong without first actually writing the session_id in the buffer.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

memset(s->s3->client_random, 0, SSL3_RANDOM_SIZE);
challenge_len = challenge_len > sizeof(clienthello.random)
? sizeof(clienthello.random) : challenge_len;
memset(clienthello.random, 0, sizeof(clienthello.random));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess I'm going to give just the opposite comment than @richsalz, but I don't like the sizeof() in this case. The SSL3_RANDOM_SIZE here is because that's what the limit is for this protocol. But clienthello is not specific to this protocol.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually reading the SSLv3 RFC about the SSLv2 client hello, we're probably doing this wrong anyway, but I wouldn't bother fixing it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hadn't thought of that, fine with either way (although perhaps turning kurt's comment into a C comment is worthwhile)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's also unlikely that this size will change, since it's really a fixed size.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. I changed it back and added a comment.

ssl/t1_lib.c Outdated
* 10.8..10.8.3 (which don't work).
*/
static void ssl_check_for_safari(SSL *s, const PACKET *pkt)
static void ssl_check_for_safari(SSL *s, CLIENTHELLO_MSG *hello)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment is outdated

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

@kroeckx
Copy link
Member

kroeckx commented Nov 6, 2016

Except for those minor comments, it looks good.

For consistency with the TLSv1.3 spec.
Don't use a sub-packet, just load it.
…dom)

The size if fixed by the protocol and won't change even if
sizeof(clienthello.random) does.
The name and type of the argument to ssl_check_for_safari() has changed.
@mattcaswell
Copy link
Member Author

New commits pushed addressing all the latest feedback.

Copy link
Contributor

@richsalz richsalz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kroeckx says looks good, and I had my feedback addressed, so I'll say approved, and you should probably mark both of us as reviewers. And then rebase the others so we can start on those. :)

@mattcaswell
Copy link
Member Author

@richsalz,thanks...I'll await @kroeckx's ok for latest changes before putting him down as reviewer

ssl/t1_lib.c Outdated
* 1 on success
* 0 on error
*/
int tls_check_client_ems_support(SSL *s, CLIENTHELLO_MSG *hello)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason this isn't a "const CLIENTHELLO_MSG *hello" here an at other places?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I made it const. There were a small number of other places which could also be changed so I did those too. Technically there are other places where the CLIENTHELLO_MSG structure itself is not changed by a function, but something pointed to by it is. I didn't change those because I think that is just too confusing.

/* |cookie| will only be initialized for DTLS. */
PACKET session_id, cipher_suites, compression, extensions, cookie;
int is_v2_record;
PACKET session_id, compression, extensions, cookie;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think session_id is an unused variable now?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or not.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, it's still used.

@kroeckx
Copy link
Member

kroeckx commented Nov 7, 2016

Looks good.

There were a few places where they could be declared const so this commit
does that.
@mattcaswell
Copy link
Member Author

New commit added. Ping @kroeckx.

@kroeckx kroeckx added reviewed branch: master Applies to master branch labels Nov 8, 2016
@mattcaswell
Copy link
Member Author

@richsalz can you reconfirm your plus one? If so I'll put you both down as reviewers.

@richsalz
Copy link
Contributor

richsalz commented Nov 9, 2016

yes, +1

@mattcaswell
Copy link
Member Author

Merged. Thanks for the review!!

@mattcaswell mattcaswell closed this Nov 9, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

branch: master Applies to master branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants