Move the existing record layer code to use the new record layer design (read side only)#18132
Move the existing record layer code to use the new record layer design (read side only)#18132mattcaswell wants to merge 69 commits intoopenssl:masterfrom
Conversation
|
KTLS will also need some more work. I plan to have a separate KTLS record method implementation - but at the moment its still mixed in with the rest of the TLS code. |
|
SSLv3 decryption code has also now been moved to the new approach. |
|
Updated with TLSv1.3 decryption code now also moved. |
13d1ad0 to
7ebe102
Compare
|
Updated to move the KTLS code to use the new read record layer. Also some significant re-org of the code so far |
|
I've now added support to move data received by a record layer object in one epoch into a record layer object in the next epoch. For example, in the case where read_ahead is set and we read a key update message followed by application data all in one go. After the key update is processed, we will be using a new record layer object so we need to move the data we already read, into the new record layer. This will become particularly important when I start implementing DTLS. |
84b3016 to
902ca53
Compare
|
The CIs are now passing for this PR except for one - which is in the external tests. Specifically the oqsprovider test: The problem seems to be that this PR amends test/ssltestlib.c - but the oqsprovider external test is patching that file. The conflict is causing the patch and hence the whole test to fail. This seems like a chicken-and-egg problem. We can't fix the oqsprovider yet to work with this PR, because it would (presumably) start failing on master. However if this PR goes in with that test failing, then its going to start failing everywhere. Probably I will have to disable the test temporarily in this PR, and then we fix it afterwards once its gone in. There's still a lot of work to do on this PR - so it will be a while before we get to that point anyway. Ping @baentsch for a heads up. Nothing to do at this stage, but we'll have to fix this once this PR goes in. |
|
ACK. The patching was introduced to allow |
Awesome! Thanks. |
|
@mattcaswell You may want to run CI again: The |
|
@baentsch Thanks! I've updated the oqs-provider submodule in this PR to the latest commit in order to pick up your latest changes. I also had to modify our oqsprovider.sh script to remove a reference to the preptests.sh script that you removed as part of your fix. Hopefully I did it right. Lets see what the CIs make of it now. |
|
Yay! CI's are green. |
c5963f2 to
b388408
Compare
|
Updated the read record layer to clean away all the remaining references to the SSL object. Just DTLS left to look at on the read record layer side of things. |
5e6fadc to
7457a65
Compare
1626c03 to
40a731d
Compare
The tls13encryption is an internal test that reaches inside libssl to test encryption/decryption of records. It needs to be amended for the new code structure so that it is testing the equivalent things as before. Reviewed-by: Hugo Landau <[email protected]> Reviewed-by: Tomas Mraz <[email protected]> (Merged from openssl#18132)
Reviewed-by: Hugo Landau <[email protected]> Reviewed-by: Tomas Mraz <[email protected]> (Merged from openssl#18132)
We would like the capability for the options/mode/read_ahead settings to be updateable after the record layer object has been instantiated. Reviewed-by: Hugo Landau <[email protected]> Reviewed-by: Tomas Mraz <[email protected]> (Merged from openssl#18132)
Reviewed-by: Hugo Landau <[email protected]> Reviewed-by: Tomas Mraz <[email protected]> (Merged from openssl#18132)
Reviewed-by: Hugo Landau <[email protected]> Reviewed-by: Tomas Mraz <[email protected]> (Merged from openssl#18132)
The read field is no longer used and can be safely removed. Reviewed-by: Hugo Landau <[email protected]> Reviewed-by: Tomas Mraz <[email protected]> (Merged from openssl#18132)
The value for epoch was being represented internally via various types: uint16_t, unsigned short, unsigned int, unsigned long We standardise on uint16_t Reviewed-by: Hugo Landau <[email protected]> Reviewed-by: Tomas Mraz <[email protected]> (Merged from openssl#18132)
This file contains design details for the old record layer and is no longer relevant for the new design. Reviewed-by: Hugo Landau <[email protected]> Reviewed-by: Tomas Mraz <[email protected]> (Merged from openssl#18132)
The 1 in DTLS1 is confusing and is removed. We also tweak the structure to always be able to track 64 packets regardless of whether we are on a 32 bit or 64 bit system. Reviewed-by: Hugo Landau <[email protected]> Reviewed-by: Tomas Mraz <[email protected]> (Merged from openssl#18132)
Some functions in the record layer were called rlayer_*, but most were called tls_*. We standardise on the latter. Reviewed-by: Hugo Landau <[email protected]> Reviewed-by: Tomas Mraz <[email protected]> (Merged from openssl#18132)
Once we free the rrl object we should NULL it to prevent a dangling ref to it. Otherwise we could get a double free. Reviewed-by: Hugo Landau <[email protected]> Reviewed-by: Tomas Mraz <[email protected]> (Merged from openssl#18132)
The SSL_AD_NO_ALERT value was defined in two places. We centralise its definition. Reviewed-by: Hugo Landau <[email protected]> Reviewed-by: Tomas Mraz <[email protected]> (Merged from openssl#18132)
Some macros were redefined in ssl3_cbc.c. We remove the redefinitions Reviewed-by: Hugo Landau <[email protected]> Reviewed-by: Tomas Mraz <[email protected]> (Merged from openssl#18132)
Some minor formatting cleanups and other minor tweaks. Reviewed-by: Hugo Landau <[email protected]> Reviewed-by: Tomas Mraz <[email protected]> (Merged from openssl#18132)
The current libssl code always ensures that the callbacks are non-null. However, the record layer itself wasn't checkthing this. We ensure it does. Reviewed-by: Hugo Landau <[email protected]> Reviewed-by: Tomas Mraz <[email protected]> (Merged from openssl#18132)
When a server responds to a second TLSv1.3 ClientHello it is required to set the legacy_record_version to 0x0303 (TLSv1.2). The client is required to ignore that field even if it is wrong. The recent changes to the read record layer in PR openssl#18132 made the record layer stricter and it was checking that the legacy_record_version was the correct value. This caused connection failures when talking to buggy servers that set the wrong legacy_record_version value. We make us more tolerant again. Fixes openssl#19051 Reviewed-by: Dmitry Belyavskiy <[email protected]> Reviewed-by: Tomas Mraz <[email protected]> (Merged from openssl#19058)
Also update the oqsprovider.sh file to not run the preptests.sh script which is no longer required Reviewed-by: Hugo Landau <[email protected]> Reviewed-by: Tomas Mraz <[email protected]> (Merged from openssl#18132) (cherry picked from commit ac837d4)
Also update the oqsprovider.sh file to not run the preptests.sh script which is no longer required Reviewed-by: Hugo Landau <[email protected]> Reviewed-by: Tomas Mraz <[email protected]> (Merged from openssl#18132) (cherry picked from commit ac837d4)
Also update the oqsprovider.sh file to not run the preptests.sh script which is no longer required Reviewed-by: Hugo Landau <[email protected]> Reviewed-by: Tomas Mraz <[email protected]> (Merged from openssl#18132) (cherry picked from commit ac837d4)
Also update the oqsprovider.sh file to not run the preptests.sh script which is no longer required Reviewed-by: Hugo Landau <[email protected]> Reviewed-by: Tomas Mraz <[email protected]> (Merged from #18132) (cherry picked from commit ac837d4)
If read_ahead is switched on, it should still work even if the data that is read cross epochs. Reviewed-by: Hugo Landau <[email protected]> Reviewed-by: Tomas Mraz <[email protected]> (Merged from openssl#18132) (cherry picked from commit f756534)
If read_ahead is switched on, it should still work even if the data that is read cross epochs. Reviewed-by: Hugo Landau <[email protected]> Reviewed-by: Tomas Mraz <[email protected]> (Merged from #18132) (cherry picked from commit f756534)
When a server responds to a second TLSv1.3 ClientHello it is required to set the legacy_record_version to 0x0303 (TLSv1.2). The client is required to ignore that field even if it is wrong. The recent changes to the read record layer in PR openssl#18132 made the record layer stricter and it was checking that the legacy_record_version was the correct value. This caused connection failures when talking to buggy servers that set the wrong legacy_record_version value. We make us more tolerant again. Fixes openssl#19051 Reviewed-by: Dmitry Belyavskiy <[email protected]> Reviewed-by: Tomas Mraz <[email protected]> (Merged from openssl#19058)
From OpenSSL commit 79a1f3e4bb62c10d9604718f6814bb8bdde4ffd6 openssl/openssl#18132
From OpenSSL commmit 11653dcd6ecbc7ff3c53f694474ece08ce4473aa openssl/openssl#18132 Also, rename the "new" function pointer to "new_record_layer" to avoid a C++ reserved name
From OpenSSL commit 34a4068cc402c38e2134a6b46d9633ad3112bfa5 openssl/openssl#18132 It doesn't yet do anything. This is a placeholder which will be filled in by susbsequent commits.
…hod.h From OpenSSL commit 0c974fc754e4b0525819ca9f6c3e124141b690ad openssl/openssl#18132
…record layer From OpenSSL commit e2d5742b1460c45bf39094ea08e4e85a8f507ea8 openssl/openssl#18132 This transfers the low level function ssl3_read_n to the new record layer. We temporarily make the read_n function a top level record layer function. Eventually, in later commits in this refactor, we will remove it as a top level function and it will just be called from read_record. But some CI cannot pass here, which will be fixed by following commits.
From OpenSSL commit 4030869d24309bfb5292e7bec41cd2b3012ba99d openssl/openssl#18132 We move the old ssl3_get_record function to conform with the new record layer design.
From OpenSSL commit 26dad42e9ca609569073463165263173ab2a27ab openssl/openssl#18132 Add a test to ensure that a connection started via DTLSv1_listen() can be completed through to handshake success. Previous DTLSv1_listen() testing only tested the function itself and did not confirm that a connection can actually be achieved using it. This is important to test some codepaths being affected by the record layer refactor.
This is very much a Work in Progress early-stage preview of what implementing the record layer design described in #17969 looks like for the existing libssl record layer implementations. Most interesting to look at at this stage is the file ssl/record/methods/tlsrecord.c and the header file based on the one in #17969 in ssl/record/recordmethod.h. I intend to break the tlsrecord.c file up into smaller files - because it is becoming too long - but that is future work.
The code as it stands compiles and the tests pass in a default build. It probably doesn't do that for some of the non-default build options. Part of the purpose is to validate the design in #17969 and this work has highlighted a number of things that need to be added to that design.
This work is far from complete. So far I have concentrated only on the "read" (i.e. decryption) side of processing. The "write" side will come later.
Most progressed is the TLSv1.0/TLSv1.1/TLSv1.2 code. This still requires some further work to remove some remaining references to SSL object internals. The intent is not to require the SSL object at all in the record layer method implementations.
The SSLv3.0/TLSv1.3 decryption crypto code has not yet been moved to the new implementation - so this side of things is currently a blend of the old and new code.
The DTLS code has had some code moved across but most of it is still using the old record layer code.
In order to get this to compile and pass the tests in its current intermediate state I've had to implement several "hacks" to get things to work. This includes adding some temporary additional function pointers into the OSSL_RECORD_LAYER structure, as well as passing the SSL object into various function calls which should not be necessary in the final end state. This will all be tidied up in time as I progress through this work.