{"title":"Lennart Grahl's Blog","link":[{"@attributes":{"href":"https:\/\/lgrahl.de\/","rel":"alternate"}},{"@attributes":{"href":"https:\/\/lgrahl.de\/feeds\/all.atom.xml","rel":"self"}}],"id":"https:\/\/lgrahl.de\/","updated":"2017-11-03T23:55:00+01:00","subtitle":"Software Engineer","entry":[{"title":"Large Data Channel Messages for Firefox","link":{"@attributes":{"href":"https:\/\/lgrahl.de\/articles\/large-dc-messages-firefox.html","rel":"alternate"}},"published":"2017-11-03T23:55:00+01:00","updated":"2017-11-03T23:55:00+01:00","author":{"name":"Lennart Grahl"},"id":"tag:lgrahl.de,2017-11-03:\/articles\/large-dc-messages-firefox.html","summary":"

At the moment, you have to be careful about the maximum message size you can send over a WebRTC data channel. Especially if the data channel has been established between Firefox and Chrome. With help from Randell Jesup, Nils Ohlmeier and plenty of other helpful guys from the #media<\/em> Mozilla IRC channel, I've added EOR<\/em> handling and determination of the maximum message size to Firefox data channels. In this article, I'll explain the technical details and the benefits of this addition.<\/p>","content":"

At the moment, you have to be careful about the maximum message size you can\nsend over a WebRTC data channel. Especially if the data channel has been\nestablished between Firefox and Chrome. With help from Randell Jesup, Nils\nOhlmeier and plenty of other helpful guys from the #media<\/em> Mozilla IRC\nchannel, I've added EOR<\/em> handling to Firefox data channels (tracked in\nbug 979417<\/a>) which will land in Firefox 57. So, what does this\nactually mean and what are the benefits?<\/p>\n

EOR... what?<\/h2>\n

SCTP<\/a> is the transport protocol used for exchanging WebRTC data channel\nmessages. Its API provides an EOR<\/em> flag which stands for end of record<\/em> and\ntells us whether a message is partial or complete. Thus, every implementation\nneeds to pay attention to this flag or message integrity will be violated.\nWhich sadly is the case for all popular data channel implementations at the\nmoment and results in harsh size limitations for data channel messages. For\nmore technical information on this and how one can work around this, take a\nlook at my previous blog posting<\/a>. But do\nnot worry, determining that size limitation will be a lot easier, soon.<\/p>\n

Unfortunately, none of the browsers today support the\nSCTP ndata extension<\/a> yet, so a single data channel message will\nmonopolise the whole SCTP assocation. This phenomenom is also known as\nhead-of-line blocking<\/em> and makes sending large messages quite painful if other\nchannels need to keep exchanging messages frequently, for example in a\nreal-time chat with file-sending capability. All messages exchanged in the chat\nwould arrive after the lengthy process of sending a gigantic file has been\ncompleted.<\/p>\n

While the ndata extension is currently in the specification process, it's\nmaking good progress and there's no excuse for holding back proper EOR handling\nanymore. Personally, I believe this should have never been an excuse for not\nhandling EOR as it has so many undesired side effects. Since I've gained\nexperience in writing a standalone data channel implementation with proper EOR\nhandling (called RAWRTC<\/a>), it was a logical choice to apply this to at\nleast one of the browsers.<\/p>\n

Exchanging Large Messages<\/h2>\n

With EOR handling, one major benefit is of course being able to send and\nreceive large messages. But hold on a second... sending large messages from\nFirefox to Firefox was possible before, wasn't it? Yep, that's true, with help\nof the deprecated PPID-based fragmentation\/reassembly mechanism. However, no\none else implemented this. A detection algorithm has been added, so Firefox\nwill fall back to use PPID-based fragmentation\/reassembly when needed: In case\nthe maximum message size attribute in the remote description's SDP is missing\nand 256 incoming streams have been negotiated for the SCTP association, it is\nassumed that the other peer is an older Firefox browser. It is not the cleanest\nof solutions but it should be adequate and will be removed once Firefox ESR has\nEOR support as well.<\/p>\n

Thanks to EOR handling, Firefox will support receiving up to 1 GiB messages.\nWhy is there a limitation? Well, theoretically we can support much more but the\nlimiting factor is available memory and the fact that the RTCDataChannel<\/code> API\nof WebRTC is too high-level<\/em>, meaning it's not possible to send or receive a\nmessage in chunks. Therefore, when you want to send a large file as a whole,\nthere's an enormous backpressure<\/a> due to the fact that the\nfile's data needs to be in memory when calling the send<\/code> method. Changing this\nrequires changes to the W3C WebRTC specification.<\/p>\n

Nevertheless, this size limitation may be increased in the future. But having a\nsize limit requires knowledge about this limitation. More precisely, that value\nmust be available before an application even tries to send a message or it\nwould be a guessing game...<\/p>\n

Maximum Message Size Indication<\/h2>\n

In the W3C WebRTC spec, there's an RTCSctpTransport<\/code> object which has a\nfield called maxMessageSize<\/code><\/a>. It maps to the attribute\nfrom the SDP and is exactly what a user application needs to look for. So, the\nfollowing example should work, right?<\/p>\n

<\/span>let<\/span> <\/span>pc<\/span> <\/span>=<\/span> <\/span>new<\/span> <\/span>RTCPeerConnection<\/span>(...);<\/span>\nlet<\/span> <\/span>dc<\/span> <\/span>=<\/span> <\/span>pc<\/span>.<\/span>createDataChannel<\/span>(<\/span>'test'<\/span>,<\/span> <\/span>{<\/span>\n    <\/span>negotiated<\/span>:<\/span> <\/span>true<\/span>,<\/span>\n    <\/span>id<\/span>:<\/span> <\/span>1<\/span>\n});<\/span>\n\ndc<\/span>.<\/span>onopen<\/span> <\/span>=<\/span> <\/span>()<\/span> <\/span>=><\/span> <\/span>{<\/span>\n    <\/span>let<\/span> <\/span>maximumMessageSize<\/span> <\/span>=<\/span> <\/span>pc<\/span>.<\/span>sctp<\/span>.<\/span>maxMessageSize<\/span>;<\/span>\n    <\/span>dc<\/span>.<\/span>send<\/span>(<\/span>new<\/span> <\/span>Uint8Array<\/span>(<\/span>maximumMessageSize<\/span>));<\/span>\n};<\/span>\n\n\/\/ Create and exchange offer\/answer here.<\/span>\n\/\/ ...<\/span>\n<\/code><\/pre><\/div>\n\n
<\/span>let<\/span> <\/span>pc<\/span> <\/span>=<\/span> <\/span>new<\/span> <\/span>RTCPeerConnection<\/span>(...);<\/span>\nlet<\/span> <\/span>maximumMessageSize<\/span>;<\/span>\nlet<\/span> <\/span>dc<\/span> <\/span>=<\/span> <\/span>pc<\/span>.<\/span>createDataChannel<\/span>(<\/span>'test'<\/span>,<\/span> <\/span>{<\/span>\n    <\/span>negotiated<\/span>:<\/span> <\/span>true<\/span>,<\/span>\n    <\/span>id<\/span>:<\/span> <\/span>1<\/span>\n});<\/span>\n\ndc<\/span>.<\/span>onopen<\/span> <\/span>=<\/span> <\/span>()<\/span> <\/span>=><\/span> <\/span>{<\/span>\n    <\/span>dc<\/span>.<\/span>send<\/span>(<\/span>new<\/span> <\/span>Uint8Array<\/span>(<\/span>maximumMessageSize<\/span>));<\/span>\n};<\/span>\n\nmySignaling<\/span>.<\/span>onRemoteDescription<\/span> <\/span>=<\/span> <\/span>(<\/span>description<\/span>)<\/span> <\/span>=><\/span> <\/span>{<\/span>\n    <\/span>\/\/ Parse a=max-message-size<\/span>\n    <\/span>maximumMessageSize<\/span> <\/span>=<\/span> <\/span>65535<\/span>;<\/span>\n    <\/span>const<\/span> <\/span>match<\/span> <\/span>=<\/span> <\/span>description<\/span>.<\/span>sdp<\/span>.<\/span>match<\/span>(<\/span>\/a=max-message-size:\\s*(\\d+)\/g<\/span>);<\/span>\n    <\/span>if<\/span> <\/span>(<\/span>match<\/span> <\/span>!==<\/span> <\/span>null<\/span> <\/span>&&<\/span> <\/span>match<\/span>.<\/span>length<\/span> <\/span>>=<\/span> <\/span>2<\/span>)<\/span> <\/span>{<\/span>\n        <\/span>maximumMessageSize<\/span> <\/span>=<\/span> <\/span>parseInt<\/span>(<\/span>match<\/span>[<\/span>1<\/span>]);<\/span>\n    <\/span>}<\/span>\n\n    <\/span>\/\/ Set remote description<\/span>\n    <\/span>...<\/span>\n};<\/span>\n\n\/\/ Create and exchange offer\/answer here.<\/span>\n\/\/ ...<\/span>\n<\/code><\/pre><\/div>\n\n
Now we know how much data we can send. In case an application sends a message\nthat exceeds this limitation, the send<\/code> method will raise an error as\ndescribed in the next section.<\/p>\n
Explicit Errors<\/h2>\n
These changes will not only make it possible to send and receive large data\nchannel messages, but will also let you know if the other peer is capable of\nreceiving a message that has been passed to the send<\/code> method in form of a\nTypeError<\/code> exception as specified<\/a> (formerly, the data\nchannel has been closed implicitly in case of such an error). This ensures\nbackwards compatibility to implementations that do not handle EOR at the moment\nas all implementations are capable of handling 64 KiB sized messages.<\/p>\n
Other Browsers<\/h2>\n
Now, it's your turn Chromium. <\/span> Let's\nraise the limit - at least well beyond 64 KiB.<\/p>\n
\n
An edited version of this article has also been released on\nMozilla's WebRTC blog<\/a>.<\/p>","category":[{"@attributes":{"term":"WebRTC"}},{"@attributes":{"term":"webrtc"}},{"@attributes":{"term":"datachannel"}},{"@attributes":{"term":"sctp"}},{"@attributes":{"term":"eor"}}]},{"title":"Demystifying WebRTC's Data Channel Message Size Limitations","link":{"@attributes":{"href":"https:\/\/lgrahl.de\/articles\/demystifying-webrtc-dc-size-limit.html","rel":"alternate"}},"published":"2016-12-01T15:10:00+01:00","updated":"2017-06-30T18:50:00+02:00","author":{"name":"Lennart Grahl"},"id":"tag:lgrahl.de,2016-12-01:\/articles\/demystifying-webrtc-dc-size-limit.html","summary":"
WebRTC's data channels have a message size limitation. But where does this limitation come from? I've read through most of Firefox's and Chromium's data channel code. In this article, I'll show you my results and give some recommendation what you can do to reliably work around this issue.<\/p>","content":"
Many application developers who utilise WebRTC data channel know that there is\na message size limitation. Some<\/a> have experimentally\ntried to find the maximum message size. However, it seems that noone has ever\nlooked deep enough to find out why that message size limitation exists in the\nfirst place and why sending messages from Firefox to Chromium in particular\ndictates a bizarrely low message size. So, I've read through most of Firefox's\nand Chromium's SCTP data channel implementations. And here are my results.<\/p>\n
Chromium<\/h2>\n
Chromium unintentionally fragments messages. Rather than implementing\nfragmentation\/reassembly on the (now deprecated) SCTP PPID (payload protocol\nidentifier) level<\/a> Chromium simply passes the data as is to the\nusrsctp<\/a> (the SCTP implementation used by both Chromium and Firefox)\nstack. Now, behaviour depends on usrsctp: When a message is greater than 256\nKiB (which is usrsctp's default maximum buffer size), Chromium simply closes\nthe data channel, because usrsctp returns EMSGSIZE<\/code> when calling\nusrsctp_sendv<\/code> which the Chromium code is not able to handle. It simply closes\nthe data channel in that case. Messages up to 256 KiB are split up into very\nsmall packets by usrsctp. These packets are then encrypted and sent on the DTLS\nlayer to the recipient.<\/p>\n
On the receiver side, usrsctp buffers the small chunks until the buffer reaches\nthe partial delivery point (which is ~64 KiB at default). It then passes the\nstill incomplete message to the browser. Chromium is supposed to buffer the\nmessage and wait for the EOR (end of record) flag which indicates that a\nmessage is complete. However, Chromium ignores that flag and simply delivers\nthe incomplete message to the user application, violating the message-oriented\nprinciple.<\/p>\n
Furthermore, Chromium does not check the return code of usrsctp_sendv<\/code> for\nEAGAIN<\/code> which means that sending a data channel message via Chromium can\npotentially lead to arbitrarily dropped packets when the internal buffer of\nusrsctp is full. Theoretically this means that you do not know how much data\nyou can reliably send. However, I've not encountered this particular type of\nerror so far, so there's probably enough time for usrsctp between send calls to\ndeliver buffered data and clear the buffers. Still, be aware that this might be\nan issue.<\/p>\n
Firefox<\/h2>\n
Current Firefox versions have two modes for sending data:<\/p>\n
    \n
  1. \n
    If the data channel is reliable and<\/strong> ordered, Firefox uses a deprecated\nfragmentation\/reassembly on SCTP PPID level<\/a>. In this mode, Firefox\ncan send an unlimited amount of data to another Firefox browser (or someone who\nhas also implemented that deprecated fragmentation and reassembly mechanism).\nThis has an unintended side-effect for Chromium browsers explained in the\nfollowing section.<\/p>\n<\/li>\n
  2. \n
    If the data channel is not reliable or<\/strong> not ordered, Firefox behaves the\nsame as Chromium, including the EOR bug. However, Firefox does not close the\ndata channel on EMSGSIZE<\/code> and also handles EAGAIN<\/code>. Older Firefox versions\nsimply discard messages on EMSGSIZE<\/code> without raising an error event at all\n(ouch!). Read the Chromium<\/em> section above for details on the EOR bug.<\/p>\n<\/li>\n<\/ol>\n
    Demystifying the 16 KiB Firefox to Chromium Message Size Limit<\/h2>\n
    First of all, this only applies to ordered and<\/strong> reliable channels! For\nunordered or<\/strong> unreliable channels, the limit remains 64 KiB.<\/p>\n
    To demistify why Firefox cannot send more than 16 KiB to Chromium as stated\nhere<\/a>, you'll have to understand that Chromium does not\nimplement the deprecated fragmentation\/reassembly implemented by Firefox for\nordered and<\/strong> reliable channels.\nAs Firefox fragments the messages into 16 KiB chunks, it sends the first chunks\nwith a different SCTP PPID (52<\/code> for Binary Partial<\/em> or 54<\/code> for String\nPartial<\/em>) than the last one (53<\/code> for Binary<\/em> or 51<\/code> for String<\/em>).\nHowever, Chromium uses the same handling mechanism for packets with PPID 52<\/code>\nand 54<\/code> or 53<\/code> and 51<\/code>, so it receives the packets one by one as sent by\nFirefox. Thus, the application gets 16 KiB messages because that's the chunk\nsize Firefox uses to fragment the message. Chromium just does not reassemble it.<\/p>\n
    Can We Fix It?<\/h2>\n
    Well, yes, but we'll have to be patient. The following subsections indicate\nwhat browser vendors have to do to get rid of the size limitation or at least\nincrease the limit.<\/p>\n
    > 16 KiB<\/h3>\n
    Either Firefox needs to drop support for the already deprecated\nfragmentation\/reassembly at SCTP PPID level or Chromium needs to implement it.\nThis is particularly annoying because it's not as easy to fix as the latter one\nbut remains the lowest common denominator for Firefox and Chromium\ninteroperation until it has been resolved.<\/p>\n
    > 64 KiB<\/h3>\n
    Both Firefox and Chromium have to obey the EOR flag to receive messages as they\nwere sent. This is an easy fix.<\/p>\n
    > 256 KiB<\/h3>\n
    While the browsers could simply increase the buffer of usrsctp, this is not a\ngood idea. A better solution would be to use the EOR flag when passing messages\nto usrsctp via a call to usrsctp_sendv<\/code>. Again, should be an easy fix.<\/p>\n
    But there's another problem and this is most probably the reason why it hasn't\nbeen done yet: A very big data channel message would lead to head-of-line\nblocking for messages sent by other data channels. To solve that problem, the\nSCTP ndata specification<\/a> has been created. Now, browser vendors\nwait until it is stable before it's being activated in the browsers. This will\ntake a bit of time.<\/p>\n
    Contribution<\/h2>\n
    Thanks to Michael T\u00fcxen for answering my questions regarding usrsctp.<\/p>\n
    TL;DR, Gimme Instructions!<\/h2>\n
    My tests and the code of both Chromium and Firefox indicate that it's safe to\nsend 64 KiB of data between current versions of Firefox and Chromium unless<\/strong>\nyou're sending from Firefox to Chromium on an ordered and<\/strong> reliable data\nchannel where 16 KiB is the maximum message size. You'll have to fragment and\nreassemble on application level.<\/p>\n
    For full support for both reliable and unreliable, ordered and unordered\nchannels, use the SaltyRTC's chunking specification<\/a>\nwhich has been implemented for JavaScript<\/a> and\nJava<\/a>. Set the chunk size to 16384<\/code> if you want to be safe.\nOtherwise, use a chunk size of 65536<\/code> in case you're sure that there'll never\nbe a Firefox browser talking to a Chromium browser. You could also implement a\ndetection mechanism for that. If you're doing that, the maximum message size\nneeds to be distributed to the other peer because the problem occurs on the\nreceiver's side.<\/p>\n
    And while you're at it, consider using SaltyRTC for seriously secure\nsignalling<\/a> to protect your WebRTC\npeer-to-peer connection. <\/span><\/p>\n
    Update - Thu, 30 Jul 2017<\/h6>\n
    The bug has been filed as bug 979417<\/a> for Firefox and as\nissue 7774<\/a> for Chromium.<\/p>\n
    Update - Fri, 03 Nov 2017<\/h6>\n
    A bugfix for this issue<\/a> has been shipped as part\nof Firefox 57. From that version on, messages of 64 KiB can be exchanged safely\nbetween Chromium and Firefox.<\/p>\n
    Update - Sun, 19 Aug 2018<\/h6>\n
    Jeroen de Borst picked up the issue that has been\nfiled for Chromium<\/a> and raised the receive limit to 256 KiB.\nWhile this is hardly pushing the limit of whats possible, it is a considerable\nimprovement. Unfortunately the violation of message integrity has not been\naddressed which means that messages larger than 256 KiB will still be delivered\nin chunks. Furthermore, Chromium does not signal the maximum message size which\nmeans that the remote peer is unable to figure out the effective maximum\nmessage size it may use to send data.<\/p>\n
    RAWRTC<\/h2>\n
    There's a standalone WebRTC data channel implementation called RAWRTC<\/strong>\nwritten by me in C that you can use to send and receive messages of arbitrary\nsize. Browser interoperation works great as well. With the changes to Firefox\n57<\/a> it's possible to send large application messages\nfrom and to the Firefox browser as well.<\/p>\n
    Check RAWRTC out on GitHub!<\/a> <\/span><\/p>","category":[{"@attributes":{"term":"WebRTC"}},{"@attributes":{"term":"webrtc"}},{"@attributes":{"term":"datachannel"}},{"@attributes":{"term":"chunking"}},{"@attributes":{"term":"saltyrtc"}}]},{"title":"Bachelor Thesis - SaltyRTC - Seriously Secure WebRTC","link":{"@attributes":{"href":"https:\/\/lgrahl.de\/articles\/ba-thesis-saltyrtc.html","rel":"alternate"}},"published":"2015-10-19T10:00:00+02:00","updated":"2017-08-06T19:20:00+02:00","author":{"name":"Lennart Grahl"},"id":"tag:lgrahl.de,2015-10-19:\/articles\/ba-thesis-saltyrtc.html","summary":"
    In my bachelor thesis, I'm going through the WebRTC protocol, in particular its weakest link - the Signalling Channel<\/em>. This work presents a solution that uses the Networking and Cryptography library<\/em> (NaCl) to provide a secure implementation of the Signalling Channel<\/em> called SaltyRTC<\/em>.<\/p>","content":"
    Download<\/h2>\n
    ba-thesis-saltyrtc-by-lennart-grahl-revised-v1.pdf<\/a><\/p>\n
    Abstract<\/h2>\n
    WebRTC is an API definition by the World Wide Web Consortium (W3C) which \nobtained a high popularity among browser-to-browser applications. One of the \ncore reasons for that popularity is the simplicity it provides to set up a \npeer-to-peer connection. Although it is often mainly used for audio and video \ncommunication, there is also a Data Channel<\/em> that allows arbitrary \nbidirectional data transfers between two peers. Datagram Transport Layer \nSecurity<\/em> (DTLS), which is based on Transport Layer Security<\/em> (TLS), is being \nused as WebRTC's security layer.<\/p>\n
    In the recent past, plenty of vulnerabilities in TLS have been revealed and it \nis reasonable to assume that there are more to come. In addition, WebRTC API \nusers have no control over the key pair generation, nor how the public keys are \nbeing exchanged between the peers. However, even more problematic is that \nWebRTC requires an implementation of a Signalling Channel<\/em> to exchange \nmetadata that is required to set up a peer-to-peer connection. This metadata \nalready contains security-relevant information that no third party should be \nable to read or modify, unless it is absolutely necessary for the use case.<\/p>\n
    This work presents a solution that uses the Networking and Cryptography \nlibrary<\/em> (NaCl) to provide another security layer for WebRTC Data Channels and \na secure implementation of the Signalling Channel<\/em>. The developed software \ncollection is called SaltyRTC<\/em> and will be released on GitHub soon.<\/i><\/p>\n
    Contribution<\/h2>\n
    Thanks to the guys at Threema<\/a> who let me fiddle around \nwith WebRTC freely and gave me the possibility to create a Threema Web Client<\/em> \nprototype.<\/p>\n
    Update - Mon, 24 Oct 2016<\/h6>\n
    The described SaltyRTC<\/em> protocol has since been carefully improved and is now \nofficially specified here<\/a>.<\/p>\n
    Update - Wed, 15 Feb 2017<\/h6>\n
    Nearly two years later, the official Threema Web Client<\/a> has been\nreleased which is based on technologies established for the prototype\nincluding the SaltyRTC<\/em> signalling protocol.\n<\/span><\/p>","category":[{"@attributes":{"term":"WebRTC"}},{"@attributes":{"term":"webrtc"}},{"@attributes":{"term":"datachannel"}},{"@attributes":{"term":"saltyrtc"}},{"@attributes":{"term":"threema"}}]}]}
But you definitely don\u2019t want to do that ugly SDP inspection yourselves, so\nsupport in adapter.js is about to be added<\/a>. I encourage you to\nupdate to the latest adapter version once that PR has been merged as it also\nsorts out various other incompatibility nits regarding the maximum message size\nfor many browser interoperability cases.<\/p>\n
But sadly, it doesn't because Firefox and all other browsers currently do not\nsupport the object API that has been borrowed from the\nORTC specification<\/a>. So, what can be done to retrieve this\ninformation? The answer lies within the SDP blob. Check this out:<\/p>\n