{"@attributes":{"version":"2.0"},"channel":{"title":"eta.st","description":"eta's personal website & blog\n","link":"https:\/\/eta.st\/","pubDate":"Mon, 15 Dec 2025 23:45:01 +0000","lastBuildDate":"Mon, 15 Dec 2025 23:45:01 +0000","generator":"Jekyll v4.3.4","item":[{"title":"Reversing UK mobile rail tickets","description":"<p>The UK has used small credit-card sized tickets to pay for train travel for years and years, since long before I was born \u2014 originally the\n<a href=\"https:\/\/en.wikipedia.org\/wiki\/APTIS_ticket_features\">APTIS ticket<\/a><sup id=\"fnref:aptis\"><a href=\"#fn:aptis\" class=\"footnote\" rel=\"footnote\" role=\"doc-noteref\">1<\/a><\/sup>,\nwhich later got replaced by a\n<a href=\"https:\/\/en.wikipedia.org\/wiki\/2014_National_Rail_ticket_features\">slightly easier to read version<\/a> printed onto the same stock.<\/p>\n\n<p><img src=\"\/assets\/img\/rsp6\/ticket.jpg\" alt=\"a National Rail paper ticket\" \/><\/p>\n\n<p>Nowadays, the industry would very much like you to ditch your paper ticket in favour of a fancy mobile barcode one (or\nan <a href=\"https:\/\/en.wikipedia.org\/wiki\/ITSO_Ltd\">ITSO<\/a> smartcard<sup id=\"fnref:itso\"><a href=\"#fn:itso\" class=\"footnote\" rel=\"footnote\" role=\"doc-noteref\">2<\/a><\/sup>); not\nonly do they not have to spend money on printing tickets but they also gain the ability to more precisely track the ticket\u2019s usage across the network\nand minimise fraud.<\/p>\n\n<p><img src=\"\/assets\/img\/rsp6\/why-mobile.png\" alt=\"promotional material about the benefits of mobile ticketing\" \/><\/p>\n\n<p>There are obvious benefits for the user too \u2014 I\u2019m willing to bet most people use the mobile tickets anyway, since they\u2019re just easier\nif you\u2019re booking your train travel in an app like <a href=\"https:\/\/www.thetrainline.com\/\">Trainline<\/a>.<\/p>\n\n<p>But what data is inside the barcode of a mobile ticket, and how do they work? Could people who aren\u2019t ticket inspectors get the data out of them?\nIt turns out that the answer is a bit more interesting than I initially expected!<\/p>\n\n<h2 id=\"initial-explorations\">Initial explorations<\/h2>\n\n<p>A mobile ticket is just an <a href=\"https:\/\/en.wikipedia.org\/wiki\/Aztec_Code\">Aztec barcode<\/a>, either displayed inside an app or on a PDF you can print out:<\/p>\n\n<p><img src=\"\/assets\/img\/rsp6\/initial-ticket.png\" alt=\"image of a Trainline ticket barcode, from Cathays (CYS) to Cardiff Queen St (CDQ), UTN TTDNQMCQF6S\" \/><\/p>\n\n<p>Googling around for prior work people had done decoding mobile tickets, I found\n<a href=\"https:\/\/de.wikipedia.org\/w\/index.php?title=Diskussion:Online-Ticket&amp;oldid=198973306#Barcode\">a bunch of discussion<\/a> (German-language link) about UIC 918.3,\na specification used by the German railway company Deutsche Bahn for their e-tickets. These also use the Aztec barcode format and looked superficially\nsimilar \u2014 and some people had already written code to read them. Maybe this could work?<\/p>\n\n<p>(Why would I expect this to work? Well, the UIC is the international standards body for railways \u2014 in Europe at least \u2014\nso it\u2019s reasonable to assume they might\u2019ve used a standard format here.)<\/p>\n\n<p>However, the formats are sadly nothing alike \u2014 decoding our UK barcode using <a href=\"https:\/\/github.com\/zxing\/zxing\">zxing<\/a>, we get:<\/p>\n\n<pre style=\"white-space: pre-wrap; overflow-wrap: anywhere;\">\n06DNQL4XHVK00TTRCGPUQWNTHPGHWBPOUTKRWXAJKGHFBAPBCTOGUZQVTZTKKDEBQXPGRWZJRJBXJZPOHNJGIPDJWEGYWJXLVPGEEZBCUUELIJMOINPRZMSDQCZJGLIZLUTQHXMTPKWCMJISUXQLORAOVYXSOLGXXGMVUDXTMHAYMBLUTKPUPFCRNNTDBBDLNWSBPDUXYKSIMJSBYBURSCPUMFBZPEUTECHTIOXAH\n<\/pre>\n\n<p>\u2026which looks nothing like you might expect a UIC barcode to look, given the latter are supposed to start with \u201c#UT\u201d\n(according to the discussion in German earlier).<\/p>\n\n<p>In fact, this is a custom standard that is only used inside the UK, as the \u201c06\u201d at the start of the data hints at; this is an \u201cRSP-6\u201d ticket\n(as in RSP for <a href=\"https:\/\/en.wikipedia.org\/wiki\/Rail_Settlement_Plan\">Rail Settlement Plan<\/a>), which Google doesn\u2019t seem to know much about.\nIt is possible to find a <a href=\"https:\/\/www.whatdotheyknow.com\/request\/rsp_6_specification_for_barcodes_2\">Freedom of Information Act request<\/a> someone made\nasking for the spec that never got a response \u2014 sadly the Rail Delivery Group (RDG), technically a private company, doesn\u2019t actually have to respond\nto such requests, so I\u2019d have to figure this out myself.<\/p>\n\n<h2 id=\"a-friendly-wolf-comes-to-help\">A friendly wolf comes to help<\/h2>\n\n<p>At this point, I basically had no idea how to continue. Comparing multiple tickets, the data seemed to be mostly random apart from some fixed headers,\nsuggesting that it was probably encrypted in some way \u2014 I couldn\u2019t just get lots of tickets and hope to find similarities between them.<\/p>\n\n<p><img src=\"\/assets\/img\/rsp6\/binwalk.png\" alt=\"photo of 'binwalk -W cdq-cys.bin pad-aml.bin', showing no shared data apart from a few random bytes and some headers\" \/><\/p>\n\n<p><em>image: output of <code class=\"language-plaintext highlighter-rouge\">binwalk -W cdq-cys.bin pad-aml.bin<\/code>, which highlights similarities and differences between two tickets<sup id=\"fnref:binwalk\"><a href=\"#fn:binwalk\" class=\"footnote\" rel=\"footnote\" role=\"doc-noteref\">3<\/a><\/sup><\/em><\/p>\n\n<p>My friend Harley (\u201c<a href=\"https:\/\/lobi.to\/\">unlobito<\/a>\u201d) had noticed me complaining about tickets in a shared group chat and had a clue for me:\nthe word \u201cmasabi\u201d, which turned out to be the name of <a href=\"https:\/\/www.masabi.com\/\">a ticketing company<\/a>.<\/p>\n\n<p>Masabi\u2019s website has <a href=\"https:\/\/www.masabi.com\/justride-uk-rail\/\">this lovely page<\/a> where they explain all about how they invented mobile\nticketing in the UK in 2007 and how the RSP6 national standard was actually written by them! They also boast about how their\n\u201cJustRide Inspect\u201d suite of apps can be used to decode these tickets.<\/p>\n\n<p><img src=\"\/assets\/img\/rsp6\/justride-web.png\" alt=\"Masabi promotional copy about their Inspect app\" \/><\/p>\n\n<p>We sadly can\u2019t just get this app off the Play Store. However, after some googling around you can totally get it from one of those less\nthan official APK rehosting websites.<\/p>\n\n<p>With the APK in hand there are a number of things we can do. We can just install it on an Android device and see whether it\u2019ll give up\nanything interesting that way; we can also try to \u201cdecompile\u201d it, to get a better idea of how the app (and the ticket parser) works.<\/p>\n\n<h2 id=\"running-the-app\">Running the app<\/h2>\n\n<p>Since I didn\u2019t have any spare throwaway Android phones and that the APK might be malware, my first step was to just run it\ninside an \n<a href=\"https:\/\/developer.android.com\/studio\/run\/managing-avds\">Android Virtual Device<\/a> (using the emulator in Android Studio).\nI reckoned this would be a bit safer than just installing it on my main Android phone.<sup id=\"fnref:avd\"><a href=\"#fn:avd\" class=\"footnote\" rel=\"footnote\" role=\"doc-noteref\">4<\/a><\/sup><\/p>\n\n<p>After a bit of fiddling about, I had it doing something:<\/p>\n\n<p><img src=\"\/assets\/img\/rsp6\/app-screen1.png\" alt=\"Inspect app showing &quot;Scan config barcode&quot; screen\" \/><\/p>\n\n<p>Unfortunately, this wasn\u2019t very useful \u2014 it wouldn\u2019t scan standard ticket barcodes in this form. The \u201cLogin manually\u201d button lets you\nchoose a Train Operating Company (TOC) to sign in as (some of the companies mentioned don\u2019t even exist any more!), which was pretty\ninteresting but not useful for our goals:<\/p>\n\n<p><img src=\"\/assets\/img\/rsp6\/app-screen2.png\" alt=\"Inspect app showing list of TOCs\" \/><\/p>\n\n<p>I don\u2019t work for a train operating company, so clearly it wasn\u2019t going to be possible for me to proceed<sup id=\"fnref:notoc\"><a href=\"#fn:notoc\" class=\"footnote\" rel=\"footnote\" role=\"doc-noteref\">5<\/a><\/sup>.\nMaybe examining the app another way could get us somewhere?<\/p>\n\n<h2 id=\"decompiling-the-app\">Decompiling the app<\/h2>\n\n<p>You can point Android Studio at an APK and have it analyse what\u2019s inside. Sort of.<\/p>\n\n<p>If you try this (via the 3-dot menu in the project chooser \u2192 \u201cProfile or Debug APK\u201d), you get something that\u2019s not entirely useful:\na bunch of weird looking \u201csmali\u201d files.<\/p>\n\n<p><img src=\"\/assets\/img\/rsp6\/smalis.png\" alt=\"Android Studio showing debug view of the APK\" \/><\/p>\n\n<p>This is because the APK only contains compiled bytecode, rather than any useful source code; as the yellow warning banner notes,\nthis is stored inside the APK in <a href=\"https:\/\/source.android.com\/docs\/core\/runtime\/dex-format\">.dex format<\/a>\n(\u201cDalvik Executable\u201d, where Dalvik is the name of the Android VM), and\n<a href=\"https:\/\/github.com\/JesusFreke\/smali\">smali<\/a> is just a human-readable representation someone invented for this format (like Assembly).<\/p>\n\n<p>What we ideally need is something that can turn the bytecode all the way back into Java. Such a tool exists in the form of\n<a href=\"https:\/\/github.com\/skylot\/jadx\">jadx<\/a>, a very handy tool that not only does just that, but can also do a bunch of other\ncool stuff like outputting a project Android Studio can load (and theoretically compile)!<\/p>\n\n<pre style=\"white-space: pre-wrap; overflow-wrap: anywhere;\">\n$ jadx --deobf -e -d out ~\/Downloads\/justride-inspect.apk\nINFO  - loading ...\nINFO  - processing ...\nINFO  - done                                                 \n$ ls out\/\napp  build.gradle  settings.gradle\n<\/pre>\n\n<h2 id=\"making-sense-of-the-decompiled-output\">Making sense of the decompiled output<\/h2>\n\n<p>When I looked in jadx\u2019s output directory I found a perfect copy of the original source code that Masabi had written,\nand my job was then very easy. Wait, no, that\u2019s a lie.<\/p>\n\n<p>Before shipping an APK to the end user, Android app developers usually run it through a number of steps to make it smaller\nincluding \u201cobfuscation\u201d \u2014 turning long class and member names like \u201cmContext\u201d into the smallest string they can get away with, like \u201ca\u201d.\nThis means that the code jadx generated is rather hard to make sense of:<\/p>\n\n<figure class=\"highlight\"><pre><code class=\"language-java\" data-lang=\"java\"><span class=\"kd\">public<\/span> <span class=\"kd\">class<\/span> <span class=\"nc\">TicketInspectActivity<\/span> <span class=\"kd\">extends<\/span> <span class=\"nc\">BaseActivity<\/span> <span class=\"kd\">implements<\/span> <span class=\"nc\">InterfaceC2526a<\/span> <span class=\"o\">{<\/span>\n\n   <span class=\"cm\">\/* renamed from: c *\/<\/span>\n   <span class=\"kd\">private<\/span> <span class=\"nc\">ViewPager<\/span> <span class=\"n\">f4615c<\/span><span class=\"o\">;<\/span>\n\n   <span class=\"cm\">\/* JADX INFO: Access modifiers changed from: private *\/<\/span>\n   <span class=\"cm\">\/* renamed from: h *\/<\/span>\n   <span class=\"kd\">public<\/span> <span class=\"nc\">C2496p<\/span> <span class=\"nf\">m1419h<\/span><span class=\"o\">()<\/span> <span class=\"o\">{<\/span>\n       <span class=\"k\">return<\/span> <span class=\"o\">(<\/span><span class=\"nc\">C2496p<\/span><span class=\"o\">)<\/span> <span class=\"k\">this<\/span><span class=\"o\">.<\/span><span class=\"na\">f4615c<\/span><span class=\"o\">.<\/span><span class=\"na\">getAdapter<\/span><span class=\"o\">();<\/span>\n   <span class=\"o\">}<\/span>\n\n   <span class=\"nd\">@Override<\/span> <span class=\"c1\">\/\/ com.masabi.app.android.ticketcheck.activities.BaseActivity<\/span>\n   <span class=\"cm\">\/* renamed from: a *\/<\/span>\n   <span class=\"kd\">public<\/span> <span class=\"kd\">final<\/span> <span class=\"kt\">void<\/span> <span class=\"nf\">mo1410a<\/span><span class=\"o\">()<\/span> <span class=\"o\">{<\/span>\n       <span class=\"kd\">super<\/span><span class=\"o\">.<\/span><span class=\"na\">mo1410a<\/span><span class=\"o\">();<\/span>\n       <span class=\"k\">if<\/span> <span class=\"o\">(<\/span><span class=\"n\">isFinishing<\/span><span class=\"o\">())<\/span> <span class=\"o\">{<\/span>\n           <span class=\"k\">return<\/span><span class=\"o\">;<\/span>\n       <span class=\"o\">}<\/span>\n       <span class=\"n\">m1419h<\/span><span class=\"o\">().<\/span><span class=\"na\">m1467a<\/span><span class=\"o\">(<\/span><span class=\"k\">this<\/span><span class=\"o\">.<\/span><span class=\"na\">f4615c<\/span><span class=\"o\">.<\/span><span class=\"na\">getCurrentItem<\/span><span class=\"o\">());<\/span>\n   <span class=\"o\">}<\/span>\n   <span class=\"cm\">\/* ... *\/<\/span>\n<span class=\"o\">}<\/span><\/code><\/pre><\/figure>\n\n<p>Mmm yes, I knew exactly what they meant when they named their class <code>C2496p<\/code>. Of course! We just need to trace the execution\nof <code>void mo1410a()<\/code> and then we\u2019ll figure it all out!<sup id=\"fnref:jadx\"><a href=\"#fn:jadx\" class=\"footnote\" rel=\"footnote\" role=\"doc-noteref\">6<\/a><\/sup><\/p>\n\n<p>Despite this looking daunting at first it\u2019s actually quite okay with the tools Android Studio gives us. Not everything is completely obfuscated:\nsome class names need to be left deobfuscated, such as activities (like this <code>TicketInspectActivity<\/code>). That lets us get some idea of\nwhere to start. The code also occasionally contains error messages that give away what the classes and methods are supposed to be:<\/p>\n\n<figure class=\"highlight\"><pre><code class=\"language-java\" data-lang=\"java\"><span class=\"cm\">\/* renamed from: com.masabi.c.a *\/<\/span>\n<span class=\"cm\">\/* loaded from: classes.dex *\/<\/span>\n<span class=\"kd\">public<\/span> <span class=\"kd\">final<\/span> <span class=\"kd\">class<\/span> <span class=\"nc\">C2666a<\/span> <span class=\"o\">{<\/span>\n   <span class=\"cm\">\/* renamed from: a *\/<\/span>\n   <span class=\"kd\">public<\/span> <span class=\"kd\">static<\/span> <span class=\"kd\">final<\/span> <span class=\"kt\">int<\/span> <span class=\"nf\">m552a<\/span><span class=\"o\">(<\/span><span class=\"nc\">Calendar<\/span> <span class=\"n\">calendar<\/span><span class=\"o\">)<\/span> <span class=\"o\">{<\/span>\n       <span class=\"k\">if<\/span> <span class=\"o\">(<\/span><span class=\"n\">calendar<\/span> <span class=\"o\">==<\/span> <span class=\"kc\">null<\/span><span class=\"o\">)<\/span> <span class=\"o\">{<\/span>\n           <span class=\"nc\">System<\/span><span class=\"o\">.<\/span><span class=\"na\">err<\/span><span class=\"o\">.<\/span><span class=\"na\">println<\/span><span class=\"o\">(<\/span><span class=\"s\">\"DateTimeUtils.packDate() ERROR - Attempt to pack a null date!\"<\/span><span class=\"o\">);<\/span>\n       <span class=\"o\">}<\/span>\n       <span class=\"k\">return<\/span> <span class=\"o\">(<\/span><span class=\"nc\">C2668c<\/span><span class=\"o\">.<\/span><span class=\"na\">m542a<\/span><span class=\"o\">(<\/span><span class=\"n\">calendar<\/span><span class=\"o\">)<\/span> <span class=\"o\">&lt;&lt;<\/span> <span class=\"mi\">16<\/span><span class=\"o\">)<\/span> <span class=\"o\">|<\/span> <span class=\"o\">(<\/span><span class=\"nc\">C2667b<\/span><span class=\"o\">.<\/span><span class=\"na\">m548a<\/span><span class=\"o\">(<\/span><span class=\"n\">calendar<\/span><span class=\"o\">)<\/span> <span class=\"o\">&amp;<\/span> <span class=\"mi\">65535<\/span><span class=\"o\">);<\/span>\n   <span class=\"o\">}<\/span>\n   <span class=\"cm\">\/* ... *\/<\/span>\n<span class=\"o\">}<\/span><\/code><\/pre><\/figure>\n\n<p>In this case, the log line lets us instantly rename <code>C2666a<\/code> \u2192 <code>DateTimeUtils<\/code>, and <code>m552a<\/code> \u2192 <code>packDate<\/code>.<\/p>\n\n<p>Android Studio also has excellent support for doing renames across an entire codebase at once, so after a long afternoon\npicking things apart it quickly started to take shape and our obfuscated code began to look something like the original source code\nmight have looked<sup id=\"fnref:osource\"><a href=\"#fn:osource\" class=\"footnote\" rel=\"footnote\" role=\"doc-noteref\">7<\/a><\/sup>.<\/p>\n\n<p><img src=\"\/assets\/img\/rsp6\/refactor-this.png\" alt=\"the Refactor \u2192 Rename menu in Android Studio\" \/><\/p>\n\n<h2 id=\"interesting-uses-of-rsa\">Interesting uses of RSA<\/h2>\n\n<p>My investigations into the app confirmed my suspicions that the data was indeed encrypted \u2014 well, not quite. Technically, the ticket data\nis actually <em>signed<\/em> with RSA and <a href=\"https:\/\/en.wikipedia.org\/wiki\/PKCS_1\">PKCS#1<\/a> (I think). Ticket issuers generate a payload containing the\nticket data, pad it a bit, and then use their RSA private key to create a signed message they put into the barcode. A ticket scanner has a set\nof the issuers\u2019 public keys on hand to verify the signature and read the original payload.<\/p>\n\n<p>As a more concrete example, some vague Rust code to do the verification and reading steps looks a bit like this:<\/p>\n\n<figure class=\"highlight\"><pre><code class=\"language-rust\" data-lang=\"rust\"><span class=\"c1\">\/\/ BigUint is an arbitrary size unsigned integer.<\/span>\n<span class=\"c1\">\/\/ The ticket is base26 encoded, so we need to undo that first:<\/span>\n<span class=\"k\">let<\/span> <span class=\"n\">ticket<\/span><span class=\"p\">:<\/span> <span class=\"n\">BigUint<\/span> <span class=\"o\">=<\/span> <span class=\"nf\">base26_decode<\/span><span class=\"p\">(<\/span><span class=\"o\">&amp;<\/span><span class=\"n\">ticket_str<\/span><span class=\"p\">[<\/span><span class=\"mi\">15<\/span><span class=\"o\">..<\/span><span class=\"p\">]);<\/span>\n<span class=\"c1\">\/\/ this is doing \u201cS^e mod N\u201d;<\/span>\n<span class=\"c1\">\/\/ i.e. part of RSA signature verification<\/span>\n<span class=\"k\">let<\/span> <span class=\"n\">message<\/span> <span class=\"o\">=<\/span> <span class=\"n\">ticket<\/span><span class=\"nf\">.modpow<\/span><span class=\"p\">(<\/span><span class=\"o\">&amp;<\/span><span class=\"n\">key<\/span><span class=\"py\">.public_exponent<\/span><span class=\"p\">,<\/span> <span class=\"o\">&amp;<\/span><span class=\"n\">key<\/span><span class=\"py\">.modulus<\/span><span class=\"p\">);<\/span>\n<span class=\"c1\">\/\/ convert big integer into raw bytes (big-endian)<\/span>\n<span class=\"k\">let<\/span> <span class=\"n\">message<\/span><span class=\"p\">:<\/span> <span class=\"nb\">Vec<\/span><span class=\"o\">&lt;<\/span><span class=\"nb\">u8<\/span><span class=\"o\">&gt;<\/span> <span class=\"o\">=<\/span> <span class=\"n\">message<\/span><span class=\"nf\">.to_bytes_be<\/span><span class=\"p\">();<\/span>\n<span class=\"c1\">\/\/ attempt to strip PKCS#1 padding; if it fails, the key is wrong<\/span>\n<span class=\"k\">if<\/span> <span class=\"k\">let<\/span> <span class=\"nf\">Some<\/span><span class=\"p\">(<\/span><span class=\"n\">unpadded<\/span><span class=\"p\">)<\/span> <span class=\"o\">=<\/span> <span class=\"nf\">strip_padding<\/span><span class=\"p\">(<\/span><span class=\"o\">&amp;<\/span><span class=\"n\">message<\/span><span class=\"p\">)<\/span> <span class=\"p\">{<\/span>\n   <span class=\"nd\">eprintln!<\/span><span class=\"p\">(<\/span><span class=\"s\">\"[+] decrypt done: {:?}\"<\/span><span class=\"p\">,<\/span> <span class=\"n\">unpadded<\/span><span class=\"p\">);<\/span>\n<span class=\"p\">}<\/span><\/code><\/pre><\/figure>\n\n<p>I\u2019m not a cryptographer, so this was all somewhat new to me! I was used to signatures being a <a href=\"https:\/\/en.wikipedia.org\/wiki\/Hash_function\">hash<\/a>\nof the original message (i.e. you\u2019d send the plaintext, and then <code>sign(hash(plaintext))<\/code> along with it), which is usually done\nso that you can <a href=\"https:\/\/crypto.stackexchange.com\/questions\/9896\/how-does-rsa-signature-verification-work\">sign messages longer than the size of your keys<\/a>.\nIn this case, they\u2019ve put the <em>whole<\/em> message inside the signature to save space on the barcode, meaning you need the public keys to read the message at all.<\/p>\n\n<p>You also can\u2019t make your own fraudulent tickets using this scheme; you\u2019d need the RSA private key of one of the ticket issuers to do that,\nor to have a custom public key added to the network of gate readers and ticket inspectors\u2019 apps, neither of which seem easy to do.<\/p>\n\n<details class=\"other-projects-details\">\n<summary class=\"other-projects-summary\">Some further details on the cryptography (click to expand)<\/summary>\n<div>\n<p>How can you tell that the ticket payload was unwrapped correctly? The payload is padded in a way that I think corresponds to some of the\nalgorithms in PKCS#1 (see <a href=\"https:\/\/www.rfc-editor.org\/rfc\/rfc8017\">RFC 8017<\/a>); it'll either be<\/p>\n\n<p>\nScheme 1: <code>padded = [0x00, 0x01, padding-string, 0x00, message]<\/code><br \/>\n(where <code>padding-string<\/code> is a length of <code>0xFF<\/code> octets)\n<\/p>\n\n<p>\nor\n<\/p>\n\n<p>\nScheme 2: <code>padded = [0x00, 0x02, padding-string, 0x00, message]<\/code><br \/>\n(where <code>padding-string<\/code> is a length of random non zero octets)\n<\/p>\n\n<p>If the payload doesn't look like either of these, the RSA operation failed, so you probably have the wrong key and should try another one.<\/p>\n<\/div>\n<\/details>\n\n<p>So the public keys are required knowledge for actually being able to decode these tickets. Where do we get those from?<\/p>\n\n<h2 id=\"obtaining-the-elusive-public-keys\">Obtaining the elusive public keys<\/h2>\n\n<p>The public keys aren\u2019t really published anywhere obvious, and reversing the Masabi app seems to indicate that it downloads the keys from a\nconfiguration server once you scan the config barcode mentioned earlier.<\/p>\n\n<figure class=\"highlight\"><pre><code class=\"language-java\" data-lang=\"java\"><span class=\"nc\">Global<\/span><span class=\"o\">.<\/span><span class=\"na\">logger<\/span><span class=\"o\">.<\/span><span class=\"na\">log<\/span><span class=\"o\">(<\/span><span class=\"n\">getClass<\/span><span class=\"o\">().<\/span><span class=\"na\">getSimpleName<\/span><span class=\"o\">(),<\/span> <span class=\"s\">\"loadAllKeys() - Fetched \"<\/span> <span class=\"o\">+<\/span> <span class=\"n\">barcodeKeysList<\/span><span class=\"o\">.<\/span><span class=\"na\">length<\/span> <span class=\"o\">+<\/span> <span class=\"s\">\" barcode keys from metadata\"<\/span><span class=\"o\">);<\/span>\n<span class=\"k\">for<\/span> <span class=\"o\">(<\/span><span class=\"kt\">int<\/span> <span class=\"n\">i<\/span> <span class=\"o\">=<\/span> <span class=\"mi\">0<\/span><span class=\"o\">;<\/span> <span class=\"n\">i<\/span> <span class=\"o\">&lt;<\/span> <span class=\"n\">barcodeKeysList<\/span><span class=\"o\">.<\/span><span class=\"na\">length<\/span><span class=\"o\">;<\/span> <span class=\"n\">i<\/span><span class=\"o\">++)<\/span> <span class=\"o\">{<\/span>\n   <span class=\"nc\">AbstractJSONObject<\/span> <span class=\"n\">key<\/span> <span class=\"o\">=<\/span> <span class=\"o\">(<\/span><span class=\"n\">barcodeKeysList<\/span><span class=\"o\">[<\/span><span class=\"n\">i2<\/span><span class=\"o\">];<\/span>\n   <span class=\"k\">if<\/span> <span class=\"o\">(<\/span><span class=\"nc\">ExtendedGlobal2<\/span><span class=\"o\">.<\/span><span class=\"na\">clock<\/span><span class=\"o\">.<\/span><span class=\"na\">getCurrentTime<\/span><span class=\"o\">()<\/span> <span class=\"o\">&lt;<\/span> <span class=\"nc\">Global<\/span><span class=\"o\">.<\/span><span class=\"na\">f4949d<\/span><span class=\"o\">.<\/span><span class=\"na\">mo915a<\/span><span class=\"o\">(<\/span><span class=\"n\">key<\/span><span class=\"o\">.<\/span><span class=\"na\">getString<\/span><span class=\"o\">(<\/span><span class=\"s\">\"expiryDate\"<\/span><span class=\"o\">))<\/span> <span class=\"o\">*<\/span> <span class=\"mi\">1000<\/span> <span class=\"o\">&amp;&amp;<\/span>\n           <span class=\"o\">(<\/span><span class=\"n\">decoder<\/span> <span class=\"o\">=<\/span> <span class=\"n\">makeDecoder<\/span><span class=\"o\">(<\/span><span class=\"n\">key<\/span><span class=\"o\">.<\/span><span class=\"na\">getString<\/span><span class=\"o\">(<\/span><span class=\"s\">\"issuerId\"<\/span><span class=\"o\">),<\/span> <span class=\"n\">key<\/span><span class=\"o\">.<\/span><span class=\"na\">getString<\/span><span class=\"o\">(<\/span><span class=\"s\">\"ticketType\"<\/span><span class=\"o\">),<\/span> <span class=\"n\">key<\/span><span class=\"o\">.<\/span><span class=\"na\">getString<\/span><span class=\"o\">(<\/span><span class=\"s\">\"modulus\"<\/span><span class=\"o\">),<\/span> <span class=\"n\">key<\/span><span class=\"o\">.<\/span><span class=\"na\">getString<\/span><span class=\"o\">(<\/span><span class=\"s\">\"exponent\"<\/span><span class=\"o\">),<\/span> <span class=\"n\">key<\/span><span class=\"o\">.<\/span><span class=\"na\">getLong<\/span><span class=\"o\">(<\/span><span class=\"s\">\"mQ\"<\/span><span class=\"o\">)))<\/span> <span class=\"o\">!=<\/span> <span class=\"kc\">null<\/span><span class=\"o\">)<\/span> <span class=\"o\">{<\/span>\n       <span class=\"n\">decoders<\/span><span class=\"o\">.<\/span><span class=\"na\">addElement<\/span><span class=\"o\">(<\/span><span class=\"n\">decoder<\/span><span class=\"o\">);<\/span>\n   <span class=\"o\">}<\/span>\n<span class=\"o\">}<\/span><\/code><\/pre><\/figure>\n\n<p>You\u2019d think this would be a dead end, since we don\u2019t have any login credentials \u2014 but they also just left some keys inside the APK\nas well. As far as I can tell no part of the app actually reads these; maybe it did in the past, or maybe they used them for testing and\nforgot to take them out of the production version of the app.<\/p>\n\n<pre>\n$ find . | grep 'keys' | grep rsp6\n.\/app\/src\/main\/assets\/keys\/rsp6_rsa_ao.dat\n.\/app\/src\/main\/assets\/keys\/rsp6_rsa_ua.dat\n.\/app\/src\/main\/assets\/keys\/rsp6_rsa_tt-qa.dat\n.\/app\/src\/main\/assets\/keys\/rsp6_rsa_tt.dat\n.\/app\/src\/main\/assets\/keys\/rsp6_rsa_t3.dat\n.\/app\/src\/main\/assets\/keys\/rsp6_rsa_t2.dat\n<\/pre>\n\n<p>The keys are split up by ticket issuer, a 2-character code that forms the first part of the ticket ID. This ticket from earlier was issued\nby Trainline, who have issuer code <strong>TT<\/strong>\u2026<\/p>\n\n<p><img src=\"\/assets\/img\/rsp6\/initial-barcode.png\" alt=\"barcode of the earlier ticket\" \/><\/p>\n\n<p>\u2026and the keys to decode this ticket are in <code>rsp6_rsa_<b>tt<\/b>.dat<\/code>. Nice!<sup id=\"fnref:datfile\"><a href=\"#fn:datfile\" class=\"footnote\" rel=\"footnote\" role=\"doc-noteref\">8<\/a><\/sup><\/p>\n\n<p>This is only a subset of all the keys that are being used today though, as I quickly discovered when\n<a href=\"https:\/\/lobi.to\/\">unlobito<\/a> gave me an <a href=\"https:\/\/www.avantiwestcoast.co.uk\/\">Avanti West Coast<\/a> ticket to decode.\nThe copy of the app I have is only from 2016 and a bunch more TOCs have started issuing mobile tickets since then!<\/p>\n\n<h2 id=\"ttkmobile\">ttkMobile<\/h2>\n\n<p>Around when I was figuring all of this out <a href=\"https:\/\/twitter.com\/puckipedia\">puck<\/a> pointed me towards the website for\n<a href=\"http:\/\/info.theticketkeeper.com\/\">The Ticket Keeper<\/a>, another firm who makes ticket validation and issuance tooling.\nThey have an iOS app for ticket inspectors called <strong>ttkMobile<\/strong>, which you can just\n<a href=\"https:\/\/apps.apple.com\/gb\/app\/ttkmobile\/id921166994\">download straight off the App Store<\/a> and use to start validating tickets at home!<sup id=\"fnref:ttk\"><a href=\"#fn:ttk\" class=\"footnote\" rel=\"footnote\" role=\"doc-noteref\">9<\/a><\/sup><\/p>\n\n<p><img src=\"\/assets\/img\/rsp6\/ttkmobile.jpg\" alt=\"screenshot of ttkMobile\" \/><\/p>\n\n<details class=\"other-projects-details\">\n<summary class=\"other-projects-summary\">Important note if you actually intend to use this app (click to expand)<\/summary>\n<p>Be warned!<\/p>\n\n<p>If you install this app, you can\u2019t ever uninstall it and expect it to work after a reinstall.\nOn first load it registers your device UUID with some server and generates a random password that it stores in local storage.\nUninstalling the app removes the password, but your device UUID doesn\u2019t change, so next time you reinstall, it won\u2019t be able\nto authenticate and it\u2019ll be useless (since it needs to grab keys and stuff to work).<\/p>\n\n<p>(\u201cBut wait,\u201d I hear you cry, \u201cisn\u2019t getting a persistent device ID exactly what Apple don\u2019t want you to do?\u201d And you\u2019d be right! Technically, I believe\nthe device UUID actually <i>does<\/i> change between installs, but they store a copy in the device keychain, which doesn\u2019t get wiped\nwhen you remove the app. This is stupid, and almost certainly a contravention of App Store policy.)<\/p>\n<\/details>\n\n<p>I don\u2019t have an iPhone, but some of my friends do. unlobito and another friend,\nEva (\u201c<a href=\"https:\/\/muffinti.me\/\">thejsa<\/a>\u201d), had a poke around and managed to get me a DRM-free<sup id=\"fnref:nodrm\"><a href=\"#fn:nodrm\" class=\"footnote\" rel=\"footnote\" role=\"doc-noteref\">10<\/a><\/sup> <a href=\"https:\/\/en.wikipedia.org\/wiki\/.ipa\"><code>.ipa<\/code><\/a>\ncontaining the app, which I could unwrap and decompile with the help of <a href=\"https:\/\/ghidra-sre.org\/\">Ghidra<\/a>.\nThis let me figure out some of the pieces of the ticket that the Masabi app didn\u2019t look at.<\/p>\n\n<p><a href=\"https:\/\/muffinti.me\/\">thejsa<\/a> also spent some time running the app through a proxy in order to find out how it\ncommunicates with the server<sup id=\"fnref:thejsa\"><a href=\"#fn:thejsa\" class=\"footnote\" rel=\"footnote\" role=\"doc-noteref\">11<\/a><\/sup>, and it turns out there\u2019s just an endpoint where you can get all of the public keys:<\/p>\n\n<pre>\n$ curl 'https:\/\/device.theticketkeeper.com\/download_keys?device_name=abc' | jq .\n{\n  \"return_code\": \"ok\",\n  \"message\": null,\n  \"keys\": {\n    \"AA\": [\n      {\n        \"valid_from\": \"20000101000000\",\n        \"valid_until\": \"29991231000000\",\n        \"public_exponent_hex\": \"10001\",\n        \"modulus_hex\": \"9140AA61F7D9A2E943C0510BACA5FA9CA7D12D78E301A36D640F2D28D8C0AA4D6A7102555CECF138E467730B797509EC1AB5BBA77CA6384BC8F483F609B121E75AE42660EDFE15EF91ADD4DA68C355F830FAAC6FFB25FBCFE1E61C7AF37C4AE8C85E264C151BD9C9AA4DE41D2756A9E260C0CC89AE2ADDD19E452A675E88DA47\",\n        \"public_key_x509\": null,\n        \"test_only\": \"N\",\n        \"updated\": \"20200313175331\"\n      },\n[etc]\n<\/pre>\n\n<p>As I mentioned earlier, this is crucial information to be able to decode tickets at all,\nso thanks go to The Ticket Keeper developers for making it available so easily!<\/p>\n\n<h3 id=\"brief-aside-on-freedom-of-information\">Brief aside on freedom of information<\/h3>\n\n<p>I don\u2019t know whether people in the industry (e.g. the Rail Delivery Group)\nwill be upset with me publishing this information or not.\nI hope they won\u2019t be: I really think the public keys should be made available to the public, along with the official specifications for decoding.\nThe tickets are signed, so it\u2019s not as if there\u2019s any practical danger \u2014 people can\u2019t use this to start forging tickets en masse, for example \u2014\nand there are lots of potential innovative uses for this data. Imagine for example a journey logger that used ticket scans to track where you\u2019d\nbeen automatically, or an expenses system that used the price information encoded in the ticket to automatically log expense requests!<\/p>\n\n<p>The railways might be run by a consortium of private companies, but they are in effect a public service owned and controlled\nby the Government<sup id=\"fnref:erma\"><a href=\"#fn:erma\" class=\"footnote\" rel=\"footnote\" role=\"doc-noteref\">12<\/a><\/sup> (as of Jan 2023), so they really should be subject to the same Freedom of Information Act provisions as other public bodies.<\/p>\n\n<p>Some people in the industry already have the right idea; in conversation with one of the Ticket Keeper developers over email,\nI was made aware that the ttkMobile app being public along with some of this data is actually an intentional choice, which is really nice to see!<\/p>\n\n<h3 id=\"bonus-etvd-logs\">Bonus: eTVD logs<\/h3>\n\n<p>The website also tells you how they have an electronic Ticket Validation Database (<a href=\"http:\/\/info.theticketkeeper.com\/services\/etvd\/\">eTVD<\/a>)\nthat has a copy of all ticket scans at gatelines and by people using their app. This is the anti-fraud thing I mentioned at the very start; this sort\nof data is presumably very useful to revenue protection staff trying to figure out systematic fare evasion, like short-faring<sup id=\"fnref:shortfare\"><a href=\"#fn:shortfare\" class=\"footnote\" rel=\"footnote\" role=\"doc-noteref\">13<\/a><\/sup>.<\/p>\n\n<p>What it doesn\u2019t tell you, though, is that the app will also give you this information unauthenticated, with nothing more than a ticket\u2019s ID (!).<\/p>\n\n<div class=\"nowplaying\">\nThis was reported to the developers as a possible security issue \/ data leak on 2023-01-19. They confirmed it was intended behaviour, but\nagreed that it would probably be a good idea to restrict it; I'm told this will happen soon.\n<\/div>\n\n<p>This information can be quite disturbingly detailed, even pinpointing the exact username of the inspector who scanned you,\nwhere you were scanned, on what exact train service you were scanned, whether it succeeded, and a bunch more stuff.\nHelpfully it\u2019ll also sometimes give you the entire barcode data, and what the ticket server thinks it decodes as, too!<\/p>\n\n<pre>\n# getting scan history information for ticket CBCZSCDPVFF\n# (this is massively cut down; there are more fields in reality)\n$ curl 'https:\/\/device.theticketkeeper.com\/get_ticket_details?device_name=abc&amp;utn=CBCZSCDPVFF' | jq '.[\"ticket_detail\"][\"scans\"]'\n[\n  {\n    \"event_time_iso\": \"2022-06-10T18:30:47\",\n    \"created\": \"2022-06-10T18:30:48\",\n    \"device_type\": \"ttkMobile\",\n    \"device_id\": 1001,\n    \"device_name\": \"f95396f5-da22-47f9-8e85-dff4b2294a5d\",\n    \"device_alias\": \"2021-TK10212\",\n    \"username\": \"JLazlo01\",\n    \"action_name\": \"Accepted\",\n    \"rsp_action_code\": 4001,\n    \"event_trigger\": \"scan\",\n    \"scan_mode\": \"clip\",\n    \"scan_nlc\": \"2728\",\n    \"validation_result\": \"warning\",\n    \"message_displayed\": \"16-25 Railcard\",\n    \"gate_id\": \"OPN-3002i[021502]\",\n    \"latitude\": 51.7824963,\n    \"longitude\": -0.2141781,\n    \"device_scan_id\": 74402,\n    \"train_uid\": \"L77572\",\n    \"departure_date\": \"2022-06-10\",\n    \"barcode\": \"06CZSCDPVFF00\u2026\",\n    \"train_info\": \"Fr1803 KGX-SKI 1D26\/GR2600\",\n    # etc\n  },\n  # etc\n]\n<\/pre>\n\n<p>So yeah, your ticket barcode \u2014 or its ID, which is often written below the code in plain text \u2014\nmight let someone access a surprising amount of detailed tracking information as to where you are and what trains you\u2019re taking!\n(Rather like\n<a href=\"https:\/\/mango.pdf.zone\/finding-former-australian-prime-minister-tony-abbotts-passport-number-on-instagram\">the booking reference they send you when you book a flight<\/a>.)<\/p>\n\n<h2 id=\"trying-this-out-for-yourself\">Trying this out for yourself<\/h2>\n\n<p>With the decompiled Masabi app and the ttkMobile app together, it wasn\u2019t too hard to work out a vague idea of what the ticket format\nwas like. I\u2019ve put together <a href=\"https:\/\/git.eta.st\/eta\/rsp6-decoder\">a small repository<\/a> with a Rust tool to decode a ticket,\nas well as a small spec with my best guess on what all the fields mean.<\/p>\n\n<p>There\u2019s also a <a href=\"https:\/\/eta.st\/tickets\/\">funky web tool<\/a> I threw together in an evening or so that\u2019ll let you point your phone at\na barcode (or upload a screenshot of one) and give you a relatively nice readout of what data\u2019s inside.\n<a href=\"https:\/\/eta.st\/tickets\/\">Give it a try!<\/a> (If you need a barcode, feel free to scroll up and use the one from this post!)<\/p>\n\n<video controls=\"\" style=\"max-width: 275px;\" autoplay=\"\" loop=\"\" muted=\"\">\n    <source src=\"\/assets\/img\/rsp6\/demo.mp4\" type=\"video\/mp4\" \/>\n\n    Sorry, your browser doesn't support embedded videos.\n<\/video>\n\n<p>Do feel free to <a href=\"\/#contact\">get in touch<\/a> if you find anything interesting or need help understanding something about the format!<\/p>\n\n<h2 id=\"acknowledgements\">Acknowledgements<\/h2>\n\n<p>Thanks to <a href=\"https:\/\/lobi.to\/\">unlobito<\/a>, <a href=\"https:\/\/twitter.com\/puckipedia\">puck<\/a>, and <a href=\"https:\/\/muffinti.me\/\">thejsa<\/a> (and assorted others\nin various chatrooms) for their help with all of this; go check those people out, too!<\/p>\n\n<hr \/>\n\n<h3 id=\"footnotes\">Footnotes<\/h3>\n\n<div class=\"footnotes\" role=\"doc-endnotes\">\n  <ol>\n    <li id=\"fn:aptis\">\n      <p>I still have fond memories of getting these as a child for local journeys to London Waterloo!\u00a0<a href=\"#fnref:aptis\" class=\"reversefootnote\" role=\"doc-backlink\">&#8617;<\/a><\/p>\n    <\/li>\n    <li id=\"fn:itso\">\n      <p><a href=\"https:\/\/lobi.to\/talks\/papertickets\/\">This talk<\/a> is a decent overview of the mess that is ITSO, if you\u2019re interested.\u00a0<a href=\"#fnref:itso\" class=\"reversefootnote\" role=\"doc-backlink\">&#8617;<\/a><\/p>\n    <\/li>\n    <li id=\"fn:binwalk\">\n      <p>I did try with more than just two tickets, but that\u2019d make for an unwieldy image.\u00a0<a href=\"#fnref:binwalk\" class=\"reversefootnote\" role=\"doc-backlink\">&#8617;<\/a><\/p>\n    <\/li>\n    <li id=\"fn:avd\">\n      <p>A VM provides a <em>decent<\/em> level of protection against running untrusted code, but it\u2019s not the end-all and be-all;\n    I\u2019d imagine the Android emulator isn\u2019t exactly hardened against people trying to break out, so this probably isn\u2019t a\n    bulletproof strategy for running shady apps in the general case.\u00a0<a href=\"#fnref:avd\" class=\"reversefootnote\" role=\"doc-backlink\">&#8617;<\/a><\/p>\n    <\/li>\n    <li id=\"fn:notoc\">\n      <p>It turns out that the app actually downloads a lot of the keys and data necessary to decode tickets using this login,\n      so even if I could somehow glitch it into starting the main ticket scanner, it wouldn\u2019t have worked.\u00a0<a href=\"#fnref:notoc\" class=\"reversefootnote\" role=\"doc-backlink\">&#8617;<\/a><\/p>\n    <\/li>\n    <li id=\"fn:jadx\">\n      <p>jadx has made things slightly easier for us by renaming some identifiers (\u201cdeobfuscation\u201d): adding some numbers and adding\n     prefixes like \u201cC\u201d for \u201cclass\u201d that help us distinguish amongst the multitude of things called \u201ca\u201d or \u201cb\u201d.\u00a0<a href=\"#fnref:jadx\" class=\"reversefootnote\" role=\"doc-backlink\">&#8617;<\/a><\/p>\n    <\/li>\n    <li id=\"fn:osource\">\n      <p>It\u2019s impossible to know without the real source what all the classes were actually called, but I can give them names that\n        make sense to <em>me<\/em>, and that\u2019s all that matters really. puck actually did manage to find an app that used some classes from\n        the Masabi SDK that was not obfuscated later on, which let me compare and see how accurate my invented names were against\n        the real ones!\u00a0<a href=\"#fnref:osource\" class=\"reversefootnote\" role=\"doc-backlink\">&#8617;<\/a><\/p>\n    <\/li>\n    <li id=\"fn:datfile\">\n      <p>The format of these .dat files is a bit weird and nonstandard, and you have to reverse the RSA decryption code to figure out\n        what it is. Or, you can be lazy like me, and just paste the decompiled code into a new Java project and treat it as a black box.\u00a0<a href=\"#fnref:datfile\" class=\"reversefootnote\" role=\"doc-backlink\">&#8617;<\/a><\/p>\n    <\/li>\n    <li id=\"fn:ttk\">\n      <p>It does expect to report back completed ticket scans by default, so you have to futz around and listen to it complain a bit before it\u2019ll work.\n    Also, the \u201cTicket Issuing\u201d button doesn\u2019t work, if you were wondering; you need a login for that.\u00a0<a href=\"#fnref:ttk\" class=\"reversefootnote\" role=\"doc-backlink\">&#8617;<\/a><\/p>\n    <\/li>\n    <li id=\"fn:nodrm\">\n      <p>As I understand it, you usually need a jailbroken phone to do this. Otherwise, you could just copy paid apps between phones and\n      share them with people who hadn\u2019t paid for them.\u00a0<a href=\"#fnref:nodrm\" class=\"reversefootnote\" role=\"doc-backlink\">&#8617;<\/a><\/p>\n    <\/li>\n    <li id=\"fn:thejsa\">\n      <p>This is also how she discovered the reason why the app doesn\u2019t work after being uninstalled. She even went so far as writing\n       a jailbreak tweak to change the device ID to make it work again, which is pretty cool!\u00a0<a href=\"#fnref:thejsa\" class=\"reversefootnote\" role=\"doc-backlink\">&#8617;<\/a><\/p>\n    <\/li>\n    <li id=\"fn:erma\">\n      <p>Yes, our railway looks privatised on the surface, but COVID actually resulted in so-called\n     <a href=\"https:\/\/www.gov.uk\/government\/speeches\/rail-update-emergency-recovery-measures-agreements\">Emergency Recovery Measures Agreements<\/a>\n     that forced the TOCs into effective state ownership and control (they\u2019re now in effect contractors who run the service, rather than\n     taking revenue risk). This means that the Government are actually responsible for things like the\n     current union dispute over jobs and working conditions, despite what some ministers would have you think!\u00a0<a href=\"#fnref:erma\" class=\"reversefootnote\" role=\"doc-backlink\">&#8617;<\/a><\/p>\n    <\/li>\n    <li id=\"fn:shortfare\">\n      <p>This refers to the practice of buying a ticket that doesn\u2019t actually cover your whole journey (e.g. skipping some stops at the start or\n          end), and gambling that you\u2019ll only get inspected between the stations where it\u2019s actually valid.\u00a0<a href=\"#fnref:shortfare\" class=\"reversefootnote\" role=\"doc-backlink\">&#8617;<\/a><\/p>\n    <\/li>\n  <\/ol>\n<\/div>\n","pubDate":"Tue, 31 Jan 2023 00:00:00 +0000","link":"https:\/\/eta.st\/2023\/01\/31\/rail-tickets.html","guid":"https:\/\/eta.st\/2023\/01\/31\/rail-tickets.html"},{"title":"Pessimism is a bad defense mechanism","description":"<div class=\"nowplaying\" style=\"background: #ffdada;\">\n<b>Warning!<\/b> This post is a mental health \/ feels post from a long time ago (in fact, <a href=\"\/nomenclature\">one gender ago<\/a>).\nIt is kept up in case some people find it useful, but I don't necessarily endorse the content in here nowadays.\n<\/div>\n\n<p>Why do some people have poor mental health? It\u2019s hard to know \u2013 there\u2019s an entire profession centred on answering this question! \u2013 but I feel like a part\nof it has to do with doing things that sabotage oneself, without necessarily being aware of the fact that this is happening. Of course, the next question\nis inevitably why people even begin to do such things; surely if sabotaging yourself leads to bad mental health, people would just not do it, and therefore\nlead happier lives.<\/p>\n\n<hr \/>\n\n<p>When you\u2019ve had a history of things not going too well, it can often be difficult to accept that a situation is actually pretty great \u2013 because doing so\ninvolves accepting the idea that things might turn out to be not so great later on, which is painful, so doing so puts you in a vulnerable position.\nSome might argue this is silly, and you should just accept the risk and enjoy the benefits of the situation \u2013 but this all depends on how things have gone\ndown in the past. If you\u2019ve opened yourself up and had a bad time, you\u2019re probably going to be inclined to avoid showing vulnerability in future.<\/p>\n\n<p>Taking the avoidance of vulnerability to its extremes often involves a heavy degree of pessimism. If you judge everything to be terrible even when it isn\u2019t,\nyou can conveniently avoid feeling bad about stuff going wrong; you already mentally prepared yourself for this outcome anyway, so there\u2019s no loss. And if\nit goes well, that\u2019s a bonus! The events in the past that hurt you because you opened yourself up and showed vulnerability can\u2019t any more; there\u2019s a far\nsmaller gap between your expectation and the reality if the reality does turn out to be bad.<\/p>\n\n<p>As the title of this post says, this is a defence mechanism. I can definitely recall times when something hurt me, often to such an extent that I felt a\nneed to burst out in tears (usually in some inopportune place, like on public transport) \u2013 but I didn\u2019t; I clenched my teeth, and muttered something\nunder my breath about the person who had hurt me being terrible all along, or the situation being doomed from the start, or some other overly-negative\nview that would manage to make the pain hurt less, somehow. Or, if it couldn\u2019t do that, it would at least keep me from visibly crying in front of all\nthe other commuters on the train.<\/p>\n\n<hr \/>\n\n<p>Side note: what even happens if you cry on a train, just in front of people? Do people look at you like some sort of insane person and just let you get\non with it in peace? Do complete strangers ask you if you\u2019re alright? Maybe it isn\u2019t actually too terrible \u2013 or maybe it\u2019s really rude. Will I ever\nfind out?<\/p>\n\n<p>For one reason or another<sup id=\"fnref:1\"><a href=\"#fn:1\" class=\"footnote\" rel=\"footnote\" role=\"doc-noteref\">1<\/a><\/sup>, I grew up with the idea that showing weakness is bad. To this day, it\u2019s something that I find very difficult to do, and I\noften criticize myself for having done it. We all need to show weakness eventually, though; sometimes things just get way too much to bear, and it all\ncomes flooding out. Unfortunately, this can perpetuate a vicious cycle: the only occasions where you open yourself up are hysterical disasters where\nyou end up <a href=\"https:\/\/www.urbandictionary.com\/define.php?term=Paragraphing\">paragraphing<\/a> one of your friends about all of your problems at once, or having\na lengthy phone call where you\u2019re the only one talking for a solid hour. This seems like a really imposing and terrible thing to do to people, so you\nconclude that showing weakness is bad, so you don\u2019t do it as much\u2026 meaning the outburst is even worse next time, because you can\u2019t hold the feelings\nback forever.<\/p>\n\n<p>But when you\u2019re trapped in the above cycle, it\u2019s incredibly tempting to just push the inevitable feelings outburst forward by a few days\/weeks\/months with the\nliberal application of some pessimism.<\/p>\n\n<hr \/>\n\n<p>Apart from this train-crying-reduction purpose, the other thing pessimism is very good at is making the actual causes of things far more vague and\nnebulous. You can avoid thinking about why something went wrong if you chalk the whole thing up to it just being a bad situation; this is particularly\nuseful when you\u2019re actually the one doing things to make a situation worse. Here\u2019s an example<sup id=\"fnref:2\"><a href=\"#fn:2\" class=\"footnote\" rel=\"footnote\" role=\"doc-noteref\">2<\/a><\/sup>: maybe you\u2019re sad about getting out of contact\nwith a friend you haven\u2019t seen in a while. If you take a pessimistic angle, you can usefully assume this is because people just naturally drift apart given\nspace, which is unfortunate but makes the whole thing seem rather impersonal: it\u2019s just a thing which is happening to you which you have no control over.<\/p>\n\n<p>This isn\u2019t how it works, though \u2013 in the above situation, the reason why you don\u2019t talk to the friend could actually be because <em>you<\/em> don\u2019t make much\nof an effort any more with them, because of the pessimism (\u201cwe\u2019re going to drift apart anyway\u201d). Another example: maybe you\u2019re upset about being fired,\napply for a new job, and don\u2019t get it. In a lot of situations, you do indeed have no control over whether or not the company will hire you \u2013 but\nif you thought you\u2019d never get a job again after getting fired and didn\u2019t bother to put in the effort for the application as a result, you\u2019re actually\nsabotaging yourself here.<\/p>\n\n<p>This ability of pessimism to confuse situations can start to ruin your life if you\u2019re not careful. If everything is terrible, what\u2019s the point in doing\nanything? Opportunities to make things better pass you by without you doing anything (because you refuse to be brave enough to actually take things\ninto your own hands and maybe fail), and can then serve as further examples of why everything is terrible when you start feeling bad about what you\u2019ve\nmissed out on, reinforcing the cycle. Because it\u2019s not always obvious that <em>you<\/em> were the one that made the situation bad, you\u2019re left feeling like\nyou can\u2019t do anything about it. Even when it\u2019s clear that you made a mistake, you might view yourself as incontrovertibly bad \u2013 sure, it was your\nfault, but you\u2019re a terrible person, so you\u2019d never have made the right decision anyway (!).<\/p>\n\n<p>Thinking like this leads to viewing life as this third-person experience where stuff just happens to you and you have to just sit there and take it \u2013\nthat you have no control over events. (Some people call this <a href=\"https:\/\/en.wikipedia.org\/wiki\/Learned_helplessness\">\u201clearned helplessness\u201d<\/a>.)\nAnd that\u2019s, to put it mildly, not good; you can get very depressed<sup id=\"fnref:3\"><a href=\"#fn:3\" class=\"footnote\" rel=\"footnote\" role=\"doc-noteref\">3<\/a><\/sup> this way!<\/p>\n\n<hr \/>\n\n<p>I would have liked to finish this blog post with some inspiring call to action about how to get out of this cycle and make all the things better in 3\nsimple steps. Unfortunately, I\u2019m not actually qualified enough to do that.<\/p>\n\n<p>Knowing what a problem is is often half the battle, however, and it\u2019s definitely possible to use this knowledge for good. The research on learned\nhelplessness actually suggests that it\u2019s not helplessness people learn, it\u2019s actually help<em>ful<\/em>ness \u2013 the default opinion of the brain is (allegedly)\nthat you can\u2019t do things to change your situation, and you have to <em>teach<\/em> it that you do actually have some power after all.<\/p>\n\n<p>So, do some things, and pay attention to their effect. I bet at least in some cases, you have more ability to change things than you think.<\/p>\n\n<hr \/>\n\n<div class=\"footnotes\" role=\"doc-endnotes\">\n  <ol>\n    <li id=\"fn:1\">\n      <p>Is going to a academically selective, all-male private school a reason? No, that couldn\u2019t <em>possibly<\/em> be related\u2026\u00a0<a href=\"#fnref:1\" class=\"reversefootnote\" role=\"doc-backlink\">&#8617;<\/a><\/p>\n    <\/li>\n    <li id=\"fn:2\">\n      <p>Frequent readers of the blog will know that the \u201cexamples\u201d are often sanitized versions of things that have recently happened.\u00a0<a href=\"#fnref:2\" class=\"reversefootnote\" role=\"doc-backlink\">&#8617;<\/a><\/p>\n    <\/li>\n    <li id=\"fn:3\">\n      <p>I think it\u2019s actually a lot <em>more<\/em> dangerous when you hold this belief underneath, but otherwise have what seems to be a relatively okay life.\n  Every so often, something will happen that makes you sad for a while, and then you\u2019ll get over it and act like nothing happened without realizing\n  what\u2019s going on \u2013 thereby ensuring that you\u2019ll continue to become occasionally sad in perpetuity!\u00a0<a href=\"#fnref:3\" class=\"reversefootnote\" role=\"doc-backlink\">&#8617;<\/a><\/p>\n    <\/li>\n  <\/ol>\n<\/div>\n","pubDate":"Wed, 30 Jun 2021 00:00:00 +0000","link":"https:\/\/eta.st\/2021\/06\/30\/self-sabotage.html","guid":"https:\/\/eta.st\/2021\/06\/30\/self-sabotage.html"},{"title":"Sending audio to LKV373 HDMI extenders","description":"<p>My <a href=\"https:\/\/benjojo.co.uk\/\">current flatmate<\/a> has a bunch of these <a href=\"https:\/\/blog.benjojo.co.uk\/post\/cheap-hdmi-capture-for-linux\">HDMI extender devices<\/a>\n(also apparently known as \u201cLKV373\u201d \/ \u201cLenkeng extenders\u201d \/ \u201cLK-3080-ACL\u201d). They\u2019re pretty nifty: HDMI goes in one end, they emit UDP over ethernet,\nand then HDMI comes out on the receiver end with surprisingly minimal latency (around 100ms).<\/p>\n\n<p><img src=\"\/assets\/img\/showtime\/extender-3.jpg\" alt=\"photo of an extender\" \/><\/p>\n\n<p><em>image: the receiver end of a transmitter-receiver pair, taken by said flatmate<\/em><\/p>\n\n<p>A couple months ago we decided it\u2019d be quite nifty to be able to play arbitrary audio out of the devices without having to plug a transmitter into someone\u2019s\nlaptop \u2013 so that we could have BBC Radio 4 or something constantly playing whenever the sound system is turned on.<\/p>\n\n<p>My flatmate\u2019s blog post, <a href=\"https:\/\/blog.benjojo.co.uk\/post\/cheap-hdmi-capture-for-linux\">\u201cLudicrously cheap HDMI capture for Linux\u201d<\/a>, outlined a way to\n<em>decode<\/em> the signal the transmitters sent (using his own tool, \n<a href=\"https:\/\/github.com\/benjojo\/de-ip-hdmi\"><code class=\"language-plaintext highlighter-rouge\">de-ip-hdmi<\/code><\/a>). But could we do the opposite,\nand <em>encode<\/em> signals the receivers would accept?<\/p>\n\n<h2 id=\"testing-without-the-actual-devices\">Testing without the actual devices<\/h2>\n\n<p>I figured I\u2019d start by looking at <code class=\"language-plaintext highlighter-rouge\">de-ip-hdmi<\/code>\u2019s source code to see how it\ndecoded the packets and then use it as a way to test the output of my\ncustom sender until it was able to decode what I was sending. It\u2019s pretty\nunlikely that I\u2019d just be able to send audio without also sending a picture,\nso I tried to figure out the picture format first.<\/p>\n\n<p><img src=\"\/assets\/img\/showtime\/sharked-sequences.png\" alt=\"video picture packet format\" \/><\/p>\n\n<p>The video picture packets have a fairly simple format: two big-endian, 16-bit\nintegers (frame number, and sequence number), and then data, up to 1020 bytes\n(making UDP packets with a 1024-byte payload, total).<\/p>\n\n<p>Extracting the output from <code class=\"language-plaintext highlighter-rouge\">de-ip-hdmi<\/code> with a transmitter running\nconfirms that the data is just a JPEG:<\/p>\n\n<div class=\"language-plaintext highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code>$ file test.mjpeg \ntest.mjpeg: JPEG image data, JFIF standard 1.01, aspect ratio, density 1x1, segment length 16, baseline, precision 8, 1920x1080, components 3\n<\/code><\/pre><\/div><\/div>\n\n<p>The frame number counts\nmonotonically up from 0, incrementing with each new frame. Since you\u2019re not\ngoing to fit a whole JPEG frame in 1020 bytes, the sequence number provides a way of\nsplitting the frame up into multiple chunks; you increment it for each chunk\nyou send within a frame, while keeping the frame number constant.<\/p>\n\n<p>The <code class=\"language-plaintext highlighter-rouge\">de-ip-hdmi<\/code> <a href=\"https:\/\/github.com\/benjojo\/de-ip-hdmi\/blob\/0bdfab96c7fbe8819977846e402c56a0b7660b47\/main.go#L117\">source code to decode<\/a>\nlooks like this:<\/p>\n\n<figure class=\"highlight\"><pre><code class=\"language-go\" data-lang=\"go\"><span class=\"c\">\/\/ taken from the de-ip-hdmi source code, in Go<\/span>\n<span class=\"n\">FrameNumber<\/span> <span class=\"o\">:=<\/span> <span class=\"kt\">uint16<\/span><span class=\"p\">(<\/span><span class=\"m\">0<\/span><span class=\"p\">)<\/span>\n<span class=\"n\">CurrentChunk<\/span> <span class=\"o\">:=<\/span> <span class=\"kt\">uint16<\/span><span class=\"p\">(<\/span><span class=\"m\">0<\/span><span class=\"p\">)<\/span>\n\n<span class=\"n\">buf<\/span> <span class=\"o\">:=<\/span> <span class=\"n\">bytes<\/span><span class=\"o\">.<\/span><span class=\"n\">NewBuffer<\/span><span class=\"p\">(<\/span><span class=\"n\">ApplicationData<\/span><span class=\"p\">[<\/span><span class=\"o\">:<\/span><span class=\"m\">2<\/span><span class=\"p\">])<\/span>\n<span class=\"n\">buf2<\/span> <span class=\"o\">:=<\/span> <span class=\"n\">bytes<\/span><span class=\"o\">.<\/span><span class=\"n\">NewBuffer<\/span><span class=\"p\">(<\/span><span class=\"n\">ApplicationData<\/span><span class=\"p\">[<\/span><span class=\"m\">2<\/span><span class=\"o\">:<\/span><span class=\"m\">4<\/span><span class=\"p\">])<\/span>\n<span class=\"n\">binary<\/span><span class=\"o\">.<\/span><span class=\"n\">Read<\/span><span class=\"p\">(<\/span><span class=\"n\">buf<\/span><span class=\"p\">,<\/span> <span class=\"n\">binary<\/span><span class=\"o\">.<\/span><span class=\"n\">BigEndian<\/span><span class=\"p\">,<\/span> <span class=\"o\">&amp;<\/span><span class=\"n\">FrameNumber<\/span><span class=\"p\">)<\/span>\n<span class=\"n\">binary<\/span><span class=\"o\">.<\/span><span class=\"n\">Read<\/span><span class=\"p\">(<\/span><span class=\"n\">buf2<\/span><span class=\"p\">,<\/span> <span class=\"n\">binary<\/span><span class=\"o\">.<\/span><span class=\"n\">BigEndian<\/span><span class=\"p\">,<\/span> <span class=\"o\">&amp;<\/span><span class=\"n\">CurrentChunk<\/span><span class=\"p\">)<\/span>\n\n<span class=\"c\">\/\/ does stuff with ApplicationData[4:]<\/span><\/code><\/pre><\/figure>\n\n<p>Seems simple enough. I wrote a small Rust <code class=\"language-plaintext highlighter-rouge\">struct<\/code> that could chunk up a\nslice of bytes into 1024-byte vectors that fit this format:<\/p>\n\n<figure class=\"highlight\"><pre><code class=\"language-rust\" data-lang=\"rust\"><span class=\"k\">struct<\/span> <span class=\"n\">FrameChunker<\/span><span class=\"o\">&lt;<\/span><span class=\"nv\">'a<\/span><span class=\"o\">&gt;<\/span> <span class=\"p\">{<\/span>\n    <span class=\"n\">inner<\/span><span class=\"p\">:<\/span> <span class=\"o\">&amp;<\/span><span class=\"nv\">'a<\/span> <span class=\"p\">[<\/span><span class=\"nb\">u8<\/span><span class=\"p\">],<\/span> <span class=\"c1\">\/\/ the data to send<\/span>\n    <span class=\"n\">frame_no<\/span><span class=\"p\">:<\/span> <span class=\"nb\">u16<\/span><span class=\"p\">,<\/span> <span class=\"c1\">\/\/ frame number<\/span>\n    <span class=\"n\">seq_no<\/span><span class=\"p\">:<\/span> <span class=\"nb\">u16<\/span> <span class=\"c1\">\/\/ sequence number<\/span>\n<span class=\"p\">}<\/span>\n\n<span class=\"k\">impl<\/span><span class=\"o\">&lt;<\/span><span class=\"nv\">'a<\/span><span class=\"o\">&gt;<\/span> <span class=\"n\">FrameChunker<\/span><span class=\"o\">&lt;<\/span><span class=\"nv\">'a<\/span><span class=\"o\">&gt;<\/span> <span class=\"p\">{<\/span>\n    <span class=\"c1\">\/\/ constructor emitted for brevity<\/span>\n\n    <span class=\"c1\">\/\/ get a new 1024 (or less)-byte chunk to send<\/span>\n    <span class=\"k\">fn<\/span> <span class=\"nf\">next_chunk<\/span><span class=\"p\">(<\/span><span class=\"o\">&amp;<\/span><span class=\"k\">mut<\/span> <span class=\"k\">self<\/span><span class=\"p\">)<\/span> <span class=\"k\">-&gt;<\/span> <span class=\"nb\">Vec<\/span><span class=\"o\">&lt;<\/span><span class=\"nb\">u8<\/span><span class=\"o\">&gt;<\/span> <span class=\"p\">{<\/span>\n        <span class=\"k\">let<\/span> <span class=\"k\">mut<\/span> <span class=\"n\">ret<\/span> <span class=\"o\">=<\/span> <span class=\"nn\">Vec<\/span><span class=\"p\">::<\/span><span class=\"nf\">with_capacity<\/span><span class=\"p\">(<\/span><span class=\"mi\">1024<\/span><span class=\"p\">);<\/span>\n        <span class=\"k\">let<\/span> <span class=\"p\">(<\/span><span class=\"n\">remaining<\/span><span class=\"p\">,<\/span> <span class=\"n\">last<\/span><span class=\"p\">)<\/span> <span class=\"o\">=<\/span> <span class=\"k\">if<\/span> <span class=\"k\">self<\/span><span class=\"py\">.inner<\/span><span class=\"nf\">.len<\/span><span class=\"p\">()<\/span> <span class=\"o\">&gt;<\/span> <span class=\"mi\">1020<\/span> <span class=\"p\">{<\/span>\n            <span class=\"p\">(<\/span><span class=\"mi\">1020<\/span><span class=\"p\">,<\/span> <span class=\"k\">false<\/span><span class=\"p\">)<\/span>\n        <span class=\"p\">}<\/span>\n        <span class=\"k\">else<\/span> <span class=\"p\">{<\/span>\n            <span class=\"p\">(<\/span><span class=\"k\">self<\/span><span class=\"py\">.inner<\/span><span class=\"nf\">.len<\/span><span class=\"p\">(),<\/span> <span class=\"k\">true<\/span><span class=\"p\">)<\/span>\n        <span class=\"p\">};<\/span>\n        <span class=\"c1\">\/\/ this is from the `byteorder` crate's `WriteBytesExt` trait<\/span>\n        <span class=\"n\">ret<\/span><span class=\"py\">.write_u16<\/span><span class=\"p\">::<\/span><span class=\"o\">&lt;<\/span><span class=\"n\">BigEndian<\/span><span class=\"o\">&gt;<\/span><span class=\"p\">(<\/span><span class=\"k\">self<\/span><span class=\"py\">.frame_no<\/span><span class=\"p\">)<\/span><span class=\"nf\">.unwrap<\/span><span class=\"p\">();<\/span>\n        <span class=\"n\">ret<\/span><span class=\"py\">.write_u16<\/span><span class=\"p\">::<\/span><span class=\"o\">&lt;<\/span><span class=\"n\">BigEndian<\/span><span class=\"o\">&gt;<\/span><span class=\"p\">(<\/span><span class=\"k\">self<\/span><span class=\"py\">.seq_no<\/span><span class=\"p\">)<\/span><span class=\"nf\">.unwrap<\/span><span class=\"p\">();<\/span>\n        <span class=\"n\">ret<\/span><span class=\"nf\">.extend<\/span><span class=\"p\">(<\/span><span class=\"o\">&amp;<\/span><span class=\"k\">self<\/span><span class=\"py\">.inner<\/span><span class=\"p\">[<\/span><span class=\"o\">..<\/span><span class=\"n\">remaining<\/span><span class=\"p\">]);<\/span>\n        <span class=\"k\">self<\/span><span class=\"py\">.inner<\/span> <span class=\"o\">=<\/span> <span class=\"o\">&amp;<\/span><span class=\"k\">self<\/span><span class=\"py\">.inner<\/span><span class=\"p\">[<\/span><span class=\"n\">remaining<\/span><span class=\"o\">..<\/span><span class=\"p\">];<\/span>\n        <span class=\"k\">self<\/span><span class=\"py\">.seq_no<\/span> <span class=\"o\">+=<\/span> <span class=\"mi\">1<\/span><span class=\"p\">;<\/span>\n        <span class=\"n\">ret<\/span>\n    <span class=\"p\">}<\/span>\n\n    <span class=\"c1\">\/\/ returns true if we're done transmitting the current frame;<\/span>\n    <span class=\"c1\">\/\/ used to know when to send the next one<\/span>\n    <span class=\"k\">fn<\/span> <span class=\"nf\">is_empty<\/span><span class=\"p\">(<\/span><span class=\"o\">&amp;<\/span><span class=\"k\">self<\/span><span class=\"p\">)<\/span> <span class=\"k\">-&gt;<\/span> <span class=\"nb\">bool<\/span> <span class=\"p\">{<\/span>\n        <span class=\"k\">self<\/span><span class=\"py\">.inner<\/span><span class=\"nf\">.is_empty<\/span><span class=\"p\">()<\/span>\n    <span class=\"p\">}<\/span>\n<span class=\"p\">}<\/span><\/code><\/pre><\/figure>\n\n<p>I then made a little 1920\u00d71080 all white JPEG in GIMP, wrote it to a file, and\nwrote some code to send out UDP multicast packets in the format the receiver\n(and <code class=\"language-plaintext highlighter-rouge\">de-ip-hdmi<\/code>) would expect:<\/p>\n\n<figure class=\"highlight\"><pre><code class=\"language-rust\" data-lang=\"rust\"><span class=\"c1\">\/\/ The magic IPv4 address for the multicast group the transmitter would send<\/span>\n<span class=\"c1\">\/\/ to.<\/span>\n<span class=\"k\">const<\/span> <span class=\"n\">MAGIC_ADDRESS<\/span><span class=\"p\">:<\/span> <span class=\"n\">Ipv4Addr<\/span> <span class=\"o\">=<\/span> <span class=\"nn\">Ipv4Addr<\/span><span class=\"p\">::<\/span><span class=\"nf\">new<\/span><span class=\"p\">(<\/span><span class=\"mi\">226<\/span><span class=\"p\">,<\/span> <span class=\"mi\">2<\/span><span class=\"p\">,<\/span> <span class=\"mi\">2<\/span><span class=\"p\">,<\/span> <span class=\"mi\">2<\/span><span class=\"p\">);<\/span>\n\n<span class=\"k\">fn<\/span> <span class=\"nf\">main<\/span><span class=\"p\">()<\/span> <span class=\"p\">{<\/span>\n    <span class=\"c1\">\/\/ load the test image data in at compile-time<\/span>\n    <span class=\"k\">let<\/span> <span class=\"n\">test_data<\/span> <span class=\"o\">=<\/span> <span class=\"nd\">include_bytes!<\/span><span class=\"p\">(<\/span><span class=\"s\">\"..\/white.mjpeg\"<\/span><span class=\"p\">);<\/span>\n\n    <span class=\"c1\">\/\/ make a UDP socket for sending<\/span>\n    <span class=\"c1\">\/\/ note: this requires creating an interface with this address<\/span>\n    <span class=\"k\">let<\/span> <span class=\"n\">mjpeg<\/span> <span class=\"o\">=<\/span> <span class=\"nn\">UdpSocket<\/span><span class=\"p\">::<\/span><span class=\"nf\">bind<\/span><span class=\"p\">(<\/span><span class=\"s\">\"192.168.168.55:2068\"<\/span><span class=\"p\">)<\/span><span class=\"nf\">.unwrap<\/span><span class=\"p\">();<\/span>\n    <span class=\"c1\">\/\/ destination address:port for video picture data<\/span>\n    <span class=\"k\">let<\/span> <span class=\"n\">mjpeg_dest<\/span><span class=\"p\">:<\/span> <span class=\"n\">SocketAddr<\/span> <span class=\"o\">=<\/span> <span class=\"s\">\"226.2.2.2:2068\"<\/span><span class=\"nf\">.parse<\/span><span class=\"p\">()<\/span><span class=\"nf\">.unwrap<\/span><span class=\"p\">();<\/span>\n\n    <span class=\"c1\">\/\/ join the multicast group<\/span>\n    <span class=\"n\">mjpeg<\/span><span class=\"nf\">.join_multicast_v4<\/span><span class=\"p\">(<\/span><span class=\"o\">&amp;<\/span><span class=\"n\">MAGIC_ADDRESS<\/span><span class=\"p\">,<\/span> <span class=\"o\">&amp;<\/span><span class=\"s\">\"192.168.168.55\"<\/span><span class=\"nf\">.parse<\/span><span class=\"p\">()<\/span><span class=\"nf\">.unwrap<\/span><span class=\"p\">())<\/span><span class=\"nf\">.unwrap<\/span><span class=\"p\">();<\/span>\n\n    <span class=\"c1\">\/\/ current frame number<\/span>\n    <span class=\"k\">let<\/span> <span class=\"k\">mut<\/span> <span class=\"n\">frame_no<\/span> <span class=\"o\">=<\/span> <span class=\"mi\">0<\/span><span class=\"p\">;<\/span>\n    <span class=\"k\">loop<\/span> <span class=\"p\">{<\/span>\n        <span class=\"k\">let<\/span> <span class=\"k\">mut<\/span> <span class=\"n\">chonker<\/span> <span class=\"o\">=<\/span> <span class=\"nn\">FrameChunker<\/span><span class=\"p\">::<\/span><span class=\"nf\">new<\/span><span class=\"p\">(<\/span><span class=\"n\">frame_no<\/span><span class=\"p\">,<\/span> <span class=\"n\">image<\/span><span class=\"p\">);<\/span>\n        <span class=\"k\">while<\/span> <span class=\"o\">!<\/span><span class=\"n\">chonker<\/span><span class=\"nf\">.is_empty<\/span><span class=\"p\">()<\/span> <span class=\"p\">{<\/span>\n            <span class=\"k\">let<\/span> <span class=\"n\">buf<\/span> <span class=\"o\">=<\/span> <span class=\"n\">chonker<\/span><span class=\"nf\">.next_chunk<\/span><span class=\"p\">();<\/span>\n            <span class=\"n\">mjpeg<\/span><span class=\"nf\">.send_to<\/span><span class=\"p\">(<\/span><span class=\"o\">&amp;<\/span><span class=\"n\">buf<\/span><span class=\"p\">,<\/span> <span class=\"n\">mjpeg_dest<\/span><span class=\"p\">)<\/span><span class=\"nf\">.unwrap<\/span><span class=\"p\">();<\/span>\n        <span class=\"p\">}<\/span>\n        <span class=\"n\">frame_no<\/span> <span class=\"o\">+=<\/span> <span class=\"mi\">1<\/span><span class=\"p\">;<\/span>\n\n        <span class=\"p\">::<\/span><span class=\"nn\">std<\/span><span class=\"p\">::<\/span><span class=\"nn\">thread<\/span><span class=\"p\">::<\/span><span class=\"nf\">sleep_ms<\/span><span class=\"p\">(<\/span><span class=\"mi\">33<\/span><span class=\"p\">);<\/span>\n    <span class=\"p\">}<\/span>\n<span class=\"p\">}<\/span><\/code><\/pre><\/figure>\n\n<p>I added a suitable network interface for the sender to use (the actual flat networking\nsetup uses <a href=\"https:\/\/en.wikipedia.org\/wiki\/Virtual_LAN\">VLAN<\/a>s for the HDMI\ntraffic, but I found out that if you just want to test then adding a loopback interface also\nworks):<\/p>\n\n<div class=\"language-plaintext highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code>$ sudo ip link add link enp5s0f3u1u1 name showtime0 type vlan id 1024\n$ sudo ip link set dev showtime0 address 00:0b:78:00:60:01\n$ sudo ip addr add 192.168.168.55\/32 dev showtime0\n<\/code><\/pre><\/div><\/div>\n\n<p>With much anticipation I then ran the code for the first time, with <code class=\"language-plaintext highlighter-rouge\">de-ip-hdmi<\/code>\nrunning in parallel \u2013 and got\u2026nothing.<\/p>\n\n<h3 id=\"adventures-in-bitshifting\">Adventures in bitshifting<\/h3>\n\n<p>Turns out I forgot something else in the packet format: a way to signal the\nlast chunk in a series of chunks for a frame. Going back to <code class=\"language-plaintext highlighter-rouge\">de-ip-hdmi<\/code>:<\/p>\n\n<figure class=\"highlight\"><pre><code class=\"language-go\" data-lang=\"go\"><span class=\"k\">if<\/span> <span class=\"kt\">uint16<\/span><span class=\"p\">(<\/span><span class=\"o\">^<\/span><span class=\"p\">(<\/span><span class=\"n\">CurrentChunk<\/span> <span class=\"o\">&gt;&gt;<\/span> <span class=\"m\">15<\/span><span class=\"p\">))<\/span> <span class=\"o\">==<\/span> <span class=\"m\">65534<\/span> <span class=\"p\">{<\/span>\n\t<span class=\"c\">\/\/ Flush the frame to output<\/span>\n\t<span class=\"c\">\/\/ [code snipped]<\/span>\n\n\t<span class=\"n\">CurrentPacket<\/span> <span class=\"o\">=<\/span> <span class=\"n\">Frame<\/span><span class=\"p\">{}<\/span>\n\t<span class=\"n\">CurrentPacket<\/span><span class=\"o\">.<\/span><span class=\"n\">Data<\/span> <span class=\"o\">=<\/span> <span class=\"nb\">make<\/span><span class=\"p\">([]<\/span><span class=\"kt\">byte<\/span><span class=\"p\">,<\/span> <span class=\"m\">0<\/span><span class=\"p\">)<\/span>\n\t<span class=\"n\">CurrentPacket<\/span><span class=\"o\">.<\/span><span class=\"n\">FrameID<\/span> <span class=\"o\">=<\/span> <span class=\"m\">0<\/span>\n\t<span class=\"n\">CurrentPacket<\/span><span class=\"o\">.<\/span><span class=\"n\">LastChunk<\/span> <span class=\"o\">=<\/span> <span class=\"m\">0<\/span>\n<span class=\"p\">}<\/span>    <\/code><\/pre><\/figure>\n\n<p>What the heck is <code class=\"language-plaintext highlighter-rouge\">uint16(^(CurrentChunk &gt;&gt; 15)) == 65534<\/code> doing? Let\u2019s break\nthat down:<\/p>\n\n<ul>\n  <li>it shifts <code class=\"language-plaintext highlighter-rouge\">CurrentChunk<\/code> to the right by 15 bits, discarding all bits apart\nfrom the most significant bit (MSB), which becomes the new least significant\nbit (LSB).<\/li>\n  <li>unary <code class=\"language-plaintext highlighter-rouge\">^<\/code> (caret) is the Go \u201clogical NOT\u201d operator, so we flip all of the\nbits in the result above<\/li>\n  <li><code class=\"language-plaintext highlighter-rouge\">65534<\/code> is a 16-bit integer with every bit set except the LSB<\/li>\n<\/ul>\n\n<p><img src=\"\/assets\/img\/showtime\/bitshift.png\" alt=\"bit shift diagrammatic explanation\" \/><\/p>\n\n<p>So\u2026 this is a <a href=\"https:\/\/mobile.twitter.com\/Benjojo12\/status\/1400616243300872193\">very long-winded<\/a>\nway of checking whether the MSB of\n<code class=\"language-plaintext highlighter-rouge\">CurrentChunk<\/code> is equal to 1 (in Rust, <code class=\"language-plaintext highlighter-rouge\">(number &amp; 0b1000_0000_0000_0000) &gt; 0<\/code>).\nAccordingly, we need to set the MSB to 1 to denote the end of a frame, so\nlet\u2019s do that in the <code class=\"language-plaintext highlighter-rouge\">FrameChunker<\/code>:<\/p>\n\n<figure class=\"highlight\"><pre><code class=\"language-rust\" data-lang=\"rust\"><span class=\"n\">ret<\/span><span class=\"py\">.write_u16<\/span><span class=\"p\">::<\/span><span class=\"o\">&lt;<\/span><span class=\"n\">BigEndian<\/span><span class=\"o\">&gt;<\/span><span class=\"p\">(<\/span><span class=\"k\">self<\/span><span class=\"py\">.frame_no<\/span><span class=\"p\">)<\/span><span class=\"nf\">.unwrap<\/span><span class=\"p\">();<\/span>\n<span class=\"n\">ret<\/span><span class=\"py\">.write_u16<\/span><span class=\"p\">::<\/span><span class=\"o\">&lt;<\/span><span class=\"n\">BigEndian<\/span><span class=\"o\">&gt;<\/span><span class=\"p\">(<\/span><span class=\"k\">self<\/span><span class=\"py\">.seq_no<\/span><span class=\"p\">)<\/span><span class=\"nf\">.unwrap<\/span><span class=\"p\">();<\/span>\n<span class=\"k\">if<\/span> <span class=\"n\">last<\/span> <span class=\"p\">{<\/span>\n    <span class=\"c1\">\/\/ it's big-endian, so ret[2] contains the most significant<\/span>\n    <span class=\"c1\">\/\/ 8 bits of the 16-bit integer<\/span>\n    <span class=\"n\">ret<\/span><span class=\"p\">[<\/span><span class=\"mi\">2<\/span><span class=\"p\">]<\/span> <span class=\"p\">|<\/span><span class=\"o\">=<\/span> <span class=\"mi\">0b1000_0000<\/span><span class=\"p\">;<\/span>\n<span class=\"p\">}<\/span><\/code><\/pre><\/figure>\n\n<h3 id=\"partial-victory\">Partial victory!<\/h3>\n\n<p>With this change applied <code class=\"language-plaintext highlighter-rouge\">de-ip-hdmi<\/code> would decode our image! Yay!<\/p>\n\n<video controls=\"\" style=\"max-width: 100%\" autoplay=\"\" loop=\"\" muted=\"\">\n    <source src=\"\/assets\/img\/showtime\/deip.mp4\" type=\"video\/mp4\" \/>\n\n    Sorry, your browser doesn't support embedded videos.\n<\/video>\n\n<p>However, the real transmitter wasn\u2019t having any of it; it refused to display a\npicture (behaving as if it couldn\u2019t see any transmitters; it says \u201cSearching\nTX\u201d when this happens).<\/p>\n\n<p><img src=\"\/assets\/img\/showtime\/searching-tx.png\" alt=\"&quot;Searching TX&quot; display\" \/><\/p>\n\n<p>Hmm.<\/p>\n\n<h2 id=\"lets-do-the-wireshark-again\">Let\u2019s do the Wireshark again<\/h2>\n\n<p>I connected a real transmitter and dumped some of the packets it sends using\n<a href=\"https:\/\/www.wireshark.org\/\">Wireshark<\/a>. Turns out there\u2019s more stuff you need\nto do.<\/p>\n\n<h3 id=\"mysterious-packets-on-port-2067\">Mysterious packets on port 2067<\/h3>\n\n<p>Here\u2019s a screenshot of some of the packets the real transmitter sends.<\/p>\n\n<p><img src=\"\/assets\/img\/showtime\/tasty-packet.png\" alt=\"wireshark screenshot\" \/><\/p>\n\n<ul>\n  <li>the (source port) 2065 \u2192 (destination port) 2066 packets are audio data.\nWe\u2019ll get to those later!<\/li>\n  <li>the 2068 \u2192 2068 packets are video picture data<\/li>\n  <li>\u2026but what are these 2067 \u2192 2067 packets?<\/li>\n<\/ul>\n\n<p><img src=\"\/assets\/img\/showtime\/2067-1.png\" alt=\"2067 packet 1\" \/><\/p>\n\n<p>Hmm, it\u2019s all zeroes. What about the next one?<\/p>\n\n<p><img src=\"\/assets\/img\/showtime\/2067-2.png\" alt=\"2067 packet 2\" \/><\/p>\n\n<p>Aha, a byte went from 00 to 01!<\/p>\n\n<p>Turns out this pattern continues; before each picture frame is sent, an\nadditional packet is sent on port 2067 of length 20, with the 5th and 6th\nbytes being a 16-bit big-endian integer, which is the same as the frame\nnumber of the frame we just sent.<\/p>\n\n<p>I\u2019m guessing this is a sort of <a href=\"https:\/\/en.wikipedia.org\/wiki\/Vertical_blanking_interval\">\u201cvsync\u201d \/ \u201cvblank\u201d packet<\/a>,\nused to tell the\nreceiver to get ready for a new frame.\nWe should probably replicate it, so I wrote some code:<\/p>\n\n<figure class=\"highlight\"><pre><code class=\"language-rust\" data-lang=\"rust\"><span class=\"k\">fn<\/span> <span class=\"nf\">make_vsync<\/span><span class=\"p\">(<\/span><span class=\"n\">frame_no<\/span><span class=\"p\">:<\/span> <span class=\"nb\">u16<\/span><span class=\"p\">)<\/span> <span class=\"k\">-&gt;<\/span> <span class=\"nb\">Vec<\/span><span class=\"o\">&lt;<\/span><span class=\"nb\">u8<\/span><span class=\"o\">&gt;<\/span> <span class=\"p\">{<\/span>\n    <span class=\"k\">let<\/span> <span class=\"k\">mut<\/span> <span class=\"n\">ret<\/span> <span class=\"o\">=<\/span> <span class=\"nn\">Vec<\/span><span class=\"p\">::<\/span><span class=\"nf\">with_capacity<\/span><span class=\"p\">(<\/span><span class=\"mi\">20<\/span><span class=\"p\">);<\/span>\n    <span class=\"n\">ret<\/span><span class=\"nf\">.write_all<\/span><span class=\"p\">(<\/span><span class=\"o\">&amp;<\/span><span class=\"p\">[<\/span><span class=\"mi\">0<\/span><span class=\"p\">,<\/span> <span class=\"mi\">0<\/span><span class=\"p\">,<\/span> <span class=\"mi\">0<\/span><span class=\"p\">,<\/span> <span class=\"mi\">0<\/span><span class=\"p\">])<\/span><span class=\"nf\">.unwrap<\/span><span class=\"p\">();<\/span>\n    <span class=\"n\">ret<\/span><span class=\"py\">.write_u16<\/span><span class=\"p\">::<\/span><span class=\"o\">&lt;<\/span><span class=\"n\">BigEndian<\/span><span class=\"o\">&gt;<\/span><span class=\"p\">(<\/span><span class=\"n\">frame_no<\/span><span class=\"p\">)<\/span><span class=\"nf\">.unwrap<\/span><span class=\"p\">();<\/span>\n    <span class=\"k\">for<\/span> <span class=\"n\">_<\/span> <span class=\"k\">in<\/span> <span class=\"mi\">0<\/span><span class=\"o\">..<\/span><span class=\"mi\">14<\/span> <span class=\"p\">{<\/span>\n        <span class=\"n\">ret<\/span><span class=\"nf\">.write_all<\/span><span class=\"p\">(<\/span><span class=\"o\">&amp;<\/span><span class=\"p\">[<\/span><span class=\"mi\">0<\/span><span class=\"p\">])<\/span><span class=\"nf\">.unwrap<\/span><span class=\"p\">();<\/span>\n    <span class=\"p\">}<\/span>\n    <span class=\"n\">ret<\/span>\n<span class=\"p\">}<\/span>\n<span class=\"c1\">\/\/ code to send this to port 2066 before sending frame packets elided<\/span><\/code><\/pre><\/figure>\n\n<p>Alas, this was not enough. The receiver continued to say \u201cSearching TX\u201d.<\/p>\n\n<h2 id=\"heartbeat-packets\">Heartbeat packets<\/h2>\n\n<p><img src=\"\/assets\/img\/showtime\/heartbeat-list.png\" alt=\"heartbeat packet list\" \/><\/p>\n\n<p>There are also some weird 512-byte payloads being sent on port 48689. These\nare easy to miss because they\u2019re only sent once every second (and so are\ncomparatively infrequent relative to the firehose of data the transmitters\nsend for pictures and video). However, you can notice them by looking at\nthe start of a packet capture since they get sent even when no picture\ndata is being sent.<\/p>\n\n<p>Let\u2019s look at a dump of some of these packet payloads over time (using\n\u201c[right click] Follow \u2192 UDP Stream\u201d in Wireshark):<\/p>\n\n<p><img src=\"\/assets\/img\/showtime\/heartbeat.png\" alt=\"heartbeat packet diagram\" \/><\/p>\n\n<p>After squinting at it for a while, you notice a few things:<\/p>\n\n<ul>\n  <li>The first three packets are identical, apart from a small 2-byte region that\ncarries on going up with each new packet (marked in red), and a sequence\nnumber incremented with each new packet (marked in pink).<\/li>\n  <li>Between the third and fourth packets, a large chunk of data (marked in blue)\nchanges to something else (marked in green), and then stays that way.<\/li>\n  <li>This change is also correlated with the video and audio starting.<\/li>\n  <li>After the change happens the packets stay identical apart from the counter.<\/li>\n<\/ul>\n\n<p>This must be some kind of heartbeat packet sent by the transmitter to tell\nthe receiver that it\u2019s there. Let\u2019s try and figure out the various bits\nin the payload in more detail.<\/p>\n\n<h3 id=\"incrementing-counter\">Incrementing counter<\/h3>\n\n<p>How is this counter being generated? Well, let\u2019s assume that\nit\u2019s a 16-bit big-endian integer, since\neverything else in this blog post has been that, and interpret the values\naccordingly.<\/p>\n\n<ul>\n  <li><code class=\"language-plaintext highlighter-rouge\">0x0458<\/code> is 1112, sent at time <em>t<\/em> = 0.0<\/li>\n  <li><code class=\"language-plaintext highlighter-rouge\">0x0842<\/code> is 2114 (delta = 1002), sent at <em>t<\/em> = 1.001<\/li>\n  <li><code class=\"language-plaintext highlighter-rouge\">0x0c2b<\/code> is 3115 (delta = 1001), sent at <em>t<\/em> = 2.002<\/li>\n  <li><code class=\"language-plaintext highlighter-rouge\">0x0ee5<\/code> is 3813 (delta = 698), sent at <em>t<\/em> = 2.700<\/li>\n<\/ul>\n\n<p><em>Hmm\u2026<\/em><\/p>\n\n<p>I think it\u2019s pretty reasonable to assume this counter just tracks time since\nthe transmitter was powered on. <em>Why<\/em> the receiver needs that information\nisn\u2019t exactly clear though (maybe for some synchronization purposes?).<\/p>\n\n<p>Anyway, we should probably do the same as the transmitter when sending our\nown packets here so we\u2019ll add a timer to the code.<\/p>\n\n<h3 id=\"other-gubbins\">Other gubbins<\/h3>\n\n<p>We have two sets of weird data:<\/p>\n\n<ul>\n  <li><code class=\"language-plaintext highlighter-rouge\">00 03 07 80 04 38 02 57 07 80 04 38 00 78 03<\/code> (when the HDMI port is active)<\/li>\n  <li><code class=\"language-plaintext highlighter-rouge\">00 10 00 00 00 00 00 00 00 00 00 00 00 78 00<\/code> (when it isn\u2019t)<\/li>\n<\/ul>\n\n<p><code class=\"language-plaintext highlighter-rouge\">07 80<\/code> and <code class=\"language-plaintext highlighter-rouge\">04 38<\/code> are repeated multiple times. Let\u2019s try decoding those\nas 16-bit big-endian integers:<\/p>\n\n<ul>\n  <li><code class=\"language-plaintext highlighter-rouge\">0x0780<\/code> is 1920<\/li>\n  <li><code class=\"language-plaintext highlighter-rouge\">0x0438<\/code> is 1080<\/li>\n<\/ul>\n\n<p>That seems familiar; isn\u2019t 1920\u00d71080 the resolution for <a href=\"https:\/\/en.wikipedia.org\/wiki\/1080p\">1080p<\/a>?\nThis data is presumably used to initialize the HDMI port on the other end.<\/p>\n\n<p>What about when the HDMI is off? Well, it looks like most of the\nfields are just zeroed out, which makes sense since it doesn\u2019t have a resolution\nor anything to send yet.\n(This causes the message \u201cCheck Tx\u2019s input signal\u201d to appear on the receiver, which\nis useful for debugging why the receiver isn\u2019t displaying anything.)<\/p>\n\n<p>In the end, I just encoded all of this data as constants (interpreting it\nas a set of 16-bit big endian integers), sending one set (the one with 1920s\nin it) if we\u2019re ready to send data, and another set (the zeroed out set)\nif we aren\u2019t.<\/p>\n\n<h3 id=\"putting-it-all-together\">Putting it all together<\/h3>\n\n<p>The code to make a heartbeat packet ended up looking like this:<\/p>\n\n<figure class=\"highlight\"><pre><code class=\"language-rust\" data-lang=\"rust\"><span class=\"k\">fn<\/span> <span class=\"nf\">make_heartbeat<\/span><span class=\"p\">(<\/span><span class=\"n\">seq_no<\/span><span class=\"p\">:<\/span> <span class=\"nb\">u16<\/span><span class=\"p\">,<\/span> <span class=\"n\">cur_ts<\/span><span class=\"p\">:<\/span> <span class=\"nb\">u16<\/span><span class=\"p\">,<\/span> <span class=\"n\">init<\/span><span class=\"p\">:<\/span> <span class=\"nb\">bool<\/span><span class=\"p\">)<\/span> <span class=\"k\">-&gt;<\/span> <span class=\"nb\">Vec<\/span><span class=\"o\">&lt;<\/span><span class=\"nb\">u8<\/span><span class=\"o\">&gt;<\/span> <span class=\"p\">{<\/span>\n    <span class=\"k\">let<\/span> <span class=\"k\">mut<\/span> <span class=\"n\">ret<\/span> <span class=\"o\">=<\/span> <span class=\"nn\">Vec<\/span><span class=\"p\">::<\/span><span class=\"nf\">with_capacity<\/span><span class=\"p\">(<\/span><span class=\"mi\">512<\/span><span class=\"p\">);<\/span>\n    <span class=\"n\">ret<\/span><span class=\"nf\">.write_all<\/span><span class=\"p\">(<\/span><span class=\"o\">&amp;<\/span><span class=\"p\">[<\/span><span class=\"mi\">0x54<\/span><span class=\"p\">,<\/span> <span class=\"mi\">0x46<\/span><span class=\"p\">,<\/span> <span class=\"mi\">0x36<\/span><span class=\"p\">,<\/span> <span class=\"mi\">0x7a<\/span><span class=\"p\">,<\/span> <span class=\"mi\">0x63<\/span><span class=\"p\">,<\/span> <span class=\"mi\">0x01<\/span><span class=\"p\">,<\/span> <span class=\"mi\">0x00<\/span><span class=\"p\">])<\/span><span class=\"nf\">.unwrap<\/span><span class=\"p\">();<\/span>\n    <span class=\"n\">ret<\/span><span class=\"py\">.write_u16<\/span><span class=\"p\">::<\/span><span class=\"o\">&lt;<\/span><span class=\"n\">BigEndian<\/span><span class=\"o\">&gt;<\/span><span class=\"p\">(<\/span><span class=\"n\">seq_no<\/span><span class=\"p\">)<\/span><span class=\"nf\">.unwrap<\/span><span class=\"p\">();<\/span>\n    <span class=\"n\">ret<\/span><span class=\"nf\">.write_all<\/span><span class=\"p\">(<\/span><span class=\"o\">&amp;<\/span><span class=\"p\">[<\/span><span class=\"mi\">0x00<\/span><span class=\"p\">,<\/span> <span class=\"mi\">0x00<\/span><span class=\"p\">,<\/span> <span class=\"mi\">0x03<\/span><span class=\"p\">,<\/span> <span class=\"mi\">0x03<\/span><span class=\"p\">,<\/span> <span class=\"mi\">0x03<\/span><span class=\"p\">,<\/span> <span class=\"mi\">0x00<\/span><span class=\"p\">,<\/span> <span class=\"mi\">0x24<\/span><span class=\"p\">,<\/span> <span class=\"mi\">0x00<\/span><span class=\"p\">,<\/span> <span class=\"mi\">0x00<\/span><span class=\"p\">])<\/span><span class=\"nf\">.unwrap<\/span><span class=\"p\">();<\/span>\n    <span class=\"k\">for<\/span> <span class=\"n\">_<\/span> <span class=\"k\">in<\/span> <span class=\"mi\">0<\/span><span class=\"o\">..<\/span><span class=\"mi\">8<\/span> <span class=\"p\">{<\/span>\n        <span class=\"n\">ret<\/span><span class=\"nf\">.write_all<\/span><span class=\"p\">(<\/span><span class=\"o\">&amp;<\/span><span class=\"p\">[<\/span><span class=\"mi\">0<\/span><span class=\"p\">])<\/span><span class=\"nf\">.unwrap<\/span><span class=\"p\">();<\/span>\n    <span class=\"p\">}<\/span>\n    <span class=\"c1\">\/\/ woohoo, magic numbers<\/span>\n    <span class=\"k\">let<\/span> <span class=\"p\">(<\/span><span class=\"n\">mode<\/span><span class=\"p\">,<\/span> <span class=\"n\">a<\/span><span class=\"p\">,<\/span> <span class=\"n\">b<\/span><span class=\"p\">,<\/span> <span class=\"n\">c<\/span><span class=\"p\">,<\/span> <span class=\"n\">d<\/span><span class=\"p\">,<\/span> <span class=\"n\">e<\/span><span class=\"p\">,<\/span> <span class=\"n\">f<\/span><span class=\"p\">,<\/span> <span class=\"n\">other<\/span><span class=\"p\">)<\/span> <span class=\"o\">=<\/span> <span class=\"k\">if<\/span> <span class=\"n\">init<\/span> <span class=\"p\">{<\/span>\n        <span class=\"p\">(<\/span><span class=\"mi\">3<\/span><span class=\"p\">,<\/span> <span class=\"mi\">1920<\/span><span class=\"p\">,<\/span> <span class=\"mi\">1080<\/span><span class=\"p\">,<\/span> <span class=\"mi\">599<\/span><span class=\"p\">,<\/span> <span class=\"mi\">1920<\/span><span class=\"p\">,<\/span> <span class=\"mi\">1080<\/span><span class=\"p\">,<\/span> <span class=\"mi\">120<\/span><span class=\"p\">,<\/span> <span class=\"mi\">3<\/span><span class=\"p\">)<\/span>\n    <span class=\"p\">}<\/span>\n    <span class=\"k\">else<\/span> <span class=\"p\">{<\/span>\n        <span class=\"p\">(<\/span><span class=\"mi\">16<\/span><span class=\"p\">,<\/span> <span class=\"mi\">0<\/span><span class=\"p\">,<\/span> <span class=\"mi\">0<\/span><span class=\"p\">,<\/span> <span class=\"mi\">0<\/span><span class=\"p\">,<\/span> <span class=\"mi\">0<\/span><span class=\"p\">,<\/span> <span class=\"mi\">0<\/span><span class=\"p\">,<\/span> <span class=\"mi\">120<\/span><span class=\"p\">,<\/span> <span class=\"mi\">0<\/span><span class=\"p\">)<\/span>\n    <span class=\"p\">};<\/span>\n    <span class=\"k\">for<\/span> <span class=\"n\">thing<\/span> <span class=\"k\">in<\/span> <span class=\"o\">&amp;<\/span><span class=\"p\">[<\/span><span class=\"n\">mode<\/span><span class=\"p\">,<\/span> <span class=\"n\">a<\/span><span class=\"p\">,<\/span> <span class=\"n\">b<\/span><span class=\"p\">,<\/span> <span class=\"n\">c<\/span><span class=\"p\">,<\/span> <span class=\"n\">d<\/span><span class=\"p\">,<\/span> <span class=\"n\">e<\/span><span class=\"p\">,<\/span> <span class=\"n\">f<\/span><span class=\"p\">]<\/span> <span class=\"p\">{<\/span>\n        <span class=\"n\">ret<\/span><span class=\"py\">.write_u16<\/span><span class=\"p\">::<\/span><span class=\"o\">&lt;<\/span><span class=\"n\">BigEndian<\/span><span class=\"o\">&gt;<\/span><span class=\"p\">(<\/span><span class=\"o\">*<\/span><span class=\"n\">thing<\/span><span class=\"p\">)<\/span><span class=\"nf\">.unwrap<\/span><span class=\"p\">();<\/span>\n    <span class=\"p\">}<\/span>\n    <span class=\"n\">ret<\/span><span class=\"nf\">.write_all<\/span><span class=\"p\">(<\/span><span class=\"o\">&amp;<\/span><span class=\"p\">[<\/span><span class=\"mi\">0<\/span><span class=\"p\">,<\/span> <span class=\"mi\">0<\/span><span class=\"p\">])<\/span><span class=\"nf\">.unwrap<\/span><span class=\"p\">();<\/span>\n    <span class=\"n\">ret<\/span><span class=\"py\">.write_u16<\/span><span class=\"p\">::<\/span><span class=\"o\">&lt;<\/span><span class=\"n\">BigEndian<\/span><span class=\"o\">&gt;<\/span><span class=\"p\">(<\/span><span class=\"n\">cur_ts<\/span><span class=\"p\">)<\/span><span class=\"nf\">.unwrap<\/span><span class=\"p\">();<\/span>\n    <span class=\"n\">ret<\/span><span class=\"nf\">.write_all<\/span><span class=\"p\">(<\/span><span class=\"o\">&amp;<\/span><span class=\"p\">[<\/span><span class=\"mi\">0<\/span><span class=\"p\">,<\/span> <span class=\"mi\">1<\/span><span class=\"p\">,<\/span> <span class=\"mi\">0<\/span><span class=\"p\">,<\/span> <span class=\"mi\">0<\/span><span class=\"p\">,<\/span> <span class=\"mi\">0<\/span><span class=\"p\">,<\/span> <span class=\"mi\">0<\/span><span class=\"p\">,<\/span> <span class=\"n\">other<\/span><span class=\"p\">,<\/span> <span class=\"mi\">0x0a<\/span><span class=\"p\">])<\/span><span class=\"nf\">.unwrap<\/span><span class=\"p\">();<\/span>\n    <span class=\"k\">for<\/span> <span class=\"n\">_<\/span> <span class=\"k\">in<\/span> <span class=\"n\">ret<\/span><span class=\"nf\">.len<\/span><span class=\"p\">()<\/span><span class=\"o\">..<\/span><span class=\"mi\">512<\/span> <span class=\"p\">{<\/span>\n        <span class=\"n\">ret<\/span><span class=\"nf\">.write_all<\/span><span class=\"p\">(<\/span><span class=\"o\">&amp;<\/span><span class=\"p\">[<\/span><span class=\"mi\">0<\/span><span class=\"p\">])<\/span><span class=\"nf\">.unwrap<\/span><span class=\"p\">();<\/span>\n    <span class=\"p\">}<\/span>\n    <span class=\"nd\">assert_eq!<\/span><span class=\"p\">(<\/span><span class=\"n\">ret<\/span><span class=\"nf\">.len<\/span><span class=\"p\">(),<\/span> <span class=\"mi\">512<\/span><span class=\"p\">);<\/span>\n    <span class=\"n\">ret<\/span>\n<span class=\"p\">}<\/span><\/code><\/pre><\/figure>\n\n<p>Putting it all together and sending heartbeat packets, vsync packets, and\nthe actual picture data\u2026 got the receiver to initialize, but output a black rectangle\ninstead of the test image.<\/p>\n\n<video controls=\"\" style=\"max-width: 100%\" autoplay=\"\" loop=\"\" muted=\"\">\n    <source src=\"\/assets\/img\/showtime\/blackscreen.mp4\" type=\"video\/mp4\" \/>\n\n    Sorry, your browser doesn't support embedded videos.\n<\/video>\n\n<p>After trying many other things, I eventually tried sending the original <code class=\"language-plaintext highlighter-rouge\">test.mjpeg<\/code> file\nI\u2019d dumped from the receiver when first trying to get this to work, instead of my own\nJPEG that I made in GIMP. That actually produced some output!<\/p>\n\n<video controls=\"\" style=\"max-width: 100%\" autoplay=\"\" loop=\"\" muted=\"\">\n    <source src=\"\/assets\/img\/showtime\/functional.mp4\" type=\"video\/mp4\" \/>\n\n    Sorry, your browser doesn't support embedded videos.\n<\/video>\n\n<p>Yay!<\/p>\n\n<p><em>(the test image was produced by writing some text on a laptop, making it full-screen, and connecting it to the transmitter)<\/em><\/p>\n\n<h2 id=\"but-what-about-audio\">But what about audio?<\/h2>\n\n<p>While this is (somewhat) useful, it doesn\u2019t do what we originally wanted; what about actually sending audio?<\/p>\n\n<h3 id=\"how-does-digital-audio-work-anyway\">How does digital audio work, anyway?<\/h3>\n\n<p>Digital audio is represented as a series of <a href=\"https:\/\/en.wikipedia.org\/wiki\/Sampling_(signal_processing)\"><em>samples<\/em><\/a> \u2013 individual measurements\nof what the voltage of your speaker should be at a given point in time (the \u201csignal\u201d), taken at a given <em>sampling rate<\/em> that\u2019s high enough to provide\na decent approximation of what the speaker should be doing to reproduce your favourite tunes.<\/p>\n\n<p><img src=\"\/assets\/img\/showtime\/sampling.png\" alt=\"sampling visualization\" \/><\/p>\n\n<p><em>image: sourced from the above linked Wikipedia article, CC0<\/em><\/p>\n\n<p>There are a number of important variables involved in this process that we need to know in order to send and receive audio in a given format:<\/p>\n\n<ul>\n  <li>what the sample rate was (measured in Hertz, aka \u201ctimes per second\u201d)<\/li>\n  <li>how the samples are stored (as floating-point numbers? as signed integers? how many bits?)<\/li>\n  <li>what the \u201cmaximum\u201d and \u201cminimum\u201d values are (for floats, we typically use <code class=\"language-plaintext highlighter-rouge\">1.0<\/code> and <code class=\"language-plaintext highlighter-rouge\">-1.0<\/code>; integer formats use <code class=\"language-plaintext highlighter-rouge\">INT_MAX<\/code> and <code class=\"language-plaintext highlighter-rouge\">INT_MIN<\/code>)<\/li>\n<\/ul>\n\n<h3 id=\"what-format-do-the-extenders-use\">What format do the extenders use?<\/h3>\n\n<p><code class=\"language-plaintext highlighter-rouge\">de-ip-hdmi<\/code> has already figured this out for us:<\/p>\n\n<figure class=\"highlight\"><pre><code class=\"language-go\" data-lang=\"go\"><span class=\"k\">if<\/span> <span class=\"n\">audio<\/span> <span class=\"p\">{<\/span>\n    <span class=\"n\">ffmpeg<\/span> <span class=\"o\">=<\/span> <span class=\"n\">exec<\/span><span class=\"o\">.<\/span><span class=\"n\">Command<\/span><span class=\"p\">(<\/span><span class=\"s\">\"ffmpeg\"<\/span><span class=\"p\">,<\/span> <span class=\"s\">\"-f\"<\/span><span class=\"p\">,<\/span> <span class=\"s\">\"mjpeg\"<\/span><span class=\"p\">,<\/span> <span class=\"s\">\"-i\"<\/span><span class=\"p\">,<\/span> <span class=\"n\">uuidpath<\/span><span class=\"p\">,<\/span> <span class=\"s\">\"-f\"<\/span><span class=\"p\">,<\/span> <span class=\"s\">\"s32be\"<\/span><span class=\"p\">,<\/span> <span class=\"s\">\"-ac\"<\/span><span class=\"p\">,<\/span> <span class=\"s\">\"2\"<\/span><span class=\"p\">,<\/span> <span class=\"s\">\"-ar\"<\/span><span class=\"p\">,<\/span> <span class=\"s\">\"44100\"<\/span><span class=\"p\">,<\/span> <span class=\"s\">\"-i\"<\/span><span class=\"p\">,<\/span> <span class=\"s\">\"pipe:0\"<\/span><span class=\"p\">,<\/span> <span class=\"s\">\"-f\"<\/span><span class=\"p\">,<\/span> <span class=\"s\">\"matroska\"<\/span><span class=\"p\">,<\/span> <span class=\"s\">\"-codec\"<\/span><span class=\"p\">,<\/span> <span class=\"s\">\"copy\"<\/span><span class=\"p\">,<\/span> <span class=\"s\">\"pipe:1\"<\/span><span class=\"p\">)<\/span>\n<span class=\"p\">}<\/span><\/code><\/pre><\/figure>\n\n<p>This is a <a href=\"https:\/\/www.ffmpeg.org\/\">ffmpeg<\/a> command line that <code class=\"language-plaintext highlighter-rouge\">de-ip-hdmi<\/code> uses to wrap the data from the extenders in a <a href=\"https:\/\/en.wikipedia.org\/wiki\/Matroska\">Matroska container<\/a>\n(a way of combining audio and images together into one video file). We can tell a few things from this:<\/p>\n\n<ul>\n  <li><code class=\"language-plaintext highlighter-rouge\">-f s32be<\/code>: the samples are stored as signed 32-bit big-endian integers\n    <ul>\n      <li>this uses <code class=\"language-plaintext highlighter-rouge\">i32::MIN<\/code> and <code class=\"language-plaintext highlighter-rouge\">i32::MAX<\/code> as minimum\/maximum (i.e. -2147483648 and 2147483647)<\/li>\n    <\/ul>\n  <\/li>\n  <li><code class=\"language-plaintext highlighter-rouge\">-ac 2<\/code>: there are 2 audio channels\n    <ul>\n      <li>the samples from each channel will be <a href=\"https:\/\/en.wikipedia.org\/wiki\/Interleaving_(data)\">interleaved<\/a>, since we just have one big audio stream<\/li>\n    <\/ul>\n  <\/li>\n  <li><code class=\"language-plaintext highlighter-rouge\">-ar 44100<\/code>: the sample rate is 44100 Hz<\/li>\n<\/ul>\n\n<p>It looks like <code class=\"language-plaintext highlighter-rouge\">de-ip-hdmi<\/code> just treats everything after the 16th byte as a set of samples in the above format:<\/p>\n\n<figure class=\"highlight\"><pre><code class=\"language-go\" data-lang=\"go\"><span class=\"c\">\/\/ Maybe there is some audio data on port 2066?<\/span>\n<span class=\"k\">if<\/span> <span class=\"n\">pkt<\/span><span class=\"o\">.<\/span><span class=\"n\">Data<\/span><span class=\"p\">[<\/span><span class=\"m\">36<\/span><span class=\"p\">]<\/span> <span class=\"o\">==<\/span> <span class=\"m\">0x08<\/span> <span class=\"o\">&amp;&amp;<\/span> <span class=\"n\">pkt<\/span><span class=\"o\">.<\/span><span class=\"n\">Data<\/span><span class=\"p\">[<\/span><span class=\"m\">37<\/span><span class=\"p\">]<\/span> <span class=\"o\">==<\/span> <span class=\"m\">0x12<\/span> <span class=\"o\">&amp;&amp;<\/span> <span class=\"o\">*<\/span><span class=\"n\">output_mkv<\/span> <span class=\"o\">&amp;&amp;<\/span> <span class=\"o\">*<\/span><span class=\"n\">audio<\/span> <span class=\"p\">{<\/span>\n    <span class=\"k\">select<\/span> <span class=\"p\">{<\/span>\n    <span class=\"k\">case<\/span> <span class=\"n\">audiodis<\/span> <span class=\"o\">&lt;-<\/span> <span class=\"n\">ApplicationData<\/span><span class=\"p\">[<\/span><span class=\"m\">16<\/span><span class=\"o\">:<\/span><span class=\"p\">]<\/span><span class=\"o\">:<\/span>\n    <span class=\"k\">default<\/span><span class=\"o\">:<\/span>\n    <span class=\"p\">}<\/span>\n    \n    <span class=\"k\">continue<\/span>\n<span class=\"p\">}<\/span><\/code><\/pre><\/figure>\n\n<p>Going back to our Wireshark dump, the first 16 bytes of every audio packet seem to be identical (and the packets are always 1008\nbytes long):<\/p>\n\n<p><img src=\"\/assets\/img\/showtime\/audiohdr.png\" alt=\"audio header in wireshark\" \/><\/p>\n\n<p>So, to send audio data we should in theory be able to write the header and then 992 (1008 - 16) bytes worth of samples into a packet\nand send it as UDP from port 2065 to port 2066.<\/p>\n\n<p>However, how frequently should we send the packets?<\/p>\n\n<h3 id=\"timing-is-everything\">Timing is everything<\/h3>\n\n<p>We need to provide <code class=\"language-plaintext highlighter-rouge\">44100 \u00d7 2 = 88200<\/code> samples each second, since we have 2 audio channels at 44100 Hz sample rate. Each packet of audio\ncontains 996 bytes of samples at 32 bits (= 4 bytes) per sample, so <code class=\"language-plaintext highlighter-rouge\">992 \/ 4 = 248<\/code> samples fit in a packet.<\/p>\n\n<p>That means we need to send a packet every <code class=\"language-plaintext highlighter-rouge\">248 \/ 88200<\/code> seconds, or roughly <strong>2.812 milliseconds<\/strong> (to 3 decimal places) \u2013 which seems to line\nup with the frequency observed in the Wireshark dump:<\/p>\n\n<p><img src=\"\/assets\/img\/showtime\/audiointerval.png\" alt=\"audio intervals in wireshark\" \/><\/p>\n\n<p><em>(e.g. 2.9222 s - 2.9195 s = 2.7ms, 2.9251s - 2.9222s = 2.9ms, etc)<\/em><\/p>\n\n<hr \/>\n\n<p>I wrote up some sample code that read samples from <code class=\"language-plaintext highlighter-rouge\">stdin<\/code> (fed in using ffmpeg) and sent a packet, then slept for 2812 nanoseconds, and then repeated.\nThis resulted in\u2026complete silence!<\/p>\n\n<p>That is, until we changed the device connected to the HDMI receiver unit (a hifi system with HDMI passthrough) to another device (a projector).\nThe projector happily played the audio we were sending\u2026 but it sounded distorted (kinda \u201ccrunchy\u201d sounding). What\u2019s going on?<\/p>\n\n<h3 id=\"jitter-is-unacceptable\">Jitter is unacceptable<\/h3>\n\n<p>A while ago, I wrote some audio code for a project called <a href=\"https:\/\/git.eta.st\/eta\/sqa\">\u201cSQA\u201d<\/a>. This taught me one of the \u2018cardinal rules\u2019\nof audio programming: jitter is unacceptable. There\u2019s a great blog post called \n<a href=\"http:\/\/www.rossbencina.com\/code\/real-time-audio-programming-101-time-waits-for-nothing\">\u201cReal-time audio programming 101: time waits for nothing\u201d<\/a> that\nexplains this in more detail but to sum up: when supplying audio samples for playback, you have a fixed amount of time to fill a buffer of a certain size,\nand you must <em>always<\/em> fill the buffer in the time given to you (or less).<\/p>\n\n<p>If you leave the buffer half-filled the audio will \u201cglitch\u201d: it\u2019ll have pops and clicks in it and\/or sound distorted, because the new samples you\u2019ve\nonly half written will stop halfway through the playback buffer and jump suddenly to a bunch of zeroes instead (or some other garbage data). This is called\na buffer under\/overrun, or an <em>xrun<\/em> for short.<\/p>\n\n<p>So, how do you make sure you never glitch? Well, doing anything that could block or take lots of time to complete (reading from a file, waiting on a mutex,\netc.) in the thread that handles audio is strictly forbidden; you\u2019ve basically got to have something else put the audio samples in a <a href=\"https:\/\/en.wikipedia.org\/wiki\/Circular_buffer\">ringbuffer<\/a>\nfor you to copy out\nin the audio thread.<\/p>\n\n<h2 id=\"final-audio-thread-code\">Final audio thread code<\/h2>\n\n<p>Armed with the above knowledge, I completely rewrote the audio thread \u2013 and it actually works!<\/p>\n\n<p>It now uses the excellent <a href=\"https:\/\/crates.io\/crates\/bounded_spsc_queue\"><code class=\"language-plaintext highlighter-rouge\">bounded_spsc_queue<\/code><\/a> ringbuffer library to ship samples from the rest of the program\nand the audio thread, and <a href=\"https:\/\/en.wikipedia.org\/wiki\/Busy_waiting\">spin-waits<\/a> (loops) until it needs to send the next packet.\n(Doing the \u201cproper\u201d thing and using something like <a href=\"https:\/\/linux.die.net\/man\/2\/nanosleep\"><code class=\"language-plaintext highlighter-rouge\">nanosleep(2)<\/code><\/a> would be more efficient \u2013 but was unreliable\ncompared to just spinning in my tests.)<\/p>\n\n<p>The full audio thread code is below:<\/p>\n\n<figure class=\"highlight\"><pre><code class=\"language-rust\" data-lang=\"rust\"><span class=\"k\">fn<\/span> <span class=\"nf\">audio_thread<\/span><span class=\"p\">(<\/span><span class=\"n\">udp<\/span><span class=\"p\">:<\/span> <span class=\"n\">UdpSocket<\/span><span class=\"p\">,<\/span> <span class=\"n\">dest<\/span><span class=\"p\">:<\/span> <span class=\"n\">SocketAddr<\/span><span class=\"p\">,<\/span> <span class=\"n\">queue<\/span><span class=\"p\">:<\/span> <span class=\"n\">Consumer<\/span><span class=\"o\">&lt;<\/span><span class=\"nb\">u8<\/span><span class=\"o\">&gt;<\/span><span class=\"p\">)<\/span> <span class=\"p\">{<\/span>\n    <span class=\"k\">let<\/span> <span class=\"k\">mut<\/span> <span class=\"n\">pkt<\/span> <span class=\"o\">=<\/span> <span class=\"p\">[<\/span><span class=\"mi\">0u8<\/span><span class=\"p\">;<\/span> <span class=\"mi\">1008<\/span><span class=\"p\">];<\/span>\n    <span class=\"c1\">\/\/ Pre-write the header on to the packet.<\/span>\n    <span class=\"k\">for<\/span> <span class=\"p\">(<\/span><span class=\"n\">i<\/span><span class=\"p\">,<\/span> <span class=\"n\">b<\/span><span class=\"p\">)<\/span> <span class=\"k\">in<\/span> <span class=\"p\">[<\/span><span class=\"mi\">0<\/span><span class=\"p\">,<\/span> <span class=\"mi\">0x55<\/span><span class=\"p\">,<\/span> <span class=\"mi\">0x55<\/span><span class=\"p\">,<\/span> <span class=\"mi\">0x55<\/span><span class=\"p\">,<\/span> <span class=\"mi\">0x55<\/span><span class=\"p\">,<\/span> <span class=\"mi\">0x55<\/span><span class=\"p\">,<\/span> <span class=\"mi\">0x55<\/span><span class=\"p\">,<\/span> <span class=\"mi\">0x55<\/span><span class=\"p\">,<\/span> <span class=\"mi\">0x55<\/span><span class=\"p\">,<\/span> <span class=\"mi\">0x55<\/span><span class=\"p\">,<\/span> <span class=\"mi\">0x55<\/span><span class=\"p\">,<\/span> <span class=\"mi\">0x55<\/span><span class=\"p\">,<\/span> <span class=\"mi\">0<\/span><span class=\"p\">,<\/span> <span class=\"mi\">0<\/span><span class=\"p\">,<\/span> <span class=\"mi\">0<\/span><span class=\"p\">]<\/span><span class=\"nf\">.iter<\/span><span class=\"p\">()<\/span><span class=\"nf\">.enumerate<\/span><span class=\"p\">()<\/span> <span class=\"p\">{<\/span>\n        <span class=\"n\">pkt<\/span><span class=\"p\">[<\/span><span class=\"n\">i<\/span><span class=\"p\">]<\/span> <span class=\"o\">=<\/span> <span class=\"o\">*<\/span><span class=\"n\">b<\/span><span class=\"p\">;<\/span>\n    <span class=\"p\">}<\/span>\n    <span class=\"k\">let<\/span> <span class=\"n\">now<\/span> <span class=\"o\">=<\/span> <span class=\"nn\">Instant<\/span><span class=\"p\">::<\/span><span class=\"nf\">now<\/span><span class=\"p\">();<\/span>\n    <span class=\"c1\">\/\/ Monotonic nanosecond counter.<\/span>\n    <span class=\"k\">let<\/span> <span class=\"n\">cur_nanos<\/span> <span class=\"o\">=<\/span> <span class=\"p\">||<\/span> <span class=\"p\">{<\/span>\n        <span class=\"nn\">Instant<\/span><span class=\"p\">::<\/span><span class=\"nf\">now<\/span><span class=\"p\">()<\/span><span class=\"nf\">.duration_since<\/span><span class=\"p\">(<\/span><span class=\"n\">now<\/span><span class=\"p\">)<\/span><span class=\"nf\">.as_micros<\/span><span class=\"p\">()<\/span>\n    <span class=\"p\">};<\/span>\n    <span class=\"c1\">\/\/ 2.812 milliseconds!<\/span>\n    <span class=\"k\">const<\/span> <span class=\"n\">INTERVAL<\/span><span class=\"p\">:<\/span> <span class=\"nb\">u128<\/span> <span class=\"o\">=<\/span> <span class=\"mi\">2812<\/span><span class=\"p\">;<\/span>\n    <span class=\"c1\">\/\/ Next time (on the counter) we need to send a packet.<\/span>\n    <span class=\"k\">let<\/span> <span class=\"k\">mut<\/span> <span class=\"n\">next<\/span> <span class=\"o\">=<\/span> <span class=\"n\">INTERVAL<\/span><span class=\"p\">;<\/span>\n    <span class=\"c1\">\/\/ Current buffer pointer.<\/span>\n    <span class=\"k\">let<\/span> <span class=\"k\">mut<\/span> <span class=\"n\">ptr<\/span> <span class=\"o\">=<\/span> <span class=\"mi\">16<\/span><span class=\"p\">;<\/span>\n    <span class=\"c1\">\/\/ Consecutive Xrun counter.<\/span>\n    <span class=\"k\">let<\/span> <span class=\"k\">mut<\/span> <span class=\"n\">num_xruns<\/span> <span class=\"o\">=<\/span> <span class=\"mi\">0<\/span><span class=\"p\">;<\/span>\n    <span class=\"k\">loop<\/span> <span class=\"p\">{<\/span>\n        <span class=\"k\">let<\/span> <span class=\"n\">cur<\/span> <span class=\"o\">=<\/span> <span class=\"nf\">cur_nanos<\/span><span class=\"p\">();<\/span>\n        <span class=\"k\">if<\/span> <span class=\"n\">cur<\/span> <span class=\"o\">&gt;=<\/span> <span class=\"n\">next<\/span> <span class=\"p\">{<\/span>\n            <span class=\"c1\">\/\/ Send the packet.<\/span>\n            <span class=\"n\">udp<\/span><span class=\"nf\">.send_to<\/span><span class=\"p\">(<\/span><span class=\"o\">&amp;<\/span><span class=\"n\">pkt<\/span><span class=\"p\">,<\/span> <span class=\"o\">&amp;<\/span><span class=\"n\">dest<\/span><span class=\"p\">)<\/span><span class=\"nf\">.unwrap<\/span><span class=\"p\">();<\/span>\n            <span class=\"c1\">\/\/ Check if we managed to fill the buf; if not, we've xrun'd.<\/span>\n            <span class=\"k\">if<\/span> <span class=\"n\">ptr<\/span> <span class=\"o\">!=<\/span> <span class=\"mi\">1008<\/span> <span class=\"p\">{<\/span>\n                <span class=\"n\">num_xruns<\/span> <span class=\"o\">+=<\/span> <span class=\"mi\">1<\/span><span class=\"p\">;<\/span>\n            <span class=\"p\">}<\/span>\n            <span class=\"k\">else<\/span> <span class=\"p\">{<\/span>\n                <span class=\"c1\">\/\/ Reset the pointer.<\/span>\n                <span class=\"n\">ptr<\/span> <span class=\"o\">=<\/span> <span class=\"mi\">16<\/span><span class=\"p\">;<\/span>\n                <span class=\"n\">num_xruns<\/span> <span class=\"o\">=<\/span> <span class=\"mi\">0<\/span><span class=\"p\">;<\/span>\n            <span class=\"p\">}<\/span>\n            <span class=\"c1\">\/\/ Advance the deadline so it's 2.8ms after the previous one.<\/span>\n            <span class=\"c1\">\/\/ We skip over if we're already going to miss it.<\/span>\n            <span class=\"k\">while<\/span> <span class=\"n\">cur<\/span> <span class=\"o\">&gt;=<\/span> <span class=\"n\">next<\/span> <span class=\"p\">{<\/span>\n                <span class=\"n\">next<\/span> <span class=\"o\">+=<\/span> <span class=\"n\">INTERVAL<\/span><span class=\"p\">;<\/span>\n            <span class=\"p\">}<\/span>\n        <span class=\"p\">}<\/span>\n        <span class=\"c1\">\/\/ Okay, now time to fill the buffer a bit.<\/span>\n        <span class=\"k\">if<\/span> <span class=\"n\">ptr<\/span> <span class=\"o\">&lt;<\/span> <span class=\"mi\">1008<\/span> <span class=\"p\">{<\/span>\n            <span class=\"k\">if<\/span> <span class=\"n\">num_xruns<\/span> <span class=\"o\">&gt;<\/span> <span class=\"mi\">5<\/span> <span class=\"p\">{<\/span>\n                <span class=\"c1\">\/\/ Let's just fill it up with silence instead.<\/span>\n                <span class=\"c1\">\/\/ (otherwise you get a buzzing effect that's highly irritating)<\/span>\n                <span class=\"n\">pkt<\/span><span class=\"p\">[<\/span><span class=\"n\">ptr<\/span><span class=\"p\">]<\/span> <span class=\"o\">=<\/span> <span class=\"mi\">0<\/span><span class=\"p\">;<\/span>\n                <span class=\"n\">ptr<\/span> <span class=\"o\">+=<\/span> <span class=\"mi\">1<\/span><span class=\"p\">;<\/span>\n            <span class=\"p\">}<\/span>\n            <span class=\"k\">else<\/span> <span class=\"p\">{<\/span>\n                <span class=\"k\">if<\/span> <span class=\"k\">let<\/span> <span class=\"nf\">Some<\/span><span class=\"p\">(<\/span><span class=\"n\">smpl<\/span><span class=\"p\">)<\/span> <span class=\"o\">=<\/span> <span class=\"n\">queue<\/span><span class=\"nf\">.try_pop<\/span><span class=\"p\">()<\/span> <span class=\"p\">{<\/span>\n                    <span class=\"n\">pkt<\/span><span class=\"p\">[<\/span><span class=\"n\">ptr<\/span><span class=\"p\">]<\/span> <span class=\"o\">=<\/span> <span class=\"n\">smpl<\/span><span class=\"p\">;<\/span>\n                    <span class=\"n\">ptr<\/span> <span class=\"o\">+=<\/span> <span class=\"mi\">1<\/span><span class=\"p\">;<\/span>\n                <span class=\"p\">}<\/span>\n            <span class=\"p\">}<\/span>\n        <span class=\"p\">}<\/span>\n        <span class=\"k\">else<\/span> <span class=\"p\">{<\/span>\n            <span class=\"p\">::<\/span><span class=\"nn\">std<\/span><span class=\"p\">::<\/span><span class=\"nn\">thread<\/span><span class=\"p\">::<\/span><span class=\"nf\">yield_now<\/span><span class=\"p\">();<\/span>\n        <span class=\"p\">}<\/span>\n    <span class=\"p\">}<\/span>\n<span class=\"p\">}<\/span><\/code><\/pre><\/figure>\n\n<h2 id=\"conclusions\">Conclusions<\/h2>\n\n<p>The code is <a href=\"https:\/\/git.eta.st\/eta\/showtime\">here<\/a>. Don\u2019t expect miracles.\n(If you actually plan to use this, you might want to <a href=\"\/#contact\">get in touch<\/a>.)<\/p>\n\n<p>There are a number of ways to feed data into it, which you choose at compile time using <a href=\"https:\/\/doc.rust-lang.org\/cargo\/reference\/features.html\">Cargo features<\/a>.\nYou can either feed in raw s32be samples from\neither stdin or TCP (it\u2019ll make a TCP server and you can connect and send stuff, but not with more than one client at the same time\u2026).\nI also ended up adding a <a href=\"https:\/\/jackaudio.org\/\">JACK<\/a> backend (using my own <a href=\"https:\/\/docs.rs\/sqa-jack\/0.6.1\/sqa_jack\/\">sqa-jack<\/a> library), if that\u2019s\nyour thing.<\/p>\n","pubDate":"Sun, 20 Jun 2021 00:00:00 +0000","link":"https:\/\/eta.st\/2021\/06\/20\/showtime.html","guid":"https:\/\/eta.st\/2021\/06\/20\/showtime.html"},{"title":"Why asynchronous Rust doesn't work","description":"<p>In 2017, I said that <a href=\"\/2017\/08\/04\/async-rust.html\">\u201casynchronous Rust programming is a disaster and a mess\u201d<\/a>. In 2021\na lot more of the Rust ecosystem has become asynchronous \u2013 such that it might be appropriate to just say that Rust\nprogramming is now a disaster and a mess. As someone who used to really love Rust, this makes me quite sad.<\/p>\n\n<p>I\u2019ve had a think about this, and I\u2019m going to attempt to explain how we got here. Many people have explained the problems\nwith asynchronous programming \u2013 the famous <a href=\"https:\/\/journal.stuffwithstuff.com\/2015\/02\/01\/what-color-is-your-function\/\">what colour is your function<\/a>\nessay, for example.<sup id=\"fnref:1\"><a href=\"#fn:1\" class=\"footnote\" rel=\"footnote\" role=\"doc-noteref\">1<\/a><\/sup> However, I think there are a number of things <em>specific to the design of Rust<\/em> that make asynchronous\nRust particularly messy, <em>on top of<\/em> the problems inherent to doing any sort of asynchronous programming.<\/p>\n\n<p>In particular,\nI actually think the design of Rust is almost fundamentally incompatible with a lot of asynchronous paradigms. It\u2019s not that\nthe people designing async were incompetent or bad at their jobs \u2013 they actually did a surprisingly good job given the\ncircumstances!<sup id=\"fnref:2\"><a href=\"#fn:2\" class=\"footnote\" rel=\"footnote\" role=\"doc-noteref\">2<\/a><\/sup> I just don\u2019t think it was ever going to work out cleanly \u2013 and to see why, you\u2019re going to have to read\na somewhat long blog post!<\/p>\n\n<h2 id=\"a-study-in-async\">A study in async<\/h2>\n\n<p>I\u2019d like to make a simple function that does some work in the background, and lets us know when it\u2019s done by running <em>another<\/em>\nfunction with the results of said background work.<\/p>\n\n<figure class=\"highlight\"><pre><code class=\"language-rust\" data-lang=\"rust\"><span class=\"k\">use<\/span> <span class=\"nn\">std<\/span><span class=\"p\">::<\/span><span class=\"n\">thread<\/span><span class=\"p\">;<\/span>\n\n<span class=\"cd\">\/\/\/ Does some strenuous work \"asynchronously\", and calls `func` with the<\/span>\n<span class=\"cd\">\/\/\/ result of the work when done.<\/span>\n<span class=\"k\">fn<\/span> <span class=\"nf\">do_work_and_then<\/span><span class=\"p\">(<\/span><span class=\"n\">func<\/span><span class=\"p\">:<\/span> <span class=\"k\">fn<\/span><span class=\"p\">(<\/span><span class=\"nb\">i32<\/span><span class=\"p\">))<\/span> <span class=\"p\">{<\/span>\n    <span class=\"nn\">thread<\/span><span class=\"p\">::<\/span><span class=\"nf\">spawn<\/span><span class=\"p\">(<\/span><span class=\"k\">move<\/span> <span class=\"p\">||<\/span> <span class=\"p\">{<\/span>\n        <span class=\"c1\">\/\/ Figuring out the meaning of life...<\/span>\n        <span class=\"nn\">thread<\/span><span class=\"p\">::<\/span><span class=\"nf\">sleep_ms<\/span><span class=\"p\">(<\/span><span class=\"mi\">1000<\/span><span class=\"p\">);<\/span> <span class=\"c1\">\/\/ gee, this takes time to do...<\/span>\n        <span class=\"c1\">\/\/ ah, that's it!<\/span>\n        <span class=\"k\">let<\/span> <span class=\"n\">result<\/span><span class=\"p\">:<\/span> <span class=\"nb\">i32<\/span> <span class=\"o\">=<\/span> <span class=\"mi\">42<\/span><span class=\"p\">;<\/span>\n        <span class=\"c1\">\/\/ let's call the `func` and tell it the good news...<\/span>\n        <span class=\"nf\">func<\/span><span class=\"p\">(<\/span><span class=\"n\">result<\/span><span class=\"p\">)<\/span>\n    <span class=\"p\">});<\/span>\n<span class=\"p\">}<\/span><\/code><\/pre><\/figure>\n\n<p>There\u2019s this idea called \u201c<a href=\"https:\/\/en.wikipedia.org\/wiki\/First-class_function\">first-class functions<\/a>\u201d which says you\ncan pass around functions as if they were objects. This would be great to have in Rust, right?<\/p>\n\n<p>See that <code class=\"language-plaintext highlighter-rouge\">func: fn(i32)<\/code>? <code class=\"language-plaintext highlighter-rouge\">fn(i32)<\/code> is the type of a function that takes in one singular <code class=\"language-plaintext highlighter-rouge\">i32<\/code> and returns nothing.\nThanks to first-class functions, I can pass a function to <code class=\"language-plaintext highlighter-rouge\">do_work_and_then<\/code> specifying what should happen next after I\u2019m\ndone with my work \u2013 like this:<\/p>\n\n<figure class=\"highlight\"><pre><code class=\"language-rust\" data-lang=\"rust\"><span class=\"k\">fn<\/span> <span class=\"nf\">main<\/span><span class=\"p\">()<\/span> <span class=\"p\">{<\/span>\n    <span class=\"nf\">do_work_and_then<\/span><span class=\"p\">(|<\/span><span class=\"n\">meaning_of_life<\/span><span class=\"p\">|<\/span> <span class=\"p\">{<\/span>\n        <span class=\"nd\">println!<\/span><span class=\"p\">(<\/span><span class=\"s\">\"oh man, I found it: {}\"<\/span><span class=\"p\">,<\/span> <span class=\"n\">meaning_of_life<\/span><span class=\"p\">);<\/span>\n    <span class=\"p\">});<\/span>\n    <span class=\"c1\">\/\/ do other stuff<\/span>\n    <span class=\"nn\">thread<\/span><span class=\"p\">::<\/span><span class=\"nf\">sleep_ms<\/span><span class=\"p\">(<\/span><span class=\"mi\">2000<\/span><span class=\"p\">);<\/span>\n<span class=\"p\">}<\/span><\/code><\/pre><\/figure>\n\n<p>Because <code class=\"language-plaintext highlighter-rouge\">do_work_and_then<\/code> is <em>asynchronous<\/em>, it returns immediately and does its thing in the background, so the control\nflow of <code class=\"language-plaintext highlighter-rouge\">main<\/code> isn\u2019t disrupted. I could do some other form of work, which would be nice (but here I just wait for 2 seconds,\nbecause there\u2019s nothing better to do). Meanwhile, when we do figure out the meaning of life, it gets printed out. Indeed,\nif you run this program, you get:<\/p>\n\n<div class=\"language-plaintext highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code>oh man, I found it: 42\n<\/code><\/pre><\/div><\/div>\n\n<p>This is really exciting; we could build whole web servers and network stuff and whatever out of this! Let\u2019s\ntry a more advanced example: I have a database I want to store the meaning of life in when I find it, and then I can run\na web server in the foreground that enables people to get it once I\u2019m done (and returns some error if I\u2019m not done yet).<\/p>\n\n<figure class=\"highlight\"><pre><code class=\"language-rust\" data-lang=\"rust\"><span class=\"k\">struct<\/span> <span class=\"n\">Database<\/span> <span class=\"p\">{<\/span>\n    <span class=\"n\">data<\/span><span class=\"p\">:<\/span> <span class=\"nb\">Vec<\/span><span class=\"o\">&lt;<\/span><span class=\"nb\">i32<\/span><span class=\"o\">&gt;<\/span>\n<span class=\"p\">}<\/span>\n<span class=\"k\">impl<\/span> <span class=\"n\">Database<\/span> <span class=\"p\">{<\/span>\n    <span class=\"k\">fn<\/span> <span class=\"nf\">store<\/span><span class=\"p\">(<\/span><span class=\"o\">&amp;<\/span><span class=\"k\">mut<\/span> <span class=\"k\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">data<\/span><span class=\"p\">:<\/span> <span class=\"nb\">i32<\/span><span class=\"p\">)<\/span> <span class=\"p\">{<\/span>\n        <span class=\"k\">self<\/span><span class=\"py\">.data<\/span><span class=\"nf\">.push<\/span><span class=\"p\">(<\/span><span class=\"n\">data<\/span><span class=\"p\">);<\/span>\n    <span class=\"p\">}<\/span>\n<span class=\"p\">}<\/span>\n\n<span class=\"k\">fn<\/span> <span class=\"nf\">main<\/span><span class=\"p\">()<\/span> <span class=\"p\">{<\/span>\n    <span class=\"k\">let<\/span> <span class=\"k\">mut<\/span> <span class=\"n\">db<\/span> <span class=\"o\">=<\/span> <span class=\"n\">Database<\/span> <span class=\"p\">{<\/span> <span class=\"n\">data<\/span><span class=\"p\">:<\/span> <span class=\"nd\">vec!<\/span><span class=\"p\">[]<\/span> <span class=\"p\">};<\/span>\n    <span class=\"nf\">do_work_and_then<\/span><span class=\"p\">(|<\/span><span class=\"n\">meaning_of_life<\/span><span class=\"p\">|<\/span> <span class=\"p\">{<\/span>\n        <span class=\"nd\">println!<\/span><span class=\"p\">(<\/span><span class=\"s\">\"oh man, I found it: {}\"<\/span><span class=\"p\">,<\/span> <span class=\"n\">meaning_of_life<\/span><span class=\"p\">);<\/span>\n        <span class=\"n\">db<\/span><span class=\"nf\">.store<\/span><span class=\"p\">(<\/span><span class=\"n\">meaning_of_life<\/span><span class=\"p\">);<\/span>\n    <span class=\"p\">});<\/span>\n    <span class=\"c1\">\/\/ I'd read from `db` here if I really were making a web server.<\/span>\n    <span class=\"c1\">\/\/ But that's beside the point, so I'm not going to.<\/span>\n    <span class=\"c1\">\/\/ (also `db` would have to be wrapped in an `Arc&lt;Mutex&lt;T&gt;&gt;`)<\/span>\n    <span class=\"nn\">thread<\/span><span class=\"p\">::<\/span><span class=\"nf\">sleep_ms<\/span><span class=\"p\">(<\/span><span class=\"mi\">2000<\/span><span class=\"p\">);<\/span>\n<span class=\"p\">}<\/span><\/code><\/pre><\/figure>\n\n<p>Let\u2019s run this\u2026oh.<\/p>\n\n<div class=\"language-plaintext highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code>error[E0308]: mismatched types\n  --&gt; src\/main.rs:27:22\n   |\n27 |       do_work_and_then(|meaning_of_life| {\n   |  ______________________^\n28 | |         println!(\"oh man, I found it: {}\", meaning_of_life);\n29 | |         db.store(meaning_of_life);\n30 | |     });\n   | |_____^ expected fn pointer, found closure\n   |\n   = note: expected fn pointer `fn(i32)`\n                 found closure `[closure@src\/main.rs:27:22: 30:6]`\n<\/code><\/pre><\/div><\/div>\n\n<p>I see.<\/p>\n\n<h2 id=\"hang-on-a-minute\">Hang on a minute\u2026<\/h2>\n\n<p>So, this is actually quite complicated. Before, the function we passed to <code class=\"language-plaintext highlighter-rouge\">do_work_and_then<\/code> was <em>pure<\/em>: it didn\u2019t have any\nassociated data, so you could just pass it around as a function pointer (<code class=\"language-plaintext highlighter-rouge\">fn(i32)<\/code>) and all was grand.\nHowever, this new function in that last example is a <em>closure<\/em>: a function object with a bit of data (a <code class=\"language-plaintext highlighter-rouge\">&amp;mut Database<\/code>)\ntacked onto it.<\/p>\n\n<p>Closures are kind of magic. We can\u2019t actually name their type \u2013 as seen above, the Rust compiler called it a\n<code class=\"language-plaintext highlighter-rouge\">[closure@src\/main.rs:27:22: 30:6]<\/code>, but we can\u2019t actually write that in valid Rust code. If you were to write it out\nexplicitly, a closure would look something like this:<\/p>\n\n<figure class=\"highlight\"><pre><code class=\"language-rust\" data-lang=\"rust\"><span class=\"k\">struct<\/span> <span class=\"n\">Closure<\/span><span class=\"o\">&lt;<\/span><span class=\"nv\">'a<\/span><span class=\"o\">&gt;<\/span> <span class=\"p\">{<\/span>\n    <span class=\"n\">data<\/span><span class=\"p\">:<\/span> <span class=\"o\">&amp;<\/span><span class=\"nv\">'a<\/span> <span class=\"k\">mut<\/span> <span class=\"n\">Database<\/span><span class=\"p\">,<\/span>\n    <span class=\"n\">func<\/span><span class=\"p\">:<\/span> <span class=\"k\">fn<\/span><span class=\"p\">(<\/span><span class=\"nb\">i32<\/span><span class=\"p\">,<\/span> <span class=\"o\">&amp;<\/span><span class=\"k\">mut<\/span> <span class=\"n\">Database<\/span><span class=\"p\">)<\/span>\n<span class=\"p\">}<\/span>\n\n<span class=\"k\">impl<\/span><span class=\"o\">&lt;<\/span><span class=\"nv\">'a<\/span><span class=\"o\">&gt;<\/span> <span class=\"n\">Closure<\/span><span class=\"o\">&lt;<\/span><span class=\"nv\">'a<\/span><span class=\"o\">&gt;<\/span> <span class=\"p\">{<\/span>\n    <span class=\"k\">fn<\/span> <span class=\"nf\">call<\/span><span class=\"p\">(<\/span><span class=\"o\">&amp;<\/span><span class=\"k\">mut<\/span> <span class=\"k\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">arg<\/span><span class=\"p\">:<\/span> <span class=\"nb\">i32<\/span><span class=\"p\">)<\/span> <span class=\"p\">{<\/span>\n        <span class=\"p\">(<\/span><span class=\"k\">self<\/span><span class=\"py\">.func<\/span><span class=\"p\">)(<\/span><span class=\"n\">arg<\/span><span class=\"p\">,<\/span> <span class=\"k\">self<\/span><span class=\"py\">.data<\/span><span class=\"p\">)<\/span>\n    <span class=\"p\">}<\/span>\n<span class=\"p\">}<\/span><\/code><\/pre><\/figure>\n\n<p>There are a number of things to unpack here.<\/p>\n\n<hr \/>\n\n<h3 id=\"an-aside-on-naming-types\">An aside on naming types<\/h3>\n\n<p>Being able to name types in Rust is quite important. With a regular old type, like <code class=\"language-plaintext highlighter-rouge\">u8<\/code>, life is easy. I can write a function\n<code class=\"language-plaintext highlighter-rouge\">fn add_one(in: u8) -&gt; u8<\/code> that takes one and returns one without any hassle.<\/p>\n\n<p>If you can\u2019t actually name a type, working with it becomes somewhat cumbersome. What you end up having to do instead is\nrefer to it using generics \u2013 for example, closures\u2019 types can\u2019t be named directly, but since they implement one of the <code class=\"language-plaintext highlighter-rouge\">Fn<\/code>\nfamily of traits, I can write functions like:<\/p>\n\n<figure class=\"highlight\"><pre><code class=\"language-rust\" data-lang=\"rust\"><span class=\"k\">fn<\/span> <span class=\"n\">closure_accepting_function<\/span><span class=\"o\">&lt;<\/span><span class=\"n\">F<\/span><span class=\"o\">&gt;<\/span><span class=\"p\">(<\/span><span class=\"n\">func<\/span><span class=\"p\">:<\/span> <span class=\"n\">F<\/span><span class=\"p\">)<\/span>\n<span class=\"k\">where<\/span>\n    <span class=\"n\">F<\/span><span class=\"p\">:<\/span> <span class=\"nf\">Fn<\/span><span class=\"p\">(<\/span><span class=\"nb\">i32<\/span><span class=\"p\">),<\/span> <span class=\"c1\">\/\/ &lt;-- look!<\/span>\n<span class=\"p\">{<\/span>\n    <span class=\"cm\">\/* do stuff *\/<\/span>\n<span class=\"p\">}<\/span><\/code><\/pre><\/figure>\n\n<p>If I want to store them in a <code class=\"language-plaintext highlighter-rouge\">struct<\/code> or something, I\u2019ll also need to do this dance with the <code class=\"language-plaintext highlighter-rouge\">where<\/code> clause every time they\u2019re\nused. This is annoying and makes things harder for me, but it\u2019s still vaguely workable. For now.<\/p>\n\n<p><img src=\"\/assets\/img\/rust-async\/msql-srv.png\" alt=\"msql-srv example code\" \/><\/p>\n\n<p><em>[image: from the <a href=\"https:\/\/github.com\/jonhoo\/msql-srv\/blob\/7bb7c06868c82b11dd736039523f95683d12b9d1\/tests\/main.rs#L26\"><code class=\"language-plaintext highlighter-rouge\">msql-srv<\/code><\/a> crate,\nshowing an example of many <code class=\"language-plaintext highlighter-rouge\">where<\/code> clauses as a result of using closures]<\/em><\/p>\n\n<hr \/>\n\n<h3 id=\"an-aside-on-radioactive-types\">An aside on \u2018radioactive\u2019 types<\/h3>\n\n<p>The way Rust is designed tends to encourage certain patterns while discouraging others. Because of ownership and lifetimes,\nhaving pieces of data that hold references to other pieces of data becomes a bit of a problem. If my type has a <code class=\"language-plaintext highlighter-rouge\">&amp;<\/code> or a <code class=\"language-plaintext highlighter-rouge\">&amp;mut<\/code>\nreference to something, Rust makes me ensure that:<\/p>\n\n<ul>\n  <li>the something in question <em>outlives<\/em> my type; you can\u2019t go and drop the thing I\u2019m referring to if I\nstill have a reference to it, otherwise my reference will become invalid<\/li>\n  <li>the something in question doesn\u2019t move while I have the reference<\/li>\n  <li>my reference to the something doesn\u2019t conflict with other references to the something (e.g. I can\u2019t have my <code class=\"language-plaintext highlighter-rouge\">&amp;<\/code> reference if\nsomething else has a <code class=\"language-plaintext highlighter-rouge\">&amp;mut<\/code> reference)<\/li>\n<\/ul>\n\n<p>So types with references in them are almost \u2018radioactive\u2019; you can keep them around for a bit (e.g. inside one particular\nfunction), but attempting to make them long-lived is usually a bit of an issue (requiring advanced tricks such as the\n<a href=\"https:\/\/doc.rust-lang.org\/std\/pin\/struct.Pin.html\"><code class=\"language-plaintext highlighter-rouge\">Pin&lt;T&gt;<\/code><\/a>\ntype which didn\u2019t even exist until a few Rust versions ago). Generally Rust doesn\u2019t really like it when you use radioactive\ntypes for too long \u2013 they make the borrow checker uneasy, because you\u2019re borrowing something for an extended period of time.<\/p>\n\n<p><img src=\"\/assets\/img\/rust-async\/rustviz.png\" alt=\"borrowing semantics visualization\" \/><\/p>\n\n<p><em>[image: from the <a href=\"https:\/\/arxiv.org\/pdf\/2011.09012.pdf\">RustViz<\/a> paper, showing borrowing semantics]<\/em><\/p>\n\n<hr \/>\n\n<p>Closures can be pretty radioactive. Look at the thing we just wrote out: it has a <code class=\"language-plaintext highlighter-rouge\">&amp;'a mut Database<\/code> reference in it!\nThat means while we\u2019re passing our <code class=\"language-plaintext highlighter-rouge\">Closure<\/code> object around, we have to be mindful of the three rules (outlives, doesn\u2019t move,\nno conflicting) \u2013 which makes things pretty hard. I can\u2019t just hand off the <code class=\"language-plaintext highlighter-rouge\">Closure<\/code> to another function (for example, the\n<code class=\"language-plaintext highlighter-rouge\">do_work_and_then<\/code> function), because then I have to make all of those rules work, and that\u2019s not necessarily easy all the time.<\/p>\n\n<p>(Not <em>all<\/em> closures are radioactive: if you make them <code class=\"language-plaintext highlighter-rouge\">move<\/code> closures, they\u2019ll take everything by value instead, and create\nclosure objects that own data instead of having radioactive references to data.\nSlightly more of a pain to deal with, but you lose the blue radiation glow the things give out when you look at them.)<\/p>\n\n<p>Also, remember what I said about being able to name types? We\u2019re not actually dealing with a nice, written-out <code class=\"language-plaintext highlighter-rouge\">Closure<\/code> object\nhere; we\u2019re dealing with something the compiler generated for us that we can\u2019t name, which is annoying. I also lied when\nI said that it was as simple as making all of your functions take <code class=\"language-plaintext highlighter-rouge\">F, where F: Fn(i32)<\/code> or something \u2013 there are actually\n<em>three<\/em> different <code class=\"language-plaintext highlighter-rouge\">Fn<\/code>-style traits, <code class=\"language-plaintext highlighter-rouge\">Fn<\/code>, <code class=\"language-plaintext highlighter-rouge\">FnMut<\/code>, and <code class=\"language-plaintext highlighter-rouge\">FnOnce<\/code>. Do you know the difference between them?<\/p>\n\n<p>So. A closure is this magical, un-nameable type that the compiler makes for us whenever we use <code class=\"language-plaintext highlighter-rouge\">|| {...}<\/code> syntax, which implements\none of three traits (and it\u2019s not immediately obvious which), and it also might be radioactive.\nTry and use one of these, and the Rust compiler is probably going to be watching you <em>very<\/em> carefully.<\/p>\n\n<hr \/>\n\n<p>The thing I really want to try and get across here is that <strong>Rust is not a language where first-class functions are ergonomic<\/strong>.\nIt\u2019s a lot easier to make some data (a <code class=\"language-plaintext highlighter-rouge\">struct<\/code>) with some functions attached (methods) than it is to make some functions with some\ndata attached (closures).<\/p>\n\n<p>Trying to use ordinary <code class=\"language-plaintext highlighter-rouge\">struct<\/code>s is downright easy:<\/p>\n\n<ul>\n  <li>they\u2019re explicitly written out by the programmer with no funky business<\/li>\n  <li>you choose what traits and methods to implement on them and how to set them out \/ implement them<\/li>\n  <li>the <code class=\"language-plaintext highlighter-rouge\">struct<\/code> can actually be referred to by other parts of the code by its type<\/li>\n<\/ul>\n\n<p>Trying to use closures is hard:<\/p>\n\n<ul>\n  <li>the compiler does some magic to make a closure type for you<\/li>\n  <li>it implements some obscure <code class=\"language-plaintext highlighter-rouge\">Fn<\/code> trait (and it\u2019s not immediately obvious which)<\/li>\n  <li>it might be radioactive (or force you to use <code class=\"language-plaintext highlighter-rouge\">move<\/code> and maybe insert a bunch of <code class=\"language-plaintext highlighter-rouge\">clone()<\/code> calls)<sup id=\"fnref:3\"><a href=\"#fn:3\" class=\"footnote\" rel=\"footnote\" role=\"doc-noteref\">3<\/a><\/sup><\/li>\n  <li>you can\u2019t actually name their type anywhere or do things like return them from a function<\/li>\n<\/ul>\n\n<p>Importantly, the restrictions applied to using closures <strong>infect<\/strong> types that contain them \u2013 if you\u2019re writing a type that contains\na closure, you\u2019ll have to make it generic over some <code class=\"language-plaintext highlighter-rouge\">Fn<\/code>-trait-implementing type parameter, and it\u2019s going to be impossible for\npeople to name your type as a result.<\/p>\n\n<p>(Other languages, like Haskell, flip this upside down: functions are everywhere, you can pass them around with reckless abandon, etc.\nOf course, these other languages usually have garbage collection to make it all work\u2026)<\/p>\n\n<hr \/>\n\n<p>Bearing this in mind, it is <em>really quite hard<\/em> to make a lot of asynchronous paradigms (like <code class=\"language-plaintext highlighter-rouge\">async<\/code>\/<code class=\"language-plaintext highlighter-rouge\">await<\/code>) work well in Rust.\nAs the <a href=\"https:\/\/journal.stuffwithstuff.com\/2015\/02\/01\/what-color-is-your-function\/\">what colour is your function<\/a> post says, <code class=\"language-plaintext highlighter-rouge\">async<\/code>\/<code class=\"language-plaintext highlighter-rouge\">await<\/code>\n(as well as things like promises, callbacks, and futures) are really a big abstraction over\n<a href=\"https:\/\/en.wikipedia.org\/wiki\/Continuation-passing_style\">continuation-passing style<\/a> \u2013 an idea closely\nrelated to the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Scheme_(programming_language)\">Scheme<\/a>\nprogramming language. Basically, the idea is you take your normal, garden-variety function and smear it out into <strong>a bunch of closures<\/strong>.\n(Well, not quite. You can read the blue links for more; I\u2019m not going to explain CPS here for the sake of brevity.)<\/p>\n\n<p>Hopefully by now you can see that making a bunch of closures is really not going to be a good idea (!)<\/p>\n\n<h2 id=\"wibbly-wobbly-scene-transition\">*wibbly wobbly scene transition*<\/h2>\n\n<p>And then fast forward a few years and you have an entire <em>language ecosystem<\/em> built on top of the idea of making these <code class=\"language-plaintext highlighter-rouge\">Future<\/code> objects\nthat actually have a load of closures inside<sup id=\"fnref:4\"><a href=\"#fn:4\" class=\"footnote\" rel=\"footnote\" role=\"doc-noteref\">4<\/a><\/sup>, and all of the problems listed above\n(hard to name, can contain references which make them radioactive, usually require using a generic <code class=\"language-plaintext highlighter-rouge\">where<\/code> clause, etc)\napply to them because of how \u201cinfectious\u201d closures are.<\/p>\n\n<p>The language people have actually been hard at\nwork to solve <em>some<\/em> (some!) of these problems by introducing features like <code class=\"language-plaintext highlighter-rouge\">impl Trait<\/code> and <code class=\"language-plaintext highlighter-rouge\">async fn<\/code> that make dealing with these\nnot <em>immediately<\/em> totally terrible, but trying to use other language features (like traits) soon makes it clear that the problems\naren\u2019t really gone; just hidden.<\/p>\n\n<p>Oh, and all the problems from <a href=\"https:\/\/journal.stuffwithstuff.com\/2015\/02\/01\/what-color-is-your-function\/\">what colour is your function<\/a>\nare still there too, by the way \u2013 on <em>top<\/em> of the Rust-specific ones.<\/p>\n\n<p>Beginner (and experienced) Rust programmers look at the state of the world as it is and try and build things on top of these shaky\nabstractions, and end up running into obscure compiler errors, and using hacks like the <a href=\"https:\/\/docs.rs\/async-trait\/0.1.42\/async_trait\/\"><code class=\"language-plaintext highlighter-rouge\">async_trait<\/code><\/a>\ncrate to glue things together, and end up with projects that depend on like 3 different versions of <code class=\"language-plaintext highlighter-rouge\">tokio<\/code> and <code class=\"language-plaintext highlighter-rouge\">futures<\/code> (perhaps\nsome <code class=\"language-plaintext highlighter-rouge\">async-std<\/code> in there if you\u2019re feeling spicy) because people have differing opinions on how to try and avoid the\n<em>fundamentally unavoidable<\/em> problems, and it\u2019s all a bit frustrating, and ultimately, all a bit sad.<\/p>\n\n<hr \/>\n\n<p>Did it really have to end this way? Was spinning up a bunch of OS threads not an acceptable\nsolution for the majority of situations? Could we have explored solutions more like Go, where a language-provided runtime makes blocking\nmore of an acceptable thing to do?<\/p>\n\n<p>Maybe we could just have kept Rust as it was circa 2016, and let the crazy non-blocking folks<sup id=\"fnref:5\"><a href=\"#fn:5\" class=\"footnote\" rel=\"footnote\" role=\"doc-noteref\">5<\/a><\/sup> write\nhand-crafted <code class=\"language-plaintext highlighter-rouge\">epoll()<\/code> loops like they do in C++. I honestly don\u2019t know, and think it\u2019s a difficult problem to solve.<\/p>\n\n<p>But as far as my money goes, I\u2019m finding it difficult to justify starting new projects in Rust when the ecosystem is like this. And, as\nI said at the start, that makes me kinda sad, because I do actually like Rust.<\/p>\n\n<hr \/>\n\n<p>(<a href=\"https:\/\/common-lisp.net\/\">Common Lisp<\/a> is pretty nice, though. We have crazy macros and parentheses and a language ecosystem that is\nolder than I am and isn\u2019t showing any signs of changing\u2026)<\/p>\n\n<hr \/>\n\n<div class=\"footnotes\" role=\"doc-endnotes\">\n  <ol>\n    <li id=\"fn:1\">\n      <p>This is really recommended reading if you aren\u2019t already familiar with it (as you\u2019ll soon see)\u00a0<a href=\"#fnref:1\" class=\"reversefootnote\" role=\"doc-backlink\">&#8617;<\/a><\/p>\n    <\/li>\n    <li id=\"fn:2\">\n      <p>Seriously \u2013 when I put out the last blog post, the actual async core team members commented saying how much they appreciated the\n  feedback, and then they actually went and made futures 1.0 better as a result. Kudos!\u00a0<a href=\"#fnref:2\" class=\"reversefootnote\" role=\"doc-backlink\">&#8617;<\/a><\/p>\n    <\/li>\n    <li id=\"fn:3\">\n      <p>You might be thinking \u201cwell, why don\u2019t you just only use <code class=\"language-plaintext highlighter-rouge\">move<\/code> closures then?\u201d \u2013 but that\u2019s beside the point; it\u2019s often a lot\n  harder to do so, because now you might have to wrap your data in an <code class=\"language-plaintext highlighter-rouge\">Arc<\/code> or something, by which point the ergonomic gains of\n  using the closure are outweighed by the borrow checker-induced pain.\u00a0<a href=\"#fnref:3\" class=\"reversefootnote\" role=\"doc-backlink\">&#8617;<\/a><\/p>\n    <\/li>\n    <li id=\"fn:4\">\n      <p>You <em>can<\/em> actually manually implement <code class=\"language-plaintext highlighter-rouge\">Future<\/code> on a regular old <code class=\"language-plaintext highlighter-rouge\">struct<\/code>. If you do this, things suddenly become a lot simpler,\n  but also you can\u2019t easily perform more async operations inside that <code class=\"language-plaintext highlighter-rouge\">struct<\/code>\u2019s methods.\u00a0<a href=\"#fnref:4\" class=\"reversefootnote\" role=\"doc-backlink\">&#8617;<\/a><\/p>\n    <\/li>\n    <li id=\"fn:5\">\n      <p>(sorry, I mean, the esteemed companies that deign to use Rust for their low-latency production services)\u00a0<a href=\"#fnref:5\" class=\"reversefootnote\" role=\"doc-backlink\">&#8617;<\/a><\/p>\n    <\/li>\n  <\/ol>\n<\/div>\n","pubDate":"Mon, 08 Mar 2021 00:00:00 +0000","link":"https:\/\/eta.st\/2021\/03\/08\/async-rust-2.html","guid":"https:\/\/eta.st\/2021\/03\/08\/async-rust-2.html"},{"title":"Getting PIV-based SSH working on a YubiKey","description":"<p>I bought a <a href=\"https:\/\/www.yubico.com\/gb\/product\/yubikey-5c-nano\/\">YubiKey 5C Nano<\/a> recently. These devices are great \u2013 I\u2019ve built a lot\nof my (metaphorical) empire on top of them, seeing as they\u2019re capable of acting as an SSH agent (store your SSH keys on them, securely!),\nan OpenPGP smartcard (do encryption and decryption on the key!), FIDO U2F \u2018security keys\u2019 (use them as a 2-factor authentication method!),\nand probably more.<\/p>\n\n<p>Getting the thing to work as an SSH agent was, however, not the easiest thing I\u2019ve ever done. There are multiple options here \u2013 you\ncan use the OpenPGP applet and then configure GnuPG to work as an SSH agent, but that\u2019s a brittle solution in my experience (<code class=\"language-plaintext highlighter-rouge\">gpg-agent<\/code>\nis quite flaky, and often requires restarting when it forgets about the YubiKey). Instead, I wanted to see whether I could use the YubiKey\u2019s\nPIV (<a href=\"https:\/\/en.wikipedia.org\/wiki\/FIPS_201\">Personal Identity Verification<\/a>) applet to get this working.<\/p>\n\n<h2 id=\"procedure\">Procedure<\/h2>\n\n<p>For the SSH agent part, we\u2019re going to use Filippo Valsorda\u2019s <a href=\"https:\/\/github.com\/FiloSottile\/yubikey-agent\">yubikey-agent<\/a>, so you\u2019ll\nwant to have that installed. In my testing, yubikey-agent\u2019s built-in <code class=\"language-plaintext highlighter-rouge\">yubikey-agent -setup<\/code> command errored out, so we\u2019ll configure the\nPIV applet of the YubiKey manually. (This\u2019ll also leave the door open for you to do other things with the PIV applet later if you like.)<\/p>\n\n<p>You\u2019ll also need Yubico\u2019s own <a href=\"https:\/\/github.com\/Yubico\/yubikey-manager\">yubikey-manager<\/a> (the <code class=\"language-plaintext highlighter-rouge\">ykman<\/code> cli tool) installed.<\/p>\n\n<p><em>Updated on 2023-11-23 to reflect the command names being moved around.<\/em><\/p>\n\n<p>Okay, here goes:<\/p>\n\n<ol>\n  <li>Ensure <code class=\"language-plaintext highlighter-rouge\">pcscd<\/code> (the PC\/SC smartcard daemon) is installed and running. This might already have been done for you by your Linux distribution\nbut, if not:\n    <ul>\n      <li><code class=\"language-plaintext highlighter-rouge\">$ sudo systemctl enable --now pcscd<\/code><\/li>\n      <li>(This is safe to run if it\u2019s already been set up; the command will just do nothing).<\/li>\n    <\/ul>\n  <\/li>\n  <li>This exercise assumes a fresh YubiKey (i.e. one where you haven\u2019t touched the PIV applet yet). If that\u2019s not the case, and you want to\n<strong>erase all of the PIV data and start afresh (losing all data encrypted with the PIV keys!)<\/strong>, use the <code class=\"language-plaintext highlighter-rouge\">ykman piv reset<\/code> command.<\/li>\n  <li>First, change the PIN from the default one (<code class=\"language-plaintext highlighter-rouge\">123456<\/code>).\n    <ul>\n      <li><code class=\"language-plaintext highlighter-rouge\">$ ykman piv access change-pin<\/code><\/li>\n      <li>As the command\u2019s help says: <em>The PIN must be between 6 and 8 characters long, and supports any type of alphanumeric characters. For\ncross-platform compatibility, numeric digits are recommended.<\/em><\/li>\n    <\/ul>\n  <\/li>\n  <li>Then, change the PUK (\u2018personal unblocking key\u2019). This is used to reset the PIN if you ever forget it.\n    <ul>\n      <li><code class=\"language-plaintext highlighter-rouge\">$ ykman piv access change-puk<\/code><\/li>\n      <li>The PUK has the same entry requirements as the PIN (i.e. also 6-8 ASCII characters).<\/li>\n      <li>It might be prudent to generate a random PUK and keep it safe (e.g. by writing it down and locking the paper away). If you lose both\nthe PIN and PUK, you will need to reset the PIV applet, losing all data encrypted with the PIV keys (and SSH access to hosts\nyou don\u2019t otherwise have access to).<\/li>\n    <\/ul>\n  <\/li>\n  <li>The YubiKey PIV applet by default has a well-known management key used to make changes to the PIV keys (etc.). It\u2019s best practice to\nchange this to something else. We\u2019ll use the option to generate a random one, store it in the YubiKey, and secure it with the PIN.\n    <ul>\n      <li><code class=\"language-plaintext highlighter-rouge\">$ ykman piv access change-management-key -pt<\/code><\/li>\n      <li><code class=\"language-plaintext highlighter-rouge\">-p<\/code>: <em>Store new management key on your YubiKey, protected by PIN. A random key will be used if no key is provided.<\/em><\/li>\n      <li><code class=\"language-plaintext highlighter-rouge\">-t<\/code>: <em>Require touch on YubiKey when prompted for management key.<\/em><\/li>\n      <li>If you\u2019re more paranoid than me, you can use the <code class=\"language-plaintext highlighter-rouge\">-g<\/code> option instead, which will generate a key for you to note down and give back\nlater. However, the extent to which you can cause damage with a management key is limited (you can delete and regenerate keys, but\nnot decrypt data or anything), so this is arguably not worth the hassle \u2013 especially since <em>forgetting<\/em> the management key\nmeans you\u2019d have to reset the PIV applet to make changes.<\/li>\n    <\/ul>\n  <\/li>\n  <li>Generate a public\/private keypair on the key, using the <code class=\"language-plaintext highlighter-rouge\">9a<\/code> (\u201cPIV Authentication\u201d) key slot. We\u2019ll use ECC (elliptic-curve cryptography)\nbecause it\u2019s fast, secure, and has vastly smaller key sizes, and configure the key to always require touches and require the PIN on\nfirst use. You might want to use different settings here.\n    <ul>\n      <li><code class=\"language-plaintext highlighter-rouge\">$ ykman piv keys generate -a ECCP256 --touch-policy ALWAYS --pin-policy ONCE 9a .\/yubikey-public.pem<\/code><\/li>\n      <li><strong>Note<\/strong>: Some outdated SSH servers\/implementations only support RSA keys. If this applies to you, leave off the <code class=\"language-plaintext highlighter-rouge\">-a<\/code> option to use\nRSA 2048 instead.<\/li>\n      <li>See <code class=\"language-plaintext highlighter-rouge\">ykman piv keys generate -h<\/code> for a full description of all available options.<\/li>\n    <\/ul>\n  <\/li>\n  <li>Use the newly generated key to make a self-signed PKCS#11 certificate to act as our SSH identity.\n    <ul>\n      <li><code class=\"language-plaintext highlighter-rouge\">$ ykman piv certificates generate -s 'my-yubikey-ssh' -d 365 9a .\/yubikey-public.pem<\/code><\/li>\n      <li>Modify the <code class=\"language-plaintext highlighter-rouge\">-s<\/code> parameter to include a human-readable description of the key or the machine the key is installed in.<\/li>\n      <li>Modify the <code class=\"language-plaintext highlighter-rouge\">-d<\/code> parameter to set how many days the key will be valid for. The value above specifices 1 year.<\/li>\n      <li>You can get rid of <code class=\"language-plaintext highlighter-rouge\">yubikey-public.pem<\/code> after this step.<\/li>\n    <\/ul>\n  <\/li>\n  <li>Enable yubikey-agent.\n    <ul>\n      <li><code class=\"language-plaintext highlighter-rouge\">$ systemctl --user enable --now yubikey-agent<\/code><\/li>\n      <li><strong>Note<\/strong>: You might need to kill any running instances of <code class=\"language-plaintext highlighter-rouge\">gpg-agent<\/code> if you had that running (and it decided to try and use your\nYubiKey), and potentially restart <code class=\"language-plaintext highlighter-rouge\">pcscd<\/code> after doing so.<\/li>\n      <li><strong>Note<\/strong>: Check that yubikey-agent started properly with <code class=\"language-plaintext highlighter-rouge\">systemctl --user status yubikey-agent<\/code>.\n        <ul>\n          <li>On my machine, it failed to start because of namespacing issues.<\/li>\n          <li>If this happens, edit the unit file with <code class=\"language-plaintext highlighter-rouge\">systemctl --user edit --full yubikey-agent<\/code>, and remove all lines under <code class=\"language-plaintext highlighter-rouge\">[Service]<\/code>\napart from <code class=\"language-plaintext highlighter-rouge\">ExecStart<\/code> and <code class=\"language-plaintext highlighter-rouge\">ExecReload<\/code>.<\/li>\n        <\/ul>\n      <\/li>\n    <\/ul>\n  <\/li>\n  <li>Update <code class=\"language-plaintext highlighter-rouge\">SSH_AUTH_SOCK<\/code> to point to the running instance of <code class=\"language-plaintext highlighter-rouge\">yubikey-agent<\/code>.\n    <ul>\n      <li><code class=\"language-plaintext highlighter-rouge\">$ export SSH_AUTH_SOCK=\/run\/user\/1000\/yubikey-agent\/yubikey-agent.sock<\/code><\/li>\n      <li>The above command assumes bash. Other shells may vary.<\/li>\n      <li>If your user ID isn\u2019t <code class=\"language-plaintext highlighter-rouge\">1000<\/code>, you\u2019ll need to change the above command (you can find the right path by <code class=\"language-plaintext highlighter-rouge\">ps aux | grep yubikey-agent<\/code>).<\/li>\n      <li><strong>Note<\/strong>: You\u2019ll want to put the above command in <code class=\"language-plaintext highlighter-rouge\">.bashrc<\/code> or similar.<\/li>\n    <\/ul>\n  <\/li>\n  <li>Get the key fingerprint for your newly generated SSH identity.\n    <ul>\n      <li><code class=\"language-plaintext highlighter-rouge\">$ ssh-add -L<\/code><\/li>\n    <\/ul>\n  <\/li>\n  <li>Copy the key fingerprint to your remote host(s), and put it in <code class=\"language-plaintext highlighter-rouge\">~\/.ssh\/authorized_keys<\/code>.<\/li>\n  <li>You should now be able to SSH using the YubiKey!\n    <ul>\n      <li><code class=\"language-plaintext highlighter-rouge\">$ ssh &lt;host&gt;<\/code><\/li>\n      <li>The first time you try this, it should pop up a window asking for the PIN, after which you\u2019ll need to touch the flashing YubiKey.<\/li>\n      <li>Subsequent attempts will only require a touch.<\/li>\n      <li>If it didn\u2019t work, check the notes under step 8. In particular, using OpenPGP and SSH simultaneously with the YubiKey usually requires\n you to kill either <code class=\"language-plaintext highlighter-rouge\">pcscd<\/code> or <code class=\"language-plaintext highlighter-rouge\">gpg-agent<\/code>, depending on what you want to do.<\/li>\n    <\/ul>\n  <\/li>\n  <li>Ensure you have a backup plan in place for when this YubiKey fails, or is lost.\n    <ul>\n      <li>The key we generated is local to the YubiKey, and cannot be exported. It can very easily be wiped by anyone running <code class=\"language-plaintext highlighter-rouge\">ykman piv reset<\/code>\n (which requires no authentication!), or you could lose the key.<\/li>\n      <li>Make sure you have another way of getting into all hosts you use this YubiKey with (encrypted regular SSH key, out-of-band console, etc.).<\/li>\n    <\/ul>\n  <\/li>\n  <li>You\u2019re done!<\/li>\n<\/ol>\n\n<hr \/>\n\n<p>Hopefully that worked. If it didn\u2019t, well, there\u2019s a comments thing below if you can be bothered, I guess?<\/p>\n","pubDate":"Sat, 06 Mar 2021 00:00:00 +0000","link":"https:\/\/eta.st\/2021\/03\/06\/yubikey-5-piv.html","guid":"https:\/\/eta.st\/2021\/03\/06\/yubikey-5-piv.html"},{"title":"Setting the tone in a group is very important","description":"<div class=\"nowplaying\" style=\"background: #ffdada;\">\n<b>Warning!<\/b> This post is a mental health \/ feels post from a long time ago (in fact, <a href=\"\/nomenclature\">one gender ago<\/a>).\nIt is kept up in case some people find it useful, but I don't necessarily endorse the content in here nowadays.\n<\/div>\n\n<p>Various people have various different opinions on how they should present themselves. Personally, I\u2019d consider myself quite a sensitive person;\nI\u2019d like to think I try quite hard to take the feelings and circumstances of other people into account when talking to them, although doing so\nis by no means easy or automatic (and I often fail at this, sometimes quite badly). Partially, this is because I often have a lot of feelings\nand circumstances going in my life myself which I\u2019d like other people to attempt to take into account \u2013 in fact, especially nowadays,\nI\u2019d argue that the overwhelming majority of people have some topics or areas that make them uncomfortable, or that it\u2019d be possible to upset\nthem with a correctly targeted remark.<\/p>\n\n<p>It\u2019s very hard to judge what might upset or offend someone, simply because there can be a <em>lot<\/em> of stuff going on behind the scenes that people\njust don\u2019t tell others about (it\u2019s almost as if having <a href=\"https:\/\/en.wikipedia.org\/wiki\/United_Kingdom\">an entire country<\/a> where not talking about\nstuff is the norm could lead to significant problems!). That said, you can at least attempt to be <em>reactive<\/em> \u2013 it\u2019s sometimes possible to detect\nthat something you\u2019ve said or done wasn\u2019t received well, and try to do less of that thing in future.<\/p>\n\n<hr \/>\n\n<p>There are, of course, cultures and groups where this sort of thing is very much not the norm. Some people \u2013 and groups of people \u2013 attempt to\nact in a sort of \u201cmacho\u201d \/ \u201ctough guy\u201d sort of way, where one is supposed to pretend that one doesn\u2019t really have feelings, or that one is immune\nto whatever life might throw in one\u2019s way. This, of course, is obviously false \u2013 everyone has feelings! \u2013 but it suits them to conduct themselves\nin this manner, because, at the end of the day, talking about one\u2019s feelings can be very hard.<\/p>\n\n<p>Maybe you weren\u2019t brought up in an environment where people do that; maybe you never saw your parents, or your school friends, cry, or be angry,\nor show feelings toward things, because they thought they had to be \u2018strong\u2019). Maybe you had people express <em>too much<\/em> of their feelings in your past\nand really got put off by it (as in, maybe <em>you<\/em> were distressed by other people having issues and now have decided that \u2018burdening\u2019 others with\nyour feelings isn\u2019t a good idea). Maybe you don\u2019t have any friends you <em>really<\/em> trust enough to be able to confide in, perhaps because they\u2019re\nall tough guys, or perhaps because you have issues trusting people for some other reason \u2013 especially in our modern society, that reason can\noften be <a href=\"https:\/\/en.wikipedia.org\/wiki\/Loneliness\">loneliness<\/a>, an epidemic that people don\u2019t actually realise the severity of.<\/p>\n\n<p>But anyway, you don\u2019t want to talk about your feelings. I get that, because I don\u2019t really want to either<sup id=\"fnref:1\"><a href=\"#fn:1\" class=\"footnote\" rel=\"footnote\" role=\"doc-noteref\">1<\/a><\/sup>. Being in that position, while\nit\u2019s not really a good thing for you long-term, isn\u2019t really <em>wrong<\/em>; you aren\u2019t hurting anyone except yourself there (although if you read the\nprevious paragraph and found it hit a bit too close to home, you should probably <a href=\"https:\/\/www.youtube.com\/watch?v=n3Xv_g3g-mA\">watch this video<\/a>).<\/p>\n\n<hr \/>\n\n<p>It\u2019s not uncommon for large-ish (i.e. more than 3 or 4) groups of people to have leaders, whether explicitly or implicitly allocated. Usually\nthere are one or two people who do a lot of the talking \u2013 who appear to set the tone and the rules of engagement for the rest of the group.\nThis doesn\u2019t always have to happen, of course, but what I\u2019m saying is it probably happens more than you realise. (It can also become painfully\nobvious when the leaders step out to go and do something else for a bit and you end up with a bunch of people who don\u2019t really know what they\nshould be doing with themselves, which is always a fun scenario!)<\/p>\n\n<p>As you probably gathered from the title, what I really want to emphasise is the whole \u201csetting the tone\u201d aspect of being a leader. This\n<a href=\"http:\/\/jsomers.net\/blog\/it-turns-out\">turns out<\/a> to be important in a bunch of ways that aren\u2019t immediately obvious. I\u2019d hypothesize that\na large part of the issues people who aren\u2019t <a href=\"https:\/\/en.wiktionary.org\/wiki\/cishet#English\">cishet<\/a> white males face in STEM fields, and especially\nprogramming \/ IT, are down to this factor; the groups people tend to hang out in are implicitly using a bunch of norms that probably aren\u2019t\nvery inclusive (think people making slightly inappropriate cheeky comments about women they fancy, but also more subtle mannerisms and ways of\ncommunicating that tend to only be shared by people from a certain background that make it harder for people not from that background to\ncommunicate). A lot of ink has been spilled about this<sup id=\"fnref:2\"><a href=\"#fn:2\" class=\"footnote\" rel=\"footnote\" role=\"doc-noteref\">2<\/a><\/sup> in terms of how companies should try to avoid it when creating teams at work,\nbecause it\u2019s a real problem.<\/p>\n\n<p>The macho people from earlier can face real issues with this sort of thing. Often, their tough-guy behaviour implicitly sets the tone for the\ngroups of people they find themselves in (which usually end up being filled with people who don\u2019t really care, or are also trying to be macho).\nTo avoid opening themselves up to the potential of insecurity, they often tend to do this more forcefully \u2013 terms like \u201cpussy\u201d, \u201cwimp\u201d, et al.\nare often employed for this purpose, wherein such people attempt to claim that people who do have feelings, are afraid of things, etc. are\nsomehow \u2018weaker\u2019 than them.<\/p>\n\n<hr \/>\n\n<p>The astute reader will think that I\u2019m writing this because someone did that to me and I\u2019m angry about it, and this blog post is my way of\nrationalizing their behaviour and asserting that I\u2019m actually a better person than them. And they\u2019d be partially right!<sup id=\"fnref:3\"><a href=\"#fn:3\" class=\"footnote\" rel=\"footnote\" role=\"doc-noteref\">3<\/a><\/sup><\/p>\n\n<p>It\u2019s more nuanced than that, however. The thing that\u2019s actually really sad about these sorts of situations is that the people responsible for\ncreating the harmful no-feelings-allowed environment are often the people most in need of a way to express their feelings (as implied earlier).\nAnd what they\u2019ve managed to do by creating such an environment is ensure they most likely won\u2019t be able to do that thing with those people \u2013\nif they try, it could get awkward (since the others aren\u2019t really happy having a more \u2018deep\u2019 conversation, and that\u2019s why they\u2019re in the group),\nor they might find themselves met with a surprising lack of sympathy (because others actually did have problems and got humiliated for them).<\/p>\n\n<p>I don\u2019t even think you can blame these people, either. They\u2019ve just found themselves in a situation that most likely isn\u2019t even their\nfault, and they don\u2019t really know what they should do to cope with it. If anything, it\u2019s probably society that teaches them to behave in this way \u2013\nand that\u2019s just a <a href=\"https:\/\/www.youtube.com\/watch?v=YL42GI-X5WA\">sad, sad situation<\/a> that\u2019s not exactly easy to fix<sup id=\"fnref:4\"><a href=\"#fn:4\" class=\"footnote\" rel=\"footnote\" role=\"doc-noteref\">4<\/a><\/sup>.<\/p>\n\n<hr \/>\n\n<div class=\"footnotes\" role=\"doc-endnotes\">\n  <ol>\n    <li id=\"fn:1\">\n      <p>No, it suits me to write pseudo-intellectual blog posts that nobody reads that vaguely hint at a whole bunch of screwed up stuff going on.\u00a0<a href=\"#fnref:1\" class=\"reversefootnote\" role=\"doc-backlink\">&#8617;<\/a><\/p>\n    <\/li>\n    <li id=\"fn:2\">\n      <p>Like a lot of the claims on this blog, this one is unsubstantiated. I think it\u2019s true though\u2026\u00a0<a href=\"#fnref:2\" class=\"reversefootnote\" role=\"doc-backlink\">&#8617;<\/a><\/p>\n    <\/li>\n    <li id=\"fn:3\">\n      <p>I mean I really don\u2019t post often, so someone has to have annoyed me for things to get this bad.\u00a0<a href=\"#fnref:3\" class=\"reversefootnote\" role=\"doc-backlink\">&#8617;<\/a><\/p>\n    <\/li>\n    <li id=\"fn:4\">\n      <p>That said, I\u2019ve seen advertising campaigns that try! I think some biscuit company teamed up with some mental health charity to promote the idea of having a cuppa and a chat about your problems with your friends, which is absolutely a good thing (even if Big Biscuit ends up profiting from it)\u00a0<a href=\"#fnref:4\" class=\"reversefootnote\" role=\"doc-backlink\">&#8617;<\/a><\/p>\n    <\/li>\n  <\/ol>\n<\/div>\n","pubDate":"Fri, 15 Jan 2021 00:00:00 +0000","link":"https:\/\/eta.st\/2021\/01\/15\/social-leadership.html","guid":"https:\/\/eta.st\/2021\/01\/15\/social-leadership.html"},{"title":"Strict COVID-19 restrictions in universities are irresponsible","description":"<p><em>This post is about mostly personal circumstances \/ issues, as well as current affairs. If that\u2019s not what you want, turn back now.<\/em><\/p>\n\n<p>At the start of the current <a href=\"https:\/\/en.wikipedia.org\/wiki\/Covid-19\">coronavirus disease 2019 (COVID-19)<\/a> pandemic,\nwe were told that \u201cflattening the curve\u201d was a good idea \u2013 i.e. attempting to limit the spread of the disease by\nstaying at home, wearing face coverings, etc. was a necessary step we should all take in order to prevent the national\nhealth services from getting overwhelmed (leading to an excess of deaths of people who could otherwise be helped).<\/p>\n\n<p>A significant number of months have passed since March, and a new wave of unsuspecting secondary school graduates\nhave descended on the UK\u2019s universities<sup id=\"fnref:1\"><a href=\"#fn:1\" class=\"footnote\" rel=\"footnote\" role=\"doc-noteref\">1<\/a><\/sup> \u2013 but, obviously, since there\u2019s still a pandemic going on, things are\ndifferent from the way they used to be. Pretty much all universities have new precautions to limit the spread of the disease,\nincluding things like<\/p>\n\n<ul>\n  <li>grouping students into (logical) \u201chouseholds\u201d, and restricting interaction between said households<\/li>\n  <li>enforcing social distancing requirements<\/li>\n  <li>enforcing face covering usage<\/li>\n  <li>limiting the number of students that can be in the same place at one time (in line with the nationwide \u201c<a href=\"https:\/\/www.gov.uk\/government\/publications\/coronavirus-covid-19-meeting-with-others-safely-social-distancing\/coronavirus-covid-19-meeting-with-others-safely-social-distancing\">rule of six<\/a>\u201d)<\/li>\n  <li>getting rid of <em>all<\/em> face-to-face tuition, and moving everything online<\/li>\n  <li>adding a curfew to, or closing, pubs and social spaces<\/li>\n<\/ul>\n\n<p>Some of these precautions involve more sacrifices on the part of the students than others; wearing face coverings is relatively\nzero-cost, and has been shown to <a href=\"https:\/\/tinyurl.com\/FAQ-aerosols\">limit the spread of the disease quite significantly<\/a><sup id=\"fnref:2\"><a href=\"#fn:2\" class=\"footnote\" rel=\"footnote\" role=\"doc-noteref\">2<\/a><\/sup>.\nHowever, the goal of the overwhelming majority of the restrictions is clear: <em>limit social interaction as far as practicable<\/em>.\n(This \u2018makes sense\u2019, because social interaction is how the virus is spread.)<\/p>\n\n<p>The point I want to express here is that having that as a goal in the context of universities is somewhat irresponsible, and seems to completely\nignores the mental health concerns of an entire year\u2019s worth of students at university right now<sup id=\"fnref:4\"><a href=\"#fn:4\" class=\"footnote\" rel=\"footnote\" role=\"doc-noteref\">3<\/a><\/sup>. Most students have left the\n(hopefully relatively comfortable) environment of secondary school to come to university \u2013 sometimes in an entirely new city, or\nindeed country. These students typically don\u2019t have many people they can talk to once they arrive, having left the vast majority\nof their friends behind from school; instead, they must somehow discover new people, usually by having a <em>lot<\/em> of spontaneous\ninteractions until they\u2019re able to bed in and start to establish some friendships.<\/p>\n\n<p>It doesn\u2019t take a genius to realise that this process is not compatible with the above stated goal of not having much social interaction.<\/p>\n\n<p>However, what I think is particularly irresponsible is the lack of discussion surrounding the consequences of not letting this process\nplay out as normal. The need for students to socialize and make friends is invariant; <a href=\"https:\/\/www.youtube.com\/watch?v=n3Xv_g3g-mA\">the feeling of loneliness is inherent to being human<\/a>\nand isn\u2019t going away any time soon, so people will (attempt to) socialize to feel less lonely, especially when placed in an unfriendly new environment.\nExamples of consequences arising from a lack of social interactions among students include<\/p>\n\n<ul>\n  <li>greater incidences of mental health problems, as loneliness creates new or exacerbates existing issues<\/li>\n  <li>a reduced ability to even notice and help with such problems, as remote learning can mask all sorts of issues that are more easily recognizable in person<\/li>\n  <li>reduced academic performance and ability, due to previously mentioned mental health problems<\/li>\n  <li>a <a href=\"https:\/\/www.theguardian.com\/education\/2020\/sep\/19\/uk-universities-predict-record-student-dropout-rate\">greater dropout rate<\/a>, leading to reduced\nincome for universities (some of which are already struggling to stay afloat)<\/li>\n  <li>in the extreme case, greater incidences of suicides<\/li>\n<\/ul>\n\n<p>It\u2019s also the case that not everyone is perfectly rule-abiding. While more meek students might follow restrictions and suffer the associated consequences,\nothers will flagrantly disobey them, a fact which has consequences of its own:<\/p>\n\n<ul>\n  <li>instead of socializing in \u2018controlled\u2019 environments, under the purview of (e.g.) student wellbeing officers, students will socialize elsewhere\n(e.g. a random park)\n    <ul>\n      <li>in these \u2018uncontrolled\u2019 environments, a greater prevalence of dangerous behaviours (excessive drinking, drug use, etc.) would be expected<\/li>\n      <li>\u2026but since these are the only opportunities available to undersocialized students, more students might end up taking unwise risks than would otherwise<\/li>\n      <li>there is already evidence to suggest more students are taking drugs <em>and dying from it<\/em> through precisely this mechanism<sup id=\"fnref:3\"><a href=\"#fn:3\" class=\"footnote\" rel=\"footnote\" role=\"doc-noteref\">4<\/a><\/sup><\/li>\n    <\/ul>\n  <\/li>\n  <li>from the perspective of the virus, the replacements for the now-banned opportunities to socialize are likely a lot worse, <em>increasing<\/em> net transmission\n    <ul>\n      <li>think people drinking from the same bottle, cramming themselves into places, not using any face coverings, \u2026 (because there\u2019s no reason for them to)<\/li>\n      <li>also see e.g. <a href=\"https:\/\/www.theguardian.com\/education\/2020\/oct\/06\/manchester-students-organising-covid-positive-parties\">Manchester students organizing COVID-19 transmission parties<\/a><\/li>\n    <\/ul>\n  <\/li>\n<\/ul>\n\n<p>A lot of the problems here tie into greater issues with the discussion of the pandemic in the media and elsewhere; a lot of people seem to\nthink that the <a href=\"https:\/\/coronavirus.data.gov.uk\/\">worrying graph of growing cases<\/a> is unquestionably something that must be dealt with immediately\n(perhaps with a lockdown, which is even worse for students). Don\u2019t get me wrong \u2013 COVID-19 is a deadly disease, and must not be underestimated.\nLetting the disease run completely unchecked throughout the population, without any restrictions whatsoever, is a terrible idea and would kill many people\nunnecessarily; a very contested document called the <a href=\"https:\/\/gbdeclaration.org\/\">Great Barrington declaration<\/a> calls for something akin to that (albeit\nwith protections in place for vulnerable members of society).<\/p>\n\n<p>The reality is that it\u2019s very difficult to come to a decision, and neither extremist view is correct; making everyone sit on their hands until a vaccine\nis available is stupid, but so is letting the disease run wild. There\u2019s much we don\u2019t know about the impacts of the virus, including whether or not\nit has long-term health implications for certain groups (and the conditions under which such long-term complications might arise) \u2013 but sensationalizing\n(e.g. evocative news headlines that attempt to instil fear as to the deadliness of the disease) does not help us come to a reasoned conclusion about risk.<\/p>\n\n<p>To conclude, then, I believe the evidence to support <em>strict<\/em> COVID-19 restrictions in UK universities is questionable, and a re-think\nabout the rationale for, and the consequences of, such strict restrictions is sorely required. It\u2019s really unclear whether the benefits conferred by\nseverely limiting social interaction (at least, imposing rules that attempt to achieve such) are worth the consequences of doing so \u2013 heck, it\u2019s even\nunclear whether people even follow the rules enough to limit transmission at all (and the recent outbreaks in universities across the nation confirm that).<\/p>\n\n<p>A lack of humane thinking seems to be the case amongst those who impose said restrictions; the problem cannot be viewed as a simple mathematical calculation\nof how to reduce cases (if reducing cases is even something worth attempting to do!), but one that leads to significant human suffering for those affected.\nWith the world being more divided and polarised than ever, it\u2019s worth trying to be <em>empathetic<\/em> \u2013 to both see the fear on the part of those pushing for a\nlockdown and limitation of cases, and to recognize the crushing impact restrictions have on the restricted.<\/p>\n\n<div class=\"footnotes\" role=\"doc-endnotes\">\n  <ol>\n    <li id=\"fn:1\">\n      <p>I\u2019m one of these, of course, which is why I\u2019m writing this.\u00a0<a href=\"#fnref:1\" class=\"reversefootnote\" role=\"doc-backlink\">&#8617;<\/a><\/p>\n    <\/li>\n    <li id=\"fn:2\">\n      <p>Even if you disagree with the evidence here, face coverings are still basically zero-cost \u2013 you really don\u2019t sacrifice much by wearing one!\u00a0<a href=\"#fnref:2\" class=\"reversefootnote\" role=\"doc-backlink\">&#8617;<\/a><\/p>\n    <\/li>\n    <li id=\"fn:4\">\n      <p>If you disagree with me, please read the whole article first before getting angry.\u00a0<a href=\"#fnref:4\" class=\"reversefootnote\" role=\"doc-backlink\">&#8617;<\/a><\/p>\n    <\/li>\n    <li id=\"fn:3\">\n      <p>I can\u2019t find a citation for this, so take this claim with a pinch of salt.\u00a0<a href=\"#fnref:3\" class=\"reversefootnote\" role=\"doc-backlink\">&#8617;<\/a><\/p>\n    <\/li>\n  <\/ol>\n<\/div>\n","pubDate":"Sun, 11 Oct 2020 00:00:00 +0000","link":"https:\/\/eta.st\/2020\/10\/11\/university-covid.html","guid":"https:\/\/eta.st\/2020\/10\/11\/university-covid.html"},{"title":"Writing as a form of relief","description":"<div class=\"nowplaying\" style=\"background: #ffdada;\">\n<b>Warning!<\/b> This post is a mental health \/ feels post from a long time ago (in fact, <a href=\"\/nomenclature\">one gender ago<\/a>).\nIt is kept up in case some people find it useful, but I don't necessarily endorse the content in here nowadays.\n<\/div>\n\n<p>The content on this blog does not get updated very frequently.<\/p>\n\n<p>This is largely because, as a somewhat permanent and public sort of thing, I have to be quite careful about what stuff I write on here,\nsince it could come back to bite me later, right? We\u2019ve all heard the stories about people getting turned down from job offers due to\nsome embarrassing stuff they posted on the social network <em>du jour<\/em><sup id=\"fnref:1\"><a href=\"#fn:1\" class=\"footnote\" rel=\"footnote\" role=\"doc-noteref\">1<\/a><\/sup><sup id=\"fnref:2\"><a href=\"#fn:2\" class=\"footnote\" rel=\"footnote\" role=\"doc-noteref\">2<\/a><\/sup>, and I generally get the impression that being careful with\nwhat you choose to write into your permanent online record is generally a good thing. (Well, when you phrase it like that\u2026)<\/p>\n\n<p>What are blogs for, though? As far as I can tell, almost nobody reads this one; there are a few stragglers who come here through Google\nbecause I posted about something like <a href=\"\/2018\/02\/07\/microg.html\">microG<\/a> or Rust programming (even though there are now <em>much<\/em> better\nresources to learn Rust out there, given the language has changed a whole load since I started learning it\u2026).<\/p>\n\n<p>Furthermore, I don\u2019t\nconsider myself the kind of person who\u2019s happy to go and do lots of writing about technical topics (for the moment, at least).\nSome people can sustain an entire blogging habit by packing things full of interesting technical content \/ deep dives \/ whatever.\nThis is great, because then the content is at least \u2018useful\u2019 to the person reading it (in some sense<sup id=\"fnref:3\"><a href=\"#fn:3\" class=\"footnote\" rel=\"footnote\" role=\"doc-noteref\">3<\/a><\/sup>), instead of being some poor sap\nwhining on about random other things happening in their life.<\/p>\n\n<p>Unfortunately, though, I\u2019m not one of these people. So, either I go about my life and don\u2019t write any part of it up on the blog, or\nI do the converse, and end up spewing things I\u2019ll probably regret later out into the web at large.<\/p>\n\n<p>There\u2019s a <a href=\"https:\/\/en.wikipedia.org\/wiki\/COVID-19_pandemic\">pandemic<\/a> on. Everyone\u2019s feeling miserable, a lot of people have very tragically\nlost their lives to the COVID-19 virus, and people are beginning to question all sorts of things about the existence we led before this all\nstarted.<\/p>\n\n<p>So, what the hell, let\u2019s just get on with it then.<\/p>\n\n<h2 id=\"the-utility-of-personal-content\">The utility of personal content<\/h2>\n\n<p>Some people might be of the opinion that personal content (like somewhat soppy blog posts) is not worth reading, and should perhaps be gotten rid\nof entirely. However \u2013 and obviously I\u2019m going to be biased here! \u2013 I don\u2019t really think so. Okay, I think the sort of content where people\ntalk about whatever mundane things they\u2019ve been getting up to (\u201cso, this week, I washed my bike, went out for a run, tinkered with Node.js a bit\u2026\u201d)\nis perhaps a bit of a waste of time \u2013 I\u2019d call that \u2018oversharing\u2019, perhaps. But I do think it\u2019s possible to read stuff about someone else\u2019s problems\nand gain some insight into how you might be able to solve your own, so I don\u2019t really want to dismiss personal content entirely.<\/p>\n\n<p>I guess there\u2019s a distinction between content that is purely <em>descriptive<\/em> \u2013 explaining how much you hate yourself, or how annoying thing X is, or\nwhatever \u2013 and content that has an <em>analytical<\/em> or <em>empathetic<\/em> component as well \u2013 trying to figure out the reasons why this is the case and provide some advice\nto people feeling the same way, or otherwise attempting to connect your own personal experience to what others may feel.\nThe former has no value to\nthe reader, really \u2013 oh, poor random internet commentator. How sad. Imagine an agony aunt column without the agony aunt\u2019s responses. How awful would that be?\nBut the latter kind of stuff can definitely be of some value; I\u2019ve read things on the web that have influenced the way I look at the world and respond to things \u2013\nmost people probably have. So it\u2019s not entirely worthless!<\/p>\n\n<h2 id=\"the-title-of-this-post\">The title of this post<\/h2>\n\n<p>In fact, I think writing about things is a great way to process and deal with said things. I\u2019m not just talking about personal or emotional matters \u2013 way back in 2016\nwhen I wrote a short <a href=\"\/2016\/03\/12\/learn-you-a-rust-for-great-good.html\">Rust tutorial series<\/a>, the aim was as much to inform others as to force me to be honest about my\nown Rust abilities; writing something up gets you to specify what exactly you mean in plain English, which can be great for identifying gaps in your knowledge, or areas\nof flawed thinking.<\/p>\n\n<p>This is partially why, as it says in the title, writing can be a form of relief; there\u2019s something about putting pen to paper that makes you feel just a bit better about\nwhatever it is you\u2019re writing about, be that your frustrations learning a new programming language or something more personal.<\/p>\n\n<p>It\u2019s also, in some ways, a lot lower friction than talking to someone about something. If you start calling up your friends and ranting to them about how much asynchronous\nprogramming paradigms suck, you eventually lose most of your friends \u2013 whereas you aren\u2019t going to annoy anyone, or take up anyone\u2019s time, by writing about things<sup id=\"fnref:4\"><a href=\"#fn:4\" class=\"footnote\" rel=\"footnote\" role=\"doc-noteref\">4<\/a><\/sup>.\n(Unless you do something crazy like start sending your friends letters in the post containing your rants. This is also a good way to lose most of your friends.)<\/p>\n\n<h2 id=\"mental-health-awareness\">\u2018Mental health awareness\u2019<\/h2>\n\n<p>Now, of course, you don\u2019t actually have to publish anything to get these benefits; simply writing something up should be enough. (This is the idea behind journaling, I think.)\nIn fact, as I discussed at the start, publishing things can be harmful to your career.<\/p>\n\n<p>However, I think it\u2019s still worth doing: I\u2019m a human being, and you are too. There are thousands of tech blogs that just talk about tech and don\u2019t talk about anything personal\nor human; there are thousands of people who only talk about technical topics on their website and never mention a thing about their private lives. I\u2019m not saying they should \u2013\nbut I do tend to think that seeing other people talk about their problems publicly can be a great motivator for you to do the same (for example, I\u2019m a big fan of\n<a href=\"https:\/\/rachelbythebay.com\/w\/\">rachelbythebay.com<\/a> and her occasional post about toxic Silicon Valley culture). To me, that\u2019s what this concept of \u2018mental health awareness\u2019\nis about (at least in part): recognizing that other people are people too, and trying to get people to talk more openly about their thoughts and feelings, instead of just keeping\nthem to themselves.<\/p>\n\n<p>So, yeah. Write (somewhat critically) about things that bother you, even if they aren\u2019t technical. It\u2019s helpful for you, and you never know what impact it\u2019ll have on somebody else!<\/p>\n\n<p>Or, you know, just don\u2019t, if you\u2019re not into that sort of thing. But I\u2019m going to give it a try.<\/p>\n\n<hr \/>\n\n<p><em>Also, the hope is that just trying to get into a semi-regular pattern of writing about \/anything\/ without much of a filter will mean that more technical stuff seeps out as well.\nWe\u2019ll see what happens!<\/em><\/p>\n\n<hr \/>\n\n<div class=\"footnotes\" role=\"doc-endnotes\">\n  <ol>\n    <li id=\"fn:1\">\n      <p>Well, people tell me this happens. I\u2019m not, ehm, <em>experienced<\/em> enough to actually have heard of this happening first-hand.\u00a0<a href=\"#fnref:1\" class=\"reversefootnote\" role=\"doc-backlink\">&#8617;<\/a><\/p>\n    <\/li>\n    <li id=\"fn:2\">\n      <p>On a related note, if you\u2019re someone who might have the capability to make me a job offer, just\u2026 do me a solid and don\u2019t read the blog, okay? <code class=\"language-plaintext highlighter-rouge\">:p<\/code>\u00a0<a href=\"#fnref:2\" class=\"reversefootnote\" role=\"doc-backlink\">&#8617;<\/a><\/p>\n    <\/li>\n    <li id=\"fn:3\">\n      <p>More on this later.\u00a0<a href=\"#fnref:3\" class=\"reversefootnote\" role=\"doc-backlink\">&#8617;<\/a><\/p>\n    <\/li>\n    <li id=\"fn:4\">\n      <p>You also are probably not going to annoy your friends by talking about personal issues if you really feel the need to talk to someone about them, since that\u2019s what friends are for! However, it doesn\u2019t feel too great having to do this a lot (where \u2018a lot\u2019 is subjectively defined) \u2013 in other words, even though your friends might not actually get annoyed, your fear of them getting annoyed (and perhaps not telling you) might be enough to make you not want to talk to them.\u00a0<a href=\"#fnref:4\" class=\"reversefootnote\" role=\"doc-backlink\">&#8617;<\/a><\/p>\n    <\/li>\n  <\/ol>\n<\/div>\n","pubDate":"Thu, 21 May 2020 00:00:00 +0000","link":"https:\/\/eta.st\/2020\/05\/21\/personal-stuff.html","guid":"https:\/\/eta.st\/2020\/05\/21\/personal-stuff.html"},{"title":"Somewhat contrived schema designing for a new chat system","description":"<p><em>[This is post 5 about designing a new chat system. Have a look at <a href=\"\/2019\/09\/10\/chat-systems.html\">the first post in the series<\/a> for more context!]<\/em><\/p>\n\n<p>This post follows on from the <a href=\"\/2019\/12\/18\/nea-schema.html\">previous one in the series<\/a><sup id=\"fnref:1\"><a href=\"#fn:1\" class=\"footnote\" rel=\"footnote\" role=\"doc-noteref\">1<\/a><\/sup>,\nwherein I had a shot at designing \/ specifying what <em>state<\/em> \u2013 persistent information, shared amongst all\nservers in a federated system \u2013 in group chats should look like. To summarize, we ended up with the group chat state containing\nthree important things:<\/p>\n\n<ul>\n  <li>a set of <em>roles<\/em>, which are a way of grouping together <em>capabilities<\/em> available to users with said roles\n    <ul>\n      <li>Remember, <em>capabilities<\/em> are simple keywords like <code class=\"language-plaintext highlighter-rouge\">speak<\/code> or <code class=\"language-plaintext highlighter-rouge\">change-topic<\/code> that represent actions users can take<\/li>\n    <\/ul>\n  <\/li>\n  <li>a list of <em>memberships<\/em> (users in the chat), together with the <em>role<\/em> for each chat member<\/li>\n  <li><em>non-user-related state<\/em>, like the chatroom topic, which sort of follows a key-value store\n    <ul>\n      <li>We figured out that allowing arbitrary stuff to be stored in a room\u2019s state was a bad idea, so this just contains\u2026some random fields we\u2019ll specify more formally later<sup id=\"fnref:2\"><a href=\"#fn:2\" class=\"footnote\" rel=\"footnote\" role=\"doc-noteref\">2<\/a><\/sup>.<\/li>\n    <\/ul>\n  <\/li>\n<\/ul>\n\n<p>In this post, we\u2019ll look into how this state will be represented in server databases, and spec out a <em>database schema<\/em> for our \nserver implementations to use<sup id=\"fnref:3\"><a href=\"#fn:3\" class=\"footnote\" rel=\"footnote\" role=\"doc-noteref\">3<\/a><\/sup>.<\/p>\n\n<h2 id=\"unpacking-our-group-chat-state-object\">Unpacking our group chat state object<\/h2>\n\n<p>We <em>could<\/em> just store the group chat state as a big JSON blob in the database (indeed, if we were using something like\n<a href=\"https:\/\/www.mongodb.com\/\">MongoDB<\/a>, that would be commonplace). However, this probably isn\u2019t a good idea \u2013 for a number of reasons:<\/p>\n\n<ul>\n  <li>we\u2019d have to retrieve the whole thing every time we wanted to access information about it, which is suboptimal\nperformance-wise<\/li>\n  <li>things in the blob could quietly become inconsistent with the rest of the database if we didn\u2019t check it all the time<\/li>\n  <li>the database wouldn\u2019t be able to enforce any schemas; we\u2019d have to do that in our application code<\/li>\n  <li>unless we (ab)use something like PostgreSQL\u2019s native JSON support, our group chat state would be completely opaque\nfrom the database\u2019s point of view \u2013 meaning it\u2019d be hard to draw links between things in there (e.g. user IDs)\nand the rest of the database<\/li>\n<\/ul>\n\n<p>These concerns are similar to the concerns <a href=\"https:\/\/en.wikipedia.org\/wiki\/Third_normal_form\">third-normal form (3NF)<\/a>, a way of structuring database\nschemas from 1971, hopes to address. Under 3NF, you store your data in a set of database tables representing various objects,\nwith each table having a <a href=\"https:\/\/en.wikipedia.org\/wiki\/Primary_key\">primary key<\/a> (an identifier or set of identifiers uniquely identifying that object).\n3NF then states that other information in each table must <em>only<\/em> tell you something about the primary key, and nothing else;\nthey aren\u2019t allowed to depend on anything other than the value of the primary key.<\/p>\n\n<p>As a more concrete example, let\u2019s say we have a User table representing a user of our chat system, where the primary key\nis a combination of a username and the user\u2019s home server. If I wanted to add a column describing what channels they\u2019re in,\nfor example, that would be fine \u2013 but if I wanted to also add the most recent messages for each channel, say\n(let\u2019s imagine you\u2019re designing a UI like WhatsApp\u2019s, with a homescreen that shows this information), that wouldn\u2019t be valid\nunder 3NF, because the most recent messages are a property of the channel, not the user. We could end up having two users\nin the same channel, and forget to keep this \u2018recent messages sent\u2019 property consistent, which would lead to confusion!<\/p>\n\n<p>So, using 3NF seems like a pretty good idea \u2013 and that\u2019s exactly what we\u2019re going to do! Our group chat state object doesn\u2019t\nfit 3NF as one large blob, so we\u2019re going to need to decompose it into a set of database tables that store the same information.<\/p>\n\n<h2 id=\"lets-do-some-schema-designing\">Let\u2019s do some schema designing!<\/h2>\n\n<h3 id=\"groupchats-our-starting-point\"><code class=\"language-plaintext highlighter-rouge\">groupchats<\/code>: our starting point<\/h3>\n\n<p>Of course, we need some object to represent a group chat. Since group chats are going to be shared across different servers,\nwe have to choose some identifier that\u2019s always going to be unique \u2013 we can\u2019t just give them textual names, otherwise they\u2019d\nbe a possibility of them clashing. Instead, we\u2019ll use a <a href=\"https:\/\/en.wikipedia.org\/wiki\/Universally_unique_identifier\">universally unique identifier (UUID)<\/a> \u2013 it does what it says on the\ntin!<\/p>\n\n<p>Our set of group chat state (for now) is in the list below. I\u2019ve also put \ud83d\udeab next to things we can\u2019t put in the <code class=\"language-plaintext highlighter-rouge\">groupchats<\/code>\ntable directly due to 3NF, and explained why.<\/p>\n\n<ul>\n  <li>topic \/ subject\n    <ul>\n      <li>This is purely a property of the group chat itself, and it doesn\u2019t depend on anything else.<\/li>\n    <\/ul>\n  <\/li>\n  <li>list of users \ud83d\udeab\n    <ul>\n      <li>This one technically could be a property of the group chat, but making it one isn\u2019t a great idea.<\/li>\n      <li>Firstly, that\u2019d mean we\u2019d have to use an array, which is generally frowned upon; it makes it harder to do things like\ndatabase table <code class=\"language-plaintext highlighter-rouge\">JOIN<\/code>s when the users are stuck in an array attribute.<\/li>\n      <li>Also, we probably want to associate some information with a user\u2019s membership, like their role. Doing that in the\n<code class=\"language-plaintext highlighter-rouge\">groupchats<\/code> table would be a big 3NF violation.<\/li>\n    <\/ul>\n  <\/li>\n  <li>list of defined roles \ud83d\udeab\n    <ul>\n      <li>Roles have capabilities associated with them, so they should be their own thing.<\/li>\n      <li>Said otherwise, our primary key is the channel\u2019s UUID, not (channel UUID, role), so storing capabilities (which\ndepend on those two things) would be a 3NF violation.<\/li>\n    <\/ul>\n  <\/li>\n  <li>mapping of users to what roles they have \ud83d\udeab\n    <ul>\n      <li>Similarly to the last item, this mapping introduces a 3NF violation.<\/li>\n      <li>We\u2019ll probably end up doing this one in a separate object, as discussed above.<\/li>\n    <\/ul>\n  <\/li>\n  <li>list of servers involved in this group chat, as well as whether they\u2019re sponsoring or not \ud83d\udeab\n    <ul>\n      <li>Ditto, really.<\/li>\n    <\/ul>\n  <\/li>\n  <li>current state version\n    <ul>\n      <li>We need to keep track of what state version we\u2019re on (remember, the state version is a monotonically incrementing integer),\nfor the purposes of our consensus algorithm.<\/li>\n    <\/ul>\n  <\/li>\n<\/ul>\n\n<p>So, now that that\u2019s all clear, we\u2019re left with group chat UUID, subject, and current state version. Here\u2019s the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Data_definition_language\">SQL DDL<\/a>:<\/p>\n\n<div class=\"language-sql highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"k\">CREATE<\/span> <span class=\"k\">TABLE<\/span> <span class=\"n\">groupchats<\/span> <span class=\"p\">(<\/span>\n    <span class=\"n\">uuid<\/span> <span class=\"n\">UUID<\/span> <span class=\"k\">PRIMARY<\/span> <span class=\"k\">KEY<\/span><span class=\"p\">,<\/span>\n    <span class=\"n\">state_ver<\/span> <span class=\"nb\">INT<\/span> <span class=\"k\">NOT<\/span> <span class=\"k\">NULL<\/span><span class=\"p\">,<\/span>\n    <span class=\"n\">subject<\/span> <span class=\"nb\">VARCHAR<\/span> <span class=\"c1\">-- can be null, if a group chat is unnamed.<\/span>\n<span class=\"p\">);<\/span>\n<\/code><\/pre><\/div><\/div>\n\n<p>(We\u2019ll include the DDL for each table in our schema.)<\/p>\n\n<h3 id=\"groupchat_roles-and-groupchat_role_capabilities-storing-group-chat-role-information\"><code class=\"language-plaintext highlighter-rouge\">groupchat_roles<\/code> and <code class=\"language-plaintext highlighter-rouge\">groupchat_role_capabilities<\/code>: storing group chat role information<\/h3>\n\n<p>Before we can actually express user memberships, we need something to store group chat role information;\nwhat roles exist, and what capabilities are associated with them. Behold:<\/p>\n\n<div class=\"language-sql highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"k\">CREATE<\/span> <span class=\"k\">TABLE<\/span> <span class=\"n\">groupchat_roles<\/span> <span class=\"p\">(<\/span>\n    <span class=\"n\">role_id<\/span> <span class=\"nb\">SERIAL<\/span> <span class=\"k\">PRIMARY<\/span> <span class=\"k\">KEY<\/span><span class=\"p\">,<\/span>\n    <span class=\"n\">groupchat_uuid<\/span> <span class=\"n\">UUID<\/span> <span class=\"k\">NOT<\/span> <span class=\"k\">NULL<\/span> <span class=\"k\">REFERENCES<\/span> <span class=\"n\">groupchats<\/span><span class=\"p\">,<\/span>\n    <span class=\"n\">role_name<\/span> <span class=\"nb\">VARCHAR<\/span> <span class=\"k\">NOT<\/span> <span class=\"k\">NULL<\/span><span class=\"p\">,<\/span>\n    <span class=\"k\">UNIQUE<\/span><span class=\"p\">(<\/span><span class=\"k\">group<\/span> <span class=\"n\">chat_uuid<\/span><span class=\"p\">,<\/span> <span class=\"n\">role_name<\/span><span class=\"p\">)<\/span>\n<span class=\"p\">);<\/span>\n<\/code><\/pre><\/div><\/div>\n\n<div class=\"language-sql highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"k\">CREATE<\/span> <span class=\"k\">TABLE<\/span> <span class=\"n\">groupchat_role_capabilities<\/span> <span class=\"p\">(<\/span>\n    <span class=\"n\">role_id<\/span> <span class=\"nb\">INT<\/span> <span class=\"k\">NOT<\/span> <span class=\"k\">NULL<\/span> <span class=\"k\">REFERENCES<\/span> <span class=\"n\">groupchat_roles<\/span><span class=\"p\">,<\/span>\n    <span class=\"n\">capability<\/span> <span class=\"nb\">VARCHAR<\/span> <span class=\"k\">NOT<\/span> <span class=\"k\">NULL<\/span><span class=\"p\">,<\/span>\n    <span class=\"k\">UNIQUE<\/span><span class=\"p\">(<\/span><span class=\"n\">role_id<\/span><span class=\"p\">,<\/span> <span class=\"n\">capability<\/span><span class=\"p\">)<\/span>\n<span class=\"p\">);<\/span>\n<\/code><\/pre><\/div><\/div>\n\n<p>A row in the <code class=\"language-plaintext highlighter-rouge\">groupchat_roles<\/code> table represents a role name in a group chat. Role names are unique per group chat, so\nthe 2-tuple <code class=\"language-plaintext highlighter-rouge\">(groupchat_uuid, role_name)<\/code> is unique; the only bit of information associated with a role\nis a list of capabilities, but we aren\u2019t going to use arrays (q.v.), so the separate <code class=\"language-plaintext highlighter-rouge\">groupchat_role_capabilities<\/code>\ntable represents capabilities granted to users with a given role.<\/p>\n\n<p>We\u2019ve given roles an internal integer ID\n(<code class=\"language-plaintext highlighter-rouge\">role_id<\/code>) just to make the primary key less annoying; the \u2018real\u2019 primary key should be <code class=\"language-plaintext highlighter-rouge\">(groupchat_uuid, role_name)<\/code>,\nbut that\u2019d be a real pain to refer to in the <code class=\"language-plaintext highlighter-rouge\">groupchat_role_capabilities<\/code> table (we\u2019d have to store both the UUID\nand the role name! So much wasted space!<sup id=\"fnref:4\"><a href=\"#fn:4\" class=\"footnote\" rel=\"footnote\" role=\"doc-noteref\">4<\/a><\/sup>), so we just use an integer instead.<\/p>\n\n<h3 id=\"groupchat_memberships-associating-users-with-group-chats\"><code class=\"language-plaintext highlighter-rouge\">groupchat_memberships<\/code>: associating users with group chats<\/h3>\n\n<p>Now that we\u2019ve got a group chat table, and a way of expressing user roles,\nwe want a way to express the set of users that are in said group chats, along with the role they have.\nThis is very simple \u2013 we have a <code class=\"language-plaintext highlighter-rouge\">(groupchat, user)<\/code> primary key, and a <code class=\"language-plaintext highlighter-rouge\">role<\/code> foreign key.<\/p>\n\n<div class=\"language-sql highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"k\">CREATE<\/span> <span class=\"k\">TABLE<\/span> <span class=\"n\">groupchat_memberships<\/span> <span class=\"p\">(<\/span>\n    <span class=\"n\">groupchat_uuid<\/span> <span class=\"n\">UUID<\/span> <span class=\"k\">NOT<\/span> <span class=\"k\">NULL<\/span> <span class=\"k\">REFERENCES<\/span> <span class=\"n\">groupchats<\/span><span class=\"p\">,<\/span>\n    <span class=\"n\">user_id<\/span> <span class=\"nb\">INT<\/span> <span class=\"k\">NOT<\/span> <span class=\"k\">NULL<\/span> <span class=\"k\">REFERENCES<\/span> <span class=\"n\">users<\/span><span class=\"p\">,<\/span>\n    <span class=\"n\">role_id<\/span> <span class=\"nb\">INT<\/span> <span class=\"k\">NOT<\/span> <span class=\"k\">NULL<\/span> <span class=\"k\">REFERENCES<\/span> <span class=\"n\">groupchat_roles<\/span><span class=\"p\">,<\/span>\n    <span class=\"k\">PRIMARY<\/span> <span class=\"k\">KEY<\/span><span class=\"p\">(<\/span><span class=\"n\">groupchat_uuid<\/span><span class=\"p\">,<\/span> <span class=\"n\">user_id<\/span><span class=\"p\">)<\/span>\n<span class=\"p\">);<\/span>\n<\/code><\/pre><\/div><\/div>\n\n<p>I\u2019m deliberately not going to mention what\u2019s in the <code class=\"language-plaintext highlighter-rouge\">users<\/code> table yet; we\u2019re going to discuss that in another blog post.<\/p>\n\n<h3 id=\"groupchat_sponsoring_servers-associating-sponsoring-servers-with-group-chats\"><code class=\"language-plaintext highlighter-rouge\">groupchat_sponsoring_servers<\/code>: associating sponsoring servers with group chats<\/h3>\n\n<p>We also need to associate servers with groups somehow. This <em>is<\/em> done for us partially, in that users reside on servers,\nand so naturally the set of servers associated with a given group chat are just the servers on which the members reside \u2013\nbut that doesn\u2019t take into account the fact that some of these servers might be <em>sponsoring servers<\/em> for the purposes\nof federation.<\/p>\n\n<p>Enter the <code class=\"language-plaintext highlighter-rouge\">groupchat_sponsoring_servers<\/code> table:<\/p>\n\n<div class=\"language-sql highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"k\">CREATE<\/span> <span class=\"k\">TABLE<\/span> <span class=\"n\">groupchat_sponsoring_servers<\/span> <span class=\"p\">(<\/span>\n    <span class=\"n\">groupchat_uuid<\/span> <span class=\"n\">UUID<\/span> <span class=\"k\">NOT<\/span> <span class=\"k\">NULL<\/span> <span class=\"k\">REFERENCES<\/span> <span class=\"n\">groupchats<\/span><span class=\"p\">,<\/span>\n    <span class=\"n\">server_id<\/span> <span class=\"nb\">INT<\/span> <span class=\"k\">NOT<\/span> <span class=\"k\">NULL<\/span> <span class=\"k\">REFERENCES<\/span> <span class=\"n\">servers<\/span><span class=\"p\">,<\/span>\n    <span class=\"k\">PRIMARY<\/span> <span class=\"k\">KEY<\/span><span class=\"p\">(<\/span><span class=\"n\">groupchat_uuid<\/span><span class=\"p\">,<\/span> <span class=\"n\">server_id<\/span><span class=\"p\">)<\/span>\n<span class=\"p\">)<\/span>\n<\/code><\/pre><\/div><\/div>\n\n<p>Again, there\u2019s this mysterious <code class=\"language-plaintext highlighter-rouge\">servers<\/code> table we haven\u2019t got to.<\/p>\n\n<hr \/>\n\n<h2 id=\"next-steps\">Next steps<\/h2>\n\n<p>We\u2019ve now established what principles we\u2019re going to use to design our schema (3NF), and we\u2019ve got 5 lovely tables that\nexpress all the group chat state stuff we\u2019ve been jabbering on about for the last two blog posts in proper SQL DDL\nthat we could actually use!<\/p>\n\n<p>We now need to tackle the other two important parts of our schema; we\u2019ve done group chats, but <em>messages<\/em> and <em>users<\/em>\nare yet to be specified. We\u2019ll need to discuss a few important points about how we design our protocol to fit both of these,\nas there\u2019s more to it than you might think! (For example, a user might seem simple \u2013 just someone on a server somewhere, right? \u2013\nbut what about them having a profile picture, or a bit of \u2018status\u2019 text describing what they\u2019re up to, or things like that?)<\/p>\n\n<p>All of that will come in the next blog post in the series, coming (hopefully quite) soon to a website near you!<\/p>\n\n<hr \/>\n\n<p><em>Hey, those deadlines really are quite scary. Wouldn\u2019t it be lovely if we had a <a href=\"\/blah\/nea-roadmap.html\">roadmap<\/a>, with\nnice intermediate dates on it, so we could actually plan stuff?<\/em><\/p>\n\n<hr \/>\n\n<div class=\"footnotes\" role=\"doc-endnotes\">\n  <ol>\n    <li id=\"fn:1\">\n      <p>Reading this post might be somewhat confusing, if you haven\u2019t read that one!\u00a0<a href=\"#fnref:1\" class=\"reversefootnote\" role=\"doc-backlink\">&#8617;<\/a><\/p>\n    <\/li>\n    <li id=\"fn:2\">\n      <p>This blog series is more \u201cbroad brushstrokes\u201d than \u201cexhaustive details\u201d \u2013 because otherwise both you and I would get horrifically bored. Don\u2019t worry, though \u2013 the exhaustive details will turn up somewhere and be featured in the final thing\u2026\u00a0<a href=\"#fnref:2\" class=\"reversefootnote\" role=\"doc-backlink\">&#8617;<\/a><\/p>\n    <\/li>\n    <li id=\"fn:3\">\n      <p>Of course, this schema isn\u2019t part of the specification. Server implementors can do whatever they want; there does, however, have to be <em>a<\/em> reference implementation out there\u2026\u00a0<a href=\"#fnref:3\" class=\"reversefootnote\" role=\"doc-backlink\">&#8617;<\/a><\/p>\n    <\/li>\n    <li id=\"fn:4\">\n      <p>If we\u2019re really wanted to save space, of course, we\u2019d refer to the <code class=\"language-plaintext highlighter-rouge\">group chats<\/code> table itself using an integer instead of a UUID (because a UUID is a blob of four integers, I think). I definitely draw the line at foreign composite primary keys, though\u2026\u00a0<a href=\"#fnref:4\" class=\"reversefootnote\" role=\"doc-backlink\">&#8617;<\/a><\/p>\n    <\/li>\n  <\/ol>\n<\/div>\n","pubDate":"Thu, 26 Dec 2019 00:00:00 +0000","link":"https:\/\/eta.st\/2019\/12\/26\/nea-schema-2.html","guid":"https:\/\/eta.st\/2019\/12\/26\/nea-schema-2.html"},{"title":"Designing group chat state for a new chat system","description":"<p><em>[This is post 4 about designing a new chat system. Have a look at <a href=\"\/2019\/09\/10\/chat-systems.html\">the first post in the series<\/a> for more context!]<\/em><\/p>\n\n<p>This is potentially the point at which the previous few blog posts, which have talked in\nvague, vacuous terms about the possibility of designing some new utopian chat system, start\nto gradually become more about actually getting the work done \u2013 purely due to the fact that,\nas previously mentioned, I have a deadline! It\u2019s been a while<sup id=\"fnref:1\"><a href=\"#fn:1\" class=\"footnote\" rel=\"footnote\" role=\"doc-noteref\">1<\/a><\/sup> since I last said anything\nabout the chat systems project \u2013 so, let\u2019s get things rolling again with a practical post about\nour database schema, shall we?<\/p>\n\n<h2 id=\"the-importance-of-a-good-schema\">The importance of a good schema<\/h2>\n\n<blockquote>\n  <p>\u201cBad programmers worry about the code. Good programmers worry about data structures and their relationships.\u201d<\/p>\n\n  <p>~ <a href=\"https:\/\/lwn.net\/Articles\/193245\/\">Linus Torvalds<\/a><\/p>\n<\/blockquote>\n\n<p>It\u2019s been said, as you can see in the quote above, that thinking hard about your <em>data structures<\/em> \u2013 the way you choose\nto store whatever data it is that you\u2019re processing \u2013 is vitally important to the success of whatever it is that you\u2019re\nbuilding \u2013 so it\u2019s worth having good ones, or else you\u2019ll end up with hacky code that doesn\u2019t quite work right<sup id=\"fnref:2\"><a href=\"#fn:2\" class=\"footnote\" rel=\"footnote\" role=\"doc-noteref\">2<\/a><\/sup>.\nIf you have the right data structures, the code usually sort of flows in to fill the gaps and make everything work right \u2013 whereas\nif you do the code first, it doesn\u2019t usually work as nicely.<\/p>\n\n<p>Now, data structures come in all shapes and sizes \u2013 there are trees, hash tables, association lists, binary heaps, and all sorts of\nother fun stuff that you\u2019d probably find in some CS textbook somewhere. We <em>could<\/em> use some of these for our new chat system \u2013 and,\nin fact, we probably will use the odd hash table or two. However, given I want to keep things nice and boring, sticking to proven,\nreliable technologies for this project<sup id=\"fnref:3\"><a href=\"#fn:3\" class=\"footnote\" rel=\"footnote\" role=\"doc-noteref\">3<\/a><\/sup>, we\u2019re probably just going to store everything in a set of <a href=\"https:\/\/en.wikipedia.org\/wiki\/Relational_database\">relational database<\/a> tables\n(i.e. rows and columns!).<\/p>\n\n<p>And, a <em>schema<\/em> is essentially a description of a set of what columns mean what, and what tables you\u2019re going to have. Which is what\nI\u2019m going to write now, so, without further ado\u2026<\/p>\n\n<h2 id=\"what-do-we-actually-want\">What do we actually want?<\/h2>\n\n<p>It\u2019s a good idea to start with a discussion of what data we have, and what we\u2019re trying to get out of that data. We\u2019ve said that we\nwant our new system to <a href=\"\/2019\/10\/10\/nea-federation-design.html\">support our own funky blend of federation<\/a>, so that\u2019s going to need to be accounted for. Naturally,\na chat service will have a bunch of <strong>users<\/strong>, each with their own usernames, passwords, emails, and other information that needs to be stored\nabout them. We\u2019ll probably have some <strong>messages<\/strong> as well, given users tend to make a lot of those when you give them the chance to.<\/p>\n\n<p>As we\u2019ve <a href=\"\/2019\/09\/26\/nea-federation.html#key-implementation-goals\">also discussed before<\/a>, the primary function of our chat service is to convey messages from their source to their recipient,\nand do so in a reliable manner. That implies further that, in addition to the messages themselves, we\u2019d also benefit from storing information\nabout message <strong>delivery<\/strong> \u2013 did the messages get through, in the end, or do we need to try again sometime?<\/p>\n\n<p>Since we\u2019re supporting <strong>group chats<\/strong>, those also need to have some information stored about them. Our federation protocol<sup id=\"fnref:4\"><a href=\"#fn:4\" class=\"footnote\" rel=\"footnote\" role=\"doc-noteref\">4<\/a><\/sup> requires us to store\nmultiple different \u2018versions\u2019 of group chat state (remember, \u2018state\u2019 refers to things like the list of members of the chat, who has admin rights,\nand what the current topic is) \u2013 because it\u2019s based on this whole funky consensus protocol stuff, we\u2019ll need to keep track of what servers have\nproposed changes, and which changes we decided to actually accept.<\/p>\n\n<h2 id=\"a-model-for-group-chat-state\">A model for group chat state<\/h2>\n\n<p>The consensus algorithms we looked into previously allow us to get a bunch of trusted servers (\u2018sponsoring servers\u2019) to agree on something, where\n\u2018something\u2019 is just some arbitrary value \u2013 usually with some kind of monotonically incrementing identifier or version number. It thus follows that\nwe need some model for what that \u2018something\u2019 will look like; how will servers communicate information about a room\u2019s state to one another?<\/p>\n\n<h3 id=\"users-administrators-and-moderation\">Users, administrators, and moderation<\/h3>\n\n<p>Actually, we haven\u2019t even specified what we want this room state to look like yet \u2013 there are still some unresolved questions around things as simple as\nhow administrator rights \/ operator powers should work in group chats. Different platforms do this in different ways, after all:<\/p>\n\n<ul>\n  <li>IRC has a system of \u2018channel modes\u2019, where users can be given flags such as <code class=\"language-plaintext highlighter-rouge\">+o<\/code> (operator; can kick, ban, etc.) and <code class=\"language-plaintext highlighter-rouge\">+v<\/code> (voice; can speak in muted channels).\n    <ul>\n      <li>Some servers then extend this, adding <code class=\"language-plaintext highlighter-rouge\">+h<\/code> (\u2018half-operator\u2019), <code class=\"language-plaintext highlighter-rouge\">+a<\/code> (\u2018admin\u2019), <code class=\"language-plaintext highlighter-rouge\">+q<\/code> (\u2018owner\u2019), and all sorts of other random gunk that confuses people\nand makes it hard to determine who can do what.<\/li>\n      <li>Of course, server administrators can override all of this and just do what they want, on most implementations.<\/li>\n    <\/ul>\n  <\/li>\n  <li>Matrix has \u2018<a href=\"https:\/\/matrix.org\/docs\/guides\/moderation\/#power-levels\">power levels<\/a>\u2019 - each user has an integer between 0 and 100 assigned to them, which determines what they\u2019re able to do. A set of rules (stored in\nthe room state, alongside the power levels) specify what power levels map to what \u2013 for example, only people with power level <code class=\"language-plaintext highlighter-rouge\">&gt;=<\/code> 50 can set the topic, etc.\n    <ul>\n      <li>You\u2019re not allowed to give people power levels higher than what you already have, and you can\u2019t take away power from people who have more or the same amount of power as you.\nThis is kinda required to make the whole thing work.<\/li>\n      <li>Because Matrix is decentralised, it\u2019s possible to get yourself into a state where everyone\u2019s lost control of a chatroom and you can\u2019t get it back. Of course, though,\nthis is quite easy to avoid, by making the software stop you from shooting yourself in the foot.<\/li>\n    <\/ul>\n  <\/li>\n  <li>WhatsApp has \u2018group admins\u2019 and\u2026that\u2019s pretty much it<sup id=\"fnref:5\"><a href=\"#fn:5\" class=\"footnote\" rel=\"footnote\" role=\"doc-noteref\">5<\/a><\/sup>. You either have admin rights, in which case you can do everything, or you don\u2019t and you can\u2019t really do much.\n    <ul>\n      <li>This is very simple for users to understand.<\/li>\n      <li>However, it makes WhatsApp completely impractical for all sorts of use cases; anyone who\u2019s found themselves kicked out of a WhatsApp chat after someone went crazy and banned everyone\nafter pleading for admin \u2018in order to help keep things civil\u2019 probably knows what I\u2019m talking about.<\/li>\n    <\/ul>\n  <\/li>\n  <li><a href=\"https:\/\/discordapp.com\/\">Discord<\/a> has \u2018roles\u2019 - there\u2019s a bit in server settings where you can create sets of permissions, called \u2018roles\u2019, that you grant to different users in order to empower them\nto do specific things<sup id=\"fnref:6\"><a href=\"#fn:6\" class=\"footnote\" rel=\"footnote\" role=\"doc-noteref\">6<\/a><\/sup>.\n    <ul>\n      <li>This is partially why Discord can play host to <em>massive<\/em> chatrooms for very popular games like PUBG and Fortnite: the system is very flexible, and suited to large-scale moderation.<\/li>\n      <li>However, it\u2019s also perhaps quite confusing for some people, especially if you\u2019re just using it for smaller-scale stuff like a private group chat.<\/li>\n      <li>Like Matrix, the roles are arranged in a kind of hierarchy; roles at the top of the hierarchy can make changes to roles below them, but not the other way round.<\/li>\n    <\/ul>\n  <\/li>\n<\/ul>\n\n<p>So, hmm, it might seem like there\u2019s a lot to choose from here \u2013 but, in fact, it\u2019s a bit more simple than you\u2019d think. It\u2019s immediately\napparent that the more flexible Matrix\/Discord systems can in fact be used to make more simple systems, like the IRC and WhatsApp ones; if all you want is group admin or not group admin,\nyou can make two Discord roles (one with all permissions, one with none), or have two power levels (100 and 0, say, with appropriate rules for each one), and you\u2019ve essentially got the\nmore simple system using your complex one. (And you can do funky things with your user interface to hide away the added power, for those who don\u2019t really need it.)<\/p>\n\n<p>Taking some inspiration from this idea, and the age-old concept of an <a href=\"https:\/\/en.wikipedia.org\/wiki\/Access-control_list\">access-control list<\/a>, here\u2019s a proposed model. We\u2019ll specify a set of <em>capabilities<\/em> that describe what actions\ncan be taken \u2013 for example, <code class=\"language-plaintext highlighter-rouge\">speak<\/code>, <code class=\"language-plaintext highlighter-rouge\">change-topic<\/code>, <code class=\"language-plaintext highlighter-rouge\">mute-user<\/code>, and so on \u2013 and then a set of <em>roles<\/em>, like the Discord roles, that are sets of capabilities. Each user gets\na role, which in turn describes what they\u2019re able to do. (If we want to make things work nicely for ex-IRC people, the roles can optionally come with a small letters, like <code class=\"language-plaintext highlighter-rouge\">+o<\/code> and <code class=\"language-plaintext highlighter-rouge\">+v<\/code>.)\nUnlike Discord and Matrix, there won\u2019t be any hierarchy to the roles. Roles can only be modified by someone holding a capability called <code class=\"language-plaintext highlighter-rouge\">change-roles<\/code>, and that\u2019ll be the end of it.\nThe sponsoring servers in our federation model will do this role check every time they receive a message or a request to change the state in some way, and refuse to apply the change if it\u2019s\nnot permitted.<\/p>\n\n<p>The list of capabilities will eventually be written in some spec document somewhere, and thereby standardised across different server implementations.\nEssentially, they\u2019ll work like <a href=\"https:\/\/ircv3.net\/specs\/core\/capability-negotiation.html#vendor-specific-capabilities\">IRCv3 capabilities<\/a>, where vendors can make their own capabilities up if they want to (prefixing them with a valid domain name\nfor whatever it is that they built).<\/p>\n\n<p>To make things easier, the <em>special capability<\/em> <code class=\"language-plaintext highlighter-rouge\">*<\/code> allows a user to make any change.<\/p>\n\n<p>In <a href=\"https:\/\/www.json.org\/json-en.html\">JSON<\/a>-esque syntax<sup id=\"fnref:7\"><a href=\"#fn:7\" class=\"footnote\" rel=\"footnote\" role=\"doc-noteref\">7<\/a><\/sup>, this would look a bit like:<\/p>\n\n<div class=\"language-json highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"p\">{<\/span><span class=\"w\">\n    <\/span><span class=\"nl\">\"roles\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">{<\/span><span class=\"w\">\n        <\/span><span class=\"nl\">\"normals\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">[<\/span><span class=\"s2\">\"speak\"<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"change-topic\"<\/span><span class=\"p\">],<\/span><span class=\"w\">\n        <\/span><span class=\"nl\">\"voiced\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">[<\/span><span class=\"s2\">\"speak\"<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"speak-in-muted-channel\"<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"change-topic\"<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"theta.eu.org\/special-extension-power\"<\/span><span class=\"p\">],<\/span><span class=\"w\">\n        <\/span><span class=\"nl\">\"admins\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">[<\/span><span class=\"s2\">\"*\"<\/span><span class=\"p\">]<\/span><span class=\"w\">\n    <\/span><span class=\"p\">},<\/span><span class=\"w\">\n    <\/span><span class=\"nl\">\"memberships\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">{<\/span><span class=\"w\">\n        <\/span><span class=\"nl\">\"server.org\/user1\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"normals\"<\/span><span class=\"p\">,<\/span><span class=\"w\">\n        <\/span><span class=\"nl\">\"server.org\/adminuser1\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"admins\"<\/span><span class=\"w\">\n    <\/span><span class=\"p\">}<\/span><span class=\"w\">\n<\/span><span class=\"p\">}<\/span><span class=\"w\">\n<\/span><\/code><\/pre><\/div><\/div>\n\n<h3 id=\"non-user-related-state\">Non-user-related state<\/h3>\n\n<p>Of course, there\u2019s also state that doesn\u2019t directly relate to users \u2013 the current group chat subject, whether or not guest users are allowed in, etc.\nSome state may have more complicated structure \u2013 for example, the Matrix people have a <a href=\"https:\/\/matrix.org\/docs\/guides\/moderation\/#banning-servers-from-rooms-server-acls\">server ACL<\/a> state event that lets you ban specific servers from taking part in rooms, which\nis pretty much its own little object with embedded arrays and booleans \u2013 which means we can\u2019t just model this state as a simple stringly-typed key-value store, i.e.<\/p>\n\n<div class=\"language-json highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"p\">{<\/span><span class=\"w\">\n    <\/span><span class=\"nl\">\"state\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">{<\/span><span class=\"w\">\n        <\/span><span class=\"nl\">\"subject\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"example\"<\/span><span class=\"p\">,<\/span><span class=\"w\">\n        <\/span><span class=\"nl\">\"other_state_key\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"other_value\"<\/span><span class=\"p\">,<\/span><span class=\"w\">\n        <\/span><span class=\"err\">\/\/<\/span><span class=\"w\"> <\/span><span class=\"err\">etc<\/span><span class=\"w\">\n    <\/span><span class=\"p\">}<\/span><span class=\"w\">\n<\/span><span class=\"p\">}<\/span><span class=\"w\">\n<\/span><\/code><\/pre><\/div><\/div>\n\n<p>The question is, though, how extensible we want things to be: can users (or servers, for that matter) store arbitrary objects in our group chat state, or do they all\nhave to follow some predetermined schema? Matrix takes the approach of letting users chuck in whatever they like (assuming they have the necessary power level) \u2013 essentially\ntreating each group chat as a mini database \u2013 while pretty much every other platform restricts room state to a set of predetermined things.<\/p>\n\n<h4 id=\"capability-negotiation\">Capability negotiation<\/h4>\n\n<p>I\u2019m not entirely sure allowing <em>arbitrary<\/em> extensibility, in the way that Matrix does, is such a good idea \u2013 for two reasons<sup id=\"fnref:8\"><a href=\"#fn:8\" class=\"footnote\" rel=\"footnote\" role=\"doc-noteref\">8<\/a><\/sup>:<\/p>\n\n<ol>\n  <li>Allowing people to just store arbitrary data in your group chat seems a bit too much like an abuse vector. It\u2019s a chat system, not a distributed database; allowing people\nto just chuck whatever stuff they want in there is a bit much!\n    <ul>\n      <li>How would you display these arbitrary state events, given you don\u2019t know anything about them?<\/li>\n      <li>You\u2019d need some limit on size, to prevent people just using you as free storage?<\/li>\n      <li>In many jurisdictions, you\u2019re responsible for content that gets put on your server. What if someone uploads stuff you\u2019re not comfortable hosting, and stashes it away\nin some event your client doesn\u2019t implement (and therefore doesn\u2019t show)?<\/li>\n    <\/ul>\n  <\/li>\n  <li>Usually, things in the group chat state are there for a reason, and it doesn\u2019t make sense for servers to just ignore them.\n    <ul>\n      <li>For example, consider when the Matrix folks rolled out the server ACL feature: servers not on the latest version of their software would just completely ignore it,\nwhich is pretty bad (because then it had no effect, as malicious actors could get into the room via the unpatched servers).\n        <ul>\n          <li>They \u2018solved\u2019 this by polling all the servers in a room for their current version number, and checking to see which ones still needed updating (which let them badger\nthe server owners until it got updated).<\/li>\n        <\/ul>\n      <\/li>\n    <\/ul>\n  <\/li>\n<\/ol>\n\n<p>Instead, it\u2019s probably better to actually have some system \u2013 like the previously-mentioned <a href=\"https:\/\/ircv3.net\/specs\/core\/capability-negotiation\">IRCv3 capability negotiation<\/a><sup id=\"fnref:9\"><a href=\"#fn:9\" class=\"footnote\" rel=\"footnote\" role=\"doc-noteref\">9<\/a><\/sup> \u2013 where servers can say \u201cyes, I support\nfeatures X, Y, and Z\u201d, thus enabling those features to be used only if all of the sponsoring servers in a group chat actually support them. This solves the issue of extensibility\nquite nicely: non-user related state is governed by a strict schema, with optional extensions for servers that have managed to negotiate that.<\/p>\n\n<h2 id=\"group-chat-state-summary\">Group Chat state summary<\/h2>\n\n<p>So, to sum up: we\u2019ve managed to get a better idea of what the blob of <em>group chat state<\/em>, shared across all servers participating in a group chat and agreed upon via the federation\nconsensus protocol, should look like: it\u2019ll contain a <em>capability<\/em>-based system for regulating what users are allowed to do what, and it\u2019ll contain a set of other pieces of strictly-typed\nnon-user-related state, like the group chat subject! This might all seem a bit abstract for now, but it\u2019ll hopefully become clearer once actual code starts getting written.\nOn that note\u2026<\/p>\n\n<h2 id=\"a-note-about-timings\">A note about timings<\/h2>\n\n<p>We managed to roughly sketch out group chat state over the course of around ~2,500 words (!), but there\u2019s still all the other stuff to spec out: users, messages, and reliable delivery\nmechanisms. In addition, there are also a number of less conceptual things to sketch out, like how we\u2019re going to ensure server-to-server transport happens securely (SSL? Encryption?<sup id=\"fnref:10\"><a href=\"#fn:10\" class=\"footnote\" rel=\"footnote\" role=\"doc-noteref\">10<\/a><\/sup>)\nand things like that.<\/p>\n\n<p>And this all has to be done by March, at the absolute latest. <em>Yay!<\/em><\/p>\n\n<p>On a more hopeful note, we do <a href=\"https:\/\/git.theta.eu.org\/nea-2019.git\/\">actually have some code<\/a> for this whole project \u2013 currently, it does the registration flow part of a standard\nIRC server (basically, the plan is to piggyback off of IRC for the client&lt;-&gt;server stuff, to get things going), and has a very untested implementation of the Paxos consensus protocol.\nWe\u2019re using <a href=\"https:\/\/common-lisp.net\/\">Common Lisp<\/a> as our main programming language, which is fun<sup id=\"fnref:11\"><a href=\"#fn:11\" class=\"footnote\" rel=\"footnote\" role=\"doc-noteref\">11<\/a><\/sup>!<\/p>\n\n<p>Anyway, the rest of the spec stuff will follow in the next blogpost (hopefully quite quickly after this one\u2026!) \u2013 if anyone actually has the tenacity to read my ~2,500 words about the annals\nof random chat protocol stuff, that is! We will also actually show how all of this translates into a database schema, which was sort of the point. Oops.<\/p>\n\n<p><em>To be continued\u2026<\/em><\/p>\n\n<hr \/>\n\n<div class=\"footnotes\" role=\"doc-endnotes\">\n  <ol>\n    <li id=\"fn:1\">\n      <p>Err, two months. I was very busy applying to universities, okay!\u00a0<a href=\"#fnref:1\" class=\"reversefootnote\" role=\"doc-backlink\">&#8617;<\/a><\/p>\n    <\/li>\n    <li id=\"fn:2\">\n      <p>A large part of my <a href=\"\/2019\/09\/10\/chat-systems.html#matrix\">criticism of Matrix<\/a> centered around the fact that they did something funky \u2013 and, in my view, perhaps unnecessary \u2013 with theirs.\u00a0<a href=\"#fnref:2\" class=\"reversefootnote\" role=\"doc-backlink\">&#8617;<\/a><\/p>\n    <\/li>\n    <li id=\"fn:3\">\n      <p>Using some fancy NoSQL database has <a href=\"http:\/\/www.sarahmei.com\/blog\/2013\/11\/11\/why-you-should-never-use-mongodb\/\">undone many a project<\/a> in the past; SQL databases have had a lot of engineering and research effort put into them over the decades to make them work reliably and quickly, and that shouldn\u2019t be given up lightly!\u00a0<a href=\"#fnref:3\" class=\"reversefootnote\" role=\"doc-backlink\">&#8617;<\/a><\/p>\n    <\/li>\n    <li id=\"fn:4\">\n      <p>Read the \u2018funky blend of federation\u2019 blog post linked earlier, if you need a refresher!\u00a0<a href=\"#fnref:4\" class=\"reversefootnote\" role=\"doc-backlink\">&#8617;<\/a><\/p>\n    <\/li>\n    <li id=\"fn:5\">\n      <p>I think they recently added some sort of limited permissions system for \u201cwho can change the topic: admins or all group members?\u201d and things like that, but this is the gist of it.\u00a0<a href=\"#fnref:5\" class=\"reversefootnote\" role=\"doc-backlink\">&#8617;<\/a><\/p>\n    <\/li>\n    <li id=\"fn:6\">\n      <p>They also serve a vanity \/ novelty purpose; look at me, I\u2019ve got a colourful name!\u00a0<a href=\"#fnref:6\" class=\"reversefootnote\" role=\"doc-backlink\">&#8617;<\/a><\/p>\n    <\/li>\n    <li id=\"fn:7\">\n      <p>Even though we might not be actually <em>using<\/em> JSON at the end of the day, it\u2019s a pretty well-understood way to describe things.\u00a0<a href=\"#fnref:7\" class=\"reversefootnote\" role=\"doc-backlink\">&#8617;<\/a><\/p>\n    <\/li>\n    <li id=\"fn:8\">\n      <p><a href=\"https:\/\/github.com\/ircv3\/ircv3-specifications\/issues\/333\">This IRCv3 issue<\/a> is about a very similar problem (allowing arbitrary client-only message tags on IRC), and is also worth a read!\u00a0<a href=\"#fnref:8\" class=\"reversefootnote\" role=\"doc-backlink\">&#8617;<\/a><\/p>\n    <\/li>\n    <li id=\"fn:9\">\n      <p>These capabilities are, unfortunately, nothing to do with the user\/administrator capabilities discussed earlier\u2026\u00a0<a href=\"#fnref:9\" class=\"reversefootnote\" role=\"doc-backlink\">&#8617;<\/a><\/p>\n    <\/li>\n    <li id=\"fn:10\">\n      <p>The Matrix people went and used <a href=\"https:\/\/perspectivessecurity.wordpress.com\/\">Perspectives<\/a> instead of regular SSL PKI for their stuff, and they eventually had to move away from it. It\u2019s worth learning from that mistake!\u00a0<a href=\"#fnref:10\" class=\"reversefootnote\" role=\"doc-backlink\">&#8617;<\/a><\/p>\n    <\/li>\n    <li id=\"fn:11\">\n      <p>It was going to be Rust, but then I started doing Lisp stuff and figured this would be more \u2018interesting\u2019\u2026\u00a0<a href=\"#fnref:11\" class=\"reversefootnote\" role=\"doc-backlink\">&#8617;<\/a><\/p>\n    <\/li>\n  <\/ol>\n<\/div>\n","pubDate":"Wed, 18 Dec 2019 00:00:00 +0000","link":"https:\/\/eta.st\/2019\/12\/18\/nea-schema.html","guid":"https:\/\/eta.st\/2019\/12\/18\/nea-schema.html"}]}}