{"title":"the website of jyn","subtitle":"i write about code, and things that bring me joy, and sometimes other things too","link":[{"@attributes":{"rel":"self","type":"application\/atom+xml","href":"https:\/\/jyn.dev\/atom.xml"}},{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev"}}],"generator":"Zola","updated":"2026-01-22T00:00:00+00:00","id":"https:\/\/jyn.dev\/atom.xml","entry":[{"title":"remotely unlocking an encrypted hard disk","published":"2026-01-22T00:00:00+00:00","updated":"2026-01-22T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/remotely-unlocking-an-encrypted-hard-disk\/"}},"id":"https:\/\/jyn.dev\/remotely-unlocking-an-encrypted-hard-disk\/","summary":"what's a few systemd services in initramfs between friends?","content":"<p>Your mission, should you choose to accept it, is to sneak into the earliest parts of the boot process, swap the startup config without breaking anything, and leave without a trace.<\/p>\n<p>Are you ready? Let's begin.<\/p>\n<h2 id=\"the-setup\">the setup<a class=\"zola-anchor\" href=\"#the-setup\" aria-label=\"Anchor link for: the-setup\"><\/a>\n<\/h2>\n<p><em>In which our heroes are introduced, and the scene is set.<\/em><\/p>\n<p>For a very long time I had a beat-up old ThinkPad that couldn\u2019t hold a charge for the life of it, especially when running Windows. It tended to die a lot when I was traveling, and I travel a lot. To save battery when I\u2019m away from home, I often ssh back into my home desktop, both so I have persistent state even if my laptop battery dies, and so I get much faster builds that don\u2019t kill the battery.<\/p>\n<p>This has two small problems:<\/p>\n<ol>\n<li>Sometimes my home loses power and the desktop shuts off.<\/li>\n<li>Sometimes when the power comes back on it has a new public IP.<\/li>\n<\/ol>\n<p>For a long time I solved 1. by enabling \u201cPower On\" after \"Restore AC Power Loss\u201d in the BIOS and 2. with <a href=\"https:\/\/tailscale.com\/kb\/1151\/what-is-tailscale\">tailscale<\/a>. However, I recently installed Arch with an encrypted boot partition, which means that boot doesn\u2019t finish until I type in the encryption password.<\/p>\n<p>Well. Well. What if I Simply put tailscale in initramfs?<\/p>\n<h2 id=\"the-plan\">the plan<a class=\"zola-anchor\" href=\"#the-plan\" aria-label=\"Anchor link for: the-plan\"><\/a>\n<\/h2>\n<p><em>In which our intrepid heroes chart the challenges to come.<\/em><\/p>\n<h3 id=\"initramfs\">initramfs<a class=\"zola-anchor\" href=\"#initramfs\" aria-label=\"Anchor link for: initramfs\"><\/a>\n<\/h3>\n<p>Oh, right. If you weren\u2019t aware, early boot in a Linux operating system<sup class=\"footnote-reference\" id=\"fr-3-1\"><a href=\"#fn-3\">1<\/a><\/sup>  is just running a full second operating system that happens to be very small, lol. That\u2019s loaded from a compressed archive file in \/boot<sup class=\"footnote-reference\" id=\"fr-4-1\"><a href=\"#fn-4\">2<\/a><\/sup> and run from memory, with no access to persistent storage. This OS running from memory is called <strong>initramfs<\/strong> (initial RAM filesystem).<\/p>\n<p>So when you see a screen like this:\n<img src=\"\/assets\/Pasted%20image%2020260121044155.png\" alt=\"\" \/>\nThat\u2019s actually a whole-ass OS, with an <code>init<\/code> PID and service management and everything. This is how, for example, <code>systemd-analyze<\/code> can show you stats about early boot \u2014 there\u2019s another copy of systemd running in initramfs, and it passes its state off to the one in the main OS.<\/p>\n<p>Well. That implies we can install things on it ^^.<\/p>\n<h3 id=\"constraints\">constraints<a class=\"zola-anchor\" href=\"#constraints\" aria-label=\"Anchor link for: constraints\"><\/a>\n<\/h3>\n<p>There\u2019s three parts to this:<\/p>\n<ol>\n<li>Networking in initramfs<\/li>\n<li>Tailscale in initramfs<\/li>\n<li>SSH in initramfs<\/li>\n<\/ol>\n<p>We also want to make this as secure as possible, so there\u2019s some more things to consider:<\/p>\n<ul>\n<li>Putting tailscale in initramfs means that it has unencrypted keys lying around.<\/li>\n<li>Tailscale keys expire (by default) after 90 days. At that point this will all break.<\/li>\n<li>You really really don\u2019t want people to get SSH access to your early boot environment.<\/li>\n<\/ul>\n<p>We can solve this in a few ways:<\/p>\n<ul>\n<li>Use Tailscale ACLs to only allow incoming connections to initramfs, not outgoing connections.<\/li>\n<li>Set the key to never expire.<\/li>\n<li>Set the SSH server to disallow all shells except the actual unlock command (<code>systemd-tty-ask-password-agent<\/code>).<\/li>\n<\/ul>\n<h3 id=\"tailscale-acls\">tailscale ACLs<a class=\"zola-anchor\" href=\"#tailscale-acls\" aria-label=\"Anchor link for: tailscale-acls\"><\/a>\n<\/h3>\n<p>Some background about Tailscale\u2019s ACLs (\u201caccess control lists\u201d). Tailscale\u2019s users are tied to their specific login method: you can, for example, add a passkey, but that passkey counts as a fully separate user than your original account. Tailscale also has \u201cgroups\u201d of users, which are what they sound like, \u201c<a href=\"https:\/\/tailscale.com\/kb\/1396\/targets#autogroups\">auto groups<\/a>\u201d, which again are what they sound like, \u201chosts\u201d, which are a machine connected to the network, and \u201ctags\u201d.<\/p>\n<p>Tags are odd, I haven't seen anything like them before. They group hosts, not users, and when you add a tag to a host, that <em>counts as its login method<\/em>, rather than the host being tied to a user account.<\/p>\n<p>A consequence of this is that the group <a href=\"https:\/\/tailscale.com\/kb\/1396\/targets#autogrouprole\"><code>autogroup:member<\/code><\/a> does <em>not<\/em> include tagged machines, because tagged machines aren\u2019t tied to a user account. (A second consequence is that you can\u2019t remove all tags from a machine without logging out and logging back in to associate it with your user account.)<\/p>\n<p>So we can write a policy like this:<\/p>\n<pre data-lang=\"json\" class=\"language-json z-code\"><code class=\"language-json\" data-lang=\"json\"><span class=\"z-source z-json\"><span class=\"z-meta z-mapping z-json\"><span class=\"z-punctuation z-section z-mapping z-begin z-json\">{<\/span>\n<\/span><\/span><span class=\"z-source z-json\"><span class=\"z-meta z-mapping z-json\">  <span class=\"z-comment z-line z-double-slash z-js\"><span class=\"z-punctuation z-definition z-comment z-json\">\/\/<\/span> Define the tags which can be applied to devices and by which users.\n<\/span><\/span><\/span><span class=\"z-source z-json\"><span class=\"z-meta z-mapping z-json\">  <\/span><span class=\"z-meta z-mapping z-key z-json\"><span class=\"z-string z-quoted z-double z-json\"><span class=\"z-punctuation z-definition z-string z-begin z-json\">&quot;<\/span>tagOwners<span class=\"z-punctuation z-definition z-string z-end z-json\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-mapping z-json\"><span class=\"z-punctuation z-separator z-mapping z-key-value z-json\">:<\/span> <\/span><span class=\"z-meta z-mapping z-value z-json\"><span class=\"z-meta z-mapping z-json\"><span class=\"z-punctuation z-section z-mapping z-begin z-json\">{<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-json\"><span class=\"z-meta z-mapping z-value z-json\"><span class=\"z-meta z-mapping z-json\">    <\/span><span class=\"z-meta z-mapping z-key z-json\"><span class=\"z-string z-quoted z-double z-json\"><span class=\"z-punctuation z-definition z-string z-begin z-json\">&quot;<\/span>tag:initrd<span class=\"z-punctuation z-definition z-string z-end z-json\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-mapping z-json\"><span class=\"z-punctuation z-separator z-mapping z-key-value z-json\">:<\/span> <\/span><span class=\"z-meta z-mapping z-value z-json\"><span class=\"z-meta z-sequence z-json\"><span class=\"z-punctuation z-section z-sequence z-begin z-json\">[<\/span><span class=\"z-string z-quoted z-double z-json\"><span class=\"z-punctuation z-definition z-string z-begin z-json\">&quot;<\/span>autogroup:admin<span class=\"z-punctuation z-definition z-string z-end z-json\">&quot;<\/span><\/span><span class=\"z-punctuation z-section z-sequence z-end z-json\">]<\/span><\/span><span class=\"z-punctuation z-separator z-mapping z-pair z-json\">,<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-json\"><span class=\"z-meta z-mapping z-value z-json\"><span class=\"z-meta z-mapping z-value z-json\">  <span class=\"z-punctuation z-section z-mapping z-end z-json\">}<\/span><\/span><span class=\"z-punctuation z-separator z-mapping z-pair z-json\">,<\/span>\n<\/span><\/span><span class=\"z-source z-json\"><span class=\"z-meta z-mapping z-value z-json\">\n<\/span><\/span><span class=\"z-source z-json\"><span class=\"z-meta z-mapping z-value z-json\">  <span class=\"z-comment z-line z-double-slash z-js\"><span class=\"z-punctuation z-definition z-comment z-json\">\/\/<\/span> Define access control lists for users, groups, autogroups, tags,\n<\/span><\/span><\/span><span class=\"z-source z-json\"><span class=\"z-meta z-mapping z-value z-json\">  <span class=\"z-comment z-line z-double-slash z-js\"><span class=\"z-punctuation z-definition z-comment z-json\">\/\/<\/span> Tailscale IP addresses, and subnet ranges.\n<\/span><\/span><\/span><span class=\"z-source z-json\"><span class=\"z-meta z-mapping z-value z-json\">  <\/span><span class=\"z-meta z-mapping z-key z-json\"><span class=\"z-string z-quoted z-double z-json\"><span class=\"z-punctuation z-definition z-string z-begin z-json\">&quot;<\/span>acls<span class=\"z-punctuation z-definition z-string z-end z-json\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-mapping z-value z-json\"><span class=\"z-punctuation z-separator z-mapping z-key-value z-json\">:<\/span> <\/span><span class=\"z-meta z-mapping z-value z-json\"><span class=\"z-meta z-sequence z-json\"><span class=\"z-punctuation z-section z-sequence z-begin z-json\">[<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-json\"><span class=\"z-meta z-mapping z-value z-json\"><span class=\"z-meta z-sequence z-json\">    <span class=\"z-meta z-mapping z-json\"><span class=\"z-punctuation z-section z-mapping z-begin z-json\">{<\/span><\/span><span class=\"z-meta z-mapping z-key z-json\"><span class=\"z-string z-quoted z-double z-json\"><span class=\"z-punctuation z-definition z-string z-begin z-json\">&quot;<\/span>action<span class=\"z-punctuation z-definition z-string z-end z-json\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-mapping z-json\"><span class=\"z-punctuation z-separator z-mapping z-key-value z-json\">:<\/span> <\/span><span class=\"z-meta z-mapping z-value z-json\"><span class=\"z-string z-quoted z-double z-json\"><span class=\"z-punctuation z-definition z-string z-begin z-json\">&quot;<\/span>accept<span class=\"z-punctuation z-definition z-string z-end z-json\">&quot;<\/span><\/span><span class=\"z-punctuation z-separator z-mapping z-pair z-json\">,<\/span> <\/span><span class=\"z-meta z-mapping z-key z-json\"><span class=\"z-string z-quoted z-double z-json\"><span class=\"z-punctuation z-definition z-string z-begin z-json\">&quot;<\/span>src<span class=\"z-punctuation z-definition z-string z-end z-json\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-mapping z-value z-json\"><span class=\"z-punctuation z-separator z-mapping z-key-value z-json\">:<\/span> <\/span><span class=\"z-meta z-mapping z-value z-json\"><span class=\"z-meta z-sequence z-json\"><span class=\"z-punctuation z-section z-sequence z-begin z-json\">[<\/span><span class=\"z-string z-quoted z-double z-json\"><span class=\"z-punctuation z-definition z-string z-begin z-json\">&quot;<\/span>autogroup:member<span class=\"z-punctuation z-definition z-string z-end z-json\">&quot;<\/span><\/span><span class=\"z-punctuation z-section z-sequence z-end z-json\">]<\/span><\/span><span class=\"z-punctuation z-separator z-mapping z-pair z-json\">,<\/span> <\/span><span class=\"z-meta z-mapping z-key z-json\"><span class=\"z-string z-quoted z-double z-json\"><span class=\"z-punctuation z-definition z-string z-begin z-json\">&quot;<\/span>dst<span class=\"z-punctuation z-definition z-string z-end z-json\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-mapping z-value z-json\"><span class=\"z-punctuation z-separator z-mapping z-key-value z-json\">:<\/span> <\/span><span class=\"z-meta z-mapping z-value z-json\"><span class=\"z-meta z-sequence z-json\"><span class=\"z-punctuation z-section z-sequence z-begin z-json\">[<\/span><span class=\"z-string z-quoted z-double z-json\"><span class=\"z-punctuation z-definition z-string z-begin z-json\">&quot;<\/span>*:*<span class=\"z-punctuation z-definition z-string z-end z-json\">&quot;<\/span><\/span><span class=\"z-punctuation z-section z-sequence z-end z-json\">]<\/span><\/span><span class=\"z-punctuation z-section z-mapping z-end z-json\">}<\/span><\/span><span class=\"z-punctuation z-separator z-sequence z-json\">,<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-json\"><span class=\"z-meta z-mapping z-value z-json\"><span class=\"z-meta z-sequence z-json\">  <span class=\"z-punctuation z-section z-sequence z-end z-json\">]<\/span><\/span><span class=\"z-punctuation z-separator z-mapping z-pair z-json\">,<\/span>\n<\/span><\/span><span class=\"z-source z-json\"><span class=\"z-meta z-mapping z-value z-json\">\n<\/span><\/span><span class=\"z-source z-json\"><span class=\"z-meta z-mapping z-value z-json\">  <span class=\"z-comment z-line z-double-slash z-js\"><span class=\"z-punctuation z-definition z-comment z-json\">\/\/<\/span> Test access rules every time they&#39;re saved.\n<\/span><\/span><\/span><span class=\"z-source z-json\"><span class=\"z-meta z-mapping z-value z-json\">  <\/span><span class=\"z-meta z-mapping z-key z-json\"><span class=\"z-string z-quoted z-double z-json\"><span class=\"z-punctuation z-definition z-string z-begin z-json\">&quot;<\/span>tests<span class=\"z-punctuation z-definition z-string z-end z-json\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-mapping z-value z-json\"><span class=\"z-punctuation z-separator z-mapping z-key-value z-json\">:<\/span> <\/span><span class=\"z-meta z-mapping z-value z-json\"><span class=\"z-meta z-sequence z-json\"><span class=\"z-punctuation z-section z-sequence z-begin z-json\">[<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-json\"><span class=\"z-meta z-mapping z-value z-json\"><span class=\"z-meta z-sequence z-json\">    <span class=\"z-meta z-mapping z-json\"><span class=\"z-punctuation z-section z-mapping z-begin z-json\">{<\/span>\n<\/span><\/span><\/span><\/span><span class=\"z-source z-json\"><span class=\"z-meta z-mapping z-value z-json\"><span class=\"z-meta z-sequence z-json\"><span class=\"z-meta z-mapping z-json\">      <\/span><span class=\"z-meta z-mapping z-key z-json\"><span class=\"z-string z-quoted z-double z-json\"><span class=\"z-punctuation z-definition z-string z-begin z-json\">&quot;<\/span>src<span class=\"z-punctuation z-definition z-string z-end z-json\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-mapping z-json\"><span class=\"z-punctuation z-separator z-mapping z-key-value z-json\">:<\/span>    <\/span><span class=\"z-meta z-mapping z-value z-json\"><span class=\"z-string z-quoted z-double z-json\"><span class=\"z-punctuation z-definition z-string z-begin z-json\">&quot;<\/span>100.76.34.8<span class=\"z-punctuation z-definition z-string z-end z-json\">&quot;<\/span><\/span><span class=\"z-punctuation z-separator z-mapping z-pair z-json\">,<\/span> <span class=\"z-comment z-line z-double-slash z-js\"><span class=\"z-punctuation z-definition z-comment z-json\">\/\/<\/span> outrageous-fortune\n<\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-json\"><span class=\"z-meta z-mapping z-value z-json\"><span class=\"z-meta z-sequence z-json\"><span class=\"z-meta z-mapping z-value z-json\">      <\/span><span class=\"z-meta z-mapping z-key z-json\"><span class=\"z-string z-quoted z-double z-json\"><span class=\"z-punctuation z-definition z-string z-begin z-json\">&quot;<\/span>accept<span class=\"z-punctuation z-definition z-string z-end z-json\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-mapping z-value z-json\"><span class=\"z-punctuation z-separator z-mapping z-key-value z-json\">:<\/span> <\/span><span class=\"z-meta z-mapping z-value z-json\"><span class=\"z-meta z-sequence z-json\"><span class=\"z-punctuation z-section z-sequence z-begin z-json\">[<\/span><span class=\"z-string z-quoted z-double z-json\"><span class=\"z-punctuation z-definition z-string z-begin z-json\">&quot;<\/span>100.102.101.127:22<span class=\"z-punctuation z-definition z-string z-end z-json\">&quot;<\/span><\/span><span class=\"z-punctuation z-separator z-sequence z-json\">,<\/span> <span class=\"z-string z-quoted z-double z-json\"><span class=\"z-punctuation z-definition z-string z-begin z-json\">&quot;<\/span>100.101.55.73:10078<span class=\"z-punctuation z-definition z-string z-end z-json\">&quot;<\/span><\/span><span class=\"z-punctuation z-section z-sequence z-end z-json\">]<\/span><\/span><span class=\"z-punctuation z-separator z-mapping z-pair z-json\">,<\/span> <span class=\"z-comment z-line z-double-slash z-js\"><span class=\"z-punctuation z-definition z-comment z-json\">\/\/<\/span> selene-initrd\n<\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-json\"><span class=\"z-meta z-mapping z-value z-json\"><span class=\"z-meta z-sequence z-json\"><span class=\"z-meta z-mapping z-value z-json\">    <span class=\"z-punctuation z-section z-mapping z-end z-json\">}<\/span><\/span><span class=\"z-punctuation z-separator z-sequence z-json\">,<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-json\"><span class=\"z-meta z-mapping z-value z-json\"><span class=\"z-meta z-sequence z-json\">    <span class=\"z-meta z-mapping z-json\"><span class=\"z-punctuation z-section z-mapping z-begin z-json\">{<\/span>\n<\/span><\/span><\/span><\/span><span class=\"z-source z-json\"><span class=\"z-meta z-mapping z-value z-json\"><span class=\"z-meta z-sequence z-json\"><span class=\"z-meta z-mapping z-json\">      <\/span><span class=\"z-meta z-mapping z-key z-json\"><span class=\"z-string z-quoted z-double z-json\"><span class=\"z-punctuation z-definition z-string z-begin z-json\">&quot;<\/span>src<span class=\"z-punctuation z-definition z-string z-end z-json\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-mapping z-json\"><span class=\"z-punctuation z-separator z-mapping z-key-value z-json\">:<\/span>  <\/span><span class=\"z-meta z-mapping z-value z-json\"><span class=\"z-string z-quoted z-double z-json\"><span class=\"z-punctuation z-definition z-string z-begin z-json\">&quot;<\/span>100.102.101.127<span class=\"z-punctuation z-definition z-string z-end z-json\">&quot;<\/span><\/span><span class=\"z-punctuation z-separator z-mapping z-pair z-json\">,<\/span> <span class=\"z-comment z-line z-double-slash z-js\"><span class=\"z-punctuation z-definition z-comment z-json\">\/\/<\/span> selene-initrd\n<\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-json\"><span class=\"z-meta z-mapping z-value z-json\"><span class=\"z-meta z-sequence z-json\"><span class=\"z-meta z-mapping z-value z-json\">      <\/span><span class=\"z-meta z-mapping z-key z-json\"><span class=\"z-string z-quoted z-double z-json\"><span class=\"z-punctuation z-definition z-string z-begin z-json\">&quot;<\/span>deny<span class=\"z-punctuation z-definition z-string z-end z-json\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-mapping z-value z-json\"><span class=\"z-punctuation z-separator z-mapping z-key-value z-json\">:<\/span> <\/span><span class=\"z-meta z-mapping z-value z-json\"><span class=\"z-meta z-sequence z-json\"><span class=\"z-punctuation z-section z-sequence z-begin z-json\">[<\/span><span class=\"z-string z-quoted z-double z-json\"><span class=\"z-punctuation z-definition z-string z-begin z-json\">&quot;<\/span>100.101.55.73:10078<span class=\"z-punctuation z-definition z-string z-end z-json\">&quot;<\/span><\/span><span class=\"z-punctuation z-section z-sequence z-end z-json\">]<\/span><\/span><span class=\"z-punctuation z-separator z-mapping z-pair z-json\">,<\/span> <span class=\"z-comment z-line z-double-slash z-js\"><span class=\"z-punctuation z-definition z-comment z-json\">\/\/<\/span> selene\n<\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-json\"><span class=\"z-meta z-mapping z-value z-json\"><span class=\"z-meta z-sequence z-json\"><span class=\"z-meta z-mapping z-value z-json\">    <span class=\"z-punctuation z-section z-mapping z-end z-json\">}<\/span><\/span><span class=\"z-punctuation z-separator z-sequence z-json\">,<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-json\"><span class=\"z-meta z-mapping z-value z-json\"><span class=\"z-meta z-sequence z-json\">  <span class=\"z-punctuation z-section z-sequence z-end z-json\">]<\/span><\/span><span class=\"z-punctuation z-separator z-mapping z-pair z-json\">,<\/span>\n<\/span><\/span><span class=\"z-source z-json\"><span class=\"z-meta z-mapping z-value z-json\"><span class=\"z-punctuation z-section z-mapping z-end z-json\">}<\/span><\/span>\n<\/span><\/code><\/pre>\n<p>This says \u201callow devices tied to a user account to access any other device, and allow no permissions at all for devices tied to a tag\u201d.<\/p>\n<p><code>selene<\/code> here is my desktop, and <code>selene-initrd<\/code> is its initramfs.  <sup class=\"footnote-reference\" id=\"fr-1-1\"><a href=\"#fn-1\">3<\/a><\/sup><\/p>\n<h3 id=\"systemd-before-boot\">systemd before boot<a class=\"zola-anchor\" href=\"#systemd-before-boot\" aria-label=\"Anchor link for: systemd-before-boot\"><\/a>\n<\/h3>\n<p>Because initramfs is just a (mostly) normal Linux system, that means it has its own <code>init<\/code>\nPID 1. On Arch, that PID is in fact just systemd. That means that we can add systemd\nservices to initramfs! There's a whole collection of them in\n<a href=\"https:\/\/github.com\/wolegis\/mkinitcpio-systemd-extras\"><code>mkinitcpio-systemd-extras<\/code><\/a>\n(<code>mkinitcpio<\/code> is the tool Arch uses to regenerate initramfs).<\/p>\n<p>We need two services: an SSH server (I went with\n<a href=\"https:\/\/github.com\/wolegis\/mkinitcpio-systemd-extras\/wiki\/Dropbear-SSH-server\"><code>dropbear<\/code><\/a>)\nand something to turn on networking, which this collection names <code>sd-network<\/code>.<\/p>\n<aside role=note class=note-container><div class=note-content>\n<p>It's possible to run <code>tailscale ssh<\/code> directly, rather than having a separate SSH server, but\nI didn't find any way to configure tailscale's SSH command, and I don't want to let anyone\nhave a shell in my initramfs.<\/p>\n<\/div><\/aside>\n<h2 id=\"the-heist\">the heist<a class=\"zola-anchor\" href=\"#the-heist\" aria-label=\"Anchor link for: the-heist\"><\/a>\n<\/h2>\n<p><em>In which our heroes execute their plan flawlessly, sneaking in without a sound.<\/em><\/p>\n<p>If you follow these steps on an Arch system, you should end up with roughly the same setup\nas I have. Most of these commands assume you are running as root.<\/p>\n<ul>\n<li>\n<p>Install the dropbear SSH server:<\/p>\n<pre data-lang=\"sh\" class=\"language-sh z-code\"><code class=\"language-sh\" data-lang=\"sh\"><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">pacman<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> -<\/span>S<\/span> dropbear<\/span>\n<\/span><\/code><\/pre>\n<\/li>\n<li>\n<p>Install the systemd packages:<\/p>\n<pre data-lang=\"sh\" class=\"language-sh z-code\"><code class=\"language-sh\" data-lang=\"sh\"><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">yay<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> -<\/span>S<\/span> mkinitcpio-systemd-extras mkinitcpio-tailscale<\/span>\n<\/span><\/code><\/pre>\n<\/li>\n<li>\n<p>Add networking (<code>sd-network<\/code>), tailscale (<code>tailscale<\/code>), and dropbear (<code>sd-dropbear<\/code>) to\n<code>\/etc\/mkinitcpio.conf<\/code>:<\/p>\n<pre data-lang=\"diff\" class=\"language-diff z-code\"><code class=\"language-diff\" data-lang=\"diff\"><span class=\"z-source z-diff\"><span class=\"z-meta z-diff z-range z-normal\"><span class=\"z-meta z-range z-normal z-diff\">1c1\n<\/span><\/span><\/span><span class=\"z-source z-diff\"><span class=\"z-markup z-deleted z-diff\"><span class=\"z-punctuation z-definition z-deleted z-diff\">&lt;<\/span> HOOKS=(base systemd autodetect microcode kms modconf block keyboard sd-vconsole plymouth sd-encrypt filesystems)\n<\/span><\/span><span class=\"z-source z-diff\"><span class=\"z-meta z-separator z-diff\"><span class=\"z-punctuation z-definition z-separator z-diff\">---<\/span>\n<\/span><\/span><span class=\"z-source z-diff\"><span class=\"z-markup z-inserted z-diff\"><span class=\"z-punctuation z-definition z-inserted z-diff\">&gt;<\/span> HOOKS=(base systemd autodetect microcode kms modconf block keyboard sd-vconsole plymouth sd-network tailscale sd-dropbear sd-encrypt filesystems)\n<\/span><\/span><\/code><\/pre>\n<\/li>\n<li>\n<p>Set up the keys for your new tailscale device:<\/p>\n<pre data-lang=\"sh\" class=\"language-sh z-code\"><code class=\"language-sh\" data-lang=\"sh\"><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">setup-initcpio-tailscale<\/span><\/span>\n<\/span><\/code><\/pre>\n<\/li>\n<li>\n<p>In <a href=\"https:\/\/login.tailscale.com\/admin\/machines\">the tailscale web console<\/a>, mark your new\ndevice with <code>tag:initrd<\/code>, and disable key expiry. It should look something like this:<\/p>\n<p><img src=\"\/assets\/Pasted%20image%2020260121213235.png\" alt=\"\" \/><\/p>\n<\/li>\n<li>\n<p>In <code>\/etc\/mkinitcpio.conf<\/code>, configure dropbear to only allow running the unlock command and nothing else:<\/p>\n<pre data-lang=\"sh\" class=\"language-sh z-code\"><code class=\"language-sh\" data-lang=\"sh\"><span class=\"z-source z-shell z-bash\"><span class=\"z-variable z-other z-readwrite z-assignment z-shell\">SD_DROPBEAR_COMMAND<\/span><span class=\"z-keyword z-operator z-assignment z-shell\">=<\/span><span class=\"z-string z-unquoted z-shell\"><span class=\"z-string z-quoted z-double z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&quot;<\/span>systemd-tty-ask-password-agent<span class=\"z-punctuation z-definition z-string z-end z-shell\">&quot;<\/span><\/span><\/span>\n<\/span><\/code><\/pre>\n<\/li>\n<li>\n<p>Tell systemd to wait forever for a decryption password. I use <code>systemd-boot<\/code>, so I edited\n<code>\/boot\/loader\/entries\/linux-cachyos<\/code>. Under <code>options<\/code>, I extended the existing\n<code>rootflags=subvol=\/@<\/code> to <code>rootflags=subvol=\/@,x-systemd.device-timeout=0<\/code>. <sup class=\"footnote-reference\" id=\"fr-2-1\"><a href=\"#fn-2\">4<\/a><\/sup><\/p>\n<\/li>\n<li>\n<p>Copy your public keys into <code>\/root\/.ssh\/authorized_keys<\/code> so they get picked up by the\ndropbear hook:<\/p>\n<pre data-lang=\"sh\" class=\"language-sh z-code\"><code class=\"language-sh\" data-lang=\"sh\"><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">cp<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> <span class=\"z-meta z-group z-expansion z-tilde\"><span class=\"z-variable z-language z-tilde z-shell\">~<\/span><\/span>\/.ssh\/authorized_keys \/root\/.ssh\/<\/span>\n<\/span><\/code><\/pre>\n<\/li>\n<li>\n<p>Generate a new public\/private keypair for use by the dropbear server.<\/p>\n<pre data-lang=\"sh\" class=\"language-sh z-code\"><code class=\"language-sh\" data-lang=\"sh\"><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">dropbearkey<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> -<\/span>t<\/span> ed25519<span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> -<\/span>f<\/span> \/etc\/dropbear\/dropbear_ed25519_host_key<\/span>\n<\/span><\/code><\/pre>\n<\/li>\n<\/ul>\n<aside role=note class=note-container><div class=note-content>\n<p>Without this, the dropbear hook will try to load keys from openssh, which means they'll be shared between early boot and your normal server. In particular that would mean your SSH server private keys would be stored unencrypted in initramfs.<\/p>\n<\/div><\/aside>\n<ul>\n<li>\n<p>Setup early networking.\n(Note: these instructions are only for Ethernet connections. If you want WiFi in early\nboot, good luck and godspeed.)<\/p>\n<ol>\n<li>Add the following config in <code>\/etc\/systemd\/network-initramfs\/10-wired.network<\/code>:<\/li>\n<\/ol>\n<pre data-lang=\"ini\" class=\"language-ini z-code\"><code class=\"language-ini\" data-lang=\"ini\"><span class=\"z-source z-genconfig\"><span class=\"z-storage z-type z-genconfig\">[Match]\n<\/span><\/span><span class=\"z-source z-genconfig\"><span class=\"z-meta z-param z-genconfig\"><span class=\"z-variable z-parameter z-genconfig\">Type<\/span><span class=\"z-keyword z-operator z-genconfig\">=<\/span><\/span>ether\n<\/span><span class=\"z-source z-genconfig\">\n<\/span><span class=\"z-source z-genconfig\"><span class=\"z-storage z-type z-genconfig\">[Network]\n<\/span><\/span><span class=\"z-source z-genconfig\"><span class=\"z-support z-constant z-genconfig\">DHCP<\/span><span class=\"z-keyword z-operator z-genconfig\">=<\/span><span class=\"z-constant z-language z-genconfig\">yes<\/span>\n<\/span><\/code><\/pre>\n<ol start=\"2\">\n<li>Register it in <code>\/etc\/mkinitcpio.conf<\/code> so it gets picked up by the <code>sd-network<\/code> hook:<\/li>\n<\/ol>\n<pre data-lang=\"sh\" class=\"language-sh z-code\"><code class=\"language-sh\" data-lang=\"sh\"><span class=\"z-source z-shell z-bash\"><span class=\"z-variable z-other z-readwrite z-assignment z-shell\">SD_NETWORK_CONFIG<\/span><span class=\"z-keyword z-operator z-assignment z-shell\">=<\/span><span class=\"z-string z-unquoted z-shell\">\/etc\/systemd\/network-initramfs<\/span>\n<\/span><\/code><\/pre>\n<p>All this rigamarole is necessary because the OS doesn't set the network interfaces to\npredictable names until late boot, so it needs some way to know which interface to use.<\/p>\n<\/li>\n<li>\n<p>Last but not least, rebuild your initramfs: <code>mkinitcpio -P<\/code>.<\/p>\n<\/li>\n<\/ul>\n<p>Next time you reboot, you should be able to ssh into <code>$(hostname)-initrd<\/code> and get a prompt\nthat looks like this:<\/p>\n<p><img src=\"\/assets\/Screenshot_20260121_222102.png\" alt=\"\" \/><\/p>\n<h2 id=\"the-getaway\">the getaway<a class=\"zola-anchor\" href=\"#the-getaway\" aria-label=\"Anchor link for: the-getaway\"><\/a>\n<\/h2>\n<p><em>In which a moral is imparted, and our scene concluded.<\/em><\/p>\n<p>The takeaway here is the same as in all my other posts: if you think something isn't\npossible to do with a computer, have you considered applying more violence?<\/p>\n<section class=\"footnotes\">\n<ol class=\"footnotes-list\">\n<li id=\"fn-3\">\n<p>and I believe in Windows, although I\u2019m less sure about that <a href=\"#fr-3-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-4\">\n<p>sometimes \/boot\/EFI <a href=\"#fr-4-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-1\">\n<p>Here \u201cinitrd\u201d stands for \u201cinitramdisk\u201d, which is another word for our initramfs system. <a href=\"#fr-1-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-2\">\n<p>See <a href=\"https:\/\/github.com\/wolegis\/mkinitcpio-systemd-extras\/wiki\/Dropbear-SSH-server#:~:text=systemd%2Edevice%2Dtimeout\">the <code>sd-dropbear<\/code>\ndocs<\/a> for more information about this. <a href=\"#fr-2-1\">\u21a9<\/a><\/p>\n<\/li>\n<\/ol>\n<\/section>\n"},{"title":"pre-commit hooks are fundamentally broken","published":"2025-12-26T00:00:00+00:00","updated":"2025-12-26T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/pre-commit-hooks-are-fundamentally-broken\/"}},"id":"https:\/\/jyn.dev\/pre-commit-hooks-are-fundamentally-broken\/","summary":"use pre-push hooks instead","content":"<p>Let's start a new Rust project.<\/p>\n<pre data-lang=\"console\" class=\"language-console z-code\"><code class=\"language-console\" data-lang=\"console\"><span class=\"z-source z-shell z-console\"><span class=\"z-keyword z-operator z-assignment z-redirection z-process z-shell\">$<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">mkdir<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> best-fizzbuzz-ever<\/span>\n<\/span><span class=\"z-source z-shell z-console\"><span class=\"z-keyword z-operator z-assignment z-redirection z-process z-shell\">$<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-support z-function z-cd z-shell\">cd<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> best-fizzbuzz-ever<\/span>\n<\/span><span class=\"z-source z-shell z-console\"><span class=\"z-keyword z-operator z-assignment z-redirection z-process z-shell\">$<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">cat<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> <span class=\"z-string z-unquoted z-heredoc z-shell\"><span class=\"z-keyword z-operator z-assignment z-redirection z-shell\">&lt;&lt;<\/span> <span class=\"z-keyword z-control z-heredoc-token z-shell\">EOF<\/span><\/span> <span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-keyword z-operator z-assignment z-redirection z-shell\">&gt;<\/span> main.rs<\/span><span class=\"z-string z-unquoted z-heredoc z-shell\">\n<\/span><\/span><\/span><span class=\"z-source z-shell z-console\"><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-string z-unquoted z-heredoc z-shell\">fn main() { for i in 0.. {\n<\/span><\/span><\/span><span class=\"z-source z-shell z-console\"><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-string z-unquoted z-heredoc z-shell\">    println (&quot;fizzbuzz&quot;);\n<\/span><\/span><\/span><span class=\"z-source z-shell z-console\"><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-string z-unquoted z-heredoc z-shell\">}}\n<\/span><\/span><\/span><span class=\"z-source z-shell z-console\"><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-string z-unquoted z-heredoc z-shell\"><span class=\"z-keyword z-control z-heredoc-token z-shell\">EOF<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-shell z-console\"><span class=\"z-keyword z-operator z-assignment z-redirection z-process z-shell\">$<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">git<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> init<\/span>\n<\/span><span class=\"z-source z-shell z-console\">Initialized empty Git repository in \/home\/jyn\/src\/third-website\/best-fizzbuzz-ever\/.git\/\n<\/span><span class=\"z-source z-shell z-console\"><span class=\"z-keyword z-operator z-assignment z-redirection z-process z-shell\">$<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">git<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> add main.rs<\/span>\n<\/span><span class=\"z-source z-shell z-console\"><span class=\"z-keyword z-operator z-assignment z-redirection z-process z-shell\">$<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">git<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> commit<span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> --<\/span>message<\/span> fizzbuzz<\/span>\n<\/span><span class=\"z-source z-shell z-console\">[main (root-commit) 661dc28] fizzbuzz\n<\/span><span class=\"z-source z-shell z-console\"> 1 file changed, 4 insertions(+)\n<\/span><span class=\"z-source z-shell z-console\"> create mode 100644 main.rs\n<\/span><\/code><\/pre>\n<p>Neat. Now let's say I add this to some list of fizzbuzz projects in different languages.\nMaybe .... <a href=\"https:\/\/github.com\/joshkunz\/fizzbuzz\">this one<\/a>.\nThey tell me I need to have \"proper formatting\" and \"use consistent style\".\nHow rude.<\/p>\n<p>Maybe I can write a pre-commit hook that checks that for me?<\/p>\n<pre data-lang=\"console\" class=\"language-console z-code\"><code class=\"language-console\" data-lang=\"console\"><span class=\"z-source z-shell z-console\"><span class=\"z-keyword z-operator z-assignment z-redirection z-process z-shell\">$<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">cat<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> <span class=\"z-string z-unquoted z-heredoc z-shell\"><span class=\"z-keyword z-operator z-assignment z-redirection z-shell\">&lt;&lt;<\/span> <span class=\"z-punctuation z-definition z-string z-begin z-shell\">&#39;<\/span><span class=\"z-keyword z-control z-heredoc-token z-shell\">EOF<\/span><span class=\"z-punctuation z-definition z-string z-end z-shell\">&#39;<\/span><\/span> <span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-keyword z-operator z-assignment z-redirection z-shell\">&gt;<\/span> pre-commit<\/span><span class=\"z-string z-unquoted z-heredoc z-shell\">\n<\/span><\/span><\/span><span class=\"z-source z-shell z-console\"><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-string z-unquoted z-heredoc z-shell\">#!\/bin\/sh\n<\/span><\/span><\/span><span class=\"z-source z-shell z-console\"><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-string z-unquoted z-heredoc z-shell\">set -eu\n<\/span><\/span><\/span><span class=\"z-source z-shell z-console\"><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-string z-unquoted z-heredoc z-shell\">for f in *.rs; do\n<\/span><\/span><\/span><span class=\"z-source z-shell z-console\"><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-string z-unquoted z-heredoc z-shell\">  rustfmt --check &quot;$f&quot;\n<\/span><\/span><\/span><span class=\"z-source z-shell z-console\"><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-string z-unquoted z-heredoc z-shell\">done\n<\/span><\/span><\/span><span class=\"z-source z-shell z-console\"><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-string z-unquoted z-heredoc z-shell\"><span class=\"z-keyword z-control z-heredoc-token z-shell\">EOF<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-shell z-console\"><span class=\"z-keyword z-operator z-assignment z-redirection z-process z-shell\">$<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">chmod<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> +x pre-commit<\/span>\n<\/span><span class=\"z-source z-shell z-console\"><span class=\"z-keyword z-operator z-assignment z-redirection z-process z-shell\">$<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">ln<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> -<\/span>s<\/span> ..\/..\/pre-commit .git\/hooks\/pre-commit<\/span>\n<\/span><span class=\"z-source z-shell z-console\"><span class=\"z-keyword z-operator z-assignment z-redirection z-process z-shell\">$<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">git<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> add pre-commit<\/span>\n<\/span><span class=\"z-source z-shell z-console\"><span class=\"z-keyword z-operator z-assignment z-redirection z-process z-shell\">$<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">git<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> commit<span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> --<\/span>message<\/span> <span class=\"z-string z-quoted z-double z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&quot;<\/span>add pre-commit hook<span class=\"z-punctuation z-definition z-string z-end z-shell\">&quot;<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-shell z-console\">Diff in \/home\/jyn\/src\/third-website\/best-fizzbuzz-ever\/src\/main.rs:1:\n<\/span><span class=\"z-source z-shell z-console\">-fn main() { for i in 0.. {\n<\/span><span class=\"z-source z-shell z-console\">-    println (&quot;fizzbuzz&quot;);\n<\/span><span class=\"z-source z-shell z-console\">-}}\n<\/span><span class=\"z-source z-shell z-console\">+fn main() {\n<\/span><span class=\"z-source z-shell z-console\">+    for i in 0.. {\n<\/span><span class=\"z-source z-shell z-console\">+        println(&quot;fizzbuzz&quot;);\n<\/span><span class=\"z-source z-shell z-console\">+    }\n<\/span><span class=\"z-source z-shell z-console\">+}\n<\/span><\/code><\/pre>\n<p>Neat! Let's commit that change.<\/p>\n<pre data-lang=\"console\" class=\"language-console z-code\"><code class=\"language-console\" data-lang=\"console\"><span class=\"z-source z-shell z-console\"><span class=\"z-keyword z-operator z-assignment z-redirection z-process z-shell\">$<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">rustfmt<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> main.rs<\/span>\n<\/span><span class=\"z-source z-shell z-console\"><span class=\"z-keyword z-operator z-assignment z-redirection z-process z-shell\">$<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">git<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> commit<span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> --<\/span>message<\/span> <span class=\"z-string z-quoted z-double z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&quot;<\/span>add pre-commit hook<span class=\"z-punctuation z-definition z-string z-end z-shell\">&quot;<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-shell z-console\">[main 3be7b87] add pre-commit hook\n<\/span><span class=\"z-source z-shell z-console\"> 1 file changed, 4 insertions(+)\n<\/span><span class=\"z-source z-shell z-console\"> create mode 100755 pre-commit\n<\/span><span class=\"z-source z-shell z-console\"><span class=\"z-keyword z-operator z-assignment z-redirection z-process z-shell\">$<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">git<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> status<\/span>\n<\/span><span class=\"z-source z-shell z-console\">On branch main\n<\/span><span class=\"z-source z-shell z-console\">Changes not staged for commit:\n<\/span><span class=\"z-source z-shell z-console\">  (use &quot;git add &lt;file&gt;...&quot; to update what will be committed)\n<\/span><span class=\"z-source z-shell z-console\">  (use &quot;git restore &lt;file&gt;...&quot; to discard changes in working directory)\n<\/span><span class=\"z-source z-shell z-console\">\tmodified:   main.rs\n<\/span><\/code><\/pre>\n<p>Oh ... We fixed the formatting, but we didn't actually stage the changes.\nThe pre-commit hook runs on the <em>working tree<\/em>, not on the <em>index<\/em>, so it didn't catch the issue.\nWe can see that the version tracked by git still has the wrong formatting:<\/p>\n<pre data-lang=\"console\" class=\"language-console z-code\"><code class=\"language-console\" data-lang=\"console\"><span class=\"z-source z-shell z-console\"><span class=\"z-keyword z-operator z-assignment z-redirection z-process z-shell\">$<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">git<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> show HEAD:main.rs<\/span>\n<\/span><span class=\"z-source z-shell z-console\">fn main() { for i in 0.. {\n<\/span><span class=\"z-source z-shell z-console\">    println (&quot;fizzbuzz&quot;);\n<\/span><span class=\"z-source z-shell z-console\">}}\n<\/span><\/code><\/pre>\n<p>Maybe we can make the script smarter?\nLet's checkout all the files in the index into a temporary directory and run our pre-commit hook there. <sup class=\"footnote-reference\" id=\"fr-4-1\"><a href=\"#fn-4\">1<\/a><\/sup><\/p>\n<pre data-lang=\"console\" class=\"language-console z-code\"><code class=\"language-console\" data-lang=\"console\"><span class=\"z-source z-shell z-console\"><span class=\"z-keyword z-operator z-assignment z-redirection z-process z-shell\">$<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">cat<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> <span class=\"z-string z-unquoted z-heredoc z-shell\"><span class=\"z-keyword z-operator z-assignment z-redirection z-shell\">&lt;&lt;<\/span> <span class=\"z-punctuation z-definition z-string z-begin z-shell\">&#39;<\/span><span class=\"z-keyword z-control z-heredoc-token z-shell\">EOF<\/span><span class=\"z-punctuation z-definition z-string z-end z-shell\">&#39;<\/span><\/span> <span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-keyword z-operator z-assignment z-redirection z-shell\">&gt;<\/span> pre-commit<\/span><span class=\"z-string z-unquoted z-heredoc z-shell\">\n<\/span><\/span><\/span><span class=\"z-source z-shell z-console\"><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-string z-unquoted z-heredoc z-shell\">#!\/bin\/sh\n<\/span><\/span><\/span><span class=\"z-source z-shell z-console\"><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-string z-unquoted z-heredoc z-shell\">set -eu\n<\/span><\/span><\/span><span class=\"z-source z-shell z-console\"><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-string z-unquoted z-heredoc z-shell\">\n<\/span><\/span><\/span><span class=\"z-source z-shell z-console\"><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-string z-unquoted z-heredoc z-shell\">tmpdir=$(mktemp -d --tmpdir &quot;$(basename &quot;$(realpath .)&quot;)-pre-commit.XXXX&quot;)\n<\/span><\/span><\/span><span class=\"z-source z-shell z-console\"><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-string z-unquoted z-heredoc z-shell\">trap &#39;rm -r &quot;$tmpdir&quot;&#39; EXIT\n<\/span><\/span><\/span><span class=\"z-source z-shell z-console\"><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-string z-unquoted z-heredoc z-shell\">git checkout-index --all --prefix=&quot;$tmpdir\/&quot;\n<\/span><\/span><\/span><span class=\"z-source z-shell z-console\"><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-string z-unquoted z-heredoc z-shell\">for f in $tmpdir\/*.rs; do\n<\/span><\/span><\/span><span class=\"z-source z-shell z-console\"><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-string z-unquoted z-heredoc z-shell\">  rustfmt --check &quot;$f&quot;\n<\/span><\/span><\/span><span class=\"z-source z-shell z-console\"><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-string z-unquoted z-heredoc z-shell\">done\n<\/span><\/span><\/span><span class=\"z-source z-shell z-console\"><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-string z-unquoted z-heredoc z-shell\"><span class=\"z-keyword z-control z-heredoc-token z-shell\">EOF<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-shell z-console\"><span class=\"z-keyword z-operator z-assignment z-redirection z-process z-shell\">$<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">git<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> add pre-commit<\/span>\n<\/span><span class=\"z-source z-shell z-console\"><span class=\"z-keyword z-operator z-assignment z-redirection z-process z-shell\">$<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">git<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> commit<span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> --<\/span>message<\/span> <span class=\"z-string z-quoted z-double z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&quot;<\/span>make pre-commit hook smarter<span class=\"z-punctuation z-definition z-string z-end z-shell\">&quot;<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-shell z-console\">Diff in \/tmp\/best-fizzbuzz-ever-pre-commit.ZNyw\/main.rs:1:\n<\/span><span class=\"z-source z-shell z-console\">-fn main() { for i in 0.. {\n<\/span><span class=\"z-source z-shell z-console\">-    println (&quot;fizzbuzz&quot;);\n<\/span><span class=\"z-source z-shell z-console\">-}}\n<\/span><span class=\"z-source z-shell z-console\">+fn main() {\n<\/span><span class=\"z-source z-shell z-console\">+    for i in 0.. {\n<\/span><span class=\"z-source z-shell z-console\">+        println(&quot;fizzbuzz&quot;);\n<\/span><span class=\"z-source z-shell z-console\">+    }\n<\/span><span class=\"z-source z-shell z-console\">+}\n<\/span><span class=\"z-source z-shell z-console\">\n<\/span><\/code><\/pre>\n<p>Yay! That caught the issue.\nNow let's add our rust program to that collection of fizzbuzz programs.<\/p>\n<pre data-lang=\"console\" class=\"language-console z-code\"><code class=\"language-console\" data-lang=\"console\"><span class=\"z-source z-shell z-console\"><span class=\"z-keyword z-operator z-assignment z-redirection z-process z-shell\">$<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">git<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> add main.rs<\/span>\n<\/span><span class=\"z-source z-shell z-console\"><span class=\"z-keyword z-operator z-assignment z-redirection z-process z-shell\">$<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">git<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> commit<span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> --<\/span>message<\/span> <span class=\"z-string z-quoted z-double z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&quot;<\/span>make pre-commit hook smarter<span class=\"z-punctuation z-definition z-string z-end z-shell\">&quot;<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-shell z-console\">[main 3cb40f6] make pre-commit hook smarter\n<\/span><span class=\"z-source z-shell z-console\"> 2 files changed, 11 insertions(+), 4 deletions(-)\n<\/span><span class=\"z-source z-shell z-console\"><span class=\"z-keyword z-operator z-assignment z-redirection z-process z-shell\">$<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">git<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> remote add upstream https:\/\/github.com\/joshkunz\/fizzbuzz<\/span>\n<\/span><span class=\"z-source z-shell z-console\"><span class=\"z-keyword z-operator z-assignment z-redirection z-process z-shell\">$<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">git<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> fetch upstream<\/span>\n<\/span><span class=\"z-source z-shell z-console\">remote: Enumerating objects: 222, done.\n<\/span><span class=\"z-source z-shell z-console\">remote: Total 222 (delta 0), reused 0 (delta 0), pack-reused 222 (from 1)\n<\/span><span class=\"z-source z-shell z-console\">Receiving objects: 100% (222\/222), 29.08 KiB | 29.08 MiB\/s, done.\n<\/span><span class=\"z-source z-shell z-console\">Resolving deltas: 100% (117\/117), done.\n<\/span><span class=\"z-source z-shell z-console\">From https:\/\/github.com\/joshkunz\/fizzbuzz\n<\/span><span class=\"z-source z-shell z-console\"> * [new branch]      master     -&gt; upstream\/master\n<\/span><span class=\"z-source z-shell z-console\"><span class=\"z-keyword z-operator z-assignment z-redirection z-process z-shell\">$<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">git<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> rebase upstream<\/span>\n<\/span><span class=\"z-source z-shell z-console\">Successfully rebased and updated refs\/heads\/main.\n<\/span><\/code><\/pre>\n<p>Maybe we'll make one last tweak...<\/p>\n<pre data-lang=\"console\" class=\"language-console z-code\"><code class=\"language-console\" data-lang=\"console\"><span class=\"z-source z-shell z-console\"><span class=\"z-keyword z-operator z-assignment z-redirection z-process z-shell\">$<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">sed<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> -<\/span>i<\/span> <span class=\"z-string z-quoted z-single z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&#39;<\/span>1i \/\/ Written by jyn<span class=\"z-punctuation z-definition z-string z-end z-shell\">&#39;<\/span><\/span> main.rs<\/span>\n<\/span><span class=\"z-source z-shell z-console\"><span class=\"z-keyword z-operator z-assignment z-redirection z-process z-shell\">$<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">git<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> commit main.rs<span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> --<\/span>message<\/span> <span class=\"z-string z-quoted z-double z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&quot;<\/span>mark who wrote fizzbuzz<span class=\"z-punctuation z-definition z-string z-end z-shell\">&quot;<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-shell z-console\">Diff in \/tmp\/best-fizzbuzz-ever-pre-commit.n1Pj\/fizzbuzz-traits.rs:4:\n<\/span><span class=\"z-source z-shell z-console\"> use std::iter;\n<\/span><span class=\"z-source z-shell z-console\">\n<\/span><span class=\"z-source z-shell z-console\"> struct FizzBuzz {\n<\/span><span class=\"z-source z-shell z-console\">-    from : i32\n<\/span><span class=\"z-source z-shell z-console\">-  , to : i32\n<\/span><span class=\"z-source z-shell z-console\">+    from: i32,\n<\/span><span class=\"z-source z-shell z-console\">+    to: i32,\n<\/span><span class=\"z-source z-shell z-console\"> }\n<\/span><span class=\"z-source z-shell z-console\">\n<\/span><span class=\"z-source z-shell z-console\"> impl FizzBuzz {\n<\/span><\/code><\/pre>\n<p>Uh. Huh. Right.\nThe code that was already here wasn't formatted according to rustfmt.\nOur script is running on every file in the git repo, so it won't let us commit.<\/p>\n<p>Maybe we can change it to only run on modified files?<\/p>\n<pre data-lang=\"console\" class=\"language-console z-code\"><code class=\"language-console\" data-lang=\"console\"><span class=\"z-source z-shell z-console\"><span class=\"z-keyword z-operator z-assignment z-redirection z-process z-shell\">$<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">cat<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> <span class=\"z-string z-unquoted z-heredoc z-shell\"><span class=\"z-keyword z-operator z-assignment z-redirection z-shell\">&lt;&lt;<\/span> <span class=\"z-punctuation z-definition z-string z-begin z-shell\">&#39;<\/span><span class=\"z-keyword z-control z-heredoc-token z-shell\">EOF<\/span><span class=\"z-punctuation z-definition z-string z-end z-shell\">&#39;<\/span><\/span> <span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-keyword z-operator z-assignment z-redirection z-shell\">&gt;<\/span> pre-commit<\/span><span class=\"z-string z-unquoted z-heredoc z-shell\">\n<\/span><\/span><\/span><span class=\"z-source z-shell z-console\"><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-string z-unquoted z-heredoc z-shell\">#!\/bin\/sh\n<\/span><\/span><\/span><span class=\"z-source z-shell z-console\"><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-string z-unquoted z-heredoc z-shell\">set -eu\n<\/span><\/span><\/span><span class=\"z-source z-shell z-console\"><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-string z-unquoted z-heredoc z-shell\">\n<\/span><\/span><\/span><span class=\"z-source z-shell z-console\"><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-string z-unquoted z-heredoc z-shell\">files=$(git diff --name-only --cached --no-ext-diff --diff-filter=d)\n<\/span><\/span><\/span><span class=\"z-source z-shell z-console\"><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-string z-unquoted z-heredoc z-shell\">\n<\/span><\/span><\/span><span class=\"z-source z-shell z-console\"><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-string z-unquoted z-heredoc z-shell\">tmpdir=$(mktemp -d --tmpdir &quot;$(basename &quot;$(realpath .)&quot;)-pre-commit.XXXX&quot;)\n<\/span><\/span><\/span><span class=\"z-source z-shell z-console\"><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-string z-unquoted z-heredoc z-shell\">trap &#39;rm -r &quot;$tmpdir&quot;&#39; EXIT\n<\/span><\/span><\/span><span class=\"z-source z-shell z-console\"><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-string z-unquoted z-heredoc z-shell\">\n<\/span><\/span><\/span><span class=\"z-source z-shell z-console\"><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-string z-unquoted z-heredoc z-shell\">printf %s &quot;$files&quot; | tr &#39;\\n&#39; &#39;\\0&#39; | xargs -0 git checkout-index --prefix=&quot;$tmpdir\/&quot;\n<\/span><\/span><\/span><span class=\"z-source z-shell z-console\"><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-string z-unquoted z-heredoc z-shell\">for f in $tmpdir\/*.rs; do\n<\/span><\/span><\/span><span class=\"z-source z-shell z-console\"><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-string z-unquoted z-heredoc z-shell\">  rustfmt --check &quot;$f&quot;\n<\/span><\/span><\/span><span class=\"z-source z-shell z-console\"><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-string z-unquoted z-heredoc z-shell\">done\n<\/span><\/span><\/span><span class=\"z-source z-shell z-console\"><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-string z-unquoted z-heredoc z-shell\"><span class=\"z-keyword z-control z-heredoc-token z-shell\">EOF<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-shell z-console\"><span class=\"z-keyword z-operator z-assignment z-redirection z-process z-shell\">$<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">git<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> commit main.rs pre-commit <span class=\"z-punctuation z-separator z-continuation z-line z-shell\">\\\n<\/span><\/span><\/span><span class=\"z-source z-shell z-console\"><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\">  --<\/span>message<\/span> <span class=\"z-string z-quoted z-double z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&quot;<\/span>update main.rs; make pre-commit even smarter<span class=\"z-punctuation z-definition z-string z-end z-shell\">&quot;<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-shell z-console\">[main f2925bc] update main.rs; make pre-commit even smarter\n<\/span><span class=\"z-source z-shell z-console\"> 2 files changed, 5 insertions(+), 1 deletion(-)\n<\/span><\/code><\/pre>\n<p>Alright. Cool.<\/p>\n<p>Let's do one last thing.\nLet's say we had an existing PR to this repo and we need to rebase it.\nMaybe it had a merge conflict, or maybe there was a fix on main that we need in order to implement our solution.<\/p>\n<pre data-lang=\"console\" class=\"language-console z-code\"><code class=\"language-console\" data-lang=\"console\"><span class=\"z-source z-shell z-console\"><span class=\"z-keyword z-operator z-assignment z-redirection z-process z-shell\">$<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">git<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> checkout upstream\/HEAD  <span class=\"z-comment z-line z-number-sign z-shell\"><span class=\"z-punctuation z-definition z-comment z-begin z-shell\">#<\/span><\/span><span class=\"z-comment z-line z-number-sign z-shell\"> Simulate an old PR by checking out an old commit<\/span><\/span>\n<\/span><span class=\"z-source z-shell z-console\">HEAD is now at 56bf3ab Adds E to the README\n<\/span><span class=\"z-source z-shell z-console\"><span class=\"z-keyword z-operator z-assignment z-redirection z-process z-shell\">$<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-support z-function z-echo z-shell\">echo<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> <span class=\"z-string z-quoted z-single z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&#39;<\/span>fn main() { println!(&quot;this counts as fizzbuzz, right?&quot;); }<span class=\"z-punctuation z-definition z-string z-end z-shell\">&#39;<\/span><\/span> <span class=\"z-keyword z-operator z-assignment z-redirection z-shell\">&gt;<\/span> print.rs<\/span>\n<\/span><span class=\"z-source z-shell z-console\"><span class=\"z-keyword z-operator z-assignment z-redirection z-process z-shell\">$<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">git<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> add print.rs<\/span>\n<\/span><span class=\"z-source z-shell z-console\"><span class=\"z-keyword z-operator z-assignment z-redirection z-process z-shell\">$<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">git<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> commit<span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> --<\/span>message<\/span> <span class=\"z-string z-quoted z-double z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&quot;<\/span>Add print.rs<span class=\"z-punctuation z-definition z-string z-end z-shell\">&quot;<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-shell z-console\">[detached HEAD 3d1bbf7] Add print.rs\n<\/span><span class=\"z-source z-shell z-console\"> 1 file changed, 1 insertion(+)\n<\/span><span class=\"z-source z-shell z-console\"> create mode 100644 print.rs\n<\/span><\/code><\/pre>\n<p>And let's also say that we want to edit the commit message.<\/p>\n<pre data-lang=\"console\" class=\"language-console z-code\"><code class=\"language-console\" data-lang=\"console\"><span class=\"z-source z-shell z-console\"><span class=\"z-keyword z-operator z-assignment z-redirection z-process z-shell\">$<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">git<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> rebase<span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> -<\/span>i<\/span> main  <span class=\"z-comment z-line z-number-sign z-shell\"><span class=\"z-punctuation z-definition z-comment z-begin z-shell\">#<\/span><\/span><span class=\"z-comment z-line z-number-sign z-shell\"> Rebase this whole branch over our main branch<\/span><\/span>\n<\/span><span class=\"z-source z-shell z-console\">reword 3d1bbf7 Add print.rs\n<\/span><span class=\"z-source z-shell z-console\"><span class=\"z-keyword z-operator z-assignment z-redirection z-process z-shell\">#<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">Rebase<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> f2925bc..3d1bbf7 onto f2925bc (1 command<\/span><span class=\"z-meta z-function-call z-shell\"><\/span>)\n<\/span><\/code><\/pre>\n<h2 id=\"now-we-really-have-a-problem\">Now, we <em>really<\/em> have a problem.<a class=\"zola-anchor\" href=\"#now-we-really-have-a-problem\" aria-label=\"Anchor link for: now-we-really-have-a-problem\"><\/a>\n<\/h2>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">Error: file `\/tmp\/best-fizzbuzz-ever-pre-commit.p3az\/*.rs` does not exist\n<\/span><span class=\"z-text z-plain\">Could not apply 3d1bbf7... Add print.rs\n<\/span><\/code><\/pre>\n<p>Two things went wrong here:<\/p>\n<ol>\n<li>Our pre-commit hook can't handle commits that don't have any Rust files.<\/li>\n<li>Our pre-commit hook <em>ran on a branch we were rebasing<\/em>. <sup class=\"footnote-reference\" id=\"fr-2-1\"><a href=\"#fn-2\">2<\/a><\/sup><\/li>\n<\/ol>\n<p>Fixing the first thing doesn't really help us, because we don't control other people's branches.\nThey might have used <code>git commit --no-verify<\/code>.\nThey might not even have a pre-commit hook installed.\nThey might have had a branch that passed the hook when they originally wrote it, but not after a rebase (e.g. if your hook is <code>cargo check<\/code> or something like that).\nThey might have had a branch that used an old version of the hook that didn't have as many checks as a later version.<\/p>\n<p>Our only real choice here is to pass <code>--no-verify<\/code> to <code>git rebase<\/code> every time we run it, and to <code>git commit<\/code> for every commit in the rebase we modify,\nand possibly even to every <code>git merge<\/code> we run outside of a rebase.<\/p>\n<p>This is because pre-commit hooks are a <em>fundamentally broken idea<\/em>.\nCode does not exist in isolation.\nCommits that are local to a developer machine do not ever go through CI.\nCommits don't even necessarily mean that that the code is ready to publish\u2014pre-commit hooks don't run on <code>git stash<\/code> for a reason!\nI don't use <code>git stash<\/code>, I use <code>git commit<\/code> so that my stashes are tied to a branch, and hooks completely break this workflow.<\/p>\n<p>More than that, pre-commit hooks are <em>preventing you from saving your work<\/em>.\nThere should be a <em>really, really good reason<\/em> to prevent you from saving your work, and IMO \"doesn't pass the test suite\" is not that.\nI have similar feelings about format-on-save hooks.<\/p>\n<p>There are a <a href=\"https:\/\/blog.plover.com\/prog\/git\/hook-disaster.html\">bunch<\/a>\nof <a href=\"https:\/\/dev.to\/afl_ext\/are-pre-commit-git-hooks-a-good-idea-i-dont-think-so-38j6\">other<\/a> <a href=\"https:\/\/github.com\/rust-lang\/rust\/issues\/77620\">footguns<\/a> with pre-commit hooks.\nThis doesn't even count the fact that nearly all pre-commit hooks are implemented in a broken way and just blindly run on the worktree, and are slow or unreliable or both.\nDon't get me started on pre-commit hooks that try to add things to the commit you're about to make, or projects that try to automatically install a hook when you run the test suite.<\/p>\n<aside role=note class=note-container><div class=note-content>\n<p>The <a href=\"https:\/\/pre-commit.com\/\"><code>pre-commit<\/code><\/a> framework (or its cousin, <a href=\"https:\/\/github.com\/lint-staged\/lint-staged\">lint-staged<\/a>) does <em>not<\/em> fix this.\nIt fixes the issues about running on the index by stashing your changes with\n<a href=\"https:\/\/git-scm.com\/docs\/git-stash#Documentation\/git-stash.txt---keep-index\"><code>--keep-index<\/code><\/a>,\nwhich works but modifies your git state.\nIt doesn't fix the issues about running during a rebase, nor does it prevent hooks from trying to add things to the current commit. <sup class=\"footnote-reference\" id=\"fr-lint-staged-1\"><a href=\"#fn-lint-staged\">3<\/a><\/sup><\/p>\n<p>\"Just don't write bad hooks\" doesn't work if I'm working on someone else's project where I don't control the hook.<\/p>\n<\/div><\/aside>\n<p>Please just don't use <code>pre-commit<\/code> hooks. Use <code>pre-push<\/code> instead. <sup class=\"footnote-reference\" id=\"fr-1-1\"><a href=\"#fn-1\">4<\/a><\/sup>\n<code>pre-push<\/code> hooks nearly avoid all of these issues.<\/p>\n<p>The only use case where I think pre-commit hooks are a good idea is for things that must <em>never<\/em> committed, that are worth interrupting a complicated rebase to prevent; namely: credentials.\nOnce credentials are committed they're quite difficult to get out, and even harder to be sure you haven't missed them.<\/p>\n<h2 id=\"tips-for-writing-a-pre-push-hook\">Tips for writing a <code>pre-push<\/code> hook<a class=\"zola-anchor\" href=\"#tips-for-writing-a-pre-push-hook\" aria-label=\"Anchor link for: tips-for-writing-a-pre-push-hook\"><\/a>\n<\/h2>\n<ul>\n<li>Run on the index, not the working tree, as described above. <sup class=\"footnote-reference\" id=\"fr-3-1\"><a href=\"#fn-3\">5<\/a><\/sup><\/li>\n<li>Only add checks that are fast and reliable. Checks that touch the network should never go in a hook. Checks that are slow and require an up-to-date build cache should never go in a hook. Checks that require credentials or a running local service should never go in a hook.<\/li>\n<li>Be as quiet as possible. This hook is running buried inside a bunch of other commands, often without the developer knowing that the hook is going to run. Don't hide other important output behind a wall of progress messages.<\/li>\n<li>Don't set the hook up automatically. Whatever tool you use that promises to make this reliable is wrong. There is not a way to do this reliably, and the number of times it's broken on me is more than I can count. Please just add docs for how to set it up manually, prominently featured in your CONTRIBUTING docs. (You do have contributing docs, right?)<\/li>\n<\/ul>\n<aside role=note class=note-container><div class=note-content>\n<p>If the hook does fail, and the changes affect an older commit than the most recent,\nyou can use a combination of <a href=\"https:\/\/github.com\/tummychow\/git-absorb\"><code>git-absorb<\/code><\/a>, <a href=\"https:\/\/git-revise.readthedocs.io\/en\/latest\/man.html\"><code>git-revise<\/code><\/a>,\nand <a href=\"https:\/\/git-scm.com\/docs\/git-rebase#Documentation\/git-rebase.txt---execcmd\"><code>git rebase -X ours --exec<\/code><\/a>\nto put them in the appropriate commit before pushing again.<\/p>\n<\/div><\/aside>\n<p>And don't write <code>pre-commit<\/code> hooks!<\/p>\n<section class=\"footnotes\">\n<ol class=\"footnotes-list\">\n<li id=\"fn-4\">\n<p>This is really quite slow on large enough repos, but there's not any real alternative. <code>git stash --keep-index<\/code> messes with git index state and also with your stashes. The only VCS that exposes a FUSE filesystem of its commits is <a href=\"https:\/\/github.com\/facebook\/sapling\/blob\/main\/eden\/fs\/docs\/Overview.md\">Sapling<\/a>, which is poorly supported outside Facebook. The best you can do is give up on looking at the whole working copy and only write hooks that read a single file at a time. <a href=\"#fr-4-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-2\">\n<p>By default this doesn't happen when running bare <code>rebase<\/code>, but the second you add <code>--interactive<\/code>, nearly anything you do runs a hook. Hooks will also run when you attempt to resolve merge conflicts. <a href=\"#fr-2-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-lint-staged\">\n<p><code>lint-staged<\/code> does actually have a <code>--fail-on-changes<\/code> flag which aborts the commit, but that still modifies the working tree, and it's not on by default. <a href=\"#fr-lint-staged-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-1\">\n<p>For more info about the difference, and a full list of possible hooks, see <a href=\"https:\/\/git-scm.com\/docs\/githooks\"><code>man 5 githooks<\/code><\/a>. <a href=\"#fr-1-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-3\">\n<p>Notice that I don't say \"only run on changed files\". That's because it's <a href=\"https:\/\/lore.kernel.org\/git\/CAHnEOG2o784dk+OpkGt-1qjRJb34=sFMJvh-JRJ3v+GNBxFywQ@mail.gmail.com\/\">not actually possible to reliably determine which branch the current commit is based on<\/a>, the best you can do is pick a random branch that looks likely. <a href=\"#fr-3-1\">\u21a9<\/a><\/p>\n<\/li>\n<\/ol>\n<\/section>\n"},{"title":"i'm just having fun","published":"2025-12-15T00:00:00+00:00","updated":"2025-12-15T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/i-m-just-having-fun\/"}},"id":"https:\/\/jyn.dev\/i-m-just-having-fun\/","summary":"programming isn't a competition","content":"<figure>\n<blockquote cite=\"https:\/\/x.com\/reillywood\/status\/1370424931477954560\">\n<p>IT IS ONLY COMPUTER<\/p>\n<\/blockquote>\n<figcaption>\n<cite ><a href=\"https:\/\/x.com\/reillywood\/status\/1370424931477954560\">Reilly Wood<\/a><\/cite>\n<\/figcaption>\n<\/figure>\n<p>i work professionally on a compiler and write about build systems in my free time and as a result people often say things to me like \"reading your posts points to me how really smart you are\" or \"reading a lot of this shit makes me feel super small\". this makes me quite uncomfortable and is not the reaction i'm seeking when i write blog posts.<\/p>\n<h2 id=\"it-s-not-a-competition\">it's not a competition<a class=\"zola-anchor\" href=\"#it-s-not-a-competition\" aria-label=\"Anchor link for: it-s-not-a-competition\"><\/a>\n<\/h2>\n<p>i mean, in some sense if you work as a professional programmer it is a competition, because the job market sucks right now. but i think usually when people say they feel dumb, it's not in the sense of \"how am i supposed to get a job when jyn exists\" but more \"jyn can do things i can't and that makes me feel bad\".<\/p>\n<h3 id=\"you-can-do-hard-things\">you can do hard things<a class=\"zola-anchor\" href=\"#you-can-do-hard-things\" aria-label=\"Anchor link for: you-can-do-hard-things\"><\/a>\n<\/h3>\n<p>all the things i know i learned by experimenting with them, or by reading books or posts or man pages or really obscure error messages. sometimes <a href=\"https:\/\/drmaciver.substack.com\/p\/how-to-do-everything\">there's a trick to it<\/a> but sometimes <a href=\"https:\/\/jordanstacey.substack.com\/p\/the-gen-z-resilience-drought\">it's just hard work<\/a>. i am not magic. <a href=\"\/theory-building-without-a-mentor\/\">you can learn these things too<\/a>.<\/p>\n<h3 id=\"everyone-has-their-own-area-of-specialization\">everyone has their own area of specialization<a class=\"zola-anchor\" href=\"#everyone-has-their-own-area-of-specialization\" aria-label=\"Anchor link for: everyone-has-their-own-area-of-specialization\"><\/a>\n<\/h3>\n<p>if you don't want to spend a bunch of time learning about how computers work, you don't have to! not knowing about gory computer internals does not make you dumb or computer illiterate or anything. everyone has their own specialty and mine is compilers and build systems. i don't know jack shit about economics or medicine! having a different specialty than me doesn't mean you're dumb.<\/p>\n<p>i really hate that computing and STEM have this mystique in our society. to the extent that engineering demonstrates intelligence, it's by <em>repeatedly forcing you to confront the results of your own mistakes<\/em>, in such a way that errors can't be ignored. there are lots of ways to do that which don't involve programming or college-level math! performance art and carpentry and running your own business or household all force you to confront your own mistakes in this way and deserve no less respect than STEM.<\/p>\n<h2 id=\"if-i-can-t-feminize-my-compiler-what-s-the-point\">if i can't feminize my compiler, what's the point?<a class=\"zola-anchor\" href=\"#if-i-can-t-feminize-my-compiler-what-s-the-point\" aria-label=\"Anchor link for: if-i-can-t-feminize-my-compiler-what-s-the-point\"><\/a>\n<\/h2>\n<p>by and large, when i learn new things about computers, it's because i'm fucking around. the fucking around is the point. if all the writing helps people learn and come up with cool new ideas, that's neat too.<\/p>\n<p>half the time the fucking around is just to make people say \"jyn NO\". half the time it's because i want to make art with my code. i really, sincerely, believe that art is one of the most important uses for a computer.<\/p>\n<p>i'm not doing this for the money. i happened to get very lucky that my passion pays very well, but i got into this industry before realizing how much programmers actually make, and now that i work for a european company i don't make US tech salaries anyway. i do it for the love of the game.<\/p>\n<p>some extracts from the jyn computer experience:<\/p>\n<a href=https:\/\/github.com\/rust-lang\/rust\/pull\/118756>\n<img src=\"&#x2F;assets&#x2F;Pasted image 20251215042122.png\"\n\t alt=\"screenshot of a rust-lang PR titled &#x27;use bold magenta instead of bold white for highlighting&#x27;. the first sentence in the description is &#x27;according to a poll of gay people in my phone, purple is the most popular color to use for highlighting&#x27;\"\n\t\n>\n<\/a>\n<br>\n<br>\n<p><img src=\"\/assets\/Pasted%20image%2020251215042623.png\" alt=\"a series of quotes such as &quot;jyn yeah you&#39;re fucking insane&quot; and &quot;what the actual fuck. but also i love it&quot;\" \/><\/p>\n<p><a href=\"https:\/\/bsky.app\/profile\/jyn.dev\/post\/3klso625ibr26\"><img src=\"\/assets\/Pasted%20image%2020251215043434.png\" alt=\"screenshot of a bluesky post by jyn that says &quot;She is humming. She is dancing. She is beautiful.&quot; below are three pictures, one J program, and a quote-tweet of A Ladder To by Jana H-S.\" \/><\/a><\/p>\n<img src=\"&#x2F;assets&#x2F;Pasted%20image%2020251215053854.png\"\n\t alt=\"screenshot of a discord conversation. it reads: jyn: for a while i had a custom build of rustc whose version string said \u201clove you love you love you\u201d in purple. jyn: i should bring that back now that i work on ferrocene. anon: You&#x27;ve heard of cargo mommy now get ready for rustc tsundere\"\n\t\n>\n<br>\n<br>\n<p><img src=\"\/assets\/Pasted%20image%2020251215043906.png\" alt=\"screenshot of a shell command showing the version of a custom-built rustc. it says &quot;Ferrocene rolling love you love you love you&quot;, with the &quot;love you&quot;s in purple.\" \/><\/p>\n<h2 id=\"my-advice\">my advice<a class=\"zola-anchor\" href=\"#my-advice\" aria-label=\"Anchor link for: my-advice\"><\/a>\n<\/h2>\n<p>you really shouldn't take advice from me lol <img src=\"\/assets\/Pasted%20image%2020251215055218.png\" alt=\"the WWII survivorship bias plane, with red dots indicating bullet holes on the wings, tail, and central body\" \/><\/p>\n<p>however! if you are determined to do so anyway, what i can do is point you towards:<\/p>\n<h3 id=\"places-to-start-fucking-around-and-finding-out\">places to start fucking around and finding out<a class=\"zola-anchor\" href=\"#places-to-start-fucking-around-and-finding-out\" aria-label=\"Anchor link for: places-to-start-fucking-around-and-finding-out\"><\/a>\n<\/h3>\n<p>highest thing i can recommend is building a tool for yourself.\nmaybe it's a spreadsheet that saves you an hour of work a week.\nmaybe it's a little website you play around with.\nmaybe it's something in RPGmaker.\nthe exact thing doesn't matter, the important part is that it's fun and you have something real at the end of it,\nwhich motivates you to keep going even when the computer is breaking in three ways you didn't even know were possible.<\/p>\n<p>second thing i can recommend is looking at things other people have built.\nyou won't understand all of it and that's ok.\npick a part of it that looks interesting and do a deep dive on how it works.<\/p>\n<p>i can recommend the following places to look when you're getting started:<\/p>\n<ul>\n<li><a href=\"https:\/\/developer.mozilla.org\/en-US\/docs\/Web\">Mozilla Development Network<\/a><\/li>\n<li><a href=\"https:\/\/wiki.archlinux.org\/title\/Main_page\">Arch Wiki<\/a><\/li>\n<li><a href=\"https:\/\/stackoverflow.com\/\">StackOverflow<\/a><\/li>\n<li><a href=\"https:\/\/www.alicemaz.com\/writing\/program.html\">alice maz, \"how I think when I think about programming\"<\/a><\/li>\n<\/ul>\n<p>most importantly, remember:\n<img src=\"\/assets\/it-is-only-computer.png\" alt=\"Practice Guide for Computer\" \/><\/p>\n"},{"title":"what is a build system, anyway?","published":"2025-12-12T00:00:00+00:00","updated":"2025-12-12T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/what-is-a-build-system-anyway\/"}},"id":"https:\/\/jyn.dev\/what-is-a-build-system-anyway\/","summary":"Some definitions and an overview of the world of build systems","content":"<p>Andrew Nesbitt recently wrote a post titled <a href=\"https:\/\/nesbitt.io\/2025\/12\/02\/what-is-a-package-manager.html\">What is a Package Manager<\/a>? This post attempts to do the same for build systems.<\/p>\n<h2 id=\"big-picture\">big picture<a class=\"zola-anchor\" href=\"#big-picture\" aria-label=\"Anchor link for: big-picture\"><\/a>\n<\/h2>\n<p>At a high level, <strong>build systems<\/strong> are tools or libraries that provide a way to <strong>define<\/strong> and <strong>execute<\/strong> a series of transformations from <strong>input<\/strong> data to <strong>output<\/strong> data that are <strong>memoized<\/strong> by <strong>caching<\/strong> them in an <strong>object store<\/strong>.<\/p>\n<p>Transformations are called <strong>steps<\/strong> or <strong>rules<\/strong> <sup class=\"footnote-reference\" id=\"fr-14-1\"><a href=\"#fn-14\">1<\/a><\/sup> and define how to execute a <strong>task<\/strong> that generates zero or more outputs from zero or more inputs.\nA rule is usually the <strong>unit of caching<\/strong>; i.e. the <strong>cache points<\/strong> are the outputs of a rule, and <strong>cache invalidations<\/strong> must happen on the inputs of a rule.\nRules can have <strong>dependencies<\/strong> on previous outputs, forming a directed graph called a <strong>dependency graph<\/strong>.\nDependencies that form a cyclic graph are called <strong>circular dependencies<\/strong> and are usually banned.<sup class=\"footnote-reference\" id=\"fr-5-1\"><a href=\"#fn-5\">2<\/a><\/sup><\/p>\n<p>Outputs that are only used by other rules, but not \u201cinteresting\u201d to the end-user, are called <strong>intermediate outputs<\/strong>.<\/p>\n<p>A output is <strong>outdated<\/strong>, <strong>dirty<\/strong>, or <strong>stale<\/strong> if one of its dependencies is modified, or, <strong>transitively<\/strong>, if one of its dependencies is outdated.\nStale outputs invalidate the cache and require the outputs to be <strong>rebuilt<\/strong>.\nAn output that is cached and not dirty is <strong>up-to-date<\/strong>.\nRules are outdated if any of their outputs are outdated.\nIf a rule has no outputs, it is always outdated.<\/p>\n<p>Each invocation of the build tool is called a <strong>build<\/strong>.\nA <strong>full build<\/strong> or <strong>clean build<\/strong> occurs when the cache is empty and all transformations are executed as a <strong>batch job<\/strong>.\nA cache is <strong>full<\/strong> if all its rules are up-to-date.\nAn <strong>incremental build<\/strong> occurs when the cache is partially full but some outputs are outdated and need to be rebuilt.\nDeleting the cache is called <strong>cleaning<\/strong>.<\/p>\n<p>A build is <strong>correct<\/strong> or <strong>sound<\/strong> if all possible incremental builds have the same result as a full build.<sup class=\"footnote-reference\" id=\"fr-1-1\"><a href=\"#fn-1\">3<\/a><\/sup>\nA build is <strong>minimal<\/strong> (occasionally <strong>optimal<\/strong>) if rules are rerun at most once per build, and only run if necessary for soundness (<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2018\/03\/build-systems.pdf\">Build Systems \u00e0 la Carte<\/a>, <a href=\"https:\/\/www.mathematik.uni-marburg.de\/~seba\/publications\/pluto-incremental-build.pdf\">Pluto<\/a>).<\/p>\n<p>In order for a build to be sound, all possible cache invalidations must be <strong>tracked<\/strong> as dependencies.<\/p>\n<p>A build system without caching is called a <strong>task runner<\/strong> or <strong>batch compiler<\/strong>.\nNote that task runners still often support dependencies even if they don't support caching.\nBuild systems with caching can emulate a task runner by only defining tasks with zero outputs, but they are usually not designed for this use case. <sup class=\"footnote-reference\" id=\"fr-7-1\"><a href=\"#fn-7\">4<\/a><\/sup><\/p>\n<p>Some examples of build systems: <code>make<\/code>, <code>docker build<\/code>, rustc.\nSome examples of task runners: <a href=\"https:\/\/just.systems\/man\/en\/\"><code>just<\/code><\/a>, shell scripts, <a href=\"https:\/\/fabiensanglard.net\/dc\/driver.php\">gcc<\/a>.<\/p>\n<h2 id=\"specifying-dependencies\">specifying dependencies<a class=\"zola-anchor\" href=\"#specifying-dependencies\" aria-label=\"Anchor link for: specifying-dependencies\"><\/a>\n<\/h2>\n<p>A build can be either <strong>inter-process<\/strong>, in which case the task is usually a single <strong>process execution<\/strong> and its input and output files, or <strong>intra-process<\/strong>, in which case a task is usually a single function call and its arguments and return values.<\/p>\n<p>In order to track dependencies, either all inputs and outputs must be <strong>declared<\/strong> in source code ahead of time, or it must be possible to <strong>infer<\/strong> them from the execution of a task.<\/p>\n<p>Build systems that track changes to a rule definition are called <strong>self-tracking<\/strong>. Past versions of the rule are called its <strong>history<\/strong> (<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2018\/03\/build-systems.pdf\">Build Systems \u00e0 la Carte<\/a>).<\/p>\n<p>The act of inferring dependencies from runtime behavior is called <strong>tracing<\/strong>.\nIf a traced rule depends on a dependency that hasn\u2019t been built yet, the build system may either error, <strong>suspend<\/strong> the task and <strong>resume<\/strong> it later once the dependency is built, or <strong>abort<\/strong> the task and <strong>restart<\/strong> it later once the dependency is built (<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2018\/03\/build-systems.pdf\">Build Systems \u00e0 la Carte<\/a>).<\/p>\n<p>Inter-process builds often declare their inputs and outputs, and intra-process builds often infer them, but this is not inherent to the definition. <sup class=\"footnote-reference\" id=\"fr-8-1\"><a href=\"#fn-8\">5<\/a><\/sup><\/p>\n<p>Some example of intra-process builds include spreadsheets, the <a href=\"https:\/\/davidlattimore.github.io\/posts\/2024\/11\/19\/designing-wilds-incremental-linking.html\">wild<\/a> linker, and memoization libraries such as python\u2019s <a href=\"https:\/\/docs.python.org\/3\/library\/functools.html#functools.cache\"><code>functools.cache<\/code><\/a>.<\/p>\n<h3 id=\"applicative-and-monadic-structure\">applicative and monadic structure<a class=\"zola-anchor\" href=\"#applicative-and-monadic-structure\" aria-label=\"Anchor link for: applicative-and-monadic-structure\"><\/a>\n<\/h3>\n<p>A build graph is <strong>applicative<\/strong> if all inputs, outputs, and rules are declared ahead of time.\nWe say in this case the graph is <strong>statically known<\/strong>.\nVery few build systems are purely applicative, almost all have an escape hatch.<\/p>\n<p>The graph is <strong>monadic<\/strong> if not all outputs are known ahead of time, or if rules can generate other rules dynamically at runtime. Inputs that aren\u2019t known ahead of time are called <strong>dynamic dependencies<\/strong>. Dynamic dependencies are weaker than a fully monadic build system, in the sense that they can <a href=\"https:\/\/buttondown.com\/hillelwayne\/archive\/the-capability-tractability-tradeoff\/\">express<\/a> fewer build graphs. <sup class=\"footnote-reference\" id=\"fr-15-1\"><a href=\"#fn-15\">6<\/a><\/sup><\/p>\n<p>Build systems that do not require declaring build rules are always monadic.<\/p>\n<p>Some examples of monadic build systems include <a href=\"https:\/\/shakebuild.com\/\">Shake<\/a>, ninja <a href=\"https:\/\/ninja-build.org\/manual.html#ref_dyndep\"><code>dyndeps<\/code><\/a>, and Cargo build scripts.<\/p>\n<p>Some examples of applicative build systems include <code>make<\/code> (with <a href=\"https:\/\/accu.org\/journals\/overload\/14\/71\/miller_2004\/\">recursive make<\/a> and <a href=\"https:\/\/www.gnu.org\/software\/make\/manual\/html_node\/Remaking-Makefiles.html\">self-rebuilding Makefiles<\/a> disallowed), Bazel (excluding native rules), and map\/reduce libraries with memoization, such as <a href=\"https:\/\/www.unison-lang.org\/articles\/distributed-datasets\/incremental-evaluation\/\">this unison program<\/a>.<\/p>\n<h3 id=\"early-cutoff\">early cutoff<a class=\"zola-anchor\" href=\"#early-cutoff\" aria-label=\"Anchor link for: early-cutoff\"><\/a>\n<\/h3>\n<p>If a dirty rule R has an outdated output, reruns, and creates a new output that matches the old one, the build system has an opportunity to avoid running later rules that depend on R.\nTaking advantage of that opportunity is called <strong>early cutoff<\/strong>.<\/p>\n<p>See <a href=\"https:\/\/rustc-dev-guide.rust-lang.org\/queries\/incremental-compilation-in-detail.html\">the rustc-dev-guide<\/a> for much more information about early cutoff. <sup class=\"footnote-reference\" id=\"fr-9-1\"><a href=\"#fn-9\">7<\/a><\/sup><\/p>\n<h3 id=\"rebuild-detection\">rebuild detection<a class=\"zola-anchor\" href=\"#rebuild-detection\" aria-label=\"Anchor link for: rebuild-detection\"><\/a>\n<\/h3>\n<p>In unsound build systems, it\u2019s possible that the build system does not accurately <strong>detect<\/strong> that it needs to rebuild.\nSuch systems sometimes offer a way to <strong>force-rerun<\/strong> a target: keeping the existing cache, but rerunning a single rule.\nFor inter-process build systems, this often involves <code>touch<\/code>ing a file to set its modification date to the current time.<\/p>\n<h2 id=\"the-executor\">the executor<a class=\"zola-anchor\" href=\"#the-executor\" aria-label=\"Anchor link for: the-executor\"><\/a>\n<\/h2>\n<p>A <strong>build executor<\/strong> runs tasks and is responsible for <strong>scheduling<\/strong> tasks in an order that respects all dependencies, often using heuristics such as dependency depth or the time taken by the task on the last run.\nThey also detect whether rule inputs have been modified, making the rule outdated; this is called <strong>rebuild detection<\/strong>.\nThe build executor is responsible for restarting or suspending tasks in build systems that support it.<\/p>\n<p>Executors usually schedule many tasks in parallel, but this is not inherent to the definition.<\/p>\n<p>Executors often provide <strong>progress reporting<\/strong>, and sometimes allow <strong>querying<\/strong> the dependency graph.\nOccasionally they trace the inputs used by the task to enforce they match the declared dependencies, or to automatically add them to an internal dependency graph.<\/p>\n<h2 id=\"inter-process-builds\">inter-process builds<a class=\"zola-anchor\" href=\"#inter-process-builds\" aria-label=\"Anchor link for: inter-process-builds\"><\/a>\n<\/h2>\n<p>In the context of inter-process builds, an <strong>artifact<\/strong> is an output file generated by a rule.<sup class=\"footnote-reference\" id=\"fr-6-1\"><a href=\"#fn-6\">8<\/a><\/sup>\nA <strong>source file<\/strong> is an input file that is specific to the current <strong>project<\/strong><sup class=\"footnote-reference\" id=\"fr-11-1\"><a href=\"#fn-11\">9<\/a><\/sup> (sometimes <strong>repository<\/strong> or <strong>workspace<\/strong>) as opposed to a <strong>system dependency<\/strong> that is reused across multiple projects.\nA project is loosely defined but generally refers to the set of all input and output files that the build system knows about, usually contained in a single directory.\nSource files can be <strong>generated<\/strong>, which means they are an output of a previous rule.<\/p>\n<p><strong>Build files<\/strong> contain rule definitions, including (but not limited to) task definitions, input and output declarations, and metadata such as a human-readable description of the rule.\nInputs are usually split into <strong>explicit inputs<\/strong> passed to the spawned process, <strong>implicit inputs<\/strong> that are tracked by the build system but not used in the task definition, and <strong>order-only inputs<\/strong> that must exist before the rule can execute, but do not invalidate the cache when modified.<\/p>\n<p>Process executions have more inputs than just files, such as the rule itself, environment variables, the current time, the current working directory, and occasionally network services or local daemons <sup class=\"footnote-reference\" id=\"fr-2-1\"><a href=\"#fn-2\">10<\/a><\/sup>.<\/p>\n<p>The set of all inputs that are not source files or command line arguments is called the <strong>environment<\/strong>.\nProcesses can be <strong>sandboxed<\/strong> to prevent them from depending on the network, a daemon, or occasionally system dependencies; this is sometimes called a <strong>sandboxed environment<\/strong> or <strong>isolated environment<\/strong>.<\/p>\n<p>System dependencies are more expansive than I think they are often understood to be.\nThey include compilers, linkers, programming language libraries <sup class=\"footnote-reference\" id=\"fr-3-1\"><a href=\"#fn-3\">11<\/a><\/sup>, and <a href=\"https:\/\/jyn.dev\/build-system-tradeoffs\/#dynamic-linking-and-platform-maintainers\">static and dynamically linked object files<\/a>, but also the dynamic loader, language runtime, and various system configuration files.\nThe subset of these dependencies needed for building a minimal program in a given language, along with various tools for inspecting and modifying the outputs at runtime, are called a <strong>toolchain<\/strong>.\nToolchains are inherently specific to a given language, but sometimes (e.g. in GCC) a single compiler will support multiple languages as inputs.<\/p>\n<p>A build is <strong>hermetic<\/strong> (rarely, <strong>self-contained<\/strong> or <strong>isolated<\/strong> <sup class=\"footnote-reference\" id=\"fr-16-1\"><a href=\"#fn-16\">12<\/a><\/sup>) if it uses no system dependencies and instead defines all its dependencies in the project (<a href=\"https:\/\/bazel.build\/basics\/hermeticity\">Bazel<\/a>).\nSandboxing and hermeticity are orthogonal axes; neither one implies the other.\nFor example, docker builds are sandboxed but not hermetic, and nix shells are hermetic but not sandboxed.<\/p>\n<p>Compiler or linkers sometimes have their own <strong>incremental caches<\/strong>.\nReusing the cache requires you to <strong>trust<\/strong> the compiler to be sound when incrementally rebuilding.\nThis is usually implicit, but hermetic or sandboxed builds require an opt-in to reuse the cache.\nBazel calls this kind of reuse a <a href=\"https:\/\/bazel.build\/versions\/7.0.0\/remote\/persistent\"><strong>persistent worker<\/strong><\/a>.<\/p>\n<h3 id=\"determinism\">determinism<a class=\"zola-anchor\" href=\"#determinism\" aria-label=\"Anchor link for: determinism\"><\/a>\n<\/h3>\n<p>A build is <strong>deterministic<\/strong> if it creates the same output every time in some specific environment.\nA build is <strong>reproducible<\/strong> if it is deterministic and also has the same output in <a href=\"https:\/\/reproducible-builds.org\/docs\/commandments\/\">any environment<\/a>, as long as the system dependencies remain the same.<\/p>\n<h3 id=\"remote-caching\">remote caching<a class=\"zola-anchor\" href=\"#remote-caching\" aria-label=\"Anchor link for: remote-caching\"><\/a>\n<\/h3>\n<p>Caching can be remote or local.\n<strong>Remote caching<\/strong> is almost always unsound unless the build is both hermetic and reproducible (i.e. its only environment dependencies are controlled by the build system).<\/p>\n<p>Downloading files from the remote cache is called <strong>materializing<\/strong> them.\nMost build systems with remote caching <strong>defer materialization<\/strong> as long as possible, since in large build graphs the cache is often too large to fit on disk.\nBuilds where the cache is never fully materialized are called <strong>shallow builds<\/strong> (<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2018\/03\/build-systems.pdf\">Build Systems \u00e0 la Carte<\/a>).<\/p>\n<p>Remote caching usually, but not necessarily, uses <strong>content addressed hashing<\/strong> in a <strong>key-value store<\/strong> to identify which artifact to download.<\/p>\n<p>Some example build systems that use remote caching: Bazel, Buck2, nix, <code>docker build<\/code>.<\/p>\n<h3 id=\"interface\">interface<a class=\"zola-anchor\" href=\"#interface\" aria-label=\"Anchor link for: interface\"><\/a>\n<\/h3>\n<p>Build systems usually have a way to run a subset of the build.\nThe identifier used to specify which part of the build you want to run is called a <strong>target<\/strong>.<sup class=\"footnote-reference\" id=\"fr-4-1\"><a href=\"#fn-4\">13<\/a><\/sup>\nTargets are usually the filenames of an artifact, but can also be abstract names of one or more rules.\nBazel-descended build systems call these names <a href=\"https:\/\/bazel.build\/concepts\/labels\"><strong>labels<\/strong><\/a>.\nMake-descended build systems call these <a href=\"https:\/\/www.gnu.org\/software\/make\/manual\/html_node\/Phony-Targets.html\"><strong>phony targets<\/strong><\/a>.\nSome build systems, such as cargo, do not use target identifiers but instead only have subcommands with arguments; the combination of arguments together specifies a set of targets.<\/p>\n<p>Some example targets:<\/p>\n<ul>\n<li><code>make all<\/code><\/li>\n<li><code>cargo build --test http_integration<\/code><\/li>\n<li><code>buck2 build :main<\/code><\/li>\n<\/ul>\n<h3 id=\"meta-build-systems\">meta-build systems<a class=\"zola-anchor\" href=\"#meta-build-systems\" aria-label=\"Anchor link for: meta-build-systems\"><\/a>\n<\/h3>\n<p>Inter-process build systems are often divided into a <strong>configuration<\/strong> step and a <strong>build<\/strong> step.\nA build system that only runs the configuration step, and requires another tool for the build step, is called a <strong>meta-build system<\/strong>.<\/p>\n<p>Usually this meta-build system <strong>discovers<\/strong> the rules that need to be executed (often through file globbing or some other programmatic way to describe dependencies), then <strong>serializes<\/strong> these rules into an <a href=\"\/i-want-a-better-action-graph-serialization\/\"><strong>action graph<\/strong><\/a>, which can be stored either in-memory or on-disk. On-disk serialized action graphs are usually themselves build files, in the sense that you can write them by hand but you wouldn't want to.<\/p>\n<p>Configuration steps usually allow the developer to choose a set of <strong>configuration flags<\/strong> (occasionally, <strong>build flags<\/strong>) that affect the generated rules.<\/p>\n<p>Some build systems also integrate directly with the <a href=\"https:\/\/nesbitt.io\/2025\/12\/02\/what-is-a-package-manager.html\"><strong>package manager<\/strong><\/a>, but this is uncommon, and usually the build system expects all packages to be pre-downloaded into a known location.<\/p>\n<p>Some examples of meta-build systems are CMake, meson, and autotools.<\/p>\n<h3 id=\"vfs\">VFS<a class=\"zola-anchor\" href=\"#vfs\" aria-label=\"Anchor link for: vfs\"><\/a>\n<\/h3>\n<p>Advanced build systems can integrate with a <strong>virtual file system<\/strong> (VFS) to check-out source control files on-demand, rather than eagerly (<a href=\"https:\/\/github.com\/facebook\/sapling?tab=readme-ov-file#ejdenfs\">EdenFS<\/a>).\nA VFS can also persistently store <strong>content hashes<\/strong> and provide efficient <strong>change detection<\/strong> for files, avoiding the need for file watching or constant re-hashing.<\/p>\n<!--\n### orchestration\nIt is common, especially in build systems that have [grown organically](https:\/\/nothingisnttrivial.com\/vines.html), for build systems to wrap multiple other build systems. We call such a system, especially in the context of **continuous integration**, a **build orchestrator**.[^17] Usually orchestrators will have only a few \u201cblessed\u201d workflows, and not all features of the underlying tool will be exposed.\n\nSome examples: [Rust\u2019s bootstrap build system](https:\/\/rustc-dev-guide.rust-lang.org\/building\/bootstrapping\/intro.html), Firefox\u2019s [mach](https:\/\/firefox-source-docs.mozilla.org\/mach\/usage.html#)\n-->\n<h2 id=\"intra-process-builds\">intra-process builds<a class=\"zola-anchor\" href=\"#intra-process-builds\" aria-label=\"Anchor link for: intra-process-builds\"><\/a>\n<\/h2>\n<p>The equivalent of system dependencies within a process is <strong>non-local state<\/strong>, including environment variables, globals, thread-locals, and class member fields (for languages where <code>this<\/code> is passed implicitly).\nEspecially tricky are function calls that do <strong>inter-process communication<\/strong> (IPC), which are basically never sound to cache.\nTracing intra-process builds is very very hard since it\u2019s easy to call a function that depends on global state without you knowing. <sup class=\"footnote-reference\" id=\"fr-19-1\"><a href=\"#fn-19\">14<\/a><\/sup><\/p>\n<p>In this intra-process context, most object stores are <strong>in-memory caches<\/strong>. A build system that supports saving (<strong>persisting<\/strong>) the cache to disk is said to have <a href=\"\/complected-and-orthogonal-persistence\/\"><strong>persistence<\/strong><\/a>. The system for persisting the cache is sometimes called a <strong>database<\/strong>, even if it is not a general-purpose database in the sense the term is normally used (<a href=\"https:\/\/salsa-rs.netlify.app\/tutorial\/db\">Salsa<\/a>).<\/p>\n<h3 id=\"tracing\">tracing<a class=\"zola-anchor\" href=\"#tracing\" aria-label=\"Anchor link for: tracing\"><\/a>\n<\/h3>\n<p>Tracing intra-process build systems are sometimes called a <strong>query system<\/strong>. <sup class=\"footnote-reference\" id=\"fr-18-1\"><a href=\"#fn-18\">15<\/a><\/sup> They work similarly to their inter-process equivalents: the interface looks like normal function calls, and the build system tracks which functions call which other functions, so it knows which to rerun later.<\/p>\n<p>Some examples of tools with tracing intra-process build systems: <a href=\"https:\/\/docs.rs\/salsa\/latest\/salsa\/\">salsa<\/a>, the <a href=\"https:\/\/rustc-dev-guide.rust-lang.org\/query.html\">rustc query system<\/a>.<\/p>\n<h3 id=\"frp\">FRP<a class=\"zola-anchor\" href=\"#frp\" aria-label=\"Anchor link for: frp\"><\/a>\n<\/h3>\n<p>Intra-process build systems that allow you to explicitly declare dependencies usually come from the background of <a href=\"https:\/\/blog.janestreet.com\/breaking-down-frp\/\"><strong>functional reactive programming<\/strong><\/a> (FRP). FRP is most often used in UI and frontend design, but many of the ideas are the same as the build systems used for compiling programs.<\/p>\n<p>Unlike any of the build systems we've talked about so far, FRP libraries let you look at <em>past versions<\/em> of your outputs, which is sometimes called <strong>remembering<\/strong> state (<a href=\"https:\/\/react.dev\/learn\/state-a-components-memory\">React<\/a>). To make this easier to reason about, rules can be written as <strong>event handlers<\/strong>.<\/p>\n<p>Some examples of libraries with dependency declarations: <a href=\"https:\/\/react.dev\/learn\/adding-interactivity\">React<\/a>.<\/p>\n<h2 id=\"so-what-counts-as-a-build-system\">so, what counts as a build system?<a class=\"zola-anchor\" href=\"#so-what-counts-as-a-build-system\" aria-label=\"Anchor link for: so-what-counts-as-a-build-system\"><\/a>\n<\/h2>\n<p>A build system is pretty much anything that lets you specify dependencies on a previous artifact \ud83d\ude04 Some more weird examples of build systems:<\/p>\n<ul>\n<li>Github Actions (jobs and workflows)<\/li>\n<li>Static site generators<\/li>\n<li>Docker-compose files<\/li>\n<li>Systemd unit files<\/li>\n<li>Excel<\/li>\n<\/ul>\n<p>Hopefully this post has given you both a vocabulary to talk about build systems and a context to compare them!<\/p>\n<h2 id=\"bibliography\">bibliography<a class=\"zola-anchor\" href=\"#bibliography\" aria-label=\"Anchor link for: bibliography\"><\/a>\n<\/h2>\n<ul>\n<li><a href=\"https:\/\/nesbitt.io\/2025\/12\/02\/what-is-a-package-manager.html\">Andrew Nesbitt, \u201cWhat is a Package Manager?\u201d<\/a><\/li>\n<li><a href=\"https:\/\/jyn.dev\/build-system-tradeoffs\/\">jyn, \u201cbuild system tradeoffs\u201d<\/a><\/li>\n<li><a href=\"https:\/\/jade.fyi\/blog\/the-postmodern-build-system\/\">Jade Lovelace, \u201cThe postmodern build system\u201d<\/a><\/li>\n<li><a href=\"https:\/\/just.systems\/man\/en\/\">Casey Rodarmor, \u201cJust Programmer's Manual\u201d<\/a><\/li>\n<li><a href=\"https:\/\/fabiensanglard.net\/dc\/driver.php\">Fabien Sanglard, \u201cDriving Compilers\u201d<\/a><\/li>\n<li><a href=\"https:\/\/rustc-dev-guide.rust-lang.org\/queries\/incremental-compilation-in-detail.html\">The Rust Project Contributors, \u201cIncremental compilation in detail\u201d<\/a><\/li>\n<li><a href=\"https:\/\/rustc-dev-guide.rust-lang.org\/query.html\">The Rust Project Contributors, \u201cQueries: demand-driven compilation\u201d<\/a><\/li>\n<li><a href=\"https:\/\/docs.python.org\/3\/library\/functools.html\">\u201cfunctools \u2014 Higher-order functions and operations on callable objects \u2014 Python 3.14.2 documentation\u201d<\/a><\/li>\n<li><a href=\"https:\/\/buttondown.com\/hillelwayne\/archive\/the-capability-tractability-tradeoff\/\">Hillel Wayne, \u201cThe Capability-Tractability Tradeoff\u201d<\/a><\/li>\n<li><a href=\"https:\/\/shakebuild.com\/\">Neil Mitchell, \u201cShake Build System\u201d<\/a><\/li>\n<li><a href=\"https:\/\/ninja-build.org\/manual.html\">\u201cThe Ninja build system\u201d<\/a><\/li>\n<li><a href=\"https:\/\/accu.org\/journals\/overload\/14\/71\/miller_2004\/\">Peter Miller, \u201cRecursive Make Considered Harmful\u201d<\/a><\/li>\n<li><a href=\"https:\/\/www.unison-lang.org\/articles\/distributed-datasets\/incremental-evaluation\/\">Rebecca Mark and Paul Chiusano,  \u201cIncremental evaluation via memoization \u00b7 Unison programming language\u201d<\/a><\/li>\n<li><a href=\"https:\/\/bazel.build\/basics\/hermeticity\">\u201cHermeticity \u00a0|\u00a0 Bazel\u201d<\/a><\/li>\n<li><a href=\"https:\/\/bazel.build\/versions\/7.0.0\/remote\/persistent\">\u201cPersistent Workers \u00a0|\u00a0 Bazel\u201d<\/a><\/li>\n<li><a href=\"https:\/\/bazel.build\/concepts\/labels\">\u201cLabels \u00a0|\u00a0 Bazel\u201d<\/a><\/li>\n<li><a href=\"https:\/\/reproducible-builds.org\/docs\/commandments\/\">\u201cCommandments of reproducible builds\u201d<\/a><\/li>\n<li><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2018\/03\/build-systems.pdf\">Mokhov et. al., Build Systems \u00e0 la Carte<\/a><\/li>\n<li><a href=\"https:\/\/www.gnu.org\/software\/make\/manual\/html_node\/Phony-Targets.html\">\u201cPhony Targets (GNU make)\u201d<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/facebook\/sapling?tab=readme-ov-file\">Facebook, \u201cSapling: A Scalable, User-Friendly Source Control System.\u201d<\/a><\/li>\n<li><a href=\"https:\/\/www.mathematik.uni-marburg.de\/~seba\/publications\/pluto-incremental-build.pdf\">Erdweg et. al., \"A Sound and Optimal Incremental Build System with Dynamic Dependencies\"<\/a><\/li>\n<li><a href=\"https:\/\/davidlattimore.github.io\/posts\/2024\/11\/19\/designing-wilds-incremental-linking.html\">David Lattimore, \u201cDesigning Wild's incremental linking\u201d<\/a><\/li>\n<li><a href=\"https:\/\/lord.io\/spreadsheets\/\">Bo Lord, \u201cHow to Recalculate a Spreadsheet\u201d<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/ninja-build\/ninja\/blob\/4b72b15aac766afe6a9a2c0dad535b0c2035a550\/src\/depfile_parser.in.cc\">\u201cninja\u2014<code>depfile_parser<\/code>\u201d<\/a><\/li>\n<li><a href=\"https:\/\/salsa-rs.netlify.app\/\">\u201csalsa - A generic framework for on-demand, incrementalized computation\u201d<\/a><\/li>\n<li><a href=\"https:\/\/salsa-rs.netlify.app\/tutorial\/db\">\u201cDefining the database struct - Salsa\u201d<\/a><\/li>\n<li><a href=\"https:\/\/blog.rust-lang.org\/2021\/05\/10\/Rust-1.52.1\/\">Felix Klock and Mark Rousskov on behalf of the Rust compiler team, \u201cAnnouncing Rust 1.52.1\u201d<\/a><\/li>\n<li><a href=\"https:\/\/blog.janestreet.com\/breaking-down-frp\/\">Yaron Minsky, \u201cJane Street Blog - Breaking down FRP \u201d<\/a><\/li>\n<li><a href=\"https:\/\/react.dev\/learn\/adding-interactivity\">\u201cAdding Interactivity \u2013 React\u201d<\/a><\/li>\n<li><a href=\"https:\/\/react.dev\/learn\/state-a-components-memory\">\u201cState: A Component's Memory \u2013 React\u201d<\/a><\/li>\n<\/ul>\n<section class=\"footnotes\">\n<ol class=\"footnotes-list\">\n<li id=\"fn-14\">\n<p>Nearly all build systems are inconsistent about whether a rule refers to an <em>abstract description<\/em> of how to build an output (i.e., can be reused for multiple sets of inputs and outputs), or a <em>concrete instantiation<\/em> of that description for a specific set of inputs and outputs. We have to live with the ambiguity, unfortunately. <a href=\"#fr-14-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-5\">\n<p>Weird things can happen here though; for example early cutoff can allow circular dependencies. This sometimes comes up for generated build.ninja files. <a href=\"#fr-5-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-1\">\n<p>The <a href=\"https:\/\/www.mathematik.uni-marburg.de\/~seba\/publications\/pluto-incremental-build.pdf\">pluto paper<\/a> defines this as \u201cafter a build, generated files consistently reflect the latest source files\u201d. Neither my definition nor pluto's definition are particularly well-defined if the build is non-deterministic. Defining this formally would probably require constructing an isomorphism between all programs with the same runtime behavior; but \u201cruntime behavior\u201d is not well-defined for a general-purpose build system that can output artifacts that are not programs. <a href=\"#fr-1-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-7\">\n<p>As we'll see later, the reverse is also true: a common design for build systems is to automatically inject cache points into an existing task runner, or to design the rule file to look as similar to a shell script or function call as possible. <a href=\"#fr-7-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-8\">\n<p>In particular, nearly all modern inter-process build systems have a limited form of tracing where they ask the compiler to generate \"dep-info\" files <sup class=\"footnote-reference\" id=\"fr-13-1\"><a href=\"#fn-13\">16<\/a><\/sup> that show which files were used (usually through imports) by a given source file. Note that this dep-info is not available until after the first time a build has run, and that this only works if the compiler supports it. <a href=\"#fr-8-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-15\">\n<p>For more information about the spectrum of designs between applicative and monadic, see <a href=\"https:\/\/jade.fyi\/blog\/the-postmodern-build-system\/\">the post-modern build system<\/a>. <a href=\"#fr-15-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-9\">\n<p>Note that the dev-guide assumes that tasks are expensive relative to the cost of constructing the graph. This is true in the context of rustc, where LLVM codegen <sup class=\"footnote-reference\" id=\"fr-10-1\"><a href=\"#fn-10\">17<\/a><\/sup> normally dominates compilation time, but it isn't true for e.g. <a href=\"https:\/\/lord.io\/spreadsheets\/\">spreadsheets<\/a>. <a href=\"#fr-9-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-6\">\n<p>It's possible for tasks to create files that aren't tracked by the build system, but these aren't called artifacts. I don't know a good word for these; \"byproducts\" is the closest but some build systems use that to mean <em>any<\/em> intermediate artifacts. <a href=\"#fr-6-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-11\">\n<p>I'm not super happy with this definition because it conflicts with how compilers use the term, but I do think it describes how most build systems think about files. <a href=\"#fr-11-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-2\">\n<p>Poorly written rules can also depend on which other rules are executing at the same time, which is called a <strong>race condition<\/strong>. Note this does not require the rule to be unsound, only for it to use intermediate files the build system doesn\u2019t know about. <a href=\"#fr-2-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-3\">\n<p>for C, header files; for other languages, usually source files or intermediate representations. <a href=\"#fr-3-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-16\">\n<p>Yes, this overlaps with the term for sandboxing. Try to avoid the word \"isolated\" if possible. <a href=\"#fr-16-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-4\">\n<p>This has no relation to a <strong>target platfom<\/strong>, which is related to cross-compiling. I wish we had better names for these things. <a href=\"#fr-4-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-19\">\n<p>I would actually describe this as much harder than tracing an inter-process build system, since <a href=\"\/i-want-a-better-build-executor\/#environment-variables\">there aren't very good systems for tracking memory access<\/a>. See <a href=\"https:\/\/blog.rust-lang.org\/2021\/05\/10\/Rust-1.52.1\/\">this post about unstable fingerprints<\/a> for an idea of what bugs this causes in practice. <a href=\"#fr-19-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-18\">\n<p>This actually has very strong analogies to the way \"query\" is used in a database context: just like a tracing query system, a database has to be able to restart a query's transaction if the data it's trying to access has been modified. <a href=\"#fr-18-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-13\">\n<p>What is a dep-info file? Good question! It's a makefile. It's literally a makefile. Don't you just love <a href=\"https:\/\/github.com\/ninja-build\/ninja\/blob\/4b72b15aac766afe6a9a2c0dad535b0c2035a550\/src\/depfile_parser.in.cc#L27-L47\">proving backslashes by induction<\/a>? <a href=\"#fr-13-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-10\">\n<p>Or, more rarely, type-checking, borrow-checking, or coherence checking. <a href=\"#fr-10-1\">\u21a9<\/a><\/p>\n<\/li>\n<\/ol>\n<\/section>\n"},{"title":"I want a better build executor","published":"2025-12-05T00:00:00+00:00","updated":"2025-12-05T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/i-want-a-better-build-executor\/"}},"id":"https:\/\/jyn.dev\/i-want-a-better-build-executor\/","summary":"I want a way to gradually transition existing builds to be hermetic.","content":"<p>This post is part 4\/4 of <a href=\"\/four-posts-about-build-systems\/\">a series about build systems<\/a>.<\/p>\n<hr \/>\n<figure>\n<blockquote cite=\"https:\/\/steveklabnik.com\/writing\/i-see-a-future-in-jj\/\">\n<p>The market fit is interesting. Git has clearly won, it has all of the mindshare, but since you can use jj to work on Git repositories, it can be adopted incrementally. This is, in my opinion, the only viable way to introduce a new VCS: it has to be able to be partially adopted.<\/p>\n<\/blockquote>\n<figcaption>\n<cite ><a href=\"https:\/\/steveklabnik.com\/writing\/i-see-a-future-in-jj\/\">Steve Klabnik<\/a><\/cite>\n<\/figcaption>\n<\/figure>\n<figure>\n<blockquote cite=\"https:\/\/www.youtube.com\/watch?v=72y2EC5fkcE\">\n<p>If you've worked with other determinism-based systems, one thing they have in common is they feel really fragile, and you have to be careful that you don't do something that breaks the determinism. But in our case, since we've created every level of the stack to support this, we can offload the determinism to the development environment and you can basically write whatever code you want without having to worry about whether it's going to break something.<\/p>\n<\/blockquote>\n<figcaption>\n<cite ><a href=\"https:\/\/www.youtube.com\/watch?v=72y2EC5fkcE\">Allan Blomquist<\/a><\/cite>\n<\/figcaption>\n<\/figure>\n<p>In <a href=\"\/i-want-a-better-action-graph-serialization\/\">my last post<\/a>, I describe an improved build graph serialization. In this post, I describe the build executor that reads those files.<\/p>\n<h2 id=\"what-is-a-build-executor\">what is a build executor?<a class=\"zola-anchor\" href=\"#what-is-a-build-executor\" aria-label=\"Anchor link for: what-is-a-build-executor\"><\/a>\n<\/h2>\n<p>Generally, there are three stages to a build:<\/p>\n<ol>\n<li>Resolving and downloading dependencies. The tool that does this is called a <a href=\"https:\/\/nesbitt.io\/2025\/12\/02\/what-is-a-package-manager.html\"><strong>package manager<\/strong><\/a>. Common examples are <code>npm<\/code>, <code>pip<\/code>, <a href=\"https:\/\/docs.conan.io\/2\/index.html\">Conan<\/a><sup class=\"footnote-reference\" id=\"fr-1-1\"><a href=\"#fn-1\">1<\/a><\/sup>, and the <a href=\"https:\/\/doc.rust-lang.org\/cargo\/reference\/resolver.html\"><code>cargo<\/code> resolver<\/a>.<\/li>\n<li>Configuring the build based on the host environment and build targets. I am not aware of any common name for this, other than maybe <strong>configure script<\/strong> (but there exist many tools for this that are not just shell scripts). Common examples are CMake, Meson, autotools, and the Cargo CLI interface (e.g. <code>--feature<\/code> and <code>--target<\/code>).<\/li>\n<li>Executing a bunch of processes and reporting on their progress. The tool that does this is called a <strong>build executor<\/strong>. Common examples are <code>make<\/code>, <code>ninja<\/code>,  <code>docker build<\/code>, and the <code>Compiling<\/code> phase of <code>cargo build<\/code>.<\/li>\n<\/ol>\n<p>There are a lot more things an executor can do than just spawning processes and showing a progress report!\nThis post explores what those are and sketches a design for a tool that could improve on current executors.<\/p>\n<h2 id=\"change-detection\">change detection<a class=\"zola-anchor\" href=\"#change-detection\" aria-label=\"Anchor link for: change-detection\"><\/a>\n<\/h2>\n<p>Ninja depends on <a href=\"https:\/\/apenwarr.ca\/log\/20181113\">mtimes, which have many issues<\/a>. Ideally, it would take notes from <a href=\"https:\/\/apenwarr.ca\/log\/20181113#:~:text=redo:%20mtime%20dependencies\"><code>redo<\/code><\/a> and look at file attributes, not just the mtime, which eliminates many more false positives.<\/p>\n<h2 id=\"querying\">querying<a class=\"zola-anchor\" href=\"#querying\" aria-label=\"Anchor link for: querying\"><\/a>\n<\/h2>\n<p>I wrote earlier about <a href=\"\/build-system-tradeoffs\/#reflection\">querying the build graph<\/a>.\nThere are two kinds of things you can query: The configuration graph (what bazel calls the <a href=\"https:\/\/bazel.build\/query\/language\">target graph<\/a>), which shows dependencies between \"human meaningful\" packages; and the <a href=\"https:\/\/bazel.build\/query\/aquery\">action graph<\/a>, which shows dependencies between files.<\/p>\n<p>Queries on the action graph live in the executor; queries on the configuration graph live in the configure script. For example, <code>cargo metadata<\/code>\/<code>cargo tree<\/code>, <code>bazel query<\/code>, and <code>cmake --graphiz<\/code> query the configuration graph; <code>ninja -t inputs<\/code> and <code>bazel aquery<\/code> query the action graph. Cargo has no stable way to query the action graph.<\/p>\n<p>Note that \u201cquerying the graph\u201d is not a binary yes\/no. Ninja's query language is much more restricted than Bazel's. Compare Ninja's syntax for querying \u201cthe command line for all C++ files used to build the target <code>\/\/:hello_world<\/code>\u201d <sup class=\"footnote-reference\" id=\"fr-3-1\"><a href=\"#fn-3\">2<\/a><\/sup>:<\/p>\n<pre data-lang=\"console\" class=\"language-console z-code\"><code class=\"language-console\" data-lang=\"console\"><span class=\"z-source z-shell z-console\"><span class=\"z-keyword z-operator z-assignment z-redirection z-process z-shell\">$<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">ninja<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> -<\/span>t<\/span> inputs hello_world<\/span> <span class=\"z-keyword z-operator z-logical z-pipe z-shell\">|<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">grep<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> <span class=\"z-string z-quoted z-single z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&#39;<\/span>\\.c++$<span class=\"z-punctuation z-definition z-string z-end z-shell\">&#39;<\/span><\/span><\/span> <span class=\"z-keyword z-operator z-logical z-pipe z-shell\">|<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">xargs<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> ninja<span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> -<\/span>t<\/span> targets<\/span> <span class=\"z-keyword z-operator z-logical z-pipe z-shell\">|<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">cut<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> -<\/span>d<\/span> :<span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> -<\/span>f<\/span> 1<\/span> <span class=\"z-keyword z-operator z-logical z-pipe z-shell\">|<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">xargs<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> ninja<span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> -<\/span>t<\/span> commands<\/span>\n<\/span><span class=\"z-source z-shell z-console\">g++ -c -o my_lib.o my_lib.cpp\n<\/span><span class=\"z-source z-shell z-console\">g++ -o hello_world hello_world.cpp my_lib.o\n<\/span><\/code><\/pre>\n<p>to Bazel's:<\/p>\n<pre data-lang=\"console\" class=\"language-console z-code\"><code class=\"language-console\" data-lang=\"console\"><span class=\"z-source z-shell z-console\"><span class=\"z-keyword z-operator z-assignment z-redirection z-process z-shell\">$<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">bazel<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> aquery <span class=\"z-string z-quoted z-single z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&#39;<\/span>inputs(&quot;.*cpp&quot;, deps(\/\/:hello_world))<span class=\"z-punctuation z-definition z-string z-end z-shell\">&#39;<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-shell z-console\">action &#39;Compiling hello_world.cpp&#39;\n<\/span><span class=\"z-source z-shell z-console\">  Mnemonic: CppCompile\n<\/span><span class=\"z-source z-shell z-console\">  Target: \/\/:hello_world\n<\/span><span class=\"z-source z-shell z-console\">  Configuration: k8-fastbuild\n<\/span><span class=\"z-source z-shell z-console\">  Execution platform: @@platforms\/\/host:host\n<\/span><span class=\"z-source z-shell z-console\">  ActionKey: 155b2cdb875736efc8d218ea790d2ef9ce698f0b1b1700d58de3c135145b1d12\n<\/span><span class=\"z-source z-shell z-console\">  Inputs: [external\/rules_cc++cc_configure_extension+local_config_cc\/builtin_include_directory_paths, external\/rules_cc++cc_configure_extension+local_config_cc\/cc_wrapper.sh, external\/rules_cc++cc_configure_extension+local_config_cc\/deps_scanner_wrapper.sh, external\/rules_cc++cc_configure_extension+local_config_cc\/validate_static_library.sh, hello_world.cpp, my_lib.h]\n<\/span><span class=\"z-source z-shell z-console\">  Outputs: [bazel-out\/k8-fastbuild\/bin\/_objs\/hello_world\/hello_world.pic.d, bazel-out\/k8-fastbuild\/bin\/_objs\/hello_world\/hello_world.pic.o]\n<\/span><span class=\"z-source z-shell z-console\">  Command Line: (exec \/nix\/store\/vr15iyyykg9zai6fpgvhcgyw7gckl78w-gcc-wrapper-14.3.0\/bin\/gcc \\\n<\/span><\/code><\/pre>\n<details><summary>full command line<\/summary>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">    -U_FORTIFY_SOURCE \\\n<\/span><span class=\"z-text z-plain\">    -fstack-protector \\\n<\/span><span class=\"z-text z-plain\">    -Wall \\\n<\/span><span class=\"z-text z-plain\">    -Wunused-but-set-parameter \\\n<\/span><span class=\"z-text z-plain\">    -Wno-free-nonheap-object \\\n<\/span><span class=\"z-text z-plain\">    -fno-omit-frame-pointer \\\n<\/span><span class=\"z-text z-plain\">    &#39;-std=c++17&#39; \\\n<\/span><span class=\"z-text z-plain\">    -MD \\\n<\/span><span class=\"z-text z-plain\">    -MF \\\n<\/span><span class=\"z-text z-plain\">    bazel-out\/k8-fastbuild\/bin\/_objs\/hello_world\/hello_world.pic.d \\\n<\/span><span class=\"z-text z-plain\">    &#39;-frandom-seed=bazel-out\/k8-fastbuild\/bin\/_objs\/hello_world\/hello_world.pic.o&#39; \\\n<\/span><span class=\"z-text z-plain\">    -fPIC \\\n<\/span><span class=\"z-text z-plain\">    -iquote \\\n<\/span><span class=\"z-text z-plain\">    . \\\n<\/span><span class=\"z-text z-plain\">    -iquote \\\n<\/span><span class=\"z-text z-plain\">    bazel-out\/k8-fastbuild\/bin \\\n<\/span><span class=\"z-text z-plain\">    -iquote \\\n<\/span><span class=\"z-text z-plain\">    external\/rules_cc+ \\\n<\/span><span class=\"z-text z-plain\">    -iquote \\\n<\/span><span class=\"z-text z-plain\">    bazel-out\/k8-fastbuild\/bin\/external\/rules_cc+ \\\n<\/span><span class=\"z-text z-plain\">    -iquote \\\n<\/span><span class=\"z-text z-plain\">    external\/bazel_tools \\\n<\/span><span class=\"z-text z-plain\">    -iquote \\\n<\/span><span class=\"z-text z-plain\">    bazel-out\/k8-fastbuild\/bin\/external\/bazel_tools \\\n<\/span><span class=\"z-text z-plain\">    -c \\\n<\/span><span class=\"z-text z-plain\">    hello_world.cpp \\\n<\/span><span class=\"z-text z-plain\">    -o \\\n<\/span><span class=\"z-text z-plain\">    bazel-out\/k8-fastbuild\/bin\/_objs\/hello_world\/hello_world.pic.o \\\n<\/span><span class=\"z-text z-plain\">    -fno-canonical-system-headers \\\n<\/span><span class=\"z-text z-plain\">    -Wno-builtin-macro-redefined \\\n<\/span><span class=\"z-text z-plain\">    &#39;-D__DATE__=&quot;redacted&quot;&#39; \\\n<\/span><span class=\"z-text z-plain\">    &#39;-D__TIMESTAMP__=&quot;redacted&quot;&#39; \\\n<\/span><span class=\"z-text z-plain\">    &#39;-D__TIME__=&quot;redacted&quot;&#39;)\n<\/span><\/code><\/pre>\n<\/details>\n<p>Bazel\u2019s language has graph operators, such as union, intersection, and filtering, that let you build up quite complex predicates. Ninja can only express one predicate at a time, with much more limited filtering\u2014but unlike Bazel, allows you to filter to individual parts of the action, like the command line invocation, without needing a full protobuf parser or trying to do text post-processing.<\/p>\n<p>I would like to see a query language that combines both these strengths: the same nested predicate structure of Bazel queries, but add a new <code>emit()<\/code> predicate that takes another predicate as an argument for complex output filtering:<\/p>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">emit(commands, inputs(&quot;.*cpp&quot;, deps(.\/src\/hello_world)))\n<\/span><\/code><\/pre>\n<p>We could even go so far as to give this a jq-like syntax:<\/p>\n<pre data-lang=\"jq\" class=\"language-jq z-code\"><code class=\"language-jq\" data-lang=\"jq\"><span class=\"z-text z-plain\">.\/src\/hello_world | deps | inputs &quot;*.c++&quot; | emit commands\n<\/span><\/code><\/pre>\n<p>For more complex predicates that have multiple sets as inputs, such as set union and intersection, we could introduce a <code>subquery<\/code> operator:<\/p>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">glob &quot;src\/**&quot; | except subquery(glob(&quot;src\/package\/**&quot;) | executable)\n<\/span><\/code><\/pre>\n<h2 id=\"tracing\">tracing<a class=\"zola-anchor\" href=\"#tracing\" aria-label=\"Anchor link for: tracing\"><\/a>\n<\/h2>\n<p>In <a href=\"\/build-system-tradeoffs\/\">my previous post<\/a>, I talked about two main uses for a tracing build system: first, to automatically add dependency edges for you; and second, to verify at runtime that no dependency edges are missing. This especially shines when the action graph has a way to express negative dependencies, because the tracing system <em>sees<\/em> every attempted file access and can add them to the graph automatically.<\/p>\n<p>For prior art, see the <a href=\"https:\/\/shakebuild.com\/\">Shake build system<\/a>. Shake is higher-level than an executor and doesn't work on an action graph, but it has <a href=\"https:\/\/neilmitchell.blogspot.com\/2020\/05\/file-tracing.html\">built-in support for file tracing<\/a> in all three of these modes: warning about incorrect edges; adding new edges to the graph when they're detected at runtime; and finally, <a href=\"https:\/\/blogs.ncl.ac.uk\/andreymokhov\/stroll\/\">fully inferring all edges from the nodes alone<\/a>.<\/p>\n<p>I would want my executor to only support linting and hard errors for missing edges. Inferring a full action graph is scary and IMO belongs in a higher-level tool, and adding dependency edges automatically can be done by a tool that wraps the executor and parses the lints.<\/p>\n<p>What's really cool about this linting system is that it allows you to gradually transition to a hermetic build over time, without frontloading all the work to when you switch to the tool.<\/p>\n<aside role=note class=note-container><div class=note-content>\n<p>The main downside of tracing is that it's highly non-portable, and in particular is very limited on macOS.<\/p>\n<p>One possible alternative I've thought of is to do a buck2-style unsandboxed hermetic builds, where you copy exactly the specified inputs into a tempdir and run the build from the tempdir. If that fails, rerun the build from the main source directory. This can't tell <em>which<\/em> dependency edges are missing, but it can tell you <em>a<\/em> dependency is missing without fully failing the build.<\/p>\n<p>The downside to <em>that<\/em> is it assumes command spawning is a pure function, which of course it's not; anything that talks to a socket is trouble because it might be stateful.<\/p>\n<\/div><\/aside>\n<h3 id=\"environment-variables\">environment variables<a class=\"zola-anchor\" href=\"#environment-variables\" aria-label=\"Anchor link for: environment-variables\"><\/a>\n<\/h3>\n<p>Tracing environment variable access is \u2026 hard. Traditionally access goes through the libc <code>getenv<\/code> function, but it\u2019s also possible to take an <code>envp<\/code> in a main function, in which case accesses are just memory reads. That means we need to trace memory reads somehow.<\/p>\n<p>On x86 machines, there\u2019s something called <a href=\"https:\/\/stackoverflow.com\/a\/77056006\">PIN<\/a> that can do this directly in the CPU without needing compile time instrumentation. On ARM there\u2019s <a href=\"https:\/\/developer.arm.com\/community\/arm-community-blogs\/b\/architectures-and-processors-blog\/posts\/statistical-profile-extension\">SPE<\/a>, which is how <a href=\"https:\/\/docs.redhat.com\/en\/documentation\/red_hat_enterprise_linux\/8\/html\/monitoring_and_managing_system_status_and_performance\/profiling-memory-accesses-with-perf-mem_monitoring-and-managing-system-status-and-performance\"><code>perf mem<\/code><\/a> works, but I\u2019m not sure whether it can be configured to track 100% of memory accesses. I need to do more research here.<\/p>\n<p>On Linux, this is all abstracted by <a href=\"https:\/\/man7.org\/linux\/man-pages\/man2\/perf_event_open.2.html\"><code>perf_event_open<\/code><\/a>. I\u2019m not sure if there\u2019s equivalent wrappers on Windows and macOS.<\/p>\n<aside role=note class=note-container><div class=note-content>\n<p>There\u2019s also <a href=\"https:\/\/dynamorio.org\/#autotoc_md180\">DynamicRIO<\/a>, which supports a bunch of platforms, but I believe it works in a similar way to QEMU, by interposing itself between the program and the CPU, which comes with a bunch of overhead. That could work as an opt-in.<\/p>\n<\/div><\/aside>\n<p>One last way to do this is with a <a href=\"https:\/\/unix.stackexchange.com\/a\/532395\">SIGSEGV signal handler<\/a>, but that requires that environment variables are in their own page of memory and therefore a linker script. This doesn\u2019t work for environment variables specifically, because they <a href=\"https:\/\/www.owlfolio.org\/development\/thread-safe-environment-variable-mutation-working-draft-2022-15\/\">aren\u2019t linker symbols in the normal sense, they get injected by the C runtime<\/a>. In general, injecting linker scripts means we\u2019re modifying the binaries being run and might cause unexpected build or runtime failures.<\/p>\n<h2 id=\"ronin-a-ninja-successor\"><code>ronin<\/code>: a ninja successor<a class=\"zola-anchor\" href=\"#ronin-a-ninja-successor\" aria-label=\"Anchor link for: ronin-a-ninja-successor\"><\/a>\n<\/h2>\n<p>Here I describe more concretely the tool I want to build, which I\u2019ve named <code>ronin<\/code>. It would read the <a href=\"\/i-want-a-better-action-graph-serialization\/#designing-a-new-action-graph\">constrained clojure action graph serialization format<\/a> (Magma) that I describe in the previous post; perhaps with a way to automatically convert Ninja files to Magma.<\/p>\n<h3 id=\"interface\">interface<a class=\"zola-anchor\" href=\"#interface\" aria-label=\"Anchor link for: interface\"><\/a>\n<\/h3>\n<p>Like <a href=\"https:\/\/github.com\/capnproto\/ekam\/\">Ekam<\/a>, Ronin would have a <code>--watch<\/code> continuous rebuild mode (but unlike Bazel and Buck2, no background server). Like Shake, It would have runtime tracing, with all of <code>--tracing=never|warn|error<\/code> options, to allow gradually transitioning to a hermetic build. And it would have bazel-like querying for the action graph, both through CLI arguments with an jq syntax and through a programmatic API.<\/p>\n<p>Finally, it would have pluggable backends for file watching, tracing, stat-ing, progress reporting, and checksums, so that it can take advantage of systems that have more features while still being reasonably fast on systems that don\u2019t. For example, on Windows stats are slow, so it would cache stat info; but on Linux stats are fast so it would just directly make a syscall.<\/p>\n<h3 id=\"architecture\">architecture<a class=\"zola-anchor\" href=\"#architecture\" aria-label=\"Anchor link for: architecture\"><\/a>\n<\/h3>\n<p>Like Ninja, Ronin would keep a command log with a history of past versions of the action graph. It would reuse the <a href=\"https:\/\/neugierig.org\/software\/blog\/2020\/05\/ninja.html#:~:text=Some%20architecture%20notes\">bipartite graph structure<\/a>, with one half being files and the other being commands. It would parse depfiles and dyndeps files just after they\u2019re built, while the cache is still hot.<\/p>\n<p>Like <a href=\"https:\/\/neugierig.org\/software\/blog\/2022\/03\/n2.html\"><code>n2<\/code><\/a>, ronin would use a single-pass approach to support early cutoff. It would hash an \"input manifest\" to decide whether to rebuild. Unlike <code>n2<\/code>, it would store a mapping from that hash back to the original manifest so you can query why a rebuild happened.<\/p>\n<p>Tracing would be built on top of a FUSE file system that tracked file access. <sup class=\"footnote-reference\" id=\"fr-2-1\"><a href=\"#fn-2\">3<\/a><\/sup><\/p>\n<p>Unlike other build systems I know, state (such as manifest hashes, content hashes, and removed outputs) would be stored in an SQLite database, not in flat files.<\/p>\n<h2 id=\"did-you-just-reinvent-buck2\">did you just reinvent buck2?<a class=\"zola-anchor\" href=\"#did-you-just-reinvent-buck2\" aria-label=\"Anchor link for: did-you-just-reinvent-buck2\"><\/a>\n<\/h2>\n<p>Kinda. Ronin takes a lot of ideas from buck2. It differs in two major ways:<\/p>\n<ul>\n<li>It does not expect to be a top-level build system. It is perfectly happy to read (and encourages) generated files from a higher level configure tool. This allows systems like CMake and Meson to mechanically translate Ninja files into this new format, so builds for existing projects can get nice things.<\/li>\n<li>It allows you to gradually transition from non-hermetic to hermetic builds, without forcing you to fix all your rules at once, and with tracing to help you find where you need to make your fixes. Buck2 doesn\u2019t support tracing at all. It technically supports non-hermetic builds, but you don't get many benefits compared to using a different build system, and it's still high cost to switch build systems <sup class=\"footnote-reference\" id=\"fr-buck-correction-1\"><a href=\"#fn-buck-correction\">4<\/a><\/sup>.<\/li>\n<\/ul>\n<p>The main advantage of Ronin is that it can slot in underneath existing build systems people are already using\u2014CMake and Meson\u2014without needing changes to your build files at all.<\/p>\n<h2 id=\"summary\">summary<a class=\"zola-anchor\" href=\"#summary\" aria-label=\"Anchor link for: summary\"><\/a>\n<\/h2>\n<p>In this post I describe what a build executor does, some features I would like to see from an executor (with a special focus on tracing), and a design for a new executor called <code>ronin<\/code> that allows existing projects generating ninja files to gradually transition to hermetic builds over time, without a \u201cflag day\u201d that requires rewriting the whole build system.<\/p>\n<p>I don\u2019t know yet if I will actually build this tool, that seems like a lot of work <sup class=\"footnote-reference\" id=\"fr-4-1\"><a href=\"#fn-4\">5<\/a><\/sup> \ud83d\ude04 but it\u2019s something I would like to exist in the world.<\/p>\n<section class=\"footnotes\">\n<ol class=\"footnotes-list\">\n<li id=\"fn-1\">\n<p>In many ways Conan <a href=\"https:\/\/docs.conan.io\/2\/tutorial\/consuming_packages\/build_simple_cmake_project.html\">profiles<\/a> are analogous to ninja files: profiles are the interface between Conan and CMake in the same way that ninja files are the interface between CMake and Ninja. Conan is the only tool I'm aware of where the split between the package manager and the configure step is explicit. <a href=\"#fr-1-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-3\">\n<p>This is not an apple to apples comparison; ideally we would name the target by the output file, not by its alias. Unfortunately output names are unpredictable and quite long in Bazel. <a href=\"#fr-3-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-2\">\n<p>macOS does not have native support for FUSE. <a href=\"https:\/\/macfuse.github.io\/\">MacFuse<\/a> exists but does not <a href=\"https:\/\/github.com\/macfuse\/macfuse\/wiki\/FUSE-Backends#limitations\">support getting the PID<\/a> of the calling process. A possible workaround would be to start a new FUSE server for each spawned process group. FUSE on Windows is possible through <a href=\"https:\/\/github.com\/billziss-gh\/winfsp\">winfsp<\/a>. <a href=\"#fr-2-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-buck-correction\">\n<p>An earlier version of this post read \"Buck2 only supports non-hermetic builds for <a href=\"https:\/\/buck2.build\/docs\/concepts\/toolchain\/\">system toolchains<\/a>, not anything else\", which is not correct. <a href=\"#fr-buck-correction-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-4\">\n<p>what if i simply took buck2 and hacked it to bits,,, <a href=\"#fr-4-1\">\u21a9<\/a><\/p>\n<\/li>\n<\/ol>\n<\/section>\n"},{"title":"I want a better action graph serialization","published":"2025-12-04T00:00:00+00:00","updated":"2025-12-04T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/i-want-a-better-action-graph-serialization\/"}},"id":"https:\/\/jyn.dev\/i-want-a-better-action-graph-serialization\/","summary":"Makefiles and Ninja files have limitations, separate from the implementing tool. How can we fix them?","content":"<p>This post is part 3\/4 of <a href=\"\/four-posts-about-build-systems\/\">a series about build systems<\/a>.\nThe next post and last post is <a href=\"\/i-want-a-better-build-executor\/\">I want a better build executor<\/a>.<\/p>\n<hr \/>\n<figure>\n<blockquote cite=\"https:\/\/lobste.rs\/s\/uwyfpy\/build_system_tradeoffs#c_ymm0ad\">\n<p>As someone who ends up getting the ping on \"my build is weird\" <em>after<\/em> it has gone through a round of \"poke it with a stick\", I would really appreciate the <em>mechanisms<\/em> for [correct dependency edges] rolling out sooner rather than later.<\/p>\n<\/blockquote>\n<figcaption>\n<cite class=username><a href=\"https:\/\/lobste.rs\/s\/uwyfpy\/build_system_tradeoffs#c_ymm0ad\">mathstuf<\/a><\/cite>\n<\/figcaption>\n<\/figure>\n<h2 id=\"what-does-action-graph-mean\">what does \"action graph\" mean?<a class=\"zola-anchor\" href=\"#what-does-action-graph-mean\" aria-label=\"Anchor link for: what-does-action-graph-mean\"><\/a>\n<\/h2>\n<p>In <a href=\"https:\/\/jyn.dev\/build-system-tradeoffs\/\">a previous post<\/a>, I talked about various approaches in the design space of build systems. In this post, I want to zero in on one particular area: action graphs.<\/p>\n<p>First, let me define \"action graph\". If you've ever used CMake, you may know that there are two steps involved: A \"configure\" step (<code>cmake -B build-dir<\/code>) and a build step (<code>make<\/code> or <code>cmake --build<\/code>). What I am interested here is what <code>cmake -B<\/code> <em>generates<\/em>, the Makefiles it has created. As <a href=\"https:\/\/neugierig.org\/software\/blog\/2020\/05\/ninja.html#:~:text=Related%20work\">the creator of ninja writes<\/a>, this is a <em>serialization<\/em> of all build steps at a given moment in time, with the ability to regenerate the graph by rerunning the configure step.<\/p>\n<p>This post explores that design space, with the goal of sketching a format that improves on the current state while also enabling incremental adoption. When I say \"design space\", I mean a serialization format where files are machine-generated by a configure step, and have few enough and restricted enough features that it's possible to make a fast <a href=\"\/i-want-a-better-build-executor\/\">build executor<\/a>.<\/p>\n<h3 id=\"who-uses-an-action-graph\">who uses an action graph?<a class=\"zola-anchor\" href=\"#who-uses-an-action-graph\" aria-label=\"Anchor link for: who-uses-an-action-graph\"><\/a>\n<\/h3>\n<p>Not all build systems serialize their action graph. <code>bazel<\/code> and <code>buck2<\/code> run persistent servers that store it in memory and allow querying it, but never serialize it to disk. For large graphs, this requires a lot of memory; <code>blaze<\/code> has actually started <a href=\"https:\/\/youtu.be\/1A8LMZ21t6Y?si=FiQ6LGdBQ7Y-aQLY\">serializing parts of its graph to reduce memory usage and startup time<\/a>.<\/p>\n<p>The nix evaluator doesn\u2019t allow querying its graph at all; nix has a very strange model where it never rebuilds because each change to your source files is a new \u201c<a href=\"https:\/\/nix.dev\/manual\/nix\/2.32\/store\/derivation\/index.html\">input-addressed derivation<\/a>\u201d and therefore requires a reconfigure. This is the main reason it\u2019s only used to package software, not as an \u201cinner\u201d build system, because that reconfigure can be very slow.<\/p>\n<aside role=note class=note-container><div class=note-content>\n<p>I\u2019ve talked to a couple Nix maintainers and they\u2019ve considered caching parts of the <em>configure<\/em> step, without caching its outputs (because there are no outputs, other than derivation files!) in order to speed this up. This is much trickier because it requires serializing parts of the evaluator state.<\/p>\n<\/div><\/aside>\n<p>Tools that <em>do<\/em> serialize their graph include CMake, Meson, and the Chrome build system (<a href=\"https:\/\/gn.googlesource.com\/gn\/\">GN<\/a>).<\/p>\n<p>Generally, serializing the graph comes in handy when:<\/p>\n<ul>\n<li>You don\u2019t have a persistent server to store it in memory. When you don\u2019t have a server, serializing makes your startup times much faster, because you don\u2019t have to rerun the configure step each time.<\/li>\n<li>You don\u2019t have a remote build cache. When you have a remote cache, the rules for <em>loading<\/em> that cache can be rather complicated because they involve network queries <sup class=\"footnote-reference\" id=\"fr-10-1\"><a href=\"#fn-10\">1<\/a><\/sup>. When you have a local cache, loading it doesn\u2019t require special support because it\u2019s just opening a file.<\/li>\n<li>You want to support querying, process spawning, and progress updates without rewriting the logic yourself for every OS (i.e. you don't want to write your own build executor).<\/li>\n<\/ul>\n<h3 id=\"design-goals\">design goals<a class=\"zola-anchor\" href=\"#design-goals\" aria-label=\"Anchor link for: design-goals\"><\/a>\n<\/h3>\n<p>In the last post I talked about 4 things one might want from a build system:<\/p>\n<ol>\n<li>a \"real\" language in the configuration step<\/li>\n<li>reflection (querying the build graph)<\/li>\n<li>file watching<\/li>\n<li>support for discovering incorrect dependency edges<\/li>\n<\/ol>\n<p>For a serialization format, we have slightly different constraints.<\/p>\n<ul>\n<li>We care about it being simple and unambiguous to load the graph from the file, so we get fast incremental rebuild speed and graph queries. In particular, we want to touch the filesystem as little as possible while loading.<\/li>\n<li>We care about supporting \"weird\" dependency edges, like <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2018\/03\/build-systems.pdf\">dynamic dependencies<\/a> and the depfiles emitted by a compiler after the first run, so that we're able to support more kinds of builds.<\/li>\n<li>And finally, we care about <em>minimizing reconfigurations<\/em>: we want to be able to express as many things as possible in the action graph so we don't have the pay the cost of rerunning the configure step. This tends to be at odds with fast graph loading; adding features at this level of the stack is very expensive!<\/li>\n<\/ul>\n<p>Throughout this post, I'll dive into detail on how these 3 overarching goals apply to the serialization format, and how well various serializations achieve that goal.<\/p>\n<h2 id=\"existing-serializations\">existing serializations<a class=\"zola-anchor\" href=\"#existing-serializations\" aria-label=\"Anchor link for: existing-serializations\"><\/a>\n<\/h2>\n<p>The first one we'll look at, because it's the default for CMake, is <code>make<\/code> and Makefiles. Make is truly in the Unix spirit: easy to implement<sup class=\"footnote-reference\" id=\"fr-2-1\"><a href=\"#fn-2\">2<\/a><\/sup>, very hard to use correctly.  Make is <a href=\"https:\/\/medium.com\/@adi.ashour\/stuff-i-learned-about-makefiles-part-2-66d87109b19b#:~:text=beware%20of%20ambiguity\">ambiguous<\/a>, <a href=\"https:\/\/medium.com\/@adi.ashour\/stuff-i-learned-about-makefiles-part-2-66d87109b19b#:~:text==%20vs%20:=\">complicated<\/a>, and makes it very easy to <a href=\"https:\/\/medium.com\/@adi.ashour\/stuff-i-learned-about-makefiles-part-1-9b85986d7c59#:~:text=pattern%20rules\">implicitly do a bunch of file system lookups<\/a>. It supports running shell commands at the top-level, which makes even loading the graph very expensive. It does do pretty well on minimizing reconfigurations, since the language is quite flexible.<\/p>\n<p>Ninja is the other generator supported by CMake. Ninja is explicitly intended to work on a serialized action graph; it's the only tool I'm aware of that is. It <a href=\"https:\/\/ninja-build.org\/manual.html#_comparison_to_make\">solves a lot of the problems of Make<\/a>: it removes many of the ambiguities; it doesn't have any form of globbing; and generally it's a much simpler and smaller language. Unfortunately, Ninja's build file format still has some limitations.<\/p>\n<p>First, it has no support for checksums. It's possible to work around that by using <a href=\"https:\/\/ninja-build.org\/manual.html#:~:text=re-stat%20the%20command\"><code>restat = true<\/code><\/a> and having a wrapper script that doesn't overwrite files unless they've changed, but that's a lot of extra work and is annoying to make portable between operating systems.<\/p>\n<p>Ninja files also have trouble expressing correct dependency edges. Let's look at a few examples, one by one. In each of these cases, we either have to reconfigure more often than we wish, or we have no way at all of expressing the dependency edge.<\/p>\n<h2 id=\"dependency-edge-issues-with-ninja\">dependency edge issues with ninja<a class=\"zola-anchor\" href=\"#dependency-edge-issues-with-ninja\" aria-label=\"Anchor link for: dependency-edge-issues-with-ninja\"><\/a>\n<\/h2>\n<h3 id=\"negative-dependencies\">negative dependencies<a class=\"zola-anchor\" href=\"#negative-dependencies\" aria-label=\"Anchor link for: negative-dependencies\"><\/a>\n<\/h3>\n<p>See <a href=\"\/negative-build-dependencies\/\">my previous post<\/a> about negative dependencies. The short version is that build files need to specify not just the files they expect to exist, but also the files they expect <em>not<\/em> to exist. There's no way to express this in a ninja file, short of reconfiguring every time a directory that might contain a negative dependency is modified, which itself has a lot of downsides.<\/p>\n<h3 id=\"renamed-files\">renamed files<a class=\"zola-anchor\" href=\"#renamed-files\" aria-label=\"Anchor link for: renamed-files\"><\/a>\n<\/h3>\n<p>Say that you have a C project with just a <code>main.c<\/code>. You rename it to <code>project.c<\/code> and ninja gives you an error that main.c no longer exists. Annoyed of editing ninja files by hand, you decide to write a generator<sup class=\"footnote-reference\" id=\"fr-4-1\"><a href=\"#fn-4\">3<\/a><\/sup> :<\/p>\n<pre data-lang=\"python\" class=\"language-python z-code\"><code class=\"language-python\" data-lang=\"python\"><span class=\"z-source z-python\"><span class=\"z-comment z-line z-number-sign z-python\"><span class=\"z-punctuation z-definition z-comment z-python\">#<\/span> build.py\n<\/span><\/span><span class=\"z-source z-python\"><span class=\"z-meta z-statement z-import z-python\"><span class=\"z-keyword z-control z-import z-python\">import<\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">ninja_syntax<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-meta z-statement z-import z-python\"><span class=\"z-keyword z-control z-import z-python\">import<\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">os<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-meta z-statement z-import z-python\"><span class=\"z-keyword z-control z-import z-python\">import<\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">sys<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-meta z-statement z-import z-python\"><span class=\"z-keyword z-control z-import z-from z-python\">from<\/span><\/span><span class=\"z-meta z-statement z-import z-python\"><span class=\"z-meta z-import-source z-python\"> <span class=\"z-meta z-import-path z-python\"><span class=\"z-meta z-import-name z-python\">os<\/span><span class=\"z-punctuation z-accessor z-dot z-python\">.<\/span><span class=\"z-meta z-import-name z-python\">path<\/span><\/span> <span class=\"z-meta z-statement z-import z-python\"><span class=\"z-keyword z-control z-import z-python\">import<\/span><\/span><\/span><\/span><span class=\"z-meta z-statement z-import z-python\"><\/span><span class=\"z-meta z-statement z-import z-python\"> <span class=\"z-meta z-generic-name z-python\">basename<\/span><span class=\"z-punctuation z-separator z-import-list z-python\">,<\/span> <span class=\"z-meta z-generic-name z-python\">splitext<\/span><\/span>\n<\/span><span class=\"z-source z-python\">\n<\/span><span class=\"z-source z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">writer<\/span><\/span> <span class=\"z-keyword z-operator z-assignment z-python\">=<\/span> <span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">ninja_syntax<\/span><\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-punctuation z-accessor z-dot z-python\">.<\/span><\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-meta z-generic-name z-python\">Writer<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">sys<\/span><span class=\"z-punctuation z-accessor z-dot z-python\">.<\/span><span class=\"z-meta z-generic-name z-python\">stdout<\/span><\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">writer<\/span><\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-punctuation z-accessor z-dot z-python\">.<\/span><\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-meta z-generic-name z-python\">rule<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\"><span class=\"z-punctuation z-definition z-string z-begin z-python\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\">python<span class=\"z-punctuation z-definition z-string z-end z-python\">&quot;<\/span><\/span><\/span><span class=\"z-punctuation z-separator z-arguments z-python\">,<\/span> <span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\"><span class=\"z-punctuation z-definition z-string z-begin z-python\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\">python $in &gt; $out<span class=\"z-punctuation z-definition z-string z-end z-python\">&quot;<\/span><\/span><\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">writer<\/span><\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-punctuation z-accessor z-dot z-python\">.<\/span><\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-meta z-generic-name z-python\">build<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\"><span class=\"z-punctuation z-definition z-string z-begin z-python\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\">build.ninja<span class=\"z-punctuation z-definition z-string z-end z-python\">&quot;<\/span><\/span><\/span><span class=\"z-punctuation z-separator z-arguments z-python\">,<\/span> <span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\"><span class=\"z-punctuation z-definition z-string z-begin z-python\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\">python<span class=\"z-punctuation z-definition z-string z-end z-python\">&quot;<\/span><\/span><\/span><span class=\"z-punctuation z-separator z-arguments z-python\">,<\/span> <span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\"><span class=\"z-punctuation z-definition z-string z-begin z-python\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\">build.py<span class=\"z-punctuation z-definition z-string z-end z-python\">&quot;<\/span><\/span><\/span><span class=\"z-punctuation z-separator z-arguments z-python\">,<\/span> <span class=\"z-variable z-parameter z-python\">implicit<\/span><span class=\"z-keyword z-operator z-assignment z-python\">=<\/span><span class=\"z-meta z-sequence z-list z-python\"><span class=\"z-punctuation z-section z-sequence z-begin z-python\">[<\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\"><span class=\"z-punctuation z-definition z-string z-begin z-python\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\">.<span class=\"z-punctuation z-definition z-string z-end z-python\">&quot;<\/span><\/span><\/span><span class=\"z-punctuation z-section z-sequence z-end z-python\">]<\/span><\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span>\n<\/span><span class=\"z-source z-python\">\n<\/span><span class=\"z-source z-python\"><span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">writer<\/span><\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-punctuation z-accessor z-dot z-python\">.<\/span><\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-meta z-generic-name z-python\">rule<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\"><span class=\"z-punctuation z-definition z-string z-begin z-python\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\">cc<span class=\"z-punctuation z-definition z-string z-end z-python\">&quot;<\/span><\/span><\/span><span class=\"z-punctuation z-separator z-arguments z-python\">,<\/span> <span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\"><span class=\"z-punctuation z-definition z-string z-begin z-python\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\">cc $in -o $out<span class=\"z-punctuation z-definition z-string z-end z-python\">&quot;<\/span><\/span><\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-meta z-statement z-loop z-for z-python\"><span class=\"z-keyword z-control z-loop z-for z-python\">for<\/span> <span class=\"z-meta z-generic-name z-python\">path<\/span> <span class=\"z-keyword z-control z-loop z-for z-in z-python\">in<\/span><\/span><span class=\"z-meta z-statement z-loop z-for z-python\"> <span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">os<\/span><\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-punctuation z-accessor z-dot z-python\">.<\/span><\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-meta z-generic-name z-python\">listdir<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-single z-python\"><span class=\"z-punctuation z-definition z-string z-begin z-python\">&#39;<\/span><\/span><\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-single z-python\">.<span class=\"z-punctuation z-definition z-string z-end z-python\">&#39;<\/span><\/span><\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span><\/span><span class=\"z-meta z-statement z-loop z-for z-python\"><span class=\"z-punctuation z-section z-block z-loop z-for z-python\">:<\/span><\/span>\n<\/span><span class=\"z-source z-python\">    <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">base<\/span><\/span>, <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">ext<\/span><\/span> <span class=\"z-keyword z-operator z-assignment z-python\">=<\/span> <span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-meta z-generic-name z-python\">splitext<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">path<\/span><\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span>\n<\/span><span class=\"z-source z-python\">    <span class=\"z-meta z-statement z-conditional z-if z-python\"><span class=\"z-keyword z-control z-conditional z-if z-python\">if<\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">ext<\/span><\/span> <span class=\"z-keyword z-operator z-comparison z-python\">==<\/span> <span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-single z-python\"><span class=\"z-punctuation z-definition z-string z-begin z-python\">&#39;<\/span><\/span><\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-single z-python\">.c<span class=\"z-punctuation z-definition z-string z-end z-python\">&#39;<\/span><\/span><\/span><span class=\"z-punctuation z-section z-block z-conditional z-if z-python\">:<\/span><\/span>\n<\/span><span class=\"z-source z-python\">        <span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">writer<\/span><\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-punctuation z-accessor z-dot z-python\">.<\/span><\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-meta z-generic-name z-python\">build<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">base<\/span><\/span><span class=\"z-punctuation z-separator z-arguments z-python\">,<\/span> <span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\"><span class=\"z-punctuation z-definition z-string z-begin z-python\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\">cc<span class=\"z-punctuation z-definition z-string z-end z-python\">&quot;<\/span><\/span><\/span><span class=\"z-punctuation z-separator z-arguments z-python\">,<\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">path<\/span><\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span>\n<\/span><\/code><\/pre>\n<p>Note this that this registers an implicit dependency on the current directory. This should automatically detect that you renamed your file and rebuild for you.<\/p>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">$ ninja\n<\/span><span class=\"z-text z-plain\">[1\/1] python build.py &gt; build.ninja\n<\/span><span class=\"z-text z-plain\">[1\/2] python build.py &gt; build.ninja\n<\/span><span class=\"z-text z-plain\">[1\/3] python build.py &gt; build.ninja\n<\/span><span class=\"z-text z-plain\">[1\/4] python build.py &gt; build.ninja\n<\/span><span class=\"z-text z-plain\">[1\/5] python build.py &gt; build.ninja\n<\/span><span class=\"z-text z-plain\">[1\/6] python build.py &gt; build.ninja\n<\/span><span class=\"z-text z-plain\">...\n<\/span><span class=\"z-text z-plain\">[1\/100] python build.py &gt; build.ninja\n<\/span><span class=\"z-text z-plain\">ninja: error: manifest &#39;build.ninja&#39; still dirty after 100 tries, perhaps system time is not set\n<\/span><\/code><\/pre>\n<p>Oh. Right. Generating build.ninja also modifies the current directory, which creates an infinite loop.<\/p>\n<p>It's possible to work around this by putting your C file in a source directory:<\/p>\n<pre data-lang=\"python\" class=\"language-python z-code\"><code class=\"language-python\" data-lang=\"python\"><span class=\"z-source z-python\"><span class=\"z-comment z-line z-number-sign z-python\"><span class=\"z-punctuation z-definition z-comment z-python\">#<\/span> build.py\n<\/span><\/span><span class=\"z-source z-python\"><span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">writer<\/span><\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-punctuation z-accessor z-dot z-python\">.<\/span><\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-meta z-generic-name z-python\">build<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\"><span class=\"z-punctuation z-definition z-string z-begin z-python\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\">build.ninja<span class=\"z-punctuation z-definition z-string z-end z-python\">&quot;<\/span><\/span><\/span><span class=\"z-punctuation z-separator z-arguments z-python\">,<\/span> <span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\"><span class=\"z-punctuation z-definition z-string z-begin z-python\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\">python<span class=\"z-punctuation z-definition z-string z-end z-python\">&quot;<\/span><\/span><\/span><span class=\"z-punctuation z-separator z-arguments z-python\">,<\/span> <span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\"><span class=\"z-punctuation z-definition z-string z-begin z-python\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\">build.py<span class=\"z-punctuation z-definition z-string z-end z-python\">&quot;<\/span><\/span><\/span><span class=\"z-punctuation z-separator z-arguments z-python\">,<\/span> <span class=\"z-variable z-parameter z-python\">implicit<\/span><span class=\"z-keyword z-operator z-assignment z-python\">=<\/span><span class=\"z-meta z-sequence z-list z-python\"><span class=\"z-punctuation z-section z-sequence z-begin z-python\">[<\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\"><span class=\"z-punctuation z-definition z-string z-begin z-python\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\">src<span class=\"z-punctuation z-definition z-string z-end z-python\">&quot;<\/span><\/span><\/span><span class=\"z-punctuation z-section z-sequence z-end z-python\">]<\/span><\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-meta z-statement z-loop z-for z-python\"><span class=\"z-keyword z-control z-loop z-for z-python\">for<\/span> <span class=\"z-meta z-generic-name z-python\">path<\/span> <span class=\"z-keyword z-control z-loop z-for z-in z-python\">in<\/span><\/span><span class=\"z-meta z-statement z-loop z-for z-python\"> <span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">os<\/span><\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-punctuation z-accessor z-dot z-python\">.<\/span><\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-meta z-generic-name z-python\">listdir<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-single z-python\"><span class=\"z-punctuation z-definition z-string z-begin z-python\">&#39;<\/span><\/span><\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-single z-python\">src<span class=\"z-punctuation z-definition z-string z-end z-python\">&#39;<\/span><\/span><\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span><\/span><span class=\"z-meta z-statement z-loop z-for z-python\"><span class=\"z-punctuation z-section z-block z-loop z-for z-python\">:<\/span><\/span>\n<\/span><span class=\"z-source z-python\">    <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">base<\/span><\/span>, <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">ext<\/span><\/span> <span class=\"z-keyword z-operator z-assignment z-python\">=<\/span> <span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-meta z-generic-name z-python\">splitext<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">path<\/span><\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span>\n<\/span><span class=\"z-source z-python\">    <span class=\"z-meta z-statement z-conditional z-if z-python\"><span class=\"z-keyword z-control z-conditional z-if z-python\">if<\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">ext<\/span><\/span> <span class=\"z-keyword z-operator z-comparison z-python\">==<\/span> <span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-single z-python\"><span class=\"z-punctuation z-definition z-string z-begin z-python\">&#39;<\/span><\/span><\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-single z-python\">.c<span class=\"z-punctuation z-definition z-string z-end z-python\">&#39;<\/span><\/span><\/span><span class=\"z-punctuation z-section z-block z-conditional z-if z-python\">:<\/span><\/span>\n<\/span><span class=\"z-source z-python\">        <span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">writer<\/span><\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-punctuation z-accessor z-dot z-python\">.<\/span><\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-meta z-generic-name z-python\">build<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-single z-python\"><span class=\"z-punctuation z-definition z-string z-begin z-python\">&#39;<\/span><\/span><\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-single z-python\">src\/<span class=\"z-punctuation z-definition z-string z-end z-python\">&#39;<\/span><\/span><\/span><span class=\"z-keyword z-operator z-arithmetic z-python\">+<\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">base<\/span><\/span><span class=\"z-punctuation z-separator z-arguments z-python\">,<\/span> <span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\"><span class=\"z-punctuation z-definition z-string z-begin z-python\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\">cc<span class=\"z-punctuation z-definition z-string z-end z-python\">&quot;<\/span><\/span><\/span><span class=\"z-punctuation z-separator z-arguments z-python\">,<\/span> <span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-single z-python\"><span class=\"z-punctuation z-definition z-string z-begin z-python\">&#39;<\/span><\/span><\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-single z-python\">src\/<span class=\"z-punctuation z-definition z-string z-end z-python\">&#39;<\/span><\/span><\/span><span class=\"z-keyword z-operator z-arithmetic z-python\">+<\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">path<\/span><\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span>\n<\/span><\/code><\/pre>\n<pre data-lang=\"console\" class=\"language-console z-code\"><code class=\"language-console\" data-lang=\"console\"><span class=\"z-source z-shell z-console\"><span class=\"z-keyword z-operator z-assignment z-redirection z-process z-shell\">$<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">tree<\/span><\/span>\n<\/span><span class=\"z-source z-shell z-console\">.\n<\/span><span class=\"z-source z-shell z-console\">\u251c\u2500\u2500 build.ninja\n<\/span><span class=\"z-source z-shell z-console\">\u251c\u2500\u2500 build.py\n<\/span><span class=\"z-source z-shell z-console\">\u251c\u2500\u2500 main.c\n<\/span><span class=\"z-source z-shell z-console\">\u251c\u2500\u2500 ninja_syntax.py\n<\/span><span class=\"z-source z-shell z-console\">\u2514\u2500\u2500 src\n<\/span><span class=\"z-source z-shell z-console\">    \u2514\u2500\u2500 main.c\n<\/span><span class=\"z-source z-shell z-console\"><span class=\"z-keyword z-operator z-assignment z-redirection z-process z-shell\">$<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">ninja<\/span><\/span>\n<\/span><span class=\"z-source z-shell z-console\">[1\/1] python build.py &gt; build.ninja\n<\/span><span class=\"z-source z-shell z-console\">[1\/2] cc src\/main.c -o src\/main\n<\/span><span class=\"z-source z-shell z-console\"><span class=\"z-keyword z-operator z-assignment z-redirection z-process z-shell\">$<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">mv<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> src\/<span class=\"z-meta z-group z-expansion z-brace z-shell\"><span class=\"z-punctuation z-section z-expansion z-brace z-begin z-shell\">{<\/span>main<span class=\"z-punctuation z-separator z-shell\">,<\/span>project<span class=\"z-punctuation z-section z-expansion z-brace z-end z-shell\">}<\/span><\/span>.c<\/span>\n<\/span><span class=\"z-source z-shell z-console\"><span class=\"z-keyword z-operator z-assignment z-redirection z-process z-shell\">$<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">ninja<\/span><\/span>\n<\/span><span class=\"z-source z-shell z-console\">[1\/1] python build.py &gt; build.ninja\n<\/span><span class=\"z-source z-shell z-console\">[1\/2] cc src\/project.c -o src\/project\n<\/span><\/code><\/pre>\n<p>There's still a problem here, though\u2014did you notice it?<\/p>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">$ ls src\n<\/span><span class=\"z-text z-plain\">main  project  project.c\n<\/span><\/code><\/pre>\n<p>Our old <code>main<\/code> target is still lying around. Ninja actually has enough information recorded to fix this: <code>ninja -t cleandead<\/code>. But it's not run automatically.<\/p>\n<p>The other problem is that this approach rebuilds far too often. In this case, we wanted to support renames, so in Ninja's model we need to depend on the whole directory. But that's not what we really depended on\u2014we only care about <code>.c<\/code> files. I would like to see a action graph format that has an event-based system, where it says \"this file was created, make any changes to the action graph necessary\", and cuts the build short if the graph wasn't changed.<\/p>\n<h3 id=\"optional-file-dependencies\">optional file dependencies<a class=\"zola-anchor\" href=\"#optional-file-dependencies\" aria-label=\"Anchor link for: optional-file-dependencies\"><\/a>\n<\/h3>\n<p>For <a href=\"https:\/\/flower.jyn.dev\/overlay.html\">flower<\/a>, I want to go further and support <em>deletions<\/em>: source files and targets that are optional, that should not fail the build if they aren't present, but should cause a rebuild if they are created, modified, or deleted. Ninja has no way of expressing this.<\/p>\n<!--\n## deleted depfiles\nNinja has a feature called \"depfiles\", where dependency information is not fully known until after the first run of a build edge. It's intended for C header files, but can be used for other things. The idea is that the dependency graph might depend on which things you dynamically require inside your source file. To support this, C compilers support `-M` [^5] to emit a `make`-compatible dependency graph for that source file, which ninja will read on the next run.\n\nThis can get us into a lot of trouble.\n\nConsider the following C project:\n\n-->\n<h3 id=\"environment-variables\">environment variables<a class=\"zola-anchor\" href=\"#environment-variables\" aria-label=\"Anchor link for: environment-variables\"><\/a>\n<\/h3>\n<p>Ninja has no way to express \u201cthis node becomes dirty when an environment variable changes\u201d. The closest you can get is hacks with <code>set<\/code> and the checksum wrapper\/restat hack, but it\u2019s a pain to express and it gets much worse if you want to depend on multiple variables.<\/p>\n<h2 id=\"designing-a-new-action-graph\">designing a new action graph<a class=\"zola-anchor\" href=\"#designing-a-new-action-graph\" aria-label=\"Anchor link for: designing-a-new-action-graph\"><\/a>\n<\/h2>\n<!-- mention incremental switch, possible to mechanically translate ninja, not meant to be hand-edited -->\n<p>At this point, we have a list of constraints for our file format:<\/p>\n<!-- 1. Reflection on the action graph -->\n<ol>\n<li>Negative dependencies<\/li>\n<li>File rename dependencies<\/li>\n<li>Optional file dependencies<\/li>\n<\/ol>\n<!-- 4. Tracing (in both linting and hard error mode) to allow gradually transitioning to a hermetic build -->\n<ol start=\"4\">\n<li>Optional checksums to reduce false positives<\/li>\n<li>Environment variable dependencies<\/li>\n<li>\"all the features of ninja\" (depfiles, <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2018\/03\/build-systems-final.pdf\">monadic builds<\/a> through <code>dyndeps<\/code><sup class=\"footnote-reference\" id=\"fr-14-1\"><a href=\"#fn-14\">4<\/a><\/sup>, a <code>generator<\/code> statement, order-only dependencies)<\/li>\n<\/ol>\n<!--\nI am not aware of any existing tool that meets these constraints. I'm considering building one on top of Shake. If anyone knows of prior art in this area, please [let me know](mailto:blog@jyn.dev) :)\n-->\n<p>Ideally, it would even be possible to mechanically translate existing .ninja files to this new format.<\/p>\n<h3 id=\"a-first-try\">a first try<a class=\"zola-anchor\" href=\"#a-first-try\" aria-label=\"Anchor link for: a-first-try\"><\/a>\n<\/h3>\n<p>This sketches out a new format that could improve over Ninja files. It could look something like this:<\/p>\n<ul>\n<li>A very very small clojure subset (just <code>def<\/code>, <code>let<\/code>, <a href=\"https:\/\/github.com\/edn-format\/edn\">EDN<\/a>, and function calls) for the text itself, no need to make loading the graph harder than necessary <sup class=\"footnote-reference\" id=\"fr-6-1\"><a href=\"#fn-6\">5<\/a><\/sup>. If people really want an equivalent of <code>include<\/code> or <code>subninja<\/code> I suppose this could learn support for <code>ns<\/code> and <code>require<\/code>, but it would have much simpler rules than Clojure's classpath. It would not have support for looping constructs, nor most of clojure's standard library.<\/li>\n<li><code>redo<\/code>-inspired dependency edges: <code>ifwritten<\/code> (for changes in file attributes), <code>ifchanged<\/code> (for changes in the checksum), <code>ifcreate<\/code> (for optional dependencies), <code>always<\/code>; plus our new <code>ifdeleted<\/code> edge.<\/li>\n<\/ul>\n<!-- - Get rid of Ninja's `rule` statement; it can be replaced with normal functions. -->\n<ul>\n<li>A <code>dyndeps<\/code> input function that can be used anywhere a file path can (e.g. in calls to <code>ifwritten<\/code>) so that the kind of edge does not depend on whether the path is known in advance or not.<\/li>\n<li>Runtime functions that determine whether the configure step needs to be re-run based on file watch events<sup class=\"footnote-reference\" id=\"fr-7-1\"><a href=\"#fn-7\">6<\/a><\/sup>. Whether there is actually a file watcher or the build system just calculates a diff on its next invocation is an implementation detail; ideally, one that's easy to slot in and out.<\/li>\n<li>\u201cphony\u201d targets would be replaced by a <code>group<\/code> statement. Groups are sets of targets. Groups cannot be used to avoid \u201cinput not found\u201d errors; that niche is filled by <code>ifcreated<\/code>.<\/li>\n<\/ul>\n<p>I\u2019d call this language Magma, since it\u2019s the simplest kind of set with closure.<\/p>\n<p>Here's a sample action graph in Magma:<\/p>\n<pre data-lang=\"clojure\" class=\"language-clojure z-code\"><code class=\"language-clojure\" data-lang=\"clojure\"><span class=\"z-source z-clojure\"><span class=\"z-comment z-line z-clojure\"><span class=\"z-punctuation z-definition z-comment\">;<\/span> In Magma, rules are simple functions.\n<\/span><\/span><span class=\"z-source z-clojure\"><span class=\"z-comment z-line z-clojure\"><span class=\"z-punctuation z-definition z-comment\">;<\/span> For each `action` that uses a rule, the function will be called with a list\n<\/span><\/span><span class=\"z-source z-clojure\"><span class=\"z-comment z-line z-clojure\"><span class=\"z-punctuation z-definition z-comment\">;<\/span> of attributes for that action.\n<\/span><\/span><span class=\"z-source z-clojure\"><span class=\"z-comment z-line z-clojure\"><span class=\"z-punctuation z-definition z-comment\">;<\/span> It returns a description of how to execute the build.\n<\/span><\/span><span class=\"z-source z-clojure\"><span class=\"z-comment z-line z-clojure\"><span class=\"z-punctuation z-definition z-comment\">;<\/span> Supported descriptions are:\n<\/span><\/span><span class=\"z-source z-clojure\"><span class=\"z-comment z-line z-clojure\"><span class=\"z-punctuation z-definition z-comment\">;<\/span> - `:command` (process spawning)\n<\/span><\/span><span class=\"z-source z-clojure\"><span class=\"z-comment z-line z-clojure\"><span class=\"z-punctuation z-definition z-comment\">;<\/span> - `:write` (writes data directly from memory to disk), and\n<\/span><\/span><span class=\"z-source z-clojure\"><span class=\"z-comment z-line z-clojure\"><span class=\"z-punctuation z-definition z-comment\">;<\/span> - `:read` (reads data directly from disk to memory), and\n<\/span><\/span><span class=\"z-source z-clojure\"><span class=\"z-comment z-line z-clojure\"><span class=\"z-punctuation z-definition z-comment\">;<\/span> - `:transform` (runs a clojure function on data in memory).\n<\/span><\/span><span class=\"z-source z-clojure\"><span class=\"z-punctuation z-section z-parens z-begin z-clojure\">(<\/span><span class=\"z-storage z-modifier z-def z-clojure\">defn<\/span> <span class=\"z-entity z-name z-function z-clojure\">cc<\/span> <span class=\"z-punctuation z-section z-brackets z-begin z-clojure\">[<\/span>vars<span class=\"z-punctuation z-section z-brackets z-end z-clojure\">]<\/span>\n<\/span><span class=\"z-source z-clojure\">  <span class=\"z-punctuation z-section z-braces z-begin z-clojure\">{<\/span><span class=\"z-constant z-other z-keyword z-clojure\"><span class=\"z-punctuation z-definition z-keyword z-clojure\">:<\/span>command<\/span> <span class=\"z-punctuation z-section z-brackets z-begin z-clojure\">[<\/span><span class=\"z-string z-quoted z-double z-clojure\"><span class=\"z-punctuation z-definition z-string z-begin z-clojure\">&quot;<\/span>cc<span class=\"z-punctuation z-definition z-string z-end z-clojure\">&quot;<\/span><\/span> <span class=\"z-punctuation z-section z-parens z-begin z-clojure\">(<\/span><span class=\"z-constant z-other z-keyword z-clojure\"><span class=\"z-punctuation z-definition z-keyword z-clojure\">:<\/span>inputs<\/span> vars<span class=\"z-punctuation z-section z-parens z-end z-clojure\">)<\/span>\n<\/span><span class=\"z-source z-clojure\">             <span class=\"z-string z-quoted z-double z-clojure\"><span class=\"z-punctuation z-definition z-string z-begin z-clojure\">&quot;<\/span>-MD<span class=\"z-punctuation z-definition z-string z-end z-clojure\">&quot;<\/span><\/span> <span class=\"z-string z-quoted z-double z-clojure\"><span class=\"z-punctuation z-definition z-string z-begin z-clojure\">&quot;<\/span>-MF<span class=\"z-punctuation z-definition z-string z-end z-clojure\">&quot;<\/span><\/span> <span class=\"z-punctuation z-section z-parens z-begin z-clojure\">(<\/span><span class=\"z-constant z-other z-keyword z-clojure\"><span class=\"z-punctuation z-definition z-keyword z-clojure\">:<\/span>depfile<\/span> vars<span class=\"z-punctuation z-section z-parens z-end z-clojure\">)<\/span>\n<\/span><span class=\"z-source z-clojure\">             <span class=\"z-string z-quoted z-double z-clojure\"><span class=\"z-punctuation z-definition z-string z-begin z-clojure\">&quot;<\/span>-o<span class=\"z-punctuation z-definition z-string z-end z-clojure\">&quot;<\/span><\/span> <span class=\"z-punctuation z-section z-parens z-begin z-clojure\">(<\/span><span class=\"z-constant z-other z-keyword z-clojure\"><span class=\"z-punctuation z-definition z-keyword z-clojure\">:<\/span>out<\/span> vars<span class=\"z-punctuation z-section z-parens z-end z-clojure\">)<\/span><span class=\"z-punctuation z-section z-brackets z-end z-clojure\">]<\/span><span class=\"z-punctuation z-section z-braces z-end z-clojure\">}<\/span><span class=\"z-punctuation z-section z-parens z-end z-clojure\">)<\/span>\n<\/span><span class=\"z-source z-clojure\">\n<\/span><span class=\"z-source z-clojure\"><span class=\"z-comment z-line z-clojure\"><span class=\"z-punctuation z-definition z-comment\">;<\/span> `action`s define a build target (here, the file `main`).\n<\/span><\/span><span class=\"z-source z-clojure\"><span class=\"z-comment z-line z-clojure\"><span class=\"z-punctuation z-definition z-comment\">;<\/span> They must have a `:rule` that states how to execute the build,\n<\/span><\/span><span class=\"z-source z-clojure\"><span class=\"z-comment z-line z-clojure\"><span class=\"z-punctuation z-definition z-comment\">;<\/span> a set  of `:inputs` that must be built before this action is run,\n<\/span><\/span><span class=\"z-source z-clojure\"><span class=\"z-comment z-line z-clojure\"><span class=\"z-punctuation z-definition z-comment\">;<\/span> and a list of `:outputs` that will be created by the `:rule`.\n<\/span><\/span><span class=\"z-source z-clojure\"><span class=\"z-comment z-line z-clojure\"><span class=\"z-punctuation z-definition z-comment\">;<\/span> They may declare a dependency on `:env`ironment variables.\n<\/span><\/span><span class=\"z-source z-clojure\"><span class=\"z-comment z-line z-clojure\"><span class=\"z-punctuation z-definition z-comment\">;<\/span> Like in ninja, a `:depfile` may integrate with the compiler to lazily register\n<\/span><\/span><span class=\"z-source z-clojure\"><span class=\"z-comment z-line z-clojure\"><span class=\"z-punctuation z-definition z-comment\">;<\/span> a dependency on header files.\n<\/span><\/span><span class=\"z-source z-clojure\"><span class=\"z-punctuation z-section z-parens z-begin z-clojure\">(<\/span><span class=\"z-variable z-function z-clojure\">action<\/span> <span class=\"z-constant z-other z-keyword z-clojure\"><span class=\"z-punctuation z-definition z-keyword z-clojure\">:<\/span>outputs<\/span> <span class=\"z-punctuation z-section z-brackets z-begin z-clojure\">[<\/span><span class=\"z-string z-quoted z-double z-clojure\"><span class=\"z-punctuation z-definition z-string z-begin z-clojure\">&quot;<\/span>main<span class=\"z-punctuation z-definition z-string z-end z-clojure\">&quot;<\/span><\/span><span class=\"z-punctuation z-section z-brackets z-end z-clojure\">]<\/span>\n<\/span><span class=\"z-source z-clojure\">  <span class=\"z-constant z-other z-keyword z-clojure\"><span class=\"z-punctuation z-definition z-keyword z-clojure\">:<\/span>rule<\/span> cc\n<\/span><span class=\"z-source z-clojure\">  <span class=\"z-comment z-line z-clojure\"><span class=\"z-punctuation z-definition z-comment\">;<\/span> Unlike in ninja, all inputs must state under which conditions the input causes a rerun.\n<\/span><\/span><span class=\"z-source z-clojure\">  <span class=\"z-comment z-line z-clojure\"><span class=\"z-punctuation z-definition z-comment\">;<\/span> `:ifwritten` means whenever the file metadata changes.\n<\/span><\/span><span class=\"z-source z-clojure\">  <span class=\"z-constant z-other z-keyword z-clojure\"><span class=\"z-punctuation z-definition z-keyword z-clojure\">:<\/span>inputs<\/span> <span class=\"z-punctuation z-section z-braces z-begin z-clojure\">{<\/span><span class=\"z-constant z-other z-keyword z-clojure\"><span class=\"z-punctuation z-definition z-keyword z-clojure\">:<\/span>ifwritten<\/span> <span class=\"z-punctuation z-section z-brackets z-begin z-clojure\">[<\/span><span class=\"z-string z-quoted z-double z-clojure\"><span class=\"z-punctuation z-definition z-string z-begin z-clojure\">&quot;<\/span>main.c<span class=\"z-punctuation z-definition z-string z-end z-clojure\">&quot;<\/span><\/span><span class=\"z-punctuation z-comma z-clojure\"><span class=\"z-comment z-punctuation z-comma z-clojure\">,<\/span><\/span> <span class=\"z-string z-quoted z-double z-clojure\"><span class=\"z-punctuation z-definition z-string z-begin z-clojure\">&quot;<\/span>helper.c<span class=\"z-punctuation z-definition z-string z-end z-clojure\">&quot;<\/span><\/span><span class=\"z-punctuation z-section z-brackets z-end z-clojure\">]<\/span><span class=\"z-punctuation z-section z-braces z-end z-clojure\">}<\/span>\n<\/span><span class=\"z-source z-clojure\">  <span class=\"z-comment z-line z-clojure\"><span class=\"z-punctuation z-definition z-comment\">;<\/span> Like in ninja, `:implicit` is exactly the same as `:inputs`,\n<\/span><\/span><span class=\"z-source z-clojure\">  <span class=\"z-comment z-line z-clojure\"><span class=\"z-punctuation z-definition z-comment\">;<\/span> but allows the `:rule` to distinguish which files should be passed to the command.\n<\/span><\/span><span class=\"z-source z-clojure\">  <span class=\"z-constant z-other z-keyword z-clojure\"><span class=\"z-punctuation z-definition z-keyword z-clojure\">:<\/span>implicit<\/span> <span class=\"z-punctuation z-section z-braces z-begin z-clojure\">{<\/span><span class=\"z-constant z-other z-keyword z-clojure\"><span class=\"z-punctuation z-definition z-keyword z-clojure\">:<\/span>ifwritten<\/span> <span class=\"z-punctuation z-section z-brackets z-begin z-clojure\">[<\/span><span class=\"z-string z-quoted z-double z-clojure\"><span class=\"z-punctuation z-definition z-string z-begin z-clojure\">&quot;<\/span>helper.h<span class=\"z-punctuation z-definition z-string z-end z-clojure\">&quot;<\/span><\/span><span class=\"z-punctuation z-section z-brackets z-end z-clojure\">]<\/span><span class=\"z-punctuation z-section z-braces z-end z-clojure\">}<\/span>\n<\/span><span class=\"z-source z-clojure\">  <span class=\"z-constant z-other z-keyword z-clojure\"><span class=\"z-punctuation z-definition z-keyword z-clojure\">:<\/span>env<\/span> <span class=\"z-punctuation z-section z-brackets z-begin z-clojure\">[<\/span><span class=\"z-string z-quoted z-double z-clojure\"><span class=\"z-punctuation z-definition z-string z-begin z-clojure\">&quot;<\/span>LIBRARY_PATH<span class=\"z-punctuation z-definition z-string z-end z-clojure\">&quot;<\/span><\/span><span class=\"z-punctuation z-comma z-clojure\"><span class=\"z-comment z-punctuation z-comma z-clojure\">,<\/span><\/span> <span class=\"z-string z-quoted z-double z-clojure\"><span class=\"z-punctuation z-definition z-string z-begin z-clojure\">&quot;<\/span>C_INCLUDE_PATH<span class=\"z-punctuation z-definition z-string z-end z-clojure\">&quot;<\/span><\/span><span class=\"z-punctuation z-section z-brackets z-end z-clojure\">]<\/span>\n<\/span><span class=\"z-source z-clojure\">  <span class=\"z-constant z-other z-keyword z-clojure\"><span class=\"z-punctuation z-definition z-keyword z-clojure\">:<\/span>depfile<\/span> <span class=\"z-string z-quoted z-double z-clojure\"><span class=\"z-punctuation z-definition z-string z-begin z-clojure\">&quot;<\/span>main.c.d<span class=\"z-punctuation z-definition z-string z-end z-clojure\">&quot;<\/span><\/span><span class=\"z-punctuation z-section z-parens z-end z-clojure\">)<\/span>\n<\/span><span class=\"z-source z-clojure\">\n<\/span><span class=\"z-source z-clojure\"><span class=\"z-comment z-line z-clojure\"><span class=\"z-punctuation z-definition z-comment\">;<\/span> Actions that don&#39;t create a file on disk are allowed.\n<\/span><\/span><span class=\"z-source z-clojure\"><span class=\"z-comment z-line z-clojure\"><span class=\"z-punctuation z-definition z-comment\">;<\/span> This is similar to a &quot;Phony&quot; target in Make, except that you can choose to not always rerun it.\n<\/span><\/span><span class=\"z-source z-clojure\"><span class=\"z-comment z-line z-clojure\"><span class=\"z-punctuation z-definition z-comment\">;<\/span> Since there&#39;s no output, we manually specify the name of the action so other actions can depend on it.\n<\/span><\/span><span class=\"z-source z-clojure\"><span class=\"z-punctuation z-section z-parens z-begin z-clojure\">(<\/span><span class=\"z-variable z-function z-clojure\">action<\/span> <span class=\"z-constant z-other z-keyword z-clojure\"><span class=\"z-punctuation z-definition z-keyword z-clojure\">:<\/span>name<\/span> <span class=\"z-string z-quoted z-double z-clojure\"><span class=\"z-punctuation z-definition z-string z-begin z-clojure\">&quot;<\/span>docker-image<span class=\"z-punctuation z-definition z-string z-end z-clojure\">&quot;<\/span><\/span>\n<\/span><span class=\"z-source z-clojure\">  <span class=\"z-constant z-other z-keyword z-clojure\"><span class=\"z-punctuation z-definition z-keyword z-clojure\">:<\/span>outputs<\/span> <span class=\"z-punctuation z-section z-brackets z-begin z-clojure\">[<\/span><span class=\"z-punctuation z-section z-brackets z-end z-clojure\">]<\/span>\n<\/span><span class=\"z-source z-clojure\">  <span class=\"z-comment z-line z-clojure\"><span class=\"z-punctuation z-definition z-comment\">;<\/span> Rules don&#39;t have to be defined separately from actions.\n<\/span><\/span><span class=\"z-source z-clojure\">  <span class=\"z-constant z-other z-keyword z-clojure\"><span class=\"z-punctuation z-definition z-keyword z-clojure\">:<\/span>rule<\/span> <span class=\"z-punctuation z-section z-parens z-begin z-clojure\">(<\/span><span class=\"z-storage z-modifier z-fn z-clojure\">fn<\/span> <span class=\"z-punctuation z-section z-brackets z-begin z-clojure\">[<\/span>_vars<span class=\"z-punctuation z-section z-brackets z-end z-clojure\">]<\/span> <span class=\"z-punctuation z-section z-braces z-begin z-clojure\">{<\/span><span class=\"z-constant z-other z-keyword z-clojure\"><span class=\"z-punctuation z-definition z-keyword z-clojure\">:<\/span>command<\/span> <span class=\"z-punctuation z-section z-brackets z-begin z-clojure\">[<\/span><span class=\"z-string z-quoted z-double z-clojure\"><span class=\"z-punctuation z-definition z-string z-begin z-clojure\">&quot;<\/span>docker<span class=\"z-punctuation z-definition z-string z-end z-clojure\">&quot;<\/span><\/span> <span class=\"z-string z-quoted z-double z-clojure\"><span class=\"z-punctuation z-definition z-string z-begin z-clojure\">&quot;<\/span>build<span class=\"z-punctuation z-definition z-string z-end z-clojure\">&quot;<\/span><\/span> <span class=\"z-string z-quoted z-double z-clojure\"><span class=\"z-punctuation z-definition z-string z-begin z-clojure\">&quot;<\/span>.<span class=\"z-punctuation z-definition z-string z-end z-clojure\">&quot;<\/span><\/span><span class=\"z-punctuation z-section z-brackets z-end z-clojure\">]<\/span><span class=\"z-punctuation z-section z-braces z-end z-clojure\">}<\/span><span class=\"z-punctuation z-section z-parens z-end z-clojure\">)<\/span>\n<\/span><span class=\"z-source z-clojure\">  <span class=\"z-comment z-line z-clojure\"><span class=\"z-punctuation z-definition z-comment\">;<\/span> We choose to trust docker&#39;s own caching here.\n<\/span><\/span><span class=\"z-source z-clojure\">  <span class=\"z-comment z-line z-clojure\"><span class=\"z-punctuation z-definition z-comment\">;<\/span> We could instead use `:implicit {:ifwritten [...]}`, but we don&#39;t.\n<\/span><\/span><span class=\"z-source z-clojure\">  <span class=\"z-constant z-other z-keyword z-clojure\"><span class=\"z-punctuation z-definition z-keyword z-clojure\">:<\/span>implicit<\/span> <span class=\"z-constant z-other z-keyword z-clojure\"><span class=\"z-punctuation z-definition z-keyword z-clojure\">:<\/span>always<\/span><span class=\"z-punctuation z-section z-parens z-end z-clojure\">)<\/span>\n<\/span><span class=\"z-source z-clojure\">\n<\/span><span class=\"z-source z-clojure\"><span class=\"z-comment z-line z-clojure\"><span class=\"z-punctuation z-definition z-comment\">;<\/span> A function that filters out paths ending with a trailing slash.\n<\/span><\/span><span class=\"z-source z-clojure\"><span class=\"z-punctuation z-section z-parens z-begin z-clojure\">(<\/span><span class=\"z-storage z-modifier z-def z-clojure\">defn<\/span> <span class=\"z-entity z-name z-function z-clojure\">files<\/span> <span class=\"z-punctuation z-section z-brackets z-begin z-clojure\">[<\/span>paths<span class=\"z-punctuation z-section z-brackets z-end z-clojure\">]<\/span>\n<\/span><span class=\"z-source z-clojure\">  <span class=\"z-punctuation z-section z-parens z-begin z-clojure\">(<\/span><span class=\"z-variable z-function z-clojure\">filter<\/span> <span class=\"z-keyword z-operator z-macro z-clojure\">#<\/span><span class=\"z-punctuation z-section z-parens z-begin z-clojure\">(<\/span><span class=\"z-variable z-function z-clojure\">not<\/span> <span class=\"z-punctuation z-section z-parens z-begin z-clojure\">(<\/span><span class=\"z-variable z-function z-clojure\">str\/ends-with?<\/span> % <span class=\"z-string z-quoted z-double z-clojure\"><span class=\"z-punctuation z-definition z-string z-begin z-clojure\">&quot;<\/span>\/<span class=\"z-punctuation z-definition z-string z-end z-clojure\">&quot;<\/span><\/span><span class=\"z-punctuation z-section z-parens z-end z-clojure\">)<\/span><span class=\"z-punctuation z-section z-parens z-end z-clojure\">)<\/span> paths<span class=\"z-punctuation z-section z-parens z-end z-clojure\">)<\/span><span class=\"z-punctuation z-section z-parens z-end z-clojure\">)<\/span>\n<\/span><span class=\"z-source z-clojure\">\n<\/span><span class=\"z-source z-clojure\"><span class=\"z-comment z-line z-clojure\"><span class=\"z-punctuation z-definition z-comment\">;<\/span> Just a normal clojure variable. We&#39;ll use this as an input later.\n<\/span><\/span><span class=\"z-source z-clojure\"><span class=\"z-punctuation z-section z-parens z-begin z-clojure\">(<\/span><span class=\"z-storage z-modifier z-def z-clojure\">def<\/span> <span class=\"z-entity z-name z-function z-clojure\">tarfile<\/span> <span class=\"z-punctuation z-section z-braces z-begin z-clojure\">{<\/span><span class=\"z-constant z-other z-keyword z-clojure\"><span class=\"z-punctuation z-definition z-keyword z-clojure\">:<\/span>ifwritten<\/span> <span class=\"z-punctuation z-section z-brackets z-begin z-clojure\">[<\/span><span class=\"z-string z-quoted z-double z-clojure\"><span class=\"z-punctuation z-definition z-string z-begin z-clojure\">&quot;<\/span>example.tar<span class=\"z-punctuation z-definition z-string z-end z-clojure\">&quot;<\/span><\/span><span class=\"z-punctuation z-section z-brackets z-end z-clojure\">]<\/span><span class=\"z-punctuation z-section z-braces z-end z-clojure\">}<\/span><span class=\"z-punctuation z-section z-parens z-end z-clojure\">)<\/span>\n<\/span><span class=\"z-source z-clojure\">\n<\/span><span class=\"z-source z-clojure\"><span class=\"z-comment z-line z-clojure\"><span class=\"z-punctuation z-definition z-comment\">;<\/span> This rule lists all files (excluding directories) inside of a tarfile.\n<\/span><\/span><span class=\"z-source z-clojure\"><span class=\"z-punctuation z-section z-parens z-begin z-clojure\">(<\/span><span class=\"z-storage z-modifier z-def z-clojure\">defn<\/span> <span class=\"z-entity z-name z-function z-clojure\">list-tar-files<\/span> <span class=\"z-punctuation z-section z-brackets z-begin z-clojure\">[<\/span>vars<span class=\"z-punctuation z-section z-brackets z-end z-clojure\">]<\/span>\n<\/span><span class=\"z-source z-clojure\">  <span class=\"z-comment z-line z-clojure\"><span class=\"z-punctuation z-definition z-comment\">;<\/span> `:rule` can be a list to run multiple rules in a row within the same action.\n<\/span><\/span><span class=\"z-source z-clojure\">  <span class=\"z-punctuation z-section z-brackets z-begin z-clojure\">[<\/span><span class=\"z-punctuation z-section z-braces z-begin z-clojure\">{<\/span><span class=\"z-constant z-other z-keyword z-clojure\"><span class=\"z-punctuation z-definition z-keyword z-clojure\">:<\/span>command<\/span> <span class=\"z-punctuation z-section z-brackets z-begin z-clojure\">[<\/span><span class=\"z-string z-quoted z-double z-clojure\"><span class=\"z-punctuation z-definition z-string z-begin z-clojure\">&quot;<\/span>tar<span class=\"z-punctuation z-definition z-string z-end z-clojure\">&quot;<\/span><\/span> <span class=\"z-string z-quoted z-double z-clojure\"><span class=\"z-punctuation z-definition z-string z-begin z-clojure\">&quot;<\/span>-tf<span class=\"z-punctuation z-definition z-string z-end z-clojure\">&quot;<\/span><\/span> <span class=\"z-punctuation z-section z-parens z-begin z-clojure\">(<\/span><span class=\"z-constant z-other z-keyword z-clojure\"><span class=\"z-punctuation z-definition z-keyword z-clojure\">:<\/span>inputs<\/span> vars<span class=\"z-punctuation z-section z-parens z-end z-clojure\">)<\/span><span class=\"z-punctuation z-section z-brackets z-end z-clojure\">]<\/span>\n<\/span><span class=\"z-source z-clojure\">    <span class=\"z-comment z-line z-clojure\"><span class=\"z-punctuation z-definition z-comment\">;<\/span> This is like a shell redirect, but saves the output to a string in memory instead of to a file.\n<\/span><\/span><span class=\"z-source z-clojure\">    <span class=\"z-comment z-line z-clojure\"><span class=\"z-punctuation z-definition z-comment\">;<\/span> The string will be passed to the next rule in the list.\n<\/span><\/span><span class=\"z-source z-clojure\">    <span class=\"z-constant z-other z-keyword z-clojure\"><span class=\"z-punctuation z-definition z-keyword z-clojure\">:<\/span>stdout<\/span> <span class=\"z-constant z-other z-keyword z-clojure\"><span class=\"z-punctuation z-definition z-keyword z-clojure\">:<\/span>string<\/span><span class=\"z-punctuation z-section z-braces z-end z-clojure\">}<\/span>\n<\/span><span class=\"z-source z-clojure\">   <span class=\"z-comment z-line z-clojure\"><span class=\"z-punctuation z-definition z-comment\">;<\/span> This filters for files, then passes the string to the next rule.\n<\/span><\/span><span class=\"z-source z-clojure\">   <span class=\"z-punctuation z-section z-braces z-begin z-clojure\">{<\/span><span class=\"z-constant z-other z-keyword z-clojure\"><span class=\"z-punctuation z-definition z-keyword z-clojure\">:<\/span>transformer<\/span> <span class=\"z-punctuation z-section z-parens z-begin z-clojure\">(<\/span><span class=\"z-storage z-modifier z-fn z-clojure\">fn<\/span> <span class=\"z-punctuation z-section z-brackets z-begin z-clojure\">[<\/span>paths<span class=\"z-punctuation z-section z-brackets z-end z-clojure\">]<\/span> <span class=\"z-punctuation z-section z-parens z-begin z-clojure\">(<\/span><span class=\"z-variable z-function z-clojure\">str\/join<\/span> <span class=\"z-string z-quoted z-double z-clojure\"><span class=\"z-punctuation z-definition z-string z-begin z-clojure\">&quot;<\/span><span class=\"z-constant z-character z-escape z-clojure\">\\n<\/span><span class=\"z-punctuation z-definition z-string z-end z-clojure\">&quot;<\/span><\/span> <span class=\"z-punctuation z-section z-parens z-begin z-clojure\">(<\/span><span class=\"z-variable z-function z-clojure\">files<\/span> <span class=\"z-punctuation z-section z-parens z-begin z-clojure\">(<\/span><span class=\"z-variable z-function z-clojure\">str\/split<\/span> <span class=\"z-keyword z-operator z-macro z-clojure\">#<\/span><span class=\"z-string z-regexp z-clojure\"><span class=\"z-punctuation z-definition z-string z-begin z-clojure\">&quot;<\/span><span class=\"z-constant z-character z-escape z-regexp\">\\n<\/span><span class=\"z-punctuation z-definition z-string z-end z-clojure\">&quot;<\/span><\/span> paths<span class=\"z-punctuation z-section z-parens z-end z-clojure\">)<\/span><span class=\"z-punctuation z-section z-parens z-end z-clojure\">)<\/span><span class=\"z-punctuation z-section z-parens z-end z-clojure\">)<\/span><span class=\"z-punctuation z-section z-parens z-end z-clojure\">)<\/span><span class=\"z-punctuation z-section z-braces z-end z-clojure\">}<\/span>\n<\/span><span class=\"z-source z-clojure\">   <span class=\"z-comment z-line z-clojure\"><span class=\"z-punctuation z-definition z-comment\">;<\/span> Finally, write the transformed data out to disk.\n<\/span><\/span><span class=\"z-source z-clojure\">   <span class=\"z-punctuation z-section z-braces z-begin z-clojure\">{<\/span><span class=\"z-constant z-other z-keyword z-clojure\"><span class=\"z-punctuation z-definition z-keyword z-clojure\">:<\/span>write<\/span> <span class=\"z-punctuation z-section z-parens z-begin z-clojure\">(<\/span><span class=\"z-constant z-other z-keyword z-clojure\"><span class=\"z-punctuation z-definition z-keyword z-clojure\">:<\/span>outputs<\/span> vars<span class=\"z-punctuation z-section z-parens z-end z-clojure\">)<\/span><span class=\"z-punctuation z-section z-braces z-end z-clojure\">}<\/span><span class=\"z-punctuation z-section z-brackets z-end z-clojure\">]<\/span><span class=\"z-punctuation z-section z-parens z-end z-clojure\">)<\/span>\n<\/span><span class=\"z-source z-clojure\">\n<\/span><span class=\"z-source z-clojure\"><span class=\"z-comment z-line z-clojure\"><span class=\"z-punctuation z-definition z-comment\">;<\/span> Generate a list of all files that will be created by extracting the tarfile.\n<\/span><\/span><span class=\"z-source z-clojure\"><span class=\"z-punctuation z-section z-parens z-begin z-clojure\">(<\/span><span class=\"z-variable z-function z-clojure\">action<\/span> <span class=\"z-constant z-other z-keyword z-clojure\"><span class=\"z-punctuation z-definition z-keyword z-clojure\">:<\/span>outputs<\/span> <span class=\"z-punctuation z-section z-brackets z-begin z-clojure\">[<\/span><span class=\"z-string z-quoted z-double z-clojure\"><span class=\"z-punctuation z-definition z-string z-begin z-clojure\">&quot;<\/span>example.tar.dd<span class=\"z-punctuation z-definition z-string z-end z-clojure\">&quot;<\/span><\/span><span class=\"z-punctuation z-section z-brackets z-end z-clojure\">]<\/span>\n<\/span><span class=\"z-source z-clojure\">  <span class=\"z-constant z-other z-keyword z-clojure\"><span class=\"z-punctuation z-definition z-keyword z-clojure\">:<\/span>rule<\/span> list-tar-files\n<\/span><span class=\"z-source z-clojure\">  <span class=\"z-constant z-other z-keyword z-clojure\"><span class=\"z-punctuation z-definition z-keyword z-clojure\">:<\/span>inputs<\/span> tarfile<span class=\"z-punctuation z-section z-parens z-end z-clojure\">)<\/span>\n<\/span><span class=\"z-source z-clojure\">\n<\/span><span class=\"z-source z-clojure\"><span class=\"z-comment z-line z-clojure\"><span class=\"z-punctuation z-definition z-comment\">;<\/span> `dynout` returns a future which `action` will resolve.\n<\/span><\/span><span class=\"z-source z-clojure\"><span class=\"z-punctuation z-section z-parens z-begin z-clojure\">(<\/span><span class=\"z-storage z-modifier z-def z-clojure\">def<\/span> <span class=\"z-entity z-name z-function z-clojure\">tar-outputs<\/span>\n<\/span><span class=\"z-source z-clojure\">  <span class=\"z-punctuation z-section z-parens z-begin z-clojure\">(<\/span><span class=\"z-variable z-function z-clojure\">dynout<\/span>\n<\/span><span class=\"z-source z-clojure\">    <span class=\"z-comment z-line z-clojure\"><span class=\"z-punctuation z-definition z-comment\">;<\/span> Recalculate the dependencies whenever our .dd file changes.\n<\/span><\/span><span class=\"z-source z-clojure\">    <span class=\"z-comment z-line z-clojure\"><span class=\"z-punctuation z-definition z-comment\">;<\/span> Note that unlike `ifwritten`, `ifchanged` calculates a checksum, it doesn&#39;t use file metadata.\n<\/span><\/span><span class=\"z-source z-clojure\">    <span class=\"z-constant z-other z-keyword z-clojure\"><span class=\"z-punctuation z-definition z-keyword z-clojure\">:<\/span>inputs<\/span> <span class=\"z-punctuation z-section z-braces z-begin z-clojure\">{<\/span><span class=\"z-constant z-other z-keyword z-clojure\"><span class=\"z-punctuation z-definition z-keyword z-clojure\">:<\/span>ifchanged<\/span> <span class=\"z-punctuation z-section z-brackets z-begin z-clojure\">[<\/span><span class=\"z-string z-quoted z-double z-clojure\"><span class=\"z-punctuation z-definition z-string z-begin z-clojure\">&quot;<\/span>example.tar.dd<span class=\"z-punctuation z-definition z-string z-end z-clojure\">&quot;<\/span><\/span><span class=\"z-punctuation z-section z-brackets z-end z-clojure\">]<\/span><span class=\"z-punctuation z-section z-braces z-end z-clojure\">}<\/span>\n<\/span><span class=\"z-source z-clojure\">    <span class=\"z-constant z-other z-keyword z-clojure\"><span class=\"z-punctuation z-definition z-keyword z-clojure\">:<\/span>rule<\/span> <span class=\"z-punctuation z-section z-parens z-begin z-clojure\">(<\/span><span class=\"z-storage z-modifier z-fn z-clojure\">fn<\/span> <span class=\"z-punctuation z-section z-brackets z-begin z-clojure\">[<\/span>vars<span class=\"z-punctuation z-section z-brackets z-end z-clojure\">]<\/span> <span class=\"z-punctuation z-section z-brackets z-begin z-clojure\">[<\/span><span class=\"z-punctuation z-section z-braces z-begin z-clojure\">{<\/span><span class=\"z-constant z-other z-keyword z-clojure\"><span class=\"z-punctuation z-definition z-keyword z-clojure\">:<\/span>read<\/span> <span class=\"z-punctuation z-section z-parens z-begin z-clojure\">(<\/span><span class=\"z-constant z-other z-keyword z-clojure\"><span class=\"z-punctuation z-definition z-keyword z-clojure\">:<\/span>inputs<\/span> vars<span class=\"z-punctuation z-section z-parens z-end z-clojure\">)<\/span><span class=\"z-punctuation z-section z-braces z-end z-clojure\">}<\/span>\n<\/span><span class=\"z-source z-clojure\">                      <span class=\"z-comment z-line z-clojure\"><span class=\"z-punctuation z-definition z-comment\">;<\/span> Our future resolves to a list of file paths.\n<\/span><\/span><span class=\"z-source z-clojure\">                      <span class=\"z-punctuation z-section z-braces z-begin z-clojure\">{<\/span><span class=\"z-constant z-other z-keyword z-clojure\"><span class=\"z-punctuation z-definition z-keyword z-clojure\">:<\/span>transformer<\/span> <span class=\"z-keyword z-operator z-macro z-clojure\">#<\/span><span class=\"z-punctuation z-section z-parens z-begin z-clojure\">(<\/span><span class=\"z-variable z-function z-clojure\">str\/split<\/span> % <span class=\"z-keyword z-operator z-macro z-clojure\">#<\/span><span class=\"z-string z-regexp z-clojure\"><span class=\"z-punctuation z-definition z-string z-begin z-clojure\">&quot;<\/span><span class=\"z-constant z-character z-escape z-regexp\">\\n<\/span><span class=\"z-punctuation z-definition z-string z-end z-clojure\">&quot;<\/span><\/span><span class=\"z-punctuation z-section z-parens z-end z-clojure\">)<\/span><span class=\"z-punctuation z-section z-braces z-end z-clojure\">}<\/span><span class=\"z-punctuation z-section z-brackets z-end z-clojure\">]<\/span><span class=\"z-punctuation z-section z-parens z-end z-clojure\">)<\/span><span class=\"z-punctuation z-section z-parens z-end z-clojure\">)<\/span><span class=\"z-punctuation z-section z-parens z-end z-clojure\">)<\/span>\n<\/span><span class=\"z-source z-clojure\">\n<\/span><span class=\"z-source z-clojure\"><span class=\"z-comment z-line z-clojure\"><span class=\"z-punctuation z-definition z-comment\">;<\/span> Dynamically add the outputs from the tarfile to the action graph.\n<\/span><\/span><span class=\"z-source z-clojure\"><span class=\"z-comment z-line z-clojure\"><span class=\"z-punctuation z-definition z-comment\">;<\/span> This schedules the action to run only after the `dynout` future is resolved.\n<\/span><\/span><span class=\"z-source z-clojure\"><span class=\"z-punctuation z-section z-parens z-begin z-clojure\">(<\/span><span class=\"z-variable z-function z-clojure\">action<\/span> <span class=\"z-constant z-other z-keyword z-clojure\"><span class=\"z-punctuation z-definition z-keyword z-clojure\">:<\/span>outputs<\/span> tar-outputs\n<\/span><span class=\"z-source z-clojure\">  <span class=\"z-constant z-other z-keyword z-clojure\"><span class=\"z-punctuation z-definition z-keyword z-clojure\">:<\/span>inputs<\/span> tarfile\n<\/span><span class=\"z-source z-clojure\">  <span class=\"z-constant z-other z-keyword z-clojure\"><span class=\"z-punctuation z-definition z-keyword z-clojure\">:<\/span>rule<\/span> <span class=\"z-punctuation z-section z-parens z-begin z-clojure\">(<\/span><span class=\"z-storage z-modifier z-fn z-clojure\">fn<\/span> <span class=\"z-punctuation z-section z-brackets z-begin z-clojure\">[<\/span>vars<span class=\"z-punctuation z-section z-brackets z-end z-clojure\">]<\/span> <span class=\"z-punctuation z-section z-braces z-begin z-clojure\">{<\/span><span class=\"z-constant z-other z-keyword z-clojure\"><span class=\"z-punctuation z-definition z-keyword z-clojure\">:<\/span>command<\/span> <span class=\"z-punctuation z-section z-brackets z-begin z-clojure\">[<\/span><span class=\"z-string z-quoted z-double z-clojure\"><span class=\"z-punctuation z-definition z-string z-begin z-clojure\">&quot;<\/span>tar<span class=\"z-punctuation z-definition z-string z-end z-clojure\">&quot;<\/span><\/span> <span class=\"z-string z-quoted z-double z-clojure\"><span class=\"z-punctuation z-definition z-string z-begin z-clojure\">&quot;<\/span>-xf<span class=\"z-punctuation z-definition z-string z-end z-clojure\">&quot;<\/span><\/span> <span class=\"z-punctuation z-section z-parens z-begin z-clojure\">(<\/span><span class=\"z-constant z-other z-keyword z-clojure\"><span class=\"z-punctuation z-definition z-keyword z-clojure\">:<\/span>inputs<\/span> vars<span class=\"z-punctuation z-section z-parens z-end z-clojure\">)<\/span><span class=\"z-punctuation z-section z-brackets z-end z-clojure\">]<\/span><span class=\"z-punctuation z-section z-braces z-end z-clojure\">}<\/span><span class=\"z-punctuation z-section z-parens z-end z-clojure\">)<\/span><span class=\"z-punctuation z-section z-parens z-end z-clojure\">)<\/span>\n<\/span><span class=\"z-source z-clojure\">\n<\/span><span class=\"z-source z-clojure\"><span class=\"z-comment z-line z-clojure\"><span class=\"z-punctuation z-definition z-comment\">;<\/span> Group together many different actions.\n<\/span><\/span><span class=\"z-source z-clojure\"><span class=\"z-comment z-line z-clojure\"><span class=\"z-punctuation z-definition z-comment\">;<\/span> This can be called from the command line as if it were its own action.\n<\/span><\/span><span class=\"z-source z-clojure\"><span class=\"z-punctuation z-section z-parens z-begin z-clojure\">(<\/span><span class=\"z-variable z-function z-clojure\">group<\/span> <span class=\"z-string z-quoted z-double z-clojure\"><span class=\"z-punctuation z-definition z-string z-begin z-clojure\">&quot;<\/span>all<span class=\"z-punctuation z-definition z-string z-end z-clojure\">&quot;<\/span><\/span> <span class=\"z-punctuation z-section z-parens z-begin z-clojure\">(<\/span><span class=\"z-variable z-function z-clojure\">concat<\/span> <span class=\"z-punctuation z-section z-brackets z-begin z-clojure\">[<\/span><span class=\"z-string z-quoted z-double z-clojure\"><span class=\"z-punctuation z-definition z-string z-begin z-clojure\">&quot;<\/span>main<span class=\"z-punctuation z-definition z-string z-end z-clojure\">&quot;<\/span><\/span> docker-image<span class=\"z-punctuation z-section z-brackets z-end z-clojure\">]<\/span> tar-outputs<span class=\"z-punctuation z-section z-parens z-end z-clojure\">)<\/span><span class=\"z-punctuation z-section z-parens z-end z-clojure\">)<\/span>\n<\/span><span class=\"z-source z-clojure\">\n<\/span><span class=\"z-source z-clojure\"><span class=\"z-comment z-line z-clojure\"><span class=\"z-punctuation z-definition z-comment\">;<\/span> Specify that the configure script shold rerun whenever a `.c` file is created or deleted.\n<\/span><\/span><span class=\"z-source z-clojure\"><span class=\"z-punctuation z-section z-parens z-begin z-clojure\">(<\/span><span class=\"z-storage z-modifier z-def z-clojure\">defn<\/span> <span class=\"z-entity z-name z-function z-clojure\">should-rerun<\/span> <span class=\"z-punctuation z-section z-brackets z-begin z-clojure\">[<\/span>event<span class=\"z-punctuation z-section z-brackets z-end z-clojure\">]<\/span>\n<\/span><span class=\"z-source z-clojure\">  <span class=\"z-punctuation z-section z-parens z-begin z-clojure\">(<\/span><span class=\"z-variable z-function z-clojure\">and<\/span> <span class=\"z-punctuation z-section z-parens z-begin z-clojure\">(<\/span><span class=\"z-variable z-function z-clojure\">ends-with?<\/span> <span class=\"z-punctuation z-section z-parens z-begin z-clojure\">(<\/span><span class=\"z-constant z-other z-keyword z-clojure\"><span class=\"z-punctuation z-definition z-keyword z-clojure\">:<\/span>path<\/span> event<span class=\"z-punctuation z-section z-parens z-end z-clojure\">)<\/span> <span class=\"z-string z-quoted z-double z-clojure\"><span class=\"z-punctuation z-definition z-string z-begin z-clojure\">&quot;<\/span>.c<span class=\"z-punctuation z-definition z-string z-end z-clojure\">&quot;<\/span><\/span><span class=\"z-punctuation z-section z-parens z-end z-clojure\">)<\/span>\n<\/span><span class=\"z-source z-clojure\">       <span class=\"z-punctuation z-section z-parens z-begin z-clojure\">(<\/span><span class=\"z-variable z-function z-clojure\">not=<\/span> <span class=\"z-constant z-other z-keyword z-clojure\"><span class=\"z-punctuation z-definition z-keyword z-clojure\">:<\/span>modified<\/span> <span class=\"z-punctuation z-section z-parens z-begin z-clojure\">(<\/span><span class=\"z-constant z-other z-keyword z-clojure\"><span class=\"z-punctuation z-definition z-keyword z-clojure\">:<\/span>kind<\/span> event<span class=\"z-punctuation z-section z-parens z-end z-clojure\">)<\/span><span class=\"z-punctuation z-section z-parens z-end z-clojure\">)<\/span><span class=\"z-punctuation z-section z-parens z-end z-clojure\">)<\/span><span class=\"z-punctuation z-section z-parens z-end z-clojure\">)<\/span>\n<\/span><span class=\"z-source z-clojure\">\n<\/span><span class=\"z-source z-clojure\"><span class=\"z-punctuation z-section z-parens z-begin z-clojure\">(<\/span><span class=\"z-variable z-function z-clojure\">action<\/span> <span class=\"z-constant z-other z-keyword z-clojure\"><span class=\"z-punctuation z-definition z-keyword z-clojure\">:<\/span>outputs<\/span> <span class=\"z-punctuation z-section z-brackets z-begin z-clojure\">[<\/span><span class=\"z-string z-quoted z-double z-clojure\"><span class=\"z-punctuation z-definition z-string z-begin z-clojure\">&quot;<\/span>build.magma<span class=\"z-punctuation z-definition z-string z-end z-clojure\">&quot;<\/span><\/span><span class=\"z-punctuation z-section z-brackets z-end z-clojure\">]<\/span>\n<\/span><span class=\"z-source z-clojure\">  <span class=\"z-constant z-other z-keyword z-clojure\"><span class=\"z-punctuation z-definition z-keyword z-clojure\">:<\/span>rule<\/span> <span class=\"z-punctuation z-section z-braces z-begin z-clojure\">{<\/span><span class=\"z-constant z-other z-keyword z-clojure\"><span class=\"z-punctuation z-definition z-keyword z-clojure\">:<\/span>command<\/span> <span class=\"z-punctuation z-section z-brackets z-begin z-clojure\">[<\/span><span class=\"z-string z-quoted z-double z-clojure\"><span class=\"z-punctuation z-definition z-string z-begin z-clojure\">&quot;<\/span>python<span class=\"z-punctuation z-definition z-string z-end z-clojure\">&quot;<\/span><\/span> <span class=\"z-string z-quoted z-double z-clojure\"><span class=\"z-punctuation z-definition z-string z-begin z-clojure\">&quot;<\/span>configure.py<span class=\"z-punctuation z-definition z-string z-end z-clojure\">&quot;<\/span><\/span><span class=\"z-punctuation z-section z-brackets z-end z-clojure\">]<\/span><span class=\"z-punctuation z-section z-braces z-end z-clojure\">}<\/span>\n<\/span><span class=\"z-source z-clojure\">  <span class=\"z-comment z-line z-clojure\"><span class=\"z-punctuation z-definition z-comment\">;<\/span> Indicate that these outputs shouldn&#39;t be deleted by `clean` rules.\n<\/span><\/span><span class=\"z-source z-clojure\">  <span class=\"z-constant z-other z-keyword z-clojure\"><span class=\"z-punctuation z-definition z-keyword z-clojure\">:<\/span>generator<\/span> <span class=\"z-constant z-language z-clojure\">true<\/span>\n<\/span><span class=\"z-source z-clojure\">  <span class=\"z-constant z-other z-keyword z-clojure\"><span class=\"z-punctuation z-definition z-keyword z-clojure\">:<\/span>env<\/span> <span class=\"z-punctuation z-section z-brackets z-begin z-clojure\">[<\/span><span class=\"z-string z-quoted z-double z-clojure\"><span class=\"z-punctuation z-definition z-string z-begin z-clojure\">&quot;<\/span>CFLAGS<span class=\"z-punctuation z-definition z-string z-end z-clojure\">&quot;<\/span><\/span><span class=\"z-punctuation z-comma z-clojure\"><span class=\"z-comment z-punctuation z-comma z-clojure\">,<\/span><\/span> <span class=\"z-string z-quoted z-double z-clojure\"><span class=\"z-punctuation z-definition z-string z-begin z-clojure\">&quot;<\/span>LDFLAGS<span class=\"z-punctuation z-definition z-string z-end z-clojure\">&quot;<\/span><\/span><span class=\"z-punctuation z-section z-brackets z-end z-clojure\">]<\/span>\n<\/span><span class=\"z-source z-clojure\">  <span class=\"z-comment z-line z-clojure\"><span class=\"z-punctuation z-definition z-comment\">;<\/span> Register a function that will run on each change to the current directory\n<\/span><\/span><span class=\"z-source z-clojure\">  <span class=\"z-comment z-line z-clojure\"><span class=\"z-punctuation z-definition z-comment\">;<\/span> and instruct whether or not the input should cause a rebuild.\n<\/span><\/span><span class=\"z-source z-clojure\">  <span class=\"z-constant z-other z-keyword z-clojure\"><span class=\"z-punctuation z-definition z-keyword z-clojure\">:<\/span>inputs<\/span> <span class=\"z-punctuation z-section z-braces z-begin z-clojure\">{<\/span><span class=\"z-constant z-other z-keyword z-clojure\"><span class=\"z-punctuation z-definition z-keyword z-clojure\">:<\/span>watch<\/span> <span class=\"z-punctuation z-section z-braces z-begin z-clojure\">{<\/span><span class=\"z-string z-quoted z-double z-clojure\"><span class=\"z-punctuation z-definition z-string z-begin z-clojure\">&quot;<\/span>.<span class=\"z-punctuation z-definition z-string z-end z-clojure\">&quot;<\/span><\/span> should-rerun<span class=\"z-punctuation z-section z-braces z-end z-clojure\">}<\/span><span class=\"z-punctuation z-section z-braces z-end z-clojure\">}<\/span><span class=\"z-punctuation z-section z-parens z-end z-clojure\">)<\/span>\n<\/span><\/code><\/pre>\n<p>Note some things about Magma:<\/p>\n<ul>\n<li>Command spawning is specified as an array <sup class=\"footnote-reference\" id=\"fr-12-1\"><a href=\"#fn-12\">7<\/a><\/sup>. No more dependency on shell quoting rules. If people want shell scripts they can put that in their configure script.<\/li>\n<li>Redirecting stdout no longer requires bash syntax, it's supported natively with the <code>:stdout<\/code> parameter of <code>:rule<\/code>.<\/li>\n<li>Build parameters can be referred to in rules through the <code>vars<\/code> argument.<\/li>\n<li><code>dynout<\/code> is a <a href=\"https:\/\/wiki.haskell.org\/Thunk\">thunk<\/a>; it only registers an intent to add edges in the future, it does not eagerly require <code>example.tar.dd<\/code> to exist.<\/li>\n<li>Our <code>watch<\/code> input edge is generalized and can apply to any rule, not just to the configure step. It executes when a file is modified (or if the tool doesn\u2019t support file watching, on each file in the calculated diff in the next tool invocation).<\/li>\n<li>Our <code>watch<\/code> edge provides the file event type, but not the file contents. This allows ronin to automatically map <code>true<\/code> results to one of the three edge kinds: <code>:ifwritten<\/code>, <code>:ifcreated<\/code>, <code>:ifdeleted<\/code>. <code>:always<\/code> and <code>:ifchanged<\/code> are not available through this API.<\/li>\n<li>We naturally distinguish between \u201cphony targets\u201d and files because the former are <code>group<\/code>s and the latter are <code>action<\/code>s. No more accidentally failing to build if an <code>all<\/code> file is created. <sup class=\"footnote-reference\" id=\"fr-13-1\"><a href=\"#fn-13\">8<\/a><\/sup><\/li>\n<li>We naturally distinguish between \u201cgroups of targets\u201d and \u201ccommands that always need to be rerun\u201d; the latter just uses <code>:inputs :always<\/code>.<\/li>\n<li>Data can be transformed in memory using clojure functions without needing a separate process invocation. No more need to use <code>sed<\/code> in your build system.<\/li>\n<\/ul>\n<!--\n- Our `group` cannot actually be used in a `:inputs` parameter because it doesn\u2019t specify the kind of edge (created\/modified\/deleted). Our mini-clojure provides other abstractions, like variables; arrays; and `map`, that render this unnecessary.\n-->\n<h3 id=\"did-you-just-reinvent-starlark\">\"did you just reinvent <a href=\"https:\/\/starlark-lang.org\/\">starlark<\/a>?\"<a class=\"zola-anchor\" href=\"#did-you-just-reinvent-starlark\" aria-label=\"Anchor link for: did-you-just-reinvent-starlark\"><\/a>\n<\/h3>\n<p>Kinda. Magma itself has a lot in common with Starlark: it's deterministic, hermetic, immutable, and can be evaluated in parallel.<\/p>\n<p>The main difference between the languages themselves is that Clojure has <code>:keywords<\/code> (equivalent to sympy symbolic variables) and Python doesn't. Some of these could be rewritten to keyword arguments, and others could be rewritten to structs, or string keys for a hashmap, or enums; but I'm not sure how much benefit there is to literally using Starlark when these files are being generated by a configure step in any case. Probably it's possible to make a 1-1 mapping between the two in any case.<\/p>\n<h3 id=\"runners\">runners<a class=\"zola-anchor\" href=\"#runners\" aria-label=\"Anchor link for: runners\"><\/a>\n<\/h3>\n<p>Buck2 has support for <a href=\"https:\/\/buck2.build\/docs\/api\/build\/RunInfo\/\"><code>RunInfo<\/code><\/a> metadata that describes how to execute a built artifact. I think this is really interesting; <code>cargo run --bin xyz -- args<\/code> is a much nicer interface than <code>make ARGS='args' xyz<\/code>, partly because of shell quoting and word splitting issues, and partly just because it's more discoverable.<\/p>\n<p>I don't have a clean idea for how to fit this into a serialization layer. \"Don't put it there and use a <code>Justfile<\/code> instead\" <em>works<\/em>, but makes it hard to do things like allow the build graph to say that an artifact needs <code>LD_LIBRARY_PATH<\/code> set or something like that, you end up duplicating the info in both files. Perhaps one option could be to attach a <code>:run<\/code> key\/value pair to <code>action<\/code>s.<\/p>\n<h3 id=\"are-you-just-piling-a-bunch-of-features-on-top-of-ninja\">\"are you just piling a bunch of features on top of ninja?\"<a class=\"zola-anchor\" href=\"#are-you-just-piling-a-bunch-of-features-on-top-of-ninja\" aria-label=\"Anchor link for: are-you-just-piling-a-bunch-of-features-on-top-of-ninja\"><\/a>\n<\/h3>\n<p>Well, yes and no. Yes, in that this has basically all the features of ninja and then some.\nBut no, because the rules here are all carefully constrained to avoid needing to do expensive file I\/O to load the build graph.\nThe most expensive new feature is <code>watch<\/code>, and it's intended to avoid an even more expensive step (rerunning the configuration step).\nIt's also limited to changed files; it can't do arbitrary globbing on the contents of the directory the way that Make pattern rules can.<\/p>\n<p>Note that this also removes some features in ninja: shell commands are gone, process spawning is much less ambiguous, <code>dyndep<\/code> files are no longer parsed automatically.\nAnd because this embeds a clojure interpreter, many things that were hard-coded in ninja can instead be library functions: <code>rule<\/code>, response files, <code>msvc_deps_prefix<\/code>, <code>in_newline<\/code>.<\/p>\n<h2 id=\"next-steps\">next steps<a class=\"zola-anchor\" href=\"#next-steps\" aria-label=\"Anchor link for: next-steps\"><\/a>\n<\/h2>\n<p>In this post, we have learned some downsides of Make and Ninja's build file formats, sketched out how they could possibly be fixed, and designed a language called Magma that has those characteristics. In the next post, I'll describe the features and design of a tool that evaluates and queries this language.<\/p>\n<section class=\"footnotes\">\n<ol class=\"footnotes-list\">\n<li id=\"fn-10\">\n<p>see e.g. <a href=\"https:\/\/github.com\/facebook\/buck2\/issues\/976#issuecomment-3007201092\">this description<\/a> of how it works in buck2 <a href=\"#fr-10-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-2\">\n<p>at least a basic version\u2014although several of the features of GNU Make get rather complicated. <a href=\"#fr-2-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-4\">\n<p><code>ninja_syntax.py<\/code> is <a href=\"https:\/\/github.com\/ninja-build\/ninja\/blob\/231db65ccf5427b16ff85b3a390a663f3c8a479f\/misc\/ninja_syntax.py\">https:\/\/github.com\/ninja-build\/ninja\/blob\/231db65ccf5427b16ff85b3a390a663f3c8a479f\/misc\/ninja_syntax.py<\/a>. <a href=\"#fr-4-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-14\">\n<p>technically these aren't true monadic builds because they're constrained a lot more than e.g. Shake rules, they can't fabricate new rules from whole cloth. but they still allow you to add more outputs to the graph at runtime. <a href=\"#fr-14-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-6\">\n<p>This goes all the way around <a href=\"https:\/\/mikehadlow.blogspot.com\/2012\/05\/configuration-complexity-clock.html\">the configuration complexity clock<\/a> and skips the \"DSL\" phase to simply give you a real language. <a href=\"#fr-6-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-7\">\n<p>This is totally based and not at all a terrible idea. <a href=\"#fr-7-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-12\">\n<p>This has a whole bunch of problems on Windows, where arguments are passed as a single string instead of an array, and each command has to reimplement its own parsing. But it will work \u201cmost\u201d of the time, and at least avoids having to deal with Powershell or CMD quoting. <a href=\"#fr-12-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-13\">\n<p>To make it possible to distinguish the two on the command line, <code>:all<\/code> could unambiguously refer to the group, like in Bazel. <a href=\"#fr-13-1\">\u21a9<\/a><\/p>\n<\/li>\n<\/ol>\n<\/section>\n"},{"title":"negative build dependencies","published":"2025-12-03T00:00:00+00:00","updated":"2025-12-03T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/negative-build-dependencies\/"}},"id":"https:\/\/jyn.dev\/negative-build-dependencies\/","summary":"A build system needs to know not just which files exist, but which shouldn't exist.","content":"<p>This post is part 2\/4 of <a href=\"\/four-posts-about-build-systems\/\">a series about build systems<\/a>.\nThe next post is <a href=\"\/i-want-a-better-action-graph-serialization\/\">I want a better action graph serialization<\/a>.<\/p>\n<hr \/>\n<p>This post is about a limitation of the dependencies you can express in a build file.\nIt uses Ninja just because it's simple and I'm very familiar with it, but the problem exists in most build systems.<\/p>\n<p>Say you have a C project with two different include paths: <code>dependency\/include<\/code> and <code>src<\/code>:<\/p>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">.\n<\/span><span class=\"z-text z-plain\">\u251c\u2500\u2500 build.ninja\n<\/span><span class=\"z-text z-plain\">\u251c\u2500\u2500 dependency\n<\/span><span class=\"z-text z-plain\">\u2502\u00a0\u00a0 \u2514\u2500\u2500 include\n<\/span><span class=\"z-text z-plain\">\u2502\u00a0\u00a0     \u2514\u2500\u2500 interface.h\n<\/span><span class=\"z-text z-plain\">\u2514\u2500\u2500 src\n<\/span><span class=\"z-text z-plain\">    \u2514\u2500\u2500 main.c\n<\/span><\/code><\/pre>\n<p>and the following build.ninja:<sup class=\"footnote-reference\" id=\"fr-depfiles-1\"><a href=\"#fn-depfiles\">1<\/a><\/sup><\/p>\n<pre data-lang=\"ninja\" class=\"language-ninja z-code\"><code class=\"language-ninja\" data-lang=\"ninja\"><span class=\"z-text z-plain\">rule cc\n<\/span><span class=\"z-text z-plain\">  command = cc -I dependency\/include -I src $in -o $out\n<\/span><span class=\"z-text z-plain\">\n<\/span><span class=\"z-text z-plain\">build src\/main: cc src\/main.c | dependency\/include\/interface.h\n<\/span><\/code><\/pre>\n<aside role=note class=note-container><div class=note-content>\n<p>In ninja, <code>| file<\/code> means an \u201cimplicit dependency\u201d, i.e. it causes a rebuild but does not get bound to <code>$in<\/code>.<\/p>\n<\/div><\/aside>\n<p>and some small implementations, just so we can see that it actually works:<\/p>\n<pre data-lang=\"c\" class=\"language-c z-code\"><code class=\"language-c\" data-lang=\"c\"><span class=\"z-source z-c\"><span class=\"z-comment z-line z-double-slash z-c\"><span class=\"z-punctuation z-definition z-comment z-c\">\/\/<\/span> src\/main.c\n<\/span><\/span><span class=\"z-source z-c\"><span class=\"z-meta z-preprocessor z-include z-c\"><span class=\"z-keyword z-control z-import z-include z-c\">#include<\/span> <span class=\"z-string z-quoted z-other z-lt-gt z-include z-c\"><span class=\"z-punctuation z-definition z-string z-begin z-c\">&lt;<\/span>stdio.h<span class=\"z-punctuation z-definition z-string z-end z-c\">&gt;<\/span><\/span>\n<\/span><\/span><span class=\"z-source z-c\"><span class=\"z-meta z-preprocessor z-include z-c\"><span class=\"z-keyword z-control z-import z-include z-c\">#include<\/span> <span class=\"z-string z-quoted z-double z-include z-c\"><span class=\"z-punctuation z-definition z-string z-begin z-c\">&quot;<\/span>interface.h<span class=\"z-punctuation z-definition z-string z-end z-c\">&quot;<\/span><\/span>\n<\/span><\/span><span class=\"z-source z-c\">\n<\/span><span class=\"z-source z-c\"><span class=\"z-storage z-type z-c\">int<\/span> <span class=\"z-meta z-function z-c\"><span class=\"z-entity z-name z-function z-c\">main<\/span><\/span><span class=\"z-meta z-function z-parameters z-c\"><span class=\"z-meta z-group z-c\"><span class=\"z-punctuation z-section z-group z-begin z-c\">(<\/span><\/span><\/span><span class=\"z-meta z-function z-parameters z-c\"><span class=\"z-meta z-group z-c\"><span class=\"z-storage z-type z-c\">void<\/span><span class=\"z-punctuation z-section z-group z-end z-c\">)<\/span><\/span><\/span><span class=\"z-meta z-function z-c\"> <\/span><span class=\"z-meta z-function z-c\"><span class=\"z-meta z-block z-c\"><span class=\"z-punctuation z-section z-block z-begin z-c\">{<\/span><\/span><\/span><span class=\"z-meta z-function z-c\"><span class=\"z-meta z-block z-c\">\n<\/span><\/span><\/span><span class=\"z-source z-c\"><span class=\"z-meta z-function z-c\"><span class=\"z-meta z-block z-c\">        <span class=\"z-meta z-function-call z-c\"><span class=\"z-support z-function z-C99 z-c\">printf<\/span><span class=\"z-meta z-group z-c\"><span class=\"z-punctuation z-section z-group z-begin z-c\">(<\/span><\/span><\/span><span class=\"z-meta z-function-call z-c\"><span class=\"z-meta z-group z-c\"><span class=\"z-string z-quoted z-double z-c\"><span class=\"z-punctuation z-definition z-string z-begin z-c\">&quot;<\/span><span class=\"z-constant z-other z-placeholder z-c\">%d<\/span><span class=\"z-constant z-character z-escape z-c\">\\n<\/span><span class=\"z-punctuation z-definition z-string z-end z-c\">&quot;<\/span><\/span><span class=\"z-punctuation z-separator z-c\">,<\/span> interface_version<\/span><\/span><span class=\"z-meta z-function-call z-c\"><span class=\"z-meta z-group z-c\"><span class=\"z-punctuation z-section z-group z-end z-c\">)<\/span><\/span><\/span><span class=\"z-punctuation z-terminator z-c\">;<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-c\"><span class=\"z-meta z-function z-c\"><span class=\"z-meta z-block z-c\"><\/span><\/span><span class=\"z-meta z-function z-c\"><span class=\"z-meta z-block z-c\"><span class=\"z-punctuation z-section z-block z-end z-c\">}<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-c\">\n<\/span><span class=\"z-source z-c\"><span class=\"z-comment z-line z-double-slash z-c\"><span class=\"z-punctuation z-definition z-comment z-c\">\/\/<\/span> dependency\/include\/interface.h\n<\/span><\/span><span class=\"z-source z-c\"><span class=\"z-storage z-modifier z-c\">const<\/span> <span class=\"z-storage z-modifier z-c\">static<\/span> <span class=\"z-storage z-type z-c\">short<\/span> interface_version <span class=\"z-keyword z-operator z-assignment z-c\">=<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-c\">1<\/span><span class=\"z-punctuation z-terminator z-c\">;<\/span>\n<\/span><\/code><\/pre>\n<p>This works ok on our first build, and when <code>dependency\/include\/interface.h<\/code> changes:<\/p>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">$ ninja\n<\/span><span class=\"z-text z-plain\">[1\/1] cc -I dependency\/include -I src src\/main.c -o src\/main\n<\/span><\/code><\/pre>\n<p>But say now that we add our own <code>src\/interface.h<\/code> file:<\/p>\n<pre data-lang=\"console\" class=\"language-console z-code\"><code class=\"language-console\" data-lang=\"console\"><span class=\"z-source z-shell z-console\"><span class=\"z-keyword z-operator z-assignment z-redirection z-process z-shell\">$<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">cat<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> src\/interface.h<\/span>\n<\/span><span class=\"z-source z-shell z-console\">const static char *version = &quot;1.0&quot;;\n<\/span><span class=\"z-source z-shell z-console\"><span class=\"z-keyword z-operator z-assignment z-redirection z-process z-shell\">$<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">ninja<\/span><\/span>\n<\/span><span class=\"z-source z-shell z-console\">ninja: no work to do.\n<\/span><\/code><\/pre>\n<p>Now we have a problem. If we remove our build artifacts, ninja <em>does<\/em> rerun, and shows us what that problem is:<\/p>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">[1\/1] cc -I dependency\/include -I src src\/main.c -o src\/main\n<\/span><span class=\"z-text z-plain\">FAILED: [code=1] src\/main\n<\/span><span class=\"z-text z-plain\">cc -I dependency\/include -I src src\/main.c -o src\/main\n<\/span><span class=\"z-text z-plain\">src\/main.c: In function \u2018main\u2019:\n<\/span><span class=\"z-text z-plain\">src\/main.c:5:24: error: \u2018interface_version\u2019 undeclared (first use in this function)\n<\/span><span class=\"z-text z-plain\">    5 |         printf(&quot;%d\\n&quot;, interface_version);\n<\/span><span class=\"z-text z-plain\">      |                        ^~~~~~~~~~~~~~~~~\n<\/span><span class=\"z-text z-plain\">src\/main.c:5:24: note: each undeclared identifier is reported only once for each function it appears in\n<\/span><span class=\"z-text z-plain\">ninja: build stopped: subcommand failed.\n<\/span><\/code><\/pre>\n<p>We switched out what <code>interface.h<\/code> resolved to, but ninja didn't know about the changed dependency.<\/p>\n<p>What we want is a way to tell it to rebuild if the file <code>src\/interface.h<\/code> is created. To my knowledge, ninja has no way of expressing this kind of negative dependency edge.<\/p>\n<aside role=note class=note-container><div class=note-content>\n<p>One possibility is to depend on the <em>whole directory<\/em> of <code>src<\/code>. The semantics of this are that <code>src<\/code> gets marked dirty whenever a file is added, moved, or deleted. This has two problems:<\/p>\n<ul>\n<li>It rebuilds too often. We only want to rerun when <code>src\/interface.h<\/code> is added, not for any other file creation.<\/li>\n<li>We don\u2019t actually have a consistent way to find all directories that need to be marked in this way. Here we just hardcoded src, but in larger builds <a href=\"https:\/\/jyn.dev\/negative-build-dependencies\/#fn-depfiles\">we would use depfiles<\/a>, and depfiles do not contain any information about negative dependencies. This matters a lot when there are a dozen+ directories in the search path!<\/li>\n<\/ul>\n<\/div><\/aside>\n<aside role=note class=note-container><div class=note-content>\n<p>Another way to avoid needing negative dependencies is to simply have a <a href=\"https:\/\/jyn.dev\/build-system-tradeoffs\/#hermetic-builds\">hermetic build<\/a>, so that our <code>cc src\/main.c | dependency\/interface.h<\/code> rule never even makes the <code>src\/interface.h<\/code> file available to the command. That works, but puts us firmly out of Ninja's design space; hermetic builds come with severe tradeoffs.<\/p>\n<\/div><\/aside>\n<aside role=note class=note-container><div class=note-content>\n<p>Yet another idea is to introduce an early-cutoff point:<\/p>\n<ol>\n<li>For each <code>.c<\/code> file, run <code>cc -M<\/code> to get a list of include files. Save that to disk.<\/li>\n<li>Whenever a directory is changed, rerun <code>cc -M<\/code>. If our list has changed, rerun a full build for that file.<\/li>\n<\/ol>\n<style>var { font-family: monospace; font-style: inherit; }<\/style>\n<p>This has the problem that it's <var>O(n<sup>3<\/sup>)<\/var>: For each file, for each include, for each directory in the search path, the preprocessor will try to <code>stat<\/code> that file to see whether it exists.<\/p>\n<p>One possible workaround is to calculate the list of include files <em>in the build system<\/em>:\nLook at the order of the search paths, list each path recursively, ignore filenames that are overridden by an include earlier in the search path.\nSave that to disk.\nNext time a directory is modified, check if the list of include files has changed. If so, only then rerun <code>cc -M<\/code> for all files.<\/p>\n<p>This requires the build system to have some quite deep knowledge of how the language works, but I do think it would work today without changes to Ninja.<\/p>\n<p>Thanks to <a href=\"https:\/\/jade.fyi\">Jade Lovelace<\/a> for this suggestion.<\/p>\n<\/div><\/aside>\n<p>Thanks to David Chisnall and Ben Boeckel (<strong>@mathstuf<\/strong>) for making me aware of this issue.<\/p>\n<section class=\"footnotes\">\n<ol class=\"footnotes-list\">\n<li id=\"fn-depfiles\">\n<p>People familiar with ninja might say this looks odd and it should use a <code>depfile<\/code> with <code>cc -M<\/code> so dependencies are tracked automatically. That makes your files shorter and means you need to run the configure step less often, but it doesn't actually solve the problem of negative dependencies. Ninja still doesn't know when it needs to regenerate the depfile. <a href=\"#fr-depfiles-1\">\u21a9<\/a><\/p>\n<\/li>\n<\/ol>\n<\/section>\n"},{"title":"brownouts reveal system boundaries","published":"2025-11-19T00:00:00+00:00","updated":"2025-11-19T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/brownouts-reveal-system-boundaries\/"}},"id":"https:\/\/jyn.dev\/brownouts-reveal-system-boundaries\/","summary":"You have backups for your critical data. Do you have backups for your critical infrastructure?","content":"<figure>\n<blockquote cite=\"https:\/\/www.newyorkfed.org\/banking\/circulars\/10923.html\">\n<p>One of the many basic tenets of internal control is that a banking organization ensure that employees in sensitive positions be absent from their duties for a minimum of two consecutive weeks. Such a requirement enhances the viability of a sound internal control environment because most frauds or embezzlements require the continual presence of the wrongdoer.<\/p>\n<\/blockquote>\n<figcaption>\n<cite ><a href=\"https:\/\/www.newyorkfed.org\/banking\/circulars\/10923.html\">Federal Reserve Bank of New York<\/a><\/cite>\n<\/figcaption>\n<\/figure>\n<figure>\n<blockquote cite=\"https:\/\/how.complexsystems.fail\/#18\">\n<p>Failure free operations require experience with failure.<\/p>\n<\/blockquote>\n<figcaption>\n<cite ><a href=\"https:\/\/how.complexsystems.fail\/#18\">How Complex Systems Fail<\/a><\/cite>\n<\/figcaption>\n<\/figure>\n<h2 id=\"uptime\">uptime<a class=\"zola-anchor\" href=\"#uptime\" aria-label=\"Anchor link for: uptime\"><\/a>\n<\/h2>\n<p>Yesterday, <a href=\"https:\/\/blog.cloudflare.com\/18-november-2025-outage\/\">Cloudflare<\/a>\u2019s global edge network was down across the world. This post is not about why that happened or how to prevent it. It\u2019s about the fact that this was inevitable. Infinite uptime <a href=\"https:\/\/how.complexsystems.fail\/\">does not exist<\/a>. If your business relies on it, sooner or later, you will get burned.<\/p>\n<p>Cloudflare\u2019s <a href=\"https:\/\/blog.cloudflare.com\/details-of-the-cloudflare-outage-on-july-2-2019\/\">last global edge outage<\/a> was on July 2, 2019. They were down yesterday for about 3 hours (with a long tail extending about another 2 and a half hours). That\u2019s an uptime of 99.99995% over the last 6 years.<\/p>\n<p>Hyperscalers like Cloudflare, AWS, and Google try very very hard to always be available, to never fail. This makes it easy to intertwine them in your architecture, so deeply you don\u2019t even know where. This is great for their business. I used to work at Cloudflare, and being intertwined like this is one of their explicit goals.<\/p>\n<h2 id=\"system-boundaries\">system boundaries<a class=\"zola-anchor\" href=\"#system-boundaries\" aria-label=\"Anchor link for: system-boundaries\"><\/a>\n<\/h2>\n<p>My company does consulting, and one of our SaaS tools is a time tracker. It was down yesterday because it relied on Cloudflare. I didn\u2019t even know until it failed! Businesses certainly don\u2019t publish their providers on their homepage. The downtime exposes dependencies that were previously hidden.<\/p>\n<p>This is especially bad for \u201ccascading\u201d dependencies, where a partner of a partner of a partner has a dependency on a hyperscaler you didn\u2019t know about. Failures like this really happen in real life; <a href=\"https:\/\/www.bloomberg.com\/opinion\/articles\/2024-11-25\/synapse-still-can-t-find-its-money\">Matt Levine writes<\/a> about one such case where a spectacular failure in a fintech caused thousands of families to lose their life savings.<\/p>\n<p>What I want to do here is make a case that cascading dependencies are bad for you, the business depending on them. Not just because you go down whenever everyone else goes down, but because depending on infinite uptime <a href=\"https:\/\/danluu.com\/postmortem-lessons\">hides error handling issues in your own architecture<\/a>. By making failures frequent enough to be normal, organizations are forced to design and practice their backup plans.<\/p>\n<h2 id=\"backup-plans\">backup plans<a class=\"zola-anchor\" href=\"#backup-plans\" aria-label=\"Anchor link for: backup-plans\"><\/a>\n<\/h2>\n<p>Backup plans don\u2019t require running your own local cloud. My blog is proxied through cloudflare; my backup plan could be \u201cfailover DNS from cloudflare to github when cloudflare is down\u201d.<\/p>\n<p>Backup plans don\u2019t have to be complicated. A hospital ER could have a backup plan of \u201ckeep patient records for everyone currently in the hospital downloaded to an offline backup  sitting in a closet somewhere\u201d, or even just \u201ckeep a printed copy next to the hospital bed\u201d.<\/p>\n<p>The important thing here is to have <em>a<\/em> backup plan, to not just blithely assume that \u201cthe internet\u201d is a magic and reliable thing.<\/p>\n<h2 id=\"testing-your-backups\">testing your backups<a class=\"zola-anchor\" href=\"#testing-your-backups\" aria-label=\"Anchor link for: testing-your-backups\"><\/a>\n<\/h2>\n<p>One way to avoid uptime reliance is brownouts, where services are down or only partially available for a predetermined amount of time. Google intentionally brownouts their internal infrastructure so that nothing relies on another service being up 100% at the time<sup class=\"footnote-reference\" id=\"fr-1-1\"><a href=\"#fn-1\">1<\/a><\/sup>. This forces errors to be constantly tested, and exposes dependency cycles.<\/p>\n<p>Another way is Chaos Monkey, pioneered at Netflix, where random things just break and you don\u2019t know which ahead of time. This requires a lot of confidence in your infrastructure, but reveals kinds of failures you didn\u2019t even think were possible.<\/p>\n<p>I would like to see a model like this for the Internet, where all service providers are required to have at least 24 hours of outages in a year. This is a bit less than 3 nines of uptime (about 5 minutes a day): enough that the service is usually up, but not so much that you can depend on it to always be up.<\/p>\n<h2 id=\"it-could-happen-here\">it could happen here<a class=\"zola-anchor\" href=\"#it-could-happen-here\" aria-label=\"Anchor link for: it-could-happen-here\"><\/a>\n<\/h2>\n<p>In my experience, and <a href=\"https:\/\/ferd.ca\/notes\/paper-the-failure-gap.html\">according to studies about failure reporting<\/a>, both people and organizations tend to chronically underestimate tail risks. Maybe you\u2019re just a personal site and you don\u2019t need 100% reliability. That\u2019s ok. But if other people depend on you, and others depend on them, and again, eventually we end up with hospitals and fire stations and water treatment plants depending on the internet. The only way I see to prevent this is to make the internet <em>unreliable<\/em> enough that they need a backup plan.<\/p>\n<p>People fail. Organizations fail. You can\u2019t control them. What you can control is whether you make them a single point of failure.<\/p>\n<p>You have backups for your critical data. Do you have backups for your critical infrastructure?<\/p>\n<section class=\"footnotes\">\n<ol class=\"footnotes-list\">\n<li id=\"fn-1\">\n<p>Of course, they don't brown-out their external-facing infra. That would lose them customers. <a href=\"#fr-1-1\">\u21a9<\/a><\/p>\n<\/li>\n<\/ol>\n<\/section>\n"},{"title":"the terminal of the future","published":"2025-11-11T00:00:00+00:00","updated":"2025-11-11T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/the-terminal-of-the-future\/"}},"id":"https:\/\/jyn.dev\/the-terminal-of-the-future\/","summary":"To redesign infrastructure, you have to allow incremental adoption, while simultaneously moving the whole design space at once.","content":"<figure>\n<blockquote cite=\"https:\/\/jvns.ca\/blog\/2025\/06\/24\/new-zine--the-secret-rules-of-the-terminal\/\">\n<p>Terminal internals are a mess. A lot of it is just the way it is because someone made a decision in the 80s and now it\u2019s impossible to change.<\/p>\n<\/blockquote>\n<figcaption>\n<cite ><a href=\"https:\/\/jvns.ca\/blog\/2025\/06\/24\/new-zine--the-secret-rules-of-the-terminal\/\">Julia Evans<\/a><\/cite>\n<\/figcaption>\n<\/figure>\n<figure>\n<blockquote cite=\"https:\/\/www.destroyallsoftware.com\/talks\/a-whole-new-world\">\n<p>This is what you have to do to redesign infrastructure. Rich [Hickey] didn't just pile some crap on top of Lisp [when building Clojure]. He took the entire Lisp and moved the whole design at once.<\/p>\n<\/blockquote>\n<figcaption>\n<cite ><a href=\"https:\/\/www.destroyallsoftware.com\/talks\/a-whole-new-world\">Gary Bernhardt<\/a><\/cite>\n<\/figcaption>\n<\/figure>\n<h2 id=\"a-mental-model-of-a-terminal\">a mental model of a terminal<a class=\"zola-anchor\" href=\"#a-mental-model-of-a-terminal\" aria-label=\"Anchor link for: a-mental-model-of-a-terminal\"><\/a>\n<\/h2>\n<p>At a very very high level, a terminal has four parts:<\/p>\n<ol>\n<li>The \"<a href=\"https:\/\/wizardzines.com\/comics\/meet-the-terminal-emulator\/\">terminal emulator<\/a>\", which is a program that renders a grid-like structure to your graphical display.<\/li>\n<li>The \"<a href=\"https:\/\/jvns.ca\/blog\/2022\/07\/20\/pseudoterminals\/\">pseudo-terminal<\/a>\" (PTY), which is a connection between the terminal emulator and a \"process group\" which receives input. This is not a program. This is a piece of state in the kernel.<\/li>\n<li>The \"shell\", which is a program that leads the \"process group\", reads and parses input, spawns processes, and generally acts as an event loop. Most environments use <a href=\"https:\/\/jvns.ca\/blog\/2017\/03\/26\/bash-quirks\/\">bash<\/a> as the default shell.<\/li>\n<li>The programs spawned by your shell, which interact with all of the above in order to receive input and send output.<\/li>\n<\/ol>\n<p>I lied a little bit above. \"input\" is not just text. It also includes <a href=\"https:\/\/man7.org\/linux\/man-pages\/man7\/signal.7.html\">signals<\/a> that can be sent to the running process. Converting keystrokes to signals is the job of the PTY.<\/p>\n<p>Similar, \"output\" is not just text. It's a stream of <a href=\"https:\/\/gist.github.com\/fnky\/458719343aabd01cfb17a3a4f7296797\">ANSI Escape Sequences<\/a> that can be used by the terminal emulator to display rich formatting.<\/p>\n<h2 id=\"what-does-a-better-terminal-look-like\">what does a better terminal look like?<a class=\"zola-anchor\" href=\"#what-does-a-better-terminal-look-like\" aria-label=\"Anchor link for: what-does-a-better-terminal-look-like\"><\/a>\n<\/h2>\n<p>I do some <a href=\"https:\/\/jyn.dev\/how-i-use-my-terminal\/\">weird things<\/a> with terminals. However, the amount of hacks I can get up to are pretty limited, because terminals are pretty limited. I won't go into all the ways they're limited, because it's been rehashed <a href=\"https:\/\/matklad.github.io\/2019\/11\/16\/a-better-shell.html\">many<\/a> <a href=\"https:\/\/github.com\/withoutboats\/notty\">times<\/a> <a href=\"https:\/\/jvns.ca\/blog\/2025\/02\/05\/some-terminal-frustrations\/\">before<\/a>. What I want to do instead is imagine what a better terminal can look like.<\/p>\n<h3 id=\"a-first-try-jupyter\">a first try: Jupyter<a class=\"zola-anchor\" href=\"#a-first-try-jupyter\" aria-label=\"Anchor link for: a-first-try-jupyter\"><\/a>\n<\/h3>\n<p>The closest thing to a terminal analog that most people are familiar with is <a href=\"https:\/\/docs.jupyter.org\/en\/latest\/\">Jupyter Notebook<\/a>. This offers a lot of cool features that are not possible in a \"traditional\" VT100 emulator:<\/p>\n<ul>\n<li>\n<p>high fidelity image rendering<\/p>\n<p><img src=\"\/assets\/jupyter%20bubble%20plot.png\" alt=\"screenshot of a JupyterLite notebook. It has some python code in a cell that generates a matplotlib chart. Beneath is the chart, rendered to raster graphics.\" \/><\/p>\n<\/li>\n<li>\n<p>a \"rerun from start\" button (or rerun the current command; or rerun only a single past command) that replaces past output instead of appending to it<\/p>\n<p><img src=\"\/assets\/jupyter%20run%20cells.png\" alt=\"\" \/><\/p>\n<\/li>\n<li>\n<p>\"views\" of source code and output that can be rewritten in place (e.g. markdown can be viewed either as source or as rendered HTML)<\/p>\n<p><img src=\"\/assets\/jupyter%20markdown%20raw.png\" alt=\"\" \/> <img src=\"\/assets\/jupyter%20markdown%20rendered.png\" alt=\"\" \/><\/p>\n<\/li>\n<li>\n<p>a built-in editor with syntax highlighting, tabs, panes, mouse support, etc.<\/p>\n<p><img src=\"\/assets\/jupyter%20window%20management.png\" alt=\"\" \/><\/p>\n<\/li>\n<\/ul>\n<h3 id=\"some-problems\">some problems<a class=\"zola-anchor\" href=\"#some-problems\" aria-label=\"Anchor link for: some-problems\"><\/a>\n<\/h3>\n<p>Jupyter works by having a \"kernel\" (in this case, a python interpreter) and a \"renderer\" (in this case, a web application displayed by the browser). You could imagine using a Jupyter Notebook with a shell as the kernel, so that you get all the nice features of Jupyter when running shell commands. However, that quickly runs into some issues:<\/p>\n<ul>\n<li>Your shell gets the commands all at once, not character-by-character, so tab-complete, syntax highlighting, and autosuggestions don't work.<\/li>\n<li>What do you do about long-lived processes? By default, Jupyter runs a cell until completion; you can cancel it, but you can't suspend, resume, interact with, nor view a process while it's running. Don't even think about running <code>vi<\/code> or <code>top<\/code>.<\/li>\n<li>The \"rerun cell\" buttons do horrible things to the state of your computer (normal Jupyter kernels have this problem too, but \"rerun all\" works better when the commands don't usually include <code>rm -rf<\/code>).<\/li>\n<li>Undo\/redo do <em>not<\/em> work. (They don't work in a normal terminal either, but people attempt to use them more when it looks like they should be able to.)<\/li>\n<\/ul>\n<p>It turns out all these problems are solveable.<\/p>\n<h2 id=\"how-does-that-work\">how does that work?<a class=\"zola-anchor\" href=\"#how-does-that-work\" aria-label=\"Anchor link for: how-does-that-work\"><\/a>\n<\/h2>\n<h3 id=\"shell-integration\">shell integration<a class=\"zola-anchor\" href=\"#shell-integration\" aria-label=\"Anchor link for: shell-integration\"><\/a>\n<\/h3>\n<p>There exists today a terminal called <a href=\"https:\/\/www.warp.dev\/\">Warp<\/a>. Warp has built native integration between the terminal and the shell, where the terminal understands where each command starts and stops, what it outputs, and what is your own input. As a result, it can render things very prettily:<\/p>\n<p><img src=\"\/assets\/warp%20terminal.png\" alt=\"\" \/><\/p>\n<p>It does this using (mostly) standard features built-in to the terminal and shell (a custom DCS): you can read their explanation <a href=\"https:\/\/www.warp.dev\/blog\/how-warp-works#:~:text=the%20reason\">here<\/a>. It's possible to do this less invasively using <a href=\"https:\/\/iterm2.com\/documentation-escape-codes.html#:~:text=shell%20integration\/\">OSC 133 escape codes<\/a>; I'm not sure why Warp didn't do this, but that's ok.<\/p>\n<p>iTerm2 does a similar thing, and this allows it to enable <a href=\"https:\/\/iterm2.com\/documentation-shell-integration.html#:~:text=enables%20numerous%20features\">really quite a lot of features<\/a>: navigating between commands with a single hotkey; notifying you when a command finishes running, showing the current command as an \"overlay\" if the output goes off the screen.<\/p>\n<h3 id=\"long-lived-processes\">long-lived processes<a class=\"zola-anchor\" href=\"#long-lived-processes\" aria-label=\"Anchor link for: long-lived-processes\"><\/a>\n<\/h3>\n<p>This is really three different things. The first is <em>interacting<\/em> with a long-lived process. The second is <em>suspending<\/em> the process without killing it. The third is <em>disconnecting<\/em> from the process, in such a way that the process state is not disturbed and is still available if you want to reconnect.<\/p>\n<h4 id=\"interacting\">interacting<a class=\"zola-anchor\" href=\"#interacting\" aria-label=\"Anchor link for: interacting\"><\/a>\n<\/h4>\n<p>To interact with a process, you need bidirectional communication, i.e. you need a \"cell output\" that is also an input. An example would be any TUI, like <code>top<\/code>, <code>gdb<\/code>, or <code>vim<\/code> <sup class=\"footnote-reference\" id=\"fr-1-1\"><a href=\"#fn-1\">1<\/a><\/sup>.  Fortunately, Jupyter is really good at this!  The whole design is around having <a href=\"https:\/\/ipywidgets.readthedocs.io\/en\/latest\/\">interactive outputs<\/a> that you can change and update.<\/p>\n<p>Additionally, I would expect my terminal to always have a \"free input cell\", as Matklad describes in <a href=\"https:\/\/matklad.github.io\/2019\/11\/16\/a-better-shell.html\">A Better Shell<\/a>, where the interactive process runs in the top half of the window and an input cell is available in the bottom half. Jupyter can do this today, but \"add a cell\" is manual, not automatic.<\/p>\n<h4 id=\"suspending\">suspending<a class=\"zola-anchor\" href=\"#suspending\" aria-label=\"Anchor link for: suspending\"><\/a>\n<\/h4>\n<p>\"Suspending\" a process is usually called \"<a href=\"https:\/\/jvns.ca\/blog\/2024\/07\/03\/reasons-to-use-job-control\/\">job control<\/a>\". There's not too much to talk about here, except that I would expect a \"modern\" terminal to show me all suspended and background processes as a de-emphasized persistent visual, kinda like how Intellij will show you \"indexing ...\" in the bottom taskbar.<\/p>\n<p><img src=\"\/assets\/intellij%20background%20tasks.png\" alt=\"\" \/><\/p>\n<h4 id=\"disconnecting\">disconnecting<a class=\"zola-anchor\" href=\"#disconnecting\" aria-label=\"Anchor link for: disconnecting\"><\/a>\n<\/h4>\n<p>There are roughly three existing approaches for disconnecting and reconnecting to a terminal session (Well, four if you count <a href=\"https:\/\/github.com\/nelhage\/reptyr\">reptyr<\/a>).<\/p>\n<ol>\n<li>\n<p>Tmux \/ Zellij \/ Screen<\/p>\n<p>These tools inject a whole extra terminal emulator between your terminal emulator and the program. They work by having a \"server\" which actually owns the PTY and renders the output, and a \"client\" that displays the output to your \"real\" terminal emulator. This model lets you detach clients, reattach them later, or even attach multiple clients at once. You can think of this as a \"batteries-included\" approach. It also has the benefit that you can <a href=\"https:\/\/jyn.dev\/how-i-use-my-terminal\/\">program<\/a> both the client and the server (although many modern terminals, like <a href=\"https:\/\/sw.kovidgoyal.net\/kitty\/\">Kitty<\/a> and <a href=\"https:\/\/wezfurlong.org\/wezterm\/\">Wezterm<\/a> are programmable now); that you can organize your tabs and windows in the terminal (although many modern desktop environments have tiling and thorough keyboard shortcuts); and that you get street cred for looking like Hackerman.<\/p>\n<p><img src=\"\/assets\/hackerman.png\" alt=\"\" \/><\/p>\n<p>The downside is that, well, now you have an extra terminal emulator running in your terminal, with <a href=\"https:\/\/sw.kovidgoyal.net\/kitty\/faq\/#i-am-using-tmux-zellij-and-have-a-problem\">all the bugs that implies<\/a>.<\/p>\n<p>iTerm actually avoids this by <a href=\"https:\/\/iterm2.com\/documentation-tmux-integration.html\">bypassing the tmux client altogether and acting as its own client<\/a> that talks directly to the server. In this mode, \"tmux tabs\" are actually iTerm tabs, \"tmux panes\" are iTerm panes, and so on. This is a good model, and I would adopt it when writing a future terminal for integration with existing tmux setups.<\/p>\n<\/li>\n<li>\n<p><a href=\"https:\/\/mosh.org\/\">Mosh<\/a><\/p>\n<p>Mosh is a really interesting place in the design space. It is not a terminal emulator replacement; instead it is an <em>ssh<\/em> replacement. Its big draw is that it supports reconnecting to your terminal session after a network interruption. It does that by <a href=\"https:\/\/mosh.org\/#:~:text=how%20mosh%20works\">running a state machine on the server and replaying an incremental diff of the viewport to the client<\/a>. This is a similar model to tmux, except that it doesn't support the \"multiplexing\" part (it expects your terminal emulator to handle that), nor scrollback (ditto). Because it has its own renderer, it has <a href=\"https:\/\/github.com\/mobile-shell\/mosh\/issues\/234\">a similar class of bugs to tmux<\/a>. One feature it <em>does<\/em> have, unlike tmux, is that the \"client\" is really running on your side of the network, so local line editing is instant.<\/p>\n<\/li>\n<li>\n<p><a href=\"https:\/\/ansuz.sooke.bc.ca\/entry\/389\">alden<\/a>\/<a href=\"https:\/\/github.com\/shell-pool\/shpool\">shpool<\/a>\/<a href=\"https:\/\/github.com\/crigler\/dtach\">dtach<\/a>\/<a href=\"https:\/\/github.com\/martanne\/abduco\">abduco<\/a>\/<a href=\"https:\/\/github.com\/yazgoo\/diss\">diss<\/a><\/p>\n<p>These all occupy a similar place in the design space: they <em>only<\/em> handle session detach\/resume with a client\/server, not networking or scrollback, and do not include their own terminal emulator. Compared to tmux and mosh, they are highly decoupled.<\/p>\n<\/li>\n<\/ol>\n<h3 id=\"rerun-and-undo-redo\">rerun and undo\/redo<a class=\"zola-anchor\" href=\"#rerun-and-undo-redo\" aria-label=\"Anchor link for: rerun-and-undo-redo\"><\/a>\n<\/h3>\n<p>I'm going to treat these together because the solution is the same: dataflow tracking.<\/p>\n<p>Take as an example <a href=\"https:\/\/plutojl.org\/\">pluto.jl<\/a>, which does this <em>today<\/em> by hooking into the Julia compiler.<\/p>\n<p><img src=\"\/assets\/pluto%20interactive%20ode.gif\" alt=\"\" \/><\/p>\n<p>Note that this updates cells live in response to previous cells that they depend on. Not pictured is that it <em>doesn't<\/em> update cells if their dependencies haven't changed. You can think of this as a spreadsheet-like Jupyter, where code is only rerun when necessary.<\/p>\n<p>You may say this is hard to generalize. The trick here is <a href=\"https:\/\/jyn.dev\/complected-and-orthogonal-persistence\/#how-far-can-we-take-this\">orthogonal persistence<\/a>. If you sandbox the processes, track all IO, and prevent things that are \"too weird\" unless they're talking to other processes in the sandbox (e.g. unix sockets and POST requests), you have really quite a lot of control over the process! This lets you treat it as a pure function of its inputs, where its inputs are \"the whole file system, all environment variables, and all process attributes\".<\/p>\n<h3 id=\"derived-features\">derived features<a class=\"zola-anchor\" href=\"#derived-features\" aria-label=\"Anchor link for: derived-features\"><\/a>\n<\/h3>\n<p>Once you have these primitives\u2014Jupyter notebook frontends, undo\/redo, automatic rerun, persistence, and shell integration\u2014you can build really quite a lot on top. And you can build it incrementally, piece-by-piece:<\/p>\n<h4 id=\"needs-a-jupyter-notebook-frontend\">needs a Jupyter notebook frontend<a class=\"zola-anchor\" href=\"#needs-a-jupyter-notebook-frontend\" aria-label=\"Anchor link for: needs-a-jupyter-notebook-frontend\"><\/a>\n<\/h4>\n<ul>\n<li><a href=\"https:\/\/blog.atuin.sh\/atuin-desktop-runbooks-that-run\/\">Runbooks<\/a> (actually, you can build these just with Jupyter and a PTY primitive).<\/li>\n<li>Terminal customization that uses normal CSS, no weird custom languages or ANSI color codes.<\/li>\n<li>Search for commands by output\/timestamp. Currently, you can search across output in the current session, or you can search across all command input history, but you don't have any kind of smart filters, and the output doesn't persist across sessions.<\/li>\n<\/ul>\n<h4 id=\"needs-shell-integration\">needs shell integration<a class=\"zola-anchor\" href=\"#needs-shell-integration\" aria-label=\"Anchor link for: needs-shell-integration\"><\/a>\n<\/h4>\n<ul>\n<li>Timestamps and execution duration for each command.<\/li>\n<li>Local line-editing, even across a network boundary.<\/li>\n<li><a href=\"https:\/\/docs.warp.dev\/terminal\/command-completions\/completions\">IntelliSense for shell commands<\/a>, without having to hit tab and with rendering that's integrated into the terminal.<\/li>\n<\/ul>\n<h4 id=\"needs-sandboxed-tracing\">needs sandboxed tracing<a class=\"zola-anchor\" href=\"#needs-sandboxed-tracing\" aria-label=\"Anchor link for: needs-sandboxed-tracing\"><\/a>\n<\/h4>\n<ul>\n<li>\"<a href=\"https:\/\/jyn.dev\/complected-and-orthogonal-persistence\/#but-why\">All the features from sandboxed tracing<\/a>\": collaborative terminals, querying files modified by a command, \"asciinema but you can edit it at runtime\", tracing build systems.<\/li>\n<li>Extend the smart search above to also search by disk state at the time the command was run.<\/li>\n<li>Extending undo\/redo to a git-like branching model (something like this is already support by <a href=\"https:\/\/elpa.gnu.org\/packages\/undo-tree.html#:~:text=undo%20systems\">emacs undo-tree<\/a>), where you have multiple \"views\" of the process tree.<\/li>\n<li>Given the undo-tree model, and since we have sandboxing, we can give an LLM access to your project, and run many of them in parallel at the same time without overwriting each others state, and in such a way that you can see what they're doing, edit it, and save it into a runbook for later use.<\/li>\n<li>A terminal in a prod environment that can't affect the state of the machine, only inspect the existing state.<\/li>\n<\/ul>\n<h2 id=\"ok-but-how-do-you-build-this\">ok but how do you build this<a class=\"zola-anchor\" href=\"#ok-but-how-do-you-build-this\" aria-label=\"Anchor link for: ok-but-how-do-you-build-this\"><\/a>\n<\/h2>\n<p>jyn, you may say, <a href=\"https:\/\/becca.ooo\/blog\/vertical-integration\/\">you can't build vertical integration in open source<\/a>. <a href=\"https:\/\/tech.lgbt\/@jyn\/112187088917827279\">you can't make money off open source projects<\/a>. <a href=\"https:\/\/jyn.dev\/you-are-in-a-box\/#switching-costs-and-growth\">the switching costs are too high<\/a>.<\/p>\n<p>All these things are true. To talk about how this is possible, we have to talk about incremental adoption.<\/p>\n<p>if I were building this, I would do it in stages, such that at each stage the thing is an improvement over its alternatives. This is how <code>jj<\/code> works and it works extremely well: it doesn't require everyone on a team to switch at once because individual people can use <code>jj<\/code>, even for single commands, without a large impact on everyone else.<\/p>\n<h3 id=\"stage-1-transactional-semantics\">stage 1: transactional semantics<a class=\"zola-anchor\" href=\"#stage-1-transactional-semantics\" aria-label=\"Anchor link for: stage-1-transactional-semantics\"><\/a>\n<\/h3>\n<p>When people think of redesigning the terminal, they always think of redesigning the terminal <em>emulator<\/em>. This is exactly the wrong place to start. People are attached to their emulators. They configure them, they make them look nice, they use their keybindings. There is a high switching cost to switching emulators because <a href=\"https:\/\/jvns.ca\/blog\/2025\/01\/11\/getting-a-modern-terminal-setup\/#everything-affects-everything-else\">everything affects everything else<\/a>. It's not <em>so<\/em> terribly high, because it's still individual and not shared across a team, but still high.<\/p>\n<p>What I would do instead is start at the CLI layer. CLI programs are great because they're easy to install and run and have very low switching costs: you can use them one-off without changing your whole workflow.<\/p>\n<p>So, I would write a CLI that implements <a href=\"https:\/\/jyn.dev\/complected-and-orthogonal-persistence\/#how-far-can-we-take-this\">transactional semantics for the terminal<\/a>. You can imagine an interface something like <code>transaction [start|rollback|commit]<\/code>, where everything run after <code>start<\/code> is undoable. There is a <em>lot<\/em> you can do with this alone, I think you could build a whole business off this.<\/p>\n<h3 id=\"stage-2-persistent-sessions\">stage 2: persistent sessions<a class=\"zola-anchor\" href=\"#stage-2-persistent-sessions\" aria-label=\"Anchor link for: stage-2-persistent-sessions\"><\/a>\n<\/h3>\n<p>Once I had transactional semantics, I would try to decouple persistence from tmux and mosh.<\/p>\n<p>To get PTY persistence, you have to introduce a client\/server model, because the kernel <em>really really<\/em> <a href=\"https:\/\/en.wikipedia.org\/wiki\/SIGHUP\">expects<\/a> both sides of a PTY to always be connected. Using commands like <a href=\"https:\/\/ansuz.sooke.bc.ca\/entry\/389\">alden<\/a>, or a library like it (it's not <em>that<\/em> complicated), lets you do this simply, without affecting the terminal emulator nor the programs running inside the PTY session.<\/p>\n<p>To get scrollback, the server could save input and output indefinitely and replay them when the client reconnects. This gets you \"native\" scrollback\u2014the terminal emulator you're already using handles it exactly like any other output, because it looks exactly like any other output\u2014while still being replayable and resumable from an arbitrary starting point. This requires some amount of parsing ANSI escape codes<sup class=\"footnote-reference\" id=\"fr-2-1\"><a href=\"#fn-2\">2<\/a><\/sup>, but it's doable with enough work.<\/p>\n<p>To get network resumption like mosh, my custom server could use <a href=\"https:\/\/eternalterminal.dev\/howitworks\/\">Eternal TCP<\/a> (possibly built on top of QUIC for efficiency). Notably, the persistence for the PTY is separate from the persistence for the network connection. Eternal TCP here is strictly an optimization: you could build this on top of a bash script that runs <code>ssh host eternal-pty attach<\/code> in a loop, it's just not as nice an experience because of network delay and packet loss. Again, composable parts allow for incremental adoption.<\/p>\n<p>At this point, you're already able to connect multiple clients to a single terminal session, like tmux, but window management is still done by your terminal emulator, not by the client\/server. If you wanted to have window management integrated, the terminal emulator could speak the tmux -CC protocol, like iTerm.<\/p>\n<p>All parts of this stage can be done independently and in parallel from the transactional semantics, but I don't think you can build a business off them, it's not enough of an improvement over the existing tools.<\/p>\n<h3 id=\"stage-3-structured-rpc\">stage 3: structured RPC<a class=\"zola-anchor\" href=\"#stage-3-structured-rpc\" aria-label=\"Anchor link for: stage-3-structured-rpc\"><\/a>\n<\/h3>\n<p>This bit depends on the client\/server model. Once you have a server interposed between the terminal emulator and the client, you can start doing really funny things like tagging I\/O with metadata. This lets all data be timestamped<sup class=\"footnote-reference\" id=\"fr-3-1\"><a href=\"#fn-3\">3<\/a><\/sup> and lets you distinguish input from output. <a href=\"https:\/\/jvns.ca\/blog\/2022\/07\/20\/pseudoterminals\/\">xterm.js<\/a> works something like this. When combined with shell integration, this even lets you distinguish shell prompts from program output, at the <em>data<\/em> layer.<\/p>\n<p>Now you can start doing really funny things, because you have a <em>structured log<\/em> of your terminal session. You can replay the log as a recording, like <a href=\"https:\/\/asciinema.org\/\">asciinema<\/a><sup class=\"footnote-reference\" id=\"fr-4-1\"><a href=\"#fn-4\">4<\/a><\/sup>; you can transform the shell prompt without rerunning all the commands; you can import it into a Jupyter Notebook or <a href=\"https:\/\/blog.atuin.sh\/atuin-desktop-runbooks-that-run\/\">Atuin Desktop<\/a>; you can save the commands and rerun them later as a script. Your terminal is data.<\/p>\n<h3 id=\"stage-4-jupyter-like-frontend\">stage 4: jupyter-like frontend<a class=\"zola-anchor\" href=\"#stage-4-jupyter-like-frontend\" aria-label=\"Anchor link for: stage-4-jupyter-like-frontend\"><\/a>\n<\/h3>\n<p>This is the very first time that we touch the terminal emulator, and it's intentionally the last step because it has the highest switching costs. This makes use of all the nice features we've built to give you a nice UI. You don't need our <code>transaction<\/code> CLI anymore unless you want nested transactions, because your whole terminal session starts in a transaction by default. You get all the features I mention <a href=\"https:\/\/jyn.dev\/the-terminal-of-the-future\/#derived-features\">above<\/a>, because we've put all the pieces together.<\/p>\n<h2 id=\"jyn-what-the-fuck\">jyn, what the fuck<a class=\"zola-anchor\" href=\"#jyn-what-the-fuck\" aria-label=\"Anchor link for: jyn-what-the-fuck\"><\/a>\n<\/h2>\n<p>This is bold and ambitious and I think building the whole thing would take about a decade. That's ok. I'm patient.<\/p>\n<p>You can help me by spreading the word :) Perhaps this post will inspire someone to start building this themselves.<\/p>\n<hr \/>\n<h2 id=\"bibliography\">bibliography<a class=\"zola-anchor\" href=\"#bibliography\" aria-label=\"Anchor link for: bibliography\"><\/a>\n<\/h2>\n<ul>\n<li><a href=\"https:\/\/www.destroyallsoftware.com\/talks\/a-whole-new-world\">Gary Bernhardt, \u201cA Whole New World\u201d<\/a><\/li>\n<li><a href=\"https:\/\/matklad.github.io\/2019\/11\/16\/a-better-shell.html\">Alex Kladov, \u201cA Better Shell\u201d<\/a><\/li>\n<li><a href=\"https:\/\/jyn.dev\/how-i-use-my-terminal\/\">jyn, \u201chow i use my terminal\u201d<\/a><\/li>\n<li><a href=\"https:\/\/jyn.dev\/complected-and-orthogonal-persistence\/\">jyn, \u201cComplected and Orthogonal Persistence\u201d<\/a><\/li>\n<li><a href=\"https:\/\/jyn.dev\/you-are-in-a-box\/\">jyn, \u201cyou are in a box\u201d<\/a><\/li>\n<li><a href=\"https:\/\/tech.lgbt\/@jyn\/112187088917827279\">jyn, \u201cthere's two costs to making money off an open source project\u2026\u201d<\/a><\/li>\n<li><a href=\"https:\/\/becca.ooo\/blog\/vertical-integration\/\">Rebecca Turner, \u201cVertical Integration is the Only Thing That Matters\u201d<\/a><\/li>\n<li><a href=\"https:\/\/jvns.ca\/blog\/2025\/06\/24\/new-zine--the-secret-rules-of-the-terminal\/\">Julia Evans, \u201cNew zine: The Secret Rules of the Terminal\u201d<\/a><\/li>\n<li><a href=\"https:\/\/wizardzines.com\/comics\/meet-the-terminal-emulator\/\">Julia Evans, \u201cmeet the terminal emulator\u201d<\/a><\/li>\n<li><a href=\"https:\/\/jvns.ca\/blog\/2022\/07\/20\/pseudoterminals\/\">Julia Evans, \u201cWhat happens when you press a key in your terminal?\u201d<\/a><\/li>\n<li><a href=\"https:\/\/jvns.ca\/blog\/2025\/01\/11\/getting-a-modern-terminal-setup\/\">Julia Evans, \u201cWhat's involved in getting a \"modern\" terminal setup?\u201d<\/a><\/li>\n<li><a href=\"https:\/\/jvns.ca\/blog\/2017\/03\/26\/bash-quirks\/\">Julia Evans, \u201cBash scripting quirks &amp; safety tips\u201d<\/a><\/li>\n<li><a href=\"https:\/\/jvns.ca\/blog\/2025\/02\/05\/some-terminal-frustrations\/\">Julia Evans, \u201cSome terminal frustrations\u201d<\/a><\/li>\n<li><a href=\"https:\/\/jvns.ca\/blog\/2024\/07\/03\/reasons-to-use-job-control\/\">Julia Evans, \u201cReasons to use your shell's job control\u201d<\/a><\/li>\n<li><a href=\"https:\/\/man7.org\/linux\/man-pages\/man7\/signal.7.html\">\u201csignal(7) - Miscellaneous Information Manual\u201d<\/a><\/li>\n<li><a href=\"https:\/\/gist.github.com\/fnky\/458719343aabd01cfb17a3a4f7296797\">Christian Petersen, \u201cANSI Escape Codes\u201d<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/withoutboats\/notty\">saoirse, \u201cwithoutboats\/notty: A new kind of terminal\u201d<\/a><\/li>\n<li><a href=\"https:\/\/docs.jupyter.org\/en\/latest\/\">Jupyter Team, \u201cProject Jupyter Documentation\u201d<\/a><\/li>\n<li><a href=\"https:\/\/www.warp.dev\/\">\u201cWarp: The Agentic Development Environment\u201d<\/a><\/li>\n<li><a href=\"https:\/\/www.warp.dev\/blog\/how-warp-works\">\u201cWarp: How Warp Works\u201d<\/a><\/li>\n<li><a href=\"https:\/\/docs.warp.dev\/terminal\/command-completions\/completions\">\u201cWarp: Completions\u201d<\/a><\/li>\n<li><a href=\"https:\/\/iterm2.com\/documentation-escape-codes.html\">George Nachman, \u201ciTerm2: Proprietary Escape Codes\u201d<\/a><\/li>\n<li><a href=\"https:\/\/iterm2.com\/documentation-shell-integration.html\">George Nachman, \u201ciTerm2: Shell Integration\u201d<\/a><\/li>\n<li><a href=\"https:\/\/iterm2.com\/documentation-tmux-integration.html\">George Nachman, \u201ciTerm2: tmux Integration\u201d<\/a><\/li>\n<li><a href=\"https:\/\/ipywidgets.readthedocs.io\/en\/latest\/\">Project Jupyter, \u201cJupyter Widgets\u201d<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/nelhage\/reptyr\">Nelson Elhage, \u201cnelhage\/reptyr: Reparent a running program to a new terminal\u201d<\/a><\/li>\n<li><a href=\"https:\/\/sw.kovidgoyal.net\/kitty\/\">Kovid Goyal, \u201ckitty\u201d<\/a><\/li>\n<li><a href=\"https:\/\/sw.kovidgoyal.net\/kitty\/faq\/\">Kovid Goyal, \u201ckitty - Frequently Asked Questions\u201d<\/a><\/li>\n<li><a href=\"https:\/\/wezfurlong.org\/wezterm\/\">Wez Furlong, \u201cWezterm\u201d<\/a><\/li>\n<li><a href=\"https:\/\/mosh.org\/\">Keith Winstein, \u201cMosh: the mobile shell\u201d<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/mobile-shell\/mosh\/issues\/234\">Keith Winstein, \u201cDisplay errors with certain characters<\/a><\/li>\n<li><a href=\"https:\/\/ansuz.sooke.bc.ca\/entry\/389\">Matthew Skala, \u201calden: detachable terminal sessions without breaking scrollback\u201d<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/shell-pool\/shpool\">Ethan Pailes, \u201cshell-pool\/shpool: Think tmux, then aim... lower\u201d<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/crigler\/dtach\">Ned T. Crigler, \u201ccrigler\/dtach: A simple program that emulates the detach feature of screen\u201d<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/martanne\/abduco\">Marc Andr\u00e9 Tanner, \u201cmartanne\/abduco: abduco provides session management\u201d<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/yazgoo\/diss\">yazgoo, \u201cyazgoo\/diss: dtach-like program \/ crate in rust\u201d<\/a><\/li>\n<li><a href=\"https:\/\/plutojl.org\/\">Fons van der Plas, \u201cPluto.jl \u2014 interactive Julia programming environment\u201d<\/a><\/li>\n<li><a href=\"https:\/\/blog.atuin.sh\/atuin-desktop-runbooks-that-run\/\">Ellie Huxtable, \u201cAtuin Desktop: Runbooks that Run\u201d<\/a><\/li>\n<li><a href=\"https:\/\/elpa.gnu.org\/packages\/undo-tree.html\">Toby Cubitt, \u201cundo-tree\u201d<\/a><\/li>\n<li><a href=\"https:\/\/en.wikipedia.org\/wiki\/SIGHUP\">\u201cSIGHUP - Wikipedia\u201d<\/a><\/li>\n<li><a href=\"https:\/\/eternalterminal.dev\/howitworks\/\">Jason Gauci, \u201cHow Eternal Terminal Works\u201d<\/a><\/li>\n<li><a href=\"https:\/\/asciinema.org\/\">Marcin Kulik, \u201cRecord and share your terminal sessions, the simple way - asciinema.org\u201d<\/a><\/li>\n<li><a href=\"https:\/\/ratatui.rs\/concepts\/backends\/alternate-screen\/\">\u201cAlternate Screen | Ratatui\u201d<\/a><\/li>\n<\/ul>\n<!--To talk about interacting with a process, we have to talk about [PTTYs](https:\/\/jvns.ca\/blog\/2022\/07\/20\/pseudoterminals\/).-->\n<!--\nMy complaint about Warp is chiefly that they didn't think big enough. Warp built a very cool tool, and then wrapped it in a product that mostly exists to make it easier to use LLMs in the terminal. There are [*so many* things](https:\/\/iterm2.com\/documentation-shell-integration.html#:~:text=enables%20numerous%20features) you could do with shell integration, and they don't seem to have done very many of them.\n-->\n<section class=\"footnotes\">\n<ol class=\"footnotes-list\">\n<li id=\"fn-1\">\n<p>there are a <em>lot<\/em> of complications here around <a href=\"https:\/\/ratatui.rs\/concepts\/backends\/alternate-screen\/\">alternate mode<\/a>, but I'm just going to skip over those for now. A simple way to handle alternate mode (that doesn't get you nice things) is just to embed a raw terminal in the output cell. <a href=\"#fr-1-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-2\">\n<p>otherwise you could start replaying output from inside an escape, which is <a href=\"https:\/\/mosh.org\/#:~:text=evil%20escape%20sequences\">not good<\/a>. I had a detailed email exchange about this with the alden author which I have not yet had time to write up into a blog post; most of the complication comes when you want to avoid replaying the <em>entire<\/em> history and only want to replay the visible viewport. <a href=\"#fr-2-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-3\">\n<p>hey, this seems awfully like <a href=\"https:\/\/asciinema.org\/\">asciinema<\/a>! <a href=\"#fr-3-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-4\">\n<p>oh, that's why it seemed like asciinema. <a href=\"#fr-4-1\">\u21a9<\/a><\/p>\n<\/li>\n<\/ol>\n<\/section>\n"},{"title":"build system tradeoffs","published":"2025-11-02T00:00:00+00:00","updated":"2025-11-02T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/build-system-tradeoffs\/"}},"id":"https:\/\/jyn.dev\/build-system-tradeoffs\/","summary":"an overview of what builds for complicated projects have to think about","content":"<p>This post is part 1\/4 of <a href=\"\/four-posts-about-build-systems\/\">a series about build systems<\/a>.\nThe next post is <a href=\"\/negative-build-dependencies\/\">negative build dependencies<\/a>.<\/p>\n<hr \/>\n<figure>\n<blockquote cite=\"https:\/\/chaos.social\/@Qyriad\/111349762684384063\">\n<p>If I am even TEMPTED to use <code>sed<\/code>, in my goddamn build system, you have lost.<\/p>\n<\/blockquote>\n<figcaption>\n<cite class=username><a href=\"https:\/\/chaos.social\/@Qyriad\/111349762684384063\">Qyriad<\/a><\/cite>\n<\/figcaption>\n<\/figure>\n<p>I am currently employed to work on <a href=\"https:\/\/rustc-dev-guide.rust-lang.org\/building\/bootstrapping\/intro.html\">the build system for the Rust compiler<\/a> (often called <code>x.py<\/code> or <code>bootstrap<\/code>). As a result, I think about a lot of build system weirdness that most people don't have to. This post aims to give an overview of what builds for complicated projects have to think about, as well as vaguely gesture in the direction of build system ideas that I like.<\/p>\n<p>This post is generally designed to be accessible to the working programmer, but I have a lot of expert blindness in this area, and sometimes assume that <a href=\"https:\/\/xkcd.com\/2501\/\">\"of <em>course<\/em> people know what a feldspar is!\"<\/a>. Apologies in advance if it's hard to follow.<\/p>\n<h2 id=\"build-concerns\">build concerns<a class=\"zola-anchor\" href=\"#build-concerns\" aria-label=\"Anchor link for: build-concerns\"><\/a>\n<\/h2>\n<p>What makes a project\u2019s build complicated?<\/p>\n<h3 id=\"running-generated-binaries\">running generated binaries<a class=\"zola-anchor\" href=\"#running-generated-binaries\" aria-label=\"Anchor link for: running-generated-binaries\"><\/a>\n<\/h3>\n<p>The first semi-complicated thing people usually want to do in their build is write an integration test. Here's a rust program which does so:<\/p>\n<pre data-lang=\"rust\" class=\"language-rust z-code\"><code class=\"language-rust\" data-lang=\"rust\"><span class=\"z-source z-rust\"><span class=\"z-comment z-line z-double-slash z-rust\"><span class=\"z-punctuation z-definition z-comment z-rust\">\/\/<\/span> tests\/integration.rs\n<\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-keyword z-other z-rust\">use<\/span> <span class=\"z-meta z-path z-rust\">std<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span><span class=\"z-meta z-path z-rust\">process<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span>Command<span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-storage z-type z-function z-rust\">fn<\/span> <\/span><span class=\"z-entity z-name z-function z-rust\">assert_success<\/span><\/span><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-begin z-rust\">(<\/span><span class=\"z-variable z-parameter z-rust\">cmd<\/span><span class=\"z-punctuation z-separator z-rust\">:<\/span> Command<\/span><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-end z-rust\">)<\/span><\/span><\/span><\/span><span class=\"z-meta z-function z-rust\"> <\/span><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\">    <span class=\"z-support z-macro z-rust\">assert!<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span>cmd<span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">status<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">unwrap<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">success<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\"><\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-annotation z-rust\"><span class=\"z-punctuation z-definition z-annotation z-rust\">#<\/span><span class=\"z-punctuation z-section z-group z-begin z-rust\">[<\/span><span class=\"z-variable z-annotation z-rust\">test<\/span><span class=\"z-punctuation z-section z-group z-end z-rust\">]<\/span><\/span>\n<\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-storage z-type z-function z-rust\">fn<\/span> <\/span><span class=\"z-entity z-name z-function z-rust\">test_my_program<\/span><\/span><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-end z-rust\">)<\/span><\/span><\/span><\/span><span class=\"z-meta z-function z-rust\"> <\/span><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\">    <span class=\"z-support z-function z-rust\">assert_success<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-meta z-path z-rust\">Command<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span>new<span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-string z-quoted z-double z-rust\"><span class=\"z-punctuation z-definition z-string z-begin z-rust\">&quot;<\/span>cargo<span class=\"z-punctuation z-definition z-string z-end z-rust\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">arg<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-string z-quoted z-double z-rust\"><span class=\"z-punctuation z-definition z-string z-begin z-rust\">&quot;<\/span>build<span class=\"z-punctuation z-definition z-string z-end z-rust\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\">    <span class=\"z-support z-function z-rust\">assert_success<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-meta z-path z-rust\">Command<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span>new<span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-string z-quoted z-double z-rust\"><span class=\"z-punctuation z-definition z-string z-begin z-rust\">&quot;<\/span>target\/debug\/my-program<span class=\"z-punctuation z-definition z-string z-end z-rust\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span>\n<\/span><\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-group z-rust\">      <span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">arg<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-string z-quoted z-double z-rust\"><span class=\"z-punctuation z-definition z-string z-begin z-rust\">&quot;<\/span>assets\/test-input.txt<span class=\"z-punctuation z-definition z-string z-end z-rust\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\"><\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span><\/span>\n<\/span><\/code><\/pre>\n<p>This instructs <a href=\"https:\/\/doc.rust-lang.org\/cargo\/\">cargo<\/a> to, when you run <code>cargo test<\/code>, compile <code>tests\/integration.rs<\/code> as a standalone program and run it, with <code>test_my_program<\/code> as the entrypoint.<\/p>\n<p>We'll come back to this program several times in this post. For now, notice that we are invoking <code>cargo build<\/code> inside of <code>cargo test<\/code>. <!-- This isn't something cargo-specific\u2014there's extensive literature about it under the name [\"recursive make\"](https:\/\/accu.org\/journals\/overload\/14\/71\/miller_2004\/).--><\/p>\n<h3 id=\"correct-dependency-tracking\">correct dependency tracking<a class=\"zola-anchor\" href=\"#correct-dependency-tracking\" aria-label=\"Anchor link for: correct-dependency-tracking\"><\/a>\n<\/h3>\n<p>I actually forgot this one in the first draft because Cargo solves this so well in the common case <sup class=\"footnote-reference\" id=\"fr-14-1\"><a href=\"#fn-14\">1<\/a><\/sup>. In many hand-written builds (<em>cough<\/em> <code>make<\/code> <em>cough<\/em>), specifying dependencies by hand is very broken, <code>-j<\/code> for parallelism simply doesn't work, and running <code>make clean<\/code> on errors is common. Needless to say, this is a bad experience.<\/p>\n<h3 id=\"cross-compiling\">cross-compiling<a class=\"zola-anchor\" href=\"#cross-compiling\" aria-label=\"Anchor link for: cross-compiling\"><\/a>\n<\/h3>\n<p>The next step up in complexity is to cross-compile code. At this point, we already start to get some idea of how involved things get:<\/p>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">$ cargo test --target aarch64-unknown-linux-gnu\n<\/span><span class=\"z-text z-plain\">   Compiling my-program v0.1.0 (\/home\/jyn\/src\/build-system-overview)\n<\/span><span class=\"z-text z-plain\">error[E0463]: can&#39;t find crate for `std`\n<\/span><span class=\"z-text z-plain\">  |\n<\/span><span class=\"z-text z-plain\">  = note: the `aarch64-unknown-linux-gnu` target may not be installed\n<\/span><span class=\"z-text z-plain\">  = help: consider downloading the target with `rustup target add aarch64-unknown-linux-gnu`\n<\/span><span class=\"z-text z-plain\">  = help: consider building the standard library from source with `cargo build -Zbuild-std`\n<\/span><\/code><\/pre>\n<p>How hard it is to cross-compile code depends greatly on not just the build system, but the language you're using and the exact platform you're targeting. The particular thing I want to point out is <em>your standard library has to come from somewhere<\/em>. In Rust, it's usually downloaded from the same place as the compiler. In bytecode and interpreted languages, like Java, JavaScript, and Python, there's no concept of cross-compilation because there is only one possible target. In C, you usually don't install the library itself, but only the headers that record the API <sup class=\"footnote-reference\" id=\"fr-1-1\"><a href=\"#fn-1\">2<\/a><\/sup>. That brings us to our next topic:<\/p>\n<h4 id=\"libc\">libc<a class=\"zola-anchor\" href=\"#libc\" aria-label=\"Anchor link for: libc\"><\/a>\n<\/h4>\n<p>Generally, people refer to one of two things when they say \"libc\". Either they mean the C standard library, <code>libc.so<\/code>, or the C runtime, <code>crt1.o<\/code>.<\/p>\n<p>Libc matters a lot for two reasons. Firstly, <a href=\"https:\/\/faultlore.com\/blah\/c-isnt-a-language\/\">C is no longer a language<\/a>, so generally the first step to porting <em>any<\/em> language to a new platform is to make sure you have a C <a href=\"https:\/\/stackoverflow.com\/a\/69006179\">toolchain<\/a> <sup class=\"footnote-reference\" id=\"fr-2-1\"><a href=\"#fn-2\">3<\/a><\/sup>. Secondly, because libc is effectively the interface to a platform, <a href=\"https:\/\/j00ru.vexillium.org\/syscalls\/nt\/64\/\">Windows<\/a>, <a href=\"https:\/\/developer.apple.com\/library\/archive\/qa\/qa1118\/_index.html\">macOS<\/a>, and <a href=\"https:\/\/marc.info\/?l=openbsd-tech&amp;m=169841790407370&amp;w=2\">OpenBSD<\/a> have no stable syscall boundary\u2014you are only allowed to talk to the kernel through their stable libraries (libc, and in the case of Windows several others too).<\/p>\n<p>To talk about <em>why<\/em> they've done this, we have to talk about:<\/p>\n<h4 id=\"dynamic-linking-and-platform-maintainers\">dynamic linking and platform maintainers<a class=\"zola-anchor\" href=\"#dynamic-linking-and-platform-maintainers\" aria-label=\"Anchor link for: dynamic-linking-and-platform-maintainers\"><\/a>\n<\/h4>\n<p><a href=\"https:\/\/clojure.org\/reference\/vars\">Many<\/a> <a href=\"https:\/\/en.wikipedia.org\/wiki\/Dynamic_dispatch\">languages<\/a> have a concept of \"<a href=\"https:\/\/en.wikipedia.org\/wiki\/Name_binding\">early binding<\/a>\", where all variable and function references are resolved at compile time, and \"<a href=\"https:\/\/en.wikipedia.org\/wiki\/Late_binding\">late binding<\/a>\", where they are resolved at runtime. C has this concept too, but it calls it \"linking\" instead of \"binding\". \"late binding\" is called \"dynamic linking\"<sup class=\"footnote-reference\" id=\"fr-3-1\"><a href=\"#fn-3\">4<\/a><\/sup>. References to late-bound variables are resolved by the <a href=\"https:\/\/linux.die.net\/man\/8\/ld.so\">\"dynamic loader\"<\/a> at program startup. Further binding can be done at runtime using <a href=\"https:\/\/man7.org\/linux\/man-pages\/man3\/dlopen.3.html\"><code>dlopen<\/code><\/a> and friends.<\/p>\n<p><a href=\"https:\/\/wiki.debian.org\/SoftwarePackaging#External_libraries\">Platform maintainers<\/a> <a href=\"https:\/\/wiki.debian.org\/StaticLinking#Downsides\">really like dynamic linking<\/a>, for the same reason <a href=\"https:\/\/blogs.gentoo.org\/mgorny\/2021\/02\/19\/the-modern-packagers-security-nightmare\/\">they dislike vendoring<\/a>: late-binding allows them to update a library for all applications on a system at once. This matters a lot for security disclosures, where there is a very short timeline between when a vulnerability is patched and announced and when attackers start exploiting it in the wild.<\/p>\n<p>Application developers <a href=\"https:\/\/vagabond.github.io\/rants\/2013\/06\/21\/z_packagers-dont-know-best\">dislike dynamic linking<\/a> for basically the same reason: it requires them to trust the platform maintainers to do a good job packaging all their dependencies, and it results in their application being deployed in scenarios that <a href=\"https:\/\/blog.yossarian.net\/2021\/02\/28\/Weird-architectures-werent-supported-to-begin-with\">they haven't considered or tested<\/a>. For example, installing openssl on Windows is really quite hard. Actually, while I was writing this, a friend overheard me say \"dynamically linking openssl\" and said \"oh god you're giving me nightmares\".<\/p>\n<p>Perhaps a good way to think about dynamically linking as commonly used is <em>a mechanism for devendoring libraries in a compiled program<\/em>. Dynamic linking has other use cases, but they are comparatively rare.<\/p>\n<p>Whether a build system (or language) makes it easy or hard to dynamically link a program is one of the major things that distinguishes it. More about that later.<\/p>\n<h3 id=\"toolchains\">toolchains<a class=\"zola-anchor\" href=\"#toolchains\" aria-label=\"Anchor link for: toolchains\"><\/a>\n<\/h3>\n<p>Ok. So. Back to cross-compiling.<\/p>\n<p>To cross-compile a program, you need:<\/p>\n<ul>\n<li>A compiler for that target. If you're using clang or Rust, this is as simple as passing <code>--target<\/code>. If you're using gcc, you need a <a href=\"https:\/\/gcc.gnu.org\/onlinedocs\/gcc-15.2.0\/gcc.pdf#Invoking%20GCC\">whole-ass extra compiler installed<\/a>.<\/li>\n<li>A standard library for that target. This is very language-specific, but at a minimum requires a working C toolchain <sup class=\"footnote-reference\" id=\"fr-4-1\"><a href=\"#fn-4\">5<\/a><\/sup>.<\/li>\n<li>A linker for that target. This is usually shipped with the compiler, but I mention it specifically because it's usually the most platform specific part. For example, \"not having the macOS linker\" is <em>the<\/em> reason that <a href=\"https:\/\/github.com\/tpoechtrager\/osxcross?tab=readme-ov-file#how-it-works\">cross-compiling to macOS is hard<\/a>.<\/li>\n<\/ul>\n<p>Where does your toolchain come from? ...<br \/>\n...<br \/>\n...<br \/>\n... It turns out <a href=\"https:\/\/rustc-dev-guide.rust-lang.org\/building\/bootstrapping\/what-bootstrapping-does.html#stages-of-bootstrapping\">this is a hard problem<\/a>. Most build systems sidestep it by \"not worrying about it\"; basically any Makefile you find is horribly broken if you update your compiler without running <code>make clean<\/code> afterwards. Cargo is a lot smarter\u2014it caches output in <code>target\/.rustc_info.json<\/code>, and rebuilds if <code>rustc --version --verbose<\/code> changes. \"How do you deal with toolchain invalidations\" is another important things that distinguishes a build system, as we'll see later.<\/p>\n<h3 id=\"environments\">environments<a class=\"zola-anchor\" href=\"#environments\" aria-label=\"Anchor link for: environments\"><\/a>\n<\/h3>\n<p>toolchains are a special case of a more general problem: your build depends on your <em>whole build environment<\/em>, not just the files passed as inputs to your compiler. That means, for instance, that people can\u2014and often do\u2014download things off the internet, <a href=\"https:\/\/rustc-dev-guide.rust-lang.org\/rustdoc.html#multiple-runs-same-output-directory\">embed previous build artifacts in later ones<\/a>, and <a href=\"https:\/\/docs.rs\/openssl\/latest\/openssl\/#vendored\">run entire nested compiler invocations<\/a>.<\/p>\n<h3 id=\"reproducible-builds\">reproducible builds<a class=\"zola-anchor\" href=\"#reproducible-builds\" aria-label=\"Anchor link for: reproducible-builds\"><\/a>\n<\/h3>\n<p>Once we get towards these higher levels of complexity, people want to start doing quite complicated things with caching. In order for caching to be sound, we need the same invocation of the compiler to emit the same output every time, which is called a <strong>reproducible build<\/strong>. This is much harder than it sounds! There are many things programs do that cause non-determinism that programmers often don\u2019t think about (for example, iterating a hashmap or a directory listing).<\/p>\n<p>At the very highest end, people want to conduct builds across multiple machines, and combine those artifacts. At this point, we can\u2019t even allow reading absolute paths, since those will be different between machines. The common tool for this is a compiler flag called <code>--remap-path-prefix<\/code>, and allows the build system to map an absolute path to a relative one. <code>--remap-path-prefix<\/code> is also how rustc is able to print the sources of the standard library when emitting diagnostics, even when running on a different machine than where it was built.<\/p>\n<h2 id=\"tradeoffs\">tradeoffs<a class=\"zola-anchor\" href=\"#tradeoffs\" aria-label=\"Anchor link for: tradeoffs\"><\/a>\n<\/h2>\n<p>At this point, we have enough information to start talking about the space of tradeoffs for a build system.<\/p>\n<h3 id=\"configuration-language\">configuration language<a class=\"zola-anchor\" href=\"#configuration-language\" aria-label=\"Anchor link for: configuration-language\"><\/a>\n<\/h3>\n<blockquote>\n<p>Putting your config in a YAML file does not make it declarative! Limiting yourself to a Turing-incomplete language does not automatically make your code easier to read!\n\u2014<a href=\"https:\/\/tech.lgbt\/@jyn\/112617315565817787\">jyn<\/a><\/p>\n<\/blockquote>\n<p>The most common unforced error I see build systems making is forcing the build configuration to be written in a custom language<sup class=\"footnote-reference\" id=\"fr-6-1\"><a href=\"#fn-6\">6<\/a><\/sup>. There are basically two reasons they do this:<\/p>\n<ul>\n<li>Programmers aren't used to treating build system code as code. This is a culture issue that's hard to change, but it's worthwhile to try anyway.<\/li>\n<li>There is some idea of making builds \"declarative\". (In fact, observant readers may observe that this corresponds to the idea of <a href=\"https:\/\/jyn.dev\/constrained-languages-are-easier-to-optimize\/\">\"constrained languages\"<\/a> I talk about in an earlier post.) This is not by itself a bad idea! The problem is it doesn't give them the properties you might want. For example, one property you might want is \"another tool can reimplement the build algorithm\". Unfortunately this quickly becomes <a href=\"https:\/\/docs.jade.fyi\/gnu\/make.html#Features-of-GNU-make\">infeasible for complicated algorithms<\/a>. Another you might want is \"what will rebuild next time a build occurs?\". You can't get this from the configuration without\u2014again\u2014reimplementing the algorithm.<\/li>\n<\/ul>\n<p>Right. So, given that making a build \"declarative\" is a lie, you may as well give programmers a real language. Some common choices are:<\/p>\n<ul>\n<li><a href=\"https:\/\/starlark-lang.org\/\">Starlark<\/a> <sup class=\"footnote-reference\" id=\"fr-7-1\"><a href=\"#fn-7\">7<\/a><\/sup><\/li>\n<li><a href=\"https:\/\/groovy-lang.org\/\">Groovy<\/a><\/li>\n<li>\u201cthe same language that the build system was written in\u201d (examples: <a href=\"https:\/\/boot-clj.github.io\/\">Clojure<\/a>, <a href=\"https:\/\/ziglang.org\/learn\/build-system\/\">Zig<\/a>, <a href=\"https:\/\/gruntjs.com\/sample-gruntfile\">JavaScript<\/a>)<\/li>\n<\/ul>\n<h3 id=\"reflection\">reflection<a class=\"zola-anchor\" href=\"#reflection\" aria-label=\"Anchor link for: reflection\"><\/a>\n<\/h3>\n<p>\"But wait, jyn!\", you may say. \"Surely you aren't suggesting a build system where you have to run a whole program every time you figure out what to rebuild??\"<\/p>\n<p>I mean ... people are doing it. But just because they're doing it doesn't mean it's a good idea, so let's look at the alternative, which is to <em>serialize your build graph<\/em>. This is easier to see than explain, so let's look at an example using the <a href=\"https:\/\/ninja-build.org\/\">Ninja build system<\/a> <sup class=\"footnote-reference\" id=\"fr-8-1\"><a href=\"#fn-8\">8<\/a><\/sup>:<\/p>\n<pre data-lang=\"ninja\" class=\"language-ninja z-code\"><code class=\"language-ninja\" data-lang=\"ninja\"><span class=\"z-text z-plain\">rule svg2pdf\n<\/span><span class=\"z-text z-plain\">  command = inkscape $in --export-text-to-path --export-pdf=$out\n<\/span><span class=\"z-text z-plain\">  description = svg2pdf $in $out\n<\/span><span class=\"z-text z-plain\">  \n<\/span><span class=\"z-text z-plain\">build pdfs\/variables.pdf: svg2pdf variables.svg\n<\/span><\/code><\/pre>\n<p>Ninjafiles give you the absolute bare minimum necessary to express your build dependencies: You get \"rules\", which explain <em>how<\/em> to build an output; \"build edges\", which state <em>when<\/em> to build; and \"variables\", which say <em>what<\/em> to build<sup class=\"footnote-reference\" id=\"fr-12-1\"><a href=\"#fn-12\">9<\/a><\/sup>. That's basically it. There's some subtleties about \"depfiles\" which can be used to dynamically add build edges while running the build rule.<\/p>\n<p>Because the features are so minimal, the files are intended to be generated, using a configure script written in one of the languages we talked about earlier. The most common generators are CMake and <a href=\"https:\/\/gn.googlesource.com\/gn\/#gn\">GN<\/a>, but you can use any language you like because the format is so simple.<\/p>\n<p>What's really cool about this is that it's trivial to parse, which means that it's very easy to write your own implementation of ninja if you want. It also means that you <em>can<\/em> get a lot of the properties we discussed before, i.e.:<\/p>\n<ul>\n<li>\"Show me all commands that are run on a full build\" (<code>ninja -t commands<\/code>)<\/li>\n<li>\"Show me all commands that will be run the next time an incremental build is run\" (<code>ninja -n -d explain<\/code>)<\/li>\n<li>\"If this <em>particular<\/em> source file is changed, what will need to be rebuilt?\" (<code>ninja -t query variables.svg<\/code>)<\/li>\n<\/ul>\n<p>It turns out these properties are very useful.<\/p>\n<aside role=note class=note-container><div class=note-content>\n<p>\"jyn, you're taking too long to get to the point!\" look I\u2019m getting there, I promise.<\/p>\n<\/div><\/aside>\n<p>The main downsides to this approach is that it has to be <em>possible<\/em> to serialize your build graph. In one sense, I see this as good, actually, because you have to think through everything your build does ahead of time. But on the other hand, if you have things like nested ninja invocations, or like our <code>cargo test<\/code> -&gt; <code>cargo build<\/code> example <a href=\"https:\/\/jyn.dev\/build-system-tradeoffs\/#running-generated-binaries\">from earlier<\/a> <sup class=\"footnote-reference\" id=\"fr-9-1\"><a href=\"#fn-9\">10<\/a><\/sup>, all the tools to query the build graph don't include the information you expect.<\/p>\n<h3 id=\"file-watching\">file watching<a class=\"zola-anchor\" href=\"#file-watching\" aria-label=\"Anchor link for: file-watching\"><\/a>\n<\/h3>\n<p>Re-stat'ing all files in a source directory is expensive. It would be much nicer if we could have a pull model instead of a push model, where the build tool gets notified of file changes and rebuilds exactly the necessary files. There are some tools with native integration for this, like <a href=\"https:\/\/gittup.org\/tup\/manual.html#:~:text=monitor\">Tup<\/a>, <a href=\"https:\/\/github.com\/capnproto\/ekam\/\">Ekam<\/a>, <a href=\"https:\/\/jj-vcs.github.io\/jj\/latest\/config\/#filesystem-monitor\">jj<\/a>, and <a href=\"https:\/\/engineering.fb.com\/2023\/04\/06\/open-source\/buck2-open-source-large-scale-build-system\/#:~:text=integrate%20with%20virtual%20file\">Buck2<\/a>, but generally it's pretty rare.<\/p>\n<p>That's ok if we have reflection, though! We can write our own file monitoring tool, ask the build system which files need to rebuilt for the changed inputs, and then tell it to rebuild only those files. That prevents it from having to recursively stat all files in the graph.<\/p>\n<p>See <a href=\"https:\/\/gittup.org\/tup\/build_system_rules_and_algorithms.pdf\">Tup's paper<\/a> for more information about the big idea here.<\/p>\n<h3 id=\"dependency-tracking\">dependency tracking<a class=\"zola-anchor\" href=\"#dependency-tracking\" aria-label=\"Anchor link for: dependency-tracking\"><\/a>\n<\/h3>\n<p>Ok, let's assume we have some build system that uses some programming language to generate a build graph, and it rebuilds exactly the necessary outputs on changes to our inputs. What exactly are our inputs?<\/p>\n<p>There are basically four major approaches to dependency tracking in the build space.<\/p>\n<h4 id=\"not-my-problem-lol\">\"not my problem, lol\"<a class=\"zola-anchor\" href=\"#not-my-problem-lol\" aria-label=\"Anchor link for: not-my-problem-lol\"><\/a>\n<\/h4>\n<p>This kind of build system externalizes all concerns out to you, the programmer. When I say \"externalizes all concerns\", I mean that you are required to write in all your dependencies by hand, and the tool doesn't help you get them right. Some examples:<\/p>\n<ul>\n<li><code>make<\/code><\/li>\n<li>Github Actions <a href=\"https:\/\/github.com\/actions\/cache\">cache actions<\/a> (and in general, most CI caching I'm aware of requires you to manually write out the files your cache depends on)<\/li>\n<li>Ansible playbooks<\/li>\n<li>Basically most build systems, this is extremely common<\/li>\n<\/ul>\n<p>A common problem with this category of build system is that people forget to mark <em>the build rule itself<\/em> as an input to the graph, resulting in dead artifacts left laying around, and as a result, <a href=\"https:\/\/jacko.io\/safety_and_soundness.html\">unsound<\/a> builds.<\/p>\n<p>In my humble opinion, this kind of tool is only useful as a serialization layer for a build graph, or if you have no other choice. Here's a nickel, kid, <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2016\/03\/hadrian.pdf\">get yourself a better build system<\/a>.<\/p>\n<h5 id=\"compiler-support\">compiler support<a class=\"zola-anchor\" href=\"#compiler-support\" aria-label=\"Anchor link for: compiler-support\"><\/a>\n<\/h5>\n<p>Sometimes build systems (CMake, Cargo, maybe others I don't know) do a little better and use the compiler's built-in support for dependency tracking (e.g. <a href=\"https:\/\/gcc.gnu.org\/onlinedocs\/gcc\/Preprocessor-Options.html#index-M\"><code>gcc -M<\/code><\/a> or <a href=\"https:\/\/doc.rust-lang.org\/rustc\/command-line-arguments.html#--emit-specifies-the-types-of-output-files-to-generate\"><code>rustc --emit=dep-info<\/code><\/a>), and automatically add dependencies on the build rules themselves. This is a lot better than nothing, and much more reliable than tracking dependencies by hand. But it still fundamentally trusts the compiler to be correct, and doesn't track environment dependencies.<\/p>\n<h4 id=\"ephemeral-state\">ephemeral state<a class=\"zola-anchor\" href=\"#ephemeral-state\" aria-label=\"Anchor link for: ephemeral-state\"><\/a>\n<\/h4>\n<p>This kind of build system always does a full build, and lets you modify the environment in arbitrary ways as you do so. This is simple, always correct, and expensive. Some examples:<\/p>\n<ul>\n<li><code>make clean &amp;&amp; make<\/code> (kinda\u2014assuming that <code>make clean<\/code> is reliable, which it often isn't.)<\/li>\n<li>Github Actions (<code>step:<\/code> rules)<\/li>\n<li>Docker containers (time between startup and shutdown)<\/li>\n<li>Initramfs (time between initial load and chroot into the full system)<\/li>\n<li>Systemd service startup rules<\/li>\n<li>Most compiler invocations (e.g. <code>cc file.c -o file<\/code>)<\/li>\n<\/ul>\n<p>These are ok if you can afford them. But they are expensive! Most people using Github Actions are only doing so because GHA is a hyperscaler giving away free CI time like there's no tomorrow. I suspect we would see far less wasted CPU-hours if people had to consider the actual costs of using them.<\/p>\n<h4 id=\"hermetic-builds\">hermetic builds<a class=\"zola-anchor\" href=\"#hermetic-builds\" aria-label=\"Anchor link for: hermetic-builds\"><\/a>\n<\/h4>\n<p>This is the kind of phrase that's well-known to people who work on build systems and basically unheard of outside it, alongside \"monadic builds\". \"hermetic\" means that <em>the only things in your environment are those you have explicitly put there<\/em>. This sometimes called \"sandboxing\", although that has unfortunate connotations about security that don't always apply here. Some examples of this:<\/p>\n<ul>\n<li>Dockerfiles (more generally, OCI images)<\/li>\n<li>nix<\/li>\n<li>Bazel \/ Buck2<\/li>\n<li><a href=\"https:\/\/github.com\/bazelbuild\/remote-apis\">\"Bazel Remote Execution Protocol\"<\/a> (not actually tied to Bazel), which lets you run an arbitrary set of build commands on a remote worker<\/li>\n<\/ul>\n<p>This has a lot of benefits! It statically guarantees that you <em>cannot<\/em> forget any of your inputs; it is 100% reliable, assuming no issues with the network or with the implementing tool \ud83d\ude43; and it gives you very very granular insight into what your dependencies actually are. Some things you can <em>do<\/em> with a hermetic build system:<\/p>\n<ul>\n<li>On a change to some source code, rerun only the affected tests. You know statically which those are, because the build tool forced you to write out the dependency edges.<\/li>\n<li>Remote caching. If you have the same environment everywhere, you can upload your cache to the cloud, and download and reuse it again on another machine. You can do this in CI\u2014but you can also do it locally! The time for a \"\"\"full build\"\"\" can be almost instantaneous because when a new engineer gets onboarded they can immediately reuse everyone else's build cache.<\/li>\n<\/ul>\n<p>The main downside is that you have to actually specify all those dependencies (if you don't, you get a hard error instead of an unsound build graph, which is the main difference between hermetic systems and \"not my problem\"). Bazel and Buck2 give you starlark, so you have a ~real<sup class=\"footnote-reference\" id=\"fr-10-1\"><a href=\"#fn-10\">11<\/a><\/sup> language in which to do it, but it's still a ton of work. Both have an enormous \"prelude\" module that just defines where you get a compiler toolchain from<sup class=\"footnote-reference\" id=\"fr-13-1\"><a href=\"#fn-13\">12<\/a><\/sup>.<\/p>\n<p>Nix can be thought of as taking this \"prelude\" idea all the way, by expanding the \"prelude\" (nixpkgs) to \"everything that's ever been packaged for NixOS\". When you write <code>import &lt;nixpkgs&gt;<\/code>, your nix build is logically in the same build graph as the nixpkgs monorepo; it just happens to have an enormous remote cache already pre-built.<\/p>\n<aside role=note class=note-container><div class=note-content>\n<p>Bazel and Buck2 don\u2019t have anything like nixpkgs, which is the main reason that using them requires a full time dedicated build engineer: that engineer has to keep writing build rules from scratch any time you add an external dependency. They also have to package any language toolchains that aren\u2019t in the prelude.<\/p>\n<\/div><\/aside>\n<p>Nix has one more interesting property, which is that all its packages compose. You can install two different versions of the same package and that's fine because they use different <a href=\"https:\/\/nix.dev\/manual\/nix\/2.24\/store\/\">store<\/a> paths. They fit together like lesbians' fingers interlock.<\/p>\n<p>Compare this to docker, which does <em>not<\/em> compose<sup class=\"footnote-reference\" id=\"fr-11-1\"><a href=\"#fn-11\">13<\/a><\/sup>. In docker, there is no way to say \"Inherit the build environment from multiple different source images\". The closest you can get is a \"multi-stage build\", where you explicitly copy over individual files from an earlier image to a later image. It can't blindly copy over all the files because some of them might want to end up at the same path, and touching fingers would be gay.<\/p>\n<h4 id=\"tracing\">tracing<a class=\"zola-anchor\" href=\"#tracing\" aria-label=\"Anchor link for: tracing\"><\/a>\n<\/h4>\n<p>The last kind I'm aware of, and the rarest I've seen, is <em>tracing<\/em> build systems. These have the same goal as hermetic build systems: they still want 100% of your dependencies to be specified. But they go about it in a different way. Rather than sandboxing your code and only allowing access to the dependencies you specify, they <em>instrument<\/em> your code, tracing its file accesses, and record the dependencies of each build step. Some examples:<\/p>\n<ul>\n<li><a href=\"https:\/\/gittup.org\/tup\/ex_a_first_tupfile.html#:~:text=a%20program%20grows\">Tup<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/rizsotto\/Bear\">Bear<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/curtsinger-lab\/riker\">Riker<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/capnproto\/ekam\/\">Ekam<\/a><\/li>\n<li><a href=\"https:\/\/rustc-dev-guide.rust-lang.org\/query.html\">The rust compiler, actually<\/a><\/li>\n<li>\"A build system with orthogonal persistence\" (<a href=\"https:\/\/jyn.dev\/complected-and-orthogonal-persistence\/#how-far-can-we-take-this\">previously<\/a>; <a href=\"https:\/\/jade.fyi\/blog\/the-postmodern-build-system\/#make-an-existing-architecture-and-os-deterministic\">previously<\/a>; <a href=\"https:\/\/nlnet.nl\/project\/Ripple\/\">previously<\/a>)<\/li>\n<\/ul>\n<p>The advantage of these is that you get all the benefits of a hermetic build system without any of the cost of having to write out your dependencies.<\/p>\n<p>The first main disadvantage is that they require the kernel to support syscall tracing, which essentially means they only work on Linux. I have Ideas\u2122 for how to get this working on macOS without disabling SIP, but they're still incomplete and not fully general; I may write a follow-up post about that. I don't yet have ideas for how this could work on Windows, but <a href=\"https:\/\/stackoverflow.com\/questions\/864839\/monitoring-certain-system-calls-done-by-a-process-in-windows\">it seems possible<\/a>.<\/p>\n<p>The second main disadvantage is that not knowing the graph up front causes many issues for the build system. In particular:<\/p>\n<ul>\n<li>If you change the graph, it doesn't find out until the next time it reruns a build. This can lead to degenerate cases where the same rule has to be run multiple times until it doesn't access any new inputs.<\/li>\n<li>If you don't cache the graph, you have that problem on <em>every edge in the graph<\/em>. <a href=\"https:\/\/github.com\/capnproto\/ekam\/issues\/63\">This is the problem Ekam has<\/a>, and makes it very slow to run full builds. Its solution is to run in \"watch\" mode, where it caches the graph in-memory instead of on-disk.<\/li>\n<li>If you do cache the graph, you can only do so for so long before it becomes prohibitively expensive to do that for all possible executions. For \"normal\" codebases this isn't a problem, but if you're Google or Facebook, this is actually a practical concern. I think it is still possible to do this with a tracing build system (by having your cache points look a lot more like many Bazel BUILD files than a single top-level Ninja file), but no one has ever tried it at that scale.<\/li>\n<li>If the same file can come from many possible places, due to multiple search paths (e.g. a <code>&lt;system&gt;<\/code> include header in C, or any import really in a JVM language), then you have a very rough time specifying what your dependencies actually are. The best ninja can do is say \u201cdepend on the whole directory containing that file\u201d, which sucks because it rebuilds <em>whenever<\/em> that directory changes, not just when your new file is added. It\u2019s possible to work around this with a (theoretical) serialization format other than Ninja, but regardless, you\u2019re adding lots of file <code>stat()<\/code>s to your hot path.<\/li>\n<li>The build system does not know which dependencies are direct (specified by you, the owner of the module being compiled) and which are transient (specified by the modules you depend on). This makes error reporting worse, and generally lets you do fewer kinds of queries on the graph.<\/li>\n<li>My friend Alexis Hunt, a build system expert, says \"there are deeper pathologies down that route of madness\". So. That's concerning.<\/li>\n<\/ul>\n<p>I have been convinced that tracing is useful as a tool to <em>generate<\/em> your build graph, but not as a tool actually used when executing it. Compare also <a href=\"https:\/\/github.com\/bazel-contrib\/bazel-gazelle\">gazelle<\/a>, which is something like that for Bazel, but based on parsing source files rather than tracking syscalls.<\/p>\n<p>Combining paradigms in this way also make it possible to verify your hermetic builds in ways that are hard to do with mere sandboxing. For example, a tracing build system can catch missing dependencies:<\/p>\n<ul>\n<li>emitting untracked <em>outputs<\/em><\/li>\n<li>overwriting source files (!),<\/li>\n<li>using an input file that was registered for a different rule<\/li>\n<\/ul>\n<p>and it can also detect non-reproducible builds:<\/p>\n<ul>\n<li>reading the current time, or absolute path to the current directory<\/li>\n<li>iterating all files in a directory (this is non-deterministic)<\/li>\n<li>machine and kernel-level sources of randomness.<\/li>\n<\/ul>\n<h2 id=\"future-work\">future work<a class=\"zola-anchor\" href=\"#future-work\" aria-label=\"Anchor link for: future-work\"><\/a>\n<\/h2>\n<p>There's more to talk about here\u2014how build systems affect the dynamics between upstream maintainers and distro packagers; how .a files are <a href=\"https:\/\/medium.com\/@eyal.itkin\/the-a-file-is-a-relic-why-static-archives-were-a-bad-idea-all-along-8cd1cf6310c5\">bad file formats<\/a>; how <a href=\"https:\/\/apenwarr.ca\/log\/20181113\">mtime comparisons are generally bad<\/a>; how configuration options make the tradeoffs much more complicated; how FUSE can let a build system integrate with a VCS to avoid downloading unnecessary files into a shallow checkout; but this post is quite long enough already.<\/p>\n<h2 id=\"takeaways\">takeaways<a class=\"zola-anchor\" href=\"#takeaways\" aria-label=\"Anchor link for: takeaways\"><\/a>\n<\/h2>\n<ul>\n<li>Most build systems do not prioritize correctness.<\/li>\n<li>Prioritizing correctness comes with severe, hard to avoid tradeoffs.<\/li>\n<li>Tracing build systems show the potential to avoid some of those tradeoffs, but are highly platform specific and come with tradeoffs of their own at large enough scale. Combining a tracing build system with a hermetic build system seems like the best of both worlds.<\/li>\n<li>Writing build rules in a \"normal\" (but constrained) programming language, then serializing them to a build graph, has surprisingly few tradeoffs. I'm not sure why more build systems don't do this.<\/li>\n<\/ul>\n<hr \/>\n<h2 id=\"bibliography\">bibliography<a class=\"zola-anchor\" href=\"#bibliography\" aria-label=\"Anchor link for: bibliography\"><\/a>\n<\/h2>\n<ul>\n<li><a href=\"https:\/\/boot-clj.github.io\/\">Alan Dipert, Micha Niskin, Joshua Smith, \u201cBoot: build tooling for Clojure\u201d<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/bazelbuild\/remote-apis\">Alexis Hunt, Ola Rozenfield, and Adrian Ludwin, \u201cbazelbuild\/remote-apis: An API for caching and execution of actions on a remote system.\u201d<\/a><\/li>\n<li><a href=\"https:\/\/andrewkelley.me\/post\/zig-cc-powerful-drop-in-replacement-gcc-clang.html\">Andrew Kelley, \u201czig cc: a Powerful Drop-In Replacement for GCC\/Clang\u201d<\/a><\/li>\n<li><a href=\"https:\/\/vagabond.github.io\/rants\/2013\/06\/21\/z_packagers-dont-know-best\">Andrew Thompson, \u201cPackagers don\u2019t know best\u201d<\/a><\/li>\n<li><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2016\/03\/hadrian.pdf\">Andrey Mokhov et. al., \u201cNon-recursive Make Considered Harmful\u201d<\/a><\/li>\n<li><a href=\"https:\/\/apenwarr.ca\/log\/20181113\">apenwarr, \u201cmtime comparison considered harmful\u201d<\/a><\/li>\n<li><a href=\"https:\/\/developer.apple.com\/library\/archive\/qa\/qa1118\/_index.html\">Apple Inc., \u201cStatically linked binaries on Mac OS X\u201d<\/a><\/li>\n<li><a href=\"https:\/\/faultlore.com\/blah\/c-isnt-a-language\/\">Aria Desires, \u201cC Isn\u2019t A Language Anymore\u201d<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/curtsinger-lab\/riker\">Charlie Curtsinger and Daniel W. Barowy, \u201ccurtsinger-lab\/riker: Always-Correct and Fast Incremental Builds from Simple Specifications\u201d<\/a><\/li>\n<li><a href=\"https:\/\/engineering.fb.com\/2023\/04\/06\/open-source\/buck2-open-source-large-scale-build-system\/\">Chris Hopman and Neil Mitchell, \u201cBuild faster with Buck2: Our open source build system\u201d<\/a><\/li>\n<li><a href=\"https:\/\/wiki.debian.org\/SoftwarePackaging\">Debian, \u201cSoftware Packaging\u201d<\/a><\/li>\n<li><a href=\"https:\/\/wiki.debian.org\/StaticLinking\">Debian, \u201cStatic Linking\u201d<\/a><\/li>\n<li><a href=\"https:\/\/nix.dev\/manual\/nix\/2.24\/store\/\">Dolstra, E., &amp; The CppNix contributors., \u201cNix Store\u201d<\/a><\/li>\n<li><a href=\"https:\/\/medium.com\/@eyal.itkin\/the-a-file-is-a-relic-why-static-archives-were-a-bad-idea-all-along-8cd1cf6310c5\">Eyal Itkin, \u201cThe .a File is a Relic: Why Static Archives Were a Bad Idea All Along\u201d<\/a><\/li>\n<li><a href=\"https:\/\/blog.rust-lang.org\/2021\/05\/10\/Rust-1.52.1\/\">Felix Klock and Mark Rousskov on behalf of the Rust compiler team, \u201cAnnouncing Rust 1.52.1\u201d<\/a><\/li>\n<li><a href=\"https:\/\/docs.jade.fyi\/gnu\/make.html\">Free Software Foundation, Inc., \u201cGNU make\u201d<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/actions\/cache\">GitHub, Inc., \u201cactions\/cache: Cache dependencies and build outputs in GitHub Actions\u201d<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/bazel-contrib\/bazel-gazelle\">Google Inc., \u201cbazel-contrib\/bazel-gazelle: a Bazel build file generator for Bazel projects\u201d<\/a><\/li>\n<li><a href=\"https:\/\/jj-vcs.github.io\/jj\/latest\/config\/\">Google LLC, \u201cJujutsu docs\u201d<\/a><\/li>\n<li><a href=\"https:\/\/docs.rs\/openssl\/latest\/openssl\/\">Jack Lloyd and Steven Fackler, \u201crust-openssl\u201d<\/a><\/li>\n<li><a href=\"https:\/\/jacko.io\/safety_and_soundness.html\">Jack O\u2019Connor, \u201cSafety and Soundness in Rust\u201d<\/a><\/li>\n<li><a href=\"https:\/\/jade.fyi\/blog\/the-postmodern-build-system\/\">Jade Lovelace, \u201cThe postmodern build system\u201d<\/a><\/li>\n<li><a href=\"https:\/\/jvns.ca\/blog\/2020\/10\/26\/ninja--a-simple-way-to-do-builds\/\">Julia Evans, \u201cninja: a simple way to do builds\u201d<\/a><\/li>\n<li><a href=\"https:\/\/jyn.dev\/complected-and-orthogonal-persistence\/\">jyn, \u201cComplected and Orthogonal Persistence\u201d<\/a><\/li>\n<li><a href=\"https:\/\/jyn.dev\/constrained-languages-are-easier-to-optimize\/\">jyn, \u201cConstrained Languages are Easier to Optimize\u201d<\/a><\/li>\n<li><a href=\"https:\/\/tech.lgbt\/@jyn\/112617315565817787\">jyn, \u201ci think i have identified what i dislike about ansible\u201d<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/capnproto\/ekam\/\">Kenton Varda, \u201cEkam Build System\u201d<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/rizsotto\/Bear\">L\u00e1szl\u00f3 Nagy, \u201crizsotto\/Bear: a tool that generates a compilation database for clang tooling\u201d<\/a><\/li>\n<li><a href=\"https:\/\/starlark-lang.org\/\">Laurent Le Brun, \u201cStarlark Programming Language\u201d<\/a><\/li>\n<li><a href=\"https:\/\/j00ru.vexillium.org\/syscalls\/nt\/64\/\">Mateusz \u201cj00ru\u201d Jurczyk, \u201cWindows X86-64 System Call Table (XP\/2003\/Vista\/7\/8\/10\/11 and Server)\u201d<\/a><\/li>\n<li><a href=\"https:\/\/blogs.gentoo.org\/mgorny\/2021\/02\/19\/the-modern-packagers-security-nightmare\/\">Micha\u0142 G\u00f3rny, \u201cThe modern packager\u2019s security nightmare\u201d<\/a><\/li>\n<li><a href=\"https:\/\/gittup.org\/tup\/ex_a_first_tupfile.html\">Mike Shal, \u201cA\u00a0First\u00a0Tupfile\u201d<\/a><\/li>\n<li><a href=\"https:\/\/gittup.org\/tup\/build_system_rules_and_algorithms.pdf\">Mike Shal, \u201cBuild System Rules and Algorithms\u201d<\/a><\/li>\n<li><a href=\"https:\/\/gittup.org\/tup\/manual.html\">Mike Shal, \u201ctup\u201d<\/a><\/li>\n<li><a href=\"https:\/\/ninja-build.org\/\">Nico Weber, \u201cNinja, a small build system with a focus on speed\u201d<\/a><\/li>\n<li><a href=\"https:\/\/nlnet.nl\/project\/Ripple\/\">NLnet, \u201cRipple\u201d<\/a><\/li>\n<li><a href=\"https:\/\/gruntjs.com\/sample-gruntfile\">OpenJS Foundation, \u201cGrunt: The JavaScript Task Runner\u201d<\/a><\/li>\n<li><a href=\"https:\/\/gcc.gnu.org\/onlinedocs\/gcc\/Preprocessor-Options.html\">\u201cPreprocessor Options (Using the GNU Compiler Collection (GCC))\u201d<\/a><\/li>\n<li><a href=\"https:\/\/xkcd.com\/2501\/\">Randall Munroe, \u201cxkcd: Average Familiarity\u201d<\/a><\/li>\n<li><a href=\"https:\/\/gcc.gnu.org\/onlinedocs\/gcc-15.2.0\/gcc.pdf\">Richard M. Stallman and the GCC Developer Community, \u201cInvoking GCC\u201d<\/a><\/li>\n<li><a href=\"https:\/\/clojure.org\/reference\/vars\">Rich Hickey, \u201cClojure - Vars and the Global Environment\u201d<\/a><\/li>\n<li><a href=\"https:\/\/langdev.stackexchange.com\/questions\/153\/what-are-the-advantages-of-requiring-forward-declaration-of-methods-fields-like\/\">Stack Exchange, \u201cWhat are the advantages of requiring forward declaration of methods\/fields like C\/C++ does?\u201d<\/a><\/li>\n<li><a href=\"https:\/\/stackoverflow.com\/questions\/864839\/monitoring-certain-system-calls-done-by-a-process-in-windows\">Stack Overflow, \u201cMonitoring certain system calls done by a process in Windows\u201d<\/a><\/li>\n<li><a href=\"https:\/\/man7.org\/linux\/man-pages\/man3\/dlopen.3.html\">System Calls Manual, \u201cdlopen(3)\u201d<\/a><\/li>\n<li><a href=\"https:\/\/linux.die.net\/man\/8\/ld.so\">System Manager\u2019s Manual, \u201cld.so(8)\u201d<\/a><\/li>\n<li><a href=\"https:\/\/groovy-lang.org\/\">The Apache Groovy project, \u201cThe Apache Groovy\u2122 programming language\u201d<\/a><\/li>\n<li><a href=\"https:\/\/gn.googlesource.com\/gn\/\">The Chromium Authors, \u201cgn\u201d<\/a><\/li>\n<li><a href=\"https:\/\/marc.info\/?l=openbsd-tech&amp;m=169841790407370&amp;w=2\">Theo de Raadt, \u201cRemoving syscall(2) from libc and kernel\u201d<\/a><\/li>\n<li><a href=\"https:\/\/rustc-dev-guide.rust-lang.org\/building\/bootstrapping\/intro.html\">The Rust Project Contributors, \u201cBootstrapping the compiler\u201d<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/rust-lang\/rust\/issues\/11937\">The Rust Project Contributors, \u201cLink using the linker directly\u201d<\/a><\/li>\n<li><a href=\"https:\/\/rustc-dev-guide.rust-lang.org\/rustdoc.html\">The Rust Project Contributors, \u201cRustdoc overview - Multiple runs, same output directory\u201d<\/a><\/li>\n<li><a href=\"https:\/\/doc.rust-lang.org\/cargo\/\">The Rust Project Contributors, \u201cThe Cargo Book\u201d<\/a><\/li>\n<li><a href=\"https:\/\/doc.rust-lang.org\/rustc\/command-line-arguments.html\">The Rust Project Contributors, \u201cCommand-line Arguments - The rustc book\u201d<\/a><\/li>\n<li><a href=\"https:\/\/rustc-dev-guide.rust-lang.org\/query.html\">The Rust Project Contributors, \u201cQueries: demand-driven compilation\u201d<\/a><\/li>\n<li><a href=\"https:\/\/rustc-dev-guide.rust-lang.org\/building\/bootstrapping\/what-bootstrapping-does.html\">The Rust Project Contributors, \u201cWhat Bootstrapping does\u201d<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/tpoechtrager\/osxcross?tab=readme-ov-file\">Thomas P\u00f6chtrager, \u201cMacOS Cross-Toolchain for Linux and *BSD\u201d<\/a><\/li>\n<li><a href=\"https:\/\/stackoverflow.com\/a\/69006179\">\u201cWhat is a compiler toolchain? - Stack Overflow\u201d<\/a><\/li>\n<li><a href=\"https:\/\/en.wikipedia.org\/wiki\/Dynamic_dispatch\">Wikipedia, \u201cDynamic dispatch\u201d<\/a><\/li>\n<li><a href=\"https:\/\/en.wikipedia.org\/wiki\/Late_binding\">Wikipedia, \u201cLate binding\u201d<\/a><\/li>\n<li><a href=\"https:\/\/en.wikipedia.org\/wiki\/Name_binding\">Wikipedia, \u201cName binding\u201d<\/a><\/li>\n<li><a href=\"https:\/\/blog.yossarian.net\/2021\/02\/28\/Weird-architectures-werent-supported-to-begin-with\">william woodruff, \u201cWeird architectures weren\u2019t supported to begin with\u201d<\/a><\/li>\n<li><a href=\"https:\/\/ziglang.org\/learn\/build-system\/\">Zig contributors, \u201cZig Build System\u201c<\/a><\/li>\n<\/ul>\n<section class=\"footnotes\">\n<ol class=\"footnotes-list\">\n<li id=\"fn-14\">\n<p>the uncommon case mostly looks like <a href=\"https:\/\/blog.rust-lang.org\/2021\/05\/10\/Rust-1.52.1\/\">incremental bugs in rustc itself<\/a>, or issues around rerunning build scripts. <a href=\"#fr-14-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-1\">\n<p>see <a href=\"https:\/\/langdev.stackexchange.com\/questions\/153\/what-are-the-advantages-of-requiring-forward-declaration-of-methods-fields-like\/\">this stackexchange post<\/a> for more discussion about the tradeoffs between forward declarations and requiring full access to the source. <a href=\"#fr-1-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-2\">\n<p>even <a href=\"https:\/\/github.com\/rust-lang\/rust\/issues\/11937\">Rust depends on crt1.o when linking<\/a>! <a href=\"#fr-2-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-3\">\n<p>early binding is called \"static linking\". <a href=\"#fr-3-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-4\">\n<p>actually, Zig solved this in <a href=\"https:\/\/andrewkelley.me\/post\/zig-cc-powerful-drop-in-replacement-gcc-clang.html\">the funniest way possible<\/a>, by bundling a C toolchain with their Zig compiler. This is a legitimately quite impressive feat. If there's any Zig contributors reading\u2014props to you, you did a great job. <a href=\"#fr-4-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-6\">\n<p>almost every build system does this, so I don't even feel compelled to name names. <a href=\"#fr-6-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-7\">\n<p>Starlark is <em>not<\/em> tied to hermetic build systems. The fact that the only common uses of it are in hermetic build systems is unfortunate. <a href=\"#fr-7-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-8\">\n<p>H.T. <a href=\"https:\/\/jvns.ca\/blog\/2020\/10\/26\/ninja--a-simple-way-to-do-builds\/\">Julia Evans<\/a> <a href=\"#fr-8-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-12\">\n<p>actually variables are more general than this, but for $in and $out this is true. <a href=\"#fr-12-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-9\">\n<p>another example is  \"rebuilding build.ninja when the build graph changes\". it's more common than you think because the language is so limited that it's easier to rerun the configure script than to try and fit the dependency info into the graph. <a href=\"#fr-9-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-10\">\n<p>not actually turing-complete <a href=\"#fr-10-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-13\">\n<p>I have been informed that the open-source version of Bazel is not actually hermetic-by-default inside of its prelude, and just uses system libraries. This is quite unfortunate; with this method of using Bazel you are getting a lot of the downsides and little of the upside. Most people I know using it are doing so in the hermetic mode. <a href=\"#fr-13-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-11\">\n<p>there's something <em>called<\/em> <code>docker compose<\/code>, but it composes containers, not images. <a href=\"#fr-11-1\">\u21a9<\/a><\/p>\n<\/li>\n<\/ol>\n<\/section>\n"},{"title":"the core of rust","published":"2025-08-21T00:00:00+00:00","updated":"2025-08-21T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/the-core-of-rust\/"}},"id":"https:\/\/jyn.dev\/the-core-of-rust\/","summary":"within Rust is a smaller language struggling to get out","content":"<p><strong>NOTE: this is not a rust tutorial.<\/strong><\/p>\n<figure>\n<blockquote cite=\"https:\/\/donsz.nl\/\">\n<p>Every year it was an incredible challenge to fit teaching Rust into lectures since you basically need all the concepts right from the start to understand a lot of programs. I never knew how to order things. The flip side was that usually when you understand all the basic components in play lots of it just fits together. i.e. there's some point where the interwovenness turns from a barrier into something incredibly valuable and helpful.<\/p>\n<\/blockquote>\n<figcaption>\n<cite ><a href=\"https:\/\/donsz.nl\/\">Jana D\u00f6nszelmann<\/a><\/cite>\n<\/figcaption>\n<\/figure>\n<h2 id=\"vision\">Vision<a class=\"zola-anchor\" href=\"#vision\" aria-label=\"Anchor link for: vision\"><\/a>\n<\/h2>\n<p>One thing I admire in a language is a strong vision. <a href=\"https:\/\/www.uiua.org\/\">Uiua<\/a>, for example, has a very strong vision: what does it take to eliminate all local named variables from a language? <a href=\"https:\/\/ziglang.org\/\">Zig<\/a> similarly has a strong vision: explicit, simple language features, easy to cross compile, drop-in replacement for C.<\/p>\n<p>Note that you don\u2019t have to agree with a language\u2019s vision to note that it <em>has<\/em> one. I expect most people to find Uiua unpleasant to program in. That\u2019s fine. You are not the target audience.<\/p>\n<p>There\u2019s a famous quote by Bjarne Strousup that goes \u201cWithin C++, there is a much smaller and cleaner language struggling to get out.\u201d Within Rust, too, there is a much smaller and cleaner language struggling to get out: one with a clear vision, goals, focus. One that is coherent, because its features <em>cohere<\/em>. This post is about that language.<\/p>\n<hr \/>\n<h2 id=\"learning-rust-requires-learning-many-things-at-once\">Learning Rust requires learning many things at once<a class=\"zola-anchor\" href=\"#learning-rust-requires-learning-many-things-at-once\" aria-label=\"Anchor link for: learning-rust-requires-learning-many-things-at-once\"><\/a>\n<\/h2>\n<p>Rust is hard to learn. Not for lack of trying\u2014many, many people have spent person-years on improving the diagnostics, documentation, and APIs\u2014but because it\u2019s complex. When people first learn the language, they are learning many different interleaving concepts:<\/p>\n<ul>\n<li>first class functions<\/li>\n<li>enums<\/li>\n<li>pattern matching<\/li>\n<li>generics<\/li>\n<li>traits<\/li>\n<li>references<\/li>\n<li>the borrow checker<\/li>\n<li><code>Send<\/code>\/<code>Sync<\/code><\/li>\n<li><code>Iterator<\/code>s<\/li>\n<\/ul>\n<p>These concepts interlock. It is very hard to learn them one at a time because they interact with each other, and each affects the design of the others. Additionally, the standard library uses all of them heavily.<\/p>\n<p>Let\u2019s look at a Rust program that does something non-trivial:<sup class=\"footnote-reference\" id=\"fr-1-1\"><a href=\"#fn-1\">1<\/a><\/sup><\/p>\n<pre data-lang=\"rust\" class=\"language-rust z-code\"><code class=\"language-rust\" data-lang=\"rust\"><span class=\"z-source z-rust\"><span class=\"z-keyword z-operator z-rust\">#<\/span><span class=\"z-keyword z-operator z-logical z-rust\">!<\/span><span class=\"z-keyword z-operator z-arithmetic z-rust\">\/<\/span>usr<span class=\"z-keyword z-operator z-arithmetic z-rust\">\/<\/span>bin<span class=\"z-keyword z-operator z-arithmetic z-rust\">\/<\/span>env <span class=\"z-keyword z-operator z-arithmetic z-rust\">-<\/span>S cargo <span class=\"z-keyword z-operator z-arithmetic z-rust\">-<\/span>Zscript\n<\/span><span class=\"z-source z-rust\"><span class=\"z-keyword z-operator z-arithmetic z-rust\">-<\/span><span class=\"z-keyword z-operator z-arithmetic z-rust\">-<\/span><span class=\"z-keyword z-operator z-arithmetic z-rust\">-<\/span>\n<\/span><span class=\"z-source z-rust\">package<span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span>edition <span class=\"z-keyword z-operator z-assignment z-rust\">=<\/span> <span class=\"z-string z-quoted z-double z-rust\"><span class=\"z-punctuation z-definition z-string z-begin z-rust\">&quot;<\/span>2024<span class=\"z-punctuation z-definition z-string z-end z-rust\">&quot;<\/span><\/span>\n<\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">[<\/span>dependencies<span class=\"z-punctuation z-section z-group z-end z-rust\">]<\/span><\/span>\n<\/span><span class=\"z-source z-rust\">notify <span class=\"z-keyword z-operator z-assignment z-rust\">=<\/span> <span class=\"z-string z-quoted z-double z-rust\"><span class=\"z-punctuation z-definition z-string z-begin z-rust\">&quot;<\/span>=8.2.0<span class=\"z-punctuation z-definition z-string z-end z-rust\">&quot;<\/span><\/span>\n<\/span><span class=\"z-source z-rust\"><span class=\"z-keyword z-operator z-arithmetic z-rust\">-<\/span><span class=\"z-keyword z-operator z-arithmetic z-rust\">-<\/span><span class=\"z-keyword z-operator z-arithmetic z-rust\">-<\/span>\n<\/span><span class=\"z-source z-rust\"><span class=\"z-keyword z-other z-rust\">use<\/span> <span class=\"z-meta z-path z-rust\">std<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span><span class=\"z-meta z-path z-rust\">path<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span>Path<span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><span class=\"z-source z-rust\"><span class=\"z-keyword z-other z-rust\">use<\/span> <span class=\"z-meta z-path z-rust\">notify<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>RecursiveMode<span class=\"z-punctuation z-separator z-rust\">,<\/span> Watcher<\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-storage z-type z-function z-rust\">fn<\/span> <\/span><span class=\"z-entity z-name z-function z-rust\">main<\/span><\/span><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-end z-rust\">)<\/span><\/span><\/span><\/span><span class=\"z-meta z-function z-rust\"> <span class=\"z-meta z-function z-return-type z-rust\"><span class=\"z-punctuation z-separator z-rust\">-&gt;<\/span> <span class=\"z-meta z-generic z-rust\"><span class=\"z-support z-type z-rust\">Result<\/span><span class=\"z-punctuation z-definition z-generic z-begin z-rust\">&lt;<\/span><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span>, <span class=\"z-meta z-path z-rust\">notify<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span>Error<span class=\"z-punctuation z-definition z-generic z-end z-rust\">&gt;<\/span><\/span><\/span> <\/span><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\">    <span class=\"z-storage z-type z-rust\">let<\/span> paths <span class=\"z-keyword z-operator z-assignment z-rust\">=<\/span> <span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">[<\/span><span class=\"z-string z-quoted z-double z-rust\"><span class=\"z-punctuation z-definition z-string z-begin z-rust\">&quot;<\/span>pages<span class=\"z-punctuation z-definition z-string z-end z-rust\">&quot;<\/span><\/span><span class=\"z-punctuation z-separator z-rust\">,<\/span> <span class=\"z-string z-quoted z-double z-rust\"><span class=\"z-punctuation z-definition z-string z-begin z-rust\">&quot;<\/span>templates<span class=\"z-punctuation z-definition z-string z-end z-rust\">&quot;<\/span><\/span><span class=\"z-punctuation z-separator z-rust\">,<\/span> <span class=\"z-string z-quoted z-double z-rust\"><span class=\"z-punctuation z-definition z-string z-begin z-rust\">&quot;<\/span>static<span class=\"z-punctuation z-definition z-string z-end z-rust\">&quot;<\/span><\/span><span class=\"z-punctuation z-section z-group z-end z-rust\">]<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\">    <span class=\"z-storage z-type z-rust\">let<\/span> <span class=\"z-storage z-modifier z-rust\">mut<\/span> watcher <span class=\"z-keyword z-operator z-assignment z-rust\">=<\/span> <span class=\"z-meta z-path z-rust\">notify<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span>recommended_watcher<span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-begin z-rust\">|<\/span><\/span><\/span><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-variable z-parameter z-rust\">result<\/span><span class=\"z-punctuation z-separator z-rust\">:<\/span> <span class=\"z-meta z-generic z-rust\"><span class=\"z-support z-type z-rust\">Result<\/span><span class=\"z-punctuation z-definition z-generic z-begin z-rust\">&lt;<\/span><span class=\"z-meta z-path z-rust\">notify<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span>Event, <span class=\"z-keyword z-operator z-rust\">_<\/span><span class=\"z-punctuation z-definition z-generic z-end z-rust\">&gt;<\/span><\/span><span class=\"z-punctuation z-section z-parameters z-end z-rust\">|<\/span><\/span> <\/span><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-group z-rust\"><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-block z-rust\">        <span class=\"z-keyword z-control z-rust\">if<\/span> <span class=\"z-storage z-type z-rust\">let<\/span> <span class=\"z-support z-type z-rust\">Ok<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span>event<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span> <span class=\"z-keyword z-operator z-assignment z-rust\">=<\/span> result <span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-group z-rust\"><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-block z-rust\">            <span class=\"z-storage z-type z-rust\">let<\/span> paths<span class=\"z-punctuation z-separator z-rust\">:<\/span> <span class=\"z-meta z-generic z-rust\"><span class=\"z-support z-type z-rust\">Vec<\/span><span class=\"z-punctuation z-definition z-generic z-begin z-rust\">&lt;<\/span><span class=\"z-support z-type z-rust\">String<\/span><span class=\"z-punctuation z-definition z-generic z-end z-rust\">&gt;<\/span><\/span> <span class=\"z-keyword z-operator z-assignment z-rust\">=<\/span> event<span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span>paths\n<\/span><\/span><\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-group z-rust\"><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-block z-rust\">                <span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">into_iter<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span>\n<\/span><\/span><\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-group z-rust\"><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-block z-rust\">                <span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">map<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-begin z-rust\">|<\/span><\/span><\/span><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-variable z-parameter z-rust\">path<\/span><span class=\"z-punctuation z-section z-parameters z-end z-rust\">|<\/span><\/span> <\/span><span class=\"z-meta z-function z-closure z-rust\">path<span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">display<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">to_string<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span>\n<\/span><\/span><\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-group z-rust\"><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-block z-rust\">                <span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">collect<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><\/span><\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-group z-rust\"><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-block z-rust\">            <span class=\"z-support z-macro z-rust\">println!<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-string z-quoted z-double z-rust\"><span class=\"z-punctuation z-definition z-string z-begin z-rust\">&quot;<\/span>:<span class=\"z-constant z-other z-placeholder z-rust\">{:?}<\/span> <span class=\"z-constant z-other z-placeholder z-rust\">{}<\/span><span class=\"z-punctuation z-definition z-string z-end z-rust\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-separator z-rust\">,<\/span> event<span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span>kind<span class=\"z-punctuation z-separator z-rust\">,<\/span> paths<span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">join<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-string z-quoted z-double z-rust\"><span class=\"z-punctuation z-definition z-string z-begin z-rust\">&quot;<\/span> <span class=\"z-punctuation z-definition z-string z-end z-rust\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><\/span><\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-group z-rust\"><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-block z-rust\">        <\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span>\n<\/span><\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-group z-rust\"><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-block z-rust\">    <\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span><\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-keyword z-operator z-rust\">?<\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\">    <span class=\"z-keyword z-control z-rust\">for<\/span> path <span class=\"z-keyword z-operator z-rust\">in<\/span> paths <span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-block z-rust\">        watcher<span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">watch<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-meta z-path z-rust\">Path<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span>new<span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span>path<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-separator z-rust\">,<\/span> <span class=\"z-meta z-path z-rust\">RecursiveMode<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span>Recursive<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-keyword z-operator z-rust\">?<\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-block z-rust\">    <\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\">    <span class=\"z-keyword z-control z-rust\">loop<\/span> <span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span> <span class=\"z-meta z-path z-rust\">std<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span><span class=\"z-meta z-path z-rust\">thread<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span>park<span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span> <\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span> <span class=\"z-comment z-line z-double-slash z-rust\"><span class=\"z-punctuation z-definition z-comment z-rust\">\/\/<\/span> sleep forever\n<\/span><\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\"><\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span><\/span>\n<\/span><\/code><\/pre>\n<p>I tried to make this program as simple as possible: I used only the simplest iterator combinators, I don't touch <code>std::mpsc<\/code> at all, I don't use async, and I don't do any complicated error handling.<\/p>\n<p>Already, this program has many interleaving concepts. I'll ignore the module system and macros, which are mostly independent of the rest of the language. To understand this program, you need to know that:<\/p>\n<ul>\n<li><code>recommended_watcher<\/code> and <code>map<\/code> take a function as an argument. In our program, that function is constructed inline as an anonymous function (closure).<\/li>\n<li>Errors are handled using something called <code>Result<\/code>, not with exceptions or error codes. I happened to use <code>fn main() -&gt; Result<\/code> and <code>?<\/code>, but you would still need to understand Result even without that, because Rust does not let you access the value inside unless you check for an error condition first.<\/li>\n<li>Result takes a generic error; in our case, <code>notify::Error<\/code>.<\/li>\n<li>Result is an data-holding enum that can be either Ok or Err, and you can check which variant it is using pattern matching.<\/li>\n<li>Iterators can be traversed either with a <code>for<\/code> loop or with <code>into_iter()<\/code>. <sup class=\"footnote-reference\" id=\"fr-2-1\"><a href=\"#fn-2\">2<\/a><\/sup> <code>for<\/code> is eager and <code>into_iter<\/code> is lazy. <code>iter<\/code> has different ownership semantics than <code>into_iter<\/code>.<\/li>\n<\/ul>\n<p>If you want to modify this program, you need to know some additional things:<\/p>\n<ul>\n<li><code>println<\/code> can only print things that implement the traits <code>Display<\/code> or <code>Debug<\/code>. As a result, <code>Path<\/code>s cannot be printed directly.<\/li>\n<li><code>path.display()<\/code> returns a struct that borrows from the path. Sending it to another thread (e.g. through a channel) won't work, because <code>event.paths<\/code> goes out of scope when the closure passed to <code>recommended_watcher<\/code> finishes running. You need to convert it to an owned value or pass <code>event.paths<\/code> as a whole.\n<ul>\n<li>As an aside, this kind of thing encourages people to break work into \"large\" chunks instead of \"small\" chunks, which I think is often good for performance in CPU-bound programs, although as always it depends.<\/li>\n<\/ul>\n<\/li>\n<li><code>recommended_watcher<\/code> only accepts functions that are <code>Send + 'static<\/code>. Small changes to this program, such as passing the current path into the closure, will give a compile error related to ownership. Fixing it requires learning the <code>move<\/code> keyword, knowing that closures borrow their arguments by default, and the meaning of <code>'static<\/code>.\n<ul>\n<li>If you are using <code>Rc&lt;RefCell&lt;String&gt;&gt;<\/code>, which is often recommended for beginners<sup class=\"footnote-reference\" id=\"fr-3-1\"><a href=\"#fn-3\">3<\/a><\/sup>, your program will need to be rewritten from scratch (either to use Arc\/Mutex or to use exterior mutability). For example, if you wanted to print changes from the main thread instead of worker threads to avoid interleaving output, you couldn't simply push to the end of an <code>all_changes<\/code> collection, you would have to use <code>Arc&lt;Mutex&lt;Vec&lt;Path&gt;&gt;&gt;<\/code> in order to communicate between threads.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>This is a <em>lot<\/em> of concepts for a 20 line program. For comparison, here is an equivalent javascript program:<\/p>\n<pre data-lang=\"javascript\" class=\"language-javascript z-code\"><code class=\"language-javascript\" data-lang=\"javascript\"><span class=\"z-source z-ts\"><span class=\"z-meta z-var z-expr z-ts\"><span class=\"z-storage z-type z-ts\">const<\/span> <span class=\"z-meta z-var-single-variable z-expr z-ts\"><span class=\"z-meta z-definition z-variable z-ts\"><span class=\"z-variable z-other z-constant z-ts\">fs<\/span><\/span> <\/span><span class=\"z-keyword z-operator z-assignment z-ts\">=<\/span> <span class=\"z-meta z-function-call z-ts\"><span class=\"z-support z-function z-ts\">require<\/span><\/span><span class=\"z-meta z-brace z-round z-ts\">(<\/span><span class=\"z-string z-quoted z-single z-ts\"><span class=\"z-punctuation z-definition z-string z-begin z-ts\">&#39;<\/span>fs<span class=\"z-punctuation z-definition z-string z-end z-ts\">&#39;<\/span><\/span><span class=\"z-meta z-brace z-round z-ts\">)<\/span><\/span><span class=\"z-punctuation z-terminator z-statement z-ts\">;<\/span>\n<\/span><span class=\"z-source z-ts\"><span class=\"z-meta z-var z-expr z-ts\"><span class=\"z-storage z-type z-ts\">const<\/span> <span class=\"z-meta z-var-single-variable z-expr z-ts\"><span class=\"z-meta z-definition z-variable z-ts\"><span class=\"z-variable z-other z-constant z-ts\">paths<\/span><\/span> <\/span><span class=\"z-keyword z-operator z-assignment z-ts\">=<\/span><span class=\"z-meta z-array z-literal z-ts\"> <span class=\"z-meta z-brace z-square z-ts\">[<\/span><span class=\"z-string z-quoted z-single z-ts\"><span class=\"z-punctuation z-definition z-string z-begin z-ts\">&#39;<\/span>pages<span class=\"z-punctuation z-definition z-string z-end z-ts\">&#39;<\/span><\/span><span class=\"z-punctuation z-separator z-comma z-ts\">,<\/span> <span class=\"z-string z-quoted z-single z-ts\"><span class=\"z-punctuation z-definition z-string z-begin z-ts\">&#39;<\/span>templates<span class=\"z-punctuation z-definition z-string z-end z-ts\">&#39;<\/span><\/span><span class=\"z-punctuation z-separator z-comma z-ts\">,<\/span> <span class=\"z-string z-quoted z-single z-ts\"><span class=\"z-punctuation z-definition z-string z-begin z-ts\">&#39;<\/span>static<span class=\"z-punctuation z-definition z-string z-end z-ts\">&#39;<\/span><\/span><span class=\"z-meta z-brace z-square z-ts\">]<\/span><\/span><\/span><span class=\"z-punctuation z-terminator z-statement z-ts\">;<\/span>\n<\/span><span class=\"z-source z-ts\"><span class=\"z-keyword z-control z-loop z-ts\">for<\/span> <span class=\"z-meta z-brace z-round z-ts\">(<\/span><span class=\"z-meta z-var z-expr z-ts\"><span class=\"z-storage z-type z-ts\">let<\/span> <span class=\"z-meta z-var-single-variable z-expr z-ts\"><span class=\"z-meta z-definition z-variable z-ts\"><span class=\"z-variable z-other z-readwrite z-ts\">path<\/span><\/span> <\/span><\/span><span class=\"z-keyword z-operator z-expression z-of z-ts\">of<\/span> <span class=\"z-variable z-other z-readwrite z-ts\">paths<\/span><span class=\"z-meta z-brace z-round z-ts\">)<\/span> <span class=\"z-meta z-block z-ts\"><span class=\"z-punctuation z-definition z-block z-ts\">{<\/span>\n<\/span><\/span><span class=\"z-source z-ts\"><span class=\"z-meta z-block z-ts\">  <span class=\"z-meta z-function-call z-ts\"><span class=\"z-variable z-other z-object z-ts\">fs<\/span><span class=\"z-punctuation z-accessor z-ts\">.<\/span><span class=\"z-support z-function z-ts\">watch<\/span><\/span><span class=\"z-meta z-brace z-round z-ts\">(<\/span><span class=\"z-variable z-other z-readwrite z-ts\">path<\/span><span class=\"z-punctuation z-separator z-comma z-ts\">,<\/span><span class=\"z-meta z-arrow z-ts\"> <span class=\"z-meta z-parameters z-ts\"><span class=\"z-punctuation z-definition z-parameters z-begin z-ts\">(<\/span><span class=\"z-variable z-parameter z-ts\">eventType<\/span><span class=\"z-punctuation z-separator z-parameter z-ts\">,<\/span> <span class=\"z-variable z-parameter z-ts\">filename<\/span><span class=\"z-punctuation z-definition z-parameters z-end z-ts\">)<\/span><\/span> <\/span><span class=\"z-meta z-arrow z-ts\"><span class=\"z-storage z-type z-function z-arrow z-ts\">=&gt;<\/span> <span class=\"z-meta z-block z-ts\"><span class=\"z-punctuation z-definition z-block z-ts\">{<\/span>\n<\/span><\/span><\/span><\/span><span class=\"z-source z-ts\"><span class=\"z-meta z-block z-ts\"><span class=\"z-meta z-arrow z-ts\"><span class=\"z-meta z-block z-ts\">    <span class=\"z-keyword z-control z-conditional z-ts\">if<\/span> <span class=\"z-meta z-brace z-round z-ts\">(<\/span><span class=\"z-variable z-other z-readwrite z-ts\">filename<\/span> <span class=\"z-keyword z-operator z-comparison z-ts\">!==<\/span> <span class=\"z-constant z-language z-null z-ts\">null<\/span><span class=\"z-meta z-brace z-round z-ts\">)<\/span> <span class=\"z-meta z-block z-ts\"><span class=\"z-punctuation z-definition z-block z-ts\">{<\/span>\n<\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-ts\"><span class=\"z-meta z-block z-ts\"><span class=\"z-meta z-arrow z-ts\"><span class=\"z-meta z-block z-ts\"><span class=\"z-meta z-block z-ts\">      <span class=\"z-meta z-function-call z-ts\"><span class=\"z-support z-class z-console z-ts\">console<\/span><span class=\"z-punctuation z-accessor z-ts\">.<\/span><span class=\"z-support z-function z-console z-ts\">log<\/span><\/span><span class=\"z-meta z-brace z-round z-ts\">(<\/span><span class=\"z-string z-template z-ts\"><span class=\"z-punctuation z-definition z-string z-template z-begin z-ts\">`<\/span><span class=\"z-meta z-template z-expression z-ts\"><span class=\"z-punctuation z-definition z-template-expression z-begin z-ts\">${<\/span><\/span><span class=\"z-meta z-template z-expression z-ts\"><span class=\"z-meta z-embedded z-line z-ts\"><span class=\"z-variable z-other z-readwrite z-ts\">eventType<\/span><\/span><span class=\"z-punctuation z-definition z-template-expression z-end z-ts\">}<\/span><\/span> <span class=\"z-meta z-template z-expression z-ts\"><span class=\"z-punctuation z-definition z-template-expression z-begin z-ts\">${<\/span><\/span><span class=\"z-meta z-template z-expression z-ts\"><span class=\"z-meta z-embedded z-line z-ts\"><span class=\"z-variable z-other z-readwrite z-ts\">filename<\/span><\/span><span class=\"z-punctuation z-definition z-template-expression z-end z-ts\">}<\/span><\/span><span class=\"z-punctuation z-definition z-string z-template z-end z-ts\">`<\/span><\/span><span class=\"z-meta z-brace z-round z-ts\">)<\/span>\n<\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-ts\"><span class=\"z-meta z-block z-ts\"><span class=\"z-meta z-arrow z-ts\"><span class=\"z-meta z-block z-ts\"><span class=\"z-meta z-block z-ts\">    <span class=\"z-punctuation z-definition z-block z-ts\">}<\/span><\/span>\n<\/span><\/span><\/span><\/span><span class=\"z-source z-ts\"><span class=\"z-meta z-block z-ts\"><span class=\"z-meta z-arrow z-ts\"><span class=\"z-meta z-block z-ts\">  <span class=\"z-punctuation z-definition z-block z-ts\">}<\/span><\/span><\/span><span class=\"z-meta z-brace z-round z-ts\">)<\/span><span class=\"z-punctuation z-terminator z-statement z-ts\">;<\/span>\n<\/span><\/span><span class=\"z-source z-ts\"><span class=\"z-meta z-block z-ts\"><span class=\"z-punctuation z-definition z-block z-ts\">}<\/span><\/span>\n<\/span><span class=\"z-source z-ts\"><span class=\"z-keyword z-control z-flow z-ts\">await<\/span> <span class=\"z-meta z-brace z-round z-ts\">(<\/span><span class=\"z-new z-expr z-ts\"><span class=\"z-keyword z-operator z-new z-ts\">new<\/span> <span class=\"z-entity z-name z-type z-ts\">Promise<\/span><span class=\"z-meta z-brace z-round z-ts\">(<\/span><span class=\"z-meta z-arrow z-ts\"><span class=\"z-meta z-parameters z-ts\"><span class=\"z-punctuation z-definition z-parameters z-begin z-ts\">(<\/span><span class=\"z-punctuation z-definition z-parameters z-end z-ts\">)<\/span><\/span> <\/span><span class=\"z-meta z-arrow z-ts\"><span class=\"z-storage z-type z-function z-arrow z-ts\">=&gt;<\/span> <span class=\"z-meta z-block z-ts\"><span class=\"z-punctuation z-definition z-block z-ts\">{<\/span><span class=\"z-punctuation z-definition z-block z-ts\">}<\/span><\/span><\/span><span class=\"z-meta z-brace z-round z-ts\">)<\/span><\/span><span class=\"z-meta z-brace z-round z-ts\">)<\/span><span class=\"z-punctuation z-terminator z-statement z-ts\">;<\/span> <span class=\"z-comment z-line z-double-slash z-ts\"><span class=\"z-punctuation z-definition z-comment z-ts\">\/\/<\/span><\/span><span class=\"z-comment z-line z-double-slash z-ts\"> sleep forever<\/span>\n<\/span><\/code><\/pre>\n<p>For this JS program, you need to understand:<\/p>\n<ul>\n<li>first class functions<\/li>\n<li>nullability<\/li>\n<li>yeah that's kinda it.<\/li>\n<\/ul>\n<p>I'm cheating a little here because <code>notify<\/code> returns a list of paths and <code>node:fs\/watch<\/code> doesn't. But only a little.<\/p>\n<p>My point is not that JS is a simpler language; that's debatable. My point is that you can do things in JS without understanding the whole language. It's very hard to do non-trivial things in Rust without understanding the whole core.<\/p>\n<h2 id=\"rust-s-core-is-interwoven-on-purpose\">Rust's core is interwoven on purpose<a class=\"zola-anchor\" href=\"#rust-s-core-is-interwoven-on-purpose\" aria-label=\"Anchor link for: rust-s-core-is-interwoven-on-purpose\"><\/a>\n<\/h2>\n<p>The previous section makes it out to seem like I'm saying all these concepts are bad. I'm not. Rather the opposite, actually. Because these language features were designed in tandem, they interplay very nicely:<\/p>\n<ul>\n<li>Enums without pattern matching <a href=\"https:\/\/en.cppreference.com\/w\/cpp\/utility\/variant.html\">are very painful to work with<\/a> and pattern matching without enums <a href=\"https:\/\/www.hillelwayne.com\/post\/python-abc\/\">has very odd semantics<\/a><\/li>\n<li><code>Result<\/code> and <code>Iterator<\/code>s are impossible to implement without generics (or duck-typing, which I think of as type-erased generics)<\/li>\n<li><code>Send<\/code>\/<code>Sync<\/code>, and the preconditions to <code>println<\/code>, are impossible to encode without traits\u2014and this often comes up in other languages, for example printing a function in clojure shows something like <code>#object[clojure.core$map 0x2e7de98a \"clojure.core$map@2e7de98a\"]<\/code>. In Rust it gives a compile error unless you opt-in with Debug.<\/li>\n<li><code>Send<\/code> \/ <code>Sync<\/code> are only possible to enforce because the borrow checker does capture analysis for closures. Java, which is <em>wildly<\/em> committed to thread-safety by the standards of most languages, cannot verify this at compile time and so has to <a href=\"https:\/\/docs.oracle.com\/en\/java\/javase\/24\/docs\/api\/java.base\/java\/text\/SimpleDateFormat.html#synchronization\">document synchronization concerns explicitly<\/a> instead.<\/li>\n<\/ul>\n<p>There are more interplays than I can easily describe in a post, and all of them are what make Rust what it is.<\/p>\n<p>Rust has other excellent language features\u2014for example the <a href=\"https:\/\/doc.rust-lang.org\/nightly\/reference\/inline-assembly.html\">inline assembly syntax<\/a> is a work of art, props to <a href=\"https:\/\/github.com\/amanieu\">Amanieu<\/a>. But they are not interwoven into the standard library in the same way, and they do not affect the way people <em>think<\/em> about writing code in the same way.<\/p>\n<h2 id=\"a-smaller-rust\">A smaller Rust<a class=\"zola-anchor\" href=\"#a-smaller-rust\" aria-label=\"Anchor link for: a-smaller-rust\"><\/a>\n<\/h2>\n<p>without.boats wrote a post in 2019 titled <a href=\"https:\/\/without.boats\/blog\/notes-on-a-smaller-rust\/\">\"Notes on a smaller Rust\"<\/a> (and a follow-up <a href=\"https:\/\/without.boats\/blog\/revisiting-a-smaller-rust\/\">revisiting<\/a> it). In a manner of speaking, that smaller Rust <em>is<\/em> the language I fell in love with when I first learned it in 2018. Rust is a lot bigger today, in many ways, and the smaller Rust is just a nostalgic rose-tinted memory. But I think it's worth studying as an example of how well <a href=\"https:\/\/en.wikipedia.org\/wiki\/Orthogonality#Computer_science\">orthogonal<\/a> features can compose when they're designed as one cohesive whole.<\/p>\n<p>If you liked this post, consider reading <a href=\"https:\/\/matklad.github.io\/2020\/07\/15\/two-beautiful-programs.html\">Two Beautiful Rust Programs<\/a> by matklad.<\/p>\n<section class=\"footnotes\">\n<ol class=\"footnotes-list\">\n<li id=\"fn-1\">\n<p>This program intentionally uses a file watcher because file IO is not possible to implement efficiently with async on Linux (and also because I wrote a file watcher recently for <a href=\"https:\/\/github.com\/jyn514\/flower\/\">flower<\/a>, so it's fresh in my mind). Tokio itself just uses a threadpool, alongside channels for notifying the future. I don\u2019t want to get into async here; this just demonstrates Send\/Sync bounds and callbacks. <a href=\"#fr-1-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-2\">\n<p>Technically, <code>for<\/code> is syntax sugar around <code>into_iter()<\/code>, but you don't need to know that for most rust programs. <a href=\"#fr-2-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-3\">\n<p>which is a terrible idea by the way, even more experienced Rust programmers often don't understand the interior mutability very well; see <a href=\"https:\/\/docs.rs\/dtolnay\/latest\/dtolnay\/macro._02__reference_types.html\">this blog post by dtolnay<\/a> on the difference between mutability and uniqueness in reference types. It would be better to suggest using owned types with exterior mutability and cloning frequently. <a href=\"#fr-3-1\">\u21a9<\/a><\/p>\n<\/li>\n<\/ol>\n<\/section>\n"},{"title":"how to communicate with intent","published":"2025-08-11T00:00:00+00:00","updated":"2025-08-11T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/how-to-communicate-with-intent\/"}},"id":"https:\/\/jyn.dev\/how-to-communicate-with-intent\/","summary":"say what you mean to say, not just the first thing on your mind","content":"<p>As you can see from this blog, I like to talk (my friends will be the first to confirm this). Just as important as knowing how to talk, though, is knowing what to say and when to listen.<\/p>\n<p>In this post I will give a few simple tips on how to improve your communication in various parts of your life. My goal is partly to help you to be a more effective communicator, and partly to reduce the number of people in my life who get on my nerves :P<\/p>\n<p>You don't always need these tips. In casual conversations and when brainstorming, it's healthy to just say the first thing on your mind. But not all conversations are casual, and conversations can slip into seriousness faster than you expect. For those situations, you need to be intentional. Otherwise, it's easy to end up with hurt feelings on both sides, or waste the time of everyone involved.<\/p>\n<h2 id=\"adapt-to-your-audience\">adapt to your audience<a class=\"zola-anchor\" href=\"#adapt-to-your-audience\" aria-label=\"Anchor link for: adapt-to-your-audience\"><\/a>\n<\/h2>\n<p>First, think about your audience. Adapt your message to the person you\u2019re speaking to. Someone learning Rust for the first time does not need an infodump about variance and type coercion, they need an introduction to enums, generics, and pattern matching. Similarly, if a barista asks you what the weird code on your screen is, don\u2019t tell them you\u2019re writing a <a href=\"https:\/\/codeberg.org\/jyn514\/flower\/\">meta-Static Site Generator in Clojure<\/a>, tell them you\u2019re building a tool to help people create websites.<\/p>\n<p>If you are writing a promotion doc, a resume, or a tutorial, don't just dump a list of everything that's relevant. Think about the structure of your document: the questions your reader is likely to have, the amount of time they are likely going to spend reading, and the order they are likely to read in. You need to be <em>legible<\/em>, which means explaining concrete impacts in terms your audience understands. It's not enough to say what's true; you have to also say why it's important.<\/p>\n<h2 id=\"consider-your-purpose\">consider your purpose<a class=\"zola-anchor\" href=\"#consider-your-purpose\" aria-label=\"Anchor link for: consider-your-purpose\"><\/a>\n<\/h2>\n<p>Consider your intended effect. If a teacher goes on Twitter saying she doesn\u2019t understand why maths is important and we should just give out A\u2019s like candy (real thing that happened on my feed!), dog piling on her is not going to change her mind. Show her a case in her life where maths would be useful, and don\u2019t talk down to her. Self-righteousness feels good in the moment, but doesn\u2019t actually achieve anything. If you just want to gloat about how other people are stupid, go play an FPS or something; Twitter has enough negativity.<\/p>\n<p>If you are writing a blog post, know why you are writing it. If you are writing to practice the skill of writing, or to have a reference document, or to share with your friends, infodumping is fine. If you are writing with a goal in mind\u2014say you want to <a href=\"https:\/\/www.joelonsoftware.com\/2002\/11\/11\/the-law-of-leaky-abstractions\/\">give a name to an idea<\/a> or <a href=\"https:\/\/devblogs.microsoft.com\/oldnewthing\/20250808-00\/?p=111447\">communicate when software can fail<\/a> or <a href=\"https:\/\/jyn.dev\/how-i-use-my-terminal\/\">enter an idea into the overton window<\/a>\u2014be intentional. Consider your audience, and the background you expect them to start from. Posting the equivalent of a wikipedia article is rarely the most effective way to instill an idea.<\/p>\n<h3 id=\"corollary-don-t-argue-for-argument-s-sake\">corollary: don't argue for argument's sake<a class=\"zola-anchor\" href=\"#corollary-don-t-argue-for-argument-s-sake\" aria-label=\"Anchor link for: corollary-don-t-argue-for-argument-s-sake\"><\/a>\n<\/h3>\n<p>Don\u2019t fight losing causes, <a href=\"https:\/\/boardgamegeek.com\/blog\/5988\/blogpost\/76876\/if-youre-going-to-do-something-that-crazy\">unless the cause is really worth it<\/a>. Someone on hacker news saying \"language A Sucks and you Should use language B instead\" is not worth arguing with. Someone who says \"language A is good in scenario X, but has shortcomings in scenario Y compared to language B\" is much more serious and worth listening to. Arguing with someone who refuses to be convinced wastes everyone\u2019s time.<\/p>\n<h2 id=\"balance-conversational-effort\">balance conversational effort<a class=\"zola-anchor\" href=\"#balance-conversational-effort\" aria-label=\"Anchor link for: balance-conversational-effort\"><\/a>\n<\/h2>\n<p>Be a good conversational partner. Ask directed probing questions: they show you are listening to the other person and invested in the topic. Saying \u201cI don\u2019t understand\u201d puts the burden on them to figure out the source of the misunderstanding. If you really aren\u2019t sure what to ask, because you\u2019re confused or the other person was vague, I like \u201csay more?\u201d as a way to leave it open ended for the other person on how to elaborate.<\/p>\n<h2 id=\"consider-implications\">consider implications<a class=\"zola-anchor\" href=\"#consider-implications\" aria-label=\"Anchor link for: consider-implications\"><\/a>\n<\/h2>\n<p>Consider the implications of how you communicate. When you say things, you are not just communicating the words you speak, you are also affecting the person you're talking to.<\/p>\n<h3 id=\"not-everyone-wants-to-hear-an-infodump\">not everyone wants to hear an infodump<a class=\"zola-anchor\" href=\"#not-everyone-wants-to-hear-an-infodump\" aria-label=\"Anchor link for: not-everyone-wants-to-hear-an-infodump\"><\/a>\n<\/h3>\n<p>If the person you're infodumping to isn't interested in the topic, infodumping anyway puts them in an awkward situation where they either have to ask you to stop or sit through a long conversation they didn't want to be in.<\/p>\n<p>Another tricky scenario is when the other person is interested, but an infodump is not the right level of detail for them right now. Perhaps they are new to the topic, or perhaps they asked a direct question. If they're still trying to get the \"big picture\", zooming in to fine-grained details will often just confuse them further.<\/p>\n<h3 id=\"apologies-are-not-the-time-to-infodump\">apologies are not the time to infodump<a class=\"zola-anchor\" href=\"#apologies-are-not-the-time-to-infodump\" aria-label=\"Anchor link for: apologies-are-not-the-time-to-infodump\"><\/a>\n<\/h3>\n<p>Info-dumping during an apology\u2014even if it\u2019s related to the thing you're apologizing for!\u2014buries the apology. More than that, it implies that you expect <em>mitigated judgement<\/em>. If there is a power dynamic between you (say a wealth gap, or you are a manager and they are an employee), that expectation of mitigated judgment implies you <em>expect to be forgiven<\/em>, and <a href=\"https:\/\/bsky.app\/profile\/did:plc:h2okxbr76w5522tailkxmidq\/post\/3lvrxmdvb3s2r\">an apology given in expectation of forgiveness is really just a request for absolution<\/a>.<\/p>\n<p>Instead, apologize directly. If you were in an altered mental state (angry, sleep-deprived, experiencing a trauma trigger), you can add at most 1-2 sentences of context asking the other person to mitigate judgement. Not all apologies need context; often \"i was wrong, i'm sorry\" is enough.<\/p>\n<h3 id=\"infodumps-will-not-prevent-all-miscommunication\">infodumps will not prevent all miscommunication<a class=\"zola-anchor\" href=\"#infodumps-will-not-prevent-all-miscommunication\" aria-label=\"Anchor link for: infodumps-will-not-prevent-all-miscommunication\"><\/a>\n<\/h3>\n<p>As we've seen above, there are times when infodumps actively hurt you. Even when they don't, though, there can be times when they aren't helping. Everyone comes to a conversation with a different background, and you cannot perfectly predict how they will respond. Rather than trying to avoid every possible miscommunication by packing the maximum amount of information\u2014Say what you mean to say. Then, address the actual miscommunication (or regular conversation!) that happens afterwards. This saves time and energy for both conversational partners.<\/p>\n<h2 id=\"be-legible\">be legible<a class=\"zola-anchor\" href=\"#be-legible\" aria-label=\"Anchor link for: be-legible\"><\/a>\n<\/h2>\n<p>The common theme of all of the above is to communicate <em>effectively<\/em> and <a href=\"https:\/\/medium.com\/@ElizAyer\/dont-ask-forgiveness-radiate-intent-d36fd22393a3\"><em>radiate intent<\/em><\/a>. Making it easy for the other person to understand both what you're saying and why you're saying it incurs a lot of goodwill, and makes it possible to say more things more bluntly than you would otherwise.<\/p>\n<p>A common trap I see people fall into is to say the first thing on their mind. This is fine for conversations between friends (although you should still consider how it affects your relationship!) but but is often counterproductive in other contexts. Slow down. Take your time. Say what you mean to say. If you don't mean to say anything, don't say anything at all.<\/p>\n"},{"title":"an engineer's perspective on hiring","published":"2025-08-08T00:00:00+00:00","updated":"2025-08-08T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/an-engineers-perspective-on-hiring\/"}},"id":"https:\/\/jyn.dev\/an-engineers-perspective-on-hiring\/","summary":"hiring in tech is broken and everyone knows it. what can we do better?","content":"<p><strong>note for my friends: this post is targeted at companies and engineering managers. i know you know that hiring sucks and companies waste your time. this is a business case for why they shouldn't do that.<\/strong><\/p>\n<h2 id=\"hiring-sucks\">hiring sucks<a class=\"zola-anchor\" href=\"#hiring-sucks\" aria-label=\"Anchor link for: hiring-sucks\"><\/a>\n<\/h2>\n<p>most companies suck at hiring. they waste everyone\u2019s time (i once had a 9-round interview pipeline!), they <a href=\"https:\/\/danluu.com\/programmer-moneyball\/\">chase the trendiest programmers<\/a>, and they <a href=\"https:\/\/medium.com\/@weswinham\/chatgpt-killed-the-tech-interview-i-tested-11-methods-and-heres-what-survived-5652a3e95190\">can\u2019t even tell programmers apart from an LLM<\/a>. in short, <a href=\"https:\/\/predr.ag\/blog\/mediocrity-can-be-a-sign-of-excellence\/\">they are not playing moneyball<\/a>.<\/p>\n<p>things are bad for interviewees too. some of the best programmers i know (think people maintaining the rust compiler) can\u2019t get jobs because <a href=\"https:\/\/hadid.dev\/posts\/living-coding\/\">they interview poorly under stress<\/a>. one with 4 years of Haskell experience and 2 years of Rust experience was labeled as \u201cnon-technical\u201d by a recruiter. and of course, companies repeatedly ghost people for weeks or months about whether they actually got a job.<\/p>\n<p>this post explores why hiring is hard, how existing approaches fail, and what a better approach could look like. my goal, of course, is to get my friends hired. <a href=\"mailto:blog@jyn.dev\">reach out to me<\/a> if you like the ideas here.<\/p>\n<h2 id=\"what-makes-a-good-interview\">what makes a good interview<a class=\"zola-anchor\" href=\"#what-makes-a-good-interview\" aria-label=\"Anchor link for: what-makes-a-good-interview\"><\/a>\n<\/h2>\n<p>before i start talking about my preferred approach, let\u2019s start by establishing some (hopefully uncontroversial) principles.<\/p>\n<p>interviews should:<\/p>\n<ol>\n<li><strong>differentiate<\/strong>. be able to tell the difference between a senior programmer and a marketer using chatgpt.<\/li>\n<li><strong>be applicable<\/strong>. reflect the actual job duties.\n<ul>\n<li>this includes coding. but it also includes architecture design, PR review, documentation, on and on and on. all good senior software engineers are generalists.<\/li>\n<\/ul>\n<\/li>\n<li><strong>think long term<\/strong>. select for applicants who will be good employees for years to come, not just in the next quarter.\n<ul>\n<li><a href=\"https:\/\/danluu.com\/people-matter\/\">people are not fungible<\/a>.<\/li>\n<li><a href=\"https:\/\/yosefk.com\/blog\/compensation-rationality-and-the-projectperson-fit.html\">there is a high cost to losing employees who are a good fit to the project<\/a>.<\/li>\n<li><a href=\"https:\/\/jyn.dev\/theory-building-without-a-mentor\/#theory-building\">there is a high cost to losing employees in general<\/a>.<\/li>\n<li>companies often over-index on crystallized knowledge over fluid intelligence. spending an additional month to find people who specialize in your tech stack, when you could have onboarded them to that stack in a month, is an advanced form of self-sabotage.<\/li>\n<\/ul>\n<\/li>\n<li><strong>be time efficient<\/strong>. spend as little time as possible interviewing.\n<ul>\n<li>engineer time is expensive.<\/li>\n<\/ul>\n<\/li>\n<li><strong>be respectful<\/strong>. respect the applicant and their time.\n<ul>\n<li>if you don't respect the applicant, you will select for people who don't respect themselves, and drive away the best applicants.<\/li>\n<li>\"but i want to select for people that don't respect themselves so i can pay them less\"\u2014get the hell off my site and don't come back.<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n<p>there is also a 6th criteria that's more controversial. let's call it <strong>taste<\/strong>. an engineer with poor <strong>taste<\/strong> can ship things very quickly at the expense of leaving a giant mess for everyone else on the team to clean up. measuring this is very hard but also very important. conversely, <a href=\"https:\/\/youtu.be\/4yleYA2giPE?si=6ukBDYiqAikWPKhp&amp;t=1348\">someone who spends time empowering the rest of their team has a multiplicative effect on their team's productivity<\/a> (c.f. <a href=\"https:\/\/www.noidea.dog\/glue\">\"Being Glue\"<\/a>).<\/p>\n<p>let's look at some common interviews and how they fare.<\/p>\n<h3 id=\"live-coding-often-called-leetcode-interviews\">live coding, often called \"leetcode interviews\"<a class=\"zola-anchor\" href=\"#live-coding-often-called-leetcode-interviews\" aria-label=\"Anchor link for: live-coding-often-called-leetcode-interviews\"><\/a>\n<\/h3>\n<p>fails on <strong>differentiating, applicability, respect, taste<\/strong>. gives very little signal about <strong>long term value<\/strong>. live coding <a href=\"https:\/\/medium.com\/@weswinham\/chatgpt-killed-the-tech-interview-i-tested-11-methods-and-heres-what-survived-5652a3e95190\">cannot distinguish a senior programmer from a marketer using chatGPT<\/a>, and most interview questions have very little to do with day-to-day responsibilities. all good software engineers are generalist and live coding does not select for generalists.<\/p>\n<p>you can augment live coding with multiple rounds of interviews, each of which tests one of the above responsibilities. but now you lose <strong>time efficiency<\/strong>; everything takes lots of engineer time. doing this despite the expense is a show of wealth, and now you are no longer playing moneyball.<\/p>\n<p>additionally, people with lots of experience often find the experience demeaning, so you are filtering out the best applicants. a friend explicitly said \"I have 18 years of experience on GitHub; if you can't tell I'm a competent programmer from that it's not a good fit.\"<\/p>\n<p>something not often thought about is that this also loses you <strong>taste<\/strong>. the code that someone puts together under pressure is not a reflection of how they normally work, and does not let you judge if your engineers will like working with them.<\/p>\n<h3 id=\"take-home-interviews\">take-home interviews<a class=\"zola-anchor\" href=\"#take-home-interviews\" aria-label=\"Anchor link for: take-home-interviews\"><\/a>\n<\/h3>\n<p>fails on <strong>differentiating<\/strong> and <strong>respect<\/strong>, and partially on <strong>applicability<\/strong>. take home interviews are very easy for chatGPT to game and have all the other problems of live interviews, except that they remove the \"interview poorly under stress\" component. but they trade off a fundamental <strong>time asymmetry<\/strong> with the applicant, which again drives away the best people.<\/p>\n<h3 id=\"architecture-design\">architecture design<a class=\"zola-anchor\" href=\"#architecture-design\" aria-label=\"Anchor link for: architecture-design\"><\/a>\n<\/h3>\n<p>this does a lot better. you can't use chatGPT to fake an architecture interview<sup class=\"footnote-reference\" id=\"fr-1-1\"><a href=\"#fn-1\">1<\/a><\/sup>. it fails at <strong>applicability<\/strong> (you don't ever see the applicant's code). at first glance it appears to give you some insight into <strong>taste<\/strong>, but often it is measuring \"how well does the applicant know the problem domain\" instead of \"how does the applicant think about design problems\", so you have to be careful about over-indexing on it.<\/p>\n<h3 id=\"meet-the-team\">\"meet the team\"<a class=\"zola-anchor\" href=\"#meet-the-team\" aria-label=\"Anchor link for: meet-the-team\"><\/a>\n<\/h3>\n<p>i haven't seen this one a lot for external interviews, but i see it very commonly for internal transfers within a company. it has much of the same tradeoffs as architecture design interviews, except it usually isn't trying to judge skills at all, mostly personality and \"fit\" (i.e. it fails on <strong>differentiating<\/strong> and partially on <strong>applicability<\/strong>). i think it makes sense in environments where the candidate has a very strong recommendation and there's little competition for the position; or if you have some other reason to highly value their skills without a formal assessment.<\/p>\n<h3 id=\"extended-essays-and-work-samples\">extended essays and work samples<a class=\"zola-anchor\" href=\"#extended-essays-and-work-samples\" aria-label=\"Anchor link for: extended-essays-and-work-samples\"><\/a>\n<\/h3>\n<p>this is an interesting one. i've only ever seen it from <a href=\"https:\/\/rfd.shared.oxide.computer\/rfd\/0003#_mechanics_of_evaluation\">Oxide Computer Company<\/a>. i like it really quite a lot. the process looks like this:<\/p>\n<ol>\n<li>the applicant submits samples of their existing work (or writes new documents specially for the interview)<\/li>\n<li>the applicant writes detailed responses to 8 questions about their values, work, career, and goals.<\/li>\n<li>the applicant goes through 9 hours of interviews with several oxide employees.<\/li>\n<\/ol>\n<p>this does really really well on nearly every criteria (including <strong>respect<\/strong>\u2014note that the time spent here is symmetric, it takes a <em>long<\/em> time for Oxide's engineers to read that much written material).<\/p>\n<p>it fails on <strong>time efficiency<\/strong>. i have not gone through this process, but based on the questions and my knowledge of who gets hired at oxide, i would expect <em>just<\/em> the written work to take at around 5-15 hours of time for a single application. given oxide and their goals, and the sheer number of people who apply there, i suspect they are ok with that tradeoff (and indeed probably value that it drives away people who aren't committed to the application). but most companies are not oxide and cannot afford this amount of time on both sides.<\/p>\n<p>if i were to take ideas from the oxide process without sacrificing too much time, i\u2019d keep \"you write the code ahead of time and discuss it in the interview\". this keeps the advantage of take-home interviews\u2014no time pressure, less stressful environment\u2014while adding a <strong>symmetric time<\/strong> component that makes talented engineers less likely to dismiss the job posting out of hand, without an enormous up-front expenditure of time. and discussing the code live filters out people who just vibecoded the whole thing (they won't be able to explain what it does!) while giving everyone else a chance to explain their thinking, helping with <strong>applicability<\/strong> and <strong>taste<\/strong>.<\/p>\n<p>this still has some <strong>time asymmetry<\/strong> if the applicant doesn\u2019t have existing open source work they want to show to an interviewer, but it\u2019s a lot less than 5-15 hours, and the company is forced to dedicate some of their own engineer time, so they have motivation not to \u201cthrow work over the wall\u201d, showing respect for the applicant.<\/p>\n<h3 id=\"code-review\">code review <sup class=\"footnote-reference\" id=\"fr-2-1\"><a href=\"#fn-2\">2<\/a><\/sup><a class=\"zola-anchor\" href=\"#code-review\" aria-label=\"Anchor link for: code-review\"><\/a>\n<\/h3>\n<p>this one i\u2019ve also only ever seen once. the format is that the interviewer writes some mediocre code ahead of time and asks the applicant how they would improve it. i did very well on this format so i'm biased, but i like it a lot. it aces all our criteria:<\/p>\n<ul>\n<li>it reverses the <strong>time asymmetry<\/strong> and reduces the amount of <strong>time spent<\/strong>. the interviewer makes one up front time commitment, the applicant makes no up front commitment, and they spend the same amount of time per interview.<\/li>\n<li>it\u2019s <strong>applicable<\/strong>: you see how the applicant gives interpersonal feedback; discussions about the code naturally lead into discussions about design; and you get information about their sense of taste.<\/li>\n<\/ul>\n<h2 id=\"imagining-a-better-interview-process\">imagining a better interview process<a class=\"zola-anchor\" href=\"#imagining-a-better-interview-process\" aria-label=\"Anchor link for: imagining-a-better-interview-process\"><\/a>\n<\/h2>\n<p>if i were a hiring manager, i would use a combo of a code review interview and a work sample discussed live, giving precedence to the code review and telling the applicant ahead of time that the work sample doesn\u2019t have to be perfect.<\/p>\n<p>programming is fundamentally a collaborative process. having the applicant collaborate on both sides (reviewing and authoring) shows you a lot about how they work, and signals to them that you care about more than the equivalent of their SAT score.<\/p>\n<p>i also suggest there always be at least one interview between the applicant and their future manager (this seems to already be common practice\u2014yay!). \"people don't quit jobs, they quit bosses\": letting them meet ahead of time saves everyone pain down the road.<\/p>\n<p>thank you for reading! i hope this inspires you to change your own hiring processes, or at least to write a comment telling me why i'm wrong ^^. you can reach me by <a href=\"mailto:blog@jyn.dev\">email<\/a> if you want to comment privately.<\/p>\n<hr \/>\n<h2 id=\"p-s-other-interview-processes\">P.S. other interview processes<a class=\"zola-anchor\" href=\"#p-s-other-interview-processes\" aria-label=\"Anchor link for: p-s-other-interview-processes\"><\/a>\n<\/h2>\n<h3 id=\"pairing\">pairing<a class=\"zola-anchor\" href=\"#pairing\" aria-label=\"Anchor link for: pairing\"><\/a>\n<\/h3>\n<p>a friend worked at a Pivotal Labs where the primary job responsibility was to pair with client developers. the interview process was for a candidate to pair with an existing employee for a whole day and \"shadow\" them. he points to <a href=\"https:\/\/www.simplermachines.com\/notes-on-the-pivotal-interview-process\/\">Nat Bennett's notes on the interview process<\/a> as a more detailed writeup.<\/p>\n<h2 id=\"p-p-s-measuring-interview-effectiveness\">P.P.S. measuring interview effectiveness<a class=\"zola-anchor\" href=\"#p-p-s-measuring-interview-effectiveness\" aria-label=\"Anchor link for: p-p-s-measuring-interview-effectiveness\"><\/a>\n<\/h2>\n<p>the most interesting comment i got in response to this post was about \"red-teaming\" the interview to see how effective it is. for example:<\/p>\n<ul>\n<li>taking existing employees and putting them through the interview process and seeing whether it suggests hiring them again. if it doesn't, it's either horribly noisy or filtering out good candidates or both.<\/li>\n<li>\"regret analysis\": following the careers of rejected candidates to see who went on to do interesting things. if so, find out which part of the interview process failed and change it.<\/li>\n<\/ul>\n<p>the friend who suggested this said both ideas were categorically rejected by management and they continued to send him mediocre resumes for people he didn't want to hire.<\/p>\n<section class=\"footnotes\">\n<ol class=\"footnotes-list\">\n<li id=\"fn-1\">\n<p>update from after publishing: a friend said they\u2019ve seen people successfully use chatgpt to game design interviews. oof. <a href=\"#fr-1-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-2\">\n<p>this section was added the day after publishing in response to feedback from senior engineers and hiring managers. <a href=\"#fr-2-1\">\u21a9<\/a><\/p>\n<\/li>\n<\/ol>\n<\/section>\n"},{"title":"you are in a box","published":"2025-07-14T00:00:00+00:00","updated":"2025-07-14T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/you-are-in-a-box\/"}},"id":"https:\/\/jyn.dev\/you-are-in-a-box\/","summary":"your data is trapped inside the box that is your program. you can only see what the program author exposes.","content":"<figure>\n<blockquote cite=\"https:\/\/drmaciver.substack.com\/i\/145700143\/you-are-in-a-box\">\n<p><strong>You are trapped in a box. You have been for a long time.<\/strong><\/p>\n<\/blockquote>\n<figcaption>\n<cite ><a href=\"https:\/\/drmaciver.substack.com\/i\/145700143\/you-are-in-a-box\">D. R. MacIver<\/a><\/cite>\n<\/figcaption>\n<\/figure>\n<figure>\n<blockquote cite=\"https:\/\/en.wikipedia.org\/wiki\/Jamie_Zawinski#Zawinski's_Law\">\n<p><strong>Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can.<\/strong><\/p>\n<\/blockquote>\n<figcaption>\n<cite ><a href=\"https:\/\/en.wikipedia.org\/wiki\/Jamie_Zawinski#Zawinski's_Law\">Zawinski's Law of Software Envelopment<\/a><\/cite>\n<\/figcaption>\n<\/figure>\n<h2 id=\"switching-costs-and-growth\">switching costs and growth<a class=\"zola-anchor\" href=\"#switching-costs-and-growth\" aria-label=\"Anchor link for: switching-costs-and-growth\"><\/a>\n<\/h2>\n<p>most tools simultaneously think too small and too big. \u201ci will let you do anything!\u201d, they promise, \u201cas long as you give up your other tools and switch to me!\u201d<\/p>\n<p>this is true of languages too. any new programming language makes an implicit claim that \u201cusing this language will give you an advantage over any other language\u201d, at least for your current problem.<\/p>\n<p>once you start using a tool for one purpose, due to switching costs, you want to keep using that tool. so you start using it for things that wasn\u2019t designed for, and as a result, tools tend to grow and grow and grow until they <a href=\"\/technical-debt-is-different-from-technical-risk\/#what-to-do-about-risk\">stagnate<\/a>. in a sense, we have replicated the business boom-and-bust cycle in our own tools.<\/p>\n<!-- \/technical-debt-is-different-from-technical-risk\/#technical-risk-means-a-program-is-hard-to-modify -->\n<!--## escaping the boom and bust cycle-->\n<h2 id=\"interoperability\">interoperability<a class=\"zola-anchor\" href=\"#interoperability\" aria-label=\"Anchor link for: interoperability\"><\/a>\n<\/h2>\n<p>there are two possible ways to escape this trap. the first is to <a href=\"https:\/\/graydon2.dreamwidth.org\/263429.html\">impose a limit on growth<\/a>, so that tools can\u2019t grow until they bust. this makes a lot of people very unhappy and is generally regarded as a bad idea.<\/p>\n<p>the second is to decrease switching costs. by making it easier to switch between tools, or to interoperate between multiple tools in the same system, there is not as much pressure to have \u201cone big hammer\u201d that gets used for every problem.<\/p>\n<h3 id=\"back-compat\">back-compat<a class=\"zola-anchor\" href=\"#back-compat\" aria-label=\"Anchor link for: back-compat\"><\/a>\n<\/h3>\n<p>tools and languages can decrease switching costs by keeping backwards compatibility with older tools, or at least being close enough that they\u2019re <a href=\"https:\/\/youtu.be\/SxdOUGdseq4?si=X9OZ975hwwzZOxpo&amp;t=346\">easy to learn<\/a> for people coming from those tools. for example, ripgrep has almost exactly the same syntax as GNU grep, and nearly every compiled language since C has kept the curly braces.<\/p>\n<!--note that you can think of different versions of a tool as having to interoperate with themselves, so you may see these techniques used even when the authors did not intend a clear interface boundary.-->\n<h3 id=\"standardization\">standardization<a class=\"zola-anchor\" href=\"#standardization\" aria-label=\"Anchor link for: standardization\"><\/a>\n<\/h3>\n<p>tools can also collaborate on standards that make it easier to interoperate. this is the basis of nearly every network protocol, since there's no guarantee that the same tool will be on the other side of the connection. to some extent this also happens for languages (most notably for C), where a language specification allows multiple different compilers to work on the same code.<\/p>\n<p>this has limitations, however, because the tool itself has to want (or be forced) to interoperate. for example, the binary format for CUDA (a framework for compiling programs to the GPU) is undocumented, so you're stuck with <a href=\"https:\/\/blog.vivekpanyam.com\/parsing-an-undocumented-file-format\">reverse engineering<\/a> or <a href=\"https:\/\/docs.vulkan.org\/guide\/latest\/what_is_spirv.html\">re-implementing<\/a> the toolchain if you want to modify it.<\/p>\n<h3 id=\"ffi\">FFI<a class=\"zola-anchor\" href=\"#ffi\" aria-label=\"Anchor link for: ffi\"><\/a>\n<\/h3>\n<p>the last \"internal\" way to talk to languages is through a \"foreign function interface\", where functions in the same process can call each other cheaply. this is hard because each language has to go <a href=\"https:\/\/faultlore.com\/blah\/c-isnt-a-language\/\">all the way down to the C ABI<\/a> before there's something remotely resembling a standard, and because two languages may have incompatible runtime properties that make FFI <a href=\"https:\/\/pkg.go.dev\/cmd\/cgo#hdr-Passing_pointers\">hard<\/a> or <a href=\"https:\/\/words.filippo.io\/rustgo\/#why-not-cgo\">slow<\/a>. languages that do encourage FFI often require you to write separate bindings for each program: for example, Rust requires you to write <code>extern \"C\"<\/code> blocks for each declaration, and python requires you to do that and also write wrappers that translate C types into python objects.<\/p>\n<p>i won't talk too much more about this\u2014the work i'm aware of in this area is mostly around <a href=\"https:\/\/webassembly.org\/\">WASM<\/a> and <a href=\"https:\/\/component-model.bytecodealliance.org\/\">WASM Components<\/a>, and there are also some efforts to <a href=\"https:\/\/github.com\/rust-lang\/rfcs\/pull\/3470\">raise the baseline for ABI<\/a> above the C level.<\/p>\n<!--for instance, take the Go language. Go has many strengths\u2014a performant green threads runtime, excellent devtools, static binaries. in exchange, you are locked into the Go ecosystem. unlike other languages (Python, Rust, JS), calls between Go and other languages are [hard](https:\/\/pkg.go.dev\/cmd\/cgo#hdr-Passing_pointers) and have a [high performance overhead](), which means that most libraries you might want to use have to be rewritten into Go. Compare this to, for example, Lua, which is intentionally designed to be easy to embed into larger applications due to its small language size and minimal runtime requirements.-->\n<h3 id=\"ipc\">IPC<a class=\"zola-anchor\" href=\"#ipc\" aria-label=\"Anchor link for: ipc\"><\/a>\n<\/h3>\n<h4 id=\"unix-shells\">Unix shells<a class=\"zola-anchor\" href=\"#unix-shells\" aria-label=\"Anchor link for: unix-shells\"><\/a>\n<\/h4>\n<p>another approach is to compose tools. the traditional way to do this is to have a shell that allows you to freely compose programs with IPC. this does unlock a lot of freedom! IPC allows programs to communicate across different languages, different ABIs, and different user-facing APIs. it also unlocks 'ad-hoc' programs, which can be thought of as <a href=\"https:\/\/gwern.net\/doc\/technology\/2004-03-30-shirky-situatedsoftware.html\">situated software<\/a> for developers themselves. consider for example the following shell pipeline:<\/p>\n<pre data-lang=\"bash\" class=\"language-bash z-code\"><code class=\"language-bash\" data-lang=\"bash\"><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">git<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> verify-pack<span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> -<\/span>v<\/span> <span class=\"z-punctuation z-separator z-continuation z-line z-shell\">\\\n<\/span><\/span><\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-arguments z-shell\">        <span class=\"z-meta z-group z-expansion z-command z-parens z-shell\"><span class=\"z-punctuation z-definition z-variable z-shell\">$<\/span><span class=\"z-punctuation z-section z-parens z-begin z-shell\">(<\/span><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">git<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> rev-parse<span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> --<\/span>git-common-dir<\/span><\/span><span class=\"z-punctuation z-section z-parens z-end z-shell\">)<\/span><\/span>\/objects\/pack\/pack-<span class=\"z-keyword z-operator z-regexp z-quantifier z-shell\">*<\/span>.idx <span class=\"z-punctuation z-separator z-continuation z-line z-shell\">\\\n<\/span><\/span><\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-arguments z-shell\"><\/span>    <span class=\"z-keyword z-operator z-logical z-pipe z-shell\">|<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">sort<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> -<\/span>k3<\/span><span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> -<\/span>n<\/span><\/span> <span class=\"z-keyword z-operator z-logical z-pipe z-shell\">|<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">cut<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> -<\/span>f1<\/span><span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> -<\/span>d<\/span><span class=\"z-string z-quoted z-single z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&#39;<\/span> <span class=\"z-punctuation z-definition z-string z-end z-shell\">&#39;<\/span><\/span> <span class=\"z-punctuation z-separator z-continuation z-line z-shell\">\\\n<\/span><\/span><\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-arguments z-shell\"><\/span>    <span class=\"z-keyword z-operator z-logical z-pipe z-shell\">|<\/span> <span class=\"z-keyword z-control z-loop z-while z-shell\">while<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-support z-function z-read z-shell\">read<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> i<\/span><span class=\"z-keyword z-operator z-logical z-continue z-shell\">;<\/span> <span class=\"z-keyword z-control z-loop z-do z-shell\">do<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">git<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> ls-tree<span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> -<\/span>r<\/span> HEAD<\/span>  <span class=\"z-keyword z-operator z-logical z-pipe z-shell\">|<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">grep<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> <span class=\"z-string z-quoted z-double z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&quot;<\/span><span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-punctuation z-definition z-variable z-shell\">$<\/span><span class=\"z-variable z-other z-readwrite z-shell\">i<\/span><\/span><span class=\"z-punctuation z-definition z-string z-end z-shell\">&quot;<\/span><\/span><\/span><span class=\"z-keyword z-operator z-logical z-continue z-shell\">;<\/span> <span class=\"z-keyword z-control z-loop z-end z-shell\">done<\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> <span class=\"z-punctuation z-separator z-continuation z-line z-shell\">\\\n<\/span><\/span><\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-arguments z-shell\"><\/span>    <span class=\"z-keyword z-operator z-logical z-pipe z-shell\">|<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">tail<\/span><\/span>\n<\/span><\/code><\/pre>\n<p>this <a href=\"https:\/\/askubuntu.com\/questions\/1259129\/\">shows the 10 largest files in the git history for the current repository<\/a>. let's set aside the readability issues for now. there are a lot of good ideas here! note that programs are interacting freely in many ways:<\/p>\n<ul>\n<li>the output of <code>git rev-parse<\/code> is passed as a CLI argument to <code>git verify-pack<\/code><\/li>\n<li>the output of <code>git verify-pack<\/code> is passed as stdin to <code>sort<\/code><\/li>\n<li>the output of <code>cut<\/code> is interpreted as a list and programmatically manipulated by <code>while read<\/code>. this kind of meta-programming is common in shell and has concise (i won't go as far as \"simple\") syntax.<\/li>\n<li>the output from the meta-programming loop is itself passed as stdin to the <code>tail<\/code> command<\/li>\n<\/ul>\n<p>the equivalent in a programming language without spawning a subprocess would be very verbose; not only that, it would require a library for the git operations in each language, bringing back the FFI issues from before (not to mention the hard work designing \"cut points\" for the API interface<sup class=\"footnote-reference\" id=\"fr-1-1\"><a href=\"#fn-1\">1<\/a><\/sup>). this shell program can be written, concisely, using only tools that already exist.<\/p>\n<p>note though that the data flow here is a DAG: pipes are one-way, and the CLI arguments are evaluated before the new program is ever spawned. as a result, it\u2019s not possible to do any kind of content negotiation (other than the programmer hard-coding it with CLI args; for example tools commonly have <code>--format=json<\/code>).<\/p>\n<p>the downside of this approach is that the interface is completely unstructured; programs work on raw bytes, and there is no common interface. it also doesn't work if the program is interactive, unless the program deliberately exposes a way to query a running server (e.g. <code>tmux list-panes<\/code> or <code>nvim --remote<\/code>). let's talk about both of those.<\/p>\n<h4 id=\"structured-ipc\">structured IPC<a class=\"zola-anchor\" href=\"#structured-ipc\" aria-label=\"Anchor link for: structured-ipc\"><\/a>\n<\/h4>\n<p><a href=\"https:\/\/learn.microsoft.com\/en-us\/powershell\/scripting\/lang-spec\/chapter-04?view=powershell-7.5\">powershell<\/a>, and more recently, <a href=\"https:\/\/www.nushell.sh\/book\/types_of_data.html\">nushell<\/a>, extend traditional unix pipelines with structured data and a typesystem. they have mechanisms for parsing arbitrary text into native types, and helper functions for common data formats.<\/p>\n<p>this is really good! i think it is the first major innovation we have seen in the shell language in many decades, and i'm glad it exists. but it does have some limitations:<\/p>\n<ul>\n<li>there is no interop between powershell and nushell.<\/li>\n<li>there is no protocol for programs to self-describe their output in a schema, so each program's output has to be special-cased by each shell.\n<ul>\n<li>powershell side-steps this by <a href=\"https:\/\/learn.microsoft.com\/en-us\/powershell\/scripting\/overview?view=powershell-7.5#scripting-language\">building on the .NET runtime<\/a>, and having native support for programs which emit .NET objects in their <a href=\"https:\/\/learn.microsoft.com\/en-us\/powershell\/module\/microsoft.powershell.core\/about\/about_output_streams?view=powershell-7.5\">output stream<\/a>. but this doesn't generalize to programs that don't run on the <a href=\"https:\/\/learn.microsoft.com\/en-us\/dotnet\/standard\/clr\">CLR<\/a>.<\/li>\n<\/ul>\n<\/li>\n<li>there is no stability guarantee between versions of a program. even tools with semi-structured JSON output are free to change the structure of the JSON, breaking whatever code parses it.<\/li>\n<\/ul>\n<p>and it is very hard to fix these limitations because there is no \"out-of-band\" communication channel that programs could use to emit a schema; the closest you could get is a \"standardized file descriptor number\", but that will lock out any program that happens to already be using that FD. we have limited kinds of reflection in the form of shell completion scripts, but they're not standardized: there's not a standard for the shell to query the program, and there's not a standard for the format the program returns. the CLI framework inside the program often <em>does<\/em> have a schema and reflection capabilities, but they're discarded the second you go over an IPC boundary.<\/p>\n<h4 id=\"rpc\">RPC<a class=\"zola-anchor\" href=\"#rpc\" aria-label=\"Anchor link for: rpc\"><\/a>\n<\/h4>\n<p>how do you get a schema? well, you establish in-band communication. RPC is theoretically about \"remote\" procedure calls, but it's just as often used for local calls. the thing that really distinguishes it is that it's in-band: you have a defined interface that emits structured information.<\/p>\n<p>RPC works really quite well! there are frameworks for <a href=\"https:\/\/protobuf.dev\/\">forwards- and backwards-compatible RPC<\/a>; types and APIs are shared across languages; and interop for a new language only requires writing bindings between that language and the on-wire format, not solving a handshake problem between all pairs of languages nor dropping down to the C ABI.<\/p>\n<p>the main downside is that it is a lot of work to add to your program. you have to extensively modify your code to fit it into the shape the framework expects, and to keep it performant you sometimes even have to modify the in-memory representation of your data structures so they can live in a contiguous buffer. you can avoid these problems, but only by giving up performance when deserializing (e.g. by parsing JSON at runtime).<\/p>\n<h2 id=\"you-are-trapped-in-a-box\">you are trapped in a box<a class=\"zola-anchor\" href=\"#you-are-trapped-in-a-box\" aria-label=\"Anchor link for: you-are-trapped-in-a-box\"><\/a>\n<\/h2>\n<p>all these <a href=\"\/operators-not-users-and-programmers#the-user-programmer-distinction\">limitations<\/a> are because <a href=\"https:\/\/web.archive.org\/web\/20210121181531\/https:\/\/djrobstep.com\/posts\/programs-are-a-prison\">programs are a prison<\/a>. your data is trapped inside the box that is your program. the commonality between all these limitations is that they require work from the program developer, and without that work you're stuck. even the data that leaves the program has to go through the narrow entrances and exits of the box, and <a href=\"https:\/\/siderea.dreamwidth.org\/1540620.html\">anything that doesn't fit is discarded<\/a>.<\/p>\n<p>some languages try to make the box bigger\u2014interop between Java, Kotlin, and Clojure is comparatively quite easy because they all run on the JVM. but at the end of the day the JVM is another box; getting a non-JVM language to talk to it is hard.<\/p>\n<p>some languages try to make the box extensible\u2014LISPs, and especially Racket, try to make it very easy to build new languages inside the box. but getting non-LISPs inside the box is hard.<\/p>\n<p>some tools try to give you individual features\u2014smalltalk gets you orthogonal persistence; pluto.jl gets you a \u201cterminal of the future\u201d; rustc gets you sub-process incremental builds. but all those features are inside a box.<\/p>\n<p>often, tools don\u2019t even try. vendor lock in, subtle or otherwise, is everywhere around us. tools with this strategy tend to be the largest, since they have both the biggest budget and the greatest incentive to prevent you from switching tools.<\/p>\n<p>and always, always, always, you are at the mercy of the program author.<\/p>\n<p>in my next post, i will discuss how we can escape this box.<\/p>\n<hr \/>\n<h2 id=\"bibliography\">bibliography<a class=\"zola-anchor\" href=\"#bibliography\" aria-label=\"Anchor link for: bibliography\"><\/a>\n<\/h2>\n<ul>\n<li><a href=\"https:\/\/drmaciver.substack.com\/i\/145700143\/you-are-in-a-box\">D. R. MacIver, \u201cThis is important\u201d<\/a><\/li>\n<li><a href=\"https:\/\/en.wikipedia.org\/wiki\/Jamie_Zawinski\">Wikipedia, \u201cZawinski\u2019s Law of Software Envelopment\u201d<\/a><\/li>\n<li><a href=\"https:\/\/graydon2.dreamwidth.org\/263429.html\">Graydon Hoare, \u201cRust 2019 and beyond: limits to (some) growth.\u201d<\/a><\/li>\n<li><a href=\"https:\/\/youtu.be\/SxdOUGdseq4?si=X9OZ975hwwzZOxpo&amp;t=346\">Rich Hickey, \u201cSimple Made Easy\u201d<\/a><\/li>\n<li><a href=\"https:\/\/blog.vivekpanyam.com\/parsing-an-undocumented-file-format\">Vivek Panyam, \u201cParsing an undocumented file format\u201d<\/a><\/li>\n<li><a href=\"https:\/\/docs.vulkan.org\/guide\/latest\/what_is_spirv.html\">The Khronos\u00ae Group Inc, \u201cVulcan Documentation: What is SPIR-V\u201d<\/a><\/li>\n<li><a href=\"https:\/\/faultlore.com\/blah\/c-isnt-a-language\/\">Aria Desires, \u201cC Isn\u2019t A Language Anymore\u201d<\/a><\/li>\n<li><a href=\"https:\/\/pkg.go.dev\/cmd\/cgo\">Google LLC, \u201cStandard library: cmd.cgo\u201d<\/a><\/li>\n<li><a href=\"https:\/\/words.filippo.io\/rustgo\/\">Filippo Valsorda, \u201crustgo: calling Rust from Go with near-zero overhead\u201d<\/a><\/li>\n<li><a href=\"https:\/\/webassembly.org\/\">WebAssembly Working Group, \u201cWebAssembly\u201d<\/a><\/li>\n<li><a href=\"https:\/\/component-model.bytecodealliance.org\/\">The Bytecode Alliance, \u201cThe WebAssembly Component Model\u201d<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/rust-lang\/rfcs\/pull\/3470\">Josh Triplett, \u201ccrABI v1\u201d<\/a><\/li>\n<li><a href=\"https:\/\/gwern.net\/doc\/technology\/2004-03-30-shirky-situatedsoftware.html\">Clay Shirky, \"Situated Software\"<\/a><\/li>\n<li><a href=\"https:\/\/learn.microsoft.com\/en-us\/powershell\/scripting\/lang-spec\/chapter-04?view=powershell-7.5\">Microsoft, \"PowerShell 7.5: 4. Types\"<\/a><\/li>\n<li><a href=\"https:\/\/learn.microsoft.com\/en-us\/powershell\/scripting\/overview?view=powershell-7.5\">Microsoft, \"PowerShell 7.5: What is PowerShell?\"<\/a><\/li>\n<li><a href=\"https:\/\/learn.microsoft.com\/en-us\/powershell\/module\/microsoft.powershell.core\/about\/about_output_streams?view=powershell-7.5\">Microsoft, \"PowerShell 7.5: about_Output_Streams\"<\/a><\/li>\n<li><a href=\"https:\/\/learn.microsoft.com\/en-us\/dotnet\/standard\/clr\">Microsoft, \".NET Execution model: Common Language Runtime (CLR) overview\"<\/a><\/li>\n<li><a href=\"https:\/\/www.nushell.sh\/book\/types_of_data.html\">Nushell Project, \"Nu Fundamentals: Types of Data\"<\/a><\/li>\n<li><a href=\"https:\/\/protobuf.dev\/\">Google LLC, \u201cProtocol Buffers\u201d<\/a><\/li>\n<li><a href=\"https:\/\/web.archive.org\/web\/20210121181531\/https:\/\/djrobstep.com\/posts\/programs-are-a-prison\">Robert Lechte, \u201cPrograms are a prison: Rethinking the fundamental building blocks of computing interfaces\u201d<\/a><\/li>\n<li><a href=\"https:\/\/siderea.dreamwidth.org\/1540620.html\">Siderea, \"Procrustean Epistemologies\"<\/a><\/li>\n<\/ul>\n<section class=\"footnotes\">\n<ol class=\"footnotes-list\">\n<li id=\"fn-1\">\n<p>blog post forthcoming <a href=\"#fr-1-1\">\u21a9<\/a><\/p>\n<\/li>\n<\/ol>\n<\/section>\n"},{"title":"constrained languages are easier to optimize","published":"2025-07-12T00:00:00+00:00","updated":"2025-07-12T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/constrained-languages-are-easier-to-optimize\/"}},"id":"https:\/\/jyn.dev\/constrained-languages-are-easier-to-optimize\/","summary":"exposing raw pointers make the optimizer\u2019s job horribly hard. high level languages can constrain your program, making more optimizations sound.","content":"<h2 id=\"jyn-what-the-fuck-are-you-talking-about\">jyn, what the fuck are you talking about<a class=\"zola-anchor\" href=\"#jyn-what-the-fuck-are-you-talking-about\" aria-label=\"Anchor link for: jyn-what-the-fuck-are-you-talking-about\"><\/a>\n<\/h2>\n<p>a recurring problem in modern \u201clow-level\u201d languages<sup class=\"footnote-reference\" id=\"fr-2-1\"><a href=\"#fn-2\">1<\/a><\/sup> is that they are hard to optimize. they <a href=\"https:\/\/queue.acm.org\/detail.cfm?id=3212479\">do not reflect the hardware<\/a>, they require doing <a href=\"https:\/\/www.ralfj.de\/blog\/2018\/07\/24\/pointers-and-bytes.html\">complex alias analysis<\/a>, and they <a href=\"https:\/\/medium.com\/@jbyj\/my-javascript-is-faster-than-your-rust-5f98fe5db1bf\">constantly allocate and deallocate memory<\/a>. <sup class=\"footnote-reference\" id=\"fr-1-1\"><a href=\"#fn-1\">2<\/a><\/sup> they looked at <a href=\"https:\/\/buttondown.com\/hillelwayne\/archive\/the-capability-tractability-tradeoff\/\">the structure\/expressiveness tradeoff<\/a> and consistently chose expressiveness.<\/p>\n<h2 id=\"what-does-a-faster-language-look-like\">what does a faster language look like<a class=\"zola-anchor\" href=\"#what-does-a-faster-language-look-like\" aria-label=\"Anchor link for: what-does-a-faster-language-look-like\"><\/a>\n<\/h2>\n<p>consider this paper on <a href=\"https:\/\/www.cs.tufts.edu\/~nr\/cs257\/archive\/duncan-coutts\/stream-fusion.pdf\">stream fusion in Haskell<\/a>. this takes a series of nested loops, each of which logically allocate an array equal in size to the input, and optimizes them down to constant space using unboxed integers. doing the same with C is inherently less general because the optimizing compiler must first prove that none of the pointers involved alias each other. in fact, optimizations are so much easier to get right in Haskell that <a href=\"https:\/\/wiki.haskell.org\/GHC\/Using_rules\">GHC exposes a mechanism for users to define them<\/a>! these optimizations are possible because of <a href=\"https:\/\/softwareengineering.stackexchange.com\/questions\/254304\/what-is-referential-transparency\">referential transparency<\/a>\u2014the compiler statically knows whether an expression can have a side effect.<\/p>\n<p>\u201chaskell is known for performance problems, why are you using it as an example. also all GC languages constantly box and unbox values, you need raw pointers to avoid that.\u201d<\/p>\n<p>GC languages do constantly box and unbox <sup class=\"footnote-reference\" id=\"fr-4-1\"><a href=\"#fn-4\">3<\/a><\/sup>, but you don\u2019t need raw pointers to avoid that. consider <a href=\"https:\/\/futhark-lang.org\">futhark<\/a>, a functional parallel language that compiles to the GPU. its benchmarks show it being <a href=\"https:\/\/futhark-lang.org\/performance.html\">up to orders of magnitude faster than sequential C<\/a> on problems that fit well into its domain. it does so by having unboxed fixed-size integers, disallowing ragged arrays, and constraining many common operations on arrays to only work if the arrays are statically known to have the same size.<\/p>\n<p>futhark is highly restrictive. consider instead SQL. SQL is a declarative language, which means the actual execution is determined by a query planner, it\u2019s not constrained by the source code. SQL has also been around for decades, which means we can compare the performance of the same code over decades. it turns out <a href=\"https:\/\/rmarcus.info\/blog\/2024\/04\/12\/pg-over-time.html\">common operations in postgres are twice as fast as they were a decade ago<\/a>. you can imagine writing SQL inline\u2014wait no it turns out C# <a href=\"https:\/\/learn.microsoft.com\/en-us\/dotnet\/csharp\/linq\/\">already has that covered<\/a>.<\/p>\n<p>SQL is not a general purpose language. but you don\u2019t need it to be! your performance issues are not evenly distributed across your code; you can identify the hotspots and choose against a language with raw pointers in favor of one more structured and therefore more amenable to optimization.<\/p>\n<h2 id=\"sometimes-you-need-raw-pointers\">sometimes you need raw pointers<a class=\"zola-anchor\" href=\"#sometimes-you-need-raw-pointers\" aria-label=\"Anchor link for: sometimes-you-need-raw-pointers\"><\/a>\n<\/h2>\n<p>there are various kinds of memory optimizations that are only possible if you have access to raw pointers; for example <a href=\"https:\/\/piotrduperas.com\/posts\/nan-boxing\">NaN boxing<\/a>, <a href=\"https:\/\/github.com\/laurelmay\/xorlist\">XOR linked lists<\/a>, and <a href=\"https:\/\/en.wikipedia.org\/wiki\/Tagged_pointer?wprov=sfti1\">tagged pointers<\/a>. sometimes you need them, which means you need a language that allows them. but these kinds of data structures are very rare! we should steer towards a general purpose language that does not expose raw pointers, and only drop down when we actually need to use them.<\/p>\n<h2 id=\"what-does-a-faster-general-purpose-language-look-like\">what does a faster general purpose language look like<a class=\"zola-anchor\" href=\"#what-does-a-faster-general-purpose-language-look-like\" aria-label=\"Anchor link for: what-does-a-faster-general-purpose-language-look-like\"><\/a>\n<\/h2>\n<p>well, Rust is a good step in the right direction: raw pointers are opt-in with <code>unsafe<\/code>; Iterators support functional paradigms that allow removing bounds checks and <a href=\"https:\/\/ntietz.com\/blog\/rusts-iterators-optimize-footgun\/\">fusing stream-like operations<\/a>; and libraries like <a href=\"https:\/\/docs.rs\/rayon\/latest\/rayon\/\">rayon<\/a> make it much easier to do multi-threaded compilation.<\/p>\n<p>but i think this is in some sense the wrong question. we should not be asking \u201cwhat language can i use everywhere for every purpose\u201d; we should build meta-languages that allow you to easily use the right tool for the job. this is already true for regular expressions and query languages; let\u2019s go further. i want inline futhark; inline CSS selectors; inline datalog; ffi between python and C that\u2019s trivially easy. the easier we make it to interop, the easier it becomes to pick the right tool for the job. <!-- TODO: link to systems thinking post -->  <!-- TODO: link to composable compilers --><\/p>\n<p>next time you hit a missed optimization, ask yourself: why was this hard for the compiler? can i imagine a language where optimizing this is easier?<\/p>\n<h2 id=\"what-have-we-learned\">what have we learned?<a class=\"zola-anchor\" href=\"#what-have-we-learned\" aria-label=\"Anchor link for: what-have-we-learned\"><\/a>\n<\/h2>\n<ul>\n<li>languages that expose raw pointers are surprisingly hard to optimize<\/li>\n<li>by constraining the language to require additional structure, the compiler has much more freedom to optimize<\/li>\n<li>by making it easier to switch between languages, we make it easier to choose the right tool for the job, increasing the performance of our code<\/li>\n<\/ul>\n<section class=\"footnotes\">\n<ol class=\"footnotes-list\">\n<li id=\"fn-2\">\n<p>this is true for all of C, C++, and unsafe Rust (and to some extent Fortran, but Fortran <a href=\"https:\/\/beza1e1.tuxen.de\/articles\/faster_than_C.html\">does not require alias analysis<\/a>). <a href=\"#fr-2-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-1\">\n<p>also, they require doing PGO ahead of time instead of collecting info dynamically at runtime. but i haven\u2019t found any benchmarks showing Java\/luaJIT programs that are faster than equivalent C, so i won\u2019t claim that JIT is inherently faster. <a href=\"#fr-1-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-4\">\n<p>true in the general case, but not always in practice. in Go and Java, the compiler needs to do escape analysis to know whether a variable can be unboxed. in Haskell, the situation is more complicated because of lazy evaluation; see <a href=\"https:\/\/youtu.be\/XiTO1EGKrhE?si=z1IqPD4KdEkqcYd9\">Alexis King on the GHC strictness analyzer<\/a> for more info. <a href=\"#fr-4-1\">\u21a9<\/a><\/p>\n<\/li>\n<\/ol>\n<\/section>\n"},{"title":"sorry for the rss screwup","published":"2025-07-06T00:00:00+00:00","updated":"2025-07-06T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/sorry-for-the-rss-screwup\/"}},"id":"https:\/\/jyn.dev\/sorry-for-the-rss-screwup\/","content":"<p>a couple days ago i pushed about 10 empty posts at once to everyone subscribed to my RSS feed. oops. sorry about that.<\/p>\n<p>i've since fixed it, but most RSS readers i've seen will cache the posts indefinitely once they're published.\nthe workaround for my own client was to delete my page and then readd it, which will unfortunately discard all your read\/unread state.<\/p>\n<p>the reason this happened is that i added a new kind of \"stub\" post, and handled that correctly on the main site, but not on the rss feed. you can see the intended layout of the stub posts <a href=\"\/computer-of-the-future\">here<\/a> if you're interested.<\/p>\n"},{"title":"operators, not users and programmers","published":"2025-07-05T00:00:00+00:00","updated":"2025-07-05T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/operators-not-users-and-programmers\/"}},"id":"https:\/\/jyn.dev\/operators-not-users-and-programmers\/","summary":"writing programs should be no harder than writing a resume","content":"<p><strong>the modern distinction between \u201cprogrammers\u201d and \u201cusers\u201d is evil and destroys agency.<\/strong><\/p>\n<h2 id=\"consider-how-the-spreadsheets-grow\">consider how the spreadsheets grow<a class=\"zola-anchor\" href=\"#consider-how-the-spreadsheets-grow\" aria-label=\"Anchor link for: consider-how-the-spreadsheets-grow\"><\/a>\n<\/h2>\n<p>spreadsheets are hugely successful. Felienne Hermans, who has spent her career studying spreadsheets, <a href=\"https:\/\/www.felienne.com\/archives\/2453\">attributes this success<\/a> to \"<em>their immediate feedback\u00a0system and their continuous deployment model<\/em>\": the spreadsheet shows you its result as soon as you open it, and it requires no steps to run other than to install Excel and double-click the file.<\/p>\n<p>Rik calls Excel \u201cmalleable software\u201d and the resulting programs <a href=\"https:\/\/nothingisnttrivial.com\/vines.html\">\u201cvine-like systems\u201d<\/a>:<\/p>\n<blockquote>\n<p>The dream of malleable software is that we can enlarge the space of possibilities, users can enjoy more freedom and get more out of their software, without having to rely on software developers to provide it for them.<\/p>\n<\/blockquote>\n<p>i would go one step further: the dream of malleable software is to unify users and programmers, such that there are just \u201coperators\u201d of a computer, and \u201cwriting a program\u201d doesn\u2019t sound any harder than \u201cwriting a resume\u201d:<\/p>\n<figure>\n<blockquote cite=\"http:\/\/www.loper-os.org\/?p=284\">\n<p>the distinction between \"user\" and \"programmer\" is an artifact of our presently barely-programmable and barely-usable computing systems.\u00a0 I would like to use the neutral word \"operator\" instead.<\/p>\n<\/blockquote>\n<figcaption>\n<cite ><a href=\"http:\/\/www.loper-os.org\/?p=284\">Stanislav<\/a><\/cite>\n<\/figcaption>\n<\/figure>\n<p>this is a relatively new distinction! if we look at the history of computing and of programming languages, we see very different patterns:<\/p>\n<figure>\n<blockquote cite=\"https:\/\/www.barnesandnoble.com\/w\/the-as-400-ibm-i-rpg-rpgiv-programming-guide-brian-w-kelly\/1124946681\">\n<p>In the 1960\u2019s the supply of programmers was not very deep so IBM and other companies trying to gain a computer sale would often have to sell the business prospect on the idea of creating its own programmer(s). Sometimes it was the shipping clerk; sometimes it was the head order taker; sometimes it was a bookkeeper, and sometimes it was a woman or man packing items in the warehouse.<\/p>\n<\/blockquote>\n<figcaption>\n<cite ><a href=\"https:\/\/www.barnesandnoble.com\/w\/the-as-400-ibm-i-rpg-rpgiv-programming-guide-brian-w-kelly\/1124946681\">Brian M. Kelly, The AS\/400 and IBM i RPG Programming Guide<\/a><\/cite>\n<\/figcaption>\n<\/figure>\n<p>this is what i want: for programming to be easy and simple enough to pick up that people can do it without specialized training in the field, so that they can write <a href=\"https:\/\/gwern.net\/doc\/technology\/2004-03-30-shirky-situatedsoftware.html\">situated software<\/a>.<\/p>\n<h2 id=\"the-user-programmer-distinction\">the \u201cuser\/programmer\u201d distinction<a class=\"zola-anchor\" href=\"#the-user-programmer-distinction\" aria-label=\"Anchor link for: the-user-programmer-distinction\"><\/a>\n<\/h2>\n<p>contrast malleable software to the systems that programmers often build:<\/p>\n<blockquote>\n<p>Many, many technologists have taken one look at an existing workflow of spreadsheets, reacted with performative disgust, and proposed the trifecta of microservices, Kubernetes and something called a \"service mesh\".<\/p>\n<\/blockquote>\n<figure>\n<blockquote cite=\"https:\/\/calpaterson.com\/bank-python.html\">\n<p>This kind of Big Enterprise technology however takes away that basic agency of those Excel users, who no longer understand the business process they run and now have to negotiate with\u00a0<a href=\"https:\/\/www.youtube.com\/watch?v=y8OnoxKotPQ\">ludicrous technology dweebs<\/a>\u00a0for each software change. The previous pliability of the spreadsheets has been completely lost.<\/p>\n<\/blockquote>\n<figcaption>\n<cite ><a href=\"https:\/\/calpaterson.com\/bank-python.html\">Cal Peterson<\/a><\/cite>\n<\/figcaption>\n<\/figure>\n<p>now, this doesn\u2019t happen because programmers are stupid and evil. it happens because the systems we build are amenable to all the features that programmers <a href=\"https:\/\/www.hillelwayne.com\/post\/what-we-can-learn\/#version-control\">take for granted<\/a>:<\/p>\n<ul>\n<li>distributed version control systems (VCS)<\/li>\n<li>automated testing (when integrated with a VCS, often called \"continuous integration\")<\/li>\n<li>gradual controlled rollouts (when integrated with a VCS, often called \"continuous deployment\")<\/li>\n<\/ul>\n<p>what i want to imagine is what it would look like to build computing systems that have those features and are also malleable.<\/p>\n<h2 id=\"what-could-a-malleable-system-with-low-technical-risk-look-like\">what could a malleable system with <a href=\"https:\/\/nothingisnttrivial.com\/technical-functional.html\">low technical risk<\/a> look like?<a class=\"zola-anchor\" href=\"#what-could-a-malleable-system-with-low-technical-risk-look-like\" aria-label=\"Anchor link for: what-could-a-malleable-system-with-low-technical-risk-look-like\"><\/a>\n<\/h2>\n<p>let's start by looking at what malleable systems already exist.<\/p>\n<h3 id=\"malleable-languages\">malleable languages<a class=\"zola-anchor\" href=\"#malleable-languages\" aria-label=\"Anchor link for: malleable-languages\"><\/a>\n<\/h3>\n<ul>\n<li>spreadsheets, as previously discussed<\/li>\n<li>WYSIWYG editors: Microsoft Word, Obsidian, Typst, Wordpress. In Word, and in Obsidian's default view, the compile step is completely hidden and it appears to the user as if the source and rendered view are the same. in other editors, the rendered view updates quickly and frequently enough that they get immediate feedback, just like a spreadsheet.<\/li>\n<li>browser devtools, where editing an element in the inspector immediately updates the rendered view<\/li>\n<li><a href=\"https:\/\/sonic-pi.net\/\">sonic-pi<\/a>, where editing code live-updates the sound played<\/li>\n<li><a href=\"\/how-i-write-blog-posts\/\">my blog<\/a><\/li>\n<\/ul>\n<p>note these aren't <em>just<\/em> hot-patching, where you edit a system while it's running. hot-patching is certainly useful and i would like to see more of it, but it doesn't unify the source and rendered version of a program, you still have to trip the condition in order to observe the change. malleable languages are a combination of:<\/p>\n<ul>\n<li>hot-patching<\/li>\n<li>live previews that makes updates appear instant<\/li>\n<li>undo\/redo for the whole system<\/li>\n<\/ul>\n<h3 id=\"malleable-version-control\">malleable version control<a class=\"zola-anchor\" href=\"#malleable-version-control\" aria-label=\"Anchor link for: malleable-version-control\"><\/a>\n<\/h3>\n<ul>\n<li>file shares: google drive; sharepoint; onedrive; etc. the tooling for diffing and reverting is very primitive compared to git or hg (in particular diffing is the responsibility of the application, not the VCS), but the primitives are there.<\/li>\n<li><a href=\"https:\/\/calpaterson.com\/bank-python.html#:~:text=barbara\">Bank Python<\/a>, where the distinction between local and persisted storage is smoothed over by the runtime<\/li>\n<\/ul>\n<p>note a pattern here: these don't require prior approval in order to use. anyone can throw a file into google drive and hit share without prior approval (sometimes this is frowned upon, in which case it gets called \"Shadow IT\").<\/p>\n<h3 id=\"malleable-deployment\">malleable deployment<a class=\"zola-anchor\" href=\"#malleable-deployment\" aria-label=\"Anchor link for: malleable-deployment\"><\/a>\n<\/h3>\n<ul>\n<li>Bank Python's <a href=\"https:\/\/calpaterson.com\/bank-python.html#:~:text=once%20described\">vouch system<\/a>, where hitting \"approve\" insta-deploys to prod. at a technical level this is basically the same as code review, but unlike normal CI systems it\u2019s not configured to require tests to pass, because anything that\u2019s too annoying will end up with people bypassing the system to build shadow IT.<\/li>\n<\/ul>\n<p>unfortunately i am not aware of a malleable testing system\u2014if you know of one, please do <a href=\"mailto:blog@jyn.dev\">tell me<\/a>!<\/p>\n<h3 id=\"putting-it-together\">putting it together<a class=\"zola-anchor\" href=\"#putting-it-together\" aria-label=\"Anchor link for: putting-it-together\"><\/a>\n<\/h3>\n<p>so, we want the following from a malleable system:<\/p>\n<ul>\n<li>hot-reloading and live previews, like spreadsheets\n<ul>\n<li>in particular, we want the user to write in the representation that makes the most sense to them, whether that's a spreadsheet or an SQL query or a text editor of markup<\/li>\n<\/ul>\n<\/li>\n<li><a href=\"..\/complected-and-orthogonal-persistence\">automatic and continuous durability<\/a>, like autosave in microsoft office products.\n<ul>\n<li>in particular this undo\/redo capability works on the whole system including derived data, not just the source code itself, so you can try new things without fear of breaking everything. the closest programmers have to this today is <code>jj<\/code>, which <a href=\"https:\/\/jj-vcs.github.io\/jj\/latest\/working-copy\/\">automatically snapshots the working tree<\/a>, but taking snapshots still requires a manual action, and you need to set up the system in advance.<\/li>\n<\/ul>\n<\/li>\n<li>distributed version control++\n<ul>\n<li>with unrestricted <em>distribution<\/em> and an easy interface, like google drive and dropbox<\/li>\n<li>with diffing\/merging\/reverting, like traditional VCS<\/li>\n<\/ul>\n<\/li>\n<li>automated and instantly triggered testing, like piper at google. this could run your tests in the background as you edit your program, with integrated build caching so that only the affected tests rerun.<\/li>\n<li>continuous deployment\n<ul>\n<li>we want explicit approval and controlled rollout, like traditional CD, so we can separate \"upload the code\" from \"run the code in prod\"<\/li>\n<li>but we want it to be trivially easy to give that approval, like double-clicking an email attachment or the vouch system. don't confuse \"controlled rollout\" with access control\u2014it's useful even if you're a solo programmer.<\/li>\n<\/ul>\n<\/li>\n<li>and of course, performance.<\/li>\n<\/ul>\n<p>this is a tall order! the rest of this series is dedicated to ideas about how to make this possible.<\/p>\n<hr \/>\n<h2 id=\"bibliography\">bibliography<a class=\"zola-anchor\" href=\"#bibliography\" aria-label=\"Anchor link for: bibliography\"><\/a>\n<\/h2>\n<ul>\n<li><a href=\"https:\/\/www.felienne.com\/archives\/2453\">Felienne Hermans, \"Proposition #1\"<\/a><\/li>\n<li><a href=\"https:\/\/nothingisnttrivial.com\/vines.html\">Rik de Kort, \"Vine-like Systems and Malleability\"<\/a><\/li>\n<li><a href=\"https:\/\/nothingisnttrivial.com\/technical-functional.html\">Rik de Kort, \"Technical and functional risk\"<\/a><\/li>\n<li><a href=\"http:\/\/www.loper-os.org\/?p=284\">Stanislav Datskovskiy, \"Seven Laws of Sane Personal Computing\"<\/a><\/li>\n<li><a href=\"https:\/\/www.barnesandnoble.com\/w\/the-as-400-ibm-i-rpg-rpgiv-programming-guide-brian-w-kelly\/1124946681\">Brian M. Kelly, <em>The AS\/400 and IBM i RPG Programming Guide<\/em><\/a><\/li>\n<li><a href=\"https:\/\/gwern.net\/doc\/technology\/2004-03-30-shirky-situatedsoftware.html\">Clay Shirky, \"Situated Software\"<\/a><\/li>\n<li><a href=\"https:\/\/calpaterson.com\/bank-python.html\">Cal Peterson, \"An oral history of Bank Python\"<\/a><\/li>\n<li><a href=\"https:\/\/www.hillelwayne.com\/post\/what-we-can-learn\/#version-control\">Hillel Wayne, \"What engineering can teach (and learn from) us\"<\/a><\/li>\n<li><a href=\"https:\/\/sonic-pi.net\/\">Sonic Pi<\/a><\/li>\n<li><a href=\"https:\/\/jj-vcs.github.io\/jj\/latest\/working-copy\/\">\"Jujutsu\u2014a version control system\"<\/a><\/li>\n<\/ul>\n<hr \/>\n<h2 id=\"p-s-what-will-people-build-with-this-power\">P.S: what will people build with this power?<a class=\"zola-anchor\" href=\"#p-s-what-will-people-build-with-this-power\" aria-label=\"Anchor link for: p-s-what-will-people-build-with-this-power\"><\/a>\n<\/h2>\n<p>i consulted a local friend of mine, a polisci major, and asked him what he would build if he were able to program. he said a penguin that walks across the screen, followed by a puffin friend for him. so, here's to penguins \ud83d\udc27\n<img src=\"\/900.jpg\" alt=\"penguin\" \/>\n<img src=\"\/Puffin_(Fratercula_arctica).jpg\" alt=\"puffin\" \/><\/p>\n"},{"title":"complected and orthogonal persistence","published":"2025-06-30T00:00:00+00:00","updated":"2025-06-30T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/complected-and-orthogonal-persistence\/"}},"id":"https:\/\/jyn.dev\/complected-and-orthogonal-persistence\/","summary":"how hard is it to save and restore program state?","content":"<blockquote>\n<p><em><strong>Everything Not Saved Will Be Lost<\/strong><\/em>\u2014Ancient Nintendo Proverb<\/p>\n<\/blockquote>\n<h2 id=\"persistence-is-hard\">persistence is hard<a class=\"zola-anchor\" href=\"#persistence-is-hard\" aria-label=\"Anchor link for: persistence-is-hard\"><\/a>\n<\/h2>\n<p>say that you are writing an editor. you don't want to lose people's work so you implement an \"autobackup\" feature, so that people can restore their unsaved changes if the program or whole computer crashes.<\/p>\n<p><a href=\"https:\/\/danluu.com\/file-consistency\/\">implementing this is hard<\/a>! the way i would do it is to serialize the data structures using something like <a href=\"https:\/\/docs.rs\/bincode\/latest\/bincode\/\">bincode<\/a> and then write them to an SQLite database so that i get crash-consistency. there are other approaches with different tradeoffs.<\/p>\n<h2 id=\"languages-with-persistence\">languages with persistence<a class=\"zola-anchor\" href=\"#languages-with-persistence\" aria-label=\"Anchor link for: languages-with-persistence\"><\/a>\n<\/h2>\n<p>this <a href=\"https:\/\/archive.cs.st-andrews.ac.uk\/papers\/download\/ABC+83b.pdf\">1983 paper<\/a> asks: why are we spending so much time rewriting this in applications, instead of doing it once in the runtime? it then introduces a language called \"PS-algol\", which supports exactly this through a library. note that arbitrary data types in the language are supported without the need for writing user code.<\/p>\n<p>it turns out that this idea is already being used in production. not in that form\u2014people don\u2019t use Algol anymore\u2014but the idea is the same. M (better known as <a href=\"https:\/\/en.wikipedia.org\/wiki\/MUMPS\">MUMPS<\/a>), <a href=\"https:\/\/calpaterson.com\/bank-python.html\">Bank Python<\/a>, and the <a href=\"https:\/\/www.devever.net\/~hl\/f\/as400guide.pdf\">IBM i<\/a><sup class=\"footnote-reference\" id=\"fr-4-1\"><a href=\"#fn-4\">1<\/a><\/sup> are still used in healthcare, financial, and insurance systems, and they work exactly like this<sup class=\"footnote-reference\" id=\"fr-5-1\"><a href=\"#fn-5\">2<\/a><\/sup>. here is a snippet of M that persists some data to a database:<\/p>\n<pre data-lang=\"mumps\" class=\"language-mumps z-code\"><code class=\"language-mumps\" data-lang=\"mumps\"><span class=\"z-source z-mumps\"> <span class=\"z-keyword\">SET<\/span> ^table(<span class=\"z-string z-quoted z-double z-mumps\"><span class=\"z-punctuation z-definition z-string z-begin z-mumps\">&quot;<\/span>column<span class=\"z-punctuation z-definition z-string z-end z-mumps\">&quot;<\/span><\/span>,<span class=\"z-string z-quoted z-double z-mumps\"><span class=\"z-punctuation z-definition z-string z-begin z-mumps\">&quot;<\/span>primary key<span class=\"z-punctuation z-definition z-string z-end z-mumps\">&quot;<\/span><\/span>)=<span class=\"z-string z-quoted z-double z-mumps\"><span class=\"z-punctuation z-definition z-string z-begin z-mumps\">&quot;<\/span>value<span class=\"z-punctuation z-definition z-string z-end z-mumps\">&quot;<\/span><\/span><span class=\"z-source z-mumps\">\n<\/span><\/span><\/code><\/pre>\n<p>and here is some Bank Python that does the same:<\/p>\n<pre data-lang=\"python\" class=\"language-python z-code\"><code class=\"language-python\" data-lang=\"python\"><span class=\"z-source z-python\"><span class=\"z-meta z-item-access z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">db<\/span><\/span><\/span><span class=\"z-meta z-item-access z-python\"><span class=\"z-punctuation z-section z-brackets z-begin z-python\">[<\/span><\/span><span class=\"z-meta z-item-access z-arguments z-python\"><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\"><span class=\"z-punctuation z-definition z-string z-begin z-python\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\">\/my_table<span class=\"z-punctuation z-definition z-string z-end z-python\">&quot;<\/span><\/span><\/span><\/span><span class=\"z-meta z-item-access z-python\"><span class=\"z-punctuation z-section z-brackets z-end z-python\">]<\/span><\/span> <span class=\"z-keyword z-operator z-assignment z-python\">=<\/span> <span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-meta z-generic-name z-python\">Table<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span>\n<\/span><\/span><span class=\"z-source z-python\"><span class=\"z-meta z-function-call z-arguments z-python\">  <span class=\"z-meta z-sequence z-list z-python\"><span class=\"z-punctuation z-section z-sequence z-begin z-python\">[<\/span><span class=\"z-meta z-sequence z-tuple z-python\"><span class=\"z-punctuation z-section z-sequence z-begin z-python\">(<\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\"><span class=\"z-punctuation z-definition z-string z-begin z-python\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\">etf<span class=\"z-punctuation z-definition z-string z-end z-python\">&quot;<\/span><\/span><\/span><span class=\"z-punctuation z-separator z-sequence z-python\">,<\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-support z-type z-python\">str<\/span><\/span><span class=\"z-punctuation z-section z-sequence z-end z-python\">)<\/span><\/span><span class=\"z-punctuation z-separator z-sequence z-python\">,<\/span> <span class=\"z-meta z-sequence z-tuple z-python\"><span class=\"z-punctuation z-section z-sequence z-begin z-python\">(<\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\"><span class=\"z-punctuation z-definition z-string z-begin z-python\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\">shares<span class=\"z-punctuation z-definition z-string z-end z-python\">&quot;<\/span><\/span><\/span><span class=\"z-punctuation z-separator z-sequence z-python\">,<\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-support z-type z-python\">float<\/span><\/span><span class=\"z-punctuation z-section z-sequence z-end z-python\">)<\/span><\/span><span class=\"z-punctuation z-section z-sequence z-end z-python\">]<\/span><\/span><span class=\"z-punctuation z-separator z-arguments z-python\">,<\/span>\n<\/span><\/span><span class=\"z-source z-python\"><span class=\"z-meta z-function-call z-arguments z-python\">  <span class=\"z-meta z-sequence z-list z-python\"><span class=\"z-punctuation z-section z-sequence z-begin z-python\">[<\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\"><span class=\"z-punctuation z-definition z-string z-begin z-python\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\">SPY<span class=\"z-punctuation z-definition z-string z-end z-python\">&quot;<\/span><\/span><\/span><span class=\"z-punctuation z-separator z-sequence z-python\">,<\/span> <span class=\"z-constant z-numeric z-float z-decimal z-python\">1200<span class=\"z-punctuation z-separator z-decimal z-python\">.<\/span>0<\/span><span class=\"z-punctuation z-section z-sequence z-end z-python\">]<\/span><\/span>\n<\/span><\/span><span class=\"z-source z-python\"><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span>\n<\/span><\/code><\/pre>\n<p>and finally some COBOL: <sup class=\"footnote-reference\" id=\"fr-3-1\"><a href=\"#fn-3\">3<\/a><\/sup><\/p>\n<pre data-lang=\"COBOL\" class=\"language-COBOL z-code\"><code class=\"language-COBOL\" data-lang=\"COBOL\"><span class=\"z-source z-cobol\"><span class=\"z-keyword z-identifiers z-cobol\">IDENTIFICATION DIVISION<\/span>.\n<\/span><span class=\"z-source z-cobol\"><span class=\"z-keyword z-verb z-cobol\">PROGRAM-ID<\/span>. <span class=\"z-meta z-symbol z-cobol\">FILES<\/span>.\n<\/span><span class=\"z-source z-cobol\"><span class=\"z-keyword z-identifiers z-cobol\">ENVIRONMENT DIVISION<\/span>.\n<\/span><span class=\"z-source z-cobol\">  <span class=\"z-keyword z-identifiers z-cobol\">INPUT-OUTPUT SECTION<\/span>.\n<\/span><span class=\"z-source z-cobol\">\t<span class=\"z-keyword z-identifiers z-cobol\">FILE-CONTROL<\/span>.\n<\/span><span class=\"z-source z-cobol\">\t  <span class=\"z-keyword z-identifiers z-cobol\">SELECT<\/span> <span class=\"z-meta z-symbol z-cobol\">TRANSACTIONS<\/span> <span class=\"z-keyword z-identifers z-cobol\">ASSIGN<\/span> <span class=\"z-keyword z-identifers z-cobol\">TO<\/span> <span class=\"z-string z-quoted z-single z-cobol\"><span class=\"z-punctuation z-definition z-string z-begin z-cobol\">&#39;<\/span>transactions.txt<span class=\"z-punctuation z-definition z-string z-end z-cobol\">&#39;<\/span><\/span>\n<\/span><span class=\"z-source z-cobol\">\t  <span class=\"z-keyword z-identifers z-cobol\">ORGANIZATION<\/span> <span class=\"z-keyword z-identifers z-cobol\">IS<\/span> <span class=\"z-keyword z-identifers z-cobol\">SEQUENTIAL<\/span>.\n<\/span><span class=\"z-source z-cobol\"><span class=\"z-keyword z-identifiers z-cobol\">DATA DIVISION<\/span>.\n<\/span><span class=\"z-source z-cobol\">  <span class=\"z-keyword z-identifiers z-cobol\">FILE SECTION<\/span>.\n<\/span><span class=\"z-source z-cobol\">\t<span class=\"z-keyword z-identifiers z-cobol\">FD<\/span> <span class=\"z-meta z-symbol z-cobol\">TRANSACTIONS<\/span>.\n<\/span><span class=\"z-source z-cobol\">\t<span class=\"z-constant z-numeric z-cobol\">01<\/span> <span class=\"z-meta z-symbol z-cobol\">TRANSACTION-STRUCT<\/span>.\n<\/span><span class=\"z-source z-cobol\">\t  <span class=\"z-constant z-numeric z-cobol\">02<\/span> <span class=\"z-meta z-symbol z-cobol\">UID<\/span> <span class=\"z-storage z-type z-picture z-cobol\">PIC 9(5).<\/span>\n<\/span><span class=\"z-source z-cobol\">\t  <span class=\"z-constant z-numeric z-cobol\">02<\/span> <span class=\"z-meta z-symbol z-cobol\">DESC<\/span> <span class=\"z-storage z-type z-picture z-cobol\">PIC X(25).<\/span>\n<\/span><span class=\"z-source z-cobol\"><span class=\"z-keyword z-identifiers z-cobol\">WORKING-STORAGE SECTION<\/span>.\n<\/span><span class=\"z-source z-cobol\">\t<span class=\"z-constant z-numeric z-cobol\">01<\/span> <span class=\"z-meta z-symbol z-cobol\">TRANSACTION-RECORD<\/span>.\n<\/span><span class=\"z-source z-cobol\">\t  <span class=\"z-constant z-numeric z-cobol\">02<\/span> <span class=\"z-meta z-symbol z-cobol\">UID<\/span> <span class=\"z-storage z-type z-picture z-cobol\">PIC 9(5)<\/span> <span class=\"z-keyword z-identifers z-cobol\">VALUE<\/span> <span class=\"z-constant z-numeric z-cobol\">12345<\/span>.\n<\/span><span class=\"z-source z-cobol\">\t  <span class=\"z-constant z-numeric z-cobol\">02<\/span> <span class=\"z-meta z-symbol z-cobol\">DESC<\/span> <span class=\"z-storage z-type z-picture z-cobol\">PIC X(25)<\/span> <span class=\"z-keyword z-identifers z-cobol\">VALUE<\/span> <span class=\"z-string z-quoted z-single z-cobol\"><span class=\"z-punctuation z-definition z-string z-begin z-cobol\">&#39;<\/span>TEST TRANSACTION<span class=\"z-punctuation z-definition z-string z-end z-cobol\">&#39;<\/span><\/span>.\n<\/span><span class=\"z-source z-cobol\"><span class=\"z-keyword z-identifiers z-cobol\">PROCEDURE DIVISION<\/span>.\n<\/span><span class=\"z-source z-cobol\">  <span class=\"z-keyword z-verbs z-cobol\">OPEN<\/span> <span class=\"z-keyword z-identifers z-cobol\">OUTPUT<\/span> <span class=\"z-meta z-symbol z-cobol\">TRANSACTIONS<\/span>\n<\/span><span class=\"z-source z-cobol\">\t<span class=\"z-keyword z-verbs z-cobol\">WRITE<\/span> <span class=\"z-meta z-symbol z-cobol\">TRANSACTION-STRUCT<\/span> <span class=\"z-keyword z-identifers z-cobol\">FROM<\/span> <span class=\"z-meta z-symbol z-cobol\">TRANSACTION-RECORD<\/span>\n<\/span><span class=\"z-source z-cobol\">  <span class=\"z-keyword z-verbs z-cobol\">CLOSE<\/span> <span class=\"z-meta z-symbol z-cobol\">TRANSACTIONS<\/span>\n<\/span><span class=\"z-source z-cobol\">  <span class=\"z-keyword z-identifiers z-cobol\">STOP RUN<\/span>.\n<\/span><\/code><\/pre>\n<p>note how in all of these, the syntax for persisting the data to disk is essentially the same as persisting it to memory (in MUMPS, persisting to memory is exactly the same, except you would write <code>SET table<\/code> instead of <code>SET ^table<\/code>).<\/p>\n<p>if you don't require the runtime to support all datatypes, there are frameworks for doing this as a library. <a href=\"https:\/\/protobuf.dev\/\">protobuf<\/a> and <a href=\"https:\/\/flatbuffers.dev\/\">flatbuffer<\/a> both autogenerate the code for a restricted set of data types, so that you only have to write the code invoking it.<\/p>\n<h2 id=\"orthogonal-persistence\">orthogonal persistence<a class=\"zola-anchor\" href=\"#orthogonal-persistence\" aria-label=\"Anchor link for: orthogonal-persistence\"><\/a>\n<\/h2>\n<p>the thing these languages and frameworks have in common is that the persistence is part of the program source code; either you have to choose a language that already has this support, or you have to do extensive modifications to the source code to persist the data at the right points. i will call this kind of persistence <em>complected persistence<\/em> because they tie together the business logic and persistence logic (see <a href=\"https:\/\/www.infoq.com\/presentations\/Simple-Made-Easy\/\">Simple Made Easy<\/a> by Rich Hickey for the history of the word \"complect\").<\/p>\n<p>there is also <em>orthogonal persistence<\/em>. <a href=\"https:\/\/en.wikipedia.org\/wiki\/Persistence_(computer_science)#Orthogonal_or_transparent_persistence\">\"orthogonal persistence\"<\/a> means your program's state is saved automatically without special work from you the programmer. in particular, the serialization is managed by the OS, database, or language runtime. as a result, you don't have to care about persistence, only about implementing business logic; the two concerns are <a href=\"https:\/\/en.wikipedia.org\/wiki\/Orthogonality#Computer_science\">orthogonal<\/a>.<\/p>\n<p>orthogonal persistence is more common than you might think. some examples:<\/p>\n<ul>\n<li><a href=\"https:\/\/en.wikipedia.org\/wiki\/Hibernation_(computing)\">hibernation<\/a> (suspend to disk). first invented in 1992 for the Compaq <a href=\"https:\/\/en.wikipedia.org\/wiki\/Compaq_LTE#LTE_Lite\">LTE Lite<\/a>. Windows has this on by default since Windows 8 (2012). MacOS has had it on by default since OS X 10.4 (2005).<\/li>\n<li>virtualized hibernation in hypervisors like VirtualBox and VMWare (usually labeled \"Save the machine state\" or something similar)<\/li>\n<\/ul>\n<h2 id=\"how-far-can-we-take-this\">how far can we take this?<a class=\"zola-anchor\" href=\"#how-far-can-we-take-this\" aria-label=\"Anchor link for: how-far-can-we-take-this\"><\/a>\n<\/h2>\n<p>these forms of orthogonal persistence work on the whole OS state. you could imagine a version that works on individual processes: swap the process to disk, restore it later. the kernel kinda already does this when it does scheduling. you can replicate it in userspace with <a href=\"https:\/\/thume.ca\/2020\/04\/18\/telefork-forking-a-process-onto-a-different-computer\/\">telefork<\/a>, which even lets you spawn a process onto another machine.<\/p>\n<p>but the rest of the OS moves on while the process is suspended: the files it accesses may have changed, the processes it was talking to over a socket may have exited. what we want is to snapshot the process state: whatever files on disk stay on disk, whatever processes it was talking to continue running. this allows you to rewind and replay the process, as if the whole thing were running in a database transaction.<\/p>\n<p>what do we need in order to do that?<\/p>\n<ul>\n<li>a filesystem that supports atomic accesses, snapshots, and transaction restarts, such as <a href=\"https:\/\/en.wikipedia.org\/wiki\/ZFS#Snapshots_and_clones\">ZFS<\/a>.<\/li>\n<li>a runtime that supports detailed tracking and replay of syscalls, such as <a href=\"https:\/\/rr-project.org\/\">rr<\/a>. this works by intercepting syscalls with <a href=\"https:\/\/man7.org\/linux\/man-pages\/man2\/ptrace.2.html\">ptrace()<\/a>, among other mechanisms, and does not require any modifications to the program executable or source code.<\/li>\n<li>a sandbox that prevents talking to processes that weren\u2019t running when the target process was spawned (unless those processes are also in the sandbox and tracked with this mechanism), such as <a href=\"https:\/\/github.com\/containers\/bubblewrap\">bubblewrap<\/a>.<\/li>\n<\/ul>\n<p>effectively, we are turning syscalls into <a href=\"http:\/\/habitatchronicles.com\/2017\/05\/what-are-capabilities\/\">capabilities<\/a>, where the capabilities we give out are \u201cexactly the syscalls the process made last time it spawned\u201d.<\/p>\n<p>note how this is possible to do today, with existing technology and kernel APIs! this doesn\u2019t require building an OS from scratch, nor rewriting all code to be in a language with tracked effects or a capability system. instead, by working at the syscall interface<sup class=\"footnote-reference\" id=\"fr-6-1\"><a href=\"#fn-6\">4<\/a><\/sup> between the program and the kernel, we can build a highly general system that applies to all the programs you already use.<\/p>\n<h2 id=\"but-why\">\u201cbut why?\u201d<a class=\"zola-anchor\" href=\"#but-why\" aria-label=\"Anchor link for: but-why\"><\/a>\n<\/h2>\n<p>note that this involves 3 different levels of tracking, which we can think of in terms of progressive enhancement:<\/p>\n<ol>\n<li>features you can get just by recording and replaying the whole process (\"tracking between processes\")<\/li>\n<li>features you can get by replaying from a specific point in the process (\"tracking within a process\")<\/li>\n<li>features you can only get with source code changes, by allowing the process to choose where it should be restored to (\"tracking that needs source code changes\")<\/li>\n<\/ol>\n<p>the editor example i gave at the beginning refers to 3; but you can get really quite a lot of things just with 1 (tracked record\/replay and transactional semantics). for example, here are some tools that would be easy to build on top:<\/p>\n<h3 id=\"needs-tracking-between-processes\">needs tracking between processes<a class=\"zola-anchor\" href=\"#needs-tracking-between-processes\" aria-label=\"Anchor link for: needs-tracking-between-processes\"><\/a>\n<\/h3>\n<ul>\n<li>collaborative terminals, where you can \u201csplit\u201d your terminal and hand a sandboxed environment of <em>your personal computer<\/em> to a colleague so they can help you debug an issue. this is more general than OCI containers because you don't need to spend time creating a dockerfile that reproduces the problem. this is more general than <a href=\"https:\/\/robert.ocallahan.org\/2017\/09\/rr-trace-portability.html\"><code>rr pack<\/code><\/a> because you can edit the program source to add printfs, or change the input you pass to it at runtime.<\/li>\n<li>\u201csave\/undo for your terminal\u201d, where you don\u2019t need to add a <a href=\"https:\/\/www.gnu.org\/software\/coreutils\/manual\/html_node\/Treating-_002f-specially.html\"><code>--no-preserve-root<\/code><\/a> flag to <code>rm<\/code>, because the underlying filesystem can just restore a snapshot. this generalizes to any command\u2014for example, you can build an arbitrary <code>git undo<\/code> command that works even if installed after the data is lost, which is <a href=\"https:\/\/blog.waleedkhan.name\/git-undo\/\">not possible today<\/a>. note that this can undo by-process, not just by point-in-time, so it is strictly more general than FS snapshots.<\/li>\n<li>query which files on disk were modified the last time you ran a command. for example you could ask \u201cwhere did this <code>curl | sh<\/code> command install its files?\u201d. the closest we have to this today is <a href=\"https:\/\/man7.org\/linux\/man-pages\/man1\/dpkg.1.html#:~:text=listfiles\"><code>dpkg --listfiles<\/code><\/a>, which only works for changes done by the package manager.<\/li>\n<\/ul>\n<h3 id=\"needs-tracking-within-a-process\">needs tracking within a process<a class=\"zola-anchor\" href=\"#needs-tracking-within-a-process\" aria-label=\"Anchor link for: needs-tracking-within-a-process\"><\/a>\n<\/h3>\n<ul>\n<li><a href=\"https:\/\/asciinema.org\/\">asciinema<\/a>, but you actually run the process instead of building your own terminal emulator. this also lets you edit the recording live instead of having to re-record from scratch.<\/li>\n<li>the <a href=\"https:\/\/jade.fyi\/blog\/the-postmodern-build-system\/#limits-of-execve-memoization\">\u201cpost-modern build system\u201d<\/a> (also needs a <a href=\"https:\/\/salsa-rs.netlify.app\/\">salsa<\/a>-like <a href=\"https:\/\/rustc-dev-guide.rust-lang.org\/queries\/incremental-compilation-in-detail.html#improving-accuracy-the-red-green-algorithm\">red-green system<\/a>)<\/li>\n<\/ul>\n<h3 id=\"needs-source-code-changes\">needs source code changes<a class=\"zola-anchor\" href=\"#needs-source-code-changes\" aria-label=\"Anchor link for: needs-source-code-changes\"><\/a>\n<\/h3>\n<ul>\n<li>\u201csave\/undo for your program\u201d, where editors and games can take advantage of cheap snapshots to use the operating system's restore mechanism instead of building their own.<\/li>\n<\/ul>\n<p>this is not an exhaustive list, just the first things on the top of my head after a couple days of thinking about it. what makes this orthogonal persistence system so useful is that all these tools are near-trivial to build on top: most of them could be done in shell scripts or a short python script, instead of needing a team of developers and a year.<\/p>\n<h2 id=\"isn-t-this-horribly-slow\">isn\u2019t this horribly slow?<a class=\"zola-anchor\" href=\"#isn-t-this-horribly-slow\" aria-label=\"Anchor link for: isn-t-this-horribly-slow\"><\/a>\n<\/h2>\n<p>not inherently. \u201cturning your file system into a database\u201c is only as slow as submitting the query is<sup class=\"footnote-reference\" id=\"fr-1-1\"><a href=\"#fn-1\">5<\/a><\/sup>\u2014and that can be quite fast when the database runs locally instead of over the network; see <a href=\"https:\/\/github.com\/spacejam\/sled?tab=readme-ov-file#performance\">sled<\/a> for an example of such a DB that has had performance tuning. rr boasts <a href=\"https:\/\/rr-project.org\/#:~:text=slowdown\">less than a 20% slowdown<\/a>. bubblewrap uses the kernel\u2019s native namespacing and to my knowledge imposes no overhead.<\/p>\n<p>now, the final system needs to be designed with performance in mind and then carefully optimized but, you know. that's doable.<\/p>\n<h2 id=\"does-this-work-across-versions-of-my-program\">does this work across versions of my program?<a class=\"zola-anchor\" href=\"#does-this-work-across-versions-of-my-program\" aria-label=\"Anchor link for: does-this-work-across-versions-of-my-program\"><\/a>\n<\/h2>\n<p>kinda. there are two possible ways to implement intra-process persistence (i.e. \"everything other than disk writes\").<\/p>\n<ol>\n<li>take a snapshot of the memory, registers, and kernel state (e.g. file descriptors). this is how <a href=\"https:\/\/thume.ca\/2020\/04\/18\/telefork-forking-a-process-onto-a-different-computer\/\">telefork<\/a> works. this only works with a single version of the executable; any change, even LTO without changing source code, will break it.<\/li>\n<li>replay all syscalls done by the process. this will work across versions, as long as the program makes the same syscalls in the same order.<\/li>\n<\/ol>\n<p>1 is cheap if you don't have much memory usage compared to CPU time.\n2 is cheap if you don't have much CPU time compared to memory usage.\nonly 2 allows you to modify the binary between saving and restoring.\nit's possible to do both for the same process, just expensive.<\/p>\n<h2 id=\"summary\">summary<a class=\"zola-anchor\" href=\"#summary\" aria-label=\"Anchor link for: summary\"><\/a>\n<\/h2>\n<ul>\n<li>persisting program state is hard and basically requires implementing a database<\/li>\n<li>persistence that does not require action from the program is called \u201corthogonal persistence\u201d<\/li>\n<li>it is possible to build orthogonal persistence for individual processes with tools that exist today, with only moderate slowdowns depending on how granular you want to be<\/li>\n<li>there are multiple possible ways to implement this system, with different perf\/generality tradeoffs<\/li>\n<li>such a system unlocks many kinds of tools by making them orders of magnitude easier to build<\/li>\n<\/ul>\n<p>many of the ideas in this post were developed in collaboration with edef. if you want to see them built, consider <a href=\"https:\/\/github.com\/sponsors\/edef1c\">sponsoring her<\/a> so she has time to work on them.<\/p>\n<hr \/>\n<h2 id=\"bibliography\">bibliography<a class=\"zola-anchor\" href=\"#bibliography\" aria-label=\"Anchor link for: bibliography\"><\/a>\n<\/h2>\n<ul>\n<li><a href=\"https:\/\/danluu.com\/file-consistency\/\">Dan Luu, \"Files are hard\"<\/a><\/li>\n<li><a href=\"https:\/\/docs.rs\/bincode\/latest\/bincode\/\">Ty Overby, Zoey Riordan, Victor Koenders, <em>bincode<\/em><\/a><\/li>\n<li><a href=\"https:\/\/archive.cs.st-andrews.ac.uk\/papers\/download\/ABC+83b.pdf\">Atkinson, M.P., Bailey, P.J., Chisholm, K.J., Cockshott, W.P. &amp; Morrison, R. \u201cPS-algol: A\nLanguage for Persistent Programming\u201d. In Proc. 10th Australian National Computer\nConference, Melbourne, Australia (1983) pp 70-79.<\/a><\/li>\n<li><a href=\"https:\/\/calpaterson.com\/bank-python.html\">Cal Paterson, \"An oral history of Bank Python\"<\/a><\/li>\n<li><a href=\"https:\/\/www.devever.net\/~hl\/f\/as400guide.pdf\">Hugo Landau, \"IBM i: An Unofficial Introduction\"<\/a><\/li>\n<li><a href=\"https:\/\/protobuf.dev\/\">Google LLC, \"Protocol Buffers\"<\/a><\/li>\n<li><a href=\"https:\/\/flatbuffers.dev\/\">Google LLC, \"FlatBuffers Docs\"<\/a><\/li>\n<li><a href=\"https:\/\/www.infoq.com\/presentations\/Simple-Made-Easy\/\">Rich Hickey, \"Simple Made Easy\"<\/a><\/li>\n<li><a href=\"https:\/\/thume.ca\/2020\/04\/18\/telefork-forking-a-process-onto-a-different-computer\/\">Tristan Hume, \"Teleforking a process onto a different computer!\"<\/a><\/li>\n<li><a href=\"https:\/\/rr-project.org\/\">Robert O\u2019Callahan et al., \"RR\"<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/containers\/bubblewrap\">Simon McVittie et al., \"bubblewrap\"<\/a><\/li>\n<li><a href=\"http:\/\/habitatchronicles.com\/2017\/05\/what-are-capabilities\/\">Chip Morningstar and F. Randall Farmer, \"What Are Capabilities?\"<\/a><\/li>\n<li><a href=\"https:\/\/robert.ocallahan.org\/2017\/09\/rr-trace-portability.html\">Robert O'Callahan, \"rr Trace Portability\"<\/a><\/li>\n<li><a href=\"https:\/\/www.gnu.org\/software\/coreutils\/manual\/html_node\/Treating-_002f-specially.html\">Free Software Foundation, Inc., \"GNU Coreutils\"<\/a><\/li>\n<li><a href=\"https:\/\/blog.waleedkhan.name\/git-undo\/\">Waleed Khan, \"git undo: We can do better\"<\/a><\/li>\n<li><a href=\"https:\/\/asciinema.org\/\">Marcin Kulik, \"Record and share your terminal sessions, the simple way.\"<\/a><\/li>\n<li><a href=\"https:\/\/jade.fyi\/blog\/the-postmodern-build-system\/#limits-of-execve-memoization\">Jade Lovelace, \"The postmodern build system\"<\/a><\/li>\n<li><a href=\"https:\/\/salsa-rs.netlify.app\/\">Salsa developrs, \"About salsa\"<\/a><\/li>\n<li><a href=\"https:\/\/rustc-dev-guide.rust-lang.org\/queries\/incremental-compilation-in-detail.html#improving-accuracy-the-red-green-algorithm\">The Rust Project contributors, \"Incremental compilation in detail\"<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/spacejam\/sled?tab=readme-ov-file#performance\">Tyler Neely, \"sled - it's all downhill from here!!!\"<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/sponsors\/edef1c\">edef<\/a><\/li>\n<li><a href=\"https:\/\/www.postgresql.org\/docs\/current\/wal-intro.html#WAL-INTRO\">The PostgreSQL Global Development Group, \"Reliability and the Write-Ahead Log: Write-Ahead Logging (WAL)\"<\/a><\/li>\n<li><a href=\"https:\/\/medium.com\/@yvanscher\/7-cobol-examples-with-explanations-ae1784b4d576\">Yvan Scher, \"7 cobol examples with explanations.\"<\/a><\/li>\n<li><a href=\"https:\/\/www.cl.cam.ac.uk\/research\/security\/ctsrd\/\">Robert N. M. Watson et al., \"CTSRD \u2013 Rethinking the hardware-software interface for security\"<\/a><\/li>\n<li><a href=\"https:\/\/webassembly.org\/\">WebAssembly Working Group, \"WebAssembly\"<\/a><\/li>\n<li><a href=\"https:\/\/doc.cat-v.org\/bell_labs\/utah2000\/utah2000.html\">Rob Pike, \"Systems Software Research is Irrelevant\"<\/a><\/li>\n<li><a href=\"https:\/\/man7.org\/linux\/man-pages\/man2\/ptrace.2.html\">System Calls Manual, \"ptrace(2)\"<\/a><\/li>\n<li><a href=\"https:\/\/man7.org\/linux\/man-pages\/man1\/dpkg.1.html#:~:text=listfiles\">dpkg suite, \"dpkg(1)\"<\/a><\/li>\n<li><a href=\"https:\/\/en.wikipedia.org\/wiki\/MUMPS\">Wikipedia, \"MUMPS\"<\/a><\/li>\n<li><a href=\"https:\/\/en.wikipedia.org\/wiki\/Persistence_(computer_science)#Orthogonal_or_transparent_persistence\">Wikipedia, \"orthogonal persistence\"<\/a><\/li>\n<li><a href=\"https:\/\/en.wikipedia.org\/wiki\/IBM_i#Technology_Independent_Machine_Interface_(TIMI)\">Wikipedia, \"IBM i\"<\/a><\/li>\n<li><a href=\"https:\/\/en.wikipedia.org\/wiki\/ZFS#Snapshots_and_clones\">Wikipedia, \"ZFS\"<\/a><\/li>\n<li><a href=\"https:\/\/en.wikipedia.org\/wiki\/Orthogonality#Computer_science\">Wikipedia, \"Orthogonality\"<\/a><\/li>\n<li><a href=\"https:\/\/en.wikipedia.org\/wiki\/Hibernation_(computing)\">Wikipedia, \"Hibernation (computing)\"<\/a><\/li>\n<li><a href=\"https:\/\/en.wikipedia.org\/wiki\/Compaq_LTE#LTE_Lite\">Wikipedia, \"Compaq LTE Lite\"<\/a><\/li>\n<\/ul>\n<section class=\"footnotes\">\n<ol class=\"footnotes-list\">\n<li id=\"fn-4\">\n<p>the IBM i will be coming up many times in this series, particularly block terminals and the object capability model. i will glaze over parts of the system that cannot be intercepted at a syscall boundary; but that said i want to point out that <a href=\"https:\/\/www.devever.net\/~hl\/f\/as400guide.pdf#page=13\">IBM POWER extensions<\/a> and <a href=\"https:\/\/en.wikipedia.org\/wiki\/IBM_i#Technology_Independent_Machine_Interface_(TIMI)\">TIMI<\/a> correspond roughly to the modern ideas of the <a href=\"https:\/\/www.cl.cam.ac.uk\/research\/security\/ctsrd\/\">CHERI<\/a> and <a href=\"https:\/\/webassembly.org\/\">WASM<\/a> projects. <a href=\"#fr-4-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-5\">\n<p>jyn, you might ask, why are all these systems so old! why legacy systems? isn't there any new code with these ideas? and the answer is no. <a href=\"https:\/\/doc.cat-v.org\/bell_labs\/utah2000\/utah2000.html\">systems research is irrelevant<\/a> and its new ideas, such as they are, do not make it into industry. <a href=\"#fr-5-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-3\">\n<p>credit <a href=\"https:\/\/medium.com\/@yvanscher\/7-cobol-examples-with-explanations-ae1784b4d576\">@yvanscher<\/a> <a href=\"#fr-3-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-6\">\n<p>on all OSes i know other than Linux, the syscall ABI is not stable and programs are expected to use libc bindings. you can do something similar by using LD_PRELOAD to intercept libc calls. <a href=\"#fr-6-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-1\">\n<p>one might think that you have to flush each write to disk before returning from write() in order to preserve ACID semantics. not so; this is exactly the problem that a <a href=\"https:\/\/www.postgresql.org\/docs\/current\/wal-intro.html#WAL-INTRO\">write-ahead-log<\/a> fixes. <a href=\"#fr-1-1\">\u21a9<\/a><\/p>\n<\/li>\n<\/ol>\n<\/section>\n"},{"title":"how i write blog posts","published":"2025-06-18T00:00:00+00:00","updated":"2025-06-18T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/how-i-write-blog-posts\/"}},"id":"https:\/\/jyn.dev\/how-i-write-blog-posts\/","summary":"the trick is to make it as easy as possible","content":"<p>this isn\u2019t about about blogging engines, don\u2019t worry. there\u2019s already plenty of writing about those. i use zola, i am mildly dissatisfied with it, i don\u2019t care enough to switch.<\/p>\n<p>no, this is how i actually write. i have an um. <em>eccentric<\/em> setup. in particular, i can write draft posts from any of my devices\u2014desktop, laptop, phone\u2014and have them show up live on a hidden subdomain of jyn.dev without any special work on my part.<\/p>\n<p>how does this work? i\u2019m glad you asked.<\/p>\n<ul>\n<li>on my desktop, i have Caddy running a reverse proxy back to a live zola server. Caddy gives me nice things like https, and makes me less worried about having public ports on the internet.\n<ul>\n<li>to get live-reloading working, Caddy also reverse-proxies websockets.<\/li>\n<\/ul>\n<\/li>\n<li>on my desktop, i have the zola content\/ directory sym-linked to a subdirectory of my obsidian notes folder.<\/li>\n<li>on all my devices, i run Obsidian Sync in the background, which automatically syncs my posts everywhere. it costs $5\/month and doesn\u2019t cause me trouble, which is a lot more than i can say for most technology.<\/li>\n<li>on laptop and mobile, i just write in obsidian, like i would for any other notes. i have a \"blog post\" template that inserts the zola header; otherwise i just write normal markdown.<\/li>\n<li>when i\u2019m ready to publish, i commit the changes to git on desktop or laptop and push to github, which updates the public facing blog.<\/li>\n<\/ul>\n<p>works great!<\/p>\n<p>normally i write outlines and ideas down on mobile, and then clean them up into prose on desktop. when i edit on desktop, i sometimes use nvim (e.g. for posts like <a href=\"https:\/\/jyn.dev\/how-i-use-my-terminal\/\">how i use my terminal<\/a> that have complicated html fragments). unlike obsidian, nvim doesn't have autosave, so i <a href=\"https:\/\/github.com\/jyn514\/dotfiles\/blob\/45f6702de2608a972615cd877993a41521f76348\/config\/nvim.lua#L242-L268\">added it myself<\/a>:<\/p>\n<pre data-lang=\"lua\" class=\"language-lua z-code\"><code class=\"language-lua\" data-lang=\"lua\"><span class=\"z-source z-lua\"><span class=\"z-comment z-line z-lua\"><span class=\"z-punctuation z-definition z-comment z-lua\">--<\/span> autosave on cursor hold\n<\/span><\/span><span class=\"z-source z-lua\"><span class=\"z-storage z-modifier z-lua\">local<\/span> <span class=\"z-variable z-other z-lua\">timers<\/span> <span class=\"z-keyword z-operator z-assignment z-lua\">=<\/span> <span class=\"z-meta z-mapping z-lua\"><span class=\"z-punctuation z-section z-block z-begin z-lua\">{<\/span><span class=\"z-punctuation z-section z-block z-end z-lua\">}<\/span><\/span>\n<\/span><span class=\"z-source z-lua\"><span class=\"z-meta z-function z-lua\"><span class=\"z-meta z-block z-lua\"><span class=\"z-storage z-type z-function z-lua\">function<\/span> <span class=\"z-meta z-name z-function\"><span class=\"z-entity z-name z-function z-lua\">autosave_enable<\/span><\/span><span class=\"z-meta z-group z-lua\"><span class=\"z-punctuation z-section z-group z-begin z-lua\">(<\/span><span class=\"z-punctuation z-section z-group z-end z-lua\">)<\/span><\/span>\n<\/span><\/span><\/span><span class=\"z-source z-lua\"><span class=\"z-meta z-function z-lua\"><span class=\"z-meta z-block z-lua\">  <span class=\"z-storage z-modifier z-lua\">local<\/span> <span class=\"z-variable z-other z-lua\">buf<\/span> <span class=\"z-keyword z-operator z-assignment z-lua\">=<\/span> <span class=\"z-variable z-other z-lua\">vim<\/span><span class=\"z-punctuation z-accessor z-lua\">.<\/span><span class=\"z-meta z-property z-lua\">api<\/span><span class=\"z-punctuation z-accessor z-lua\">.<\/span><span class=\"z-meta z-property z-lua\"><span class=\"z-variable z-function z-lua\">nvim_get_current_buf<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-lua\"><span class=\"z-meta z-group z-lua\"><span class=\"z-punctuation z-section z-group z-begin z-lua\">(<\/span><span class=\"z-punctuation z-section z-group z-end z-lua\">)<\/span><\/span><\/span>\n<\/span><\/span><\/span><span class=\"z-source z-lua\"><span class=\"z-meta z-function z-lua\"><span class=\"z-meta z-block z-lua\">  <span class=\"z-keyword z-control z-conditional z-lua\">if<\/span> <span class=\"z-variable z-other z-lua\">timers<\/span><span class=\"z-meta z-brackets z-lua\"><span class=\"z-punctuation z-section z-brackets z-begin z-lua\">[<\/span><span class=\"z-variable z-other z-lua\">buf<\/span><span class=\"z-punctuation z-section z-brackets z-end z-lua\">]<\/span><\/span> <span class=\"z-meta z-block z-lua\"><span class=\"z-keyword z-control z-conditional z-lua\">then<\/span> <span class=\"z-keyword z-control z-return z-lua\">return<\/span> <span class=\"z-keyword z-control z-end z-lua\">end<\/span><\/span>\n<\/span><\/span><\/span><span class=\"z-source z-lua\"><span class=\"z-meta z-function z-lua\"><span class=\"z-meta z-block z-lua\">\n<\/span><\/span><\/span><span class=\"z-source z-lua\"><span class=\"z-meta z-function z-lua\"><span class=\"z-meta z-block z-lua\">  <span class=\"z-storage z-modifier z-lua\">local<\/span> <span class=\"z-variable z-other z-lua\">buf_name<\/span> <span class=\"z-keyword z-operator z-assignment z-lua\">=<\/span> <span class=\"z-variable z-other z-lua\">vim<\/span><span class=\"z-punctuation z-accessor z-lua\">.<\/span><span class=\"z-meta z-property z-lua\">fn<\/span><span class=\"z-punctuation z-accessor z-lua\">.<\/span><span class=\"z-meta z-property z-lua\"><span class=\"z-variable z-function z-lua\">expand<\/span><\/span> <span class=\"z-meta z-function-call z-arguments z-lua\"><span class=\"z-string z-quoted z-single z-lua\"><span class=\"z-punctuation z-definition z-string z-begin z-lua\">&#39;<\/span>%<span class=\"z-punctuation z-definition z-string z-end z-lua\">&#39;<\/span><\/span><\/span>\n<\/span><\/span><\/span><span class=\"z-source z-lua\"><span class=\"z-meta z-function z-lua\"><span class=\"z-meta z-block z-lua\">  <span class=\"z-variable z-other z-lua\">vim<\/span><span class=\"z-punctuation z-accessor z-lua\">.<\/span><span class=\"z-meta z-property z-lua\"><span class=\"z-variable z-function z-lua\">notify<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-lua\"><span class=\"z-meta z-group z-lua\"><span class=\"z-punctuation z-section z-group z-begin z-lua\">(<\/span><span class=\"z-string z-quoted z-double z-lua\"><span class=\"z-punctuation z-definition z-string z-begin z-lua\">&quot;<\/span>autosaving <span class=\"z-punctuation z-definition z-string z-end z-lua\">&quot;<\/span><\/span><span class=\"z-keyword z-operator z-concatenation z-lua\">..<\/span><span class=\"z-variable z-other z-lua\">buf_name<\/span><span class=\"z-punctuation z-section z-group z-end z-lua\">)<\/span><\/span><\/span>\n<\/span><\/span><\/span><span class=\"z-source z-lua\"><span class=\"z-meta z-function z-lua\"><span class=\"z-meta z-block z-lua\">  <span class=\"z-variable z-other z-lua\">timers<\/span><span class=\"z-meta z-brackets z-lua\"><span class=\"z-punctuation z-section z-brackets z-begin z-lua\">[<\/span><span class=\"z-variable z-other z-lua\">buf<\/span><span class=\"z-punctuation z-section z-brackets z-end z-lua\">]<\/span><\/span> <span class=\"z-keyword z-operator z-assignment z-lua\">=<\/span> <span class=\"z-variable z-other z-lua\">vim<\/span><span class=\"z-punctuation z-accessor z-lua\">.<\/span><span class=\"z-meta z-property z-lua\">api<\/span><span class=\"z-punctuation z-accessor z-lua\">.<\/span><span class=\"z-meta z-property z-lua\"><span class=\"z-variable z-function z-lua\">nvim_create_autocmd<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-lua\"><span class=\"z-meta z-group z-lua\"><span class=\"z-punctuation z-section z-group z-begin z-lua\">(<\/span><span class=\"z-string z-quoted z-double z-lua\"><span class=\"z-punctuation z-definition z-string z-begin z-lua\">&quot;<\/span>CursorHold<span class=\"z-punctuation z-definition z-string z-end z-lua\">&quot;<\/span><\/span><span class=\"z-punctuation z-separator z-comma z-lua\">,<\/span> <span class=\"z-meta z-mapping z-lua\"><span class=\"z-punctuation z-section z-block z-begin z-lua\">{<\/span>\n<\/span><\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-lua\"><span class=\"z-meta z-function z-lua\"><span class=\"z-meta z-block z-lua\"><span class=\"z-meta z-function-call z-arguments z-lua\"><span class=\"z-meta z-group z-lua\"><span class=\"z-meta z-mapping z-lua\">    <\/span><span class=\"z-meta z-mapping z-key z-lua\"><span class=\"z-string z-unquoted z-key z-lua\">desc<\/span><\/span><span class=\"z-meta z-mapping z-lua\"> <\/span><span class=\"z-meta z-mapping z-lua\"><span class=\"z-punctuation z-separator z-key-value z-lua\">=<\/span><\/span><span class=\"z-meta z-mapping z-value z-lua\"> <span class=\"z-string z-quoted z-double z-lua\"><span class=\"z-punctuation z-definition z-string z-begin z-lua\">&quot;<\/span>Save <span class=\"z-punctuation z-definition z-string z-end z-lua\">&quot;<\/span><\/span><span class=\"z-keyword z-operator z-concatenation z-lua\">..<\/span><span class=\"z-variable z-other z-lua\">buf_name<\/span><span class=\"z-keyword z-operator z-concatenation z-lua\">..<\/span><span class=\"z-string z-quoted z-double z-lua\"><span class=\"z-punctuation z-definition z-string z-begin z-lua\">&quot;<\/span> on change<span class=\"z-punctuation z-definition z-string z-end z-lua\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-mapping z-lua\"><span class=\"z-punctuation z-separator z-field z-lua\">,<\/span>\n<\/span><\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-lua\"><span class=\"z-meta z-function z-lua\"><span class=\"z-meta z-block z-lua\"><span class=\"z-meta z-function-call z-arguments z-lua\"><span class=\"z-meta z-group z-lua\"><span class=\"z-meta z-mapping z-lua\">    <\/span><span class=\"z-meta z-mapping z-key z-lua\"><span class=\"z-entity z-name z-function z-lua\">callback<\/span><\/span><span class=\"z-meta z-mapping z-lua\"> <\/span><span class=\"z-meta z-mapping z-lua\"><span class=\"z-punctuation z-separator z-key-value z-lua\">=<\/span><\/span><span class=\"z-meta z-mapping z-value z-lua\"> <span class=\"z-meta z-function z-lua\"><span class=\"z-meta z-block z-lua\"><span class=\"z-storage z-type z-function z-lua\">function<\/span><span class=\"z-meta z-group z-lua\"><span class=\"z-punctuation z-section z-group z-begin z-lua\">(<\/span><span class=\"z-punctuation z-section z-group z-end z-lua\">)<\/span><\/span> <span class=\"z-variable z-other z-lua\">vim<\/span><span class=\"z-punctuation z-accessor z-lua\">.<\/span><span class=\"z-meta z-property z-lua\">api<\/span><span class=\"z-punctuation z-accessor z-lua\">.<\/span><span class=\"z-meta z-property z-lua\"><span class=\"z-variable z-function z-lua\">nvim_buf_call<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-lua\"><span class=\"z-meta z-group z-lua\"><span class=\"z-punctuation z-section z-group z-begin z-lua\">(<\/span><span class=\"z-variable z-other z-lua\">buf<\/span><span class=\"z-punctuation z-separator z-comma z-lua\">,<\/span>\n<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-lua\"><span class=\"z-meta z-function z-lua\"><span class=\"z-meta z-block z-lua\"><span class=\"z-meta z-function-call z-arguments z-lua\"><span class=\"z-meta z-group z-lua\"><span class=\"z-meta z-mapping z-value z-lua\"><span class=\"z-meta z-function z-lua\"><span class=\"z-meta z-block z-lua\"><span class=\"z-meta z-function-call z-arguments z-lua\"><span class=\"z-meta z-group z-lua\">      <span class=\"z-meta z-function z-lua\"><span class=\"z-meta z-block z-lua\"><span class=\"z-storage z-type z-function z-lua\">function<\/span><span class=\"z-meta z-group z-lua\"><span class=\"z-punctuation z-section z-group z-begin z-lua\">(<\/span><span class=\"z-punctuation z-section z-group z-end z-lua\">)<\/span><\/span> <span class=\"z-variable z-other z-lua\">vim<\/span><span class=\"z-punctuation z-accessor z-lua\">.<\/span><span class=\"z-meta z-property z-lua\"><span class=\"z-variable z-function z-lua\">cmd<\/span><\/span> <span class=\"z-meta z-function-call z-arguments z-lua\"><span class=\"z-string z-quoted z-double z-lua\"><span class=\"z-punctuation z-definition z-string z-begin z-lua\">&quot;<\/span>silent update<span class=\"z-punctuation z-definition z-string z-end z-lua\">&quot;<\/span><\/span><\/span> <span class=\"z-keyword z-control z-end z-lua\">end<\/span><\/span><\/span>\n<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-lua\"><span class=\"z-meta z-function z-lua\"><span class=\"z-meta z-block z-lua\"><span class=\"z-meta z-function-call z-arguments z-lua\"><span class=\"z-meta z-group z-lua\"><span class=\"z-meta z-mapping z-value z-lua\"><span class=\"z-meta z-function z-lua\"><span class=\"z-meta z-block z-lua\"><span class=\"z-meta z-function-call z-arguments z-lua\"><span class=\"z-meta z-group z-lua\">    <span class=\"z-punctuation z-section z-group z-end z-lua\">)<\/span><\/span><\/span> <span class=\"z-keyword z-control z-end z-lua\">end<\/span><\/span><\/span> <\/span><span class=\"z-meta z-mapping z-lua\"><span class=\"z-punctuation z-section z-block z-end z-lua\">}<\/span><\/span><span class=\"z-punctuation z-section z-group z-end z-lua\">)<\/span><\/span><\/span>\n<\/span><\/span><\/span><span class=\"z-source z-lua\"><span class=\"z-meta z-function z-lua\"><span class=\"z-meta z-block z-lua\"><span class=\"z-keyword z-control z-end z-lua\">end<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-lua\"><span class=\"z-variable z-other z-lua\">vim<\/span><span class=\"z-punctuation z-accessor z-lua\">.<\/span><span class=\"z-meta z-property z-lua\">api<\/span><span class=\"z-punctuation z-accessor z-lua\">.<\/span><span class=\"z-meta z-property z-lua\"><span class=\"z-variable z-function z-lua\">nvim_create_user_command<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-lua\"><span class=\"z-meta z-group z-lua\"><span class=\"z-punctuation z-section z-group z-begin z-lua\">(<\/span><span class=\"z-string z-quoted z-single z-lua\"><span class=\"z-punctuation z-definition z-string z-begin z-lua\">&#39;<\/span>AutoSave<span class=\"z-punctuation z-definition z-string z-end z-lua\">&#39;<\/span><\/span><span class=\"z-punctuation z-separator z-comma z-lua\">,<\/span> <span class=\"z-variable z-other z-lua\">autosave_enable<\/span><span class=\"z-punctuation z-separator z-comma z-lua\">,<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-lua\"><span class=\"z-meta z-function-call z-arguments z-lua\"><span class=\"z-meta z-group z-lua\">\t<span class=\"z-meta z-mapping z-lua\"><span class=\"z-punctuation z-section z-block z-begin z-lua\">{<\/span><\/span><span class=\"z-meta z-mapping z-key z-lua\"><span class=\"z-string z-unquoted z-key z-lua\">desc<\/span><\/span><span class=\"z-meta z-mapping z-lua\"> <\/span><span class=\"z-meta z-mapping z-lua\"><span class=\"z-punctuation z-separator z-key-value z-lua\">=<\/span><\/span><span class=\"z-meta z-mapping z-value z-lua\"> <span class=\"z-string z-quoted z-double z-lua\"><span class=\"z-punctuation z-definition z-string z-begin z-lua\">&quot;<\/span>Start saving each second on change<span class=\"z-punctuation z-definition z-string z-end z-lua\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-mapping z-lua\"><span class=\"z-punctuation z-section z-block z-end z-lua\">}<\/span><\/span><span class=\"z-punctuation z-section z-group z-end z-lua\">)<\/span><\/span><\/span>\n<\/span><\/code><\/pre>\n<p>the Caddyfile is also quite simple:<\/p>\n<pre data-lang=\"Caddyfile\" class=\"language-Caddyfile z-code\"><code class=\"language-Caddyfile\" data-lang=\"Caddyfile\"><span class=\"z-source z-Caddyfile\"><span class=\"z-variable z-other z-Caddyfile\">subdomain.jyn.dev<\/span><span class=\"z-punctuation z-definition z-bracket z-Caddyfile\"> {<\/span>\n<\/span><span class=\"z-source z-Caddyfile\">        <span class=\"z-comment z-line z-Caddyfile\"># start with `zola serve --drafts --base-url \/`<\/span>\n<\/span><span class=\"z-source z-Caddyfile\">        <span class=\"z-support z-function z-Caddyfile\">reverse_proxy<\/span> localhost:1111<span class=\"z-punctuation z-definition z-bracket z-Caddyfile\"> {<\/span>\n<\/span><span class=\"z-source z-Caddyfile\">                <span class=\"z-keyword z-other z-Caddyfile\">fail_duration<\/span> 10s\n<\/span><span class=\"z-source z-Caddyfile\"><span class=\"z-punctuation z-definition z-bracket z-Caddyfile\">        }<\/span>\n<\/span><span class=\"z-source z-Caddyfile\">        <span class=\"z-support z-function z-Caddyfile\">handle_errors<\/span> 502 503<span class=\"z-punctuation z-definition z-bracket z-Caddyfile\"> {<\/span>\n<\/span><span class=\"z-source z-Caddyfile\">                <span class=\"z-keyword z-other z-Caddyfile\">header<\/span> Content-Type text\/html\n<\/span><span class=\"z-source z-Caddyfile\">                <span class=\"z-keyword z-other z-Caddyfile\">respond<\/span> 503<span class=\"z-punctuation z-definition z-bracket z-Caddyfile\"> {<\/span>\n<\/span><span class=\"z-source z-Caddyfile\">                        <span class=\"z-keyword z-other z-Caddyfile\">body<\/span> <span class=\"z-string z-quoted z-double z-Caddyfile\">&quot;&lt;!doctype html&gt;&lt;html&gt;&lt;body&gt;subdomain.jyn.dev is down right n\n<\/span><\/span><span class=\"z-source z-Caddyfile\"><span class=\"z-string z-quoted z-double z-Caddyfile\">ow; DM jyn to bring it back up. or just visit &lt;a href=&#39;https:\/\/jyn.dev&#39;&gt;jyn.dev&lt;\/a&gt;.&lt;\/b\n<\/span><\/span><span class=\"z-source z-Caddyfile\"><span class=\"z-string z-quoted z-double z-Caddyfile\">ody&gt;&lt;\/html&gt;&quot;<\/span>\n<\/span><span class=\"z-source z-Caddyfile\"><span class=\"z-punctuation z-definition z-bracket z-Caddyfile\">                }<\/span>\n<\/span><span class=\"z-source z-Caddyfile\"><span class=\"z-punctuation z-definition z-bracket z-Caddyfile\">        }<\/span>\n<\/span><span class=\"z-source z-Caddyfile\"><span class=\"z-punctuation z-definition z-bracket z-Caddyfile\">}<\/span>\n<\/span><span class=\"z-source z-Caddyfile\"><span class=\"z-comment z-line z-Caddyfile\"># websocket proxy<\/span>\n<\/span><span class=\"z-source z-Caddyfile\"><span class=\"z-variable z-other z-Caddyfile\">subdomain.jyn.dev:1025<\/span><span class=\"z-punctuation z-definition z-bracket z-Caddyfile\"> {<\/span>\n<\/span><span class=\"z-source z-Caddyfile\">        <span class=\"z-support z-function z-Caddyfile\">reverse_proxy<\/span> localhost:1024 {}\n<\/span><span class=\"z-source z-Caddyfile\"><span class=\"z-punctuation z-definition z-bracket z-Caddyfile\">}<\/span>\n<\/span><\/code><\/pre>\n<p>the one downside of this is that i get very little visibility on mobile onto why things are not syncing to the desktop. to make up for this, my phone and desktop are on the same <a href=\"https:\/\/tailscale.com\/kb\/1151\/what-is-tailscale\">tailnet<\/a>, which allows me to ssh in remotely to check up on the zola server (i\u2019ve never had to check up on the Caddy server). i like <a href=\"https:\/\/termius.com\/\">Termius<\/a> for this.<\/p>\n<p>note some things about this setup:<\/p>\n<ul>\n<li>i have live reloading all the way through, regardless of the editor or device i am using to edit the post.<\/li>\n<li>because it's just a public url, it's very easy to share with my friends and ask for feedback on early posts, without making the posts visible to random people on hacker news.<\/li>\n<li>if i ever want to take down the site, i just kill the zola server. it defaults to off when i start my computer.<\/li>\n<\/ul>\n"},{"title":"how i use my terminal","published":"2025-06-16T00:00:00+00:00","updated":"2025-06-16T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/how-i-use-my-terminal\/"}},"id":"https:\/\/jyn.dev\/how-i-use-my-terminal\/","summary":"i have gone a little above and beyond trying to get all the features of VSCode","content":"<p>this is a whole blog post because it is \"outside the overton window\"; it usually takes at least a video before people even understand the thing i am trying to describe. so, here's the video:<\/p>\n<p><video controls><source src=\"\/assets\/terminal-recording.mp4\"><\/video><\/p>\n<p>the steps here that tend to surprise people are <button class=timestamp>0:11<\/button>\n, <button class=timestamp>0:21<\/button>\n, and <button class=timestamp>0:41<\/button>\n. when i say \"surprise\" i don't just mean that people are surprised that i've set this up, but they are surprised this is possible at all.<\/p>\n<p>here's what happens in that video:<\/p>\n<ol>\n<li><button class=timestamp>0:00<\/button>\n I start with Windows Terminal open on my laptop.<\/li>\n<li><button class=timestamp>0:02<\/button>\n I hit <kbd>ctrl + shift + 5<\/kbd>, which opens a new terminal tab which <code>ssh<\/code>'s to my home desktop and immediately launches tmux.<\/li>\n<li><button class=timestamp>0:03<\/button>\n tmux launches my default shell, <code>zsh<\/code>. zsh shows a prompt, while loading the full config asynchronously<\/li>\n<li><button class=timestamp>0:08<\/button>\n i use <code>zoxide<\/code> to fuzzy find a recent directory<\/li>\n<li><button class=timestamp>0:09<\/button>\n i start typing a ripgrep command. zsh autofills the command since i've typed it before and i accept it with <kbd>ctrl + f<\/kbd>.<\/li>\n<li><button class=timestamp>0:11<\/button>\n i hit <kbd>ctrl + k<\/kbd><kbd>f<\/kbd>, which tells tmux to search all output in the scrollback for filenames. the filenames are highlighted in blue.<\/li>\n<li><button class=timestamp>0:12<\/button>\n i hold <kbd>n<\/kbd> to navigate through the files. there are a lot of them, so it takes me a bit to find the one i'm looking for.<\/li>\n<li><button class=timestamp>0:21<\/button>\n i press <kbd>o<\/kbd> to open the selected file in my default application (<code>nvim<\/code>). tmux launches it in a new pane. note that this is still running <em>on the remote server<\/em>; it is opening a remote file in a remote tmux pane. i do not need to have this codebase cloned locally on my laptop.<\/li>\n<li><button class=timestamp>0:26<\/button>\n i try to navigate to several references using rust-analyzer, which fails because RA doesn't understand the macros in this file. at <button class=timestamp>0:32<\/button>\n i finally find one which works and navigate to it.<\/li>\n<li><button class=timestamp>0:38<\/button>\n i hit <kbd>ctrl + k<\/kbd><kbd>h<\/kbd>, which tells tmux to switch focus back to the left pane.<\/li>\n<li><button class=timestamp>0:39<\/button>\n i hit <kbd>n<\/kbd> again. the pane is still in \"copy-mode\", so all the files from before are still the focus of the search. they are highlighted again and tmux selects the next file in search order.<\/li>\n<li><button class=timestamp>0:41<\/button>\n i hit <kbd>o<\/kbd>, which opens a different file than before, but in the <em>same<\/em> instance of <code>nvim<\/code>.<\/li>\n<li><button class=timestamp>0:43<\/button>\n i hit <kbd><\/kbd><kbd>b<\/kbd>, which shows my open file buffers. in particular, this shows that the earlier file is still open. i switch back and forth between the two files a couple times before ending the stream.<\/li>\n<\/ol>\n<h2 id=\"but-why\">but why??<a class=\"zola-anchor\" href=\"#but-why\" aria-label=\"Anchor link for: but-why\"><\/a>\n<\/h2>\n<p>i got annoyed at VSCode a while back for being laggy, especially when the vim plugin was running, and at having lots of keybind conflicts between the editor, vim plugin, terminal, and window management. i tried zed but at the time it was quite immature (and still had the problem of lots of keybind conflicts).<\/p>\n<p>i switched to using nvim in the terminal, but quickly got annoyed at how much time i spent copy-pasting filenames into the editor; in particular i would often copy-paste files with columns from ripgrep, get a syntax error, and then have to edit them before actually opening the file. this was quite annoying. what i wanted was an equivalent of ctrl-click in vscode, where i could take an arbitrary file path and have it open as smoothly as i could navigate to it. so, i started using tmux and built it myself.<\/p>\n<p>people sometimes ask me why i use tmux. this is why! this is the whole reason! (well, this and session persistence.) terminals are stupidly powerful and most of them expose almost none of it to you as the user. i like tmux, despite its age, bugs, and antiquated syntax, because it's very extensible in this way.<\/p>\n<h2 id=\"how-it-works\">how it works<a class=\"zola-anchor\" href=\"#how-it-works\" aria-label=\"Anchor link for: how-it-works\"><\/a>\n<\/h2>\n<h3 id=\"search-all-scrollback-for-filenames\">search all scrollback for filenames<a class=\"zola-anchor\" href=\"#search-all-scrollback-for-filenames\" aria-label=\"Anchor link for: search-all-scrollback-for-filenames\"><\/a>\n<\/h3>\n<p>this is done purely with tmux config:<\/p>\n<pre data-lang=\"tmux\" class=\"language-tmux z-code\"><code class=\"language-tmux\" data-lang=\"tmux\"><span class=\"z-source z-tmux\"><span class=\"z-comment z-line z-number-sign z-tmux\"><span class=\"z-punctuation z-definition z-comment z-tmux\">#<\/span> i am so sorry\n<\/span><\/span><span class=\"z-source z-tmux\"><span class=\"z-comment z-line z-number-sign z-tmux\"><span class=\"z-punctuation z-definition z-comment z-tmux\">#<\/span> see `search-regex.sh` for wtf this means\n<\/span><\/span><span class=\"z-source z-tmux\"><span class=\"z-comment z-line z-number-sign z-tmux\"><span class=\"z-punctuation z-definition z-comment z-tmux\">#<\/span> TODO: include shell variable names\n<\/span><\/span><span class=\"z-source z-tmux\"><span class=\"z-support z-function z-tmux\">bind-key<\/span> f <span class=\"z-support z-function z-tmux\">copy-mode<\/span> \\; <span class=\"z-support z-function z-tmux\">send-keys<\/span> -X search-backward \\\n<\/span><span class=\"z-source z-tmux\">  <span class=\"z-string z-quoted z-single z-tmux\"><span class=\"z-punctuation z-definition z-string z-begin z-tmux\">&#39;<\/span>(^|\/|<span class=\"z-constant z-character z-escape z-tmux\">\\&lt;<\/span>|[[:space:]&quot;])((<span class=\"z-constant z-character z-escape z-tmux\">\\.<\/span>|<span class=\"z-constant z-character z-escape z-tmux\">\\.<\/span><span class=\"z-constant z-character z-escape z-tmux\">\\.<\/span>)|[[:alnum:]~_&quot;-]*)((\/[][[:alnum:]_.#$%&amp;+=@&quot;-]+)+([\/ &quot;]|<span class=\"z-constant z-character z-escape z-tmux\">\\.<\/span>([][[:alnum:]_.#$%&amp;+=@&quot;-]+(:[0-9]+)?(:[0-9]+)?)|[][[:alnum:]_.#$%&amp;+=@&quot;-]+(:[0-9]+)(:[0-9]+)?)|(\/[][[:alnum:]_.#$%&amp;+=@&quot;-]+){2,}([\/ &quot;]|<span class=\"z-constant z-character z-escape z-tmux\">\\.<\/span>([][[:alnum:]_.#$%&amp;+=@&quot;-]+(:[0-9]+)?(:[0-9]+)?)|[][[:alnum:]_.#$%&amp;+=@&quot;-]+(:[0-9]+)(:[0-9]+)?)?|(<span class=\"z-constant z-character z-escape z-tmux\">\\.<\/span>|<span class=\"z-constant z-character z-escape z-tmux\">\\.<\/span><span class=\"z-constant z-character z-escape z-tmux\">\\.<\/span>)\/([][[:alnum:]_.#$%&amp;+=@&quot;-]+(:[0-9]+)?(:[0-9]+)?))<span class=\"z-punctuation z-definition z-string z-end z-tmux\">&#39;<\/span><\/span>\n<\/span><\/code><\/pre>\n<p>and this is the contents of <code>search-regex.sh<\/code>:<\/p>\n<pre data-lang=\"sh\" class=\"language-sh z-code\"><code class=\"language-sh\" data-lang=\"sh\"><span class=\"z-source z-shell z-bash\"><span class=\"z-variable z-other z-readwrite z-assignment z-shell\">start_delim<\/span><span class=\"z-keyword z-operator z-assignment z-shell\">=<\/span><span class=\"z-string z-unquoted z-shell\"><span class=\"z-string z-quoted z-single z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&#39;<\/span>(^|\/|\\&lt;|[[:space:]&quot;])<span class=\"z-punctuation z-definition z-string z-end z-shell\">&#39;<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-shell z-bash\">\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-variable z-other z-readwrite z-assignment z-shell\">relative_path<\/span><span class=\"z-keyword z-operator z-assignment z-shell\">=<\/span><span class=\"z-string z-unquoted z-shell\"><span class=\"z-string z-quoted z-single z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&#39;<\/span>(\\.|\\.\\.)<span class=\"z-punctuation z-definition z-string z-end z-shell\">&#39;<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-variable z-other z-readwrite z-assignment z-shell\">start_path<\/span><span class=\"z-keyword z-operator z-assignment z-shell\">=<\/span><span class=\"z-string z-unquoted z-shell\"><span class=\"z-string z-quoted z-double z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&quot;<\/span>(<span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-punctuation z-definition z-variable z-shell\">$<\/span><span class=\"z-variable z-other z-readwrite z-shell\">relative_path<\/span><\/span>|[[:alnum:]~_<span class=\"z-constant z-character z-escape z-shell\">\\&quot;<\/span>-]*)<span class=\"z-punctuation z-definition z-string z-end z-shell\">&quot;<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-shell z-bash\">\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-variable z-other z-readwrite z-assignment z-shell\">component<\/span><span class=\"z-keyword z-operator z-assignment z-shell\">=<\/span><span class=\"z-string z-unquoted z-shell\"><span class=\"z-string z-quoted z-single z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&#39;<\/span>[][[:alnum:]_.#$%&amp;+=@&quot;-]<span class=\"z-punctuation z-definition z-string z-end z-shell\">&#39;<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-variable z-other z-readwrite z-assignment z-shell\">intermediate_paths<\/span><span class=\"z-keyword z-operator z-assignment z-shell\">=<\/span><span class=\"z-string z-unquoted z-shell\"><span class=\"z-string z-quoted z-double z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&quot;<\/span>(\/<span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-punctuation z-definition z-variable z-shell\">$<\/span><span class=\"z-variable z-other z-readwrite z-shell\">component<\/span><\/span>+)<span class=\"z-punctuation z-definition z-string z-end z-shell\">&quot;<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-shell z-bash\">\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-variable z-other z-readwrite z-assignment z-shell\">line_no<\/span><span class=\"z-keyword z-operator z-assignment z-shell\">=<\/span><span class=\"z-string z-unquoted z-shell\"><span class=\"z-string z-quoted z-single z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&#39;<\/span>(:[0-9]+)<span class=\"z-punctuation z-definition z-string z-end z-shell\">&#39;<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-variable z-other z-readwrite z-assignment z-shell\">file_end<\/span><span class=\"z-keyword z-operator z-assignment z-shell\">=<\/span><span class=\"z-string z-unquoted z-shell\"><span class=\"z-string z-quoted z-double z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&quot;<\/span>(<span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-punctuation z-definition z-variable z-shell\">$<\/span><span class=\"z-variable z-other z-readwrite z-shell\">component<\/span><\/span>+<span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-punctuation z-definition z-variable z-shell\">$<\/span><span class=\"z-variable z-other z-readwrite z-shell\">line_no<\/span><\/span>?<span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-punctuation z-definition z-variable z-shell\">$<\/span><span class=\"z-variable z-other z-readwrite z-shell\">line_no<\/span><\/span>?)<span class=\"z-punctuation z-definition z-string z-end z-shell\">&quot;<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-variable z-other z-readwrite z-assignment z-shell\">end<\/span><span class=\"z-keyword z-operator z-assignment z-shell\">=<\/span><span class=\"z-string z-unquoted z-shell\"><span class=\"z-string z-quoted z-double z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&quot;<\/span>([\/ <span class=\"z-constant z-character z-escape z-shell\">\\&quot;<\/span>]|\\.<span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-punctuation z-definition z-variable z-shell\">$<\/span><span class=\"z-variable z-other z-readwrite z-shell\">file_end<\/span><\/span>|<span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-punctuation z-definition z-variable z-shell\">$<\/span><span class=\"z-variable z-other z-readwrite z-shell\">component<\/span><\/span>+<span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-punctuation z-definition z-variable z-shell\">$<\/span><span class=\"z-variable z-other z-readwrite z-shell\">line_no<\/span><\/span><span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-punctuation z-definition z-variable z-shell\">$<\/span><span class=\"z-variable z-other z-readwrite z-shell\">line_no<\/span><\/span>?)<span class=\"z-punctuation z-definition z-string z-end z-shell\">&quot;<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-shell z-bash\">\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-support z-function z-echo z-shell\">echo<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> <span class=\"z-string z-quoted z-double z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&quot;<\/span><span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-punctuation z-definition z-variable z-shell\">$<\/span><span class=\"z-variable z-other z-readwrite z-shell\">start_delim<\/span><\/span><span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-punctuation z-definition z-variable z-shell\">$<\/span><span class=\"z-variable z-other z-readwrite z-shell\">start_path<\/span><\/span>(<span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-punctuation z-definition z-variable z-shell\">$<\/span><span class=\"z-punctuation z-section z-expansion z-parameter z-begin z-shell\">{<\/span><\/span><span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-variable z-other z-readwrite z-shell\">intermediate_paths<\/span><\/span><span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-punctuation z-section z-expansion z-parameter z-end z-shell\">}<\/span><\/span>+<span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-punctuation z-definition z-variable z-shell\">$<\/span><span class=\"z-variable z-other z-readwrite z-shell\">end<\/span><\/span>|<span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-punctuation z-definition z-variable z-shell\">$<\/span><span class=\"z-punctuation z-section z-expansion z-parameter z-begin z-shell\">{<\/span><\/span><span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-variable z-other z-readwrite z-shell\">intermediate_paths<\/span><\/span><span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-punctuation z-section z-expansion z-parameter z-end z-shell\">}<\/span><\/span>{2,}<span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-punctuation z-definition z-variable z-shell\">$<\/span><span class=\"z-variable z-other z-readwrite z-shell\">end<\/span><\/span>?|<span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-punctuation z-definition z-variable z-shell\">$<\/span><span class=\"z-variable z-other z-readwrite z-shell\">relative_path<\/span><\/span>\/<span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-punctuation z-definition z-variable z-shell\">$<\/span><span class=\"z-variable z-other z-readwrite z-shell\">file_end<\/span><\/span>)<span class=\"z-punctuation z-definition z-string z-end z-shell\">&quot;<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-shell z-bash\">\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-comment z-line z-number-sign z-shell\"><span class=\"z-punctuation z-definition z-comment z-begin z-shell\">#<\/span><\/span><span class=\"z-comment z-line z-number-sign z-shell\"> test cases omitted for brevity<\/span><span class=\"z-comment z-line z-number-sign z-shell\">\n<\/span><\/span><\/code><\/pre>\n<p>i will not go through the whole regex, but uh. there you go. i spent more time on this than i probably should have.<\/p>\n<h3 id=\"open-selected-file-in-a-new-pane-running-nvim\">open selected file in a new pane running nvim<a class=\"zola-anchor\" href=\"#open-selected-file-in-a-new-pane-running-nvim\" aria-label=\"Anchor link for: open-selected-file-in-a-new-pane-running-nvim\"><\/a>\n<\/h3>\n<p>this is actually a trick; there are many steps here.<\/p>\n<h4 id=\"open-selected-file-in-default-application\">open selected file in default application<a class=\"zola-anchor\" href=\"#open-selected-file-in-default-application\" aria-label=\"Anchor link for: open-selected-file-in-default-application\"><\/a>\n<\/h4>\n<p>this part is not so bad. tmux again.<\/p>\n<pre data-lang=\"tmux\" class=\"language-tmux z-code\"><code class=\"language-tmux\" data-lang=\"tmux\"><span class=\"z-source z-tmux\"><span class=\"z-comment z-line z-number-sign z-tmux\"><span class=\"z-punctuation z-definition z-comment z-tmux\">#<\/span> `cd` is important in case this is a relative path. `echo | bash` is to perform tilde expansion.\n<\/span><\/span><span class=\"z-source z-tmux\"><span class=\"z-support z-function z-tmux\">bind-key<\/span> -T <span class=\"z-support z-function z-tmux\">copy-mode<\/span>-vi o  <span class=\"z-support z-function z-tmux\">send-keys<\/span> -X copy-pipe \\\n<\/span><span class=\"z-source z-tmux\">    <span class=\"z-string z-quoted z-single z-tmux\"><span class=\"z-punctuation z-definition z-string z-begin z-tmux\">&#39;<\/span>cd #{pane_current_path}; xargs -I {} echo &quot;echo {}&quot; | bash | xargs open<span class=\"z-punctuation z-definition z-string z-end z-tmux\">&#39;<\/span><\/span> \\; \\\n<\/span><span class=\"z-source z-tmux\">  if -F <span class=\"z-string z-quoted z-double z-tmux\"><span class=\"z-punctuation z-definition z-string z-begin z-tmux\">&quot;<\/span>#{alternate_on}<span class=\"z-punctuation z-definition z-string z-end z-tmux\">&quot;<\/span><\/span> { <span class=\"z-support z-function z-tmux\">send-keys<\/span> -X cancel }\n<\/span><\/code><\/pre>\n<p>i also have a version that always opens an editor in the current pane, instead of launching in the default application. for example i use <a href=\"https:\/\/fx.wtf\/\"><code>fx<\/code><\/a> by default to view json files, but <code>nvim<\/code> to edit them.<\/p>\n<pre data-lang=\"tmux\" class=\"language-tmux z-code\"><code class=\"language-tmux\" data-lang=\"tmux\"><span class=\"z-source z-tmux\"><span class=\"z-comment z-line z-number-sign z-tmux\"><span class=\"z-punctuation z-definition z-comment z-tmux\">#<\/span> save the buffer, then open an editor in the current pane\n<\/span><\/span><span class=\"z-source z-tmux\"><span class=\"z-support z-function z-tmux\">bind-key<\/span> -T <span class=\"z-support z-function z-tmux\">copy-mode<\/span>-vi O <span class=\"z-support z-function z-tmux\">send-keys<\/span> -X copy-pipe-and-cancel \\\n<\/span><span class=\"z-source z-tmux\">    <span class=\"z-string z-quoted z-single z-tmux\"><span class=\"z-punctuation z-definition z-string z-begin z-tmux\">&#39;<\/span>tmux send-keys &quot;C-q&quot;; xargs -I {} tmux send-keys &quot;${EDITOR:-vi} {}&quot;; tmux send-keys &quot;C-m&quot;<span class=\"z-punctuation z-definition z-string z-end z-tmux\">&#39;<\/span><\/span>\n<\/span><\/code><\/pre>\n<h4 id=\"open-a-new-pane-running-nvim\">open a new pane running nvim<a class=\"zola-anchor\" href=\"#open-a-new-pane-running-nvim\" aria-label=\"Anchor link for: open-a-new-pane-running-nvim\"><\/a>\n<\/h4>\n<p>here is the trick. i have created <a href=\"https:\/\/github.com\/jyn514\/dotfiles\/blob\/master\/bin\/hx-hax\">a shell script<\/a> (actually a perl script) that is the default application for all text files.<\/p>\n<aside role=note class=note-container><div class=note-content>\n<p>setting up that many file associations by hand is a pain. i will write a separate blog post about the scripts that install my dotfiles onto a system. i don't use Nix partly because all my friends who use Nix have <em>even weirder<\/em> bugs than they already had, and partly because i don't like the philosophy of not being able to install things at runtime. i want to install things at runtime and <em>track<\/em> that i did so. that's a separate post too.<\/p>\n<\/div><\/aside>\n<p>the relevant part is this:<\/p>\n<pre data-lang=\"perl\" class=\"language-perl z-code\"><code class=\"language-perl\" data-lang=\"perl\"><span class=\"z-source z-perl\"><span class=\"z-meta z-comment z-perl\"><span class=\"z-comment z-line z-number-sign z-perl\"><span class=\"z-punctuation z-definition z-comment z-perl\">#<\/span> don&#39;t use `` so that args can have embedded pipes\n<\/span><\/span><\/span><span class=\"z-source z-perl\"><span class=\"z-storage z-type z-variable z-perl\">my<\/span> <span class=\"z-variable z-other z-readwrite z-perl\"><span class=\"z-punctuation z-definition z-variable z-perl\">@<\/span>split<\/span> <span class=\"z-keyword z-operator z-assignment z-perl\">=<\/span> <span class=\"z-punctuation z-section z-group z-begin z-perl\">(<\/span><span class=\"z-meta z-string z-perl\"><span class=\"z-string z-quoted z-single z-perl\"><span class=\"z-punctuation z-definition z-string z-begin z-perl\">&#39;<\/span>tmux<span class=\"z-punctuation z-definition z-string z-end z-perl\">&#39;<\/span><\/span><\/span><span class=\"z-punctuation z-separator z-sequence z-perl\">,<\/span> <span class=\"z-meta z-string z-perl\"><span class=\"z-string z-quoted z-single z-perl\"><span class=\"z-punctuation z-definition z-string z-begin z-perl\">&#39;<\/span>split-window<span class=\"z-punctuation z-definition z-string z-end z-perl\">&#39;<\/span><\/span><\/span><span class=\"z-punctuation z-separator z-sequence z-perl\">,<\/span> <span class=\"z-meta z-string z-perl\"><span class=\"z-string z-quoted z-single z-perl\"><span class=\"z-punctuation z-definition z-string z-begin z-perl\">&#39;<\/span>-h<span class=\"z-punctuation z-definition z-string z-end z-perl\">&#39;<\/span><\/span><\/span><span class=\"z-punctuation z-separator z-sequence z-perl\">,<\/span> <span class=\"z-meta z-string z-perl\"><span class=\"z-string z-quoted z-single z-perl\"><span class=\"z-punctuation z-definition z-string z-begin z-perl\">&#39;<\/span>-P<span class=\"z-punctuation z-definition z-string z-end z-perl\">&#39;<\/span><\/span><\/span><span class=\"z-punctuation z-separator z-sequence z-perl\">,<\/span> <span class=\"z-meta z-string z-perl\"><span class=\"z-string z-quoted z-single z-perl\"><span class=\"z-punctuation z-definition z-string z-begin z-perl\">&#39;<\/span>-F<span class=\"z-punctuation z-definition z-string z-end z-perl\">&#39;<\/span><\/span><\/span><span class=\"z-punctuation z-separator z-sequence z-perl\">,<\/span> <span class=\"z-meta z-string z-perl\"><span class=\"z-string z-quoted z-single z-perl\"><span class=\"z-punctuation z-definition z-string z-begin z-perl\">&#39;<\/span>&quot;#{pane_id}&quot;<span class=\"z-punctuation z-definition z-string z-end z-perl\">&#39;<\/span><\/span><\/span><span class=\"z-punctuation z-separator z-sequence z-perl\">,<\/span> <span class=\"z-variable z-other z-readwrite z-perl\"><span class=\"z-punctuation z-definition z-variable z-perl\">$<\/span>editor<\/span><span class=\"z-punctuation z-separator z-sequence z-perl\">,<\/span> <span class=\"z-variable z-other z-readwrite z-perl\"><span class=\"z-punctuation z-definition z-variable z-perl\">@<\/span>args<\/span><span class=\"z-punctuation z-section z-group z-end z-perl\">)<\/span><span class=\"z-punctuation z-terminator z-statement z-perl\">;<\/span>\n<\/span><span class=\"z-source z-perl\"><span class=\"z-meta z-function-call z-perl\"><span class=\"z-support z-function z-perl\">open<\/span><\/span><span class=\"z-punctuation z-section z-group z-begin z-perl\">(<\/span><span class=\"z-storage z-type z-variable z-perl\">my<\/span> <span class=\"z-variable z-other z-readwrite z-perl\"><span class=\"z-punctuation z-definition z-variable z-perl\">$<\/span>fd<\/span><span class=\"z-punctuation z-separator z-sequence z-perl\">,<\/span> <span class=\"z-meta z-string z-perl\"><span class=\"z-string z-quoted z-single z-perl\"><span class=\"z-punctuation z-definition z-string z-begin z-perl\">&#39;<\/span>-|<span class=\"z-punctuation z-definition z-string z-end z-perl\">&#39;<\/span><\/span><\/span><span class=\"z-punctuation z-separator z-sequence z-perl\">,<\/span> <span class=\"z-variable z-other z-readwrite z-perl\"><span class=\"z-punctuation z-definition z-variable z-perl\">@<\/span>split<\/span><span class=\"z-punctuation z-section z-group z-end z-perl\">)<\/span> <span class=\"z-keyword z-operator z-logical z-perl\">||<\/span> <span class=\"z-keyword z-control z-flow z-die z-perl\">die<\/span> <span class=\"z-meta z-string z-perl\"><span class=\"z-string z-quoted z-double z-perl\"><span class=\"z-punctuation z-definition z-string z-begin z-perl\">&quot;<\/span>can&#39;t open pipeline: <\/span><span class=\"z-meta z-interpolation z-perl\"><span class=\"z-variable z-language z-perl\"><span class=\"z-punctuation z-definition z-variable z-perl\">$<\/span>!<\/span><\/span><span class=\"z-string z-quoted z-double z-perl\"><span class=\"z-punctuation z-definition z-string z-end z-perl\">&quot;<\/span><\/span><\/span><span class=\"z-punctuation z-terminator z-statement z-perl\">;<\/span>\n<\/span><\/code><\/pre>\n<p>this bounces <em>back<\/em> to tmux. in particular, this is being very dumb and assuming that tmux is running on the machine where the file is, which happens to be the case here. this is not too bad to ensure - i just use a separate terminal <em>emulator<\/em> tab for each instance of tmux i care about; for example i will often have open one Windows Terminal tab for WSL on my local laptop, one for my desktop, and one for a remote work machine via a VPN.<\/p>\n<aside role=note class=note-container><div class=note-content>\n<p>there's actually even more going on here\u2014for example i am translating the <code>file:line:column<\/code> syntax to something vim understands, and overriding <code>xdg-open<\/code> so that it doesn't error out on the <code>:line<\/code>\u2014but for the most part it's straightforward and not that interesting.<\/p>\n<\/div><\/aside>\n<h3 id=\"open-a-file-in-a-running-instance-of-nvim\">open a file in a running instance of nvim<a class=\"zola-anchor\" href=\"#open-a-file-in-a-running-instance-of-nvim\" aria-label=\"Anchor link for: open-a-file-in-a-running-instance-of-nvim\"><\/a>\n<\/h3>\n<p>this is a perl script that scripts tmux to send keys to a running instance of nvim (actually the same perl script as before, so that both of these can be bound to the same keybind regardless of whether nvim is already open or not):<\/p>\n<pre data-lang=\"perl\" class=\"language-perl z-code\"><code class=\"language-perl\" data-lang=\"perl\"><span class=\"z-source z-perl\"><span class=\"z-storage z-type z-variable z-perl\">my<\/span> <span class=\"z-variable z-other z-readwrite z-perl\"><span class=\"z-punctuation z-definition z-variable z-perl\">$<\/span>current_window<\/span><span class=\"z-keyword z-operator z-assignment z-perl\">=<\/span> <span class=\"z-meta z-function-call z-perl\"><span class=\"z-variable z-function z-perl\">trim<\/span><\/span> <span class=\"z-meta z-string z-perl\"><span class=\"z-string z-quoted z-backtick z-perl\"><span class=\"z-punctuation z-definition z-string z-begin z-perl\">`<\/span>tmux display-message -p &quot;#{window_id}&quot;<span class=\"z-punctuation z-definition z-string z-end z-perl\">`<\/span><\/span><\/span><span class=\"z-punctuation z-terminator z-statement z-perl\">;<\/span>\n<\/span><span class=\"z-source z-perl\"><span class=\"z-storage z-type z-variable z-perl\">my<\/span> <span class=\"z-variable z-other z-readwrite z-perl\"><span class=\"z-punctuation z-definition z-variable z-perl\">$<\/span>pane<\/span> <span class=\"z-keyword z-operator z-assignment z-perl\">=<\/span> <span class=\"z-meta z-function-call z-perl\"><span class=\"z-variable z-function z-perl\">trim<\/span><\/span> <span class=\"z-meta z-string z-perl\"><span class=\"z-string z-quoted z-backtick z-perl\"><span class=\"z-punctuation z-definition z-string z-begin z-perl\">`<\/span>tmux list-panes -a <span class=\"z-constant z-character z-escape z-perl\">\\\\<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-perl\"><span class=\"z-meta z-string z-perl\"><span class=\"z-string z-quoted z-backtick z-perl\">  -f &#39;#{&amp;&amp;:#{==:#{window_id},<\/span><span class=\"z-meta z-interpolation z-perl\"><span class=\"z-variable z-other z-readwrite z-perl\"><span class=\"z-punctuation z-definition z-variable z-perl\">$<\/span>current_window<\/span><\/span><span class=\"z-string z-quoted z-backtick z-perl\">},#{==:#{pane_current_command},<\/span><span class=\"z-meta z-interpolation z-perl\"><span class=\"z-variable z-other z-readwrite z-perl\"><span class=\"z-punctuation z-definition z-variable z-perl\">$<\/span>editor<\/span><\/span><span class=\"z-string z-quoted z-backtick z-perl\">}}&#39; <span class=\"z-constant z-character z-escape z-perl\">\\\\<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-perl\"><span class=\"z-meta z-string z-perl\"><span class=\"z-string z-quoted z-backtick z-perl\">  -F &#39;#{pane_id}&#39;<span class=\"z-punctuation z-definition z-string z-end z-perl\">`<\/span><\/span><\/span><span class=\"z-punctuation z-terminator z-statement z-perl\">;<\/span>\n<\/span><span class=\"z-source z-perl\">\n<\/span><span class=\"z-source z-perl\"><span class=\"z-meta z-comment z-perl\"><span class=\"z-comment z-line z-number-sign z-perl\"><span class=\"z-punctuation z-definition z-comment z-perl\">#<\/span> ...\n<\/span><\/span><\/span><span class=\"z-source z-perl\">\n<\/span><span class=\"z-source z-perl\"><span class=\"z-meta z-comment z-perl\"><span class=\"z-comment z-line z-number-sign z-perl\"><span class=\"z-punctuation z-definition z-comment z-perl\">#<\/span> exit copy mode so we don&#39;t send these commands directly to tmux\n<\/span><\/span><\/span><span class=\"z-source z-perl\"><span class=\"z-meta z-string z-perl\"><span class=\"z-string z-quoted z-backtick z-perl\"><span class=\"z-punctuation z-definition z-string z-begin z-perl\">`<\/span>tmux send-keys -t <\/span><span class=\"z-meta z-interpolation z-perl\"><span class=\"z-variable z-other z-readwrite z-perl\"><span class=\"z-punctuation z-definition z-variable z-perl\">$<\/span>pane<\/span><\/span><span class=\"z-string z-quoted z-backtick z-perl\"> -X cancel 2&gt;\/dev\/null<span class=\"z-punctuation z-definition z-string z-end z-perl\">`<\/span><\/span><\/span><span class=\"z-punctuation z-terminator z-statement z-perl\">;<\/span>\n<\/span><span class=\"z-source z-perl\"><span class=\"z-meta z-comment z-perl\"><span class=\"z-comment z-line z-number-sign z-perl\"><span class=\"z-punctuation z-definition z-comment z-perl\">#<\/span> Escape for some reason doesn&#39;t get sent as the escape key if it shows up next to any other keys???\n<\/span><\/span><\/span><span class=\"z-source z-perl\"><span class=\"z-meta z-string z-perl\"><span class=\"z-string z-quoted z-backtick z-perl\"><span class=\"z-punctuation z-definition z-string z-begin z-perl\">`<\/span>tmux send-keys -t <\/span><span class=\"z-meta z-interpolation z-perl\"><span class=\"z-variable z-other z-readwrite z-perl\"><span class=\"z-punctuation z-definition z-variable z-perl\">$<\/span>pane<\/span><\/span><span class=\"z-string z-quoted z-backtick z-perl\"> Escape<span class=\"z-punctuation z-definition z-string z-end z-perl\">`<\/span><\/span><\/span><span class=\"z-punctuation z-terminator z-statement z-perl\">;<\/span>\n<\/span><span class=\"z-source z-perl\"><span class=\"z-storage z-type z-variable z-perl\">my<\/span> <span class=\"z-variable z-other z-readwrite z-perl\"><span class=\"z-punctuation z-definition z-variable z-perl\">$<\/span>args<\/span> <span class=\"z-keyword z-operator z-assignment z-perl\">=<\/span> <span class=\"z-meta z-function-call z-perl\"><span class=\"z-support z-function z-perl\">join<\/span><\/span> <span class=\"z-meta z-string z-perl\"><span class=\"z-string z-quoted z-single z-perl\"><span class=\"z-punctuation z-definition z-string z-begin z-perl\">&#39;<\/span> <span class=\"z-punctuation z-definition z-string z-end z-perl\">&#39;<\/span><\/span><\/span><span class=\"z-punctuation z-separator z-sequence z-perl\">,<\/span> <span class=\"z-variable z-other z-readwrite z-perl\"><span class=\"z-punctuation z-definition z-variable z-perl\">@<\/span>args<\/span><span class=\"z-punctuation z-terminator z-statement z-perl\">;<\/span>\n<\/span><span class=\"z-source z-perl\"><span class=\"z-storage z-type z-variable z-perl\">my<\/span> <span class=\"z-variable z-other z-readwrite z-perl\"><span class=\"z-punctuation z-definition z-variable z-perl\">$<\/span>cmd<\/span> <span class=\"z-keyword z-operator z-assignment z-perl\">=<\/span> <span class=\"z-variable z-other z-readwrite z-perl\"><span class=\"z-punctuation z-definition z-variable z-perl\">$<\/span>editor<\/span> <span class=\"z-keyword z-operator z-logical z-perl\">eq<\/span> <span class=\"z-meta z-string z-perl\"><span class=\"z-string z-quoted z-single z-perl\"><span class=\"z-punctuation z-definition z-string z-begin z-perl\">&#39;<\/span>nvim<span class=\"z-punctuation z-definition z-string z-end z-perl\">&#39;<\/span><\/span><\/span> <span class=\"z-keyword z-operator z-logical z-perl\">?<\/span> <span class=\"z-meta z-string z-perl\"><span class=\"z-string z-quoted z-single z-perl\"><span class=\"z-punctuation z-definition z-string z-begin z-perl\">&#39;<\/span>drop<span class=\"z-punctuation z-definition z-string z-end z-perl\">&#39;<\/span><\/span><\/span> <span class=\"z-keyword z-operator z-logical z-perl\">:<\/span> <span class=\"z-meta z-string z-perl\"><span class=\"z-string z-quoted z-single z-perl\"><span class=\"z-punctuation z-definition z-string z-begin z-perl\">&#39;<\/span>open<span class=\"z-punctuation z-definition z-string z-end z-perl\">&#39;<\/span><\/span><\/span><span class=\"z-punctuation z-terminator z-statement z-perl\">;<\/span>\n<\/span><span class=\"z-source z-perl\"><span class=\"z-meta z-string z-perl\"><span class=\"z-string z-quoted z-backtick z-perl\"><span class=\"z-punctuation z-definition z-string z-begin z-perl\">`<\/span>tmux send-keys -t <\/span><span class=\"z-meta z-interpolation z-perl\"><span class=\"z-variable z-other z-readwrite z-perl\"><span class=\"z-punctuation z-definition z-variable z-perl\">$<\/span>pane<\/span><\/span><span class=\"z-string z-quoted z-backtick z-perl\"> &quot;:<\/span><span class=\"z-meta z-interpolation z-perl\"><span class=\"z-variable z-other z-readwrite z-perl\"><span class=\"z-punctuation z-definition z-variable z-perl\">$<\/span>cmd<\/span><\/span><span class=\"z-string z-quoted z-backtick z-perl\"> <\/span><span class=\"z-meta z-interpolation z-perl\"><span class=\"z-variable z-other z-readwrite z-perl\"><span class=\"z-punctuation z-definition z-variable z-perl\">$<\/span>args<\/span><\/span><span class=\"z-string z-quoted z-backtick z-perl\">&quot; Enter<span class=\"z-punctuation z-definition z-string z-end z-perl\">`<\/span><\/span><\/span><span class=\"z-punctuation z-terminator z-statement z-perl\">;<\/span>\n<\/span><span class=\"z-source z-perl\"><span class=\"z-meta z-string z-perl\"><span class=\"z-string z-quoted z-backtick z-perl\"><span class=\"z-punctuation z-definition z-string z-begin z-perl\">`<\/span>tmux select-pane -t <\/span><span class=\"z-meta z-interpolation z-perl\"><span class=\"z-variable z-other z-readwrite z-perl\"><span class=\"z-punctuation z-definition z-variable z-perl\">$<\/span>pane<\/span><\/span><span class=\"z-string z-quoted z-backtick z-perl\"> -Z<span class=\"z-punctuation z-definition z-string z-end z-perl\">`<\/span><\/span><\/span><span class=\"z-punctuation z-terminator z-statement z-perl\">;<\/span>\n<\/span><\/code><\/pre>\n<h2 id=\"consequences-of-this-setup\">consequences of this setup<a class=\"zola-anchor\" href=\"#consequences-of-this-setup\" aria-label=\"Anchor link for: consequences-of-this-setup\"><\/a>\n<\/h2>\n<ul>\n<li>i don't need a fancy terminal locally; something with nice fonts is enough. all the fancy things are done through tmux, which is good because it means they work on Windows too without needing to install a separate terminal.<\/li>\n<li>the editor thing works even if the editor doesn't support remote scripting. nvim <em>does<\/em> support RPC, but this setup also worked back when i used <code>helix<\/code> and <code>kakoune<\/code>.<\/li>\n<li>i <em>could<\/em> have written this such that the fancy terminal emulator scripts were in my editor, not in tmux (e.g. <code>:terminal<\/code> in nvim). but again this locks me into the editor; and the built-in terminals in editors are usually not very good.<\/li>\n<\/ul>\n<h2 id=\"ok-but-do-you-really-want-to-use-tmux\">ok, but do you really want to use tmux<a class=\"zola-anchor\" href=\"#ok-but-do-you-really-want-to-use-tmux\" aria-label=\"Anchor link for: ok-but-do-you-really-want-to-use-tmux\"><\/a>\n<\/h2>\n<p>well. well. now that you mention it. the last thing keeping me on tmux was session persistence and <a href=\"https:\/\/ansuz.sooke.bc.ca\/entry\/389\">Ansuz has just released a standalone tool that does persistence and nothing else<\/a>. so. i plan to switch to <a href=\"https:\/\/sw.kovidgoyal.net\/kitty\/\">kitty<\/a> in the near future, which lets me keep all these scripts and does not require shoving a whole second terminal emulator inside my terminal emulator, which hopefully will reduce the number of weird mysterious bugs i encounter on a regular basis.<\/p>\n<p>the reason i picked kitty over <a href=\"https:\/\/wezterm.org\/\">wezterm<\/a> is that ssh integration works by integrating with the shell, not by launching a server process, so it doesn't need to be installed on the remote. this mattered less for tmux because tmux is everywhere, but hardly anywhere has wezterm installed by default.<\/p>\n<h2 id=\"was-it-worth-it\">... was it worth it?<a class=\"zola-anchor\" href=\"#was-it-worth-it\" aria-label=\"Anchor link for: was-it-worth-it\"><\/a>\n<\/h2>\n<p>honestly, yeah. i spend quite a lot less time fighting my editor these days.<\/p>\n<ul>\n<li>it's <em>much<\/em> easier to debug when something goes wrong (vscode's debugging tools are mostly for plugin extension authors and running them is non-trivial). with vim plugins i can just add <code>print<\/code> statements to the lua source and see what's happening.<\/li>\n<li>all my keybinds make sense to me!<\/li>\n<li>my editor is less laggy.<\/li>\n<li>my terminal is much easier to script through tmux than through writing a VSCode plugin, which usually involves setting up a whole typescript toolchain and context-switching into a new project<\/li>\n<\/ul>\n<p>that said, i cannot in good conscience recommend this to anyone else. all my scripts are fragile and will probably break if you look at them wrong, which is not ideal if you haven't written them yourself and don't know where to start debugging them.<\/p>\n<h2 id=\"ok-but-this-looks-nice-i-want-this\">ok but this looks nice i want this<a class=\"zola-anchor\" href=\"#ok-but-this-looks-nice-i-want-this\" aria-label=\"Anchor link for: ok-but-this-looks-nice-i-want-this\"><\/a>\n<\/h2>\n<p>if you do want something similar without writing your own tools, i can recommend:<\/p>\n<ul>\n<li><a href=\"https:\/\/fishshell.com\/\">fish<\/a> + <a href=\"https:\/\/github.com\/ajeetdsouza\/zoxide\/\">zoxide<\/a> + <a href=\"https:\/\/github.com\/junegunn\/fzf\">fzf<\/a>. that gets you steps 4, 5, and kinda sorta-ish 6.<\/li>\n<li>\"builtin functionality in your editor\" - fuzzy find, full text search, tabs and windows, and \"open recent file\" are all commonly supported.<\/li>\n<li><a href=\"https:\/\/git.causal.agency\/src\/tree\/bin\/qf.c\">qf<\/a>, which gets you the \"select files in terminal output\" part of 6, kinda. you have to remember to pipe your output to it though, so it doesn't work after the fact and it doesn't work if your tool is interactive. note that it hard-codes a vi-like CLI (<code>vi +line file.ext<\/code>), so you may need to fork it or still add a script that takes the place of $EDITOR. see <a href=\"https:\/\/jvns.ca\/blog\/2025\/06\/10\/how-to-compile-a-c-program\/\">julia evans' most recent post<\/a> for more info.<\/li>\n<li><a href=\"https:\/\/github.com\/kilobyte\/e\">e<\/a>, which gets you the \"translate <code>file:line<\/code> into something your editor recognizes\" part of 8, kinda. i had never heard of this tool until i wrote my own with literally the exactly the same name that did literally exactly the same thing, forgot to put it in PATH, and got a suggestion from <code>command-not-found<\/code> asking if i wanted to install it, lol.<\/li>\n<li><code>vim --remote filename<\/code> or <code>code filename<\/code> or <code>emacsclient filename<\/code>, all of which get you 12, kinda. the problem with this is that they don't all support <code>file:line<\/code>, and it means you have to modify this whenever you switch editors. admittedly most people don't switch editors that often, lol.<\/li>\n<\/ul>\n<h2 id=\"what-have-we-learned\">what have we learned?<a class=\"zola-anchor\" href=\"#what-have-we-learned\" aria-label=\"Anchor link for: what-have-we-learned\"><\/a>\n<\/h2>\n<ul>\n<li>terminals are a lot more powerful than people think! by using terminals that let you script them, you can do quite a lot of things.<\/li>\n<li>you can kinda sorta replicate most of these features without scripting your terminal, as long as you don't mind tying yourself to an editor.<\/li>\n<li>doing this requires quite a lot of work, because no one who builds these tools thought of these features ahead of time.<\/li>\n<\/ul>\n<p>hopefully this was interesting! i am always curious what tools people use and how - feel free to <a href=\"mailto:blog@jyn.dev\">email me<\/a> about your own setup :)<\/p>\n"},{"title":"theory building without a mentor","published":"2025-05-24T00:00:00+00:00","updated":"2025-05-24T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/theory-building-without-a-mentor\/"}},"id":"https:\/\/jyn.dev\/theory-building-without-a-mentor\/","summary":"learning how a program is intended to work is hard. here's how to do it.","content":"<p><strong>NOTE: if you are just here for the how-to guide, <a href=\"https:\/\/jyn.dev\/theory-building-without-a-mentor\/#recreating-a-theory\">click here<\/a> to skip the philosophizing.<\/strong><\/p>\n<h2 id=\"theory-building\">theory building<a class=\"zola-anchor\" href=\"#theory-building\" aria-label=\"Anchor link for: theory-building\"><\/a>\n<\/h2>\n<p>Peter Naur wrote a famous article in 1985 called <a href=\"https:\/\/gist.github.com\/onlurking\/fc5c81d18cfce9ff81bc968a7f342fb1\">Programming as Theory Building<\/a>. it has some excellent ideas, such as:<\/p>\n<blockquote>\n<p>programming must be the programmers\u2019 building up knowledge of a certain kind, knowledge taken to be basically the programmers\u2019 immediate possession, any documentation being an auxiliary product.<\/p>\n<\/blockquote>\n<blockquote>\n<p>solutions suggested by group B [who did not possess a theory of the program] [\u2026] effectively destroyed its power and simplicity. The members of group A [who did possess a theory] were able to spot these cases instantly and could propose simple and effective solutions, framed entirely within the existing structure.<\/p>\n<\/blockquote>\n<blockquote>\n<p>the program text and its documentation proved insufficient as a carrier of the most important design ideas<\/p>\n<\/blockquote>\n<p>i think this article is excellent, and highly recommend reading it in full. however, i want to discuss one particular idea Naur mentions:<\/p>\n<blockquote>\n<p>For a new programmer to come to possess an existing theory of a program it is insufficient that he or she has the opportunity to become familiar with the program text and other documentation. What is required is that the new programmer has the opportunity to work in close contact with the programmers who already possess the theory [...] program revival, that is reestablishing the theory of a program merely from the documentation, is strictly impossible.<\/p>\n<\/blockquote>\n<p>i do not think it is true that it is impossible to recover a theory of the program merely from the code and docs. my day job, and indeed one of my most prized skills when i interview for jobs, is creating a theory of programs from their text and documentation alone. this blog post is about how i do that, and how you can too.<\/p>\n<h2 id=\"theory-modification\">theory modification<a class=\"zola-anchor\" href=\"#theory-modification\" aria-label=\"Anchor link for: theory-modification\"><\/a>\n<\/h2>\n<p>Naur also says in the article:<\/p>\n<blockquote>\n<p>\u201cin a certain sense there can be no question of theory modification, only program modification\u201d<\/p>\n<\/blockquote>\n<p>i think this is wrong: theory modification is exactly what Ward Cunningham describes as \"consolidation\" in his 1992 article on <a href=\"https:\/\/c2.com\/doc\/oopsla92.html\">Technical Debt<\/a>. i highly recommend the original article, but the basic idea is that over time, your understanding of how the program <em>should<\/em> behave changes, and you modify and refactor your program to match that idea. this happens in all programs, but the modification is easier in programs with little <a href=\"https:\/\/jyn.dev\/technical-debt-is-different-from-technical-risk\/\">technical risk<\/a>.<\/p>\n<p>furthermore, this theory modification often happens unintentionally over time as people are added and removed from teams. as <a href=\"https:\/\/blog.ceejbot.com\/posts\/dysfunction-junction\/#fn:1\">ceejbot puts it<\/a>:<\/p>\n<blockquote>\n<p>This is Conway\u2019s Law over time. Teams are immutable: adding or removing a person to a team produces a different team. After enough change, the team is different enough that it no longer recognizes itself in the software system it produces. The result is people being vaguely unhappy about software that might be working perfectly well.<\/p>\n<\/blockquote>\n<p>i bring this up to note that you will never recover the <em>same<\/em> theory as the original programmers (at least, not without talking to them directly). the most you can do is to recover one similar enough that it does not require large changes to the program. in other words, you are creating a new theory of the program, and may end up having to adapt the program to your new theory.<\/p>\n<h2 id=\"recreating-a-theory\">recreating a theory<a class=\"zola-anchor\" href=\"#recreating-a-theory\" aria-label=\"Anchor link for: recreating-a-theory\"><\/a>\n<\/h2>\n<p>this is useful both when fixing bugs and when adding new features; i will focus on new features because i want to emphasize that these skills are useful any time you modify a program. for a focus on debugging, see Julia Evans' <a href=\"https:\/\/wizardzines.com\/zines\/debugging-guide\/\">Pocket Guide to Debugging<\/a>.<\/p>\n<p>this post is about creating theories at the \"micro\" level, for small portions of the program. i hope to make a post about the \"macro\" level in the future, since that's what really lets you start making design decisions about a program.<\/p>\n<aside role=note class=note-container><div class=note-content>\n<p>i recently made a <a href=\"https:\/\/github.com\/neovim\/neovim\/pull\/33339\/\">PR to neovim<\/a>, having never worked on neovim before; i'll use that as an example going forward.\ni highly recommend following along with a piece of code you want to learn more about. if you don't have one in mind, i have hidden all the examples behind a drop-down menu, so you can try to apply the ideas on your own before seeing how i use them.<\/p>\n<p>the investigation i did in this blog post was based off <a href=\"https:\/\/github.com\/neovim\/neovim\/tree\/57d99a515f57454370b6c122545ea53685d22d1b\">neovim commit 57d99a5<\/a>.<\/p>\n<p><button id=expandAll>Click here to open all notes.<\/button><\/p>\n<\/div><\/aside>\n<h3 id=\"where-to-start\">where to start<a class=\"zola-anchor\" href=\"#where-to-start\" aria-label=\"Anchor link for: where-to-start\"><\/a>\n<\/h3>\n<p>to start off, you need an idea of what change you want to make to the program. almost always, programs are too large for you to get an idea of the whole program at once. instead, you need to focus on theory-building for the parts you care about, and only understand the rest of the program to the extent that the parts you care about interact with it.<\/p>\n<aside role=note class=note-container><div class=note-content>\n<p>in my neovim PR, i cared about the <a href=\"https:\/\/neovim.io\/doc\/user\/windows.html#%3Adrop\"><code>:drop<\/code><\/a> command, which opens a file if it isn't loaded, or switches to the relevant buffer if it is. specifically i wanted to extend the \"switch to the relevant buffer\" part to also respect <a href=\"https:\/\/neovim.io\/doc\/user\/editing.html#%2Bcmd\"><code>+cmd<\/code><\/a>, so that i could pass it a line number.<\/p>\n<\/div><\/aside>\n<h3 id=\"finding-the-parts-you-care-about\">finding the parts you care about<a class=\"zola-anchor\" href=\"#finding-the-parts-you-care-about\" aria-label=\"Anchor link for: finding-the-parts-you-care-about\"><\/a>\n<\/h3>\n<p>there are several ways to get started here. the simplest is just finding the relevant part of the code or docs\u2014if you can provoke an error that's related to the part of the code you're changing, you can search for that error directly. often, knowing <em>how<\/em> execution reaches that state is very helpful, which you can do by getting a backtrace. you can get backtraces for output from arbitrary programs with <a href=\"https:\/\/jade.fyi\/blog\/debugging-rr-children\/\">liberal use of rr<\/a>, but if you're debugging rustc specifically, there's actually a built-in flag for this, so you can just use <a href=\"https:\/\/rustc-dev-guide.rust-lang.org\/compiler-debugging.html#getting-a-backtrace-for-errors\"><code>rustc file.rs -Z treat-err-as-bug<\/code><\/a>.<\/p>\n<aside role=note class=note-container><details class=note-content><summary><code>:drop<\/code><\/summary>\n<p>for <code>:drop<\/code>, this didn't work: it was documented on <a href=\"https:\/\/neovim.io\/doc\/user\/windows.html#%3Adrop\">neovim's site<\/a>, but i didn't know a <code>:drop<\/code>-specific error to search for.<\/p>\n<\/details><\/aside>\n<p>if this doesn't print an error message, or if it's not possible to get a recording of the program, things are harder. you want to look for something you already know the name of; search for literal strings with that name, or substrings that might form part of a template.<\/p>\n<aside role=note class=note-container><details class=note-content><summary><code>:drop<\/code><\/summary>\n<p>for <code>:drop<\/code> i searched for the literal string <code>\"drop\"<\/code>, since <em>something<\/em> needs to parse commands and it's not super common for it to be on its own in a string. that pulled up the following hits:<\/p>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">$ rg &#39;&quot;drop&quot;&#39; src\n<\/span><span class=\"z-text z-plain\">src\/nvim\/ex_docmd.c:4196:  } else if (STRICMP(p, &quot;drop&quot;) == 0) {\n<\/span><span class=\"z-text z-plain\">src\/nvim\/ex_docmd.c:4213:    &quot;drop&quot;,\n<\/span><span class=\"z-text z-plain\">src\/nvim\/ex_docmd.c:4302:    \/\/ &quot;drop&quot;.\n<\/span><span class=\"z-text z-plain\">src\/nvim\/eval.c:7146:    len += 7 + 4;  \/\/ &quot; ++bad=&quot; + &quot;keep&quot; or &quot;drop&quot;\n<\/span><\/code><\/pre>\n<p><code>ex_docmd.c<\/code> looked promising, so i read the code around there.<\/p>\n<\/details><\/aside>\n<h3 id=\"reading-source-code\">reading source code<a class=\"zola-anchor\" href=\"#reading-source-code\" aria-label=\"Anchor link for: reading-source-code\"><\/a>\n<\/h3>\n<p>sometimes triggering the condition is hard, so instead i read the source code to reverse-engineer the stack trace. seeing all possible call sites of a function is instructive in itself, and you can usually narrow it down to only a few callers by skimming what the callers are doing. i highly recommend using an LSP for this part since the advantage comes from seeing <em>all<\/em> possible callers, not just most, and regex is less reliable than proper name resolution.<\/p>\n<aside role=note class=note-container><details class=note-content><summary><code>:drop<\/code><\/summary>\n<p>it turned out that none of the code i found in my search was for <code>:drop<\/code> itself, but i did find it was in a function named <code>get_bad_opt<\/code>. <code>get_bad_opt<\/code> had only one caller, <code>getargopt<\/code>. that was called by <code>do_one_cmd<\/code>. the doc-comment on <code>do_one_cmd<\/code> mentions that it parses the string, but i am not used to having documentation so i went up one level too far to <code>do_cmdline<\/code>. at that point, looking at the call site of <code>do_one_cmd<\/code>, i realized i had gone too far because it was passing in the whole string of the Ex command line. i found a more relevant part of the code by looking at the uses of <code>cmdlinep<\/code> in <code>do_one_cmd<\/code>:<\/p>\n<pre data-lang=\"c\" class=\"language-c z-code\"><code class=\"language-c\" data-lang=\"c\"><span class=\"z-source z-c\">      <span class=\"z-storage z-type z-c\">char<\/span> <span class=\"z-keyword z-operator z-c\">*<\/span>cmdname <span class=\"z-keyword z-operator z-assignment z-c\">=<\/span> after_modifier <span class=\"z-keyword z-operator z-ternary z-c\">?<\/span> after_modifier <span class=\"z-keyword z-operator z-ternary z-c\">:<\/span> <span class=\"z-keyword z-operator z-c\">*<\/span>cmdlinep<span class=\"z-punctuation z-terminator z-c\">;<\/span>\n<\/span><\/code><\/pre>\n<p>i got lucky - this was not actually the code i cared about, but the bit i did care about had a similar name, so i found it by searching for <code>cmdname<\/code>:<\/p>\n<pre data-lang=\"c\" class=\"language-c z-code\"><code class=\"language-c\" data-lang=\"c\"><span class=\"z-source z-c\">  <span class=\"z-comment z-line z-double-slash z-c\"><span class=\"z-punctuation z-definition z-comment z-c\">\/\/<\/span> 6. Parse arguments.  Then check for errors.\n<\/span><\/span><span class=\"z-source z-c\">  <span class=\"z-keyword z-control z-c\">if<\/span> <span class=\"z-meta z-group z-c\"><span class=\"z-punctuation z-section z-group z-begin z-c\">(<\/span><span class=\"z-keyword z-operator z-arithmetic z-c\">!<\/span><span class=\"z-meta z-function-call z-c\"><span class=\"z-variable z-function z-c\">IS_USER_CMDIDX<\/span><span class=\"z-meta z-group z-c\"><span class=\"z-punctuation z-section z-group z-begin z-c\">(<\/span><\/span><\/span><span class=\"z-meta z-function-call z-c\"><span class=\"z-meta z-group z-c\">ea<span class=\"z-punctuation z-accessor z-c\">.<\/span><span class=\"z-variable z-other z-member z-c\">cmdidx<\/span><\/span><\/span><span class=\"z-meta z-function-call z-c\"><span class=\"z-meta z-group z-c\"><span class=\"z-punctuation z-section z-group z-end z-c\">)<\/span><\/span><\/span><span class=\"z-punctuation z-section z-group z-end z-c\">)<\/span><\/span> <span class=\"z-meta z-block z-c\"><span class=\"z-punctuation z-section z-block z-begin z-c\">{<\/span>\n<\/span><\/span><span class=\"z-source z-c\"><span class=\"z-meta z-block z-c\">    ea<span class=\"z-punctuation z-accessor z-c\">.<\/span><span class=\"z-variable z-other z-member z-c\">argt<\/span> <span class=\"z-keyword z-operator z-assignment z-c\">=<\/span> cmdnames<span class=\"z-meta z-brackets z-c\"><span class=\"z-punctuation z-section z-brackets z-begin z-c\">[<\/span><span class=\"z-meta z-group z-c\"><span class=\"z-punctuation z-section z-group z-begin z-c\">(<\/span><span class=\"z-storage z-type z-c\">int<\/span><span class=\"z-punctuation z-section z-group z-end z-c\">)<\/span><\/span>ea<span class=\"z-punctuation z-accessor z-c\">.<\/span><span class=\"z-variable z-other z-member z-c\">cmdidx<\/span><span class=\"z-punctuation z-section z-brackets z-end z-c\">]<\/span><\/span><span class=\"z-punctuation z-accessor z-c\">.<\/span><span class=\"z-variable z-other z-member z-c\">cmd_argt<\/span><span class=\"z-punctuation z-terminator z-c\">;<\/span>\n<\/span><\/span><span class=\"z-source z-c\"><span class=\"z-meta z-block z-c\">  <span class=\"z-punctuation z-section z-block z-end z-c\">}<\/span><\/span>\n<\/span><\/code><\/pre>\n<p>from there i went to the definition of <code>cmdnames<\/code> (in <code>build\/src\/nvim\/auto\/ex_cmds_defs.generated.h<\/code>) and found <code>\"drop\"<\/code> in that file:<\/p>\n<pre data-lang=\"c\" class=\"language-c z-code\"><code class=\"language-c\" data-lang=\"c\"><span class=\"z-source z-c\">  <span class=\"z-meta z-brackets z-c\"><span class=\"z-punctuation z-section z-brackets z-begin z-c\">[<\/span>CMD_drop<span class=\"z-punctuation z-section z-brackets z-end z-c\">]<\/span><\/span> <span class=\"z-keyword z-operator z-assignment z-c\">=<\/span> <span class=\"z-meta z-block z-c\"><span class=\"z-punctuation z-section z-block z-begin z-c\">{<\/span>\n<\/span><\/span><span class=\"z-source z-c\"><span class=\"z-meta z-block z-c\">    <span class=\"z-punctuation z-accessor z-c\">.<\/span><span class=\"z-variable z-other z-member z-c\">cmd_name<\/span> <span class=\"z-keyword z-operator z-assignment z-c\">=<\/span> <span class=\"z-string z-quoted z-double z-c\"><span class=\"z-punctuation z-definition z-string z-begin z-c\">&quot;<\/span>drop<span class=\"z-punctuation z-definition z-string z-end z-c\">&quot;<\/span><\/span><span class=\"z-punctuation z-separator z-c\">,<\/span>\n<\/span><\/span><span class=\"z-source z-c\"><span class=\"z-meta z-block z-c\">    <span class=\"z-punctuation z-accessor z-c\">.<\/span><span class=\"z-variable z-other z-member z-c\">cmd_func<\/span> <span class=\"z-keyword z-operator z-assignment z-c\">=<\/span> <span class=\"z-meta z-group z-c\"><span class=\"z-punctuation z-section z-group z-begin z-c\">(<\/span>ex_func_T<span class=\"z-punctuation z-section z-group z-end z-c\">)<\/span><\/span><span class=\"z-keyword z-operator z-c\">&amp;<\/span>ex_drop<span class=\"z-punctuation z-separator z-c\">,<\/span>\n<\/span><\/span><span class=\"z-source z-c\"><span class=\"z-meta z-block z-c\">    <span class=\"z-punctuation z-accessor z-c\">.<\/span><span class=\"z-variable z-other z-member z-c\">cmd_preview_func<\/span> <span class=\"z-keyword z-operator z-assignment z-c\">=<\/span> <span class=\"z-constant z-language z-c\">NULL<\/span><span class=\"z-punctuation z-separator z-c\">,<\/span>\n<\/span><\/span><span class=\"z-source z-c\"><span class=\"z-meta z-block z-c\">    <span class=\"z-punctuation z-accessor z-c\">.<\/span><span class=\"z-variable z-other z-member z-c\">cmd_argt<\/span> <span class=\"z-keyword z-operator z-assignment z-c\">=<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-c\">147854<span class=\"z-storage z-type z-numeric z-c\">L<\/span><\/span><span class=\"z-punctuation z-separator z-c\">,<\/span>\n<\/span><\/span><span class=\"z-source z-c\"><span class=\"z-meta z-block z-c\">    <span class=\"z-punctuation z-accessor z-c\">.<\/span><span class=\"z-variable z-other z-member z-c\">cmd_addr_type<\/span> <span class=\"z-keyword z-operator z-assignment z-c\">=<\/span> ADDR_NONE\n<\/span><\/span><span class=\"z-source z-c\"><span class=\"z-meta z-block z-c\">  <span class=\"z-punctuation z-section z-block z-end z-c\">}<\/span><\/span><span class=\"z-punctuation z-separator z-c\">,<\/span>\n<\/span><\/code><\/pre>\n<p>and from there found that the function i cared about was called <code>ex_drop<\/code>.<\/p>\n<p>if i had been a little more careful, i could have found <code>CMD_drop<\/code> sooner with <code>rg -ul '\"drop\"'<\/code> (this time without filtering out hidden files or limiting to the source directory). but this way worked fine as well.<\/p>\n<\/details><\/aside>\n<h3 id=\"verifying-your-understanding\">verifying your understanding<a class=\"zola-anchor\" href=\"#verifying-your-understanding\" aria-label=\"Anchor link for: verifying-your-understanding\"><\/a>\n<\/h3>\n<p>do mini experiments: if you see an error emitted in nearby code, try to trigger it so that you verify you're looking in the right place. when debugging, i often use process of elimination to narrow down callers: if an error would have been emitted if a certain code path was taken, or if there would have been more or less logging, i can be sure that code i am looking at was not run.<\/p>\n<p>the simplest experiment is just <code>exit(1)<\/code>; it's easy to notice and doesn't change the state of the program, and it can't fail. other experiments could include \"adding custom logging\" or \"change the behavior of the function\", which let you perform multiple experiments at once and understand how the function impacts its callers.<\/p>\n<p>for more complicated code, i like to use a debugger, which lets you see much more of the state at once. if possible, in-editor debuggers are really nice\u2014vscode, and <a href=\"https:\/\/zed.dev\/debugger\">since recently, zed<\/a>, have one built-in; for nvim i use <a href=\"https:\/\/github.com\/rcarriga\/nvim-dap-ui\">nvim-dap-ui<\/a>. you can also just use a debugger in a terminal. some experiments i like to try:<\/p>\n<ul>\n<li>breaking at a function to make sure it is executed<\/li>\n<li>printing local variables<\/li>\n<li>setting hardware watchpoints on memory to see where something is modified (this especially shines <a href=\"https:\/\/rr-project.org\/#:~:text=more%20powerful%20is%20reverse%20execution\">in combination with a time-travel debugger<\/a>)<\/li>\n<\/ul>\n<aside role=note class=note-container><details class=note-content><summary><code>:drop<\/code><\/summary>\n<p>for <code>:drop<\/code>, i was quite confident i had found the right code, so i didn't bother with any experiments. there are other cases where it's more useful; i made an earlier <a href=\"https:\/\/github.com\/tmux\/tmux\/pull\/4399\">PR to tmux<\/a> where there were many different places search happened, so verifying i was looking at the right one was very helpful. specifically i added <code>exit(1)<\/code> to the function i thought was the right place, since debug logging in tmux is non-trivial to access.<\/p>\n<p>i rarely use a debugger for adding new code; mostly i use it for debugging existing code. programs complicated enough that i need a debugger just to understand control flow usually have a client\/server model that also makes them harder to debug, so i don't bother and just read the source code.<\/p>\n<\/details><\/aside>\n<h3 id=\"writing-new-code\">writing new code<a class=\"zola-anchor\" href=\"#writing-new-code\" aria-label=\"Anchor link for: writing-new-code\"><\/a>\n<\/h3>\n<p>reading source code is also useful for finding examples of how to use an API. often it handles edge cases you wouldn't know about by skimming, and uses helper functions that make your life simpler. your goal is to make your change as similar to the existing codebase as possible, both to reduce the risk of bugs and to increase the chance the maintainer likes your change.<\/p>\n<p>when i write new code, i will usually copy a small snippet from elsewhere in the codebase and modify it to my needs. i try to copy at most 10-15 lines; more than that indicates that i should try to reuse or create a higher-level API.<\/p>\n<aside role=note class=note-container><details class=note-content><summary><code>:drop<\/code><\/summary>\n<p>once in <code>ex_drop<\/code>, i skimmed the code and found a snippet looked like it was handling existing files:<\/p>\n<pre data-lang=\"c\" class=\"language-c z-code\"><code class=\"language-c\" data-lang=\"c\"><span class=\"z-source z-c\">  <span class=\"z-meta z-function-call z-c\"><span class=\"z-variable z-function z-c\">FOR_ALL_TAB_WINDOWS<\/span><span class=\"z-meta z-group z-c\"><span class=\"z-punctuation z-section z-group z-begin z-c\">(<\/span><\/span><\/span><span class=\"z-meta z-function-call z-c\"><span class=\"z-meta z-group z-c\">tp<span class=\"z-punctuation z-separator z-c\">,<\/span> wp<\/span><span class=\"z-meta z-group z-c\"><span class=\"z-punctuation z-section z-group z-end z-c\">)<\/span><\/span><\/span> <span class=\"z-meta z-block z-c\"><span class=\"z-punctuation z-section z-block z-begin z-c\">{<\/span>\n<\/span><\/span><span class=\"z-source z-c\"><span class=\"z-meta z-block z-c\">    <span class=\"z-keyword z-control z-c\">if<\/span> <span class=\"z-meta z-group z-c\"><span class=\"z-punctuation z-section z-group z-begin z-c\">(<\/span>wp<span class=\"z-punctuation z-accessor z-c\">-&gt;<\/span>w_buffer <span class=\"z-keyword z-operator z-comparison z-c\">==<\/span> buf<span class=\"z-punctuation z-section z-group z-end z-c\">)<\/span><\/span> <span class=\"z-meta z-block z-c\"><span class=\"z-punctuation z-section z-block z-begin z-c\">{<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-c\"><span class=\"z-meta z-block z-c\"><span class=\"z-meta z-block z-c\">      <span class=\"z-meta z-function-call z-c\"><span class=\"z-variable z-function z-c\">goto_tabpage_win<\/span><span class=\"z-meta z-group z-c\"><span class=\"z-punctuation z-section z-group z-begin z-c\">(<\/span><\/span><\/span><span class=\"z-meta z-function-call z-c\"><span class=\"z-meta z-group z-c\">tp<span class=\"z-punctuation z-separator z-c\">,<\/span> wp<\/span><\/span><span class=\"z-meta z-function-call z-c\"><span class=\"z-meta z-group z-c\"><span class=\"z-punctuation z-section z-group z-end z-c\">)<\/span><\/span><\/span><span class=\"z-punctuation z-terminator z-c\">;<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-c\"><span class=\"z-meta z-block z-c\"><span class=\"z-meta z-block z-c\">      curwin<span class=\"z-punctuation z-accessor z-c\">-&gt;<\/span>w_arg_idx <span class=\"z-keyword z-operator z-assignment z-c\">=<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-c\">0<\/span><span class=\"z-punctuation z-terminator z-c\">;<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-c\"><span class=\"z-meta z-block z-c\"><span class=\"z-meta z-block z-c\">      <span class=\"z-keyword z-control z-c\">if<\/span> <span class=\"z-meta z-group z-c\"><span class=\"z-punctuation z-section z-group z-begin z-c\">(<\/span><span class=\"z-keyword z-operator z-arithmetic z-c\">!<\/span><span class=\"z-meta z-function-call z-c\"><span class=\"z-variable z-function z-c\">bufIsChanged<\/span><span class=\"z-meta z-group z-c\"><span class=\"z-punctuation z-section z-group z-begin z-c\">(<\/span><\/span><\/span><span class=\"z-meta z-function-call z-c\"><span class=\"z-meta z-group z-c\">curbuf<\/span><\/span><span class=\"z-meta z-function-call z-c\"><span class=\"z-meta z-group z-c\"><span class=\"z-punctuation z-section z-group z-end z-c\">)<\/span><\/span><\/span><span class=\"z-punctuation z-section z-group z-end z-c\">)<\/span><\/span> <span class=\"z-meta z-block z-c\"><span class=\"z-punctuation z-section z-block z-begin z-c\">{<\/span>\n<\/span><\/span><\/span><\/span><span class=\"z-source z-c\"><span class=\"z-meta z-block z-c\"><span class=\"z-meta z-block z-c\"><span class=\"z-meta z-block z-c\">        <span class=\"z-storage z-modifier z-c\">const<\/span> <span class=\"z-storage z-type z-c\">int<\/span> save_ar <span class=\"z-keyword z-operator z-assignment z-c\">=<\/span> curbuf<span class=\"z-punctuation z-accessor z-c\">-&gt;<\/span>b_p_ar<span class=\"z-punctuation z-terminator z-c\">;<\/span>\n<\/span><\/span><\/span><\/span><span class=\"z-source z-c\"><span class=\"z-meta z-block z-c\"><span class=\"z-meta z-block z-c\"><span class=\"z-meta z-block z-c\">\n<\/span><\/span><\/span><\/span><span class=\"z-source z-c\"><span class=\"z-meta z-block z-c\"><span class=\"z-meta z-block z-c\"><span class=\"z-meta z-block z-c\">        <span class=\"z-comment z-line z-double-slash z-c\"><span class=\"z-punctuation z-definition z-comment z-c\">\/\/<\/span> reload the file if it is newer\n<\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-c\"><span class=\"z-meta z-block z-c\"><span class=\"z-meta z-block z-c\"><span class=\"z-meta z-block z-c\">        curbuf<span class=\"z-punctuation z-accessor z-c\">-&gt;<\/span>b_p_ar <span class=\"z-keyword z-operator z-assignment z-c\">=<\/span> <span class=\"z-constant z-language z-c\">true<\/span><span class=\"z-punctuation z-terminator z-c\">;<\/span>\n<\/span><\/span><\/span><\/span><span class=\"z-source z-c\"><span class=\"z-meta z-block z-c\"><span class=\"z-meta z-block z-c\"><span class=\"z-meta z-block z-c\">        <span class=\"z-meta z-function-call z-c\"><span class=\"z-variable z-function z-c\">buf_check_timestamp<\/span><span class=\"z-meta z-group z-c\"><span class=\"z-punctuation z-section z-group z-begin z-c\">(<\/span><\/span><\/span><span class=\"z-meta z-function-call z-c\"><span class=\"z-meta z-group z-c\">curbuf<\/span><\/span><span class=\"z-meta z-function-call z-c\"><span class=\"z-meta z-group z-c\"><span class=\"z-punctuation z-section z-group z-end z-c\">)<\/span><\/span><\/span><span class=\"z-punctuation z-terminator z-c\">;<\/span>\n<\/span><\/span><\/span><\/span><span class=\"z-source z-c\"><span class=\"z-meta z-block z-c\"><span class=\"z-meta z-block z-c\"><span class=\"z-meta z-block z-c\">        curbuf<span class=\"z-punctuation z-accessor z-c\">-&gt;<\/span>b_p_ar <span class=\"z-keyword z-operator z-assignment z-c\">=<\/span> save_ar<span class=\"z-punctuation z-terminator z-c\">;<\/span>\n<\/span><\/span><\/span><\/span><span class=\"z-source z-c\"><span class=\"z-meta z-block z-c\"><span class=\"z-meta z-block z-c\"><span class=\"z-meta z-block z-c\">      <span class=\"z-punctuation z-section z-block z-end z-c\">}<\/span><\/span>\n<\/span><\/span><\/span><span class=\"z-source z-c\"><span class=\"z-meta z-block z-c\"><span class=\"z-meta z-block z-c\">      <span class=\"z-keyword z-control z-c\">if<\/span> <span class=\"z-meta z-group z-c\"><span class=\"z-punctuation z-section z-group z-begin z-c\">(<\/span>curbuf<span class=\"z-punctuation z-accessor z-c\">-&gt;<\/span>b_ml<span class=\"z-punctuation z-accessor z-c\">.<\/span><span class=\"z-variable z-other z-member z-c\">ml_flags<\/span> <span class=\"z-keyword z-operator z-c\">&amp;<\/span> ML_EMPTY<span class=\"z-punctuation z-section z-group z-end z-c\">)<\/span><\/span> <span class=\"z-meta z-block z-c\"><span class=\"z-punctuation z-section z-block z-begin z-c\">{<\/span>\n<\/span><\/span><\/span><\/span><span class=\"z-source z-c\"><span class=\"z-meta z-block z-c\"><span class=\"z-meta z-block z-c\"><span class=\"z-meta z-block z-c\">        <span class=\"z-meta z-function-call z-c\"><span class=\"z-variable z-function z-c\">ex_rewind<\/span><span class=\"z-meta z-group z-c\"><span class=\"z-punctuation z-section z-group z-begin z-c\">(<\/span><\/span><\/span><span class=\"z-meta z-function-call z-c\"><span class=\"z-meta z-group z-c\">eap<\/span><\/span><span class=\"z-meta z-function-call z-c\"><span class=\"z-meta z-group z-c\"><span class=\"z-punctuation z-section z-group z-end z-c\">)<\/span><\/span><\/span><span class=\"z-punctuation z-terminator z-c\">;<\/span>\n<\/span><\/span><\/span><\/span><span class=\"z-source z-c\"><span class=\"z-meta z-block z-c\"><span class=\"z-meta z-block z-c\"><span class=\"z-meta z-block z-c\">      <span class=\"z-punctuation z-section z-block z-end z-c\">}<\/span><\/span>\n<\/span><\/span><\/span><span class=\"z-source z-c\"><span class=\"z-meta z-block z-c\"><span class=\"z-meta z-block z-c\">      <span class=\"z-keyword z-control z-flow z-return z-c\">return<\/span><span class=\"z-punctuation z-terminator z-c\">;<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-c\"><span class=\"z-meta z-block z-c\"><span class=\"z-meta z-block z-c\">    <span class=\"z-punctuation z-section z-block z-end z-c\">}<\/span><\/span>\n<\/span><\/span><span class=\"z-source z-c\"><span class=\"z-meta z-block z-c\">  <span class=\"z-punctuation z-section z-block z-end z-c\">}<\/span><\/span>\n<\/span><\/code><\/pre>\n<p>the bug here is not any code that is present; instead it's code that's missing. i had to figure out where <code>+cmd<\/code> was stored and how to process it.\nso, i repeated a similar process for <code>+cmd<\/code>. this time i had something more to start with - i knew the command structure was named <code>eap<\/code> and had type <code>exarg_t<\/code>.\nlooking at the definition of <code>struct exarg<\/code> showed me what i wanted:<\/p>\n<pre data-lang=\"sh\" class=\"language-sh z-code\"><code class=\"language-sh\" data-lang=\"sh\"><span class=\"z-source z-shell z-bash\">  <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">char<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> <span class=\"z-keyword z-operator z-regexp z-quantifier z-shell\">*<\/span>do_ecmd_cmd<\/span><span class=\"z-keyword z-operator z-logical z-continue z-shell\">;<\/span>            <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">\/\/\/<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-keyword z-operator z-assignment z-redirection z-shell\">&lt;<\/span> +command arg to be used in edited file<\/span>\n<\/span><\/code><\/pre>\n<p>looking for <code>do_ecmd_cmd<\/code>, i found <code>do_exbuffer<\/code> (with a helpful comment saying it was responsible for <code>:buffer<\/code>) which called <code>do_cmdline_cmd<\/code>, and in turn <code>do_cmdline<\/code>.\nlooking at the callers of <code>do_cmdline<\/code> i found <code>do_ecmd<\/code>, which handles <code>:edit<\/code>. <code>:edit<\/code> has exactly the behavior i wanted for <code>:drop<\/code>, so i copied its behavior:<\/p>\n<pre data-lang=\"c\" class=\"language-c z-code\"><code class=\"language-c\" data-lang=\"c\"><span class=\"z-source z-c\">    <span class=\"z-meta z-function-call z-c\"><span class=\"z-variable z-function z-c\">do_cmdline<\/span><span class=\"z-meta z-group z-c\"><span class=\"z-punctuation z-section z-group z-begin z-c\">(<\/span><\/span><\/span><span class=\"z-meta z-function-call z-c\"><span class=\"z-meta z-group z-c\">command<span class=\"z-punctuation z-separator z-c\">,<\/span> <span class=\"z-constant z-language z-c\">NULL<\/span><span class=\"z-punctuation z-separator z-c\">,<\/span> <span class=\"z-constant z-language z-c\">NULL<\/span><span class=\"z-punctuation z-separator z-c\">,<\/span> DOCMD_VERBOSE<\/span><span class=\"z-meta z-group z-c\"><span class=\"z-punctuation z-section z-group z-end z-c\">)<\/span><\/span><\/span><span class=\"z-punctuation z-terminator z-c\">;<\/span>\n<\/span><\/code><\/pre>\n<p>out of caution, i also looked at the other places in the function that handled <code>command<\/code>, and it's a good thing i did, because i found this wild snippet above:<\/p>\n<pre data-lang=\"c\" class=\"language-c z-code\"><code class=\"language-c\" data-lang=\"c\"><span class=\"z-source z-c\">  <span class=\"z-keyword z-control z-c\">if<\/span> <span class=\"z-meta z-group z-c\"><span class=\"z-punctuation z-section z-group z-begin z-c\">(<\/span><span class=\"z-meta z-group z-c\"><span class=\"z-punctuation z-section z-group z-begin z-c\">(<\/span>command <span class=\"z-keyword z-operator z-comparison z-c\">!=<\/span> <span class=\"z-constant z-language z-c\">NULL<\/span> <span class=\"z-keyword z-operator z-arithmetic z-c\">||<\/span> newlnum <span class=\"z-keyword z-operator z-comparison z-c\">&gt;<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-c\">0<\/span><span class=\"z-punctuation z-section z-group z-end z-c\">)<\/span><\/span>\n<\/span><\/span><span class=\"z-source z-c\"><span class=\"z-meta z-group z-c\">      <span class=\"z-keyword z-operator z-arithmetic z-c\">&amp;&amp;<\/span> <span class=\"z-keyword z-operator z-c\">*<\/span><span class=\"z-meta z-function-call z-c\"><span class=\"z-variable z-function z-c\">get_vim_var_str<\/span><span class=\"z-meta z-group z-c\"><span class=\"z-punctuation z-section z-group z-begin z-c\">(<\/span><\/span><\/span><span class=\"z-meta z-function-call z-c\"><span class=\"z-meta z-group z-c\">VV_SWAPCOMMAND<\/span><\/span><span class=\"z-meta z-function-call z-c\"><span class=\"z-meta z-group z-c\"><span class=\"z-punctuation z-section z-group z-end z-c\">)<\/span><\/span><\/span> <span class=\"z-keyword z-operator z-comparison z-c\">==<\/span> NUL<span class=\"z-punctuation z-section z-group z-end z-c\">)<\/span><\/span> <span class=\"z-meta z-block z-c\"><span class=\"z-punctuation z-section z-block z-begin z-c\">{<\/span>\n<\/span><\/span><span class=\"z-source z-c\"><span class=\"z-meta z-block z-c\">    <span class=\"z-comment z-line z-double-slash z-c\"><span class=\"z-punctuation z-definition z-comment z-c\">\/\/<\/span> Set v:swapcommand for the SwapExists autocommands.\n<\/span><\/span><\/span><span class=\"z-source z-c\"><span class=\"z-meta z-block z-c\">    <span class=\"z-storage z-modifier z-c\">const<\/span> <span class=\"z-support z-type z-sys-types z-c\">size_t<\/span> len <span class=\"z-keyword z-operator z-assignment z-c\">=<\/span> <span class=\"z-meta z-group z-c\"><span class=\"z-punctuation z-section z-group z-begin z-c\">(<\/span>command <span class=\"z-keyword z-operator z-comparison z-c\">!=<\/span> <span class=\"z-constant z-language z-c\">NULL<\/span><span class=\"z-punctuation z-section z-group z-end z-c\">)<\/span><\/span> <span class=\"z-keyword z-operator z-ternary z-c\">?<\/span> <span class=\"z-meta z-function-call z-c\"><span class=\"z-support z-function z-C99 z-c\">strlen<\/span><span class=\"z-meta z-group z-c\"><span class=\"z-punctuation z-section z-group z-begin z-c\">(<\/span><\/span><\/span><span class=\"z-meta z-function-call z-c\"><span class=\"z-meta z-group z-c\">command<\/span><\/span><span class=\"z-meta z-function-call z-c\"><span class=\"z-meta z-group z-c\"><span class=\"z-punctuation z-section z-group z-end z-c\">)<\/span><\/span><\/span> <span class=\"z-keyword z-operator z-arithmetic z-c\">+<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-c\">3<\/span> <span class=\"z-keyword z-operator z-ternary z-c\">:<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-c\">30<\/span><span class=\"z-punctuation z-terminator z-c\">;<\/span>\n<\/span><\/span><span class=\"z-source z-c\"><span class=\"z-meta z-block z-c\">    <span class=\"z-storage z-type z-c\">char<\/span> <span class=\"z-keyword z-operator z-c\">*<\/span><span class=\"z-storage z-modifier z-c\">const<\/span> p <span class=\"z-keyword z-operator z-assignment z-c\">=<\/span> <span class=\"z-meta z-function-call z-c\"><span class=\"z-variable z-function z-c\">xmalloc<\/span><span class=\"z-meta z-group z-c\"><span class=\"z-punctuation z-section z-group z-begin z-c\">(<\/span><\/span><\/span><span class=\"z-meta z-function-call z-c\"><span class=\"z-meta z-group z-c\">len<\/span><\/span><span class=\"z-meta z-function-call z-c\"><span class=\"z-meta z-group z-c\"><span class=\"z-punctuation z-section z-group z-end z-c\">)<\/span><\/span><\/span><span class=\"z-punctuation z-terminator z-c\">;<\/span>\n<\/span><\/span><span class=\"z-source z-c\"><span class=\"z-meta z-block z-c\">    <span class=\"z-keyword z-control z-c\">if<\/span> <span class=\"z-meta z-group z-c\"><span class=\"z-punctuation z-section z-group z-begin z-c\">(<\/span>command <span class=\"z-keyword z-operator z-comparison z-c\">!=<\/span> <span class=\"z-constant z-language z-c\">NULL<\/span><span class=\"z-punctuation z-section z-group z-end z-c\">)<\/span><\/span> <span class=\"z-meta z-block z-c\"><span class=\"z-punctuation z-section z-block z-begin z-c\">{<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-c\"><span class=\"z-meta z-block z-c\"><span class=\"z-meta z-block z-c\">      <span class=\"z-meta z-function-call z-c\"><span class=\"z-variable z-function z-c\">vim_snprintf<\/span><span class=\"z-meta z-group z-c\"><span class=\"z-punctuation z-section z-group z-begin z-c\">(<\/span><\/span><\/span><span class=\"z-meta z-function-call z-c\"><span class=\"z-meta z-group z-c\">p<span class=\"z-punctuation z-separator z-c\">,<\/span> len<span class=\"z-punctuation z-separator z-c\">,<\/span> <span class=\"z-string z-quoted z-double z-c\"><span class=\"z-punctuation z-definition z-string z-begin z-c\">&quot;<\/span>:<span class=\"z-constant z-other z-placeholder z-c\">%s<\/span><span class=\"z-constant z-character z-escape z-c\">\\r<\/span><span class=\"z-punctuation z-definition z-string z-end z-c\">&quot;<\/span><\/span><span class=\"z-punctuation z-separator z-c\">,<\/span> command<\/span><\/span><span class=\"z-meta z-function-call z-c\"><span class=\"z-meta z-group z-c\"><span class=\"z-punctuation z-section z-group z-end z-c\">)<\/span><\/span><\/span><span class=\"z-punctuation z-terminator z-c\">;<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-c\"><span class=\"z-meta z-block z-c\"><span class=\"z-meta z-block z-c\">    <span class=\"z-punctuation z-section z-block z-end z-c\">}<\/span><\/span> <span class=\"z-keyword z-control z-c\">else<\/span> <span class=\"z-meta z-block z-c\"><span class=\"z-punctuation z-section z-block z-begin z-c\">{<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-c\"><span class=\"z-meta z-block z-c\"><span class=\"z-meta z-block z-c\">      <span class=\"z-meta z-function-call z-c\"><span class=\"z-variable z-function z-c\">vim_snprintf<\/span><span class=\"z-meta z-group z-c\"><span class=\"z-punctuation z-section z-group z-begin z-c\">(<\/span><\/span><\/span><span class=\"z-meta z-function-call z-c\"><span class=\"z-meta z-group z-c\">p<span class=\"z-punctuation z-separator z-c\">,<\/span> len<span class=\"z-punctuation z-separator z-c\">,<\/span> <span class=\"z-string z-quoted z-double z-c\"><span class=\"z-punctuation z-definition z-string z-begin z-c\">&quot;<\/span>%<span class=\"z-punctuation z-definition z-string z-end z-c\">&quot;<\/span><\/span> PRId64 <span class=\"z-string z-quoted z-double z-c\"><span class=\"z-punctuation z-definition z-string z-begin z-c\">&quot;<\/span>G<span class=\"z-punctuation z-definition z-string z-end z-c\">&quot;<\/span><\/span><span class=\"z-punctuation z-separator z-c\">,<\/span> <span class=\"z-meta z-group z-c\"><span class=\"z-punctuation z-section z-group z-begin z-c\">(<\/span><span class=\"z-support z-type z-stdint z-c\">int64_t<\/span><span class=\"z-punctuation z-section z-group z-end z-c\">)<\/span><\/span>newlnum<\/span><\/span><span class=\"z-meta z-function-call z-c\"><span class=\"z-meta z-group z-c\"><span class=\"z-punctuation z-section z-group z-end z-c\">)<\/span><\/span><\/span><span class=\"z-punctuation z-terminator z-c\">;<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-c\"><span class=\"z-meta z-block z-c\"><span class=\"z-meta z-block z-c\">    <span class=\"z-punctuation z-section z-block z-end z-c\">}<\/span><\/span>\n<\/span><\/span><span class=\"z-source z-c\"><span class=\"z-meta z-block z-c\">    <span class=\"z-meta z-function-call z-c\"><span class=\"z-variable z-function z-c\">set_vim_var_string<\/span><span class=\"z-meta z-group z-c\"><span class=\"z-punctuation z-section z-group z-begin z-c\">(<\/span><\/span><\/span><span class=\"z-meta z-function-call z-c\"><span class=\"z-meta z-group z-c\">VV_SWAPCOMMAND<span class=\"z-punctuation z-separator z-c\">,<\/span> p<span class=\"z-punctuation z-separator z-c\">,<\/span> <span class=\"z-keyword z-operator z-arithmetic z-c\">-<\/span><span class=\"z-constant z-numeric z-integer z-decimal z-c\">1<\/span><\/span><\/span><span class=\"z-meta z-function-call z-c\"><span class=\"z-meta z-group z-c\"><span class=\"z-punctuation z-section z-group z-end z-c\">)<\/span><\/span><\/span><span class=\"z-punctuation z-terminator z-c\">;<\/span>\n<\/span><\/span><span class=\"z-source z-c\"><span class=\"z-meta z-block z-c\">    did_set_swapcommand <span class=\"z-keyword z-operator z-assignment z-c\">=<\/span> <span class=\"z-constant z-language z-c\">true<\/span><span class=\"z-punctuation z-terminator z-c\">;<\/span>\n<\/span><\/span><span class=\"z-source z-c\"><span class=\"z-meta z-block z-c\">    <span class=\"z-meta z-function-call z-c\"><span class=\"z-variable z-function z-c\">xfree<\/span><span class=\"z-meta z-group z-c\"><span class=\"z-punctuation z-section z-group z-begin z-c\">(<\/span><\/span><\/span><span class=\"z-meta z-function-call z-c\"><span class=\"z-meta z-group z-c\">p<\/span><\/span><span class=\"z-meta z-function-call z-c\"><span class=\"z-meta z-group z-c\"><span class=\"z-punctuation z-section z-group z-end z-c\">)<\/span><\/span><\/span><span class=\"z-punctuation z-terminator z-c\">;<\/span>\n<\/span><\/span><span class=\"z-source z-c\"><span class=\"z-meta z-block z-c\">  <span class=\"z-punctuation z-section z-block z-end z-c\">}<\/span><\/span>\n<\/span><\/code><\/pre>\n<p>i refactored this into a helper function and then called it from both the original <code>:edit<\/code> command and my new code in <code>:drop<\/code>.<\/p>\n<\/details><\/aside>\n<h3 id=\"testing-new-code\">testing new code<a class=\"zola-anchor\" href=\"#testing-new-code\" aria-label=\"Anchor link for: testing-new-code\"><\/a>\n<\/h3>\n<p>this works in much the same way. try to find existing tests by using <a href=\"https:\/\/jyn.dev\/theory-building-without-a-mentor\/#finding-the-parts-you-care-about\">the same techniques as finding the code you care about<\/a>. read them; write them using existing examples. tests are also code, after all.<\/p>\n<p>test suites usually have better documentation than the code itself, since adding new tests is much more common than modifying any particular section of code; see if you can find the docs. i look for <code>CONTRIBUTING.md<\/code> files, and if i don't find them i fall back to skimming the readme. sometimes there are is also a <code>README.md<\/code> in the folder where the tests are located, although these tend to be somewhat out of date.<\/p>\n<p>i care a lot about iteration times, so i try and find how to run individual tests. that info is usually in the README, or sometime you can figure it out from the test command's <code>--help<\/code> output.<\/p>\n<p>run your tests! ideally, create and run your tests <em>before<\/em> modifying the code so that you can see that they start to pass after your change. tests are extra important when you don't already understand the code, because they help you verify that your new theory is correct. run existing tests as well; run those before you make changes so you know which failures are spurious (a surprisingly high number of codebases have flaky or environment-dependent tests).<\/p>\n<aside role=note class=note-container><details class=note-content><summary><code>:drop<\/code><\/summary>\n<p>i started by looking for existing tests for <code>:drop<\/code>:<\/p>\n<pre data-lang=\"sh\" class=\"language-sh z-code\"><code class=\"language-sh\" data-lang=\"sh\"><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">$<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> rg :drop test<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">test\/functional\/ex_cmds\/drop_spec.lua:8:describe<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\">(<span class=\"z-string z-quoted z-single z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&#39;<\/span>:drop<span class=\"z-punctuation z-definition z-string z-end z-shell\">&#39;<\/span><\/span>, function(<\/span><span class=\"z-meta z-function-call z-shell\"><\/span>)\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">test\/functional\/ex_cmds\/drop_spec.lua:75:<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\">      :drop Xdrop_modified.txt<\/span>           <span class=\"z-keyword z-operator z-logical z-pipe z-shell\">|<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">test\/old\/testdir\/test_winfixbuf.vim:1094:<span class=\"z-string z-quoted z-double z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&quot;<\/span> Fail :drop but :drop! is allowed\n<\/span><\/span><\/span><\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\"><span class=\"z-string z-quoted z-double z-shell\">test\/old\/testdir\/test_excmd.vim:92:<span class=\"z-punctuation z-definition z-string z-end z-shell\">&quot;<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> Test for the :drop command<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">test\/old\/testdir\/test_excmd.vim:775:<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\">  call term_sendkeys(buf, <span class=\"z-string z-quoted z-double z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&quot;<\/span>:drop Xdrop_modified.txt\\&lt;CR&gt;<span class=\"z-punctuation z-definition z-string z-end z-shell\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-function-call z-shell\"><\/span>)\n<\/span><\/code><\/pre>\n<p>fortunately this had results right away and i was able to start adding my new test. <code>CONTRIBUTING.md<\/code> had a pointer to <code>test\/README.md<\/code> which documented <code>TEST_FILE<\/code> and <code>mak functionaltest<\/code>. neovim has very good internal tooling and when my <code>screen:expect()<\/code> call failed it gave me a very helpful pointer to <code>screen:snapshot_util<\/code>.<\/p>\n<\/details><\/aside>\n<h2 id=\"what-have-we-learned\">what have we learned?<a class=\"zola-anchor\" href=\"#what-have-we-learned\" aria-label=\"Anchor link for: what-have-we-learned\"><\/a>\n<\/h2>\n<ul>\n<li><a href=\"https:\/\/gist.github.com\/onlurking\/fc5c81d18cfce9ff81bc968a7f342fb1\">programming is theory building<\/a>.<\/li>\n<li>recovering a theory from code and docs alone is hard, but possible.<\/li>\n<li>most programs are too large for you to understand them all at once. decide on your goal and learn just enough to accomplish it.<\/li>\n<li>reading source code is surprisingly rewarding.<\/li>\n<li>match the existing code as closely as you can until you are sure you have a working theory.<\/li>\n<\/ul>\n<p>hopefully this was helpful! i am told by my friends that i am unusually good at this skill, so i am interested whether this post was effective at teaching it. if you have any questions, or if you just want to get in contact, feel free to reach out <a href=\"mailto:blog@jyn.dev\">via email<\/a>.<\/p>\n<!--say you want to change the output (e.g. you are adding a new field to JSON emitted by the program). i like to start by breaking it down along interface boundaries: finding the code that creates the JSON, then breaking it down into smaller and smaller pieces, ideally to a single function. if there are multiple possible places that could have emitted it, see if you can determine which of them are reachable from your input. you can usually get an idea of which parts of the program are reachable statically just by reading the source code, even if you have no trace and aren't able to replicate locally.-->\n"},{"title":"technical debt is different from technical risk","published":"2025-05-23T00:00:00+00:00","updated":"2025-05-23T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/technical-debt-is-different-from-technical-risk\/"}},"id":"https:\/\/jyn.dev\/technical-debt-is-different-from-technical-risk\/","summary":"\"technical debt\" is about updating your understanding of the program over time. \"technical risk\" is about sacrificing your ability to make changes for speed of development in the short term.","content":"<h2 id=\"technical-debt-is-commonly-misunderstood\">technical debt is commonly misunderstood<a class=\"zola-anchor\" href=\"#technical-debt-is-commonly-misunderstood\" aria-label=\"Anchor link for: technical-debt-is-commonly-misunderstood\"><\/a>\n<\/h2>\n<p>the phrase \"technical debt\" at this point is very common in programming circles. however, i think the way this phrase is commonly used is misleading and in some cases actively harmful. here is a statement of the common usage by <a href=\"https:\/\/mastodon.pub.solar\/@mezza\/114219114805175449\">a random commenter on my fediverse posts<\/a>:<\/p>\n<blockquote>\n<p>tech debt is [...] debt in the literal sense that you took a shortcut to get to your product state. You've saved time by moving quick, and are now paying interest by development slowing down.<\/p>\n<\/blockquote>\n<p>contrast this to the original statement of technical debt in <a href=\"https:\/\/c2.com\/doc\/oopsla92.html\">Ward Cunningham's paper from 1992<\/a>:<\/p>\n<blockquote>\n<p>Shipping first time code is like going into debt. A little debt speeds development so long as it is paid back promptly with a rewrite. [...] The danger occurs when the debt is not repaid. Every minute spent on not-quite-right code counts as interest on that debt. Entire engineering organizations can be brought to a stand-still under the debt load of an unconsolidated implementation.<\/p>\n<\/blockquote>\n<p>Ward isn't comparing \"shortcuts\" or \"move fast and break things\" to \"good code\"\u2014he's comparing iterative design (often called \"agile\") to the waterfall method <sup class=\"footnote-reference\" id=\"fr-smalltalk-1\"><a href=\"#fn-smalltalk\">1<\/a><\/sup>:<\/p>\n<blockquote>\n<p>The traditional waterfall development cycle has endeavored to avoid programming catastrophy by working out a program in detail before programming begins. [...] However, using our debt analogy, we recognize this amounts to preserving the concept of payment up-front and in-full.<\/p>\n<\/blockquote>\n<p>Finally, I want to quote a <a href=\"https:\/\/wiki.c2.com\/?WardExplainsDebtMetaphor\">follow-up statement from Ward<\/a> in 2006, which is closely related to <a href=\"https:\/\/gist.github.com\/onlurking\/fc5c81d18cfce9ff81bc968a7f342fb1\">Programming As Theory Building<\/a> by Peter Naur:<\/p>\n<blockquote>\n<p>A lot of bloggers at least have explained the debt metaphor and confused it, I think, with the idea that you could write code poorly with the intention of doing a good job later and thinking that that was the primary source of debt. I'm never in favor of writing code poorly, but I am in favor of writing code to reflect your current understanding of a problem even if that understanding is partial.<\/p>\n<\/blockquote>\n<p>It seems pretty clear at this point that Ward is describing something different from the common usage (I think \"technical debt\" is a good term for Ward's original idea). What then can we call the common usage? I like <em>technical risk<\/em>.<\/p>\n<h2 id=\"technical-risk-means-a-program-is-hard-to-modify\">technical risk means a program is hard to modify<a class=\"zola-anchor\" href=\"#technical-risk-means-a-program-is-hard-to-modify\" aria-label=\"Anchor link for: technical-risk-means-a-program-is-hard-to-modify\"><\/a>\n<\/h2>\n<p>Whenever you modify a program's behavior, you incur a risk that you introduce a bug. Over time, as the code is used more, the number of bugs tends to decrease as you fix them. <a href=\"https:\/\/security.googleblog.com\/2024\/09\/eliminating-memory-safety-vulnerabilities-Android.html#:~:text=the%20math\">Two studies in 2021 and 2022<\/a> (one by the Android security team, one by Usenix security) found empirically that memory vulnerabilities decay <em>exponentially<\/em> over time. So you have an inherent tension between minimizing your changes so that your code gets less buggy over time and modifying your code so that your program becomes more useful.<\/p>\n<p>When people talk about \"technical debt\", what I am calling \"technical risk\", they mean \"modifying the code has a very high risk\"\u2014any kind of modification has a high chance of introducing bugs, not only when adding new features but also when doing refactors and bug fixes. Even the most trivial changes become painful and time-consuming, and the right tail of your time distribution increases dramatically.<\/p>\n<p>Furthermore, when we say \"this program has a lot of tech debt\", we are implicitly arguing \"the risk of a refactor is lower than the risk of it eventually breaking when we make <em>some other change<\/em>\". We are <em>gambling<\/em> that the risk of a refactor (either in time or breakage) is worth the decreased risk going forward.<\/p>\n<p>Note that you cannot overcome technical risk simply by spending more time; in this way it is unlike technical debt. With sufficient technical risk, simply predicting how long a change will take becomes hard. Due to the risk of regressions you must spend more time testing; but because of the complexity of the program, creating tests is also time-consuming, and it is less likely that you can test exhaustively, so there is a higher risk that your tests don't catch all regressions. Eventually changing the program without regressions becomes nearly impossible, and people fork or reimplement the program from scratch (what Ward describes as \"the interest is total\").<\/p>\n<h2 id=\"all-programs-have-risk\">all programs have risk<a class=\"zola-anchor\" href=\"#all-programs-have-risk\" aria-label=\"Anchor link for: all-programs-have-risk\"><\/a>\n<\/h2>\n<p>The common understanding of \"tech debt\" is that it only applies to programs that were built hastily or without planning. \"tech risk\" is much more broad than that, though.\nIt also applies to old programs which no longer have anyone that <a href=\"https:\/\/gist.github.com\/onlurking\/fc5c81d18cfce9ff81bc968a7f342fb1\">understands their theory<\/a>; new code if it's sufficiently complicated (stateful, non-local, \"just a complicated algorithm\", etc); and large programs that are too big for any one person to understand in full.\nIn fact, most code has some amount of risk, simply because it isn't usually worth making readability the number 1 priority (and readability differs from programmer to programmer).<\/p>\n<h2 id=\"bad-code-misses-the-point\">\"bad code\" misses the point<a class=\"zola-anchor\" href=\"#bad-code-misses-the-point\" aria-label=\"Anchor link for: bad-code-misses-the-point\"><\/a>\n<\/h2>\n<p>Hillel Wayne recently wrote a post titled <a href=\"https:\/\/buttondown.com\/hillelwayne\/archive\/write-the-most-clever-code-you-possibly-can\/\">Write the most clever code you possibly can<\/a>. At one point he says this:<\/p>\n<blockquote>\n<p>I've also talked to people who think that datatypes besides lists and hashmaps are too clever to use, that most optimizations are too clever to bother with, and even that functions and classes are too clever and code should be a linear script.<\/p>\n<\/blockquote>\n<p>This is an extreme example, but it reflects a pattern I often see: people think any code that uses complicated features is \"too clever\" and therefore bad.\nThis comes up a lot for \"weird\" syscalls or libc functions, like <code>setjmp<\/code>\/<code>longjmp<\/code> and <code>fork<\/code>.\nI think this misses the point. What makes something technical risk is the <em>risk<\/em>, the inertia when you try to modify it, the likelihood of bugs.\nHaving a steep learning curve is not the same as being hard to modify, because once you learn it once, future changes become easier.<\/p>\n<h2 id=\"feature-flags-are-the-taste-of-the-lotus\">feature flags are the taste of the lotus<a class=\"zola-anchor\" href=\"#feature-flags-are-the-taste-of-the-lotus\" aria-label=\"Anchor link for: feature-flags-are-the-taste-of-the-lotus\"><\/a>\n<\/h2>\n<p>Once your risk is high enough, and if you don't have the option of reducing complexity, people tend to work around the risk with feature flags or configuration options. These flags avoid the new behavior altogether in the default case, such that \"changing the program\" is decoupled from \"changing the behavior\".<\/p>\n<p>In my experience this can be good in moderation\u2014but if every new change requires a feature flag, and you never go back and remove old flags, then you're in trouble, because the flags themselves are adding complexity and risk. Each new change has to consider not just the default case, but all possible combinations of flags in a combinatorial explosion. You see this with things like tmux, OracleDB, and vim, all of which tend to accumulate options without removing them. Consider <a href=\"https:\/\/news.ycombinator.com\/item?id=18442941\">this quote<\/a> from someone who claimed to work at Oracle:<\/p>\n<blockquote>\n<p>Sometimes one needs to understand the values and the effects of 20 different flag to predict how the code would behave in different situations. Sometimes 100s too! I am not exaggerating. The only reason why this product is still surviving and still works is due to literally millions of tests!<\/p>\n<\/blockquote>\n<p>This is an extreme case, but in my experience it is absolutely representative of what happens to sufficiently large codebases over time. Once things are this bad you are \"locked in\" to the feature flag model\u2014there's too many to remove (and your users may be depending on them!), but you cannot understand the interactions of all their combinations, so you gate new changes behind more new flags just to be sure.<\/p>\n<h2 id=\"what-to-do-about-risk\">what to do about risk?<a class=\"zola-anchor\" href=\"#what-to-do-about-risk\" aria-label=\"Anchor link for: what-to-do-about-risk\"><\/a>\n<\/h2>\n<p>this post is kinda scary! it tells a story of codebases that grow more and more bogged down over time, despite people's best efforts, until they eventually die because they can't be modified.<\/p>\n<p>i think things are not actually so dire as they seem. firstly, you always have the option to do ongoing refactors, reducing the risk of changes. with ongoing maintenance like this, even extremely complex programs can be maintained for years or decades; i think <a href=\"https:\/\/rustc-dev-guide.rust-lang.org\/about-this-guide.html#constant-change\">the rust compiler<\/a> is a good example of such a program.<\/p>\n<p>secondly, rebuilding systems is good, actually, because it lets us learn from the lessons of the past. oracledb, tmux, and vim all have younger competitors (e.g. sqlite, zellij, and helix) that are more nimble. even more than that, new systems have the opportunity to be built on a different paradigm (e.g. sqlite runs in-process instead of as a daemon) with different tradeoffs. this is the classic case of <a href=\"https:\/\/en.wikipedia.org\/wiki\/Disruptive_innovation\">disruptive innovation<\/a>.<\/p>\n<p>to some extent, people or teams can get \"locked in\" to existing systems, especially if they are highly configurable or changing to a new system would be high risk for the organization (e.g. migrating to a new database is extremely high risk for almost anyone), but this can be mitigated by open file formats (such as <a href=\"https:\/\/www.sqlite.org\/fileformat2.html\">sqlite's database file<\/a>) and backwards compatibility for the old options (such as in neovim).<\/p>\n<h2 id=\"in-conclusion\">in conclusion<a class=\"zola-anchor\" href=\"#in-conclusion\" aria-label=\"Anchor link for: in-conclusion\"><\/a>\n<\/h2>\n<ul>\n<li>\"technical debt\" as commonly understood is different from its origins.<\/li>\n<li>the original \"technical debt\" referred to iterative design.<\/li>\n<li>the common meaning is about programs that are hard to change, and i refer to it as \"technical risk\".<\/li>\n<li>all programs have technical risk to greater or lesser degree; you can decrease it but never eliminate it altogether.<\/li>\n<li>once risk grows sufficiently high, changes become hard enough that they have to be gated behind feature flags. the program eventually stagnates and is rewritten.<\/li>\n<\/ul>\n<section class=\"footnotes\">\n<ol class=\"footnotes-list\">\n<li id=\"fn-smalltalk\">\n<p>Actually, if you read the post more closely, he is saying something even more interesting: iterative development is only possible <em>because<\/em> his company is using a language (smalltalk) that has privacy boundaries. <a href=\"#fr-smalltalk-1\">\u21a9<\/a><\/p>\n<\/li>\n<\/ol>\n<\/section>\n"},{"title":"sftp sandboxing","published":"2025-05-05T00:00:00+00:00","updated":"2025-05-05T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/sftp-sandboxing\/"}},"id":"https:\/\/jyn.dev\/sftp-sandboxing\/","summary":"unix sure is a system","content":"<p>consider the following problem:<\/p>\n<ul>\n<li>you want to share a 300 MB directory<\/li>\n<li>with a single other person, not publicly<\/li>\n<li>over a public network<\/li>\n<\/ul>\n<p>any hosting service that lets you serve 300 MB costs money. also, for sufficient file sizes the wasted upload becomes noticeable. it would be much nicer if we could peer-to-peer this.<\/p>\n<p>sftp to the rescue!<\/p>\n<p>small problem: sftp shows you a <em>lot<\/em> of the system state. for one thing, you have read access to basically every file on the system, and write access to anything owned by your user. for another, the easiest way to set up sftp is through an ssh server, which is. well. it has \"shell\" in the name for a reason.<\/p>\n<p>so! here is how to set up a read-only sftp server which doesn't allow any other kind of access.<\/p>\n<p>First, add the following to <code>\/etc\/ssh\/sshd_config<\/code>:<\/p>\n<pre data-lang=\"sshd\" class=\"language-sshd z-code\"><code class=\"language-sshd\" data-lang=\"sshd\"><span class=\"z-text z-plain\">Subsystem       sftp    internal-sftp\n<\/span><span class=\"z-text z-plain\">Match User myuser\n<\/span><span class=\"z-text z-plain\">\tChrootDirectory %h\n<\/span><span class=\"z-text z-plain\">\tForceCommand internal-sftp -R\n<\/span><span class=\"z-text z-plain\">\tDisableForwarding yes\n<\/span><\/code><\/pre>\n<p>Then run the following commands in a root shell (e.g. with <code>sudo -i<\/code>):<\/p>\n<pre data-lang=\"sh\" class=\"language-sh z-code\"><code class=\"language-sh\" data-lang=\"sh\"><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">mkdir<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> \/empty<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">chmod<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> a-w \/empty<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">useradd<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> -<\/span>s<\/span> \/usr\/sbin\/nologin<span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> -<\/span>m<\/span><span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> --<\/span>skel<\/span> \/empty myuser<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">chown<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> root:root \/home\/myuser<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">chmod<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> a+rx \/home\/myuser<\/span>\n<\/span><\/code><\/pre>\n<p>Finally, create <code>\/home\/myuser\/.ssh\/authorized_keys<\/code> in any way you wish<\/p>\n<p>This creates a sftp session that looks like this:<\/p>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">$ ssh myuser@localhost\n<\/span><span class=\"z-text z-plain\">This service allows sftp connections only.\n<\/span><span class=\"z-text z-plain\">Connection to localhost closed.\n<\/span><span class=\"z-text z-plain\">$ sftp myuser@localhost\n<\/span><span class=\"z-text z-plain\">Connected to localhost.\n<\/span><span class=\"z-text z-plain\">sftp&gt; pwd\n<\/span><span class=\"z-text z-plain\">Remote working directory: \/\n<\/span><span class=\"z-text z-plain\">sftp&gt; ls\n<\/span><span class=\"z-text z-plain\">myfile\n<\/span><span class=\"z-text z-plain\">sftp&gt; ls ..\n<\/span><span class=\"z-text z-plain\">..\/myfile\n<\/span><span class=\"z-text z-plain\">sftp&gt; ls ..\/..\/..\n<\/span><span class=\"z-text z-plain\">..\/..\/..\/myfile\n<\/span><span class=\"z-text z-plain\">sftp&gt; mkdir x\n<\/span><span class=\"z-text z-plain\">remote mkdir &quot;\/x&quot;: Permission denied\n<\/span><span class=\"z-text z-plain\">sftp&gt; chmod 644\tmyfile\n<\/span><span class=\"z-text z-plain\">Changing mode on \/myfile\n<\/span><span class=\"z-text z-plain\">remote setstat &quot;\/myfile&quot;: Permission denied\n<\/span><span class=\"z-text z-plain\">sftp&gt; get myfile\n<\/span><span class=\"z-text z-plain\">Fetching \/myfile to myfile\n<\/span><span class=\"z-text z-plain\">myfile       100%   46MB 899.5MB\/s   00:00\n<\/span><\/code><\/pre>\n<hr \/>\n<p>Some explanations and references:<\/p>\n<p><a href=\"https:\/\/man.openbsd.org\/cgi-bin\/man.cgi\/OpenBSD-current\/man5\/sshd_config.5#Subsystem\"><code>Subsystem<\/code><\/a> is documented as follows:<\/p>\n<blockquote>\n<p>Configures an external subsystem (e.g. file transfer daemon). [...] the name <code>internal-sftp<\/code> implements an in-process SFTP server. [...] It accepts the same command line arguments as <code>sftp-server<\/code> [...]<\/p>\n<\/blockquote>\n<p>what arguments can we pass to <a href=\"https:\/\/man.openbsd.org\/sftp-server#R\"><code>sftp-server<\/code><\/a>?<\/p>\n<blockquote>\n<p><code>-R<\/code>\tPlaces this instance of <code>sftp-server<\/code> into a read-only mode. Attempts to open files for writing, as well as other operations that change the state of the filesystem, will be denied.<\/p>\n<\/blockquote>\n<p>just what we want!<\/p>\n<p><a href=\"https:\/\/man.openbsd.org\/cgi-bin\/man.cgi\/OpenBSD-current\/man5\/sshd_config.5#ForceCommand\"><code>ForceCommand internal-sftp<\/code><\/a> is a cute little feature of openssh:<\/p>\n<blockquote>\n<p>Specifying a command of <code>internal-sftp<\/code> will force the use of an in-process SFTP server that requires no support files when used with <code>ChrootDirectory<\/code>.<\/p>\n<\/blockquote>\n<p>In particular, this denies shell and command access.<\/p>\n<p>What is <code>ChrootDirectory<\/code>?<\/p>\n<blockquote>\n<p>Specifies the pathname of a directory to <a href=\"https:\/\/man.openbsd.org\/chroot.2\">chroot(2)<\/a> to after authentication. At session startup <a href=\"https:\/\/man.openbsd.org\/sshd.8\">sshd(8)<\/a> checks that all components of the pathname are root-owned directories which are not writable by group or others. After the chroot, <a href=\"https:\/\/man.openbsd.org\/sshd.8\">sshd(8)<\/a> changes the working directory to the user's home directory. [...] For file transfer sessions using SFTP no additional configuration of the environment is necessary if the in-process sftp-server is used [...]<\/p>\n<\/blockquote>\n<p>cute! an easy way to deny even reading other parts of the rest of the system.<\/p>\n<p>why does it require the path to be owned by root? <a href=\"https:\/\/lists.mindrot.org\/pipermail\/openssh-unix-dev\/2009-May\/027651.html\">https:\/\/lists.mindrot.org\/pipermail\/openssh-unix-dev\/2009-May\/027651.html<\/a><\/p>\n<blockquote>\n<p>If you ever let the user chroot to this directory and execute his\nhard-linked \/bin\/su, he can become root within that directory and then\nescape the chroot. Even if you could prevent him from escaping chroot,\nhe can create device nodes and operate directly on filesystems, mount\n\/proc and operate on external processes, etc. It should be clear that\nthis is Very Bad (tm).<\/p>\n<\/blockquote>\n<p>cool cool cool. love unix. this sure is a tool we use to build software.<\/p>\n<p><a href=\"https:\/\/man.openbsd.org\/cgi-bin\/man.cgi\/OpenBSD-current\/man5\/sshd_config.5#Match\"><code>Match<\/code><\/a> is surprisingly complicated in the general case but simple enough here.\n<a href=\"https:\/\/man.openbsd.org\/cgi-bin\/man.cgi\/OpenBSD-current\/man5\/sshd_config.5#DisableForwarding\"><code>DisableForwarding<\/code><\/a> is probably not <em>strictly<\/em> necessary but we don't want people using this file server as a jumpbox to whatever other networks it's connected to, nor as e.g. a tor exit node.<\/p>\n<p>lastly i want to point out that <code>useradd<\/code> by default creates a user with no login password, which takes care of people guessing passwords for this without the proper ssh public key. i have <code>PasswordAuthentication no<\/code> set in <code>sshd_config<\/code>, but this allows disabling the password for just your sftp access without disabling it altogether. alternatively you could put it under the <code>Match User<\/code>.<\/p>\n<hr \/>\n<p><strong>NOTE:<\/strong> an earlier version of this post suggested <code>Subsystem sftp internal-sftp -R<\/code> and <code>ForceCommand internal-sftp<\/code>. that disables write access for <em>all<\/em> users, not just the selected user. the new version only disables it for the selected user.<\/p>\n"},{"title":"tools","published":"2025-05-03T00:00:00+00:00","updated":"2025-05-03T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/tools\/"}},"id":"https:\/\/jyn.dev\/tools\/","summary":"tools i use or want to use","content":"<p>i care a lot about my tools. i have very high standards for tools along some axes, and low standards along others\u2014but that's the topic of a different blog post. this post is about the tools i <em>do<\/em> use, and about tools i have my eye on and may use in the future. hopefully it will be useful to other people, such as programmers, sysadmins, etc.<\/p>\n<p>this is also not a description of <em>how<\/em> i use these tools, which also deserves its own blog post (i pride myself on integrating tools that were not designed to integrate with each other). this is just a list of the tools themselves.<\/p>\n<p>i have a version of <a href=\"https:\/\/jyn.dev\/tools\/#tools-i-have-my-eye-on\">tools i have my eye on<\/a> for myself that i keep regularly updated. i will probably not keep this public copy updated, unless i find a tool that's sufficiently good i want to tell people about it.<\/p>\n<p>i'm always on the lookout for more useful tools. if you have a tool you use regularly that's not on this list (and not specific to a single language ecosystem), please <a href=\"mailto:blog@jyn.dev\">let me know<\/a>. the one exception is containerization software. i am interested in tools to <em>interact<\/em> with containers, but at this time i am not looking for more container hosts (podman, kubernetes, vagrant, lxc, systemd-nspawn, firejail, chroot, etc).<\/p>\n<h2 id=\"tools-i-use\">tools i use<a class=\"zola-anchor\" href=\"#tools-i-use\" aria-label=\"Anchor link for: tools-i-use\"><\/a>\n<\/h2>\n<h3 id=\"basics\">basics<a class=\"zola-anchor\" href=\"#basics\" aria-label=\"Anchor link for: basics\"><\/a>\n<\/h3>\n<p>you have probably seen someone else recommend these tools. it's not just hype. they are useful.<\/p>\n<ul>\n<li><a href=\"https:\/\/github.com\/BurntSushi\/ripgrep\">ripgrep<\/a>: file contents searching<\/li>\n<li><a href=\"https:\/\/github.com\/sharkdp\/fd\">fd<\/a>: file name searching<\/li>\n<li><a href=\"https:\/\/jqlang.org\/\">jq<\/a>: json processing. note that most data formats can be converted to json, which i recommend over <a href=\"https:\/\/github.com\/mikefarah\/yq\">yq<\/a> or other format-specific filtering tools; the exception is XML and things isomorphic to it, which <a href=\"https:\/\/fwslc.blogspot.com\/2014\/10\/json-m.html\">cannot be represented in json with fidelity<\/a>. i tried a rewrite, <a href=\"https:\/\/github.com\/01mf02\/jaq\">jaq<\/a>, that promised to be faster, but it had fewer features and it turns out that i rarely care about the speed of jq.<\/li>\n<li><a href=\"https:\/\/man7.org\/linux\/man-pages\/man1\/killall.1.html\"><code>killall<\/code><\/a>: kill matching processes<\/li>\n<li><a href=\"https:\/\/linux.die.net\/man\/1\/pkill\"><code>pkill<\/code><\/a>: kill specific matching process<\/li>\n<\/ul>\n<h3 id=\"shell\">shell<a class=\"zola-anchor\" href=\"#shell\" aria-label=\"Anchor link for: shell\"><\/a>\n<\/h3>\n<p><a href=\"https:\/\/docs.jade.fyi\/zsh\/zsh.html#Introduction-1\">zsh<\/a>. i cannot in good conscience recommend this to anyone else; most people will be better served by <a href=\"https:\/\/fishshell.com\/\">fish<\/a>. i just have sunk cost from learning entirely too much bash syntax in my misspent youth.<\/p>\n<p>i tried <a href=\"https:\/\/xon.sh\/\">xonsh<\/a> and was annoyed by how poorly python variables interact with regular shell variables.<\/p>\n<p>i was very pleasantly surprised by <a href=\"https:\/\/learn.microsoft.com\/en-us\/powershell\/\">powershell<\/a>, which it turns out runs on linux these days. it has actual data types instead of just strings, which means you aren't constantly doing string munging in a language not designed for it. i would probably use it interactively if i were willing to give up my sunk cost on bash syntax.<\/p>\n<p>i tried <a href=\"https:\/\/www.nushell.sh\/\">nushell<\/a>, which is like powershell but not made by microsoft and without the .NET integration. i found it extremely verbose - it's a good scripting language but not a good interactive shell. also it breaks a bunch of bash syntax for no reason, even when the syntax is unambiguous. (much of bash's syntax <em>is<\/em> ambiguous and i can forgive breaking that.)<\/p>\n<h3 id=\"editor\">editor<a class=\"zola-anchor\" href=\"#editor\" aria-label=\"Anchor link for: editor\"><\/a>\n<\/h3>\n<p>i use <a href=\"https:\/\/neovim.io\/\">neovim<\/a> with a <a href=\"https:\/\/github.com\/jyn514\/dotfiles\/blob\/master\/config\/nvim.lua\">truly absurd amount of custom configuration<\/a>. i cannot in good conscience recommend this to anyone else; most people will be better served by vscode and a few plugins.<\/p>\n<p>i tried various other editors and was disappointed by all of them. <code>kakoune<\/code> required more configuration than i was willing to put in to just to get a \"basic\" experience. <code>helix<\/code> was extraordinarily resistant to being configured past very basic key remapping (for a while i forked it, but this didn't scale very well for the amount of configuration i wanted). VSCode dropped keystrokes and was generally laggy, both of which got worse when i installed a vim plugin. <code>zed<\/code> had various issues with the integrated terminal and window management (although this may have changed since i tried it in mid-2024). i have not tried <code>emacs<\/code> and dread the day i do because it will probably suck up weeks of my time.<\/p>\n<h3 id=\"terminal-multiplexing\">terminal multiplexing<a class=\"zola-anchor\" href=\"#terminal-multiplexing\" aria-label=\"Anchor link for: terminal-multiplexing\"><\/a>\n<\/h3>\n<p>i use <a href=\"https:\/\/github.com\/jyn514\/tmux\/tree\/working\">a fork<\/a> of tmux with <a href=\"https:\/\/github.com\/jyn514\/dotfiles\/blob\/master\/config\/tmux.conf\">more absurd amounts of configuration<\/a>. again, i cannot in good conscience recommend this to anyone else. most people will be better served by the integrated terminal in vscode (i spent a couple weeks on trying to get back ctrl-click for filepaths alone). vscode also works on windows MSVC, unlike terminal multiplexers.<\/p>\n<p>various people have recommended <a href=\"https:\/\/zellij.dev\/\">zellij<\/a> to me. i think zellij is good if you like the default keybinds, or if you use a terminal multiplexer infrequently enough that having the basic commands on-screen is helpful for you. i found that the default keybinds interfered with a bunch of programs and didn't look further into it.<\/p>\n<p>wezterm and kitty are not really in the running because they don't have session save\/resume.<\/p>\n<h3 id=\"debugging\">debugging<a class=\"zola-anchor\" href=\"#debugging\" aria-label=\"Anchor link for: debugging\"><\/a>\n<\/h3>\n<ul>\n<li><a href=\"https:\/\/docs.jade.fyi\/zsh\/zsh.html#index-XTRACE\"><code>set -x<\/code><\/a>: trace shell programs<\/li>\n<li><a href=\"https:\/\/git-scm.com\/book\/en\/v2\/Git-Internals-Environment-Variables#_debugging\"><code>GIT_TRACE<\/code><\/a>: trace git commands<\/li>\n<li><a href=\"https:\/\/strace.io\/\">strace<\/a>: trace system calls. this doesn't exist on macOS; it has a counterpart, dtruss, which <a href=\"https:\/\/poweruser.blog\/using-dtrace-with-sip-enabled-3826a352e64b\">kinda sorta mostly works<\/a>.<\/li>\n<li><a href=\"https:\/\/rr-project.org\/\">rr<\/a>: time travel debugging. this doesn't exist on macOS; if you know a counterpart <em>please<\/em> <a href=\"mailto:blog@jyn.dev\">let me know<\/a>.<\/li>\n<\/ul>\n<h3 id=\"c-c-tools\">C\/C++ tools<a class=\"zola-anchor\" href=\"#c-c-tools\" aria-label=\"Anchor link for: c-c-tools\"><\/a>\n<\/h3>\n<ul>\n<li>for cmake projects: <a href=\"https:\/\/cmake.org\/cmake\/help\/latest\/manual\/cmake.1.html#cmdoption-cmake-G\"><code>cmake -G Ninja<\/code><\/a>. uses <a href=\"https:\/\/ninja-build.org\/manual.html#_comparison_to_make\"><code>ninja<\/code><\/a> instead of <code>make<\/code>, improving compile times and with a much nicer progress bar.<\/li>\n<li>for cmake projects: <a href=\"https:\/\/cmake.org\/cmake\/help\/latest\/variable\/CMAKE_EXPORT_COMPILE_COMMANDS.html\"><code>cmake -D CMAKE_EXPORT_COMPILE_COMMANDS=On<\/code><\/a>. Generates a <code>compile_commands.json<\/code> file that most LSPs know how to read.<\/li>\n<li>for everything else: <a href=\"https:\/\/github.com\/rizsotto\/Bear\"><code>bear<\/code><\/a>. like above, but works for arbitrary build systems.<\/li>\n<\/ul>\n<h3 id=\"rust-tools\">rust tools<a class=\"zola-anchor\" href=\"#rust-tools\" aria-label=\"Anchor link for: rust-tools\"><\/a>\n<\/h3>\n<ul>\n<li><a href=\"https:\/\/github.com\/Canop\/bacon\/\"><code>bacon<\/code><\/a>: background code checker<\/li>\n<li><a href=\"https:\/\/doc.rust-lang.org\/cargo\/commands\/cargo-tree.html#tree-options\"><code>cargo tree -i<\/code><\/a>: figure out why a package was built<\/li>\n<li><a href=\"https:\/\/doc.crates.io\/contrib\/implementation\/debugging.html#logging\"><code>CARGO_LOG<\/code><\/a>: trace <a href=\"https:\/\/doc.rust-lang.org\/cargo\/\"><code>cargo<\/code><\/a> commands<\/li>\n<li><a href=\"https:\/\/rustc-dev-guide.rust-lang.org\/tracing.html\"><code>RUSTC_LOG<\/code><\/a>: trace <a href=\"https:\/\/doc.rust-lang.org\/rustc\/\"><code>rustc<\/code><\/a> commands<\/li>\n<li><a href=\"https:\/\/github.com\/rust-lang\/cargo-bisect-rustc\"><code>cargo bisect-rustc<\/code><\/a>: figure out when a compiler regression happened<\/li>\n<\/ul>\n<h3 id=\"everything-else\">\"everything else\"<a class=\"zola-anchor\" href=\"#everything-else\" aria-label=\"Anchor link for: everything-else\"><\/a>\n<\/h3>\n<ul>\n<li><a href=\"https:\/\/github.com\/watchexec\/watchexec\/\"><code>watchexec<\/code><\/a>: bacon but generalized to arbitrary files and commands. i find this quite useful for a variety of tasks, including minimizing bugs.<\/li>\n<li><a href=\"https:\/\/github.com\/aristocratos\/btop\">btop<\/a>: like top but easier to use and easier to understand the output<\/li>\n<li><a href=\"https:\/\/tailscale.com\/\">tailscale<\/a>: software-defined networking. i mainly use it for easy NAT punching and DDNS, but it's much more flexible than that.<\/li>\n<li><a href=\"https:\/\/obsidian.md\/\">obsidian<\/a>: flexible and pretty markdown editor. i use it for journaling and drafting blog posts. <a href=\"https:\/\/obsidian.md\/sync\">obsidian sync<\/a> is not required but \"just works\" and makes it much more convenient; it's nice to have access to the same files everywhere.<\/li>\n<li><a href=\"https:\/\/developer.mozilla.org\/en-US\/docs\/Web\/URI\/Reference\/Fragment\/Text_fragments\"><code>#:~:text=xyz<\/code><\/a>: link to specifically the text \"xyz\" on a web page. works in firefox since around the end of 2024, and in chrome since a lot earlier.<\/li>\n<\/ul>\n<h2 id=\"tools-i-have-my-eye-on\">tools i have my eye on<a class=\"zola-anchor\" href=\"#tools-i-have-my-eye-on\" aria-label=\"Anchor link for: tools-i-have-my-eye-on\"><\/a>\n<\/h2>\n<p>i either use these occasionally enough i need a reference, or haven't used them yet but want to.<\/p>\n<h3 id=\"debugging-1\">debugging<a class=\"zola-anchor\" href=\"#debugging-1\" aria-label=\"Anchor link for: debugging-1\"><\/a>\n<\/h3>\n<ul>\n<li><a href=\"https:\/\/man7.org\/linux\/man-pages\/man1\/strace.1.html#:~:text=y,decode-fds\"><code>strace -y<\/code><\/a>: show absolute paths<\/li>\n<li><a href=\"https:\/\/man7.org\/linux\/man-pages\/man1\/strace.1.html#:~:text=summary-only\"><code>strace -c<\/code><\/a>: show performance stats<\/li>\n<li><a href=\"https:\/\/crates.io\/crates\/elf-info\"><code>elf-info<\/code><\/a>: ELF inspector<\/li>\n<li><a href=\"https:\/\/binsider.dev\/blog\/v020\/\"><code>binsider<\/code><\/a>: another ELF inspector<\/li>\n<li><a href=\"https:\/\/docs.racket-lang.org\/zordoz\/index.html\"><code>zordoz<\/code><\/a>: racket decompiler<\/li>\n<li><a href=\"https:\/\/github.com\/dnSpyEx\/dnSpy\"><code>dnSpyEx<\/code><\/a>: .NET disassembler<\/li>\n<li><a href=\"https:\/\/github.com\/Syllo\/nvtop\"><code>nvtop<\/code><\/a>: gpu monitoring<\/li>\n<li><a href=\"https:\/\/linux.die.net\/man\/1\/xdpyinfo\"><code>xdpyinfo<\/code><\/a>: X11 debugging<\/li>\n<\/ul>\n<h3 id=\"operating-systems\">operating systems<a class=\"zola-anchor\" href=\"#operating-systems\" aria-label=\"Anchor link for: operating-systems\"><\/a>\n<\/h3>\n<ul>\n<li><a href=\"https:\/\/github.com\/pop-os\/launcher\/blob\/master\/plugins\/src\/web\/config.ron\">popos launcher config<\/a>, since i couldn't find docs for it (local copy in <code>\/usr\/lib\/pop-launcher\/plugins\/web\/config.ron<\/code>). this is what runs custom searches when you hit Super-R on PopOS 22.04.<\/li>\n<li><a href=\"https:\/\/github.com\/uobikiemukot\/yaft?tab=readme-ov-file\">yaft<\/a>: linux framebuffer terminal<\/li>\n<li><a href=\"https:\/\/github.com\/kellyjonbrazil\/jc\">jc<\/a>: structured parsing for unix tools<\/li>\n<li><a href=\"https:\/\/docs.fedoraproject.org\/en-US\/quick-docs\/cups-debug-scanning-issues\/\">CUPS debugging<\/a><\/li>\n<li><a href=\"https:\/\/yggdrasil-network.github.io\/about.html\">yggrasil-network<\/a>: \"ipv6 for everyone\"<\/li>\n<\/ul>\n<h3 id=\"containers\">containers<a class=\"zola-anchor\" href=\"#containers\" aria-label=\"Anchor link for: containers\"><\/a>\n<\/h3>\n<ul>\n<li><a href=\"https:\/\/github.com\/containers\/bubblewrap\">bubblewrap<\/a>: flexible adhoc sandboxing<\/li>\n<li><a href=\"https:\/\/github.com\/Mic92\/cntr\">cntr<\/a>: overlayfs that works with a running container<\/li>\n<\/ul>\n<h3 id=\"development\">development<a class=\"zola-anchor\" href=\"#development\" aria-label=\"Anchor link for: development\"><\/a>\n<\/h3>\n<ul>\n<li><a href=\"https:\/\/crates.io\/crates\/cargo-bounds\"><code>cargo-bounds<\/code><\/a>: check your crate semver bounds are correct<\/li>\n<li><a href=\"https:\/\/victoriametrics.com\/products\/victorialogs\/\">VictoriaLogs<\/a>: structured database for logs<\/li>\n<li><a href=\"https:\/\/cron.help\/\">cron.help<\/a>: crontab reference<\/li>\n<li><a href=\"https:\/\/hurl.dev\">hurl.dev<\/a>: HTTP test assertion framework\n<ul>\n<li><a href=\"https:\/\/docs.racket-lang.org\/riposte\/\">riposte<\/a>: similar for <a href=\"https:\/\/racket-lang.org\/\">racket<\/a><\/li>\n<\/ul>\n<\/li>\n<li><a href=\"https:\/\/rosie-lang.org\/\">rosie-lang<\/a>: maintainable, composable regexes. really these are PEG parsers but they are designed to look similar to regex.<\/li>\n<li><a href=\"https:\/\/github.com\/brushtechnology\/fabricate\">fabricate<\/a>: <a href=\"https:\/\/gittup.org\/tup\/\">tup<\/a> but python<\/li>\n<li><a href=\"https:\/\/spoofax.dev\/\">spoofax<\/a>: dsl builder<\/li>\n<li><a href=\"https:\/\/github.com\/Ericsson\/codechecker\">codechecker<\/a>: advanced static analysis for c\/c++<\/li>\n<li><a href=\"https:\/\/google.github.io\/zx\/getting-started\">zx<\/a>: terse process spawning in js<\/li>\n<li><a href=\"https:\/\/jsonnet.org\">jsonnet<\/a>: like <a href=\"https:\/\/dhall-lang.org\/\">dhall<\/a> but lua-coded instead of haskell-coded<\/li>\n<li><a href=\"https:\/\/sql.datapage.app\/\">generate a webpage from an sql query<\/a><\/li>\n<\/ul>\n<h3 id=\"editing-and-diffing\">editing and diffing<a class=\"zola-anchor\" href=\"#editing-and-diffing\" aria-label=\"Anchor link for: editing-and-diffing\"><\/a>\n<\/h3>\n<ul>\n<li><a href=\"https:\/\/ast-grep.github.io\/\">ast-grep<\/a>. language-aware structured search and replace.<\/li>\n<li><a href=\"https:\/\/diffoscope.org\/\">diffoscope<\/a>: recursive diffing<\/li>\n<li>fix tabs and spaces: <code>unexpand -t 4 foo.c | sponge foo.c<\/code><\/li>\n<\/ul>\n<h3 id=\"graphics\">graphics<a class=\"zola-anchor\" href=\"#graphics\" aria-label=\"Anchor link for: graphics\"><\/a>\n<\/h3>\n<ul>\n<li><a href=\"https:\/\/typst.app\/\">typst<\/a>: <a href=\"https:\/\/www.latex-project.org\/\">latex<\/a> but not evil<\/li>\n<li><a href=\"https:\/\/excalidraw.com\/\">excalidraw<\/a>: online diagram editor<\/li>\n<li><a href=\"https:\/\/github.com\/Aloxaf\/silicon\">silicon<\/a>: pretty screenshots of source code<\/li>\n<\/ul>\n<h3 id=\"benchmarking\">benchmarking<a class=\"zola-anchor\" href=\"#benchmarking\" aria-label=\"Anchor link for: benchmarking\"><\/a>\n<\/h3>\n<ul>\n<li><a href=\"https:\/\/github.com\/andrewrk\/poop\"><code>poop<\/code><\/a>: like <a href=\"https:\/\/github.com\/sharkdp\/hyperfine\">hyperfine<\/a> but also reports memory usage<\/li>\n<li><a href=\"https:\/\/github.com\/sharkdp\/hyperfine#parameterized-benchmarks\"><code>hyperfine --parameter-list<\/code><\/a>: parameterized benchmarking<\/li>\n<li><a href=\"https:\/\/man7.org\/linux\/man-pages\/man1\/strace.1.html#:~:text=summary-wall-clock\"><code>strace -wc<\/code><\/a>: measure syscall timing<\/li>\n<li><a href=\"https:\/\/man7.org\/linux\/man-pages\/man1\/ltrace.1.html#:~:text=count%20time\"><code>ltrace -c<\/code><\/a>: measure library call timing<\/li>\n<li><a href=\"https:\/\/linux.die.net\/man\/1\/x11perf\"><code>x11perf<\/code><\/a>: X11 performance testing<\/li>\n<li><a href=\"https:\/\/manpages.debian.org\/testing\/freetype2-demos\/ftbench.1.en.html\"><code>ftbench<\/code><\/a>: font performance testing<\/li>\n<li><a href=\"https:\/\/man.openbsd.org\/ts#i\"><code>ts -i<\/code><\/a>: show the elapsed time between each line of output in a pipe. useful with <code>bash -x<\/code>.<\/li>\n<li><a href=\"https:\/\/docs.jade.fyi\/zsh\/zsh.html#The-zsh_002fzprof-Module\"><code>zmodload zsh\/zprof; zprof<\/code><\/a>: benchmark zsh startup<\/li>\n<\/ul>\n"},{"title":"Terrible Horrible No Good Very Bad Python","published":"2025-03-28T00:00:00+00:00","updated":"2025-03-28T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/terrible-horrible-no-good-very-bad-python\/"}},"id":"https:\/\/jyn.dev\/terrible-horrible-no-good-very-bad-python\/","summary":"new and exciting ways to write buggy code","content":"<p>time for everyone's favorite game!!<\/p>\n<p>what does this code do?<\/p>\n<pre data-lang=\"python\" class=\"language-python z-code\"><code class=\"language-python\" data-lang=\"python\"><span class=\"z-source z-python\"><span class=\"z-meta z-function z-python\"><span class=\"z-storage z-type z-function z-python\"><span class=\"z-keyword z-declaration z-function z-python\">def<\/span><\/span> <span class=\"z-entity z-name z-function z-python\"><span class=\"z-meta z-generic-name z-python\">foo<\/span><\/span><\/span><span class=\"z-meta z-function z-parameters z-python\"><span class=\"z-punctuation z-section z-parameters z-begin z-python\">(<\/span><\/span><span class=\"z-meta z-function z-parameters z-python\"><span class=\"z-punctuation z-section z-parameters z-end z-python\">)<\/span><\/span><span class=\"z-meta z-function z-python\"><span class=\"z-punctuation z-section z-function z-begin z-python\">:<\/span><\/span>\n<\/span><span class=\"z-source z-python\">  <span class=\"z-meta z-statement z-exception z-try z-python\"><span class=\"z-keyword z-control z-exception z-try z-python\">try<\/span><span class=\"z-punctuation z-section z-block z-exception z-try z-python\">:<\/span><\/span>\n<\/span><span class=\"z-source z-python\">    <span class=\"z-keyword z-control z-flow z-return z-python\">return<\/span> <span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">os<\/span><\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-punctuation z-accessor z-dot z-python\">.<\/span><\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-meta z-generic-name z-python\">_exit<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span>\n<\/span><span class=\"z-source z-python\">  <span class=\"z-meta z-statement z-exception z-finally z-python\"><span class=\"z-keyword z-control z-exception z-finally z-python\">finally<\/span><span class=\"z-punctuation z-section z-block z-exception z-finally z-python\">:<\/span><\/span>\n<\/span><span class=\"z-source z-python\">    <span class=\"z-keyword z-control z-flow z-return z-python\">return<\/span> <span class=\"z-constant z-language z-python\">False<\/span>\n<\/span><span class=\"z-source z-python\">\n<\/span><span class=\"z-source z-python\"><span class=\"z-meta z-statement z-import z-python\"><span class=\"z-keyword z-control z-import z-python\">import<\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">os<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-meta z-generic-name z-python\">foo<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span>\n<\/span><\/code><\/pre>\n<style type=\"text\/css\">\n    ol { list-style-type: upper-alpha; }\n<\/style>\n<p>does it:<\/p>\n<ol>\n<li>return None<\/li>\n<li>return False<\/li>\n<li>throw an exception<\/li>\n<li>exit the process without printing anything<\/li>\n<li>something even more devious<\/li>\n<\/ol>\n<p>sit with it. have a good think. explain your answers.<\/p>\n<p>ready?<\/p>\n<details><summary>ok fine what does it do<\/summary>\n<p>returns <code>False<\/code>. want to know why?<\/p>\n<details><summary>yes just tell me already >:(<\/summary>\n<p>normally, <code>os._exit<\/code> exits the process without running \"cleanup handlers\" (<code>finally<\/code> blocks). however, it takes one argument. this snippet forgets to pass in the exit code, so instead of exiting, it throws <code>TypeError<\/code>. then the <code>finally<\/code> block silently swallows the exception because of the <code>return<\/code>.<\/p>\n<p><code>return<\/code> from a <code>finally<\/code> block is in fact so commonly misused that the python developers <a href=\"https:\/\/discuss.python.org\/t\/pep-765-disallow-return-break-continue-that-exit-a-finally-block\/71348\">plan<\/a> to make it emit a <code>SyntaxWarning<\/code> in a future release.<\/p>\n<p>one might be mislead that <code>import os<\/code> comes after the function is defined. but python has dynamic scoping, so that's fine.<\/p>\n<p>one might also mix up <code>sys.exit<\/code> with <code>os._exit<\/code>. <code>sys.exit<\/code> works by raising a <a href=\"https:\/\/docs.python.org\/3\/library\/exceptions.html#SystemExit\"><code>SystemExit<\/code><\/a> exception, which would be caught and swallowed by the <code>finally<\/code> block. but <a href=\"https:\/\/docs.python.org\/3\/library\/os.html#os._exit\"><code>_exit<\/code><\/a> directly exits the process:<\/p>\n<blockquote>\n<p>Exit the process with status n, without calling cleanup handlers, flushing stdio buffers, etc.<\/p>\n<\/blockquote>\n<p>in fact, it doesn't even call atexit handlers, not even if we directly use libc:<\/p>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">&gt;&gt;&gt; import atexit\n<\/span><span class=\"z-text z-plain\">&gt;&gt;&gt; atexit.register(lambda: print(&#39;hi&#39;))\n<\/span><span class=\"z-text z-plain\">&lt;function &lt;lambda&gt; at 0x73c740cf2830&gt;\n<\/span><span class=\"z-text z-plain\">&gt;&gt;&gt; from ctypes import *\n<\/span><span class=\"z-text z-plain\">&gt;&gt;&gt; libc = cdll.LoadLibrary(&quot;libc.so.6&quot;)\n<\/span><span class=\"z-text z-plain\">&gt;&gt;&gt; libc.on_exit(CFUNCTYPE(c_int, c_voidp)(lambda status, _: print(status)))\n<\/span><span class=\"z-text z-plain\">0\n<\/span><span class=\"z-text z-plain\">&gt;&gt;&gt; import os\n<\/span><span class=\"z-text z-plain\">&gt;&gt;&gt; os._exit(1)\n<\/span><span class=\"z-text z-plain\"># no output\n<\/span><\/code><\/pre>\n<\/details>\n<\/details>\n<p>yes someone did write this code by accident and yes they were very confused. i thought it was a bug in cpython until i figured it out.<\/p>\n<p>you're welcome!!<\/p>\n"},{"title":"Scheduling Do-Not-Disturb in GNOME","published":"2025-02-22T00:00:00+00:00","updated":"2025-02-22T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/do-not-disturb-in-gnome\/"}},"id":"https:\/\/jyn.dev\/do-not-disturb-in-gnome\/","content":"<h2 id=\"do-not-disturb\">Do Not Disturb<a class=\"zola-anchor\" href=\"#do-not-disturb\" aria-label=\"Anchor link for: do-not-disturb\"><\/a>\n<\/h2>\n<p>GNOME has a little button that lets you turn on Do-Not-Disturb for notifications:<\/p>\n<img src=\"&#x2F;assets&#x2F;Pasted image 20250222135047.png\"\n\t alt=\"Gnome notifications menu\"\n\t\n>\n<p>Unfortunately, it has <a href=\"https:\/\/gitlab.gnome.org\/GNOME\/gnome-control-center\/-\/issues\/2200\">no way of scheduling DnD<\/a>.<\/p>\n<p>Good news, though! It does support turning on DnD through the CLI: <code>gsettings set org.gnome.desktop.notifications show-banners false<\/code>. I put that in a script named <code>toggle-dnd<\/code> in my dotfiles:<\/p>\n<pre data-lang=\"sh\" class=\"language-sh z-code\"><code class=\"language-sh\" data-lang=\"sh\"><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">$<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> cat bin\/toggle-dnd<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-comment z-line z-number-sign z-shell\"><span class=\"z-punctuation z-definition z-comment z-begin z-shell\">#<\/span><\/span><span class=\"z-comment z-line z-number-sign z-shell\">!\/bin\/sh<\/span><span class=\"z-comment z-line z-number-sign z-shell\">\n<\/span><\/span><span class=\"z-source z-shell z-bash\">\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-conditional z-case z-shell\"><span class=\"z-keyword z-control z-conditional z-case z-shell\">case<\/span> <span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-punctuation z-definition z-variable z-shell\">$<\/span><span class=\"z-punctuation z-section z-expansion z-parameter z-begin z-shell\">{<\/span><\/span><span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-variable z-other z-readwrite z-shell\">1<\/span><\/span><span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-keyword z-operator z-assignment z-shell\">:-<\/span><\/span><span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-punctuation z-section z-expansion z-parameter z-end z-shell\">}<\/span><\/span> <span class=\"z-keyword z-control z-in z-shell\">in<\/span>\n<\/span><\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-conditional z-case z-shell\">\t<span class=\"z-comment z-line z-number-sign z-shell\"><span class=\"z-punctuation z-definition z-comment z-begin z-shell\">#<\/span><\/span><span class=\"z-comment z-line z-number-sign z-shell\"> &quot;show-banners&quot; is reversed from what you would expect &quot;do not disturb&quot; to mean<\/span><span class=\"z-comment z-line z-number-sign z-shell\">\n<\/span><\/span><\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-conditional z-case z-shell\">\t<\/span><span class=\"z-meta z-conditional z-case z-clause z-patterns z-shell\">true<span class=\"z-meta z-conditional z-case z-shell\"><span class=\"z-keyword z-control z-conditional z-patterns z-end z-shell\">)<\/span><\/span><\/span><span class=\"z-meta z-conditional z-case z-clause z-commands z-shell\"> <span class=\"z-variable z-other z-readwrite z-assignment z-shell\">new<\/span><span class=\"z-keyword z-operator z-assignment z-shell\">=<\/span><span class=\"z-string z-unquoted z-shell\">false<\/span><\/span><span class=\"z-meta z-conditional z-case z-clause z-commands z-shell\"><span class=\"z-punctuation z-terminator z-case z-clause z-shell\">;;<\/span><\/span><span class=\"z-meta z-conditional z-case z-clause z-patterns z-shell\">\n<\/span><\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-conditional z-case z-clause z-patterns z-shell\">\t<\/span><span class=\"z-meta z-conditional z-case z-clause z-patterns z-shell\">false<span class=\"z-meta z-conditional z-case z-clause z-patterns z-shell\"><span class=\"z-keyword z-control z-conditional z-patterns z-end z-shell\">)<\/span><\/span><\/span><span class=\"z-meta z-conditional z-case z-clause z-commands z-shell\"> <span class=\"z-variable z-other z-readwrite z-assignment z-shell\">new<\/span><span class=\"z-keyword z-operator z-assignment z-shell\">=<\/span><span class=\"z-string z-unquoted z-shell\">true<\/span><\/span><span class=\"z-meta z-conditional z-case z-clause z-commands z-shell\"><span class=\"z-punctuation z-terminator z-case z-clause z-shell\">;;<\/span><\/span><span class=\"z-meta z-conditional z-case z-clause z-patterns z-shell\">\n<\/span><\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-conditional z-case z-clause z-patterns z-shell\">\t<\/span><span class=\"z-meta z-conditional z-case z-clause z-patterns z-shell\"><span class=\"z-keyword z-operator z-regexp z-quantifier z-shell\">*<\/span><span class=\"z-meta z-conditional z-case z-clause z-patterns z-shell\"><span class=\"z-keyword z-control z-conditional z-patterns z-end z-shell\">)<\/span><\/span><\/span><span class=\"z-meta z-conditional z-case z-clause z-commands z-shell\">\n<\/span><\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-conditional z-case z-clause z-commands z-shell\">\t<span class=\"z-keyword z-control z-conditional z-if z-shell\">if<\/span> <span class=\"z-support z-function z-test z-begin z-shell\">[<\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> <span class=\"z-string z-quoted z-double z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&quot;<\/span><span class=\"z-meta z-group z-expansion z-command z-parens z-shell\"><span class=\"z-punctuation z-definition z-variable z-shell\">$<\/span><span class=\"z-punctuation z-section z-parens z-begin z-shell\">(<\/span><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">gsettings<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> get org.gnome.desktop.notifications show-banners<\/span><span class=\"z-punctuation z-section z-parens z-end z-shell\">)<\/span><\/span><span class=\"z-punctuation z-definition z-string z-end z-shell\">&quot;<\/span><\/span> <span class=\"z-keyword z-operator z-logical z-shell\">=<\/span> true <span class=\"z-support z-function z-test z-end z-shell\">]<\/span><\/span>\n<\/span><\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-conditional z-case z-clause z-commands z-shell\">\t\t<span class=\"z-keyword z-control z-conditional z-then z-shell\">then<\/span> <span class=\"z-variable z-other z-readwrite z-assignment z-shell\">new<\/span><span class=\"z-keyword z-operator z-assignment z-shell\">=<\/span><span class=\"z-string z-unquoted z-shell\">false<\/span>\n<\/span><\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-conditional z-case z-clause z-commands z-shell\">\t\t<span class=\"z-keyword z-control z-conditional z-else z-shell\">else<\/span> <span class=\"z-variable z-other z-readwrite z-assignment z-shell\">new<\/span><span class=\"z-keyword z-operator z-assignment z-shell\">=<\/span><span class=\"z-string z-unquoted z-shell\">true<\/span>\n<\/span><\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-conditional z-case z-clause z-commands z-shell\">\t<span class=\"z-keyword z-control z-conditional z-end z-shell\">fi<\/span>\n<\/span><\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-conditional z-case z-clause z-commands z-shell\">\t<\/span><span class=\"z-meta z-conditional z-case z-clause z-commands z-shell\"><span class=\"z-punctuation z-terminator z-case z-clause z-shell\">;;<\/span><\/span><span class=\"z-meta z-conditional z-case z-clause z-patterns z-shell\">\n<\/span><\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-conditional z-case z-clause z-patterns z-shell\"><span class=\"z-keyword z-control z-conditional z-end z-shell\">esac<\/span><\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">gsettings<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> set org.gnome.desktop.notifications show-banners <span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-punctuation z-definition z-variable z-shell\">$<\/span><span class=\"z-variable z-other z-readwrite z-shell\">new<\/span><\/span><\/span>\n<\/span><\/code><\/pre>\n<h2 id=\"scheduling\">scheduling<a class=\"zola-anchor\" href=\"#scheduling\" aria-label=\"Anchor link for: scheduling\"><\/a>\n<\/h2>\n<p>I tried putting that in cron<sup class=\"footnote-reference\" id=\"fr-1-1\"><a href=\"#fn-1\">1<\/a><\/sup>, had a sneaking suspicion it wouldn't work, set it to run every minute, and saw this very unhelpful line of logging:<\/p>\n<pre data-lang=\"console\" class=\"language-console z-code\"><code class=\"language-console\" data-lang=\"console\"><span class=\"z-source z-shell z-console\"><span class=\"z-keyword z-operator z-assignment z-redirection z-process z-shell\">$<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">journalctl<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> --<\/span>unit<\/span> cron<span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> --<\/span>since<\/span> <span class=\"z-string z-quoted z-single z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&#39;<\/span>5m ago<span class=\"z-punctuation z-definition z-string z-end z-shell\">&#39;<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-shell z-console\">Feb 22 12:00:01 pop-os CRON[1623131]: (CRON) info (No MTA installed, discarding output)\n<\/span><\/code><\/pre>\n<p>Ok, fine. Let's pipe the output to the system log<sup class=\"footnote-reference\" id=\"fr-2-1\"><a href=\"#fn-2\">2<\/a><\/sup>, since clearly cron can't handle that itself.<\/p>\n<pre data-lang=\"crontab\" class=\"language-crontab z-code\"><code class=\"language-crontab\" data-lang=\"crontab\"><span class=\"z-text z-plain\">* * * * * bash -lc &#39;org.gnome.desktop.notifications show-banners false 2&gt;&amp;1 | logger -t toggle-dnd&#39;\n<\/span><\/code><\/pre>\n<p>That at least shows more useful output.<\/p>\n<pre data-lang=\"console\" class=\"language-console z-code\"><code class=\"language-console\" data-lang=\"console\"><span class=\"z-source z-shell z-console\"><span class=\"z-keyword z-operator z-assignment z-redirection z-process z-shell\">$<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">journalctl<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> -<\/span>t<\/span> toggle-dnd<\/span>\n<\/span><span class=\"z-source z-shell z-console\">Feb 22 05:59:01 pop-os toggle-dnd[1376772]: \/bin\/sh: 1: toggle-dnd: not found\n<\/span><\/code><\/pre>\n<p>Oh right. Cron is running things with a default PATH. Technically there are ways to configure this <sup class=\"footnote-reference\" id=\"fr-3-1\"><a href=\"#fn-3\">3<\/a><\/sup>, but the simple solution is just to run a bash login shell which sources all the directories i would normally have in a shell. at this point, however, it is getting somewhat annoying to test via cron, so let's replicate cron's environment:<\/p>\n<pre data-lang=\"console\" class=\"language-console z-code\"><code class=\"language-console\" data-lang=\"console\"><span class=\"z-source z-shell z-console\"><span class=\"z-keyword z-operator z-assignment z-redirection z-process z-shell\">$<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">env<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> -<\/span>i<\/span> HOME=<span class=\"z-string z-quoted z-double z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&quot;<\/span><span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-punctuation z-definition z-variable z-shell\">$<\/span><span class=\"z-variable z-other z-readwrite z-shell\">HOME<\/span><\/span><span class=\"z-punctuation z-definition z-string z-end z-shell\">&quot;<\/span><\/span> TERM=<span class=\"z-string z-quoted z-double z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&quot;<\/span><span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-punctuation z-definition z-variable z-shell\">$<\/span><span class=\"z-variable z-other z-readwrite z-shell\">TERM<\/span><\/span><span class=\"z-punctuation z-definition z-string z-end z-shell\">&quot;<\/span><\/span> PS1=<span class=\"z-string z-quoted z-single z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&#39;<\/span>$ <span class=\"z-punctuation z-definition z-string z-end z-shell\">&#39;<\/span><\/span> HISTSIZE=-1 HISTFILE= bash<span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> --<\/span>norc<\/span><span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> --<\/span>noprofile<\/span><\/span>\n<\/span><span class=\"z-source z-shell z-console\"><span class=\"z-keyword z-operator z-assignment z-redirection z-process z-shell\">$<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-support z-function z-echo z-shell\">echo<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> <span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-punctuation z-definition z-variable z-shell\">$<\/span><span class=\"z-variable z-other z-readwrite z-shell\">PATH<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-shell z-console\">\/usr\/local\/sbin:\/usr\/local\/bin:\/usr\/sbin:\/usr\/bin:\/sbin:\/bin\n<\/span><span class=\"z-source z-shell z-console\"><span class=\"z-keyword z-operator z-assignment z-redirection z-process z-shell\">$<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">bash<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> -<\/span>l<\/span><\/span>\n<\/span><span class=\"z-source z-shell z-console\"><span class=\"z-keyword z-operator z-assignment z-redirection z-process z-shell\">$<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-support z-function z-echo z-shell\">echo<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> <span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-punctuation z-definition z-variable z-shell\">$<\/span><span class=\"z-variable z-other z-readwrite z-shell\">PATH<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-shell z-console\">\/home\/jyn\/Documents\/node-v20.12.2-linux-x64\/bin:\/home\/jyn\/.local\/bin:\/home\/jyn\/src\/dotfiles\/bin:\/home\/jyn\/.local\/lib\/cargo\/bin:\/snap\/bin:\/usr\/games:\/home\/jyn\/perl5\/bin:\/usr\/local\/sbin:\/usr\/local\/bin:\/usr\/sbin:\/usr\/bin:\/sbin:\/bin:\/snap\/bin\n<\/span><\/code><\/pre>\n<p>neat. let's run  make sure our command runs.\nbefore:<\/p>\n<img src=\"&#x2F;assets&#x2F;Pasted image 20250222141930.png\"\n\t alt=\"Left side of the Gnome topbar\"\n\t\n>\n<p>after running <code>toggle-dnd<\/code> in our login shell:<\/p>\n<img src=\"&#x2F;assets&#x2F;Pasted image 20250222141930.png\"\n\t alt=\"Still the left side of the gnome topbar. In fact this is the exact same image as before.\"\n\t\n>\n<p>... nothing happened.\nwe can confirm this on the CLI:<\/p>\n<pre data-lang=\"console\" class=\"language-console z-code\"><code class=\"language-console\" data-lang=\"console\"><span class=\"z-source z-shell z-console\"><span class=\"z-keyword z-operator z-assignment z-redirection z-process z-shell\">$<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">gsettings<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> get org.gnome.desktop.notifications show-banners<\/span>\n<\/span><span class=\"z-source z-shell z-console\">true\n<\/span><span class=\"z-source z-shell z-console\"><span class=\"z-keyword z-operator z-assignment z-redirection z-process z-shell\">$<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">gsettings<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> set org.gnome.desktop.notifications show-banners false<\/span>\n<\/span><span class=\"z-source z-shell z-console\"><span class=\"z-keyword z-operator z-assignment z-redirection z-process z-shell\">$<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">gsettings<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> get org.gnome.desktop.notifications show-banners<\/span>\n<\/span><span class=\"z-source z-shell z-console\">true\n<\/span><\/code><\/pre>\n<h2 id=\"dbus\">DBUS<a class=\"zola-anchor\" href=\"#dbus\" aria-label=\"Anchor link for: dbus\"><\/a>\n<\/h2>\n<p>at this point i started to get annoyed and ran <code>systemctl --user status<\/code> in hopes of writing a systemd timer instead. fortunately, i did that inside the bash login shell, which gave me this helpful error message:<\/p>\n<pre data-lang=\"console\" class=\"language-console z-code\"><code class=\"language-console\" data-lang=\"console\"><span class=\"z-source z-shell z-console\"><span class=\"z-keyword z-operator z-assignment z-redirection z-process z-shell\">$<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">systemctl<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> --<\/span>user<\/span> status<\/span>\n<\/span><span class=\"z-source z-shell z-console\">Failed to connect to bus: $DBUS_SESSION_BUS_ADDRESS and $XDG_RUNTIME_DIR not defined (consider using --machine=&lt;user&gt;@.host --user to connect to bus of other user)\n<\/span><\/code><\/pre>\n<p>It turns out that both <code>gsettings<\/code> and <code>systemctl<\/code> are trying to communicate over DBUS, and DBUS is linked to your \"user session\", set when you login. Unsetting environment variables disables DBUS <sup class=\"footnote-reference\" id=\"fr-4-1\"><a href=\"#fn-4\">4<\/a><\/sup>.<\/p>\n<p>I found a helpful <a href=\"https:\/\/askubuntu.com\/a\/1468012\">stackoverflow post<\/a> that helps us reconnect to DBUS <sup class=\"footnote-reference\" id=\"fr-5-1\"><a href=\"#fn-5\">5<\/a><\/sup>:<\/p>\n<pre data-lang=\"sh\" class=\"language-sh z-code\"><code class=\"language-sh\" data-lang=\"sh\"><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-storage z-modifier z-shell\">export<\/span> <span class=\"z-variable z-other z-readwrite z-assignment z-shell\">XDG_RUNTIME_DIR<\/span><span class=\"z-keyword z-operator z-assignment z-shell\">=<\/span><span class=\"z-string z-unquoted z-shell\"><span class=\"z-string z-quoted z-double z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&quot;<\/span>\/run\/user\/<span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-punctuation z-definition z-variable z-shell\">$<\/span><span class=\"z-variable z-other z-readwrite z-shell\">UID<\/span><\/span><span class=\"z-punctuation z-definition z-string z-end z-shell\">&quot;<\/span><\/span><\/span><\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-storage z-modifier z-shell\">export<\/span> <span class=\"z-variable z-other z-readwrite z-assignment z-shell\">DBUS_SESSION_BUS_ADDRESS<\/span><span class=\"z-keyword z-operator z-assignment z-shell\">=<\/span><span class=\"z-string z-unquoted z-shell\"><span class=\"z-string z-quoted z-double z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&quot;<\/span>unix:path=<span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-punctuation z-definition z-variable z-shell\">$<\/span><span class=\"z-punctuation z-section z-expansion z-parameter z-begin z-shell\">{<\/span><\/span><span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-variable z-other z-readwrite z-shell\">XDG_RUNTIME_DIR<\/span><\/span><span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-punctuation z-section z-expansion z-parameter z-end z-shell\">}<\/span><\/span>\/bus<span class=\"z-punctuation z-definition z-string z-end z-shell\">&quot;<\/span><\/span><\/span><\/span>\n<\/span><\/code><\/pre>\n<p>Let's write a little abstraction for that, too <sup class=\"footnote-reference\" id=\"fr-6-1\"><a href=\"#fn-6\">6<\/a><\/sup>.<\/p>\n<pre data-lang=\"bash\" class=\"language-bash z-code\"><code class=\"language-bash\" data-lang=\"bash\"><span class=\"z-source z-shell z-bash\"><span class=\"z-comment z-line z-number-sign z-shell\"><span class=\"z-punctuation z-definition z-comment z-begin z-shell\">#<\/span><\/span><span class=\"z-comment z-line z-number-sign z-shell\">!\/usr\/bin\/env bash<\/span><span class=\"z-comment z-line z-number-sign z-shell\">\n<\/span><\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-comment z-line z-number-sign z-shell\"><span class=\"z-punctuation z-definition z-comment z-begin z-shell\">#<\/span><\/span><span class=\"z-comment z-line z-number-sign z-shell\"> Run a command in an environment where DBUS commands (e.g. `systemd --user`, `gsettings`) are available<\/span><span class=\"z-comment z-line z-number-sign z-shell\">\n<\/span><\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-support z-function z-colon z-shell\">:<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> <span class=\"z-string z-quoted z-double z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&quot;<\/span><span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-punctuation z-definition z-variable z-shell\">$<\/span><span class=\"z-punctuation z-section z-expansion z-parameter z-begin z-shell\">{<\/span><\/span><span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-variable z-other z-readwrite z-shell\">XDG_RUNTIME_DIR<\/span><\/span><span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-keyword z-operator z-assignment z-shell\">:=<\/span><\/span><span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-string z-quoted z-double z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&quot;<\/span>\/run\/user\/<span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-punctuation z-definition z-variable z-shell\">$<\/span><span class=\"z-variable z-other z-readwrite z-shell\">UID<\/span><\/span><span class=\"z-punctuation z-definition z-string z-end z-shell\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-punctuation z-section z-expansion z-parameter z-end z-shell\">}<\/span><\/span><span class=\"z-punctuation z-definition z-string z-end z-shell\">&quot;<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-support z-function z-colon z-shell\">:<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> <span class=\"z-string z-quoted z-double z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&quot;<\/span><span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-punctuation z-definition z-variable z-shell\">$<\/span><span class=\"z-punctuation z-section z-expansion z-parameter z-begin z-shell\">{<\/span><\/span><span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-variable z-other z-readwrite z-shell\">DBUS_SESSION_BUS_ADDRESS<\/span><\/span><span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-keyword z-operator z-assignment z-shell\">:=<\/span><\/span><span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-string z-quoted z-double z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&quot;<\/span>unix:path=<span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-punctuation z-definition z-variable z-shell\">$<\/span><span class=\"z-punctuation z-section z-expansion z-parameter z-begin z-shell\">{<\/span><\/span><span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-variable z-other z-readwrite z-shell\">XDG_RUNTIME_DIR<\/span><\/span><span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-punctuation z-section z-expansion z-parameter z-end z-shell\">}<\/span><\/span>\/bus<span class=\"z-punctuation z-definition z-string z-end z-shell\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-punctuation z-section z-expansion z-parameter z-end z-shell\">}<\/span><\/span><span class=\"z-punctuation z-definition z-string z-end z-shell\">&quot;<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-storage z-modifier z-shell\">export<\/span> <span class=\"z-variable z-other z-readwrite z-assignment z-shell\">XDG_RUNTIME_DIR<\/span> <span class=\"z-variable z-other z-readwrite z-assignment z-shell\">DBUS_SESSION_BUS_ADDRESS<\/span><\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-support z-function z-exec z-shell\">exec<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> <span class=\"z-string z-quoted z-double z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&quot;<\/span><span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-punctuation z-definition z-variable z-shell\">$<\/span><span class=\"z-variable z-language z-shell\">@<\/span><\/span><span class=\"z-punctuation z-definition z-string z-end z-shell\">&quot;<\/span><\/span><\/span>\n<\/span><\/code><\/pre>\n<p>Now, finally, we can put the pieces together:<\/p>\n<pre data-lang=\"console\" class=\"language-console z-code\"><code class=\"language-console\" data-lang=\"console\"><span class=\"z-source z-shell z-console\"><span class=\"z-keyword z-operator z-assignment z-redirection z-process z-shell\">$<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">crontab<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> -<\/span>l<\/span><\/span>\n<\/span><span class=\"z-source z-shell z-console\">0 20 * * * bash -lc &#39;dbus-run-user toggle-dnd true  2&gt;&amp;1 | logger -t toggle-dnd&#39;\n<\/span><span class=\"z-source z-shell z-console\">0 8  * * * bash -lc &#39;dbus-run-user toggle-dnd false 2&gt;&amp;1 | logger -t toggle-dnd&#39;\n<\/span><\/code><\/pre>\n<h3 id=\"p-s\">P.S.<a class=\"zola-anchor\" href=\"#p-s\" aria-label=\"Anchor link for: p-s\"><\/a>\n<\/h3>\n<h4 id=\"x11\">X11<a class=\"zola-anchor\" href=\"#x11\" aria-label=\"Anchor link for: x11\"><\/a>\n<\/h4>\n<p>just setting up DBUS doesn't set up our X11 environment again. I didn't happen to need that. but now that we have DBUS, you can retrieve it pretty easily:<\/p>\n<pre data-lang=\"console\" class=\"language-console z-code\"><code class=\"language-console\" data-lang=\"console\"><span class=\"z-source z-shell z-console\"><span class=\"z-keyword z-operator z-assignment z-redirection z-process z-shell\">$<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">dbus-run-user<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> systemctl<span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> --<\/span>user<\/span> show-environment<\/span> <span class=\"z-keyword z-operator z-logical z-pipe z-shell\">|<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">grep<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> ^DISPLAY<\/span>\n<\/span><span class=\"z-source z-shell z-console\">DISPLAY=:1\n<\/span><\/code><\/pre>\n<p>You can do something similar for <code>XAUTHORITY<\/code>, <code>TMUX*<\/code>, and <code>XDG_*<\/code> environment variables. Note that <code>systemctl --user<\/code> may be using a tmux session or pane that no longer exists; use caution.<\/p>\n<h4 id=\"why-were-you-doing-this-in-the-first-place\">why were you doing this in the first place<a class=\"zola-anchor\" href=\"#why-were-you-doing-this-in-the-first-place\" aria-label=\"Anchor link for: why-were-you-doing-this-in-the-first-place\"><\/a>\n<\/h4>\n<p>hahahaha so i had the foolish idea that this would get discord to silence notifications at night. <a href=\"https:\/\/support.discord.com\/hc\/en\/community\/posts\/22549582088343-Respect-desktop-s-Do-Not-Disturb-mode-for-desktop-notifications\">it does not do that<\/a>. i ended up just turning off desktop notification sounds altogether.<\/p>\n<section class=\"footnotes\">\n<ol class=\"footnotes-list\">\n<li id=\"fn-1\">\n<p>\"why not a systemd user timer\" because i don't know a helper that lets you write the timer schedule interactively, the way that <a href=\"https:\/\/cron.help\/\">https:\/\/cron.help\/<\/a> works for crontabs, and because systemd's documentation is smeared across a ton of different man pages. cron is just <code>crontab -e<\/code>. <a href=\"#fr-1-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-2\">\n<p>\"why not set cron to automatically write output to the system log\" hahahahaha there's no way to do that. unless you're using specifically <a href=\"https:\/\/man.archlinux.org\/man\/extra\/cronie\/cron.8.en#OPTIONS\"><code>cronie<\/code><\/a>, which isn't packaged in Ubuntu 22.04. <a href=\"#fr-2-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-3\">\n<p>see <a href=\"https:\/\/manpages.ubuntu.com\/manpages\/trusty\/man5\/crontab.5.html\"><code>man 5 crontab<\/code><\/a>; also this differs depending which version of cron you have installed. this link for instance does not match the man page i have installed locally for <code>man 5 crontab<\/code>. <a href=\"#fr-3-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-4\">\n<p>it's almost like <a href=\"https:\/\/blog.sunfishcode.online\/no-ghosts\/\">using environment variables to communicate data through a system is a bad idea<\/a>! <a href=\"#fr-4-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-5\">\n<p>note that recommends using <code>machinectl<\/code> or <code>systemctl --machine=$USER@localhost --user<\/code> instead. but both of those don't work in this environment: <code>machinectl<\/code> requires interactive login, and <code>--machine<\/code> just doesn't work at all:\n<code>Failed to connect to bus: Host is down<\/code>\n<code>Failed to list units: Transport endpoint is not connected<\/code> <a href=\"#fr-5-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-6\">\n<p>This uses bash because that sets <code>$UID<\/code> for us automatically. Theoretically we could do this with sh and <code>id -u<\/code>, but it's more of a pain than it's worth. <a href=\"#fr-6-1\">\u21a9<\/a><\/p>\n<\/li>\n<\/ol>\n<\/section>\n"},{"title":"building your own ","published":"2024-10-24T00:00:00+00:00","updated":"2024-10-24T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/rustc-driver\/"}},"id":"https:\/\/jyn.dev\/rustc-driver\/","summary":"what happens when you run <code>cargo clippy<\/code>?","content":"<h2 id=\"a-deeper-rabbit-hole-than-expected\">a deeper rabbit hole than expected<a class=\"zola-anchor\" href=\"#a-deeper-rabbit-hole-than-expected\" aria-label=\"Anchor link for: a-deeper-rabbit-hole-than-expected\"><\/a>\n<\/h2>\n<p>what happens when you run <code>cargo clippy<\/code>?<\/p>\n<p>well, we can ask cargo what it does:<\/p>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">$ cargo clippy -v\n<\/span><span class=\"z-text z-plain\">    Checking example v0.1.0 (\/home\/jyn\/src\/example)\n<\/span><span class=\"z-text z-plain\">     Running `\/home\/jyn\/.rustup\/toolchains\/nightly-x86_64-unknown-linux-gnu\/bin\/clippy-driver \/home\/jyn\/.rustup\/toolchains\/nightly-x86_64-unknown-linux-gnu\/bin\/rustc --crate-name example --edition=2021 src\/main.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --diagnostic-width=124 --crate-type bin --emit=dep-info,metadata -C embed-bitcode=no -C debuginfo=2 --check-cfg &#39;cfg(docsrs)&#39; --check-cfg &#39;cfg(feature, values())&#39; -C metadata=f3baefdd4f0d88a8 -C extra-filename=-f3baefdd4f0d88a8 --out-dir \/home\/jyn\/.cargo\/target\/debug\/deps -C incremental=\/home\/jyn\/.cargo\/target\/debug\/incremental -L dependency=\/home\/jyn\/.cargo\/target\/debug\/deps`\n<\/span><span class=\"z-text z-plain\">    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.67s\n<\/span><\/code><\/pre>\n<p>that's kinda weird! it's running something called <code>clippy-driver<\/code>? and passing <code>bin\/rustc<\/code> as an argument to it? what's going on there? is it running any other programs afterwards?<\/p>\n<p>we can find that out too:<\/p>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">; strace -f -e execve cargo clippy\n<\/span><\/code><\/pre>\n<details><summary>all of strace's output<\/summary>\n<p>note this is massaged slightly for clarity; the actual command i ran was<\/p>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">cargo clean; strace -f -e execve,clone3 -s 100 cargo clippy 2&gt;&amp;1 | sed &#39;s\/Process \/\\nProcess \/; s\/execve([^,]*, \\([^]]*\\).*\/execve(\\1])\/&#39; | rg -v &#39;resumed&gt;&#39; | rg &#39;(\\[pid[^]]*\\] )?(execve.*|clone3|Process.*)&#39; -o | sed &#39;s\/clone3\/clone3()\/&#39;\n<\/span><\/code><\/pre>\n<p>followed by some manual cleanup.<\/p>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">execve([&quot;cargo&quot;, &quot;clippy&quot;])\n<\/span><span class=\"z-text z-plain\">execve([&quot;\/home\/jyn\/.local\/lib\/rustup\/toolchains\/nightly-x86_64-unknown-linux-gnu\/bin\/cargo&quot;, &quot;clippy&quot;])\n<\/span><span class=\"z-text z-plain\">execve([&quot;\/home\/jyn\/.local\/lib\/cargo\/bin\/cargo-clippy&quot;, &quot;clippy&quot;])\n<\/span><span class=\"z-text z-plain\">execve([&quot;\/home\/jyn\/.local\/lib\/rustup\/toolchains\/nightly-x86_64-unknown-linux-gnu\/bin\/cargo-clippy&quot;, &quot;clippy&quot;])\n<\/span><span class=\"z-text z-plain\">clone3(): Process 6125 attached\n<\/span><span class=\"z-text z-plain\">[pid  6125] execve([&quot;\/home\/jyn\/.local\/lib\/rustup\/toolchains\/nightly-x86_64-unknown-linux-gnu\/bin\/cargo&quot;, &quot;check&quot;])\n<\/span><span class=\"z-text z-plain\">[pid  6125] clone3(): Process 6126 attached\n<\/span><span class=\"z-text z-plain\">[pid  6126] execve([&quot;\/home\/jyn\/.local\/lib\/rustup\/toolchains\/nightly-x86_64-unknown-linux-gnu\/bin\/clippy-driver&quot;, &quot;\/home\/jyn\/.local\/lib\/rustup\/toolchains\/nightly-x86_64-unknown-linux-gnu\/bin\/rustc&quot;, &quot;-vV&quot;])\n<\/span><span class=\"z-text z-plain\">[pid  6125] clone3(): Process 6127 attached\n<\/span><span class=\"z-text z-plain\">[pid  6127] execve([&quot;\/home\/jyn\/.local\/lib\/rustup\/toolchains\/nightly-x86_64-unknown-linux-gnu\/bin\/clippy-driver&quot;, &quot;\/home\/jyn\/.local\/lib\/rustup\/toolchains\/nightly-x86_64-unknown-linux-gnu\/bin\/rustc&quot;, &quot;-&quot;, &quot;--crate-name&quot;, &quot;___&quot;, &quot;--print=file-names&quot;, &quot;--crate-type&quot;, &quot;bin&quot;, &quot;--crate-type&quot;, &quot;rlib&quot;, &quot;--crate-type&quot;, &quot;dylib&quot;, &quot;--crate-type&quot;, &quot;cdylib&quot;, &quot;--crate-type&quot;, &quot;staticlib&quot;, &quot;--crate-type&quot;, &quot;proc-macro&quot;, &quot;--check-cfg&quot;, &quot;cfg()&quot;])\n<\/span><span class=\"z-text z-plain\">[pid  6125] clone3(): Process 6129 attached\n<\/span><span class=\"z-text z-plain\">[pid  6129] execve([&quot;\/home\/jyn\/.local\/lib\/rustup\/toolchains\/nightly-x86_64-unknown-linux-gnu\/bin\/clippy-driver&quot;, &quot;\/home\/jyn\/.local\/lib\/rustup\/toolchains\/nightly-x86_64-unknown-linux-gnu\/bin\/rustc&quot;, &quot;-&quot;, &quot;--crate-name&quot;, &quot;___&quot;, &quot;--print=file-names&quot;, &quot;--crate-type&quot;, &quot;bin&quot;, &quot;--crate-type&quot;, &quot;rlib&quot;, &quot;--crate-type&quot;, &quot;dylib&quot;, &quot;--crate-type&quot;, &quot;cdylib&quot;, &quot;--crate-type&quot;, &quot;staticlib&quot;, &quot;--crate-type&quot;, &quot;proc-macro&quot;, &quot;--print=sysroot&quot;, &quot;--print=split-debuginfo&quot;, &quot;--print=crate-name&quot;, &quot;--print=cfg&quot;])\n<\/span><span class=\"z-text z-plain\">[pid  6125] clone3(): Process 6131 attached\n<\/span><span class=\"z-text z-plain\">[pid  6131] execve([&quot;\/home\/jyn\/.local\/lib\/rustup\/toolchains\/nightly-x86_64-unknown-linux-gnu\/bin\/clippy-driver&quot;, &quot;\/home\/jyn\/.local\/lib\/rustup\/toolchains\/nightly-x86_64-unknown-linux-gnu\/bin\/rustc&quot;, &quot;-vV&quot;])\n<\/span><span class=\"z-text z-plain\">[pid  6125] clone3(): Process 6134 attached\n<\/span><span class=\"z-text z-plain\">[pid  6134] execve([&quot;\/home\/jyn\/.local\/lib\/rustup\/toolchains\/nightly-x86_64-unknown-linux-gnu\/bin\/clippy-driver&quot;, &quot;\/home\/jyn\/.local\/lib\/rustup\/toolchains\/nightly-x86_64-unknown-linux-gnu\/bin\/rustc&quot;, &quot;--crate-name&quot;, &quot;example&quot;, &quot;--edition=2021&quot;, &quot;src\/main.rs&quot;, &quot;--error-format=json&quot;, &quot;--json=diagnostic-rendered-ansi,artifacts,future-incompat&quot;, &quot;--crate-type&quot;, &quot;bin&quot;, &quot;--emit=dep-info,metadata&quot;, &quot;-C&quot;, &quot;embed-bitcode=no&quot;, &quot;-C&quot;, &quot;debuginfo=2&quot;, &quot;--check-cfg&quot;, &quot;cfg(docsrs)&quot;, &quot;--check-cfg&quot;, &quot;cfg(feature, values())&quot;, &quot;-C&quot;, &quot;metadata=f3baefdd4f0d88a8&quot;, &quot;-C&quot;, &quot;extra-filename=-f3baefdd4f0d88a8&quot;, &quot;--out-dir&quot;, &quot;\/home\/jyn\/.local\/lib\/cargo\/target\/debug\/deps&quot;, &quot;-C&quot;, &quot;incremental=\/home\/jyn\/.local\/lib\/cargo\/target\/debug\/incremental&quot;, &quot;-L&quot;, &quot;dependency=\/home\/jyn\/.local\/lib\/cargo\/target\/debug\/deps&quot;])\n<\/span><\/code><\/pre>\n<\/details>\n<p>wow, we have like 10 different commands running here! <sup class=\"footnote-reference\" id=\"fr-paths-1\"><a href=\"#fn-paths\">1<\/a><\/sup><\/p>\n<ol>\n<li><code>~\/.cargo\/bin\/cargo clippy -v<\/code><\/li>\n<li><code>~\/.rustup\/toolchains\/nightly-x86_64-unknown-linux-gnu\/bin\/cargo clippy -v<\/code><\/li>\n<li><code>~\/.cargo\/bin\/cargo-clippy -v<\/code><\/li>\n<li><code>~\/.rustup\/...\/cargo-clippy clippy -v<\/code><\/li>\n<li><code>~\/.rustup\/...\/cargo check -v<\/code><\/li>\n<li><code>~\/.rustup\/...\/clippy-driver ~\/.rustup\/...\/rustc -vV<\/code><\/li>\n<li><code>~\/.rustup\/...\/clippy-driver ~\/.rustup\/..\/rustc - --print=file-names<\/code><\/li>\n<li><code>~\/.rustup\/...\/clippy-driver ~\/.rustup\/...\/rustc -- --print=sysroot<\/code><\/li>\n<li><code>~\/.rustup\/...\/clippy-driver ~\/.rustup\/...\/rustc -vV<\/code><\/li>\n<li><code>~\/.rustup\/...\/clippy-driver ~\/.rustup\/..\/rustc src\/main.rs<\/code><\/li>\n<\/ol>\n<p>that's a lot of indirection to compile a simple hello world! let's dig into why it happens.<\/p>\n<h2 id=\"rustup-toolchain-proxies\">Rustup toolchain proxies<a class=\"zola-anchor\" href=\"#rustup-toolchain-proxies\" aria-label=\"Anchor link for: rustup-toolchain-proxies\"><\/a>\n<\/h2>\n<p>twice, we see the same binary executed first from <code>~\/.cargo<\/code>, then again from <code>~\/.rustup<\/code>: for <code>cargo<\/code> and for <code>cargo-clippy<\/code>. what's the difference between these?<\/p>\n<p>let's take a look at the first one that runs:<\/p>\n<details><summary>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">; strace -e readlink,openat -z ~\/.local\/lib\/cargo\/bin\/cargo\n<\/span><span class=\"z-text z-plain\">readlink(&quot;\/proc\/self\/exe&quot;, &quot;\/home\/jyn\/.local\/lib\/cargo\/bin\/cargo&quot;, 256) = 36\n<\/span><span class=\"z-text z-plain\">openat(AT_FDCWD, &quot;\/home\/jyn\/.local\/lib\/rustup\/settings.toml&quot;, O_RDONLY|O_CLOEXEC) = 3\n<\/span><\/code><\/pre>\n<\/summary>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">; strace -e readlink,openat -s 100 -z ~\/.local\/lib\/cargo\/bin\/cargo 2&gt;&amp;1| rg &#39;readlink|openat&#39; | rg -v &#39;&quot;\/(lib|etc)&#39;\n<\/span><span class=\"z-text z-plain\">openat(AT_FDCWD, &quot;\/proc\/self\/maps&quot;, O_RDONLY|O_CLOEXEC) = 3\n<\/span><span class=\"z-text z-plain\">readlink(&quot;\/proc\/self\/exe&quot;, &quot;\/home\/jyn\/.local\/lib\/cargo\/bin\/cargo&quot;, 256) = 36\n<\/span><span class=\"z-text z-plain\">openat(AT_FDCWD, &quot;\/home\/jyn\/.local\/lib\/rustup\/settings.toml&quot;, O_RDONLY|O_CLOEXEC) = 3\n<\/span><span class=\"z-text z-plain\">openat(AT_FDCWD, &quot;\/home\/jyn\/.local\/lib\/rustup\/toolchains&quot;, O_RDONLY|O_CLOEXEC) = 3\n<\/span><span class=\"z-text z-plain\">openat(3, &quot;nightly-x86_64-unknown-linux-gnu&quot;, O_RDONLY|O_NOCTTY|O_CLOEXEC) = 4\n<\/span><span class=\"z-text z-plain\">openat(AT_FDCWD, &quot;\/home\/jyn\/.local\/lib\/rustup\/toolchains\/nightly-x86_64-unknown-linux-gnu\/lib\/rustlib\/rust-installer-version&quot;, O_RDONLY|O_CLOEXEC) = 3\n<\/span><span class=\"z-text z-plain\">openat(AT_FDCWD, &quot;\/home\/jyn\/.local\/lib\/rustup\/toolchains\/nightly-x86_64-unknown-linux-gnu\/lib\/rustlib\/multirust-channel-manifest.toml&quot;, O_RDONLY|O_CLOEXEC) = 3\n<\/span><span class=\"z-text z-plain\">openat(AT_FDCWD, &quot;\/home\/jyn\/.local\/lib\/rustup\/toolchains\/nightly-x86_64-unknown-linux-gnu\/lib\/rustlib\/multirust-config.toml&quot;, O_RDONLY|O_CLOEXEC) = 3\n<\/span><span class=\"z-text z-plain\">readlink(&quot;\/proc\/self\/exe&quot;, &quot;\/home\/jyn\/.local\/lib\/rustup\/toolchains\/nightly-x86_64-unknown-linux-gnu\/bin\/cargo&quot;, 4096) = 81\n<\/span><span class=\"z-text z-plain\">openat(AT_FDCWD, &quot;\/proc\/self\/maps&quot;, O_RDONLY|O_CLOEXEC) = 3\n<\/span><\/code><\/pre>\n<\/details>\n<p>it turns out this is something called a <a href=\"https:\/\/rust-lang.github.io\/rustup\/concepts\/proxies.html\">rustup proxy<\/a>. this is a little shim that <a href=\"https:\/\/blog.axo.dev\/2024\/07\/an-app-by-any-other-name\">hardlinks<\/a> from the proxy (in this case <code>cargo<\/code>) to rustup. rustup then looks at its own executable name, notices that it's cargo, and picks the right toolchain to run the actual cargo binary.<\/p>\n<p>you can actually ask rustup to just find the path without running it:<\/p>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">; rustup which cargo\n<\/span><span class=\"z-text z-plain\">\/home\/jyn\/.rustup\/toolchains\/nightly-x86_64-unknown-linux-gnu\/bin\/cargo\n<\/span><\/code><\/pre>\n<p>and you can see that the real cargo doesn't support rustup features like toolchain selectors:<\/p>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">; cargo +nightly --version\n<\/span><span class=\"z-text z-plain\">cargo 1.81.0-nightly (4dcbca118 2024-06-11)\n<\/span><span class=\"z-text z-plain\">; $(rustup which cargo) +nightly --version\n<\/span><span class=\"z-text z-plain\">error: no such command: `+nightly`\n<\/span><span class=\"z-text z-plain\">\n<\/span><span class=\"z-text z-plain\">        Cargo does not handle `+toolchain` directives.\n<\/span><span class=\"z-text z-plain\">        Did you mean to invoke `cargo` through `rustup` instead?\n<\/span><\/code><\/pre>\n<p>you can even verify for yourself that <code>~\/.cargo\/bin\/cargo<\/code> is just a hardlink by comparing the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Inode\">inodes<\/a>:<\/p>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">; ls -i \/home\/jyn\/.cargo\/bin\/rustup\n<\/span><span class=\"z-text z-plain\">32294517 \/home\/jyn\/.cargo\/bin\/rustup\n<\/span><span class=\"z-text z-plain\">; ls -i \/home\/jyn\/.cargo\/bin\/rustc\n<\/span><span class=\"z-text z-plain\">32294517 \/home\/jyn\/.cargo\/bin\/rustc\n<\/span><\/code><\/pre>\n<h2 id=\"cargo-extensions\">cargo extensions<a class=\"zola-anchor\" href=\"#cargo-extensions\" aria-label=\"Anchor link for: cargo-extensions\"><\/a>\n<\/h2>\n<p>ok, we've figured out 2\/10 of the processes. what are the other 8?<\/p>\n<p>the first one that runs after <code>cargo clippy<\/code> is ... <code>cargo-clippy<\/code> (with a <em>-<\/em> dash, not a space). this is a <a href=\"https:\/\/doc.rust-lang.org\/cargo\/reference\/external-tools.html#custom-subcommands\">cargo extension<\/a>; <code>cargo clippy<\/code> is doing very little here other than setting <code>CARGO<\/code> in the environment to point back to itself. why does it need to do that? because <code>cargo-clippy<\/code> is about to call back to cargo: that's process 5, <code>cargo check<\/code>.<\/p>\n<p>what's changed is that <a href=\"https:\/\/github.com\/rust-lang\/rust-clippy\/blob\/cefa31a5243b90c0c606e2fdb3fc3e036a8bec16\/src\/main.rs#L109-L126\"><code>cargo-clippy<\/code> has set<\/a> <a href=\"https:\/\/doc.rust-lang.org\/cargo\/reference\/config.html#buildrustc-workspace-wrapper\"><code>RUSTC_WORKSPACE_WRAPPER<\/code><\/a> to <code>clippy-driver<\/code>. that means that instead of invoking rustc on each crate, it's going to invoke clippy-driver. and because it's a <code>WRAPPER<\/code>, it gets passed <code>rustc<\/code> as an argument. the intended use-case is for tools like <a href=\"https:\/\/github.com\/mozilla\/sccache\"><code>sccache<\/code><\/a> that transparently wrap rustc before invoking it; <a href=\"https:\/\/github.com\/rust-lang\/rust-clippy\/blob\/9cf416dc6ec9d7ebc8df299ca970e4a92efa2596\/src\/driver.rs#L229-L255\">clippy never invokes rustc as a process<\/a>, but there's no equivalent of <a href=\"https:\/\/doc.rust-lang.org\/cargo\/reference\/config.html#buildrustc\"><code>RUSTC<\/code><\/a> that's only used for workspaces.<\/p>\n<h2 id=\"sysroots-and-dynamic-libraries\">sysroots and dynamic libraries<a class=\"zola-anchor\" href=\"#sysroots-and-dynamic-libraries\" aria-label=\"Anchor link for: sysroots-and-dynamic-libraries\"><\/a>\n<\/h2>\n<p>now, at this point you might imagine that we have enough info to write our own clippy-like tool that gets invoked instead of rustc. if we look at <a href=\"https:\/\/github.com\/rust-lang\/rust\/blob\/b8bb2968ce1e44d01520c9d59ee6299ed66df3f9\/compiler\/rustc\/src\/main.rs\"><code>compiler\/rustc\/src\/main.rs<\/code> in rust-lang\/rust<\/a>, other than some weird jemalloc stuff, all it does is call <code>rustc_driver::main<\/code>. let's write a toy program that does that.<\/p>\n<pre data-lang=\"rust\" class=\"language-rust z-code\"><code class=\"language-rust\" data-lang=\"rust\"><span class=\"z-source z-rust\"><span class=\"z-punctuation z-terminator z-rust\">;<\/span> cargo new driver\n<\/span><span class=\"z-source z-rust\"><span class=\"z-punctuation z-terminator z-rust\">;<\/span> cd driver\n<\/span><span class=\"z-source z-rust\"><span class=\"z-punctuation z-terminator z-rust\">;<\/span> echo <span class=\"z-keyword z-operator z-rust\">&#39;<\/span><span class=\"z-meta z-annotation z-rust\"><span class=\"z-punctuation z-definition z-annotation z-rust\">#!<\/span><span class=\"z-punctuation z-section z-group z-begin z-rust\">[<\/span><span class=\"z-variable z-annotation z-rust\">feature<\/span><span class=\"z-meta z-annotation z-parameters z-rust\"><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><\/span><span class=\"z-meta z-annotation z-parameters z-rust\"><span class=\"z-meta z-group z-rust\">rustc_private<\/span><\/span><span class=\"z-meta z-annotation z-parameters z-rust\"><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><\/span><span class=\"z-punctuation z-section z-group z-end z-rust\">]<\/span><\/span>\n<\/span><span class=\"z-source z-rust\"><span class=\"z-keyword z-other z-rust\">extern<\/span> <span class=\"z-keyword z-other z-rust\">crate<\/span> rustc_driver<span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-storage z-type z-function z-rust\">fn<\/span> <\/span><span class=\"z-entity z-name z-function z-rust\">main<\/span><\/span><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-end z-rust\">)<\/span><\/span><\/span><\/span><span class=\"z-meta z-function z-rust\"> <\/span><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\">  <span class=\"z-support z-macro z-rust\">println!<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-string z-quoted z-double z-rust\"><span class=\"z-punctuation z-definition z-string z-begin z-rust\">&quot;<\/span>this is a custom driver!<span class=\"z-punctuation z-definition z-string z-end z-rust\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\">  <span class=\"z-meta z-path z-rust\">rustc_driver<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span>main<span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\"><\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span><\/span><span class=\"z-keyword z-operator z-rust\">&#39;<\/span> <span class=\"z-keyword z-operator z-comparison z-rust\">&gt;<\/span> src<span class=\"z-keyword z-operator z-arithmetic z-rust\">\/<\/span>main<span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span>rs\n<\/span><span class=\"z-source z-rust\"><span class=\"z-punctuation z-terminator z-rust\">;<\/span> cargo <span class=\"z-keyword z-operator z-arithmetic z-rust\">+<\/span>nightly run\n<\/span><span class=\"z-source z-rust\">error<span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">[<\/span><span class=\"z-constant z-other z-rust\">E0463<\/span><span class=\"z-punctuation z-section z-group z-end z-rust\">]<\/span><\/span><span class=\"z-punctuation z-separator z-rust\">:<\/span> can<span class=\"z-storage z-modifier z-lifetime z-rust\">&#39;t<\/span> find <span class=\"z-keyword z-other z-rust\">crate<\/span> <span class=\"z-keyword z-control z-rust\">for<\/span> `rustc_driver`\n<\/span><span class=\"z-source z-rust\"> <span class=\"z-keyword z-operator z-arithmetic z-rust\">-<\/span><span class=\"z-meta z-function z-return-type z-rust\"><span class=\"z-punctuation z-separator z-rust\">-&gt;<\/span> src<\/span><span class=\"z-keyword z-operator z-arithmetic z-rust\">\/<\/span>main<span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span>rs<span class=\"z-punctuation z-separator z-rust\">:<\/span><span class=\"z-constant z-numeric z-integer z-decimal z-rust\">2<\/span><span class=\"z-punctuation z-separator z-rust\">:<\/span><span class=\"z-constant z-numeric z-integer z-decimal z-rust\">1<\/span>\n<\/span><span class=\"z-source z-rust\">  <span class=\"z-keyword z-operator z-bitwise z-rust\">|<\/span>\n<\/span><span class=\"z-source z-rust\"><span class=\"z-constant z-numeric z-integer z-decimal z-rust\">2<\/span> <span class=\"z-keyword z-operator z-bitwise z-rust\">|<\/span> <span class=\"z-keyword z-other z-rust\">extern<\/span> <span class=\"z-keyword z-other z-rust\">crate<\/span> rustc_driver<span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><span class=\"z-source z-rust\">  <span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-begin z-rust\">|<\/span><\/span><\/span><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"> ^^^^^^^^^^^^^^^^^^^^^^^^^^ can<span class=\"z-storage z-modifier z-lifetime z-rust\">&#39;t<\/span> find crate<\/span>\n<\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-closure z-rust\">  <\/span><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-keyword z-operator z-bitwise z-rust\">|<\/span><\/span>\n<\/span><span class=\"z-source z-rust\">  <span class=\"z-keyword z-operator z-assignment z-rust\">=<\/span> help<span class=\"z-punctuation z-separator z-rust\">:<\/span> maybe you need to install the missing components with<span class=\"z-punctuation z-separator z-rust\">:<\/span> `rustup component add rust<span class=\"z-keyword z-operator z-arithmetic z-rust\">-<\/span>src rustc<span class=\"z-keyword z-operator z-arithmetic z-rust\">-<\/span>dev llvm<span class=\"z-keyword z-operator z-arithmetic z-rust\">-<\/span>tools<span class=\"z-keyword z-operator z-arithmetic z-rust\">-<\/span>preview`\n<\/span><\/code><\/pre>\n<p>oh. huh. that's kinda weird. let's follow those instructions for now though.<\/p>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">; rustup component add rust-src rustc-dev llvm-tools-preview\n<\/span><span class=\"z-text z-plain\">; cargo +nightly run\n<\/span><span class=\"z-text z-plain\">   Compiling driver v0.1.0 (\/home\/jyn\/src\/driver)\n<\/span><span class=\"z-text z-plain\">    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.08s\n<\/span><span class=\"z-text z-plain\">     Running `\/home\/jyn\/.cargo\/target\/debug\/driver`\n<\/span><span class=\"z-text z-plain\">this is a custom driver!\n<\/span><span class=\"z-text z-plain\">Usage: rustc [OPTIONS] INPUT\n<\/span><span class=\"z-text z-plain\">...\n<\/span><\/code><\/pre>\n<p>this is pretty cool! let's install it for our user and see if we can run it on a rust file.<\/p>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">; cargo install --path . --debug\n<\/span><span class=\"z-text z-plain\">  Installing driver v0.1.0 (\/home\/jyn\/src\/driver)\n<\/span><span class=\"z-text z-plain\">    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.01s\n<\/span><span class=\"z-text z-plain\">  Installing \/home\/jyn\/.cargo\/bin\/driver\n<\/span><span class=\"z-text z-plain\">   Installed package `driver v0.1.0 (\/home\/jyn\/src\/driver)` (executable `driver`)\n<\/span><span class=\"z-text z-plain\">; driver src\/main.rs\n<\/span><span class=\"z-text z-plain\">driver: error while loading shared libraries: librustc_driver-5396912e8af1f65d.so: cannot open shared object file: No such file or directory\n<\/span><\/code><\/pre>\n<p>well that's unfortunate. what happened here?<\/p>\n<p>when we use <code>cargo run<\/code> for our driver, it sets some environment variables. in particular, it sets a variable called <code>LD_LIBRARY_PATH<\/code>:<\/p>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">; strace -s 1000 -v -e execve cargo run 2&gt;&amp;1 &gt;\/dev\/null | rg debug\/driver | rg -o &#39;LD_LIBRARY_PATH=[^&quot;]*&#39;\n<\/span><span class=\"z-text z-plain\">LD_LIBRARY_PATH=\/home\/jyn\/.cargo\/target\/debug\/deps:\/home\/jyn\/.cargo\/target\/debug:\/home\/jyn\/.rustup\/toolchains\/nightly-x86_64-unknown-linux-gnu\/lib\/rustlib\/x86_64-unknown-linux-gnu\/lib:\/home\/jyn\/.rustup\/toolchains\/nightly-x86_64-unknown-linux-gnu\/lib\n<\/span><\/code><\/pre>\n<p>and indeed if we set that variable, our driver works again:<\/p>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">; env &quot;$(strace -s 1000 -v -e execve cargo run 2&gt;&amp;1 &gt;\/dev\/null | rg debug\/driver | rg -o &#39;LD_LIBRARY_PATH=[^&quot;]*&#39;)&quot; driver src\/main.rs\n<\/span><span class=\"z-text z-plain\">this is a custom driver!\n<\/span><\/code><\/pre>\n<p>what does it do?<\/p>\n<p>well, <a href=\"https:\/\/tldp.org\/HOWTO\/Program-Library-HOWTO\/shared-libraries.html#AEN80\">the linux documentation project<\/a> documents it as follows:<\/p>\n<blockquote>\n<p>You can temporarily substitute a different library for this particular execution. In Linux, the environment variable LD_LIBRARY_PATH is a colon-separated set of directories where libraries should be searched for first, before the standard set of directories; this is useful when debugging a new library or using a nonstandard library for special purposes<\/p>\n<\/blockquote>\n<p>\"library\" here means a shared object library. on linux, these files end with <code>.so<\/code>; on Windows, <code>.dll<\/code> (for \"dynamic-link library\") ; on MacOS, <code>.dylib<\/code>. these are object files that contain code (\"symbols\" in linker terms) that are loaded by the linker at runtime, after the rest of your program is loaded into memory. for example, if we look at the rustc binary we copied our driver's code from, it's almost empty:<\/p>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">;; stat -c %s $(rustup which rustc) | numfmt --to=iec\n<\/span><span class=\"z-text z-plain\">2.6M\n<\/span><\/code><\/pre>\n<p>instead, almost all the code is in a shared object:<\/p>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">; stat -c %s \/home\/jyn\/.rustup\/toolchains\/nightly-x86_64-unknown-linux-gnu\/lib\/librustc_driver-5396912e8af1f65d.so | numfmt --to=iec\n<\/span><span class=\"z-text z-plain\">136M\n<\/span><\/code><\/pre>\n<p>that shared object gets loaded at runtime by every rustc_driver: <code>rustc<\/code>, <code>clippy<\/code>, <code>rustdoc<\/code>, <code>miri<\/code> - and our new <code>driver<\/code> tool. it's actually shipped with every toolchain; if you look at the stable toolchain in <code>.rustup<\/code> you'll see it there too. however, what you <em>won't<\/em> see is the <code>.rmeta<\/code> files in the toolchain directory: <sup class=\"footnote-reference\" id=\"fr-target-libdir-1\"><a href=\"#fn-target-libdir\">2<\/a><\/sup><\/p>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">; ls \/home\/jyn\/.rustup\/toolchains\/nightly-x86_64-unknown-linux-gnu\/lib\/rustlib\/x86_64-unknown-linux-gnu\/lib\/*.rmeta | wc -l\n<\/span><span class=\"z-text z-plain\">228\n<\/span><\/code><\/pre>\n<p>those are the files we just installed with <code>rustup component add rustc-dev<\/code>. they give rustc the type information it needs to check our <code>driver<\/code> code against the type signatures of all the <code>rustc_private<\/code> internal crates.<\/p>\n<h2 id=\"building-a-driver\">building a driver<a class=\"zola-anchor\" href=\"#building-a-driver\" aria-label=\"Anchor link for: building-a-driver\"><\/a>\n<\/h2>\n<p>so! we have learned a bunch of stuff:<\/p>\n<ol>\n<li>when we run <code>cargo<\/code> or <code>rustc<\/code>, that's actually running a \"rustup proxy\" that picks the toolchain version to use<\/li>\n<li>when we run <code>cargo clippy<\/code>, that's doing a complicated back and forth between cargo and various clippy executables<\/li>\n<li>when we run <code>rustc<\/code> or <code>clippy-driver<\/code>, that's loading a shared object called <code>librustc_driver.so<\/code> at runtime<\/li>\n<\/ol>\n<p>that's enough info we can now build our own driver!<\/p>\n<p>first, let's make sure we can always run our driver even if we're not going through cargo. it turns out there's a linker flag for this called <a href=\"https:\/\/man.openbsd.org\/man1\/ld.bfd.1#rpath\"><code>-rpath<\/code><\/a>, which does essentially the same thing as <code>LD_LIBRARY_PATH<\/code> but at link time instead of at runtime. to get the path we need, we can use <a href=\"https:\/\/doc.rust-lang.org\/stable\/rustc\/command-line-arguments.html#--print-print-compiler-information\"><code>rustc --print=sysroot<\/code><\/a> (we can get the right version of <code>rustc<\/code> from an env variable cargo sets, <a href=\"https:\/\/doc.rust-lang.org\/cargo\/reference\/environment-variables.html#environment-variables-cargo-sets-for-build-scripts\"><code>RUSTC<\/code><\/a>). and finally, to tell cargo to pass that flag to the linker, we can write a build script that uses <a href=\"https:\/\/doc.rust-lang.org\/cargo\/reference\/build-scripts.html#rustc-link-arg\"><code>cargo:rustc-link-arg<\/code><\/a>. <sup class=\"footnote-reference\" id=\"fr-rpath-origin-1\"><a href=\"#fn-rpath-origin\">3<\/a><\/sup><\/p>\n<pre data-lang=\"rust\" class=\"language-rust z-code\"><code class=\"language-rust\" data-lang=\"rust\"><span class=\"z-source z-rust\"><span class=\"z-comment z-line z-double-slash z-rust\"><span class=\"z-punctuation z-definition z-comment z-rust\">\/\/<\/span> build.rs\n<\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-storage z-type z-function z-rust\">fn<\/span> <\/span><span class=\"z-entity z-name z-function z-rust\">main<\/span><\/span><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-end z-rust\">)<\/span><\/span><\/span><\/span><span class=\"z-meta z-function z-rust\"> <\/span><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\">    <span class=\"z-storage z-type z-rust\">let<\/span> rustc <span class=\"z-keyword z-operator z-assignment z-rust\">=<\/span> <span class=\"z-meta z-path z-rust\">std<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span><span class=\"z-meta z-path z-rust\">env<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span>var<span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-string z-quoted z-double z-rust\"><span class=\"z-punctuation z-definition z-string z-begin z-rust\">&quot;<\/span>RUSTC<span class=\"z-punctuation z-definition z-string z-end z-rust\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">unwrap<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\">    <span class=\"z-storage z-type z-rust\">let<\/span> output <span class=\"z-keyword z-operator z-assignment z-rust\">=<\/span> <span class=\"z-meta z-path z-rust\">std<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span><span class=\"z-meta z-path z-rust\">process<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span><span class=\"z-meta z-path z-rust\">Command<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span>new<span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span>rustc<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">arg<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-string z-quoted z-double z-rust\"><span class=\"z-punctuation z-definition z-string z-begin z-rust\">&quot;<\/span>--print=sysroot<span class=\"z-punctuation z-definition z-string z-end z-rust\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">output<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\">    <span class=\"z-storage z-type z-rust\">let<\/span> stdout <span class=\"z-keyword z-operator z-assignment z-rust\">=<\/span> <span class=\"z-support z-type z-rust\">String<\/span><span class=\"z-meta z-path z-rust\"><span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span>from_utf8<span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span>output<span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">unwrap<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span>stdout<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">unwrap<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\">    <span class=\"z-storage z-type z-rust\">let<\/span> sysroot <span class=\"z-keyword z-operator z-assignment z-rust\">=<\/span> stdout<span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">trim_end<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\">    <span class=\"z-support z-macro z-rust\">println!<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-string z-quoted z-double z-rust\"><span class=\"z-punctuation z-definition z-string z-begin z-rust\">&quot;<\/span>cargo:rustc-link-arg=-Wl,-rpath=<span class=\"z-constant z-other z-placeholder z-rust\">{sysroot}<\/span>\/lib<span class=\"z-punctuation z-definition z-string z-end z-rust\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\"><\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span><\/span>\n<\/span><\/code><\/pre>\n<p>next, we want to be able to invoke this as <code>cargo driver<\/code>, so we need to write a <code>cargo-driver<\/code> program.<\/p>\n<pre data-lang=\"rust\" class=\"language-rust z-code\"><code class=\"language-rust\" data-lang=\"rust\"><span class=\"z-source z-rust\"><span class=\"z-comment z-line z-double-slash z-rust\"><span class=\"z-punctuation z-definition z-comment z-rust\">\/\/<\/span> src\/bin\/cargo-driver.rs\n<\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-keyword z-other z-rust\">use<\/span> <span class=\"z-meta z-path z-rust\">std<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span>env<span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-storage z-type z-function z-rust\">fn<\/span> <\/span><span class=\"z-entity z-name z-function z-rust\">main<\/span><\/span><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-end z-rust\">)<\/span><\/span><\/span><\/span><span class=\"z-meta z-function z-rust\"> <span class=\"z-meta z-function z-return-type z-rust\"><span class=\"z-punctuation z-separator z-rust\">-&gt;<\/span> <span class=\"z-meta z-generic z-rust\"><span class=\"z-support z-type z-rust\">Result<\/span><span class=\"z-punctuation z-definition z-generic z-begin z-rust\">&lt;<\/span><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span>, <span class=\"z-storage z-type z-rust\">i32<\/span><span class=\"z-punctuation z-definition z-generic z-end z-rust\">&gt;<\/span><\/span><\/span> <\/span><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\">    <span class=\"z-storage z-type z-rust\">let<\/span> cargo <span class=\"z-keyword z-operator z-assignment z-rust\">=<\/span> <span class=\"z-meta z-path z-rust\">env<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span>var<span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-string z-quoted z-double z-rust\"><span class=\"z-punctuation z-definition z-string z-begin z-rust\">&quot;<\/span>CARGO<span class=\"z-punctuation z-definition z-string z-end z-rust\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">unwrap_or<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-string z-quoted z-double z-rust\"><span class=\"z-punctuation z-definition z-string z-begin z-rust\">&quot;<\/span>cargo<span class=\"z-punctuation z-definition z-string z-end z-rust\">&quot;<\/span><\/span><span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">into<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\">    <span class=\"z-storage z-type z-rust\">let<\/span> <span class=\"z-storage z-modifier z-rust\">mut<\/span> cmd <span class=\"z-keyword z-operator z-assignment z-rust\">=<\/span> <span class=\"z-meta z-path z-rust\">std<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span><span class=\"z-meta z-path z-rust\">process<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span><span class=\"z-meta z-path z-rust\">Command<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span>new<span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span>cargo<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\">    <span class=\"z-storage z-type z-rust\">let<\/span> driver <span class=\"z-keyword z-operator z-assignment z-rust\">=<\/span> <span class=\"z-meta z-path z-rust\">env<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span>current_exe<span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">unwrap<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">with_file_name<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-string z-quoted z-double z-rust\"><span class=\"z-punctuation z-definition z-string z-begin z-rust\">&quot;<\/span>driver<span class=\"z-punctuation z-definition z-string z-end z-rust\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\">    <span class=\"z-storage z-type z-rust\">let<\/span> status <span class=\"z-keyword z-operator z-assignment z-rust\">=<\/span> cmd<span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">arg<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-string z-quoted z-double z-rust\"><span class=\"z-punctuation z-definition z-string z-begin z-rust\">&quot;<\/span>build<span class=\"z-punctuation z-definition z-string z-end z-rust\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">env<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-string z-quoted z-double z-rust\"><span class=\"z-punctuation z-definition z-string z-begin z-rust\">&quot;<\/span>RUSTC<span class=\"z-punctuation z-definition z-string z-end z-rust\">&quot;<\/span><\/span><span class=\"z-punctuation z-separator z-rust\">,<\/span> driver<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">status<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">unwrap<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\">    <span class=\"z-keyword z-control z-rust\">match<\/span> status<span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">code<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span> <span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-block z-rust\">        <span class=\"z-support z-type z-rust\">Some<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-constant z-numeric z-integer z-decimal z-rust\">0<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span> <span class=\"z-keyword z-operator z-rust\">=&gt;<\/span> <span class=\"z-support z-type z-rust\">Ok<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-separator z-rust\">,<\/span>\n<\/span><\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-block z-rust\">        <span class=\"z-support z-type z-rust\">Some<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span>other<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span> <span class=\"z-keyword z-operator z-rust\">=&gt;<\/span> <span class=\"z-support z-type z-rust\">Err<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span>other<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-separator z-rust\">,<\/span>\n<\/span><\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-block z-rust\">        <span class=\"z-support z-type z-rust\">None<\/span> <span class=\"z-keyword z-operator z-rust\">=&gt;<\/span> <span class=\"z-support z-type z-rust\">Err<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-keyword z-operator z-arithmetic z-rust\">-<\/span><span class=\"z-constant z-numeric z-integer z-decimal z-rust\">1<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-separator z-rust\">,<\/span>\n<\/span><\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-block z-rust\">    <\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\"><\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span><\/span>\n<\/span><\/code><\/pre>\n<p>note that this works fine when using <code>cargo install<\/code>, but when using <code>cargo run --bin cargo-driver<\/code> locally, cargo doesn't rebuild all your binaries automatically - you need to manually run <code>cargo build<\/code> first to make sure both executables are updated. <sup class=\"footnote-reference\" id=\"fr-bindeps-1\"><a href=\"#fn-bindeps\">4<\/a><\/sup><\/p>\n<h2 id=\"we-get-a-little-silly-with-it-3\">we get a little silly with it :3<a class=\"zola-anchor\" href=\"#we-get-a-little-silly-with-it-3\" aria-label=\"Anchor link for: we-get-a-little-silly-with-it-3\"><\/a>\n<\/h2>\n<p>now let's make a program that does something fun! i'm going to disable the <code>unsafe<\/code> checker, so that we can write unsafe programs in \"safe\" rust.<\/p>\n<p>first, let's tell rust-analyzer that we're using rustc internals in this crate, so we get go-to-definition and other nice things:<\/p>\n<pre data-lang=\"toml\" class=\"language-toml z-code\"><code class=\"language-toml\" data-lang=\"toml\"><span class=\"z-source z-toml\"><span class=\"z-comment z-line z-number-sign z-toml\"><span class=\"z-punctuation z-definition z-comment z-toml\">#<\/span> Cargo.toml<\/span>\n<\/span><span class=\"z-source z-toml\"><span class=\"z-punctuation z-definition z-table z-begin z-toml\">[<\/span><span class=\"z-meta z-tag z-table z-toml\"><span class=\"z-entity z-name z-table z-toml\">package<\/span><span class=\"z-punctuation z-separator z-table z-toml\">.<\/span><span class=\"z-entity z-name z-table z-toml\">metadata<\/span><span class=\"z-punctuation z-separator z-table z-toml\">.<\/span><span class=\"z-entity z-name z-table z-toml\">rust-analyzer<\/span><\/span><span class=\"z-punctuation z-definition z-table z-end z-toml\">]<\/span>\n<\/span><span class=\"z-source z-toml\"><span class=\"z-comment z-line z-number-sign z-toml\"><span class=\"z-punctuation z-definition z-comment z-toml\">#<\/span> This package uses #[feature(rustc_private)]<\/span>\n<\/span><span class=\"z-source z-toml\"><span class=\"z-meta z-tag z-key z-toml\"><span class=\"z-entity z-name z-tag z-toml\">rustc_private<\/span><\/span> <span class=\"z-punctuation z-definition z-key-value z-toml\">=<\/span> <span class=\"z-constant z-language z-toml\">true<\/span>\n<\/span><\/code><\/pre>\n<p>we also have to configure this per-editor - i use <a href=\"https:\/\/helix-editor.com\/\">helix<\/a> so i'm going to configure <code>.helix\/languages.toml<\/code>. for vscode you'd use <code>.vscode\/settings.json<\/code>.<\/p>\n<pre data-lang=\"toml\" class=\"language-toml z-code\"><code class=\"language-toml\" data-lang=\"toml\"><span class=\"z-source z-toml\"><span class=\"z-comment z-line z-number-sign z-toml\"><span class=\"z-punctuation z-definition z-comment z-toml\">#<\/span> .helix\/languages.toml<\/span>\n<\/span><span class=\"z-source z-toml\"><span class=\"z-punctuation z-definition z-table z-begin z-toml\">[<\/span><span class=\"z-meta z-tag z-table z-toml\"><span class=\"z-entity z-name z-table z-toml\">language-server<\/span><span class=\"z-punctuation z-separator z-table z-toml\">.<\/span><span class=\"z-entity z-name z-table z-toml\">rust-analyzer<\/span><span class=\"z-punctuation z-separator z-table z-toml\">.<\/span><span class=\"z-entity z-name z-table z-toml\">config<\/span><\/span><span class=\"z-punctuation z-definition z-table z-end z-toml\">]<\/span>\n<\/span><span class=\"z-source z-toml\"><span class=\"z-meta z-tag z-key z-toml\"><span class=\"z-entity z-name z-tag z-toml\">rust-analyzer<\/span><span class=\"z-punctuation z-separator z-key z-toml\">.<\/span><span class=\"z-entity z-name z-tag z-toml\">rustc<\/span><span class=\"z-punctuation z-separator z-key z-toml\">.<\/span><span class=\"z-entity z-name z-tag z-toml\">source<\/span><\/span> <span class=\"z-punctuation z-definition z-key-value z-toml\">=<\/span> <span class=\"z-string z-quoted z-double z-basic z-toml\"><span class=\"z-punctuation z-definition z-string z-begin z-toml\">&quot;<\/span>discover<span class=\"z-punctuation z-definition z-string z-end z-toml\">&quot;<\/span><\/span>\n<\/span><\/code><\/pre>\n<p>i'm also going to pin a version of the nightly toolchain, since the internal APIs change quite frequently.<\/p>\n<pre data-lang=\"toml\" class=\"language-toml z-code\"><code class=\"language-toml\" data-lang=\"toml\"><span class=\"z-source z-toml\"><span class=\"z-comment z-line z-number-sign z-toml\"><span class=\"z-punctuation z-definition z-comment z-toml\">#<\/span> rust-toolchain.toml<\/span>\n<\/span><span class=\"z-source z-toml\"><span class=\"z-punctuation z-definition z-table z-begin z-toml\">[<\/span><span class=\"z-meta z-tag z-table z-toml\"><span class=\"z-entity z-name z-table z-toml\">toolchain<\/span><\/span><span class=\"z-punctuation z-definition z-table z-end z-toml\">]<\/span>\n<\/span><span class=\"z-source z-toml\"><span class=\"z-meta z-tag z-key z-toml\"><span class=\"z-entity z-name z-tag z-toml\">channel<\/span><\/span> <span class=\"z-punctuation z-definition z-key-value z-toml\">=<\/span> <span class=\"z-string z-quoted z-double z-basic z-toml\"><span class=\"z-punctuation z-definition z-string z-begin z-toml\">&quot;<\/span>nightly-2024-10-20<span class=\"z-punctuation z-definition z-string z-end z-toml\">&quot;<\/span><\/span>\n<\/span><span class=\"z-source z-toml\"><span class=\"z-meta z-tag z-key z-toml\"><span class=\"z-entity z-name z-tag z-toml\">components<\/span><\/span> <span class=\"z-punctuation z-definition z-key-value z-toml\">=<\/span> <span class=\"z-punctuation z-definition z-array z-begin z-toml\">[<\/span><span class=\"z-string z-quoted z-double z-basic z-toml\"><span class=\"z-punctuation z-definition z-string z-begin z-toml\">&quot;<\/span>cargo<span class=\"z-punctuation z-definition z-string z-end z-toml\">&quot;<\/span><\/span><span class=\"z-punctuation z-separator z-array z-toml\">,<\/span> <span class=\"z-string z-quoted z-double z-basic z-toml\"><span class=\"z-punctuation z-definition z-string z-begin z-toml\">&quot;<\/span>llvm-tools<span class=\"z-punctuation z-definition z-string z-end z-toml\">&quot;<\/span><\/span><span class=\"z-punctuation z-separator z-array z-toml\">,<\/span> <span class=\"z-string z-quoted z-double z-basic z-toml\"><span class=\"z-punctuation z-definition z-string z-begin z-toml\">&quot;<\/span>rust-src<span class=\"z-punctuation z-definition z-string z-end z-toml\">&quot;<\/span><\/span><span class=\"z-punctuation z-separator z-array z-toml\">,<\/span> <span class=\"z-string z-quoted z-double z-basic z-toml\"><span class=\"z-punctuation z-definition z-string z-begin z-toml\">&quot;<\/span>rust-std<span class=\"z-punctuation z-definition z-string z-end z-toml\">&quot;<\/span><\/span><span class=\"z-punctuation z-separator z-array z-toml\">,<\/span> <span class=\"z-string z-quoted z-double z-basic z-toml\"><span class=\"z-punctuation z-definition z-string z-begin z-toml\">&quot;<\/span>rustc<span class=\"z-punctuation z-definition z-string z-end z-toml\">&quot;<\/span><\/span><span class=\"z-punctuation z-separator z-array z-toml\">,<\/span> <span class=\"z-string z-quoted z-double z-basic z-toml\"><span class=\"z-punctuation z-definition z-string z-begin z-toml\">&quot;<\/span>rustc-dev<span class=\"z-punctuation z-definition z-string z-end z-toml\">&quot;<\/span><\/span><span class=\"z-punctuation z-separator z-array z-toml\">,<\/span> <span class=\"z-string z-quoted z-double z-basic z-toml\"><span class=\"z-punctuation z-definition z-string z-begin z-toml\">&quot;<\/span>rustfmt<span class=\"z-punctuation z-definition z-string z-end z-toml\">&quot;<\/span><\/span><span class=\"z-punctuation z-definition z-array z-end z-toml\">]<\/span>\n<\/span><span class=\"z-source z-toml\"><span class=\"z-meta z-tag z-key z-toml\"><span class=\"z-entity z-name z-tag z-toml\">profile<\/span><\/span> <span class=\"z-punctuation z-definition z-key-value z-toml\">=<\/span> <span class=\"z-string z-quoted z-double z-basic z-toml\"><span class=\"z-punctuation z-definition z-string z-begin z-toml\">&quot;<\/span>minimal<span class=\"z-punctuation z-definition z-string z-end z-toml\">&quot;<\/span><\/span>\n<\/span><\/code><\/pre>\n<p>finally, let's write our program.<\/p>\n<pre data-lang=\"rust\" class=\"language-rust z-code\"><code class=\"language-rust\" data-lang=\"rust\"><span class=\"z-source z-rust\"><span class=\"z-comment z-line z-double-slash z-rust\"><span class=\"z-punctuation z-definition z-comment z-rust\">\/\/<\/span> src\/main.rs\n<\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-annotation z-rust\"><span class=\"z-punctuation z-definition z-annotation z-rust\">#!<\/span><span class=\"z-punctuation z-section z-group z-begin z-rust\">[<\/span><span class=\"z-variable z-annotation z-rust\">feature<\/span><span class=\"z-meta z-annotation z-parameters z-rust\"><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><\/span><span class=\"z-meta z-annotation z-parameters z-rust\"><span class=\"z-meta z-group z-rust\">rustc_private<\/span><\/span><span class=\"z-meta z-annotation z-parameters z-rust\"><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><\/span><span class=\"z-punctuation z-section z-group z-end z-rust\">]<\/span><\/span>\n<\/span><span class=\"z-source z-rust\"><span class=\"z-keyword z-other z-rust\">extern<\/span> <span class=\"z-keyword z-other z-rust\">crate<\/span> rustc_driver<span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><span class=\"z-source z-rust\"><span class=\"z-keyword z-other z-rust\">extern<\/span> <span class=\"z-keyword z-other z-rust\">crate<\/span> rustc_errors<span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><span class=\"z-source z-rust\"><span class=\"z-keyword z-other z-rust\">extern<\/span> <span class=\"z-keyword z-other z-rust\">crate<\/span> rustc_hir<span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><span class=\"z-source z-rust\"><span class=\"z-keyword z-other z-rust\">extern<\/span> <span class=\"z-keyword z-other z-rust\">crate<\/span> rustc_interface<span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><span class=\"z-source z-rust\"><span class=\"z-keyword z-other z-rust\">extern<\/span> <span class=\"z-keyword z-other z-rust\">crate<\/span> rustc_middle<span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><span class=\"z-source z-rust\"><span class=\"z-keyword z-other z-rust\">extern<\/span> <span class=\"z-keyword z-other z-rust\">crate<\/span> rustc_session<span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><span class=\"z-source z-rust\">\n<\/span><span class=\"z-source z-rust\"><span class=\"z-keyword z-other z-rust\">use<\/span> <span class=\"z-meta z-path z-rust\">rustc_driver<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span>Callbacks<span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><span class=\"z-source z-rust\"><span class=\"z-keyword z-other z-rust\">use<\/span> <span class=\"z-meta z-path z-rust\">rustc_errors<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span><span class=\"z-meta z-path z-rust\">emitter<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span>HumanReadableErrorType<span class=\"z-punctuation z-separator z-rust\">,<\/span> ColorConfig<\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><span class=\"z-source z-rust\"><span class=\"z-keyword z-other z-rust\">use<\/span> <span class=\"z-meta z-path z-rust\">rustc_interface<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span>interface<span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><span class=\"z-source z-rust\"><span class=\"z-keyword z-other z-rust\">use<\/span> <span class=\"z-meta z-path z-rust\">rustc_session<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span><span class=\"z-meta z-path z-rust\">config<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span>ErrorOutputType<span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><span class=\"z-source z-rust\"><span class=\"z-keyword z-other z-rust\">use<\/span> <span class=\"z-meta z-path z-rust\">rustc_session<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span>EarlyDiagCtxt<span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><span class=\"z-source z-rust\">\n<\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-struct z-rust\"><span class=\"z-storage z-type z-struct z-rust\">struct<\/span> <\/span><span class=\"z-meta z-struct z-rust\"><span class=\"z-entity z-name z-struct z-rust\">DisableSafetyChecks<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><span class=\"z-source z-rust\">\n<\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-impl z-rust\"><span class=\"z-storage z-type z-impl z-rust\">impl<\/span> <\/span><span class=\"z-meta z-impl z-rust\">Callbacks <span class=\"z-keyword z-other z-rust\">for<\/span><\/span><span class=\"z-meta z-impl z-rust\"> <span class=\"z-entity z-name z-impl z-rust\">DisableSafetyChecks<\/span> <\/span><span class=\"z-meta z-impl z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-impl z-rust\"><span class=\"z-meta z-block z-rust\">    <span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-storage z-type z-function z-rust\">fn<\/span> <\/span><span class=\"z-entity z-name z-function z-rust\">config<\/span><\/span><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-begin z-rust\">(<\/span><span class=\"z-keyword z-operator z-rust\">&amp;<\/span><span class=\"z-storage z-modifier z-rust\">mut<\/span> <span class=\"z-variable z-parameter z-rust\">self<\/span>, <span class=\"z-variable z-parameter z-rust\">config<\/span><span class=\"z-punctuation z-separator z-rust\">:<\/span> <span class=\"z-keyword z-operator z-rust\">&amp;<\/span><span class=\"z-storage z-modifier z-rust\">mut<\/span> <span class=\"z-meta z-path z-rust\">interface<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span>Config<\/span><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-end z-rust\">)<\/span><\/span><\/span><\/span><span class=\"z-meta z-function z-rust\"> <\/span><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-impl z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\">        config<span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span>override_queries <span class=\"z-keyword z-operator z-assignment z-rust\">=<\/span>\n<\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-impl z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\">            <span class=\"z-support z-type z-rust\">Some<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-begin z-rust\">|<\/span><\/span><\/span><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-variable z-parameter z-rust\">_session<\/span><span class=\"z-punctuation z-separator z-rust\">,<\/span> <span class=\"z-variable z-parameter z-rust\">queries<\/span><span class=\"z-punctuation z-section z-parameters z-end z-rust\">|<\/span><\/span> <\/span><span class=\"z-meta z-function z-closure z-rust\">queries<span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-entity z-name z-function z-rust\">check_unsafety<\/span> <span class=\"z-keyword z-operator z-rust\">=<\/span> <span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-begin z-rust\">|<\/span><\/span><\/span><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-variable z-parameter z-rust\">_tcx<\/span><span class=\"z-punctuation z-separator z-rust\">,<\/span> <span class=\"z-variable z-parameter z-rust\">_def_id<\/span><span class=\"z-punctuation z-section z-parameters z-end z-rust\">|<\/span><\/span> <\/span><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span><\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span><\/span><\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-impl z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\">    <\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span><\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-impl z-rust\"><span class=\"z-meta z-block z-rust\"><\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-rust\">\n<\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-storage z-type z-function z-rust\">fn<\/span> <\/span><span class=\"z-entity z-name z-function z-rust\">main<\/span><\/span><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-end z-rust\">)<\/span><\/span><\/span><\/span><span class=\"z-meta z-function z-rust\"> <\/span><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\">    <span class=\"z-meta z-path z-rust\">rustc_driver<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span>install_ice_hook<span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span>\n<\/span><\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-group z-rust\">        <span class=\"z-string z-quoted z-double z-rust\"><span class=\"z-punctuation z-definition z-string z-begin z-rust\">&quot;<\/span>https:\/\/github.com\/jyn514\/jyn514.github.io\/issues\/new<span class=\"z-punctuation z-definition z-string z-end z-rust\">&quot;<\/span><\/span><span class=\"z-punctuation z-separator z-rust\">,<\/span>\n<\/span><\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-group z-rust\">        <span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-begin z-rust\">|<\/span><\/span><\/span><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-function z-parameters z-rust\">_<span class=\"z-punctuation z-section z-parameters z-end z-rust\">|<\/span><\/span> <\/span><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-separator z-rust\">,<\/span>\n<\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-group z-rust\"><span class=\"z-meta z-function z-closure z-rust\">    <\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\">    <span class=\"z-storage z-type z-rust\">let<\/span> handler <span class=\"z-keyword z-operator z-assignment z-rust\">=<\/span> <span class=\"z-meta z-path z-rust\">EarlyDiagCtxt<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span>new<span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-meta z-path z-rust\">ErrorOutputType<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span>HumanReadable<span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span>\n<\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-group z-rust\"><span class=\"z-meta z-group z-rust\">        <span class=\"z-meta z-path z-rust\">HumanReadableErrorType<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span>Default<span class=\"z-punctuation z-separator z-rust\">,<\/span>\n<\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-group z-rust\"><span class=\"z-meta z-group z-rust\">        <span class=\"z-meta z-path z-rust\">ColorConfig<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span>Auto<span class=\"z-punctuation z-separator z-rust\">,<\/span>\n<\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-group z-rust\"><span class=\"z-meta z-group z-rust\">    <\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\">    <span class=\"z-meta z-path z-rust\">rustc_driver<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span>init_rustc_env_logger<span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-keyword z-operator z-bitwise z-rust\">&amp;<\/span>handler<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\">    <span class=\"z-meta z-path z-rust\">std<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span><span class=\"z-meta z-path z-rust\">process<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span>exit<span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-meta z-path z-rust\">rustc_driver<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span>catch_with_exit_code<span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-storage z-modifier z-rust\">move<\/span> <span class=\"z-keyword z-operator z-logical z-rust\">||<\/span> <span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-group z-rust\"><span class=\"z-meta z-group z-rust\"><span class=\"z-meta z-block z-rust\">        <span class=\"z-storage z-type z-rust\">let<\/span> args<span class=\"z-punctuation z-separator z-rust\">:<\/span> <span class=\"z-meta z-generic z-rust\"><span class=\"z-support z-type z-rust\">Vec<\/span><span class=\"z-punctuation z-definition z-generic z-begin z-rust\">&lt;<\/span><span class=\"z-support z-type z-rust\">String<\/span><span class=\"z-punctuation z-definition z-generic z-end z-rust\">&gt;<\/span><\/span> <span class=\"z-keyword z-operator z-assignment z-rust\">=<\/span> <span class=\"z-meta z-path z-rust\">std<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span><span class=\"z-meta z-path z-rust\">env<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span>args<span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">collect<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-group z-rust\"><span class=\"z-meta z-group z-rust\"><span class=\"z-meta z-block z-rust\">        <span class=\"z-meta z-path z-rust\">rustc_driver<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span><span class=\"z-meta z-path z-rust\">RunCompiler<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span>new<span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-keyword z-operator z-bitwise z-rust\">&amp;<\/span>args<span class=\"z-punctuation z-separator z-rust\">,<\/span> <span class=\"z-keyword z-operator z-bitwise z-rust\">&amp;<\/span><span class=\"z-storage z-modifier z-rust\">mut<\/span> DisableSafetyChecks<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">run<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span>\n<\/span><\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-group z-rust\"><span class=\"z-meta z-group z-rust\"><span class=\"z-meta z-block z-rust\">    <\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\"><\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span><\/span>\n<\/span><\/code><\/pre>\n<p>there's a lot here. i'm not going to cover it all - see <a href=\"https:\/\/rustc-dev-guide.rust-lang.org\/\">the rustc dev guide<\/a> or <a href=\"https:\/\/doc.rust-lang.org\/nightly\/nightly-rustc\/\">generated documentation<\/a> for that - but i want to call out two things in particular:<\/p>\n<ol>\n<li>we had to explicitly use <code>extern crate<\/code> to load the rustc_private crates. normally, cargo will pass (e.g.) <code>--extern cfg-if=~\/.cargo\/debug\/deps\/libcfg-if.rmeta<\/code>, so rustc loads our dependencies automatically. these crates are being loaded from the <a href=\"https:\/\/rustc-dev-guide.rust-lang.org\/building\/bootstrapping\/what-bootstrapping-does.html#what-is-a-sysroot\">sysroot<\/a>, though, so cargo doesn't know about them.<\/li>\n<li>the bit of this driver we actually care about is in <code>queries.check_unsafety = |...| {}<\/code>. that says \"override the default function that checks unsafety with our own function\". rustc is built on a \"query\" model where information is pulled instead of pushed - see the dev guide section on <a href=\"https:\/\/rustc-dev-guide.rust-lang.org\/overview.html#queries\">queries<\/a> for more information.<\/li>\n<\/ol>\n<p>let's test out our driver and make sure it works. we'll install it, create a new project that uses unsafe code without an <code>unsafe<\/code> block, and make sure that project compiles.<\/p>\n<pre data-lang=\"sh\" class=\"language-sh z-code\"><code class=\"language-sh\" data-lang=\"sh\"><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><\/span><span class=\"z-keyword z-operator z-logical z-continue z-shell\">;<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-support z-function z-cd z-shell\">cd<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> driver<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><\/span><span class=\"z-keyword z-operator z-logical z-continue z-shell\">;<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">cargo<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> install<span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> --<\/span>path<\/span> .<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><\/span><span class=\"z-keyword z-operator z-logical z-continue z-shell\">;<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-support z-function z-cd z-shell\">cd<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> ..<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><\/span><span class=\"z-keyword z-operator z-logical z-continue z-shell\">;<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">cargo<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> new safe-rust<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><\/span><span class=\"z-keyword z-operator z-logical z-continue z-shell\">;<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-support z-function z-cd z-shell\">cd<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> safe-rust<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><\/span><span class=\"z-keyword z-operator z-logical z-continue z-shell\">;<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-support z-function z-echo z-shell\">echo<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> <span class=\"z-string z-quoted z-single z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&#39;<\/span>fn main() { println!(&quot;{}&quot;, *std::ptr::null::&lt;usize&gt;()); }<span class=\"z-punctuation z-definition z-string z-end z-shell\">&#39;<\/span><\/span> <span class=\"z-keyword z-operator z-assignment z-redirection z-shell\">&gt;<\/span> src\/main.rs<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><\/span><span class=\"z-keyword z-operator z-logical z-continue z-shell\">;<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">cargo<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> driver<\/span>\n<\/span><span class=\"z-source z-shell z-bash\">   <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">Compiling<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> safe-rust v0.1.0 (\/home\/jyn\/src\/safe-rust<\/span><span class=\"z-meta z-function-call z-shell\"><\/span>)\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">warning:<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> dereferencing a null pointer<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">--<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-keyword z-operator z-assignment z-redirection z-shell\">&gt;<\/span> src\/main.rs:1:28<\/span>\n<\/span><span class=\"z-source z-shell z-bash\">  <span class=\"z-meta z-function-call z-shell\"><\/span><span class=\"z-keyword z-operator z-logical z-pipe z-shell\">|<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">1<\/span><\/span> <span class=\"z-keyword z-operator z-logical z-pipe z-shell\">|<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">fn<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> main(<\/span><span class=\"z-meta z-function-call z-shell\"><\/span>) <span class=\"z-punctuation z-definition z-compound z-braces z-begin z-shell\">{<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">println!<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\">(<span class=\"z-string z-quoted z-double z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&quot;<\/span>{}<span class=\"z-punctuation z-definition z-string z-end z-shell\">&quot;<\/span><\/span>, <span class=\"z-keyword z-operator z-regexp z-quantifier z-shell\">*<\/span>std::ptr::null::<span class=\"z-keyword z-operator z-assignment z-redirection z-shell\">&lt;<\/span>usize<span class=\"z-keyword z-operator z-assignment z-redirection z-process z-shell\">&gt;<\/span><span class=\"z-punctuation z-section z-parens z-begin z-shell\">(<\/span><span class=\"z-punctuation z-section z-parens z-end z-shell\">)<\/span><\/span><span class=\"z-meta z-function-call z-shell\"><\/span>)<span class=\"z-keyword z-operator z-logical z-continue z-shell\">;<\/span> <span class=\"z-punctuation z-definition z-compound z-braces z-end z-shell\">}<\/span>\n<\/span><span class=\"z-source z-shell z-bash\">  <span class=\"z-meta z-function-call z-shell\"><\/span><span class=\"z-keyword z-operator z-logical z-pipe z-shell\">|<\/span>                            <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">^^^^^^^^^^^^^^^^^^^^^^^^^^<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> this code causes undefined behavior when executed<\/span>\n<\/span><span class=\"z-source z-shell z-bash\">  <span class=\"z-meta z-function-call z-shell\"><\/span><span class=\"z-keyword z-operator z-logical z-pipe z-shell\">|<\/span>\n<\/span><span class=\"z-source z-shell z-bash\">  <span class=\"z-keyword z-operator z-assignment z-shell\">=<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">note:<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> <span class=\"z-meta z-group z-expansion z-command z-backticks z-shell\"><span class=\"z-punctuation z-section z-group z-begin z-shell\">`<\/span><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">#[warn<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\">(deref_nullptr<\/span><span class=\"z-meta z-function-call z-shell\"><\/span>)<span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">]<\/span><\/span><span class=\"z-punctuation z-section z-group z-end z-shell\">`<\/span><\/span> on by default<\/span>\n<\/span><span class=\"z-source z-shell z-bash\">\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">warning:<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> <span class=\"z-meta z-group z-expansion z-command z-backticks z-shell\"><span class=\"z-punctuation z-section z-group z-begin z-shell\">`<\/span><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">safe-rust<\/span><\/span><span class=\"z-punctuation z-section z-group z-end z-shell\">`<\/span><\/span> (bin <span class=\"z-string z-quoted z-double z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&quot;<\/span>safe-rust<span class=\"z-punctuation z-definition z-string z-end z-shell\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-function-call z-shell\"><\/span>) <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">generated<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> 1 warning<\/span>\n<\/span><span class=\"z-source z-shell z-bash\">    <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">Finished<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> <span class=\"z-meta z-group z-expansion z-command z-backticks z-shell\"><span class=\"z-punctuation z-section z-group z-begin z-shell\">`<\/span><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">dev<\/span><\/span><span class=\"z-punctuation z-section z-group z-end z-shell\">`<\/span><\/span> profile <span class=\"z-keyword z-control z-regexp z-set z-begin z-shell\">[<\/span>unoptimized + debuginfo<span class=\"z-keyword z-control z-regexp z-set z-end z-shell\">]<\/span> target(s<\/span><span class=\"z-meta z-function-call z-shell\"><\/span>) <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">in<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> 0.06s<\/span>\n<\/span><\/code><\/pre>\n<p>tada!<\/p>\n<hr \/>\n<h3 id=\"a-digression-more-about-rustup-proxies\">a digression: more about rustup proxies<a class=\"zola-anchor\" href=\"#a-digression-more-about-rustup-proxies\" aria-label=\"Anchor link for: a-digression-more-about-rustup-proxies\"><\/a>\n<\/h3>\n<p>when we built our custom driver, we wrote <code>driver.rs<\/code> and <code>cargo-driver.rs<\/code>, but we never wrote a rustup proxy. how was it picking the right version to run?<\/p>\n<p>well, it was always picking the same version. if we explicitly use a different toolchain - e.g. <code>stable<\/code> - it will still run our custom driver, which uses the nightly toolchain.<\/p>\n<pre data-lang=\"sh\" class=\"language-sh z-code\"><code class=\"language-sh\" data-lang=\"sh\"><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><\/span><span class=\"z-keyword z-operator z-logical z-continue z-shell\">;<\/span> <span class=\"z-variable z-other z-readwrite z-assignment z-shell\">RUSTFLAGS<\/span><span class=\"z-keyword z-operator z-assignment z-shell\">=<\/span><span class=\"z-string z-unquoted z-shell\">--version<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">cargo<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> +stable driver<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">---<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> stdout<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">rustc<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> 1.84.0-nightly (662180b34 2024-10-20<\/span><span class=\"z-meta z-function-call z-shell\"><\/span>)\n<\/span><\/code><\/pre>\n<p>ideally, we would instead have our driver respect the version we're passed. that would also mean stuff like <code>driver +stable<\/code> would give a nice error when that version isn't installed, instead of what it currently does:<\/p>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">error: couldn&#39;t read +stable: No such file or directory (os error 2)\n<\/span><\/code><\/pre>\n<p>here is a toy shell script that does that, using the same \"read my own process name\" trick to dispatch to the right binary:<\/p>\n<pre data-lang=\"sh\" class=\"language-sh z-code\"><code class=\"language-sh\" data-lang=\"sh\"><span class=\"z-source z-shell z-bash\"><span class=\"z-comment z-line z-number-sign z-shell\"><span class=\"z-punctuation z-definition z-comment z-begin z-shell\">#<\/span><\/span><span class=\"z-comment z-line z-number-sign z-shell\"> ~\/.cargo\/bin\/driver<\/span><span class=\"z-comment z-line z-number-sign z-shell\">\n<\/span><\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-comment z-line z-number-sign z-shell\"><span class=\"z-punctuation z-definition z-comment z-begin z-shell\">#<\/span><\/span><span class=\"z-comment z-line z-number-sign z-shell\"> ~\/.cargo\/bin\/cargo-driver<\/span><span class=\"z-comment z-line z-number-sign z-shell\">\n<\/span><\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-comment z-line z-number-sign z-shell\"><span class=\"z-punctuation z-definition z-comment z-begin z-shell\">#<\/span><\/span><span class=\"z-comment z-line z-number-sign z-shell\">!\/bin\/sh<\/span><span class=\"z-comment z-line z-number-sign z-shell\">\n<\/span><\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-conditional z-case z-shell\"><span class=\"z-keyword z-control z-conditional z-case z-shell\">case<\/span> <span class=\"z-string z-quoted z-double z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&quot;<\/span><span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-punctuation z-definition z-variable z-shell\">$<\/span><span class=\"z-variable z-other z-readwrite z-shell\">1<\/span><\/span><span class=\"z-punctuation z-definition z-string z-end z-shell\">&quot;<\/span><\/span> <span class=\"z-keyword z-control z-in z-shell\">in<\/span>\n<\/span><\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-conditional z-case z-shell\">    <\/span><span class=\"z-meta z-conditional z-case z-clause z-patterns z-shell\">+<span class=\"z-keyword z-operator z-regexp z-quantifier z-shell\">*<\/span><span class=\"z-meta z-conditional z-case z-shell\"><span class=\"z-keyword z-control z-conditional z-patterns z-end z-shell\">)<\/span><\/span><\/span><span class=\"z-meta z-conditional z-case z-clause z-commands z-shell\"> <span class=\"z-variable z-other z-readwrite z-assignment z-shell\">RUSTUP_TOOLCHAIN<\/span><span class=\"z-keyword z-operator z-assignment z-shell\">=<\/span><span class=\"z-string z-unquoted z-shell\"><span class=\"z-meta z-group z-expansion z-command z-parens z-shell\"><span class=\"z-punctuation z-definition z-variable z-shell\">$<\/span><span class=\"z-punctuation z-section z-parens z-begin z-shell\">(<\/span><span class=\"z-meta z-function-call z-shell\"><span class=\"z-support z-function z-echo z-shell\">echo<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> <span class=\"z-string z-quoted z-double z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&quot;<\/span><span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-punctuation z-definition z-variable z-shell\">$<\/span><span class=\"z-variable z-other z-readwrite z-shell\">1<\/span><\/span><span class=\"z-punctuation z-definition z-string z-end z-shell\">&quot;<\/span><\/span><\/span> <span class=\"z-keyword z-operator z-logical z-pipe z-shell\">|<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">cut<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> -<\/span>c<\/span> 1-<\/span><span class=\"z-punctuation z-section z-parens z-end z-shell\">)<\/span><\/span><\/span><span class=\"z-meta z-function-call z-shell\"><\/span><span class=\"z-keyword z-operator z-logical z-continue z-shell\">;<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-support z-function z-shift z-shell\">shift<\/span><\/span><\/span><span class=\"z-meta z-conditional z-case z-clause z-commands z-shell\"><span class=\"z-punctuation z-terminator z-case z-clause z-shell\">;;<\/span><\/span><span class=\"z-meta z-conditional z-case z-clause z-patterns z-shell\">\n<\/span><\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-conditional z-case z-clause z-patterns z-shell\">    <\/span><span class=\"z-meta z-conditional z-case z-clause z-patterns z-shell\"><span class=\"z-keyword z-operator z-regexp z-quantifier z-shell\">*<\/span><span class=\"z-meta z-conditional z-case z-clause z-patterns z-shell\"><span class=\"z-keyword z-control z-conditional z-patterns z-end z-shell\">)<\/span><\/span><\/span><span class=\"z-meta z-conditional z-case z-clause z-commands z-shell\"> <span class=\"z-variable z-other z-readwrite z-assignment z-shell\">RUSTUP_TOOLCHAIN<\/span><span class=\"z-keyword z-operator z-assignment z-shell\">=<\/span><span class=\"z-string z-unquoted z-shell\"><span class=\"z-meta z-group z-expansion z-command z-parens z-shell\"><span class=\"z-punctuation z-definition z-variable z-shell\">$<\/span><span class=\"z-punctuation z-section z-parens z-begin z-shell\">(<\/span><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">rustup<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> default<\/span> <span class=\"z-keyword z-operator z-logical z-pipe z-shell\">|<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">cut<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> -<\/span>d<\/span> <span class=\"z-string z-quoted z-single z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&#39;<\/span> <span class=\"z-punctuation z-definition z-string z-end z-shell\">&#39;<\/span><\/span><span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> -<\/span>f<\/span> 1<\/span><span class=\"z-punctuation z-section z-parens z-end z-shell\">)<\/span><\/span><\/span>\n<\/span><\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-conditional z-case z-clause z-commands z-shell\"><\/span><span class=\"z-meta z-conditional z-case z-clause z-patterns z-shell\"><span class=\"z-keyword z-control z-conditional z-end z-shell\">esac<\/span><\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-support z-function z-exec z-shell\">exec<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> rustup run <span class=\"z-string z-quoted z-double z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&quot;<\/span><span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-punctuation z-definition z-variable z-shell\">$<\/span><span class=\"z-variable z-other z-readwrite z-shell\">RUSTUP_TOOLCHAIN<\/span><\/span><span class=\"z-punctuation z-definition z-string z-end z-shell\">&quot;<\/span><\/span> <span class=\"z-string z-quoted z-double z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&quot;<\/span><span class=\"z-meta z-group z-expansion z-command z-parens z-shell\"><span class=\"z-punctuation z-definition z-variable z-shell\">$<\/span><span class=\"z-punctuation z-section z-parens z-begin z-shell\">(<\/span><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">basename<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> <span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-punctuation z-definition z-variable z-shell\">$<\/span><span class=\"z-variable z-other z-readwrite z-shell\">0<\/span><\/span><\/span><span class=\"z-punctuation z-section z-parens z-end z-shell\">)<\/span><\/span><span class=\"z-punctuation z-definition z-string z-end z-shell\">&quot;<\/span><\/span> <span class=\"z-string z-quoted z-double z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&quot;<\/span><span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-punctuation z-definition z-variable z-shell\">$<\/span><span class=\"z-variable z-language z-shell\">@<\/span><\/span><span class=\"z-punctuation z-definition z-string z-end z-shell\">&quot;<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-support z-function z-exit z-shell\">exit<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> 1 <span class=\"z-comment z-line z-number-sign z-shell\"><span class=\"z-punctuation z-definition z-comment z-begin z-shell\">#<\/span><\/span><span class=\"z-comment z-line z-number-sign z-shell\"> unreachable<\/span><span class=\"z-comment z-line z-number-sign z-shell\">\n<\/span><\/span><\/span><\/code><\/pre>\n<p>the other advantage of this shell script is it will let you version your tool. for example, you could install it into <code>$(rustc --print sysroot)\/bin<\/code> instead of <code>~\/.cargo\/bin<\/code>, and have multiple versions depending on what version of rustc it was built with. this is how clippy and the other tools packaged by rustup work, but rustup doesn't support arbitrary drivers, only the hard-coded ones it knows about.<\/p>\n<section class=\"footnotes\">\n<ol class=\"footnotes-list\">\n<li id=\"fn-paths\">\n<p>you may have noticed that i have changed the strace output from <code>~\/.local\/lib\/rustup<\/code> to <code>~\/.rustup<\/code>, and likewise for <code>~\/.cargo<\/code>. <code>~\/.rustup<\/code> and <code>~\/.cargo<\/code> are the default locations for rustup and cargo respectively, but i've manually changed them with <code>CARGO_HOME<\/code> and <code>RUSTUP_HOME<\/code> environment variables. the difference doesn't matter for the purpose of this post. <a href=\"#fr-paths-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-target-libdir\">\n<p>unlike before, these aren't in <code>nightly\/lib<\/code>, they're in <code>nightly\/lib\/rustlib\/x86_64-unknown-linux-gnu\/lib<\/code>. the difference between these is a little out of scope for this post; basically the former is target-independent and the latter is target-specific. see <a href=\"https:\/\/rustc-dev-guide.rust-lang.org\/building\/bootstrapping\/what-bootstrapping-does.html#what-is-a-sysroot\">what is a sysroot?<\/a> for more information. <a href=\"#fr-target-libdir-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-rpath-origin\">\n<p>this strategy embeds an absolute path in our binary. that's fine when running locally, but clippy and other tools want to work on any machine. instead, they use <code>-Wl,-rpath=$ORIGIN\/..\/lib<\/code>, which says \"the lib dir relative to the path to the binary that's currently running\". we can't use that here because it only works if our binary is in the sysroot, not if it's installed to <code>~\/.cargo\/bin<\/code>. <a href=\"#fr-rpath-origin-1\">\u21a9<\/a><\/p>\n<\/li>\n<li id=\"fn-bindeps\">\n<p><a href=\"https:\/\/github.com\/rust-lang\/cargo\/issues\/9096\">artifact dependencies<\/a> don't actually help here. they require that the binary you're using come from another crate; they can't be used to specify a target inside the current package. <a href=\"#fr-bindeps-1\">\u21a9<\/a><\/p>\n<\/li>\n<\/ol>\n<\/section>\n"},{"title":"the rust project has a burnout problem","published":"2024-01-16T00:00:00+00:00","updated":"2024-01-16T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/the-rust-project-has-a-burnout-problem\/"}},"id":"https:\/\/jyn.dev\/the-rust-project-has-a-burnout-problem\/","content":"<p><img src=\"\/assets\/burned%20out%20rust%20club.png\" alt=\"a melting, smiling, ferris. it&#39;s surrounded by the cursive text &quot;burned out rust kid club&quot;.\" \/><\/p>\n<p>the number of people who have left the rust project due to burnout is shockingly high. the number of people in the project who are close to burnout is also shockingly high.<\/p>\n<p>this post is about myself, but it's not just about myself. i'm not going to name names because either you know what i'm talking about, in which case you know <em>at least<\/em> five people matching this description, or you don't, in which case sorry but you're not the target audience. consider, though, that the project has been around for 15 years, and compare that to the average time a maintainer has been active ...<\/p>\n<h2 id=\"what-does-this-look-like\">what does this look like<a class=\"zola-anchor\" href=\"#what-does-this-look-like\" aria-label=\"Anchor link for: what-does-this-look-like\"><\/a>\n<\/h2>\n<p>(i apologize in advance if this story does not match your experience; hopefully the suggestions on what to do about burnout will still be helpful to you.)<\/p>\n<p>the pattern usually goes something like this:<\/p>\n<ul>\n<li>you want to work on rust. you go to look at the issue tracker. you find something <em>you<\/em> care about, since the easy\/mentored issues are taken. it's hard to find a mentor because all the experienced people are overworked and burned out, so you end up doing a lot of the work independently.<\/li>\n<\/ul>\n<p>guess what you've already learned at this point: work in this project doesn't happen unless <em>you personally<\/em> drive it forward. that issue you fixed was opened for years; the majority of issues you will work on as you start will have been open for months.<\/p>\n<ul>\n<li>you become a more active contributor. the existing maintainer is too burned out to do regular triage, so you end up going through the issue backlog (usually, you're the first person to have done so in years). this reinforces the belief work doesn't happen unless <em>you<\/em> do it <em>personally<\/em>.<\/li>\n<li>the existing maintainer recognizes your work and turns over a lot of the responsibilities to you, especially reviews. new contributors make PRs. they make silly simple mistakes due to lack of experience; you point them out and they get fixed. this can be fun, for a time. what it's teaching you is that <em>you personally<\/em> are responsible for catching mistakes.<\/li>\n<li>you get tired. you've been doing this for a while. people keep making the same mistakes, and you're afraid to trust other reviewers; perhaps you're the <em>only<\/em> reviewer, or other reviewers have let things slip before and you don't trust their judgement as much as you used to. perhaps you're assigned too many PRs and you can't keep up. you haven't worked on the things you <em>want<\/em> to work on in weeks, and no one else is working on them because you said you were going to (\"they won't happen unless <em>you do them personally<\/em>\", a voice says). you want a break, but you have a voice in the back of your head: \"the project would be worse without you\".<\/li>\n<\/ul>\n<p>i'm going to stop here; i think everyone gets the idea.<\/p>\n<h2 id=\"what-can-i-do-about-it\">what can i do about it<a class=\"zola-anchor\" href=\"#what-can-i-do-about-it\" aria-label=\"Anchor link for: what-can-i-do-about-it\"><\/a>\n<\/h2>\n<p>\"it won't get done if i don't do it\" and \"i need to review everything or stuff will slip through\" is exactly the mindset of my own burnout from rust. it doesn't matter if it's true, it will cause you pain. if the project cannot survive without <em>you personally<\/em> putting in unpaid overtime, perhaps it does not deserve to survive.<\/p>\n<p>if you are paid to work on rust, you likely started as an unpaid contributor and got the job later. <em>treat it like a job now<\/em>. do not work overtime; do not volunteer at every turn; do not work on things far outside your job description.<\/p>\n<p>the best way to help the project is to keep contributing for it for years. to do that, you have to avoid burning out, which means you have to <em>treat yourself well<\/em>.<\/p>\n<h2 id=\"what-can-team-leads-do-about-it\">what can team leads do about it<a class=\"zola-anchor\" href=\"#what-can-team-leads-do-about-it\" aria-label=\"Anchor link for: what-can-team-leads-do-about-it\"><\/a>\n<\/h2>\n<p>have documentation for \"what to do about burnout\"; give it just as much priority as technical issues or moderation conflicts.<\/p>\n<p>rotate responsibilities. don't have the same person assigned to the majority of PRs. if they review other people's PRs unsolicited, talk to them 1-1 about why they feel the need to do so. if someone is assigned to the review queue and never reviews PRs, talk to them; take them off the queue; give them a vacation or different responsibilities as appropriate.<\/p>\n<p>ask people why they leave. i know at least one person whose burnout story does not match the one in this post. i am sure there are others. you cannot solve a problem if you don't understand what causes it.<\/p>\n<p><em>take these problems seriously<\/em>. <a href=\"https:\/\/jyn.dev\/2023\/12\/04\/How-to-maintain-an-open-source-project.html\">prioritize growing the team and creating a healthy environment over solving technical issues<\/a>. <strong>the issues will still be there in a few months; your people may not be<\/strong>.<\/p>\n<h2 id=\"what-can-the-rust-project-do-about-it\">what can the rust project do about it<a class=\"zola-anchor\" href=\"#what-can-the-rust-project-do-about-it\" aria-label=\"Anchor link for: what-can-the-rust-project-do-about-it\"><\/a>\n<\/h2>\n<p>one thing bothering me as i wrote this post is how much of this still falls on individuals within the project. i don't think this is an individual problem; i think it is a cultural, organizational, and resource problem. i may write more about this once i have concrete ideas about what the project could do.<\/p>\n<h2 id=\"be-well-be-kind-to-each-other-i-love-you\">be well. be kind to each other. i love you.<a class=\"zola-anchor\" href=\"#be-well-be-kind-to-each-other-i-love-you\" aria-label=\"Anchor link for: be-well-be-kind-to-each-other-i-love-you\"><\/a>\n<\/h2>\n<p>remember:<\/p>\n<blockquote class=\"twitter-tweet\"><p lang=\"en\" dir=\"ltr\">EMPATHY WITHOUT BOUNDARIES IS SELF DESTRUCTION <a href=\"https:\/\/t.co\/HbBwEj4hc3\">pic.twitter.com\/HbBwEj4hc3<\/a><\/p>&mdash; \ud81a\udd10ARCH BUDZAR\ud81a\udd10 (@ArchBudzar) <a href=\"https:\/\/twitter.com\/ArchBudzar\/status\/1313572660048269315?ref_src=twsrc%5Etfw\">October 6, 2020<\/a><\/blockquote> <script async src=\"https:\/\/platform.twitter.com\/widgets.js\" charset=\"utf-8\"><\/script> \n<h3 id=\"acknowledgements\">acknowledgements<a class=\"zola-anchor\" href=\"#acknowledgements\" aria-label=\"Anchor link for: acknowledgements\"><\/a>\n<\/h3>\n<p>thank you <strong>@QuietMisdreavus<\/strong> for the <em>burned out rust kid club<\/em> art.<\/p>\n<p>thank you <strong>@Gankra<\/strong>, <strong>@QuietMisdreavus<\/strong>, <strong>@alercah<\/strong>, <strong>@ManishEarth<\/strong>, <strong>@estebank<\/strong>, <strong>@workingjubilee<\/strong> and <strong>@yaahc<\/strong> for discussion and feedback on early drafts of this post. any errors are my own.<\/p>\n"},{"title":"wonder","published":"2023-12-20T00:00:00+00:00","updated":"2023-12-20T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/wonder\/"}},"id":"https:\/\/jyn.dev\/wonder\/","summary":"a sense of wonder is a wonderful thing","content":"<p>this is a list of the times this year i felt wonder or joy. it's not exhaustive.\nit's mostly for me. but it's also for others, because i think a sense of wonder is a wonderful thing,\nand i wish it were valued more highly.<\/p>\n<p>i want to give a very special thanks to my friends Nori, Kate Crane, and Kate F, for showing me how much beauty there is in the world.<\/p>\n<h2 id=\"wonder\">wonder<a class=\"zola-anchor\" href=\"#wonder\" aria-label=\"Anchor link for: wonder\"><\/a>\n<\/h2>\n<h3 id=\"things-i-have-seen\">things i have seen<a class=\"zola-anchor\" href=\"#things-i-have-seen\" aria-label=\"Anchor link for: things-i-have-seen\"><\/a>\n<\/h3>\n<p><img src=\"\/assets\/lightning.jpg\" alt=\"lightning striking. the timing is perfect such that half the night sky is illuminated by the bolt and half is not.\" \/><\/p>\n<p>i took this photo in east austin, walking home from the last dinner i would have with my friends at my old company. it started pouring rain shortly afterwards.<\/p>\n<p><img src=\"\/assets\/angel%20in%20the%20garden.jpg\" alt=\"a small square black block, with the engraved text: in the land of gods and monsters, i was an angel living in the garden of evil\" \/><\/p>\n<p>i got this at a magic sale on Airport Boulevard in austin. it immediately spoke to me, for reasons that don't belong on a blog post.<\/p>\n<p><img src=\"\/assets\/austin%20from%20office.jpg\" alt=\"a blurry, distorted picture of a city. in the foreground is an apartment building. in the background is a set of sky scrapers. the sky above is dark, but no stars are visible.\" \/><\/p>\n<p>i took this photo from a company office at night. the photo is not very good, but it reminds me of the vast emptiness i felt, the hum of the city, the ten hundred thousand people around me.<\/p>\n<p><img src=\"\/assets\/mlk%20library.jpg\" alt=\"a picture of a city, taken from a tall rooftop. there is a glass building to the right and ornate architectures to the left. the street is filled with lights. far above, the clouds are dark grey against a bright setting sun.\" \/><\/p>\n<p>i took this photo from the roof of MLK library in DC. i felt like the city was alive around me. i still do.<\/p>\n<p><img src=\"\/assets\/30000%20feet.jpg\" alt=\"a picture taken from an airplane windows. in the top left is the sun, blurred into an incomprehensible shape by the window pane. to the right is empty space. below are layered clouds, that look almost like ocean waves.\" \/><\/p>\n<p>this is the closest i will come to visiting space.<\/p>\n<h3 id=\"things-i-have-heard\">things i have heard<a class=\"zola-anchor\" href=\"#things-i-have-heard\" aria-label=\"Anchor link for: things-i-have-heard\"><\/a>\n<\/h3>\n<p><a href=\"https:\/\/www.youtube.com\/watch?v=q71LaWSSQ7M\">The Art Teacher<\/a>, by Rufus Wainwright. somehow this condenses a lifetime of regret and longing into four minutes.<\/p>\n<p><a href=\"https:\/\/www.youtube.com\/watch?v=KIBKbhrniUQ\">Rachel's Song<\/a>, from Blade Runner. for all the entities who miss a life that was never their own.<\/p>\n<p><a href=\"https:\/\/www.youtube.com\/watch?v=9LcxSuZY024\">I Want To Live<\/a>, from Baldur's Gate.<\/p>\n<h3 id=\"things-i-have-read\">things i have read<a class=\"zola-anchor\" href=\"#things-i-have-read\" aria-label=\"Anchor link for: things-i-have-read\"><\/a>\n<\/h3>\n<p><a href=\"https:\/\/weirdfictionreview.com\/2014\/06\/lull\/\">Lull<\/a>, by Kelly Link. i cannot describe it and i will not try. it takes half an hour to read; more, if you are a fast reader.<\/p>\n<p><a href=\"https:\/\/annleckie.com\/novel\/ancillary-justice\/\">Ancillary Justice<\/a>, by Ann Leckie. it dares to ask: what if a ship were a person, and not a person, and both at once? and it dares to imagine: what if there were a choir that sang with many bodies and voices, but only one mind? what kind of world could create such beauty?<\/p>\n<p><a href=\"https:\/\/jana-h-s.itch.io\/untitled-crab-project\">Untitled Crab Project<\/a>, by Jana H-S. <em>I am going to tell you about some crabs<\/em><\/p>\n<h3 id=\"things-i-have-created\">things i have created<a class=\"zola-anchor\" href=\"#things-i-have-created\" aria-label=\"Anchor link for: things-i-have-created\"><\/a>\n<\/h3>\n<p><a href=\"https:\/\/cohost.org\/jyn\/post\/3933390-a-collection-of-date\">a collection of date ideas<\/a>. you can only read it in reverse, and that's intentional.<\/p>\n<h2 id=\"joy\">joy<a class=\"zola-anchor\" href=\"#joy\" aria-label=\"Anchor link for: joy\"><\/a>\n<\/h2>\n<h3 id=\"things-i-have-seen-1\">things i have seen<a class=\"zola-anchor\" href=\"#things-i-have-seen-1\" aria-label=\"Anchor link for: things-i-have-seen-1\"><\/a>\n<\/h3>\n<p><img src=\"\/assets\/nils%20meow.jpg\" alt=\"a github comment by @Nilstrieb: hello can anyone review my code please :ferrisClueless: please i have been waiting for weeks now all you do is meow please just one approval please\" \/><\/p>\n<p>i burst into laughter every time i see this.<\/p>\n<video controls>\n    <source src=\"\/assets\/fuck%20you%20chicago%20and%20northwestern%20railroad%20company.mp4\">\n<\/video>\n<p>FUCK YOU CHICAGO AND NORTHWESTERN RAILROAD COMPANY<\/p>\n<h3 id=\"things-i-have-created-1\">things i have created<a class=\"zola-anchor\" href=\"#things-i-have-created-1\" aria-label=\"Anchor link for: things-i-have-created-1\"><\/a>\n<\/h3>\n<p><a href=\"https:\/\/github.com\/rust-lang\/rust\/pull\/118756\">use bold magenta instead of bold white for highlighting<\/a>. this is both very gay and legitimately useful. i would like more of my life to be like this, please.<\/p>\n<p><a href=\"https:\/\/github.com\/rust-lang\/triagebot\/pull\/1756\">make triagebot cuter<\/a><\/p>\n<p><img src=\"\/assets\/pendant-selfie.jpg\" alt=\"mirror selfie!! i&#39;m wearing a long sleeved burgundy shirt that hugs my curves and jeans that are just visible at the waist. my nails are painted glossy black. i&#39;m wearing a bronze-ish metal watch, a brown bracelet, and a delicate necklace with a picture of a flower. my hair is half up and drapes over my left shoulder. i&#39;m smiling wide, bringing out my laugh lines, although my mouth is closed.\" \/><\/p>\n<p>this is the first time i felt like a woman.<\/p>\n"},{"title":"How to maintain an Open Source project","published":"2023-12-04T00:00:00+00:00","updated":"2023-12-04T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/how-to-maintain-an-open-source-project\/"}},"id":"https:\/\/jyn.dev\/how-to-maintain-an-open-source-project\/","summary":"How to maintain a project without burning yourself out","content":"<p>Open source is unique in that <em>energy<\/em>, not time or money, is the limiting factor. The existential threats are maintainer burnout and an imploding community, not failing to make payroll. As a result, it\u2019s very important to do three things:<\/p>\n<ol>\n<li>Recruit new maintainers as frequently as possible.<\/li>\n<li>Ruthlessly prioritize the energy of existing maintainers.<\/li>\n<li>Be kind to your users.<\/li>\n<\/ol>\n<p>All other concerns are in some sense secondary; it doesn\u2019t matter if a project has lots of useful features if it\u2019s been abandoned for 4 years and no longer compiles, or if it has no users.<\/p>\n<p>Note that \"be kind to your users\" is not the same as doing everything they ask for. You can and should say no to features. Being kind is really more simple stuff \u2013 don\u2019t be rude in discussions, don\u2019t make breaking changes unless there\u2019s a good reason (there\u2019s tools for this, like <a href=\"https:\/\/github.com\/obi1kenobi\/cargo-semver-checks#cargo-semver-checks\">cargo-semver-checks<\/a>!), make it easy for people to submit bug reports and ask questions. If you close an issue as WONTFIX, tell them why; if it's not too much effort, perhaps say the circumstances under which you might change your mind, or give an example of another way to solve their problem.<\/p>\n<h2 id=\"balancing-priorities\">Balancing priorities<a class=\"zola-anchor\" href=\"#balancing-priorities\" aria-label=\"Anchor link for: balancing-priorities\"><\/a>\n<\/h2>\n<p>One might notice that all three of those bullets are contradictory: recruiting new maintainers takes energy from existing maintainers; a feature that\u2019s fun for a developer to write may not be useful for users, or vice versa; the more features a project has, the more complicated it gets and the harder it is to onboard new maintainers. Balancing these concerns is the cornerstone of an open source project.<\/p>\n<p>Here are a few concrete examples of how to meet those goals:<\/p>\n<ul>\n<li>Automate everything you find boring. CI, releases, updating dependencies, formatting, documentation are all good candidates here. Remember that it\u2019s better to e.g. have a release that\u2019s missing pre-compiled binaries than no release at all. There are many tools that can do this for you; a few examples in the rust ecosystem are Github Actions, <a href=\"https:\/\/opensource.axo.dev\/cargo-dist\/\">cargo-dist<\/a>, dependabot, rustfmt, and rustdoc.<\/li>\n<li>Post about the project frequently on social media or other platforms. This is both a good motivator for you to see what you\u2019ve accomplished and a great way to get other people\u2019s interest, either as users or maintainers. Don\u2019t wait until you have something polished, a half-baked post is better than no post at all. Start a platform dedicated to your project where you can talk with users and contributors; this can be as simple as a Discord server or as complicated as a self-hosted Zulip instance. The important thing is, again, that it should be fun for you to set up.<\/li>\n<li>Document how to develop the project, and how to write documentation. The former pays dividends \u2013 not just for recruiting new maintainers, but also for reminding yourself how the project works when you come back from vacation in two weeks. The latter reduces your burden for writing documentation by making it easy for other people to help you write it.<\/li>\n<li>Take vacations, even extended vacations. If you know you'll be on vacation ahead of time, be kind to your users or fellow maintainers by telling them you're gone. If you have automated review assignments, set it up to allow people to take breaks.<\/li>\n<\/ul>\n<h2 id=\"culture\">Culture<a class=\"zola-anchor\" href=\"#culture\" aria-label=\"Anchor link for: culture\"><\/a>\n<\/h2>\n<p>Past those concrete things, though, it\u2019s important to set a culture for the project that makes you eager to work on it.<\/p>\n<ul>\n<li>Develop a taste for when to say no to features. This is especially hard because it changes as a project grows. Early on, features are easy and fun to add, and open up lots of possibilities for your users. Later, they get harder to add, and new features make the tool more complicated for your existing users (who after all, started using your tool when it was more simpler) and new users (who have to read through all your documentation); and you have enough users it becomes less important to grow your user base.<\/li>\n<li>Find people who are as excited about the project as you are. Invite constructive criticism \u2013 it\u2019s a great way for you to learn and improve the project at the same time! \u2013 but at the same time make sure you balance it with positive feedback so you don\u2019t get overwhelmed.<\/li>\n<li>Don\u2019t let yourself get burned out. This is in some ways the hardest part. There is only so much work you can do every week before you get tired. Pushing yourself past that will work, temporarily, but at the cost of draining your energy and making you less willing to work on the project in the future. You are not a company and do not have an SLA to your users; if they are really harmed they can use an older release or submit a patch themselves.<\/li>\n<\/ul>\n<h2 id=\"recruiting-maintainers\">Recruiting maintainers<a class=\"zola-anchor\" href=\"#recruiting-maintainers\" aria-label=\"Anchor link for: recruiting-maintainers\"><\/a>\n<\/h2>\n<p>Most people will never ask to be maintainers. Either they don't want to be presumptuous, or they don't feel experienced enough, or it just won't occur to them. Instead, you should reach out to them.<\/p>\n<p>As a general rule, I suggest leaning towards trusting people with merge privileges early. Anyone who's active in the project is likely going to stick around for a while; you can always revert code they merge, and you don't have to give them publish access. This is a great way to make people feel like part of the project, which in turns makes them more likely to keep contributing.<\/p>\n<p>I have a lot more to say on this, but most of it has already been said by my friend Alice Cecile in <a href=\"https:\/\/www.youtube.com\/watch?v=xM7bI2OPPLQ\">this wonderful talk<\/a>. Go watch it. Seriously, go watch it, it's only 20 minutes and none of them are wasted.<\/p>\n<p>Projects can become self-sustaining rather quickly \u2013 you really only need to recruit one or two other people, get five or ten users, and have a decent codebase to expand on. Once you\u2019ve done that, you can mentor your other contributors into taking a leading role, and, eventually, step back from your role as the founder.<\/p>\n"},{"title":"Why is Rust's build system uniquely hard to use?","published":"2023-01-12T00:00:00+00:00","updated":"2023-01-12T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/bootstrapping-rust-in-2023\/"}},"id":"https:\/\/jyn.dev\/bootstrapping-rust-in-2023\/","summary":"Goals for improving Rust's build system and making it easier to understand","content":"<p><em>This post will assume you have watched <a href=\"https:\/\/www.youtube.com\/watch?v=oUIjG-y4zaA\">https:\/\/www.youtube.com\/watch?v=oUIjG-y4zaA<\/a>.\nYou may also find it helpful to read <a href=\"https:\/\/rustc-dev-guide.rust-lang.org\/building\/bootstrapping\/what-bootstrapping-does.html#stages-of-bootstrapping\">https:\/\/rustc-dev-guide.rust-lang.org\/building\/bootstrapping\/what-bootstrapping-does.html#stages-of-bootstrapping<\/a>, but I won't assume prior knowledge of the information there.<\/em><\/p>\n<!-- This is a post about how I would like Rust's bootstrapping system to work in an ideal world. -->\n<!-- I am not sure all these changes are feasible - they certainly can be done at a technical -->\n<!-- blah blah coordination is hard -->\n<h2 id=\"why-is-bootstrap-confusing\">Why is bootstrap confusing?<a class=\"zola-anchor\" href=\"#why-is-bootstrap-confusing\" aria-label=\"Anchor link for: why-is-bootstrap-confusing\"><\/a>\n<\/h2>\n<p>People get confused by Rust's build system a lot. I have been trying for a while to figure out what\nmakes Rust uniquely hard here, as a lot of the people who are confused are experienced compiler\nengineers who have used staged compilers in the past. Here are some theories I have.<\/p>\n<h3 id=\"is-stage-0-is-uniquely-confusing\">Is \"stage 0\" is uniquely confusing?<a class=\"zola-anchor\" href=\"#is-stage-0-is-uniquely-confusing\" aria-label=\"Anchor link for: is-stage-0-is-uniquely-confusing\"><\/a>\n<\/h3>\n<h4 id=\"what-do-other-compilers-call-stage-0\">What do other compilers call \"stage 0\"?<a class=\"zola-anchor\" href=\"#what-do-other-compilers-call-stage-0\" aria-label=\"Anchor link for: what-do-other-compilers-call-stage-0\"><\/a>\n<\/h4>\n<p><a href=\"https:\/\/gcc.gnu.org\/install\/build.html\">https:\/\/gcc.gnu.org\/install\/build.html<\/a> refers to a \"3-stage\" build, and names the stages \"stage1\", \"stage2\", \"stage3\".\nIt also references a \"native compiler\". As far as I can tell, \"native compiler\" corresponds to what Rust calls \"stage 0\", and stage 1\/2\/3 are all exactly equivalent, i.e:<\/p>\n<ul>\n<li>stage0 is a pre-existing compiler, which is assumed to already exist (or in rust's case, a downloaded beta compiler).<\/li>\n<li>stage1 is the sources from latest master, built by stage0, and has a different ABI from stage2 and stage3.<\/li>\n<li>stage2 is the same sources, built by stage1.<\/li>\n<li>stage3 is byte-for-byte identical with stage2, only useful for verifying reproducible builds.<\/li>\n<\/ul>\n<p><a href=\"https:\/\/llvm.org\/docs\/AdvancedBuilds.html#bootstrap-builds\">https:\/\/llvm.org\/docs\/AdvancedBuilds.html#bootstrap-builds<\/a> says \"In a simple two-stage bootstrap build, we build clang using the system compiler, then use that just-built clang to build clang again.\" which again seems to match GCC and Rust.<\/p>\n<p><a href=\"https:\/\/gitlab.haskell.org\/ghc\/ghc\/blob\/master\/hadrian\/README.md#staged-compilation\">https:\/\/gitlab.haskell.org\/ghc\/ghc\/blob\/master\/hadrian\/README.md#staged-compilation<\/a> seems to match GCC, Clang, and Rust.<\/p>\n<!-- TOOD: mention that there are other self-hosted compilers that don't use stages? would make it clear that having \"too close\" a model to other compilers is confusing -->\n<h4 id=\"why-would-stage-0-be-confusing\">Why would \"stage 0\" be confusing?<a class=\"zola-anchor\" href=\"#why-would-stage-0-be-confusing\" aria-label=\"Anchor link for: why-would-stage-0-be-confusing\"><\/a>\n<\/h4>\n<p>Maybe treating this as \"just\" another stage, rather than naming it \"native compiler\" or \"system compiler\", is confusing.\nThat alone seems unlikely though; \"0\" at least to me seems like a good indication that it's not being built from source.\nSo renaming stage 0 to \"bootstrap compiler\" or \"pre-compiled compiler\", while helpful, seems unlikely to clear up the confusion.<\/p>\n<h3 id=\"is-building-the-standard-library-confusing\">Is building the standard library confusing?<a class=\"zola-anchor\" href=\"#is-building-the-standard-library-confusing\" aria-label=\"Anchor link for: is-building-the-standard-library-confusing\"><\/a>\n<\/h3>\n<h4 id=\"what-do-other-compilers-do\">What do other compilers do?<a class=\"zola-anchor\" href=\"#what-do-other-compilers-do\" aria-label=\"Anchor link for: what-do-other-compilers-do\"><\/a>\n<\/h4>\n<p>GCC and Clang do not build their standard libraries from source. Instead, they use the same dynamically linked system standard library for all stages, including the \"stage 0\" or \"system compiler\" stage.<\/p>\n<p>(As an aside, that's <em>horrifying<\/em>, C doesn't have a standardized ABI and so this can cause miscompilations even if there are no bugs in the standard library or gcc itself: <a href=\"https:\/\/faultlore.com\/blah\/c-isnt-a-language\/#c-doesnt-actually-have-an-abi\">https:\/\/faultlore.com\/blah\/c-isnt-a-language\/#c-doesnt-actually-have-an-abi<\/a>)<\/p>\n<p>GHC <em>does<\/em> build its standard library from source. It has two parts to its standard library:<\/p>\n<ol>\n<li><code>GHC.Base<\/code>, which can only be compiled by exactly one version of GHC. This is a runtime, analogous to <code>crt0.o<\/code> or <a href=\"https:\/\/stdrs.dev\/nightly\/x86_64-unknown-linux-gnu\/std\/rt\/index.html\"><code>std::rt<\/code><\/a>.<\/li>\n<li>Everything else in the <code>GHC.*<\/code> namespace. This can be compiled by any version of GHC.<\/li>\n<\/ol>\n<h4 id=\"what-does-rust-do\">What does Rust do?<a class=\"zola-anchor\" href=\"#what-does-rust-do\" aria-label=\"Anchor link for: what-does-rust-do\"><\/a>\n<\/h4>\n<p>Rust also builds its standard library from source. It has three parts:<\/p>\n<ol>\n<li><code>core<\/code><\/li>\n<li><code>alloc<\/code><\/li>\n<li><code>std<\/code><\/li>\n<\/ol>\n<p>All three can be built by exactly <em>two<\/em> versions of <code>rustc<\/code>:<\/p>\n<ol>\n<li>The previous beta compiler.<\/li>\n<li>The in-tree sources, versioned in the same git repo.<\/li>\n<\/ol>\n<p>As discussed in my RustConf talk, they distinguish the two with <code>cfg(bootstrap)<\/code>; when set, we're using beta, when unset, we're using the in-tree sources.<\/p>\n<h4 id=\"why-would-this-be-confusing\">Why would this be confusing?<a class=\"zola-anchor\" href=\"#why-would-this-be-confusing\" aria-label=\"Anchor link for: why-would-this-be-confusing\"><\/a>\n<\/h4>\n<p>Having to support two versions of the compiler seems to be unique to Rust's standard library. C bypasses the question altogether by not having language intrinsics in the standard library and supporting any compiler version; Haskell only requires one version of the compiler to be supported.<\/p>\n<p>Having to support two versions is in fact the original motivation for this post, since it causes\nlots of pain for changes that modify both the compiler and standard library; see\n<a href=\"https:\/\/github.com\/rust-lang\/rust\/pull\/84863\">https:\/\/github.com\/rust-lang\/rust\/pull\/84863<\/a> for an example that modifies both in the same PR and\n<a href=\"https:\/\/github.com\/rust-lang\/rust\/pull\/99917\">https:\/\/github.com\/rust-lang\/rust\/pull\/99917<\/a> for an example that depends on changes to the\ncompiler that haven't yet landed on beta.<\/p>\n<h3 id=\"what-can-we-do-about-it\">What can we do about it?<a class=\"zola-anchor\" href=\"#what-can-we-do-about-it\" aria-label=\"Anchor link for: what-can-we-do-about-it\"><\/a>\n<\/h3>\n<p>Supporting two versions is not an intrinsic requirement. We do it for two reasons:<\/p>\n<ol>\n<li>It allows testing changes to the standard library without having to first build the compiler.\n<a href=\"https:\/\/github.com\/rust-lang\/rust\/issues\/65031\">Building the compiler is painfully slow<\/a>.<\/li>\n<li>It allows <a href=\"https:\/\/rustc-dev-guide.rust-lang.org\/building\/bootstrapping\/what-bootstrapping-does.html#complications-of-bootstrapping\">using nightly standard library features<\/a> in the compiler before they land on beta.<\/li>\n<\/ol>\n<p>1 is \"just\" implementation work to fix: if there are no changes to the compiler, we can download CI artifacts for that commit and use those instead. There are <a href=\"https:\/\/github.com\/rust-lang\/rust\/issues\/81930\">a few bugs to fix<\/a> but they're surmountable.<\/p>\n<p>2 is harder. Either we add <code>cfg(bootstrap)<\/code> to the compiler to use a different implementation when building with stage0 than stage1, or we stop using nightly standard library features until they reach beta. See <a href=\"https:\/\/rust-lang.zulipchat.com\/#narrow\/stream\/131828-t-compiler\/topic\/Building.20rustc.20with.20beta.20libstd\/near\/209899890\">https:\/\/rust-lang.zulipchat.com\/#narrow\/stream\/131828-t-compiler\/topic\/Building.20rustc.20with.20beta.20libstd\/near\/209899890<\/a> for a very (very) long discussion of the tradeoffs here.<\/p>\n<p>There are some more benefits to supporting a single version not discussed there:<\/p>\n<ul>\n<li>Rebasing over master recompiles much less code. Modifying a single line in <code>core<\/code> no longer requires rebuilding the world; only changes to the compiler require the compiler to be rebuilt.<\/li>\n<li>Modifying the standard library locally don't require rebuilding the compiler. This is especially relevant to people who are changing how the standard library interacts with the compiler; we would be able to remove <code>--keep-stage-std 0<\/code> and all associated footguns as a workflow altogether.<\/li>\n<li>Creating a new Rust release requires touching <em>drastically<\/em> less <code>cfg(bootstrap)<\/code> since lang items no longer need to be modified, only small parts of the compiler.<\/li>\n<\/ul>\n<p>I've put together some data on how often using those features before they hit beta happens in\npractice, and - at least from 1.61.0 onwards - it appears it <em>never<\/em> happens. See\n<a href=\"https:\/\/github.com\/jyn514\/rust\/tree\/versions-used\">https:\/\/github.com\/jyn514\/rust\/tree\/versions-used<\/a> for how that data was gathered (run\n<code>.\/collect_new_versions.sh<\/code>). What's more common is renaming a method before it's stable; see\n<a href=\"https:\/\/github.com\/rust-lang\/rust\/pull\/79805\/files\">https:\/\/github.com\/rust-lang\/rust\/pull\/79805\/files<\/a> for an example. The <code>cfg(bootstrap)<\/code> code to\nhandle this in the compiler should be pretty simple.<\/p>\n<p>I've talked to people on both T-libs and T-compiler and they say that removing <code>cfg(bootstrap)<\/code> would be an <em>enormous<\/em> help. Some testimonials:<\/p>\n<blockquote>\n<p><a href=\"https:\/\/github.com\/thomcc\">@thomcc<\/a>: Yes please. It's a huge headache. It also frequently comes up as a reason not to let the const-eval team experiment with stuff, since we know <code>~const<\/code> is likely going away and don't want to deal with the \"lol we <code>cfg(bootstrap)<\/code>ed off all of core::iter\".<\/p>\n<\/blockquote>\n<blockquote>\n<p><a href=\"https:\/\/github.com\/fee1-dead\">@fee1-dead<\/a>: Working on const traits makes the bootstrap issue very apparent because almost all bugs would be found from attempting to use the feature in the standard library. Fixes for those bugs would need to wait six weeks before finally released as the beta compiler, which slows down development and evolution of the feature.<\/p>\n<\/blockquote>\n<blockquote>\n<p><a href=\"https:\/\/github.com\/m-ou-se\">@m-ou-se<\/a>: Sometimes it gets super messy with all the <code>cfg(bootstrap)<\/code> stuff for things relying on built-in macros or new lang items. Please fix cfg bootstrap hell :D<\/p>\n<\/blockquote>\n<blockquote>\n<p><a href=\"https:\/\/github.com\/workingjubilee\">@workingjubilee<\/a>: Using beta stdlib would make it much easier to experiment outside the compiler\/stdlib proper.<\/p>\n<\/blockquote>\n<blockquote>\n<p><a href=\"https:\/\/github.com\/nilstrieb\">@Nilstrieb<\/a>: I'd rather have a few cfgs in the compiler when necessary instead of cfgs in std all the time.<\/p>\n<\/blockquote>\n<h4 id=\"but-that-just-moves-the-cfg-bootstrap-to-the-compiler\">\"But that just moves the cfg(bootstrap) to the compiler!\"<a class=\"zola-anchor\" href=\"#but-that-just-moves-the-cfg-bootstrap-to-the-compiler\" aria-label=\"Anchor link for: but-that-just-moves-the-cfg-bootstrap-to-the-compiler\"><\/a>\n<\/h4>\n<p>A common (incorrect) objection is that, after this change, adding new language items would require adding <code>cfg(bootstrap)<\/code> to the compiler.\nThis is false. The compiler <em>also<\/em> only has to support building one version of <code>std<\/code> after this change. The only time the bootstrap standard library is involved is when <em>building the compiler<\/em>. Unlike how <code>std<\/code> is intrinsically tied to the compiler due to lang items, the compiler doesn't intrinsically depend on implementation details of the standard library; it only uses them for dogfooding.<\/p>\n<p>(I don't want to hear about how <code>lang<\/code> items are ideologically impure. I don't care. It's not changing.)<\/p>\n<p>Here is a graph of what the build would like before:<\/p>\n<table><thead><tr><th>component<\/th><th>built-by<\/th><th>building<\/th><\/tr><\/thead><tbody>\n<tr><td>compiler<\/td><td>1 std<\/td><td>1 std<\/td><\/tr>\n<tr><td>std<\/td><td>2 compilers<\/td><td>NA<\/td><\/tr>\n<\/tbody><\/table>\n<p>and after:<\/p>\n<table><thead><tr><th>component<\/th><th>built-by<\/th><th>building<\/th><\/tr><\/thead><tbody>\n<tr><td>compiler<\/td><td>2 std<\/td><td>1 std<\/td><\/tr>\n<tr><td>std<\/td><td>1 compiler<\/td><td>NA<\/td><\/tr>\n<\/tbody><\/table>\n<h3 id=\"is-the-stage-terminology-itself-confusing\">Is the \"stage\" terminology itself confusing?<a class=\"zola-anchor\" href=\"#is-the-stage-terminology-itself-confusing\" aria-label=\"Anchor link for: is-the-stage-terminology-itself-confusing\"><\/a>\n<\/h3>\n<h4 id=\"what-do-other-compilers-do-1\">What do other compilers do?<a class=\"zola-anchor\" href=\"#what-do-other-compilers-do-1\" aria-label=\"Anchor link for: what-do-other-compilers-do-1\"><\/a>\n<\/h4>\n<p>For GHC, <code>build stage1:exe:ghc-bin<\/code> builds stage1 GHC with the stage0 compiler.<\/p>\n <!-- https:\/\/discourse.llvm.org\/t\/how-to-run-individual-phases-of-a-2-or-3-stage-build\/2596\/2 -->\n<p>For Clang, <code>ninja stage2<\/code> builds the stage2 clang with the stage1 compiler and <code>ninja clang-bootstrap-deps<\/code> builds the stage1 clang with the stage0 compiler.<\/p>\n<!-- https:\/\/github.com\/ocaml\/ocaml\/blob\/64ef2d0ce1eb7d5f09ac6cde1a78f74b62804cc6\/Makefile#L698-L702 -->\n<!-- https:\/\/github.com\/ocaml\/ocaml\/blob\/64ef2d0ce1eb7d5f09ac6cde1a78f74b62804cc6\/Makefile#L833-L841 -->\n<p>OCaml uses <code>make coreall<\/code> to build the stage1 OCaml with the bootstrap compiler and <code>make bootstrap<\/code> to build a full bootstrap compiler.<\/p>\n<!-- zig is a pain in the ass to build on windows because it needs llvm installed -->\n<!-- pypy doesn't have stages -->\n<!-- swift isn't self-hosted https:\/\/github.com\/apple\/swift\/blob\/2c7b0b22831159396fe0e98e5944e64a483c356e\/www\/FAQ.rst -->\n<!-- http:\/\/www.sbcl.org\/porting.html is unclear and I don't feel like building it -->\n<h4 id=\"what-does-rust-do-1\">What does Rust do?<a class=\"zola-anchor\" href=\"#what-does-rust-do-1\" aria-label=\"Anchor link for: what-does-rust-do-1\"><\/a>\n<\/h4>\n<p><code>x build --stage 0 rustc<\/code> builds stage1 rustc with the stage0 compiler.\n<code>x build --stage 1 rustc<\/code> builds stage2 rustc with the stage1 compiler.<\/p>\n<p>This is off-by-one from how <em>every other modern compiler<\/em> counts stages.<\/p>\n<p><a href=\"https:\/\/rustc-dev-guide.rust-lang.org\/building\/bootstrapping\/what-bootstrapping-does.html#understanding-stages-of-bootstrap\">https:\/\/rustc-dev-guide.rust-lang.org\/building\/bootstrapping\/what-bootstrapping-does.html#understanding-stages-of-bootstrap<\/a>\nspends several paragraphs talking about how <code>build --stage N<\/code> means \"build with stage N\", not \"create the\ncompiler that lives in the stage N sysroot\".  All the people I've talked to who say this\nmeaning of <code>--stage N<\/code> is intuitive have been using <code>x.py<\/code> for several years and are experts in the\nsystem. Nearly all the people I've talked to who find it confusing are either new to the compiler,\nor contribute regularly but aren't experts in Rust's build system - even those who are experienced\nin bootstrapping compilers for other languages!<\/p>\n<p>In the words of <a href=\"https:\/\/github.com\/nilstrieb\">@Nilstrieb<\/a>:<\/p>\n<blockquote>\n<p>You build a <em>target<\/em>. The focus is always <em>what<\/em> you build.<\/p>\n<\/blockquote>\n<!-- TODO: talk with manish about how to make this less combative -->\n<p>We are not meeting that intuition today with x.py.  It seems unfortunate to have a system that only\nmakes sense if you've used it for a long time and are accustomed to it. If you're using it for\nseveral years, you have time to relearn the system.<\/p>\n<h4 id=\"what-can-we-do-instead\">What can we do instead?<a class=\"zola-anchor\" href=\"#what-can-we-do-instead\" aria-label=\"Anchor link for: what-can-we-do-instead\"><\/a>\n<\/h4>\n<p>I would like to introduce four new flags:<\/p>\n<ul>\n<li><code>--bootstrap-sysroot<\/code><\/li>\n<li><code>--dev-sysroot<\/code><\/li>\n<li><code>--dist-sysroot<\/code><\/li>\n<li><code>--reproducible-sysroot<\/code><\/li>\n<\/ul>\n<p>This correspond closely, but not exactly, to <code>--stage 0\/1\/2\/3<\/code> (respectively).\nHere is a conversion guide between the two: <img src=\"\/assets\/bootstrap-sysroot-conversion.png\" alt=\"conversion guide (see \/assets\/bootstrapping.tex for source)\" \/><\/p>\n<p>I propose <em>not<\/em> putting this in the dev-guide, but creating an inside-rust post which we link to in bootstrap's changelog.\nThe idea is for people who've already been using x.py to see the guide, but not people learning the tool for the first time.\nWe would keep <code>--stage<\/code> for a time, but eventually deprecate it.<\/p>\n<p>I want to call out a few interesting properties of these flags:<\/p>\n<ul>\n<li><code>build --bootstrap-sysroot std<\/code> makes it more clear how <em>strange<\/em> it is that <code>std<\/code> is built before rustc. This isn't something people will need to think about if we change <code>std<\/code> to only need to support one compiler.<\/li>\n<li>The names are self-describing. People don't have to wonder whether <code>--stage 1<\/code> is the flag they want or not; <code>--dev-sysroot<\/code> makes it clear it is.<\/li>\n<li>The only times <code>--bootstrap-sysroot<\/code> will be used is for\n<ul>\n<li><code>doc<\/code> \/ <code>clippy<\/code> (to use the beta tools instead of recompiling, although <code>--dev-sysroot<\/code> will still be supported). Given that <code>download-rustc<\/code> will be a blessed workflow, we may want to drop support for this in the future.<\/li>\n<li><code>build expand-yaml-anchors<\/code> (or other bootstrap tools); <em>not<\/em> the standard library or compiler.<\/li>\n<\/ul>\n<\/li>\n<li><code>check<\/code> will only ever support one stage (<code>--dev-sysroot<\/code> for the compiler and std; <code>--bootstrap-sysroot<\/code> for bootstrap tools)<\/li>\n<li><code>test --dev-sysroot ui<\/code> now matches the sysroot of <code>build compiler<\/code> (!). I am planning to make <code>test rustc_data_structures --dev-sysroot<\/code> compile <code>rustc_data_strucutres<\/code> (like <code>--stage 0<\/code> today) for consistency, and so that we are always testing the compiler <em>that will end up<\/em> in the given sysroot.<\/li>\n<li><code>build --dev-sysroot rustdoc<\/code> now matches the sysroot of <code>build --dev-sysroot compiler<\/code> (!). Hopefully we can also do this for clippy and miri.<\/li>\n<\/ul>\n<p>To make the new flags easier to learn, we can name the sysroots directories after the flags: <code>build\/host\/{bootstrap,dev,dist}-sysroot<\/code>.\nI have not yet decided if we should introduce a <code>reproducible-sysroot<\/code> or not; see <a href=\"https:\/\/github.com\/rust-lang\/rust\/issues\/90244#issuecomment-1120649548\">https:\/\/github.com\/rust-lang\/rust\/issues\/90244#issuecomment-1120649548<\/a> for some of the difficulties involved.<\/p>\n<h2 id=\"misc-breaking-changes\">\"Misc breaking changes\"<a class=\"zola-anchor\" href=\"#misc-breaking-changes\" aria-label=\"Anchor link for: misc-breaking-changes\"><\/a>\n<\/h2>\n<p>Skimmed over in the previous sections is how to get people to use <code>download-rustc<\/code>.\nIf we enable it unconditionally for everyone, distros will get very upset.\nIf we don't enable it, people working on the standard library will have horrifically long compile times.<\/p>\n<p>To avoid this, we could <em>require<\/em> people to set a profile for <code>config.toml<\/code>. To avoid making distros\nhate us too much, <code>.\/configure<\/code> would set the <code>user<\/code> profile; running <code>.\/x.py build<\/code> without\ncreating a profile would give a hard error pointing you to <code>x.py setup<\/code>. We would still treat an\nempty config.toml as opting-in to no profile.<\/p>\n<h2 id=\"summary-of-future-work\">Summary of future work<a class=\"zola-anchor\" href=\"#summary-of-future-work\" aria-label=\"Anchor link for: summary-of-future-work\"><\/a>\n<\/h2>\n<p>Here are all the proposed changes in this post, gathered in one place:<\/p>\n<ul>\n<li>Change <code>.\/configure<\/code> to set <code>profile = \"user\"<\/code>.<\/li>\n<li>Make <code>profile<\/code> required in config.toml.<\/li>\n<li>Fix the existing bugs in <code>download-rustc<\/code>.<\/li>\n<li>Enable <code>download-rustc = \"if-unchanged\"<\/code> by default for the <code>library<\/code> profile.<\/li>\n<li>Get rid of <code>build --stage 0 std<\/code>. The compiler will be unconditionally built with beta std, not nightly std.\n<ul>\n<li>Get rid of <code>stage0-sysroot<\/code>.<\/li>\n<li>Make sure <code>download-rustc<\/code> doesn't build the compiler from source if there are only library changes; this needs to be careful to still rebuild stage 2 rustc if there are library changes.<\/li>\n<\/ul>\n<\/li>\n<li>Rename <code>build\/host\/stage{0,1,2}<\/code> to <code>build\/host\/{bootstrap,dev,dist}-sysroot<\/code>.<\/li>\n<li>Add <code>--{bootstrap,dev,dist}-sysroot<\/code> flags.\n<ul>\n<li>When doing this, clippy and miri will start using the same flags as rustdoc. See <a href=\"https:\/\/rust-lang.zulipchat.com\/#narrow\/stream\/326414-t-infra.2Fbootstrap\/topic\/Stage.20numbering.20for.20tools\/near\/298189698\">https:\/\/rust-lang.zulipchat.com\/#narrow\/stream\/326414-t-infra.2Fbootstrap\/topic\/Stage.20numbering.20for.20tools\/near\/298189698<\/a> for how tool flags are numbered today; after this change, both <code>build --dev-sysroot rustdoc<\/code> and <code>build --dev-sysroot clippy<\/code> will build rustc once, as a library.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>That's a lot of breaking changes and a lot of work, for things we are not sure will make the user\nexperience easier.  To avoid multiple breaking changes in short succession, I propose making all the\nchanges at once, and inviting people to try out the changes from a branch before merging them. If we\nhave trouble getting user feedback, I could create a standalone binary which uses the new\n<code>*-sysroot<\/code> flags even on the master branch. Note that this will not be possible for the changes\nremoving <code>cfg(bootstrap)<\/code> from std, since that requires changes outside of bootstrap.<\/p>\n<h2 id=\"questions-concerns-hate-mail\">Questions? Concerns? Hate mail?<a class=\"zola-anchor\" href=\"#questions-concerns-hate-mail\" aria-label=\"Anchor link for: questions-concerns-hate-mail\"><\/a>\n<\/h2>\n<p>Feel free to contact me (<code>@jyn<\/code>) in <a href=\"https:\/\/rust-lang.zulipchat.com\/#narrow\/stream\/326414-t-infra.2Fbootstrap\">https:\/\/rust-lang.zulipchat.com\/#narrow\/stream\/326414-t-infra.2Fbootstrap<\/a>.<\/p>\n"},{"title":"Git cheats","published":"2022-09-02T00:00:00+00:00","updated":"2022-09-02T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/git-cheats\/"}},"id":"https:\/\/jyn.dev\/git-cheats\/","summary":"tips and tricks for dealing with git's horrible error messages","content":"<p>A small note: this will be much less organized and thought out than my other blog posts because I have been unable to convince myself to write a blog in about 2 years. Instead these are some notes to myself that happen to be public for anyone who finds them useful.<\/p>\n<p>A small note written after spending 5 minutes on this post: holy shit this is so deranged I'm turning into Gankra lmao<\/p>\n<h1 id=\"git\">Git<a class=\"zola-anchor\" href=\"#git\" aria-label=\"Anchor link for: git\"><\/a>\n<\/h1>\n<p>big fan of the data model.\nnot a fan of the UI.<\/p>\n<p>Ok so before anything else, don't panic. Your code is probably still there somewhere you just can't find it. If you <em>stop running commands<\/em> right now, you can probably get it back.<\/p>\n<h2 id=\"i-had-a-commit-and-then-i-checked-out-a-different-branch-and-now-i-can-t-find-the-commit-again-sob\">I had a commit and then I checked out a different branch and now I can't find the commit again \ud83d\ude2d<a class=\"zola-anchor\" href=\"#i-had-a-commit-and-then-i-checked-out-a-different-branch-and-now-i-can-t-find-the-commit-again-sob\" aria-label=\"Anchor link for: i-had-a-commit-and-then-i-checked-out-a-different-branch-and-now-i-can-t-find-the-commit-again-sob\"><\/a>\n<\/h2>\n<p>Run <code>git reflog<\/code>, it shows you all the recent commits you were on.<\/p>\n<h2 id=\"someone-told-me-to-rebase-and-i-did-that-but-now-git-says-failed-to-push-some-refs-why-is-this-happening\">someone told me to rebase and I did that but now git says \"failed to push some refs\"??? why is this happening<a class=\"zola-anchor\" href=\"#someone-told-me-to-rebase-and-i-did-that-but-now-git-says-failed-to-push-some-refs-why-is-this-happening\" aria-label=\"Anchor link for: someone-told-me-to-rebase-and-i-did-that-but-now-git-says-failed-to-push-some-refs-why-is-this-happening\"><\/a>\n<\/h2>\n<p>The error probably looks like this:<\/p>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">$ git push\n<\/span><span class=\"z-text z-plain\">To github.com:jyn514\/rust.git\n<\/span><span class=\"z-text z-plain\"> ! [rejected]                simplify-storage -&gt; simplify-storage (non-fast-forward)\n<\/span><span class=\"z-text z-plain\">error: failed to push some refs to &#39;git@github.com:jyn514\/rust.git&#39;\n<\/span><span class=\"z-text z-plain\">hint: Updates were rejected because the tip of your current branch is behind\n<\/span><span class=\"z-text z-plain\">hint: its remote counterpart. Integrate the remote changes (e.g.\n<\/span><span class=\"z-text z-plain\">hint: &#39;git pull ...&#39;) before pushing again.\n<\/span><span class=\"z-text z-plain\">hint: See the &#39;Note about fast-forwards&#39; in &#39;git push --help&#39; for details.\n<\/span><\/code><\/pre>\n<p>This is totally normal. Git just gives absolutely awful error messages. Run <code>git push --force-with-lease<\/code> and it should work fine.<\/p>\n<p>Be careful using <code>--force<\/code>. 99% of the time it works. The other 1% someone else pushed to your branch and you just deleted their work. They will not be happy.<\/p>\n<h2 id=\"wait-but-i-ran-git-pull-before-i-looked-at-your-blog-now-what-do-i-do\">wait but I ran git pull before I looked at your blog now what do I do???<a class=\"zola-anchor\" href=\"#wait-but-i-ran-git-pull-before-i-looked-at-your-blog-now-what-do-i-do\" aria-label=\"Anchor link for: wait-but-i-ran-git-pull-before-i-looked-at-your-blog-now-what-do-i-do\"><\/a>\n<\/h2>\n<p>Your latest commit probably looks like this:<\/p>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">commit 88fdea3dd2e876a92960601c019e729401e832ab (HEAD -&gt; simplify-storage)\n<\/span><span class=\"z-text z-plain\">Merge: d6bd3ef8662 6b22a42e011\n<\/span><span class=\"z-text z-plain\">Author: jyn &lt;github@jyn.dev&gt;\n<\/span><span class=\"z-text z-plain\">Date:   Fri Sep 2 19:24:33 2022 -0500\n<\/span><span class=\"z-text z-plain\">\n<\/span><span class=\"z-text z-plain\">    Merge github.com:jyn514\/rust into simplify-storage\n<\/span><\/code><\/pre>\n<p>Get rid of it. We don't want it. If you have changes since the latest commit, save them somewhere (e.g. <code>git stash<\/code>).\nThen run <code>git reset --hard HEAD~<\/code>. Now go to the step above about <code>--force-with-lease<\/code>.<\/p>\n<h2 id=\"i-have-a-detached-head\">I have a detached HEAD<a class=\"zola-anchor\" href=\"#i-have-a-detached-head\" aria-label=\"Anchor link for: i-have-a-detached-head\"><\/a>\n<\/h2>\n<h3 id=\"wtf-does-that-mean\">WTF does that mean<a class=\"zola-anchor\" href=\"#wtf-does-that-mean\" aria-label=\"Anchor link for: wtf-does-that-mean\"><\/a>\n<\/h3>\n<p>You have turned into the Nick the Headless Horseman. Have fun trying to join the Headless Hunt.<\/p>\n<h3 id=\"no-stop-i-came-here-for-help\">No stop I came here for help<a class=\"zola-anchor\" href=\"#no-stop-i-came-here-for-help\" aria-label=\"Anchor link for: no-stop-i-came-here-for-help\"><\/a>\n<\/h3>\n<p>Git doesn't think you're on a branch or tag.<\/p>\n<p>This is <em>not<\/em> the same as being on a commit that's not in any branch or tag. It's perfectly possible\nto be on the same commit as the latest <code>main<\/code> and still have a detached HEAD. It just means that\nany new commit you make in this state won't be on a branch.<\/p>\n<h3 id=\"how-does-this-help-me-i-don-t-understand\">How does this help me I don't understand<a class=\"zola-anchor\" href=\"#how-does-this-help-me-i-don-t-understand\" aria-label=\"Anchor link for: how-does-this-help-me-i-don-t-understand\"><\/a>\n<\/h3>\n<p>Everything is fine. All your work is still here. You just need to figure out what you want to do next.<\/p>\n<h4 id=\"i-don-t-care-about-any-of-the-work-i-ve-done-in-the-last-day-i-just-want-a-git-repo-that-works-i-will-wipe-this-directory-if-you-don-t-help\">I don't care about any of the work I've done in the last day I just want a git repo that works I will wipe this directory if you don't help<a class=\"zola-anchor\" href=\"#i-don-t-care-about-any-of-the-work-i-ve-done-in-the-last-day-i-just-want-a-git-repo-that-works-i-will-wipe-this-directory-if-you-don-t-help\" aria-label=\"Anchor link for: i-don-t-care-about-any-of-the-work-i-ve-done-in-the-last-day-i-just-want-a-git-repo-that-works-i-will-wipe-this-directory-if-you-don-t-help\"><\/a>\n<\/h4>\n<p><strong>DO NOT DO THIS IF YOU HAVE WORK YOU WANT TO SAVE!!!<\/strong><\/p>\n<p>Run <code>git checkout --force $(git rev-parse --abbrev-ref origin\/HEAD)<\/code>. If that gives an error for any reason try <code>git checkout --force origin\/master <\/code>or <code>origin\/main<\/code> instead. If none of those work IDK what to tell you maybe wiping the repo is the best idea after all. Good luck.<\/p>\n<h4 id=\"i-have-work-i-want-to-save-please-help-me-fix-this-i-have-a-deadline-in-an-hour\">I have work I want to save please help me fix this I have a deadline in an hour<a class=\"zola-anchor\" href=\"#i-have-work-i-want-to-save-please-help-me-fix-this-i-have-a-deadline-in-an-hour\" aria-label=\"Anchor link for: i-have-work-i-want-to-save-please-help-me-fix-this-i-have-a-deadline-in-an-hour\"><\/a>\n<\/h4>\n<p>Ok. Pick which branch you want to put your work on. I'm going to pick <code>bark<\/code> because branches have bark. haha<\/p>\n<p>Does the commit you're on have the work you want to save?<\/p>\n<h5 id=\"yes-how-do-i-get-it-to-work\">yes how do I get it to work<a class=\"zola-anchor\" href=\"#yes-how-do-i-get-it-to-work\" aria-label=\"Anchor link for: yes-how-do-i-get-it-to-work\"><\/a>\n<\/h5>\n<p>Run <code>git branch --delete bark &amp;&amp; git checkout --branch bark<\/code>. This is the least invasive way to fix things;\nwhatever untracked code is in your working directory won't be modified, and the old branch will still be in your reflog if you really need it. If you don't know what that means, that's ok, you're on the right branch now and you can push it wherever you need.<\/p>\n<h5 id=\"no-please-i-don-t-know-how-i-got-into-this-state-i-really-hate-git-right-now\">no please I don't know how I got into this state I really hate git right now<a class=\"zola-anchor\" href=\"#no-please-i-don-t-know-how-i-got-into-this-state-i-really-hate-git-right-now\" aria-label=\"Anchor link for: no-please-i-don-t-know-how-i-got-into-this-state-i-really-hate-git-right-now\"><\/a>\n<\/h5>\n<p>Figure out which commit has the work you want to save. Then run <code>git checkout &lt;that commit&gt;<\/code> and then follow the steps about branches immediately above.<\/p>\n<h2 id=\"git-tells-me-something-about-would-overwrite-untracked-files-idk-what-that-means-i-just-want-to-pull-the-latest-branch\">Git tells me something about \"would overwrite untracked files\"? idk what that means I just want to pull the latest branch<a class=\"zola-anchor\" href=\"#git-tells-me-something-about-would-overwrite-untracked-files-idk-what-that-means-i-just-want-to-pull-the-latest-branch\" aria-label=\"Anchor link for: git-tells-me-something-about-would-overwrite-untracked-files-idk-what-that-means-i-just-want-to-pull-the-latest-branch\"><\/a>\n<\/h2>\n<p>I'm going to assume the error looks something like this:<\/p>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">error: The following untracked working tree files would be overwritten by reset:\n<\/span><span class=\"z-text z-plain\">        src\/tools\/rls\/Cargo.toml\n<\/span><span class=\"z-text z-plain\">        src\/tools\/rls\/README.md\n<\/span><span class=\"z-text z-plain\">Please move or remove them before you reset.\n<\/span><span class=\"z-text z-plain\">Aborting\n<\/span><span class=\"z-text z-plain\">fatal: could not move back to 9ba169a73acfa9c9875b76eec09e9a91cc6246df\n<\/span><\/code><\/pre>\n<p>There are three things I can think of that could be going wrong here:<\/p>\n<ol>\n<li>\n<p>You made some changes and didn't commit them. Commit them now. Go. Do it. <code>git add src\/tools\/rls &amp;&amp; git commit<\/code>.<\/p>\n<\/li>\n<li>\n<p>This is in a submodule and git got confused. If you don't know what a submodule is, mentally replace it with \"code I have never modified and will never want to modify\". You can ignore this by running <code>git submodule deinit -f src\/tools\/rls<\/code> (or whichever directory the submodule starts in - you can check with <code>git submodule status<\/code>).<\/p>\n<p><strong>DO NOT DO THIS<\/strong> if you have made changes to that directory you want to save. Git will destroy all you hold dear. It is a sharp and powerful tool. You might call it ... a subtle knife.<\/p>\n<\/li>\n<li>\n<p>This used to be a submodule and it's no longer a submodule. You tried the <code>deinit<\/code> thing above and it didn't help. Try <code>rm -rf src\/tools\/rls<\/code>. This does exactly what it looks like, don't do it if you want to save your work.<\/p>\n<\/li>\n<\/ol>\n<p>If none of those fix it, glhf sucks to be you<\/p>\n<h2 id=\"i-m-in-rust-lang-rust-and-cargo-lock-keeps-showing-modifications-but-i-definitely-didn-t-touch-anything\">I'm in rust-lang\/rust and Cargo.lock keeps showing modifications but I definitely didn't touch anything ???<a class=\"zola-anchor\" href=\"#i-m-in-rust-lang-rust-and-cargo-lock-keeps-showing-modifications-but-i-definitely-didn-t-touch-anything\" aria-label=\"Anchor link for: i-m-in-rust-lang-rust-and-cargo-lock-keeps-showing-modifications-but-i-definitely-didn-t-touch-anything\"><\/a>\n<\/h2>\n<p>yes this is a very specific problem. no I'm not going to take it out of the blog.<\/p>\n<p>This happens because your submodules are out of date. Run <code>.\/x.py --help<\/code>, which updates submodules for you.\nUnder the hood this is running the equivalent of <code>git submodule update --init --recursive<\/code> which makes sure all the rust tool submodules are up to date.<\/p>\n<h3 id=\"that-didn-t-help\">that didn't help.<a class=\"zola-anchor\" href=\"#that-didn-t-help\" aria-label=\"Anchor link for: that-didn-t-help\"><\/a>\n<\/h3>\n<p>sucks to be you<\/p>\n<h2 id=\"i-made-a-commit-and-then-i-realized-i-want-to-make-more-changes-but-also-people-getting-annoyed-when-i-add-lots-of-commits-what-do-i-do\">I made a commit and then I realized I want to make more changes but also people getting annoyed when I add lots of commits what do I do<a class=\"zola-anchor\" href=\"#i-made-a-commit-and-then-i-realized-i-want-to-make-more-changes-but-also-people-getting-annoyed-when-i-add-lots-of-commits-what-do-i-do\" aria-label=\"Anchor link for: i-made-a-commit-and-then-i-realized-i-want-to-make-more-changes-but-also-people-getting-annoyed-when-i-add-lots-of-commits-what-do-i-do\"><\/a>\n<\/h2>\n<p>First make your changes. Then run <code>git add .<\/code><\/p>\n<p>If you want to change the latest commit you made, run <code>git commit --amend<\/code>.<\/p>\n<p>If you want to change an earlier commit, run <code>git commit --fixup &lt;earlier commit&gt; &amp;&amp; git rebase -i --autosquash &lt;earlier commit&gt;~<\/code>. Type the <code>~<\/code> literally but remove the <code>&lt;&gt;<\/code>.<\/p>\n<p>If you don't know which commit you want to change, but you know it's a commit you made since you made this branch, install\n<a href=\"https:\/\/github.com\/tummychow\/git-absorb\/\">https:\/\/github.com\/tummychow\/git-absorb\/<\/a> then run the <code>autosquash<\/code> command from above. Absorb does magic to pick the right commits.<\/p>\n<p>If you want to change a commit that's already on master, don't. Just don't.\nIf you're <em>really<\/em>, <em>truly<\/em> convinced you need to, and you're <em>sure<\/em> no one else minds (which in practice is probably never true), you can use <a href=\"https:\/\/github.com\/newren\/git-filter-repo\">https:\/\/github.com\/newren\/git-filter-repo<\/a>.<\/p>\n<h2 id=\"i-made-a-merge-commit-but-now-someone-is-telling-me-merge-commits-aren-t-allowed-how-do-i-fix-this-please-i-already-spent-3-hours-on-this-i-don-t-want-to-spend-3-more\">I made a merge commit but now someone is telling me merge commits aren't allowed how do I fix this?? please I already spent 3 hours on this I don't want to spend 3 more<a class=\"zola-anchor\" href=\"#i-made-a-merge-commit-but-now-someone-is-telling-me-merge-commits-aren-t-allowed-how-do-i-fix-this-please-i-already-spent-3-hours-on-this-i-don-t-want-to-spend-3-more\" aria-label=\"Anchor link for: i-made-a-merge-commit-but-now-someone-is-telling-me-merge-commits-aren-t-allowed-how-do-i-fix-this-please-i-already-spent-3-hours-on-this-i-don-t-want-to-spend-3-more\"><\/a>\n<\/h2>\n<p>Run <code>git rebase -i $(git rev-parse --abbrev-ref origin\/HEAD)<\/code> (or whatever your default branch is). Then go up to the bit about \"failed to push some refs\" above.<\/p>\n<p>Yes this will probably cause conflicts. No I don't know how to avoid that, probably some nonsense with <code>git reset $(git merge-base $(git rev-parse --abbrev-ref origin\/HEAD)) &amp;&amp; git add . &amp;&amp; git commit<\/code> or whatever. Don't blame me if that doesn't work, I didn't test it. Also it throws away all the history, you keep the work but not the commit messages, you have to fish them up again with <code>git reflog<\/code> or <code>git for-each-ref --sort=committerdate refs\/heads\/ | tail -n1 | cut -d' ' -f1 | xargs git log<\/code>.<\/p>\n<h2 id=\"aaa\">AAA<a class=\"zola-anchor\" href=\"#aaa\" aria-label=\"Anchor link for: aaa\"><\/a>\n<\/h2>\n<p>yeah no that's it have fun kid<\/p>\n<p>ok no actually one more thing \u2014<\/p>\n<p>can I talk for a second about how absolutely <em>awful<\/em> every error message I've quoted here is. they are so bad I don't even refer to them when explaining what went wrong.\nhalf the time they point you to completely the wrong solution; the other half they give no indication at all of why something went wrong.<\/p>\n<p>this is not how it should be. things can be better. here is an error message for the \"push after rebase\" that is actually useful:<\/p>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">$ git push\n<\/span><span class=\"z-text z-plain\">To git@github.com:jyn514\/rust.git (simplify-storage -&gt; origin\/simplify-storage)\n<\/span><span class=\"z-text z-plain\">error: failed to push 1 commit to &#39;origin&#39;\n<\/span><span class=\"z-text z-plain\">hint: Both your current branch and the remote branch have changed since the last time you pushed changes.\n<\/span><span class=\"z-text z-plain\">hint: The commits locally and in your remote have the same descriptions, and you just finished a rebase.\n<\/span><span class=\"z-text z-plain\">hint: If you want to overwrite the commits on the remote, use &#39;git push --force-with-lease&#39; to push.\n<\/span><span class=\"z-text z-plain\">note: See https:\/\/git-scm.com\/book\/en\/v2\/Git-Branching-Rebasing for more information about rebasing.\n<\/span><\/code><\/pre>\n<p>See how that's helpful? see how you understand more after reading it than before? unlike the absolutely useless error before<\/p>\n<p>god ok that's it for real I'm done now<\/p>\n<h3 id=\"update-now-that-this-is-apparently-popular\">update now that this is apparently popular<a class=\"zola-anchor\" href=\"#update-now-that-this-is-apparently-popular\" aria-label=\"Anchor link for: update-now-that-this-is-apparently-popular\"><\/a>\n<\/h3>\n<p>go read https:\/\/rustc-dev-guide.rust-lang.org\/git.html too, it's the most comprehensive document on rebase workflows that I've seen. no the official docs don't count, if you don't explain the common errors you run into it's not comprehensive. also the manpages are useless unless you already know what the command does in which case WHAT WAS THE POINT OF THE MAN PAGE<\/p>\n<p>also go read my twitter where you can read more nonsense like me complaining about how I can't convince myself to go to sleep <a href=\"https:\/\/twitter.com\/jynelson514\/\">https:\/\/twitter.com\/jynelson514\/<\/a><\/p>\n"},{"title":"bootstrapping: the once and future compiler","published":"2022-08-05T00:00:00+00:00","updated":"2022-08-05T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/talks\/bootstrapping\/"}},"id":"https:\/\/jyn.dev\/talks\/bootstrapping\/","content":{"@attributes":{"type":"html"}}},{"title":"async() => { await }","published":"2022-02-09T00:00:00+00:00","updated":"2022-02-09T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/talks\/async-await\/"}},"id":"https:\/\/jyn.dev\/talks\/async-await\/","content":{"@attributes":{"type":"html"}}},{"title":"I'll Rust With You: the song","published":"2021-09-10T00:00:00+00:00","updated":"2021-09-10T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/i-ll-rust-with-you-the-song\/"}},"id":"https:\/\/jyn.dev\/i-ll-rust-with-you-the-song\/","content":"<h1 id=\"i-ll-rust-with-you-the-song\">I'll Rust With You: the song<a class=\"zola-anchor\" href=\"#i-ll-rust-with-you-the-song\" aria-label=\"Anchor link for: i-ll-rust-with-you-the-song\"><\/a>\n<\/h1>\n<p>Niko Matsakis, co-lead for the Rust language team, recently published a song titled <a href=\"https:\/\/smallcultfollowing.com\/babysteps\/blog\/2021\/05\/26\/edition-the-song\/\">\"Edition: the\nsong\"<\/a>. I decided I was not to be outdone and sang a cover of \"I'll Rust With You\", by\n<a href=\"https:\/\/steampoweredgiraffe.com\/\">Steam Powered Giraffe<\/a>. The song has absolutely no relation to the Rust language other than\nthe title, I just think it's fun.<\/p>\n<h3 id=\"video\">Video<a class=\"zola-anchor\" href=\"#video\" aria-label=\"Anchor link for: video\"><\/a>\n<\/h3>\n<iframe width=\"560\" height=\"315\" src=\"https:\/\/www.youtube.com\/embed\/chqv-MY1PGQ\" title=\"YouTube video player\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen=\"\"><\/iframe>\n"},{"title":"borrow-checker FAQs","published":"2021-03-20T00:00:00+00:00","updated":"2021-03-20T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/borrow-checker-faqs\/"}},"id":"https:\/\/jyn.dev\/borrow-checker-faqs\/","summary":"tips and tricks for dealing with the borrow-checker","content":"<p>I got lots of positive feedback about <a href=\"https:\/\/jyn514.github.io\/2020\/09\/05\/Rust-in-2021.html#faq\">the FAQ\nsection<\/a> in my Rust 2020 blog post, so\nI'm trying that format again for another topic that's been requested a lot: How to fix common\nborrow-checker issues. This isn't meant to explain how or why the borrow checker works the way it\ndoes (see <a href=\"https:\/\/doc.rust-lang.org\/nomicon\/lifetimes.html\">The Nomicon<\/a> or <a href=\"https:\/\/matklad.github.io\/2020\/07\/15\/two-beautiful-programs.html\">Two Beautiful Rust\nPrograms<\/a> for that), just how\nto work around some of its current limitations.<\/p>\n<p>In a small break from the format, the 'questions' will instead be Rust code and the accompanying\ncompiler error. I make no pretense that the code is doing useful work, it's just meant to give\nexamples of errors.<\/p>\n<h1 id=\"faq\">FAQ<a class=\"zola-anchor\" href=\"#faq\" aria-label=\"Anchor link for: faq\"><\/a>\n<\/h1>\n<blockquote>\n<p>How do I modify elements in a collection while also modifying the collection?<\/p>\n<\/blockquote>\n<p><a href=\"https:\/\/play.rust-lang.org\/?version=stable&amp;mode=debug&amp;edition=2018&amp;gist=90cf38acb25060d159d46933ba3a0608\">Playground<\/a><\/p>\n<pre data-lang=\"rust\" class=\"language-rust z-code\"><code class=\"language-rust\" data-lang=\"rust\"><span class=\"z-source z-rust\"><span class=\"z-storage z-type z-rust\">let<\/span> <span class=\"z-storage z-modifier z-rust\">mut<\/span> queue <span class=\"z-keyword z-operator z-assignment z-rust\">=<\/span> <span class=\"z-support z-macro z-rust\">vec!<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">[<\/span><span class=\"z-constant z-numeric z-integer z-decimal z-rust\">1<\/span><span class=\"z-punctuation z-separator z-rust\">,<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-rust\">2<\/span><span class=\"z-punctuation z-separator z-rust\">,<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-rust\">3<\/span><span class=\"z-punctuation z-section z-group z-end z-rust\">]<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><span class=\"z-source z-rust\"><span class=\"z-keyword z-control z-rust\">for<\/span> elem <span class=\"z-keyword z-operator z-rust\">in<\/span> queue<span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">iter_mut<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span> <span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-block z-rust\">    <span class=\"z-keyword z-operator z-arithmetic z-rust\">*<\/span>elem <span class=\"z-keyword z-operator z-assignment z-rust\">+=<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-rust\">2<\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-block z-rust\">    queue<span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">push<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-keyword z-operator z-arithmetic z-rust\">*<\/span>elem <span class=\"z-keyword z-operator z-arithmetic z-rust\">+<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-rust\">1<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-block z-rust\"><\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span>\n<\/span><\/code><\/pre>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">error[E0499]: cannot borrow `queue` as mutable more than once at a time\n<\/span><span class=\"z-text z-plain\"> --&gt; src\/main.rs:5:9\n<\/span><span class=\"z-text z-plain\">  |\n<\/span><span class=\"z-text z-plain\">3 |     for elem in queue.iter_mut() {\n<\/span><span class=\"z-text z-plain\">  |                 ----------------\n<\/span><span class=\"z-text z-plain\">  |                 |\n<\/span><span class=\"z-text z-plain\">  |                 first mutable borrow occurs here\n<\/span><span class=\"z-text z-plain\">  |                 first borrow later used here\n<\/span><span class=\"z-text z-plain\">4 |         *elem += 2;\n<\/span><span class=\"z-text z-plain\">5 |         queue.push(*elem + 1);\n<\/span><span class=\"z-text z-plain\">  |         ^^^^^ second mutable borrow occurs here\n<\/span><\/code><\/pre>\n<p>This happens because for-loops borrow the iterator for the <em>whole<\/em> loop, not just for a single\niteration. You can work around it by using <code>while let<\/code> instead:<\/p>\n<pre data-lang=\"rust\" class=\"language-rust z-code\"><code class=\"language-rust\" data-lang=\"rust\"><span class=\"z-source z-rust\"><span class=\"z-storage z-type z-rust\">let<\/span> <span class=\"z-storage z-modifier z-rust\">mut<\/span> queue <span class=\"z-keyword z-operator z-assignment z-rust\">=<\/span> <span class=\"z-support z-macro z-rust\">vec!<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">[<\/span><span class=\"z-constant z-numeric z-integer z-decimal z-rust\">1<\/span><span class=\"z-punctuation z-separator z-rust\">,<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-rust\">2<\/span><span class=\"z-punctuation z-separator z-rust\">,<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-rust\">3<\/span><span class=\"z-punctuation z-section z-group z-end z-rust\">]<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><span class=\"z-source z-rust\"><span class=\"z-keyword z-control z-rust\">while<\/span> <span class=\"z-storage z-type z-rust\">let<\/span> <span class=\"z-support z-type z-rust\">Some<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-storage z-modifier z-rust\">mut<\/span> elem<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span> <span class=\"z-keyword z-operator z-assignment z-rust\">=<\/span> queue<span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">pop<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span> <span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-block z-rust\">    elem <span class=\"z-keyword z-operator z-assignment z-rust\">+=<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-rust\">2<\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-block z-rust\">    queue<span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">push<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span>elem <span class=\"z-keyword z-operator z-arithmetic z-rust\">+<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-rust\">1<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-block z-rust\"><\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span>\n<\/span><\/code><\/pre>\n<p>Note that this has a behavior change: unlike the previous loop, this removes the element from the\nqueue before modifying it. You can get the behavior from before back by adding <code>queue.insert(0, elem)<\/code> (although at that point you may want to use a\n<a href=\"https:\/\/doc.rust-lang.org\/std\/collections\/struct.VecDeque.html\"><code>VecDeque<\/code><\/a>).<\/p>\n<blockquote>\n<p>I have disjoint fields in a struct, but because I use one in a closure, I can't use the other.<\/p>\n<\/blockquote>\n<pre data-lang=\"rust\" class=\"language-rust z-code\"><code class=\"language-rust\" data-lang=\"rust\"><span class=\"z-source z-rust\"><span class=\"z-meta z-struct z-rust\"><span class=\"z-storage z-type z-struct z-rust\">struct<\/span> <\/span><span class=\"z-meta z-struct z-rust\"><span class=\"z-entity z-name z-struct z-rust\">S<\/span> <\/span><span class=\"z-meta z-struct z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-struct z-rust\"><span class=\"z-meta z-block z-rust\">    <span class=\"z-variable z-other z-member z-rust\">elems<\/span><span class=\"z-punctuation z-separator z-type z-rust\">:<\/span> <span class=\"z-meta z-generic z-rust\"><span class=\"z-support z-type z-rust\">Vec<\/span><span class=\"z-punctuation z-definition z-generic z-begin z-rust\">&lt;<\/span><span class=\"z-storage z-type z-rust\">i32<\/span><span class=\"z-punctuation z-definition z-generic z-end z-rust\">&gt;<\/span><\/span>,\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-struct z-rust\"><span class=\"z-meta z-block z-rust\">    <span class=\"z-variable z-other z-member z-rust\">metadata<\/span><span class=\"z-punctuation z-separator z-type z-rust\">:<\/span> <span class=\"z-storage z-type z-rust\">i32<\/span>,\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-struct z-rust\"><span class=\"z-meta z-block z-rust\"><\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-rust\">\n<\/span><span class=\"z-source z-rust\"><span class=\"z-storage z-type z-rust\">let<\/span> <span class=\"z-storage z-modifier z-rust\">mut<\/span> s <span class=\"z-keyword z-operator z-assignment z-rust\">=<\/span> S <span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-block z-rust\">    elems<span class=\"z-punctuation z-separator z-rust\">:<\/span> <span class=\"z-support z-macro z-rust\">vec!<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">[<\/span><span class=\"z-constant z-numeric z-integer z-decimal z-rust\">1<\/span><span class=\"z-punctuation z-separator z-rust\">,<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-rust\">2<\/span><span class=\"z-punctuation z-separator z-rust\">,<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-rust\">3<\/span><span class=\"z-punctuation z-section z-group z-end z-rust\">]<\/span><\/span><span class=\"z-punctuation z-separator z-rust\">,<\/span>\n<\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-block z-rust\">    metadata<span class=\"z-punctuation z-separator z-rust\">:<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-rust\">0<\/span><span class=\"z-punctuation z-separator z-rust\">,<\/span>\n<\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-block z-rust\"><\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-path z-rust\">std<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span><span class=\"z-meta z-path z-rust\">thread<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span>spawn<span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-begin z-rust\">|<\/span><\/span><\/span><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-end z-rust\">|<\/span><\/span> <\/span><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-group z-rust\"><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-block z-rust\">    <span class=\"z-keyword z-control z-rust\">for<\/span> x <span class=\"z-keyword z-operator z-rust\">in<\/span> s<span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span>elems <span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-group z-rust\"><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-block z-rust\">        <span class=\"z-support z-macro z-rust\">println!<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-string z-quoted z-double z-rust\"><span class=\"z-punctuation z-definition z-string z-begin z-rust\">&quot;<\/span><span class=\"z-constant z-other z-placeholder z-rust\">{}<\/span><span class=\"z-punctuation z-definition z-string z-end z-rust\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-separator z-rust\">,<\/span> x<span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-group z-rust\"><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-block z-rust\">    <\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span>\n<\/span><\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-group z-rust\"><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-block z-rust\"><\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span><\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><span class=\"z-source z-rust\">\n<\/span><span class=\"z-source z-rust\"><span class=\"z-support z-macro z-rust\">println!<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-string z-quoted z-double z-rust\"><span class=\"z-punctuation z-definition z-string z-begin z-rust\">&quot;<\/span><span class=\"z-constant z-other z-placeholder z-rust\">{}<\/span><span class=\"z-punctuation z-definition z-string z-end z-rust\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-separator z-rust\">,<\/span> s<span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span>metadata<span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><\/code><\/pre>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">error[E0382]: borrow of moved value: `s`\n<\/span><span class=\"z-text z-plain\">  --&gt; src\/main.rs:18:16\n<\/span><span class=\"z-text z-plain\">   |\n<\/span><span class=\"z-text z-plain\">8  | let s = S {\n<\/span><span class=\"z-text z-plain\">   |     - move occurs because `s` has type `S`, which does not implement the `Copy` trait\n<\/span><span class=\"z-text z-plain\">...\n<\/span><span class=\"z-text z-plain\">12 | std::thread::spawn(|| {\n<\/span><span class=\"z-text z-plain\">   |                    -- value moved into closure here\n<\/span><span class=\"z-text z-plain\">13 |     for x in s.elems {\n<\/span><span class=\"z-text z-plain\">   |              ------- variable moved due to use in closure\n<\/span><span class=\"z-text z-plain\">...\n<\/span><span class=\"z-text z-plain\">18 | println!(&quot;{}&quot;, s.metadata);\n<\/span><span class=\"z-text z-plain\">   |                ^^^^^^^^^^ value borrowed here after move\n<\/span><\/code><\/pre>\n<p>This happens because the closure captures the whole <code>s<\/code> struct, not just the fields it needs.\nYou can explicitly say which fields to capture by adding a <code>let<\/code> statement:<\/p>\n<pre data-lang=\"rust\" class=\"language-rust z-code\"><code class=\"language-rust\" data-lang=\"rust\"><span class=\"z-source z-rust\"><span class=\"z-storage z-type z-rust\">let<\/span> elems <span class=\"z-keyword z-operator z-assignment z-rust\">=<\/span> s<span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span>elems<span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-path z-rust\">std<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span><span class=\"z-meta z-path z-rust\">thread<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span>spawn<span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-begin z-rust\">|<\/span><\/span><\/span><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-end z-rust\">|<\/span><\/span> <\/span><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-group z-rust\"><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-block z-rust\">    <span class=\"z-keyword z-control z-rust\">for<\/span> x <span class=\"z-keyword z-operator z-rust\">in<\/span> elems <span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-group z-rust\"><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-block z-rust\">        <span class=\"z-support z-macro z-rust\">println!<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-string z-quoted z-double z-rust\"><span class=\"z-punctuation z-definition z-string z-begin z-rust\">&quot;<\/span><span class=\"z-constant z-other z-placeholder z-rust\">{}<\/span><span class=\"z-punctuation z-definition z-string z-end z-rust\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-separator z-rust\">,<\/span> x<span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-group z-rust\"><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-block z-rust\">    <\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span>\n<\/span><\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-group z-rust\"><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-block z-rust\"><\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span><\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><\/code><\/pre>\n<p>This is done automatically for you with <code>#![feature(capture_disjoint_fields)]<\/code>, which will\nhopefully be <a href=\"https:\/\/github.com\/rust-lang\/project-rfc-2229\/milestones\">enabled by default<\/a> in the 2021 edition.<\/p>\n<p>Similar issues happen for structs that aren't <code>Send<\/code> or <code>Sync<\/code>:<\/p>\n<pre data-lang=\"rust\" class=\"language-rust z-code\"><code class=\"language-rust\" data-lang=\"rust\"><span class=\"z-source z-rust\"><span class=\"z-keyword z-other z-rust\">use<\/span> <span class=\"z-meta z-path z-rust\">std<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span><span class=\"z-meta z-path z-rust\">cell<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span>RefCell<span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-struct z-rust\"><span class=\"z-storage z-type z-struct z-rust\">struct<\/span> <\/span><span class=\"z-meta z-struct z-rust\"><span class=\"z-entity z-name z-struct z-rust\">S<\/span> <\/span><span class=\"z-meta z-struct z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-struct z-rust\"><span class=\"z-meta z-block z-rust\">    <span class=\"z-variable z-other z-member z-rust\">elems<\/span><span class=\"z-punctuation z-separator z-type z-rust\">:<\/span> <span class=\"z-meta z-generic z-rust\"><span class=\"z-support z-type z-rust\">Vec<\/span><span class=\"z-punctuation z-definition z-generic z-begin z-rust\">&lt;<\/span><span class=\"z-storage z-type z-rust\">i32<\/span><span class=\"z-punctuation z-definition z-generic z-end z-rust\">&gt;<\/span><\/span>,\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-struct z-rust\"><span class=\"z-meta z-block z-rust\">    <span class=\"z-variable z-other z-member z-rust\">metadata<\/span><span class=\"z-punctuation z-separator z-type z-rust\">:<\/span> <span class=\"z-meta z-generic z-rust\">RefCell<span class=\"z-punctuation z-definition z-generic z-begin z-rust\">&lt;<\/span><span class=\"z-storage z-type z-rust\">i32<\/span><span class=\"z-punctuation z-definition z-generic z-end z-rust\">&gt;<\/span><\/span>,\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-struct z-rust\"><span class=\"z-meta z-block z-rust\"><\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-rust\">\n<\/span><span class=\"z-source z-rust\"><span class=\"z-storage z-type z-rust\">let<\/span> s <span class=\"z-keyword z-operator z-assignment z-rust\">=<\/span> S <span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-block z-rust\">    elems<span class=\"z-punctuation z-separator z-rust\">:<\/span> <span class=\"z-support z-macro z-rust\">vec!<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">[<\/span><span class=\"z-constant z-numeric z-integer z-decimal z-rust\">1<\/span><span class=\"z-punctuation z-separator z-rust\">,<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-rust\">2<\/span><span class=\"z-punctuation z-separator z-rust\">,<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-rust\">3<\/span><span class=\"z-punctuation z-section z-group z-end z-rust\">]<\/span><\/span><span class=\"z-punctuation z-separator z-rust\">,<\/span>\n<\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-block z-rust\">    metadata<span class=\"z-punctuation z-separator z-rust\">:<\/span> <span class=\"z-meta z-path z-rust\">RefCell<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span>new<span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-constant z-numeric z-integer z-decimal z-rust\">0<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-separator z-rust\">,<\/span>\n<\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-block z-rust\"><\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-path z-rust\">std<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span><span class=\"z-meta z-path z-rust\">thread<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span>spawn<span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-begin z-rust\">|<\/span><\/span><\/span><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-end z-rust\">|<\/span><\/span> <\/span><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-group z-rust\"><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-block z-rust\">    <span class=\"z-keyword z-control z-rust\">for<\/span> x <span class=\"z-keyword z-operator z-rust\">in<\/span> <span class=\"z-keyword z-operator z-bitwise z-rust\">&amp;<\/span>s<span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span>elems <span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-group z-rust\"><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-block z-rust\">        <span class=\"z-support z-macro z-rust\">println!<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-string z-quoted z-double z-rust\"><span class=\"z-punctuation z-definition z-string z-begin z-rust\">&quot;<\/span><span class=\"z-constant z-other z-placeholder z-rust\">{}<\/span><span class=\"z-punctuation z-definition z-string z-end z-rust\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-separator z-rust\">,<\/span> x<span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-group z-rust\"><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-block z-rust\">    <\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span>\n<\/span><\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-group z-rust\"><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-block z-rust\"><\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span><\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><span class=\"z-source z-rust\">\n<\/span><span class=\"z-source z-rust\"><span class=\"z-support z-macro z-rust\">println!<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-string z-quoted z-double z-rust\"><span class=\"z-punctuation z-definition z-string z-begin z-rust\">&quot;<\/span><span class=\"z-constant z-other z-placeholder z-rust\">{}<\/span><span class=\"z-punctuation z-definition z-string z-end z-rust\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-separator z-rust\">,<\/span> s<span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span>metadata<span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">borrow<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><\/code><\/pre>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">error[E0277]: `RefCell&lt;i32&gt;` cannot be shared between threads safely\n<\/span><span class=\"z-text z-plain\">   --&gt; src\/main.rs:16:1\n<\/span><span class=\"z-text z-plain\">    |\n<\/span><span class=\"z-text z-plain\">16  | std::thread::spawn(|| {\n<\/span><span class=\"z-text z-plain\">    | ^^^^^^^^^^^^^^^^^^ `RefCell&lt;i32&gt;` cannot be shared between threads safely\n<\/span><span class=\"z-text z-plain\">    |\n<\/span><span class=\"z-text z-plain\">    = help: within `S`, the trait `Sync` is not implemented for `RefCell&lt;i32&gt;`\n<\/span><span class=\"z-text z-plain\">    = note: required because it appears within the type `S`\n<\/span><span class=\"z-text z-plain\">    = note: required because of the requirements on the impl of `Send` for `&amp;S`\n<\/span><span class=\"z-text z-plain\">    = note: required because it appears within the type `[closure@src\/main.rs:16:20: 20:2]`\n<\/span><span class=\"z-text z-plain\">\n<\/span><span class=\"z-text z-plain\">error: aborting due to previous error\n<\/span><\/code><\/pre>\n<p>If you only need a few of the fields, you can add explicit <code>let<\/code> statements in the same way.<\/p>\n<pre data-lang=\"rust\" class=\"language-rust z-code\"><code class=\"language-rust\" data-lang=\"rust\"><span class=\"z-source z-rust\"><span class=\"z-storage z-type z-rust\">let<\/span> elems <span class=\"z-keyword z-operator z-assignment z-rust\">=<\/span> s<span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span>elems<span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-path z-rust\">std<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span><span class=\"z-meta z-path z-rust\">thread<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span>spawn<span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-begin z-rust\">|<\/span><\/span><\/span><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-end z-rust\">|<\/span><\/span> <\/span><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-group z-rust\"><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-block z-rust\">    <span class=\"z-keyword z-control z-rust\">for<\/span> x <span class=\"z-keyword z-operator z-rust\">in<\/span> elems <span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-group z-rust\"><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-block z-rust\">        <span class=\"z-support z-macro z-rust\">println!<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-string z-quoted z-double z-rust\"><span class=\"z-punctuation z-definition z-string z-begin z-rust\">&quot;<\/span><span class=\"z-constant z-other z-placeholder z-rust\">{}<\/span><span class=\"z-punctuation z-definition z-string z-end z-rust\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-separator z-rust\">,<\/span> x<span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-group z-rust\"><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-block z-rust\">    <\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span>\n<\/span><\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-group z-rust\"><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-block z-rust\"><\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span><\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><span class=\"z-source z-rust\"><span class=\"z-support z-macro z-rust\">println!<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-string z-quoted z-double z-rust\"><span class=\"z-punctuation z-definition z-string z-begin z-rust\">&quot;<\/span><span class=\"z-constant z-other z-placeholder z-rust\">{}<\/span><span class=\"z-punctuation z-definition z-string z-end z-rust\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-separator z-rust\">,<\/span> s<span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span>metadata<span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">borrow<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><\/code><\/pre>\n<blockquote>\n<p>Moving a line of code into a separate function makes it fail to compile.<\/p>\n<\/blockquote>\n<p>There are actually quite a few things that will cause this. The most common are:<\/p>\n<ol>\n<li>The compiler can no longer tell you're using disjoint fields.<\/li>\n<li>You wrote the wrong lifetimes (or the elided lifetimes are wrong).<\/li>\n<\/ol>\n<p>An example of 1 (<a href=\"https:\/\/play.rust-lang.org\/?version=stable&amp;mode=debug&amp;edition=2018&amp;gist=2f84148a276b0c8595a8023363f2f6a7\">Playground<\/a>):<\/p>\n<pre data-lang=\"rust\" class=\"language-rust z-code\"><code class=\"language-rust\" data-lang=\"rust\"><span class=\"z-source z-rust\"><span class=\"z-meta z-struct z-rust\"><span class=\"z-storage z-type z-struct z-rust\">struct<\/span> <\/span><span class=\"z-meta z-struct z-rust\"><span class=\"z-entity z-name z-struct z-rust\">S<\/span> <\/span><span class=\"z-meta z-struct z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-struct z-rust\"><span class=\"z-meta z-block z-rust\">    <span class=\"z-variable z-other z-member z-rust\">elems<\/span><span class=\"z-punctuation z-separator z-type z-rust\">:<\/span> <span class=\"z-meta z-generic z-rust\"><span class=\"z-support z-type z-rust\">Vec<\/span><span class=\"z-punctuation z-definition z-generic z-begin z-rust\">&lt;<\/span><span class=\"z-storage z-type z-rust\">i32<\/span><span class=\"z-punctuation z-definition z-generic z-end z-rust\">&gt;<\/span><\/span>,\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-struct z-rust\"><span class=\"z-meta z-block z-rust\">    <span class=\"z-variable z-other z-member z-rust\">metadata<\/span><span class=\"z-punctuation z-separator z-type z-rust\">:<\/span> <span class=\"z-storage z-type z-rust\">i32<\/span>,\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-struct z-rust\"><span class=\"z-meta z-block z-rust\"><\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-rust\">\n<\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-impl z-rust\"><span class=\"z-storage z-type z-impl z-rust\">impl<\/span> <\/span><span class=\"z-meta z-impl z-rust\"><span class=\"z-entity z-name z-impl z-rust\">S<\/span> <\/span><span class=\"z-meta z-impl z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-impl z-rust\"><span class=\"z-meta z-block z-rust\">    <span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-storage z-type z-function z-rust\">fn<\/span> <\/span><span class=\"z-entity z-name z-function z-rust\">set_metadata<\/span><\/span><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-begin z-rust\">(<\/span><span class=\"z-keyword z-operator z-rust\">&amp;<\/span><span class=\"z-storage z-modifier z-rust\">mut<\/span> <span class=\"z-variable z-parameter z-rust\">self<\/span>, <span class=\"z-variable z-parameter z-rust\">val<\/span><span class=\"z-punctuation z-separator z-rust\">:<\/span> <span class=\"z-storage z-type z-rust\">i32<\/span><\/span><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-end z-rust\">)<\/span><\/span><\/span><\/span><span class=\"z-meta z-function z-rust\"> <\/span><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-impl z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\">        <span class=\"z-variable z-language z-rust\">self<\/span><span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span>metadata <span class=\"z-keyword z-operator z-assignment z-rust\">=<\/span> val<span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-impl z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\">    <\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span><\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-impl z-rust\"><span class=\"z-meta z-block z-rust\"><\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-rust\">\n<\/span><span class=\"z-source z-rust\"><span class=\"z-storage z-type z-rust\">let<\/span> <span class=\"z-storage z-modifier z-rust\">mut<\/span> s <span class=\"z-keyword z-operator z-assignment z-rust\">=<\/span> S <span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-block z-rust\">    elems<span class=\"z-punctuation z-separator z-rust\">:<\/span> <span class=\"z-support z-macro z-rust\">vec!<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">[<\/span><span class=\"z-constant z-numeric z-integer z-decimal z-rust\">1<\/span><span class=\"z-punctuation z-separator z-rust\">,<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-rust\">2<\/span><span class=\"z-punctuation z-separator z-rust\">,<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-rust\">3<\/span><span class=\"z-punctuation z-section z-group z-end z-rust\">]<\/span><\/span><span class=\"z-punctuation z-separator z-rust\">,<\/span>\n<\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-block z-rust\">    metadata<span class=\"z-punctuation z-separator z-rust\">:<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-rust\">0<\/span><span class=\"z-punctuation z-separator z-rust\">,<\/span>\n<\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-block z-rust\"><\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><span class=\"z-source z-rust\"><span class=\"z-storage z-type z-rust\">let<\/span> _elems <span class=\"z-keyword z-operator z-assignment z-rust\">=<\/span> s<span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span>elems<span class=\"z-punctuation z-terminator z-rust\">;<\/span> <span class=\"z-comment z-line z-double-slash z-rust\"><span class=\"z-punctuation z-definition z-comment z-rust\">\/\/<\/span> moves out of `elems`\n<\/span><\/span><span class=\"z-source z-rust\">s<span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span>metadata <span class=\"z-keyword z-operator z-assignment z-rust\">=<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-rust\">1<\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span> <span class=\"z-comment z-line z-double-slash z-rust\"><span class=\"z-punctuation z-definition z-comment z-rust\">\/\/<\/span> works fine\n<\/span><\/span><span class=\"z-source z-rust\">s<span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">set_metadata<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-constant z-numeric z-integer z-decimal z-rust\">1<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span> <span class=\"z-comment z-line z-double-slash z-rust\"><span class=\"z-punctuation z-definition z-comment z-rust\">\/\/<\/span> breaks because `elems` is moved\n<\/span><\/span><\/code><\/pre>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">error[E0382]: borrow of partially moved value: `s`\n<\/span><span class=\"z-text z-plain\">  --&gt; src\/main.rs:22:1\n<\/span><span class=\"z-text z-plain\">   |\n<\/span><span class=\"z-text z-plain\">20 | let elems = s.elems; \/\/ moves out of `elems`\n<\/span><span class=\"z-text z-plain\">   |             ------- value partially moved here\n<\/span><span class=\"z-text z-plain\">21 | s.metadata = 1; \/\/ works fine\n<\/span><span class=\"z-text z-plain\">22 | s.set_metadata(1); \/\/ breaks because `elems` is moved\n<\/span><span class=\"z-text z-plain\">   | ^ value borrowed here after partial move\n<\/span><span class=\"z-text z-plain\">   |\n<\/span><span class=\"z-text z-plain\">   = note: partial move occurs because `s.elems` has type `Vec&lt;i32&gt;`, which does not implement the `Copy` trait\n<\/span><\/code><\/pre>\n<p>There is no simple fix for this; this is the main reason getters and setters are discouraged in\nRust. You can either manually inline the code, or change your struct so that the relevant parts\nuse composition instead of all being in the same struct:<\/p>\n<pre data-lang=\"rust\" class=\"language-rust z-code\"><code class=\"language-rust\" data-lang=\"rust\"><span class=\"z-source z-rust\"><span class=\"z-meta z-struct z-rust\"><span class=\"z-storage z-type z-struct z-rust\">struct<\/span> <\/span><span class=\"z-meta z-struct z-rust\"><span class=\"z-entity z-name z-struct z-rust\">S<\/span> <\/span><span class=\"z-meta z-struct z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-struct z-rust\"><span class=\"z-meta z-block z-rust\">    <span class=\"z-variable z-other z-member z-rust\">elems<\/span><span class=\"z-punctuation z-separator z-type z-rust\">:<\/span> <span class=\"z-meta z-generic z-rust\"><span class=\"z-support z-type z-rust\">Vec<\/span><span class=\"z-punctuation z-definition z-generic z-begin z-rust\">&lt;<\/span><span class=\"z-storage z-type z-rust\">i32<\/span><span class=\"z-punctuation z-definition z-generic z-end z-rust\">&gt;<\/span><\/span>,\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-struct z-rust\"><span class=\"z-meta z-block z-rust\">    <span class=\"z-variable z-other z-member z-rust\">metadata<\/span><span class=\"z-punctuation z-separator z-type z-rust\">:<\/span> Inner,\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-struct z-rust\"><span class=\"z-meta z-block z-rust\"><\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-struct z-rust\"><span class=\"z-storage z-type z-struct z-rust\">struct<\/span> <\/span><span class=\"z-meta z-struct z-rust\"><span class=\"z-entity z-name z-struct z-rust\">Inner<\/span><\/span><span class=\"z-meta z-struct z-rust\"><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-storage z-type z-rust\">i32<\/span><\/span><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><span class=\"z-source z-rust\">\n<\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-impl z-rust\"><span class=\"z-storage z-type z-impl z-rust\">impl<\/span> <\/span><span class=\"z-meta z-impl z-rust\"><span class=\"z-entity z-name z-impl z-rust\">Inner<\/span> <\/span><span class=\"z-meta z-impl z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-impl z-rust\"><span class=\"z-meta z-block z-rust\">    <span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-storage z-type z-function z-rust\">fn<\/span> <\/span><span class=\"z-entity z-name z-function z-rust\">set_metadata<\/span><\/span><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-begin z-rust\">(<\/span><span class=\"z-keyword z-operator z-rust\">&amp;<\/span><span class=\"z-storage z-modifier z-rust\">mut<\/span> <span class=\"z-variable z-parameter z-rust\">self<\/span>, <span class=\"z-variable z-parameter z-rust\">val<\/span><span class=\"z-punctuation z-separator z-rust\">:<\/span> <span class=\"z-storage z-type z-rust\">i32<\/span><\/span><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-end z-rust\">)<\/span><\/span><\/span><\/span><span class=\"z-meta z-function z-rust\"> <\/span><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-impl z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\">        <span class=\"z-keyword z-operator z-arithmetic z-rust\">*<\/span><span class=\"z-variable z-language z-rust\">self<\/span> <span class=\"z-keyword z-operator z-assignment z-rust\">=<\/span> Inner<span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span>val<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-impl z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\">    <\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span><\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-impl z-rust\"><span class=\"z-meta z-block z-rust\"><\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-rust\">\n<\/span><span class=\"z-source z-rust\"><span class=\"z-storage z-type z-rust\">let<\/span> <span class=\"z-storage z-modifier z-rust\">mut<\/span> s <span class=\"z-keyword z-operator z-assignment z-rust\">=<\/span> S <span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-block z-rust\">    elems<span class=\"z-punctuation z-separator z-rust\">:<\/span> <span class=\"z-support z-macro z-rust\">vec!<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">[<\/span><span class=\"z-constant z-numeric z-integer z-decimal z-rust\">1<\/span><span class=\"z-punctuation z-separator z-rust\">,<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-rust\">2<\/span><span class=\"z-punctuation z-separator z-rust\">,<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-rust\">3<\/span><span class=\"z-punctuation z-section z-group z-end z-rust\">]<\/span><\/span><span class=\"z-punctuation z-separator z-rust\">,<\/span>\n<\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-block z-rust\">    metadata<span class=\"z-punctuation z-separator z-rust\">:<\/span> Inner<span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-constant z-numeric z-integer z-decimal z-rust\">0<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-separator z-rust\">,<\/span>\n<\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-block z-rust\"><\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><span class=\"z-source z-rust\"><span class=\"z-storage z-type z-rust\">let<\/span> _elems <span class=\"z-keyword z-operator z-assignment z-rust\">=<\/span> s<span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span>elems<span class=\"z-punctuation z-terminator z-rust\">;<\/span> <span class=\"z-comment z-line z-double-slash z-rust\"><span class=\"z-punctuation z-definition z-comment z-rust\">\/\/<\/span> moves out of `elems`\n<\/span><\/span><span class=\"z-source z-rust\">s<span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span>metadata <span class=\"z-keyword z-operator z-assignment z-rust\">=<\/span> Inner<span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-constant z-numeric z-integer z-decimal z-rust\">1<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span> <span class=\"z-comment z-line z-double-slash z-rust\"><span class=\"z-punctuation z-definition z-comment z-rust\">\/\/<\/span> works fine\n<\/span><\/span><span class=\"z-source z-rust\">s<span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span>metadata<span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">set_metadata<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-constant z-numeric z-integer z-decimal z-rust\">1<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span> <span class=\"z-comment z-line z-double-slash z-rust\"><span class=\"z-punctuation z-definition z-comment z-rust\">\/\/<\/span> works fine\n<\/span><\/span><\/code><\/pre>\n<p>An example of 2 (<a href=\"https:\/\/play.rust-lang.org\/?version=stable&amp;mode=debug&amp;edition=2018&amp;gist=99a4597a50b4c7915325ef1caa287040\">Playground<\/a>):<\/p>\n<pre data-lang=\"rust\" class=\"language-rust z-code\"><code class=\"language-rust\" data-lang=\"rust\"><span class=\"z-source z-rust\"><span class=\"z-meta z-struct z-rust\"><span class=\"z-storage z-type z-struct z-rust\">struct<\/span> <\/span><span class=\"z-meta z-struct z-rust\"><span class=\"z-meta z-generic z-rust\"><span class=\"z-entity z-name z-struct z-rust\">S<\/span><span class=\"z-meta z-generic z-rust\"><span class=\"z-punctuation z-definition z-generic z-begin z-rust\">&lt;<\/span><span class=\"z-storage z-modifier z-lifetime z-rust\">&#39;a<\/span><span class=\"z-punctuation z-definition z-generic z-end z-rust\">&gt;<\/span><\/span><\/span><\/span><span class=\"z-meta z-struct z-rust\"><\/span><span class=\"z-meta z-struct z-rust\"><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-keyword z-operator z-rust\">&amp;<\/span><span class=\"z-storage z-modifier z-lifetime z-rust\">&#39;a<\/span> <span class=\"z-storage z-type z-rust\">i32<\/span><\/span><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><span class=\"z-source z-rust\">\n<\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-impl z-rust\"><span class=\"z-storage z-type z-impl z-rust\">impl<\/span> <\/span><span class=\"z-meta z-impl z-rust\"><span class=\"z-entity z-name z-impl z-rust\">S<\/span><span class=\"z-meta z-generic z-rust\"><span class=\"z-punctuation z-definition z-generic z-begin z-rust\">&lt;<\/span>&#39;<span class=\"z-keyword z-operator z-rust\">_<\/span><span class=\"z-punctuation z-definition z-generic z-end z-rust\">&gt;<\/span><\/span> <\/span><span class=\"z-meta z-impl z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-impl z-rust\"><span class=\"z-meta z-block z-rust\">    <span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-storage z-type z-function z-rust\">fn<\/span> <\/span><span class=\"z-entity z-name z-function z-rust\">inner<\/span><\/span><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-begin z-rust\">(<\/span><span class=\"z-keyword z-operator z-rust\">&amp;<\/span><span class=\"z-variable z-parameter z-rust\">self<\/span><\/span><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-end z-rust\">)<\/span><\/span><\/span><\/span><span class=\"z-meta z-function z-rust\"> <span class=\"z-meta z-function z-return-type z-rust\"><span class=\"z-punctuation z-separator z-rust\">-&gt;<\/span> <span class=\"z-keyword z-operator z-rust\">&amp;<\/span><span class=\"z-storage z-type z-rust\">i32<\/span><\/span> <\/span><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-impl z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\">        <span class=\"z-variable z-language z-rust\">self<\/span><span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-constant z-numeric z-integer z-decimal z-rust\">0<\/span>\n<\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-impl z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\">    <\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span><\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-impl z-rust\"><span class=\"z-meta z-block z-rust\"><\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-rust\">\n<\/span><span class=\"z-source z-rust\"><span class=\"z-storage z-type z-rust\">let<\/span> x <span class=\"z-keyword z-operator z-assignment z-rust\">=<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-rust\">0<\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><span class=\"z-source z-rust\"><span class=\"z-storage z-type z-rust\">let<\/span> _p <span class=\"z-keyword z-operator z-assignment z-rust\">=<\/span> <span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-block z-rust\">    <span class=\"z-storage z-type z-rust\">let<\/span> s <span class=\"z-keyword z-operator z-assignment z-rust\">=<\/span> S<span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-keyword z-operator z-bitwise z-rust\">&amp;<\/span>x<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-block z-rust\">    s<span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-constant z-numeric z-integer z-decimal z-rust\">0<\/span>\n<\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-block z-rust\"><\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><span class=\"z-source z-rust\">\n<\/span><span class=\"z-source z-rust\"><span class=\"z-storage z-type z-rust\">let<\/span> _q <span class=\"z-keyword z-operator z-assignment z-rust\">=<\/span> <span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-block z-rust\">    <span class=\"z-storage z-type z-rust\">let<\/span> s <span class=\"z-keyword z-operator z-assignment z-rust\">=<\/span> S<span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-keyword z-operator z-bitwise z-rust\">&amp;<\/span>x<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-block z-rust\">    s<span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">inner<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span> <span class=\"z-comment z-line z-double-slash z-rust\"><span class=\"z-punctuation z-definition z-comment z-rust\">\/\/<\/span> error: does not live long enough\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-block z-rust\"><\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><\/code><\/pre>\n<p>This one is tricky to spot: the issue isn't the code you wrote, but rather the code you <em>didn't<\/em> write.\nIf you desugar <code>inner()<\/code>, it would be something like<\/p>\n<pre data-lang=\"rust\" class=\"language-rust z-code\"><code class=\"language-rust\" data-lang=\"rust\"><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-storage z-type z-function z-rust\">fn<\/span> <\/span><span class=\"z-entity z-name z-function z-rust\">inner<\/span><\/span><span class=\"z-meta z-generic z-rust\"><span class=\"z-punctuation z-definition z-generic z-begin z-rust\">&lt;<\/span><span class=\"z-storage z-modifier z-lifetime z-rust\">&#39;s<\/span><span class=\"z-punctuation z-definition z-generic z-end z-rust\">&gt;<\/span><\/span><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-begin z-rust\">(<\/span><span class=\"z-keyword z-operator z-rust\">&amp;<\/span><span class=\"z-storage z-modifier z-lifetime z-rust\">&#39;s<\/span> <span class=\"z-variable z-parameter z-rust\">self<\/span><\/span><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-end z-rust\">)<\/span><\/span><\/span><\/span><span class=\"z-meta z-function z-rust\"> <span class=\"z-meta z-function z-return-type z-rust\"><span class=\"z-punctuation z-separator z-rust\">-&gt;<\/span> <span class=\"z-keyword z-operator z-rust\">&amp;<\/span><span class=\"z-storage z-modifier z-lifetime z-rust\">&#39;s<\/span> <span class=\"z-storage z-type z-rust\">i32<\/span><\/span> <\/span><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\">    <span class=\"z-variable z-language z-rust\">self<\/span><span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-constant z-numeric z-integer z-decimal z-rust\">0<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\"><\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span><\/span>\n<\/span><\/code><\/pre>\n<p>But this is unnecessarily restrictive - <code>self.0<\/code> lives longer than <code>self<\/code>. The fix is to write the lifetime yourself:<\/p>\n<pre data-lang=\"rust\" class=\"language-rust z-code\"><code class=\"language-rust\" data-lang=\"rust\"><span class=\"z-source z-rust\"><span class=\"z-meta z-impl z-rust\"><span class=\"z-storage z-type z-impl z-rust\">impl<\/span><\/span><span class=\"z-meta z-impl z-rust\"><span class=\"z-meta z-generic z-rust\"><span class=\"z-punctuation z-definition z-generic z-begin z-rust\">&lt;<\/span><span class=\"z-storage z-modifier z-lifetime z-rust\">&#39;a<\/span><span class=\"z-punctuation z-definition z-generic z-end z-rust\">&gt;<\/span><\/span><\/span><span class=\"z-meta z-impl z-rust\"> <span class=\"z-entity z-name z-impl z-rust\">S<\/span><span class=\"z-meta z-generic z-rust\"><span class=\"z-punctuation z-definition z-generic z-begin z-rust\">&lt;<\/span><span class=\"z-storage z-modifier z-lifetime z-rust\">&#39;a<\/span><span class=\"z-punctuation z-definition z-generic z-end z-rust\">&gt;<\/span><\/span> <\/span><span class=\"z-meta z-impl z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-impl z-rust\"><span class=\"z-meta z-block z-rust\">    <span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-storage z-type z-function z-rust\">fn<\/span> <\/span><span class=\"z-entity z-name z-function z-rust\">inner<\/span><\/span><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-begin z-rust\">(<\/span><span class=\"z-keyword z-operator z-rust\">&amp;<\/span><span class=\"z-variable z-parameter z-rust\">self<\/span><\/span><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-end z-rust\">)<\/span><\/span><\/span><\/span><span class=\"z-meta z-function z-rust\"> <span class=\"z-meta z-function z-return-type z-rust\"><span class=\"z-punctuation z-separator z-rust\">-&gt;<\/span> <span class=\"z-keyword z-operator z-rust\">&amp;<\/span><span class=\"z-storage z-modifier z-lifetime z-rust\">&#39;a<\/span> <span class=\"z-storage z-type z-rust\">i32<\/span><\/span> <\/span><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-impl z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\">        <span class=\"z-variable z-language z-rust\">self<\/span><span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-constant z-numeric z-integer z-decimal z-rust\">0<\/span>\n<\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-impl z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\">    <\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span><\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-impl z-rust\"><span class=\"z-meta z-block z-rust\"><\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span><\/span>\n<\/span><\/code><\/pre>\n<p>There are a lot more variants of this. I might publish a follow-up post with more.<\/p>\n<blockquote>\n<p>How can I use an iterator from a struct when I also need mutable access? I know that my changes won't affect the iterator.<\/p>\n<\/blockquote>\n<p><a href=\"https:\/\/play.rust-lang.org\/?version=stable&amp;mode=debug&amp;edition=2018&amp;gist=d5324bfe8d3429c6f40fe09bd52c7d04\">Playground<\/a><\/p>\n<pre data-lang=\"rust\" class=\"language-rust z-code\"><code class=\"language-rust\" data-lang=\"rust\"><span class=\"z-source z-rust\"><span class=\"z-comment z-line z-double-slash z-rust\"><span class=\"z-punctuation z-definition z-comment z-rust\">\/\/<\/span> same `s` from earlier example\n<\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-struct z-rust\"><span class=\"z-storage z-type z-struct z-rust\">struct<\/span> <\/span><span class=\"z-meta z-struct z-rust\"><span class=\"z-entity z-name z-struct z-rust\">S<\/span> <\/span><span class=\"z-meta z-struct z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-struct z-rust\"><span class=\"z-meta z-block z-rust\">    <span class=\"z-variable z-other z-member z-rust\">elems<\/span><span class=\"z-punctuation z-separator z-type z-rust\">:<\/span> <span class=\"z-meta z-generic z-rust\"><span class=\"z-support z-type z-rust\">Vec<\/span><span class=\"z-punctuation z-definition z-generic z-begin z-rust\">&lt;<\/span><span class=\"z-storage z-type z-rust\">i32<\/span><span class=\"z-punctuation z-definition z-generic z-end z-rust\">&gt;<\/span><\/span>,\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-struct z-rust\"><span class=\"z-meta z-block z-rust\">    <span class=\"z-variable z-other z-member z-rust\">metadata<\/span><span class=\"z-punctuation z-separator z-type z-rust\">:<\/span> <span class=\"z-storage z-type z-rust\">i32<\/span>,\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-struct z-rust\"><span class=\"z-meta z-block z-rust\"><\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-rust\">\n<\/span><span class=\"z-source z-rust\"><span class=\"z-storage z-type z-rust\">let<\/span> <span class=\"z-storage z-modifier z-rust\">mut<\/span> s <span class=\"z-keyword z-operator z-assignment z-rust\">=<\/span> S <span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-block z-rust\">    elems<span class=\"z-punctuation z-separator z-rust\">:<\/span> <span class=\"z-support z-macro z-rust\">vec!<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">[<\/span><span class=\"z-constant z-numeric z-integer z-decimal z-rust\">1<\/span><span class=\"z-punctuation z-separator z-rust\">,<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-rust\">2<\/span><span class=\"z-punctuation z-separator z-rust\">,<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-rust\">3<\/span><span class=\"z-punctuation z-section z-group z-end z-rust\">]<\/span><\/span><span class=\"z-punctuation z-separator z-rust\">,<\/span>\n<\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-block z-rust\">    metadata<span class=\"z-punctuation z-separator z-rust\">:<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-rust\">0<\/span><span class=\"z-punctuation z-separator z-rust\">,<\/span>\n<\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-block z-rust\"><\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><span class=\"z-source z-rust\">\n<\/span><span class=\"z-source z-rust\"><span class=\"z-keyword z-control z-rust\">for<\/span> x <span class=\"z-keyword z-operator z-rust\">in<\/span> s<span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span>elems<span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">iter<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span> <span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-block z-rust\">    <span class=\"z-support z-function z-rust\">takes_s<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-keyword z-operator z-arithmetic z-rust\">*<\/span>x<span class=\"z-punctuation z-separator z-rust\">,<\/span> <span class=\"z-keyword z-operator z-bitwise z-rust\">&amp;<\/span><span class=\"z-storage z-modifier z-rust\">mut<\/span> s<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-block z-rust\"><\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span>\n<\/span><span class=\"z-source z-rust\">\n<\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-storage z-type z-function z-rust\">fn<\/span> <\/span><span class=\"z-entity z-name z-function z-rust\">takes_s<\/span><\/span><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-begin z-rust\">(<\/span><span class=\"z-variable z-parameter z-rust\">x<\/span><span class=\"z-punctuation z-separator z-rust\">:<\/span> <span class=\"z-storage z-type z-rust\">i32<\/span>, <span class=\"z-variable z-parameter z-rust\">s<\/span><span class=\"z-punctuation z-separator z-rust\">:<\/span> <span class=\"z-keyword z-operator z-rust\">&amp;<\/span><span class=\"z-storage z-modifier z-rust\">mut<\/span> S<\/span><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-end z-rust\">)<\/span><\/span><\/span><\/span><span class=\"z-meta z-function z-rust\"> <\/span><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\">    s<span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span>metadata <span class=\"z-keyword z-operator z-assignment z-rust\">=<\/span> x<span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\"><\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span><\/span>\n<\/span><\/code><\/pre>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">error[E0502]: cannot borrow `s` as mutable because it is also borrowed as immutable\n<\/span><span class=\"z-text z-plain\">  --&gt; src\/main.rs:13:17\n<\/span><span class=\"z-text z-plain\">   |\n<\/span><span class=\"z-text z-plain\">12 | for x in s.elems.iter() {\n<\/span><span class=\"z-text z-plain\">   |          --------------\n<\/span><span class=\"z-text z-plain\">   |          |\n<\/span><span class=\"z-text z-plain\">   |          immutable borrow occurs here\n<\/span><span class=\"z-text z-plain\">   |          immutable borrow later used here\n<\/span><span class=\"z-text z-plain\">13 |     takes_s(*x, &amp;mut s);\n<\/span><span class=\"z-text z-plain\">   |                 ^^^^^^ mutable borrow occurs here\n<\/span><\/code><\/pre>\n<p>You can <code>.collect<\/code> the iterator before using it:<\/p>\n<pre data-lang=\"rust\" class=\"language-rust z-code\"><code class=\"language-rust\" data-lang=\"rust\"><span class=\"z-source z-rust\"><span class=\"z-storage z-type z-rust\">let<\/span> iter<span class=\"z-punctuation z-separator z-rust\">:<\/span> <span class=\"z-meta z-generic z-rust\"><span class=\"z-support z-type z-rust\">Vec<\/span><span class=\"z-punctuation z-definition z-generic z-begin z-rust\">&lt;<\/span><span class=\"z-keyword z-operator z-rust\">_<\/span><span class=\"z-punctuation z-definition z-generic z-end z-rust\">&gt;<\/span><\/span> <span class=\"z-keyword z-operator z-assignment z-rust\">=<\/span> s<span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span>elems<span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">iter<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">copied<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">collect<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><span class=\"z-source z-rust\"><span class=\"z-keyword z-control z-rust\">for<\/span> x <span class=\"z-keyword z-operator z-rust\">in<\/span> iter <span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-block z-rust\">    <span class=\"z-support z-function z-rust\">takes_s<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span>x<span class=\"z-punctuation z-separator z-rust\">,<\/span> <span class=\"z-keyword z-operator z-bitwise z-rust\">&amp;<\/span><span class=\"z-storage z-modifier z-rust\">mut<\/span> s<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-block z-rust\"><\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span>\n<\/span><\/code><\/pre>\n<p>Note that this isn't ideal in several ways:<\/p>\n<ul>\n<li>If you're wrong that <code>takes_s<\/code> doesn't affect the iterator, it changes the behavior.<\/li>\n<li>It makes the iterator eager instead of lazy, which means you have to allocate a new collection\n(you don't strictly have to use <code>Vec<\/code>, but there's no reason to use anything else).<\/li>\n<li>It requires either copying or cloning the elements.<\/li>\n<\/ul>\n<p>However, in some cases, there's no alternative. See <a href=\"https:\/\/github.com\/rust-lang\/rust\/pull\/82020#discussion_r575905338\">this Rustdoc\nPR<\/a> for a real-world example.<\/p>\n"},{"title":{"pre":{"code":"&'borrow mut dyn FnMut(BrokenLink<'input>) -> CowStr<'input>"}},"published":"2020-12-08T00:00:00+00:00","updated":"2020-12-08T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/borrow-mut-dyn-fnmut-brokenlink-cowstring-and-other-valid-rust-programs\/"}},"id":"https:\/\/jyn.dev\/borrow-mut-dyn-fnmut-brokenlink-cowstring-and-other-valid-rust-programs\/","summary":"Why HRTB are my least-favorite part of Rust","content":"<p>This is a story about type signatures, Higher Ranked Trait Bounds (<a href=\"https:\/\/doc.rust-lang.org\/nomicon\/hrtb.html\">HRTB<\/a>), and the most\nconfusing diagnostics bug I've seen in the Rust compiler. Along the way we'll learn how\n<a href=\"https:\/\/github.com\/raphlinus\/pulldown-cmark\"><code>pulldown-cmark<\/code><\/a> has been trying to fix the same API for 3 different releases, and discover\nthat some bugs only appear at the compile time of downstream crates.<\/p>\n<p>If you don't know what some of the words mean, there is <a href=\"https:\/\/jyn.dev\/borrow-mut-dyn-fnmut-brokenlink-cowstring-and-other-valid-rust-programs\/#appendix\">an appendix<\/a>. Unfortunately,\nthis article needs too much background knowledge to fit all of it in a blog post.<\/p>\n<h2 id=\"background\">Background<a class=\"zola-anchor\" href=\"#background\" aria-label=\"Anchor link for: background\"><\/a>\n<\/h2>\n<p>Our story starts with a simple change to rustdoc: It wants to <a href=\"https:\/\/github.com\/rust-lang\/rust\/pull\/79781\">hide the <code>[]<\/code><\/a> around <a href=\"https:\/\/doc.rust-lang.org\/rustdoc\/linking-to-items-by-name.html\">intra-doc links<\/a> in search results.\n(For context, search results have links stripped since otherwise the relative links would be broken, which means rustdoc needs to do some preprocessing of the markdown.)\nThat would turn screenshots like this:<\/p>\n<p><img src=\"https:\/\/user-images.githubusercontent.com\/37223377\/101309556-92908680-3801-11eb-8420-0609e7af4e92.png\" alt=\"Before\" \/><\/p>\n<p>into this:<\/p>\n<p><img src=\"https:\/\/user-images.githubusercontent.com\/37223377\/101309591-a0dea280-3801-11eb-85e1-620549f64bf6.png\" alt=\"After\" \/><\/p>\n<p>Rustdoc does this in several <a href=\"https:\/\/github.com\/rust-lang\/rust\/blob\/d32c80467db39672fa612e1519564ad5fd473e91\/src\/librustdoc\/html\/markdown.rs#L1066\">different<\/a> <a href=\"https:\/\/github.com\/rust-lang\/rust\/blob\/d32c80467db39672fa612e1519564ad5fd473e91\/src\/librustdoc\/html\/markdown.rs#L1151\">places<\/a>, so\nnaturally this should be pulled out into a single function that can be reused. This is a callback\nto pulldown's markdown <a href=\"https:\/\/docs.rs\/pulldown-cmark\/0.8.0\/pulldown_cmark\/struct.Parser.html\"><code>Parser<\/code><\/a>.<\/p>\n<pre data-lang=\"rust\" class=\"language-rust z-code\"><code class=\"language-rust\" data-lang=\"rust\"><span class=\"z-source z-rust\"><span class=\"z-comment z-line z-double-slash z-rust\"><span class=\"z-punctuation z-definition z-comment z-rust\">\/\/<\/span> Omitted: in real life, this would only mark intra-doc links as valid, not all broken links.\n<\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-storage z-type z-function z-rust\">fn<\/span> <\/span><span class=\"z-entity z-name z-function z-rust\">summary_broken_link_callback<\/span><\/span><span class=\"z-meta z-generic z-rust\"><span class=\"z-punctuation z-definition z-generic z-begin z-rust\">&lt;<\/span><span class=\"z-storage z-modifier z-lifetime z-rust\">&#39;a<\/span><span class=\"z-punctuation z-definition z-generic z-end z-rust\">&gt;<\/span><\/span><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-begin z-rust\">(<\/span><span class=\"z-variable z-parameter z-rust\">link<\/span><span class=\"z-punctuation z-separator z-rust\">:<\/span> <span class=\"z-meta z-generic z-rust\">BrokenLink<span class=\"z-punctuation z-definition z-generic z-begin z-rust\">&lt;<\/span><span class=\"z-storage z-modifier z-lifetime z-rust\">&#39;a<\/span><span class=\"z-punctuation z-definition z-generic z-end z-rust\">&gt;<\/span><\/span><\/span><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-end z-rust\">)<\/span><\/span><\/span><\/span><span class=\"z-meta z-function z-rust\"> <span class=\"z-meta z-function z-return-type z-rust\"><span class=\"z-punctuation z-separator z-rust\">-&gt;<\/span> <span class=\"z-meta z-generic z-rust\"><span class=\"z-support z-type z-rust\">Option<\/span><span class=\"z-punctuation z-definition z-generic z-begin z-rust\">&lt;<\/span><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-meta z-generic z-rust\">CowStr<span class=\"z-punctuation z-definition z-generic z-begin z-rust\">&lt;<\/span><span class=\"z-storage z-modifier z-lifetime z-rust\">&#39;a<\/span><span class=\"z-punctuation z-definition z-generic z-end z-rust\">&gt;<\/span><\/span>, <span class=\"z-meta z-generic z-rust\">CowStr<span class=\"z-punctuation z-definition z-generic z-begin z-rust\">&lt;<\/span><span class=\"z-storage z-modifier z-lifetime z-rust\">&#39;a<\/span><span class=\"z-punctuation z-definition z-generic z-end z-rust\">&gt;<\/span><\/span><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><span class=\"z-punctuation z-definition z-generic z-end z-rust\">&gt;<\/span><\/span><\/span> <\/span><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\">    <span class=\"z-support z-type z-rust\">Some<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-string z-quoted z-double z-rust\"><span class=\"z-punctuation z-definition z-string z-begin z-rust\">&quot;<\/span>#<span class=\"z-punctuation z-definition z-string z-end z-rust\">&quot;<\/span><\/span><span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">into<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-separator z-rust\">,<\/span> link<span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span>reference<span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">to_owned<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">into<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\"><\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span><\/span>\n<\/span><\/code><\/pre>\n<p>Except that doesn't work:<\/p>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">error[E0621]: explicit lifetime required in the type of `md`\n<\/span><span class=\"z-text z-plain\">    --&gt; src\/librustdoc\/html\/markdown.rs:1065:9\n<\/span><span class=\"z-text z-plain\">     |\n<\/span><span class=\"z-text z-plain\">1050 | fn markdown_summary_with_limit(md: &amp;str, length_limit: usize) -&gt; (String, bool) {\n<\/span><span class=\"z-text z-plain\">     |                                    ---- help: add explicit lifetime `&#39;static` to the type of `md`: `&amp;&#39;static str`\n<\/span><span class=\"z-text z-plain\">...\n<\/span><span class=\"z-text z-plain\">1065 |         Parser::new_with_broken_link_callback(md, summary_opts(), Some(&amp;mut summary_broken_link_callback))\n<\/span><span class=\"z-text z-plain\">     |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ lifetime `&#39;static` required\n<\/span><span class=\"z-text z-plain\">\n<\/span><span class=\"z-text z-plain\">error: higher-ranked subtype error\n<\/span><span class=\"z-text z-plain\">    --&gt; src\/librustdoc\/html\/markdown.rs:1065:72\n<\/span><span class=\"z-text z-plain\">     |\n<\/span><span class=\"z-text z-plain\">1065 |         Parser::new_with_broken_link_callback(md, summary_opts(), Some(&amp;mut summary_broken_link_callback))\n<\/span><span class=\"z-text z-plain\">     |                                                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n<\/span><span class=\"z-text z-plain\">\n<\/span><span class=\"z-text z-plain\">error[E0716]: temporary value dropped while borrowed\n<\/span><span class=\"z-text z-plain\">    --&gt; src\/librustdoc\/html\/markdown.rs:1065:77\n<\/span><span class=\"z-text z-plain\">     |\n<\/span><span class=\"z-text z-plain\">1065 |         Parser::new_with_broken_link_callback(md, summary_opts(), Some(&amp;mut summary_broken_link_callback))\n<\/span><span class=\"z-text z-plain\">     |         --------------------------------------------------------------------^^^^^^^^^^^^^^^^^^^^^^^^^^^^--\n<\/span><span class=\"z-text z-plain\">     |         |                                                                   |                            |\n<\/span><span class=\"z-text z-plain\">     |         |                                                                   |                            temporary value is freed at the end of this statement\n<\/span><span class=\"z-text z-plain\">     |         |                                                                   creates a temporary which is freed while still in use\n<\/span><span class=\"z-text z-plain\">     |         argument requires that borrow lasts for `&#39;static`\n<\/span><\/code><\/pre>\n<h2 id=\"lifetime-annotations-action-at-a-distance\">Lifetime annotations: action at a distance<a class=\"zola-anchor\" href=\"#lifetime-annotations-action-at-a-distance\" aria-label=\"Anchor link for: lifetime-annotations-action-at-a-distance\"><\/a>\n<\/h2>\n<p>What on earth is going on here? We have a simple function that takes a link and returns it, right?\nWhat does that have to do with \"higher-ranked subtypes\" and <code>'static<\/code>? To find out, we have to look\nat the definition of <a href=\"https:\/\/docs.rs\/pulldown-cmark\/0.8.0\/pulldown_cmark\/struct.Parser.html#method.new_with_broken_link_callback\"><code>new_with_broken_link_callback()<\/code><\/a>.<\/p>\n<pre data-lang=\"rust\" class=\"language-rust z-code\"><code class=\"language-rust\" data-lang=\"rust\"><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-storage z-modifier z-rust\">pub<\/span> <span class=\"z-storage z-type z-function z-rust\">fn<\/span> <\/span><span class=\"z-entity z-name z-function z-rust\">new_with_broken_link_callback<\/span><\/span><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-begin z-rust\">(<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-parameters z-rust\">    <span class=\"z-variable z-parameter z-rust\">text<\/span><span class=\"z-punctuation z-separator z-rust\">:<\/span> <span class=\"z-keyword z-operator z-rust\">&amp;<\/span><span class=\"z-storage z-modifier z-lifetime z-rust\">&#39;a<\/span> <span class=\"z-storage z-type z-rust\">str<\/span>,\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-parameters z-rust\">    <span class=\"z-variable z-parameter z-rust\">options<\/span><span class=\"z-punctuation z-separator z-rust\">:<\/span> Options,\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-parameters z-rust\">    <span class=\"z-variable z-parameter z-rust\">broken_link_callback<\/span><span class=\"z-punctuation z-separator z-rust\">:<\/span> <span class=\"z-meta z-generic z-rust\"><span class=\"z-support z-type z-rust\">Option<\/span><span class=\"z-punctuation z-definition z-generic z-begin z-rust\">&lt;<\/span><span class=\"z-keyword z-operator z-rust\">&amp;<\/span><span class=\"z-storage z-modifier z-lifetime z-rust\">&#39;a<\/span> <span class=\"z-storage z-modifier z-rust\">mut<\/span> dyn <span class=\"z-support z-type z-rust\">FnMut<\/span><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-meta z-generic z-rust\">BrokenLink<span class=\"z-punctuation z-definition z-generic z-begin z-rust\">&lt;<\/span>&#39;<span class=\"z-keyword z-operator z-rust\">_<\/span><span class=\"z-punctuation z-definition z-generic z-end z-rust\">&gt;<\/span><\/span><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span> <span class=\"z-punctuation z-separator z-generic z-rust\">-&gt;<\/span> <span class=\"z-meta z-generic z-rust\"><span class=\"z-support z-type z-rust\">Option<\/span><span class=\"z-punctuation z-definition z-generic z-begin z-rust\">&lt;<\/span><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-meta z-generic z-rust\">CowStr<span class=\"z-punctuation z-definition z-generic z-begin z-rust\">&lt;<\/span><span class=\"z-storage z-modifier z-lifetime z-rust\">&#39;a<\/span><span class=\"z-punctuation z-definition z-generic z-end z-rust\">&gt;<\/span><\/span>, <span class=\"z-meta z-generic z-rust\">CowStr<span class=\"z-punctuation z-definition z-generic z-begin z-rust\">&lt;<\/span><span class=\"z-storage z-modifier z-lifetime z-rust\">&#39;a<\/span><span class=\"z-punctuation z-definition z-generic z-end z-rust\">&gt;<\/span><\/span><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><span class=\"z-punctuation z-definition z-generic z-end z-rust\">&gt;<\/span><\/span><span class=\"z-punctuation z-definition z-generic z-end z-rust\">&gt;<\/span><\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><\/span><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-end z-rust\">)<\/span><\/span><\/span><\/span><span class=\"z-meta z-function z-rust\"> <span class=\"z-meta z-function z-return-type z-rust\"><span class=\"z-punctuation z-separator z-rust\">-&gt;<\/span> <span class=\"z-meta z-generic z-rust\">Parser<span class=\"z-punctuation z-definition z-generic z-begin z-rust\">&lt;<\/span><span class=\"z-storage z-modifier z-lifetime z-rust\">&#39;a<\/span><span class=\"z-punctuation z-definition z-generic z-end z-rust\">&gt;<\/span><\/span><\/span> <\/span><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span> <span class=\"z-comment z-block z-rust\"><span class=\"z-punctuation z-definition z-comment z-rust\">\/*<\/span> ... <span class=\"z-punctuation z-definition z-comment z-rust\">*\/<\/span><\/span> <\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span><\/span>\n<\/span><\/code><\/pre>\n<p>Wow! There's a lot going on here, especially in <code>broken_link_callback<\/code>. Let's take that type apart a little:<\/p>\n<ul>\n<li><code>Option<\/code> means the callback is optional; if you pass in <code>None<\/code> it will just do nothing.<\/li>\n<li><code>&amp;'a mut<\/code> is a mutable reference.<\/li>\n<li><code>FnMut(BrokenLink&lt;'_&gt;) -&gt; Option&lt;(CowStr&lt;'a&gt;, CowStr&lt;'a&gt;)&gt;<\/code> is a trait for 'functions taking <code>BrokenLink<\/code> and returning a tuple of <code>CowStr<\/code>'.<\/li>\n<li>Because <code>FnMut<\/code> is a <em>trait<\/em> and not a type, <code>dyn<\/code> turns it into a type (i.e. this is <a href=\"https:\/\/stackoverflow.com\/questions\/28961957\/example-of-runtime-polymorphism-in-java\">runtime polymorphism<\/a>).<\/li>\n<\/ul>\n<p>But there's something funny here - the <em>same<\/em> lifetime <code>'a<\/code> is used for both the reference to the\nfunction, and the types it outputs. That's what made our function break: since\n<code>summary_broken_link_callback<\/code> is a function, any reference to it has a <code>'static<\/code> lifetime.\nFortunately, Rust will implicitly <a href=\"https:\/\/doc.rust-lang.org\/nomicon\/lifetimes.html#the-area-covered-by-a-lifetime\">reborrow<\/a> the reference if it lives too long. Unfortunately,\nwe don't know how long it <em>should<\/em> live. That's why the error is mentioning <code>'static<\/code>; since it can't figure out the right lifetime, it falls back to the longest possible.\nBy looking at the definition of <a href=\"https:\/\/docs.rs\/pulldown-cmark\/0.8.0\/src\/pulldown_cmark\/parse.rs.html#2051-2062\"><code>Parser<\/code><\/a>, we can see why that gives an error about <code>md<\/code>:<\/p>\n<pre data-lang=\"rust\" class=\"language-rust z-code\"><code class=\"language-rust\" data-lang=\"rust\"><span class=\"z-source z-rust\"><span class=\"z-meta z-struct z-rust\"><span class=\"z-storage z-modifier z-rust\">pub<\/span> <span class=\"z-storage z-type z-struct z-rust\">struct<\/span> <\/span><span class=\"z-meta z-struct z-rust\"><span class=\"z-meta z-generic z-rust\"><span class=\"z-entity z-name z-struct z-rust\">Parser<\/span><span class=\"z-meta z-generic z-rust\"><span class=\"z-punctuation z-definition z-generic z-begin z-rust\">&lt;<\/span><span class=\"z-storage z-modifier z-lifetime z-rust\">&#39;a<\/span><span class=\"z-punctuation z-definition z-generic z-end z-rust\">&gt;<\/span><\/span><\/span><\/span><span class=\"z-meta z-struct z-rust\"> <\/span><span class=\"z-meta z-struct z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-struct z-rust\"><span class=\"z-meta z-block z-rust\">    <span class=\"z-variable z-other z-member z-rust\">md<\/span><span class=\"z-punctuation z-separator z-type z-rust\">:<\/span> <span class=\"z-keyword z-operator z-rust\">&amp;<\/span><span class=\"z-storage z-modifier z-lifetime z-rust\">&#39;a<\/span> <span class=\"z-storage z-type z-rust\">str<\/span>,\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-struct z-rust\"><span class=\"z-meta z-block z-rust\">    <span class=\"z-variable z-other z-member z-rust\">broken_link_callback<\/span><span class=\"z-punctuation z-separator z-type z-rust\">:<\/span> <span class=\"z-meta z-generic z-rust\">BrokenLinkCallback<span class=\"z-punctuation z-definition z-generic z-begin z-rust\">&lt;<\/span><span class=\"z-storage z-modifier z-lifetime z-rust\">&#39;a<\/span><span class=\"z-punctuation z-definition z-generic z-end z-rust\">&gt;<\/span><\/span>,\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-struct z-rust\"><span class=\"z-meta z-block z-rust\">    <span class=\"z-comment z-line z-double-slash z-rust\"><span class=\"z-punctuation z-definition z-comment z-rust\">\/\/<\/span> ... some fields omitted\n<\/span><\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-struct z-rust\"><span class=\"z-meta z-block z-rust\"><\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-rust\">\n<\/span><span class=\"z-source z-rust\"><span class=\"z-storage z-modifier z-rust\">pub<\/span> <span class=\"z-storage z-type z-type z-rust\">type<\/span> <span class=\"z-entity z-name z-type z-rust\">BrokenLinkCallback<\/span><span class=\"z-keyword z-operator z-comparison z-rust\">&lt;<\/span><span class=\"z-storage z-modifier z-lifetime z-rust\">&#39;a<\/span><span class=\"z-keyword z-operator z-comparison z-rust\">&gt;<\/span> <span class=\"z-keyword z-operator z-assignment z-rust\">=<\/span>\n<\/span><span class=\"z-source z-rust\">    <span class=\"z-meta z-generic z-rust\"><span class=\"z-support z-type z-rust\">Option<\/span><span class=\"z-punctuation z-definition z-generic z-begin z-rust\">&lt;<\/span><span class=\"z-keyword z-operator z-rust\">&amp;<\/span><span class=\"z-storage z-modifier z-lifetime z-rust\">&#39;a<\/span> <span class=\"z-storage z-modifier z-rust\">mut<\/span> dyn <span class=\"z-support z-type z-rust\">FnMut<\/span><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span>BrokenLink<span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span> <span class=\"z-punctuation z-separator z-generic z-rust\">-&gt;<\/span> <span class=\"z-meta z-generic z-rust\"><span class=\"z-support z-type z-rust\">Option<\/span><span class=\"z-punctuation z-definition z-generic z-begin z-rust\">&lt;<\/span><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-meta z-generic z-rust\">CowStr<span class=\"z-punctuation z-definition z-generic z-begin z-rust\">&lt;<\/span><span class=\"z-storage z-modifier z-lifetime z-rust\">&#39;a<\/span><span class=\"z-punctuation z-definition z-generic z-end z-rust\">&gt;<\/span><\/span>, <span class=\"z-meta z-generic z-rust\">CowStr<span class=\"z-punctuation z-definition z-generic z-begin z-rust\">&lt;<\/span><span class=\"z-storage z-modifier z-lifetime z-rust\">&#39;a<\/span><span class=\"z-punctuation z-definition z-generic z-end z-rust\">&gt;<\/span><\/span><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><span class=\"z-punctuation z-definition z-generic z-end z-rust\">&gt;<\/span><\/span><span class=\"z-punctuation z-definition z-generic z-end z-rust\">&gt;<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><\/code><\/pre>\n<p>So there's <em>many<\/em> lifetimes that have been tied together here:<\/p>\n<ol>\n<li>The lifetime of the input, <code>md<\/code><\/li>\n<li>The lifetime of the borrow, <code>&amp;mut broken_link_callback<\/code><\/li>\n<li>The lifetime of the function <em>outputs<\/em>, which are unnamed here (pulldown <a href=\"https:\/\/docs.rs\/pulldown-cmark\/0.8.0\/src\/pulldown_cmark\/parse.rs.html#2380\">calls them<\/a> <code>url<\/code> and <code>title<\/code>).<\/li>\n<\/ol>\n<p>And that explains all the errors:<\/p>\n<ol>\n<li><code>md<\/code> is a temporary, but <code>&amp;mut broken_link_callback<\/code> is <code>'static<\/code><\/li>\n<li><code>&amp;mut broken_link_callback<\/code> can't find an appropriate lifetime for the borrow, so it gives up altogether with \"higher-ranked subtype error\"<\/li>\n<li>The arguments are temporaries, so the outputs are temporaries too (because we declared <code>broken_link_callback<\/code> as <code>fn summary_broken_link_callback&lt;'a&gt;(link: BrokenLink&lt;'a&gt;) -&gt; Option&lt;(CowStr&lt;'a&gt;, CowStr&lt;'a&gt;)&gt;<\/code>), but <code>&amp;mut broken_link_callback<\/code> is <code>'static<\/code><\/li>\n<\/ol>\n<h2 id=\"how-do-you-solve-a-problem-like-a-lifetime\"><a href=\"https:\/\/www.youtube.com\/watch?v=s-VRyQprlu8\">How do you solve a problem like a lifetime?<\/a><a class=\"zola-anchor\" href=\"#how-do-you-solve-a-problem-like-a-lifetime\" aria-label=\"Anchor link for: how-do-you-solve-a-problem-like-a-lifetime\"><\/a>\n<\/h2>\n<p>Notice that the bug here is <em>not<\/em> in <code>broken_link_callback<\/code> - it's in the API itself, which is\ntying the lifetimes together unnecessarily. If you write <code>broken_link_callback<\/code> on its own, it\nwill compile and run just fine, it's only the way that it interacts with <code>Parser<\/code> that breaks\nthings. The fix is not to tie the lifetimes together:<\/p>\n<pre data-lang=\"rust\" class=\"language-rust z-code\"><code class=\"language-rust\" data-lang=\"rust\"><span class=\"z-source z-rust\"><span class=\"z-meta z-struct z-rust\"><span class=\"z-storage z-modifier z-rust\">pub<\/span> <span class=\"z-storage z-type z-struct z-rust\">struct<\/span> <\/span><span class=\"z-meta z-struct z-rust\"><span class=\"z-meta z-generic z-rust\"><span class=\"z-entity z-name z-struct z-rust\">Parser<\/span><span class=\"z-meta z-generic z-rust\"><span class=\"z-punctuation z-definition z-generic z-begin z-rust\">&lt;<\/span><span class=\"z-storage z-modifier z-lifetime z-rust\">&#39;input<\/span>, <span class=\"z-storage z-modifier z-lifetime z-rust\">&#39;callback<\/span><span class=\"z-punctuation z-definition z-generic z-end z-rust\">&gt;<\/span><\/span><\/span><\/span><span class=\"z-meta z-struct z-rust\"> <\/span><span class=\"z-meta z-struct z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-struct z-rust\"><span class=\"z-meta z-block z-rust\">    <span class=\"z-variable z-other z-member z-rust\">text<\/span><span class=\"z-punctuation z-separator z-type z-rust\">:<\/span> <span class=\"z-keyword z-operator z-rust\">&amp;<\/span><span class=\"z-storage z-modifier z-lifetime z-rust\">&#39;input<\/span> <span class=\"z-storage z-type z-rust\">str<\/span>,\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-struct z-rust\"><span class=\"z-meta z-block z-rust\">    <span class=\"z-variable z-other z-member z-rust\">broken_link_callback<\/span><span class=\"z-punctuation z-separator z-type z-rust\">:<\/span> <span class=\"z-meta z-generic z-rust\">BrokenLinkCallback<span class=\"z-punctuation z-definition z-generic z-begin z-rust\">&lt;<\/span><span class=\"z-storage z-modifier z-lifetime z-rust\">&#39;input<\/span>, <span class=\"z-storage z-modifier z-lifetime z-rust\">&#39;callback<\/span><span class=\"z-punctuation z-definition z-generic z-end z-rust\">&gt;<\/span><\/span>,\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-struct z-rust\"><span class=\"z-meta z-block z-rust\"><\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-rust\">\n<\/span><span class=\"z-source z-rust\"><span class=\"z-storage z-modifier z-rust\">pub<\/span> <span class=\"z-storage z-type z-type z-rust\">type<\/span> <span class=\"z-entity z-name z-type z-rust\">BrokenLinkCallback<\/span><span class=\"z-keyword z-operator z-comparison z-rust\">&lt;<\/span><span class=\"z-storage z-modifier z-lifetime z-rust\">&#39;input<\/span><span class=\"z-punctuation z-separator z-rust\">,<\/span> <span class=\"z-storage z-modifier z-lifetime z-rust\">&#39;borrow<\/span><span class=\"z-keyword z-operator z-comparison z-rust\">&gt;<\/span> <span class=\"z-keyword z-operator z-assignment z-rust\">=<\/span>\n<\/span><span class=\"z-source z-rust\">    <span class=\"z-meta z-generic z-rust\"><span class=\"z-support z-type z-rust\">Option<\/span><span class=\"z-punctuation z-definition z-generic z-begin z-rust\">&lt;<\/span><span class=\"z-keyword z-operator z-rust\">&amp;<\/span><span class=\"z-storage z-modifier z-lifetime z-rust\">&#39;borrow<\/span> <span class=\"z-storage z-modifier z-rust\">mut<\/span> dyn <span class=\"z-support z-type z-rust\">FnMut<\/span><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-meta z-generic z-rust\">BrokenLink<span class=\"z-punctuation z-definition z-generic z-begin z-rust\">&lt;<\/span><span class=\"z-storage z-modifier z-lifetime z-rust\">&#39;input<\/span><span class=\"z-punctuation z-definition z-generic z-end z-rust\">&gt;<\/span><\/span><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span> <span class=\"z-punctuation z-separator z-generic z-rust\">-&gt;<\/span> <span class=\"z-meta z-generic z-rust\"><span class=\"z-support z-type z-rust\">Option<\/span><span class=\"z-punctuation z-definition z-generic z-begin z-rust\">&lt;<\/span><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-meta z-generic z-rust\">CowStr<span class=\"z-punctuation z-definition z-generic z-begin z-rust\">&lt;<\/span><span class=\"z-storage z-modifier z-lifetime z-rust\">&#39;input<\/span><span class=\"z-punctuation z-definition z-generic z-end z-rust\">&gt;<\/span><\/span>, <span class=\"z-meta z-generic z-rust\">CowStr<span class=\"z-punctuation z-definition z-generic z-begin z-rust\">&lt;<\/span><span class=\"z-storage z-modifier z-lifetime z-rust\">&#39;input<\/span><span class=\"z-punctuation z-definition z-generic z-end z-rust\">&gt;<\/span><\/span><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><span class=\"z-punctuation z-definition z-generic z-end z-rust\">&gt;<\/span><\/span><span class=\"z-punctuation z-definition z-generic z-end z-rust\">&gt;<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><\/code><\/pre>\n<p>Notice that 1. and 3. are still tied together; this is because pulldown has an API that yields\n<a href=\"https:\/\/docs.rs\/pulldown-cmark\/0.8.0\/pulldown_cmark\/struct.Parser.html#associatedtype.Item\">events with the lifetime of the input<\/a>. To use the output of the callback in the\n<code>Iterator<\/code> implementation, the outputs have to live as long as the full markdown input, not just\nthe current link.<\/p>\n<p>An interesting note is that <code>broken_link_callback<\/code> had a similar issue over overspecifying lifetimes:\nif you change the type signature to<\/p>\n<pre data-lang=\"rust\" class=\"language-rust z-code\"><code class=\"language-rust\" data-lang=\"rust\"><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-storage z-type z-function z-rust\">fn<\/span> <\/span><span class=\"z-entity z-name z-function z-rust\">summary_broken_link_callback<\/span><\/span><span class=\"z-meta z-generic z-rust\"><span class=\"z-punctuation z-definition z-generic z-begin z-rust\">&lt;<\/span><span class=\"z-storage z-modifier z-lifetime z-rust\">&#39;a<\/span><span class=\"z-punctuation z-definition z-generic z-end z-rust\">&gt;<\/span><\/span><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-begin z-rust\">(<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-parameters z-rust\">\t<span class=\"z-variable z-parameter z-rust\">link<\/span><span class=\"z-punctuation z-separator z-rust\">:<\/span> <span class=\"z-meta z-generic z-rust\">BrokenLink<span class=\"z-punctuation z-definition z-generic z-begin z-rust\">&lt;<\/span>&#39;<span class=\"z-keyword z-operator z-rust\">_<\/span><span class=\"z-punctuation z-definition z-generic z-end z-rust\">&gt;<\/span><\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><\/span><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-end z-rust\">)<\/span><\/span><\/span><\/span><span class=\"z-meta z-function z-rust\"> <span class=\"z-meta z-function z-return-type z-rust\"><span class=\"z-punctuation z-separator z-rust\">-&gt;<\/span> <span class=\"z-meta z-generic z-rust\"><span class=\"z-support z-type z-rust\">Option<\/span><span class=\"z-punctuation z-definition z-generic z-begin z-rust\">&lt;<\/span><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-meta z-generic z-rust\">CowStr<span class=\"z-punctuation z-definition z-generic z-begin z-rust\">&lt;<\/span><span class=\"z-storage z-modifier z-lifetime z-rust\">&#39;a<\/span><span class=\"z-punctuation z-definition z-generic z-end z-rust\">&gt;<\/span><\/span>, <span class=\"z-meta z-generic z-rust\">CowStr<span class=\"z-punctuation z-definition z-generic z-begin z-rust\">&lt;<\/span><span class=\"z-storage z-modifier z-lifetime z-rust\">&#39;a<\/span><span class=\"z-punctuation z-definition z-generic z-end z-rust\">&gt;<\/span><\/span><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><span class=\"z-punctuation z-definition z-generic z-end z-rust\">&gt;<\/span><\/span><\/span> <\/span><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\">  <span class=\"z-support z-type z-rust\">Some<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-string z-quoted z-double z-rust\"><span class=\"z-punctuation z-definition z-string z-begin z-rust\">&quot;<\/span>#<span class=\"z-punctuation z-definition z-string z-end z-rust\">&quot;<\/span><\/span><span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">into<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-separator z-rust\">,<\/span> link<span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span>reference<span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">to_owned<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">into<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-block z-rust\"><\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span><\/span>\n<\/span><\/code><\/pre>\n<p>it will suddenly compile. In particular, the lifetime of the input is no longer tied to the lifetime of the output,\nwhich works because <code>link.reference<\/code> is copied, not borrowed.<\/p>\n<p>For more information about this problem, including strange and disconcerting errors that show up\nif you give lifetimes the wrong annotations, see <a href=\"https:\/\/github.com\/raphlinus\/pulldown-cmark\/issues\/509\">the issue<\/a> against pulldown, as well as <a href=\"https:\/\/github.com\/raphlinus\/pulldown-cmark\/pull\/510\">the\nPR<\/a> I made fixing it.<\/p>\n<h2 id=\"bonus-wait-where-s-my-diagnostics-bug-i-was-promised-a-diagnostics-bug\">Bonus: Wait, where's my diagnostics bug? I was promised a diagnostics bug!<a class=\"zola-anchor\" href=\"#bonus-wait-where-s-my-diagnostics-bug-i-was-promised-a-diagnostics-bug\" aria-label=\"Anchor link for: bonus-wait-where-s-my-diagnostics-bug-i-was-promised-a-diagnostics-bug\"><\/a>\n<\/h2>\n<p>For some reason, the error in a stand-alone program is different from the one when you compile <code>broken_link_callback<\/code> as part of rustdoc:<\/p>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">error: implementation of `FnOnce` is not general enough\n<\/span><span class=\"z-text z-plain\">   --&gt; src\/lib.rs:8:80\n<\/span><span class=\"z-text z-plain\">    |\n<\/span><span class=\"z-text z-plain\">8   |       for _ in Parser::new_with_broken_link_callback(txt, Options::empty(), Some(&amp;mut callback)) {\n<\/span><span class=\"z-text z-plain\">    |                                                                                  ^^^^^^^^^^^^^ implementation of `FnOnce` is not general enough\n<\/span><span class=\"z-text z-plain\">    | \n<\/span><span class=\"z-text z-plain\">   ::: \/home\/jyn\/.local\/lib\/rustup\/toolchains\/stable-x86_64-unknown-linux-gnu\/lib\/rustlib\/src\/rust\/library\/core\/src\/ops\/function.rs:219:1\n<\/span><span class=\"z-text z-plain\">    |\n<\/span><span class=\"z-text z-plain\">219 | \/ pub trait FnOnce&lt;Args&gt; {\n<\/span><span class=\"z-text z-plain\">220 | |     \/\/\/ The returned type after the call operator is used.\n<\/span><span class=\"z-text z-plain\">221 | |     #[lang = &quot;fn_once_output&quot;]\n<\/span><span class=\"z-text z-plain\">222 | |     #[stable(feature = &quot;fn_once_output&quot;, since = &quot;1.12.0&quot;)]\n<\/span><span class=\"z-text z-plain\">...   |\n<\/span><span class=\"z-text z-plain\">227 | |     extern &quot;rust-call&quot; fn call_once(self, args: Args) -&gt; Self::Output;\n<\/span><span class=\"z-text z-plain\">228 | | }\n<\/span><span class=\"z-text z-plain\">    | |_- trait `FnOnce` defined here\n<\/span><span class=\"z-text z-plain\">    |\n<\/span><span class=\"z-text z-plain\">    = note: `for&lt;&#39;a&gt; fn(pulldown_cmark::BrokenLink&lt;&#39;a&gt;) -&gt; Option&lt;(pulldown_cmark::CowStr&lt;&#39;a&gt;, pulldown_cmark::CowStr&lt;&#39;a&gt;)&gt; {callback}` must implement `FnOnce&lt;(pulldown_cmark::BrokenLink&lt;&#39;_&gt;,)&gt;`\n<\/span><span class=\"z-text z-plain\">    = note: ...but `FnOnce&lt;(pulldown_cmark::BrokenLink&lt;&#39;_&gt;,)&gt;` is actually implemented for the type `for&lt;&#39;a&gt; fn(pulldown_cmark::BrokenLink&lt;&#39;a&gt;) -&gt; Option&lt;(pulldown_cmark::CowStr&lt;&#39;a&gt;, pulldown_cmark::CowStr&lt;&#39;a&gt;)&gt; {callback}`\n<\/span><\/code><\/pre>\n<p>In particular, the <code>note:<\/code> makes no sense: it shows the same types above and below!\nThis is a long-lived <a href=\"https:\/\/github.com\/rust-lang\/rust\/issues\/79643\">diagnostics issue<\/a> with the compiler itself, going back <a href=\"https:\/\/github.com\/rust-lang\/rust\/issues\/41078\">at least until 2017<\/a>.<\/p>\n<h2 id=\"appendix\">Appendix<a class=\"zola-anchor\" href=\"#appendix\" aria-label=\"Anchor link for: appendix\"><\/a>\n<\/h2>\n<ul>\n<li><a href=\"https:\/\/doc.rust-lang.org\/nomicon\/hrtb.html\">HRTB<\/a><\/li>\n<li><a href=\"https:\/\/doc.rust-lang.org\/rustdoc\/\">Rustdoc<\/a><\/li>\n<li><a href=\"https:\/\/doc.rust-lang.org\/rustdoc\/linking-to-items-by-name.html\">intra-doc links<\/a><\/li>\n<li><a href=\"https:\/\/doc.rust-lang.org\/nomicon\/lifetimes.html#the-area-covered-by-a-lifetime\">reborrowing<\/a><\/li>\n<li>I wrote more about <a href=\"https:\/\/stackoverflow.com\/questions\/28961957\/example-of-runtime-polymorphism-in-java\">runtime polymorphism<\/a> (which is like <code>virtual<\/code> in C++ or any method call in Python) in an <a href=\"https:\/\/acm.cse.sc.edu\/assets\/2020-09-09\">ACM talk<\/a>.<\/li>\n<\/ul>\n"},{"title":"The intra-doc links saga","published":"2020-10-23T00:00:00+00:00","updated":"2020-10-23T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/the-intra-doc-links-saga\/"}},"id":"https:\/\/jyn.dev\/the-intra-doc-links-saga\/","summary":"How I helped stabilize intra-doc links","content":"<p>One of my projects for the last 4 months or so has been working on\n<a href=\"https:\/\/github.com\/rust-lang\/rust\/issues\/43466\">'intra-doc links'<\/a>, a feature of <code>rustdoc<\/code>\nthat lets you link to items by name. That feature will be stable in 4 weeks in\nRust 1.48.0!<\/p>\n<p><strong>@Manishearth<\/strong> and I wrote a longer blog post about intra-doc links which you\ncan read on <a href=\"https:\/\/blog.rust-lang.org\/inside-rust\/2020\/09\/17\/stabilizing-intra-doc-links.html\">the official Rust blog<\/a>.<\/p>\n"},{"title":"Rust in 2021","published":"2020-09-05T00:00:00+00:00","updated":"2020-09-05T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/rust-in-2021\/"}},"id":"https:\/\/jyn.dev\/rust-in-2021\/","summary":"Rust in 2021 should focus on discoverability","content":"<h2 id=\"who-is-this-guy-anyway\">Who is this guy anyway?<a class=\"zola-anchor\" href=\"#who-is-this-guy-anyway\" aria-label=\"Anchor link for: who-is-this-guy-anyway\"><\/a>\n<\/h2>\n<p>Hello, it's me! I'm a somewhat new contributor to Rust and I'm about three blog posts behind. Here they are all at once:<\/p>\n<ul>\n<li><a href=\"https:\/\/jyn.dev\/building-docs-rs\/\">I work on docs.rs!<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/jyn514\/saltwater\/\">I wrote a C compiler in Rust<\/a><\/li>\n<li>I spend way too much time <a href=\"https:\/\/github.com\/rust-lang\/rust\/pulls?q=is%3Apr+label%3AA-intra-doc-links+author%3Ajyn514+\">working on intra-doc links<\/a><\/li>\n<li>I <a href=\"https:\/\/github.com\/rust-lang\/compiler-team\/issues\/326\">help<\/a> <a href=\"https:\/\/github.com\/rust-lang\/rustc-dev-guide\/pulls?q=+is%3Apr+author%3Ajyn514+\">make<\/a> it easier to contribute to the Rust compiler<\/li>\n<\/ul>\n<h2 id=\"wait-we-re-writing-a-blog-post-here\">Wait, we're writing a blog post here.<a class=\"zola-anchor\" href=\"#wait-we-re-writing-a-blog-post-here\" aria-label=\"Anchor link for: wait-we-re-writing-a-blog-post-here\"><\/a>\n<\/h2>\n<p>In particular I want to talk about that last point.\nI spend way too much time on Discord and I commonly run into questions like this:<\/p>\n<blockquote>\n<p>How do I get the first element of a vector if I don't care about the rest? <code>vec[0]<\/code> gives me a reference.<\/p>\n<\/blockquote>\n<blockquote>\n<p>How does docs.rs calculate the percentage of items documented in a crate?<\/p>\n<\/blockquote>\n<blockquote>\n<p>Why doesn't <code>async || { ... }<\/code> work? The compiler told me to add <code>async<\/code> if I want to use <code>await<\/code>!<\/p>\n<\/blockquote>\n<blockquote>\n<p>Why am I getting a borrow check error when I try to add something to a map if it's not yet there?<\/p>\n<\/blockquote>\n<blockquote>\n<p>How can I use generic types in an <code>extern \"C\"<\/code> function? Is this even possible? (This one I asked!)<\/p>\n<\/blockquote>\n<blockquote>\n<p>How do I link to the latest version of a subpage of my docs on docs.rs?<\/p>\n<\/blockquote>\n<blockquote>\n<p>Is there a guide on how to add a new target or promote a target to tier 2?<\/p>\n<\/blockquote>\n<blockquote>\n<p>How can I use C or C++ functions from Rust? (not just the C standard library, but other libraries)<\/p>\n<\/blockquote>\n<blockquote>\n<p>Can I have an API that takes either a value or a reference?<\/p>\n<\/blockquote>\n<blockquote>\n<p>Can I have a logging function that will print then return a value?<\/p>\n<\/blockquote>\n<blockquote>\n<p>Where did <code>arg_enum!<\/code> end up in clap 3? It's not at <code>clap::arg_enum<\/code> like it was before.<\/p>\n<\/blockquote>\n<blockquote>\n<p>Is there a way to do this with iterator methods instead?<\/p>\n<\/blockquote>\n<p>All of these have <a href=\"https:\/\/jyn.dev\/rust-in-2021\/#faq\">answers<\/a> that are simple to understand after the fact, but\nare very hard to figure out if you don't already know them.<\/p>\n<h2 id=\"discoverability\">Discoverability<a class=\"zola-anchor\" href=\"#discoverability\" aria-label=\"Anchor link for: discoverability\"><\/a>\n<\/h2>\n<p>As a frequent contributor to the compiler, the theme I want to focus on in 2021 is <strong>discoverability<\/strong>.\nRust and the Rust ecosystem have a lot of features, but it can be hard to find them all,\nor even to know that they exist - you can't search if you don't know what to search for!<\/p>\n<h3 id=\"in-the-compiler\">In the compiler<a class=\"zola-anchor\" href=\"#in-the-compiler\" aria-label=\"Anchor link for: in-the-compiler\"><\/a>\n<\/h3>\n<p>Rust has a well-earned reputation for good error messages.\nI want to continue to expand those to catch more common errors and guide you\nin the right direction. There is a <a href=\"https:\/\/github.com\/rust-lang\/rust\/pull\/73996\">lot<\/a> of\n<a href=\"https:\/\/github.com\/rust-lang\/rust\/pull\/76171\">great<\/a> <a href=\"https:\/\/github.com\/rust-lang\/rust\/pull\/75931\">work<\/a> going on this area:\nA giant thank you to <strong>@estebank<\/strong>, <strong>@da-x<\/strong>, and everyone else working on improving diagnostics!<\/p>\n<h3 id=\"in-libraries\">In libraries<a class=\"zola-anchor\" href=\"#in-libraries\" aria-label=\"Anchor link for: in-libraries\"><\/a>\n<\/h3>\n<p>This one is harder - libraries can't give error messages,\nthe best they can do is write documentation.\nI want to see more examples of ways to solve errors <em>using the library<\/em>.\nFor example, imagine if this error:<\/p>\n<pre data-lang=\"rust\" class=\"language-rust z-code\"><code class=\"language-rust\" data-lang=\"rust\"><span class=\"z-source z-rust\">error<span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">[<\/span><span class=\"z-constant z-other z-rust\">E0499<\/span><span class=\"z-punctuation z-section z-group z-end z-rust\">]<\/span><\/span><span class=\"z-punctuation z-separator z-rust\">:<\/span> cannot borrow `<span class=\"z-keyword z-operator z-arithmetic z-rust\">*<\/span>map` <span class=\"z-keyword z-operator z-rust\">as<\/span> mutable more than once at a time\n<\/span><span class=\"z-source z-rust\">  <span class=\"z-keyword z-operator z-arithmetic z-rust\">-<\/span><span class=\"z-meta z-function z-return-type z-rust\"><span class=\"z-punctuation z-separator z-rust\">-&gt;<\/span> src<\/span><span class=\"z-keyword z-operator z-arithmetic z-rust\">\/<\/span>main<span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span>rs<span class=\"z-punctuation z-separator z-rust\">:<\/span><span class=\"z-constant z-numeric z-integer z-decimal z-rust\">14<\/span><span class=\"z-punctuation z-separator z-rust\">:<\/span><span class=\"z-constant z-numeric z-integer z-decimal z-rust\">13<\/span>\n<\/span><span class=\"z-source z-rust\">   <span class=\"z-keyword z-operator z-bitwise z-rust\">|<\/span>\n<\/span><span class=\"z-source z-rust\"><span class=\"z-constant z-numeric z-integer z-decimal z-rust\">6<\/span>  <span class=\"z-keyword z-operator z-bitwise z-rust\">|<\/span>   <span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-storage z-type z-function z-rust\">fn<\/span> <\/span><span class=\"z-entity z-name z-function z-rust\">get_default<\/span><\/span><span class=\"z-meta z-generic z-rust\"><span class=\"z-punctuation z-definition z-generic z-begin z-rust\">&lt;<\/span><span class=\"z-storage z-modifier z-lifetime z-rust\">&#39;m<\/span>, K, V<span class=\"z-punctuation z-definition z-generic z-end z-rust\">&gt;<\/span><\/span><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-begin z-rust\">(<\/span><span class=\"z-variable z-parameter z-rust\">map<\/span><span class=\"z-punctuation z-separator z-rust\">:<\/span> <span class=\"z-keyword z-operator z-rust\">&amp;<\/span><span class=\"z-storage z-modifier z-lifetime z-rust\">&#39;m<\/span> <span class=\"z-storage z-modifier z-rust\">mut<\/span> <span class=\"z-meta z-generic z-rust\">HashMap<span class=\"z-punctuation z-definition z-generic z-begin z-rust\">&lt;<\/span>K, V<span class=\"z-punctuation z-definition z-generic z-end z-rust\">&gt;<\/span><\/span>, <span class=\"z-variable z-parameter z-rust\">key<\/span><span class=\"z-punctuation z-separator z-rust\">:<\/span> K<\/span><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-end z-rust\">)<\/span><\/span><\/span><\/span><span class=\"z-meta z-function z-rust\"> <span class=\"z-meta z-function z-return-type z-rust\"><span class=\"z-punctuation z-separator z-rust\">-&gt;<\/span> <span class=\"z-keyword z-operator z-rust\">&amp;<\/span><span class=\"z-storage z-modifier z-lifetime z-rust\">&#39;m<\/span> <span class=\"z-storage z-modifier z-rust\">mut<\/span> V\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-rust\"><span class=\"z-meta z-function z-return-type z-rust\">   <\/span><\/span><span class=\"z-keyword z-operator z-bitwise z-rust\">|<\/span>                  <span class=\"z-keyword z-operator z-arithmetic z-rust\">-<\/span><span class=\"z-keyword z-operator z-arithmetic z-rust\">-<\/span> lifetime `<span class=\"z-storage z-modifier z-lifetime z-rust\">&#39;m<\/span>` defined here\n<\/span><span class=\"z-source z-rust\"><span class=\"z-keyword z-operator z-range z-rust\">...<\/span>\n<\/span><span class=\"z-source z-rust\"><span class=\"z-constant z-numeric z-integer z-decimal z-rust\">11<\/span> <span class=\"z-keyword z-operator z-bitwise z-rust\">|<\/span>       <span class=\"z-keyword z-control z-rust\">match<\/span> map<span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">get_mut<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-keyword z-operator z-bitwise z-rust\">&amp;<\/span>key<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span> <span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-block z-rust\">   <span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-begin z-rust\">|<\/span><\/span><\/span><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-function z-parameters z-rust\">       -     --- first mutable borrow occurs here<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-function z-closure z-rust\">   <\/span><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-keyword z-operator z-bitwise z-rust\">|<\/span>  <span class=\"z-constant z-other z-rust\">_____<\/span><span class=\"z-keyword z-operator z-bitwise z-rust\">|<\/span><\/span>\n<\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-block z-rust\">   <span class=\"z-keyword z-operator z-bitwise z-rust\">|<\/span> <span class=\"z-keyword z-operator z-bitwise z-rust\">|<\/span>\n<\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-constant z-numeric z-integer z-decimal z-rust\">12<\/span> <span class=\"z-keyword z-operator z-bitwise z-rust\">|<\/span> <span class=\"z-keyword z-operator z-bitwise z-rust\">|<\/span>         <span class=\"z-support z-type z-rust\">Some<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span>value<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span> <span class=\"z-keyword z-operator z-rust\">=&gt;<\/span> value<span class=\"z-punctuation z-separator z-rust\">,<\/span>\n<\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-constant z-numeric z-integer z-decimal z-rust\">13<\/span> <span class=\"z-keyword z-operator z-bitwise z-rust\">|<\/span> <span class=\"z-keyword z-operator z-bitwise z-rust\">|<\/span>         <span class=\"z-support z-type z-rust\">None<\/span> <span class=\"z-keyword z-operator z-rust\">=&gt;<\/span> <span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-constant z-numeric z-integer z-decimal z-rust\">14<\/span> <span class=\"z-keyword z-operator z-bitwise z-rust\">|<\/span> <span class=\"z-keyword z-operator z-bitwise z-rust\">|<\/span>             map<span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">insert<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span>key<span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">clone<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-separator z-rust\">,<\/span> <span class=\"z-meta z-path z-rust\">V<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span>default<span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-terminator z-rust\">;<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-block z-rust\">   <span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-begin z-rust\">|<\/span><\/span><\/span><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"> <span class=\"z-punctuation z-section z-parameters z-end z-rust\">|<\/span><\/span>             <\/span><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-keyword z-operator z-bitwise z-rust\">^<\/span><span class=\"z-keyword z-operator z-bitwise z-rust\">^<\/span><span class=\"z-keyword z-operator z-bitwise z-rust\">^<\/span> second mutable borrow occurs here<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-constant z-numeric z-integer z-decimal z-rust\">15<\/span> <span class=\"z-keyword z-operator z-bitwise z-rust\">|<\/span> <span class=\"z-keyword z-operator z-bitwise z-rust\">|<\/span>             map<span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">get_mut<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-keyword z-operator z-bitwise z-rust\">&amp;<\/span>key<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">unwrap<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-constant z-numeric z-integer z-decimal z-rust\">16<\/span> <span class=\"z-keyword z-operator z-bitwise z-rust\">|<\/span> <span class=\"z-keyword z-operator z-bitwise z-rust\">|<\/span>         <\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span>\n<\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-constant z-numeric z-integer z-decimal z-rust\">17<\/span> <span class=\"z-keyword z-operator z-bitwise z-rust\">|<\/span> <span class=\"z-keyword z-operator z-bitwise z-rust\">|<\/span>     <\/span><span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-end z-rust\">}<\/span><\/span>\n<\/span><span class=\"z-source z-rust\">   <span class=\"z-keyword z-operator z-bitwise z-rust\">|<\/span> <span class=\"z-keyword z-operator z-bitwise z-rust\">|<\/span><span class=\"z-constant z-other z-rust\">_____<\/span><span class=\"z-keyword z-operator z-arithmetic z-rust\">-<\/span> returning this value requires that `<span class=\"z-keyword z-operator z-arithmetic z-rust\">*<\/span>map` is borrowed <span class=\"z-keyword z-control z-rust\">for<\/span> `<span class=\"z-storage z-modifier z-lifetime z-rust\">&#39;m<\/span>`\n<\/span><\/code><\/pre>\n<p>also said this:<\/p>\n<pre data-lang=\"rust\" class=\"language-rust z-code\"><code class=\"language-rust\" data-lang=\"rust\"><span class=\"z-source z-rust\">   <span class=\"z-keyword z-operator z-assignment z-rust\">=<\/span> note<span class=\"z-punctuation z-separator z-rust\">:<\/span> this pattern is valid<span class=\"z-punctuation z-separator z-rust\">,<\/span> but not currently recognized by the borrow<span class=\"z-keyword z-operator z-arithmetic z-rust\">-<\/span>checker<span class=\"z-punctuation z-separator z-rust\">:<\/span> https<span class=\"z-punctuation z-separator z-rust\">:<\/span><span class=\"z-comment z-line z-double-slash z-rust\"><span class=\"z-punctuation z-definition z-comment z-rust\">\/\/<\/span>doc.rust-lang.org\/nomicon\/lifetime-mismatch.html#improperly-reduced-borrows\n<\/span><\/span><span class=\"z-source z-rust\">   <span class=\"z-keyword z-operator z-assignment z-rust\">=<\/span> help<span class=\"z-punctuation z-separator z-rust\">:<\/span> try using the `Entry` <span class=\"z-constant z-other z-rust\">API<\/span><span class=\"z-punctuation z-separator z-rust\">:<\/span>\n<\/span><span class=\"z-source z-rust\">   <span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-begin z-rust\">|<\/span><\/span><\/span><span class=\"z-meta z-function z-closure z-rust\">\n<\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-function z-closure z-rust\"><\/span><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-constant z-numeric z-integer z-decimal z-rust\">11<\/span> <span class=\"z-keyword z-operator z-bitwise z-rust\">|<\/span>       map<span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">entry<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><span class=\"z-keyword z-operator z-bitwise z-rust\">&amp;<\/span>key<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span><span class=\"z-support z-function z-rust\">or_default<\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">(<\/span><\/span><span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-end z-rust\">)<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-rust\">   <span class=\"z-keyword z-operator z-bitwise z-rust\">|<\/span>       <span class=\"z-keyword z-operator z-bitwise z-rust\">^<\/span><span class=\"z-keyword z-operator z-bitwise z-rust\">^<\/span><span class=\"z-keyword z-operator z-bitwise z-rust\">^<\/span><span class=\"z-keyword z-operator z-bitwise z-rust\">^<\/span><span class=\"z-keyword z-operator z-bitwise z-rust\">^<\/span><span class=\"z-keyword z-operator z-bitwise z-rust\">^<\/span><span class=\"z-keyword z-operator z-bitwise z-rust\">^<\/span><span class=\"z-keyword z-operator z-bitwise z-rust\">^<\/span><span class=\"z-keyword z-operator z-bitwise z-rust\">^<\/span><span class=\"z-keyword z-operator z-bitwise z-rust\">^<\/span><span class=\"z-keyword z-operator z-bitwise z-rust\">^<\/span><span class=\"z-keyword z-operator z-bitwise z-rust\">^<\/span><span class=\"z-keyword z-operator z-bitwise z-rust\">^<\/span><span class=\"z-keyword z-operator z-bitwise z-rust\">^<\/span><span class=\"z-keyword z-operator z-bitwise z-rust\">^<\/span><span class=\"z-keyword z-operator z-bitwise z-rust\">^<\/span><span class=\"z-keyword z-operator z-bitwise z-rust\">^<\/span><span class=\"z-keyword z-operator z-bitwise z-rust\">^<\/span><span class=\"z-keyword z-operator z-bitwise z-rust\">^<\/span><span class=\"z-keyword z-operator z-bitwise z-rust\">^<\/span><span class=\"z-keyword z-operator z-bitwise z-rust\">^<\/span><span class=\"z-keyword z-operator z-bitwise z-rust\">^<\/span><span class=\"z-keyword z-operator z-bitwise z-rust\">^<\/span><span class=\"z-keyword z-operator z-bitwise z-rust\">^<\/span><span class=\"z-keyword z-operator z-bitwise z-rust\">^<\/span><span class=\"z-keyword z-operator z-bitwise z-rust\">^<\/span><span class=\"z-keyword z-operator z-bitwise z-rust\">^<\/span><span class=\"z-keyword z-operator z-bitwise z-rust\">^<\/span>\n<\/span><\/code><\/pre>\n<p>Another example:<\/p>\n<pre data-lang=\"rust\" class=\"language-rust z-code\"><code class=\"language-rust\" data-lang=\"rust\"><span class=\"z-source z-rust\">error<span class=\"z-meta z-group z-rust\"><span class=\"z-punctuation z-section z-group z-begin z-rust\">[<\/span><span class=\"z-constant z-other z-rust\">E0433<\/span><span class=\"z-punctuation z-section z-group z-end z-rust\">]<\/span><\/span><span class=\"z-punctuation z-separator z-rust\">:<\/span> failed to resolve<span class=\"z-punctuation z-separator z-rust\">:<\/span> could not find `arg_enum` <span class=\"z-keyword z-operator z-rust\">in<\/span> `clap`\n<\/span><span class=\"z-source z-rust\"> <span class=\"z-keyword z-operator z-arithmetic z-rust\">-<\/span><span class=\"z-meta z-function z-return-type z-rust\"><span class=\"z-punctuation z-separator z-rust\">-&gt;<\/span> src<\/span><span class=\"z-keyword z-operator z-arithmetic z-rust\">\/<\/span>lib<span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span>rs<span class=\"z-punctuation z-separator z-rust\">:<\/span><span class=\"z-constant z-numeric z-integer z-decimal z-rust\">2<\/span><span class=\"z-punctuation z-separator z-rust\">:<\/span><span class=\"z-constant z-numeric z-integer z-decimal z-rust\">10<\/span>\n<\/span><span class=\"z-source z-rust\">  <span class=\"z-keyword z-operator z-bitwise z-rust\">|<\/span>\n<\/span><span class=\"z-source z-rust\"><span class=\"z-constant z-numeric z-integer z-decimal z-rust\">2<\/span> <span class=\"z-keyword z-operator z-bitwise z-rust\">|<\/span>     <span class=\"z-meta z-path z-rust\">clap<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span>arg_enum<span class=\"z-keyword z-operator z-logical z-rust\">!<\/span> <span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-block z-rust\">  <span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-begin z-rust\">|<\/span><\/span><\/span><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-function z-parameters z-rust\">           ^^^^^^^^ could not find `arg_enum` in `clap`<\/span>\n<\/span><\/span><\/span><\/code><\/pre>\n<p>Could instead be:<\/p>\n<pre data-lang=\"rust\" class=\"language-rust z-code\"><code class=\"language-rust\" data-lang=\"rust\"><span class=\"z-source z-rust\">error<span class=\"z-punctuation z-separator z-rust\">:<\/span> `arg_enum` is now a derive <span class=\"z-invalid z-illegal z-rust\">macro<\/span>\n<\/span><span class=\"z-source z-rust\"> <span class=\"z-keyword z-operator z-arithmetic z-rust\">-<\/span><span class=\"z-meta z-function z-return-type z-rust\"><span class=\"z-punctuation z-separator z-rust\">-&gt;<\/span> src<\/span><span class=\"z-keyword z-operator z-arithmetic z-rust\">\/<\/span>lib<span class=\"z-punctuation z-accessor z-dot z-rust\">.<\/span>rs<span class=\"z-punctuation z-separator z-rust\">:<\/span><span class=\"z-constant z-numeric z-integer z-decimal z-rust\">2<\/span><span class=\"z-punctuation z-separator z-rust\">:<\/span><span class=\"z-constant z-numeric z-integer z-decimal z-rust\">10<\/span>\n<\/span><span class=\"z-source z-rust\">  <span class=\"z-keyword z-operator z-bitwise z-rust\">|<\/span>\n<\/span><span class=\"z-source z-rust\"><span class=\"z-constant z-numeric z-integer z-decimal z-rust\">2<\/span> <span class=\"z-keyword z-operator z-bitwise z-rust\">|<\/span>     <span class=\"z-meta z-path z-rust\">clap<span class=\"z-punctuation z-accessor z-rust\">::<\/span><\/span>arg_enum<span class=\"z-keyword z-operator z-logical z-rust\">!<\/span> <span class=\"z-meta z-block z-rust\"><span class=\"z-punctuation z-section z-block z-begin z-rust\">{<\/span>\n<\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-block z-rust\">  <span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-function z-parameters z-rust\"><span class=\"z-punctuation z-section z-parameters z-begin z-rust\">|<\/span><\/span><\/span><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-meta z-function z-parameters z-rust\">           ^^^^^^^^ <span class=\"z-variable z-parameter z-rust\">note<\/span><span class=\"z-punctuation z-separator z-rust\">:<\/span> this code was valid in clap 2.0<span class=\"z-punctuation z-separator z-rust\">,<\/span> but has changed in 3.0<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-rust\"><span class=\"z-meta z-block z-rust\"><span class=\"z-meta z-function z-closure z-rust\">  <\/span><span class=\"z-meta z-function z-closure z-rust\"><span class=\"z-keyword z-operator z-assignment z-rust\">=<\/span> help<span class=\"z-punctuation z-separator z-rust\">:<\/span> <span class=\"z-keyword z-other z-rust\">use<\/span> a derive instead<span class=\"z-punctuation z-separator z-rust\">:<\/span> `<span class=\"z-meta z-annotation z-rust\"><span class=\"z-punctuation z-definition z-annotation z-rust\">#<\/span><span class=\"z-punctuation z-section z-group z-begin z-rust\">[<\/span><span class=\"z-variable z-annotation z-rust\">clap<\/span>::<span class=\"z-variable z-annotation z-rust\">arg_enum<\/span><span class=\"z-punctuation z-section z-group z-end z-rust\">]<\/span><\/span>`<\/span>\n<\/span><\/span><\/code><\/pre>\n<p>Not only would it help fix the code, but it would help you learn more about the library and the Rust language!\nI'm sure there's many things making this difficult, but I think it's a great goal to reach for.<\/p>\n<h3 id=\"in-devtools\">In devtools<a class=\"zola-anchor\" href=\"#in-devtools\" aria-label=\"Anchor link for: in-devtools\"><\/a>\n<\/h3>\n<p>This one I think has both the most room for improvement and needs the least effort to fix.\nCurrently, a lot of functionality is 'hidden' behind <code>cargo<\/code> subcommands.\nFor examples, <code>cargo test -- --ignored<\/code> and <code>cargo test -- --test-threads 1<\/code>\nare very difficult to find unless you see existing examples online.\nEven experienced rustaceans don't know about many rustdoc options because they need\n<code>cargo rustdoc -- -h<\/code>, not <code>cargo doc -h<\/code> or <code>cargo doc -- -h<\/code>.<\/p>\n<p>I think we should both document these options better and make them easier to find\nby experimenting.<\/p>\n<p>Another improvement I see in devtools is distinguishing between stable and unstable features.\nRight now the way to do this is different for every tool:<\/p>\n<ul>\n<li><code>cargo<\/code> has <code>cargo-features = [\"...\"]<\/code><\/li>\n<li><code>rustfmt<\/code> has <code>unstable_features = true<\/code>, including on the stable channel<\/li>\n<li><code>rustdoc<\/code> enables most features by default, even without <code>#![feature(...)]<\/code><\/li>\n<\/ul>\n<p>This causes confusion: In the <a href=\"https:\/\/github.com\/rust-lang\/rust\/issues\/63305\">words of <strong>@abonader<\/strong><\/a>,<\/p>\n<blockquote>\n<p>I assume it works on stable if there's no feature flags.<\/p>\n<\/blockquote>\n<p>I think there should be a consistent story here that makes it easier to see\nwhat's nightly and what isn't (even I <a href=\"https:\/\/github.com\/rust-lang\/rust\/pull\/75953\">got confused<\/a> by this!).<\/p>\n<h3 id=\"in-the-community\">In the community<a class=\"zola-anchor\" href=\"#in-the-community\" aria-label=\"Anchor link for: in-the-community\"><\/a>\n<\/h3>\n<p>It can be difficult to know where discussions take place.\nThere are many different places:<\/p>\n<ul>\n<li>Zulip<\/li>\n<li>Discord<\/li>\n<li>rustc-dev-guide<\/li>\n<li>Forge<\/li>\n<li>Blog posts (usually discussed on reddit)<\/li>\n<li>internals.rust-lang.org<\/li>\n<li>users.rust-lang.org<\/li>\n<li>MCPs in rust-lang\/compiler<\/li>\n<li>MCPs in rust-lang\/lang<\/li>\n<li>RFCs in rust-lang\/rfcs<\/li>\n<li>FCPs in rust-lang\/rust<\/li>\n<li>Various READMEs in the compiler itself (although most of those point to external links now)<\/li>\n<\/ul>\n<p>This makes it hard for new contributors to find information\nand hard for frequent contributors to stay up to date on the current status of issues.<\/p>\n<p>I think we should consolidate documentation into fewer places and discussion\ninto fewer channels. This would make it easier to understand what's going on\nwithout having to spend lots of time chasing down leads (in both senses \ud83d\ude09).<\/p>\n<h2 id=\"what-do-you-think-could-be-more-discoverable\">What do <em>you<\/em> think could be more discoverable?<a class=\"zola-anchor\" href=\"#what-do-you-think-could-be-more-discoverable\" aria-label=\"Anchor link for: what-do-you-think-could-be-more-discoverable\"><\/a>\n<\/h2>\n<p>Is there a feature of Rust you really enjoy that people don't know about?\nAre you looking for a feature that <em>seems<\/em> like it should exist but you haven't been able to find?\nPlease let me know! Good ways to bring this up are writing other blog posts or opening an issue.<\/p>\n<h2 id=\"summary\">Summary<a class=\"zola-anchor\" href=\"#summary\" aria-label=\"Anchor link for: summary\"><\/a>\n<\/h2>\n<p>I think Rust is a great language with a bright future.\nI want to make it easier not just to get started, but to\nexplore and learn more about Rust and the Rust community.<\/p>\n<h3 id=\"faq\">FAQ<a class=\"zola-anchor\" href=\"#faq\" aria-label=\"Anchor link for: faq\"><\/a>\n<\/h3>\n<p>You can't have questions without answers!<\/p>\n<blockquote>\n<p>How do I get the first element of a vector if I don't care about the rest? <code>vec[0]<\/code> gives me a reference.<\/p>\n<\/blockquote>\n<p><code>vec.into_iter().next()<\/code> or <code>vec.truncate(1); vec.pop()<\/code> depending on when you want to drop the items (<a href=\"https:\/\/rust-lang.zulipchat.com\/#narrow\/stream\/219381-t-libs\/topic\/Eager.20skipping.20for.20iterators\/near\/209045693\">discussion<\/a>).<\/p>\n<blockquote>\n<p>How does docs.rs calculate the percentage of items documented in a crate?<\/p>\n<\/blockquote>\n<p><a href=\"https:\/\/doc.rust-lang.org\/rustdoc\/unstable-features.html#--show-coverage-calculate-the-percentage-of-items-with-documentation\"><code>rustdoc --show-coverage<\/code><\/a>.<\/p>\n<blockquote>\n<p>Why doesn't <code>async || { ... }<\/code> work? The compiler told me to add <code>async<\/code> if I want to use <code>await<\/code>!<\/p>\n<\/blockquote>\n<p>Use <code>async { ... }<\/code> instead. I opened <a href=\"https:\/\/github.com\/rust-lang\/rust\/issues\/76011\">an issue<\/a> to add a suggestion for this.<\/p>\n<blockquote>\n<p>Why am I getting a borrow check error when I try to add something to a map if it's not yet there?<\/p>\n<\/blockquote>\n<p>This is a limitation of the borrow checker; use <a href=\"https:\/\/doc.rust-lang.org\/std\/collections\/hash_map\/enum.Entry.html\"><code>entry()<\/code><\/a> instead.<\/p>\n<blockquote>\n<p>How can I use generic types in an <code>extern \"C\"<\/code> function? Is this even possible? (This one I asked!)<\/p>\n<\/blockquote>\n<p>It works like a normal function pointer! You can simply use <code>my_c_func(my_callback::&lt;T&gt; as fn()<\/code>,\nand declare <code>my_callback<\/code> like a normal generic function that happens to have <code>extern \"C\"<\/code> at the front.<\/p>\n<blockquote>\n<p>How do I link to the latest version of a subpage of my docs on docs.rs?<\/p>\n<\/blockquote>\n<p><code>\/my-crate\/latest\/my_crate\/module\/kind.name.html<\/code>; <a href=\"https:\/\/docs.rs\/about\/redirections\">docs<\/a>.<\/p>\n<blockquote>\n<p>Is there a guide on how to add a new target or promote a target to tier 2?<\/p>\n<\/blockquote>\n<p>There is an <a href=\"https:\/\/github.com\/rust-lang\/rfcs\/pull\/2803\">RFC in progress<\/a> and docs for <a href=\"https:\/\/rustc-dev-guide.rust-lang.org\/building\/new-target.html\">adding a new target<\/a>.\nHowever I'm not aware of any documentation for promoting a target from tier 3 to tier 2,\nor from 'tier 2 cross-compiled' to 'tier 2 hosted' (see the RFC).<\/p>\n<blockquote>\n<p>How can I use C or C++ functions from Rust? (not just the C standard library, but other libraries)<\/p>\n<\/blockquote>\n<p>You probably want either <a href=\"https:\/\/github.com\/dtolnay\/cxx\/\">bindgen<\/a> or <a href=\"https:\/\/github.com\/dtolnay\/cxx\/\">cxx<\/a>. For the reverse, calling Rust from C or C++,\ntake a look at <a href=\"https:\/\/github.com\/eqrion\/cbindgen\">cbindgen<\/a>.<\/p>\n<blockquote>\n<p>Can I have an API that takes either a value or a reference?<\/p>\n<\/blockquote>\n<p>Yes, <a href=\"https:\/\/doc.rust-lang.org\/std\/borrow\/trait.Borrow.html\"><code>Borrow&lt;T&gt;<\/code><\/a>.<\/p>\n<p>NOTE: this answer originally said <a href=\"https:\/\/doc.rust-lang.org\/std\/convert\/trait.AsRef.html\"><code>AsRef&lt;T&gt;<\/code><\/a>,\nwhich doesn't always work - there's no blanket <code>impl AsRef&lt;T&gt; for &amp;T<\/code>\nlike there is for <code>Borrow<\/code>. This difference is itself a discoverability problem! Thanks to <strong>@raphlinus<\/strong> for pointing out the difference.<\/p>\n<blockquote>\n<p>Can I have a logging function that will print then return a value?<\/p>\n<\/blockquote>\n<p>Yes, and it's even in the standard library: <a href=\"https:\/\/doc.rust-lang.org\/std\/macro.dbg.html\"><code>dbg!<\/code><\/a>.\nUnfortunately it doesn't work with <code>log<\/code> or <code>tracing<\/code>,\nbut it's not too hard to write your own wrapper around those.<\/p>\n<blockquote>\n<p>Where did <code>arg_enum!<\/code> end up in clap 3? It's not at <code>clap::arg_enum<\/code> like it was before.<\/p>\n<\/blockquote>\n<p>It's now a <a href=\"https:\/\/github.com\/clap-rs\/clap\/blob\/master\/clap_derive\/examples\/arg_enum.rs\">derive macro<\/a>.<\/p>\n<blockquote>\n<p>Is there a way to do this with iterator methods instead?<\/p>\n<\/blockquote>\n<p>Yes.<\/p>\n<h3 id=\"updates\">Updates!<a class=\"zola-anchor\" href=\"#updates\" aria-label=\"Anchor link for: updates\"><\/a>\n<\/h3>\n<p>This was surprisingly popular <a href=\"https:\/\/www.reddit.com\/r\/rust\/comments\/inxwka\/rust_in_2021_discoverability\/\">on reddit<\/a>! I'm answering a few of the most relevant questions here so everyone can see them.<\/p>\n<blockquote>\n<p>Is there a way a library can give a custom message on trait errors?<\/p>\n<\/blockquote>\n<p>Yes, <a href=\"https:\/\/github.com\/rust-lang\/rust\/issues\/29628\"><code>rustc_on_unimplemented<\/code><\/a>. Unfortunately it's not intended to be stabilized,\nbut I'd be happy to see an RFC for it!<\/p>\n<blockquote>\n<p>Is there something like Hoogle for Rust, where you give it a type signature and it suggests possible implementations?<\/p>\n<\/blockquote>\n<p>Yes, rustdoc has <a href=\"https:\/\/doc.rust-lang.org\/std\/rc\/struct.Rc.html?search=*%20-%3E%20String\">'type-based search'<\/a>. Press <code>?<\/code> in any rustdoc page for more information about it.<\/p>\n<blockquote>\n<p>Couldn't the maintainers of clap make <code>arg_enum!<\/code> deprecated and point to the derive macro?<\/p>\n<\/blockquote>\n<p>No, because both the old and new macro have the same name in the same namespace.\nSo adding both would be a compile error. Fortunately this means the compiler can help out, I <a href=\"https:\/\/github.com\/rust-lang\/rust\/issues\/76429\">opened an issue<\/a> for it to do so.<\/p>\n"},{"title":"Building Docs.rs","published":"2019-11-26T00:00:00+00:00","updated":"2019-11-26T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/building-docs-rs\/"}},"id":"https:\/\/jyn.dev\/building-docs-rs\/","summary":"How I because the 2nd-most prolific contributor to docs.rs","content":"<h2 id=\"what-is-docs-rs\">What is docs.rs?<a class=\"zola-anchor\" href=\"#what-is-docs-rs\" aria-label=\"Anchor link for: what-is-docs-rs\"><\/a>\n<\/h2>\n<p><a href=\"https:\/\/docs.rs\/\">docs.rs<\/a> is a site dedicated to hosting documentation for Rust projects.\nIt automatically builds documentation for every package published to\n<a href=\"https:\/\/crates.io\/\">crates.io<\/a> using <code>cargo<\/code> and <code>rustdoc<\/code>.\nUnfortunately, because <code>rustdoc<\/code> is tied so closely to the compiler,\nthis requires building every package from source using <code>cargo doc<\/code>.\nAs a result, I've heard several members of the docs.rs team describe it as\n'Remote Code Execution as a Service' (since <code>build.rs<\/code> files can\n<a href=\"https:\/\/doc.rust-lang.org\/cargo\/reference\/build-scripts.html\">execute arbitrary code<\/a>).<\/p>\n<p>Just from this description you can see that hosting docs.rs is an enormous job\nand requires a ton of compute resources. It gets worse when you consider that\ndocs.rs has to <em>store<\/em> all this documentation somewhere. In fact, it's in\nan AWS storage bucket and gets uploaded after every build.<\/p>\n<h2 id=\"how-did-i-get-involved\">How did I get involved?<a class=\"zola-anchor\" href=\"#how-did-i-get-involved\" aria-label=\"Anchor link for: how-did-i-get-involved\"><\/a>\n<\/h2>\n<p>My main rust project is <a href=\"https:\/\/github.com\/jyn514\/rcc\">rcc<\/a>, a C compiler written in Rust.\nI use <a href=\"https:\/\/github.com\/CraneStation\/cranelift\">Cranelift<\/a> to generate the assembly\ncode; Cranelift is a code generator like <a href=\"https:\/\/llvm.org\/\">LLVM<\/a>.\nCranelift releases very quickly - I first started using it in version\n<a href=\"https:\/\/github.com\/jyn514\/rcc\/commit\/9f5573d\"><code>0.36<\/code><\/a> and it's currently on\n<a href=\"https:\/\/docs.rs\/cranelift-codegen\/0.51.0\/cranelift_codegen\/\"><code>0.51<\/code><\/a>.\nBecause I used the docs so much when I was getting started,\nFirefox got convinced that I always wanted to visit <code>0.36<\/code> - see also the\n<a href=\"https:\/\/github.com\/rust-lang\/rust\/issues\/9461\">rustdoc issue<\/a> about canonical URLs.<\/p>\n<p>Fortunately, docs.rs has a little link you can click to go to the latest version.\nUnfortunately, docs.rs used to redirect you to the home page whenever you clicked that link,\nso if you were looking up\n<a href=\"https:\/\/docs.rs\/cranelift-codegen\/0.42.0\/cranelift_codegen\/ir\/trait.InstBuilder.html\"><code>cranelift_codegen::ir::InstBuilder<\/code><\/a>\nbefore, you'd have to click through 3 links to get back there on the latest version.\nThis has made a lot of people very angry and been widely regarded as a bad move.\nThere had actually been <a href=\"https:\/\/github.com\/rust-lang\/docs.rs\/issues\/200\">an issue open<\/a>\nfor about a year and a half at this point, although it wasn't getting much attention.<\/p>\n<p>I decided since I was running into the issue so much, I may as well contribute a patch.<\/p>\n<h2 id=\"what-happened\">What happened?<a class=\"zola-anchor\" href=\"#what-happened\" aria-label=\"Anchor link for: what-happened\"><\/a>\n<\/h2>\n<p>At that point, docs.rs was using <a href=\"https:\/\/www.vagrantup.com\/\">Vagrant<\/a> to run\nbuilds locally, because it needed to have a sandboxed environment to execute\narbitrary code on. When I tried using it, it took about 20 minutes and then I\n<a href=\"https:\/\/github.com\/rust-lang\/docs.rs\/issues\/200#issuecomment-539771094\">got an ugly error<\/a> without any explanation.\nRemember, this isn't even the hard part of the change, I just wanted to get it running from master.<\/p>\n<p>I chatted with <a href=\"https:\/\/github.com\/pietroalbini\">Pietro<\/a> on Discord\nand it turns out that docs.rs had recently\n<a href=\"https:\/\/github.com\/rust-lang\/docs.rs\/pull\/407\">switched to a different sandbox<\/a>,\nbut the Vagrantfile never got updated. Since I don't know really how to use Vagrant\nor Ruby (and no one was volunteering to fix it), I didn't have any hope of getting\nthat working. I asked what I could do and someone pointed me to\n<a href=\"https:\/\/github.com\/rust-lang\/docs.rs\/wiki\/Self-hosting-outside-the-Vagrant-VM\">the wiki<\/a>.\nI took one look and knew there was no way I wanted to do this - it's about 5 pages,\nand 30 manually steps, most of which have to be done in order.\nSo I figured, hey, I'm a developer, let's automate this!<\/p>\n<p>Long story short, I ended up <a href=\"https:\/\/github.com\/rust-lang\/docs.rs\/pull\/432\">writing a <code>Dockerfile<\/code><\/a>\nand Pietro was so impressed that he started <a href=\"https:\/\/github.com\/rust-lang\/docs.rs\/pull\/455\">using it in production<\/a> :D<\/p>\n<p>After a month and what has been called<\/p>\n<blockquote>\n<p><a href=\"https:\/\/users.rust-lang.org\/t\/rust-2020-growth\/34956\/43\">Unprecedented (but greatly appreciated) levels of yak shaving<\/a><\/p>\n<\/blockquote>\n<p>I finally <a href=\"https:\/\/github.com\/rust-lang\/docs.rs\/pull\/454\">got the original change merged<\/a> :)<\/p>\n<h2 id=\"what-next\">What next?<a class=\"zola-anchor\" href=\"#what-next\" aria-label=\"Anchor link for: what-next\"><\/a>\n<\/h2>\n<p>Pietro and <a href=\"https:\/\/github.com\/QuietMisdreavus\">QuietMisdreavus<\/a> were so impressed\nwith my work that they invited me to the team! As of November 1st, I have push access\nto <code>docs.rs<\/code> :) (Pietro still handles the deploys)<\/p>\n<p>I've been helping with <a href=\"https:\/\/github.com\/rust-lang\/docs.rs\/pull\/487\">a<\/a>\n<a href=\"https:\/\/github.com\/rust-lang\/docs.rs\/pull\/485\">few<\/a>\n<a href=\"https:\/\/github.com\/rust-lang\/docs.rs\/pull\/476\">other<\/a>\n<a href=\"https:\/\/github.com\/rust-lang\/docs.rs\/pull\/468\">PRs<\/a>\n<a href=\"https:\/\/github.com\/rust-lang\/docs.rs\/pull\/460\">since<\/a>\nand plan to contribute for the foreseeable future!<\/p>\n<h2 id=\"acknowledgments\">Acknowledgments<a class=\"zola-anchor\" href=\"#acknowledgments\" aria-label=\"Anchor link for: acknowledgments\"><\/a>\n<\/h2>\n<p>Thank you so much to Pietro and QuietMisdreavus for helping me get started with\nthe project! My favorite part of working on docs.rs is giving and receiving feedback\nfrom my friends.<\/p>\n<p>Thank you to <a href=\"https:\/\/users.rust-lang.org\/u\/17cupsofcoffee\">\/u\/17cupsofcoffee<\/a>\nfor prompting me to write this blog post!<\/p>\n<p>And finally, thanks to everyone in the Rust community for making this an awesome\nlanguage to work on :D<\/p>\n"},{"title":"Intro to Reverse Engineering","published":"2019-11-11T00:00:00+00:00","updated":"2019-11-11T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/talks\/assembly\/"}},"id":"https:\/\/jyn.dev\/talks\/assembly\/","content":{"@attributes":{"type":"html"}}},{"title":"Intro to C","published":"2019-11-05T00:00:00+00:00","updated":"2019-11-05T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/talks\/c\/"}},"id":"https:\/\/jyn.dev\/talks\/c\/","content":{"@attributes":{"type":"html"}}},{"title":"Intro to Linux","published":"2019-09-16T00:00:00+00:00","updated":"2019-09-16T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/talks\/linux\/"}},"id":"https:\/\/jyn.dev\/talks\/linux\/","content":{"@attributes":{"type":"html"}}},{"title":"Intro to Python","published":"2019-09-11T00:00:00+00:00","updated":"2019-09-11T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/talks\/python\/"}},"id":"https:\/\/jyn.dev\/talks\/python\/","content":{"@attributes":{"type":"html"}}},{"title":"Fun with gaming on Linux","published":"2019-08-04T00:00:00+00:00","updated":"2019-08-04T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/fun-with-gaming-on-linux\/"}},"id":"https:\/\/jyn.dev\/fun-with-gaming-on-linux\/","summary":"How I got Multiplayer Civilization 5 to work on Linux","content":"<h2 id=\"the-problem\">The Problem<a class=\"zola-anchor\" href=\"#the-problem\" aria-label=\"Anchor link for: the-problem\"><\/a>\n<\/h2>\n<p>As my family frequently reminds me, most of my tech problems are of my own making.<\/p>\n<p>For instance, I wanted to play <a href=\"https:\/\/civilization.com\/civilization-5\/\">Civilization 5<\/a> with a friend of mine through Steam.\nI have a Linux\/Windows dual-boot with a shared\n<a href=\"https:\/\/en.wikipedia.org\/wiki\/NTFS\">NTFS<\/a>\n<a href=\"https:\/\/en.wikipedia.org\/wiki\/Disk_partitioning\">partition<\/a> so that I can share files between the operating systems.\nI run Linux by default.\nI also have Hibernate disabled on Windows because I can't use the shared partition\nfrom Linux if Windows is hibernated -\nWindows <a href=\"https:\/\/www.tuxera.com\/community\/ntfs-3g-faq\/#fullyshut\">doesn't see the new changes<\/a>.<\/p>\n<p>This led to a perfect storm of interactions, since running Civ required<\/p>\n<ul>\n<li>Rebooting the machine<\/li>\n<li>Waiting about 10 minutes for Windows to cold-start from a HDD (remember that Hibernate is disabled)<\/li>\n<li>Waiting another 5-10 minutes for Steam and Civ to launch<\/li>\n<\/ul>\n<p>Needless to say, this was frustrating for both my friend and I.<\/p>\n<h2 id=\"trying-new-things\">Trying new things<a class=\"zola-anchor\" href=\"#trying-new-things\" aria-label=\"Anchor link for: trying-new-things\"><\/a>\n<\/h2>\n<p>I found recently that <a href=\"https:\/\/fosspost.org\/tutorials\/enable-steam-play-on-linux-to-run-windows-games\">Steam has a compatibility layer<\/a>\nfor playing Windows games from other OSs.\nI downloaded it and gave it a try on a single-player game.\nIt works great, I have no complaints.<\/p>\n<p>The first time I tried to play with my friend, though, I got repeated issues saying\n'Could not connect to host'.\nRelated, I block outgoing network traffic by default at the firewall level.\nUnfortunately, Civ uses random ports to connect, so I can't whitelist it like I do for HTTP, SSH, etc.\nMy normal solution here is to have a group called <code>internet<\/code> which is whitelisted by\nIPTables (the native firewall software on Linux, similar to Windows Firewall).\nIf I want to, for example, chat via Discord, I run <code>sg internet discord<\/code>\nwhich allows unrestricted network traffic for that single Discord instance.<\/p>\n<p>When I tried this with Steam, however, I ran into all sorts of strange errors.\nSometimes Civ would launch but show network errors when I tried to connect.\nSometimes Steam wouldn't recognize my game library, mounted on the NTFS partition so I don't have to redownload 5 GB of assets when I reboot.\nI tried re-adding the library but got the message 'The partition where the game library is mounted must allow programs to be executed'.\nThis was exceedingly odd because I execute programs on that partition all the time.<\/p>\n<h2 id=\"strange-and-maddening-rules\">Strange and Maddening Rules<a class=\"zola-anchor\" href=\"#strange-and-maddening-rules\" aria-label=\"Anchor link for: strange-and-maddening-rules\"><\/a>\n<\/h2>\n<p>I finally tracked the error down to Steam trying to create and run a shell script unsuccessfully.\nFun fact: <code>strings steamclient.so<\/code> shows, among other things, a complete shell script:<\/p>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">...\n<\/span><span class=\"z-text z-plain\">.steam_exec_test.sh\n<\/span><span class=\"z-text z-plain\">#!\/bin\/sh\n<\/span><span class=\"z-text z-plain\">exit 0\n<\/span><span class=\"z-text z-plain\">Couldn&#39;t write %s: %s\n<\/span><span class=\"z-text z-plain\">...\n<\/span><\/code><\/pre>\n<p>The issue was that my NTFS drive only recognized me, UID 1000 and GID 1000, as a user.\nAny file written by any other user or group would have the ownership set to <code>root:root<\/code>.\nWorse yet, the OS would then deny all permissions to you (since you're not root).<\/p>\n<p>I spent about a half hour reading through lots of manuals about Linux &lt;=&gt; Windows permissions\n(here's <a href=\"https:\/\/www.tuxera.com\/community\/ntfs-3g-advanced\/ownership-and-permissions\/#usermapping\">the manual<\/a> if you're looking for it).\nThe solution I came up with was to explicitly add the <code>internet<\/code> group as me in <code>&lt;mountpoint&gt;\/.NTFS-3G\/UserMapping<\/code>.<\/p>\n<p>Now everything works!<\/p>\n<h2 id=\"even-more-info\">Even more info<a class=\"zola-anchor\" href=\"#even-more-info\" aria-label=\"Anchor link for: even-more-info\"><\/a>\n<\/h2>\n<ul>\n<li><a href=\"https:\/\/github.com\/jyn514\/dotfiles\/blob\/master\/lib\/iptables\">My IPTables configuration file<\/a><\/li>\n<li><a href=\"\/assets\/UserMapping.txt\">My <code>UserMapping<\/code> file<\/a><\/li>\n<\/ul>\n"},{"title":"Buffer Overflows and Stacks and Assembly, Oh My!","published":"2019-04-29T00:00:00+00:00","updated":"2019-04-29T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/buffer-overflows-and-stacks-and-assembly-oh-my\/"}},"id":"https:\/\/jyn.dev\/buffer-overflows-and-stacks-and-assembly-oh-my\/","content":"<p>Google has a 'Intro' Capture The Flag competition at\n<a href=\"https:\/\/capturetheflag.withgoogle.com\/#beginners\/\">https:\/\/capturetheflag.withgoogle.com\/#beginners\/<\/a>.\nI've been trying it out because, why not?\nThe challenges aren't easy by any means - they range from javascript crypto\n(let me know if you find out how to break JS Safe!) to SQL injections to steganography.\nThe challenge this post about, however, is a reverse engineering challenge: three, in fact.<\/p>\n<p>10-second summary of reverse engineering: you have a binary and no source code,\nand you want to exploit it somehow. In this case, the binary is across a network\nconnection, so you can't modify the code, you have to use what's there already.\nIf you want to try it yourself, the binary is <a href=\"\/assets\/mngmnt\">here<\/a>,\nit runs on 64-bit Linux 2.4+ (but may work on other machines, I haven't tried).\nI highly recommend going through the whole CTF first, though.<\/p>\n<h2 id=\"challenge-1\">Challenge 1<a class=\"zola-anchor\" href=\"#challenge-1\" aria-label=\"Anchor link for: challenge-1\"><\/a>\n<\/h2>\n<blockquote>\n<p>Do some black box testing here, it'll go well with your hat.\n<code>nc mngmnt-iface.ctfcompetition.com 1337<\/code><\/p>\n<\/blockquote>\n<p>When you connect, you get a screen saying<\/p>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">=== Management Interface ===\n<\/span><span class=\"z-text z-plain\"> 1) Service access\n<\/span><span class=\"z-text z-plain\"> 2) Read EULA\/patch notes\n<\/span><span class=\"z-text z-plain\"> 3) Quit\n<\/span><\/code><\/pre>\n<p>Let's read the patchnotes!<\/p>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">2\n<\/span><span class=\"z-text z-plain\">The following patchnotes were found:\n<\/span><span class=\"z-text z-plain\"> - Version0.2\n<\/span><span class=\"z-text z-plain\"> - Version0.3\n<\/span><span class=\"z-text z-plain\">Which patchnotes should be shown?\n<\/span><span class=\"z-text z-plain\">Version0.2\n<\/span><span class=\"z-text z-plain\"># Release 0.2\n<\/span><span class=\"z-text z-plain\"> - Updated library X to version 0.Y\n<\/span><span class=\"z-text z-plain\"> - Fixed path traversal bug\n<\/span><span class=\"z-text z-plain\"> - Improved the UX\n<\/span><span class=\"z-text z-plain\">=== Management Interface ===\n<\/span><span class=\"z-text z-plain\"> 1) Service access\n<\/span><span class=\"z-text z-plain\"> 2) Read EULA\/patch notes\n<\/span><span class=\"z-text z-plain\"> 3) Quit\n<\/span><span class=\"z-text z-plain\">2\n<\/span><span class=\"z-text z-plain\">The following patchnotes were found:\n<\/span><span class=\"z-text z-plain\"> - Version0.2\n<\/span><span class=\"z-text z-plain\"> - Version0.3\n<\/span><span class=\"z-text z-plain\">Which patchnotes should be shown?\n<\/span><span class=\"z-text z-plain\">Version0.3\n<\/span><span class=\"z-text z-plain\"># Version 0.3\n<\/span><span class=\"z-text z-plain\"> - Rollback of version 0.2 because of random reasons\n<\/span><span class=\"z-text z-plain\"> - Blah Blah\n<\/span><span class=\"z-text z-plain\"> - Fix random reboots at 2:32 every second Friday when it&#39;s new-moon.\n<\/span><\/code><\/pre>\n<p>Well, look at that. There's a path traversal bug:\nfixed in .2 and rolled back in .3 for 'random reasons'.\nA path traversal bug is when you have a public server which is meant to serve\nfiles in the starting directory, but instead servers files from the whole machine if\nyou use relative paths. An example:<\/p>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">=== Management Interface ===\n<\/span><span class=\"z-text z-plain\"> 1) Service access\n<\/span><span class=\"z-text z-plain\"> 2) Read EULA\/patch notes\n<\/span><span class=\"z-text z-plain\"> 3) Quit\n<\/span><span class=\"z-text z-plain\">2\n<\/span><span class=\"z-text z-plain\">The following patchnotes were found:\n<\/span><span class=\"z-text z-plain\"> - Version0.2\n<\/span><span class=\"z-text z-plain\"> - Version0.3\n<\/span><span class=\"z-text z-plain\">Which patchnotes should be shown?\n<\/span><span class=\"z-text z-plain\">..\/..\/..\/..\/..\/etc\/passwd\n<\/span><span class=\"z-text z-plain\">root:x:0:0:root:\/root:\/bin\/bash\n<\/span><span class=\"z-text z-plain\">...\n<\/span><span class=\"z-text z-plain\">user:x:1337:1337::\/home\/user:\n<\/span><\/code><\/pre>\n<p>From here you have a couple options to get the flag: you can either guess\n('fuzz') the filename, or you can download the binary and see what file it's\ntrying to open. Let's do the second.<\/p>\n<p>But wait, how do we get the binary? It turns out on linux there's a <code>proc<\/code> filesystem\nwhich <em>looks<\/em> like it has a bunch of files but really is a bunch of kernel info about\nthe running system. What that means for us is we can get the program without knowing the filename or current directory.<\/p>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">=== Management Interface ===\n<\/span><span class=\"z-text z-plain\"> 1) Service access\n<\/span><span class=\"z-text z-plain\"> 2) Read EULA\/patch notes\n<\/span><span class=\"z-text z-plain\"> 3) Quit\n<\/span><span class=\"z-text z-plain\">2\n<\/span><span class=\"z-text z-plain\">The following patchnotes were found:\n<\/span><span class=\"z-text z-plain\"> - Version0.2\n<\/span><span class=\"z-text z-plain\"> - Version0.3\n<\/span><span class=\"z-text z-plain\">Which patchnotes should be shown?\n<\/span><span class=\"z-text z-plain\">..\/..\/..\/..\/..\/proc\/self\/exe\n<\/span><span class=\"z-text z-plain\">ELF&gt;PAAA@X\ufffd@8\n<\/span><span class=\"z-text z-plain\">... more binary follows ...\n<\/span><\/code><\/pre>\n<p><code>\/proc<\/code> is the start of the filesystem, <code>self<\/code> is the directory for the currently running process\n(in this case the vulnerable program), and <code>exe<\/code> is the executable file as it was\nwhen the process started running. So now that we've got the binary, we can take a look at what it's doing.<\/p>\n<h3 id=\"enter-gdb\">Enter GDB<a class=\"zola-anchor\" href=\"#enter-gdb\" aria-label=\"Anchor link for: enter-gdb\"><\/a>\n<\/h3>\n<p>If you don't know how to use GDB, I highly recommend learning. Not only will it\nbe useful for debugging your own code, but it's the main way I know of to\nreverse engineer executables.<\/p>\n<p>The first thing I normally do is look at the main function:<\/p>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">gdb -q .\/mngmnt\n<\/span><span class=\"z-text z-plain\">Reading symbols from .\/mngmnt...done.\n<\/span><span class=\"z-text z-plain\">(gdb) break main\n<\/span><span class=\"z-text z-plain\">Breakpoint 1 at 0x41414660: file main.cpp, line 133.\n<\/span><span class=\"z-text z-plain\">(gdb) run\n<\/span><span class=\"z-text z-plain\">Starting program: \/home\/jyn\/Documents\/Programming\/challenges\/googlectf\/mngmnt\n<\/span><span class=\"z-text z-plain\">\n<\/span><span class=\"z-text z-plain\">Breakpoint 1, main (argc=1, argv=0x7fffffffdc28) at main.cpp:133\n<\/span><span class=\"z-text z-plain\">133     main.cpp: No such file or directory.\n<\/span><span class=\"z-text z-plain\">(gdb) disas\n<\/span><span class=\"z-text z-plain\">Dump of assembler code for function main(int, char**):\n<\/span><span class=\"z-text z-plain\">   0x0000000041414648 &lt;+0&gt;:     push   %rbp\n<\/span><span class=\"z-text z-plain\">   0x0000000041414649 &lt;+1&gt;:     mov    %rsp,%rbp\n<\/span><span class=\"z-text z-plain\">   0x000000004141464c &lt;+4&gt;:     sub    $0x1e0,%rsp\n<\/span><span class=\"z-text z-plain\">   0x0000000041414653 &lt;+11&gt;:    mov    %edi,-0x1d4(%rbp)\n<\/span><span class=\"z-text z-plain\">   0x0000000041414659 &lt;+17&gt;:    mov    %rsi,-0x1e0(%rbp)\n<\/span><span class=\"z-text z-plain\">=&gt; 0x0000000041414660 &lt;+24&gt;:    mov    0x201a99(%rip),%rax        # 0x41616100 &lt;stdin@@GLIBC_2.2.5&gt;\n<\/span><span class=\"z-text z-plain\">\n<\/span><span class=\"z-text z-plain\">... lots of assembly follows, about 600 lines ...\n<\/span><\/code><\/pre>\n<p>Well that's a lot to look at. There's a function called <code>primary_login<\/code>\nabout halfway down though, that looks promising.<\/p>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">(gdb) break primary_login()\n<\/span><span class=\"z-text z-plain\">Breakpoint 2 at 0x41414576: file main.cpp, line 112.\n<\/span><span class=\"z-text z-plain\">(gdb) continue\n<\/span><span class=\"z-text z-plain\">Continuing.\n<\/span><span class=\"z-text z-plain\">=== Management Interface ===\n<\/span><span class=\"z-text z-plain\"> 1) Service access\n<\/span><span class=\"z-text z-plain\"> 2) Read EULA\/patch notes\n<\/span><span class=\"z-text z-plain\"> 3) Quit\n<\/span><span class=\"z-text z-plain\">1\n<\/span><span class=\"z-text z-plain\">\n<\/span><span class=\"z-text z-plain\">Breakpoint 2, primary_login () at main.cpp:112\n<\/span><span class=\"z-text z-plain\">112     in main.cpp\n<\/span><span class=\"z-text z-plain\">(gdb) disas\n<\/span><span class=\"z-text z-plain\">Dump of assembler code for function primary_login():\n<\/span><span class=\"z-text z-plain\">   0x000000004141456b &lt;+0&gt;:     push   %rbp\n<\/span><span class=\"z-text z-plain\">   0x000000004141456c &lt;+1&gt;:     mov    %rsp,%rbp\n<\/span><span class=\"z-text z-plain\">   0x000000004141456f &lt;+4&gt;:     sub    $0x110,%rsp\n<\/span><span class=\"z-text z-plain\">=&gt; 0x0000000041414576 &lt;+11&gt;:    lea    0x62b(%rip),%rdi        # 0x41414ba8\n<\/span><span class=\"z-text z-plain\">   0x000000004141457d &lt;+18&gt;:    callq  0x400a90 &lt;puts@plt&gt;\n<\/span><span class=\"z-text z-plain\">   0x0000000041414582 &lt;+23&gt;:    mov    $0x0,%esi\n<\/span><span class=\"z-text z-plain\">   0x0000000041414587 &lt;+28&gt;:    lea    0x49e(%rip),%rdi        # 0x41414a2c &lt;_ZL9FLAG_FILE&gt;\n<\/span><span class=\"z-text z-plain\">   0x000000004141458e &lt;+35&gt;:    mov    $0x0,%eax\n<\/span><span class=\"z-text z-plain\">   0x0000000041414593 &lt;+40&gt;:    callq  0x400be0 &lt;open@plt&gt;\n<\/span><span class=\"z-text z-plain\">... about 200 more lines of assembly ...\n<\/span><\/code><\/pre>\n<p>What's <code>_ZL9FLAG_FILE<\/code>?<\/p>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">(gdb) printf &quot;%s\\n&quot;, 0x41414a2c\n<\/span><span class=\"z-text z-plain\">flag\n<\/span><\/code><\/pre>\n<p>After all that, that was it? Well, at least we know how to get the flag now.\nOne last trick - we can get the current directory of the running process with\n<code>\/proc\/self\/cwd<\/code>:<\/p>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">nc mngmnt-iface.ctfcompetition.com 1337\n<\/span><span class=\"z-text z-plain\">=== Management Interface ===\n<\/span><span class=\"z-text z-plain\"> 1) Service access\n<\/span><span class=\"z-text z-plain\"> 2) Read EULA\/patch notes\n<\/span><span class=\"z-text z-plain\"> 3) Quit\n<\/span><span class=\"z-text z-plain\">2\n<\/span><span class=\"z-text z-plain\">The following patchnotes were found:\n<\/span><span class=\"z-text z-plain\"> - Version0.2\n<\/span><span class=\"z-text z-plain\"> - Version0.3\n<\/span><span class=\"z-text z-plain\">Which patchnotes should be shown?\n<\/span><span class=\"z-text z-plain\">..\/..\/..\/..\/..\/proc\/self\/cwd\/flag\n<\/span><span class=\"z-text z-plain\">CTF{I_luv_buggy_sOFtware}\n<\/span><\/code><\/pre>\n<h2 id=\"challenge-2\">Challenge 2<a class=\"zola-anchor\" href=\"#challenge-2\" aria-label=\"Anchor link for: challenge-2\"><\/a>\n<\/h2>\n<p>That got our feet wet. The next challenge is a bit trickier.\nCan we get through the login screen?<\/p>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">nc mngmnt-iface.ctfcompetition.com 1337&#39;\n<\/span><span class=\"z-text z-plain\">=== Management Interface ===\n<\/span><span class=\"z-text z-plain\"> 1) Service access\n<\/span><span class=\"z-text z-plain\"> 2) Read EULA\/patch notes\n<\/span><span class=\"z-text z-plain\"> 3) Quit\n<\/span><span class=\"z-text z-plain\">1\n<\/span><span class=\"z-text z-plain\">Please enter the backdoo^Wservice password:\n<\/span><span class=\"z-text z-plain\">CTF{I_luv_buggy_sOFtware}\n<\/span><span class=\"z-text z-plain\">! Two factor authentication required !\n<\/span><span class=\"z-text z-plain\">Please enter secret secondary password:\n<\/span><\/code><\/pre>\n<p>Oh boy, what now? Let's look at the program again.<\/p>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">gdb .\/mngmnt\n<\/span><span class=\"z-text z-plain\">Reading symbols from .\/mngmnt...done.\n<\/span><span class=\"z-text z-plain\">(gdb) disas primary_login\n<\/span><span class=\"z-text z-plain\">Dump of assembler code for function primary_login():\n<\/span><span class=\"z-text z-plain\">... lots of code ...\n<\/span><span class=\"z-text z-plain\">   0x0000000041414627 &lt;+188&gt;:   jne    0x41414630 &lt;primary_login()+197&gt;\n<\/span><span class=\"z-text z-plain\">   0x0000000041414629 &lt;+190&gt;:   callq  0x41414446 &lt;secondary_login()&gt;\n<\/span><span class=\"z-text z-plain\">   0x000000004141462e &lt;+195&gt;:   jmp    0x41414646 &lt;primary_login()+219&gt;\n<\/span><span class=\"z-text z-plain\">   0x0000000041414630 &lt;+197&gt;:   lea    0x5b9(%rip),%rdi        # 0x41414bf0\n<\/span><span class=\"z-text z-plain\">   0x0000000041414637 &lt;+204&gt;:   callq  0x400a90 &lt;puts@plt&gt;\n<\/span><span class=\"z-text z-plain\">   0x000000004141463c &lt;+209&gt;:   mov    $0x1,%edi\n<\/span><span class=\"z-text z-plain\">---Type &lt;return&gt; to continue, or q &lt;return&gt; to quit---\n<\/span><span class=\"z-text z-plain\">   0x0000000041414641 &lt;+214&gt;:   callq  0x400aa0 &lt;exit@plt&gt;\n<\/span><span class=\"z-text z-plain\">   0x0000000041414646 &lt;+219&gt;:   leaveq\n<\/span><span class=\"z-text z-plain\">   0x0000000041414647 &lt;+220&gt;:   retq\n<\/span><span class=\"z-text z-plain\">End of assembler dump.\n<\/span><\/code><\/pre>\n<p><code>secondary_login<\/code> looks promising, what's that look like?\nNote: this is really long because we do need to read it this time.<\/p>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">(gdb) disas secondary_login\n<\/span><span class=\"z-text z-plain\">Dump of assembler code for function secondary_login():\n<\/span><span class=\"z-text z-plain\">   0x0000000041414446 &lt;+0&gt;:     push   %rbp\n<\/span><span class=\"z-text z-plain\">   0x0000000041414447 &lt;+1&gt;:     mov    %rsp,%rbp\n<\/span><span class=\"z-text z-plain\">   0x000000004141444a &lt;+4&gt;:     sub    $0x90,%rsp\n<\/span><span class=\"z-text z-plain\">   0x0000000041414451 &lt;+11&gt;:    lea    0x6d8(%rip),%rdi        # 0x41414b30\n<\/span><span class=\"z-text z-plain\">   0x0000000041414458 &lt;+18&gt;:    callq  0x400a90 &lt;puts@plt&gt;\n<\/span><span class=\"z-text z-plain\">   0x000000004141445d &lt;+23&gt;:    lea    0x6f4(%rip),%rdi        # 0x41414b58\n<\/span><span class=\"z-text z-plain\">   0x0000000041414464 &lt;+30&gt;:    callq  0x400a90 &lt;puts@plt&gt;\n<\/span><span class=\"z-text z-plain\">   0x0000000041414469 &lt;+35&gt;:    lea    -0x90(%rbp),%rax\n<\/span><span class=\"z-text z-plain\">   0x0000000041414470 &lt;+42&gt;:    mov    %rax,%rsi\n<\/span><span class=\"z-text z-plain\">   0x0000000041414473 &lt;+45&gt;:    lea    0x706(%rip),%rdi        # 0x41414b80\n<\/span><span class=\"z-text z-plain\">   0x000000004141447a &lt;+52&gt;:    mov    $0x0,%eax\n<\/span><span class=\"z-text z-plain\">   0x000000004141447f &lt;+57&gt;:    callq  0x400b00 &lt;scanf@plt&gt;\n<\/span><span class=\"z-text z-plain\">   0x0000000041414484 &lt;+62&gt;:    lea    -0x90(%rbp),%rax\n<\/span><span class=\"z-text z-plain\">   0x000000004141448b &lt;+69&gt;:    mov    %rax,%rdi\n<\/span><span class=\"z-text z-plain\">   0x000000004141448e &lt;+72&gt;:    callq  0x400b10 &lt;strlen@plt&gt;\n<\/span><span class=\"z-text z-plain\">   0x0000000041414493 &lt;+77&gt;:    mov    %rax,-0x10(%rbp)\n<\/span><span class=\"z-text z-plain\">   0x0000000041414497 &lt;+81&gt;:    movq   $0x0,-0x8(%rbp)\n<\/span><span class=\"z-text z-plain\">   0x000000004141449f &lt;+89&gt;:    mov    -0x8(%rbp),%rax\n<\/span><span class=\"z-text z-plain\">   0x00000000414144a3 &lt;+93&gt;:    cmp    -0x10(%rbp),%rax\n<\/span><span class=\"z-text z-plain\">   0x00000000414144a7 &lt;+97&gt;:    jae    0x414144d6 &lt;secondary_login()+144&gt;\n<\/span><span class=\"z-text z-plain\">   0x00000000414144a9 &lt;+99&gt;:    lea    -0x90(%rbp),%rdx\n<\/span><span class=\"z-text z-plain\">   0x00000000414144b0 &lt;+106&gt;:   mov    -0x8(%rbp),%rax\n<\/span><span class=\"z-text z-plain\">   0x00000000414144b4 &lt;+110&gt;:   add    %rdx,%rax\n<\/span><span class=\"z-text z-plain\">   0x00000000414144b7 &lt;+113&gt;:   movzbl (%rax),%eax\n<\/span><span class=\"z-text z-plain\">   0x00000000414144ba &lt;+116&gt;:   xor    $0xffffffc7,%eax\n<\/span><span class=\"z-text z-plain\">   0x00000000414144bd &lt;+119&gt;:   mov    %eax,%ecx\n<\/span><span class=\"z-text z-plain\">   0x00000000414144bf &lt;+121&gt;:   lea    -0x90(%rbp),%rdx\n<\/span><span class=\"z-text z-plain\">   0x00000000414144c6 &lt;+128&gt;:   mov    -0x8(%rbp),%rax\n<\/span><span class=\"z-text z-plain\">   0x00000000414144ca &lt;+132&gt;:   add    %rdx,%rax\n<\/span><span class=\"z-text z-plain\">   0x00000000414144cd &lt;+135&gt;:   mov    %cl,(%rax)\n<\/span><span class=\"z-text z-plain\">   0x00000000414144cf &lt;+137&gt;:   addq   $0x1,-0x8(%rbp)\n<\/span><span class=\"z-text z-plain\">   0x00000000414144d4 &lt;+142&gt;:   jmp    0x4141449f &lt;secondary_login()+89&gt;\n<\/span><span class=\"z-text z-plain\">   0x00000000414144d6 &lt;+144&gt;:   cmpq   $0x23,-0x10(%rbp)\n<\/span><span class=\"z-text z-plain\">   0x00000000414144db &lt;+149&gt;:   jne    0x41414537 &lt;secondary_login()+241&gt;\n<\/span><span class=\"z-text z-plain\">   0x00000000414144dd &lt;+151&gt;:   mov    0x55c(%rip),%rax        # 0x41414a40 &lt;_ZL4FLAG&gt;\n<\/span><span class=\"z-text z-plain\">   0x00000000414144e4 &lt;+158&gt;:   mov    0x55d(%rip),%rdx        # 0x41414a48 &lt;_ZL4FLAG+8&gt;\n<\/span><span class=\"z-text z-plain\">   0x00000000414144eb &lt;+165&gt;:   mov    %rax,-0x90(%rbp)\n<\/span><span class=\"z-text z-plain\">   0x00000000414144f2 &lt;+172&gt;:   mov    %rdx,-0x88(%rbp)\n<\/span><span class=\"z-text z-plain\">   0x00000000414144f9 &lt;+179&gt;:   mov    0x550(%rip),%rax        # 0x41414a50 &lt;_ZL4FLAG+16&gt;\n<\/span><span class=\"z-text z-plain\">   0x0000000041414500 &lt;+186&gt;:   mov    0x551(%rip),%rdx        # 0x41414a58 &lt;_ZL4FLAG+24&gt;\n<\/span><span class=\"z-text z-plain\">   0x0000000041414507 &lt;+193&gt;:   mov    %rax,-0x80(%rbp)\n<\/span><span class=\"z-text z-plain\">   0x000000004141450b &lt;+197&gt;:   mov    %rdx,-0x78(%rbp)\n<\/span><span class=\"z-text z-plain\">   0x000000004141450f &lt;+201&gt;:   movzwl 0x54a(%rip),%eax        # 0x41414a60 &lt;_ZL4FLAG+32&gt;\n<\/span><span class=\"z-text z-plain\">   0x0000000041414516 &lt;+208&gt;:   mov    %ax,-0x70(%rbp)\n<\/span><span class=\"z-text z-plain\">   0x000000004141451a &lt;+212&gt;:   movzbl 0x541(%rip),%eax        # 0x41414a62 &lt;_ZL4FLAG+34&gt;\n<\/span><span class=\"z-text z-plain\">   0x0000000041414521 &lt;+219&gt;:   mov    %al,-0x6e(%rbp)\n<\/span><span class=\"z-text z-plain\">   0x0000000041414524 &lt;+222&gt;:   lea    -0x90(%rbp),%rax\n<\/span><span class=\"z-text z-plain\">---Type &lt;return&gt; to continue, or q &lt;return&gt; to quit---\n<\/span><span class=\"z-text z-plain\">   0x000000004141452b &lt;+229&gt;:   test   %rax,%rax\n<\/span><span class=\"z-text z-plain\">   0x000000004141452e &lt;+232&gt;:   je     0x41414537 &lt;secondary_login()+241&gt;\n<\/span><span class=\"z-text z-plain\">   0x0000000041414530 &lt;+234&gt;:   mov    $0x1,%eax\n<\/span><span class=\"z-text z-plain\">   0x0000000041414535 &lt;+239&gt;:   jmp    0x4141453c &lt;secondary_login()+246&gt;\n<\/span><span class=\"z-text z-plain\">   0x0000000041414537 &lt;+241&gt;:   mov    $0x0,%eax\n<\/span><span class=\"z-text z-plain\">   0x000000004141453c &lt;+246&gt;:   test   %al,%al\n<\/span><span class=\"z-text z-plain\">   0x000000004141453e &lt;+248&gt;:   je     0x41414553 &lt;secondary_login()+269&gt;\n<\/span><span class=\"z-text z-plain\">   0x0000000041414540 &lt;+250&gt;:   lea    0x63f(%rip),%rdi        # 0x41414b86\n<\/span><span class=\"z-text z-plain\">   0x0000000041414547 &lt;+257&gt;:   callq  0x400a90 &lt;puts@plt&gt;\n<\/span><span class=\"z-text z-plain\">   0x000000004141454c &lt;+262&gt;:   callq  0x4141428e &lt;command_line()&gt;\n<\/span><span class=\"z-text z-plain\">   0x0000000041414551 &lt;+267&gt;:   jmp    0x41414569 &lt;secondary_login()+291&gt;\n<\/span><span class=\"z-text z-plain\">   0x0000000041414553 &lt;+269&gt;:   lea    0x63a(%rip),%rdi        # 0x41414b94\n<\/span><span class=\"z-text z-plain\">   0x000000004141455a &lt;+276&gt;:   callq  0x400a90 &lt;puts@plt&gt;\n<\/span><span class=\"z-text z-plain\">   0x000000004141455f &lt;+281&gt;:   mov    $0x1,%edi\n<\/span><span class=\"z-text z-plain\">   0x0000000041414564 &lt;+286&gt;:   callq  0x400aa0 &lt;exit@plt&gt;\n<\/span><span class=\"z-text z-plain\">   0x0000000041414569 &lt;+291&gt;:   leaveq\n<\/span><span class=\"z-text z-plain\">   0x000000004141456a &lt;+292&gt;:   retq\n<\/span><\/code><\/pre>\n<p>There's a few things of note here. First is the <code>command_line<\/code> function,\nwhich looks promising. Next is <code>_ZL4FLAG<\/code>, which seems useful. What's there?<\/p>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">(gdb) printf &quot;%s\\n&quot;, 0x41414a40\n<\/span><span class=\"z-text z-plain\">\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\/bin\/sh\n<\/span><\/code><\/pre>\n<p>That's not helpful, it's binary data. And why does it have a shell at the end there?\nAnd finally, how do we get to <code>command_line<\/code>? Let's trace the execution.\nIf you've heard of 'basic blocks' in compilers, that's basically what we're going\nto do now: split up the code into segments divided by jumps.<\/p>\n<p>First we start at +0. Then execution is linear until line +93, where there's a\n<code>cmp<\/code> (compare) and a <code>jae<\/code> (jump above equal) to +144. At 144, it compares\n-0x10(%rbp) to 0x23, or 35 in decimal.<\/p>\n<p>That's funny, the number of <code>\ufffd<\/code> characters in the data above is 35.\nCoincidence? I think not. What happens if we just put in a string with 35 characters?<\/p>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">! Two factor authentication required !\n<\/span><span class=\"z-text z-plain\">Please enter secret secondary password:\n<\/span><span class=\"z-text z-plain\">aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\n<\/span><span class=\"z-text z-plain\">Authenticated\n<\/span><span class=\"z-text z-plain\">&gt;\n<\/span><\/code><\/pre>\n<p>Wait, what? That's it? Where's the flag?\nIt turns out this program forgot to check that you had the right password.\nI couldn't figure out what was going on, so I cheated and looked at\nhttps:\/\/jhalon.github.io\/2018-google-ctf-beginners-pwn-solutions-1\/\n(which is a much better writeup). Long story short, there's an inner loop\ndoing funny things with <code>xor<\/code>. If we xor the binary at <code>_ZL4FLAG<\/code> with\n<code>0xc7<\/code>, we get back the flag. GDB and python will both be helpful here.<\/p>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">(gdb) print\/x _ZL4FLAG\n<\/span><span class=\"z-text z-plain\">$5 = {0x84, 0x93, 0x81, 0xbc, 0x93, 0xb0, 0xa8, 0x98, 0x97, 0xa6, 0xb4, 0x94, 0xb0, 0xa8,\n<\/span><span class=\"z-text z-plain\">  0xb5, 0x83, 0xbd, 0x98, 0x85, 0xa2, 0xb3, 0xb3, 0xa2, 0xb5, 0x98, 0xb3, 0xaf, 0xf3, 0xa9,\n<\/span><span class=\"z-text z-plain\">  0x98, 0xf6, 0x98, 0xac, 0xf8, 0xba}\n<\/span><span class=\"z-text z-plain\">(gdb) quit\n<\/span><span class=\"z-text z-plain\">$ python -q\n<\/span><span class=\"z-text z-plain\">&gt;&gt;&gt; import re\n<\/span><span class=\"z-text z-plain\">&gt;&gt;&gt; hexes = &quot;&quot;&quot;0x84, 0x93, 0x81, 0xbc, 0x93, 0xb0, 0xa8, 0x98, 0x97, 0xa6, 0xb4, 0x94, 0xb0, 0xa8,\n<\/span><span class=\"z-text z-plain\">...   0xb5, 0x83, 0xbd, 0x98, 0x85, 0xa2, 0xb3, 0xb3, 0xa2, 0xb5, 0x98, 0xb3, 0xaf, 0xf3, 0xa9,\n<\/span><span class=\"z-text z-plain\">...   0x98, 0xf6, 0x98, 0xac, 0xf8, 0xba&quot;&quot;&quot;\n<\/span><span class=\"z-text z-plain\">&gt;&gt;&gt; chars = &#39;&#39;.join(hexes.split()).replace(&#39;,&#39;, &#39;&#39;).replace(&#39;0x&#39;, &#39;&#39;)\n<\/span><span class=\"z-text z-plain\">&gt;&gt;&gt; chars\n<\/span><span class=\"z-text z-plain\">&#39;849381bc93b0a89897a6b494b0a8b583bd9885a2b3b3a2b598b3aff3a998f698acf8ba&#39;\n<\/span><span class=\"z-text z-plain\">&gt;&gt;&gt; &#39;&#39;.join([chr(int(c, 16) ^ 0xc7) for c in re.findall(&#39;..&#39;, chars)])\n<\/span><span class=\"z-text z-plain\">&#39;CTF{Two_PasSworDz_Better_th4n_1_k?}&#39;\n<\/span><\/code><\/pre>\n<p>Quick explanation of that awful and hacky code:\n<code>print\/x<\/code> means print as hexidecimal instead of decimal or binary.\n<code>''.join(hexes.split())<\/code> is a hacky way of removing whitespace.\nThe <code>replace<\/code> calls are just getting rid of formatting we don't care about.\n<code>re.findall('..', chars)<\/code> splits the strings into 2-character sequences.\nThe rest is a list comprehesion which converts each string to an int,\nxors it with hexidecimal c7, then finally joins the whole list into a string.<\/p>\n<p>Side note: originally this was even more hacky - I used gdb's <code>&lt;address&gt;@n<\/code> syntax\nwhich gives you back the memory in architecture-endian order. 1\/10 do not recommend.<\/p>\n<!--\n\n## Challenge 3\n> It's a sure bet that they can't handle their memory properly\n\nThe last challenge was also the hardest to exploit. Once you enter the second flag,\nyou get a shell without any instructions:\n\n```\n$  .\/mngmnt\n=== Management Interface ===\n 1) Service access\n 2) Read EULA\/patch notes\n 3) Quit\n1\nPlease enter the backdoo^Wservice password:\nflag\n! Two factor authentication required !\nPlease enter secret secondary password:\nCTF{Two_PasSworDz_Better_th4n_1_k?}\nAuthenticated\n> help\nUnknown command 'help'\n> ls\nUnknown command 'ls'\n> whoami\nUnknown command 'whoami'\n```\n\nLet's see if we can find what's going on. Looking back up at the assembly,\nwe see that it calls `command_line` once you put in the second password.\nWhat's that doing?\n\n```\ngdb .\/mngmnt\nReading symbols from .\/mngmnt...done.\n(gdb) disas command_line\nDump of assembler code for function command_line():\n   0x000000004141428e <+0>:     push   %rbp\n   0x000000004141428f <+1>:     mov    %rsp,%rbp\n   0x0000000041414292 <+4>:     push   %rbx\n   0x0000000041414293 <+5>:     sub    $0x128,%rsp\n   0x000000004141429a <+12>:    lea    0x7ca(%rip),%rdi        # 0x41414a6b\n   0x00000000414142a1 <+19>:    mov    $0x0,%eax\n   0x00000000414142a6 <+24>:    callq  0x400a70 <printf@plt>\n   0x00000000414142ab <+29>:    lea    -0x30(%rbp),%rax\n   0x00000000414142af <+33>:    mov    %rax,%rdi\n   0x00000000414142b2 <+36>:    callq  0x4141423a <getsx(char*)>\n   0x00000000414142b7 <+41>:    mov    0x201e77(%rip),%eax        # 0x41616134 <_ZL13cmds_executed>\n   0x00000000414142bd <+47>:    add    $0x1,%eax\n   0x00000000414142c0 <+50>:    mov    %eax,0x201e6e(%rip)        # 0x41616134 <_ZL13cmds_executed>\n   0x00000000414142c6 <+56>:    lea    -0x30(%rbp),%rax\n   0x00000000414142ca <+60>:    lea    0x79d(%rip),%rsi        # 0x41414a6e\n   0x00000000414142d1 <+67>:    mov    %rax,%rdi\n   0x00000000414142d4 <+70>:    callq  0x400ba0 <strcmp@plt>\n   0x00000000414142d9 <+75>:    test   %eax,%eax\n   0x00000000414142db <+77>:    jne    0x414142ee <command_line()+96>\n   0x00000000414142dd <+79>:    lea    0x78f(%rip),%rdi        # 0x41414a73\n   0x00000000414142e4 <+86>:    callq  0x400a90 <puts@plt>\n   0x00000000414142e9 <+91>:    jmpq   0x4141443c <command_line()+430>\n   0x00000000414142ee <+96>:    lea    -0x30(%rbp),%rax\n   0x00000000414142f2 <+100>:   lea    0x77f(%rip),%rsi        # 0x41414a78\n   0x00000000414142f9 <+107>:   mov    %rax,%rdi\n   0x00000000414142fc <+110>:   callq  0x400ba0 <strcmp@plt>\n   0x0000000041414301 <+115>:   test   %eax,%eax\n   0x0000000041414303 <+117>:   jne    0x41414313 <command_line()+133>\n   0x0000000041414305 <+119>:   lea    0x774(%rip),%rdi        # 0x41414a80\n   0x000000004141430c <+126>:   callq  0x400a90 <puts@plt>\n... snip ...\n   0x00000000414143ff <+369>:   lea    0x708(%rip),%rdi        # 0x41414b0e\n   0x0000000041414406 <+376>:   callq  0x400a90 <puts@plt>\n   0x000000004141440b <+381>:   lea    -0x130(%rbp),%rax\n   0x0000000041414412 <+388>:   mov    %rax,%rdi\n   0x0000000041414415 <+391>:   callq  0x400ae0 <system@plt>\n   0x000000004141441a <+396>:   jmpq   0x4141429a <command_line()+12>\n---Type <return> to continue, or q <return> to quit---\n   0x000000004141441f <+401>:   lea    -0x30(%rbp),%rax\n   0x0000000041414423 <+405>:   mov    %rax,%rsi\n   0x0000000041414426 <+408>:   lea    0x6ec(%rip),%rdi        # 0x41414b19\n   0x000000004141442d <+415>:   mov    $0x0,%eax\n   0x0000000041414432 <+420>:   callq  0x400a70 <printf@plt>\n   0x0000000041414437 <+425>:   jmpq   0x4141429a <command_line()+12>\n   0x000000004141443c <+430>:   add    $0x128,%rsp\n   0x0000000041414443 <+437>:   pop    %rbx\n   0x0000000041414444 <+438>:   pop    %rbp\n   0x0000000041414445 <+439>:   retq\n```\n\nThis looks long and compilicated, but it's really doing the same thing\nover and over: comparing a string to a bunch of other strings.\nThat begs the question: What other strings?\n\n```\n(gdb) printf \"%s\\n\", 0x41414a6e\nquit\n(gdb) printf \"%s\\n\", 0x41414a78\nversion\n(gdb) printf \"%s\\n\", 0x41414a8c\nshell\n(gdb) printf \"%s\\n\", 0x41414ac3\necho\n(gdb) printf \"%s\\n\", 0x41414ac8\ndebug\n```\n\nLet's see what these do.\n\n```\n> version\nVersion 0.3\n> shell\nSecurity made us disable the shell, sorry!\n> echo hi\nhi\n> debug\nDebug data dump:\n pid=30575 cmds executed=0x41616134->8 Mappings:\n\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\n       \u2502 File: \/proc\/30575\/maps\n\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\n   1   \u2502 00400000-00401000 r-xp 00000000 08:04 315016                             \/home\/jyn\/Documents\/Programming\/challenges\/googlectf\/mngmnt\n   2   \u2502 41414000-41415000 r-xp 00014000 08:04 315016                             \/home\/jyn\/Documents\/Programming\/challenges\/googlectf\/mngmnt\n   3   \u2502 41615000-41616000 r--p 00015000 08:04 315016                             \/home\/jyn\/Documents\/Programming\/challenges\/googlectf\/mngmnt\n   4   \u2502 41616000-41617000 rw-p 00016000 08:04 315016                             \/home\/jyn\/Documents\/Programming\/challenges\/googlectf\/mngmnt\n   5   \u2502 41617000-41638000 rw-p 00000000 00:00 0                                  [heap]\n   6   \u2502 7ffff70a5000-7ffff728c000 r-xp 00000000 08:05 1840331                    \/lib\/x86_64-linux-gnu\/libc-2.27.so\n   7   \u2502 7ffff728c000-7ffff748c000 ---p 001e7000 08:05 1840331                    \/lib\/x86_64-linux-gnu\/libc-2.27.so\n   8   \u2502 7ffff748c000-7ffff7490000 r--p 001e7000 08:05 1840331                    \/lib\/x86_64-linux-gnu\/libc-2.27.so\n   9   \u2502 7ffff7490000-7ffff7492000 rw-p 001eb000 08:05 1840331                    \/lib\/x86_64-linux-gnu\/libc-2.27.so\n  10   \u2502 7ffff7492000-7ffff7496000 rw-p 00000000 00:00 0 \n  11   \u2502 7ffff7496000-7ffff74ad000 r-xp 00000000 08:05 1840694                    \/lib\/x86_64-linux-gnu\/libgcc_s.so.1\n  12   \u2502 7ffff74ad000-7ffff76ac000 ---p 00017000 08:05 1840694                    \/lib\/x86_64-linux-gnu\/libgcc_s.so.1\n  13   \u2502 7ffff76ac000-7ffff76ad000 r--p 00016000 08:05 1840694                    \/lib\/x86_64-linux-gnu\/libgcc_s.so.1\n  14   \u2502 7ffff76ad000-7ffff76ae000 rw-p 00017000 08:05 1840694                    \/lib\/x86_64-linux-gnu\/libgcc_s.so.1\n  15   \u2502 7ffff76ae000-7ffff784b000 r-xp 00000000 08:05 1840394                    \/lib\/x86_64-linux-gnu\/libm-2.27.so\n  16   \u2502 7ffff784b000-7ffff7a4a000 ---p 0019d000 08:05 1840394                    \/lib\/x86_64-linux-gnu\/libm-2.27.so\n  17   \u2502 7ffff7a4a000-7ffff7a4b000 r--p 0019c000 08:05 1840394                    \/lib\/x86_64-linux-gnu\/libm-2.27.so\n  18   \u2502 7ffff7a4b000-7ffff7a4c000 rw-p 0019d000 08:05 1840394                    \/lib\/x86_64-linux-gnu\/libm-2.27.so\n  19   \u2502 7ffff7a4c000-7ffff7bc5000 r-xp 00000000 08:05 1706663                    \/usr\/lib\/x86_64-linux-gnu\/libstdc++.so.6.0.25\n  20   \u2502 7ffff7bc5000-7ffff7dc5000 ---p 00179000 08:05 1706663                    \/usr\/lib\/x86_64-linux-gnu\/libstdc++.so.6.0.25\n  21   \u2502 7ffff7dc5000-7ffff7dcf000 r--p 00179000 08:05 1706663                    \/usr\/lib\/x86_64-linux-gnu\/libstdc++.so.6.0.25\n  22   \u2502 7ffff7dcf000-7ffff7dd1000 rw-p 00183000 08:05 1706663                    \/usr\/lib\/x86_64-linux-gnu\/libstdc++.so.6.0.25\n  23   \u2502 7ffff7dd1000-7ffff7dd5000 rw-p 00000000 00:00 0 \n  24   \u2502 7ffff7dd5000-7ffff7dfc000 r-xp 00000000 08:05 1840303                    \/lib\/x86_64-linux-gnu\/ld-2.27.so\n  25   \u2502 7ffff7fc4000-7ffff7fca000 rw-p 00000000 00:00 0 \n  26   \u2502 7ffff7ff7000-7ffff7ffa000 r--p 00000000 00:00 0                          [vvar]\n  27   \u2502 7ffff7ffa000-7ffff7ffc000 r-xp 00000000 00:00 0                          [vdso]\n  28   \u2502 7ffff7ffc000-7ffff7ffd000 r--p 00027000 08:05 1840303                    \/lib\/x86_64-linux-gnu\/ld-2.27.so\n  29   \u2502 7ffff7ffd000-7ffff7ffe000 rw-p 00028000 08:05 1840303                    \/lib\/x86_64-linux-gnu\/ld-2.27.so\n  30   \u2502 7ffff7ffe000-7ffff7fff000 rw-p 00000000 00:00 0 \n  31   \u2502 7ffffffdd000-7ffffffff000 rw-p 00000000 00:00 0                          [stack]\n  32   \u2502 ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]\n> quit\nBye!\n```\n\nWell, no dice. That debug info is a little strange.\nAnd why have a shell option if it's disabled?\nLet's look at that more closely.\n\n```\n   0x0000000041414317 <+137>:   lea    0x76e(%rip),%rsi        # 0x41414a8c\n   0x000000004141431e <+144>:   mov    %rax,%rdi\n   0x0000000041414321 <+147>:   callq  0x400ba0 <strcmp@plt>\n   0x0000000041414326 <+152>:   test   %eax,%eax\n   0x0000000041414328 <+154>:   jne    0x41414353 <command_line()+197>\n   0x000000004141432a <+156>:   movzbl 0x201e07(%rip),%eax        # 0x41616138 <_ZL13shell_enabled>\n   0x0000000041414331 <+163>:   xor    $0x1,%eax\n   0x0000000041414334 <+166>:   test   %al,%al\n   0x0000000041414336 <+168>:   je     0x41414349 <command_line()+187>\n   0x0000000041414338 <+170>:   lea    0x759(%rip),%rdi        # 0x41414a98\n   0x000000004141433f <+177>:   callq  0x400a90 <puts@plt>\n   0x0000000041414344 <+182>:   jmpq   0x4141429a <command_line()+12>\n   0x0000000041414349 <+187>:   callq  0x41414227 <debug_shell()>\n```\n\nThis is basically doing three things - checking if we input 'shell',\nchecking if `_ZL13shell_enabled` is 0, and if so launching a shell on the host.\nUnfortunately, `_ZL13shell_enabled` is _always_ 0. What can we do?\n\nWell, the challenge description mentioned memory.\nCould there be a buffer overrun somewhere? Let's open gdb and check it out.\nFor this bit I'm actually going to use a gdb addon called\n[peda](https:\/\/github.com\/longld\/peda) which makes debugging much nicer\nwithout changing any behaviour.\n\nWhat happens if we put in a really long string?\n\n```\n\n!-->\n<p>(Challenge 3 will probably not be forthcoming, I finished it but never got around to writing it up.)<\/p>\n"},{"title":"Reverse Engineering x86 assembly","published":"2018-09-29T00:00:00+00:00","updated":"2018-09-29T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/reverse-engineering-x86-assembly\/"}},"id":"https:\/\/jyn.dev\/reverse-engineering-x86-assembly\/","summary":"Basics of decompiling binaries on linux with GDB","content":"<h2 id=\"intro\">Intro<a class=\"zola-anchor\" href=\"#intro\" aria-label=\"Anchor link for: intro\"><\/a>\n<\/h2>\n<p>For those of you not familiar with the C toolchain, it goes something like this:<\/p>\n<p>Source -&gt; preprocessed source -&gt; assembly -&gt; object file -&gt; binary<\/p>\n<p>The second step (preprocessed -&gt; assembly) is the hardest to undo.\nIt removes labels, debug symbols, and turns a high-level control flow into assembly instructions.\nHowever, since you <em>need<\/em> a binary to run a program, if you can undo that step,\nyou can see what's going on.<\/p>\n<h2 id=\"gdb-basics\">GDB Basics<a class=\"zola-anchor\" href=\"#gdb-basics\" aria-label=\"Anchor link for: gdb-basics\"><\/a>\n<\/h2>\n<p>GDB is a debugger. It steps through a compiled program, line by line, and shows you\nthe changes after every step.<\/p>\n<p>Let's take a simple hello-world program and step through it.<\/p>\n<pre data-lang=\"C\" class=\"language-C z-code\"><code class=\"language-C\" data-lang=\"C\"><span class=\"z-source z-c\"><span class=\"z-meta z-preprocessor z-include z-c\"><span class=\"z-keyword z-control z-import z-include z-c\">#include<\/span> <span class=\"z-string z-quoted z-other z-lt-gt z-include z-c\"><span class=\"z-punctuation z-definition z-string z-begin z-c\">&lt;<\/span>stdio.h<span class=\"z-punctuation z-definition z-string z-end z-c\">&gt;<\/span><\/span>\n<\/span><\/span><span class=\"z-source z-c\">\n<\/span><span class=\"z-source z-c\"><span class=\"z-storage z-type z-c\">int<\/span> <span class=\"z-meta z-function z-c\"><span class=\"z-entity z-name z-function z-c\">main<\/span><\/span><span class=\"z-meta z-function z-parameters z-c\"><span class=\"z-meta z-group z-c\"><span class=\"z-punctuation z-section z-group z-begin z-c\">(<\/span><\/span><\/span><span class=\"z-meta z-function z-parameters z-c\"><span class=\"z-meta z-group z-c\"><span class=\"z-storage z-type z-c\">void<\/span><span class=\"z-punctuation z-section z-group z-end z-c\">)<\/span><\/span><\/span><span class=\"z-meta z-function z-c\"> <\/span><span class=\"z-meta z-function z-c\"><span class=\"z-meta z-block z-c\"><span class=\"z-punctuation z-section z-block z-begin z-c\">{<\/span><\/span><\/span><span class=\"z-meta z-function z-c\"><span class=\"z-meta z-block z-c\">\n<\/span><\/span><\/span><span class=\"z-source z-c\"><span class=\"z-meta z-function z-c\"><span class=\"z-meta z-block z-c\">  <span class=\"z-meta z-function-call z-c\"><span class=\"z-support z-function z-C99 z-c\">puts<\/span><span class=\"z-meta z-group z-c\"><span class=\"z-punctuation z-section z-group z-begin z-c\">(<\/span><\/span><\/span><span class=\"z-meta z-function-call z-c\"><span class=\"z-meta z-group z-c\"><span class=\"z-string z-quoted z-double z-c\"><span class=\"z-punctuation z-definition z-string z-begin z-c\">&quot;<\/span>Hello, world!<span class=\"z-punctuation z-definition z-string z-end z-c\">&quot;<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-c\"><span class=\"z-meta z-group z-c\"><span class=\"z-punctuation z-section z-group z-end z-c\">)<\/span><\/span><\/span><span class=\"z-punctuation z-terminator z-c\">;<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-c\"><span class=\"z-meta z-function z-c\"><span class=\"z-meta z-block z-c\">  <span class=\"z-keyword z-control z-flow z-return z-c\">return<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-c\">0<\/span><span class=\"z-punctuation z-terminator z-c\">;<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-c\"><span class=\"z-meta z-function z-c\"><span class=\"z-meta z-block z-c\"><\/span><\/span><span class=\"z-meta z-function z-c\"><span class=\"z-meta z-block z-c\"><span class=\"z-punctuation z-section z-block z-end z-c\">}<\/span><\/span><\/span>\n<\/span><\/code><\/pre>\n<p>Passing <code>-g<\/code> to <code>gcc<\/code> means preserve debug symbols, so we know where we are in the original source code.\nPassing <code>-q<\/code> to <code>gdb<\/code> means don't print 10 lines of copyright info.\nNote that we have to add a breakpoint in the main function, or it will run without ever stopping.<\/p>\n<pre data-lang=\"sh\" class=\"language-sh z-code\"><code class=\"language-sh\" data-lang=\"sh\"><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">$<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> gcc<span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> -<\/span>g<\/span> hello_world.c<span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> -<\/span>o<\/span> hello<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">$<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> gdb<span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> -<\/span>q<\/span> hello<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">Reading<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> symbols from hello...done.<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-punctuation z-definition z-compound z-begin z-shell\">(<\/span><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">gdb<\/span><\/span><span class=\"z-punctuation z-definition z-compound z-end z-shell\">)<\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> list<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">1       <span class=\"z-comment z-line z-number-sign z-shell\"><span class=\"z-punctuation z-definition z-comment z-begin z-shell\">#<\/span><\/span><span class=\"z-comment z-line z-number-sign z-shell\">include &lt;stdio.h&gt;<\/span><span class=\"z-comment z-line z-number-sign z-shell\">\n<\/span><\/span><\/span><\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">2<\/span><\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">3<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\">       int main(void<\/span><span class=\"z-meta z-function-call z-shell\"><\/span>) <span class=\"z-punctuation z-definition z-compound z-braces z-begin z-shell\">{<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">4<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\">         puts(<span class=\"z-string z-quoted z-double z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&quot;<\/span>Hello, world!<span class=\"z-punctuation z-definition z-string z-end z-shell\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-function-call z-shell\"><\/span>)<span class=\"z-keyword z-operator z-logical z-continue z-shell\">;<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">5<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\">         return 0<\/span><span class=\"z-keyword z-operator z-logical z-continue z-shell\">;<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">6<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\">       <\/span><span class=\"z-punctuation z-definition z-compound z-braces z-end z-shell\">}<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">7<\/span><\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-punctuation z-definition z-compound z-begin z-shell\">(<\/span><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">gdb<\/span><\/span><span class=\"z-punctuation z-definition z-compound z-end z-shell\">)<\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> break main<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">Breakpoint<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> 1 at 0x4004db: file hello_world.c, line 4.<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-punctuation z-definition z-compound z-begin z-shell\">(<\/span><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">gdb<\/span><\/span><span class=\"z-punctuation z-definition z-compound z-end z-shell\">)<\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> run<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">Starting<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> program: \/home\/jyn\/hello<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">Breakpoint<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> 1, main (<\/span><span class=\"z-meta z-function-call z-shell\"><\/span>) <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">at<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> tmp.c:4<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">4<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\">         puts(<span class=\"z-string z-quoted z-double z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&quot;<\/span>Hello, world!<span class=\"z-punctuation z-definition z-string z-end z-shell\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-function-call z-shell\"><\/span>)<span class=\"z-keyword z-operator z-logical z-continue z-shell\">;<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-punctuation z-definition z-compound z-begin z-shell\">(<\/span><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">gdb<\/span><\/span><span class=\"z-punctuation z-definition z-compound z-end z-shell\">)<\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> next<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">Hello,<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> world!<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">5<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\">         return 0<\/span><span class=\"z-keyword z-operator z-logical z-continue z-shell\">;<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-punctuation z-definition z-compound z-begin z-shell\">(<\/span><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">gdb<\/span><\/span><span class=\"z-punctuation z-definition z-compound z-end z-shell\">)<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">6<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\">       <\/span><span class=\"z-meta z-function-call z-shell\"><\/span>}\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-punctuation z-definition z-compound z-begin z-shell\">(<\/span><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">gdb<\/span><\/span><span class=\"z-punctuation z-definition z-compound z-end z-shell\">)<\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> continue<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">Continuing.<\/span><\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">[Inferior<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> 1 (process 28690<\/span><span class=\"z-meta z-function-call z-shell\"><\/span>) <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">exited<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> normally]<\/span>\n<\/span><\/code><\/pre>\n<p>You can also show variables as you step through.\n(Note that <code>return 0<\/code> is optional in the main function)<\/p>\n<pre data-lang=\"C\" class=\"language-C z-code\"><code class=\"language-C\" data-lang=\"C\"><span class=\"z-source z-c\"><span class=\"z-meta z-preprocessor z-include z-c\"><span class=\"z-keyword z-control z-import z-include z-c\">#include<\/span> <span class=\"z-string z-quoted z-other z-lt-gt z-include z-c\"><span class=\"z-punctuation z-definition z-string z-begin z-c\">&lt;<\/span>stdio.h<span class=\"z-punctuation z-definition z-string z-end z-c\">&gt;<\/span><\/span>\n<\/span><\/span><span class=\"z-source z-c\">\n<\/span><span class=\"z-source z-c\"><span class=\"z-storage z-type z-c\">int<\/span> <span class=\"z-meta z-function z-c\"><span class=\"z-entity z-name z-function z-c\">main<\/span><\/span><span class=\"z-meta z-function z-parameters z-c\"><span class=\"z-meta z-group z-c\"><span class=\"z-punctuation z-section z-group z-begin z-c\">(<\/span><\/span><\/span><span class=\"z-meta z-function z-parameters z-c\"><span class=\"z-meta z-group z-c\"><span class=\"z-storage z-type z-c\">void<\/span><span class=\"z-punctuation z-section z-group z-end z-c\">)<\/span><\/span><\/span><span class=\"z-meta z-function z-c\"> <\/span><span class=\"z-meta z-function z-c\"><span class=\"z-meta z-block z-c\"><span class=\"z-punctuation z-section z-block z-begin z-c\">{<\/span><\/span><\/span><span class=\"z-meta z-function z-c\"><span class=\"z-meta z-block z-c\">\n<\/span><\/span><\/span><span class=\"z-source z-c\"><span class=\"z-meta z-function z-c\"><span class=\"z-meta z-block z-c\">  <span class=\"z-storage z-type z-c\">char<\/span> <span class=\"z-keyword z-operator z-c\">*<\/span>var <span class=\"z-keyword z-operator z-assignment z-c\">=<\/span> <span class=\"z-string z-quoted z-double z-c\"><span class=\"z-punctuation z-definition z-string z-begin z-c\">&quot;<\/span>Hello, world!<span class=\"z-punctuation z-definition z-string z-end z-c\">&quot;<\/span><\/span><span class=\"z-punctuation z-terminator z-c\">;<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-c\"><span class=\"z-meta z-function z-c\"><span class=\"z-meta z-block z-c\">  var <span class=\"z-keyword z-operator z-assignment z-c\">=<\/span> <span class=\"z-string z-quoted z-double z-c\"><span class=\"z-punctuation z-definition z-string z-begin z-c\">&quot;<\/span>changed my mind<span class=\"z-punctuation z-definition z-string z-end z-c\">&quot;<\/span><\/span><span class=\"z-punctuation z-terminator z-c\">;<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-c\"><span class=\"z-meta z-function z-c\"><span class=\"z-meta z-block z-c\">  <span class=\"z-meta z-function-call z-c\"><span class=\"z-support z-function z-C99 z-c\">puts<\/span><span class=\"z-meta z-group z-c\"><span class=\"z-punctuation z-section z-group z-begin z-c\">(<\/span><\/span><\/span><span class=\"z-meta z-function-call z-c\"><span class=\"z-meta z-group z-c\">var<\/span><\/span><span class=\"z-meta z-function-call z-c\"><span class=\"z-meta z-group z-c\"><span class=\"z-punctuation z-section z-group z-end z-c\">)<\/span><\/span><\/span><span class=\"z-punctuation z-terminator z-c\">;<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-c\"><span class=\"z-meta z-function z-c\"><span class=\"z-meta z-block z-c\"><\/span><\/span><span class=\"z-meta z-function z-c\"><span class=\"z-meta z-block z-c\"><span class=\"z-punctuation z-section z-block z-end z-c\">}<\/span><\/span><\/span>\n<\/span><\/code><\/pre>\n<pre data-lang=\"sh\" class=\"language-sh z-code\"><code class=\"language-sh\" data-lang=\"sh\"><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">$<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> gcc<span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> -<\/span>g<\/span> var.c<span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> -<\/span>o<\/span> var<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">$<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> gdb<span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> -<\/span>q<\/span> var<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">Reading<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> symbols from var...done.<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-punctuation z-definition z-compound z-begin z-shell\">(<\/span><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">gdb<\/span><\/span><span class=\"z-punctuation z-definition z-compound z-end z-shell\">)<\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> break main<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">Breakpoint<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> 1 at 0x4004df: file var.c, line 4.<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-punctuation z-definition z-compound z-begin z-shell\">(<\/span><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">gdb<\/span><\/span><span class=\"z-punctuation z-definition z-compound z-end z-shell\">)<\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> run<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">Starting<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> program: \/home\/jyn\/var<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">Breakpoint<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> 1, main (<\/span><span class=\"z-meta z-function-call z-shell\"><\/span>) <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">at<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> var.c:4<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">4<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\">         char <span class=\"z-keyword z-operator z-regexp z-quantifier z-shell\">*<\/span>var = <span class=\"z-string z-quoted z-double z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&quot;<\/span>Hello, world!<span class=\"z-punctuation z-definition z-string z-end z-shell\">&quot;<\/span><\/span><\/span><span class=\"z-keyword z-operator z-logical z-continue z-shell\">;<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-punctuation z-definition z-compound z-begin z-shell\">(<\/span><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">gdb<\/span><\/span><span class=\"z-punctuation z-definition z-compound z-end z-shell\">)<\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> print var<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\"><span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-punctuation z-definition z-variable z-shell\">$<\/span><span class=\"z-variable z-other z-readwrite z-shell\">1<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> = 0x0<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-punctuation z-definition z-compound z-begin z-shell\">(<\/span><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">gdb<\/span><\/span><span class=\"z-punctuation z-definition z-compound z-end z-shell\">)<\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> n<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">5<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\">         var = <span class=\"z-string z-quoted z-double z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&quot;<\/span>changed my mind<span class=\"z-punctuation z-definition z-string z-end z-shell\">&quot;<\/span><\/span><\/span><span class=\"z-keyword z-operator z-logical z-continue z-shell\">;<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-punctuation z-definition z-compound z-begin z-shell\">(<\/span><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">gdb<\/span><\/span><span class=\"z-punctuation z-definition z-compound z-end z-shell\">)<\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> print var<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\"><span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-punctuation z-definition z-variable z-shell\">$<\/span><span class=\"z-variable z-other z-readwrite z-shell\">2<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> = 0x4005a0 <span class=\"z-string z-quoted z-double z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&quot;<\/span>Hello, world!<span class=\"z-punctuation z-definition z-string z-end z-shell\">&quot;<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-punctuation z-definition z-compound z-begin z-shell\">(<\/span><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">gdb<\/span><\/span><span class=\"z-punctuation z-definition z-compound z-end z-shell\">)<\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> n<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">6<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\">         puts(var<\/span><span class=\"z-meta z-function-call z-shell\"><\/span>)<span class=\"z-keyword z-operator z-logical z-continue z-shell\">;<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-punctuation z-definition z-compound z-begin z-shell\">(<\/span><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">gdb<\/span><\/span><span class=\"z-punctuation z-definition z-compound z-end z-shell\">)<\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> print var<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\"><span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-punctuation z-definition z-variable z-shell\">$<\/span><span class=\"z-variable z-other z-readwrite z-shell\">3<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> = 0x4005ae <span class=\"z-string z-quoted z-double z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&quot;<\/span>changed my mind<span class=\"z-punctuation z-definition z-string z-end z-shell\">&quot;<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-punctuation z-definition z-compound z-begin z-shell\">(<\/span><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">gdb<\/span><\/span><span class=\"z-punctuation z-definition z-compound z-end z-shell\">)<\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> n<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">changed<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> my mind<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">7<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\">       <\/span><span class=\"z-meta z-function-call z-shell\"><\/span>}\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-punctuation z-definition z-compound z-begin z-shell\">(<\/span><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">gdb<\/span><\/span><span class=\"z-punctuation z-definition z-compound z-end z-shell\">)<\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> c<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">Continuing.<\/span><\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">[Inferior<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> 1 (process 29542<\/span><span class=\"z-meta z-function-call z-shell\"><\/span>) <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">exited<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> normally]<\/span>\n<\/span><\/code><\/pre>\n<p>This post is continued in <a href=\"https:\/\/jyn.dev\/buffer-overflows-and-stacks-and-assembly-oh-my\/\">Buffer Overflows and Stacks and Assembly, Oh My<\/a>.<\/p>\n"},{"title":"Keepass on Linux and Android","published":"2018-08-24T00:00:00+00:00","updated":"2018-08-24T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/keepass-on-linux-and-android\/"}},"id":"https:\/\/jyn.dev\/keepass-on-linux-and-android\/","summary":"File synchronization on Linux","content":"<p>Previously, I wrote about\n<a href=\"\/2018-02-24-Password-Safety.html\">using a password manager<\/a>.\nHowever, the disadvantage of using an audited, local manager like\n<a href=\"https:\/\/keepass.info\">Keepass<\/a> is that it's hard to share passwords between devices.\nYou can put the encrypted database in a file-sharing service like Google Drive,\nbut that means you need a sync client on all of your devices, and Google\n<a href=\"https:\/\/abevoelker.github.io\/how-long-since-google-said-a-google-drive-linux-client-is-coming\/\">doesn't have one for Linux<\/a>.<\/p>\n<p>Fortunately, it's Linux, so there are alternatives.<\/p>\n<p>My current favorite sync client is <a href=\"https:\/\/github.com\/astrada\/google-drive-ocamlfuse\">google-drive-ocamlfuse<\/a>.\nAlthough it's annoying to have to <a href=\"https:\/\/github.com\/astrada\/google-drive-ocamlfuse\/wiki\/Installation\">build from source<\/a>\nand is <a href=\"https:\/\/github.com\/astrada\/google-drive-ocamlfuse#usage\">command-line only<\/a>,\nit lets you treat Drive as a user-land file system using <a href=\"https:\/\/github.com\/libfuse\/libfuse\">FUSE<\/a>,\nwhich is particularly nice for clients that aren't web-aware (like keepass).\nIt also has an excellent <a href=\"https:\/\/github.com\/astrada\/google-drive-ocamlfuse\/wiki\/Automounting\">walkthrough<\/a> on how to automount on boot\n(this is Linux, nothing is automatic).<\/p>\n<h2 id=\"appendix\">Appendix<a class=\"zola-anchor\" href=\"#appendix\" aria-label=\"Anchor link for: appendix\"><\/a>\n<\/h2>\n<ul>\n<li>\n<p>Why not just use Dropbox? It doesn't support NTFS drives\n(or anything other than <a href=\"https:\/\/www.dropbox.com\/help\/desktop-web\/system-requirements#desktop\">ext4<\/a>).<\/p>\n<\/li>\n<li>\n<p>Why not use an encrypted service, like <a href=\"https:\/\/mega.nz\/\">MEGA<\/a>\nor <a href=\"https:\/\/spideroak.com\/\">Spideroak<\/a>? They don't support WebDAV,\nand I don't feel like going through a file system every time I update passwords\n(most phone clients only allow file access through a specific app).<\/p>\n<\/li>\n<li>\n<p>Why not use &lt;some other service&gt; that (supports WebDAV or syncs automatically on phones) and has a Linux client? I haven't heard of &lt;some other service&gt;.<\/p>\n<\/li>\n<\/ul>\n"},{"title":"Rewriting Jython","published":"2018-06-09T00:00:00+00:00","updated":"2018-06-09T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/rewriting-jython\/"}},"id":"https:\/\/jyn.dev\/rewriting-jython\/","summary":"I got bored one day and started rewriting Jython","content":"<h2 id=\"intro\">Intro<a class=\"zola-anchor\" href=\"#intro\" aria-label=\"Anchor link for: intro\"><\/a>\n<\/h2>\n<p>Have you heard of Jython? It's a variant of Python written in Java instead of C;\nit lets you use native Java classes in python code.\nThe most common use case is integration with large existing Java codebases:\nYou can reuse Java code and still get the readability and conciseness of python.<\/p>\n<p>I was thinking about types in Python, particularly how it allows\nprimitive operators to be overridden (+-\/*%). For those unfamiliar with Python classes,\ncheck out my <a href=\"https:\/\/jyn.dev\/object-oriented-python\/\">previous post<\/a>;\nessentially it uses methods like <code>__add__<\/code>, <code>__sub__<\/code>, etc. as interfaces.\nThis lets you do things like<\/p>\n<pre data-lang=\"python\" class=\"language-python z-code\"><code class=\"language-python\" data-lang=\"python\"><span class=\"z-source z-python\"><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span> <span class=\"z-meta z-sequence z-list z-python\"><span class=\"z-punctuation z-section z-sequence z-begin z-python\">[<\/span><span class=\"z-constant z-numeric z-integer z-decimal z-python\">4<\/span><span class=\"z-punctuation z-section z-sequence z-end z-python\">]<\/span><\/span> <span class=\"z-keyword z-operator z-arithmetic z-python\">+<\/span> <span class=\"z-meta z-sequence z-list z-python\"><span class=\"z-punctuation z-section z-sequence z-begin z-python\">[<\/span><span class=\"z-constant z-numeric z-integer z-decimal z-python\">3<\/span><span class=\"z-punctuation z-section z-sequence z-end z-python\">]<\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-meta z-sequence z-list z-python\"><span class=\"z-punctuation z-section z-sequence z-begin z-python\">[<\/span><span class=\"z-constant z-numeric z-integer z-decimal z-python\">4<\/span><span class=\"z-punctuation z-separator z-sequence z-python\">,<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-python\">3<\/span><span class=\"z-punctuation z-section z-sequence z-end z-python\">]<\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span> <span class=\"z-meta z-sequence z-list z-python\"><span class=\"z-punctuation z-section z-sequence z-begin z-python\">[<\/span><span class=\"z-constant z-numeric z-integer z-decimal z-python\">4<\/span><span class=\"z-punctuation z-section z-sequence z-end z-python\">]<\/span><\/span> <span class=\"z-keyword z-operator z-arithmetic z-python\">*<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-python\">3<\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-meta z-sequence z-list z-python\"><span class=\"z-punctuation z-section z-sequence z-begin z-python\">[<\/span><span class=\"z-constant z-numeric z-integer z-decimal z-python\">4<\/span><span class=\"z-punctuation z-separator z-sequence z-python\">,<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-python\">4<\/span><span class=\"z-punctuation z-separator z-sequence z-python\">,<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-python\">4<\/span><span class=\"z-punctuation z-section z-sequence z-end z-python\">]<\/span><\/span>\n<\/span><\/code><\/pre>\n<p>which would throw nasty exceptions in most other languages.<\/p>\n<p>Because python allows any type to be called with these methods, it's the\nresponsibility of the class implementing the interface to ensure type safety,\nfor example <code>42 - '2'<\/code> will throw <code>TypeError: unsupported operand type(s) for -: 'int' and 'str'<\/code>.<\/p>\n<p>I thought to myself, this looks doable.<\/p>\n<h2 id=\"reflections\">Reflections<a class=\"zola-anchor\" href=\"#reflections\" aria-label=\"Anchor link for: reflections\"><\/a>\n<\/h2>\n<p>Java has a rarely-used but very powerful <a href=\"https:\/\/docs.oracle.com\/javase\/tutorial\/reflect\/\">reflection API<\/a>.\nThis allows you to do anything from get the class of a variable at runtime\n(and do different things depending on which class it is)\nto get the methods and constructors of an unknown class.\nMy (primitive) implementation of Jython make heavy use of reflections to use the\nbuiltin Java classes and avoiding writing an interpreter from scratch.<\/p>\n<h3 id=\"duck-typing\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/Duck_typing\">Duck Typing<\/a><a class=\"zola-anchor\" href=\"#duck-typing\" aria-label=\"Anchor link for: duck-typing\"><\/a>\n<\/h3>\n<p>Since python doesn't enforce types, we won't either. That means we have to call\nmethods at runtime, without knowing the class of the variable.\n<a href=\"https:\/\/docs.oracle.com\/javase\/8\/docs\/api\/java\/lang\/Class.html#getMethods--\">Java lets you do this!<\/a><\/p>\n<p>[\/\/]: # Secondly, since we want to allow inheritance, we have to find the constructors at runtime.\n[\/\/]: # This <a href=\"https:\/\/docs.oracle.com\/javase\/8\/docs\/api\/java\/lang\/Class.html#getConstructors--\">can be done<\/a> as well,\n[\/\/]: # but since we want the types to match, we have to iterate all the constructors until we find one that's compatible.<\/p>\n<h2 id=\"api\">API<a class=\"zola-anchor\" href=\"#api\" aria-label=\"Anchor link for: api\"><\/a>\n<\/h2>\n<p>The current API is exactly one class, <code>NumberWrapper<\/code>, which does nothing more than\n<a href=\"https:\/\/en.wikipedia.org\/wiki\/Type_conversion#Type_promotion\">type promotion<\/a> (and demotion).\nI'm currently working on a <code>ListWrapper<\/code> which would demonstrate the power of\nduck typing more clearly.\nBecause this is a toy project (and because parsing is hard), I have not implemented\nan interpreter; the sample usage would be <code>add(new NumberWrapper(5), new NumberWrapper(-.14))<\/code>.<\/p>\n<p>The full code is available <a href=\"\/assets\/jython.java\">here<\/a>.\nIts output is as follows:<\/p>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">$ javac jython.java -Xlint:unchecked\n<\/span><span class=\"z-text z-plain\">$ java main\n<\/span><span class=\"z-text z-plain\">3.5\n<\/span><\/code><\/pre>\n<p>Note that despite the extensive use of duck typing there are no type warnings;\n<code>javac<\/code> emits warnings only for casts, not calls to the reflection API.<\/p>\n"},{"title":"Copying Generic Arrays in Java","published":"2018-03-29T00:00:00+00:00","updated":"2018-03-29T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/copying-generic-arrays-in-java\/"}},"id":"https:\/\/jyn.dev\/copying-generic-arrays-in-java\/","summary":"The problem that's been bothering me for a week","content":"<h1 id=\"intro\">Intro<a class=\"zola-anchor\" href=\"#intro\" aria-label=\"Anchor link for: intro\"><\/a>\n<\/h1>\n<p>Copying an array.\nIt sounds so simple - run a quick <code>for<\/code> loop,\nor better yet, <code>System.arraycopy<\/code>.<\/p>\n<pre data-lang=\"java\" class=\"language-java z-code\"><code class=\"language-java\" data-lang=\"java\"><span class=\"z-source z-java\"><span class=\"z-support z-class z-java\">System<\/span><span class=\"z-punctuation z-accessor z-dot z-java\">.<\/span><span class=\"z-meta z-function-call z-java\"><span class=\"z-variable z-function z-java\">arraycopy<\/span><span class=\"z-punctuation z-section z-parens z-begin z-java\">(<\/span>array<span class=\"z-punctuation z-separator z-comma z-java\">,<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-java\">0<\/span><span class=\"z-punctuation z-separator z-comma z-java\">,<\/span> tmp<span class=\"z-punctuation z-separator z-comma z-java\">,<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-java\">0<\/span><span class=\"z-punctuation z-separator z-comma z-java\">,<\/span> array<span class=\"z-punctuation z-accessor z-dot z-java\">.<\/span>length<span class=\"z-punctuation z-section z-parens z-end z-java\">)<\/span><\/span><span class=\"z-punctuation z-terminator z-java\">;<\/span>\n<\/span><\/code><\/pre>\n<p>What if you need to perform a destructive operation? <sup><a href=\"#1\">[1]<\/a><\/sup>\nIn my Data Structures class, we're asked to implement heap sort\nwithout modifying the original array.\nThis is simple enough: use <code>array.clone<\/code>.\nIt preserves the order of the original array while giving you a new instance.<\/p>\n<pre data-lang=\"java\" class=\"language-java z-code\"><code class=\"language-java\" data-lang=\"java\"><span class=\"z-source z-java\"><span class=\"z-support z-class z-java\">T<\/span><span class=\"z-storage z-modifier z-array z-java\">[]<\/span> tmp <span class=\"z-meta z-assignment z-rhs z-java\"><span class=\"z-keyword z-operator z-assignment z-java\">=<\/span> array<span class=\"z-punctuation z-accessor z-dot z-java\">.<\/span><span class=\"z-meta z-function-call z-java\"><span class=\"z-variable z-function z-java\">clone<\/span><span class=\"z-punctuation z-section z-parens z-begin z-java\">(<\/span><span class=\"z-punctuation z-section z-parens z-end z-java\">)<\/span><\/span><\/span><span class=\"z-punctuation z-terminator z-java\">;<\/span>\n<\/span><\/code><\/pre>\n<p>Remember that this is a destructive operation, however;\nyou need a new third array in which to store the result.\nOnly one problem: Java doesn't allow generic arrays to be created.\nTrying to do so is a compile-time error:<\/p>\n<pre data-lang=\"java\" class=\"language-java z-code\"><code class=\"language-java\" data-lang=\"java\"><span class=\"z-source z-java\"><span class=\"z-meta z-class z-java\"><span class=\"z-meta z-class z-identifier z-java\"><span class=\"z-storage z-type z-java\">class<\/span> <span class=\"z-entity z-name z-class z-java\">Generic<\/span><\/span><span class=\"z-meta z-generic z-declaration z-java\"><span class=\"z-punctuation z-definition z-generic z-begin z-java\">&lt;<\/span><span class=\"z-variable z-parameter z-type z-java\">T<\/span><span class=\"z-punctuation z-definition z-generic z-end z-java\">&gt;<\/span><\/span> <span class=\"z-meta z-class z-body z-java\"><span class=\"z-meta z-block z-java\"><span class=\"z-punctuation z-section z-block z-begin z-java\">{<\/span>\n<\/span><\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-class z-java\"><span class=\"z-meta z-class z-body z-java\"><span class=\"z-meta z-block z-java\">    <span class=\"z-meta z-method z-java\"><span class=\"z-meta z-method z-identifier z-java\"><span class=\"z-entity z-name z-function z-constructor z-java\">Generic<\/span><\/span> <span class=\"z-meta z-method z-parameters z-java\"><span class=\"z-meta z-parens z-java\"><span class=\"z-punctuation z-section z-parens z-begin z-java\">(<\/span><span class=\"z-punctuation z-section z-parens z-end z-java\">)<\/span><\/span><\/span> <span class=\"z-meta z-method z-java\"><span class=\"z-meta z-method z-body z-java\"><span class=\"z-punctuation z-section z-block z-begin z-java\">{<\/span><\/span><\/span><\/span><span class=\"z-meta z-method z-java\"><span class=\"z-meta z-method z-body z-java\">\n<\/span><\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-class z-java\"><span class=\"z-meta z-class z-body z-java\"><span class=\"z-meta z-block z-java\"><span class=\"z-meta z-method z-java\"><span class=\"z-meta z-method z-body z-java\">        <span class=\"z-support z-class z-java\">System<\/span><span class=\"z-punctuation z-accessor z-dot z-java\">.<\/span>out<span class=\"z-punctuation z-accessor z-dot z-java\">.<\/span><span class=\"z-meta z-function-call z-java\"><span class=\"z-variable z-function z-java\">println<\/span><span class=\"z-punctuation z-section z-parens z-begin z-java\">(<\/span><span class=\"z-meta z-instantiation z-java\"><span class=\"z-keyword z-other z-storage z-new z-java\">new<\/span> <span class=\"z-support z-class z-java\">T<\/span><span class=\"z-meta z-brackets z-array-initialization z-java\"><span class=\"z-punctuation z-section z-brackets z-begin z-java\">[<\/span><span class=\"z-constant z-numeric z-integer z-decimal z-java\">0<\/span><span class=\"z-punctuation z-section z-brackets z-end z-java\">]<\/span><\/span><\/span><span class=\"z-punctuation z-section z-parens z-end z-java\">)<\/span><\/span><span class=\"z-punctuation z-terminator z-java\">;<\/span>\n<\/span><\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-class z-java\"><span class=\"z-meta z-class z-body z-java\"><span class=\"z-meta z-block z-java\"><span class=\"z-meta z-method z-java\"><span class=\"z-meta z-method z-body z-java\">    <span class=\"z-punctuation z-section z-block z-end z-java\">}<\/span><\/span><\/span>\n<\/span><\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-class z-java\"><span class=\"z-meta z-class z-body z-java\"><span class=\"z-meta z-block z-java\">\n<\/span><\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-class z-java\"><span class=\"z-meta z-class z-body z-java\"><span class=\"z-meta z-block z-java\">    <span class=\"z-storage z-modifier z-java\">public<\/span> <span class=\"z-storage z-modifier z-java\">static<\/span> <span class=\"z-meta z-method z-java\"><span class=\"z-storage z-type z-void z-java\">void<\/span> <span class=\"z-meta z-method z-identifier z-java\"><span class=\"z-entity z-name z-function z-java\">main<\/span><\/span><span class=\"z-meta z-method z-parameters z-java\"><span class=\"z-meta z-parens z-java\"><span class=\"z-punctuation z-section z-parens z-begin z-java\">(<\/span><span class=\"z-support z-class z-java\">String<\/span><span class=\"z-storage z-modifier z-array z-java\">[]<\/span> <span class=\"z-variable z-parameter z-java\">args<\/span><span class=\"z-punctuation z-section z-parens z-end z-java\">)<\/span><\/span><\/span> <span class=\"z-meta z-method z-java\"><span class=\"z-meta z-method z-body z-java\"><span class=\"z-punctuation z-section z-block z-begin z-java\">{<\/span><\/span><\/span><\/span><span class=\"z-meta z-method z-java\"><span class=\"z-meta z-method z-body z-java\">\n<\/span><\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-class z-java\"><span class=\"z-meta z-class z-body z-java\"><span class=\"z-meta z-block z-java\"><span class=\"z-meta z-method z-java\"><span class=\"z-meta z-method z-body z-java\">        <span class=\"z-meta z-instantiation z-java\"><span class=\"z-keyword z-other z-storage z-new z-java\">new<\/span> <span class=\"z-support z-class z-java\">Generic<\/span><span class=\"z-meta z-generic z-java\"><span class=\"z-punctuation z-definition z-generic z-begin z-java\">&lt;<\/span><span class=\"z-support z-class z-java\">String<\/span><span class=\"z-punctuation z-definition z-generic z-end z-java\">&gt;<\/span><\/span><span class=\"z-meta z-parens z-constructor-arguments z-java\"><span class=\"z-punctuation z-section z-parens z-begin z-java\">(<\/span><span class=\"z-punctuation z-section z-parens z-end z-java\">)<\/span><\/span><\/span><span class=\"z-punctuation z-terminator z-java\">;<\/span>\n<\/span><\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-class z-java\"><span class=\"z-meta z-class z-body z-java\"><span class=\"z-meta z-block z-java\"><span class=\"z-meta z-method z-java\"><span class=\"z-meta z-method z-body z-java\">    <span class=\"z-punctuation z-section z-block z-end z-java\">}<\/span><\/span><\/span>\n<\/span><\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-class z-java\"><span class=\"z-meta z-class z-body z-java\"><span class=\"z-meta z-block z-java\"><span class=\"z-punctuation z-section z-block z-end z-java\">}<\/span><\/span><\/span><\/span>\n<\/span><\/code><\/pre>\n<pre data-lang=\"sh\" class=\"language-sh z-code\"><code class=\"language-sh\" data-lang=\"sh\"><span class=\"z-source z-shell z-bash\"> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\"><span class=\"z-meta z-group z-expansion z-job z-shell\"><span class=\"z-punctuation z-definition z-variable z-job z-shell\">%<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> javac Generic.java<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">Generic.java:3:<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> error: generic array creation<\/span>\n<\/span><span class=\"z-source z-shell z-bash\">        <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">System.out.println<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\">(new T<span class=\"z-keyword z-control z-regexp z-set z-begin z-shell\">[<\/span>0<span class=\"z-keyword z-control z-regexp z-set z-end z-shell\">]<\/span><\/span><span class=\"z-meta z-function-call z-shell\"><\/span>)<span class=\"z-keyword z-operator z-logical z-continue z-shell\">;<\/span>\n<\/span><span class=\"z-source z-shell z-bash\">                           <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">^<\/span><\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">1<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> error<\/span>\n<\/span><\/code><\/pre>\n<p>Fine, let's do some <a href=\"https:\/\/stackoverflow.com\/a\/530289\">ugly casting<\/a>.<\/p>\n<p><code>T[] array = (T[]) new Object[0];<\/code><\/p>\n<h1 id=\"strange-happenings\">Strange happenings<a class=\"zola-anchor\" href=\"#strange-happenings\" aria-label=\"Anchor link for: strange-happenings\"><\/a>\n<\/h1>\n<p>So far, this has all been pretty standard.\nJava has <a href=\"https:\/\/docs.oracle.com\/javase\/tutorial\/java\/generics\/erasure.html\">type erasure<\/a>, Java generics are a pain, yadda yadda.\nNow we get to the strange part of this post.<\/p>\n<pre data-lang=\"java\" class=\"language-java z-code\"><code class=\"language-java\" data-lang=\"java\"><span class=\"z-source z-java\"><span class=\"z-meta z-class z-java\"><span class=\"z-meta z-class z-identifier z-java\"><span class=\"z-storage z-type z-java\">class<\/span> <span class=\"z-entity z-name z-class z-java\">Copy<\/span><\/span><span class=\"z-meta z-generic z-declaration z-java\"><span class=\"z-punctuation z-definition z-generic z-begin z-java\">&lt;<\/span><span class=\"z-variable z-parameter z-type z-java\">T<\/span><span class=\"z-punctuation z-definition z-generic z-end z-java\">&gt;<\/span><\/span> <span class=\"z-meta z-class z-body z-java\"><span class=\"z-meta z-block z-java\"><span class=\"z-punctuation z-section z-block z-begin z-java\">{<\/span>\n<\/span><\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-class z-java\"><span class=\"z-meta z-class z-body z-java\"><span class=\"z-meta z-block z-java\">    <span class=\"z-storage z-modifier z-java\">private<\/span> <span class=\"z-storage z-modifier z-java\">final<\/span> <span class=\"z-support z-class z-java\">T<\/span><span class=\"z-storage z-modifier z-array z-java\">[]<\/span> <span class=\"z-meta z-field z-java\">array<\/span><span class=\"z-punctuation z-terminator z-java\">;<\/span>\n<\/span><\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-class z-java\"><span class=\"z-meta z-class z-body z-java\"><span class=\"z-meta z-block z-java\">    <span class=\"z-meta z-method z-java\"><span class=\"z-meta z-method z-identifier z-java\"><span class=\"z-entity z-name z-function z-constructor z-java\">Copy<\/span><\/span><span class=\"z-meta z-method z-parameters z-java\"><span class=\"z-meta z-parens z-java\"><span class=\"z-punctuation z-section z-parens z-begin z-java\">(<\/span><span class=\"z-support z-class z-java\">T<\/span><span class=\"z-storage z-modifier z-array z-java\">[]<\/span> <span class=\"z-variable z-parameter z-java\">array<\/span><span class=\"z-punctuation z-section z-parens z-end z-java\">)<\/span><\/span><\/span> <span class=\"z-meta z-method z-java\"><span class=\"z-meta z-method z-body z-java\"><span class=\"z-punctuation z-section z-block z-begin z-java\">{<\/span><\/span><\/span><\/span><span class=\"z-meta z-method z-java\"><span class=\"z-meta z-method z-body z-java\">\n<\/span><\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-class z-java\"><span class=\"z-meta z-class z-body z-java\"><span class=\"z-meta z-block z-java\"><span class=\"z-meta z-method z-java\"><span class=\"z-meta z-method z-body z-java\">        <span class=\"z-variable z-language z-java\">this<\/span><span class=\"z-punctuation z-accessor z-dot z-java\">.<\/span>array <span class=\"z-meta z-assignment z-rhs z-java\"><span class=\"z-keyword z-operator z-assignment z-java\">=<\/span> array<\/span><span class=\"z-punctuation z-terminator z-java\">;<\/span>\n<\/span><\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-class z-java\"><span class=\"z-meta z-class z-body z-java\"><span class=\"z-meta z-block z-java\"><span class=\"z-meta z-method z-java\"><span class=\"z-meta z-method z-body z-java\">    <span class=\"z-punctuation z-section z-block z-end z-java\">}<\/span><\/span><\/span>\n<\/span><\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-class z-java\"><span class=\"z-meta z-class z-body z-java\"><span class=\"z-meta z-block z-java\">\n<\/span><\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-class z-java\"><span class=\"z-meta z-class z-body z-java\"><span class=\"z-meta z-block z-java\">    <span class=\"z-support z-class z-java\">T<\/span><span class=\"z-storage z-modifier z-array z-java\">[]<\/span> <span class=\"z-meta z-method z-java\"><span class=\"z-meta z-method z-identifier z-java\"><span class=\"z-entity z-name z-function z-java\">getNew<\/span><\/span><span class=\"z-meta z-method z-parameters z-java\"><span class=\"z-meta z-parens z-java\"><span class=\"z-punctuation z-section z-parens z-begin z-java\">(<\/span><span class=\"z-punctuation z-section z-parens z-end z-java\">)<\/span><\/span><\/span> <span class=\"z-meta z-method z-java\"><span class=\"z-meta z-method z-body z-java\"><span class=\"z-punctuation z-section z-block z-begin z-java\">{<\/span><\/span><\/span><\/span><span class=\"z-meta z-method z-java\"><span class=\"z-meta z-method z-body z-java\">\n<\/span><\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-class z-java\"><span class=\"z-meta z-class z-body z-java\"><span class=\"z-meta z-block z-java\"><span class=\"z-meta z-method z-java\"><span class=\"z-meta z-method z-body z-java\">        <span class=\"z-support z-class z-java\">T<\/span><span class=\"z-storage z-modifier z-array z-java\">[]<\/span> result <span class=\"z-meta z-assignment z-rhs z-java\"><span class=\"z-keyword z-operator z-assignment z-java\">=<\/span> <span class=\"z-meta z-parens z-java\"><span class=\"z-punctuation z-section z-parens z-begin z-java\">(<\/span><span class=\"z-support z-class z-java\">T<\/span><span class=\"z-storage z-modifier z-array z-java\">[]<\/span><span class=\"z-punctuation z-section z-parens z-end z-java\">)<\/span><\/span> <span class=\"z-meta z-instantiation z-java\"><span class=\"z-keyword z-other z-storage z-new z-java\">new<\/span> <span class=\"z-support z-class z-java\">Object<\/span><span class=\"z-meta z-brackets z-array-initialization z-java\"><span class=\"z-punctuation z-section z-brackets z-begin z-java\">[<\/span>array<span class=\"z-punctuation z-accessor z-dot z-java\">.<\/span>length<span class=\"z-punctuation z-section z-brackets z-end z-java\">]<\/span><\/span><\/span><\/span><span class=\"z-punctuation z-terminator z-java\">;<\/span>\n<\/span><\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-class z-java\"><span class=\"z-meta z-class z-body z-java\"><span class=\"z-meta z-block z-java\"><span class=\"z-meta z-method z-java\"><span class=\"z-meta z-method z-body z-java\">        <span class=\"z-keyword z-control z-loop z-for z-java\">for<\/span> <span class=\"z-meta z-parens z-java\"><span class=\"z-punctuation z-section z-parens z-begin z-java\">(<\/span><span class=\"z-storage z-type z-primitive z-java\">int<\/span> i <span class=\"z-meta z-assignment z-rhs z-java\"><span class=\"z-keyword z-operator z-assignment z-java\">=<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-java\">0<\/span><\/span><span class=\"z-punctuation z-terminator z-java\">;<\/span> i <span class=\"z-keyword z-operator z-comparison z-java\">&lt;<\/span> array<span class=\"z-punctuation z-accessor z-dot z-java\">.<\/span>length<span class=\"z-punctuation z-terminator z-java\">;<\/span> i<span class=\"z-keyword z-operator z-increment-decrement z-java\">++<\/span><span class=\"z-punctuation z-section z-parens z-end z-java\">)<\/span><\/span> <span class=\"z-meta z-block z-java\"><span class=\"z-punctuation z-section z-block z-begin z-java\">{<\/span>\n<\/span><\/span><\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-class z-java\"><span class=\"z-meta z-class z-body z-java\"><span class=\"z-meta z-block z-java\"><span class=\"z-meta z-method z-java\"><span class=\"z-meta z-method z-body z-java\"><span class=\"z-meta z-block z-java\">            result[i] <span class=\"z-meta z-assignment z-rhs z-java\"><span class=\"z-keyword z-operator z-assignment z-java\">=<\/span> array[i]<\/span><span class=\"z-punctuation z-terminator z-java\">;<\/span>\n<\/span><\/span><\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-class z-java\"><span class=\"z-meta z-class z-body z-java\"><span class=\"z-meta z-block z-java\"><span class=\"z-meta z-method z-java\"><span class=\"z-meta z-method z-body z-java\"><span class=\"z-meta z-block z-java\">        <span class=\"z-punctuation z-section z-block z-end z-java\">}<\/span><\/span>\n<\/span><\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-class z-java\"><span class=\"z-meta z-class z-body z-java\"><span class=\"z-meta z-block z-java\"><span class=\"z-meta z-method z-java\"><span class=\"z-meta z-method z-body z-java\">        <span class=\"z-keyword z-control z-flow z-return z-java\">return<\/span> result<span class=\"z-punctuation z-terminator z-java\">;<\/span>\n<\/span><\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-class z-java\"><span class=\"z-meta z-class z-body z-java\"><span class=\"z-meta z-block z-java\"><span class=\"z-meta z-method z-java\"><span class=\"z-meta z-method z-body z-java\">    <span class=\"z-punctuation z-section z-block z-end z-java\">}<\/span><\/span><\/span>\n<\/span><\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-class z-java\"><span class=\"z-meta z-class z-body z-java\"><span class=\"z-meta z-block z-java\">\n<\/span><\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-class z-java\"><span class=\"z-meta z-class z-body z-java\"><span class=\"z-meta z-block z-java\">    <span class=\"z-storage z-modifier z-java\">public<\/span> <span class=\"z-storage z-modifier z-java\">static<\/span> <span class=\"z-meta z-method z-java\"><span class=\"z-storage z-type z-void z-java\">void<\/span> <span class=\"z-meta z-method z-identifier z-java\"><span class=\"z-entity z-name z-function z-java\">main<\/span><\/span><span class=\"z-meta z-method z-parameters z-java\"><span class=\"z-meta z-parens z-java\"><span class=\"z-punctuation z-section z-parens z-begin z-java\">(<\/span><span class=\"z-support z-class z-java\">String<\/span><span class=\"z-storage z-modifier z-array z-java\">[]<\/span> <span class=\"z-variable z-parameter z-java\">args<\/span><span class=\"z-punctuation z-section z-parens z-end z-java\">)<\/span><\/span><\/span> <span class=\"z-meta z-method z-java\"><span class=\"z-meta z-method z-body z-java\"><span class=\"z-punctuation z-section z-block z-begin z-java\">{<\/span><\/span><\/span><\/span><span class=\"z-meta z-method z-java\"><span class=\"z-meta z-method z-body z-java\">\n<\/span><\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-class z-java\"><span class=\"z-meta z-class z-body z-java\"><span class=\"z-meta z-block z-java\"><span class=\"z-meta z-method z-java\"><span class=\"z-meta z-method z-body z-java\">        <span class=\"z-support z-class z-java\">Integer<\/span><span class=\"z-storage z-modifier z-array z-java\">[]<\/span> arr <span class=\"z-meta z-assignment z-rhs z-java\"><span class=\"z-keyword z-operator z-assignment z-java\">=<\/span> <span class=\"z-meta z-instantiation z-java\"><span class=\"z-keyword z-other z-storage z-new z-java\">new<\/span> <span class=\"z-support z-class z-java\">Copy<\/span><span class=\"z-meta z-generic z-java\"><span class=\"z-punctuation z-definition z-generic z-begin z-java\">&lt;<\/span><span class=\"z-support z-class z-java\">Integer<\/span><span class=\"z-punctuation z-definition z-generic z-end z-java\">&gt;<\/span><\/span><span class=\"z-meta z-parens z-constructor-arguments z-java\"><span class=\"z-punctuation z-section z-parens z-begin z-java\">(<\/span><span class=\"z-meta z-instantiation z-java\"><span class=\"z-keyword z-other z-storage z-new z-java\">new<\/span> <span class=\"z-support z-class z-java\">Integer<\/span><span class=\"z-meta z-brackets z-array-initialization z-java\"><span class=\"z-punctuation z-section z-brackets z-begin z-java\">[<\/span><span class=\"z-punctuation z-section z-brackets z-end z-java\">]<\/span><\/span> <span class=\"z-meta z-braces z-array-initialization z-java\"><span class=\"z-punctuation z-section z-braces z-begin z-java\">{<\/span><span class=\"z-constant z-numeric z-integer z-decimal z-java\">1<\/span><span class=\"z-punctuation z-separator z-comma z-java\">,<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-java\">2<\/span><span class=\"z-punctuation z-separator z-comma z-java\">,<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-java\">3<\/span><span class=\"z-punctuation z-section z-braces z-end z-java\">}<\/span><\/span><\/span><span class=\"z-punctuation z-section z-parens z-end z-java\">)<\/span><\/span><\/span><span class=\"z-punctuation z-accessor z-dot z-java\">.<\/span><span class=\"z-meta z-function-call z-java\"><span class=\"z-variable z-function z-java\">getNew<\/span><span class=\"z-punctuation z-section z-parens z-begin z-java\">(<\/span><span class=\"z-punctuation z-section z-parens z-end z-java\">)<\/span><\/span><\/span><span class=\"z-punctuation z-terminator z-java\">;<\/span>\n<\/span><\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-class z-java\"><span class=\"z-meta z-class z-body z-java\"><span class=\"z-meta z-block z-java\"><span class=\"z-meta z-method z-java\"><span class=\"z-meta z-method z-body z-java\">    <span class=\"z-punctuation z-section z-block z-end z-java\">}<\/span><\/span><\/span>\n<\/span><\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-class z-java\"><span class=\"z-meta z-class z-body z-java\"><span class=\"z-meta z-block z-java\"><span class=\"z-punctuation z-section z-block z-end z-java\">}<\/span><\/span><\/span><\/span>\n<\/span><\/code><\/pre>\n<pre data-lang=\"sh\" class=\"language-sh z-code\"><code class=\"language-sh\" data-lang=\"sh\"><span class=\"z-source z-shell z-bash\"> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\"><span class=\"z-meta z-group z-expansion z-job z-shell\"><span class=\"z-punctuation z-definition z-variable z-job z-shell\">%<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> javac copy.java<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">Note:<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> copy.java uses unchecked or unsafe operations.<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">Note:<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> Recompile with<span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> -<\/span>Xlint<\/span>:unchecked for details.<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\"><span class=\"z-meta z-group z-expansion z-job z-shell\"><span class=\"z-punctuation z-definition z-variable z-job z-shell\">%<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> java copy <\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">Exception<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> in thread <span class=\"z-string z-quoted z-double z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&quot;<\/span>main<span class=\"z-punctuation z-definition z-string z-end z-shell\">&quot;<\/span><\/span> java.lang.ClassCastException: [Ljava.lang.Object<\/span><span class=\"z-keyword z-operator z-logical z-continue z-shell\">;<\/span> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">cannot<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> be cast to [Ljava.lang.Integer<\/span><span class=\"z-keyword z-operator z-logical z-continue z-shell\">;<\/span>\n<\/span><span class=\"z-source z-shell z-bash\">        <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">at<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> copy.main(copy.java:16<\/span><span class=\"z-meta z-function-call z-shell\"><\/span>)\n<\/span><\/code><\/pre>\n<p>What?!\u203d\nNot only are we getting a <code>ClassCastException<\/code> trying to convert <code>Integer<\/code> to <code>Integer<\/code>,\nbut we're getting it at runtime, not compile-time!\nWorse yet, there's no traceback - the error pointed to <code>main<\/code>, not to <code>getNew<\/code>!<\/p>\n<h1 id=\"my-solution\">My solution<a class=\"zola-anchor\" href=\"#my-solution\" aria-label=\"Anchor link for: my-solution\"><\/a>\n<\/h1>\n<p>What happens when you declare a new array in Java?\nYou might assume that this is translated directly to Java bytecode, but you would be incorrect.\n<code>javac<\/code> actually translates this to <code>java.lang.reflect.Array.newInstance<\/code>, which in turn\n<a href=\"http:\/\/hg.openjdk.java.net\/jdk\/jdk\/file\/1f9dd2360b17\/src\/java.base\/share\/classes\/java\/lang\/reflect\/Array.java#l76\">calls<\/a>\nthe native method <code>java.lang.reflect.Array.newArray<\/code>.\nIn fact, you can call this method yourself, but only if you have the class type.\nType parameters don't cut it.<\/p>\n<p>Well, if you have an instance of the class, you can call <code>instance.getClass()<\/code>.\nIf you don't, you can (in this particular case) return a null pointer,\nsince the array must be empty.<\/p>\n<pre data-lang=\"java\" class=\"language-java z-code\"><code class=\"language-java\" data-lang=\"java\"><span class=\"z-source z-java\"> <span class=\"z-keyword z-operator z-arithmetic z-java\">%<\/span> cat copy<span class=\"z-punctuation z-accessor z-dot z-java\">.<\/span>java\n<\/span><span class=\"z-source z-java\"><span class=\"z-meta z-class z-java\"><span class=\"z-meta z-class z-identifier z-java\"><span class=\"z-storage z-type z-java\">class<\/span> <span class=\"z-entity z-name z-class z-java\">Copy<\/span><\/span><span class=\"z-meta z-generic z-declaration z-java\"><span class=\"z-punctuation z-definition z-generic z-begin z-java\">&lt;<\/span><span class=\"z-variable z-parameter z-type z-java\">T<\/span><span class=\"z-punctuation z-definition z-generic z-end z-java\">&gt;<\/span><\/span> <span class=\"z-meta z-class z-body z-java\"><span class=\"z-meta z-block z-java\"><span class=\"z-punctuation z-section z-block z-begin z-java\">{<\/span>\n<\/span><\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-class z-java\"><span class=\"z-meta z-class z-body z-java\"><span class=\"z-meta z-block z-java\">\n<\/span><\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-class z-java\"><span class=\"z-meta z-class z-body z-java\"><span class=\"z-meta z-block z-java\">    <span class=\"z-storage z-modifier z-java\">private<\/span> <span class=\"z-storage z-modifier z-java\">final<\/span> <span class=\"z-support z-class z-java\">T<\/span><span class=\"z-storage z-modifier z-array z-java\">[]<\/span> <span class=\"z-meta z-field z-java\">array<\/span><span class=\"z-punctuation z-terminator z-java\">;<\/span>\n<\/span><\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-class z-java\"><span class=\"z-meta z-class z-body z-java\"><span class=\"z-meta z-block z-java\">\n<\/span><\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-class z-java\"><span class=\"z-meta z-class z-body z-java\"><span class=\"z-meta z-block z-java\">    <span class=\"z-meta z-method z-java\"><span class=\"z-meta z-method z-identifier z-java\"><span class=\"z-entity z-name z-function z-constructor z-java\">Copy<\/span><\/span><span class=\"z-meta z-method z-parameters z-java\"><span class=\"z-meta z-parens z-java\"><span class=\"z-punctuation z-section z-parens z-begin z-java\">(<\/span><span class=\"z-support z-class z-java\">T<\/span><span class=\"z-storage z-modifier z-array z-java\">[]<\/span> <span class=\"z-variable z-parameter z-java\">array<\/span><span class=\"z-punctuation z-section z-parens z-end z-java\">)<\/span><\/span><\/span> <span class=\"z-meta z-method z-java\"><span class=\"z-meta z-method z-body z-java\"><span class=\"z-punctuation z-section z-block z-begin z-java\">{<\/span><\/span><\/span><\/span><span class=\"z-meta z-method z-java\"><span class=\"z-meta z-method z-body z-java\">\n<\/span><\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-class z-java\"><span class=\"z-meta z-class z-body z-java\"><span class=\"z-meta z-block z-java\"><span class=\"z-meta z-method z-java\"><span class=\"z-meta z-method z-body z-java\">        <span class=\"z-variable z-language z-java\">this<\/span><span class=\"z-punctuation z-accessor z-dot z-java\">.<\/span>array <span class=\"z-meta z-assignment z-rhs z-java\"><span class=\"z-keyword z-operator z-assignment z-java\">=<\/span> array<\/span><span class=\"z-punctuation z-terminator z-java\">;<\/span>\n<\/span><\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-class z-java\"><span class=\"z-meta z-class z-body z-java\"><span class=\"z-meta z-block z-java\"><span class=\"z-meta z-method z-java\"><span class=\"z-meta z-method z-body z-java\">    <span class=\"z-punctuation z-section z-block z-end z-java\">}<\/span><\/span><\/span>\n<\/span><\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-class z-java\"><span class=\"z-meta z-class z-body z-java\"><span class=\"z-meta z-block z-java\">\n<\/span><\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-class z-java\"><span class=\"z-meta z-class z-body z-java\"><span class=\"z-meta z-block z-java\">    <span class=\"z-support z-class z-java\">T<\/span><span class=\"z-storage z-modifier z-array z-java\">[]<\/span> <span class=\"z-meta z-method z-java\"><span class=\"z-meta z-method z-identifier z-java\"><span class=\"z-entity z-name z-function z-java\">getNew<\/span><\/span><span class=\"z-meta z-method z-parameters z-java\"><span class=\"z-meta z-parens z-java\"><span class=\"z-punctuation z-section z-parens z-begin z-java\">(<\/span><span class=\"z-punctuation z-section z-parens z-end z-java\">)<\/span><\/span><\/span> <span class=\"z-meta z-method z-java\"><span class=\"z-meta z-method z-body z-java\"><span class=\"z-punctuation z-section z-block z-begin z-java\">{<\/span><\/span><\/span><\/span><span class=\"z-meta z-method z-java\"><span class=\"z-meta z-method z-body z-java\">\n<\/span><\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-class z-java\"><span class=\"z-meta z-class z-body z-java\"><span class=\"z-meta z-block z-java\"><span class=\"z-meta z-method z-java\"><span class=\"z-meta z-method z-body z-java\">        <span class=\"z-keyword z-control z-conditional z-if z-java\">if<\/span> <span class=\"z-meta z-parens z-java\"><span class=\"z-punctuation z-section z-parens z-begin z-java\">(<\/span>array<span class=\"z-punctuation z-accessor z-dot z-java\">.<\/span>length <span class=\"z-keyword z-operator z-comparison z-java\">==<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-java\">0<\/span><span class=\"z-punctuation z-section z-parens z-end z-java\">)<\/span><\/span> <span class=\"z-keyword z-control z-flow z-return z-java\">return<\/span> <span class=\"z-constant z-language z-java\">null<\/span><span class=\"z-punctuation z-terminator z-java\">;<\/span>\n<\/span><\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-class z-java\"><span class=\"z-meta z-class z-body z-java\"><span class=\"z-meta z-block z-java\"><span class=\"z-meta z-method z-java\"><span class=\"z-meta z-method z-body z-java\">        <span class=\"z-support z-class z-java\">T<\/span><span class=\"z-storage z-modifier z-array z-java\">[]<\/span> result <span class=\"z-meta z-assignment z-rhs z-java\"><span class=\"z-keyword z-operator z-assignment z-java\">=<\/span> <span class=\"z-meta z-parens z-java\"><span class=\"z-punctuation z-section z-parens z-begin z-java\">(<\/span><span class=\"z-support z-class z-java\">T<\/span><span class=\"z-storage z-modifier z-array z-java\">[]<\/span><span class=\"z-punctuation z-section z-parens z-end z-java\">)<\/span><\/span> <span class=\"z-meta z-path z-java\"><span class=\"z-support z-type z-package z-java\">java<\/span><span class=\"z-punctuation z-accessor z-dot z-java\">.<\/span><span class=\"z-support z-type z-package z-java\">lang<\/span><span class=\"z-punctuation z-accessor z-dot z-java\">.<\/span><span class=\"z-support z-type z-package z-java\">reflect<\/span><span class=\"z-punctuation z-accessor z-dot z-java\">.<\/span><span class=\"z-support z-class z-java\">Array<\/span><\/span><span class=\"z-punctuation z-accessor z-dot z-java\">.<\/span><span class=\"z-meta z-function-call z-java\"><span class=\"z-variable z-function z-java\">newInstance<\/span><span class=\"z-punctuation z-section z-parens z-begin z-java\">(<\/span>array[<span class=\"z-constant z-numeric z-integer z-decimal z-java\">0<\/span>]<span class=\"z-punctuation z-accessor z-dot z-java\">.<\/span><span class=\"z-meta z-function-call z-java\"><span class=\"z-variable z-function z-java\">getClass<\/span><span class=\"z-punctuation z-section z-parens z-begin z-java\">(<\/span><span class=\"z-punctuation z-section z-parens z-end z-java\">)<\/span><\/span><span class=\"z-punctuation z-separator z-comma z-java\">,<\/span> array<span class=\"z-punctuation z-accessor z-dot z-java\">.<\/span>length<span class=\"z-punctuation z-section z-parens z-end z-java\">)<\/span><\/span><\/span><span class=\"z-punctuation z-terminator z-java\">;<\/span>\n<\/span><\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-class z-java\"><span class=\"z-meta z-class z-body z-java\"><span class=\"z-meta z-block z-java\"><span class=\"z-meta z-method z-java\"><span class=\"z-meta z-method z-body z-java\">\n<\/span><\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-class z-java\"><span class=\"z-meta z-class z-body z-java\"><span class=\"z-meta z-block z-java\"><span class=\"z-meta z-method z-java\"><span class=\"z-meta z-method z-body z-java\">        <span class=\"z-keyword z-control z-loop z-for z-java\">for<\/span> <span class=\"z-meta z-parens z-java\"><span class=\"z-punctuation z-section z-parens z-begin z-java\">(<\/span><span class=\"z-storage z-type z-primitive z-java\">int<\/span> i <span class=\"z-meta z-assignment z-rhs z-java\"><span class=\"z-keyword z-operator z-assignment z-java\">=<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-java\">0<\/span><\/span><span class=\"z-punctuation z-terminator z-java\">;<\/span> i <span class=\"z-keyword z-operator z-comparison z-java\">&lt;<\/span> array<span class=\"z-punctuation z-accessor z-dot z-java\">.<\/span>length<span class=\"z-punctuation z-terminator z-java\">;<\/span> i<span class=\"z-keyword z-operator z-increment-decrement z-java\">++<\/span><span class=\"z-punctuation z-section z-parens z-end z-java\">)<\/span><\/span> <span class=\"z-meta z-block z-java\"><span class=\"z-punctuation z-section z-block z-begin z-java\">{<\/span>\n<\/span><\/span><\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-class z-java\"><span class=\"z-meta z-class z-body z-java\"><span class=\"z-meta z-block z-java\"><span class=\"z-meta z-method z-java\"><span class=\"z-meta z-method z-body z-java\"><span class=\"z-meta z-block z-java\">            result[i] <span class=\"z-meta z-assignment z-rhs z-java\"><span class=\"z-keyword z-operator z-assignment z-java\">=<\/span> array[i]<\/span><span class=\"z-punctuation z-terminator z-java\">;<\/span>\n<\/span><\/span><\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-class z-java\"><span class=\"z-meta z-class z-body z-java\"><span class=\"z-meta z-block z-java\"><span class=\"z-meta z-method z-java\"><span class=\"z-meta z-method z-body z-java\"><span class=\"z-meta z-block z-java\">        <span class=\"z-punctuation z-section z-block z-end z-java\">}<\/span><\/span>\n<\/span><\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-class z-java\"><span class=\"z-meta z-class z-body z-java\"><span class=\"z-meta z-block z-java\"><span class=\"z-meta z-method z-java\"><span class=\"z-meta z-method z-body z-java\">\n<\/span><\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-class z-java\"><span class=\"z-meta z-class z-body z-java\"><span class=\"z-meta z-block z-java\"><span class=\"z-meta z-method z-java\"><span class=\"z-meta z-method z-body z-java\">        <span class=\"z-keyword z-control z-flow z-return z-java\">return<\/span> result<span class=\"z-punctuation z-terminator z-java\">;<\/span>\n<\/span><\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-class z-java\"><span class=\"z-meta z-class z-body z-java\"><span class=\"z-meta z-block z-java\"><span class=\"z-meta z-method z-java\"><span class=\"z-meta z-method z-body z-java\">    <span class=\"z-punctuation z-section z-block z-end z-java\">}<\/span><\/span><\/span>\n<\/span><\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-class z-java\"><span class=\"z-meta z-class z-body z-java\"><span class=\"z-meta z-block z-java\">\n<\/span><\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-class z-java\"><span class=\"z-meta z-class z-body z-java\"><span class=\"z-meta z-block z-java\">    <span class=\"z-storage z-modifier z-java\">public<\/span> <span class=\"z-storage z-modifier z-java\">static<\/span> <span class=\"z-meta z-method z-java\"><span class=\"z-storage z-type z-void z-java\">void<\/span> <span class=\"z-meta z-method z-identifier z-java\"><span class=\"z-entity z-name z-function z-java\">main<\/span><\/span><span class=\"z-meta z-method z-parameters z-java\"><span class=\"z-meta z-parens z-java\"><span class=\"z-punctuation z-section z-parens z-begin z-java\">(<\/span><span class=\"z-support z-class z-java\">String<\/span><span class=\"z-storage z-modifier z-array z-java\">[]<\/span> <span class=\"z-variable z-parameter z-java\">args<\/span><span class=\"z-punctuation z-section z-parens z-end z-java\">)<\/span><\/span><\/span> <span class=\"z-meta z-method z-java\"><span class=\"z-meta z-method z-body z-java\"><span class=\"z-punctuation z-section z-block z-begin z-java\">{<\/span><\/span><\/span><\/span><span class=\"z-meta z-method z-java\"><span class=\"z-meta z-method z-body z-java\">\n<\/span><\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-class z-java\"><span class=\"z-meta z-class z-body z-java\"><span class=\"z-meta z-block z-java\"><span class=\"z-meta z-method z-java\"><span class=\"z-meta z-method z-body z-java\">        <span class=\"z-support z-class z-java\">System<\/span><span class=\"z-punctuation z-accessor z-dot z-java\">.<\/span>out<span class=\"z-punctuation z-accessor z-dot z-java\">.<\/span><span class=\"z-meta z-function-call z-java\"><span class=\"z-variable z-function z-java\">println<\/span><span class=\"z-punctuation z-section z-parens z-begin z-java\">(<\/span><span class=\"z-meta z-instantiation z-java\"><span class=\"z-keyword z-other z-storage z-new z-java\">new<\/span> <span class=\"z-support z-class z-java\">Copy<\/span><span class=\"z-meta z-generic z-java\"><span class=\"z-punctuation z-definition z-generic z-begin z-java\">&lt;<\/span><span class=\"z-support z-class z-java\">Integer<\/span><span class=\"z-punctuation z-definition z-generic z-end z-java\">&gt;<\/span><\/span><span class=\"z-meta z-parens z-constructor-arguments z-java\"><span class=\"z-punctuation z-section z-parens z-begin z-java\">(<\/span><span class=\"z-meta z-instantiation z-java\"><span class=\"z-keyword z-other z-storage z-new z-java\">new<\/span> <span class=\"z-support z-class z-java\">Integer<\/span><span class=\"z-meta z-brackets z-array-initialization z-java\"><span class=\"z-punctuation z-section z-brackets z-begin z-java\">[<\/span><span class=\"z-punctuation z-section z-brackets z-end z-java\">]<\/span><\/span> <span class=\"z-meta z-braces z-array-initialization z-java\"><span class=\"z-punctuation z-section z-braces z-begin z-java\">{<\/span><span class=\"z-constant z-numeric z-integer z-decimal z-java\">1<\/span><span class=\"z-punctuation z-separator z-comma z-java\">,<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-java\">2<\/span><span class=\"z-punctuation z-separator z-comma z-java\">,<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-java\">3<\/span><span class=\"z-punctuation z-section z-braces z-end z-java\">}<\/span><\/span><\/span><span class=\"z-punctuation z-section z-parens z-end z-java\">)<\/span><\/span><\/span><span class=\"z-punctuation z-accessor z-dot z-java\">.<\/span><span class=\"z-meta z-function-call z-java\"><span class=\"z-variable z-function z-java\">getNew<\/span><span class=\"z-punctuation z-section z-parens z-begin z-java\">(<\/span><span class=\"z-punctuation z-section z-parens z-end z-java\">)<\/span><\/span><span class=\"z-punctuation z-section z-parens z-end z-java\">)<\/span><\/span><span class=\"z-punctuation z-terminator z-java\">;<\/span>\n<\/span><\/span><\/span><\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-class z-java\"><span class=\"z-meta z-class z-body z-java\"><span class=\"z-meta z-block z-java\"><span class=\"z-meta z-method z-java\"><span class=\"z-meta z-method z-body z-java\">    <span class=\"z-punctuation z-section z-block z-end z-java\">}<\/span><\/span><\/span>\n<\/span><\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-class z-java\"><span class=\"z-meta z-class z-body z-java\"><span class=\"z-meta z-block z-java\"><span class=\"z-punctuation z-section z-block z-end z-java\">}<\/span><\/span><\/span><\/span>\n<\/span><\/code><\/pre>\n<pre data-lang=\"sh\" class=\"language-sh z-code\"><code class=\"language-sh\" data-lang=\"sh\"><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\"><span class=\"z-meta z-group z-expansion z-job z-shell\"><span class=\"z-punctuation z-definition z-variable z-job z-shell\">%<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> javac <span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-punctuation z-definition z-variable z-shell\">$<\/span><span class=\"z-variable z-language z-shell\">_<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">Note:<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> copy.java uses unchecked or unsafe operations.<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">Note:<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> Recompile with<span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> -<\/span>Xlint<\/span>:unchecked for details.<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">jyn@debian-thinkpad<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> <span class=\"z-meta z-group z-expansion z-tilde\"><span class=\"z-variable z-language z-tilde z-shell\">~<\/span><\/span>\/Documents\/Programming\/Java\/test<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\"><span class=\"z-meta z-group z-expansion z-job z-shell\"><span class=\"z-punctuation z-definition z-variable z-job z-shell\">%<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> java copy <\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">[Ljava.lang.Integer<\/span><\/span><span class=\"z-keyword z-operator z-logical z-continue z-shell\">;<\/span><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">@28d93b30<\/span><\/span>\n<\/span><\/code><\/pre>\n<h1 id=\"conclusion\">Conclusion<a class=\"zola-anchor\" href=\"#conclusion\" aria-label=\"Anchor link for: conclusion\"><\/a>\n<\/h1>\n<p>I still have no idea what's going on here - my best guess is that due to type erasure,\nthe <a href=\"https:\/\/stackoverflow.com\/a\/95679\">JIT compiler<\/a> doesn't know that the objects in\n<code>array<\/code> are actually valid <code>Integer<\/code>s.<\/p>\n<div class=\"footnote\" name=\"1\"><sup>1<\/sup>For the purpose of this post, I assume that only a shallow copy is necessary.<\/div>\n"},{"title":"Object Oriented Python","published":"2018-03-03T00:00:00+00:00","updated":"2018-03-03T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/object-oriented-python\/"}},"id":"https:\/\/jyn.dev\/object-oriented-python\/","summary":"creating classes and objects in Python","content":"<h1 id=\"intro\">Intro<a class=\"zola-anchor\" href=\"#intro\" aria-label=\"Anchor link for: intro\"><\/a>\n<\/h1>\n<p>Python is a strange language.\nYou can do a lot with it, up to and including shooting your own foot off.<\/p>\n<h2 id=\"python-allows\">Python allows<a class=\"zola-anchor\" href=\"#python-allows\" aria-label=\"Anchor link for: python-allows\"><\/a>\n<\/h2>\n<ul>\n<li>array splicing<\/li>\n<\/ul>\n<pre data-lang=\"python\" class=\"language-python z-code\"><code class=\"language-python\" data-lang=\"python\"><span class=\"z-source z-python\"><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">l<\/span><\/span> <span class=\"z-keyword z-operator z-assignment z-python\">=<\/span> <span class=\"z-meta z-sequence z-list z-python\"><span class=\"z-punctuation z-section z-sequence z-begin z-python\">[<\/span><span class=\"z-constant z-numeric z-integer z-decimal z-python\">1<\/span><span class=\"z-punctuation z-separator z-sequence z-python\">,<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-python\">2<\/span><span class=\"z-punctuation z-separator z-sequence z-python\">,<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-python\">3<\/span><span class=\"z-punctuation z-separator z-sequence z-python\">,<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-python\">4<\/span><span class=\"z-punctuation z-separator z-sequence z-python\">,<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-python\">5<\/span><span class=\"z-punctuation z-section z-sequence z-end z-python\">]<\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span> <span class=\"z-meta z-item-access z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">l<\/span><\/span><\/span><span class=\"z-meta z-item-access z-python\"><span class=\"z-punctuation z-section z-brackets z-begin z-python\">[<\/span><\/span><span class=\"z-meta z-item-access z-arguments z-python\"><span class=\"z-punctuation z-separator z-slice z-python\">:<\/span><span class=\"z-constant z-numeric z-integer z-decimal z-python\">2<\/span><\/span><span class=\"z-meta z-item-access z-python\"><span class=\"z-punctuation z-section z-brackets z-end z-python\">]<\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-meta z-sequence z-list z-python\"><span class=\"z-punctuation z-section z-sequence z-begin z-python\">[<\/span><span class=\"z-constant z-numeric z-integer z-decimal z-python\">1<\/span><span class=\"z-punctuation z-separator z-sequence z-python\">,<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-python\">2<\/span><span class=\"z-punctuation z-section z-sequence z-end z-python\">]<\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span> <span class=\"z-meta z-item-access z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">l<\/span><\/span><\/span><span class=\"z-meta z-item-access z-python\"><span class=\"z-punctuation z-section z-brackets z-begin z-python\">[<\/span><\/span><span class=\"z-meta z-item-access z-arguments z-python\"><span class=\"z-constant z-numeric z-integer z-decimal z-python\">2<\/span><span class=\"z-punctuation z-separator z-slice z-python\">:<\/span><\/span><span class=\"z-meta z-item-access z-python\"><span class=\"z-punctuation z-section z-brackets z-end z-python\">]<\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-meta z-sequence z-list z-python\"><span class=\"z-punctuation z-section z-sequence z-begin z-python\">[<\/span><span class=\"z-constant z-numeric z-integer z-decimal z-python\">3<\/span><span class=\"z-punctuation z-separator z-sequence z-python\">,<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-python\">4<\/span><span class=\"z-punctuation z-separator z-sequence z-python\">,<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-python\">5<\/span><span class=\"z-punctuation z-section z-sequence z-end z-python\">]<\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span> <span class=\"z-meta z-item-access z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">l<\/span><\/span><\/span><span class=\"z-meta z-item-access z-python\"><span class=\"z-punctuation z-section z-brackets z-begin z-python\">[<\/span><\/span><span class=\"z-meta z-item-access z-arguments z-python\"><span class=\"z-punctuation z-separator z-slice z-python\">:<\/span><span class=\"z-punctuation z-separator z-slice z-python\">:<\/span><span class=\"z-constant z-numeric z-integer z-decimal z-python\">2<\/span><\/span><span class=\"z-meta z-item-access z-python\"><span class=\"z-punctuation z-section z-brackets z-end z-python\">]<\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-meta z-sequence z-list z-python\"><span class=\"z-punctuation z-section z-sequence z-begin z-python\">[<\/span><span class=\"z-constant z-numeric z-integer z-decimal z-python\">1<\/span><span class=\"z-punctuation z-separator z-sequence z-python\">,<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-python\">3<\/span><span class=\"z-punctuation z-separator z-sequence z-python\">,<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-python\">5<\/span><span class=\"z-punctuation z-section z-sequence z-end z-python\">]<\/span><\/span>\n<\/span><\/code><\/pre>\n<ul>\n<li>appending lists with the '+' operator<\/li>\n<\/ul>\n<pre data-lang=\"python\" class=\"language-python z-code\"><code class=\"language-python\" data-lang=\"python\"><span class=\"z-source z-python\"><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span> <span class=\"z-meta z-sequence z-list z-python\"><span class=\"z-punctuation z-section z-sequence z-begin z-python\">[<\/span><span class=\"z-constant z-numeric z-integer z-decimal z-python\">1<\/span><span class=\"z-punctuation z-separator z-sequence z-python\">,<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-python\">2<\/span><span class=\"z-punctuation z-separator z-sequence z-python\">,<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-python\">3<\/span><span class=\"z-punctuation z-section z-sequence z-end z-python\">]<\/span><\/span> <span class=\"z-keyword z-operator z-arithmetic z-python\">+<\/span> <span class=\"z-meta z-sequence z-list z-python\"><span class=\"z-punctuation z-section z-sequence z-begin z-python\">[<\/span><span class=\"z-constant z-numeric z-integer z-decimal z-python\">4<\/span><span class=\"z-punctuation z-separator z-sequence z-python\">,<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-python\">5<\/span><span class=\"z-punctuation z-separator z-sequence z-python\">,<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-python\">6<\/span><span class=\"z-punctuation z-section z-sequence z-end z-python\">]<\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-meta z-sequence z-list z-python\"><span class=\"z-punctuation z-section z-sequence z-begin z-python\">[<\/span><span class=\"z-constant z-numeric z-integer z-decimal z-python\">1<\/span><span class=\"z-punctuation z-separator z-sequence z-python\">,<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-python\">2<\/span><span class=\"z-punctuation z-separator z-sequence z-python\">,<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-python\">3<\/span><span class=\"z-punctuation z-separator z-sequence z-python\">,<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-python\">4<\/span><span class=\"z-punctuation z-separator z-sequence z-python\">,<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-python\">5<\/span><span class=\"z-punctuation z-separator z-sequence z-python\">,<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-python\">6<\/span><span class=\"z-punctuation z-section z-sequence z-end z-python\">]<\/span><\/span>\n<\/span><\/code><\/pre>\n<ul>\n<li>using the first string of a progam as documentation<\/li>\n<\/ul>\n<pre data-lang=\"bash\" class=\"language-bash z-code\"><code class=\"language-bash\" data-lang=\"bash\"><span class=\"z-source z-shell z-bash\"> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\"><span class=\"z-meta z-group z-expansion z-job z-shell\"><span class=\"z-punctuation z-definition z-variable z-job z-shell\">%<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> cat test.py<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\"><span class=\"z-string z-quoted z-single z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&#39;<\/span><span class=\"z-punctuation z-definition z-string z-end z-shell\">&#39;<\/span><\/span><span class=\"z-string z-quoted z-single z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&#39;<\/span>This is a docstring.<span class=\"z-punctuation z-definition z-string z-end z-shell\">&#39;<\/span><\/span><span class=\"z-string z-quoted z-single z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&#39;<\/span><span class=\"z-punctuation z-definition z-string z-end z-shell\">&#39;<\/span><\/span><\/span><\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">print<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\">(__doc__<\/span><span class=\"z-meta z-function-call z-shell\"><\/span>)\n<\/span><span class=\"z-source z-shell z-bash\"> <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\"><span class=\"z-meta z-group z-expansion z-job z-shell\"><span class=\"z-punctuation z-definition z-variable z-job z-shell\">%<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> python <span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-punctuation z-definition z-variable z-shell\">$<\/span><span class=\"z-variable z-language z-shell\">_<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">This<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> is a docstring.<\/span>\n<\/span><\/code><\/pre>\n<ul>\n<li>renaming objects as they are imported<\/li>\n<\/ul>\n<pre data-lang=\"python\" class=\"language-python z-code\"><code class=\"language-python\" data-lang=\"python\"><span class=\"z-source z-python\"><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span> <span class=\"z-meta z-statement z-import z-python\"><span class=\"z-keyword z-control z-import z-from z-python\">from<\/span><\/span><span class=\"z-meta z-statement z-import z-python\"><span class=\"z-meta z-import-source z-python\"> <span class=\"z-meta z-import-path z-python\"><span class=\"z-meta z-import-name z-python\">sys<\/span><\/span> <span class=\"z-meta z-statement z-import z-python\"><span class=\"z-keyword z-control z-import z-python\">import<\/span><\/span><\/span><\/span><span class=\"z-meta z-statement z-import z-python\"><\/span><span class=\"z-meta z-statement z-import z-python\"> <span class=\"z-meta z-generic-name z-python\">stdin<\/span> <span class=\"z-keyword z-control z-import z-as z-python\">as<\/span> <span class=\"z-meta z-generic-name z-python\">s<\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">s<\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-keyword z-operator z-comparison z-python\">&lt;<\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">_io<\/span><span class=\"z-punctuation z-accessor z-dot z-python\">.<\/span><span class=\"z-meta z-generic-name z-python\">TextIOWrapper<\/span><\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">name<\/span><\/span><span class=\"z-keyword z-operator z-assignment z-python\">=<\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-single z-python\"><span class=\"z-punctuation z-definition z-string z-begin z-python\">&#39;<\/span><\/span><\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-single z-python\">&lt;stdin&gt;<span class=\"z-punctuation z-definition z-string z-end z-python\">&#39;<\/span><\/span><\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">mode<\/span><\/span><span class=\"z-keyword z-operator z-assignment z-python\">=<\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-single z-python\"><span class=\"z-punctuation z-definition z-string z-begin z-python\">&#39;<\/span><\/span><\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-single z-python\">r<span class=\"z-punctuation z-definition z-string z-end z-python\">&#39;<\/span><\/span><\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">encoding<\/span><\/span><span class=\"z-keyword z-operator z-assignment z-python\">=<\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-single z-python\"><span class=\"z-punctuation z-definition z-string z-begin z-python\">&#39;<\/span><\/span><\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-single z-python\">UTF-8<span class=\"z-punctuation z-definition z-string z-end z-python\">&#39;<\/span><\/span><\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span>\n<\/span><\/code><\/pre>\n<ul>\n<li>modifying the type of a variable at runtime<\/li>\n<\/ul>\n<pre data-lang=\"python\" class=\"language-python z-code\"><code class=\"language-python\" data-lang=\"python\"><span class=\"z-source z-python\"><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span> <span class=\"z-meta z-statement z-import z-python\"><span class=\"z-keyword z-control z-import z-python\">import<\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">sys<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">x<\/span><\/span> <span class=\"z-keyword z-operator z-assignment z-python\">=<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-python\">2<\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">x<\/span><\/span> <span class=\"z-keyword z-operator z-assignment z-python\">=<\/span> <span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-single z-python\"><span class=\"z-punctuation z-definition z-string z-begin z-python\">&#39;<\/span><\/span><\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-single z-python\">hi<span class=\"z-punctuation z-definition z-string z-end z-python\">&#39;<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">x<\/span><\/span> <span class=\"z-keyword z-operator z-assignment z-python\">=<\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">sys<\/span><span class=\"z-punctuation z-accessor z-dot z-python\">.<\/span><span class=\"z-meta z-generic-name z-python\">stdin<\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span> <span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-support z-function z-builtin z-python\">print<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">x<\/span><\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-keyword z-operator z-comparison z-python\">&lt;<\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">_io<\/span><span class=\"z-punctuation z-accessor z-dot z-python\">.<\/span><span class=\"z-meta z-generic-name z-python\">TextIOWrapper<\/span><\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">name<\/span><\/span><span class=\"z-keyword z-operator z-assignment z-python\">=<\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-single z-python\"><span class=\"z-punctuation z-definition z-string z-begin z-python\">&#39;<\/span><\/span><\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-single z-python\">&lt;stdin&gt;<span class=\"z-punctuation z-definition z-string z-end z-python\">&#39;<\/span><\/span><\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">mode<\/span><\/span><span class=\"z-keyword z-operator z-assignment z-python\">=<\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-single z-python\"><span class=\"z-punctuation z-definition z-string z-begin z-python\">&#39;<\/span><\/span><\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-single z-python\">r<span class=\"z-punctuation z-definition z-string z-end z-python\">&#39;<\/span><\/span><\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">encoding<\/span><\/span><span class=\"z-keyword z-operator z-assignment z-python\">=<\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-single z-python\"><span class=\"z-punctuation z-definition z-string z-begin z-python\">&#39;<\/span><\/span><\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-single z-python\">UTF-8<span class=\"z-punctuation z-definition z-string z-end z-python\">&#39;<\/span><\/span><\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span>\n<\/span><\/code><\/pre>\n<ul>\n<li>within a function, using a variable that has not yet been declared<\/li>\n<\/ul>\n<pre data-lang=\"python\" class=\"language-python z-code\"><code class=\"language-python\" data-lang=\"python\"><span class=\"z-source z-python\"><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-invalid z-illegal z-name z-python\">def<\/span><\/span> <span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-meta z-generic-name z-python\">add_global<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span><span class=\"z-punctuation z-separator z-annotation z-variable z-python\">:<\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-constant z-language z-python\">...<\/span>     <span class=\"z-storage z-modifier z-global z-python\">global<\/span> <span class=\"z-meta z-generic-name z-python\">x<\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-constant z-language z-python\">...<\/span>     <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">x<\/span><\/span> <span class=\"z-keyword z-operator z-assignment z-augmented z-python\">+=<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-python\">2<\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-constant z-language z-python\">...<\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">x<\/span><\/span> <span class=\"z-keyword z-operator z-assignment z-python\">=<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-python\">0<\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span> <span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-meta z-generic-name z-python\">add_global<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span><span class=\"z-punctuation z-terminator z-statement z-python\">;<\/span> <span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-support z-function z-builtin z-python\">print<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">x<\/span><\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-constant z-numeric z-integer z-decimal z-python\">2<\/span>\n<\/span><\/code><\/pre>\n<p><br><br><\/p>\n<h2 id=\"python-disallows\">Python disallows<a class=\"zola-anchor\" href=\"#python-disallows\" aria-label=\"Anchor link for: python-disallows\"><\/a>\n<\/h2>\n<ul>\n<li>declaring types<\/li>\n<\/ul>\n<pre data-lang=\"python\" class=\"language-python z-code\"><code class=\"language-python\" data-lang=\"python\"><span class=\"z-source z-python\"><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-support z-type z-python\">int<\/span><\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">x<\/span><\/span> <span class=\"z-keyword z-operator z-assignment z-python\">=<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-python\">2<\/span>\n<\/span><span class=\"z-source z-python\">  <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">File<\/span><\/span> <span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\"><span class=\"z-punctuation z-definition z-string z-begin z-python\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\">&lt;stdin&gt;<span class=\"z-punctuation z-definition z-string z-end z-python\">&quot;<\/span><\/span><\/span>, <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">line<\/span><\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-python\">1<\/span>\n<\/span><span class=\"z-source z-python\">    <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-support z-type z-python\">int<\/span><\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">x<\/span><\/span> <span class=\"z-keyword z-operator z-assignment z-python\">=<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-python\">2<\/span>\n<\/span><span class=\"z-source z-python\">        <span class=\"z-keyword z-operator z-arithmetic z-python\">^<\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-support z-type z-exception z-python\">SyntaxError<\/span><\/span><span class=\"z-punctuation z-separator z-annotation z-variable z-python\">:<\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">invalid<\/span><\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">syntax<\/span><\/span>\n<\/span><\/code><\/pre>\n<ul>\n<li>throwing errors before runtime<\/li>\n<\/ul>\n<pre data-lang=\"python\" class=\"language-python z-code\"><code class=\"language-python\" data-lang=\"python\"><span class=\"z-source z-python\"><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-invalid z-illegal z-name z-python\">def<\/span><\/span> <span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-meta z-generic-name z-python\">add_local<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span><span class=\"z-punctuation z-separator z-annotation z-variable z-python\">:<\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-constant z-language z-python\">...<\/span>     <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">x<\/span><\/span> <span class=\"z-keyword z-operator z-assignment z-augmented z-python\">+=<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-python\">2<\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-constant z-language z-python\">...<\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span> <span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-meta z-generic-name z-python\">add_local<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-meta z-generic-name z-python\">Traceback<\/span><\/span><\/span> <\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">most<\/span><\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">recent<\/span><\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">call<\/span><\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">last<\/span><\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span><span class=\"z-punctuation z-separator z-annotation z-variable z-python\">:<\/span>\n<\/span><span class=\"z-source z-python\">  <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">File<\/span><\/span> <span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\"><span class=\"z-punctuation z-definition z-string z-begin z-python\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\">&lt;stdin&gt;<span class=\"z-punctuation z-definition z-string z-end z-python\">&quot;<\/span><\/span><\/span>, <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">line<\/span><\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-python\">1<\/span>, <span class=\"z-keyword z-operator z-logical z-python\">in<\/span> <span class=\"z-keyword z-operator z-comparison z-python\">&lt;<\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">module<\/span><\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span>\n<\/span><span class=\"z-source z-python\">  <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">File<\/span><\/span> <span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\"><span class=\"z-punctuation z-definition z-string z-begin z-python\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\">&lt;stdin&gt;<span class=\"z-punctuation z-definition z-string z-end z-python\">&quot;<\/span><\/span><\/span>, <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">line<\/span><\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-python\">2<\/span>, <span class=\"z-keyword z-operator z-logical z-python\">in<\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">add_local<\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-support z-type z-exception z-python\">UnboundLocalError<\/span><\/span><span class=\"z-punctuation z-separator z-annotation z-variable z-python\">:<\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">local<\/span><\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">variable<\/span><\/span> <span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-single z-python\"><span class=\"z-punctuation z-definition z-string z-begin z-python\">&#39;<\/span><\/span><\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-single z-python\">x<span class=\"z-punctuation z-definition z-string z-end z-python\">&#39;<\/span><\/span><\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">referenced<\/span><\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">before<\/span><\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">assignment<\/span><\/span>\n<\/span><\/code><\/pre>\n<ul>\n<li>declaring and assigning a global variable in the same statement<\/li>\n<\/ul>\n<pre data-lang=\"python\" class=\"language-python z-code\"><code class=\"language-python\" data-lang=\"python\"><span class=\"z-source z-python\"><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-invalid z-illegal z-name z-python\">def<\/span><\/span> <span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-meta z-generic-name z-python\">myfunc<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span><span class=\"z-punctuation z-separator z-annotation z-variable z-python\">:<\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-constant z-language z-python\">...<\/span>     <span class=\"z-storage z-modifier z-global z-python\">global<\/span> <span class=\"z-meta z-generic-name z-python\">x<\/span> <span class=\"z-invalid z-illegal z-name z-storage z-python\">=<\/span> <span class=\"z-invalid z-illegal z-name z-storage z-python\">1<\/span>\n<\/span><span class=\"z-source z-python\">  <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">File<\/span><\/span> <span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\"><span class=\"z-punctuation z-definition z-string z-begin z-python\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\">&lt;stdin&gt;<span class=\"z-punctuation z-definition z-string z-end z-python\">&quot;<\/span><\/span><\/span>, <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">line<\/span><\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-python\">2<\/span>\n<\/span><span class=\"z-source z-python\">    <span class=\"z-storage z-modifier z-global z-python\">global<\/span> <span class=\"z-meta z-generic-name z-python\">x<\/span> <span class=\"z-invalid z-illegal z-name z-storage z-python\">=<\/span> <span class=\"z-invalid z-illegal z-name z-storage z-python\">1<\/span>\n<\/span><span class=\"z-source z-python\">             <span class=\"z-keyword z-operator z-arithmetic z-python\">^<\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-support z-type z-exception z-python\">SyntaxError<\/span><\/span><span class=\"z-punctuation z-separator z-annotation z-variable z-python\">:<\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">invalid<\/span><\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">syntax<\/span><\/span>\n<\/span><\/code><\/pre>\n<ul>\n<li>bad indentation<\/li>\n<\/ul>\n<pre data-lang=\"python\" class=\"language-python z-code\"><code class=\"language-python\" data-lang=\"python\"><span class=\"z-source z-python\"><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span> <span class=\"z-meta z-statement z-loop z-for z-python\"><span class=\"z-keyword z-control z-loop z-for z-python\">for<\/span> <span class=\"z-meta z-generic-name z-python\">i<\/span> <span class=\"z-keyword z-control z-loop z-for z-in z-python\">in<\/span><\/span><span class=\"z-meta z-statement z-loop z-for z-python\"> <span class=\"z-meta z-sequence z-list z-python\"><span class=\"z-punctuation z-section z-sequence z-begin z-python\">[<\/span><span class=\"z-constant z-numeric z-integer z-decimal z-python\">1<\/span><span class=\"z-punctuation z-separator z-sequence z-python\">,<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-python\">2<\/span><span class=\"z-punctuation z-separator z-sequence z-python\">,<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-python\">3<\/span><span class=\"z-punctuation z-section z-sequence z-end z-python\">]<\/span><\/span><\/span><span class=\"z-meta z-statement z-loop z-for z-python\"><span class=\"z-punctuation z-section z-block z-loop z-for z-python\">:<\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-constant z-language z-python\">...<\/span> <span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-support z-function z-builtin z-python\">print<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">i<\/span><\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span>\n<\/span><span class=\"z-source z-python\">  <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">File<\/span><\/span> <span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\"><span class=\"z-punctuation z-definition z-string z-begin z-python\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\">&lt;stdin&gt;<span class=\"z-punctuation z-definition z-string z-end z-python\">&quot;<\/span><\/span><\/span>, <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">line<\/span><\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-python\">2<\/span>\n<\/span><span class=\"z-source z-python\">    <span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-support z-function z-builtin z-python\">print<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">i<\/span><\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span>\n<\/span><span class=\"z-source z-python\">        <span class=\"z-keyword z-operator z-arithmetic z-python\">^<\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-support z-type z-exception z-python\">IndentationError<\/span><\/span><span class=\"z-punctuation z-separator z-annotation z-variable z-python\">:<\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">expected<\/span><\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">an<\/span><\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">indented<\/span><\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">block<\/span><\/span>\n<\/span><\/code><\/pre>\n<ul>\n<li><a href=\"https:\/\/stackoverflow.com\/questions\/13591970\/\">tail call optimization<\/a><\/li>\n<li><a href=\"https:\/\/www.python.org\/dev\/peps\/pep-3103\/\">switch statements<\/a><\/li>\n<\/ul>\n<p>In this article, I go over some of the things you can do with object-oriented python,\nsome things you should do, and some you really shouldn't.<\/p>\n<h1 id=\"objects-in-python\">Objects in python<a class=\"zola-anchor\" href=\"#objects-in-python\" aria-label=\"Anchor link for: objects-in-python\"><\/a>\n<\/h1>\n<p>In other object-oriented languages I am familiar with, notably Java,\neach object has its own methods you must learn to use.\n<a href=\"https:\/\/stackoverflow.com\/questions\/1965500\/\"><code>size<\/code>, <code>length<\/code>, and <code>length()<\/code><\/a>,\nfor example, are easily confused even for experienced developers.<\/p>\n<p>In python, there is only one function, and it is not a property of the object.<\/p>\n<pre data-lang=\"python\" class=\"language-python z-code\"><code class=\"language-python\" data-lang=\"python\"><span class=\"z-source z-python\"><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span> <span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-support z-function z-builtin z-python\">len<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-meta z-sequence z-list z-python\"><span class=\"z-punctuation z-section z-sequence z-begin z-python\">[<\/span><span class=\"z-constant z-numeric z-integer z-decimal z-python\">1<\/span><span class=\"z-punctuation z-separator z-sequence z-python\">,<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-python\">2<\/span><span class=\"z-punctuation z-separator z-sequence z-python\">,<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-python\">3<\/span><span class=\"z-punctuation z-section z-sequence z-end z-python\">]<\/span><\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-constant z-numeric z-integer z-decimal z-python\">3<\/span>\n<\/span><\/code><\/pre>\n<p>Similarly, all array-like objects can be treated as arrays:<\/p>\n<pre data-lang=\"python\" class=\"language-python z-code\"><code class=\"language-python\" data-lang=\"python\"><span class=\"z-source z-python\"><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span> <span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\"><span class=\"z-punctuation z-definition z-string z-begin z-python\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\">135<span class=\"z-punctuation z-definition z-string z-end z-python\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-item-access z-python\"><span class=\"z-punctuation z-section z-brackets z-begin z-python\">[<\/span><\/span><span class=\"z-meta z-item-access z-arguments z-python\"><span class=\"z-constant z-numeric z-integer z-decimal z-python\">2<\/span><\/span><span class=\"z-meta z-item-access z-python\"><span class=\"z-punctuation z-section z-brackets z-end z-python\">]<\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-single z-python\"><span class=\"z-punctuation z-definition z-string z-begin z-python\">&#39;<\/span><\/span><\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-single z-python\">5<span class=\"z-punctuation z-definition z-string z-end z-python\">&#39;<\/span><\/span><\/span>\n<\/span><\/code><\/pre>\n<h2 id=\"how-is-this-done\">How is this done?<a class=\"zola-anchor\" href=\"#how-is-this-done\" aria-label=\"Anchor link for: how-is-this-done\"><\/a>\n<\/h2>\n<p>In Java, C++, and every other language I can think of,\nobjects are essentially implemented as very complicated structs:\nthere are pointers to data and void pointers to functions.\nAll object pointers are void pointers at runtime;\nthis is safe because types are checked at compile time.\nHow would you implement a global type-independent <code>length<\/code> function\nwhen you don't know what length means for that object?\nThe closest you could get is a wrapper:<\/p>\n<pre data-lang=\"java\" class=\"language-java z-code\"><code class=\"language-java\" data-lang=\"java\"><span class=\"z-source z-java\"><span class=\"z-storage z-modifier z-java\">public<\/span> <span class=\"z-storage z-modifier z-java\">static<\/span> <span class=\"z-storage z-type z-primitive z-java\">int<\/span> <span class=\"z-meta z-function-call z-java\"><span class=\"z-variable z-function z-java\">length<\/span><span class=\"z-punctuation z-section z-parens z-begin z-java\">(<\/span><span class=\"z-support z-class z-java\">T<\/span> object<span class=\"z-punctuation z-section z-parens z-end z-java\">)<\/span><\/span> <span class=\"z-meta z-block z-java\"><span class=\"z-punctuation z-section z-block z-begin z-java\">{<\/span>\n<\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-block z-java\">    <span class=\"z-keyword z-control z-conditional z-switch z-java\">switch<\/span><span class=\"z-meta z-parens z-java\"><span class=\"z-punctuation z-section z-parens z-begin z-java\">(<\/span>object<span class=\"z-punctuation z-accessor z-dot z-java\">.<\/span><span class=\"z-variable z-language z-java\">class<\/span><span class=\"z-punctuation z-section z-parens z-end z-java\">)<\/span><\/span> <span class=\"z-meta z-block z-java\"><span class=\"z-punctuation z-section z-block z-begin z-java\">{<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-block z-java\"><span class=\"z-meta z-block z-java\">    <span class=\"z-keyword z-control z-conditional z-case z-java\">case<\/span> <span class=\"z-support z-class z-java\">Array<\/span><span class=\"z-punctuation z-accessor z-dot z-java\">.<\/span><span class=\"z-variable z-language z-java\">class<\/span><span class=\"z-keyword z-operator z-ternary z-java\">:<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-block z-java\"><span class=\"z-meta z-block z-java\">        <span class=\"z-keyword z-control z-flow z-return z-java\">return<\/span> object<span class=\"z-punctuation z-accessor z-dot z-java\">.<\/span>length<span class=\"z-punctuation z-terminator z-java\">;<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-block z-java\"><span class=\"z-meta z-block z-java\">    <span class=\"z-keyword z-control z-conditional z-case z-java\">case<\/span> <span class=\"z-support z-class z-java\">List<\/span><span class=\"z-punctuation z-accessor z-dot z-java\">.<\/span><span class=\"z-variable z-language z-java\">class<\/span><span class=\"z-keyword z-operator z-ternary z-java\">:<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-block z-java\"><span class=\"z-meta z-block z-java\">        <span class=\"z-keyword z-control z-flow z-return z-java\">return<\/span> object<span class=\"z-punctuation z-accessor z-dot z-java\">.<\/span><span class=\"z-meta z-function-call z-java\"><span class=\"z-variable z-function z-java\">size<\/span><span class=\"z-punctuation z-section z-parens z-begin z-java\">(<\/span><span class=\"z-punctuation z-section z-parens z-end z-java\">)<\/span><\/span><span class=\"z-punctuation z-terminator z-java\">;<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-block z-java\"><span class=\"z-meta z-block z-java\">    <span class=\"z-keyword z-control z-conditional z-case z-java\">case<\/span> <span class=\"z-support z-class z-java\">String<\/span><span class=\"z-punctuation z-accessor z-dot z-java\">.<\/span><span class=\"z-variable z-language z-java\">class<\/span><span class=\"z-keyword z-operator z-ternary z-java\">:<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-block z-java\"><span class=\"z-meta z-block z-java\">        <span class=\"z-keyword z-control z-flow z-return z-java\">return<\/span> object<span class=\"z-punctuation z-accessor z-dot z-java\">.<\/span><span class=\"z-meta z-function-call z-java\"><span class=\"z-variable z-function z-java\">length<\/span><span class=\"z-punctuation z-section z-parens z-begin z-java\">(<\/span><span class=\"z-punctuation z-section z-parens z-end z-java\">)<\/span><\/span><span class=\"z-punctuation z-terminator z-java\">;<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-block z-java\"><span class=\"z-meta z-block z-java\">    <span class=\"z-punctuation z-accessor z-dot z-java\">.<\/span><span class=\"z-punctuation z-accessor z-dot z-java\">.<\/span><span class=\"z-punctuation z-accessor z-dot z-java\">.<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-java\"><span class=\"z-meta z-block z-java\"><span class=\"z-meta z-block z-java\"><span class=\"z-punctuation z-section z-block z-end z-java\">}<\/span><\/span>\n<\/span><\/span><\/code><\/pre>\n<p>It's a nightmare just to write 3 of those switch cases;\nimagine how awful it would be for every object in <code>java.util<\/code>.<\/p>\n<p>Well, it turns out, that wrapper is essentially what python does.\nWhen you run <code>len(object)<\/code>, all it does is <code>return object.__len__()<\/code>!\nIt's the same for every other attribute -\naccessing an item, iterating, the whole\n<a href=\"https:\/\/en.wikipedia.org\/wiki\/Shebang_(Unix)\">shebang<\/a>.<\/p>\n<pre data-lang=\"python\" class=\"language-python z-code\"><code class=\"language-python\" data-lang=\"python\"><span class=\"z-source z-python\"><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-invalid z-illegal z-name z-python\">class<\/span><\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">obj<\/span><\/span><span class=\"z-punctuation z-separator z-annotation z-variable z-python\">:<\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-constant z-language z-python\">...<\/span>     <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-invalid z-illegal z-name z-python\">def<\/span><\/span> <span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-support z-function z-magic z-python\">__len__<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-language z-python\">self<\/span><\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span><span class=\"z-punctuation z-separator z-annotation z-variable z-python\">:<\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-constant z-language z-python\">...<\/span>         <span class=\"z-keyword z-control z-flow z-return z-python\">return<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-python\">4<\/span>  <span class=\"z-comment z-line z-number-sign z-python\"><span class=\"z-punctuation z-definition z-comment z-python\">#<\/span> guarenteed by IEEE to be random\n<\/span><\/span><span class=\"z-source z-python\"><span class=\"z-constant z-language z-python\">...<\/span>     <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-invalid z-illegal z-name z-python\">def<\/span><\/span> <span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-support z-function z-magic z-python\">__getitem__<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-language z-python\">self<\/span><\/span><span class=\"z-punctuation z-separator z-arguments z-python\">,<\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">index<\/span><\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span><span class=\"z-punctuation z-separator z-annotation z-variable z-python\">:<\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-constant z-language z-python\">...<\/span>         <span class=\"z-keyword z-control z-flow z-return z-python\">return<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-python\">3<\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-constant z-language z-python\">...<\/span>     <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-invalid z-illegal z-name z-python\">def<\/span><\/span> <span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-support z-function z-magic z-python\">__iter__<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-language z-python\">self<\/span><\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span><span class=\"z-punctuation z-separator z-annotation z-variable z-python\">:<\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-constant z-language z-python\">...<\/span>         <span class=\"z-keyword z-control z-flow z-yield z-python\">yield<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-python\">1<\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-constant z-language z-python\">...<\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">o<\/span><\/span> <span class=\"z-keyword z-operator z-assignment z-python\">=<\/span> <span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-meta z-generic-name z-python\">obj<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span> <span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-support z-function z-builtin z-python\">len<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">o<\/span><\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-constant z-numeric z-integer z-decimal z-python\">4<\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span> <span class=\"z-meta z-item-access z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">o<\/span><\/span><\/span><span class=\"z-meta z-item-access z-python\"><span class=\"z-punctuation z-section z-brackets z-begin z-python\">[<\/span><\/span><span class=\"z-meta z-item-access z-arguments z-python\"><span class=\"z-constant z-numeric z-integer z-decimal z-python\">613<\/span><\/span><span class=\"z-meta z-item-access z-python\"><span class=\"z-punctuation z-section z-brackets z-end z-python\">]<\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-constant z-numeric z-integer z-decimal z-python\">3<\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span> <span class=\"z-meta z-statement z-loop z-for z-python\"><span class=\"z-keyword z-control z-loop z-for z-python\">for<\/span> <span class=\"z-meta z-generic-name z-python\">i<\/span> <span class=\"z-keyword z-control z-loop z-for z-in z-python\">in<\/span><\/span><span class=\"z-meta z-statement z-loop z-for z-python\"> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">o<\/span><\/span><\/span><span class=\"z-meta z-statement z-loop z-for z-python\"><span class=\"z-punctuation z-section z-block z-loop z-for z-python\">:<\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-constant z-language z-python\">...<\/span>     <span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-support z-function z-builtin z-python\">print<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">i<\/span><\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-constant z-language z-python\">...<\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-constant z-numeric z-integer z-decimal z-python\">1<\/span>\n<\/span><\/code><\/pre>\n<p>We have to <code>yield<\/code> 1 instead of <code>return<\/code> because <code>iter<\/code> takes a\n<a href=\"https:\/\/wiki.python.org\/moin\/Generators\">generator<\/a>.<\/p>\n<h2 id=\"using-objects\">Using objects<a class=\"zola-anchor\" href=\"#using-objects\" aria-label=\"Anchor link for: using-objects\"><\/a>\n<\/h2>\n<p>So, now we know how objects work -\nall we have to do is implement the builtin methods.\nHow do you extend a class, though?\nJava's entire paradigm is built around inheritance.<\/p>\n<p>You extend classes in Python by passing them to the constructor:<\/p>\n<pre data-lang=\"python\" class=\"language-python z-code\"><code class=\"language-python\" data-lang=\"python\"><span class=\"z-source z-python\"><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-invalid z-illegal z-name z-python\">class<\/span><\/span> <span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-meta z-generic-name z-python\">my_list<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-support z-type z-python\">list<\/span><\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span><span class=\"z-punctuation z-separator z-annotation z-variable z-python\">:<\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-constant z-language z-python\">...<\/span>     <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-invalid z-illegal z-name z-python\">def<\/span><\/span> <span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-meta z-generic-name z-python\">my_function<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-language z-python\">self<\/span><\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span><span class=\"z-punctuation z-separator z-annotation z-variable z-python\">:<\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-constant z-language z-python\">...<\/span>             <span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-support z-function z-builtin z-python\">print<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\"><span class=\"z-punctuation z-definition z-string z-begin z-python\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\">Hello from my_list!<span class=\"z-punctuation z-definition z-string z-end z-python\">&quot;<\/span><\/span><\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-constant z-language z-python\">...<\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span> <span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-meta z-generic-name z-python\">my_list<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-meta z-sequence z-list z-empty z-python\"><span class=\"z-punctuation z-section z-sequence z-begin z-python\">[<\/span><span class=\"z-punctuation z-section z-sequence z-end z-python\">]<\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span> <span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-meta z-generic-name z-python\">my_list<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-meta z-sequence z-list z-python\"><span class=\"z-punctuation z-section z-sequence z-begin z-python\">[<\/span><span class=\"z-constant z-numeric z-integer z-decimal z-python\">1<\/span><span class=\"z-punctuation z-separator z-sequence z-python\">,<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-python\">2<\/span><span class=\"z-punctuation z-section z-sequence z-end z-python\">]<\/span><\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-meta z-sequence z-list z-python\"><span class=\"z-punctuation z-section z-sequence z-begin z-python\">[<\/span><span class=\"z-constant z-numeric z-integer z-decimal z-python\">1<\/span><span class=\"z-punctuation z-separator z-sequence z-python\">,<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-python\">2<\/span><span class=\"z-punctuation z-section z-sequence z-end z-python\">]<\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span> <span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-meta z-generic-name z-python\">my_list<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-meta z-sequence z-list z-python\"><span class=\"z-punctuation z-section z-sequence z-begin z-python\">[<\/span><span class=\"z-constant z-numeric z-integer z-decimal z-python\">1<\/span><span class=\"z-punctuation z-separator z-sequence z-python\">,<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-python\">2<\/span><span class=\"z-punctuation z-section z-sequence z-end z-python\">]<\/span><\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span><span class=\"z-meta z-item-access z-python\"><span class=\"z-punctuation z-section z-brackets z-begin z-python\">[<\/span><\/span><span class=\"z-meta z-item-access z-arguments z-python\"><span class=\"z-constant z-numeric z-integer z-decimal z-python\">0<\/span><\/span><span class=\"z-meta z-item-access z-python\"><span class=\"z-punctuation z-section z-brackets z-end z-python\">]<\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-constant z-numeric z-integer z-decimal z-python\">1<\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span> <span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-meta z-generic-name z-python\">my_list<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-meta z-sequence z-list z-python\"><span class=\"z-punctuation z-section z-sequence z-begin z-python\">[<\/span><span class=\"z-constant z-numeric z-integer z-decimal z-python\">1<\/span><span class=\"z-punctuation z-separator z-sequence z-python\">,<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-python\">2<\/span><span class=\"z-punctuation z-section z-sequence z-end z-python\">]<\/span><\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span><span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-punctuation z-accessor z-dot z-python\">.<\/span><\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-meta z-generic-name z-python\">my_function<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">Hello<\/span><\/span> <span class=\"z-meta z-statement z-import z-python\"><span class=\"z-keyword z-control z-import z-from z-python\">from<\/span><\/span><span class=\"z-meta z-statement z-import z-python\"><span class=\"z-meta z-import-source z-python\"> <span class=\"z-meta z-import-path z-python\"><span class=\"z-meta z-import-name z-python\">my_list<\/span><\/span><\/span><\/span>!\n<\/span><\/code><\/pre>\n<p>And what's up with that pesky <code>self<\/code>, anyway?<\/p>\n<pre data-lang=\"python\" class=\"language-python z-code\"><code class=\"language-python\" data-lang=\"python\"><span class=\"z-source z-python\"><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-invalid z-illegal z-name z-python\">class<\/span><\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">my_class<\/span><\/span><span class=\"z-punctuation z-separator z-annotation z-variable z-python\">:<\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-constant z-language z-python\">...<\/span>     <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-invalid z-illegal z-name z-python\">def<\/span><\/span> <span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-meta z-generic-name z-python\">my_function<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span><span class=\"z-punctuation z-separator z-annotation z-variable z-python\">:<\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-constant z-language z-python\">...<\/span>             <span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-support z-function z-builtin z-python\">print<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\"><span class=\"z-punctuation z-definition z-string z-begin z-python\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\">Hello from my_function!<span class=\"z-punctuation z-definition z-string z-end z-python\">&quot;<\/span><\/span><\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-constant z-language z-python\">...<\/span>     <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-invalid z-illegal z-name z-python\">def<\/span><\/span> <span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-meta z-generic-name z-python\">my_other_function<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-language z-python\">self<\/span><\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span><span class=\"z-punctuation z-separator z-annotation z-variable z-python\">:<\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-constant z-language z-python\">...<\/span>             <span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-support z-function z-builtin z-python\">print<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\"><span class=\"z-punctuation z-definition z-string z-begin z-python\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\">Hello from my_other_function!<span class=\"z-punctuation z-definition z-string z-end z-python\">&quot;<\/span><\/span><\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-constant z-language z-python\">...<\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">o<\/span><\/span> <span class=\"z-keyword z-operator z-assignment z-python\">=<\/span> <span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-meta z-generic-name z-python\">my_class<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span> <span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">o<\/span><\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-punctuation z-accessor z-dot z-python\">.<\/span><\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-meta z-generic-name z-python\">my_function<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-meta z-generic-name z-python\">Traceback<\/span><\/span><\/span> <\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">most<\/span><\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">recent<\/span><\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">call<\/span><\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">last<\/span><\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span><span class=\"z-punctuation z-separator z-annotation z-variable z-python\">:<\/span>\n<\/span><span class=\"z-source z-python\">  <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">File<\/span><\/span> <span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\"><span class=\"z-punctuation z-definition z-string z-begin z-python\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\">&lt;stdin&gt;<span class=\"z-punctuation z-definition z-string z-end z-python\">&quot;<\/span><\/span><\/span>, <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">line<\/span><\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-python\">1<\/span>, <span class=\"z-keyword z-operator z-logical z-python\">in<\/span> <span class=\"z-keyword z-operator z-comparison z-python\">&lt;<\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">module<\/span><\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-support z-type z-exception z-python\">TypeError<\/span><\/span><span class=\"z-punctuation z-separator z-annotation z-variable z-python\">:<\/span> <span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-meta z-generic-name z-python\">my_function<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">takes<\/span><\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-python\">0<\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">positional<\/span><\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">arguments<\/span><\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">but<\/span><\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-python\">1<\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">was<\/span><\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">given<\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span> <span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">o<\/span><\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-punctuation z-accessor z-dot z-python\">.<\/span><\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-meta z-generic-name z-python\">my_other_function<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">Hello<\/span><\/span> <span class=\"z-meta z-statement z-import z-python\"><span class=\"z-keyword z-control z-import z-from z-python\">from<\/span><\/span><span class=\"z-meta z-statement z-import z-python\"><span class=\"z-meta z-import-source z-python\"> <span class=\"z-meta z-import-path z-python\"><span class=\"z-meta z-import-name z-python\">my_other_function<\/span><\/span><\/span><\/span>!\n<\/span><\/code><\/pre>\n<p>It turns out that Python automatically passes <code>self<\/code> to each method when it's called.\nThe reason for this is one of my least favorite parts of Python:\nthere's no class namespace. If you want to refer to an attribute of the instance,\nyou have to use <code>self<\/code>.<\/p>\n<pre data-lang=\"python\" class=\"language-python z-code\"><code class=\"language-python\" data-lang=\"python\"><span class=\"z-source z-python\"><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-invalid z-illegal z-name z-python\">class<\/span><\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">obj<\/span><\/span><span class=\"z-punctuation z-separator z-annotation z-variable z-python\">:<\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-constant z-language z-python\">...<\/span>     <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-invalid z-illegal z-name z-python\">def<\/span><\/span> <span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-support z-function z-magic z-python\">__init__<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-language z-python\">self<\/span><\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span><span class=\"z-punctuation z-separator z-annotation z-variable z-python\">:<\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-constant z-language z-python\">...<\/span>         <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-language z-python\">self<\/span><span class=\"z-punctuation z-accessor z-dot z-python\">.<\/span><span class=\"z-meta z-generic-name z-python\">num<\/span><\/span> <span class=\"z-keyword z-operator z-assignment z-python\">=<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-python\">3<\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-constant z-language z-python\">...<\/span>     <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-invalid z-illegal z-name z-python\">def<\/span><\/span> <span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-meta z-generic-name z-python\">f<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-language z-python\">self<\/span><\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span><span class=\"z-punctuation z-separator z-annotation z-variable z-python\">:<\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-constant z-language z-python\">...<\/span>         <span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-support z-function z-builtin z-python\">print<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">num<\/span><\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-constant z-language z-python\">...<\/span>     <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-invalid z-illegal z-name z-python\">def<\/span><\/span> <span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-meta z-generic-name z-python\">f2<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-language z-python\">self<\/span><\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span><span class=\"z-punctuation z-separator z-annotation z-variable z-python\">:<\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-constant z-language z-python\">...<\/span>         <span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-support z-function z-builtin z-python\">print<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-language z-python\">self<\/span><span class=\"z-punctuation z-accessor z-dot z-python\">.<\/span><span class=\"z-meta z-generic-name z-python\">num<\/span><\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-constant z-language z-python\">...<\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">o<\/span><\/span> <span class=\"z-keyword z-operator z-assignment z-python\">=<\/span> <span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-meta z-generic-name z-python\">obj<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">o<\/span><span class=\"z-punctuation z-accessor z-dot z-python\">.<\/span><span class=\"z-meta z-generic-name z-python\">num<\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-constant z-numeric z-integer z-decimal z-python\">3<\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span> <span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">o<\/span><\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-punctuation z-accessor z-dot z-python\">.<\/span><\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-meta z-generic-name z-python\">f<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-meta z-generic-name z-python\">Traceback<\/span><\/span><\/span> <\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">most<\/span><\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">recent<\/span><\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">call<\/span><\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">last<\/span><\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span><span class=\"z-punctuation z-separator z-annotation z-variable z-python\">:<\/span>\n<\/span><span class=\"z-source z-python\">  <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">File<\/span><\/span> <span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\"><span class=\"z-punctuation z-definition z-string z-begin z-python\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\">&lt;stdin&gt;<span class=\"z-punctuation z-definition z-string z-end z-python\">&quot;<\/span><\/span><\/span>, <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">line<\/span><\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-python\">1<\/span>, <span class=\"z-keyword z-operator z-logical z-python\">in<\/span> <span class=\"z-keyword z-operator z-comparison z-python\">&lt;<\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">module<\/span><\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span>\n<\/span><span class=\"z-source z-python\">  <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">File<\/span><\/span> <span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\"><span class=\"z-punctuation z-definition z-string z-begin z-python\">&quot;<\/span><\/span><\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-double z-python\">&lt;stdin&gt;<span class=\"z-punctuation z-definition z-string z-end z-python\">&quot;<\/span><\/span><\/span>, <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">line<\/span><\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-python\">5<\/span>, <span class=\"z-keyword z-operator z-logical z-python\">in<\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">f<\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-support z-type z-exception z-python\">NameError<\/span><\/span><span class=\"z-punctuation z-separator z-annotation z-variable z-python\">:<\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">name<\/span><\/span> <span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-single z-python\"><span class=\"z-punctuation z-definition z-string z-begin z-python\">&#39;<\/span><\/span><\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-single z-python\">num<span class=\"z-punctuation z-definition z-string z-end z-python\">&#39;<\/span><\/span><\/span> <span class=\"z-keyword z-operator z-logical z-python\">is<\/span> <span class=\"z-keyword z-operator z-logical z-python\">not<\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">defined<\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span><span class=\"z-keyword z-operator z-comparison z-python\">&gt;<\/span> <span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">o<\/span><\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-punctuation z-accessor z-dot z-python\">.<\/span><\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-meta z-generic-name z-python\">f2<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-constant z-numeric z-integer z-decimal z-python\">3<\/span>\n<\/span><\/code><\/pre>\n<h2 id=\"constructors-and-destructors\">Constructors and Destructors<a class=\"zola-anchor\" href=\"#constructors-and-destructors\" aria-label=\"Anchor link for: constructors-and-destructors\"><\/a>\n<\/h2>\n<p>So . . . about those.<\/p>\n<p>When you instantiate an object in python, it calls\n<a href=\"https:\/\/stackoverflow.com\/q\/674304\"><code>__new__<\/code> then <code>__init__<\/code><\/a>, in that order.\n<code>__init__<\/code> is <code>None<\/code> by default; <code>__new__<\/code> may or may not be.\nIt's considered bad practice to define <code>__new__<\/code> in your program.<\/p>\n<p>When an object is destroyed through garbage collection,\n<code>__del__<\/code> is called.<\/p>\n<p>This sounds simple enough so far.<\/p>\n<p>If <code>__init__<\/code> is interrupted, <code>__del__<\/code>\n<a href=\"https:\/\/stackoverflow.com\/a\/2433847\">may or may not be called<\/a>.<\/p>\n<p>If you use a <code>with<\/code> statement, i.e. <code>with open(filename) as f:<\/code>,\n<em>neither <code>__init__<\/code> nor <code>__del__<\/code> are called<\/em>; <code>__enter__<\/code> and <code>__exit__<\/code>\nare called instead.<\/p>\n<p>I'm still confused by these.<\/p>\n<h2 id=\"conclusion\">Conclusion<a class=\"zola-anchor\" href=\"#conclusion\" aria-label=\"Anchor link for: conclusion\"><\/a>\n<\/h2>\n<p>While python certainly supports objects,\nI think it is harder to create objects in python than in other languages,\nespecially compared to the simplicity of the rest of the language.\nYou shouldn't have to know how the garbage collector handles references\nto create a deconstructor (I haven't seen any syntax for deconstructors I liked,\nbut that's a different topic). You especially should <em>not<\/em> have to create\nmethods with two underscores before and afterwards to have a useful class.<\/p>\n<p><code>&lt;\/rant&gt;<\/code><\/p>\n<h2 id=\"example\">Example<a class=\"zola-anchor\" href=\"#example\" aria-label=\"Anchor link for: example\"><\/a>\n<\/h2>\n<p>I'll leave you with an example from my research project:<\/p>\n<pre data-lang=\"python\" class=\"language-python z-code\"><code class=\"language-python\" data-lang=\"python\"><span class=\"z-source z-python\"><span class=\"z-meta z-statement z-import z-python\"><span class=\"z-keyword z-control z-import z-from z-python\">from<\/span><\/span><span class=\"z-meta z-statement z-import z-python\"><span class=\"z-meta z-import-source z-python\"> <span class=\"z-meta z-import-path z-python\"><span class=\"z-meta z-import-name z-python\">keras<\/span><span class=\"z-punctuation z-accessor z-dot z-python\">.<\/span><span class=\"z-meta z-import-name z-python\">utils<\/span><span class=\"z-punctuation z-accessor z-dot z-python\">.<\/span><span class=\"z-meta z-import-name z-python\">data_utils<\/span><\/span> <span class=\"z-meta z-statement z-import z-python\"><span class=\"z-keyword z-control z-import z-python\">import<\/span><\/span><\/span><\/span><span class=\"z-meta z-statement z-import z-python\"><\/span><span class=\"z-meta z-statement z-import z-python\"> <span class=\"z-meta z-generic-name z-python\">Sequence<\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-meta z-statement z-import z-python\"><span class=\"z-keyword z-control z-import z-from z-python\">from<\/span><\/span><span class=\"z-meta z-statement z-import z-python\"><span class=\"z-meta z-import-source z-python\"> <span class=\"z-meta z-import-path z-python\"><span class=\"z-meta z-import-name z-python\">pandas<\/span><\/span> <span class=\"z-meta z-statement z-import z-python\"><span class=\"z-keyword z-control z-import z-python\">import<\/span><\/span><\/span><\/span><span class=\"z-meta z-statement z-import z-python\"><\/span><span class=\"z-meta z-statement z-import z-python\"> <span class=\"z-meta z-generic-name z-python\">read_csv<\/span><\/span>\n<\/span><span class=\"z-source z-python\">\n<\/span><span class=\"z-source z-python\"><span class=\"z-meta z-class z-python\"><span class=\"z-storage z-type z-class z-python\"><span class=\"z-keyword z-declaration z-class z-python\">class<\/span><\/span> <span class=\"z-entity z-name z-class z-python\"><span class=\"z-meta z-generic-name z-python\">FileGenerator<\/span><\/span><span class=\"z-meta z-class z-inheritance z-python\"><span class=\"z-punctuation z-section z-inheritance z-begin z-python\">(<\/span><\/span><\/span><span class=\"z-meta z-class z-inheritance z-python\"><span class=\"z-entity z-other z-inherited-class z-python\">Sequence<\/span><span class=\"z-punctuation z-section z-inheritance z-end z-python\">)<\/span><\/span><span class=\"z-meta z-class z-python\"><span class=\"z-punctuation z-section z-class z-begin z-python\">:<\/span><\/span>\n<\/span><span class=\"z-source z-python\"><span class=\"z-meta z-function z-python\">    <span class=\"z-storage z-type z-function z-python\"><span class=\"z-keyword z-declaration z-function z-python\">def<\/span><\/span> <span class=\"z-entity z-name z-function z-python\"><span class=\"z-support z-function z-magic z-python\">__init__<\/span><\/span><\/span><span class=\"z-meta z-function z-parameters z-python\"><span class=\"z-punctuation z-section z-parameters z-begin z-python\">(<\/span><\/span><span class=\"z-meta z-function z-parameters z-python\"><span class=\"z-variable z-parameter z-python\">self<\/span><span class=\"z-punctuation z-separator z-parameters z-python\">,<\/span> <span class=\"z-variable z-parameter z-python\">data<\/span><\/span><span class=\"z-meta z-function z-parameters z-default-value z-python\"><span class=\"z-keyword z-operator z-assignment z-python\">=<\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-single z-python\"><span class=\"z-punctuation z-definition z-string z-begin z-python\">&#39;<\/span><\/span><\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-single z-python\">AmPEP-master\/combined_sequences.txt<span class=\"z-punctuation z-definition z-string z-end z-python\">&#39;<\/span><\/span><\/span><\/span><span class=\"z-meta z-function z-parameters z-python\"><span class=\"z-punctuation z-separator z-parameters z-python\">,<\/span> <span class=\"z-variable z-parameter z-python\">batch_size<\/span><\/span><span class=\"z-meta z-function z-parameters z-default-value z-python\"><span class=\"z-keyword z-operator z-assignment z-python\">=<\/span><span class=\"z-constant z-numeric z-integer z-decimal z-python\">128<\/span><\/span><span class=\"z-meta z-function z-parameters z-python\"><span class=\"z-punctuation z-separator z-parameters z-python\">,<\/span>\n<\/span><\/span><span class=\"z-source z-python\"><span class=\"z-meta z-function z-parameters z-python\">                 <span class=\"z-variable z-parameter z-python\">ratio<\/span><\/span><span class=\"z-meta z-function z-parameters z-default-value z-python\"><span class=\"z-keyword z-operator z-assignment z-python\">=<\/span><span class=\"z-constant z-numeric z-integer z-decimal z-python\">3<\/span><\/span><span class=\"z-meta z-function z-parameters z-python\"><span class=\"z-punctuation z-separator z-parameters z-python\">,<\/span> <span class=\"z-variable z-parameter z-python\">labels<\/span><\/span><span class=\"z-meta z-function z-parameters z-default-value z-python\"><span class=\"z-keyword z-operator z-assignment z-python\">=<\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-single z-python\"><span class=\"z-punctuation z-definition z-string z-begin z-python\">&#39;<\/span><\/span><\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-single z-python\">AmPEP-master\/labels.txt<span class=\"z-punctuation z-definition z-string z-end z-python\">&#39;<\/span><\/span><\/span><\/span><span class=\"z-meta z-function z-parameters z-python\"><span class=\"z-punctuation z-section z-parameters z-end z-python\">)<\/span><\/span><span class=\"z-meta z-function z-python\"><span class=\"z-punctuation z-section z-function z-begin z-python\">:<\/span><\/span>\n<\/span><span class=\"z-source z-python\">        <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-language z-python\">self<\/span><span class=\"z-punctuation z-accessor z-dot z-python\">.<\/span><span class=\"z-meta z-generic-name z-python\">positive_matches<\/span><\/span> <span class=\"z-keyword z-operator z-assignment z-python\">=<\/span> <span class=\"z-constant z-numeric z-integer z-decimal z-python\">3286<\/span>\n<\/span><span class=\"z-source z-python\">        <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-language z-python\">self<\/span><span class=\"z-punctuation z-accessor z-dot z-python\">.<\/span><span class=\"z-meta z-generic-name z-python\">filename<\/span><\/span> <span class=\"z-keyword z-operator z-assignment z-python\">=<\/span> <span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-support z-function z-builtin z-python\">open<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">filename<\/span><\/span><span class=\"z-punctuation z-separator z-arguments z-python\">,<\/span> <span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-single z-python\"><span class=\"z-punctuation z-definition z-string z-begin z-python\">&#39;<\/span><\/span><\/span><span class=\"z-meta z-string z-python\"><span class=\"z-string z-quoted z-single z-python\">r<span class=\"z-punctuation z-definition z-string z-end z-python\">&#39;<\/span><\/span><\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span>\n<\/span><span class=\"z-source z-python\">        <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-language z-python\">self<\/span><span class=\"z-punctuation z-accessor z-dot z-python\">.<\/span><span class=\"z-meta z-generic-name z-python\">batch_size<\/span><\/span> <span class=\"z-keyword z-operator z-assignment z-python\">=<\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">batch_size<\/span><\/span>\n<\/span><span class=\"z-source z-python\">        <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-language z-python\">self<\/span><span class=\"z-punctuation z-accessor z-dot z-python\">.<\/span><span class=\"z-meta z-generic-name z-python\">length<\/span><\/span> <span class=\"z-keyword z-operator z-assignment z-python\">=<\/span> <span class=\"z-meta z-group z-python\"><span class=\"z-punctuation z-section z-group z-begin z-python\">(<\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">positive_matches<\/span><\/span> <span class=\"z-keyword z-operator z-arithmetic z-python\">+<\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">positive_matches<\/span><\/span> <span class=\"z-keyword z-operator z-arithmetic z-python\">*<\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">ratio<\/span><\/span><span class=\"z-punctuation z-section z-group z-end z-python\">)<\/span><\/span> <span class=\"z-keyword z-operator z-arithmetic z-python\">\/<\/span><span class=\"z-keyword z-operator z-arithmetic z-python\">\/<\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">batch_size<\/span><\/span>\n<\/span><span class=\"z-source z-python\">\n<\/span><span class=\"z-source z-python\"><span class=\"z-meta z-function z-python\">    <span class=\"z-storage z-type z-function z-python\"><span class=\"z-keyword z-declaration z-function z-python\">def<\/span><\/span> <span class=\"z-entity z-name z-function z-python\"><span class=\"z-support z-function z-magic z-python\">__len__<\/span><\/span><\/span><span class=\"z-meta z-function z-parameters z-python\"><span class=\"z-punctuation z-section z-parameters z-begin z-python\">(<\/span><\/span><span class=\"z-meta z-function z-parameters z-python\"><span class=\"z-variable z-parameter z-python\">self<\/span><span class=\"z-punctuation z-section z-parameters z-end z-python\">)<\/span><\/span><span class=\"z-meta z-function z-python\"><span class=\"z-punctuation z-section z-function z-begin z-python\">:<\/span><\/span>\n<\/span><span class=\"z-source z-python\">        <span class=\"z-keyword z-control z-flow z-return z-python\">return<\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-language z-python\">self<\/span><span class=\"z-punctuation z-accessor z-dot z-python\">.<\/span><span class=\"z-meta z-generic-name z-python\">length<\/span><\/span>\n<\/span><span class=\"z-source z-python\">\n<\/span><span class=\"z-source z-python\"><span class=\"z-meta z-function z-python\">    <span class=\"z-storage z-type z-function z-python\"><span class=\"z-keyword z-declaration z-function z-python\">def<\/span><\/span> <span class=\"z-entity z-name z-function z-python\"><span class=\"z-support z-function z-magic z-python\">__getitem__<\/span><\/span><\/span><span class=\"z-meta z-function z-parameters z-python\"><span class=\"z-punctuation z-section z-parameters z-begin z-python\">(<\/span><\/span><span class=\"z-meta z-function z-parameters z-python\"><span class=\"z-variable z-parameter z-python\">self<\/span><span class=\"z-punctuation z-separator z-parameters z-python\">,<\/span> <span class=\"z-variable z-parameter z-python\">index<\/span><span class=\"z-punctuation z-section z-parameters z-end z-python\">)<\/span><\/span><span class=\"z-meta z-function z-python\"><span class=\"z-punctuation z-section z-function z-begin z-python\">:<\/span><\/span>\n<\/span><span class=\"z-source z-python\">        <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">data<\/span><\/span> <span class=\"z-keyword z-operator z-assignment z-python\">=<\/span> <span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-meta z-generic-name z-python\">read_csv<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-language z-python\">self<\/span><span class=\"z-punctuation z-accessor z-dot z-python\">.<\/span><span class=\"z-meta z-generic-name z-python\">data<\/span><\/span><span class=\"z-punctuation z-separator z-arguments z-python\">,<\/span> <span class=\"z-variable z-parameter z-python\">header<\/span><span class=\"z-keyword z-operator z-assignment z-python\">=<\/span><span class=\"z-constant z-language z-python\">None<\/span><span class=\"z-punctuation z-separator z-arguments z-python\">,<\/span> <span class=\"z-variable z-parameter z-python\">skiprows<\/span><span class=\"z-keyword z-operator z-assignment z-python\">=<\/span><span class=\"z-meta z-group z-python\"><span class=\"z-punctuation z-section z-group z-begin z-python\">(<\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">index<\/span><\/span> <span class=\"z-keyword z-operator z-arithmetic z-python\">*<\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">batch_size<\/span><\/span><span class=\"z-punctuation z-section z-group z-end z-python\">)<\/span><\/span><span class=\"z-punctuation z-separator z-arguments z-python\">,<\/span>\n<\/span><\/span><span class=\"z-source z-python\"><span class=\"z-meta z-function-call z-arguments z-python\">                        <span class=\"z-variable z-parameter z-python\">nrows<\/span><span class=\"z-keyword z-operator z-assignment z-python\">=<\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">batch_size<\/span><\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span><span class=\"z-meta z-item-access z-python\"><span class=\"z-punctuation z-section z-brackets z-begin z-python\">[<\/span><\/span><span class=\"z-meta z-item-access z-arguments z-python\"><span class=\"z-constant z-numeric z-integer z-decimal z-python\">0<\/span><\/span><span class=\"z-meta z-item-access z-python\"><span class=\"z-punctuation z-section z-brackets z-end z-python\">]<\/span><\/span>\n<\/span><span class=\"z-source z-python\">        <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">labels<\/span><\/span> <span class=\"z-keyword z-operator z-assignment z-python\">=<\/span> <span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-meta z-generic-name z-python\">read_csv<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-language z-python\">self<\/span><span class=\"z-punctuation z-accessor z-dot z-python\">.<\/span><span class=\"z-meta z-generic-name z-python\">labels<\/span><\/span><span class=\"z-punctuation z-separator z-arguments z-python\">,<\/span> <span class=\"z-variable z-parameter z-python\">header<\/span><span class=\"z-keyword z-operator z-assignment z-python\">=<\/span><span class=\"z-constant z-language z-python\">None<\/span><span class=\"z-punctuation z-separator z-arguments z-python\">,<\/span> <span class=\"z-variable z-parameter z-python\">skiprows<\/span><span class=\"z-keyword z-operator z-assignment z-python\">=<\/span><span class=\"z-meta z-group z-python\"><span class=\"z-punctuation z-section z-group z-begin z-python\">(<\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">index<\/span><\/span> <span class=\"z-keyword z-operator z-arithmetic z-python\">*<\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">batch_size<\/span><\/span><span class=\"z-punctuation z-section z-group z-end z-python\">)<\/span><\/span><span class=\"z-punctuation z-separator z-arguments z-python\">,<\/span>\n<\/span><\/span><span class=\"z-source z-python\"><span class=\"z-meta z-function-call z-arguments z-python\">                        <span class=\"z-variable z-parameter z-python\">nrows<\/span><span class=\"z-keyword z-operator z-assignment z-python\">=<\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">batch_size<\/span><\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span><span class=\"z-meta z-item-access z-python\"><span class=\"z-punctuation z-section z-brackets z-begin z-python\">[<\/span><\/span><span class=\"z-meta z-item-access z-arguments z-python\"><span class=\"z-constant z-numeric z-integer z-decimal z-python\">0<\/span><\/span><span class=\"z-meta z-item-access z-python\"><span class=\"z-punctuation z-section z-brackets z-end z-python\">]<\/span><\/span>\n<\/span><span class=\"z-source z-python\">        <span class=\"z-keyword z-control z-flow z-return z-python\">return<\/span> <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">data<\/span><\/span>, <span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">labels<\/span><\/span>\n<\/span><span class=\"z-source z-python\">\n<\/span><span class=\"z-source z-python\"><span class=\"z-meta z-function z-python\">    <span class=\"z-storage z-type z-function z-python\"><span class=\"z-keyword z-declaration z-function z-python\">def<\/span><\/span> <span class=\"z-entity z-name z-function z-python\"><span class=\"z-support z-function z-magic z-python\">__iter__<\/span><\/span><\/span><span class=\"z-meta z-function z-parameters z-python\"><span class=\"z-punctuation z-section z-parameters z-begin z-python\">(<\/span><\/span><span class=\"z-meta z-function z-parameters z-python\"><span class=\"z-variable z-parameter z-python\">self<\/span><span class=\"z-punctuation z-section z-parameters z-end z-python\">)<\/span><\/span><span class=\"z-meta z-function z-python\"><span class=\"z-punctuation z-section z-function z-begin z-python\">:<\/span><\/span>\n<\/span><span class=\"z-source z-python\">        <span class=\"z-keyword z-control z-flow z-return z-python\">return<\/span> <span class=\"z-meta z-group z-python\"><span class=\"z-punctuation z-section z-group z-begin z-python\">(<\/span><span class=\"z-meta z-item-access z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-language z-python\">self<\/span><\/span><\/span><span class=\"z-meta z-item-access z-python\"><span class=\"z-punctuation z-section z-brackets z-begin z-python\">[<\/span><\/span><span class=\"z-meta z-item-access z-arguments z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-meta z-generic-name z-python\">i<\/span><\/span><\/span><span class=\"z-meta z-item-access z-python\"><span class=\"z-punctuation z-section z-brackets z-end z-python\">]<\/span><\/span> <span class=\"z-meta z-expression z-generator z-python\"><span class=\"z-keyword z-control z-loop z-for z-generator z-python\">for<\/span> <span class=\"z-meta z-generic-name z-python\">i<\/span> <span class=\"z-keyword z-control z-loop z-for z-in z-python\">in<\/span><\/span> <span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-support z-function z-builtin z-python\">range<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-meta z-function-call z-python\"><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-function z-python\"><span class=\"z-support z-function z-builtin z-python\">len<\/span><\/span><\/span><\/span><span class=\"z-meta z-function-call z-arguments z-python\"><span class=\"z-punctuation z-section z-arguments z-begin z-python\">(<\/span><span class=\"z-meta z-qualified-name z-python\"><span class=\"z-variable z-language z-python\">self<\/span><\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span><span class=\"z-punctuation z-section z-arguments z-end z-python\">)<\/span><\/span><span class=\"z-punctuation z-section z-group z-end z-python\">)<\/span><\/span>\n<\/span><\/code><\/pre>\n"},{"title":"Password Safety","published":"2018-02-24T00:00:00+00:00","updated":"2018-02-24T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/password-safety\/"}},"id":"https:\/\/jyn.dev\/password-safety\/","summary":"What a password manager is and how to use one","content":"<h1 id=\"introduction\">Introduction<a class=\"zola-anchor\" href=\"#introduction\" aria-label=\"Anchor link for: introduction\"><\/a>\n<\/h1>\n<p>On the web today, there many different services we use.\nWe use email, Facebook, GitHub, Google, Twitter, dozens of various services.\nEach of these require their own username and password.<\/p>\n<p>We're told, of course, to use a different password for each account.\n<a href=\"https:\/\/www.troyhunt.com\/science-of-password-selection\/\">Only 70% of us<\/a>, however, actually do so.\nHow are we supposed to remember a <a href=\"https:\/\/www.nist.gov\/publications\/character-strings-memory-and-passwords-what-recall-study-can-tell-us\">dozen random passwords<\/a>, each different?\nTraditionally, we use the same password for each,\nperhaps with <a href=\"https:\/\/reusablesec.blogspot.com\/2010\/10\/new-paper-on-password-security-metrics.html\">slight differences<\/a>.\nThis makes it very easy for hackers to <a href=\"https:\/\/xkcd.com\/792\/\">get into your accounts<\/a>.<\/p>\n<p>There are very few common remedies for this -\nthere's ways to <a href=\"https:\/\/haveibeenpwned.com\/\">check if you've been hacked<\/a>\nand <a href=\"https:\/\/en.wikipedia.org\/wiki\/Multi-factor_authentication\">multi-factor auth<\/a>,\nbut most peoples' response is to either write sticky notes of passwords,\nor ignore the problem altogether.<\/p>\n<p>This does work if you don't <a href=\"\/assets\/password.jpg\">leave your passwords lying around<\/a>.\nIt is inconvenient, however, and doesn't scale to a large password database.\nWhat you should be using is a <a href=\"https:\/\/en.wikipedia.org\/wiki\/Password_manager\">password manager<\/a>.<\/p>\n<h2 id=\"password-managers\">Password Managers<a class=\"zola-anchor\" href=\"#password-managers\" aria-label=\"Anchor link for: password-managers\"><\/a>\n<\/h2>\n<p>A password manager is an easy way to keep track of passwords.\nIt can be anything from an <a href=\"https:\/\/support.office.com\/en-us\/article\/Protect-an-Excel-file-7359d4ae-7213-4ac2-b058-f75e9311b599\">encrypted Excel Spreadsheet<\/a>\nto <a href=\"https:\/\/duckduckgo.com\/?q=browser+manage+password+saving\">your browser's builtin manager<\/a>.\nIt takes one 'master password', and in return keeps track of everything else for you.\nThere's no need to remember multiple passwords,\nit's easy to meet any complexity requirement,\nand it doesn't matter how often you're required to change your passwords\n(which is <a href=\"http:\/\/cs.unc.edu\/~fabian\/papers\/PasswordExpire.pdf\">bad practice<\/a>, by the way.<\/p>\n<h2 id=\"what-to-use\">What to use<a class=\"zola-anchor\" href=\"#what-to-use\" aria-label=\"Anchor link for: what-to-use\"><\/a>\n<\/h2>\n<p>Wikipedia has <a href=\"https:\/\/en.wikipedia.org\/wiki\/List_of_password_managers\">an extensive list<\/a>\nof password managers.<\/p>\n<p>When choosing a manager, you should consider several things:<\/p>\n<ul>\n<li>What features are you looking for? What would be a dealbreaker?<\/li>\n<li>How much security are you looking for? What is your\n<a href=\"https:\/\/en.wikipedia.org\/wiki\/Threat_model\">threat model<\/a>?<\/li>\n<li>How much convenience are you willing to sacrifice?<\/li>\n<\/ul>\n<p>I recommend <a href=\"https:\/\/keepass.info\">Keepass<\/a> because it<\/p>\n<ul>\n<li>encrypts your databases with <a href=\"https:\/\/keepass.info\/help\/base\/security.html\">state of the art encryption<\/a><\/li>\n<li>organizes your passwords <a href=\"https:\/\/keepass.info\/features.html#lnkgroups\">by groups<\/a><\/li>\n<li>stores usernames, URLs, and\n<a href=\"https:\/\/keepass.info\/features.html#lnktimes\">arbitrary data<\/a><\/li>\n<li>randomly <a href=\"https:\/\/keepass.info\/help\/base\/pwgenerator.html\">generates passwords<\/a><\/li>\n<li>is <a href=\"https:\/\/keepass.info\/download.html\">open source<\/a><\/li>\n<li>is fully interoperable with many <a href=\"https:\/\/en.wikipedia.org\/wiki\/KeePass#Unofficial_KeePass_derivatives\">other programs<\/a><\/li>\n<li>has <a href=\"https:\/\/keepass.info\/features.html\">many other features<\/a><\/li>\n<\/ul>\n<p>Once you start using a password manager, you can start using passwords like\n<code>W(xv|u7N''fAs,{t|J<\/code> for all your accounts.<\/p>\n<h2 id=\"appendix\">Appendix<a class=\"zola-anchor\" href=\"#appendix\" aria-label=\"Anchor link for: appendix\"><\/a>\n<\/h2>\n<ul>\n<li>If you're not on Windows, or just want more features, check out <a href=\"https:\/\/keepassxc.org\/\">Keepassxc<\/a>\n(Keepass Cross-Platform Community Edition). It supports global autotype and\n<a href=\"https:\/\/en.wikipedia.org\/wiki\/Time-based_One-time_Password_Algorithm\">timed one-time passwords<\/a>, a.k.a. two-factor auth.<\/li>\n<li>I've written a <a href=\"https:\/\/drive.google.com\/open?id=15u4uXxC5K7v2Llsu8L4JVDcFNjyavW4pKBJ2GW5YV5M\">presentation on secure passwords<\/a><\/li>\n<li><a href=\"https:\/\/security.stackexchange.com\/q\/6682\">StackOverflow on password reuse<\/a><\/li>\n<li>The figure for reuse <a href=\"https:\/\/www.lightbluetouchpaper.org\/2011\/02\/09\/measuring-password-re-use-empirically\/\">may be lower<\/a> than I've claimed,\nbut it's still a minimum of 30%<\/li>\n<\/ul>\n"},{"title":"Speed Up!","published":"2018-02-14T00:00:00+00:00","updated":"2018-02-14T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/speed-up\/"}},"id":"https:\/\/jyn.dev\/speed-up\/","summary":"This site is now a lot faster.","content":"<p>Good news! As of <a href=\"https:\/\/github.com\/jyn514\/jyn514.github.io\/commit\/525779f74014e46b11d0d035314806fa234a58d4\">today<\/a>,\nmy site now takes less than a tenth of the time to load.<\/p>\n<p>I found an excellent <a href=\"https:\/\/gtmetrix.com\/\">site<\/a> which tests page load speeds.\nIt told me that my site (which is mostly static pages)\nwas taking a full 4.3 seconds to load.\nI was very confused by this - I don't have any large pages that I'm aware of;\nthe <a href=\"https:\/\/developer.mozilla.org\/en-US\/docs\/Web\/CSS\">CSS<\/a> is <a href=\"https:\/\/en.wikipedia.org\/wiki\/Minification_(programming)\">minified<\/a> and there's almost no JavaScript.<\/p>\n<p>It turns out that the nifty DNSSEC testers I'd put in <a href=\"https:\/\/github.com\/jyn514\/jyn514.github.io\/commit\/988425efe8b17ea75509f8f940d2db7dd042db4b\">the other day<\/a>\nwere taking a second each to load.\nNow, a second doesn't sound like a lot, but it makes a <a href=\"https:\/\/developers.google.com\/web\/fundamentals\/performance\/rail#ux\">big difference<\/a>\nto how users perceive the site. 3.5 seconds - the speed up I've since gotten -\nis an eternity on the web.<\/p>\n<p>Honestly, I feel a lot more comfortable without the DNSSEC testers.\nThey always felt suspiciously like analytics (which I firmly oppose),\neven though I didn't see the results.<\/p>\n<p>The site is now at 223 milliseconds per load and counting.\nThe slowest things still present are Open Sans (loaded from Google fonts, nearly half the load time)\nand the second DNS tester.\nHowever, I've been told the site looks a lot nicer than before, so I'm loath to remove the nice fonts.\nIf I were self-hosting, I could fix this with caching, but I unfortunately don't control GitHub's servers.\nLet me know if you have any suggestions!<\/p>\n<h1 id=\"appendix\">Appendix<a class=\"zola-anchor\" href=\"#appendix\" aria-label=\"Anchor link for: appendix\"><\/a>\n<\/h1>\n<ul>\n<li>Google has a <a href=\"https:\/\/testmysite.thinkwithgoogle.com\">'mobile experience' site<\/a> that I tried first.\nUnfortunately, it kept giving me errors.<\/li>\n<li>Proudly inspired by <a href=\"https:\/\/danluu.com\/octopress-speedup\/\">this post<\/a><\/li>\n<\/ul>\n"},{"title":"Using PGP Keys","published":"2018-02-11T00:00:00+00:00","updated":"2018-02-11T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/using-pgp-keys\/"}},"id":"https:\/\/jyn.dev\/using-pgp-keys\/","summary":"Why and How to use encryption","content":"<h1 id=\"what-is-gpg\">What is GPG?<a class=\"zola-anchor\" href=\"#what-is-gpg\" aria-label=\"Anchor link for: what-is-gpg\"><\/a>\n<\/h1>\n<p><a href=\"https:\/\/gnupg.org\/\">GPG<\/a> is a program for encrypting your files, including<\/p>\n<ul>\n<li>Documents<\/li>\n<li>Music<\/li>\n<li>Programs<\/li>\n<li>Anything else you can store digitally<\/li>\n<\/ul>\n<p><a href=\"https:\/\/www.xkcd.com\/1475\/\">Technically<\/a>, GPG implements the <a href=\"https:\/\/www.openpgp.org\/\">PGP<\/a> standard,\nwhich is <a href=\"https:\/\/www.ietf.org\/rfc\/rfc4880.txt\">mandated<\/a> by the <a href=\"https:\/\/www.ietf.org\/\">IETF\u00ae<\/a>.<\/p>\n<h2 id=\"what-is-encryption\">What is encryption?<a class=\"zola-anchor\" href=\"#what-is-encryption\" aria-label=\"Anchor link for: what-is-encryption\"><\/a>\n<\/h2>\n<p><a href=\"https:\/\/en.wikipedia.org\/wiki\/Encryption\">Encryption<\/a> is a way of ensuring that <em>only<\/em> people you choose\nare able to read your information. Encryption goes back thousands of years\nto the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Caesar_cipher\">Roman empire<\/a>. It is <a href=\"https:\/\/doesmysiteneedhttps.com\/\">used every day<\/a>\nin the <a href=\"https:\/\/en.wikipedia.org\/wiki\/HTTPS\"><code>HTTPS<\/code><\/a> protocol, which ensures that the websites you visit\nare only visible to you, and that the site has not <a href=\"https:\/\/gist.github.com\/ryankearney\/4146814\">been changed in transit<\/a>.<\/p>\n<p>Cryptography therefore has two major functions:<\/p>\n<ul>\n<li>To encrypt data: to make sure only people you choose can read it, and<\/li>\n<li>To sign data: to ensure that information you read has not been altered by third parties.<\/li>\n<\/ul>\n<h2 id=\"how-do-i-get-gpg\">How do I get GPG?<a class=\"zola-anchor\" href=\"#how-do-i-get-gpg\" aria-label=\"Anchor link for: how-do-i-get-gpg\"><\/a>\n<\/h2>\n<p>GPG is only fully-featured when used from the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Command-line_interface\">command line<\/a>.\nIf that sounds foreign to you, I have a quick-and-dirty guide on <a href=\"https:\/\/github.com\/jyn514\/215-resources\/blob\/master\/tutorials\/ShellIntro.pdf\">GitHub<\/a>.<\/p>\n<p>Some binary releases:<\/p>\n<ul>\n<li>Debian\/Ubuntu: <code>sudo apt-get install gnupg<\/code><\/li>\n<li>Windows: <a href=\"https:\/\/gpg4win.org\/download.html\">GPG4Win<\/a><\/li>\n<li>macOS:\n<ul>\n<li>without brew: <a href=\"https:\/\/gpgtools.org\/\">GPGtools.org<\/a><\/li>\n<li>with <code>brew<\/code> installed: <code>brew install gnupg <\/code><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<h2 id=\"how-do-i-use-gpg\">How do I use GPG?<a class=\"zola-anchor\" href=\"#how-do-i-use-gpg\" aria-label=\"Anchor link for: how-do-i-use-gpg\"><\/a>\n<\/h2>\n<p>Now we get to the fun stuff! GPG uses what's called <a href=\"https:\/\/en.wikipedia.org\/wiki\/Public-key_cryptography\">asymmetric encryption<\/a>,\nwhich allows <em>anyone<\/em> to send secure messages to you, but only you to read them.\nIn order to take full advantage, you'll need to<\/p>\n<ol>\n<li>Make a key<\/li>\n<li>Publish your key<\/li>\n<li>Start signing things<\/li>\n<\/ol>\n<h3 id=\"make-a-key\">Make a key<a class=\"zola-anchor\" href=\"#make-a-key\" aria-label=\"Anchor link for: make-a-key\"><\/a>\n<\/h3>\n<p>You need a <a href=\"https:\/\/whatisapassphrase.com\/\">passphrase<\/a> to use a GPG key.\nThis prevents anyone from using your key.<\/p>\n<pre data-lang=\"sh\" class=\"language-sh z-code\"><code class=\"language-sh\" data-lang=\"sh\"><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">gpg<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> --<\/span>quick-gen-key<\/span> <span class=\"z-string z-quoted z-double z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&quot;<\/span>Joe Shmoe &lt;joe@email.example.com&gt;<span class=\"z-punctuation z-definition z-string z-end z-shell\">&quot;<\/span><\/span><\/span>\n<\/span><\/code><\/pre>\n<p>Enter your passphrase into the prompt that pops up.<\/p>\n<h3 id=\"publish-your-key\">Publish your key<a class=\"zola-anchor\" href=\"#publish-your-key\" aria-label=\"Anchor link for: publish-your-key\"><\/a>\n<\/h3>\n<pre data-lang=\"sh\" class=\"language-sh z-code\"><code class=\"language-sh\" data-lang=\"sh\"><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">gpg<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> --<\/span>send-keys<\/span><\/span>\n<\/span><\/code><\/pre>\n<h3 id=\"start-signing-things\">Start signing things<a class=\"zola-anchor\" href=\"#start-signing-things\" aria-label=\"Anchor link for: start-signing-things\"><\/a>\n<\/h3>\n<ul>\n<li>Sign things with git:\n<code>gpg config --global commit.gpgsign true<\/code><\/li>\n<li>Sign your emails with supporting clients:\n<ul>\n<li><a href=\"https:\/\/www.mozilla.org\/en-US\/thunderbird\/\">Thunderbird<\/a> and <a href=\"https:\/\/www.enigmail.net\/index.php\/en\/\">Enigmail<\/a> for desktop<\/li>\n<li><a href=\"https:\/\/www.kde.org\/applications\/internet\/kmail\/\">Kmail<\/a> for KDE<\/li>\n<li><a href=\"https:\/\/k9mail.github.io\/\">K9mail<\/a> and <a href=\"https:\/\/openkeychain.org\/\">OpenKeychain<\/a>\nfor Android<\/li>\n<li><a href=\"https:\/\/gnupg.org\/software\/swlist.html#mutt\">Mutt<\/a> for the terminal<\/li>\n<\/ul>\n<\/li>\n<li>Join <a href=\"https:\/\/keybase.io\/\">Keybase<\/a><\/li>\n<\/ul>\n<h2 id=\"more-information\">More information<a class=\"zola-anchor\" href=\"#more-information\" aria-label=\"Anchor link for: more-information\"><\/a>\n<\/h2>\n<ul>\n<li><a href=\"https:\/\/www.gnupg.org\/gph\/en\/manual\/book1.html\">GPG handbook<\/a><\/li>\n<li><a href=\"https:\/\/gnupg.org\/software\/frontends.html\">More frontends<\/a><\/li>\n<li><a href=\"https:\/\/www.glump.net\/howto\/cryptography\/practical-introduction-to-gnu-privacy-guard-in-windows\">A much more thorough explanation<\/a><\/li>\n<li><a href=\"https:\/\/www.openpgp.org\/\">Homepage<\/a><\/li>\n<li><a href=\"https:\/\/gnupg.org\/\">GPG homepage<\/a><\/li>\n<li><a href=\"https:\/\/emailselfdefense.fsf.org\/en\/\">FSF<\/a> on email self-defense<\/li>\n<\/ul>\n<h2 id=\"appendix\">Appendix<a class=\"zola-anchor\" href=\"#appendix\" aria-label=\"Anchor link for: appendix\"><\/a>\n<\/h2>\n<h3 id=\"prompts-are-repeated\">Prompts are repeated<a class=\"zola-anchor\" href=\"#prompts-are-repeated\" aria-label=\"Anchor link for: prompts-are-repeated\"><\/a>\n<\/h3>\n<p>Note that the following prompt appears <em>twice<\/em> when you generate a key:<\/p>\n<pre class=\"z-code\"><code><span class=\"z-text z-plain\">We need to generate a lot of random bytes.\n<\/span><span class=\"z-text z-plain\">It is a good idea to perform some other action (type on the keyboard,\n<\/span><span class=\"z-text z-plain\">move the mouse, utilize the disks) during the prime generation;\n<\/span><span class=\"z-text z-plain\">this gives the random number generator a better chance to gain enough entropy.\n<\/span><\/code><\/pre>\n<p>This is because there are actually two keys being generated:\n1 private key and 1 public. The private you will store on your computer\nsecurely. The public you will upload to a keyserver for anyone to see.<\/p>\n<h3 id=\"what-if-i-m-not-comfortable-with-shell-quoting\">What if I'm not comfortable with shell quoting?<a class=\"zola-anchor\" href=\"#what-if-i-m-not-comfortable-with-shell-quoting\" aria-label=\"Anchor link for: what-if-i-m-not-comfortable-with-shell-quoting\"><\/a>\n<\/h3>\n<p>Use <code>gpg --gen-key<\/code> or <code>gpg --full-gen-key<\/code>.<\/p>\n"},{"title":"THERE IS A CAR IN SPACE","published":"2018-02-06T00:00:00+00:00","updated":"2018-02-06T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/there-is-a-car-in-space\/"}},"id":"https:\/\/jyn.dev\/there-is-a-car-in-space\/","content":"<p>As of today, February 6th 2018, from around 3:50 PM, there is a\n<a href=\"https:\/\/en.wikipedia.org\/wiki\/Elon_Musk%27s_Tesla_Roadster\">Tesla Roadster<\/a>\nheaded to Mars.<\/p>\n<p>Today marked the first successful launch of the SpaceX\n<a href=\"https:\/\/en.wikipedia.org\/wiki\/Falcon_Heavy\">Falcon Heavy<\/a>, a rocket ship than\ncan carry other rocket ships.\nNot only did the Heavy, which can carry more payload\nthan any other ship besides the Saturn V, launch successfully,\nbut its rocket boosters <em>then deattached and landed themselves<\/em>, successfully,\nso that they can be reused for the next launch.<\/p>\n<p>The car is currently in geostationary orbit within the\n<a href=\"https:\/\/en.wikipedia.org\/wiki\/Van_Allen_radiation_belt\">Van Allen radiation belt<\/a>, testing radiation shielding\nfor <a href=\"https:\/\/arstechnica.com\/science\/2018\/02\/at-the-pad-elon-musk-sizes-up-the-falcon-heavys-chance-of-success\/\">\"Air Force intel missions\"<\/a>.\nIt has <a href=\"https:\/\/www.youtube.com\/watch?v=aBr2kKAHN6M\"><em>live<\/em> coverage<\/a>\nand will launch for Mars within the next 8 hours.<\/p>\n<p>To reiterate:<\/p>\n<p><strong>THERE IS A CAR IN SPACE.<\/strong><\/p>\n<p>That is all.<\/p>\n"},{"title":"Using DNSSEC","published":"2018-01-06T00:00:00+00:00","updated":"2018-01-06T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/using-dnssec\/"}},"id":"https:\/\/jyn.dev\/using-dnssec\/","summary":"How and why to enable DNSSEC","content":"<p>I was on StackOverflow the other day when I saw\n<a href=\"https:\/\/unix.stackexchange.com\/questions\/90035\/how-to-set-dns-resolver-in-fedora-using-network-manager\">a nifty post<\/a> on using custom DNS servers with\n<a href=\"https:\/\/wiki.gnome.org\/Projects\/NetworkManager\">NetworkManager<\/a>.\nAs you can see, in the middle of the post there's a link to\n<a href=\"https:\/\/dnssec.vs.uni-due.de\">https:\/\/dnssec.vs.uni-due.de<\/a>.\nBeing a curious sort, I opened it up\nand saw that it's a DNSSEC tester!<\/p>\n<p><a href=\"https:\/\/en.wikipedia.org\/wiki\/Domain_Name_System_Security_Extensions\">DNSSEC<\/a> is a protocol for ensuring that <a href=\"https:\/\/en.wikipedia.org\/wiki\/Domain_Name_System\">DNS<\/a> results are accurate,\nusing <a href=\"https:\/\/en.wikipedia.org\/wiki\/Public-key_cryptography\">public-key cryptographpy<\/a>.\nIt prevents <a href=\"https:\/\/en.wikipedia.org\/wiki\/DNS_spoofing\">cache poisoning<\/a> and <a href=\"https:\/\/en.wikipedia.org\/wiki\/DNS_hijacking\">dns hijacking<\/a>.\nIt also prevents <a href=\"https:\/\/blog.xs4all.nl\/xs4all-moet-adressen-pirate-bay-voorlopig-blokkeren\/\">court-ordered DNS bans<\/a> and\n<a href=\"https:\/\/blog.wired.com\/27bstroke6\/2008\/04\/isps-error-page.html\">hidden redirects<\/a>, both of which violate <a href=\"https:\/\/www.ietf.org\/rfc.html\">RFC<\/a> standards.<\/p>\n<p>Unfortunately, most computers in the world do <em>not<\/em> use DNSSEC by default.\nMost end-users use DNS servers provided by their <a href=\"https:\/\/en.wikipedia.org\/wiki\/Internet_service_provider\">ISP<\/a>, which often\ndon't support DNSSEC at all. Fortunately, configuring it is a simple process for which\nGoogle has an <a href=\"https:\/\/developers.google.com\/speed\/public-dns\/docs\/using\">extensive walkthrough<\/a>,\nas well as <a href=\"https:\/\/developers.google.com\/speed\/public-dns\/\">an overview<\/a> of DNS in general.<\/p>\n<h1 id=\"appendix\">Appendix<a class=\"zola-anchor\" href=\"#appendix\" aria-label=\"Anchor link for: appendix\"><\/a>\n<\/h1>\n<ul>\n<li>You may also want to <a href=\"https:\/\/test-ipv6.com\">see if you can access IPv6 sites<\/a><\/li>\n<li>Google has a <a href=\"https:\/\/dns.google.com\">web interface<\/a> for DNS lookups<\/li>\n<li><a href=\"https:\/\/jpmens.net\/2011\/10\/21\/automating-unbound-for-dnssec-on-your-workstation\/\">Run your own<\/a> DNS server!<\/li>\n<li>I've included DNSSEC testers in my own site.\nOpen up the <a href=\"https:\/\/webmasters.stackexchange.com\/questions\/8525\/\">browser console<\/a> to take a look!<\/li>\n<\/ul>\n"},{"title":"Setting up Jekyll for GitHub Pages","published":"2017-12-28T00:00:00+00:00","updated":"2017-12-28T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/setting-up-jekyll\/"}},"id":"https:\/\/jyn.dev\/setting-up-jekyll\/","summary":"The story of how I set up this site","content":"<p>Jekyll is a wonderful program. The more I use it, the more I like it.\nIt's customizable, automatically parses markdown, and uses a template system\nthat makes it very easy to create a consistent style. Its only flaw is that\nit depends on <a href=\"https:\/\/jyn.dev\/setting-up-jekyll\/#appendix\">rubygems<\/a>.<\/p>\n<p>Jekyll does get a little getting used to, however.\nIn this article, I'll go over the basics:<\/p>\n<ol>\n<li>Installing<\/li>\n<li>Creating a site<\/li>\n<li>Customizing a site<\/li>\n<li>Creating content<\/li>\n<\/ol>\n<p>Note that I assume some basic familiarity with <a href=\"https:\/\/git-scm.com\/book\/en\/v2\">Git<\/a>\nand the commandline, which will be covered in another post.<\/p>\n<h2 id=\"installing\">Installing<a class=\"zola-anchor\" href=\"#installing\" aria-label=\"Anchor link for: installing\"><\/a>\n<\/h2>\n<p>If you don't have <a href=\"https:\/\/www.ruby-lang.org\/en\/documentation\/installation\/\">rubygems<\/a>\ninstalled, you'll need it. See also\n<a href=\"https:\/\/jyn.dev\/setting-up-jekyll\/#appendix\">footnote 1<\/a>.<\/p>\n<pre data-lang=\"sh\" class=\"language-sh z-code\"><code class=\"language-sh\" data-lang=\"sh\"><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">gem<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> install jekyll<\/span>\n<\/span><\/code><\/pre>\n<h2 id=\"creating-a-site\">Creating a site<a class=\"zola-anchor\" href=\"#creating-a-site\" aria-label=\"Anchor link for: creating-a-site\"><\/a>\n<\/h2>\n<pre data-lang=\"sh\" class=\"language-sh z-code\"><code class=\"language-sh\" data-lang=\"sh\"><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">jekyll<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> new <span class=\"z-keyword z-operator z-assignment z-redirection z-shell\">&lt;<\/span>directory<span class=\"z-keyword z-operator z-assignment z-redirection z-shell\">&gt;<\/span>\n<\/span><\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-support z-function z-cd z-shell\">cd<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> <span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-punctuation z-definition z-variable z-shell\">$<\/span><span class=\"z-variable z-language z-shell\">_<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">jekyll<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> serve<\/span>\n<\/span><\/code><\/pre>\n<p>Congratulations! Your site is now live\n(at <a href=\"http:\/\/localhost:4000\">http:\/\/localhost:4000<\/a> by default).<\/p>\n<h2 id=\"customizing-your-site\">Customizing your site<a class=\"zola-anchor\" href=\"#customizing-your-site\" aria-label=\"Anchor link for: customizing-your-site\"><\/a>\n<\/h2>\n<p>\"Your awesome title\" is a pretty terribly name for a site.\nGo ahead and edit it in <code>_config.yml<\/code>.\nThere's lots of other juicy config to change in there,\nquick rundown <a href=\"https:\/\/jekyllrb.com\/docs\/configuration\/\">here<\/a>.<\/p>\n<h3 id=\"other-things-to-edit\">Other things to edit<a class=\"zola-anchor\" href=\"#other-things-to-edit\" aria-label=\"Anchor link for: other-things-to-edit\"><\/a>\n<\/h3>\n<p>Jekyll uses <a href=\"https:\/\/jekyll.github.io\/minima\/\">minima<\/a>\nby default; find where it is with <code>bundle show minima<\/code>.<\/p>\n<pre data-lang=\"sh\" class=\"language-sh z-code\"><code class=\"language-sh\" data-lang=\"sh\"><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">cp<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> -<\/span>r<\/span> <span class=\"z-meta z-group z-expansion z-command z-parens z-shell\"><span class=\"z-punctuation z-definition z-variable z-shell\">$<\/span><span class=\"z-punctuation z-section z-parens z-begin z-shell\">(<\/span><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">bundle<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> show minima<\/span><span class=\"z-punctuation z-section z-parens z-end z-shell\">)<\/span><\/span>\/<span class=\"z-keyword z-operator z-regexp z-quantifier z-shell\">*<\/span> <span class=\"z-keyword z-operator z-assignment z-redirection z-shell\">&lt;<\/span>directory<span class=\"z-keyword z-operator z-assignment z-redirection z-shell\">&gt;<\/span>\n<\/span><\/span><\/code><\/pre>\n<ul>\n<li>CSS: <code>_sass\/minima\/<\/code><\/li>\n<li>Page layouts: <code>_layouts\/<\/code><\/li>\n<li>Headers and footers: <code>_includes\/<\/code><\/li>\n<li>404 page: <code>404.html<\/code><\/li>\n<\/ul>\n<h2 id=\"creating-content\">Creating content<a class=\"zola-anchor\" href=\"#creating-content\" aria-label=\"Anchor link for: creating-content\"><\/a>\n<\/h2>\n<p>Jekyll expects a certain format from its templates. I've made an\n<a href=\"https:\/\/github.com\/jyn514\/jyn514.github.io\/blob\/master\/scripts\/new_post\">script<\/a>\nthat will handle the metadata automatically.<\/p>\n<p>The content itself can be in one of three formats:<\/p>\n<ul>\n<li><a href=\"https:\/\/daringfireball.net\/projects\/markdown\/\">Markdown<\/a><\/li>\n<li>HTML<\/li>\n<li>Plain text<\/li>\n<\/ul>\n<p>The <a href=\"https:\/\/github.com\/jyn514\/jyn514.github.io\/\">source<\/a>\nof my site is also available as an example.<\/p>\n<h2 id=\"appendix\">Appendix<a class=\"zola-anchor\" href=\"#appendix\" aria-label=\"Anchor link for: appendix\"><\/a>\n<\/h2>\n<ul>\n<li>If, like me, you got a permissions error -<\/li>\n<\/ul>\n<pre data-lang=\"sh\" class=\"language-sh z-code\"><code class=\"language-sh\" data-lang=\"sh\"><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">jyn@debian-thinkpad:\/usr\/local\/src\/second-website$<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> gem install jekyll<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">ERROR:<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\">  While executing gem ... (Errno::EACCES<\/span><span class=\"z-meta z-function-call z-shell\"><\/span>)\n<\/span><span class=\"z-source z-shell z-bash\">    <span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">Permission<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> denied @ dir_s_mkdir - \/var\/lib\/ruby\/2.3.0\/gem\/specs<\/span>\n<\/span><\/code><\/pre>\n<p>then you probably installed with a package manager. Unfortunately,\nyou'll have to reinstall gem; I'm not aware of any way around this.\nSince installing on a system-wide basis requires root permissions,\n\/var\/lib\/ruby is only read\/writable for root.<\/p>\n<ul>\n<li>If you want to edit where gems are stored, you'll have to edit\nthe rubygem script itself. Find the ruby library (in my case,\n<code>\/usr\/lib\/ruby<\/code>) and ```sh\ncd 2.3.0\/rubygems\nsed -i \"s\/File.join Gem.user_home, '.gem'\/File.join Gem.user_home, '.local', 'lib', 'gem'\/\" **<\/li>\n<\/ul>\n<pre class=\"z-code\"><code><\/code><\/pre>\n"},{"title":"Hosting an SSH Server","published":"2017-12-27T00:00:00+00:00","updated":"2017-12-27T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/hosting-an-ssh-server\/"}},"id":"https:\/\/jyn.dev\/hosting-an-ssh-server\/","summary":"I decided one day that I really wanted a remote server.","content":"<p>I decided one day over break that I really wanted a remote server.\nI had a spare laptop, a router at home, and far too much free time.\nThe obvious solution was to combine the three.<\/p>\n<p>It ended up being much simpler than I expected to set up.\nOn my old laptop, I installed an SSH server:<\/p>\n<pre data-lang=\"sh\" class=\"language-sh z-code\"><code class=\"language-sh\" data-lang=\"sh\"><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">sudo<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> apt-get install openssh-server<\/span>\n<\/span><\/code><\/pre>\n<p>I made a few changes to the default config:<\/p>\n<pre data-lang=\"sh\" class=\"language-sh z-code\"><code class=\"language-sh\" data-lang=\"sh\"><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">sed<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> -<\/span>i<\/span> <span class=\"z-string z-quoted z-single z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&#39;<\/span>\n<\/span><\/span><\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-string z-quoted z-single z-shell\">s\/Port: 22\/Port: 666\/\n<\/span><\/span><\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-string z-quoted z-single z-shell\">s\/PermitRootLogin without-password\/PermitRootLogin no\/\n<\/span><\/span><\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-string z-quoted z-single z-shell\">s\/#PasswordAuthentication yes\/PasswordAuthentication no\/\n<\/span><\/span><\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-string z-quoted z-single z-shell\">s\/PrintMotd no\/PrintMotd yes\/\n<\/span><\/span><\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-string z-quoted z-single z-shell\"><span class=\"z-punctuation z-definition z-string z-end z-shell\">&#39;<\/span><\/span> \/etc\/ssh\/sshd_config<\/span>\n<\/span><\/code><\/pre>\n<p>Created and added an SSH key:<\/p>\n<pre data-lang=\"sh\" class=\"language-sh z-code\"><code class=\"language-sh\" data-lang=\"sh\"><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">ssh-keygen<\/span><\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-support z-function z-eval z-shell\">eval<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> <span class=\"z-meta z-group z-expansion z-command z-parens z-shell\"><span class=\"z-punctuation z-definition z-variable z-shell\">$<\/span><span class=\"z-punctuation z-section z-parens z-begin z-shell\">(<\/span><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">ssh-agent<\/span><\/span><span class=\"z-punctuation z-section z-parens z-end z-shell\">)<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">ssh-add<\/span><\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">ssh-copy-id<\/span><\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">sudo<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> service ssh restart<\/span>\n<\/span><\/code><\/pre>\n<p>And copied it to my new computer:<\/p>\n<pre data-lang=\"sh\" class=\"language-sh z-code\"><code class=\"language-sh\" data-lang=\"sh\"><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">jyn@debian-acer:<span class=\"z-meta z-group z-expansion z-tilde\"><span class=\"z-variable z-language z-tilde z-shell\">~<\/span><\/span>$<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> cp <span class=\"z-meta z-group z-expansion z-tilde\"><span class=\"z-variable z-language z-tilde z-shell\">~<\/span><\/span>\/.ssh\/id_rsa<span class=\"z-keyword z-operator z-regexp z-quantifier z-shell\">*<\/span> \/media\/jyn\/usb<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">jyn@debian-acer:<span class=\"z-meta z-group z-expansion z-tilde\"><span class=\"z-variable z-language z-tilde z-shell\">~<\/span><\/span>$<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> sudo umount <span class=\"z-meta z-group z-expansion z-parameter z-shell\"><span class=\"z-punctuation z-definition z-variable z-shell\">$<\/span><span class=\"z-variable z-language z-shell\">_<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">jyn@debian-thinkpad:<span class=\"z-meta z-group z-expansion z-tilde\"><span class=\"z-variable z-language z-tilde z-shell\">~<\/span><\/span>$<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> sudo mount \/dev\/sdb1 \/mnt<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">cp<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> \/mnt\/id_rsa<span class=\"z-keyword z-operator z-regexp z-quantifier z-shell\">*<\/span> <span class=\"z-meta z-group z-expansion z-tilde\"><span class=\"z-variable z-language z-tilde z-shell\">~<\/span><\/span>\/.ssh<\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">ssh-add<\/span><\/span>\n<\/span><\/code><\/pre>\n<p>SSH now worked perfectly when I typed in the IP address.\nI was moving back to USC in a few weeks though,\nmy server would be behind a firewall.\nNot to worry though, with a mutter of approval from my dad,\nI forwarded port 666 on the router to 666 on debian-acer.<\/p>\n<p>Worked like a charm.<\/p>\n<h1 id=\"appendix\">Appendix<a class=\"zola-anchor\" href=\"#appendix\" aria-label=\"Anchor link for: appendix\"><\/a>\n<\/h1>\n<ul>\n<li>If, like me, you have an outdated laptop\nwhich refuses to start networking at boot, try this:<\/li>\n<\/ul>\n<pre data-lang=\"sh\" class=\"language-sh z-code\"><code class=\"language-sh\" data-lang=\"sh\"><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">sudo<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> sh<span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> -<\/span>c<\/span> <span class=\"z-string z-quoted z-single z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&#39;<\/span>echo &quot;# Ethernet\n<\/span><\/span><\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-string z-quoted z-single z-shell\">auto eth0\n<\/span><\/span><\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-string z-quoted z-single z-shell\">iface eth0 inet dhcp\n<\/span><\/span><\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-string z-quoted z-single z-shell\">&quot; &gt;&gt; \/etc\/network\/interfaces<span class=\"z-punctuation z-definition z-string z-end z-shell\">&#39;<\/span><\/span><\/span>\n<\/span><\/code><\/pre>\n<ul>\n<li>More elegant way to specify ssh ports:<\/li>\n<\/ul>\n<pre data-lang=\"sh\" class=\"language-sh z-code\"><code class=\"language-sh\" data-lang=\"sh\"><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-variable z-function z-shell\">sudo<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> sh<span class=\"z-variable z-parameter z-option z-shell\"><span class=\"z-punctuation z-definition z-parameter z-shell\"> -<\/span>c<\/span> <span class=\"z-string z-quoted z-single z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&#39;<\/span>echo &quot;192.168.1.13   home &gt;&gt; \/etc\/hosts&quot;<span class=\"z-punctuation z-definition z-string z-end z-shell\">&#39;<\/span><\/span><\/span>\n<\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-shell\"><span class=\"z-support z-function z-echo z-shell\">echo<\/span><\/span><span class=\"z-meta z-function-call z-arguments z-shell\"> <span class=\"z-string z-quoted z-single z-shell\"><span class=\"z-punctuation z-definition z-string z-begin z-shell\">&#39;<\/span>Host home\n<\/span><\/span><\/span><span class=\"z-source z-shell z-bash\"><span class=\"z-meta z-function-call z-arguments z-shell\"><span class=\"z-string z-quoted z-single z-shell\">Port 666<span class=\"z-punctuation z-definition z-string z-end z-shell\">&#39;<\/span><\/span> <span class=\"z-keyword z-operator z-assignment z-redirection z-shell\">&gt;&gt;<\/span> <span class=\"z-meta z-group z-expansion z-tilde\"><span class=\"z-variable z-language z-tilde z-shell\">~<\/span><\/span>\/.ssh\/config<\/span>\n<\/span><\/code><\/pre>\n"},{"title":"Hello, World!","published":"2017-12-23T00:00:00+00:00","updated":"2017-12-23T00:00:00+00:00","author":{"name":"\n            \n              jyn\n            \n          "},"link":{"@attributes":{"rel":"alternate","type":"text\/html","href":"https:\/\/jyn.dev\/first-post\/"}},"id":"https:\/\/jyn.dev\/first-post\/","summary":"Hello, World!","content":"<p><strong>Hello, World!<\/strong><\/p>\n<p>This is my first post on the new and improved site using <a href=\"https:\/\/github.com\/jekyll\/jekyll\">jekyll<\/a>.\nIt will (hopefully) be the first of many. Feel free to explore the source code <a href=\"https:\/\/github.com\/jyn514\/second-website\">here<\/a>.<\/p>\n<p>I started this site from an offhand comment by my good friend <a href=\"https:\/\/github.com\/charlesdaniels\">Charles Daniels<\/a>.\nWe were working on a <a href=\"https:\/\/github.com\/charlesdaniels\/bitshuffle\">side project<\/a> together when I suggested\nsigning release commits with <a href=\"https:\/\/gnupg.org\/\">GPG<\/a>.\nHe was surprised I had experience with PGP keys, and his first question was \"Do you have a blog?\"<\/p>\n<p>From that simple question sprung this site. I hope you enjoy!<\/p>\n"}]}