{"@attributes":{"version":"2.0"},"channel":{"title":"Home on Glyphack","link":"https:\/\/glyphack.com\/","description":"Recent content in Home on Glyphack","generator":"Hugo -- gohugo.io","language":"en","lastBuildDate":"Tue, 21 Jul 2026 00:00:00 +0000","item":[{"title":"It's getting harder to focus every day","link":"https:\/\/glyphack.com\/attention\/","pubDate":"Wed, 15 Jul 2026 00:00:00 +0000","guid":"https:\/\/glyphack.com\/attention\/","description":"<p>I&rsquo;m feeling it right now. I had to set a timer for 15 minute on my computer and block all distractions to write this.\nIf I didn&rsquo;t force myself to focus I would easily get distracted by something after few minutes.\nEven when I&rsquo;m doing things that I&rsquo;ve been waiting to do it, I still feel the urge to do something else.<\/p>\n<p>I don&rsquo;t know how and when this happened.\nDuring the last few years I was always studying, working, and doing open source.\nAnd actually got stuff done.\nDoing all of those at the same time requires paying attention to what I wanted to do and ignore the noise.<\/p>\n<p>Nowadays, I&rsquo;m lucky if I get 1 hour of focused time.\nJust to be clear, my goal is not to work 90 hours a week or anything crazy.\nThat is not possible for me.\nI just want the hour I spend programming, learning, or writing to be just one activity.\nBut instead I spend 10 minute on something then I get distracted, and try to focus again.<\/p>\n<p>Ideally, I want to be able to plan to work on something for long hours without any distraction.\nAfter that time get back online check for messages and other things.<\/p>\n<p>The distractions are not one particular thing. A few examples:<\/p>\n<ul>\n<li>I want to do something and I remember there&rsquo;s a post related to this. I go to find it and in between I click on some links and end up reading something completely unrelated.<\/li>\n<li>I am waiting for something then I go browse the web and I get distracted.<\/li>\n<li>I&rsquo;m working on something and I face a challenge I have to think for 10 minutes. I get up to get some water and check my phone along the way and get distracted. Sometimes I get distracted by making the bed.<\/li>\n<\/ul>\n<p>It feels like my brain finds a way to do something else and avoid painful situations like boredom or hard work.<\/p>\n<hr>\n<p>It reminds me of when I was in high school talking to a friend about how laying in bed with your phone can kill hours without you noticing it.\nIt was circa 2015, back then <a href=\"https:\/\/world.hey.com\/dhh\/the-totalitarians-of-the-attention-economy-3e239524\" rel=\"noopener\" target=\"_blank\">attention hungry<\/a> apps were less powerful but still lure a teenager&rsquo;s mind for few hours.\nI learned that these apps should be used very carefully.\nAt that time I made a decision to never have a charger near my bed.\nBack then it was mostly about my phone, because when I was on the computer I was either reading, or programming, or playing a game.\nEven if I wasn&rsquo;t in the mood to think, I played a strategic game or chess.\nThese activities exercise the mind and are fun.\nAll of them were an intentional activity.\nI didn&rsquo;t do any passive activities like browsing or chatting on my computer.<\/p>\n<p>Later I discovered HackerNoon and Medium, it was the first website that I was browsing whenever I was bored behind the computer.\nThis meant that I had a way to get out boredom easily.\nIt used to have some high quality content. It inspired me to do some projects and learn more programming.\nI found channels like <a href=\"https:\/\/www.youtube.com\/@CSDojo\/videos\" rel=\"noopener\" target=\"_blank\">CSDojo<\/a> there.\nNowadays I don&rsquo;t even open them.\nThey are filled with slop or click bait articles, probably because of monetization incentives.\nThen hackernews, and YouTube and others took their place.\nI also found some good people and blogs along the way.\nI learned to keep a <a href=\"https:\/\/r.glyphack.com\/s\/s\" rel=\"noopener\" target=\"_blank\">reading list<\/a> from people I like to read when I&rsquo;m on the bus.<\/p>\n<p>I slowly found more activities for when I&rsquo;m bored.\nThis made it harder to focus on hard things for me.\nWhat kept me on track was that there was no way out.\nI had to work out some algebra problems. I kept myself to a very high bar of understanding what I do.\nAnd I did everything in LaTeX so I couldn&rsquo;t copy from someone else. I was the only one typing them.<\/p>\n<hr>\n<p>The first time I saw people not putting the effort and still get the reward for it was at work.\nYou might wonder, aren&rsquo;t people in the university constantly cheating and copying homework, and get good grades?\nWell yes, but when you talked with someone in that group it was clear that he is clueless about the subject.\nA good grade didn&rsquo;t mean much to me at that time. And as a student cheating does not get you that far.<\/p>\n<p>At work it is different.\nMy days are mostly spend meetings and over chat.\nThe balance between <a href=\"https:\/\/www.spakhm.com\/bullshit-ratio\" rel=\"noopener\" target=\"_blank\">actual work and bullshit<\/a> is skewed.\nI saw people who were barely doing any work and just talk are successful.\nAs long as people give 10% of their attention to work they are considered fine in most environments.\nPreviously I did <a href=\"https:\/\/glyphack.com\/tracking-time\/\" rel=\"noopener\" target=\"_blank\">an experiment<\/a> to track my time and found out that I spent 8 hours chatting on slack in a week.<\/p>\n<p>I don&rsquo;t care how employers want to shatter employee&rsquo;s focus.\nBut this made me get used to distractions when I&rsquo;m programming.\nI&rsquo;m trying to undo this damage.<\/p>\n<hr>\n<p>The next big change is more usage of LLMs.\nI find myself in this situation too many times, where I outsource something to an LLM and then I start working on something else.\nAnd while doing this I keep thinking about what it&rsquo;s doing.\nOr when I&rsquo;m thinking about something I start chatting with an LLM about my idea and instead of getting started on something I&rsquo;m in research mode only for hours.<\/p>\n<p>It&rsquo;s good that I&rsquo;m able to ask something else to research some topic for me.\nBut the productivity only comes if I can move on to something else and not think about it.\nI don&rsquo;t have any notifications turned on so it doesn&rsquo;t distract me.\nBut I still find myself thinking about what I just asked it to do and I cannot focus on something else.<\/p>\n<p>At the same time if I&rsquo;m spending the time interactively with an LLM I feel slow.\nI have to wait for the response and I have to correct every response coming out.\nThe best use of AI seems to be outsourcing what they can do end to end without error.<\/p>\n<p>Why do I keep doing this? Presumably because it&rsquo;s easy and fast, and productive.\nIf I realize I have to do something I can write it down to do it later or I can just ask the LLM to do it.\nThe problem only shows up when I start doing so many things at once because it&rsquo;s actually doing the thing.\nThen I have multiple things on my mind and can&rsquo;t focus really. I have to check on it and guide it in the right direction every now and then.<\/p>\n<p>And this is overstimulating, in a way that doing something without LLM sometimes is boring.\nYou don&rsquo;t see the results as fast.\nWhich makes focusing harder.<\/p>\n<hr>\n<p>So how can I regain my ability to focus?\nSometimes I <a href=\"https:\/\/www.youtube.com\/channel\/UCcmQIfYP9cdK291cvDIsllg\" rel=\"noopener\" target=\"_blank\">live stream<\/a> what I&rsquo;m doing just because with a camera I cannot escape from hard challenges by grabbing my phone.\nI used to co-work with my friends over discord.\nUnfortunately it&rsquo;s not possible anymore because people in Iran cannot have a stable internet connection nowadays.<\/p>\n<p>It&rsquo;s incredibly hard to commit to something for a long period of time.\nIf what I&rsquo;m doing is going to take multiple days to have a result I have less motivations to do it.\nMeanwhile, vibe coding small utility scripts is fun I keep doing it whenever I see a friction.\nWhenever I am stuck I can throw my problem into it and wait strengthen this habit of waiting for an answer from someone as opposed to work through problems.\nAnd they are fast in getting back the results.\nSo next time I have to read a paper to understand the subject I will be more reluctant because I can get a faster result through them.<\/p>\n<p>I&rsquo;m changing some habits to replace the current ones.\nIf I don&rsquo;t feel motivated enough to do anything I get up and pick up a book to read.\nI&rsquo;m keeping a <a href=\"https:\/\/glyphack.com\/s\/garden\/\">Garden<\/a> in my balcony is that when I&rsquo;m tired I can move the soil around and plant some pots and prune plants.\nI find this to be a less addictive than say, watching a movie.\nWhen the motivation comes back I can stop it easily.<\/p>\n<hr>\n<p>My goal was not to find an answer for this problem.\nI wanted to see what&rsquo;s going on and why I am not doing anything inconsequential in the past few months.\nAnyway the timer I set to write this really helped.\nI spent a lot more time to write this but it gave me the initial motivation to write.<\/p>\n"},{"title":"This plant is two years old","link":"https:\/\/glyphack.com\/2g\/","pubDate":"Sun, 28 Jun 2026 00:00:00 +0000","guid":"https:\/\/glyphack.com\/2g\/","description":"<p>I planted my first plant in this apartment nearly three years ago.\nIt was the result of a walk through a supermarket with my friend, where I suddenly saw the stuff needed for gardening and tried planting some herbs.<\/p>\n<p>I had some moderately happy plants from the seeds that I bought.\nOne day I was wondering to myself if I can get a capsicum plan from the seeds of bell peppers in the home.\nI took about 4 bell peppers and extracted their seeds.\nI planted the seed in the soil and watered it.\nAfter a month of watering I didn&rsquo;t see anything growing.\nSuddenly, after 5 months a the capsicum appeared.\nThe seeds were waiting for the right temperature to start sprouting.<\/p>\n<p>This plant is now 2 years old.\nI have harvested about 5 small bell peppers from it. It&rsquo;s beautiful and flowers during spring.\nEach year some branches die during the winter. Some leaves get infested with pests and I have to cut it but it continues to grow.<\/p>\n<p>\n\n\n\n\n\n\n\n<figure class=\"\">\n\n    \n    \n      \n        \n        <div class=\"img-container\" style=\"--w: 768; --h: 1024;\">\n            <img loading=\"lazy\" alt=\"bell-pepper-plant.jpeg\" src=\"https:\/\/glyphack.com\/2g\/bell-pepper-plant_hu_a1997f720b420e7d.jpeg\" width=\"768\" height=\"1024\">\n        <\/div>\n      \n    \n\n    \n<\/figure>\n<\/p>\n<p>\n\n\n\n\n\n\n\n<figure class=\"\">\n\n    \n    \n      \n        \n        <div class=\"img-container\" style=\"--w: 768; --h: 1024;\">\n            <img loading=\"lazy\" alt=\"bell-pepper-flowers.jpeg\" src=\"https:\/\/glyphack.com\/2g\/bell-pepper-flowers_hu_a18b04b3bf644aac.jpeg\" width=\"768\" height=\"1024\">\n        <\/div>\n      \n    \n\n    \n<\/figure>\n<\/p>\n<p>I&rsquo;m not sure how long it&rsquo;s going to continue, I&rsquo;ve heard that these plants <a href=\"https:\/\/en.wikipedia.org\/wiki\/Bolting_(horticulture)\" rel=\"noopener\" target=\"_blank\">stop<\/a> producing edible parts at some point.\nBut anyway I&rsquo;m glad that I like plants. It makes the house beautiful and you produce something you can eat. It feels great.<\/p>\n<p>This year it started to grow some baby capsicum plants.\n\n\n\n\n\n\n\n\n<figure class=\"\">\n\n    \n    \n      \n        \n        <div class=\"img-container\" style=\"--w: 768; --h: 1024;\">\n            <img loading=\"lazy\" alt=\"tiny-bell-pepper.jpeg\" src=\"https:\/\/glyphack.com\/2g\/tiny-bell-pepper_hu_77b26e8d0bf99729.jpeg\" width=\"768\" height=\"1024\">\n        <\/div>\n      \n    \n\n    \n<\/figure>\n<\/p>\n<p>The hardest part of maintaining it was the pests.\nThese are susceptible to <a href=\"https:\/\/en.wikipedia.org\/wiki\/Spider_mite\" rel=\"noopener\" target=\"_blank\">spider mites<\/a>. They usually live behind the leaves and eat the juice inside them and slowly kill them.\nI usually just wash the leaves with water and spray with pest killers.\nLast year another funny thing happened.\nI found a mint plant on the street. It was completely healthy so I took it home.\nAnd that&rsquo;s how I brought <a href=\"https:\/\/en.wikipedia.org\/wiki\/Whitefly\" rel=\"noopener\" target=\"_blank\">Whitefly<\/a> into my home.\nIt was really hard to get them out. They quickly spread among my plants including this one.\nOne night I gathered all of my pots and started cleaning the leaves. Whiteflies mostly live on the leaves.<\/p>\n<p>\n\n\n\n\n\n\n\n<figure class=\"\">\n\n    \n    \n      \n        \n        <div class=\"img-container\" style=\"--w: 4032; --h: 3024;\">\n            <img loading=\"lazy\" alt=\"whitefly-capsicum.jpeg\" src=\"https:\/\/glyphack.com\/2g\/whitefly-capsicum_hu_2f819b1253960d19.jpeg\" width=\"4032\" height=\"3024\">\n        <\/div>\n      \n    \n\n    \n<\/figure>\n<\/p>\n<p>\n\n\n\n\n\n\n\n<figure class=\"\">\n\n    \n    \n      \n        \n        <div class=\"img-container\" style=\"--w: 3024; --h: 4032;\">\n            <img loading=\"lazy\" alt=\"whitefly-tomato.jpeg\" src=\"https:\/\/glyphack.com\/2g\/whitefly-tomato_hu_724297b41e60e94a.jpeg\" width=\"3024\" height=\"4032\">\n        <\/div>\n      \n    \n\n    \n<\/figure>\n<\/p>\n<p>You might think that I learned my lesson to not bring plants from the street into my home.\nI didn&rsquo;t, just last week I brought 3 pots inside. 2 of them were very dry olive trees. One of them was <a href=\"https:\/\/en.wikipedia.org\/wiki\/Hydrangea_macrophylla\" rel=\"noopener\" target=\"_blank\">Hydrangea<\/a>. The Hydrangea had a lot of spider mites around it, I wasn&rsquo;t even sure if it&rsquo;s going to survive.\nBut every new plant brought from outside can teach me something new.\nI don&rsquo;t like my own plants to be stressed or nearly dying. But when I find one on the streets it&rsquo;s time to learn how to bring a plant back to life.<\/p>\n<p>\n\n\n\n\n\n\n\n<figure class=\"\">\n\n    \n    \n      \n        \n        <div class=\"img-container\" style=\"--w: 952; --h: 1269;\">\n            <img loading=\"lazy\" alt=\"olive-2026-06-17.png\" src=\"https:\/\/glyphack.com\/2g\/olive-2026-06-17_hu_65fd7204b66b122c.png\" width=\"952\" height=\"1269\">\n        <\/div>\n      \n    \n\n    \n<\/figure>\n<\/p>\n<p>\n\n\n\n\n\n\n\n<figure class=\"\">\n\n    \n    \n      \n        \n        <div class=\"img-container\" style=\"--w: 960; --h: 1280;\">\n            <img loading=\"lazy\" alt=\"hydrangea-macrophylla-bad.png\" src=\"https:\/\/glyphack.com\/2g\/hydrangea-macrophylla-bad_hu_12dbcceb1719565d.png\" width=\"960\" height=\"1280\">\n        <\/div>\n      \n    \n\n    \n<\/figure>\n\n\n\n\n\n\n\n\n\n<figure class=\"\">\n\n    \n    \n      \n        \n        <div class=\"img-container\" style=\"--w: 3024; --h: 4032;\">\n            <img loading=\"lazy\" alt=\"hydragea-macrophylla-stem.jpeg\" src=\"https:\/\/glyphack.com\/2g\/hydragea-macrophylla-stem_hu_e91e27ce79623d18.jpeg\" width=\"3024\" height=\"4032\">\n        <\/div>\n      \n    \n\n    \n<\/figure>\n<\/p>\n<p>I never planned to have this many plants inside. I&rsquo;m glad that it happened.\nWhen I&rsquo;m not feeling alright or need something to pass the time I always have some soil and pots to plant something.\nOr there&rsquo;s some pruning to do.\nThis is a far superior option to scrolling on the phone.\nPlants are beautiful. They are also alive. It&rsquo;s not just growing but you see them <a href=\"https:\/\/en.wikipedia.org\/wiki\/Plant_perception_(physiology)\" rel=\"noopener\" target=\"_blank\">reacting<\/a> to the environment. They crawl around objects, react to temperature and sun.\nGet some plants. Use the shovel and move the soil around when you need time to pass.<\/p>\n"},{"title":"Devlog 8: Fuzzing ty and dotfile improvements","link":"https:\/\/glyphack.com\/dv-8\/","pubDate":"Tue, 02 Jun 2026 00:05:46 +0200","guid":"https:\/\/glyphack.com\/dv-8\/","description":"<p>It&rsquo;s been a while since I wrote a dev log.\nI was working on some networking project and I didn&rsquo;t really think about other stuff for a few weeks.\nNow I&rsquo;m slowly getting back to writing more blogs.<\/p>\n<p>My plan is to share more notes on my blog like my current <a href=\"https:\/\/glyphack.com\/reading-list\/\">bookshelf<\/a>.\nI&rsquo;ll probably start with sharing my travels.<\/p>\n<h2 class=\"heading\" id=\"fuzzing-ty\">\n  Fuzzing Ty\n  <a class=\"anchor\" href=\"#fuzzing-ty\">#<\/a>\n<\/h2>\n<p>Fuzzing is a cool technique. With fuzzing you can discover crashes in a codebase that you haven&rsquo;t worked in before.\nYou might not find the most interesting crashes.\nBut you find crashes.<\/p>\n<p>So what&rsquo;s fuzzing?\nYou have a huge set of input data for your program.\nThis is called a corpus.\nA fuzzer program can feed input to your program and detect if it crashes.\nIt can also modify the input to generate new input.\nThe loop is to generate input data, feed it into the program, check for a crash, mutate again.<\/p>\n<p>The mutation can be based on code coverage.\nThe fuzzer can explore random changes to input and try to hit new paths in the program.<\/p>\n<p>It all started after seeing <a href=\"https:\/\/github.com\/astral-sh\/ruff\/pull\/23146\" rel=\"noopener\" target=\"_blank\">This PR<\/a>.\nI wondered if I could find a couple of these panics by fuzzing ty.\nSo I spent a day writing a fuzzer to see how it turns out.\nI didn&rsquo;t have prior experience with fuzzing.<\/p>\n<p>The most famous fuzzing tool is probably <a href=\"https:\/\/aflplus.plus\/\" rel=\"noopener\" target=\"_blank\">AFL++<\/a>. But for ty there was already a fuzzer setup with Rust Fuzz so I used it.\nI read through <a href=\"https:\/\/rust-fuzz.github.io\/book\/cargo-fuzz\/tutorial.html\" rel=\"noopener\" target=\"_blank\">Rust Fuzz Book<\/a> and wrote a fuzzer for autocompletion in ty, and I found three crashes!<\/p>\n<ul>\n<li><a href=\"https:\/\/github.com\/astral-sh\/ty\/issues\/3087\" rel=\"noopener\" target=\"_blank\">Debug version panics on autocompletion when cursor is between two brackets<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/astral-sh\/ty\/issues\/3123\" rel=\"noopener\" target=\"_blank\">Excessive runtime for nested <code>OrderedDict<\/code> instances<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/astral-sh\/ty\/issues\/3272\" rel=\"noopener\" target=\"_blank\">Calling Enum with a single argument panics<\/a><\/li>\n<\/ul>\n<p>I suspect ty&rsquo;s fuzzer has not been used for a long time.\nFollowing the <a href=\"https:\/\/github.com\/astral-sh\/ruff\/blob\/61d78a19ece136d300290249beb2fac2cea5a266\/fuzz\/README.md#L32\" rel=\"noopener\" target=\"_blank\">instructions<\/a>, I ran <code>cargo +nightly fuzz run<\/code> and I immediately got an error.<\/p>\n<p>This build command was for M1 Macs. I&rsquo;m on Apple silicon but I&rsquo;m not on M1.\nAnyway, I then tried the other command that was for non-M1 Macs.\nAnother error.<\/p>\n<p>At this point I started reading the error message to figure out the problem:<\/p>\n<div class=\"code-block\">\n  <pre tabindex=\"0\"><code>  = note: Undefined symbols for architecture arm64:\n            &#34;___sanitizer_cov_8bit_counters_init&#34;, referenced from:\n                _sancov.module_ctor_8bit_counters in pep440_rs-bed188ac979b131c.pep440_rs.4bdb810c36179955-cgu.0.rcgu.o\n                _sancov.module_ctor_8bit_counters in libunicode_width-eea06fe4ff42950d.rlib[3](unicode_width-eea06fe4ff42950d.unicode_width.1584ecc128ad03be-cgu.0.rcgu.o)\n...\nld: symbol(s) not found for architecture arm64\nclang: error: linker command failed with exit code 1 (use -v to see invocation)<\/code><\/pre>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>I was already running with <code>-s none<\/code>, which should disable all sanitizers.\nIf a codebase has a lot of <code>unsafe<\/code> code, sanitizers help to uncover more memory issues.\nIn the case of ty, I knew there could be crashes in non-unsafe code because of assertions.<\/p>\n<p>So I searched around and another solution was to provide a stub for <a href=\"https:\/\/clang.llvm.org\/docs\/SanitizerCoverage.html\" rel=\"noopener\" target=\"_blank\">Sanitizer Coverage<\/a>.\nIn the error above the <code>___sanitizer_cov_8bit_counters_init<\/code> definition is missing.\nI can define this symbol and provide it to the compiler.\nEach definition has <code>__attribute__((weak))<\/code>.\nThis would allow these definitions to be overridden.<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-cpp\" data-lang=\"cpp\"><span style=\"display:flex;\"><span><span style=\"color:#427b58\">#include<\/span> <span style=\"color:#427b58;font-style:italic\">&lt;stdint.h&gt;<\/span><span style=\"color:#427b58\">\n<\/span><\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>__attribute__((weak)) <span style=\"color:#b57614\">void<\/span> __sanitizer_cov_8bit_counters_init(<span style=\"color:#b57614\">uint8_t<\/span> <span style=\"color:#af3a03\">*<\/span>start,\n<\/span><\/span><span style=\"display:flex;\"><span>                                                              <span style=\"color:#b57614\">uint8_t<\/span> <span style=\"color:#af3a03\">*<\/span>stop) {}\n<\/span><\/span><span style=\"display:flex;\"><span>__attribute__((weak)) <span style=\"color:#b57614\">void<\/span> __sanitizer_cov_pcs_init(<span style=\"color:#af3a03\">const<\/span> uintptr_t <span style=\"color:#af3a03\">*<\/span>pcs_beg,\n<\/span><\/span><span style=\"display:flex;\"><span>                                                    <span style=\"color:#af3a03\">const<\/span> uintptr_t <span style=\"color:#af3a03\">*<\/span>pcs_end) {}\n<\/span><\/span><span style=\"display:flex;\"><span>__attribute__((weak)) <span style=\"color:#b57614\">void<\/span> __sanitizer_cov_trace_const_cmp1(<span style=\"color:#b57614\">uint8_t<\/span> a,\n<\/span><\/span><span style=\"display:flex;\"><span>                                                            <span style=\"color:#b57614\">uint8_t<\/span> b) {}\n<\/span><\/span><span style=\"display:flex;\"><span>__attribute__((weak)) <span style=\"color:#b57614\">void<\/span> __sanitizer_cov_trace_const_cmp2(<span style=\"color:#b57614\">uint16_t<\/span> a,\n<\/span><\/span><span style=\"display:flex;\"><span>                                                            <span style=\"color:#b57614\">uint16_t<\/span> b) {}\n<\/span><\/span><span style=\"display:flex;\"><span>__attribute__((weak)) <span style=\"color:#b57614\">void<\/span> __sanitizer_cov_trace_const_cmp4(<span style=\"color:#b57614\">uint32_t<\/span> a,\n<\/span><\/span><span style=\"display:flex;\"><span>                                                            <span style=\"color:#b57614\">uint32_t<\/span> b) {}\n<\/span><\/span><span style=\"display:flex;\"><span>__attribute__((weak)) <span style=\"color:#b57614\">void<\/span> __sanitizer_cov_trace_cmp1(<span style=\"color:#b57614\">uint8_t<\/span> a, <span style=\"color:#b57614\">uint8_t<\/span> b) {}\n<\/span><\/span><span style=\"display:flex;\"><span>__attribute__((weak)) <span style=\"color:#b57614\">void<\/span> __sanitizer_cov_trace_cmp2(<span style=\"color:#b57614\">uint16_t<\/span> a, <span style=\"color:#b57614\">uint16_t<\/span> b) {}\n<\/span><\/span><span style=\"display:flex;\"><span>__attribute__((weak)) <span style=\"color:#b57614\">void<\/span> __sanitizer_cov_trace_cmp4(<span style=\"color:#b57614\">uint32_t<\/span> a, <span style=\"color:#b57614\">uint32_t<\/span> b) {}\n<\/span><\/span><span style=\"display:flex;\"><span>__attribute__((weak)) <span style=\"color:#b57614\">void<\/span> __sanitizer_cov_trace_cmp8(<span style=\"color:#b57614\">uint64_t<\/span> a, <span style=\"color:#b57614\">uint64_t<\/span> b) {}\n<\/span><\/span><span style=\"display:flex;\"><span>__attribute__((weak)) <span style=\"color:#b57614\">void<\/span> __sanitizer_cov_trace_switch(<span style=\"color:#b57614\">uint64_t<\/span> val,\n<\/span><\/span><span style=\"display:flex;\"><span>                                                        <span style=\"color:#b57614\">uint64_t<\/span> <span style=\"color:#af3a03\">*<\/span>cases) {}\n<\/span><\/span><span style=\"display:flex;\"><span>__attribute__((weak)) <span style=\"color:#b57614\">void<\/span> __sanitizer_cov_trace_div4(<span style=\"color:#b57614\">uint32_t<\/span> val) {}\n<\/span><\/span><span style=\"display:flex;\"><span>__attribute__((weak)) <span style=\"color:#b57614\">void<\/span> __sanitizer_cov_trace_pc_indir(uintptr_t callee) {}<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>Finally compiled!\nI&rsquo;m not an expert in this topic and don&rsquo;t know what the correct solution is.\nThe reason I&rsquo;m fine with this is that I have disabled sanitizers with <code>-s none<\/code>.\nI assume this means sanitizers will not be used during fuzzing.\nSo an empty stub should be fine.<\/p>\n<p>Now what are we testing in ty?<\/p>\n<p>The fuzzer randomly generates Python code.\nIt then inserts the cursor at every line\/column combination possible and calls the <code>completion<\/code> function.\nThis function will return autocomplete suggestions for that line and column.\nIf there&rsquo;s a crash, the fuzzer will report it.<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-rust\" data-lang=\"rust\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">fn<\/span> <span style=\"color:#b57614\">do_fuzz<\/span>(source: <span style=\"color:#af3a03\">&amp;<\/span>[<span style=\"color:#b57614\">u8<\/span>]) -&gt; Corpus {\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">let<\/span> code <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#af3a03\">match<\/span> std::<span style=\"color:#b57614\">str<\/span>::from_utf8(source) {\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#b57614\">Ok<\/span>(s) <span style=\"color:#af3a03\">if<\/span> <span style=\"color:#af3a03\">!<\/span>s.is_empty() <span style=\"color:#af3a03\">=&gt;<\/span> s,\n<\/span><\/span><span style=\"display:flex;\"><span>        _ <span style=\"color:#af3a03\">=&gt;<\/span> <span style=\"color:#af3a03\">return<\/span> Corpus::Reject,\n<\/span><\/span><span style=\"display:flex;\"><span>    };\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">let<\/span> parsed <span style=\"color:#af3a03\">=<\/span> parse_unchecked(code, ParseOptions::from(Mode::Module));\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">if<\/span> parsed.has_invalid_syntax() {\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">return<\/span> Corpus::Reject;\n<\/span><\/span><span style=\"display:flex;\"><span>    }\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">let<\/span> (<span style=\"color:#af3a03\">mut<\/span> db, python_file) <span style=\"color:#af3a03\">=<\/span> setup_db();\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">let<\/span> settings <span style=\"color:#af3a03\">=<\/span> CompletionSettings { auto_import: false };\n<\/span><\/span><span style=\"display:flex;\"><span>    db.write_file(<span style=\"color:#d3869b\">PYTHON_PATH<\/span>, code).unwrap();\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">for<\/span> offset <span style=\"color:#af3a03\">in<\/span> (<span style=\"color:#8f3f71\">0<\/span><span style=\"color:#af3a03\">..=<\/span>code.len()).filter(<span style=\"color:#af3a03\">|&amp;<\/span>i<span style=\"color:#af3a03\">|<\/span> code.is_char_boundary(i)) {\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">let<\/span> text_offset <span style=\"color:#af3a03\">=<\/span> TextSize::from(offset <span style=\"color:#af3a03\">as<\/span> <span style=\"color:#b57614\">u32<\/span>);\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">let<\/span> _ <span style=\"color:#af3a03\">=<\/span> completion(<span style=\"color:#af3a03\">&amp;<\/span>db, <span style=\"color:#af3a03\">&amp;<\/span>settings, python_file, text_offset);\n<\/span><\/span><span style=\"display:flex;\"><span>    }\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    Corpus::Keep\n<\/span><\/span><span style=\"display:flex;\"><span>}\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#b57614\">fuzz_target!<\/span>(<span style=\"color:#af3a03\">|<\/span>data: <span style=\"color:#af3a03\">&amp;<\/span>[<span style=\"color:#b57614\">u8<\/span>]<span style=\"color:#af3a03\">|<\/span> {\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">let<\/span> _ <span style=\"color:#af3a03\">=<\/span> do_fuzz(data);\n<\/span><\/span><span style=\"display:flex;\"><span>});<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>In fuzzing you usually leave the full argument generation to the fuzzer.\nThe code above doesn&rsquo;t do that.\nI let the fuzzer generate the source.\nI try all the positions in a file and there&rsquo;s a higher chance of finding a crash.<\/p>\n<p>Out of curiosity, I also tried another approach called <a href=\"https:\/\/rust-fuzz.github.io\/book\/cargo-fuzz\/structure-aware-fuzzing.html\" rel=\"noopener\" target=\"_blank\">Structure-Aware Fuzzing<\/a>.\nThe idea is that you define your input and the fuzzer generates it instead of a byte array.<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-rust\" data-lang=\"rust\"><span style=\"display:flex;\"><span><span style=\"color:#427b58\">#[derive(Debug)]<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">struct<\/span> CompletionInput {\n<\/span><\/span><span style=\"display:flex;\"><span>    source: <span style=\"color:#b57614\">String<\/span>,\n<\/span><\/span><span style=\"display:flex;\"><span>    cursor_offset: <span style=\"color:#b57614\">usize<\/span>,\n<\/span><\/span><span style=\"display:flex;\"><span>}\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">impl<\/span><span style=\"color:#af3a03\">&lt;<\/span><span style=\"color:#79740e;font-weight:bold\">&#39;a<\/span><span style=\"color:#af3a03\">&gt;<\/span> Arbitrary<span style=\"color:#af3a03\">&lt;<\/span><span style=\"color:#79740e;font-weight:bold\">&#39;a<\/span><span style=\"color:#af3a03\">&gt;<\/span> <span style=\"color:#af3a03\">for<\/span> CompletionInput {\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">fn<\/span> <span style=\"color:#b57614\">arbitrary<\/span>(u: <span style=\"color:#af3a03\">&amp;<\/span>mut Unstructured<span style=\"color:#af3a03\">&lt;<\/span><span style=\"color:#79740e;font-weight:bold\">&#39;a<\/span><span style=\"color:#af3a03\">&gt;<\/span>) -&gt; arbitrary::<span style=\"color:#b57614\">Result<\/span><span style=\"color:#af3a03\">&lt;<\/span><span style=\"color:#b57614\">Self<\/span><span style=\"color:#af3a03\">&gt;<\/span> {\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">let<\/span> source <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#b57614\">String<\/span>::arbitrary(u)<span style=\"color:#af3a03\">?<\/span>;\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">if<\/span> source.is_empty() {\n<\/span><\/span><span style=\"display:flex;\"><span>            <span style=\"color:#af3a03\">return<\/span> <span style=\"color:#b57614\">Err<\/span>(arbitrary::Error::IncorrectFormat);\n<\/span><\/span><span style=\"display:flex;\"><span>        }\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">let<\/span> boundaries: <span style=\"color:#b57614\">Vec<\/span><span style=\"color:#af3a03\">&lt;<\/span><span style=\"color:#b57614\">usize<\/span><span style=\"color:#af3a03\">&gt;<\/span> <span style=\"color:#af3a03\">=<\/span> (<span style=\"color:#8f3f71\">0<\/span><span style=\"color:#af3a03\">..=<\/span>source.len())\n<\/span><\/span><span style=\"display:flex;\"><span>            .filter(<span style=\"color:#af3a03\">|&amp;<\/span>i<span style=\"color:#af3a03\">|<\/span> source.is_char_boundary(i))\n<\/span><\/span><span style=\"display:flex;\"><span>            .collect();\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">let<\/span> <span style=\"color:#af3a03\">&amp;<\/span>offset <span style=\"color:#af3a03\">=<\/span> u.choose(<span style=\"color:#af3a03\">&amp;<\/span>boundaries)<span style=\"color:#af3a03\">?<\/span>;\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#b57614\">Ok<\/span>(<span style=\"color:#b57614\">Self<\/span> {\n<\/span><\/span><span style=\"display:flex;\"><span>            source,\n<\/span><\/span><span style=\"display:flex;\"><span>            cursor_offset: offset,\n<\/span><\/span><span style=\"display:flex;\"><span>        })\n<\/span><\/span><span style=\"display:flex;\"><span>    }\n<\/span><\/span><span style=\"display:flex;\"><span>}<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>And the logic for the fuzzer is to just find valid inputs and run the <code>completion<\/code> function.<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-rust\" data-lang=\"rust\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">fn<\/span> <span style=\"color:#b57614\">do_fuzz<\/span>(input: CompletionInput) -&gt; Corpus {\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">let<\/span> code <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#af3a03\">&amp;<\/span>input.source;\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">let<\/span> offset <span style=\"color:#af3a03\">=<\/span> input.cursor_offset;\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">let<\/span> parsed <span style=\"color:#af3a03\">=<\/span> parse_unchecked(code, ParseOptions::from(Mode::Module));\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">if<\/span> parsed.has_invalid_syntax() {\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">return<\/span> Corpus::Reject;\n<\/span><\/span><span style=\"display:flex;\"><span>    }\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">let<\/span> (<span style=\"color:#af3a03\">mut<\/span> db, python_file) <span style=\"color:#af3a03\">=<\/span> setup_db();\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">let<\/span> settings <span style=\"color:#af3a03\">=<\/span> CompletionSettings { auto_import: false };\n<\/span><\/span><span style=\"display:flex;\"><span>    db.write_file(<span style=\"color:#d3869b\">PYTHON_PATH<\/span>, code).unwrap();\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">let<\/span> text_offset <span style=\"color:#af3a03\">=<\/span> TextSize::from(offset <span style=\"color:#af3a03\">as<\/span> <span style=\"color:#b57614\">u32<\/span>);\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">let<\/span> _ <span style=\"color:#af3a03\">=<\/span> completion(<span style=\"color:#af3a03\">&amp;<\/span>db, <span style=\"color:#af3a03\">&amp;<\/span>settings, python_file, text_offset);\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    Corpus::Keep\n<\/span><\/span><span style=\"display:flex;\"><span>}<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>I found two crashes with the first approach and one with the second one.\nI don&rsquo;t think there&rsquo;s a winner here.\nBoth methods can be useful.\nTrying all positions is slower.\nAlso you lose the benefit of the fuzzer saving the exact input that causes the crash.<\/p>\n<p>Found crashes, what now?<\/p>\n<p>Leaving a fuzzer on a project that is not stable yet, like ty, is expected to generate lots of crashes.\nIt&rsquo;s good to report them.\nBut it&rsquo;s important to not flood the issues by copying and pasting what the fuzzer is giving us. (Sounds familiar, doesn&rsquo;t it?)<\/p>\n<p>Make sure to not report one problem multiple times.\nMultiple crashes might share the same root cause.\nReporting that once with all the examples is enough.<\/p>\n<p>Use <code>creduce<\/code> to <a href=\"https:\/\/bernsteinbear.com\/blog\/creduce\/\" rel=\"noopener\" target=\"_blank\">minimize the input<\/a>.\nUsually fuzzer inputs are large, and the culprit is just in one or two lines.\nMinimizing is taking a large input and reducing it to the absolute minimum that still causes the crash.\nDebugging with a single line of Python code is a lot easier than debugging with a large program.<\/p>\n<p>And finally, send patches too.\nCharlie fixed <a href=\"https:\/\/github.com\/astral-sh\/ty\/issues\/3272\" rel=\"noopener\" target=\"_blank\">this<\/a> 20 minutes after I opened it and didn&rsquo;t give me a chance.<\/p>\n<p>This was a fun journey!\nIt took me two days and I learned a lot.<\/p>\n<p>I liked cargo fuzz. Next time I&rsquo;ll try the <a href=\"https:\/\/aflplus.plus\/libafl-book\/introduction.html\" rel=\"noopener\" target=\"_blank\">LibAFL<\/a> library to make a custom fuzzer.<\/p>\n<h2 class=\"heading\" id=\"g-command\">\n  <code>,g<\/code> Command\n  <a class=\"anchor\" href=\"#g-command\">#<\/a>\n<\/h2>\n<p>I am a fan of customizing git configuration.\nI started customizing my <a href=\"https:\/\/github.com\/glyphack\/dotfiles\/blob\/0bcb61f1e2812acf9807ecff527060817f8306c7\/gitconf\/.gitconfig#L1\" rel=\"noopener\" target=\"_blank\"><code>.gitconfig<\/code><\/a> a few years ago.<\/p>\n<p>But I barely used git CLI wrappers, or GUI apps that aim to replace git.\nThe operations I do with git are simple and similar.\nAfter you&rsquo;ve undone a commit a few times you memorize it and don&rsquo;t need a GUI for it.\nAnother advantage of using the git CLI is that it works everywhere.\nI do most of my git operations in fugitive in Vim with the same git commands.<\/p>\n<p>But I finally made a git CLI wrapper!\nIt all started with a command I made some years ago. The command <code>myprs<\/code> used <code>gh<\/code> to list my pull requests using <a href=\"https:\/\/github.com\/charmbracelet\/gum\" rel=\"noopener\" target=\"_blank\">gum<\/a> for selecting the branch to check out to.\nMy programming is mostly oriented around pull requests.\nHaving something that lists all the stuff I have open was a huge win for me.\nOr if you work in a large repo with lots of contributors it&rsquo;s easier to switch between your branches because you don&rsquo;t see others&rsquo; branches.\nLater I learned there are <a href=\"https:\/\/www.youtube.com\/watch?v=GKBq5Xo_B6I\" rel=\"noopener\" target=\"_blank\">ways<\/a> to do this with git itself too.<\/p>\n<p>But there are other things that I do repetitively.<\/p>\n<ul>\n<li>Find name of the base branch (main\/master)<\/li>\n<li>Deleting branches that are already merged<\/li>\n<li>Keeping the worktrees in a specific place so I can keep track of them.<\/li>\n<\/ul>\n<p>The result is the <code>,g<\/code> command.\nIt uses <a href=\"https:\/\/github.com\/charmbracelet\/gum\" rel=\"noopener\" target=\"_blank\">gum<\/a> to make it easy to list and choose stuff.<\/p>\n<p><code>,g new branch-name<\/code> makes a new branch.\nWith the <code>-w<\/code> flag it makes it a worktree. Worktrees are stored in <code>{worktree-path}\/{project}-{branch}<\/code>. This makes it easier to search and manage.\nNew branches are created from the upstream base branch.\nThe main\/master is resolved by the command.<\/p>\n<p><code>,g co<\/code> lets me switch between my branches or pull requests.\nBefore switching, anything in the current branch is stashed.\nWhen I come back to it, saved changes are unstashed.\nI used to do this with a subcommand <code>git wip<\/code> but I ditched that for an automated one.\nMaybe I lose a change one day and go back to manual.<\/p>\n<p>The command <code>,g done<\/code> checks out to base and deletes the branch and its worktree.\nI also migrated my pull request scripts over so with <code>,g pr create<\/code> I can create them quickly.\n<code>,g pr review<\/code> copies a message with pull request link to share with others for review.\nI use <code>,g coauthored glyphack<\/code> to generate a co-authored-by message.\nThere are more commands I added based on the things I do frequently and need multiple command invocations.\nYou can check it out in my <a href=\"https:\/\/github.com\/Glyphack\/dotfiles\/blob\/9f39ecf04c60900a94c8d1632a20b6432efe4b14\/fish\/functions\/%2Cg.fish#L1\" rel=\"noopener\" target=\"_blank\">dotfiles<\/a>.<\/p>\n<p>It&rsquo;s written in fish. I ported it to fish with an LLM from bash scripts that I had over the years.\nI&rsquo;m comfortable with Python for these kinds of things but I don&rsquo;t know if it&rsquo;s important enough for me to rewrite it.<\/p>\n<h2 class=\"heading\" id=\"vm2-command-for-running-containers\">\n  vm2 command for running containers\n  <a class=\"anchor\" href=\"#vm2-command-for-running-containers\">#<\/a>\n<\/h2>\n<p>It is scary to install anything on your computer these days.\nEach app has <a href=\"https:\/\/pointersgonewild.com\/2026-05-11-dependencies-are-a-liability\" rel=\"noopener\" target=\"_blank\">a shit ton of dependencies<\/a>.\nNearly every week a dependency is compromised.<\/p>\n<p>But I like to install coding agents and try them.\nAt the same time I&rsquo;m not confident in installing and running them on my system.<\/p>\n<p>I used to use <a href=\"https:\/\/github.com\/lynaghk\/vibe\" rel=\"noopener\" target=\"_blank\">vibe<\/a> for running agents.\nI love how it&rsquo;s implemented and how customizable it is.\nBut two things made me reconsider:<\/p>\n<ul>\n<li>I know nothing about containers. Discovering networking <a href=\"https:\/\/github.com\/lynaghk\/vibe\/issues\/27\" rel=\"noopener\" target=\"_blank\">problems<\/a> is not for me.<\/li>\n<li>Some features like <code>exec<\/code> are already provided in containers. But another tool means implementing them again.<\/li>\n<\/ul>\n<p>So I tried a different approach with <a href=\"https:\/\/github.com\/apple\/container\" rel=\"noopener\" target=\"_blank\">Apple Container<\/a>.\nI need a container with the tools I want to use.\nThen run the container with directories mounted.\nTo simplify the workflow, I built a small CLI for it.\nI can also customize the volumes and environment variables that are passed to the container.<\/p>\n<p>Check it out <a href=\"https:\/\/github.com\/Glyphack\/dotfiles\/tree\/be7d14e92480294cd59a83903dfaa7bb7be8f0f2\/scripts\/vm2\" rel=\"noopener\" target=\"_blank\">here<\/a>.<\/p>\n<p>Here&rsquo;s how it works:<\/p>\n<p>Run <code>vm2 build<\/code> to build the image.\nRunning <code>vm2<\/code> in any directory creates a container and mounts:<\/p>\n<ul>\n<li>current working directory<\/li>\n<li>configurations like <code>.claude<\/code><\/li>\n<li>the <code>.git<\/code> folder is included in the container. If it&rsquo;s a worktree, the <code>.git<\/code> is resolved and mounted.<\/li>\n<\/ul>\n<p>Git provides useful tools like <code>git bisect<\/code> for the agent when debugging.\nBut git push is disabled by default with a <code>pre-push<\/code> git hook.\nSo the agent cannot accidentally push garbage.<\/p>\n<p>This is partially extensible. You can write a Python script that runs and prints what customization to apply.\nIt can be setting environment variables, mounting a directory, or running a shell command after starting the container.<\/p>\n<p>The format of output is:<\/p>\n<div class=\"code-block\">\n  <pre tabindex=\"0\"><code>VOLUME src:dest\nSETUP command1;command2\nENV VAR1=VAL1<\/code><\/pre>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>I use it mostly for running agents.\nSometimes I use it when I need to run a project and I don&rsquo;t trust the dependencies.<\/p>\n<h2 class=\"heading\" id=\"links\">\n  Links\n  <a class=\"anchor\" href=\"#links\">#<\/a>\n<\/h2>\n<p><a href=\"http:\/\/www.noiseprotocol.org\" rel=\"noopener\" target=\"_blank\">Noise Protocol Framework<\/a><\/p>\n<p>I love the idea. Noise provides building blocks for creating secure communication channels.\nThis is useful for sending information back and forth over a public network.\nSince it provides the tools, you can build the security protocol in your app.\nYou manage it however you like.\nFor someone like me without a background in cryptography and security, this is the closest I can get to doing my own security.<\/p>\n<p>It&rsquo;s based on Diffie-Hellman.\nYou choose the algorithms to use and then use APIs to read and write messages.\n<a href=\"https:\/\/docs.rs\/snow\/\" rel=\"noopener\" target=\"_blank\">snow<\/a> crate implements the framework in Rust.\nI used it and it was great.<\/p>\n<p>The scenarios it protects against are:<\/p>\n<ol>\n<li>Encrypt messages: An attacker who sees the messages cannot read the data.<\/li>\n<li>Replay protection: An attacker cannot send the same bytes on behalf of the sender.<\/li>\n<li>Man in the middle: An attacker cannot impersonate the receiver and fool the client to exchange keys.<\/li>\n<\/ol>\n<p>If you like to learn more watch this <a href=\"https:\/\/www.youtube.com\/watch?v=3gipxdJ22iM\" rel=\"noopener\" target=\"_blank\">talk<\/a> from Trevor Perrin (author) and build something with it!<\/p>\n<p><a href=\"https:\/\/www.scattered-thoughts.net\/writing\/borrow-checking-surprises\/\" rel=\"noopener\" target=\"_blank\">Borrow-checking surprises<\/a><\/p>\n<p>It&rsquo;s like JavaScript puzzles of what will be printed, but for Rust borrow checker. I only hit 1 case of this.<\/p>\n<p><a href=\"https:\/\/en.wikipedia.org\/wiki\/DigiNotar\" rel=\"noopener\" target=\"_blank\">DigiNotar<\/a><\/p>\n<p>I found this when I was learning more about security with the Noise framework.\nInternet certificates that guarantee that the website you are seeing is the actual website are just an agreement.\nThe authorities can simply lie to you.<\/p>\n<blockquote>\n<p>The company was hacked in June 2011 and it issued hundreds of fraudulent\u00a0certificates, some of which were used for\u00a0man-in-the-middle attacks\u00a0on Iranian\u00a0Gmail\u00a0users.<\/p>\n<\/blockquote>\n<p><a href=\"https:\/\/matklad.github.io\/2026\/05\/12\/software-architecture\" rel=\"noopener\" target=\"_blank\">Learning Software Architecture<\/a><\/p>\n<p>Matklad was the one who saved me from the system design hell.\nThings that I heard from people about software design when I started programming were garbage.\nIf of Kafka vs another queue means software architecture to you, read this.\nIt helps you to get out of the boxes and arrow mentality of software design.<\/p>\n<p><a href=\"https:\/\/www.experimental-history.com\/p\/the-illusion-of-moral-decline\" rel=\"noopener\" target=\"_blank\">The illusion of moral decline<\/a><\/p>\n<p>This writing argues that morality is not declining.\nThe survey data from different countries, ages, etc. all agree that morality is declining.\nEspecially since the time they were born. Which if true is contradictory.\nSo something must make people feel that during their time something bad is happening.<\/p>\n<p>In the end it argues that it&rsquo;s a bias that we think this is happening.\nI am not convinced by the answer.\nBut I&rsquo;m happy to know that having nostalgia is not just for me and everyone has it.\nNow I can notice when it&rsquo;s affecting me.<\/p>\n<p><a href=\"https:\/\/youtube.com\/watch?v=EKWGGDXe5MA\" rel=\"noopener\" target=\"_blank\">Richard Feynman on Hardware, Software, and Heuristics<\/a><\/p>\n<p>Feynman explains to students how a computer works. How and where to use it.\nI&rsquo;m amazed by the level of detail he goes into in this introductory lecture.\nThis is the most accessible explanation of computer organization I&rsquo;ve seen.<\/p>\n<p>In the end he discusses the topic of &ldquo;Can machines think?&rdquo;.\nI was reading about the same topic in The Art of Doing Science and Engineering.\nBoth answers are very similar. This subject in the book has more depth. Read it if you liked the discussion in the lecture.<\/p>\n<p><a href=\"https:\/\/slatestarcodex.com\/2013\/06\/30\/the-lottery-of-fascinations\" rel=\"noopener\" target=\"_blank\">The Lottery of Fascinations<\/a><\/p>\n<p>This helps you to not feel guilty next time when you feel you should like some subject.<\/p>\n<p><a href=\"https:\/\/guzey.com\/productivity\/\" rel=\"noopener\" target=\"_blank\">Every productivity thought I&rsquo;ve ever had<\/a><\/p>\n<p>I haven&rsquo;t read a piece on productivity for a while.\nI liked it because it started out with a great tip and said that no productivity system works.\nWhich means this system probably won&rsquo;t work, so it&rsquo;s honest. Great!<\/p>\n"},{"title":"Application as a Database","link":"https:\/\/glyphack.com\/appdb\/","pubDate":"Thu, 02 Apr 2026 08:53:48 +0200","guid":"https:\/\/glyphack.com\/appdb\/","description":"<p>When I started programming professionally, I got recommended some programming books that teach you how to write good code and use well known patterns.\nYears after that I feel it was just preference of the author for how to write code.\nThere&rsquo;s only one important note about coding style and that&rsquo;s <a href=\"https:\/\/glyphack.com\/s\/the-practice-of-programming\/\" rel=\"noopener\" target=\"_blank\">consistency<\/a>.<\/p>\n<p>So after a while of programming I have formed my preference: Make data available throughout the whole program, and to not duplicate any data.\nI&rsquo;m going to explain this style in this post.<\/p>\n<h2 class=\"heading\" id=\"make-data-accessible\">\n  Make Data Accessible\n  <a class=\"anchor\" href=\"#make-data-accessible\">#<\/a>\n<\/h2>\n<p>Every part of your program should be able to lookup whatever is needed from the context.\nSimilar to how a web server has a database and every endpoint handler has access to the database.<\/p>\n<p>This is against the idea of isolating pieces of code behind interfaces.\nEvery layer you add to your program means duplicating data structures.\nEach layer loses some information when you copy the data to next layer.<\/p>\n<p>If your program has a user object and you also have a layer for interacting with database then you also have a user object in database.\nYour code needs to copy bytes around for transforming application user to database user.\nThis means a lot of useless code for copying and hurting performance of your application.<\/p>\n<p>How many layers can we remove? PostgreSQL can <a href=\"https:\/\/github.com\/sivers\/sivers#html-in-postgresql\" rel=\"noopener\" target=\"_blank\">return a complete HTTP response<\/a> if it fits the needs of your application why not?<\/p>\n<p>I like information hiding. It&rsquo;s good to hide complexity of something behind an interface.\nBut hiding the data makes changing program harder since you don&rsquo;t have access to the data to perform something.\nThis means constantly passing more arguments and values to each function to perform tasks.<\/p>\n<p><strong>How do you allow access to data everywhere?<\/strong><\/p>\n<p>One way to do this is to have a database class that contains all of your application information and pass that through classes.\nThis means passing something throughout the program.\nThis is a trade off, it changes all of your program. But as a result you have access to everything.\nThen in every part of the program you can access other data structures using the database.<\/p>\n<p>Another way of doing this is to just include all the information available when passing values back from functions.\nWhenever in a program I call an external endpoint I pass the full response object back.\nThe function that calls external service only knows how to handle HTTP errors in general way.\nBut other parts of the program can do what is more appropriate in that context.\nSo instead of taking a response and transforming it into some internal representation I have create an object that wraps the response with additional information.\nThe downside is obviously more memory usage.\nBut this is something you can get rid of later once you are sure the data is not needed.<\/p>\n<p>I&rsquo;ve seen this idea to have a database like object being used in other projects like:<\/p>\n<ul>\n<li>Atom in Clojure<\/li>\n<li>ty and Rust analyzer<\/li>\n<\/ul>\n<p>A simple compiler architecture performs these passes on the code:<\/p>\n<div class=\"code-block\">\n  <pre tabindex=\"0\"><code>Parser -&gt; Semantic Analyzer -&gt; Type Checker -&gt; Code Generator<\/code><\/pre>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>While it can be seen as something that you can just perform each pass and just pass the output to next step. This solution is not optimal.\nFor the type checker to provide good error messages it needs the source code, or to suggest a fix.\nWhich means not only you need output of the parser but you also need the input.\nYou don&rsquo;t take something transform it into another object that loses information.\nInstead, you add more information to what you already have.<\/p>\n<p>To solve this problem you can create a data structure containing all the information produced by your program.\nIn every stage you have access to this data structure.<\/p>\n<p>Without this structure you would have something like:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-python\" data-lang=\"python\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">parse<\/span>(source: <span style=\"color:#b57614\">str<\/span>) <span style=\"color:#af3a03\">-&gt;<\/span> <span style=\"color:#b57614\">dict<\/span>: <span style=\"color:#af3a03\">...<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#928374;font-style:italic\"># Now in type check you don&#39;t have the source anymore.<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#928374;font-style:italic\"># you can pass the source as well<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#928374;font-style:italic\"># This means a group of related values that have to be passed everywhere.<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">type_check<\/span>(ast: <span style=\"color:#b57614\">dict<\/span>) <span style=\"color:#af3a03\">-&gt;<\/span> <span style=\"color:#b57614\">dict<\/span>: <span style=\"color:#af3a03\">...<\/span><\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>Example of this structure:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-python\" data-lang=\"python\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">class<\/span> CompilerDB:\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">__init__<\/span>(<span style=\"color:#b57614\">self<\/span>, source: <span style=\"color:#b57614\">str<\/span>):\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>source <span style=\"color:#af3a03\">=<\/span> source\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>ast <span style=\"color:#af3a03\">=<\/span> {}      \n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>symbols <span style=\"color:#af3a03\">=<\/span> {}  \n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>diagnostics <span style=\"color:#af3a03\">=<\/span> []\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">parse<\/span>(db: CompilerDB): <span style=\"color:#af3a03\">...<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">type_check<\/span>(db: CompilerDB): <span style=\"color:#af3a03\">...<\/span><\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p><a href=\"https:\/\/petevilter.me\/post\/datalog-typechecking\" rel=\"noopener\" target=\"_blank\">This is a good introduction for full implementation<\/a>.<\/p>\n<p>You can either have an object that contains AST from parser, symbol table from semantic analyzer.<\/p>\n<p>Or you can store the data in <a href=\"https:\/\/www.dataorienteddesign.com\/dodbook\/node3.html#SECTION00340000000000000000\" rel=\"noopener\" target=\"_blank\">normalized<\/a> way and keep references to it. A function in symbol table can have a node ID to reference its source code from AST.\nThink about what&rsquo;s the relation between these objects. Since they need to reference each using IDs.\nGoing with latter and having IDs also helps with easier caching and incremental computation in your program.<\/p>\n<h2 class=\"heading\" id=\"data-deduplication\">\n  Data Deduplication\n  <a class=\"anchor\" href=\"#data-deduplication\">#<\/a>\n<\/h2>\n<p>After every part of the program is able to access any information you can easily avoid duplicating logic and data.\nThe reason is now you can avoid all the intermediate objects that only transferred one value from a layer to another.\nWhich means now for every concept you have one class for it in your program.\nOnce you have the data structures, think about ownership of the values.\nEach value would have one owner and others would store a pointer to that data.\nThis is the source of truth for that concept.<\/p>\n<p>Without the data access you need the copy of values from class to class so you have access to that value in places where it&rsquo;s needed.\nWithout that limitation, you don&rsquo;t need to copy anything.<\/p>\n<p>But there&rsquo;s another gotcha here. You might end up duplicating values from objects in different places.\nIn the example:<\/p>\n<div class=\"code-block\">\n  <pre tabindex=\"0\"><code>players: Dict[str, Player]<\/code><\/pre>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>Where key of this hashmap is player username.\nThis data modeling is duplicating the player username.\nThis means whenever you need to update a player, you also need to update this hashmap as well.\nBut it&rsquo;s not gonna stop at the hashmap since you start referring to the player using username throughout the code.<\/p>\n<p>Instead, using a stable ID (like <a href=\"https:\/\/verdagon.dev\/blog\/generational-references\" rel=\"noopener\" target=\"_blank\">generational indexes<\/a>) would be a good choice.<\/p>\n<p>Also you need to create structs that can hold all the information related to the same concept in one class or struct.\nThis helps with having a struct holding all information about something and just having an ID to access that struct means you have all the information.<\/p>\n<p>Once you have that a function like <code>get_player(db, id)<\/code> can return you the player and you can access username from the player object.<\/p>\n<p>There&rsquo;s one place where I like to have methods to access or set a field on an object. It&rsquo;s for easier code navigation.\nLSPs don&rsquo;t have a way to only show usages of an attribute where it was assigned. In that case having a method that assigns the attribute can help with finding them more easily.\nSimilarly constructor methods can help with finding all places where something is created.<\/p>\n<hr>\n<p>The push in software architecture to isolate pieces of code and data is popular in gigantic codebases.\nIn that environment, guiding programmers with the isolation works better.<\/p>\n<p>But in most cases, you can free yourself from the arbitrary constraints and write code with less indirection.<\/p>\n<p>And ask yourself <a href=\"https:\/\/www.hytradboi.com\/\" rel=\"noopener\" target=\"_blank\">have you tried rubbing a database on it?<\/a><\/p>\n"},{"title":"Devlog 7: I made a code review tool","link":"https:\/\/glyphack.com\/dv-7\/","pubDate":"Wed, 04 Mar 2026 08:51:24 +0100","guid":"https:\/\/glyphack.com\/dv-7\/","description":"<p>Last month I created a new project. It&rsquo;s called <a href=\"https:\/\/github.com\/Glyphack\/towelie\" rel=\"noopener\" target=\"_blank\">Towelie<\/a>.\nTowelie is a tool for reviewing code locally, similar to the GitHub pull request review tool.<\/p>\n<p>But why?\nLike most people, I also don&rsquo;t like the AI slop produced by AI agents.\nBut I found that with some guidance, I can get what I want for certain tasks.\nI have a loop that looks like this when I&rsquo;m writing code with an agent:<\/p>\n<ul>\n<li>agent writes code<\/li>\n<li>I review using Towelie<\/li>\n<li>I give comments to the agent<\/li>\n<\/ul>\n<p>My most effective AI usage is when I give a task that it can handle.\nGive the agent a good way to verify its work and wait until it finishes the code.\nThen instead of trying the feature I read the code. I can find stupid mistakes easier that way.\nI also stay informed about the code this way.<\/p>\n<h2 class=\"heading\" id=\"workflow\">\n  Workflow\n  <a class=\"anchor\" href=\"#workflow\">#<\/a>\n<\/h2>\n<p>As I said, I start by giving AI some task that I think it can do by itself.\nThen I can return to my own work and leave it to work until it&rsquo;s finished.<\/p>\n<p>Once it has the complete code I run <code>towelie<\/code>:<\/p>\n<p>\n\n\n\n\n\n\n\n<figure class=\"\">\n\n    \n    \n      \n        \n        <div class=\"img-container\" style=\"--w: 3830; --h: 1742;\">\n            <img loading=\"lazy\" alt=\"Towelie\" src=\"https:\/\/glyphack.com\/dv-7\/tw-1_hu_436c400568032d4a.png\" width=\"3830\" height=\"1742\">\n        <\/div>\n      \n    \n\n    \n<\/figure>\n<\/p>\n<p>Here I can select the changes I want to look for (branch, commit, or even not committed changes) and review them.<\/p>\n<p>If I find something that I don&rsquo;t like I leave a comment:<\/p>\n<p>\n\n\n\n\n\n\n\n<figure class=\"\">\n\n    \n    \n      \n        \n        <div class=\"img-container\" style=\"--w: 3826; --h: 1092;\">\n            <img loading=\"lazy\" alt=\"Towelie\" src=\"https:\/\/glyphack.com\/dv-7\/tw-2_hu_3b815fd80248b97f.png\" width=\"3826\" height=\"1092\">\n        <\/div>\n      \n    \n\n    \n<\/figure>\n<\/p>\n<p>Then I can click finish review and have this copied to my clipboard:<\/p>\n<div class=\"code-block\">\n  <pre tabindex=\"0\"><code>Here&#39;s the review of the user. Go over the comments and resolve them. You can use git commands to get more context about some line changes if you need more information to implement the comment.\n\nOverall notes:\nThis is almost good. I want to make sure the CommitRef datastructure is used through the review controller and we don&#39;t have logic and code duplication.\n\n---\n\nweb\/src\/controllers\/review_controller.ts lines 536-537 on the new code (after the change)\n\n```\n      commit_ref: commitRef,\n      commit_sha: commitSha,\n```\n\n```\nThe commit_ref and commit_sha are not always available. If user selects uncommitted changes there is no sha. So instead just define one variable that is called ref that either has the committed,uncommited, etc. or the commit sha. When it&#39;s commit sha then prefix is with commit sha: ...\n```<\/code><\/pre>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>I copy this to my AI agent, and repeat the review until I&rsquo;m happy.<\/p>\n<h2 class=\"heading\" id=\"how-does-it-work\">\n  How does it work?\n  <a class=\"anchor\" href=\"#how-does-it-work\">#<\/a>\n<\/h2>\n<p>Towelie starts a server that serves the diff review tool.\nIt currently uses <a href=\"https:\/\/www.npmjs.com\/package\/diff2html\" rel=\"noopener\" target=\"_blank\"><code>diff2html<\/code><\/a> library for rendering the diff.\nThere are custom adjustments to <code>diff2html<\/code> to allow selecting lines and commenting on them.\nFor this I&rsquo;m using <a href=\"https:\/\/stimulus.hotwired.dev\/\" rel=\"noopener\" target=\"_blank\">Stimulus<\/a>.\nThis was my first experience using Stimulus, and I like it.\nIt&rsquo;s small and easy to work with. You can learn it in under an hour.<\/p>\n<hr>\n<h2 class=\"heading\" id=\"other-things\">\n  Other Things\n  <a class=\"anchor\" href=\"#other-things\">#<\/a>\n<\/h2>\n<p>I found <a href=\"https:\/\/github.com\/trishume\/dotfiles\/blob\/master\/hammerspoon\/hammerspoon.symlink\/init.lua\" rel=\"noopener\" target=\"_blank\">Tristan Hume&rsquo;s Hammerspoon config<\/a>.\nI borrowed some nice improvements for my own dotfiles.\nMy window manager now knows the last focused window and I can switch to it.\nDid you know that he created <a href=\"https:\/\/www.hammerspoon.org\/docs\/hs.noises.html\" rel=\"noopener\" target=\"_blank\">this module<\/a> that makes your mac respond to sounds?<\/p>\n<p><a href=\"https:\/\/www.youtube.com\/watch?v=b2F-DItXtZs\" rel=\"noopener\" target=\"_blank\">This channel<\/a> is so funny.\nAnd so relevant.<\/p>\n<p>I recently learned about <a href=\"https:\/\/bernsteinbear.com\/blog\/creduce\/\" rel=\"noopener\" target=\"_blank\">c reduce<\/a>.\nI wish I had it when I was debugging crashes in <a href=\"https:\/\/glyphack.com\/ty-self\/\">ty<\/a>. I will definitely use it next time I am debugging crashes.<\/p>\n<p>I am starting to make more scripts in my projects.\nIt&rsquo;s more convenient, and <code>make<\/code> is not convenient for random scripting.\nAfter reading <a href=\"https:\/\/matklad.github.io\/2026\/01\/27\/make-ts.html\" rel=\"noopener\" target=\"_blank\">make.ts<\/a> I&rsquo;m thinking about making the same setup with <code>uv<\/code>.\nRight now I&rsquo;m also using Typescript, but I&rsquo;m more comfortable with Python.<\/p>\n<p>I finally learned why git has conflicts when I am working on branch A from master, and branch B from A, then when A is merged to master and I rebase B, I get conflicts.\nThe solution is <a href=\"https:\/\/stackoverflow.com\/questions\/71019922\/git-rebase-a-branch-on-another-after-parent-branch-is-merged-to-master\" rel=\"noopener\" target=\"_blank\">this<\/a>.<\/p>\n<div class=\"code-block\">\n  <pre tabindex=\"0\"><code>git rebase --onto master branch-a(commit hash or the merge commit)<\/code><\/pre>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>I used AI to improve the design for Towelie frontend.\nI gave the same prompt to ChatGPT, Claude, and V0.\nThe result was that Claude built a much better-looking UI in the first shot and I decided to iterate on that.\nWhat V0 and ChatGPT made was pretty similar, but it looked really basic and I didn&rsquo;t continue with prompting.<\/p>\n<details>\n    <summary>Prompt<\/summary>\nHello, I want you to make a design for a web application. This web application is a local terminal application, but when you open it, when you run it, it will open a web page for you that you can interact with the application. What this does is that it's an offline code review tool where you can launch it and you will see the diff of the repository, git repository that you are at. You can select different branches, you can select the base branch, the other branch you want to check the diff and then you can select commits. You will see the file tree of the changes and you will see the files and you would see the lines and you can comment on the lines. And when it's done, there is a button for a finish review where you click and then you get your review copied to the clipboard. It's useful to use it with coding tools, AI coding agents. So you can review their code and give them back feedback in bulk. Make a design for it. I'm mostly looking for something minimal but also beautiful. Not too much clutter but it also should look nice. Like the color, typography, these kind of things. What are the important parts here?\nStart with checking the requirements and ask me if there are any questions about it.\nDon't want the full app just html and style with tailwind. Run it for me so I can see it.\nGenerate as a single HTML file tailwind. No js for functionality.\n<\/details>\n<p>I made a <a href=\"https:\/\/github.com\/Glyphack\/dotfiles\/blob\/2b56c48ea21fde3a65dbfdf98374882103c20e2a\/raycast\/commands\/md-link.sh#L1\" rel=\"noopener\" target=\"_blank\">new raycast command<\/a> that pastes the current URL in my clipboard as a markdown link.\nIt automatically fetches the title. Titles are so long and full of fluff these days.<\/p>\n<p>I usually use <a href=\"https:\/\/writewithharper.com\/\" rel=\"noopener\" target=\"_blank\">Harper<\/a> when I&rsquo;m writing to catch grammar and spelling mistakes.\nSometimes it&rsquo;s too noisy with the false positives about my writing. So I also tried using more LLMs to check my writing and teach me my mistakes.\nI still think Harper is a good tool (and similar tools), but I feel with LLMs it&rsquo;s nicer that you get a review with the context of what I&rsquo;m writing.<\/p>\n<p>Over the last couple of weeks I found myself having more projects open at the same time.\nI usually have a couple of tabs open in my terminal.\nSo when having multiple projects it becomes a lot of tabs and hard to jump between them.\nI asked Claude to make a workspace switcher for me, but it was not successful initially.\nI found <a href=\"https:\/\/github.com\/MLFlexer\/smart_workspace_switcher.wezterm\/\" rel=\"noopener\" target=\"_blank\">smart_workspace_switcher.wezterm<\/a> and browsed some wezterm docs and guided it myself.\nFinally it was able to make a <a href=\"https:\/\/github.com\/Glyphack\/dotfiles\/blob\/2b56c48ea21fde3a65dbfdf98374882103c20e2a\/wezterm\/wezterm.lua#L228\" rel=\"noopener\" target=\"_blank\">good session manager<\/a>.\nWhenever I hit CMD + Enter I can switch between workspaces, create a new one, or delete one. It has fuzzy searching too.<\/p>\n<p>I started making more functions for my fish shell. Just to make hard things easier.\nFor example, I used to have this problem where I generated some output in the terminal and needed to send it in Slack.\nThe problem was that I had to go to Slack and navigate to the correct folder.\nInstead I made <a href=\"https:\/\/github.com\/Glyphack\/dotfiles\/blob\/master\/fish\/functions\/%2Ccp.fish\" rel=\"noopener\" target=\"_blank\">this script<\/a> that copies the file to my clipboard so I run the command and just go to Slack to paste.<\/p>\n<p>I also finally decided to make my own git wrapper <code>,g<\/code>.\nIt&rsquo;s for frequent stuff that I do:<\/p>\n<ul>\n<li>Update main and create a new branch from it<\/li>\n<li>List branches that I committed to and select one to switch\nAnd more stuff that you can check <a href=\"https:\/\/github.com\/Glyphack\/dotfiles\/blob\/master\/fish\/functions\/%2Cg.fish\" rel=\"noopener\" target=\"_blank\">here<\/a><\/li>\n<\/ul>\n"},{"title":"Reverse Engineering Philips Hue light strip to control from PC","link":"https:\/\/glyphack.com\/huec\/","pubDate":"Wed, 25 Feb 2026 22:25:08 +0100","guid":"https:\/\/glyphack.com\/huec\/","description":"<p>I was looking for a way to control my <a href=\"https:\/\/amzn.to\/4r7oN4M\" rel=\"noopener\" target=\"_blank\">Philips Hue light strip<\/a> without their terrible app<sup id=\"fnref:1\"><a href=\"#fn:1\" class=\"footnote-ref\" role=\"doc-noteref\">1<\/a><\/sup>.\nAll my searches led to this conclusion: you need to buy a <a href=\"https:\/\/amzn.to\/4aN5NDR\" rel=\"noopener\" target=\"_blank\">Hue Bridge<\/a> to control the lamp from a PC.\nBut I don&rsquo;t want to have another device just to do what my PC is capable of doing right now.<\/p>\n<p>I want my light to turn on and off automatically every day without paying for another device. I also want to control it from my desk without grabbing my phone.<\/p>\n<p>I published the end result of this project in <a href=\"https:\/\/github.com\/Glyphack\/hue-control\" rel=\"noopener\" target=\"_blank\">huec<\/a>, a CLI app that lets you control Philips Hue lights.\nHere&rsquo;s a quick demo:<\/p>\n<div style=\"position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden;\">\n\t\t\t<iframe allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share; fullscreen\" loading=\"eager\" referrerpolicy=\"strict-origin-when-cross-origin\" src=\"https:\/\/www.youtube.com\/embed\/isGCe3Zvm54?autoplay=0&amp;controls=1&amp;end=0&amp;loop=0&amp;mute=0&amp;start=0\" style=\"position: absolute; top: 0; left: 0; width: 100%; height: 100%; border:0;\" title=\"YouTube video\"><\/iframe><p><a href=\"https:\/\/www.youtube.com\/embed\/isGCe3Zvm54?autoplay=0&amp;controls=1&amp;end=0&amp;loop=0&amp;mute=0&amp;start=0\">View embedded content<\/a><\/p>\n\t\t<\/div>\n\n<p>Here I discuss the journey of discovering the protocol, explaining how power, brightness, color, and alarms are controlled.<\/p>\n<h2 class=\"heading\" id=\"use-cases\">\n  Use Cases\n  <a class=\"anchor\" href=\"#use-cases\">#<\/a>\n<\/h2>\n<p><strong>Turn lights on and off every day<\/strong><\/p>\n<p>I have two alarms for my light to turn on at 07:00 and turn off at 08:00.\nTo have this repeat every day I run the following command:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-text\" data-lang=\"text\"><span style=\"display:flex;\"><span>huec alarms enable --all<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p><strong>Timer using the lamps<\/strong><\/p>\n<p>I have a 5-minute timer on the light.\nUsing this script I can start this timer.<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-py\" data-lang=\"py\"><span style=\"display:flex;\"><span>result <span style=\"color:#af3a03\">=<\/span> run(<span style=\"color:#79740e\">&#34;uv run huec alarms list --json&#34;<\/span>)\n<\/span><\/span><span style=\"display:flex;\"><span>alarms <span style=\"color:#af3a03\">=<\/span> json<span style=\"color:#af3a03\">.<\/span>loads(result<span style=\"color:#af3a03\">.<\/span>stdout)\n<\/span><\/span><span style=\"display:flex;\"><span>matches <span style=\"color:#af3a03\">=<\/span> [a <span style=\"color:#af3a03\">for<\/span> a <span style=\"color:#af3a03\">in<\/span> alarms <span style=\"color:#af3a03\">if<\/span> a[<span style=\"color:#79740e\">&#34;name&#34;<\/span>] <span style=\"color:#af3a03\">==<\/span> <span style=\"color:#79740e\">&#34;Timer&#34;<\/span>]\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">if<\/span> <span style=\"color:#af3a03\">not<\/span> matches:\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#b57614\">print<\/span>(<span style=\"color:#79740e\">&#34;No alarm named &#39;Timer&#39; found&#34;<\/span>, file<span style=\"color:#af3a03\">=<\/span>sys<span style=\"color:#af3a03\">.<\/span>stderr)\n<\/span><\/span><span style=\"display:flex;\"><span>    sys<span style=\"color:#af3a03\">.<\/span>exit(<span style=\"color:#8f3f71\">1<\/span>)\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>alarm_id <span style=\"color:#af3a03\">=<\/span> matches[<span style=\"color:#8f3f71\">0<\/span>][<span style=\"color:#79740e\">&#34;id&#34;<\/span>]\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#b57614\">print<\/span>(<span style=\"color:#79740e\">f<\/span><span style=\"color:#79740e\">&#34;Enabling alarm ID: <\/span><span style=\"color:#79740e\">{<\/span>alarm_id<span style=\"color:#79740e\">}<\/span><span style=\"color:#79740e\">&#34;<\/span>)\n<\/span><\/span><span style=\"display:flex;\"><span>run(<span style=\"color:#79740e\">f<\/span><span style=\"color:#79740e\">&#34;uv run huec alarms enable --id <\/span><span style=\"color:#79740e\">{<\/span>alarm_id<span style=\"color:#79740e\">}<\/span><span style=\"color:#79740e\">&#34;<\/span>)<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p><strong>Turn on after unlocking my Mac<\/strong><\/p>\n<p>Using <a href=\"https:\/\/www.hammerspoon.org\/\" rel=\"noopener\" target=\"_blank\">Hammerspoon<\/a> I <a href=\"https:\/\/github.com\/Glyphack\/dotfiles\/blob\/2b56c48ea21fde3a65dbfdf98374882103c20e2a\/hammerspoon\/init.lua#L663\" rel=\"noopener\" target=\"_blank\">set<\/a> the lights to turn on when I unlock my Mac:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-lua\" data-lang=\"lua\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">function<\/span> <span style=\"color:#b57614\">ToggleLights<\/span>(eventType)\n<\/span><\/span><span style=\"display:flex;\"><span>\t<span style=\"color:#af3a03\">if<\/span> eventType <span style=\"color:#af3a03\">==<\/span> hs.caffeinate.watcher.screensDidUnlock <span style=\"color:#af3a03\">or<\/span> eventType <span style=\"color:#af3a03\">==<\/span> hs.caffeinate.watcher.systemDidWake <span style=\"color:#af3a03\">then<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>\t\tfishRunCommand(<span style=\"color:#79740e\">&#34;huec power on&#34;<\/span>)\n<\/span><\/span><span style=\"display:flex;\"><span>\t<span style=\"color:#af3a03\">end<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">end<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">local<\/span> ToggleLights <span style=\"color:#af3a03\">=<\/span> hs.caffeinate.watcher.new(toggleLights)\n<\/span><\/span><span style=\"display:flex;\"><span>ToggleLights:start()<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>Without further ado, let\u2019s see what I figured out about controlling the light.<\/p>\n<!-- End of introduction -->\n<p>It all started when I found <a href=\"https:\/\/github.com\/dmtrKovalenko\/blendr\/\" rel=\"noopener\" target=\"_blank\">Blendr<\/a>.\nBlendr connects to Bluetooth Low Energy (BLE) devices and lets you browse their services and characteristics.<\/p>\n<p>Characteristics are a place where the light stores some data.\nYou can get data from a characteristic or write to it.\nA service is simply a group of characteristics.\nFor example a service could be for changing color and brightness.<\/p>\n<p>Here&rsquo;s how the output looked for my lamp:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-text\" data-lang=\"text\"><span style=\"display:flex;\"><span>Service Device Information (0x1800)\n<\/span><\/span><span style=\"display:flex;\"><span>  Manufacturer Name String (0x2A29) [Read]\n<\/span><\/span><span style=\"display:flex;\"><span>  Model Number String (0x2A24) [Read]\n<\/span><\/span><span style=\"display:flex;\"><span>  Software Revision String (0x2A28) [Read]\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>Service 932c32bd-0001-47a2-835a-a8d455b859dd\n<\/span><\/span><span style=\"display:flex;\"><span>  932c32bd-0001-47a2-835a-a8d455b859dd [Read]\n<\/span><\/span><span style=\"display:flex;\"><span>  932c32bd-0002-47a2-835a-a8d455b859dd [Read, Write, Notify]\n<\/span><\/span><span style=\"display:flex;\"><span>  932c32bd-0003-47a2-835a-a8d455b859dd [Read, Write, Notify]\n<\/span><\/span><span style=\"display:flex;\"><span>  932c32bd-0004-47a2-835a-a8d455b859dd [Read, Write, Notify]\n<\/span><\/span><span style=\"display:flex;\"><span>  932c32bd-0005-47a2-835a-a8d455b859dd [Read, Write, Notify]\n<\/span><\/span><span style=\"display:flex;\"><span>  932c32bd-0006-47a2-835a-a8d455b859dd [Write]\n<\/span><\/span><span style=\"display:flex;\"><span>  932c32bd-0007-47a2-835a-a8d455b859dd [Read, Write, Notify]\n<\/span><\/span><span style=\"display:flex;\"><span>  932c32bd-1005-47a2-835a-a8d455b859dd [Read, Write]\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>Service 97fe6561-0001-4f62-86e9-b71ee2da3d22\n<\/span><\/span><span style=\"display:flex;\"><span>  97fe6561-0001-4f62-86e9-b71ee2da3d22 [Read]\n<\/span><\/span><span style=\"display:flex;\"><span>  97fe6561-0003-4f62-86e9-b71ee2da3d22 [Read, Write]\n<\/span><\/span><span style=\"display:flex;\"><span>  97fe6561-0004-4f62-86e9-b71ee2da3d22 [Write]\n<\/span><\/span><span style=\"display:flex;\"><span>  97fe6561-0005-4f62-86e9-b71ee2da3d22 [Write]\n<\/span><\/span><span style=\"display:flex;\"><span>  97fe6561-0006-4f62-86e9-b71ee2da3d22 [Read, Write]\n<\/span><\/span><span style=\"display:flex;\"><span>  97fe6561-0008-4f62-86e9-b71ee2da3d22 [Write, Notify]\n<\/span><\/span><span style=\"display:flex;\"><span>  97fe6561-1001-4f62-86e9-b71ee2da3d22 [Read, Write, Notify]\n<\/span><\/span><span style=\"display:flex;\"><span>  97fe6561-2001-4f62-86e9-b71ee2da3d22 [Read, Write]\n<\/span><\/span><span style=\"display:flex;\"><span>  97fe6561-2002-4f62-86e9-b71ee2da3d22 [Write]\n<\/span><\/span><span style=\"display:flex;\"><span>  97fe6561-2004-4f62-86e9-b71ee2da3d22 [Write]\n<\/span><\/span><span style=\"display:flex;\"><span>  97fe6561-a001-4f62-86e9-b71ee2da3d22 [Write]\n<\/span><\/span><span style=\"display:flex;\"><span>  97fe6561-a002-4f62-86e9-b71ee2da3d22 [Read]\n<\/span><\/span><span style=\"display:flex;\"><span>  97fe6561-a003-4f62-86e9-b71ee2da3d22 [Read, Write]\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>Service 9da2ddf1-0001-44d0-909c-3f3d3cb34a7b\n<\/span><\/span><span style=\"display:flex;\"><span>  9da2ddf1-0001-44d0-909c-3f3d3cb34a7b [Write, Notify]\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>Service b8843add-0001-4aa1-8794-c3f462030bda\n<\/span><\/span><span style=\"display:flex;\"><span>  b8843add-0001-4aa1-8794-c3f462030bda [Read]\n<\/span><\/span><span style=\"display:flex;\"><span>  b8843add-0002-4aa1-8794-c3f462030bda [Write, Notify]\n<\/span><\/span><span style=\"display:flex;\"><span>  b8843add-0003-4aa1-8794-c3f462030bda [Write]\n<\/span><\/span><span style=\"display:flex;\"><span>  b8843add-0004-4aa1-8794-c3f462030bda [Read]<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>Some characteristics have &ldquo;Read&rdquo; in front of them.\nThis means you can read their values using Blendr.<\/p>\n<p>Now the question is, what does each characteristic do?\nThere are two ways to find this out:<\/p>\n<ol>\n<li>Randomly write data into different characteristics to see if the lamp reacts. For example we can write <code>0x00<\/code> into all characteristics and see when the lamp turns off. This requires guessing what value turns the light on and off and what value changes the color.<\/li>\n<li>Use the app to change properties of the lamp and then read values using Blendr.<\/li>\n<\/ol>\n<p>I turned the lamp off using Philips Hue app and checked what characteristic has <code>0x00<\/code> in it.\nIt was the <code>932c32bd-0002-47a2-835a-a8d455b859dd<\/code>, and that was the characteristic that controls power.<\/p>\n<p>To send and receive data from the lamp there is <a href=\"https:\/\/bleak.readthedocs.io\/en\/latest\/\" rel=\"noopener\" target=\"_blank\">Bleak<\/a>.<\/p>\n<p>By knowing the name of the lamp(you can get it from Blendr) you can connect to the lamp using:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-py\" data-lang=\"py\"><span style=\"display:flex;\"><span>POWER_UUID <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#79740e\">&#39;932c32bd-0002-47a2-835a-a8d455b859dd&#39;<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">async<\/span> <span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">connect_to_light<\/span>(name: <span style=\"color:#b57614\">str<\/span>, timeout: <span style=\"color:#b57614\">float<\/span> <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#8f3f71\">10.0<\/span>) <span style=\"color:#af3a03\">-&gt;<\/span> BleakClient:\n<\/span><\/span><span style=\"display:flex;\"><span>    device <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#af3a03\">await<\/span> BleakScanner<span style=\"color:#af3a03\">.<\/span>find_device_by_name(name, timeout<span style=\"color:#af3a03\">=<\/span>timeout)\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">if<\/span> <span style=\"color:#af3a03\">not<\/span> device:\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">raise<\/span> <span style=\"color:#fb4934\">SystemExit<\/span>(<span style=\"color:#79740e\">f<\/span><span style=\"color:#79740e\">&#34;Device &#39;<\/span><span style=\"color:#79740e\">{<\/span>name<span style=\"color:#79740e\">}<\/span><span style=\"color:#79740e\">&#39; not found.&#34;<\/span>)\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    client <span style=\"color:#af3a03\">=<\/span> BleakClient(device, timeout<span style=\"color:#af3a03\">=<\/span>timeout)\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">await<\/span> client<span style=\"color:#af3a03\">.<\/span>connect(timeout<span style=\"color:#af3a03\">=<\/span>timeout)\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">return<\/span> client\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">await<\/span> client<span style=\"color:#af3a03\">.<\/span>write_gatt_char(POWER_UUID, <span style=\"color:#79740e\">b<\/span><span style=\"color:#79740e\">&#34;<\/span><span style=\"color:#79740e\">\\x01<\/span><span style=\"color:#79740e\">&#34;<\/span>)  <span style=\"color:#928374;font-style:italic\"># turn on<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">await<\/span> client<span style=\"color:#af3a03\">.<\/span>write_gatt_char(POWER_UUID, <span style=\"color:#79740e\">b<\/span><span style=\"color:#79740e\">&#34;<\/span><span style=\"color:#79740e\">\\x00<\/span><span style=\"color:#79740e\">&#34;<\/span>)  <span style=\"color:#928374;font-style:italic\"># turn off<\/span><\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>The <code>client<\/code> can then be used to read and write values.<\/p>\n<h2 class=\"heading\" id=\"color\">\n  Color\n  <a class=\"anchor\" href=\"#color\">#<\/a>\n<\/h2>\n<p>There are multiple characteristics that update when you change the color the light color:<\/p>\n<ul>\n<li><code>932c32bd-0003-47a2-835a-a8d455b859dd<\/code> changes with brightness<\/li>\n<li><code>932c32bd-0005-47a2-835a-a8d455b859dd<\/code> changes with color (if I do warm white then cool white stays the same)<\/li>\n<li><code>932c32bd-0007-47a2-835a-a8d455b859dd<\/code> changes with everything.<\/li>\n<\/ul>\n<p>You can find out the pattern by changing the light color and observing the characteristic values.<\/p>\n<p>Cool white:<\/p>\n01 01 01 02 01 FE 03 02 9C 00\n\n<p>Warm white:<\/p>\n01 01 01 02 01 FE 03 02 5A 01\n\n<p>The temperature values are in <a href=\"https:\/\/en.wikipedia.org\/wiki\/Mired\" rel=\"noopener\" target=\"_blank\">mireds<\/a>.\nWarm white and cool white only differ in bytes 8-9.<\/p>\n<p>When I set it to another color the bytes change to this format:<\/p>\n<p>For example, here&rsquo;s the packet for red<\/p>\n01 01 01 02 01 fe 04 04 c5 af 51 4e\n\n<p>The color is encoded in <a href=\"https:\/\/en.wikipedia.org\/wiki\/CIE_1931_color_space\" rel=\"noopener\" target=\"_blank\">CIE xy<\/a> format.<\/p>\n<p>Philips Hue <a href=\"https:\/\/developers.meethue.com\/develop\/application-design-guidance\/color-conversion-formulas-rgb-to-xy-and-back\/#xy-to-rgb-color\" rel=\"noopener\" target=\"_blank\">developer docs<\/a> require login! So I asked Claude to figure out what this format is and how to convert from RGB.<\/p>\n<ol>\n<li>Convert the 8-bit number from R\/G\/B into a number between 0 and 1<\/li>\n<li>Linearize the numbers based on this formula <code>if g &gt; 0.04045 then g \/ 12.92 else ((g + 0.055) \/ 1.055) ^ 2.4<\/code><\/li>\n<li>Apply D65 <a href=\"https:\/\/en.wikipedia.org\/wiki\/Adobe_RGB_color_space#Reference_viewing_conditions\" rel=\"noopener\" target=\"_blank\">matrix<\/a> transformation, one full matrix example is <a href=\"https:\/\/www.image-engineering.de\/library\/technotes\/958-how-to-convert-between-srgb-and-ciexyz\" rel=\"noopener\" target=\"_blank\">here<\/a>.<\/li>\n<\/ol>\n<p>You can play around with it in the box below:<\/p>\nNot available in RSS.\n\n<p>When you run the app in interactive mode with <code>huec interactive<\/code>, it will open up a browser page and run a server.\nThe browser displays a color picker and calculates the payload for the color based on the explanations above.\nThe server accepts the payload and sends it to the light using Bleak.<\/p>\n<p>The <code>set_color<\/code> function below sends the packet to the lamp:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-py\" data-lang=\"py\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">async<\/span> <span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">set_color<\/span>(<span style=\"color:#b57614\">self<\/span>, data: <span style=\"color:#b57614\">bytes<\/span>) <span style=\"color:#af3a03\">-&gt;<\/span> <span style=\"color:#af3a03\">None<\/span>:\n<\/span><\/span><span style=\"display:flex;\"><span>    COLOR_UUID <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#79740e\">&#34;932c32bd-0007-47a2-835a-a8d455b859dd&#34;<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">await<\/span> <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>client<span style=\"color:#af3a03\">.<\/span>write_gatt_char(COLOR_UUID, data, response<span style=\"color:#af3a03\">=<\/span><span style=\"color:#af3a03\">True<\/span>)<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<h2 class=\"heading\" id=\"alarms\">\n  Alarms\n  <a class=\"anchor\" href=\"#alarms\">#<\/a>\n<\/h2>\n<p>Alarms in the Philips app are a functionality to turn on\/off the light at a specific time or create a countdown to flash the lights.\nOnce an alarm fires, it deactivates and must be manually re-enabled to go off again the next day.<\/p>\n<p>Similar to how I discovered how colors work I tried to look into what characteristics change when I create an alarm.\nBut I didn&rsquo;t see anything changing.<\/p>\n<p>I needed to see what my phone was doing to create alarms.<\/p>\n<p>For capturing Bluetooth packets there are tools like Wireshark.\nThese tools allow you to see what data software running on the system is sending and where it&rsquo;s going.\nI was using macOS + iOS. For this combination there is:<\/p>\n<ul>\n<li><a href=\"https:\/\/developer.apple.com\/bluetooth\/\" rel=\"noopener\" target=\"_blank\">Bluetooth Packet Logger<\/a><\/li>\n<li><a href=\"https:\/\/developer.apple.com\/services-account\/download?path=\/iOS\/iOS_Logs\/iOSBluetoothLogging.mobileconfig\" rel=\"noopener\" target=\"_blank\">Bluetooth logging config for iOS<\/a><sup id=\"fnref:2\"><a href=\"#fn:2\" class=\"footnote-ref\" role=\"doc-noteref\">2<\/a><\/sup><\/li>\n<\/ul>\n<p>Install Packet Logger on your computer and the profile on your iPhone.\nThen, connect the phone to the computer.\nStart using the Philips Hue app, and you will see the packets being sent or received.<\/p>\n<p>After setting up the tools I checked what was happening when the app connects to the light.\nThe logs looked like this:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-text\" data-lang=\"text\"><span style=\"display:flex;\"><span>0x005A  Hue lightstrip pl  Write Request - Handle:0x0068 - Value: 0311 00  \n<\/span><\/span><span style=\"display:flex;\"><span>\tWrite Request - Handle:0x0068 - Value: 0311 00\n<\/span><\/span><span style=\"display:flex;\"><span>\tOpcode: 0x12\n<\/span><\/span><span style=\"display:flex;\"><span>\tAttribute Handle: 0x0068 (104)\n<\/span><\/span><span style=\"display:flex;\"><span>\tValue: 0311 00\n<\/span><\/span><span style=\"display:flex;\"><span>0x005A  Hue lightstrip pl  Channel ID: 0x0004  Length: 0x0006 (06) [ 12 68 00 03 11 00 ]  \n<\/span><\/span><span style=\"display:flex;\"><span>\tChannel ID: 0x0004  Length: 0x0006 (06) [ 12 68 00 03 11 00 ]\n<\/span><\/span><span style=\"display:flex;\"><span>\tL2CAP Payload:\n<\/span><\/span><span style=\"display:flex;\"><span>\t00000000: 1268 0003 1100                           .h....\n<\/span><\/span><span style=\"display:flex;\"><span>0x005A  Hue lightstrip pl  Data [Handle: 0x005A, Packet Boundary Flags: 0x0, Length: 0x000A (10)]  \n<\/span><\/span><span style=\"display:flex;\"><span>0x005A  Hue lightstrip pl  Write Response  \n<\/span><\/span><span style=\"display:flex;\"><span>\tWrite Response\n<\/span><\/span><span style=\"display:flex;\"><span>\tOpcode: 0x13\n<\/span><\/span><span style=\"display:flex;\"><span>0x005A  Hue lightstrip pl  Channel ID: 0x0004  Length: 0x0001 (01) [ 13 ]  \n<\/span><\/span><span style=\"display:flex;\"><span>\tChannel ID: 0x0004  Length: 0x0001 (01) [ 13 ]\n<\/span><\/span><span style=\"display:flex;\"><span>\tL2CAP Payload:\n<\/span><\/span><span style=\"display:flex;\"><span>\t00000000: 13                                       .\n<\/span><\/span><span style=\"display:flex;\"><span>0x005A  Hue lightstrip pl  Data [Handle: 0x005A, Packet Boundary Flags: 0x2, Length: 0x0005 (5)]  \n<\/span><\/span><span style=\"display:flex;\"><span>0x005A  Hue lightstrip pl  Handle Value Notification - Handle:0x0068 - Value: 0300 1100  \n<\/span><\/span><span style=\"display:flex;\"><span>\tHandle Value Notification - Handle:0x0068 - Value: 0300 1100\n<\/span><\/span><span style=\"display:flex;\"><span>\tOpcode: 0x1B\n<\/span><\/span><span style=\"display:flex;\"><span>\tAttribute Handle: 0x0068 (104)\n<\/span><\/span><span style=\"display:flex;\"><span>0x005A  Hue lightstrip pl  Channel ID: 0x0004  Length: 0x0007 (07) [ 1B 68 00 03 00 11 00 ]  \n<\/span><\/span><span style=\"display:flex;\"><span>\tChannel ID: 0x0004  Length: 0x0007 (07) [ 1B 68 00 03 00 11 00 ]\n<\/span><\/span><span style=\"display:flex;\"><span>\tL2CAP Payload:\n<\/span><\/span><span style=\"display:flex;\"><span>\t00000000: 1B68 0003 0011 00                        .h.....\n<\/span><\/span><span style=\"display:flex;\"><span>0x005A  Hue lightstrip pl  Data [Handle: 0x005A, Packet Boundary Flags: 0x2, Length: 0x000B (11)]  \n<\/span><\/span><span style=\"display:flex;\"><span>0x005A  Hue lightstrip pl  Handle Value Notification - Handle:0x0068 - Value: 0411 00FF FF  \n<\/span><\/span><span style=\"display:flex;\"><span>\tHandle Value Notification - Handle:0x0068 - Value: 0411 00FF FF\n<\/span><\/span><span style=\"display:flex;\"><span>\tOpcode: 0x1B\n<\/span><\/span><span style=\"display:flex;\"><span>\tAttribute Handle: 0x0068 (104)\n<\/span><\/span><span style=\"display:flex;\"><span>0x005A  Hue lightstrip pl  Channel ID: 0x0004  Length: 0x0008 (08) [ 1B 68 00 04 11 00 FF FF ]  \n<\/span><\/span><span style=\"display:flex;\"><span>\tChannel ID: 0x0004  Length: 0x0008 (08) [ 1B 68 00 04 11 00 FF FF ]\n<\/span><\/span><span style=\"display:flex;\"><span>\tL2CAP Payload:\n<\/span><\/span><span style=\"display:flex;\"><span>\t00000000: 1B68 0004 1100 FFFF                      .h......\n<\/span><\/span><span style=\"display:flex;\"><span>0x005A  Hue lightstrip pl  Data [Handle: 0x005A, Packet Boundary Flags: 0x2, Length: 0x000C (12)]  \n<\/span><\/span><span style=\"display:flex;\"><span>\tPacket Boundary Flags: [10] 0x02 - First Flushable Packet Of Higher Layer Message (Start Of An L2CAP Packet)\n<\/span><\/span><span style=\"display:flex;\"><span>\tBroadcast Flags: [00] 0x00 - Point-to-point\n<\/span><\/span><span style=\"display:flex;\"><span>\tData (0x000C Bytes)\n<\/span><\/span><span style=\"display:flex;\"><span>0x0000  00:00:00:00:00:00  00000000: 5A20 0C00 0800 0400 1B68 0004 1100 FFFF  Z .......h......  \n<\/span><\/span><span style=\"display:flex;\"><span>0x005A  Hue lightstrip pl  Number Of Completed Packets - Handle: 0x005A - Packets: 0x0001    \n<\/span><\/span><span style=\"display:flex;\"><span>\tParameter Length: 5 (0x05)\n<\/span><\/span><span style=\"display:flex;\"><span>\tNumber Of Handles: 0x01\n<\/span><\/span><span style=\"display:flex;\"><span>\tConnection Handle: 0x005A\n<\/span><\/span><span style=\"display:flex;\"><span>\tNumber Of Packets: 0x0001<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>I asked Claude to figure out what the light was doing and gave it the context about what I was looking for.\nIt figured out that the app performs this process:<\/p>\n<ol>\n<li>Write <code>00<\/code> to a characteristic.<\/li>\n<li>The characteristic replies with current alarm IDs.<\/li>\n<li>The app writes each alarm ID to the characteristic again and receives more information about that alarm.<\/li>\n<\/ol>\n<p>So I learned that characteristics can also reply.\nThis happens through subscriptions.\nFrom the first list of characteristics you can see some have read and write properties.\nSome characteristics have write and notify properties.\nYou can write to these characteristics and receive a response.<\/p>\n<p>Here&rsquo;s the code to do this:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-py\" data-lang=\"py\"><span style=\"display:flex;\"><span>ALARM_ID <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#79740e\">&#34;9da2ddf1-0001-44d0-909c-3f3d3cb34a7b&#34;<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>notifications <span style=\"color:#af3a03\">=<\/span> asyncio<span style=\"color:#af3a03\">.<\/span>Queue()\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">on_alarm_notification<\/span>(sender, data: <span style=\"color:#b57614\">bytearray<\/span>):\n<\/span><\/span><span style=\"display:flex;\"><span>    notifications<span style=\"color:#af3a03\">.<\/span>put_nowait(data)\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">await<\/span> client<span style=\"color:#af3a03\">.<\/span>start_notify(ALARM_ID, on_alarm_notification)\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">await<\/span> client<span style=\"color:#af3a03\">.<\/span>write_gatt_char(ALARM_ID, <span style=\"color:#b57614\">bytes<\/span>([<span style=\"color:#8f3f71\">0x00<\/span>]))\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>response <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#af3a03\">await<\/span> asyncio<span style=\"color:#af3a03\">.<\/span>wait_for(notifications<span style=\"color:#af3a03\">.<\/span>get(), timeout<span style=\"color:#af3a03\">=<\/span><span style=\"color:#8f3f71\">5.0<\/span>)<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>You can see that in the Packet Logger logs there is only a handle. There is no characteristic ID.\nI tried a simple approach: I subscribed to all characteristics and then wrote <code>00<\/code> payload to all and checked which one replied.\nThat&rsquo;s how I got the characteristic.<\/p>\n<p>The alarm characteristic (<code>9da2ddf1-0001-44d0-909c-3f3d3cb34a7b<\/code>) is like a server.\nYou can write different messages to it and subscribe to it to get back responses as notifications.<\/p>\n<p>When the app connects it writes this to the characteristic:<\/p>\n00\n\n<p>Then the characteristic responds back with the list of alarm IDs:<\/p>\n00 00 07 02 2C 00 2D 00\n\n<p>To read the alarm details using its ID, we construct this message:<\/p>\n02 2C 00 00 00\n\n<p>This gives us the full alarm details:<\/p>\n02 00 2C 00 35 00 00 00 00 01 00 60 55 95 69 00 09 01 01 01 06 01 09 08 01 7D 22 01 D4 0C 13 8D 81 B9 4A 4C AA 42 B9 9A CE C6 2D 88 00 FF FF FF FF 0A 4D 6F 72 6E 69 6E 67 20 75 70 01\n\n<p>That&rsquo;s it. You can read and parse the alarm.<\/p>\n<p>Can we create any alarm we like now?\nWhen I took this same message and just tried to create an alarm by substituting my timestamp and name the lamp was not creating the alarm.<\/p>\n<p>So I checked what the app sends to the lamp to create an alarm:<\/p>\n01 FF FF 00 01 00 D8 B6 8E 69 00 09 01 01 01 06 01 09 08 01 5B 19 01 94 D1 84 84 B7 51 43 DA A8 67 A9 2F 02 11 0C 8D 00 FF FF FF FF 01 41 01\n\n<p>After the alarm write succeeds there will be these two notifications which reply with the ID of the alarm:<\/p>\n01 00 FF FF 1E 00\n\n04 FF FF 1E 00\n\n<p>Then I tried sending the same payload to the lamp to create an alarm.\nBut the alarm never actually got created.\nSo then I checked what happens when I create the same alarm twice via the app.<\/p>\n<p>First alarm creation payload:<\/p>\n01 FF FF 00 01 00 30 5C 91 69 00 09 01 01 01 06 01 09 08 01 65 1F 01 FB D0 61 C2 5B 63 40 F6 AA 71 BB 49 E1 86 F0 C9 00 FF FF FF FF 07 57 61 6B 65 20 75 70 01\n\n<p>Second alarm creation, same configuration:<\/p>\n01 FF FF 00 01 00 30 5C 91 69 00 09 01 01 01 06 01 09 08 01 65 1F 01 85 97 FE 88 1C C1 46 47 A1 9D 9F 6A 8C 2C 29 7B 00 FF FF FF FF 07 57 61 6B 65 20 75 70 01\n\n<p>As you see the mystery bytes change.\nThere&rsquo;s no change in the alarm configuration.\nThis suggests that the app generates these bytes as a checksum.\nThe lamp checks the checksum to verify if the alarm is valid or not.\nThis means if I just repeat this with any configuration I want, it&rsquo;s not going to work.<\/p>\n<p>After spending some time on it<sup id=\"fnref:3\"><a href=\"#fn:3\" class=\"footnote-ref\" role=\"doc-noteref\">3<\/a><\/sup> I decided to come up with another solution to control alarms.\nMy goal was to have an alarm that repeats every day.\nWhat if I could just change the active byte of the alarm and it would be enabled every day?<\/p>\n<p>Then I created an alarm and turned it off and on again in the app.\nI already knew how to read alarm information.\nI was able to read the alarm info and see what had changed.<\/p>\n<p>Create a new alarm called &ldquo;Test&rdquo;:<\/p>\n01 FF FF 00 01 00 40 89 90 69 00 09 01 01 01 06 01 09 08 01 65 1C 01 EF 55 72 FE F8 17 4B 67 AC ED 26 72 1F CF AA 24 00 FF FF FF FF 04 54 65 73 74 01\n\n<p>Response confirming the alarm was created with ID 1:<\/p>\n01 00 FF FF 01 00\n\n04 FF FF 01 00\n\n<p>I turned off the alarm via the app, then I read the alarm details. Alarm active byte is 0:<\/p>\n02 00 06 00 2F 00 00 00 00 00 00 C0 DA 91 69 00 09 01 01 01 06 01 09 08 01 65 1C 01 EF 55 72 FE F8 17 4B 67 AC ED 26 72 1F CF AA 24 00 FF FF FF FF 04 54 65 73 74 01\n\n<p>Then I turned it on again for tomorrow via the app. This is the edit request that sets the active byte to 1:<\/p>\n01 02 00 00 01 00 C0 DA 91 69 00 09 01 01 01 06 01 09 08 01 65 1C 01 EF 55 72 FE F8 17 4B 67 AC ED 26 72 1F CF AA 24 00 FF FF FF FF 04 54 65 73 74 01\n\n<p>Responses confirming the edit:<\/p>\n01 00 02 00 03 00\n\n04 02 00 03 00\n\n<p>Reading the alarm again after re-enabling byte 9 is now <code>01<\/code> (active):<\/p>\n02 00 07 00 2F 00 00 00 00 01 00 C0 DA 91 69 00 09 01 01 01 06 01 09 08 01 65 1C 01 EF 55 72 FE F8 17 4B 67 AC ED 26 72 1F CF AA 24 00 FF FF FF FF 04 54 65 73 74 01\n\n<p>So in order to turn on the alarm for the next day I have to do two things:<\/p>\n<ul>\n<li>Change the active byte to <code>01<\/code>.<\/li>\n<li>Update the timestamp to be the next day. The alarm timestamp contains date and time. Time stamp is in UTC.<\/li>\n<\/ul>\n<h3 class=\"heading\" id=\"timers\">\n  Timers\n  <a class=\"anchor\" href=\"#timers\">#<\/a>\n<\/h3>\n<p>The Philips app also has a feature called timer.\nYou can start a timer and after it reaches 0 the light starts flashing.<\/p>\n02 00 38 00 28 00 00 00 00 00 01 b9 ba 94 69 01 01 02 1d 01 50 c1 53 49 69 60 40 b1 b3 38 46 6b c3 bb 42 58 03 2c 01 00 00 05 54 69 6d 65 72 01\n\n<p>Timers can be turned on and off in the same way.\nSo the code that turns alarms on and off works on timers too.<\/p>\n<h3 class=\"heading\" id=\"deleting-alarms\">\n  Deleting Alarms\n  <a class=\"anchor\" href=\"#deleting-alarms\">#<\/a>\n<\/h3>\n<p>Deleting alarms happen using the same characteristic.<\/p>\n<p>By setting first byte to <code>03<\/code> we can make a delete alarm request. For example:<\/p>\n<p>Delete alarm with ID 30:<\/p>\n03 1E 00\n\n<p>Response confirming the deletion:<\/p>\n03 00 1E 00\n\n04 1E 00 FF FF\n\n<hr>\n<div class=\"footnotes\" role=\"doc-endnotes\">\n<hr>\n<ol>\n<li id=\"fn:1\">\n<p>They have one app per device. Each app is slow and unresponsive. They don&rsquo;t have good features. When I open the light app I need to wait a few seconds before it loads. If you want the lamp to turn on on a routine you need to pay extra. It&rsquo;s a mess. I don&rsquo;t want another Philips device in my home.&#160;<a href=\"#fnref:1\" class=\"footnote-backref\" role=\"doc-backlink\">&#x21a9;&#xfe0e;<\/a><\/p>\n<\/li>\n<li id=\"fn:2\">\n<p>You need an Apple account to download it. I hate this because when I was in Iran many of these tools were blocked because you can&rsquo;t easily create an account.&#160;<a href=\"#fnref:2\" class=\"footnote-backref\" role=\"doc-backlink\">&#x21a9;&#xfe0e;<\/a><\/p>\n<\/li>\n<li id=\"fn:3\">\n<p>I created more alarms with different configurations trying to figure out the mystery bytes but I did not find a pattern. Let me know if you do!<\/p>\n<details>\n<summary>alarm creation packets<\/summary>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-text\" data-lang=\"text\"><span style=\"display:flex;\"><span>Wake up 07:00 sunrise fade in 30 min\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>01FF FF00 0100 D8B6 8E69 0009 0101 0106 0109 0801 5B19 0194 D184 84B7 5143 DAA8 67A9 2F02 110C 8D00 FFFF FFFF 0141 01\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>--\n<\/span><\/span><span style=\"display:flex;\"><span>Wake up 06:50 sunrise fade in 20 min\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>01FF FF00 0100 D8B6 8E69 0009 0101 0106 0109 0801 6519 01CA 492E A08E 6A48 6883 4FC0 1C5B 8E8F 4700 FFFF FFFF 0141 01\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>--\n<\/span><\/span><span style=\"display:flex;\"><span>Wake up 06:50 sunrise fade in 10 min\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>01FF FF00 0100 30B9 8E69 0009 0101 0106 0109 0801 7D19 01FA 3FD8 C1E2 304D 1E81 948B AE5E C246 3000 FFFF FFFF 0141 01\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>--\n<\/span><\/span><span style=\"display:flex;\"><span>Wake up 07:00 full brightness fade in 30 min\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>010C 0000 0100 D8B6 8E69 000E 0101 0102 01FE 0302 BF01 0502 5046 1901 2114 F58F E794 40F1 86C4 BF6A 8529 73C4 00FF FFFF FF01 4101\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>--\n<\/span><\/span><span style=\"display:flex;\"><span>Wake up 06:50 full brightness fade in 20 min\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>01FF FF00 0100 D8B6 8E69 000E 0101 0102 01FE 0302 BF01 0502 E02E 1901 BE74 4F8A 71FA 464D 8C10 910D 7983 676C 00FF FFFF FF01 4101\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>--\n<\/span><\/span><span style=\"display:flex;\"><span>Wake up 06:50 full brightness fade in 10 min\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>01FF FF00 0100 30B9 8E69 000E 0101 0102 01FE 0302 BF01 0502 7017 1901 AA85 9C87 96F4 4A08 A30D 26A7 E9E0 629B 00FF FFFF FF01 4101\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>--\n<\/span><\/span><span style=\"display:flex;\"><span>Wake up 06:50 sunrise fade in 30 min\n<\/span><\/span><span style=\"display:flex;\"><span>01FF FF00 0100 80B4 8E69 0009 0101 0106 0109 0801 5B19 012D A26C 130F A94B F882 E9C3 215C 27A4 8700 FFFF FFFF 0141 01\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>--\n<\/span><\/span><span style=\"display:flex;\"><span>Wake up 06:50 full brightness fade in 30 min\n<\/span><\/span><span style=\"display:flex;\"><span>01FF FF00 0100 80B4 8E69 000E 0101 0102 01FE 0302 BF01 0502 5046 1901 1478 FFD5 B1F1 431E AB18 E212 A720 34D6 00FF FFFF FF01 4101<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<\/details>\n&#160;<a href=\"#fnref:3\" class=\"footnote-backref\" role=\"doc-backlink\">&#x21a9;&#xfe0e;<\/a><\/li>\n<\/ol>\n<\/div>\n"},{"title":"Talk: type checking Python in Rust","link":"https:\/\/glyphack.com\/f-26\/","pubDate":"Mon, 02 Feb 2026 10:38:44 +0100","guid":"https:\/\/glyphack.com\/f-26\/","description":"<p>ty is one of the great examples of a challenging project that is simple to contribute to.\nDuring last year I contributed to it and I learned a lot.\nAbout type checking, structuring a complex project and taming complexity.<\/p>\n<div style=\"position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden;\">\n\t\t\t<iframe allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share; fullscreen\" loading=\"eager\" referrerpolicy=\"strict-origin-when-cross-origin\" src=\"https:\/\/www.youtube.com\/embed\/Lz-by27piIY?autoplay=0&amp;controls=1&amp;end=0&amp;loop=0&amp;mute=0&amp;start=0\" style=\"position: absolute; top: 0; left: 0; width: 100%; height: 100%; border:0;\" title=\"YouTube video\"><\/iframe><p><a href=\"https:\/\/www.youtube.com\/embed\/Lz-by27piIY?autoplay=0&amp;controls=1&amp;end=0&amp;loop=0&amp;mute=0&amp;start=0\">View embedded content<\/a><\/p>\n\t\t<\/div>\n\n\n![](fosdem3.jpg)\n![](fosdem2.jpg)\n![](fosdem1.jpg)\n\n\n<h2 class=\"heading\" id=\"slides\">\n  Slides\n  <a class=\"anchor\" href=\"#slides\">#<\/a>\n<\/h2>\n<p><a href=\"https:\/\/glyphack.com\/f-26\/ty-f26.pdf\">ty: Adventures of type-checking Python in Rust<\/a><\/p>\n<h2 class=\"heading\" id=\"links\">\n  Links\n  <a class=\"anchor\" href=\"#links\">#<\/a>\n<\/h2>\n<ul>\n<li><a href=\"https:\/\/github.com\/astral-sh\/ruff\" rel=\"noopener\" target=\"_blank\">ty codebase<\/a> ty code lives in the Ruff repository<\/li>\n<li><a href=\"https:\/\/github.com\/astral-sh\/ty\" rel=\"noopener\" target=\"_blank\">ty repo<\/a><\/li>\n<li><a href=\"https:\/\/play.ty.dev\" rel=\"noopener\" target=\"_blank\">ty Playground<\/a><\/li>\n<\/ul>\n"},{"title":"NMRD","link":"https:\/\/glyphack.com\/nmrd\/","pubDate":"Thu, 22 Jan 2026 10:07:53 +0100","guid":"https:\/\/glyphack.com\/nmrd\/","description":"<p><em>Disclaimer: This is an imaginary conversation happening in a world where AI is not as capable as good programmer.\nOf course it&rsquo;s wrong and incorrect in the current world we live in.\nSo don&rsquo;t take it seriously.\nThe content is copied from this <a href=\"https:\/\/www.youtube.com\/watch?v=4OstpOap9KU\" rel=\"noopener\" target=\"_blank\">timeless<\/a> video.<\/em><\/p>\n<hr>\n<p>T: I see you&rsquo;re working on a new look.\nMy guess is that the I \u2764\ufe0f Java t-shirt is meant to be ironic.<\/p>\n<p>P: Doctor, I&rsquo;m recreating myself as a swashbuckling hacker savant with a dark side who is actually a nice guy underneath once you crack his world-be-damned nonchalance.<\/p>\n<p>T: Have you adopted a programming language to match your new persona?<\/p>\n<p>P: I haven&rsquo;t decided, Rust is the front runner.<\/p>\n<p>T: Why not Clojure?<\/p>\n<p>P: Oh please.<\/p>\n<p>T: Let&rsquo;s talk about another work experience that affected you.<\/p>\n<p>P: Well, like most software companies we went through an AI phase.\nWhen our new CTO came on board he immediately hired a team of experts to automate our development process with AI.\nJust because his buddy told him AI writes 90% of the code in other companies.\nHe didn&rsquo;t want to be left behind.\nThe prospect of having half the staff and twice the productivity was simply more than he could resist.<\/p>\n<p>T: I see many patients who lost jobs to AI.<\/p>\n<p>P: My team oversaw the work and prior to our kickoff meeting we received a lecture on how the new team was very sensitive to criticism.\nApparently some people call their work slop.\nWe had to make sure that no one lost face.<\/p>\n<p>T: Maintaining face is very important in many cultures.<\/p>\n<p>P: In software culture maintaining face is not hard, it&rsquo;s very simple.\nThe rule of thumb is don&rsquo;t formally submit code that looks like it was written by two cats copulating on top of a keyboard.\nI mean holy Christ, we would review their code when we got into the office and once we stopped laughing and pissing ourselves and regained some composure we sat around and tried to figure out how we could tell these imbeciles that their code sucked balls without hurting their precious feelings.<\/p>\n<p>P: Providing constructive feedback is always challenging, and imagine on top of that we had to submit my feedback to the AI agent.\nYou instantly get the &ldquo;You&rsquo;re absolutely correct!&rdquo; only to have the same mistake happen in the next PR.\nFor fuck&rsquo;s sake, the only thing constructive that we could have done was to use their source files as random keys for SSL ciphers.\nWe ended up rewriting their generated code and committing it back.<\/p>\n<p>P: They&rsquo;d pick up the next day totally oblivious to the fact that the code had changed, was legible, and actually worked.\nIgnoring all the patterns in the code and start generating the shit again.\nThey rewrote our app every time a new model was released.\nEvery day they came up with a new markdown file to fix their AI.\nThey promised this is the last file needed and then the AI will not forget things.\nBut the agent always found something new to mess up, and I had to write yet another file.<\/p>\n<p>T: What happened?<\/p>\n<p>P: This happened again and again until the project was completed.\nThe CTO was so happy with the work that he fired my team.<\/p>\n<p>T: But you&rsquo;re the reason the project was successful.<\/p>\n<p>P: I was honored to be a part of one of the universe&rsquo;s great pound-you-in-the-ass ironies.\nIn the end they kept shipping slop and the company went out of business.\nWhen they found out who wrote the original code they blamed my team for not giving in to the vibe.\nThe CTO was celebrated as a genius tactician of startup efficiency and appointed by the VC to another helpless company.\nYou can read about it on X.<\/p>\n<p>T: That CTO is a total douchebag.<\/p>\n<p>P: The man was, um, I&rsquo;m not actually sure there&rsquo;s a word to describe this man.\nIt&rsquo;s a combination of extreme arrogance and utter stupidity.\nDouchebaggery is close but it&rsquo;s not what I&rsquo;m looking for.<\/p>\n<p>T: What you&rsquo;re describing on one hand is classic narcissism.\nThe first sign is an exaggerated sense of self-importance.\nDid you observe this?<\/p>\n<p>P: On a daily basis.<\/p>\n<p>T: What about a preoccupation with fantasies of unlimited success, power, or brilliance?<\/p>\n<p>P: All the fucking time.<\/p>\n<p>T: Requires excessive admiration?<\/p>\n<p>P: Constantly.<\/p>\n<p>T: Has a sense of entitlement?<\/p>\n<p>P: Overwhelmingly.<\/p>\n<p>T: Shows arrogant, haughty, patronizing, or contemptuous behaviors or attitudes?<\/p>\n<p>P: My God, there&rsquo;s a name for this?<\/p>\n<p>T: It&rsquo;s called narcissistic personality disorder.<\/p>\n<p>P: Holy shit, but there&rsquo;s still an element that I&rsquo;m struggling to describe.\nIt&rsquo;s almost as if he&rsquo;s mentally retarded.<\/p>\n<p>T: Mental retardation among senior management in most companies is quite rare.<\/p>\n<p>P: I&rsquo;m no expert but this man exhibited classic retard behavior in every meeting I saw him in.\nIf after one of his self-absorbed oratories he were to return to his chair and drool coffee onto his shirt while making fart noises with his mouth, no one would give it a second thought.\nDoctor, I believe he has full-blown narcissistic mental retardation disorder.<\/p>\n<p>T: That&rsquo;s not a real condition.<\/p>\n<p>P: I can&rsquo;t thank you enough for helping me work through this.\nIt&rsquo;s so liberating to know that my tortured existence under the leadership of this shit-for-brains was no more an injustice than catching a cold from a random sneeze.\nHad I only known at the time, narcissistic mental retardation disorder, it makes perfect sense.\nHe has narcissistic mental retardation disorder.<\/p>\n<p>T: I see our time is up.\nThis has been productive.<\/p>\n<hr>\n<p>Now before you go.\nMy argument is that if you shut off your brain and let AI do the thinking you get slop.\nThat&rsquo;s the current situation.\nWhen you use it and you know what you are doing you get good results.\nThink about it, don&rsquo;t pick a side. See what works and what does not.<\/p>\n"},{"title":"Devlog 6: I Can Make a CPU with My Logic Gate Simulator","link":"https:\/\/glyphack.com\/dv-6\/","pubDate":"Sun, 21 Dec 2025 21:43:56 +0100","guid":"https:\/\/glyphack.com\/dv-6\/","description":"<p>I can finally start making a CPU using <a href=\"https:\/\/github.com\/Glyphack\/simu\" rel=\"noopener\" target=\"_blank\">my logic gate simulator<\/a>.<\/p>\n<p>Try it yourself <a href=\"https:\/\/glyphack.github.io\/simu\/\" rel=\"noopener\" target=\"_blank\">here<\/a>!<\/p>\n<p>It supports basic things you expect from a logic gate simulator.\nDrag gates onto the canvas.\nAttach them together using wires.\nWires stick to pins like magnets.\nYou can detach a wire by selecting it and then move it.<\/p>\nsimu-1.mp4\n\n<p>You can create a connection between two gates by dragging a pin to another.\nWhen a wire is connected to a gate it stays connected even if the gate moves.\nMoving a wire detaches it from its connections.\nA wire can be split into two. A new wire is drawn from the middle, which you can connect elsewhere.\nYou can select gates to move them around.\nWhen you select things you can use copy and paste for each instance.<\/p>\n<p>Simulator is simple and it runs real-time.\nIt supports feedback loops.\nYou can build memory circuits like <a href=\"https:\/\/en.wikipedia.org\/wiki\/Flip-flop_(electronics)\" rel=\"noopener\" target=\"_blank\">Flip Flops<\/a>.<\/p>\nsimu-2.mp4\n\n<p>The next feature that I needed was a way to create circuits.\nA CPU contains thousands of similar circuits. Like Registers.\nIn Simu you can select gates and create a module from them.\nThe module input\/output is mapped to input\/output pins that are unconnected.<\/p>\nsimu-3.mp4\n\n<p>To create a module you need to first free up some pins.\nFree pins are what is considered input\/output of the module.<\/p>\n<p>The module behaves as if those gates were placed directly in the circuit.\nModules are note special.\nWhenever a module is created the state of its gates and connections is saved and when it&rsquo;s added to the circuit the members of the module are added to the circuit.\nConnecting something to the module is similar to connecting it to what&rsquo;s inside the module.\nI made this decision to make modules easier to integrate with rest of the code.\nThis makes it possible to be able to see what exactly is happening in the module. Live as the simulation is running.\nModules are just a faster way to place multiple gates with their connections in the circuit.\nIt&rsquo;s also possible to view inside modules and the state of internal gates.<\/p>\n<p>The panel on the left side and the debug logs window are development tools.\nThey provide everything that is happening in the app on the screen to <a href=\"https:\/\/bernsteinbear.com\/blog\/walking-around\/\" rel=\"noopener\" target=\"_blank\">walk around<\/a> the app.<\/p>\n<p>That&rsquo;s all there is to the UI.\nThe Rest of it are small functionalities to create a circuit faster.\nThe next part is about the design and code of the application<\/p>\n<hr>\n<h2 class=\"heading\" id=\"references-using-ids\">\n  References Using IDs\n  <a class=\"anchor\" href=\"#references-using-ids\">#<\/a>\n<\/h2>\n<p>The main struct in Simu is DB. It holds all the objects on the space and looks like this:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-rust\" data-lang=\"rust\"><span style=\"display:flex;\"><span><span style=\"color:#427b58\">#[derive(Default, serde::Deserialize, serde::Serialize, Debug, Clone)]<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">pub<\/span> <span style=\"color:#af3a03\">struct<\/span> Circuit {\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#928374;font-style:italic\">\/\/ Type registry for each instance id\n<\/span><\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">pub<\/span> types: SlotMap<span style=\"color:#af3a03\">&lt;<\/span>InstanceId, InstanceKind<span style=\"color:#af3a03\">&gt;<\/span>,\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#928374;font-style:italic\">\/\/ Per-kind payloads keyed off the primary key space\n<\/span><\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">pub<\/span> gates: SecondaryMap<span style=\"color:#af3a03\">&lt;<\/span>InstanceId, Gate<span style=\"color:#af3a03\">&gt;<\/span>,\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">pub<\/span> wires: SecondaryMap<span style=\"color:#af3a03\">&lt;<\/span>InstanceId, Wire<span style=\"color:#af3a03\">&gt;<\/span>,\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">pub<\/span> connections: HashSet<span style=\"color:#af3a03\">&lt;<\/span>Connection<span style=\"color:#af3a03\">&gt;<\/span>,\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">..<\/span>. Rest of the objects\n<\/span><\/span><span style=\"display:flex;\"><span>}<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p><a href=\"https:\/\/github.com\/Glyphack\/simu\/blob\/35177cdb0dbabd1ec884a8331cfb70233a4d628d\/src\/db.rs#L60\" rel=\"noopener\" target=\"_blank\">source<\/a><\/p>\n<p><a href=\"https:\/\/docs.rs\/slotmap\/1.1.1\" rel=\"noopener\" target=\"_blank\">Slotmap<\/a> is similar to an array.\nUpon inserting a new instance it returns an ID and that&rsquo;s why everything is a map from <code>InstanceId<\/code> to the actual data.\nEverything in the program refers to other things using the ID.<\/p>\n<p>Connections just hold some IDs of which pins from what ID are connected.<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-rust\" data-lang=\"rust\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">pub<\/span> <span style=\"color:#af3a03\">struct<\/span> Connection {\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">pub<\/span> a: Pin,\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">pub<\/span> b: Pin,\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">pub<\/span> kind: ConnectionKind,\n<\/span><\/span><span style=\"display:flex;\"><span>}\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">pub<\/span> <span style=\"color:#af3a03\">struct<\/span> Pin {\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">pub<\/span> ins: InstanceId,\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">pub<\/span> index: <span style=\"color:#b57614\">u32<\/span>,\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">pub<\/span> kind: PinKind,\n<\/span><\/span><span style=\"display:flex;\"><span>}<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p><a href=\"https:\/\/github.com\/Glyphack\/simu\/blob\/35177cdb0dbabd1ec884a8331cfb70233a4d628d\/src\/connection_manager.rs#L20\" rel=\"noopener\" target=\"_blank\">source<\/a><\/p>\n<p>As an example to get to instances connected to a pin with this data structure we have to:<\/p>\n<ol>\n<li>Find connections including that pin<\/li>\n<li>Get the other pin in those connections<\/li>\n<li>Get the instance ID of those pins<\/li>\n<li>Find what is the type of that instance ID<\/li>\n<li>Lookup the instance ID in its associated map(wires, gates, etc.)<\/li>\n<\/ol>\n<p>This is harder to do than having objects embedded into each other and having access like <code>pin.instance.pos<\/code>.\nBut it introduces borrow checker issues.\nWhich is painful to deal with. And requires a lot of code.<\/p>\n<p>In this program I know when and where something is not needed and how to clean up the object.\nI don&rsquo;t have anything against smart pointers, my reasoning was similar here: I know when and how the clean up should happen I can just code it.\nIf I had a more complex object lifetime then I could consider them.<\/p>\n<p>Another major benefit of using IDs is being able to save and load circuits easily.\nThere&rsquo;s almost no serialization code to persist state of the program to the disk.<\/p>\n<h2 class=\"heading\" id=\"simulation-logic\">\n  Simulation Logic\n  <a class=\"anchor\" href=\"#simulation-logic\">#<\/a>\n<\/h2>\n<p>Simulation logic is straightforward except for feedback loops.\nLet me explain.<\/p>\n<p>In some circuits we feed the output of the circuit back to itself.\nThis is for creating &ldquo;memory&rdquo; so the circuit works based on its previous state.\nWhat happens in these circuits is this:<\/p>\n<ol>\n<li>Gate A output is connected to Gate B<\/li>\n<li>Gate B output is connected to Gate A<\/li>\n<\/ol>\n<p>Now in order to calculate the output of Gate A we need to calculate Gate B.\nOne solution here is to assume a fallback value for output of Gate A and then we can calculate Gate B.\nOnce B is determined we use that to calculate A.\nBut we are not done yet. We assumed a fallback value for A.\nNow that we have fallback value of A output and calculated value we need to compare them.\nIf the computed value is different from the assumption then we need to redo the computation this time with the new computed value.\nThis operation goes on until the value we get out of computing is same as what we assumed at the beginning.\nIs it guaranteed to get to this point? Well no, you can connect output of a NAND to itself.\nThis causes the gate output to toggle between on and off.<\/p>\n<p>Following the same algorithm above:<\/p>\n<ol>\n<li>Assume output is 0<\/li>\n<li>Input is 0<\/li>\n<li>NAND with 0 input is 1 -&gt; contradicts assumption<\/li>\n<\/ol>\n<p>No matter how many times we iterate, a NAND connected to itself is going to toggle.\nSo we need an upper bound of times we try to iterate to get to a stable point.\nThis number is 10 in the program.\nMeaning that the program will simulate the whole circuit at most 10 times and check if the cycles are resolved.\nNote that this only happens when there is complex cycle in the program, most of cycles resolve in 6 or 7 iterations.\nMy plan for more optimization here is to resolve each cycle in isolation.\nThis will require <a href=\"https:\/\/en.wikipedia.org\/wiki\/Strongly_connected_component\" rel=\"noopener\" target=\"_blank\">SCC<\/a>.\nThis means that if a cycle contains 5 instances out of 100 in the circuit only those 5 would be re computed multiple times.<\/p>\n<hr>\n<p>I always wanted to try making a better logisim.\nThat was the program I used as a student to make a 16-bit CPU based on <a href=\"https:\/\/www.goodreads.com\/book\/show\/224131.Computer_System_Architecture\" rel=\"noopener\" target=\"_blank\">Mano&rsquo;s book<\/a>.\nThat software is good but it&rsquo;s not fast and responsive. It does not run in the browser and the UI is not good.<\/p>\n<p>My first goal is to start making my own CPU. Along the way I discover more UI features that I need to make it faster and better.\nAnd after that maybe I can add a way to design quizzes.\nIt would be fun to solve puzzles for learning logic gates.\nI wish more things were interactive when I was learning.\nBut I can build them for others.<\/p>\n"},{"title":"Ty Test Suite","link":"https:\/\/glyphack.com\/ty-test\/","pubDate":"Wed, 10 Dec 2025 16:12:40 +0100","guid":"https:\/\/glyphack.com\/ty-test\/","description":"<p><a href=\"https:\/\/github.com\/astral-sh\/ty\" rel=\"noopener\" target=\"_blank\">Ty<\/a> tests use markdown.\nIt looked strange, but I&rsquo;m sold to the idea now.\nWhen I started programming all I found online on testing was about unit tests, end-to-end, black-box, white-box.\nSo why don\u2019t more tests look like this?<\/p>\n<h2 class=\"heading\" id=\"unit-tests-are-verbose\">\n  Unit Tests Are Verbose\n  <a class=\"anchor\" href=\"#unit-tests-are-verbose\">#<\/a>\n<\/h2>\n<p>Say you\u2019re writing a parser and want to add a unit test. It usually ends up looking like this example from <a href=\"https:\/\/glyphack.com\/s\/write-an-interpreter-in-go\/\" rel=\"noopener\" target=\"_blank\">Writing An Interpreter In Go<\/a>:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-go\" data-lang=\"go\"><span style=\"display:flex;\"><span><span style=\"color:#928374;font-style:italic\">\/\/ parser\/parser_test.go<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">func<\/span> <span style=\"color:#b57614\">TestFunctionLiteralParsing<\/span>(t <span style=\"color:#af3a03\">*<\/span>testing.T) {\n<\/span><\/span><span style=\"display:flex;\"><span> input <span style=\"color:#af3a03\">:=<\/span> <span style=\"color:#79740e\">`fn(x, y) { x + y; }`<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span> l <span style=\"color:#af3a03\">:=<\/span> lexer.<span style=\"color:#b57614\">New<\/span>(input)\n<\/span><\/span><span style=\"display:flex;\"><span> p <span style=\"color:#af3a03\">:=<\/span> <span style=\"color:#b57614\">New<\/span>(l)\n<\/span><\/span><span style=\"display:flex;\"><span> program <span style=\"color:#af3a03\">:=<\/span> p.<span style=\"color:#b57614\">ParseProgram<\/span>()\n<\/span><\/span><span style=\"display:flex;\"><span> <span style=\"color:#b57614\">checkParserErrors<\/span>(t, p)\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span> <span style=\"color:#af3a03\">if<\/span> <span style=\"color:#b57614\">len<\/span>(program.Statements) <span style=\"color:#af3a03\">!=<\/span> <span style=\"color:#8f3f71\">1<\/span> {\n<\/span><\/span><span style=\"display:flex;\"><span>  t.<span style=\"color:#b57614\">Fatalf<\/span>(<span style=\"color:#79740e\">&#34;program.Body does not contain %d statements. got=%d\\n&#34;<\/span>,\n<\/span><\/span><span style=\"display:flex;\"><span>   <span style=\"color:#8f3f71\">1<\/span>, <span style=\"color:#b57614\">len<\/span>(program.Statements))\n<\/span><\/span><span style=\"display:flex;\"><span> }\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span> stmt, ok <span style=\"color:#af3a03\">:=<\/span> program.Statements[<span style=\"color:#8f3f71\">0<\/span>].(<span style=\"color:#af3a03\">*<\/span>ast.ExpressionStatement)\n<\/span><\/span><span style=\"display:flex;\"><span> <span style=\"color:#af3a03\">if<\/span> !ok {\n<\/span><\/span><span style=\"display:flex;\"><span>  t.<span style=\"color:#b57614\">Fatalf<\/span>(<span style=\"color:#79740e\">&#34;program.Statements[0] is not ast.ExpressionStatement. got=%T&#34;<\/span>,\n<\/span><\/span><span style=\"display:flex;\"><span>   program.Statements[<span style=\"color:#8f3f71\">0<\/span>])\n<\/span><\/span><span style=\"display:flex;\"><span> }\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span> function, ok <span style=\"color:#af3a03\">:=<\/span> stmt.Expression.(<span style=\"color:#af3a03\">*<\/span>ast.FunctionLiteral)\n<\/span><\/span><span style=\"display:flex;\"><span> <span style=\"color:#af3a03\">if<\/span> !ok {\n<\/span><\/span><span style=\"display:flex;\"><span>  t.<span style=\"color:#b57614\">Fatalf<\/span>(<span style=\"color:#79740e\">&#34;stmt.Expression is not ast.FunctionLiteral. got=%T&#34;<\/span>,\n<\/span><\/span><span style=\"display:flex;\"><span>   stmt.Expression)\n<\/span><\/span><span style=\"display:flex;\"><span> }\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span> <span style=\"color:#af3a03\">if<\/span> <span style=\"color:#b57614\">len<\/span>(function.Parameters) <span style=\"color:#af3a03\">!=<\/span> <span style=\"color:#8f3f71\">2<\/span> {\n<\/span><\/span><span style=\"display:flex;\"><span>  t.<span style=\"color:#b57614\">Fatalf<\/span>(<span style=\"color:#79740e\">&#34;function literal parameters wrong. want 2, got=%d\\n&#34;<\/span>,\n<\/span><\/span><span style=\"display:flex;\"><span>   <span style=\"color:#b57614\">len<\/span>(function.Parameters))\n<\/span><\/span><span style=\"display:flex;\"><span> }\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span> <span style=\"color:#b57614\">testLiteralExpression<\/span>(t, function.Parameters[<span style=\"color:#8f3f71\">0<\/span>], <span style=\"color:#79740e\">&#34;x&#34;<\/span>)\n<\/span><\/span><span style=\"display:flex;\"><span> <span style=\"color:#b57614\">testLiteralExpression<\/span>(t, function.Parameters[<span style=\"color:#8f3f71\">1<\/span>], <span style=\"color:#79740e\">&#34;y&#34;<\/span>)\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span> <span style=\"color:#af3a03\">if<\/span> <span style=\"color:#b57614\">len<\/span>(function.Body.Statements) <span style=\"color:#af3a03\">!=<\/span> <span style=\"color:#8f3f71\">1<\/span> {\n<\/span><\/span><span style=\"display:flex;\"><span>  t.<span style=\"color:#b57614\">Fatalf<\/span>(<span style=\"color:#79740e\">&#34;function.Body.Statements has not 1 statements. got=%d\\n&#34;<\/span>,\n<\/span><\/span><span style=\"display:flex;\"><span>   <span style=\"color:#b57614\">len<\/span>(function.Body.Statements))\n<\/span><\/span><span style=\"display:flex;\"><span> }\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span> bodyStmt, ok <span style=\"color:#af3a03\">:=<\/span> function.Body.Statements[<span style=\"color:#8f3f71\">0<\/span>].(<span style=\"color:#af3a03\">*<\/span>ast.ExpressionStatement)\n<\/span><\/span><span style=\"display:flex;\"><span> <span style=\"color:#af3a03\">if<\/span> !ok {\n<\/span><\/span><span style=\"display:flex;\"><span>  t.<span style=\"color:#b57614\">Fatalf<\/span>(<span style=\"color:#79740e\">&#34;function body stmt is not ast.ExpressionStatement. got=%T&#34;<\/span>,\n<\/span><\/span><span style=\"display:flex;\"><span>   function.Body.Statements[<span style=\"color:#8f3f71\">0<\/span>])\n<\/span><\/span><span style=\"display:flex;\"><span> }\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span> <span style=\"color:#b57614\">testInfixExpression<\/span>(t, bodyStmt.Expression, <span style=\"color:#79740e\">&#34;x&#34;<\/span>, <span style=\"color:#79740e\">&#34;+&#34;<\/span>, <span style=\"color:#79740e\">&#34;y&#34;<\/span>)\n<\/span><\/span><span style=\"display:flex;\"><span>}<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>This test case is a beast<sup id=\"fnref:1\"><a href=\"#fn:1\" class=\"footnote-ref\" role=\"doc-noteref\">1<\/a><\/sup>.\nAll this code just to check a one-line function parses correctly.\nIf we decide to update this test to check, say a function with 3 parameters we need to modify multiple places in this test.\nWhat you can do in this case is to define a function for it:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-go\" data-lang=\"go\"><span style=\"display:flex;\"><span>fn <span style=\"color:#b57614\">checkParameters<\/span>(params <span style=\"color:#af3a03\">*<\/span>function.Parameters, expected []<span style=\"color:#b57614\">string<\/span>) {\n<\/span><\/span><span style=\"display:flex;\"><span> <span style=\"color:#928374;font-style:italic\">\/\/ Check expected parametes are in params in order<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>}<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>So then you can test different combinations of parameters with less lines of code.\nAlso if you change how parameters are stored in <code>function.Parameters<\/code> you don&rsquo;t need to update individual tests.\nWe need lots of tiny helpers just to keep tests readable.<\/p>\n<h2 class=\"heading\" id=\"snapshot-the-output\">\n  Snapshot the Output\n  <a class=\"anchor\" href=\"#snapshot-the-output\">#<\/a>\n<\/h2>\n<p>Complex output makes assertions messy and huge.<\/p>\n<p>I learned this technique of storing output of tests from <a href=\"https:\/\/github.com\/tjdevries\/vim9jit\" rel=\"noopener\" target=\"_blank\">vim9jit<\/a>.<\/p>\n<p>For example, <a href=\"https:\/\/github.com\/tjdevries\/vim9jit\/blob\/master\/crates\/vim9-parser\/testdata\/snapshots\/assign.vim\" rel=\"noopener\" target=\"_blank\">this test<\/a> is just the program input.\nThe test runner feeds it to the parser and stores the output in a snapshot file next to it.\nOn the next run, it compares the parser\u2019s output to the stored snapshot instead of checking dozens of individual assertions.<\/p>\n<p>For example,  is just the input of the program.\nIt feeds the input to the parser and stores the <a href=\"https:\/\/github.com\/tjdevries\/vim9jit\/blob\/master\/crates\/vim9-parser\/testdata\/output\/vim9_parser__test__assign.snap\" rel=\"noopener\" target=\"_blank\">output<\/a> next to the test.\nNext time the tests run it compares the output of the parser to the previous stored snapshot.\nIn this method instead of individual asserts we check the output of the parser.<\/p>\n<p>This works especially well for programs that transform data, like parsers and compilers, but you can use it anywhere.\nAll you need is a method to print state of the program as text.\nAnd you most likely already need to <a href=\"https:\/\/bernsteinbear.com\/blog\/walking-around\/\" rel=\"noopener\" target=\"_blank\">reveal the internals<\/a> of the program for debugging.\nWhich makes it even more appealing the code is not just for tests.<\/p>\n<p>I used snapshot testing on my project for a while, but hit another problem: snapshots can get <a href=\"https:\/\/github.com\/Glyphack\/enderpy\/blob\/dc53f04f8223653b272bc8e806a1d59d33667b4e\/typechecker\/test_data\/output\/enderpy_python_type_checker__checker__tests__specialtypes_none.snap#L98\" rel=\"noopener\" target=\"_blank\">huge<\/a>.\nChecking large snapshots requires having input and output open side by side to compare.\nI found myself not reviewing the output carefully because the output was massive.\nSo snapshots cut boilerplate, but they also make it easy to hide bugs inside massive dumps of text.<\/p>\n<h2 class=\"heading\" id=\"literate-testing\">\n  Literate Testing\n  <a class=\"anchor\" href=\"#literate-testing\">#<\/a>\n<\/h2>\n<p>Ty\u2019s tests live in markdown files.\nIt&rsquo;s very close to the idea of <a href=\"https:\/\/en.wikipedia.org\/wiki\/Literate_programming#Workflow\" rel=\"noopener\" target=\"_blank\">literate programming<\/a>.<\/p>\n<blockquote>\n<p>Implementing literate programming consists of two steps:<\/p>\n<ol>\n<li>Weaving: Generating a comprehensive document about the program and its maintenance.<\/li>\n<li>Tangling: Generating machine executable code<\/li>\n<\/ol>\n<\/blockquote>\n<p>You write examples and documentation, and they\u00a0are\u00a0the <a href=\"https:\/\/github.com\/astral-sh\/ruff\/blob\/main\/crates\/ty_python_semantic\/resources\/mdtest\/assignment\/annotations.md\" rel=\"noopener\" target=\"_blank\">tests<\/a>:<\/p>\n<p>\n\n\n\n\n\n\n\n<figure class=\"\">\n\n    \n    \n      \n        \n        <div class=\"img-container\" style=\"--w: 2580; --h: 1494;\">\n            <img loading=\"lazy\" alt=\"Ty Test Example\" src=\"https:\/\/glyphack.com\/ty-test\/ty-test-example_hu_1b4471d7b10dc369.png\" width=\"2580\" height=\"1494\">\n        <\/div>\n      \n    \n\n    \n<\/figure>\n<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-python\" data-lang=\"python\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">f<\/span>() <span style=\"color:#af3a03\">-&gt;<\/span> <span style=\"color:#b57614\">int<\/span>:\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">return<\/span> <span style=\"color:#8f3f71\">1<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span> \n<\/span><\/span><span style=\"display:flex;\"><span>reveal_type(f())  <span style=\"color:#928374;font-style:italic\"># revealed: int<\/span><\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>This test checks how function return types are type-checked.\nAdding another test case is easy. You just paste the Python code.\nIf you get a bug report, you can paste the failing code straight into the tests and reproduce it without extra boilerplate.<\/p>\n<ol>\n<li>You write the Python code you want to test.<\/li>\n<li>You call\u00a0<code>reveal_type<\/code>\u00a0on an expression and write the expected type in a comment.<\/li>\n<li>Test runner passes this code to Ty type checker.<\/li>\n<li>The test runner feeds this file to Ty, which evaluates each\u00a0<code>reveal_type<\/code>\u00a0call and compares the displayed type to the comment.<\/li>\n<\/ol>\n<p><code>reveal_type<\/code> is not the only way to assert.\nThere are more functions like <code>generic_context<\/code> to <a href=\"https:\/\/github.com\/Glyphack\/ruff\/blob\/bcddab6680f5026718094a630e2891f79488a55a\/crates\/ty_python_semantic\/resources\/mdtest\/generics\/scoping.md\" rel=\"noopener\" target=\"_blank\">check<\/a> the generic type information.<\/p>\n<p>Python also uses <a href=\"https:\/\/github.com\/python\/typing\/blob\/main\/conformance\/tests\/annotations_methods.py\" rel=\"noopener\" target=\"_blank\">this style<\/a> of testing for type checker conformance tests.\nYou want to test different type checkers.\nThey all take Python programs as input.\nSo let&rsquo;s write test cases as Python programs.<\/p>\n<h2 class=\"heading\" id=\"other-programs\">\n  Other Programs\n  <a class=\"anchor\" href=\"#other-programs\">#<\/a>\n<\/h2>\n<p>If a program you&rsquo;re working on is not transforming text then applying this idea directly is not possible.\nBut so do API servers (JSON responses), database systems (query results), even UI frameworks (rendered HTML).\nSo by writing the code that reveals the internals as text it is possible to test using the above techniques.<\/p>\n<div class=\"footnotes\" role=\"doc-endnotes\">\n<hr>\n<ol>\n<li id=\"fn:1\">\n<p>In case you&rsquo;re wondering how does a simpler version of this look like. This is example of the <a href=\"https:\/\/github.com\/Glyphack\/enderpy\/blob\/dc53f04f8223653b272bc8e806a1d59d33667b4e\/parser\/test_data\/inputs\/function_def.py#L1\" rel=\"noopener\" target=\"_blank\">input<\/a> and <a href=\"https:\/\/github.com\/Glyphack\/enderpy\/blob\/dc53f04f8223653b272bc8e806a1d59d33667b4e\/parser\/test_data\/output\/enderpy_python_parser__parser__parser__tests__function_def.snap#L11-L46\" rel=\"noopener\" target=\"_blank\">output<\/a>. The output file is long. But if you have a <a href=\"https:\/\/insta.rs\/\" rel=\"noopener\" target=\"_blank\">good tool<\/a> for snapshot tests it&rsquo;s easy to review changes. Otherwise you can split the big input file to smaller ones and have smaller output.&#160;<a href=\"#fnref:1\" class=\"footnote-backref\" role=\"doc-backlink\">&#x21a9;&#xfe0e;<\/a><\/p>\n<\/li>\n<\/ol>\n<\/div>\n"},{"title":"Simple Expertise","link":"https:\/\/glyphack.com\/se\/","pubDate":"Sat, 25 Oct 2025 11:09:19 +0200","guid":"https:\/\/glyphack.com\/se\/","description":"<p>If you think about how we judge skills you&rsquo;d see we do it by questioning the advanced stuff.<\/p>\n<p>Some think this is right. If a person has experience in something they must know the advanced topics that beginners haven\u2019t heard of.\nThis is what schools do: after every level, the questions get harder.\nI\u2019ve sat through more physics than Galileo and Newton combined.\nAm I a good physicist?<\/p>\n<p>That&rsquo;s how my mindset changed.<\/p>\n<p>I was sitting at a talk by <a href=\"https:\/\/github.com\/kelseyhightower\" rel=\"noopener\" target=\"_blank\">Kelsey Hightower<\/a><sup id=\"fnref:1\"><a href=\"#fn:1\" class=\"footnote-ref\" role=\"doc-noteref\">1<\/a><\/sup>.\nHe said in every interview he wants to learn something new about the person.\nAnd he asks the candidate to write a simple program and watches everything he does.\nSo he did this live in the talk.<\/p>\n<p>He opened the terminal, then Vim, and wrote a Go hello world.<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-go\" data-lang=\"go\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">package<\/span> main\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">import<\/span> <span style=\"color:#79740e\">&#34;fmt&#34;<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">func<\/span> <span style=\"color:#b57614\">main<\/span>() {\n<\/span><\/span><span style=\"display:flex;\"><span> fmt.<span style=\"color:#b57614\">Println<\/span>(<span style=\"color:#79740e\">&#34;hello world&#34;<\/span>)\n<\/span><\/span><span style=\"display:flex;\"><span>}<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>He Compiled it and asked how big is the binary? No one knew.\nHe ran <code>ls -lh<\/code> and it was 2Mb.<\/p>\n<p>He asked was what is making this file 2 megabytes?\nPeople suggested complex reasons: debug symbols, Go runtime, system libraries.<\/p>\n<p>He asked How can we apply these suggestions to make the binary smaller?\nAgain silence in the room.\nIdeas were easy to remember and say, but no one thought about applying them.<\/p>\n<p>He proceeded to change it to:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-go\" data-lang=\"go\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">package<\/span> main\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">func<\/span> <span style=\"color:#b57614\">main<\/span>() {\n<\/span><\/span><span style=\"display:flex;\"><span> <span style=\"color:#b57614\">println<\/span>(<span style=\"color:#79740e\">&#34;hello world&#34;<\/span>)\n<\/span><\/span><span style=\"display:flex;\"><span>}<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>These questions can continue,\nhow many instructions do you think the two programs have?<\/p>\n<p>Expertise shows up in simple problems, because simple problems expose fundamentals.\nIn programming we are working on top of <a href=\"https:\/\/iuliangulea.com\/pyramid-of-mastery\" rel=\"noopener\" target=\"_blank\">layers of abstraction<\/a>.\nAbstractions can be good when you know how they work and you use them but otherwise they are some artificial limit.\nAn abstraction is designed for a use case in mind.\nWhen it does not work the truth is not that it cannot be done, but it&rsquo;s to break through the layer.\nDiscover how to achieve the goal from basic principles.\nIt&rsquo;s like saying a program cannot be smaller because of the compiler.\nThe compiler is just a word. The truth is what did the compiler put into the program.<\/p>\n<p>This happens in other topics.\nIf someone asks why leaves are green and I say it&rsquo;s because they contain chlorophyll do I really know what I&rsquo;m talking about?\nThat&rsquo;s the answer everyone gets in school.<\/p>\n<p>How does it pay off to see what&rsquo;s under the hood?<\/p>\n<p>Kelsey told a story that his colleague found a bug in go compiler because he looked at the instructions when noticed the program is not fast.\nIt&rsquo;s easier to shrug it off and say the compiler did some magic work and this is the result.\nThat never leads to interesting findings.<\/p>\n<p>This was also my own experience.\nI thought contributing to <a href=\"https:\/\/glyphack.com\/contributing-to-python-docs\/\" rel=\"noopener\" target=\"_blank\">Python<\/a> required some magical preparation.\nThe reality is, just read what&rsquo;s in the manual then check what&rsquo;s going on.\nIf they don&rsquo;t match you are onto something.<\/p>\n<p>If you want to feel confident about your knowledge, start with the simplest task and keep asking fundamental questions.\nDon&rsquo;t take magical words as answers. The answers are simple.<\/p>\n<div class=\"footnotes\" role=\"doc-endnotes\">\n<hr>\n<ol>\n<li id=\"fn:1\">\n<p>He didn&rsquo;t prepare any slides. It was more a discussion. Presentation was three slides with one big word &ldquo;AI&rdquo; and it got bigger in each slide. He asked everyone to invest in themselves. Our work is the AI training data so there&rsquo;s still value in thinking. Don&rsquo;t delegate your thinking to AI.\n\n\n\n\n\n\n\n\n<figure class=\"\">\n\n    \n    \n      \n        \n        <div class=\"img-container\" style=\"--w: 5712; --h: 4284;\">\n            <img loading=\"lazy\" alt=\"Kelsey\" src=\"https:\/\/glyphack.com\/se\/kelsey_hu_7f14c4fc5e64cf83.jpeg\" width=\"5712\" height=\"4284\">\n        <\/div>\n      \n    \n\n    \n<\/figure>\n&#160;<a href=\"#fnref:1\" class=\"footnote-backref\" role=\"doc-backlink\">&#x21a9;&#xfe0e;<\/a><\/p>\n<\/li>\n<\/ol>\n<\/div>\n"},{"title":"Devlog 5: Simu, Ty, Book Reader","link":"https:\/\/glyphack.com\/dv-5\/","pubDate":"Sun, 28 Sep 2025 10:52:46 +0200","guid":"https:\/\/glyphack.com\/dv-5\/","description":"<h2 class=\"heading\" id=\"chat-with-new-people\">\n  Chat with New People\n  <a class=\"anchor\" href=\"#chat-with-new-people\">#<\/a>\n<\/h2>\n<p>Met <a href=\"https:\/\/amirhn.com\/\" rel=\"noopener\" target=\"_blank\">Amir<\/a>. Turns out it was a good decision to live stream writing Redis in C from scratch. You find new friends by doing it.\nMet Matt from <a href=\"https:\/\/blinkinlabs.com\/about\" rel=\"noopener\" target=\"_blank\">Blinkinlabs<\/a>.<\/p>\n<h2 class=\"heading\" id=\"simu\">\n  Simu\n  <a class=\"anchor\" href=\"#simu\">#<\/a>\n<\/h2>\n<p>About a month ago I started building a logic gate simulator software. My goal is to run it in web browser and have enough features to make a CPU with it.\nI think this will be useful for teaching people about CPUs.<\/p>\n<p>Worked on:<\/p>\n<ul>\n<li>Polishing the UI, You can click drag a pin to connect it to another pin.<\/li>\n<li>Various dragging actions for dragging and keeping the connection, drag both ends of the wire and drag only one end of the wire.<\/li>\n<li>Draft for connecting a wire to middle of another wire, live coding <a href=\"https:\/\/youtu.be\/ROll1Qz64OQ\" rel=\"noopener\" target=\"_blank\">here<\/a>(Alert: it&rsquo;s in persian.)<\/li>\n<\/ul>\n<p>You can watch me <a href=\"https:\/\/youtu.be\/ROll1Qz64OQ?t=295\" rel=\"noopener\" target=\"_blank\">here<\/a> using it to build a multiplexer. The UI is finally easy and fast to use.<\/p>\n<h2 class=\"heading\" id=\"ty\">\n  Ty\n  <a class=\"anchor\" href=\"#ty\">#<\/a>\n<\/h2>\n<p>Ty worked on the <a href=\"https:\/\/glyphack.com\/ty-self\/\">self feature<\/a> a bit and then I&rsquo;m taking a break. I&rsquo;ll get back this week.\nMost of the blocking issues are resolved by others. They did the super hard part.\nWe are close to have Self type in Ty!<\/p>\n<h2 class=\"heading\" id=\"book-reader\">\n  Book Reader\n  <a class=\"anchor\" href=\"#book-reader\">#<\/a>\n<\/h2>\n<p>I had this project from a long time ago and I stopped using it. It&rsquo;s a book reader with builtin translation.\nI made it when I was reading a <a href=\"https:\/\/nl.wikipedia.org\/wiki\/Max_Havelaar_(boek)\" rel=\"noopener\" target=\"_blank\">Max Havelaar<\/a> But I lost my interest.\nNow my girlfriend is using it so I spend more time working on it. The good part is that it still compiles after few months of not touching it. I will write more about it.<\/p>\n<h2 class=\"heading\" id=\"books\">\n  Books\n  <a class=\"anchor\" href=\"#books\">#<\/a>\n<\/h2>\n<p>I started reading [Data Oriented Design](\/synced\/Data Oriented Design) when I worked on Simu.\nIt&rsquo;s a good book, I never had correct training on data normalization and this book had new stuff for me right away.<\/p>\n<p>As I read more of the book it got more abstract and it was talking about topics that are hard to apply. For example &ldquo;Avoid enums&rdquo; But the author does not talk about the huge amount of code you need to write when you avoid enums. I think overall there are useful ideas in the book but on the other hand it&rsquo;s not as practical as I expected it to be.<\/p>\n<p>While reading it my friend Kevin sent me <a href=\"https:\/\/kyju.org\/blog\/rustconf-2018-keynote\/\" rel=\"noopener\" target=\"_blank\">this blog post<\/a> which had more examples of writing code in DoD way.<\/p>\n<p>[Creativity, Inc.](\/synced\/Creativity, Inc.) was in my reading list for a long time.\nI heard good things about it from a lot of blogs and people.<\/p>\n<p>First chapters are cool. It talks about the history of Pixar, how Ed got interested in making animations with computers and how long it took for him to get funded and pursue this dream.<\/p>\n<p>Everything else is generic management advice. Things like people should review each others work and be able to give honest opinion, you can&rsquo;t control everything so leave room for randomness in your calculations.<\/p>\n<p>So not so exciting of non-fiction choices.<\/p>\n<h2 class=\"heading\" id=\"various-links\">\n  Various Links\n  <a class=\"anchor\" href=\"#various-links\">#<\/a>\n<\/h2>\n<ul>\n<li><a href=\"https:\/\/geohot.github.io\/\/blog\/jekyll\/update\/2025\/09\/12\/ai-coding.html\" rel=\"noopener\" target=\"_blank\">https:\/\/geohot.github.io\/\/blog\/jekyll\/update\/2025\/09\/12\/ai-coding.html<\/a><\/li>\n<li><a href=\"https:\/\/borretti.me\/article\/lessons-writing-compiler\" rel=\"noopener\" target=\"_blank\">https:\/\/borretti.me\/article\/lessons-writing-compiler<\/a><\/li>\n<li><a href=\"https:\/\/en.wikipedia.org\/wiki\/Chris_McCandless\" rel=\"noopener\" target=\"_blank\">https:\/\/en.wikipedia.org\/wiki\/Chris_McCandless<\/a><\/li>\n<li><a href=\"https:\/\/www.recurse.com\/self-directives\" rel=\"noopener\" target=\"_blank\">https:\/\/www.recurse.com\/self-directives<\/a><\/li>\n<li><a href=\"https:\/\/bernsteinbear.com\/blog\/walking-around\" rel=\"noopener\" target=\"_blank\">https:\/\/bernsteinbear.com\/blog\/walking-around<\/a><\/li>\n<li><a href=\"https:\/\/commandcenter.blogspot.com\/2023\/12\/simplicity.html\" rel=\"noopener\" target=\"_blank\">https:\/\/commandcenter.blogspot.com\/2023\/12\/simplicity.html<\/a><\/li>\n<li><a href=\"https:\/\/www.youtube.com\/watch?v=JRTLSxGf_6w\" rel=\"noopener\" target=\"_blank\">https:\/\/www.youtube.com\/watch?v=JRTLSxGf_6w<\/a><\/li>\n<\/ul>\n"},{"title":"Adding Support for Self to Ty","link":"https:\/\/glyphack.com\/ty-self\/","pubDate":"Thu, 18 Sep 2025 19:20:37 +0200","guid":"https:\/\/glyphack.com\/ty-self\/","description":"<p>I am reading <a href=\"https:\/\/glyphack.com\/s\/no-ordinary-genius\/\" rel=\"noopener\" target=\"_blank\">No Ordinary Genius<\/a> and the book is saying how nice it is when some technical writing includes everything you need to understand a subject. In its own words:<\/p>\n<blockquote>\n<p>I find it very discomforting now that there are many books that claim to explain a subject, in which there&rsquo;s some concept or other which is very poorly explained, and therefore it&rsquo;s not there\u2014no matter how hard you study that book, you&rsquo;ll never come out the other end.\nYou&rsquo;d have to know the concept wasn&rsquo;t there, and that&rsquo;s hard to do when you&rsquo;re first learning a subject. The problem is not that you are foolish or incapable, but that the writing isn&rsquo;t there. I was lucky in that the sources I had always had everything in them; even if it was in very condensed form, it was all explained very carefully.<\/p>\n<\/blockquote>\n<p>Also, reading through the implementation of <a href=\"https:\/\/jellezijlstra.github.io\/pep695.html\" rel=\"noopener\" target=\"_blank\">PEP 695<\/a>, it&rsquo;s a great example of the case above, where it explains the subject perfectly.<\/p>\n<p>I didn&rsquo;t write this way before, but I&rsquo;m trying my best this time.<\/p>\n<p>After my last attempt at writing a <a href=\"https:\/\/glyphack.com\/dv-2\/\" rel=\"noopener\" target=\"_blank\">Python type checker<\/a>, and failing, I started looking into <a href=\"https:\/\/github.com\/astral-sh\/ty\/\" rel=\"noopener\" target=\"_blank\">Ty<\/a> to learn how Python typing works and how to implement this in Rust without blood and sweat.\nSo I&rsquo;ve been casually contributing to Ty in my free time.\nAbout 5 months ago, when I was browsing Ty issues I found that it <a href=\"https:\/\/github.com\/astral-sh\/ty\/issues\/159\" rel=\"noopener\" target=\"_blank\">does not support the <code>Self<\/code> type yet<\/a>.\nIt looked like a fun thing, so I started working on it.<\/p>\n<p>In this post, I explain how this was implemented, the areas of code it touched, and the challenges along the way.<\/p>\n<h2 class=\"heading\" id=\"self-type\">\n  Self type\n  <a class=\"anchor\" href=\"#self-type\">#<\/a>\n<\/h2>\n<p><code>self<\/code> is a type in Python that refers to the class itself.<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-python\" data-lang=\"python\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">class<\/span> Person:\n<\/span><\/span><span style=\"display:flex;\"><span>  <span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">name<\/span>(<span style=\"color:#b57614\">self<\/span>): <span style=\"color:#af3a03\">...<\/span> <span style=\"color:#928374;font-style:italic\"># self is referring to Person<\/span><\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>This part of the <a href=\"https:\/\/typing.python.org\/en\/latest\/spec\/generics.html#self\" rel=\"noopener\" target=\"_blank\">typing spec<\/a> explains a strategy to determine a type for the <code>self<\/code> argument in the above code.<\/p>\n<p>How does this <code>self<\/code> type work under the hood? It&rsquo;s all <a href=\"https:\/\/en.wikipedia.org\/wiki\/Syntactic_sugar\" rel=\"noopener\" target=\"_blank\">syntactic sugar<\/a>:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-python\" data-lang=\"python\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">from<\/span> <span style=\"color:#79740e\">typing<\/span> <span style=\"color:#af3a03\">import<\/span> TypeVar\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>Self <span style=\"color:#af3a03\">=<\/span> TypeVar(<span style=\"color:#79740e\">&#34;Self&#34;<\/span>, bound<span style=\"color:#af3a03\">=<\/span><span style=\"color:#79740e\">&#34;Person&#34;<\/span>)\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">class<\/span> Person:\n<\/span><\/span><span style=\"display:flex;\"><span>  <span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">name<\/span>(<span style=\"color:#b57614\">self<\/span>: Self): <span style=\"color:#af3a03\">...<\/span> <span style=\"color:#928374;font-style:italic\"># self is referring to Person<\/span><\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>And if you are not familiar with <code>TypeVar<\/code>, it simply means:<\/p>\n<ol>\n<li>Any type that can be used in place of <code>Person<\/code> can be used in place of the <code>Self<\/code> type.<\/li>\n<li>Any type that is a subclass of <code>Person<\/code> can also be used in place of the <code>Self<\/code> variable.<\/li>\n<\/ol>\n<p>How do you annotate <code>self<\/code> arguments in methods?\nWriting a type variable just for every class would be tedious.\nWe can leave that to typeshed.<\/p>\n<p>The <code>typing<\/code> module provides <code>Self<\/code> as a symbol that any Python code can use to refer to the class. So we can replace the handmade <code>Self<\/code> in the code above with <code>from typing import Self<\/code>.<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-python\" data-lang=\"python\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">from<\/span> <span style=\"color:#79740e\">typing<\/span> <span style=\"color:#af3a03\">import<\/span> Self\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">class<\/span> Person:\n<\/span><\/span><span style=\"display:flex;\"><span>  <span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">name<\/span>(<span style=\"color:#b57614\">self<\/span>: Self): <span style=\"color:#af3a03\">...<\/span> <span style=\"color:#928374;font-style:italic\"># same as the previous code<\/span><\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<h2 class=\"heading\" id=\"what-is-the-type-of-self\">\n  What Is the Type of <code>Self<\/code>\n  <a class=\"anchor\" href=\"#what-is-the-type-of-self\">#<\/a>\n<\/h2>\n<p>Now, for a type checker, the question would be what the type of a variable should be when it&rsquo;s annotated with <code>Self<\/code>. Normally, you would check the definition for the symbol to see what it defines.<\/p>\n<p>If we look at the actual <a href=\"https:\/\/github.com\/python\/typeshed\/blob\/cb0fbd891336df9410d297de863fb3dd9731067c\/stdlib\/typing.pyi#L255\" rel=\"noopener\" target=\"_blank\">definition<\/a> of <code>Self<\/code> in typeshed, we find:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-py\" data-lang=\"py\"><span style=\"display:flex;\"><span>Self: _SpecialForm<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>What is a <code>_SpecialForm<\/code>?<\/p>\n<p>Special forms in typeshed are types that do not have a complete definition in the code; instead, the behavior of that type is defined by the typing spec. There are many types in Python like this: <code>Optional<\/code>, <code>Literal<\/code>, etc.<\/p>\n<p>So, based on the spec, the suggested strategy for implementing <code>Self<\/code> in a type checker would be:<\/p>\n<ol>\n<li>When you encounter <code>Self<\/code>, add a synthetic <code>TypeVar<\/code> above the usage.<\/li>\n<li>Replace <code>Self<\/code> with the new type the type checker just added.<\/li>\n<li>Use the new type for the rest of type checking.<\/li>\n<\/ol>\n<p>Also, there are <a href=\"https:\/\/typing.python.org\/en\/latest\/spec\/generics.html#valid-locations-for-self\" rel=\"noopener\" target=\"_blank\">invalid<\/a> uses of <code>Self<\/code>. They don&rsquo;t change how type checking happens, but they help with eliminating some edge cases.<\/p>\n<p>One question could be, why wouldn&rsquo;t we have a type in our type system called <code>Self<\/code> that contains what class it was used on so you can know what this type means during type checking?\nI guess this can be done, but one nice feature of changing <code>Self<\/code> into a <code>TypeVar<\/code> is that you don&rsquo;t need to touch the rest of the type checker. It will be handled by the same code that handles a <code>TypeVar<\/code> so there will be less code (and fewer bugs, because the logic would otherwise be duplicated).<\/p>\n<p>But implementing this in a type checker would be a few more steps than just this explanation.\nThat&rsquo;s why I love learning by making instead of just reading about it.<\/p>\n<h2 class=\"heading\" id=\"support-for-typingself-annotation\">\n  Support for <code>typing.Self<\/code> Annotation\n  <a class=\"anchor\" href=\"#support-for-typingself-annotation\">#<\/a>\n<\/h2>\n<p>This step is the easiest one: if the user annotates anything with <code>Self<\/code>, then perform the above replacement and return the correct type for <code>Self<\/code>.\nI did this in <a href=\"https:\/\/github.com\/astral-sh\/ruff\/pull\/17689\" rel=\"noopener\" target=\"_blank\">this PR<\/a>.<\/p>\n<h3 class=\"heading\" id=\"binding-self\">\n  Binding Self\n  <a class=\"anchor\" href=\"#binding-self\">#<\/a>\n<\/h3>\n<p>This looked simple to me at the time, but now after a few months of tinkering with the <code>Self<\/code> type support I know more edge cases.\nOne decision here is where we should bind <code>Self<\/code>.<\/p>\n<p>Binding a type variable means which class or function (or module) should contain the definition for the type variable. Maybe this is clearer with the new type parameter syntax defined in <a href=\"https:\/\/peps.python.org\/pep-0695\/\" rel=\"noopener\" target=\"_blank\">PEP 695<\/a>:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-python\" data-lang=\"python\"><span style=\"display:flex;\"><span><span style=\"color:#928374;font-style:italic\"># Option 1 bind to class<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">class<\/span> Person[Self: <span style=\"color:#79740e\">&#34;Person&#34;<\/span>]:\n<\/span><\/span><span style=\"display:flex;\"><span>  <span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">name<\/span>(<span style=\"color:#b57614\">self<\/span>: Self): <span style=\"color:#af3a03\">...<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#928374;font-style:italic\"># Option 2 bind to method<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">class<\/span> Person:\n<\/span><\/span><span style=\"display:flex;\"><span>  <span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">name<\/span>[Self: <span style=\"color:#79740e\">&#34;Person&#34;<\/span>](<span style=\"color:#b57614\">self<\/span>: Self): <span style=\"color:#af3a03\">...<\/span><\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>In the code above, first, the type variable is bound to the <code>Person<\/code> class, and the second example binds it to the method.<\/p>\n<p>The difference between these approaches is in situations where the user creates a subclass of the class. From <a href=\"https:\/\/discuss.python.org\/t\/unsoundness-of-contravariant-self-type\/86338\" rel=\"noopener\" target=\"_blank\">this discussion<\/a>:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-python\" data-lang=\"python\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">from<\/span> <span style=\"color:#79740e\">typing<\/span> <span style=\"color:#af3a03\">import<\/span> Self\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">class<\/span> Base:\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">foo<\/span>(<span style=\"color:#b57614\">self<\/span>, x: Self) <span style=\"color:#af3a03\">-&gt;<\/span> <span style=\"color:#af3a03\">None<\/span>:\n<\/span><\/span><span style=\"display:flex;\"><span>         <span style=\"color:#af3a03\">pass<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">class<\/span> Derived(Base):\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">foo<\/span>(<span style=\"color:#b57614\">self<\/span>, x: Self) <span style=\"color:#af3a03\">-&gt;<\/span> <span style=\"color:#af3a03\">None<\/span>:\n<\/span><\/span><span style=\"display:flex;\"><span>        x<span style=\"color:#af3a03\">.<\/span>bar()\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">bar<\/span>(<span style=\"color:#b57614\">self<\/span>) <span style=\"color:#af3a03\">-&gt;<\/span> <span style=\"color:#af3a03\">None<\/span>:\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">pass<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">test<\/span>(a: Base, b: Base) <span style=\"color:#af3a03\">-&gt;<\/span> <span style=\"color:#af3a03\">None<\/span>:\n<\/span><\/span><span style=\"display:flex;\"><span>    a<span style=\"color:#af3a03\">.<\/span>foo(b)\n<\/span><\/span><span style=\"display:flex;\"><span>test(Derived(), Base())<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>In this example, the method <code>foo<\/code> that redefines the <code>foo<\/code> method in <code>Base<\/code> does so in an incompatible manner. <code>Base.foo<\/code> considers <code>x<\/code> to be <code>Base<\/code>-like, but <code>Derived.foo<\/code> considers <code>x<\/code> to be <code>Derived<\/code>-like.<\/p>\n<p>If the <code>Self<\/code> type variable is bound to the method, it&rsquo;s more natural for the type checker to reject this override.\nBecause when checking the signatures we don&rsquo;t need extra information and it&rsquo;s clear that overriding something that accepted the base class with the child class violates the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Liskov_substitution_principle\" rel=\"noopener\" target=\"_blank\">Liskov Substitution Principle<\/a>.<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-python\" data-lang=\"python\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">class<\/span> Base:\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">foo<\/span>[Self: <span style=\"color:#79740e\">&#34;Base&#34;<\/span>](<span style=\"color:#b57614\">self<\/span>, x: Self) <span style=\"color:#af3a03\">-&gt;<\/span> <span style=\"color:#af3a03\">None<\/span>: <span style=\"color:#af3a03\">...<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">class<\/span> Derived(Base):\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">foo<\/span>[Self: <span style=\"color:#79740e\">&#34;Derived&#34;<\/span>](<span style=\"color:#b57614\">self<\/span>, x: Self) <span style=\"color:#af3a03\">-&gt;<\/span> <span style=\"color:#af3a03\">None<\/span>: <span style=\"color:#af3a03\">...<\/span><\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>It&rsquo;s not impossible to achieve the same behavior by binding <code>Self<\/code> to the class but this now makes it more trivial to detect problems like this. A type checker is full of exceptions and special cases already and being able to remove problems like this is a good thing.<\/p>\n<p>So that&rsquo;s how <a href=\"https:\/\/github.com\/astral-sh\/ruff\/pull\/20366\" rel=\"noopener\" target=\"_blank\">Ty binds the <code>Self<\/code> argument<\/a>.<\/p>\n<h2 class=\"heading\" id=\"implicit-self-type\">\n  Implicit <code>self<\/code> type\n  <a class=\"anchor\" href=\"#implicit-self-type\">#<\/a>\n<\/h2>\n<p>The next part of this change is unannotated <code>self<\/code> usage.\nMost Python code in the wild does not annotate the <code>self<\/code> argument with <code>typing.Self<\/code> so the <code>self<\/code> argument is considered to have this type annotation implicitly.<\/p>\n<p>For a type checker we need to assume that the following:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-python\" data-lang=\"python\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">class<\/span> A:\n<\/span><\/span><span style=\"display:flex;\"><span>  <span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">foo<\/span>(<span style=\"color:#b57614\">self<\/span>): <span style=\"color:#af3a03\">...<\/span><\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>is equivalent to:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-python\" data-lang=\"python\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">class<\/span> A:\n<\/span><\/span><span style=\"display:flex;\"><span>  <span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">foo<\/span>(<span style=\"color:#b57614\">self<\/span>: Self): <span style=\"color:#af3a03\">...<\/span><\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>So next I worked on adding this implicit annotation to the <code>self<\/code> argument.<\/p>\n<p>This implementation would touch two main areas of code:<\/p>\n<ul>\n<li>Uses of <code>self<\/code>: when you type <code>self<\/code> in method bodies<\/li>\n<li>Method calls: when you call a method on a class like <code>Foo().bar()<\/code><\/li>\n<\/ul>\n<h2 class=\"heading\" id=\"self-in-method-bodies\">\n  <code>self<\/code> in method bodies\n  <a class=\"anchor\" href=\"#self-in-method-bodies\">#<\/a>\n<\/h2>\n<p>Imagine the type checker wants to know what is the return type of <code>A.foo<\/code> in this code:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-python\" data-lang=\"python\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">class<\/span> A:\n<\/span><\/span><span style=\"display:flex;\"><span>  <span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">foo<\/span>(<span style=\"color:#b57614\">self<\/span>):\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">return<\/span> <span style=\"color:#b57614\">self<\/span><\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>Ty looks up <code>self<\/code> in the function scope, finds that it refers to the first argument.\nAfter performing a couple of checks (are we in a class? Is it a <code>classmethod<\/code> or <code>staticmethod<\/code>?)\nit can identify that we are pointing to the first argument of a normal method, which has <code>self<\/code>, so it considers the type annotation to be <code>typing.Self<\/code>.<\/p>\n<p>And that&rsquo;s the whole logic I implement in <a href=\"https:\/\/github.com\/astral-sh\/ruff\/pull\/18473\" rel=\"noopener\" target=\"_blank\">this PR<\/a>.<\/p>\n<p>But if you look at the scrollbar you see we are not finished yet and there is something awaiting us.\nThat&rsquo;s true.<\/p>\n<p>After adding this implicit annotation, Ty has to do a lot more work than in previous versions.\nBecause the type of <code>self<\/code> was previously unknown, it bypassed a lot of type-checking rules.<\/p>\n<p>After I made and sent the PR the CI identified a crash in the new code.\nTy has a nice CI.\nIn each PR it runs the new version of the type checker on a bunch of Python codebases using <a href=\"https:\/\/github.com\/hauntsaninja\/mypy_primer\" rel=\"noopener\" target=\"_blank\">mypy_primer<\/a> and compares the new Ty and old Ty diagnostics.\nThe idea is very similar to <a href=\"https:\/\/en.wikipedia.org\/wiki\/Fuzzing\" rel=\"noopener\" target=\"_blank\">fuzzing<\/a>.\nBy running the type checker on numerous codebases some bugs are surfaced that are hard to think of.<\/p>\n<p>This CI is also nice for checking impact of a new change across the Python ecosystem.\nYou can implement a new rule and see how many new diagnostics are we going to emit and are they correct.<\/p>\n<p>This stage of testing uncovered a couple of panics in my code.<\/p>\n<h3 class=\"heading\" id=\"salsa\">\n  Salsa\n  <a class=\"anchor\" href=\"#salsa\">#<\/a>\n<\/h3>\n<p>To get to what caused the panic we need to take a step back and look at Ty data structures for representing types.<\/p>\n<p>Ty uses a library called <a href=\"https:\/\/github.com\/salsa-rs\/salsa\" rel=\"noopener\" target=\"_blank\">salsa-rs<\/a>, which is an incremental recompilation tool.\nIn case of a type checker, it helps when you are checking multiple Python files and one of the files changes; you only need to check the changed file without recomputing types for the whole project.<\/p>\n<p>This is an example defined type using Salsa interned structs:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-rust\" data-lang=\"rust\"><span style=\"display:flex;\"><span><span style=\"color:#427b58\">#[salsa::interned(debug, heap_size=ruff_memory_usage::heap_size)]<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#427b58\">#[derive(PartialOrd, Ord)]<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">pub<\/span> <span style=\"color:#af3a03\">struct<\/span> ClassLiteral<span style=\"color:#af3a03\">&lt;<\/span><span style=\"color:#79740e;font-weight:bold\">&#39;db<\/span><span style=\"color:#af3a03\">&gt;<\/span> {\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#79740e\">\/\/\/ Name of the class at definition\n<\/span><\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#427b58\">#[returns(ref)]<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">pub<\/span>(<span style=\"color:#af3a03\">crate<\/span>) name: ast::name::Name,\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">pub<\/span>(<span style=\"color:#af3a03\">crate<\/span>) body_scope: ScopeId<span style=\"color:#af3a03\">&lt;<\/span><span style=\"color:#79740e;font-weight:bold\">&#39;db<\/span><span style=\"color:#af3a03\">&gt;<\/span>,\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">pub<\/span>(<span style=\"color:#af3a03\">crate<\/span>) known: <span style=\"color:#b57614\">Option<\/span><span style=\"color:#af3a03\">&lt;<\/span>KnownClass<span style=\"color:#af3a03\">&gt;<\/span>,\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#79740e\">\/\/\/ If this class is deprecated, this holds the deprecation message.\n<\/span><\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">pub<\/span>(<span style=\"color:#af3a03\">crate<\/span>) deprecated: <span style=\"color:#b57614\">Option<\/span><span style=\"color:#af3a03\">&lt;<\/span>DeprecatedInstance<span style=\"color:#af3a03\">&lt;<\/span><span style=\"color:#79740e;font-weight:bold\">&#39;db<\/span><span style=\"color:#af3a03\">&gt;&gt;<\/span>,\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">pub<\/span>(<span style=\"color:#af3a03\">crate<\/span>) dataclass_params: <span style=\"color:#b57614\">Option<\/span><span style=\"color:#af3a03\">&lt;<\/span>DataclassParams<span style=\"color:#af3a03\">&gt;<\/span>,\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">pub<\/span>(<span style=\"color:#af3a03\">crate<\/span>) dataclass_transformer_params: <span style=\"color:#b57614\">Option<\/span><span style=\"color:#af3a03\">&lt;<\/span>DataclassTransformerParams<span style=\"color:#af3a03\">&gt;<\/span>,\n<\/span><\/span><span style=\"display:flex;\"><span>}<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p><a href=\"https:\/\/github.com\/astral-sh\/ruff\/blob\/59c8fda3f8f3bf2cc5c1ae34e7ca9dbea4d0278f\/crates\/ty_python_semantic\/src\/types\/class.rs#L1331\" rel=\"noopener\" target=\"_blank\">source<\/a><\/p>\n<p>Notice the <code>salsa::interned<\/code> usage and <code>db<\/code> lifetime.<\/p>\n<p>Interning a struct means when creating an instance of this struct Salsa will store the value and give you an ID; you can later use this ID and query the Salsa DB to get back the object.\nSimilar to how pointers work, but in this case this is an ID, and I think it helps with making the type serializable so you can do incremental computation nicely.<\/p>\n<p>Also, having IDs is nicer to work with because you can pass values by copying and have fewer issues with the borrow checker.<\/p>\n<p>Other than incremental recompilation, Salsa provides another awesome feature for type checkers, <a href=\"https:\/\/salsa-rs.github.io\/salsa\/cycles.html\" rel=\"noopener\" target=\"_blank\">cycle handling<\/a>.<\/p>\n<p>Let&rsquo;s start with a simple example where this would be useful.\nThis was the first infinite recursion bug I faced while type checking the Python standard library:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-python\" data-lang=\"python\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">class<\/span> str(Sequence[<span style=\"color:#b57614\">str<\/span>]):\n<\/span><\/span><span style=\"display:flex;\"><span>  <span style=\"color:#af3a03\">...<\/span><\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p><a href=\"https:\/\/github.com\/python\/typeshed\/blob\/cb0fbd891336df9410d297de863fb3dd9731067c\/stdlib\/builtins.pyi#L476\" rel=\"noopener\" target=\"_blank\">source<\/a><\/p>\n<p>Let&rsquo;s see this code through the lens of a type checker.\nThere is a function like this:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-rust\" data-lang=\"rust\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">fn<\/span> <span style=\"color:#b57614\">infer_class_type<\/span>(class_def: ClassDefinition) -&gt; ClassType:\n<\/span><\/span><span style=\"display:flex;\"><span>  <span style=\"color:#928374;font-style:italic\">\/\/ We definitely need to check the base classes of this class to find what its type is\n<\/span><\/span><\/span><span style=\"display:flex;\"><span>  <span style=\"color:#af3a03\">for<\/span> base <span style=\"color:#af3a03\">in<\/span> class_def.bases:\n<\/span><\/span><span style=\"display:flex;\"><span>    let base_type <span style=\"color:#af3a03\">=<\/span> infer_class_type(base);\n<\/span><\/span><span style=\"display:flex;\"><span>  <span style=\"color:#928374;font-style:italic\">\/\/ rest of the code\n<\/span><\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>This is how the above code will have infinite recursion:<\/p>\n<ol>\n<li>Try to infer type of <code>Str<\/code><\/li>\n<li>Infer type of <code>Sequence[Str]<\/code> base to know what things <code>Str<\/code> inherits from the base<\/li>\n<li><code>Sequence<\/code> has a generic parameter and here we are using <code>Str<\/code> to figure out what the type of <code>Sequence[Str]<\/code> is we need to infer <code>Str<\/code><\/li>\n<li>We are back to step 1<\/li>\n<\/ol>\n<p>This is called a cycle, and we need a mechanism to stop it. The good news is that Salsa has a cycle resolution algorithm. It works based on fixed-point iteration.<\/p>\n<p>What happens in this case is that at step 3, when we are going back to step 1, Salsa can detect this (because it tracks the function calls) and Ty in this case can provide a cycle fallback value.\nThis will tell salsa to consider <code>Str<\/code> as <code>Any<\/code> and finish the recursive calls.\nThis way we end up with some type for <code>Str<\/code>. Is this enough? No.\nNext is that we repeat this process now with the type we have for <code>Str<\/code> and at some point this should converge.\nMeaning that determining the type again with the value we got from the previous process should give the same type.\nThis way you can be sure the outcome type is final.<\/p>\n<h3 class=\"heading\" id=\"cycle-panic-accumulating-literals\">\n  Cycle Panic: Accumulating Literals\n  <a class=\"anchor\" href=\"#cycle-panic-accumulating-literals\">#<\/a>\n<\/h3>\n<p>So now we can go back to the implicit self annotation and explore the panic that happened.\nThe code that caused a panic could be reduced to:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-python\" data-lang=\"python\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">class<\/span> A:\n<\/span><\/span><span style=\"display:flex;\"><span>  <span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">__init__<\/span>(<span style=\"color:#b57614\">self<\/span>):\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>n <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#8f3f71\">1<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>    \n<\/span><\/span><span style=\"display:flex;\"><span>  <span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">incr<\/span>(<span style=\"color:#b57614\">self<\/span>):\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>n <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>n <span style=\"color:#af3a03\">+<\/span> <span style=\"color:#8f3f71\">1<\/span><\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>This example is a bit more complicated to understand why there is infinite recursion.<\/p>\n<p>First Ty tries to be very precise in the returned type.\nWhile you can say that type of <code>n<\/code> is an integer.\nTy considers that <code>n<\/code> could be <code>Literal[1 | 2 | ...]<\/code> It knows that the <code>Literal<\/code> type can be expanded many times because it does not precisely know how many times <code>incr<\/code> might be called.\nTy tries to infer <code>self.n<\/code> and it sees that it requires to know about <code>self.n<\/code>.\n<code>self.n<\/code> is initially 1 so then <code>self.n<\/code> will be 2. Now with the value of 2 we infer <code>self.n<\/code> again (because of the fixed-point iteration algorithm from previous section.)\nThis process happens for 200 times and Salsa panics because the cycle did not converge.<\/p>\n<p>The fix for this would be easy, we just need an upper bound on how long we want to consider <code>self.n<\/code> a literal value before it becomes an integer.\nIf we set this upper limit to anything lower than 200 then in one of the iterations Salsa will see that <code>self.n<\/code> had type <code>int<\/code> and the result is also <code>int<\/code> so the cycle stops.<\/p>\n<h3 class=\"heading\" id=\"cycle-panic-divergent-value\">\n  Cycle Panic: Divergent Value\n  <a class=\"anchor\" href=\"#cycle-panic-divergent-value\">#<\/a>\n<\/h3>\n<p>After fixing this I found <a href=\"https:\/\/github.com\/astral-sh\/ty\/issues\/692\" rel=\"noopener\" target=\"_blank\">another panic<\/a> happening. This panic has a different nature.\nThis is a reduced version of the original code that reproduces the issue:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-python\" data-lang=\"python\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">from<\/span> <span style=\"color:#79740e\">typing<\/span> <span style=\"color:#af3a03\">import<\/span> Literal\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">class<\/span> Toggle:\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">__init__<\/span>(<span style=\"color:#b57614\">self<\/span>: <span style=\"color:#79740e\">&#34;Toggle&#34;<\/span>):\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">if<\/span> <span style=\"color:#af3a03\">not<\/span> <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>x:\n<\/span><\/span><span style=\"display:flex;\"><span>            <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>x: Literal[<span style=\"color:#af3a03\">True<\/span>] <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#af3a03\">True<\/span><\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>In this strange-looking code the definition of <code>self.x<\/code> is under an if statement that checks for <code>self.x<\/code> so this is an access before defining the value.\nFor the type checker it does not matter if the code is invalid or not. It will type-check it to provide feedback for the user.<\/p>\n<p>Now let&rsquo;s see through the lens of the type checker what happens when inferring the type of <code>self.x<\/code>:<\/p>\n<ol>\n<li>This attribute is guarded by an if statement, we first need to check if the condition is met then the value is True<\/li>\n<li>Condition is <code>self.x<\/code> so we need to infer type of <code>self.x<\/code><\/li>\n<li>Salsa cycle recovery assumes the type of <code>self.x<\/code> is unknown and the if would be false<\/li>\n<li>Type of <code>self.x<\/code> is <code>Literal[True]<\/code> now.<\/li>\n<li>To complete the fixed point iteration now infer type of <code>self.x<\/code> again with the determined type<\/li>\n<li>Type of <code>self.x<\/code> is unknown because <code>not self.x<\/code> is false and the if does not execute so <code>self.x<\/code> would not be defined<\/li>\n<li>The cycle recovery goes back to step 5 now with the type unknown and it repeats\nIn the end, the type of <code>self.x<\/code> will alternate between unknown and <code>Literal[True]<\/code>.\nBut it will never be one type.<\/li>\n<\/ol>\n<p>This example is also nice because it highlights why convergence for the cycles is important.\nIf we simply substitute a value for <code>self.x<\/code> when there is a cycle and conclude we are not seeing all the possible cases.\nWe either see <code>Literal[True]<\/code> type or unbound value and a runtime error case.\nAnd in this case seeing the <code>Literal[True]<\/code> sounds like a better idea because the <code>self.x<\/code> might be set by the user, outside of the function.\nIt&rsquo;s hard to decide these.\nBecause the Python code could run without a runtime error but type-checking would be more complex.<\/p>\n<p><a href=\"https:\/\/github.com\/sharkdp\" rel=\"noopener\" target=\"_blank\">David<\/a> suggested <a href=\"https:\/\/github.com\/astral-sh\/ruff\/pull\/19579\" rel=\"noopener\" target=\"_blank\">a solution<\/a> that fixed this issue.<\/p>\n<p>The solution here is to disable the reachability analysis when an attribute is not defined.\nThis means that when Ty is checking the code above instead of trying to see if the <code>not self.x<\/code> is true or not it considers that it is either true or false.\nThen based on if we consider this ambiguous value true or false to run the if or not.<\/p>\n<p>The solution involves keeping track of all the names and attribute accesses that happen when type checking an expression and returning a boolean to indicate if all subexpressions are definitely defined.\nDefinitely defined means it won&rsquo;t cause a runtime error when you use that expression.\nIn the case above if we don&rsquo;t assign <code>x<\/code> to the instance before calling the method it results in a runtime error.<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-rust\" data-lang=\"rust\"><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">let<\/span> inference <span style=\"color:#af3a03\">=<\/span> infer_expression_types(db, expression);\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">if<\/span> <span style=\"color:#af3a03\">!<\/span>inference.all_places_definitely_bound() {\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">return<\/span> Truthiness::Ambiguous;\n<\/span><\/span><span style=\"display:flex;\"><span>    }\n<\/span><\/span><span style=\"display:flex;\"><span>  <span style=\"color:#928374;font-style:italic\">\/\/ if it is bound then check the truthiness of the inferred type\n<\/span><\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<h3 class=\"heading\" id=\"performance-regression-1\">\n  Performance Regression 1\n  <a class=\"anchor\" href=\"#performance-regression-1\">#<\/a>\n<\/h3>\n<p><a href=\"https:\/\/github.com\/astral-sh\/ty\/issues\/758\" rel=\"noopener\" target=\"_blank\">https:\/\/github.com\/astral-sh\/ty\/issues\/758<\/a><\/p>\n<p>The next problem that I found using <code>mypy_primer<\/code> was that the code was running slow.\nThe result was scary:<\/p>\n<div class=\"code-block\">\n  <pre tabindex=\"0\"><code>\/tmp\/mypy_primer\/ty_old\/target\/debug\/ty on mongo-python-driver took 1.96s\n\/tmp\/mypy_primer\/ty_new\/target\/debug\/ty on mongo-python-driver took 98.62s<\/code><\/pre>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>The new version of the code is almost 50 times slower.<\/p>\n<p>Since this was not the only one, I used <a href=\"https:\/\/gist.github.com\/Glyphack\/6f430f90c3c28954f89216c7b87b61d4\" rel=\"noopener\" target=\"_blank\">this Python script<\/a> to list projects that ran slow in type checking or never finished.<\/p>\n<p>This case was slow was because of attribute assignments guarded by themselves.<\/p>\n<p>The guard means more than only if statements, for example in this code:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-python\" data-lang=\"python\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">class<\/span> A:\n<\/span><\/span><span style=\"display:flex;\"><span>  <span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">a<\/span>(<span style=\"color:#b57614\">self<\/span>):\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>foo <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#af3a03\">False<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>bar <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#af3a03\">False<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>    \n<\/span><\/span><span style=\"display:flex;\"><span>  <span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">f<\/span>(<span style=\"color:#b57614\">self<\/span>):\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">if<\/span> <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>foo:\n<\/span><\/span><span style=\"display:flex;\"><span>      <span style=\"color:#af3a03\">return<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">if<\/span> <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>bar:\n<\/span><\/span><span style=\"display:flex;\"><span>      <span style=\"color:#af3a03\">raise<\/span> <span style=\"color:#fb4934\">Exception<\/span>()\n<\/span><\/span><span style=\"display:flex;\"><span>    \n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>baz <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#8f3f71\">1<\/span><\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>For the type checker the <code>self.baz<\/code> is defined if <code>self.bar<\/code> and <code>self.foo<\/code> are false. So its definition is guarded by those checks.\nThis is what reachability analyzer does in the type checker to find what is reachable and thus defined and what is not.<\/p>\n<p>Also note that <code>self.baz<\/code> is an implicit attribute. Ty does not know if <code>A.a<\/code> is always called before <code>A.f<\/code>. So it needs to consider both it might or might not be defined. Usually type checkers don&rsquo;t emit diagnostic for accessing these attributes. A lot of Python code is written like this.<\/p>\n<p>Now getting back to the problem. If <code>baz<\/code> is guarded by <code>bar<\/code> and <code>foo<\/code> if we add another line to make <code>foo<\/code> and <code>bar<\/code> dependant on <code>baz<\/code> then we have a cycle and we get closer to the <a href=\"https:\/\/github.com\/astral-sh\/ty\/issues\/758\" rel=\"noopener\" target=\"_blank\">problem<\/a> I faced.<\/p>\n<p>It&rsquo;s not hard to identify there&rsquo;s a long cycle happening. Salsa has traces so by running Ty with <code>-vvv<\/code> we can get the traces and it looks something like this:<\/p>\n<div class=\"code-block\">\n  <pre tabindex=\"0\"><code>TRACE ty_project::db: Salsa event: Event { thread_id: ThreadId(2), kind: WillIterateCycle { database_key: member_lookup_with_policy_(Id(f4e2)), iteration_count: IterationCount(1), fell_back: false } }<\/code><\/pre>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>To decode what the Id <code>f4e2<\/code> is referring to we just need to check rest of the traces. Find somewhere the Id is mentioned and try to map it to source code.<\/p>\n<p>I wrote so many <code>eprintln!<\/code>s in the code to figure out what is going wrong.<\/p>\n<p>And when implementing the fix in the previous topic, I was benchmarking this as well and the cycle count was reducing with the fix here too, but it was not enough to make it fast.<\/p>\n<p>To improve the performance here, David found <a href=\"https:\/\/github.com\/astral-sh\/ruff\/pull\/20128\" rel=\"noopener\" target=\"_blank\">a solution<\/a> to disable checking if an implicit instance attribute is bound or not.<\/p>\n<p>Here we don&rsquo;t want to give an error if user tries to access <code>A.foo<\/code> because the method <code>A.a<\/code> might be called before and the attribute actually exists.\nThese are called implicit attributes that are not defined in the <code>__init__<\/code> method and instead are implicitly added to the instance in other ways.<\/p>\n<h3 class=\"heading\" id=\"performance-regression-2\">\n  Performance Regression 2\n  <a class=\"anchor\" href=\"#performance-regression-2\">#<\/a>\n<\/h3>\n<p>The <a href=\"https:\/\/github.com\/astral-sh\/ty\/issues\/1111\" rel=\"noopener\" target=\"_blank\">next problem<\/a> was found when type checking the SymPy codebase. The problem is not specific to SymPy though. It also happened on other repositories.<\/p>\n<p>The problem happens when there are attributes which all depend on each other for values.\nThe reason is unknown to me. I think it&rsquo;s a little bit related to the previous performance regression.\nSomehow when these attributes depend on each other when A needs to be inferred we need B and for B we need C and for C we need A and B so cycle counts start increasing.\nThis means Ty spends a lot of time iterating to find a value for A, B, and C but it needs <em>a lot<\/em> of iterations to converge.<\/p>\n<p>This was a big problem, it meant that type checking on some packages took a long time.\nAnd I could not fix this issue.\nEven with the previous one I spent too many days adding prints to the code to understand what is happening and understand flow of the code.\nBut this one was harder than that.\nI had to leave it to Astral team to figure this part out.<\/p>\n<p>The fix isn&rsquo;t there yet.\nWe&rsquo;ll get there eventually and the fix for this one might be in the <a href=\"https:\/\/github.com\/salsa-rs\/salsa\/issues\/841\" rel=\"noopener\" target=\"_blank\">Salsa library<\/a>.\nIt would have been fun if I had the time to investigate this myself and learn more about Salsa.<\/p>\n<h2 class=\"heading\" id=\"self-in-method-calls\">\n  <code>self<\/code> in method calls\n  <a class=\"anchor\" href=\"#self-in-method-calls\">#<\/a>\n<\/h2>\n<p>Why does inferring <code>self<\/code> in method calls require a different solution than the one above?<\/p>\n<p>Consider the following examples:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-python\" data-lang=\"python\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">class<\/span> A:\n<\/span><\/span><span style=\"display:flex;\"><span>  <span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">foo<\/span>(<span style=\"color:#b57614\">self<\/span>):\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#b57614\">print<\/span>(<span style=\"color:#b57614\">self<\/span>) <span style=\"color:#928374;font-style:italic\"># the type checker sees self and tries to find it in the scope.<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>a <span style=\"color:#af3a03\">=<\/span> A()\n<\/span><\/span><span style=\"display:flex;\"><span>a<span style=\"color:#af3a03\">.<\/span>foo() <span style=\"color:#928374;font-style:italic\"># Here Python implicitly passes `a` as the first argument of `foo`. So that&#39;s what the type checker does as well.<\/span><\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>In the first part of the code when we refer to <code>self<\/code> in the method body the type checker finds that variable and sees that it&rsquo;s a method argument. Then the code that we talked in previous section takes care of assigning its type implicitly.<\/p>\n<p>In the second part of the code when we call any method on a class we are passing the instance as the value for the <code>self<\/code> argument, even though it&rsquo;s not passed in the parentheses.\nYou can check this by doing <code>A.foo(A())<\/code> and see that it&rsquo;s just syntactic sugar for it.<\/p>\n<p>In the second example Ty checks whether you are passing valid arguments to the method, i.e., whether <code>a<\/code> is assignable to <code>self<\/code>.<\/p>\n<p>This requires two different codes to handle the type checking.\nI think the real reason is because of generics.\nWhen we are inferring types in a method body the function or class is not specialized yet.\nMeaning that we don&rsquo;t know what type parameters are referring to (<code>typing.Self<\/code> or others.)\nBut when a call is happening the value is specialized, so it makes sense that we need some separate code to handle these.<\/p>\n<p>When functions are called Ty <a href=\"https:\/\/github.com\/astral-sh\/ruff\/blob\/59c8fda3f8f3bf2cc5c1ae34e7ca9dbea4d0278f\/crates\/ty_python_semantic\/src\/types\/signatures.rs#L1156\" rel=\"noopener\" target=\"_blank\">makes a signature<\/a> for the function and tries to match the passed arguments with defined ones.<\/p>\n<p>And here I needed to add the extra code that <a href=\"https:\/\/github.com\/astral-sh\/ruff\/pull\/18007\" rel=\"noopener\" target=\"_blank\">would add the synthetic type annotation<\/a> to <code>self<\/code>.<\/p>\n<p>With these kinds of changes you expect some performance regressions.\nBecause previously a type was unknown and a lot of type checking rules did not apply but now they are.\nBut along the way we found one speed improvement that could also simplify the type inference.\nThe trick was to annotate the <code>self<\/code> with the class name if the function is not already generic:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-python\" data-lang=\"python\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">class<\/span> A:\n<\/span><\/span><span style=\"display:flex;\"><span>  <span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">foo<\/span>(<span style=\"color:#b57614\">self<\/span>: <span style=\"color:#79740e\">&#34;A&#34;<\/span>): <span style=\"color:#af3a03\">...<\/span><\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>Here annotating <code>self<\/code> with <code>&quot;A&quot;<\/code> or the type variable version would not make any difference.\nAnd since the method is not generic we don&rsquo;t make it generic only because it has <code>self<\/code>.\nRemember, we bind the <code>Self<\/code> type variable to the method, so the method that contains <code>self<\/code> has a type variable, and is therefore generic.\nWhen a method is generic on every use we need to solve the generic parameters.\nMeaning to check how the method is used and replace an example <code>T<\/code> generic parameter with what it should be at the use.\nAnd this is redundant here, so I tried to apply this, and the result was a couple hundred milliseconds faster when type checking the pandas package. It had the biggest difference around 500 milliseconds.<\/p>\n<p>The credit for this goes to <a href=\"https:\/\/github.com\/carljm\" rel=\"noopener\" target=\"_blank\">Carl<\/a>.<\/p>\n<p>As you can see in the PR above this change surfaced some issues around handling generics.\nSince previously the <code>self<\/code> type was unknown a lot of assignability issues were not reported.<\/p>\n<p>But there was one way to make progress on this PR easier. Since these issues could be reproduced on main.\nI could file issues for them (here is <a href=\"https:\/\/github.com\/astral-sh\/ty\/issues\/1131\" rel=\"noopener\" target=\"_blank\">an example<\/a>) to fix them in a separate PR.\nDavid helped me a lot with resolving these issues. So sadly I don&rsquo;t have more information to share about the issues.<\/p>\n<h2 class=\"heading\" id=\"future-work\">\n  Future Work\n  <a class=\"anchor\" href=\"#future-work\">#<\/a>\n<\/h2>\n<p>As of now, the support for implicit <code>self<\/code> type is not in Ty yet.\nThere are some improvements to add to have a correct implementation that works on different Python codebases.\nProbably in a couple of weeks, it&rsquo;s going to land in Ty.\nI spent time on this feature and the challenge made me happy.\nI&rsquo;m glad to see it&rsquo;s getting close.<\/p>\n<h2 class=\"heading\" id=\"acknowledgments\">\n  Acknowledgments\n  <a class=\"anchor\" href=\"#acknowledgments\">#<\/a>\n<\/h2>\n<p>What I implemented here was mostly guided by the Astral team. If you count overall their contribution was far more than me on this topic.<\/p>\n<p>David helped implement fixes for false-positives and panics related to implicit <code>self<\/code> to help me make progress.<\/p>\n<p>Carl provided a lot of guidance for the implementation.<\/p>\n<p>Douglas helped me a lot with generics and. He <a href=\"https:\/\/github.com\/astral-sh\/ruff\/pull\/19604\" rel=\"noopener\" target=\"_blank\">solved<\/a> the first blocker for supporting <code>self<\/code> in method calls.<\/p>\n"},{"title":"The best lasagna I've ever had","link":"https:\/\/glyphack.com\/lasagna\/","pubDate":"Sat, 12 Jul 2025 00:16:52 +0200","guid":"https:\/\/glyphack.com\/lasagna\/","description":"<p>I went to Italy in March with my friends from first grade.\nThere are a lot of different places to see in Rome, Vatican Palace, Pantheon, and Colosseum.\nBut for me there was something else I missed and wanted to see again.\nIt was <a href=\"https:\/\/maps.app.goo.gl\/WYi3bWYaKYAiWC5N7\" rel=\"noopener\" target=\"_blank\">Borghiciana Pastificio<\/a> where I had the most delicious lasagna of my life<sup id=\"fnref:1\"><a href=\"#fn:1\" class=\"footnote-ref\" role=\"doc-noteref\">1<\/a><\/sup>.<\/p>\n<p>The restaurant is small compared to others.\nOnly about 16 people can sit inside so there&rsquo;s a line in front of it all the time.\nYou order while waiting and when you get there the food is ready.\nIt feels unpretentious and welcoming.\nThe waitress was delighted to see me again.\nI didn&rsquo;t expect that given how many people they see in a year.<\/p>\n<p>The owner work as other people in the restaurant.\nFrom cooking in the kitchen to serving tables.\nHe also pours you a drink if you compliment the food.\nHe sat at our table and started chatting about how he started the place.\nHe said he\u2019s run the restaurant for seven years.\nHasn&rsquo;t expanded even though the restaurant got more famous in recent years.\nHis success stems from reviews.<\/p>\n<p>But he also mentioned that this fame sometimes backlashes.\nSome arrive with high expectations but are deterred by long lines and the small space.\nHis vision is to give delicious food to people and make them happy.<\/p>\n<p>This reminded me of the idea that you can win without <a href=\"https:\/\/world.hey.com\/jason\/an-alternative-to-competition-ff57f4bc\" rel=\"noopener\" target=\"_blank\">competing<\/a> with everyone.\nThese days I have more respect for products and people who try to make something good than the ones trying to collect scores.<\/p>\n<p>I don&rsquo;t think anyone who tries to impress by some score sells a bad product.\nBut it is related. It increases the chances because it can distract you from the actual product and diverge the focus on other things.<\/p>\n<p>\n\n\n\n\n\n\n\n<figure class=\"\">\n\n    \n    \n      \n        \n        <div class=\"img-container\" style=\"--w: 1280; --h: 960;\">\n            <img loading=\"lazy\" alt=\"\" src=\"https:\/\/glyphack.com\/lasagna\/borghiciana_hu_d8dba6de1d7c7dd8.jpg\" width=\"1280\" height=\"960\">\n        <\/div>\n      \n    \n\n    \n<\/figure>\n<\/p>\n<div class=\"footnotes\" role=\"doc-endnotes\">\n<hr>\n<ol>\n<li id=\"fn:1\">\n<p>Also try out <a href=\"https:\/\/maps.app.goo.gl\/FKEbRmEcpfhv9NCd8\" rel=\"noopener\" target=\"_blank\">Gelateria La Romana<\/a> if you end up eating here.&#160;<a href=\"#fnref:1\" class=\"footnote-backref\" role=\"doc-backlink\">&#x21a9;&#xfe0e;<\/a><\/p>\n<\/li>\n<\/ol>\n<\/div>\n"},{"title":"I Made ReadsThis to Share and Find Good Blogs","link":"https:\/\/glyphack.com\/rt\/","pubDate":"Sun, 20 Apr 2025 13:20:54 +0200","guid":"https:\/\/glyphack.com\/rt\/","description":"<p>I created a site to share your favorite RSS Feeds in a page.\nThe idea comes from one of my favorite sites <a href=\"https:\/\/usesthis.com\" rel=\"noopener\" target=\"_blank\">usesthis.com<\/a>.<\/p>\n<p><a href=\"https:\/\/r.glyphack.com\" rel=\"noopener\" target=\"_blank\">Try it out<\/a> and send me your links.\nI love to know interesting blogs around the internet. Here is <a href=\"https:\/\/r.glyphack.com\/s\/s\" rel=\"noopener\" target=\"_blank\">mine<\/a>.<\/p>\n<p>I spend great time reading stuff on the internet. I started this in high school.\nI think it&rsquo;s getting harder everyday to find good content.\nFor me a good writing is the one with thoughts behind it.\nWhich is hard to get in the world that everyone is fighting for your attention.<\/p>\n<p>So I started collecting blogs that I like. I decided that I&rsquo;m gonna be reading only those posts.\nI limited the number of posts I see and I can choose from to read. But the quality went higher.\nFrom time to time, I share these blogs with my friends. They share blogs with me too. And I love it.\nNow I have a collection that I really like. I might add more or less stuff to them but it&rsquo;s almost enough for my entire life.<\/p>\n<p>So I created this website where we can share and find interesting blogs.\nIt&rsquo;s essentially, a recommendation system created by people.<\/p>\n"},{"title":"Devlog 4: I made a chrome extension","link":"https:\/\/glyphack.com\/dv-4\/","pubDate":"Sat, 12 Apr 2025 11:49:07 +0200","guid":"https:\/\/glyphack.com\/dv-4\/","description":"<p>I&rsquo;m not sure if some time comes that I finally can say I know vim. But I don&rsquo;t think that would happen before I <a href=\"https:\/\/www.youtube.com\/watch?v=rT-fbLFOCy0\" rel=\"noopener\" target=\"_blank\">read the whole manual<\/a>.<\/p>\n<p>This week I learned these new commands:<\/p>\n<p>I&rsquo;ve always used <code>ctrl-o<\/code> to move to last place I was editing in vim. Turns out there&rsquo;s a keybinding to move to the previous file that was open it&rsquo;s  <strong><code>Ctrl-^<\/code>\u00a0(or\u00a0<code>Ctrl-6<\/code>)<\/strong>.\nThis command switches to the\u00a0last visited file.\nIt provides quick toggling between two files if you do it repeatedly.<\/p>\n<p><a href=\"https:\/\/vim.fandom.com\/wiki\/Power_of_g\" rel=\"noopener\" target=\"_blank\"><code>:g<\/code><\/a> is powerful. To move\/delete things that have a pattern.\nYou are probably familiar with <code>gj<\/code> and <code>gk<\/code> but there&rsquo;s also <code>gq<\/code> (or <code>gw<\/code> if the other did not work.) <code>gw<\/code> helps to split a long line into smaller lines.<\/p>\n<p>Other things:<\/p>\n<ul>\n<li>I made a chrome extension, <a href=\"https:\/\/chromewebstore.google.com\/detail\/readwise-reader-importer\/biaidjfcmkeeiidenndhkdaldkljaipi?authuser=1&amp;hl=en\" rel=\"noopener\" target=\"_blank\">Readwise Reader Importer<\/a> to import links into Readwise. I wanted this tool myself for importing youtube playlists to Readwise. This time I decided to build it as an extension so I can share with others. It got 10 users! I did most of the work using Claude code. It costed around 10 euros, but I&rsquo;m happy with the resulting look.<\/li>\n<li>I moved <a href=\"https:\/\/glyphack.com\/reading-list\/\" rel=\"noopener\" target=\"_blank\">my reading list<\/a> off Notion and to my blog. I was tired of me entering books I read and notes about them in Obsidian and the Notion was rarely updated. So I found a <a href=\"https:\/\/glyphack.com\/ob\/\" rel=\"noopener\" target=\"_blank\">solution<\/a> to update my blog with Obsidian, and then show the book notes just as any other page on my blog. The bonus is that I can make it more beautiful in the future.<\/li>\n<li>I&rsquo;ve heard about this language called ungrammar that is used in Rust to generate CSTs. I did not know anything about it. Thanks to this <a href=\"https:\/\/github.com\/astral-sh\/ruff\/issues\/15655\" rel=\"noopener\" target=\"_blank\">issue in Ruff<\/a> , I did some work related to generating AST. There&rsquo;s <a href=\"https:\/\/www.youtube.com\/watch?v=EIXb9mX_o9s\" rel=\"noopener\" target=\"_blank\">this nice<\/a> video that explains how it&rsquo;s used in the Rust Analyzer code.<\/li>\n<li>I implemented <a href=\"https:\/\/redis.io\/docs\/latest\/develop\/data-types\/streams\/\" rel=\"noopener\" target=\"_blank\">Redis streams<\/a> in toy <a href=\"https:\/\/github.com\/Glyphack\/redis-clone\" rel=\"noopener\" target=\"_blank\">clone of Redis<\/a>. I took some time doing this, I wanted to use an array instead of a linked list. This would make a good candidate for a blog post so I won&rsquo;t go into details. Redis uses <a href=\"https:\/\/en.wikipedia.org\/wiki\/Radix_tree#:~:text=In%20computer%20science%2C%20a%20radix,is%20merged%20with%20its%20parent.\" rel=\"noopener\" target=\"_blank\">radix tree<\/a> to implement streams(according to AI)<\/li>\n<li>I read <a href=\"https:\/\/sive.rs\/su\" rel=\"noopener\" target=\"_blank\">https:\/\/sive.rs\/su<\/a> and decided to keep my own URLs shorter too.<\/li>\n<\/ul>\n<p>That&rsquo;s it for this week.<\/p>\n"},{"title":"Blogging with Obsidian","link":"https:\/\/glyphack.com\/ob\/","pubDate":"Wed, 09 Apr 2025 21:25:18 +0200","guid":"https:\/\/glyphack.com\/ob\/","description":"<p>My blog is a bunch of files that are given to a program that converts them to HTML and CSS files, which is what you see here.<\/p>\n<p>Also, Obsidian uses files to store my notes. This means it gives the most freedom than any other tool out there.<\/p>\n<p>Since I&rsquo;ve been writing both on my blog and in Obsidian there were times that I wanted to share stuff outside my Obsidian and link it in my website.\nOne way to do this is using Obsidian Publish, but after reading the <a href=\"https:\/\/www.reddit.com\/r\/ObsidianMD\/comments\/16e5jek\/comment\/jzv38ja\/?utm_source=share&amp;utm_medium=web3x&amp;utm_name=web3xcss&amp;utm_term=1&amp;utm_content=share_button\" rel=\"noopener\" target=\"_blank\">creator of Obsidian himself suggesting alternatives to it<\/a> I searched for ways to publish my notes on my blog.<\/p>\n<p>There are great plugins that make this thing called digital garden (basically a website that you can browse) from Obsidian vault:<\/p>\n<ul>\n<li><a href=\"http:\/\/github.com\/oleeskild\/obsidian-digital-garden\" rel=\"noopener\" target=\"_blank\">http:\/\/github.com\/oleeskild\/obsidian-digital-garden<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/jackyzha0\/quartz\" rel=\"noopener\" target=\"_blank\">https:\/\/github.com\/jackyzha0\/quartz<\/a><\/li>\n<\/ul>\n<p>While they are good I was looking for something that I can use to generate the site myself.\nSo I can have my notes in the same theme as my blog, so it looks like the same page.<\/p>\n<p>That&rsquo;s how I found <a href=\"https:\/\/github.com\/Enveloppe\/obsidian-enveloppe\" rel=\"noopener\" target=\"_blank\">Obsidian Enveloppe<\/a>, a plugin that helps you push files from Obsidian to Github.\nI&rsquo;m satisfied with the tool, it does the job with little complexity and composes well with user workflow.<\/p>\n<p>So as a first step I decided to give it a shot, moving my reading list off Notion to my Blog.<\/p>\n<p>This will allow me to use Obsidian as my editing tool and my Blog as the publishing tool.<\/p>\n<h2 class=\"heading\" id=\"my-setup\">\n  My setup\n  <a class=\"anchor\" href=\"#my-setup\">#<\/a>\n<\/h2>\n<p>I create notes for books I read with my notes for the book.\nThese notes have special <a href=\"https:\/\/gohugo.io\/content-management\/front-matter\/\" rel=\"noopener\" target=\"_blank\">front matter<\/a> <code>category: [Books]<\/code> to specify it&rsquo;s a book.<\/p>\n<p>Then I publish these notes to my blog repository in a special folder that I use to sync my notes. This sync is a one way sync. I&rsquo;m not going to update these notes on Blog anymore.<\/p>\n<p>The published notes will appear in my blog by default. Since Hugo just picks up all the markdown files and turns them into a page.<\/p>\n<p>But these pages are not linked by default since they are not blog entries.\nSo I created this HTML page, that lists all the notes with &ldquo;Books&rdquo; category:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-html\" data-lang=\"html\"><span style=\"display:flex;\"><span>{{ define &#34;main&#34; }}\n<\/span><\/span><span style=\"display:flex;\"><span>&lt;<span style=\"color:#9d0006\">article<\/span> <span style=\"color:#79740e;font-weight:bold\">class<\/span><span style=\"color:#af3a03\">=<\/span><span style=\"color:#79740e\">&#34;post&#34;<\/span>&gt;\n<\/span><\/span><span style=\"display:flex;\"><span>  &lt;<span style=\"color:#9d0006\">header<\/span>&gt;\n<\/span><\/span><span style=\"display:flex;\"><span>    &lt;<span style=\"color:#9d0006\">h1<\/span> <span style=\"color:#79740e;font-weight:bold\">class<\/span><span style=\"color:#af3a03\">=<\/span><span style=\"color:#79740e\">&#34;post-title&#34;<\/span>&gt;{{ .Title }}&lt;\/<span style=\"color:#9d0006\">h1<\/span>&gt;\n<\/span><\/span><span style=\"display:flex;\"><span>  &lt;\/<span style=\"color:#9d0006\">header<\/span>&gt;\n<\/span><\/span><span style=\"display:flex;\"><span>  &lt;<span style=\"color:#9d0006\">div<\/span> <span style=\"color:#79740e;font-weight:bold\">class<\/span><span style=\"color:#af3a03\">=<\/span><span style=\"color:#79740e\">&#34;post-content&#34;<\/span>&gt;\n<\/span><\/span><span style=\"display:flex;\"><span>    {{ .Content }}\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    &lt;<span style=\"color:#9d0006\">h2<\/span>&gt;Books&lt;\/<span style=\"color:#9d0006\">h2<\/span>&gt;\n<\/span><\/span><span style=\"display:flex;\"><span>    {{ $syncPages := where .Site.RegularPages &#34;Section&#34; &#34;synced&#34; }} {{ if\n<\/span><\/span><span style=\"display:flex;\"><span>    $syncPages }}\n<\/span><\/span><span style=\"display:flex;\"><span>    &lt;<span style=\"color:#9d0006\">ul<\/span>&gt;\n<\/span><\/span><span style=\"display:flex;\"><span>      {{ range $syncPages }}\n<\/span><\/span><span style=\"display:flex;\"><span>      &lt;<span style=\"color:#9d0006\">li<\/span>&gt;\n<\/span><\/span><span style=\"display:flex;\"><span>        &lt;<span style=\"color:#9d0006\">a<\/span> <span style=\"color:#79740e;font-weight:bold\">href<\/span><span style=\"color:#af3a03\">=<\/span><span style=\"color:#79740e\">&#34;{{ .RelPermalink }}&#34;<\/span>&gt;{{ .File.BaseFileName }}&lt;\/<span style=\"color:#9d0006\">a<\/span>&gt;\n<\/span><\/span><span style=\"display:flex;\"><span>      &lt;\/<span style=\"color:#9d0006\">li<\/span>&gt;\n<\/span><\/span><span style=\"display:flex;\"><span>      {{ end }}\n<\/span><\/span><span style=\"display:flex;\"><span>    &lt;\/<span style=\"color:#9d0006\">ul<\/span>&gt;\n<\/span><\/span><span style=\"display:flex;\"><span>    {{ else }}\n<\/span><\/span><span style=\"display:flex;\"><span>    &lt;<span style=\"color:#9d0006\">p<\/span>&gt;No sync pages found matching the criteria.&lt;\/<span style=\"color:#9d0006\">p<\/span>&gt;\n<\/span><\/span><span style=\"display:flex;\"><span>    {{ end }}\n<\/span><\/span><span style=\"display:flex;\"><span>  &lt;\/<span style=\"color:#9d0006\">div<\/span>&gt;\n<\/span><\/span><span style=\"display:flex;\"><span>&lt;\/<span style=\"color:#9d0006\">article<\/span>&gt;\n<\/span><\/span><span style=\"display:flex;\"><span>{{ end }}<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>And I used it as the layout for a markdown page:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-markdown\" data-lang=\"markdown\"><span style=\"display:flex;\"><span><span style=\"color:#79740e\">---<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#9d0006\">title<\/span>: <span style=\"color:#79740e\">&#34;My Reading List&#34;<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#9d0006\">layout<\/span>: reading-list\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#9d0006\">draft<\/span>: <span style=\"color:#af3a03\">false<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#79740e\">---<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>Here are some of the books I&#39;ve read and plan to read.<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>Now I have a <code>\/reading-list<\/code> page on my website, that filters all the notes related to books. And show them.<\/p>\n<p>This shows how simple and without any magic you can publish Obsidian notes.\nThere are unlimited customizations that can be done. Like showing the books in a specific order or format. I left that out since it&rsquo;s lengthy and does not fit in the scope of this post.\nSince I&rsquo;m not using some dependency I am free to touch the HTML and change how my published notes look.<\/p>\n<p>I like to see more people sharing their internal notes. If you do, share it with me. I like to read it.<\/p>\n<p>Next I like to get create email subscriptions and comments for my blog.<\/p>\n"},{"title":"After Interview","link":"https:\/\/glyphack.com\/after-interview\/","pubDate":"Thu, 06 Mar 2025 08:41:39 +0100","guid":"https:\/\/glyphack.com\/after-interview\/","description":"<p>Biggest companies put candidates through lengthy interview process, yet still fire a lot of bad hires.\nWhat if the problem is not the interview, but what happens after?\nSmarter onboarding is the key to identify if a new hire is a great fit.<\/p>\n<p>In my experience and what I&rsquo;ve heard from others, the onboarding is slow.\nFirst few weeks filled with trivial work and going through a lot of documents.\nSometimes even the laptop and required access is not ready.\nBasically the first few weeks(or months) the new hire barely touches the product.<\/p>\n<p>You cannot distinguish between a motivated skilled person and a boring person with no passion without seeing the actual work.\nTo identify a bad hire there is an evaluation process after a couple of months.\nThis evaluation happens late because judging without seeing work is impossible.\nDifficult problems are the ones showing the strengths and weaknesses. If the person is suitable for this job and if they enjoy it.<\/p>\n<p>My solution is to pair the new hire with a mentor and real work on the product.\nHelp people experience real work from day one and observe how well they do it.\nDoes it sound risky? That&rsquo;s the point.\nThere&rsquo;s no guarantee that it will be a success and that&rsquo;s okay.\nYou can still learn about this person more in this situation.\nEveryone makes mistakes and there&rsquo;s a way to move forward.\nThe new hire can understand how well the company is at handling hard situations. And company sees how new hire acts.<\/p>\n<blockquote>\n<p>The kindest thing you can do to a new team member is to involve them in something real and challenging right away. Don&rsquo;t squander weeks of new-job enthusiasm with baby rails and play tasks. Get them into the deep end right from the start.\n<a href=\"https:\/\/world.hey.com\/dhh\/start-them-in-the-deep-end-8c9c77fe\" rel=\"noopener\" target=\"_blank\">https:\/\/world.hey.com\/dhh\/start-them-in-the-deep-end-8c9c77fe<\/a><\/p>\n<\/blockquote>\n<p>How does the new approach help with identifying good candidates?\nYou can have a faster feedback cycle.\nIt&rsquo;s better for everyone to find out if they enjoy the ride or not.<\/p>\n<p>It&rsquo;s not about figuring out if the new hire makes mistakes or not, but to find strengths and weaknesses.<\/p>\n<ul>\n<li>Does he understand what he&rsquo;s doing? Is he repeating something without understanding?<\/li>\n<li>What happens when he make a mistake? Is he going to blame or fix?<\/li>\n<li>How does he approach an unknown problem? Is he asking questions? Is he trying out solutions or waiting for someone else to solve it?<\/li>\n<\/ul>\n<p>This approach is not only useful for the company but also for the new hire.\nBoth parties know sooner if this a good match.\nIf you hired someone new, welcome the person with real work with risk.<\/p>\n"},{"title":"Devlog 3: Coding a redis clone in C and things I learned","link":"https:\/\/glyphack.com\/dv-3\/","pubDate":"Sat, 15 Feb 2025 18:02:46 +0100","guid":"https:\/\/glyphack.com\/dv-3\/","description":"<p>About two months ago I started the build your own Redis challenge from <a href=\"https:\/\/codecrafters.io\" rel=\"noopener\" target=\"_blank\">https:\/\/codecrafters.io<\/a>.\nI decided to do this in C. Initially I was just curious to do it in C. Doing it in C thought me a lot of stuff that otherwise I would have not learned. Another bonus point was that I could tweak the performance to be on par with Redis server.<\/p>\n<p>C always seemed like an impossible language to me. Working so much in garbage collected languages with rich standard libraries made me think C is hard.\nNow I think C is not hard. Whatever you write gets executed the way you wrote it. Little abstractions make it a great language to implement what you want and have control over your program.\nTopics like Async programming, managing memory are broad topics. I agree that it&rsquo;s hard to achieve the same level of concurrency that you have in Python in C. But writing a small version that works for a specific use case is not impossible.\nIt&rsquo;s possible to implement hash map, Async, memory arena(for easier memory management), and your own string type with a few lines of C code, thanks to blogs like <a href=\"https:\/\/nullprogram.com\" rel=\"noopener\" target=\"_blank\">null program<\/a><\/p>\n<p>I think I&rsquo;ll use the below techniques I learned for any C program I create. I wish it was easier to package them so I can reuse it in different projects.<\/p>\n<p>The source code is at <a href=\"https:\/\/github.com\/Glyphack\/redis-clone\" rel=\"noopener\" target=\"_blank\">glyphack\/redis-clone<\/a>.<\/p>\n<p>The below are things I learned about C programming.<\/p>\n<h2 class=\"heading\" id=\"awesome-compiler-flags\">\n  Awesome Compiler Flags\n  <a class=\"anchor\" href=\"#awesome-compiler-flags\">#<\/a>\n<\/h2>\n<p>I don&rsquo;t know why nobody told me this. C can have stacktraces. It can detect race conditions. It can detect use after free. It can do a lot of stuff by just adding more compiler flags.<\/p>\n<p>You can compile your program with flags:<\/p>\n<ul>\n<li><code>-fsanitize=undefined<\/code> to crash on undefined behavior scenarios<\/li>\n<li><code>-fsanitize=thread<\/code> to crash when threads have race condition(this is actually what powers Golang race detector)<\/li>\n<\/ul>\n<p>And if there is a race condition in your program you will see something like:<\/p>\n<div class=\"code-block\">\n  <pre tabindex=\"0\"><code>==================\nWARNING: ThreadSanitizer: data race (pid=12345)\n  Read of size 4 at 0x000000601040 by thread T2:\n    #0 thread_func (source.c:7) in main\n    #1 start_thread (pthread_create.c:XXX)\n\n  Previous write of size 4 at 0x000000601040 by thread T1:\n    #0 thread_func (source.c:7) in main\n    #1 start_thread (pthread_create.c:XXX)\n\n  Location is global &#39;shared_var&#39; defined in source.c:5\n==================<\/code><\/pre>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>I built my program in debug mode with these flags and ran a tester that would insert and get values from the server. It detected a lot of bugs for me while pointing out the exact line the problem happened.<\/p>\n<h2 class=\"heading\" id=\"memory-arena\">\n  Memory Arena\n  <a class=\"anchor\" href=\"#memory-arena\">#<\/a>\n<\/h2>\n<p>I used a <a href=\"https:\/\/nullprogram.com\/blog\/2023\/09\/27\/\" rel=\"noopener\" target=\"_blank\">memory arena<\/a> to minimize the number of <code>malloc<\/code> and <code>free<\/code> calls in the code.\nIt makes code faster but in the end I realized how much simplified the code gets.\nIt&rsquo;s basically like having a garbage collector and you know when it happens.<\/p>\n<h2 class=\"heading\" id=\"custom-string-type\">\n  Custom String Type\n  <a class=\"anchor\" href=\"#custom-string-type\">#<\/a>\n<\/h2>\n<p>One of the problems I faced soon after working with C was that, sometimes I wanted to keep a reference to middle of a giant string. Imagine you get a request and extract a field name from it. Now you have two options, either <code>memcpy<\/code> that substring into a new string and add a <code>\\0<\/code> or make a pointer to the starting offset of the substring.\nWhen you keep the offset you cannot use it in most of other places because you don&rsquo;t know the end of this string, and C strings end with <code>\\0<\/code>.\nI followed the advice in <a href=\"https:\/\/nullprogram.com\/blog\/2023\/10\/08\/\" rel=\"noopener\" target=\"_blank\">this post<\/a> and it made code a lot smaller(no extra string creation and <code>malloc<\/code> calls) and simpler because working with a string when you know the length is just easier. Plus if you are worried you would lose the benefits of C string functions don&rsquo;t worry there is not much functionality there. You can implement it yourself.<\/p>\n<h2 class=\"heading\" id=\"hash-map\">\n  Hash Map\n  <a class=\"anchor\" href=\"#hash-map\">#<\/a>\n<\/h2>\n<p>I followed <a href=\"https:\/\/nullprogram.com\/blog\/2023\/09\/30\/\" rel=\"noopener\" target=\"_blank\">this post<\/a> to implement a hash map. Initially my program was using a thread per connection so I tried to extend the lock free version to work with my program.\nIn the end I implemented Asynchronous code to handle multiple connections, removed the threads, and just used the hash map that is explain in the post.<\/p>\n<h2 class=\"heading\" id=\"redis-replication\">\n  Redis Replication\n  <a class=\"anchor\" href=\"#redis-replication\">#<\/a>\n<\/h2>\n<p>The Redis replication is initially simple to implement I did not implement the full protocol.\nThe basic functionality is to handshake with the master node and master node has to keep a list of replicas to forward write messages to.\nThe way multiple nodes stay in sync is by using replication offset that master checks for periodically. I did not implement any recovery case for when a replica is out of sync.<\/p>\n<h2 class=\"heading\" id=\"asynchronous-programming\">\n  Asynchronous Programming\n  <a class=\"anchor\" href=\"#asynchronous-programming\">#<\/a>\n<\/h2>\n<p>This is my favorite topic. I finally got a clue what actually happens in higher level languages. Before actually doing this I read and heard some words but it all felt like buzzwords to me.\nWhat does it mean each coroutine has a stack? Why can&rsquo;t you run a blocking task inside an asynchronous function? I learned all after I implemented this.<\/p>\n<p>The interesting part is, implementing basic asynchronous I\/O is not hard, making it general and cross platform is. This is what other programming languages did.<\/p>\n<p>To serve multiple clients concurrently we need a way to read from all of them without ever waiting for one client and <strong>blocking<\/strong> others.\nSo what we really need is, a way to tell which clients are ready to read, which are ready to write and instead of waiting for the ones that are not ready just skip and serve other clients.\nI recommend following <a href=\"https:\/\/build-your-own.org\/redis\/05_event_loop_intro\" rel=\"noopener\" target=\"_blank\">this guide<\/a>. I ended up switching to <code>kqueue<\/code> from <code>poll<\/code> to achieve better performance on MacOS.<\/p>\n<hr>\n<p>I think this database can be a good base program to expand to any kind of future databases I want to write.\nI can reuse the existing protocol to exchange messages and add my custom logic or <a href=\"https:\/\/eli.thegreenplace.net\/2020\/implementing-raft-part-0-introduction\/\" rel=\"noopener\" target=\"_blank\">implement Raft<\/a>.<\/p>\n"},{"title":"Can Data Fool You?","link":"https:\/\/glyphack.com\/fooled-by-data\/","pubDate":"Wed, 29 Jan 2025 22:32:33 +0100","guid":"https:\/\/glyphack.com\/fooled-by-data\/","description":"<p>Recently, A personal experience showed me how something widely accepted might be wrong.\nI didn&rsquo;t want to lose this opportunity to write about it.<\/p>\n<p>If you haven&rsquo;t been living under a cave, you know data-driven decision making is a must these days.\nSo, if you cannot make a decision just run both under some experiment and decide based on the result.\nIf you have a problem that is hard to answer, you just try different answers.\nThen you pick the one that makes more money or whatever thing you want.<\/p>\n<p>Imagine you don&rsquo;t know what&rsquo;s the maximum price you can sell a book that people keep buying it.\nYou can sell it more expensive and expect less sales while having more profit on each book or the other way around.\nNow the data boss comes up with this neat idea.\nLet&rsquo;s sell the book with two different prices in two markets and see how much profit do we make.\nSounds good right?\nImagine maybe one market was Inevitably going to like the book less.\nYou can imagine a million reasons that this might happen.\nYou know how accidentally stuff go viral.<\/p>\n<p>When you are done with the experiment you try to find the patterns and decide what to do based on the data.\nUnless something is terribly wrong(10x books sold in one area) you don&rsquo;t suspect the result.\n&ldquo;Oh, maybe it&rsquo;s expensive and people didn&rsquo;t buy it.&rdquo;, you say forgetting how many things can affect this.\nAnother problem with these experiments is how long are you going to continue this. There is no answer, and you can slow yourself down significantly.<\/p>\n<p>Few days ago I was working on some piece of code.\nBuried down deep in codebase everyone forgot about it once it was launched and the metrics looked good.\nOnly one point was missed, the calculations were wrong.\nNow the person working on this before me definitely relied on the metrics to assure the change is making good impact.\nAnd indeed the change was good overall because the metrics showed that compared to the product before change.<\/p>\n<p>This is why I prefer to use thinking instead when it&rsquo;s possible to decide about something.\nWhen one solution is obviously worth (code that does not work) why bother trying it?\nThe moment you see the charts go up you attribute that to the code you changed no matter correct or wrong.\nCould it be that the code is wrong, but the result is better due to some other thing that we don&rsquo;t know?\nSo you lose the opportunity to think more, maybe the wrong code is better because something else is broken.<\/p>\n<p>My experience is limited, but you can check more <a href=\"https:\/\/www.goodreads.com\/book\/show\/13530973-antifragile\" rel=\"noopener\" target=\"_blank\">examples<\/a> on why you might misunderstand a phenomenon and come up with wrong answers and <a href=\"https:\/\/www.youtube.com\/watch?v=QBe8lJdpvDU\" rel=\"noopener\" target=\"_blank\">how data can fool you<\/a><\/p>\n<p>Remember learning multiplication tables?\nYour teacher didn&rsquo;t say &ldquo;just keep guessing numbers until your test scores improve.&rdquo;\nThat would be absurd. Yet somehow, that&rsquo;s exactly what we&rsquo;re doing with our code: throw something at the wall, check if the metrics went up, ship it if they did.<\/p>\n<p>Maybe imposing constraints would get us out of this situation. You don&rsquo;t have all the time to try everything, just think what would be useful.<\/p>\n<p>Here&rsquo;s the thing about understanding versus just observing: if you see something work and think you know why, you should be able to do it again.\nThat&rsquo;s the difference between actual understanding and just having a good story about what happened.\nLet me put it this way: if you run an A\/B test and conclude &ldquo;Oh, users love blue buttons!&rdquo; then you should be able to predict when blue buttons will work again.<\/p>\n<p>So, data is good, but why not combine it with some thinking?<\/p>\n"},{"title":"On Tracking Time","link":"https:\/\/glyphack.com\/tracking-time\/","pubDate":"Tue, 31 Dec 2024 18:00:17 +0100","guid":"https:\/\/glyphack.com\/tracking-time\/","description":"<p>One week I suddenly realized I did so much during the week but there&rsquo;s nothing to show for it.\nYou know that feeling that you don&rsquo;t have anything to tell your friend that you did that week.<\/p>\n<p>I gave time tracking a shot to see what will I find.\nRescue time looked like a simple tool that gets the job done. I used Activity Watch before and the main lacking feature was syncing and the mobile app (at the time they now have a new mobile app.)<\/p>\n<p>Turns out my feeling was right, next week I saw around 8 hours of my week during work time spent in meeting, Slack in discussions.\nOther than that it showed me how much time I spend on social media and other things that does not achieve anything.\nSo basically I was spending a lot of time on things that don&rsquo;t produce anything. It&rsquo;s not fun to talk with someone about how many work discussions they had last week.<\/p>\n<p>After a couple of months I notice the change in how I use my time.\nI don&rsquo;t really use the time tracking to find out how much time I spend on what.\nI just use it to find out if I spend too much time idling and not doing anything or not.\nThese days even if I don&rsquo;t use it any more I will keep the few habits it helped me to create.<\/p>\n<p><strong>Use devices purposely<\/strong><\/p>\n<p>Use the phone or the computer for a purpose.\nThis is basically what all the app blockers do. They want to block the distractions so you think okay what did I want to do?<\/p>\n<p>Sometimes there is nothing to do with your phone. You just want the time to pass or rest a bit.\nWhat about reading a book, sleeping or doing something that requires no effort?<\/p>\n<p><strong>Think about total time spent<\/strong><\/p>\n<p>It&rsquo;s easy to notice when you spend 1 hour in a day scrolling through content and not doing anything.\nBut it&rsquo;s harder to notice it if it&rsquo;s half an hour every day. But if you think about it that results in 3.5 hours per week.\n3 hours sounds a lot really a lot can be done with that time, even adding it to the sleep sounds better.<\/p>\n<p>I tracked time for the full month and saw how much time I&rsquo;m spending on things that are not really useful.\nI tried to limit them. As a result I just did what was necessary.\nFor example what I mentioned all the time in discussions at work. I limited that to a certain amount per day. This forced me to say what I had to say in slack and continue doing what I was doing.<\/p>\n<p><strong>Do not care too much about the graphs<\/strong><\/p>\n<p>These apps usually give you a lot of scores, graphs and data to play with.\nI don&rsquo;t think it&rsquo;s useful to look at them.\nIt&rsquo;s a distraction, and it can drag you down into optimizing the wrong thing.<\/p>\n<p>The most important part is that, you spend time on what you want and distractions are not in your way.\nThis can be seen by how much time you are spending on the distractions during the day.\nSo I think if the distractions are taking very little time. Then the rest of the time is free.\nFree time plus boredom is all needed to do what you want to do. Unless you don&rsquo;t want <a href=\"https:\/\/paulgraham.com\/want.html\" rel=\"noopener\" target=\"_blank\">what you want<\/a>.<\/p>\n<hr>\n<p>I don&rsquo;t know if time tracking is necessary or not.\nMaybe after a while these habits are shaped, so it&rsquo;s not needed any more.\nI think if there&rsquo;s something that grabs your attention, and you want to do it, you will do it. No focus or do not disturb mode needed for true captivation.\nBut all the other times it&rsquo;s easy to get distracted and not find the intriguing.<\/p>\n<p>Update 2026-01-26: I stopped using time tracking software.\nI got value out of it. It helped me notice patterns and habits. I don&rsquo;t want to optimize my life around screen time metrics.<\/p>\n"},{"title":"Dev Tools at Work","link":"https:\/\/glyphack.com\/dev-tools-at-work\/","pubDate":"Sun, 29 Dec 2024 19:32:07 +0100","guid":"https:\/\/glyphack.com\/dev-tools-at-work\/","description":"<p>I was reading this old post from Brad Fitzpatrick, talking about why he thought open source contributors suddenly disappear after joining Google:<\/p>\n<blockquote>\n<ul>\n<li>They&rsquo;re busy. Google seems to suck everybody&rsquo;s free time, and then some. It&rsquo;s not that Google is forcing them to work all the time, but they are anyway because there are so many cool things that can be done. I often joke that I have seven 20% projects.<\/li>\n<li>The Google development environment is so nice. The source control, build system, code review tools, debuggers, profilers, submit queues, continuous builds, test bots, documentation, and all associated machinery and processes are incredibly well done. It&rsquo;s very easy to hack on anything, anywhere and submit patches to anybody, and notably: to find who or what list to submit patches to. Generally submitting a patch is the best way to even start a discussion about a feature, showing that you&rsquo;re serious, even if your patch is wrong.<\/li>\n<\/ul>\n<\/blockquote>\n<p><a href=\"https:\/\/brad.livejournal.com\/2409049.html\" rel=\"noopener\" target=\"_blank\">https:\/\/brad.livejournal.com\/2409049.html<\/a><\/p>\n<p>I cannot imagine how the work is so fun there than people vanish into the air after joining google.\nOr how someone gets caught up in seven different 20% projects because the environment lets you do that much work.<\/p>\n<p>I like being busy when there is a lot to work on.\nMost of the time what happens is that people are busy because they are kept busy by the tasks around work.<\/p>\n<p>Let me tell you a story about John. (It&rsquo;s not a real John.)<\/p>\n<p>John works at a 50 people company. He works on tools for sales and marketing to do their work faster.\nHis tools are mostly scripts. He makes a website to make the interaction with scripts easier.<\/p>\n<p>One day he has an idea of a new mini product.\nNothing earth-shattering, mind you - just a simple integration between two tools that would save the marketing team from having to copy-paste discount codes. In Brad&rsquo;s Google utopia, John would just write the code, submit a patch, and boom - problem solved, everyone&rsquo;s happy.<\/p>\n<p>First, John has to talk to Sarah. But wait, he can&rsquo;t just <em>talk<\/em> to Sarah.\nThat would be far too efficient.\nHe has to go through Sarah&rsquo;s manager, who needs to talk to Sarah, who then needs to talk back to her manager, who needs to update some road map that probably hasn&rsquo;t been looked at since last OKR planning.<\/p>\n<p>So this simple conversation that could lead to a possibly good thing cannot happen simply.\nIt Needs to happen in a group chat when everyone says their opinion (which takes time from the readers as well as the author) and have some back and forth discussion.<\/p>\n<p>Because of this John has to be more thoughtful about the suggestions.\nIt&rsquo;s okay to suggest something not useful a few times but if it becomes a couple of times per week then everyone will get tired from him.<\/p>\n<p>Although he does not actively give ideas and prototype them. He posts a few every month or so, then he goes off writing a long proposal for both teams to accept it and then moves it to their engineering road map to be done somewhere in the next months.\nHe&rsquo;s lucky if he can deliver 1 of his suggestions every quarter.\nThis seems dangerous, he might get fired right?\nNo, John&rsquo;s a hero because he&rsquo;s mastered the art of producing artifacts - not actual, useful code, mind you, but the kind of artifacts that look good in performance reviews. Meeting notes. Project proposals. Progress updates. It&rsquo;s like a cargo cult of productivity.<\/p>\n<p>Few months later the something similar happens.\nSarah wants a small new feature to make her work more enjoyable.\nEngineering and sales teams chat, they decide that this bug is not easy to fix.\nBut there is an easy work around that adds few more steps to Sarah&rsquo;s work.\nSo they decide to fix the bug when the engineering team has some free time.\nThe free time definitely does not happen a lot.\nSarah&rsquo;s work is not much slower but it&rsquo;s annoying, she might make a mistake because someone else product is not working.\nBut it&rsquo;s hard to measure how annoying the product is in numbers, so people only measure the time it saves from the work.<\/p>\n<hr>\n<p>I&rsquo;ve seen this story happening over and over again, and I&rsquo;ve been on the both sides.<\/p>\n<p>John can&rsquo;t even prototype the damn thing. Because in this brave new world of corporate efficiency, actually <em>building something<\/em> to see if it works is considered too risky. Better to spend six months writing proposals about the thing you want to build than actually building it in a week to see if it&rsquo;s any good.<\/p>\n<p>I&rsquo;m not saying all planning, design, and management is unnecessary.\nIt&rsquo;s definitely easier to do it right at the beginning than to migrate a live system.\nI&rsquo;m okay with thinking carefully about these critical components.\nInstead of slowing down everything, just try out most of the stuff but think about irreversible or hard to migrate decisions like how data is stored.<\/p>\n<p>Also doing something gives you more information.\nThe feedback from acting tells where to go next.\nThis is something you don&rsquo;t get as much from sitting and discussing an idea.<\/p>\n<p>If people don&rsquo;t have the authority to prototype something then the rate of generating and trying out ideas will decrease.\nA lot of times the ideas will not be useful and be discarded.\nTrying out ideas teaches people and give them more experience.\nMakes them better in generating next ideas.\nJust shipping the thing and deciding what to do next is faster and more fun.<\/p>\n<p>We should allow people use their curiosity and don&rsquo;t block them with processes.<\/p>\n<p><strong>Having good tools<\/strong><\/p>\n<p>There are countless times I found projects that have out-of-date documents, misconfigured tools, examples that won&rsquo;t run anymore.\nI honestly have no idea how someone can be maintaining a project where <code>Makefile<\/code> is broken. Do you even use it?<\/p>\n<p>It takes time and skill to create and maintain good tools.\nAnd I think the reason is that the incentive for working on tools is not there.\nThe author of the projects make this beautiful <code>README<\/code>, Makefile, example code that would make everyone get up and running with project in seconds. Why? Because they&rsquo;re trying to get that promotion<\/p>\n<p>Fast-forward six months and the project document is out of date, the Makefile is so broken that you make your own scripts to work on the project.\nWhen tools and projects don&rsquo;t have good developer tools you can&rsquo;t just start working on a project to evaluate an idea.\nWant to try out a new idea on the codebase? Good luck with that.<\/p>\n<p>The typical experience goes something like this: you clone the repo (assuming you can find it), follow the setup instructions, and then you hit some obscure error message as result of mixing 10 tools. With the poor error handling you have to dig in the code, find which one is not working as expected and start fixing code or your environment.<\/p>\n<p>And this isn&rsquo;t just annoying - it&rsquo;s toxic to innovation. How the hell is anyone supposed to prototype big ideas when they can&rsquo;t even get the damn thing to compile?<\/p>\n<p>Document the contributing process. Constantly update the docs and review that in the code review.\nMake sure the examples work, the tools integrate nicely in a project.\nFor example if your project has a debugger make sure your changes won&rsquo;t break the debugger.\nMake sure the app runs locally.\nTest this whenever someone new joins the team.<\/p>\n<p>You know what? Maybe we should try out this crazy idea of letting people actually do work and see the result.\nI&rsquo;m just saying, MAYBE, MAYBE just doing something helps you find out if something is good.\nMaybe just write the documentation and update it instead of writing proposals about writing documentations and how the build system should work (when it&rsquo;s broken).<\/p>\n<hr>\n<p>I&rsquo;m sad that I haven&rsquo;t worked at a place that matches what Brad said about Google. But it&rsquo;s probably about google at that time and things might be different now.\nThe bright side is that open source is much better these days. I experienced this in open source projects. Good tools, fast builds, and great developer setup guides.\nPeople care about good experience there.<\/p>\n<p>P.S. If you&rsquo;re reading this from Google circa 2009, please send help. And your build system and your developer tools.<\/p>\n"},{"title":"Devlog 2","link":"https:\/\/glyphack.com\/dv-2\/","pubDate":"Sat, 02 Nov 2024 11:11:05 +0100","guid":"https:\/\/glyphack.com\/dv-2\/","description":"<p>Random notes from past month.<\/p>\n<p><strong>New Projects<\/strong><\/p>\n<p>I spent about a year building <a href=\"https:\/\/github.com\/Glyphack\/enderpy\" rel=\"noopener\" target=\"_blank\">my own tool-chain<\/a> for building a python type checker. It inspired by what ruff was doing for Python linting and wanted to do the same for type checker.\nA few months ago I found out that astral team are building a type checker. So I decided to redirect my energy toward that.\nBuilding a type checker for a language that is not designed for typing is hard.\nSometimes you need to know what will be the exact behaviour of a type, and you see Pyright and Mypy have differences.\nSo naturally this requires you to do more research and figure out the specs yourself.<\/p>\n<p>I was getting a lot of my guidance from the astral team. Because I&rsquo;m not a python typing expert.\nSo I think this would be a better approach to contribute to that project and achieve my goal.\nAlso, Rust is a hard language and a lot of the time I felt like the language was stopping me from doing what I want.\nSo I had to read a lot on how to use it.\nWhen contributing to another project, you have other people who will help with this kind of stuff.\nSo this is even better I don&rsquo;t need to fight the language any more, I can read their code and learn and help with typing.<\/p>\n<p>Aside from that,\nI&rsquo;m building a C compiler from scratch with my friend. The goal is for it to compile itself.\n<a href=\"https:\/\/github.com\/keyvank\/30cc\" rel=\"noopener\" target=\"_blank\">https:\/\/github.com\/keyvank\/30cc<\/a><\/p>\n<p>For some time I&rsquo;m going to write my own projects in something other than Rust.\nIt was hard for me to work with it and focus on the project.\nFor learning projects I want to do it myself with minimal dependencies.\nRust can be tricky, and you need a dependency to save yourself from writing unsafe code.\nOr the code becomes verbose and requires a lot of typing and organizing it.<\/p>\n<p><strong>Terminal Workflow Improvements<\/strong><\/p>\n<p>On of the things I&rsquo;ve been wanting for a long time was the ability to jump to start of a command output in terminal. Imagine when you run something, and it outputs a lot of things.\nWhen you want to see the beginning of the output or just read the logs from the beginning you need to scroll up.\nThis turns out to be easy to do but requires configuration for the terminal and shell you are using.\nFor Wezterm and Fish I did the following:<\/p>\n<p>Set key bindings for <a href=\"https:\/\/wezterm.org\/config\/lua\/keyassignment\/ScrollToPrompt.html\" rel=\"noopener\" target=\"_blank\">ScrollToPrompt<\/a> action.\nCreate a fish function to emit the characters that marks the output of the command before executing a command.<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-sh\" data-lang=\"sh\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">function<\/span> pre_command --on-event fish_preexec\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#b57614\">printf<\/span> <span style=\"color:#79740e\">&#39;\\033]133;A\\033\\\\&#39;<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>end<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>This feature to run specific functions on an event in fish is really powerful. You can build custom workflows around your work. For example, you can do some project specific setup when entering a folder with this snippet:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-sh\" data-lang=\"sh\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">function<\/span> some_setup --on-variable PWD\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">if<\/span> <span style=\"color:#b57614\">test<\/span> <span style=\"color:#79740e\">&#34;<\/span>$PWD<span style=\"color:#79740e\">&#34;<\/span> <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#79740e\">&#34;<\/span>$PROGRAMMING_DIR<span style=\"color:#79740e\">\/&#34;<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>  <span style=\"color:#928374;font-style:italic\"># do some stuff<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>    end\n<\/span><\/span><span style=\"display:flex;\"><span>end<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p><strong>Neovim<\/strong><\/p>\n<p>If you know how to fold all the functions by default please let me know.\n<a href=\"https:\/\/old.reddit.com\/r\/neovim\/comments\/1g41rjy\/can_neovim_do_this_already_with_treesitter\/\" rel=\"noopener\" target=\"_blank\">https:\/\/old.reddit.com\/r\/neovim\/comments\/1g41rjy\/can_neovim_do_this_already_with_treesitter\/<\/a><\/p>\n<p>I&rsquo;m proud of myself for writing these two simple commands:<\/p>\n<ol>\n<li>Key binding to insert a hyperlink in markdown file on visual selection<\/li>\n<li>Command to go to the test file of the current go file I have open. Very useful at my job<\/li>\n<\/ol>\n<div class=\"code-block\">\n  <pre tabindex=\"0\"><code>vim.api.nvim_create_user_command(&#34;Link&#34;, function(opts)\n local start_pos = vim.fn.getpos(&#34;&#39;&lt;&#34;)\n local end_pos = vim.fn.getpos(&#34;&#39;&gt;&#34;)\n\n local selected_text = vim.fn.getline(start_pos[2]):sub(start_pos[3], end_pos[3])\n\n vim.api.nvim_command(&#34;normal! gv&#34;)\n if selected_text:match(&#34;^http&#34;) then\n  vim.fn.setreg(&#39;&#34;&#39;, &#34;[](&#34; .. selected_text .. &#34;)&#34;)\n  vim.api.nvim_command(&#34;normal! P&#34;)\n  local new_pos = { start_pos[2], start_pos[3] - 1 }\n  vim.api.nvim_win_set_cursor(0, new_pos)\n else\n  vim.fn.setreg(&#39;&#34;&#39;, &#34;[&#34; .. selected_text .. &#34;]()&#34;)\n  vim.api.nvim_command(&#34;normal! P&#34;)\n  local new_pos = { start_pos[2], start_pos[3] + #selected_text + 2 }\n  vim.api.nvim_win_set_cursor(0, new_pos)\n end\nend, { range = true })\n\nvim.keymap.set(&#34;v&#34;, &#34;&lt;leader&gt;k&#34;, &#34;:Link&lt;CR&gt;&#34;, { noremap = true, silent = true })\n\nvim.api.nvim_create_user_command(&#34;GotoTest&#34;, function()\n local current_file = vim.fn.expand(&#34;%:p&#34;)\n local file_type = vim.bo.filetype\n local test_file\n\n if file_type == &#34;go&#34; then\n  test_file = vim.fn.fnamemodify(current_file, &#34;:r&#34;) .. &#34;_test.go&#34;\n else\n  vim.api.nvim_err_writeln(&#34;Test file location not defined for filetype: &#34; .. file_type)\n  return\n end\n\n if vim.fn.filereadable(test_file) == 1 then\n  vim.cmd(&#34;edit &#34; .. test_file)\n else\n  vim.api.nvim_err_writeln(&#34;Test file not found: &#34; .. test_file)\n end\nend, {})<\/code><\/pre>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>I had a problem that when I connected my laptop to a new screen Flameshot would not capture the whole screen from the new screen in the screenshots.\nI could not find a way to resolve this so I wrote this hammerspoon script to restart the app when I connect it to a new monitor:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-lua\" data-lang=\"lua\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">local<\/span> <span style=\"color:#af3a03\">function<\/span> <span style=\"color:#b57614\">screenCallback<\/span>(layout)\n<\/span><\/span><span style=\"display:flex;\"><span> <span style=\"color:#af3a03\">if<\/span> layout <span style=\"color:#af3a03\">==<\/span> <span style=\"color:#af3a03\">true<\/span> <span style=\"color:#af3a03\">then<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>  print(<span style=\"color:#79740e\">&#34;Screen did not change&#34;<\/span>)\n<\/span><\/span><span style=\"display:flex;\"><span>  <span style=\"color:#af3a03\">return<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span> <span style=\"color:#af3a03\">end<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span> setPrimary()\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span> <span style=\"color:#af3a03\">local<\/span> flameshot_bundle <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#79740e\">&#34;\/Applications\/flameshot.app&#34;<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span> <span style=\"color:#af3a03\">local<\/span> flameshot <span style=\"color:#af3a03\">=<\/span> hs.application.find(flameshot_bundle, <span style=\"color:#af3a03\">false<\/span>, <span style=\"color:#af3a03\">false<\/span>)\n<\/span><\/span><span style=\"display:flex;\"><span> <span style=\"color:#af3a03\">if<\/span> flameshot <span style=\"color:#af3a03\">then<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>  flameshot:kill()\n<\/span><\/span><span style=\"display:flex;\"><span> <span style=\"color:#af3a03\">end<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span> hs.application.open(flameshot_bundle)\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">end<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>hs.screen.watcher.newWithActiveScreen(screenCallback):start()<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<hr>\n<ul>\n<li>I knew about PyPy, but I didn&rsquo;t know they have a full tool chain for creating interpreters. Until I watched this <a href=\"https:\/\/www.youtube.com\/watch?v=p8fCq16XTH4\" rel=\"noopener\" target=\"_blank\">Tsoding video<\/a><\/li>\n<li>Kay Lack&rsquo;s YouTube channel is one of the best things I found last month. High quality videos about computers and programming.\nThis one is about <a href=\"https:\/\/youtube.com\/watch?v=DiXMoBMWMmA&amp;si=yqldVom-i92x7iSA\" rel=\"noopener\" target=\"_blank\">regex<\/a> and <a href=\"https:\/\/www.youtube.com\/watch?v=GU8MnZI0snA\" rel=\"noopener\" target=\"_blank\">this one<\/a> assembly.<\/li>\n<\/ul>\n"},{"title":"Writing a DNS server From Scratch","link":"https:\/\/glyphack.com\/dns-from-scratch\/","pubDate":"Sat, 12 Oct 2024 13:09:53 +0200","guid":"https:\/\/glyphack.com\/dns-from-scratch\/","description":"<p>I decided to build another system from scratch just to explore a new topic and learn things.\nThis time I chose the <a href=\"https:\/\/app.codecrafters.io\/courses\/DNS-server\" rel=\"noopener\" target=\"_blank\">Code Crafters&rsquo;s DNS challenge<\/a><\/p>\n<p>This is not a tutorial on how to do it but some notes to motivate you to do it.\nThere&rsquo;s already <a href=\"https:\/\/app.codecrafters.io\/courses\/DNS-server\" rel=\"noopener\" target=\"_blank\">excellent material<\/a> on how to do it with code examples here.\nIf you are already curious, then start building your own.<\/p>\n<p>I really enjoyed the way challenge is organized.\nIt&rsquo;s very small steps in which you implement something from the spec.\nYou don&rsquo;t build fake things that later become real at a certain stage.\nIt&rsquo;s all working software from the beginning.\nFor example, in the DNS challenge you can use what you build with the <code>dig<\/code> command from the first step.\nIt does not actually work, but it&rsquo;s the first thing you need when building a DNS server.\nSomething that just spits out correct bytes.<\/p>\n<p>This encourages you to not plan for what components your app should have or how to represent the request and response.\nYou build the thing that shows you a result as soon as possible.\nI was writing my code in the main function with no structs or anything. Because you really don&rsquo;t need it.<\/p>\n<p>This helps a lot to make progress and stay motivated.\nA lot of the time when you are learning something new you try to see what is the whole subject and how much you have to learn and what are the topics.\nBut this also slows you down and makes it uninteresting.\nPrimarily, because the joy of learning is destroyed.\nAlso, because any complex topic is big enough to show you how much time you need to spend to become good that can push you away from learning it.\nThis helps you to just focus on the given problem at the time.\nYou can apply this in your own projects as well.\nInstead of listing every single thing you need to do to succeed focus on the next result you can see and learn to get there.<\/p>\n<p><strong>Things I learned<\/strong><\/p>\n<p>This was my first time working with the bits and bytes in golang.\nI had only done it in assembly before and almost forgot how it&rsquo;s done.\nAnd golang has a useful set of packages to do help with this.<\/p>\n<p>Also, the <a href=\"https:\/\/datatracker.ietf.org\/doc\/html\/rfc1035#autoid-44\" rel=\"noopener\" target=\"_blank\">compression method<\/a> is interesting, you don&rsquo;t need a general algorithm that compresses the data when you know patterns in your data.\nThe compression does not work like normal compressions, and instead it relies on the fact that the DNS query has a lot of domain names with similar labels (each word between the <code>.<\/code> in domain is a label) so it says a label once and refer to it later.<\/p>\n<p>Start with the smallest thing that gives result.<\/p>\n<p><strong>Improvements<\/strong><\/p>\n<p>In the last stage of the challenge you implement a DNS server that takes in the requests and resolves them using another DNS resolver. The other resolver is provided by codecrafters.\nI tried to use my resolver with <code>8.8.8.8:53<\/code> and it was not working.\nI found the reason is that my DNS parser does not still support all the information that <code>dig<\/code> command sends by default. And also I haven&rsquo;t implemented compression for encoding DNS request to bytes.\nI haven&rsquo;t checked how the codecrafters software works but maybe I would try to contribute this later.<\/p>\n"},{"title":"How to Discover Interests","link":"https:\/\/glyphack.com\/explore\/","pubDate":"Sun, 29 Sep 2024 21:31:32 +0200","guid":"https:\/\/glyphack.com\/explore\/","description":"<p>I was talking with my girlfriend recently about how to find something you are passionate about.<\/p>\n<p>Having something you\u2019re truly passionate about gives you a reason to get up in the morning and makes life more enjoyable.\nIt\u2019s more than just a hobby, it\u2019s <a href=\"https:\/\/paulgraham.com\/genius.html\" rel=\"noopener\" target=\"_blank\">one of the key ingredients<\/a> to doing great work.<\/p>\n<p>But how can you find it?\nOne way is to notice what is drawing your attention.\nAlthough this is a required condition for the answer, but it&rsquo;s not enough.\nYou need genuine interest.\nPlenty of things seem interesting at first, but the desire fades when you dig deeper.\nIt\u2019s easy to think you\u2019d enjoy doing what successful people do, but you don\u2019t always take action.\nYour mind tends to filter out things you\u2019re not truly interested in.<\/p>\n<p>Imagine you\u2019re trying to help a 10-year-old figure this out.\nWhat often happens is parents suggest a prestigious profession: \u201cDo you want to be a doctor, lawyer, or engineer?\u201d\nThen they explain what each does day-to-day.\nBut the kid\u2019s decision is usually based on surface-level factors\u2014how respected the job is, the salary, or what society thinks, without really understanding what those professions involve.\nParents usually follow up with, \u201cStudy hard, get into a good university, and become one of these.\u201d\nAt this point, the kid is thinking, \u201cWhat do I do next? Just get good grades in every subject to become a lawyer?\u201d<\/p>\n<p>Not only this kind of advice does not get the kid anywhere.\nIt also contains a trap from what I described earlier.\nUsing prestigious professions is a way to make the suggestions look more interesting.\nIt&rsquo;s not bad to do this, but we should be aware that it can create a false sense of interest.<\/p>\n<p>And this problem is not only in parents advice, but on the internet as well.\nIt might sound a good idea to lookup introduction to different professions online and make a judgement based on that.\nBut introductory materials online are very high level.\nFor example if you search for how to become a programmer the first pages you&rsquo;re going to land on is horrible.\nThey are going to you the list of programming languages with how many jobs are on the market for each one, or list of different areas that you can work in how much you get paid for each one.\nLeaving one thinking which one should they use first.\nWhat it should do instead is to give them a taste of doing it.\nThe search should lead them into some kind of experiment where they program a bit and see if fall in love with it<sup id=\"fnref:1\"><a href=\"#fn:1\" class=\"footnote-ref\" role=\"doc-noteref\">1<\/a><\/sup>.<\/p>\n<p>Another way to look at it is to see what is the next step they can take.\nThe kid is already learning some subjects in school.\nThey must enjoy one of them more than the others and might have some hobbies already.\nIf they like painting, encourage them to pursue it further.\nMaybe by learning about famous painters and their stories.\nIf they enjoy math, suggest they look up concepts they don\u2019t understand.\nFor instance, when learning about Gauss&rsquo;s formula for summing numbers, ask: Where was he from? What else did he do? They might check out his Wikipedia page and discover other theorems, clicking through to learn more.\nThey will also see he was not only a mathematician, so by clicking on those links you discover more professions.\nThe hours spent on this are productive time, because they show you new ideas.<\/p>\n<p>I think the reason we are not thinking to provide a next step for kids is that the outcome is unpredictable.\nParents want their kids to end up in one of the professions they think is a good choice.\nIf the desired outcome for someone is that their kid must become an X they tell them next step to practice that and see if they enjoy it.\nThey also don&rsquo;t want risks for their kid, so they point out something that sounds safe and good enough.<\/p>\n<p>When I look at my past I find a lot of good stuff when I was exploring.\nExploring is about being curious and pursuing whatever you don\u2019t know when you come across it. If you find yourself continually spending time on a subject, that\u2019s a sign of genuine interest.\nAlso exploring will give you more options to choose compared to when you start.\nAt the beginning you have one topic in mind but when you go deep you find more topics.\nTake our Gauss example: now you know ten more things about him to study. You learn where he\u2019s from and might explore more about that country and its language. You discover he wasn\u2019t just a mathematician, leading you to other professions he had. This exploration reveals many new paths to consider.<\/p>\n<p>Some people might think it&rsquo;s pointless to just read about anything you hear.\nAnd while it sounds true, but it&rsquo;s the source of new things.\nUnless you have found the topic you still need this source.\nAnd when you find new paths continue exploring until you find genuine interest.\nThen start working on the topic and see if the motivation and interest continues.\nYou don&rsquo;t need to fake enthusiasm for yourself.\nMove on similar topics that you discovered or change the topic entirely if it&rsquo;s boring.<\/p>\n<p>I think this is the way to show kids you they can discover what they want to do.\nSchools often fail at this, they are making kids focused on the predefined topics and nothing else to get good grades.\nThe home works and tests different combinations of the same concepts you learned, and nothing new or extra is needed to pass.\nNot only you don&rsquo;t get any benefits if you go deep into subjects.\nBut you also don&rsquo;t even get the chance to read about the subject because you have to study the important bits that show up in the exam.\nThey cannot even see where the things they are learning are useful until very late in the education.<\/p>\n<p>This is not only useful for teenagers.\nBut also as an adult I still have the issue.\nI face this question a lot of the times.\nAnd for adults it&rsquo;s even a harder problem to solve.\nDuring school time if you did not want to do anything you were just forced to do school stuff for the majority of the day and spend time that way.\nYou can ignore the question since you already have huge pile of homework, classes and exams that seem like progress to distract yourself.<\/p>\n<p>But as an adult you have less work imposed on yourself.\nSo you have more time and if you don&rsquo;t have something that gets you excited or make you happy doing you are forced to spend time with something to pass the day.\nMostly TV, YouTube, social media, 5 second clips.\nOr you can drown yourself in your job and treat it like school.\nA very simple trick to understand if you use the job to distract yourself is if you are looking for something to spend your time on and then find the job as the answer then it&rsquo;s not the actual answer.<\/p>\n<p>So as an adult you still need to keep the habit of exploring things you don&rsquo;t understand and stay curios.\nBut as an adult one of the ingredients is harder to find: Random subjects and topics that you are forced to learn.\nIn school, you are given some books and materials, perfect or not you have some initial point.\nYou can start learning about topics you hear but don&rsquo;t know about.\nYou can get ideas by being curious. Books, writings, talking with others.\nBut the point is that action is needed.\nAnd from that initial source you can again create more paths and find the interesting things.<\/p>\n<p>Do not forget that the same challenges in the school can apply here.\nYou might be very busy during the job to look into other areas or learn about them.\nSome problems in life might take the time and energy from you to explore unknown topics.<\/p>\n<p>And when you encounter questions you don&rsquo;t know the answer to write about it.\nI was reading <a href=\"https:\/\/www.lesswrong.com\/posts\/ii4xtogen7AyYmN6B\/learning-by-writing\" rel=\"noopener\" target=\"_blank\">learning by writing<\/a> a few weeks ago which explains the point perfectly.\nYou can use the writing to answer a question, and discover more questions as you write.<\/p>\n<p>Do not worry that this does not guarantee a predictable outcome.\nThe fact that outcome is not determined is the strength because each person has to determine the path for themselves.<\/p>\n<div class=\"footnotes\" role=\"doc-endnotes\">\n<hr>\n<ol>\n<li id=\"fn:1\">\n<p><a href=\"https:\/\/www.norvig.com\/21-days.html\" rel=\"noopener\" target=\"_blank\">https:\/\/www.norvig.com\/21-days.html<\/a> is a good starting point for someone to learn about programming.&#160;<a href=\"#fnref:1\" class=\"footnote-backref\" role=\"doc-backlink\">&#x21a9;&#xfe0e;<\/a><\/p>\n<\/li>\n<\/ol>\n<\/div>\n"},{"title":"A better go test","link":"https:\/\/glyphack.com\/better-gotest\/","pubDate":"Sat, 24 Aug 2024 15:15:06 +0200","guid":"https:\/\/glyphack.com\/better-gotest\/","description":"<p>My job now involves doing some Golang work and this post is about how <code>go test<\/code> command can be improved.<\/p>\n<p>Until now, I never actually thought about improving the test command for a language.\nBut with Golang I have serious problems with reporting test results on command line.\nTest output is not readable, The usual test flags you expect are not there.<\/p>\n<ul>\n<li><a href=\"https:\/\/github.com\/golang\/go\/pull\/62714\" rel=\"noopener\" target=\"_blank\">Fail fast option does not work<\/a><\/li>\n<li><a href=\"https:\/\/stackoverflow.com\/questions\/25380799\/listing-of-pass-and-failed-test-cases-in-go\" rel=\"noopener\" target=\"_blank\">You cannot see the list of failed tests<\/a><\/li>\n<li><a href=\"https:\/\/docs.pytest.org\/en\/7.1.x\/how-to\/cache.html\" rel=\"noopener\" target=\"_blank\">Does not offer rerun options like pytest<\/a><\/li>\n<\/ul>\n<p>I noticed the problem when I started working on a quite large golang project with a lot of tests.\nWhen I run the tests it starts printing a lot of information, most of it are not important.<\/p>\n<p>I found <a href=\"https:\/\/github.com\/gotestyourself\/gotestsum\" rel=\"noopener\" target=\"_blank\">gotestsum<\/a> package which seems to do what I want.\nI like it, but it seems to focus on other things that I don&rsquo;t find problematic.\nFor example, it has watch option, which is not really needed for the test runner.\nYou have other tools to watch and run commands.<\/p>\n<p>Happy that I found an opportunity I created <a href=\"https:\/\/github.com\/Glyphack\/gotest\" rel=\"noopener\" target=\"_blank\">gotest<\/a>.\nIt&rsquo;s a very simple tool. It takes in the output of <code>go test<\/code> command and organizes output to a human friendly output. I&rsquo;m planning to add more commands and see how far can I improve the testing on CLI. You don&rsquo;t have to install an IDE just for running tests easily, it&rsquo;s not Java.<\/p>\n<p><strong>Why not contributing to Golang?<\/strong><\/p>\n<p>I think some of these features could be added in the <code>go test<\/code>, and I&rsquo;m going to use this project as an experiment to see how they turn out to be.<\/p>\n<p>If you are interested let me know.<\/p>\n"},{"title":"Pytest Dev Sprint 2024","link":"https:\/\/glyphack.com\/pytest-dev-sprint-2024\/","pubDate":"Tue, 20 Aug 2024 19:35:10 +0200","guid":"https:\/\/glyphack.com\/pytest-dev-sprint-2024\/","description":"<p>It was about 3 Months ago that a interesting message showed upon my GitHub feed.\nIt was from <a href=\"https:\/\/github.com\/The-Compiler\" rel=\"noopener\" target=\"_blank\">The-Compiler<\/a> arranging a 5 day <a href=\"https:\/\/github.com\/pytest-dev\/sprint\/\" rel=\"noopener\" target=\"_blank\">Pytest development sprint<\/a> in Austria.\nI was following him for his open source work.\nI finally got access to something interesting from following people whom work I like.<\/p>\n<p>I did not know much about the event, my guess was that there is going to be some coding and meeting other Pytest community.\nSo I shared the news with <a href=\"https:\/\/github.com\/farbodahm\" rel=\"noopener\" target=\"_blank\">my friend<\/a>, and we signed up for it.<\/p>\n<p>This was the first in person open source development event I ever joined.\nThe uncertainty give a mix of excitement and nervousness.<\/p>\n<p>The sprint was organized at <a href=\"https:\/\/www.omicronenergy.com\/en\/\" rel=\"noopener\" target=\"_blank\">Omicron<\/a> in <a href=\"https:\/\/www.openstreetmap.org\/#map=15\/47.3075\/9.6207\" rel=\"noopener\" target=\"_blank\">Klaus, Vorarlberg<\/a>.\nWe were staying at a hotel in Feldkirch the hotel was right above the train station.\nEvery morning we took a 10-minute train ride form Feldkirch to Klaus.<\/p>\n<p>Other than us there were 5 pytest maintainers and 2 other contributors from Omicron. Genuinely helpful people.<\/p>\n<p>I worked on two refactorings during that week.\nPytest codebase is very old, I found some pieces from 12 years ago.\nThese are written when a lot of python features were not introduced.\nSo there are a lot of improvements both to code and to functionality by using better methods that are introduced to python.<\/p>\n<p>Pytest codebase is big, has a lot of features, but it was easy for me to navigate through it.\nCode is kept close together, not much abstractions and mostly python structures themselves are used.\nI wish software at companies were like this.<\/p>\n<p>The first issue I fixed was to make pytest fixtures an actual object and showing better error messages when a fixture is used in test summary.\nPreviously fixtures were created by using monkey patching a function to wrap a fixture and in the code there were checks on functions to find if it is a function wrapper or not.\nI replaced it with a new class to contain the fixtures, and it is then easier to check for fixtures in the code using <code>isinstance<\/code> calls.<\/p>\n<p>The second refactor I worked on turned out to be way bigger and ambitious than I imagined.\nIt&rsquo;s related to the beautiful error messages you see when tests fail that show you debug information.\nThis is done by rewriting the assert statement in tests and adding additional information.\nThere was this file in the codebase called <code>assert_rewrite.py<\/code> I basically started to rewrite that file.\nThat file I think is one of the main reasons why pytest is so pleasant to work with.\nIt takes in your plain tests and rewrites the AST to make them more informative and the result is printed out. So when your test fails you see nice error message with debugging traces in the output following your test code that failed.\nThis genius trick required genius code. Which is hard to understand.\nWe wanted to make some adjustments in that part and realized there is an opportunity to rewrite it and make it simpler.\nThere are definitely more ways to do something that was done 11 years ago.\nI don&rsquo;t think there is very big gains in refactoring this part.\nIt&rsquo;s more of a fun challenge and maybe the new code allow adding more features.<\/p>\n<p>That week I had a great time. Felt really alive and happy.\nFrom the morning I woke up and got ready to go and program.\nThe place that we had and the room full of people ready for talking about problems and reviewing your work.<\/p>\n<p>I don&rsquo;t know why it is so much different than having a job as a programmer.\nMaybe <a href=\"https:\/\/world.hey.com\/dhh\/i-won-t-let-you-pay-me-for-my-open-source-d7cf4568\" rel=\"noopener\" target=\"_blank\">money really ruins the joy<\/a>.<\/p>\n<p>On the 4th day we went to visit the water power plant near Feldkirch.\nA lot of stuff were mechanical.\nStats were shown in the mechanical gauges.\nI don&rsquo;t blame them if they don&rsquo;t trust in software enough for this.\nWe can&rsquo;t even get basic CRUD apps to work these days.<\/p>\n<p>I&rsquo;m glad that I had followed Florian, and saw the announcement in my GitHub feed. Another good reason to find good people and follow them on GitHub.<\/p>\n<p>Oh lastly, I started using <a href=\"https:\/\/qutebrowser.org\/doc\/quickstart.html\" rel=\"noopener\" target=\"_blank\">qutebrowser<\/a> during this sprint. I&rsquo;m very happy with my decision. It&rsquo;s an efficient and fast, just like vim. It works with everything except for crappy websites. For example google sometimes does not let you log in to an account from this browser (how much more they have to work to convince you that they are taking control of the web and browsers?).\nBut I&rsquo;m using it as my development browser now. It works perfect for websites showing information. Let&rsquo;s see if I will start creating scripts to have fun and automate some more stuff in the browser.<\/p>\n<p>Follow people you like on GitHub. On LinkedIn, you find show offs and announcements by sales and PR on GitHub you find real things.<\/p>\n"},{"title":"Camping at Vresselse Bos","link":"https:\/\/glyphack.com\/vresselse-camping\/","pubDate":"Wed, 29 May 2024 22:55:24 +0200","guid":"https:\/\/glyphack.com\/vresselse-camping\/","description":"<p>I had some time in between switching jobs and decided to go for a camping trip.\nMy first time in the Netherlands.<\/p>\n<p>After looking up some places I chose to go to <a href=\"https:\/\/nl.wikipedia.org\/wiki\/Vresselse_Bos\" rel=\"noopener\" target=\"_blank\">Vresselse Bos<\/a>\nI rented a tent from Airbnb.\nThe tent was close to a village called Nijnsel.\nThe way to get there was train to Eindhoven and then a bus to Nijnsel, then a 40-minute walk from there.\nAfter about an hour walk to the east side after exiting the village there is a forest.\nThe tent itself and the area surrounding it was quite big.\nAfter entering the fence gate, there was a long path surrounded with trees to the tent.\nInside were wooden furniture that gave you the feeling of living in the woods and a warm blanket to survive.\nAnd most importantly, a large barrel of drinking water.<\/p>\n<p>In front of the tent there was a <a href=\"https:\/\/en.wikipedia.org\/wiki\/Dry_toilet\" rel=\"noopener\" target=\"_blank\">dry toilet<\/a> and shower.\nShower was good, but I soon learned that water cannot stop a hungry mosquito.\nAnd on the right side there was a fire pit with tree trunks around it to sit.<\/p>\n<p>The shirt 24 hours I was too tired to do anything after the long walk with my backpack.\nSo I sat down under the trees and watched birds and sunset.<\/p>\n<p>I started the second day with preparing breakfast.\nI brought food with myself for the whole trip.\nApples and bananas, Lentils and red lentils(the best thing seriously), Potato eggplant Tomato, Eggs, Walnuts and Pistachios.\nThese were easy to keep in a cold area and none of them got rotten.<\/p>\n<p>My favorite meal lentil soup: Pour water over lentil and red lentils with olive oil pepper and salt, leave it for 45 minutes, and then you got a soup.\nWell actually the original recipe does not have red lentils, but I discovered this myself.\nThe other dish I made there was <a href=\"https:\/\/en.wikipedia.org\/wiki\/Mirza_ghassemi\" rel=\"noopener\" target=\"_blank\">Mirza Ghasemi<\/a>.\nIt&rsquo;s very good with grilled eggplant.<\/p>\n<p>I brought some food back because I was nothing eating as much as normal days.\nIt sounds contradictory I&rsquo;m really curious about the reason. Maybe because I was not eating on schedule but just when I was hungry.<\/p>\n<p>On second and third day of the trip, I was super excited and energized.\nI did not have internet, but it was not boring.<\/p>\n<p>I went for hiking, the forest was massive.\nYou could stand in the middle and look around and don&rsquo;t see anything other than trees and plants in horizon.\nThe nature is unpredictable and beautiful.\nNo two footsteps were the same, the height is different or the moisture of the soil.\nI was giving full attention when walking there.\nSome branches falling down occasionally, so I had to keep an eye.\nThere were a lot of canker worms crawling up to a tree.\nI hit a lot of them, and it&rsquo;s hard to get them off, another reason to watch steps.\nSpecially in the first few days I was not paying full attention to the surroundings.<\/p>\n<p>I saw squirrels, cats but no wild big animal there.\nBut there were a lot of insects and I got a lot of bites from them.\nThe most interesting thing I found was an ant nest. There were a lot of them and when they walked I could hear the sound of them walking. I brought them sugar the other day, but they did not like it. They were more interesting in getting their food in the natural way.<\/p>\n<p>It&rsquo;s the complete opposite of daily life were every step and route is predictable.<\/p>\n<p>I then ate lunch and did some reading, then another hike.\nIt was like I can not finish the day.\nI did a lot but still was afternoon.\nThere are a lot of hours in a day if there is no internet.<\/p>\n<p>On forth day I started using my laptop to read blogs and programming.\nI downloaded a few blogs I liked before going there, so I had access to those.\nI had also downloaded Python and Rust docs.\nIf you think it&rsquo;s not possible to program without Google, it is.\nSome people might even say it&rsquo;s not possible to program without ChatGPT, sad.\nIt was a pleasant experience to do these activities in the jungle.\nNo distractions, No time limit, and bird sounds.<\/p>\n<p>In the last days I recorded some videos talking about different stuff, just thinking out loud.\nThey are on my <a href=\"https:\/\/www.youtube.com\/@glyphack\/videos\" rel=\"noopener\" target=\"_blank\">YouTube channel<\/a> and I liked it.\nIt&rsquo;s very similar writing.<\/p>\n<hr>\n<p>Overall the experience was wonderful.\nI got up with bird noises when the sun went up and slept with it went down.\nI was away form the daily life stress.\nWhat time is it? Who cares.\nI was more focused there, there was nothing to steal my attention other than nature.\nI felt different physically as well not sure how much of it was because I was not under stress.<\/p>\n<p>Since I had to spend time on basic needs like food I was more physically active and enjoyed the meals more.\nI also eat less that I normally eat, 1 or 2 meals per day only soup some days.\nAlthough I was more active I was feeling full.\nThe thing was that I was eating more slowly and enjoying the moment.<\/p>\n<p>About the no internet part, well I wanted to turn it off to not have distractions and the micro stress shock that it gives me.\nOf course, I needed the information on it.<\/p>\n<p>I mentioned that I downloaded a few pages before I go there.\nI used this <code>wget<\/code> command to download any site:<\/p>\n<div class=\"code-block\">\n  <pre tabindex=\"0\"><code>wget -m -k -p -E -np --limit-rate=200k<\/code><\/pre>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>And as much as it&rsquo;s hard to believe, it&rsquo;s doable.\nI read and coded for a few hours each day.\nThe only bummer is that you cannot open links inside posts, I just saved them for when I&rsquo;m back.\nIt is kind a good thing because you don&rsquo;t get distracted while reading.\nI&rsquo;m sad that my link collector requires internet to work, time to question choices.<\/p>\n<p>I read a bunch of blogs and books there all worthy to read.<\/p>\n<p>The blogs read there:<\/p>\n<ul>\n<li><a href=\"https:\/\/www.unqualified-reservations.org\" rel=\"noopener\" target=\"_blank\">https:\/\/www.unqualified-reservations.org<\/a><\/li>\n<li><a href=\"http:\/\/www.catb.org\/~esr\/\" rel=\"noopener\" target=\"_blank\">http:\/\/www.catb.org\/~esr\/<\/a><\/li>\n<li><a href=\"https:\/\/ranprieur.com\" rel=\"noopener\" target=\"_blank\">https:\/\/ranprieur.com<\/a><\/li>\n<li><a href=\"https:\/\/datagenetics.com\" rel=\"noopener\" target=\"_blank\">https:\/\/datagenetics.com<\/a> my favorite math puzzle blog<\/li>\n<li><a href=\"https:\/\/starslatecodex.com\" rel=\"noopener\" target=\"_blank\">https:\/\/starslatecodex.com<\/a><\/li>\n<li><a href=\"https:\/\/geohot.github.io\/blog\/\" rel=\"noopener\" target=\"_blank\">https:\/\/geohot.github.io\/blog\/<\/a><\/li>\n<li><a href=\"https:\/\/lukesmith.xyz\" rel=\"noopener\" target=\"_blank\">https:\/\/lukesmith.xyz<\/a><\/li>\n<li><a href=\"https:\/\/notrelated.xyz\" rel=\"noopener\" target=\"_blank\">https:\/\/notrelated.xyz<\/a><\/li>\n<li><a href=\"https:\/\/paulgraham.com\" rel=\"noopener\" target=\"_blank\">https:\/\/paulgraham.com<\/a><\/li>\n<\/ul>\n<p>I also took two longer writings:<\/p>\n<ul>\n<li>Antifragile by Nassim Taleb<\/li>\n<li><a href=\"https:\/\/www.washingtonpost.com\/wp-srv\/national\/longterm\/unabomber\/manifesto.decsn.htm\" rel=\"noopener\" target=\"_blank\">Unabomber Manifesto<\/a> read the Washington post version and spend their bandwidth<\/li>\n<li><a href=\"http:\/\/localroger.com\/prime-intellect\" rel=\"noopener\" target=\"_blank\">The Metamorphosis of Prime Intellect<\/a><\/li>\n<\/ul>\n<p>I also worked on my project <a href=\"https:\/\/github.com\/Glyphack\/enderpy\" rel=\"noopener\" target=\"_blank\">Enderpy<\/a> there, I made great progress toward adding the LSP hover action to the editor.<\/p>\n<p>Now that I&rsquo;m back I try to make my life more like that.\nRemove the stress and distractions.\nDo more physical activities specially to provide my basic needs.\nWorry less about stuff, and use the thinking to think about what I like.<\/p>\n"},{"title":"Why Is It Hard to Do Real Work","link":"https:\/\/glyphack.com\/doing-real-work\/","pubDate":"Fri, 26 Apr 2024 21:48:35 +0200","guid":"https:\/\/glyphack.com\/doing-real-work\/","description":"<p>When I first read <a href=\"https:\/\/www.paulgraham.com\/procrastination.html\" rel=\"noopener\" target=\"_blank\">good and bad procrastination<\/a> by Paul Graham, I was amazed how much my perspective changed about working.<\/p>\n<blockquote>\n<p>There are three variants of procrastination, depending on what you do instead of working on something: you could work on (a) nothing, (b) something less important, or (c) something more important. That last type, I&rsquo;d argue, is good procrastination.<\/p>\n<p>The most dangerous form of procrastination is unacknowledged type-B procrastination, because it doesn&rsquo;t feel like procrastination. You&rsquo;re &ldquo;getting things done.&rdquo; Just the wrong things.\nAny advice about procrastination that concentrates on crossing things off your to-do list is not only incomplete, but positively misleading, if it doesn&rsquo;t consider the possibility that the to-do list is itself a form of type-B procrastination. In fact, possibility is too weak a word. Nearly everyone&rsquo;s is. Unless you&rsquo;re working on the biggest things you could be working on, you&rsquo;re type-B procrastinating, no matter how much you&rsquo;re getting done.<\/p>\n<\/blockquote>\n<p>I learned that the most dangerous way to procrastinate is to do some work but not the important work.\nThe kind of work that results to something great.\nYou can be working your ass off and be busy all the time.\nBut why don\u2019t you have a result from all that work?\nBecause those hours spent working were not going to produce anything at all.<\/p>\n<p>I was thinking what are the ways to do more real work and less procrastination.<\/p>\n<p>I write up a list of things I find interesting and important.\nIt&rsquo;s a weekly list which I carry over to next week.\nWhenever I&rsquo;m doing something I can check if that work was something I found important before or not.<\/p>\n<p>This helps me with finding:<\/p>\n<ul>\n<li>work that I do which I did not put under the list<\/li>\n<li>work that I put under the list but I don&rsquo;t spend time on.\nAnswering both of them is helpful to spend time wisely.<\/li>\n<\/ul>\n<p>One of the ways to avoid doing work is being busy deciding non important stuff.\nThere are many decisions to make daily optimizing all is not possible.\nIn these situations I try to get it done, and move on.\nImagine if it&rsquo;s deciding between something that does not matter much, I&rsquo;d roll a die.\nRolling a dice for stuff that are not the main goal is good because it helps to make a decision and have more time.\nWhen I free up enough time with useless stuff, I get bored then I do something else. I can repeat this process over and over to get to important work finally.<\/p>\n<p>But there&rsquo;s also another kind of errand that does not finish.\nThings like cleaning the house, calling others, checking inboxes.\nI can be doing these stuff day after day.\nIn case of social media every hour(or minute!) so the dice based decision making does not work.\nIn this case, the best way is to resist the temptation.\nFor necessary ones(housekeeping, chatting with friends), schedule a specific amount of time to only do them within the allotted time.<\/p>\n<p>When I decide to do real work, other urgent works appear.\nYet, these tasks are often made up excuses.\nI realize this could be a coping mechanism to avoid the important work I intended to do.\nWhen this happens I pay more attention to the work and try to fix the underlying reason that I don&rsquo;t want to do the work.\nIt might be that it&rsquo;s too big of a work and sounds impossible.\nWhen something doesn&rsquo;t sound doable it reduces the motivation to do it.\nBreaking to smaller achievable tasks help.<\/p>\n<p>Sometimes the distractions are so reachable that avoiding it requires effort.\nIn this case making it harder to reach would be better.\nIf I sit down to write(like right now) I turn off wifi to write the first draft.\nIf I cannot remember something or want to check it I put a note.\nAfter finishing, I can use the internet to address those notes.\nThis gets harder with things like programming where I need to look up libraries or install them, <a href=\"https:\/\/twitter.com\/mitchellh\/status\/1781840288300097896\" rel=\"noopener\" target=\"_blank\">but doable<\/a>.\nPrepare a list of tasks, download the resources offline, get it done.<\/p>\n<p>Sometimes the reason to procrastinate about something comes from how it&rsquo;s done.\nI was watching an old talk from DHH about his experience building 37signals.\nIn the talk <a href=\"https:\/\/youtu.be\/MlhAkNWC1qo?t=1190\" rel=\"noopener\" target=\"_blank\">he mentions<\/a> how he limited the amount of hours he could be putting into building their products.\nHow could limiting the hours produce better results? One reason is that you have no room for procrastination.\nIf I do something for only 1 hour a day, that hour is either spent on the work or gone.\nAnd when it&rsquo;s gone I can feel it because I did not get anything done.\nBut if I don&rsquo;t limit the time, I can be spending the whole day on that task.\nAnd the truth is, having a lot of time for doing something makes room for delays.\nIf something comes up you can say &ldquo;Oh I have the whole day for work so let&rsquo;s get this done before that.&rdquo;\nBut this is not possible with one hour time limit.\nIf you decide to do the errand in that hour you don&rsquo;t get anything done.\nWhich alerts the brain more.\nIt&rsquo;s better to waste time in obvious ways than with fake work.\nBecause fake work requires more effort to notice.<\/p>\n<p>The worst kind of effort is the one that produces nothing and fighting that is the first step to get to boredom. After staring at the blank page for a while, writing will follow.<\/p>\n"},{"title":"Expertise Beyond Validation","link":"https:\/\/glyphack.com\/expertise-beyond-validation\/","pubDate":"Tue, 16 Apr 2024 21:48:35 +0200","guid":"https:\/\/glyphack.com\/expertise-beyond-validation\/","description":"<p>I was walking in Rome a few weeks ago.\nThis idea that less educated people with less resources and money built such beautiful buildings was so fascinating to me.\nEven the most ordinary-looking buildings are constructed with care.\nThey have fantastic arches, curved ceilings.\nThey are a work of art.<\/p>\n<p>We see this pattern everywhere.\nMy favorite writers like <a href=\"https:\/\/sive.rs\/\" rel=\"noopener\" target=\"_blank\">Derek Sivers<\/a>, <a href=\"http:\/\/www.paulgraham.com\/articles.html\" rel=\"noopener\" target=\"_blank\">Paul Graham<\/a>, and <a href=\"https:\/\/slatestarcodex.com\/\" rel=\"noopener\" target=\"_blank\">Scott Alexander<\/a> are not full time authors.\nThe writers who truly changed my perspective and showed me new ways were not just writers, but thinkers.\nThey write, it&rsquo;s in form of blogs mostly.\nEach blog post gives a message, and it&rsquo;s interesting that you don&rsquo;t need a whole book to convey a message.\nAnd there are tons of authors who write books full time but not as good.\nThey can write more being able writing more is not the guarantee to quality.<\/p>\n<p>The same phenomenon happens in software.\nProgrammers outperform full time corporate software engineers.\nSomehow <a href=\"https:\/\/github.com\/excalidraw\/excalidraw\" rel=\"noopener\" target=\"_blank\">Excalidraw<\/a> manages to be faster and more good-looking than Miro, and be open source and free.\nOf course, I&rsquo;m not saying the people who build this superior software are not employees of these companies.\nA lot of these people work there or worked there.\nThe important part is why they can build a better product without having resources these companies can provide?<\/p>\n<p>So there&rsquo;s definitely something that helps some individuals without validations outperform so-called experts in a topic.\nI&rsquo;m calling it validations because in Today&rsquo;s world it&rsquo;s more than only certifications.\nPeople used to be impressed by university degrees only, now by your employer.\nWhile it&rsquo;s not a common thing that causes this, I think it&rsquo;s worth to dig deeper.<\/p>\n<p><strong>Motivation<\/strong><\/p>\n<p>Pursuing mastery and perfection requires strong motivation.\nIt&rsquo;s either (a) an external force, as with ancient builders who constructed for the king, or (b) intrinsic motivation.\nBoth of these forces are really strong.\nThe first one sounds very cruel, but this is the same thing that pushes a startup to succeed.\nThey don&rsquo;t want to <a href=\"https:\/\/paulgraham.com\/die.html\" rel=\"noopener\" target=\"_blank\">die<\/a>.<\/p>\n<p>Now which motivation is better?\nMotivated by money or fear and joy?\nI think the latter is better.\nAlso, money by itself can be a poor motivation sometimes.\nMoney is a proxy to showing value.\nBuilding a product is the value.\nWhenever there&rsquo;s a chance to go for the real thing, proxies are worthless.\nAlso, most of the time you don&rsquo;t get paid based on the quality.\nThis decreases the quality in paid work.<\/p>\n<p>This advantage helps the expert tribe to care more about the work.\nCaring means they make it perfect, not because of the money but because they have unlimited source of motivation.\nThat&rsquo;s why these projects worth doing, money is not the ultimate goal.\nIt&rsquo;s <a href=\"https:\/\/world.hey.com\/dhh\/it-must-be-worth-it-even-if-it-doesn-t-work-1e7f49fc\" rel=\"noopener\" target=\"_blank\">the satisfaction<\/a>.<\/p>\n<p><strong>Practicality<\/strong><\/p>\n<p>People coming without any background to a topic to do something don&rsquo;t know the rules and frameworks.\nThey are there to build something.\nIt&rsquo;s not about following best practices or showing off your theoretical knowledge.\nThey are flexible to use any technique that works.<\/p>\n<p>This is especially true in software.\nA lot of companies are just slowed down due to following industry best practices.\nIf you focus on am I using the right pattern instead of is my product good your will be slow.\nPractices are tools to be used, not the ultimate end goal.\nCompanies can be successful and not follow all the practices.\nThere are a lot of rules, you don&rsquo;t need all of them.\nAnd also there are a lot of things that is <strong>not<\/strong> in the industry, but you need it.\nWhen you are too focused on what you can do and cannot do you loose the option to invent new techniques.\nSometime inventing a new technique is just borrowing from other fields.<\/p>\n<p>So focusing on practical stuff matters.\nDoing whatever is needed in order to succeed matters.\nAirBnb founder <a href=\"https:\/\/twitter.com\/StartupArchive_\/status\/1737446769519124584\" rel=\"noopener\" target=\"_blank\">did the photography of their hosts<\/a> at the beginning.\nIf they were too rigid about what practices they should follow they would not do this.\nThat&rsquo;s why it&rsquo;s so odd to hear this.\nOther professionals rarely do this.<\/p>\n<p><strong>Flexibility<\/strong><\/p>\n<p>People working on their own ideas are flexible.\nThey can focus on what they like to do, and can choose to change what they want to do.<\/p>\n<p>The reason writing blog posts is more flexible than writing a book is that you can discover what you want to write as you write.\nYou are free to jump from topic to topic.\nEven write new topics that discards previous ideas.\nWith a book you need to start with what you want to write.<\/p>\n<p>There&rsquo;s also the problem of a deadline.\nFor example a software project with a deadline needs to finish on time.\nWhen it&rsquo;s not finished the project manager needs to explain why to their bosses.\nSo they get some more time and push for the project to finish.\nIt&rsquo;s obvious that a lot of corners are cut here to meet the deadline.\nBut there are also a lot of corners cut in startups.\nThen why they are different?\nIn startups, it&rsquo;s so easy to discard previous code and write another version.\nReddit did this, they even changed the programming language from Lisp to Python.\nThis requires a lot of sign off and meetings in companies.\nSo if you end up with a half ass work at a company it will be there for a long time.\nBut in a startup, sure no problem just rewrite it next week.\nMake it better, and have fun.<\/p>\n<p><strong>Fear of Failure<\/strong><\/p>\n<p>Certified people in a topic tend to fear more from failure.\nWhen failure happens it&rsquo;s like as if their certifications were not valid, which takes away the credibility.\nOutsiders have nothing to lose.\nThey either have skill and it&rsquo;s in the result or don&rsquo;t.\nIf they realize they are not skilled enough is not a bad thing, they practice more.\nWhat are you going to do about certifications? Get another one?<\/p>\n"},{"title":"Best Place to Work at as a Programmer","link":"https:\/\/glyphack.com\/best-job\/","pubDate":"Fri, 12 Apr 2024 20:39:20 +0200","guid":"https:\/\/glyphack.com\/best-job\/","description":"<p>Despite the title this isn&rsquo;t meant to name a specific company.\nI&rsquo;m trying to figure out for people like myself who enjoy programming what the best place to work would be like.<\/p>\n<p>I think there&rsquo;s fundamental problem at most companies right now.\nEverything is fake. Of course, we have goals, but they are artificial.\nThey are there because companies need goals, to show to investors.\nBut they are not the real goals of the company. The goal is the customers, and the problem it solves.\nBut this way is <a href=\"https:\/\/geohot.github.io\/blog\/jekyll\/update\/2023\/07\/20\/a-disgusting-playbook-copy.html\" rel=\"noopener\" target=\"_blank\">the playbook<\/a> for getting money and being famous.\nThis is not true for all the companies, some are actually solving problems selling product and making money.\nA miner knows the value in their work because they create wealth.\nSometimes I don&rsquo;t even know where my salary comes from.<\/p>\n<p>But it&rsquo;s also not so easy to get into good companies.\nHaving those skills is not something that comes from working on a regular software dev job.\nYou&rsquo;re not gonna suddenly jump from building web forms to building autonomous cars, it requires other skills, and practice.<\/p>\n<p>So ultimately I think some companies are places were you can find value in the work.\nThey are a small number, and hard to get into.\nThe next point is also in successful companies not all the teams are doing work equally valuable.\nThat means just passing the interview and getting in is not enough, you have to get into the right team, with the right skill.\nIn the end it&rsquo;s not just a name, it&rsquo;s the skill and people you work with that matters.\nYou might set the goal to work at a good company thinking it leads to valuable work, but that&rsquo;s the wrong goal.<\/p>\n<p>So what&rsquo;s the way to escape? Open source.\nOpen source is the place where results and performance matters.\nIn open source you just don&rsquo;t get promoted because the budget allows that. You build useful stuff and acquire skill.\nIt&rsquo;s not about artificial titles and goals anymore, it&rsquo;s about what actually is being built.<\/p>\n<p>That&rsquo;s why I like companies that start with open source more than others.\nCompanies like <a href=\"https:\/\/comma.ai\/\" rel=\"noopener\" target=\"_blank\">Comma<\/a>, Sentry, Hashicorp, and <a href=\"https:\/\/astral.sh\/\" rel=\"noopener\" target=\"_blank\">Astral<\/a>, they all had an excellent product before acting like a big company.<\/p>\n<p>The next question is how to get to work on things that are actually valuable?\nUnlike big companies that have a big gate keeping you out open source is open to everyone.\nIf today you want to work with a more skilled person than yourself on a project, open source allows that.\nJust find a project, read the code and improve it.\nYou get to work with people created a valuable programming language, a database, a web framework.\nWithout having to pass an interview or any other gate.\nThe interesting part is, many of good people that are at big companies, that you want to work with, are in open source communities.<\/p>\n<p>It can be even easier to get in touch with them in open source than to get a job at their company.\nI&rsquo;ve had better chance to contact someone asking questions and feedback in open source and getting feedback than senior people in companies.\nThey are just more available.<\/p>\n<p>That&rsquo;s why it&rsquo;s the best place to work at.\nYou don&rsquo;t have to play interview game to get in, then play the politic game to get do the things you find valuable.\nIt requires more effort to work, that&rsquo;s expected.\nYou are not doing anything other than the important work itself, so the work seems harder.\nFill your day with meetings and discussions, and you see how easier (and more pointless) it gets.<\/p>\n<p>But open source doesn&rsquo;t pay.\nThat is true and there&rsquo;s not much to do about it.\nSpecially when you are just starting there&rsquo;s nothing to get money from.\nBut I think after building something useful, it&rsquo;s possible to get money out of it.\nYou just need the product first.\nThese days the situation is better, VCs&rsquo; are putting funding open source projects.\nBut that should not be a goal, it&rsquo;s just an indicator that there&rsquo;s money.\nYou won&rsquo;t starve. You get to do valuable work, and enjoy it.<\/p>\n<p>What makes open source more sustainable than a low paying job is that you can work on valuable things.\nJobs are structured in a way that pay is not directly related to the value.\nWithout garbage collectors, the world would be a mess, but they are not paid well.\nSame phenomenon described <a href=\"https:\/\/strikemag.org\/bullshit-jobs\/\" rel=\"noopener\" target=\"_blank\">Bullshit Jobs<\/a>.<\/p>\n"},{"title":"How to infer type for Generic types in Python?","link":"https:\/\/glyphack.com\/python-generics-type-inference\/","pubDate":"Mon, 25 Mar 2024 16:46:15 +0100","guid":"https:\/\/glyphack.com\/python-generics-type-inference\/","description":"<p>I implemented Generic classes and functions this week in Enderpy. I&rsquo;m happy that now my code can now read and infer types of <a href=\"https:\/\/github.com\/python\/typing\/blob\/main\/conformance\/tests\/generics_basic.py#L114\" rel=\"noopener\" target=\"_blank\">this conformance test<\/a>.<\/p>\n<p>I chose to skip implementing generics with syntax <code>def f[T](): T<\/code> because of the <a href=\"https:\/\/peps.python.org\/pep-0695\/#scoping-behavior\" rel=\"noopener\" target=\"_blank\">scoping behavior<\/a>.\nThey are also tested extensively in this <a href=\"https:\/\/github.com\/python\/typing\/blob\/main\/conformance\/tests\/generics_syntax_scoping.py\" rel=\"noopener\" target=\"_blank\">test case<\/a>.<\/p>\n<p>Another strange thing I found in the <code>typeshed<\/code> repo is that in the <code>sys\/__init__.py<\/code> file there is a <code>import sys<\/code> in the beginning.\nMaking this a cyclic import. I think the reason is to use the <code>sys.version<\/code> and <code>sys.platform<\/code> in the type definitions.\nBut in the type checker I manually skip resolving this import because it resolves to itself.<\/p>\n<p>The current implementation I came up with for the generic does not infer the actual type of generic in the type evaluation phase. So if the type checker asks the type of parameter that is generic type it gets back a generic parameter node in the returned type.\nI&rsquo;m planning to add the functionality to infer the type of the generic parameter based on the types passed as the generic type to type checker.<\/p>\n<p>This is how the code works right now:<\/p>\n<ol>\n<li>The type parameters are inserted in the symbol table like other variables.<\/li>\n<li>When the type evaluator resolves a type annotation that is referring to typing. TypeVar it considers that a type parameter type.<\/li>\n<li>When the classes have a base class of <code>typing.Generic[T]<\/code> the type evaluator tries to find the type parameter, and adds the type parameter to the inferred class type.<\/li>\n<\/ol>\n<p>The next step is to continue the generics test cases and implement the type inference for the generic types.<\/p>\n"},{"title":"Don't be afraid to Rewrite","link":"https:\/\/glyphack.com\/rewrite\/","pubDate":"Tue, 12 Mar 2024 16:24:34 +0100","guid":"https:\/\/glyphack.com\/rewrite\/","description":"<p>I recently started to notice this pattern that some teams create products and move on. Maybe some maintenance until no one sends more bug reports.\nThey just literally move on from that product, to the next one.<\/p>\n<p>Now this is not new, when I was a consultant this was exactly what companies expect. Work on something and make it to the finish line and leave.\nIt makes sense, you don&rsquo;t want to keep them around because it&rsquo;s costly.\nBut at a product company? This does not make sense, why wouldn&rsquo;t you want the product to grow?<\/p>\n<p>A typical example is that you are working on a new product for the company. You usually have a timeline of when are you going to launch, when to do testing and when to fully release.\nYou work during this time, and after making it to the finish line, you release. Then the company expects all your focus and energy on the next thing.\nThis happens because the company wants to grow in many areas, so they don&rsquo;t want your focus in only on area.<\/p>\n<p>But the problem is they do it in a way that if you touch a software after deadline it looks like a bad thing. It looks like you missed the release.\nBut it&rsquo;s actually most of the time you have more information when you work on something.\nThis expectation is what I don&rsquo;t think is right, that you need to deliver and project should be good without you. Otherwise, they say you created something that needs maintenance and is considered bad.<\/p>\n<p>But this is wrong.<\/p>\n<p>The first reason is that when you develop the product, you learn more.\nSo after doing the initial development.\nThere are just some things that you can never discover in the requirements phase.\nThe reason is not that the person did a bad job in the design or coding.\nWhen you make something for the first time, you don&rsquo;t know some unknowns.\nIt&rsquo;s because they will know more when they make something that works. And for any problem you just realize the problems with your solution when you apply it.\nSo if you stop the development after the release, you loose that extra knowledge.\nUse the initial development phase to discover them, and even ignore them but fix them afterwards.<\/p>\n<p>The other problem with this mindset is that people stop caring about the product after it goes live.\nIf they find issues, they tend to ignore them if they are small enough.\nThey would not care about 2% of the cases where the endpoint times out.\nBecause, if you committed to the date and delivered, if you go back and touch the product, it means you did not deliver it on time.\nNow if this is applied to the every feature and product you are left with a product that has issues in every feature.\nOne page is slow, another one crashes. They only happen to you because you are in that 2%.<\/p>\n<p>I don&rsquo;t consider rewriting and refactoring bad even after you deliver.\nThere are a lot of things that are just not discoverable before you dive deep into a problem.\nSo by encouraging people to deliver and forget, you create this culture that everything is created to be good enough for release.<\/p>\n<p>I don&rsquo;t mind rewriting what I worked on for few days or weeks. It&rsquo;s better than lying to yourself that I made a technical debt. It&rsquo;s a lie because it&rsquo;s not a tech debt, it&rsquo;s a defect, and you know it.<\/p>\n<p>So if you do something for the first time. Expect to fail to come up with the right solution.\nAccept the fact that you might do it wrong, and need to rewrite it.\nTeams, should not consider this as a delayed lunch.\nIt&rsquo;s just that the product is live but, we want to make it better.<\/p>\n<p>That&rsquo;s all I had to say. I saw this pattern in consultancy and said okay.\nBut I also saw it in product companies, and I&rsquo;m shocked. I don&rsquo;t want to see it in my own work.<\/p>\n<p>So don&rsquo;t be afraid to realized you made a mistake, that&rsquo;s part of the journey.\nUse it and rewrite your software to be better.<\/p>\n"},{"title":"Devlog 1: Contributing to Ruff, Profiling, Python Types Conformance Tests","link":"https:\/\/glyphack.com\/dv-1\/","pubDate":"Sat, 03 Feb 2024 10:56:48 +0100","guid":"https:\/\/glyphack.com\/dv-1\/","description":"<p>Last week I wanted to start contributing to rust.\nI was working on Adding <a href=\"https:\/\/github.com\/astral-sh\/ruff\/pull\/9513\" rel=\"noopener\" target=\"_blank\">uninitialized attribute access check<\/a> to Ruff.<\/p>\n<p>I did it and learned a lot about how to track attributes in Python code.\nA gist of it would be, you need to go over the class, in each function when something is assigned to a name you need to check if that name is self or cls.\nAnd you do this by checking if it matches the first argument of that function.\nThen if the function is a class method it&rsquo;s cls and otherwise self.<\/p>\n<p>I also learned about profiling.\nAfter finishing the implementation I realized the benchmarks are failing.\nSo I need see how did I mess up the performance. It is not because of the rule but because of the code I added to the visitor to keep track of attribute initialization and access.\nBut first we need to profile.<\/p>\n<p>I found two resources for doing it.\nMaybe I can do a separate note on Rust profiling.\n<a href=\"https:\/\/nnethercote.github.io\/perf-book\/\" rel=\"noopener\" target=\"_blank\">Rust Performance Book<\/a> which has a profiling section.\n<a href=\"https:\/\/docs.astral.sh\/ruff\/contributing\/#profiling-projects\" rel=\"noopener\" target=\"_blank\">Amazing guide for Ruff Only<\/a><\/p>\n<p>The interesting part is that Macos is not good for profiling, or at least I could not easily learn to use the tools.\nI used cargo instruments, the output can be opened with instruments app. Instruments app is dog shit.\nI expected some kind of home page, documentation or something when I search for it like <a href=\"https:\/\/jetbrains.com\/help\/idea\/profiler-intro.html\" rel=\"noopener\" target=\"_blank\">this<\/a>.\nBut nothing.<\/p>\n<p>So I could not find traces for the functions I added(skill issue.) I gave up.<\/p>\n<p>In the end I ended up using <a href=\"https:\/\/github.com\/mstange\/samply\" rel=\"noopener\" target=\"_blank\">Samply<\/a> which was better.<\/p>\n<p>I also used the cargo benchmark and <a href=\"https:\/\/github.com\/BurntSushi\/critcmp\" rel=\"noopener\" target=\"_blank\">critcmp<\/a> to compare results between my commits and found the perf issue.<\/p>\n<p>It was caused because I added a new vector to each scope to keep track of undefined attribute accesses.\nBut I realized I can just have a global vector for the whole file and store the undefined attribute along with it&rsquo;s scope.<\/p>\n<p>With a vector on every scope and many allocations:<\/p>\n<div class=\"code-block\">\n  <pre tabindex=\"0\"><code>linter\/default-rules\/large\/dataset.py       1.00    455.7\u00b16.28\u00b5s    89.3 MB\/sec    1.14   519.6\u00b119.69\u00b5s    78.3 MB\/sec\nlinter\/default-rules\/numpy\/ctypeslib.py     1.00     86.9\u00b11.62\u00b5s   191.5 MB\/sec    1.14     99.3\u00b16.53\u00b5s   167.8 MB\/sec\nlinter\/default-rules\/numpy\/globals.py       1.00     12.5\u00b10.19\u00b5s   236.4 MB\/sec    1.05     13.1\u00b10.10\u00b5s   225.1 MB\/sec\nlinter\/default-rules\/pydantic\/types.py      1.00    194.5\u00b14.85\u00b5s   131.2 MB\/sec    1.16   226.3\u00b142.70\u00b5s   112.7 MB\/sec\nlinter\/default-rules\/unicode\/pypinyin.py    1.00     31.7\u00b10.29\u00b5s   132.6 MB\/sec    1.06     33.7\u00b11.64\u00b5s   124.7 MB\/sec<\/code><\/pre>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>After using a global vector for the whole program:<\/p>\n<div class=\"code-block\">\n  <pre tabindex=\"0\"><code>linter\/default-rules\/large\/dataset.py       1.00    455.7\u00b16.28\u00b5s    89.3 MB\/sec    1.03    469.9\u00b15.46\u00b5s    86.6 MB\/sec\nlinter\/default-rules\/numpy\/ctypeslib.py     1.00     86.9\u00b11.62\u00b5s   191.5 MB\/sec    1.02     88.8\u00b11.47\u00b5s   187.6 MB\/sec\nlinter\/default-rules\/numpy\/globals.py       1.00     12.5\u00b10.19\u00b5s   236.4 MB\/sec    1.04     13.0\u00b10.10\u00b5s   227.1 MB\/sec\nlinter\/default-rules\/pydantic\/types.py      1.00    194.5\u00b14.85\u00b5s   131.2 MB\/sec    1.03    201.2\u00b14.70\u00b5s   126.8 MB\/sec\nlinter\/default-rules\/unicode\/pypinyin.py    1.00     31.7\u00b10.29\u00b5s   132.6 MB\/sec    1.03     32.7\u00b10.28\u00b5s   128.7 MB\/sec<\/code><\/pre>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>I also learned that codespeed is a wonderful tool for exploring performance changes between my commits.\n<a href=\"https:\/\/codspeed.io\/astral-sh\/ruff\/branches\/Glyphack:linter-pylint-E0203\" rel=\"noopener\" target=\"_blank\">Example<\/a>, next time I use this.<\/p>\n<p>For enderpy I was looking for a test suite that I can develop against until my type checker is complete.\nLuckily it exists! You can view it <a href=\"https:\/\/github.com\/python\/typing\/tree\/main\/conformance\" rel=\"noopener\" target=\"_blank\">here<\/a>.\nIt does not have a basic test case were you only have functions and variables but that one is easy to come up with myself.<\/p>\n"},{"title":"Five Thousands Lines of Kotlin","link":"https:\/\/glyphack.com\/five-thousands-lines-of-kotlin\/","pubDate":"Sun, 21 Jan 2024 14:43:03 +0100","guid":"https:\/\/glyphack.com\/five-thousands-lines-of-kotlin\/","description":"<p>This post is about my impression of Kotlin as a language after, well you guessed it, writing about five thousand lines of it. I will go through what I don&rsquo;t like about it and whether I would use this language on my own(spoiler: I won&rsquo;t unless I have to.) I started using Kotlin only because of the new job. In my job I worked on server applications only, with the added spice of enterprise software practices.<\/p>\n<p>You&rsquo;ve probably heard that Kotlin is much better compared to Java. That is true, it&rsquo;s like a unicorn compared to a horse, but in the end it&rsquo;s just Java with extra features. Kotlin has more features for dealing with nulls, the best feature is the <a href=\"https:\/\/kotlinlang.org\/docs\/null-safety.html#safe-calls\" rel=\"noopener\" target=\"_blank\"><code>.?<\/code> operator<\/a>. This allows you to eliminate the <code>if (x == null)<\/code> from code. I wish they had the same for exceptions.<\/p>\n<p><strong>Exception Handling<\/strong><\/p>\n<p>Continuing with the language features, I think Kotlin lacks tools for <a href=\"https:\/\/kotlinlang.org\/docs\/exceptions.html#checked-exceptions\" rel=\"noopener\" target=\"_blank\">specifying exception types<\/a> thrown by a function, at least in Java you can annotate what exceptions a function can raise, but Kotlin does not support that.\nI think Java is even better in this regard.\nThis results in ugly try catches on <code>Exception<\/code> everywhere.\nI&rsquo;m not sure why they thought it was a good idea.\nAnyone who used a language with support for specifying error types in the function signature knows that it makes it much easier to use the functions and handle exceptions correctly.\nIn contrast in Kotlin you either have to read through the docs or function code and hope you handled each exception. If you really don&rsquo;t want to fail, you are left with ugly try caches everywhere because you need to catch all exceptions from each statement.<\/p>\n<p><strong>Editor Support<\/strong><\/p>\n<p>Kotlin does not have an official LSP.\nLSP stands for Language Server Protocol. Almost all editors work with LSP to provide autocompletion and diagnostics. So the red lines you see in VScode telling what is wrong comes from that.<\/p>\n<p><a href=\"https:\/\/discuss.kotlinlang.org\/t\/any-plan-for-supporting-language-server-protocol\/2471\" rel=\"noopener\" target=\"_blank\">The reason<\/a> JetBrains does not invest time on LSP is because they want to spend the time on their own editor.\nI understand they created really good IDEs and obviously they have a good editor for Kotlin.\nBut drifting away from open source standards makes it harder for people to use language where they are comfortable.<\/p>\n<p>I used kotlin-language-server while using Kotlin.\nOne of my problems with it is the slow startup. But this is improved in the recent months. I also <a href=\"https:\/\/github.com\/neovim\/nvim-lspconfig\/pull\/2930\" rel=\"noopener\" target=\"_blank\">added this feature<\/a> to Neovim lsp config which helped a lot.\nKotlin is really complex as a language for a couple of people to maintain a language server for it.<\/p>\n<p><strong>Language Features<\/strong><\/p>\n<p>Speaking of how hard it is to maintain a language server for Kotlin we get to the next point.\nI think Kotlin has a lot of features, and that is what makes it hard to analyze Kotlin code:<\/p>\n<ul>\n<li>Class modifiers like <code>open<\/code><\/li>\n<li>There are 4 visibility modifiers<\/li>\n<li>Classes can have 2 constructors\nThese are not very complex for end user but makes it harder for language tool developers.\nI don&rsquo;t think this is needed. This probably satisfies people with boundary fetishes, but you don&rsquo;t need them to get the job done.<\/li>\n<\/ul>\n<p>But there are also features that makes it harder for end users to reason about the code.<\/p>\n<p>The <a href=\"https:\/\/kotlinlang.org\/docs\/extensions.html\" rel=\"noopener\" target=\"_blank\">extension functions<\/a> are one, you can mess up the <code>this<\/code> pointer in a class very easily:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-kotlin\" data-lang=\"kotlin\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">class<\/span> Hello {\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">fun<\/span> <span style=\"color:#b57614\">sayHi<\/span>() {\n<\/span><\/span><span style=\"display:flex;\"><span>        print(<span style=\"color:#79740e\">&#34;I say &#34;<\/span>.toHi())\n<\/span><\/span><span style=\"display:flex;\"><span>    }\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">fun<\/span> <span style=\"color:#b57614\">String<\/span>.toHi(): String {\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">return<\/span> <span style=\"color:#af3a03\">this<\/span>.uppercase() + <span style=\"color:#79740e\">&#34;hi&#34;<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>    }\n<\/span><\/span><span style=\"display:flex;\"><span>}\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">fun<\/span> <span style=\"color:#b57614\">main<\/span>() {\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">val<\/span> h = Hello()\n<\/span><\/span><span style=\"display:flex;\"><span>    h.sayHi()\n<\/span><\/span><span style=\"display:flex;\"><span>}<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>In this code the <code>this<\/code> in <code>toHi()<\/code> function is not referring to the Hello class anymore.<\/p>\n<p>The next one is the <a href=\"https:\/\/kotlinlang.org\/docs\/type-safe-builders.html\" rel=\"noopener\" target=\"_blank\">builder syntax<\/a>(it&rsquo;s ironically called type safe builders).\nwith builders, you can write code like:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-kotlin\" data-lang=\"kotlin\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">class<\/span> Hello {\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">fun<\/span> <span style=\"color:#b57614\">sayHi<\/span>(str: String) {\n<\/span><\/span><span style=\"display:flex;\"><span>        buildHello {\n<\/span><\/span><span style=\"display:flex;\"><span>         name=str\n<\/span><\/span><span style=\"display:flex;\"><span>        }.say()\n<\/span><\/span><span style=\"display:flex;\"><span>}<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>This code can create a new class inside the buildHello scope, and then call the say function on it. This looks nice but imagine you want to make the argument name clearer:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-kotlin\" data-lang=\"kotlin\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">fun<\/span> <span style=\"color:#b57614\">sayHi<\/span>(name: String) {\n<\/span><\/span><span style=\"display:flex;\"><span> buildHello {\n<\/span><\/span><span style=\"display:flex;\"><span>  name=name\n<\/span><\/span><span style=\"display:flex;\"><span> }.say()<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>Now the point you missed is that name in the buildHello scope is referring to the attribute name of the builder. So that name is not referencing the name in the argument. This is seriously tricky to catch.<\/p>\n<p>You might look and say hey, this makes my code cleaner. Yeah sure, one minute you&rsquo;re writing clean, concise code, and the next you&rsquo;re lost in a sea of extension functions and builders. Good luck debugging.<\/p>\n<p>So my final take on the language features, it&rsquo;s too much!<\/p>\n<p><strong>Libraries<\/strong><\/p>\n<p>Kotlin has access to JVM libraries. Which is a big plus.\nThe only problem I faced was that the null safety is not fully respected when using Java code with <code>NotNull<\/code> annotation.\n@NotNull, indicates that we must never call our method with a\u00a0null\u00a0if we want to avoid an exception.\nWhich is kind of a surprise because being null safe means being null safe everywhere.\nBut in these functions you can call something that can be null.\nIntelliJ and the compiler won&rsquo;t complain about this as well.<\/p>\n<p>But same with Java I don&rsquo;t like that the libraries in the ecosystem rely so much language features as their API. Just give me a goddamn function not an annotation, or a builder, or plugins.<\/p>\n<p>There are some Ktor features that fail during the runtime because it cannot find some plugin that is needed to work properly. Why must this be a runtime problem?<\/p>\n<p><strong>Why Kotlin Won and Not Scala?<\/strong><\/p>\n<p>This question annoys my mind a lot. I also worked with Scala and I think it&rsquo;s certainly a better language. Maybe that&rsquo;s for another post.<\/p>\n<p>But I also stopped doing Scala for one reason, the functional programming community took over the language and every library tried to add functional paradigm and type safety to everything (talking to you, ZIO). Again, just expose a goddamn function.<\/p>\n"},{"title":"A Better Keyboard","link":"https:\/\/glyphack.com\/better-keyboard\/","pubDate":"Sat, 30 Dec 2023 10:54:35 +0100","guid":"https:\/\/glyphack.com\/better-keyboard\/","description":"<p>Imagine you want to make a better keyboard.\nSeems like a hard challenge for every company.\nWhat if &lsquo;better&rsquo; meant compressing multiple key presses into one? Or have shortcut for your frequent actions.\nIf you minimize the effort to use it then you are making it better for yourself.<\/p>\n<p>Challenge lies in the keyboard&rsquo;s limited keys, and hard to press combinations like <code>ctrl + alt + any key<\/code>.<\/p>\n<p>Most of us learn to use tools as they are.\nBut most of the time the product is not tailored to your needs out of the box.<\/p>\n<p>None of the products produced are going to be designed based on your specific needs.\nOne of the advantages of trying to use keyboard to make repetitive tasks easier is to reduce the attention needed for them.\nWhen these tasks will be easy enough that doing them <a href=\"https:\/\/www.scattered-thoughts.net\/writing\/moving-faster\/\" rel=\"noopener\" target=\"_blank\">won&rsquo;t require attention<\/a>.\nJust like how when you learn touch typing, and suddenly you are just writing instead of looking at the keyboard, or frequently press the wrong key.\nYou get faster.<\/p>\n<p>Before going into details, keep in mind that the goal is to make your workflow easier.\nSome of these suggestions might be useful and others might be not.\nTake away the ideas with yourself and adjust it accordingly.<\/p>\n<p>I implemented the improvements using the following tools:<\/p>\n<ul>\n<li><a href=\"https:\/\/www.hammerspoon.org\/\" rel=\"noopener\" target=\"_blank\">Hammerspoon<\/a><\/li>\n<li><a href=\"https:\/\/karabiner-elements.pqrs.org\/\" rel=\"noopener\" target=\"_blank\">Karabiner<\/a><\/li>\n<\/ul>\n<p>These tools are exclusively for Mac. There are other alternatives for other platforms.<\/p>\n<h2 class=\"heading\" id=\"remapping-keys\">\n  Remapping Keys\n  <a class=\"anchor\" href=\"#remapping-keys\">#<\/a>\n<\/h2>\n<p>Our keyboards are not designed for heavy usage of shortcuts.\nSo you can start making shortcuts for stuff by setting them to <code>ctrl + T<\/code>,<\/p>\n<p>This method presents two significant challenges:<\/p>\n<ul>\n<li>The easily accessible keys are already assigned, such as <code>CMD + T<\/code>.<\/li>\n<li>Complex combinations become cumbersome: try pressing <code>CMD + ALT + ctrl + T<\/code>.<\/li>\n<\/ul>\n<p>The idea is that some keys can be used to do more than one thing.\nWhat keys can be used like this? Let&rsquo;s take a look at different keys.<\/p>\n<ul>\n<li>Keys you hold down to change how <em>other<\/em> keys behave, but that (usually) don&rsquo;t do anything if you use them on their own (like Shift and Control).\n<ul>\n<li><code>Shift<\/code><\/li>\n<li><code>Control<\/code><\/li>\n<li><code>Alt<\/code><\/li>\n<li><code>Command<\/code><\/li>\n<li><code>Fn<\/code><\/li>\n<\/ul>\n<\/li>\n<li>Keys that you press and release but don&rsquo;t want to &ldquo;repeat&rdquo; as you hold them (like Escape or Insert).\n<ul>\n<li><code>Escape<\/code><\/li>\n<li><code>Caps lock<\/code><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>You can use the keys that are designed to only be held to do a new thing if they are pressed. Or use the keys that are designed to be pressed to do another thing if they are held.<\/p>\n<p>For example, I have set the following setting for Caps lock keys:<\/p>\n<ul>\n<li>On hold: hyper key <code>ctrl + SHIFT + ALT<\/code><\/li>\n<li>On press: escape<\/li>\n<\/ul>\n<p>Why hyper?\nBecause this new key press cannot conflict with any other shortcuts\nThis allows you to create shortcuts like: hyper + H\/J\/K\/L which is pretty comfortable to press.<\/p>\n<p>I used to have CAPS lock set to <code>CMD + ctrl + SHIFT + ALT<\/code>.\nBut I noticed that there is a MacOS specific key binding for taking a system snapshot with <code>CMD + ctrl + shift + alt + ,<\/code> which cannot be disabled and my system froze when I accidentally pressed this key.\nSo I stopped using it and switched to the above combination instead.<\/p>\n<p>You can do this using a <a href=\"https:\/\/karabiner-elements.pqrs.org\/docs\/manual\/configuration\/configure-complex-modifications\/#create-your-own-rules\" rel=\"noopener\" target=\"_blank\">complex modification<\/a> in Karabiner:<\/p>\n<div class=\"code-block\">\n  <pre tabindex=\"0\"><code>{\n    &#34;description&#34;: &#34;Capslock to Hyper&#34;,\n    &#34;manipulators&#34;: [\n        {\n            &#34;description&#34;: &#34;Click to Capslock, Hold to Hyper&#34;,\n            &#34;from&#34;: {\n                &#34;key_code&#34;: &#34;caps_lock&#34;,\n                &#34;modifiers&#34;: {\n                    &#34;optional&#34;: [\n                        &#34;any&#34;\n                    ]\n                }\n            },\n            &#34;to&#34;: [\n                {\n                    &#34;key_code&#34;: &#34;right_shift&#34;,\n                    &#34;modifiers&#34;: [\n                        &#34;right_control&#34;,\n                        &#34;right_option&#34;\n                    ]\n                }\n            ],\n            &#34;to_if_alone&#34;: [\n                {\n                    &#34;key_code&#34;: &#34;escape&#34;\n                }\n            ],\n            &#34;type&#34;: &#34;basic&#34;\n        }\n    ]\n}<\/code><\/pre>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>This key now can be used as your new shortcut key.\n<code>hyper + t<\/code> can be mapped to an action globally and does not conflict with anything.\nSo any key on the keyboard can now be used for shortcuts, allowing numerous customization.<\/p>\n<p>If there are keys on your keyboard that you don&rsquo;t use you can map them to frequently used keys.\nFor example for vim users, the right command key on macs can be remapped to control.\nThis makes pressing vim shortcuts like <code>ctrl + A<\/code> easier.<\/p>\n<p>I have a split keyboard, so keys under my thumbs are easy to press, and I remapped them to do more stuff than usual.<\/p>\n<p>\n\n\n\n\n\n\n\n<figure class=\"\">\n\n    \n    \n      \n        <div class=\"img-container\">\n            <img loading=\"lazy\" alt=\"\" src=\"https:\/\/glyphack.com\/better-keyboard\/split-keyboard-remapping.excalidraw.svg\">\n        <\/div>\n      \n    \n\n    \n<\/figure>\n<\/p>\n<p>I changed a lot of keys since then but the ideas are useful. Just find the keys that works best for you.<\/p>\n<h2 class=\"heading\" id=\"window-switching\">\n  Window Switching\n  <a class=\"anchor\" href=\"#window-switching\">#<\/a>\n<\/h2>\n<p>There are some default hotkeys on every system like <code>alt+tab<\/code>.\nThis gives you a bit more advantage over switching windows with a mouse, <a href=\"https:\/\/www.youtube.com\/watch?app=desktop&amp;v=Px0_8J0Wb-s\" rel=\"noopener\" target=\"_blank\">but there is room for improvements<\/a>.\nThese shortcuts are designed for general problems.\nYou can improve it for your own workflow.<\/p>\n<p>First example is alt tabbing to switch windows.\nThis simple thing that you probably do 100 times a day requires to:<\/p>\n<ul>\n<li>Take hand off home row<\/li>\n<li>Press them multiple times to find the window you want<\/li>\n<li>And if you have multiple windows then press <code>ctrl+tab<\/code> or <code>command+~<\/code> to get there\nWouldn&rsquo;t it be good if you could do 80% of these window switches with a single shortcut that is more ergonomic?\nIt depends, I switch between Browser, terminal and note app multiple times most of the time.\nYou can assign a hotkey for these and only use alt tab for when you need to switch infrequently used windows.<\/li>\n<\/ul>\n<p>Here&rsquo;s the solution I use based on <a href=\"https:\/\/rakhesh.com\/coding\/using-hammerspoon-to-switch-apps\/\" rel=\"noopener\" target=\"_blank\">this post<\/a><\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-lua\" data-lang=\"lua\"><span style=\"display:flex;\"><span>WINDOW_MANAGEMENT_KEY <span style=\"color:#af3a03\">=<\/span> { <span style=\"color:#79740e\">&#34;alt&#34;<\/span>, <span style=\"color:#79740e\">&#34;command&#34;<\/span>, <span style=\"color:#79740e\">&#34;ctrl&#34;<\/span> }\n<\/span><\/span><span style=\"display:flex;\"><span>WINDOWS_SHORTCUTS <span style=\"color:#af3a03\">=<\/span> {\n<\/span><\/span><span style=\"display:flex;\"><span> { <span style=\"color:#79740e\">&#34;J&#34;<\/span>, <span style=\"color:#79740e\">&#34;Brave Browser&#34;<\/span> },\n<\/span><\/span><span style=\"display:flex;\"><span> { <span style=\"color:#79740e\">&#34;K&#34;<\/span>, <span style=\"color:#79740e\">&#34;WezTerm&#34;<\/span> },\n<\/span><\/span><span style=\"display:flex;\"><span>}\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">local<\/span> <span style=\"color:#af3a03\">function<\/span> <span style=\"color:#b57614\">launchOrFocusOrRotate<\/span>(app)\n<\/span><\/span><span style=\"display:flex;\"><span> <span style=\"color:#af3a03\">local<\/span> focusedWindow <span style=\"color:#af3a03\">=<\/span> hs.window.focusedWindow()\n<\/span><\/span><span style=\"display:flex;\"><span> <span style=\"color:#af3a03\">if<\/span> focusedWindow <span style=\"color:#af3a03\">==<\/span> <span style=\"color:#af3a03\">nil<\/span> <span style=\"color:#af3a03\">then<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>  hs.application.launchOrFocus(app)\n<\/span><\/span><span style=\"display:flex;\"><span>  <span style=\"color:#af3a03\">return<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span> <span style=\"color:#af3a03\">end<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span> <span style=\"color:#af3a03\">local<\/span> focusedWindowApp <span style=\"color:#af3a03\">=<\/span> focusedWindow:application()\n<\/span><\/span><span style=\"display:flex;\"><span> <span style=\"color:#af3a03\">local<\/span> focusedWindowAppName <span style=\"color:#af3a03\">=<\/span> focusedWindowApp:name()\n<\/span><\/span><span style=\"display:flex;\"><span> <span style=\"color:#af3a03\">local<\/span> focusedWindowPath <span style=\"color:#af3a03\">=<\/span> focusedWindowApp:path()\n<\/span><\/span><span style=\"display:flex;\"><span> <span style=\"color:#af3a03\">local<\/span> appNameOnDisk <span style=\"color:#af3a03\">=<\/span> string.gsub(focusedWindowPath, <span style=\"color:#79740e\">&#34;\/Applications\/&#34;<\/span>, <span style=\"color:#79740e\">&#34;&#34;<\/span>)\n<\/span><\/span><span style=\"display:flex;\"><span> <span style=\"color:#af3a03\">local<\/span> appNameOnDisk <span style=\"color:#af3a03\">=<\/span> string.gsub(appNameOnDisk, <span style=\"color:#79740e\">&#34;.app&#34;<\/span>, <span style=\"color:#79740e\">&#34;&#34;<\/span>)\n<\/span><\/span><span style=\"display:flex;\"><span> <span style=\"color:#af3a03\">local<\/span> appNameOnDisk <span style=\"color:#af3a03\">=<\/span> string.gsub(appNameOnDisk, <span style=\"color:#79740e\">&#34;\/System\/Library\/CoreServices\/&#34;<\/span>, <span style=\"color:#79740e\">&#34;&#34;<\/span>)\n<\/span><\/span><span style=\"display:flex;\"><span> <span style=\"color:#af3a03\">if<\/span> focusedWindow <span style=\"color:#af3a03\">and<\/span> appNameOnDisk <span style=\"color:#af3a03\">==<\/span> app <span style=\"color:#af3a03\">then<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>  <span style=\"color:#af3a03\">local<\/span> currentApp <span style=\"color:#af3a03\">=<\/span> hs.application.get(focusedWindowAppName)\n<\/span><\/span><span style=\"display:flex;\"><span>  <span style=\"color:#af3a03\">local<\/span> appWindows <span style=\"color:#af3a03\">=<\/span> currentApp:allWindows()\n<\/span><\/span><span style=\"display:flex;\"><span>  <span style=\"color:#928374;font-style:italic\">-- https:\/\/www.hammerspoon.org\/docs\/hs.application.html#allWindows<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>  <span style=\"color:#928374;font-style:italic\">-- A table of zero or more hs.window objects owned by the application. From the current space.<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>  <span style=\"color:#af3a03\">if<\/span> <span style=\"color:#af3a03\">#<\/span>appWindows <span style=\"color:#af3a03\">==<\/span> <span style=\"color:#8f3f71\">1<\/span> <span style=\"color:#af3a03\">then<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>   currentApp:hide()\n<\/span><\/span><span style=\"display:flex;\"><span>   <span style=\"color:#af3a03\">return<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>  <span style=\"color:#af3a03\">end<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>  <span style=\"color:#af3a03\">if<\/span> <span style=\"color:#af3a03\">#<\/span>appWindows <span style=\"color:#af3a03\">&gt;<\/span> <span style=\"color:#8f3f71\">0<\/span> <span style=\"color:#af3a03\">then<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>   <span style=\"color:#928374;font-style:italic\">-- It seems that this list order changes after one window get focused,<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>   <span style=\"color:#928374;font-style:italic\">-- Let&#39;s directly bring the last one to focus every time<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>   <span style=\"color:#928374;font-style:italic\">-- https:\/\/www.hammerspoon.org\/docs\/hs.window.html#focus<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>   <span style=\"color:#af3a03\">if<\/span> app <span style=\"color:#af3a03\">==<\/span> <span style=\"color:#79740e\">&#34;Finder&#34;<\/span> <span style=\"color:#af3a03\">then<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#928374;font-style:italic\">-- If the app is Finder the window count returned is one more than the actual count, so I subtract<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>    appWindows[<span style=\"color:#af3a03\">#<\/span>appWindows <span style=\"color:#af3a03\">-<\/span> <span style=\"color:#8f3f71\">1<\/span>]:focus()\n<\/span><\/span><span style=\"display:flex;\"><span>   <span style=\"color:#af3a03\">else<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>    appWindows[<span style=\"color:#af3a03\">#<\/span>appWindows]:focus()\n<\/span><\/span><span style=\"display:flex;\"><span>   <span style=\"color:#af3a03\">end<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>  <span style=\"color:#af3a03\">else<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>   hs.application.launchOrFocus(app)\n<\/span><\/span><span style=\"display:flex;\"><span>  <span style=\"color:#af3a03\">end<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span> <span style=\"color:#af3a03\">else<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>  hs.application.launchOrFocus(app)\n<\/span><\/span><span style=\"display:flex;\"><span> <span style=\"color:#af3a03\">end<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">end<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">for<\/span> _, shortcut <span style=\"color:#af3a03\">in<\/span> ipairs(WINDOWS_SHORTCUTS) <span style=\"color:#af3a03\">do<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span> hs.hotkey.bind(WINDOW_MANAGEMENT_KEY, shortcut[<span style=\"color:#8f3f71\">1<\/span>], <span style=\"color:#af3a03\">function<\/span>()\n<\/span><\/span><span style=\"display:flex;\"><span>  launchOrFocusOrRotate(shortcut[<span style=\"color:#8f3f71\">2<\/span>])\n<\/span><\/span><span style=\"display:flex;\"><span> <span style=\"color:#af3a03\">end<\/span>)\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">end<\/span><\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>The lua table can be easily expanded to open more applications.<\/p>\n<h2 class=\"heading\" id=\"shortcut-for-frequent-actions\">\n  Shortcut for frequent actions\n  <a class=\"anchor\" href=\"#shortcut-for-frequent-actions\">#<\/a>\n<\/h2>\n<p>Other than switching apps there are some useful tools that is nice to have at hand.<\/p>\n<p>Some suggestions are:<\/p>\n<ul>\n<li><a href=\"https:\/\/www.raycast.com\/extensions\/calendar\" rel=\"noopener\" target=\"_blank\">Viewing calendar and reminders<\/a><\/li>\n<li><a href=\"https:\/\/www.raycast.com\/extensions\/clipboard-history\" rel=\"noopener\" target=\"_blank\">Clipboard history<\/a><\/li>\n<li><a href=\"https:\/\/www.raycast.com\/raycast\/browser-bookmarks\" rel=\"noopener\" target=\"_blank\">Fuzzy find and open browser bookmarks<\/a><\/li>\n<li><a href=\"https:\/\/www.hammerspoon.org\/docs\/hs.grid.html#:~:text=To%20resize%2Fmove%20the%20window,upper%2Dleft%20of%20the%20window.\" rel=\"noopener\" target=\"_blank\">Splitting, resizing &amp; moving windows<\/a><\/li>\n<li><a href=\"https:\/\/www.raycast.com\/changelog\/1-19-0\" rel=\"noopener\" target=\"_blank\">Fuzzy find open windows<\/a><\/li>\n<\/ul>\n<p>Raycast is easier to use for things you need to browse and search.\nFor actions that don&rsquo;t include search and selection Hammerspoon is good.<\/p>\n<p>These mappings can look like this:<\/p>\n<div class=\"code-block\">\n  <pre tabindex=\"0\"><code>hyper + A move window to left half of the screen\nhyper + S move window to bottom half of the screen\nhyper + D move window to right half of the screen\nhyper + W move window to top half of the screen\nhyper + L clipboard history\nhyper + ; browser bookmarks\nhyper + M search open windows<\/code><\/pre>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>The following Hammerspoon config allows moving windows with shortcuts:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-lua\" data-lang=\"lua\"><span style=\"display:flex;\"><span><span style=\"color:#928374;font-style:italic\">-- left<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>hs.hotkey.bind(WINDOW_MANAGEMENT_KEY, <span style=\"color:#79740e\">&#34;a&#34;<\/span>, <span style=\"color:#af3a03\">function<\/span>()\n<\/span><\/span><span style=\"display:flex;\"><span> hs.window.focusedWindow():moveToUnit({ <span style=\"color:#8f3f71\">0<\/span>, <span style=\"color:#8f3f71\">0<\/span>, <span style=\"color:#8f3f71\">0.5<\/span>, <span style=\"color:#8f3f71\">1<\/span> })\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">end<\/span>)\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#928374;font-style:italic\">-- right<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>hs.hotkey.bind(WINDOW_MANAGEMENT_KEY, <span style=\"color:#79740e\">&#34;d&#34;<\/span>, <span style=\"color:#af3a03\">function<\/span>()\n<\/span><\/span><span style=\"display:flex;\"><span> hs.window.focusedWindow():moveToUnit({ <span style=\"color:#8f3f71\">0.5<\/span>, <span style=\"color:#8f3f71\">0<\/span>, <span style=\"color:#8f3f71\">0.5<\/span>, <span style=\"color:#8f3f71\">1<\/span> })\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">end<\/span>)\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#928374;font-style:italic\">-- up<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>hs.hotkey.bind(WINDOW_MANAGEMENT_KEY, <span style=\"color:#79740e\">&#34;w&#34;<\/span>, <span style=\"color:#af3a03\">function<\/span>()\n<\/span><\/span><span style=\"display:flex;\"><span> hs.window.focusedWindow():moveToUnit({ <span style=\"color:#8f3f71\">0<\/span>, <span style=\"color:#8f3f71\">0<\/span>, <span style=\"color:#8f3f71\">1<\/span>, <span style=\"color:#8f3f71\">0.5<\/span> })\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">end<\/span>)\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#928374;font-style:italic\">-- down<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>hs.hotkey.bind(WINDOW_MANAGEMENT_KEY, <span style=\"color:#79740e\">&#34;s&#34;<\/span>, <span style=\"color:#af3a03\">function<\/span>()\n<\/span><\/span><span style=\"display:flex;\"><span> hs.window.focusedWindow():moveToUnit({ <span style=\"color:#8f3f71\">0<\/span>, <span style=\"color:#8f3f71\">0.5<\/span>, <span style=\"color:#8f3f71\">1<\/span>, <span style=\"color:#8f3f71\">0.5<\/span> })\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">end<\/span>)\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#928374;font-style:italic\">-- center<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>hs.hotkey.bind(WINDOW_MANAGEMENT_KEY, <span style=\"color:#79740e\">&#34;c&#34;<\/span>, <span style=\"color:#af3a03\">function<\/span>()\n<\/span><\/span><span style=\"display:flex;\"><span> hs.window.focusedWindow():centerOnScreen()\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">end<\/span>)\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#928374;font-style:italic\">-- full screen<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>hs.hotkey.bind(WINDOW_MANAGEMENT_KEY, <span style=\"color:#79740e\">&#34;i&#34;<\/span>, <span style=\"color:#af3a03\">function<\/span>()\n<\/span><\/span><span style=\"display:flex;\"><span> hs.window.focusedWindow():moveToUnit({ <span style=\"color:#8f3f71\">0<\/span>, <span style=\"color:#8f3f71\">0<\/span>, <span style=\"color:#8f3f71\">1<\/span>, <span style=\"color:#8f3f71\">1<\/span> })\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">end<\/span>)<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>You can also use a layout mode for moving windows:\n<a href=\"https:\/\/github.com\/jasonrudolph\/keyboard#window-layout-mode\" rel=\"noopener\" target=\"_blank\">https:\/\/github.com\/jasonrudolph\/keyboard#window-layout-mode<\/a><\/p>\n<h2 class=\"heading\" id=\"symbol-layers\">\n  Symbol Layers\n  <a class=\"anchor\" href=\"#symbol-layers\">#<\/a>\n<\/h2>\n<p>If you code a lot this will be your favorite section.\nHave you noticed how hard it is to type <code>_<\/code>?\nYou need to take fingers off the home row, hold shift and press a key.\nBoth keys are pressed with pinky fingers.<\/p>\n<p>I find the idea <a href=\"https:\/\/gist.github.com\/gsinclair\/f4ab34da53034374eb6164698a0a8ace\" rel=\"noopener\" target=\"_blank\">here<\/a>,\nit suggests to map <code>(holding s)+k<\/code> to a symbol like <code>-<\/code>.<\/p>\n<p>The idea is very similar to how we define different toggle and press behavior to keys.\nWith Karabiner, you can modify s key to act like normal s but when pressed simultaneously with k become <code>-<\/code>.<\/p>\n<p>Concerned about accidentally typing <code>sk<\/code> or <code>-<\/code>? You can adjust the speed at which the shortcut triggers with the\n<a href=\"https:\/\/karabiner-elements.pqrs.org\/docs\/json\/complex-modifications-manipulator-definition\/to\/hold-down-milliseconds\/\" rel=\"noopener\" target=\"_blank\">hold down option<\/a>.<\/p>\n<p>So imagine the following layout:<\/p>\n<div class=\"code-block\">\n  <pre tabindex=\"0\"><code>    y u i o\na   h j k l\n    n m , .<\/code><\/pre>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>When you hold a with left hand and any of the right keys it can be mapped to a symbol.<\/p>\n<p>Here&rsquo;s my layout. I mention the key you need to hold on the left and what new keys are mapped to on the right.<\/p>\n<p><code>s<\/code> for symbols:<\/p>\n<div class=\"code-block\">\n  <pre tabindex=\"0\"><code>    y ` u # i $ o %\ns   h ~ j - k - l !\n    n   m + , + . @<\/code><\/pre>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p><code>f<\/code> for delimiters:<\/p>\n<div class=\"code-block\">\n  <pre tabindex=\"0\"><code>    y   u { i } o ^\nf   h &lt; j ( k ) l &amp;\n    n &gt; m [ , ] . *<\/code><\/pre>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>So just set these stuff for different combinations that are hard to press.\nI even have <code>a+u<\/code> for <code>tab<\/code> and <code>a+i<\/code> for <code>ctrl+tab<\/code>.<\/p>\n<p>For implementing this in Karabiner follow the guide above.\nSince Karabiner uses json files for configuration writing all of this by hand is time consuming. You can use a tool like Goku (brew install yqrashawn\/goku\/goku) instead.\nHere is my <a href=\"https:\/\/github.com\/Glyphack\/dotfiles\/blob\/20b97e675532db0bf8d73068fec1dd3050ad2fc5\/karabiner\/karabiner.json#L1\" rel=\"noopener\" target=\"_blank\">giant json<\/a> Karabiner configuration.<\/p>\n<p>For doing normal JSON you need the following Karabiner rules for :<\/p>\n<div class=\"code-block\">\n  <pre tabindex=\"0\"><code>{\n                &#34;from&#34;: {\n                  &#34;key_code&#34;: &#34;u&#34;,\n                  &#34;modifiers&#34;: {\n                    &#34;optional&#34;: [\n                      &#34;any&#34;\n                    ]\n                  }\n                },\n                &#34;to&#34;: [\n                  {\n                    &#34;key_code&#34;: &#34;open_bracket&#34;,\n                    &#34;modifiers&#34;: [\n                      &#34;left_shift&#34;\n                    ]\n                  }\n                ],\n                &#34;conditions&#34;: [\n                  {\n                    &#34;name&#34;: &#34;f-mode&#34;,\n                    &#34;value&#34;: 1,\n                    &#34;type&#34;: &#34;variable_if&#34;\n                  }\n                ],\n                &#34;type&#34;: &#34;basic&#34;\n              },\n              {\n                &#34;type&#34;: &#34;basic&#34;,\n                &#34;parameters&#34;: {\n                  &#34;basic.simultaneous_threshold_milliseconds&#34;: 250\n                },\n                &#34;to&#34;: [\n                  {\n                    &#34;set_variable&#34;: {\n                      &#34;name&#34;: &#34;f-mode&#34;,\n                      &#34;value&#34;: 1\n                    }\n                  },\n                  {\n                    &#34;key_code&#34;: &#34;open_bracket&#34;,\n                    &#34;modifiers&#34;: [\n                      &#34;left_shift&#34;\n                    ]\n                  }\n                ],\n                &#34;from&#34;: {\n                  &#34;simultaneous&#34;: [\n                    {\n                      &#34;key_code&#34;: &#34;f&#34;\n                    },\n                    {\n                      &#34;key_code&#34;: &#34;u&#34;\n                    }\n                  ],\n                  &#34;simultaneous_options&#34;: {\n                    &#34;detect_key_down_uninterruptedly&#34;: true,\n                    &#34;key_down_order&#34;: &#34;strict&#34;,\n                    &#34;key_up_order&#34;: &#34;strict_inverse&#34;,\n                    &#34;key_up_when&#34;: &#34;any&#34;,\n                    &#34;to_after_key_up&#34;: [\n                      {\n                        &#34;set_variable&#34;: {\n                          &#34;name&#34;: &#34;f-mode&#34;,\n                          &#34;value&#34;: 0\n                        }\n                      }\n                    ]\n                  }\n                }\n              }<\/code><\/pre>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<h2 class=\"heading\" id=\"not-only-speed-but-ergonomics\">\n  Not Only Speed but ergonomics\n  <a class=\"anchor\" href=\"#not-only-speed-but-ergonomics\">#<\/a>\n<\/h2>\n<p>These customization help with removing uncomfortable keys you need to press.\nThe benefits are immediate - less strain on your fingers and wrists due to reduced movement, and a mind unburdened from managing mundane tasks like locating the Terminal window.<\/p>\n"},{"title":"TIL Secret to Open Source Contribution & Contributing to Python Docs","link":"https:\/\/glyphack.com\/contributing-to-python-docs\/","pubDate":"Tue, 19 Sep 2023 22:23:48 +0200","guid":"https:\/\/glyphack.com\/contributing-to-python-docs\/","description":"<p>I think I learned something about contributing to open source that, if I knew a couple of years back, I could have done much more open source contributions.<\/p>\n<p>A while back, I started creating a <a href=\"https:\/\/github.com\/Glyphack\/enderpy\" rel=\"noopener\" target=\"_blank\">hand-written parser for Python<\/a>.\nI ended up also contributing some fixes to Python docs.\nThis was particularly interesting to me because it did not require really advanced knowledge, and the stuff there that was incorrect or outdated had been there for years.\nSo why didn&rsquo;t I do this earlier?<\/p>\n<p>It was because I never exposed myself to the opportunity.\nI usually avoided reading docs from start to finish or diving into the code of the tool I was using.<\/p>\n<p>The <a href=\"https:\/\/github.com\/python\/cpython\/pull\/104986\" rel=\"noopener\" target=\"_blank\">first<\/a> &amp; <a href=\"https:\/\/github.com\/python\/cpython\/pull\/104986\" rel=\"noopener\" target=\"_blank\">second PR<\/a> were the result of reading the grammar and <code>ast<\/code> package docs and finding inconsistencies.<\/p>\n<p>Also after I was working on my type checker I started reading PEPs and playing around with other Python type checkers such as pyright.\nThen suddenly <a href=\"https:\/\/github.com\/quora\/pyanalyze\/issues\/707\" rel=\"noopener\" target=\"_blank\">I found<\/a> a rule in PEP-586(<a href=\"https:\/\/peps.python.org\/pep-0586\/#illegal-parameters-for-literal-at-type-check-time\" rel=\"noopener\" target=\"_blank\">https:\/\/peps.python.org\/pep-0586\/#illegal-parameters-for-literal-at-type-check-time<\/a>)\nwhich was not possible to implement with Python ast structure.\nI haven&rsquo;t started an issue for this one yet because it requires more effort but it&rsquo;s another opportunity.<\/p>\n<p>I think that when we start programming journey, it&rsquo;s best to be exposed to these opportunities of reading the actual framework\/tool documentation.\nOr just, in general, look more into the source\/docs rather than reaching for tutorials.<\/p>\n"},{"title":"Compilers Resources","link":"https:\/\/glyphack.com\/compiler-resources\/","pubDate":"Fri, 15 Sep 2023 19:50:19 +0200","guid":"https:\/\/glyphack.com\/compiler-resources\/","description":"<p>This post is a compilation of great resources I found while building a type checker for Python.\nThese resources are free and highly focused on specific topics, making them ideal for learning by doing rather than going through extensive materials.<\/p>\n<h2 class=\"heading\" id=\"parser\">\n  Parser\n  <a class=\"anchor\" href=\"#parser\">#<\/a>\n<\/h2>\n<p>There are different ways to approach parsing. You can either write one by hand or use a parser generator.\nFor compilers or interpreters, you can use a parser generator.\nHowever, if you&rsquo;re working on tools like formatters or language servers, your parser needs to handle broken code gracefully. This can be either done with a tool like treesitter that can handle broken code to some extent and also by writing your own. Of course writing your own is more fun.<\/p>\n<ul>\n<li><a href=\"https:\/\/oxc-project.github.io\/javascript-parser-in-rust\/\" rel=\"noopener\" target=\"_blank\">&ldquo;Write JS Parser in Rust&rdquo;<\/a> by Boshen is an excellent introductory guide.<\/li>\n<li>For resilient parsing, check out this tutorial on <a href=\"https:\/\/matklad.github.io\/2023\/05\/21\/resilient-ll-parsing-tutorial.html\" rel=\"noopener\" target=\"_blank\">resilient LL parsing<\/a>.<\/li>\n<li>Your language&rsquo;s official documentation. For Python, there is <a href=\"https:\/\/docs.python.org\/3\/library\/ast.html\" rel=\"noopener\" target=\"_blank\">Python AST module<\/a>.<\/li>\n<li>Look into implementation of open source linters or compilers. <a href=\"https:\/\/github.com\/RustPython\/Parser\/blob\/main\/parser\/src\/lexer.rs\" rel=\"noopener\" target=\"_blank\">RustPython Lexer<\/a> is a good one for python.<\/li>\n<\/ul>\n<h2 class=\"heading\" id=\"compilers--interpreters\">\n  Compilers &amp; Interpreters\n  <a class=\"anchor\" href=\"#compilers--interpreters\">#<\/a>\n<\/h2>\n<p><a href=\"https:\/\/craftinginterpreters.com\/\" rel=\"noopener\" target=\"_blank\">&ldquo;Crafting Interpreters&rdquo;<\/a> is an essential resource for compilers. I recommend reading it chapter by chapter as you build your project.<\/p>\n<p>For a comprehensive understanding of relevant topics, consider following the <a href=\"http:\/\/openclassroom.stanford.edu\/MainFolder\/CoursePage.php?course=Compilers\" rel=\"noopener\" target=\"_blank\">Stanford Compilers Class<\/a>. Although I haven&rsquo;t watched it personally, I found this <a href=\"https:\/\/pgrandinetti.github.io\/compilers\/\" rel=\"noopener\" target=\"_blank\">guide<\/a> based on the class quite helpful.<\/p>\n<p>You can find examples of implemented programming languages and use them as a reference\n<a href=\"https:\/\/plzoo.andrej.com\/language\/poly.html\" rel=\"noopener\" target=\"_blank\">Programming Languages Zoo<\/a> is one resource for this.<\/p>\n<h3 class=\"heading\" id=\"symbol-table\">\n  Symbol Table\n  <a class=\"anchor\" href=\"#symbol-table\">#<\/a>\n<\/h3>\n<p>For symbol table you need to check the language implementation and know the scoping rules, private\/public, and different kinds of symbols. There&rsquo;s no all in one solution.<\/p>\n<p>This <a href=\"https:\/\/eli.thegreenplace.net\/2010\/09\/18\/python-internals-symbol-tables-part-1\/\" rel=\"noopener\" target=\"_blank\">series on the Python symbol table implementation<\/a>\nfrom Eli Bendersky is useful for learning how does a symbol table works.<\/p>\n<p>RustPython&rsquo;s <a href=\"https:\/\/rustpython.github.io\/website\/rustpython_compiler\/symboltable\/struct.SymbolTable.html\" rel=\"noopener\" target=\"_blank\">SymbolTable<\/a> implementation.<\/p>\n<h2 class=\"heading\" id=\"semantic-analyzer\">\n  Semantic Analyzer\n  <a class=\"anchor\" href=\"#semantic-analyzer\">#<\/a>\n<\/h2>\n<p>While resources specific to the semantic analysis phase are scarce, you can find inspiration and solutions in existing projects:<\/p>\n<ul>\n<li>MyPy <a href=\"https:\/\/github.com\/python\/mypy\/wiki\/Semantic-Analyzer\" rel=\"noopener\" target=\"_blank\">wiki<\/a>.<\/li>\n<li>Pyright&rsquo;s <a href=\"https:\/\/github.com\/microsoft\/pyright\/blob\/eb98cdda4ecfb4d2ce2fb1d4b9ce7848ab439c32\/packages\/pyright-internal\/src\/analyzer\/binder.ts\" rel=\"noopener\" target=\"_blank\">binder.ts<\/a> is an example of you would do it.<\/li>\n<\/ul>\n<h2 class=\"heading\" id=\"type-checking\">\n  Type Checking\n  <a class=\"anchor\" href=\"#type-checking\">#<\/a>\n<\/h2>\n<p>For type checking you are mostly interested in the type rules in that language.\nTherefore it&rsquo;s good to check other type checker implementations.\nThey will teach you the rules and how to do it.<\/p>\n<ul>\n<li>design of <a href=\"https:\/\/github.com\/quora\/pyanalyze\/blob\/master\/docs\/design.md\" rel=\"noopener\" target=\"_blank\">pyanalyze<\/a> for Python.<\/li>\n<li>For MyPy, the <a href=\"https:\/\/github.com\/python\/mypy\/wiki\/Type-Checker\" rel=\"noopener\" target=\"_blank\">Type Checker<\/a> wiki.<\/li>\n<li>internal details of <a href=\"https:\/\/github.com\/quora\/pyanalyze\/blob\/master\/docs\/design.md\" rel=\"noopener\" target=\"_blank\">Jedi language server<\/a>.<\/li>\n<li>Pyright <a href=\"https:\/\/github.com\/microsoft\/pyright\/blob\/main\/docs\/internals.md\" rel=\"noopener\" target=\"_blank\">internals<\/a><\/li>\n<\/ul>\n<h2 class=\"heading\" id=\"lsp-language-server-protocol\">\n  LSP (Language Server Protocol)\n  <a class=\"anchor\" href=\"#lsp-language-server-protocol\">#<\/a>\n<\/h2>\n<p>For a comprehensive understanding of language servers, file systems, updates, and testing, check out this <a href=\"https:\/\/www.youtube.com\/playlist?list=PLhb66M_x9UmrqXhQuIpWC5VgTdrGxMx3y\" rel=\"noopener\" target=\"_blank\">Explaining Rust AnalyzerYouTube playlist<\/a> from <a href=\"https:\/\/matklad.github.io\/\" rel=\"noopener\" target=\"_blank\">Matkald<\/a>.<\/p>\n<p><a href=\"https:\/\/microsoft.github.io\/language-server-protocol\/specifications\/lsp\/3.17\/specification\/\" rel=\"noopener\" target=\"_blank\">LSP specifications<\/a> are very easy to read. It&rsquo;s long but you don&rsquo;t need everything in the beginning.\nTo skip the part of defning every structure yourself you can use <a href=\"https:\/\/github.com\/ebkalderon\/tower-lsp\" rel=\"noopener\" target=\"_blank\">Tower LSP<\/a>.<\/p>\n<h2 class=\"heading\" id=\"linters\">\n  Linters\n  <a class=\"anchor\" href=\"#linters\">#<\/a>\n<\/h2>\n<p>Same as with type checking, for linters it&rsquo;s best to look into implementations and learn from them.\nSpecially linters have a lot in common with compilers and interpreters because they just emit a human readable error instead of machine code.<\/p>\n<p>The following tools are useful to understand how analysis is done and errors are reported:<\/p>\n<ul>\n<li><a href=\"https:\/\/github.com\/web-infra-dev\/oxc\" rel=\"noopener\" target=\"_blank\">oxc<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/astral-sh\/ruff\" rel=\"noopener\" target=\"_blank\">Ruff<\/a><\/li>\n<\/ul>\n<h2 class=\"heading\" id=\"final-words\">\n  Final Words\n  <a class=\"anchor\" href=\"#final-words\">#<\/a>\n<\/h2>\n<p>Compilers are super fun. If you have more resources please send them to me.<\/p>\n"},{"title":"Trying Out Learning In Public","link":"https:\/\/glyphack.com\/my-learn-in-public-workflow\/","pubDate":"Thu, 18 May 2023 14:07:58 +0200","guid":"https:\/\/glyphack.com\/my-learn-in-public-workflow\/","description":"<p>A while back I read about <a href=\"https:\/\/www.swyx.io\/learn-in-public\" rel=\"noopener\" target=\"_blank\">learning in public<\/a>\nfrom swyx:<\/p>\n<blockquote>\n<p>At some point people will start asking you for help because of all the stuff you put out.<\/p>\n<p>80% of developers are \u201cdark\u201d, they dont write or speak or participate in public tech discourse.<\/p>\n<\/blockquote>\n<p>The idea is promising people who learns in public get more reputaiton.\nThis extra reputation makes it easier to find friends, <a href=\"https:\/\/simonwillison.net\/2021\/Jul\/17\/standing-out\/\" rel=\"noopener\" target=\"_blank\">get jobs<\/a>\nand so on.<\/p>\n<p>Starting to learn in public is a hard path,\nthe people that are learning in public like swyx and <a href=\"https:\/\/nicolevanderhoeven.com\/\" rel=\"noopener\" target=\"_blank\">nicole van derhoeven<\/a>\neach have their own way, and both are successful with it.\nSo there&rsquo;s no one way to do this.<\/p>\n<p>I&rsquo;m not measuring the success only in terms of follower\/subscriber but also the\ntheir consistency in publishing.\nOne interesting aspect of people who are consistent in their work is that,\nafter you read one of their post there are a lot more to read from them when\nyou enjoy their work.\nMaybe that&rsquo;s why you rarely find one fascinating writing and when you\nsearch for the writer they haven&rsquo;t wrote anything else.\nThe consistency of their work make them better.<\/p>\n<p>Swyx has some <a href=\"https:\/\/www.swyx.io\/learning-gears\" rel=\"noopener\" target=\"_blank\">nice hacks<\/a> to start doing this.\nSimon Wilson has another suggestion which is write down your <a href=\"https:\/\/simonwillison.net\/2021\/May\/2\/one-year-of-tils\/\" rel=\"noopener\" target=\"_blank\">daily TILs<\/a>.<\/p>\n<p>My goal is to write down a plan for how to be a public learner.\nThis plan is for myself so you might want to adjust some parts before adopting it.<\/p>\n<p>First step is to be aware of what you learn every day.\nI use Readwise with anything I read or watch almost all the time.<\/p>\n<p>When reading something and I found some interesting idea, I add a TIL tag to it.\nEveryday I have a reminder to 10 minutes to review the TIL notes I created that day.<\/p>\n<p>Now with this newly learned piece there are two possible options:<\/p>\n<ol>\n<li>It&rsquo;s a small point so I can just share it as a post or a tweet<\/li>\n<li>It&rsquo;s part of a bigger idea, and I want to create a creative exhaust from it<\/li>\n<\/ol>\n<p>For items belonging to the second category I spend more time to write my own thinking\nfrom the idea.\nThis is like a blogpost but I write in in Obsidian, my note taking app.\nIt helps to gather ideas related to a specific topic inside a page.<\/p>\n<p>I like to keep a list of content I can create in the future.\nwhenever I write a page for something I learned that means I have\na topic to do reasearch on so I put it in the content list.<\/p>\n<p>I like to set a schedule to remind myself about picking up\nthe stuff I left. So just by setting a 2 hour time block every week I can\nresearch more on one of the topics on my list and write down all the stuff I know.\nOr not even writing down but to connect different notes I have on the topic.\nIt might become a post worthy content or not, anyway It&rsquo;s there and can be shared.<\/p>\n<p>So it&rsquo;s really simlple but you need to keep the list and ideas somewhere,\notherwise you never know what to talk about when it comes to sharing.\nAfter a while when the knowledge piles up in a topic.\nYou can spend small time connecting all your learnings together,\nand create learning exhaust.<\/p>\n<p>There&rsquo;s another aspect of learning in public, which is to share what you&rsquo;re doing.<\/p>\n<p><a href=\"https:\/\/twitter.com\/willmcgugan\" rel=\"noopener\" target=\"_blank\">WillMcGugan<\/a> shares their progress on building\nTextual on twitter.\nThis is not particularly same as other public learners mentioned above,\nbut it brings the same benefits.\nBy doing this people know him more and he can meet new poeple.<\/p>\n<p>I met <a href=\"https:\/\/twitter.com\/isidentical\" rel=\"noopener\" target=\"_blank\">Batuhan Taskaya<\/a> just by posting my\njourney on building a python parser on twitter.\nHe helped me with implementing tricky parts of parser and\ncontributing to Python.<\/p>\n<p>The takeaway is to share updates when you are building in public.\nNot only when you finish building something but also along the way.<\/p>\n<p>So this is all about starting small and being consistent,\nand having fun along the way.<\/p>\n"},{"title":"Building a Web Crawler in Golang","link":"https:\/\/glyphack.com\/build-a-crawler-in-golang\/","pubDate":"Mon, 20 Mar 2023 18:17:24 +0100","guid":"https:\/\/glyphack.com\/build-a-crawler-in-golang\/","description":"<!-- vim-markdown-toc GFM -->\n<ul>\n<li><a href=\"https:\/\/glyphack.com\/build-a-crawler-in-golang\/#introduction\">Introduction<\/a><\/li>\n<li><a href=\"https:\/\/glyphack.com\/build-a-crawler-in-golang\/#but-why-building-another-crawler\">But Why Building Another Crawler?<\/a><\/li>\n<li><a href=\"https:\/\/glyphack.com\/build-a-crawler-in-golang\/#high-level-design\">High Level Design<\/a>\n<ul>\n<li><a href=\"https:\/\/glyphack.com\/build-a-crawler-in-golang\/#url-frontier\">URL Frontier<\/a><\/li>\n<li><a href=\"https:\/\/glyphack.com\/build-a-crawler-in-golang\/#selector\">Selector<\/a><\/li>\n<li><a href=\"https:\/\/glyphack.com\/build-a-crawler-in-golang\/#workers\">Workers<\/a><\/li>\n<li><a href=\"https:\/\/glyphack.com\/build-a-crawler-in-golang\/#fetcher\">Fetcher<\/a><\/li>\n<li><a href=\"https:\/\/glyphack.com\/build-a-crawler-in-golang\/#content-processor\">Content Processor<\/a><\/li>\n<li><a href=\"https:\/\/glyphack.com\/build-a-crawler-in-golang\/#link-extractor\">Link Extractor<\/a><\/li>\n<\/ul>\n<\/li>\n<li><a href=\"https:\/\/glyphack.com\/build-a-crawler-in-golang\/#implementation\">Implementation<\/a>\n<ul>\n<li><a href=\"https:\/\/glyphack.com\/build-a-crawler-in-golang\/#lets-talk-about-channels\">Let&rsquo;s talk about channels<\/a><\/li>\n<li><a href=\"https:\/\/glyphack.com\/build-a-crawler-in-golang\/#storage\">Storage<\/a><\/li>\n<li><a href=\"https:\/\/glyphack.com\/build-a-crawler-in-golang\/#parser\">Parser<\/a><\/li>\n<li><a href=\"https:\/\/glyphack.com\/build-a-crawler-in-golang\/#processor\">Processor<\/a><\/li>\n<li><a href=\"https:\/\/glyphack.com\/build-a-crawler-in-golang\/#distribute-and-collect-result-from-workers\">Distribute and Collect Result from Workers<\/a><\/li>\n<li><a href=\"https:\/\/glyphack.com\/build-a-crawler-in-golang\/#worker\">Worker<\/a><\/li>\n<li><a href=\"https:\/\/glyphack.com\/build-a-crawler-in-golang\/#extracting-links\">Extracting Links<\/a><\/li>\n<li><a href=\"https:\/\/glyphack.com\/build-a-crawler-in-golang\/#saving-content\">Saving Content<\/a><\/li>\n<li><a href=\"https:\/\/glyphack.com\/build-a-crawler-in-golang\/#running-processors\">Running Processors<\/a><\/li>\n<li><a href=\"https:\/\/glyphack.com\/build-a-crawler-in-golang\/#failed-urls\">Failed URLs<\/a><\/li>\n<li><a href=\"https:\/\/glyphack.com\/build-a-crawler-in-golang\/#html-parser\">HTML parser<\/a><\/li>\n<li><a href=\"https:\/\/glyphack.com\/build-a-crawler-in-golang\/#putting-it-all-together\">Putting it All Together<\/a><\/li>\n<\/ul>\n<\/li>\n<li><a href=\"https:\/\/glyphack.com\/build-a-crawler-in-golang\/#conclusion\">Conclusion<\/a><\/li>\n<\/ul>\n<!-- vim-markdown-toc -->\n<h2 class=\"heading\" id=\"introduction\">\n  Introduction\n  <a class=\"anchor\" href=\"#introduction\">#<\/a>\n<\/h2>\n<p>Web crawler is a program that explores the Internet,\nby going to different websites and following any link it finds.<\/p>\n<p>Crawlers are interesting because they provide a way to gather data\nfrom the internet.\nThis is especially useful for data mining purposes.<\/p>\n<p>You can find the full implementation in the <a href=\"https:\/\/github.com\/Glyphack\/crawler\" rel=\"noopener\" target=\"_blank\">GitHub repository<\/a>.<\/p>\n<p><a href=\"https:\/\/cacm.acm.org\/blogs\/blog-cacm\/153780-data-mining-the-web-via-crawling\/fulltext\" rel=\"noopener\" target=\"_blank\">This post<\/a>\nprovides a good introduction to building a crawler.<\/p>\n<h2 class=\"heading\" id=\"but-why-building-another-crawler\">\n  But Why Building Another Crawler?\n  <a class=\"anchor\" href=\"#but-why-building-another-crawler\">#<\/a>\n<\/h2>\n<p>I wrote down my reasons in the <a href=\"https:\/\/glyphack.com\/rate-limiter-python-2\/\">rate limiter post<\/a>\non why I&rsquo;m building this stuff from scratch.\nThe short answer is that it seems simple until you try it.<\/p>\n<p>After reading through this project and implementing yourself,\nyou will have a good understanding of how to write concurrent\napplications in Go.<\/p>\n<h2 class=\"heading\" id=\"high-level-design\">\n  High Level Design\n  <a class=\"anchor\" href=\"#high-level-design\">#<\/a>\n<\/h2>\n<p>Let&rsquo;s look into what components a crawler is made of, this helps\nto structure our code.<\/p>\n<p>The following diagram shows the execution flow of our program and\nresponsibilities of components:<\/p>\n<p>\n\n\n\n\n\n\n\n<figure class=\"\">\n\n    \n    \n      \n        <div class=\"img-container\">\n            <img loading=\"lazy\" alt=\"crawler-diagram\" src=\"https:\/\/glyphack.com\/build-a-crawler-in-golang\/crawler-diagram.excalidraw.svg\">\n        <\/div>\n      \n    \n\n    \n<\/figure>\n<\/p>\n<p>Let&rsquo;s break it down:<\/p>\n<h3 class=\"heading\" id=\"url-frontier\">\n  URL Frontier\n  <a class=\"anchor\" href=\"#url-frontier\">#<\/a>\n<\/h3>\n<p>URL Frontier is a collection of URLs that are going to be crawled.\nIt supports adding &amp; consuming new URLs as we discover links in fetched pages.<\/p>\n<h3 class=\"heading\" id=\"selector\">\n  Selector\n  <a class=\"anchor\" href=\"#selector\">#<\/a>\n<\/h3>\n<p>To consume the URLs from frontier we can get them one by one.\nBut this can cause problem if we want to distribute the URLs between multiple workers.<\/p>\n<p>The technique used here is called fan-out.<\/p>\n<p>For example if some URLs are more important to crawl first, and each worker gets\nthe next URL to crawl then those special URLs can&rsquo;t be crawled first.\nAnother usefulness of this component is distributing URLs from one host to one worker.\nSo each worker can make sure to not send too many requests to a single Host.\nThe best practice is to wait 2 seconds between requests for the same Host.<\/p>\n<h3 class=\"heading\" id=\"workers\">\n  Workers\n  <a class=\"anchor\" href=\"#workers\">#<\/a>\n<\/h3>\n<p>Each worker consumes from queues that selector fills and fetches the URL.<\/p>\n<p>The worker must handle failures and retry when it fails to fetch a URL.\nEach worker also keeps track of URLs fetched to be polite.<\/p>\n<h3 class=\"heading\" id=\"fetcher\">\n  Fetcher\n  <a class=\"anchor\" href=\"#fetcher\">#<\/a>\n<\/h3>\n<p>This components is the reverse of selector component, it gathers\nresults from different workers to a single collection.<\/p>\n<p>This operation is called fan-in which is useful here because we\ncan simplify the processor operations because it only needs to\nconsume from a single result channel.<\/p>\n<h3 class=\"heading\" id=\"content-processor\">\n  Content Processor\n  <a class=\"anchor\" href=\"#content-processor\">#<\/a>\n<\/h3>\n<p>After we get the result from each worker we ran different\ncontent processors on the result, this can be tasks like extracting\nnew URLs or saving pages to the disk.<\/p>\n<p>Also note that this component does not apply a single\nlogic on all results. We can register different processors,\nlike a plugin system to expand this component.<\/p>\n<p>Later we discuss how we can use strategy design pattern to\nimplement this in code.<\/p>\n<h3 class=\"heading\" id=\"link-extractor\">\n  Link Extractor\n  <a class=\"anchor\" href=\"#link-extractor\">#<\/a>\n<\/h3>\n<p>The link extractor is a special processor we create\nthat uses a parser to parse the page content and insert URLs\nback to frontier.<\/p>\n<h2 class=\"heading\" id=\"implementation\">\n  Implementation\n  <a class=\"anchor\" href=\"#implementation\">#<\/a>\n<\/h2>\n<h3 class=\"heading\" id=\"lets-talk-about-channels\">\n  Let&rsquo;s talk about channels\n  <a class=\"anchor\" href=\"#lets-talk-about-channels\">#<\/a>\n<\/h3>\n<p>channels are going to be used heavily in the implementation.\nI suggest you to make sure you understand <a href=\"https:\/\/go.dev\/tour\/concurrency\/1\" rel=\"noopener\" target=\"_blank\">fundamentals<\/a>\nof channels.\nbefore reading the rest of this post.<\/p>\n<p>We can start with frontier since it&rsquo;s not dependent on any other component.<\/p>\n<p>I&rsquo;ll create a new package frontier:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-go\" data-lang=\"go\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">type<\/span> Frontier <span style=\"color:#af3a03\">struct<\/span> {\n<\/span><\/span><span style=\"display:flex;\"><span>    urls        <span style=\"color:#af3a03\">chan<\/span> <span style=\"color:#af3a03\">*<\/span>url.URL\n<\/span><\/span><span style=\"display:flex;\"><span>    history     <span style=\"color:#af3a03\">map<\/span>[url.URL]time.Time\n<\/span><\/span><span style=\"display:flex;\"><span>    exclude     []<span style=\"color:#b57614\">string<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>}\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">func<\/span> <span style=\"color:#b57614\">NewFrontier<\/span>(initialUrls []url.URL, exclude []<span style=\"color:#b57614\">string<\/span>) <span style=\"color:#af3a03\">*<\/span>Frontier {\n<\/span><\/span><span style=\"display:flex;\"><span>    history <span style=\"color:#af3a03\">:=<\/span> <span style=\"color:#b57614\">make<\/span>(<span style=\"color:#af3a03\">map<\/span>[url.URL]time.Time)\n<\/span><\/span><span style=\"display:flex;\"><span>    f <span style=\"color:#af3a03\">:=<\/span> <span style=\"color:#af3a03\">&amp;<\/span>Frontier{\n<\/span><\/span><span style=\"display:flex;\"><span>        urls:    <span style=\"color:#b57614\">make<\/span>(<span style=\"color:#af3a03\">chan<\/span> <span style=\"color:#af3a03\">*<\/span>url.URL, <span style=\"color:#b57614\">len<\/span>(initialUrls)),\n<\/span><\/span><span style=\"display:flex;\"><span>        history: history,\n<\/span><\/span><span style=\"display:flex;\"><span>        exclude: exclude,\n<\/span><\/span><span style=\"display:flex;\"><span>    }\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">for<\/span> _, u <span style=\"color:#af3a03\">:=<\/span> <span style=\"color:#af3a03\">range<\/span> initialUrls {\n<\/span><\/span><span style=\"display:flex;\"><span>        f.<span style=\"color:#b57614\">Add<\/span>(<span style=\"color:#af3a03\">&amp;<\/span>u)\n<\/span><\/span><span style=\"display:flex;\"><span>    }\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">return<\/span> f\n<\/span><\/span><span style=\"display:flex;\"><span>}<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>The frontier uses a channel of urls to keep the added URLs.\nSince the channel is consumed then we keep a <code>history<\/code> of visited URLs.\nHistory can be later used to check whether we seen a URL or not.<\/p>\n<p>The <code>terminating<\/code> attribute is used so we can have a graceful termination.\nSince another goroutine is going to read from this channel, we might\nwant to wait until all the URLs are consumed and meanwhile don&rsquo;t add any new URLs.<\/p>\n<p>Next we need a method to add a new url.<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-go\" data-lang=\"go\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">func<\/span> (f <span style=\"color:#af3a03\">*<\/span>Frontier) <span style=\"color:#b57614\">Add<\/span>(url <span style=\"color:#af3a03\">*<\/span>url.URL) <span style=\"color:#b57614\">bool<\/span> {\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">if<\/span> f.terminating {\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">return<\/span> <span style=\"color:#af3a03\">false<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>    }\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">if<\/span> f.<span style=\"color:#b57614\">Seen<\/span>(url) {\n<\/span><\/span><span style=\"display:flex;\"><span>        log.<span style=\"color:#b57614\">WithFields<\/span>(log.Fields{\n<\/span><\/span><span style=\"display:flex;\"><span>            <span style=\"color:#79740e\">&#34;url&#34;<\/span>: url,\n<\/span><\/span><span style=\"display:flex;\"><span>        }).<span style=\"color:#b57614\">Info<\/span>(<span style=\"color:#79740e\">&#34;Already seen&#34;<\/span>)\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">return<\/span> <span style=\"color:#af3a03\">false<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>    }\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">for<\/span> _, pattern <span style=\"color:#af3a03\">:=<\/span> <span style=\"color:#af3a03\">range<\/span> f.exclude {\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">if<\/span> pattern <span style=\"color:#af3a03\">==<\/span> url.Host {\n<\/span><\/span><span style=\"display:flex;\"><span>            log.<span style=\"color:#b57614\">WithFields<\/span>(log.Fields{\n<\/span><\/span><span style=\"display:flex;\"><span>                <span style=\"color:#79740e\">&#34;url&#34;<\/span>: url,\n<\/span><\/span><span style=\"display:flex;\"><span>            }).<span style=\"color:#b57614\">Info<\/span>(<span style=\"color:#79740e\">&#34;Excluded&#34;<\/span>)\n<\/span><\/span><span style=\"display:flex;\"><span>            <span style=\"color:#af3a03\">return<\/span> <span style=\"color:#af3a03\">false<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>        }\n<\/span><\/span><span style=\"display:flex;\"><span>    }\n<\/span><\/span><span style=\"display:flex;\"><span>    f.history[<span style=\"color:#af3a03\">*<\/span>url] = time.<span style=\"color:#b57614\">Now<\/span>()\n<\/span><\/span><span style=\"display:flex;\"><span>    f.urls <span style=\"color:#af3a03\">&lt;-<\/span> url\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">return<\/span> <span style=\"color:#af3a03\">true<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>}\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">func<\/span> (f <span style=\"color:#af3a03\">*<\/span>Frontier) <span style=\"color:#b57614\">Seen<\/span>(url <span style=\"color:#af3a03\">*<\/span>url.URL) <span style=\"color:#b57614\">bool<\/span> {\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">if<\/span> lastFetch, ok <span style=\"color:#af3a03\">:=<\/span> f.history[<span style=\"color:#af3a03\">*<\/span>url]; ok {\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">return<\/span> time.<span style=\"color:#b57614\">Since<\/span>(lastFetch) &lt; <span style=\"color:#8f3f71\">2<\/span><span style=\"color:#af3a03\">*<\/span>time.Hour\n<\/span><\/span><span style=\"display:flex;\"><span>    }\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">return<\/span> <span style=\"color:#af3a03\">false<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>}\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">func<\/span> (f <span style=\"color:#af3a03\">*<\/span>Frontier) <span style=\"color:#b57614\">Get<\/span>() <span style=\"color:#af3a03\">chan<\/span> <span style=\"color:#af3a03\">*<\/span>url.URL {\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">return<\/span> f.urls\n<\/span><\/span><span style=\"display:flex;\"><span>}<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>This method simply checks if the url is seen or not and if it&rsquo;s not\nexcluded adds it to the channel.<\/p>\n<p>This function is blocking unless another goroutine is consuming from the urls channel.\nWhy is this important?\nBecause if we run the Add in a blocking way without consuming the urls\nwe will block the goroutine &amp; it&rsquo;s a deadlock.<\/p>\n<p>The <code>Get<\/code> function does not provide any abstraction here, but I like the idea that\nconsumers don&rsquo;t have to know which channel they need to consume from.<\/p>\n<p>In case you are wondering what log package I&rsquo;m using, it&rsquo;s <a href=\"https:\/\/github.com\/sirupsen\/logrus\" rel=\"noopener\" target=\"_blank\">logrus<\/a>.<\/p>\n<p>The next step is to create the component and orchestrates the crawl process.<\/p>\n<p>Let&rsquo;s first define the configuration that user can pass to the crawler.<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-go\" data-lang=\"go\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">type<\/span> Config <span style=\"color:#af3a03\">struct<\/span> {\n<\/span><\/span><span style=\"display:flex;\"><span>    MaxRedirects    <span style=\"color:#b57614\">int<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>    RevisitDelay    time.Duration\n<\/span><\/span><span style=\"display:flex;\"><span>    WorkerCount     <span style=\"color:#b57614\">int<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>    ExcludePatterns []<span style=\"color:#b57614\">string<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>}<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-go\" data-lang=\"go\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">package<\/span> crawler\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">import<\/span> (\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#79740e\">&#34;net\/url&#34;<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    log <span style=\"color:#79740e\">&#34;github.com\/sirupsen\/logrus&#34;<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#79740e\">&#34;github.com\/glyphack\/crawler\/internal\/frontier&#34;<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#79740e\">&#34;github.com\/glyphack\/crawler\/internal\/parser&#34;<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#79740e\">&#34;github.com\/glyphack\/crawler\/internal\/storage&#34;<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>)\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">type<\/span> Crawler <span style=\"color:#af3a03\">struct<\/span> {\n<\/span><\/span><span style=\"display:flex;\"><span>    config         <span style=\"color:#af3a03\">*<\/span>Config\n<\/span><\/span><span style=\"display:flex;\"><span>    frontier       <span style=\"color:#af3a03\">*<\/span>frontier.Frontier\n<\/span><\/span><span style=\"display:flex;\"><span>    storage        storage.Storage\n<\/span><\/span><span style=\"display:flex;\"><span>    contentParsers []parser.Parser\n<\/span><\/span><span style=\"display:flex;\"><span>    deadLetter     <span style=\"color:#af3a03\">chan<\/span> <span style=\"color:#af3a03\">*<\/span>url.URL\n<\/span><\/span><span style=\"display:flex;\"><span>    processors     []Processor\n<\/span><\/span><span style=\"display:flex;\"><span>}\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">func<\/span> <span style=\"color:#b57614\">NewCrawler<\/span>(initialUrls []url.URL, contentStorage storage.Storage, config <span style=\"color:#af3a03\">*<\/span>Config) <span style=\"color:#af3a03\">*<\/span>Crawler {\n<\/span><\/span><span style=\"display:flex;\"><span>    deadLetter <span style=\"color:#af3a03\">:=<\/span> <span style=\"color:#b57614\">make<\/span>(<span style=\"color:#af3a03\">chan<\/span> <span style=\"color:#af3a03\">*<\/span>url.URL)\n<\/span><\/span><span style=\"display:flex;\"><span>    contentParser <span style=\"color:#af3a03\">:=<\/span> []parser.Parser{<span style=\"color:#af3a03\">&amp;<\/span>parser.HtmlParser{}}\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">return<\/span> <span style=\"color:#af3a03\">&amp;<\/span>Crawler{\n<\/span><\/span><span style=\"display:flex;\"><span>        frontier:       frontier.<span style=\"color:#b57614\">NewFrontier<\/span>(initialUrls, config.ExcludePatterns),\n<\/span><\/span><span style=\"display:flex;\"><span>        storage:        contentStorage,\n<\/span><\/span><span style=\"display:flex;\"><span>        contentParsers: contentParser,\n<\/span><\/span><span style=\"display:flex;\"><span>        deadLetter:     deadLetter,\n<\/span><\/span><span style=\"display:flex;\"><span>        config:         config,\n<\/span><\/span><span style=\"display:flex;\"><span>    }\n<\/span><\/span><span style=\"display:flex;\"><span>}\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">func<\/span> (c <span style=\"color:#af3a03\">*<\/span>Crawler) <span style=\"color:#b57614\">AddContentParser<\/span>(contentParser parser.Parser) {\n<\/span><\/span><span style=\"display:flex;\"><span>    a.contentParsers = <span style=\"color:#b57614\">append<\/span>(c.contentParsers, contentParser)\n<\/span><\/span><span style=\"display:flex;\"><span>}\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">func<\/span> (c <span style=\"color:#af3a03\">*<\/span>Crawler) <span style=\"color:#b57614\">AddExcludePattern<\/span>(pattern <span style=\"color:#b57614\">string<\/span>) {\n<\/span><\/span><span style=\"display:flex;\"><span>    c.config.ExcludePatterns = <span style=\"color:#b57614\">append<\/span>(c.config.ExcludePatterns, pattern)\n<\/span><\/span><span style=\"display:flex;\"><span>}\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">func<\/span> (c <span style=\"color:#af3a03\">*<\/span>Crawler) <span style=\"color:#b57614\">AddProcessor<\/span>(processor Processor) {\n<\/span><\/span><span style=\"display:flex;\"><span>    c.processors = <span style=\"color:#b57614\">append<\/span>(c.processors, processor)\n<\/span><\/span><span style=\"display:flex;\"><span>}<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>Config comes from the user and by making it a separate struct we can easily modify\nit without changing the parameters we pass to create the crawler.\nWe keep a deadLetter channel for the failed URLs to have a retry mechanism.<\/p>\n<p>The crawler takes in other components let&rsquo;s break them down:<\/p>\n<h3 class=\"heading\" id=\"storage\">\n  Storage\n  <a class=\"anchor\" href=\"#storage\">#<\/a>\n<\/h3>\n<p>Storage is an interface that exposes method to save data.\nThis helps with extending the processor without changing it&rsquo;s code.<\/p>\n<p>Whatever storage implementation we use we need to implement\nthe following methods:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-go\" data-lang=\"go\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">package<\/span> storage\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">type<\/span> Storage <span style=\"color:#af3a03\">interface<\/span> {\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#b57614\">Get<\/span>(path <span style=\"color:#b57614\">string<\/span>) (<span style=\"color:#b57614\">string<\/span>, <span style=\"color:#b57614\">error<\/span>)\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#b57614\">Set<\/span>(path <span style=\"color:#b57614\">string<\/span>, value <span style=\"color:#b57614\">string<\/span>) <span style=\"color:#b57614\">error<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#b57614\">Delete<\/span>(path <span style=\"color:#b57614\">string<\/span>) <span style=\"color:#b57614\">error<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>}<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<h3 class=\"heading\" id=\"parser\">\n  Parser\n  <a class=\"anchor\" href=\"#parser\">#<\/a>\n<\/h3>\n<p>Instead of parsing the content in the crawler we can provide an implementation\nfor the file types we want to parse.\nWe can have a single parser that handles all the file types but\nthis way is much easier to extend.<\/p>\n<p>But why do we need the parser?\nAfter we fetch the page we need to parse it to get\nthe links from it and add it to our frontier.<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-go\" data-lang=\"go\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">package<\/span> parser\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">type<\/span> Token <span style=\"color:#af3a03\">struct<\/span> {\n<\/span><\/span><span style=\"display:flex;\"><span>    Name  <span style=\"color:#b57614\">string<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>    Value <span style=\"color:#b57614\">string<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>}\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">type<\/span> Parser <span style=\"color:#af3a03\">interface<\/span> {\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#b57614\">IsSupportedExtension<\/span>(extension <span style=\"color:#b57614\">string<\/span>) <span style=\"color:#b57614\">bool<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#b57614\">Parse<\/span>(content <span style=\"color:#b57614\">string<\/span>) ([]Token, <span style=\"color:#b57614\">error<\/span>)\n<\/span><\/span><span style=\"display:flex;\"><span>}<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>Parser can check the file extension to see if it&rsquo;s supported,\nAnd parse the file into tokens.<\/p>\n<p>The token is parsed information from the content.\nThis is a nice way to extend the material we parse from the page later.\nCurrently we only care about <code>a<\/code> tags which are links.<\/p>\n<h3 class=\"heading\" id=\"processor\">\n  Processor\n  <a class=\"anchor\" href=\"#processor\">#<\/a>\n<\/h3>\n<p>Following the same idea with parsers, we can provide the crawler\nprocesses to apply on the web pages.<\/p>\n<p>Some typical processes are:<\/p>\n<ul>\n<li>Saving the page<\/li>\n<li>Extracting links from the page<\/li>\n<\/ul>\n<p>Let&rsquo;s define the interface based on the required actions.<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-go\" data-lang=\"go\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">type<\/span> Processor <span style=\"color:#af3a03\">interface<\/span> {\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#b57614\">Process<\/span>(CrawlResult) <span style=\"color:#b57614\">error<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>}<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>The process function takes in the crawl result which we&rsquo;ll define later.\nThe function is only going to return an error.\nSince a lot of operations can be done in this function we are not returning any value.<\/p>\n<h3 class=\"heading\" id=\"distribute-and-collect-result-from-workers\">\n  Distribute and Collect Result from Workers\n  <a class=\"anchor\" href=\"#distribute-and-collect-result-from-workers\">#<\/a>\n<\/h3>\n<p>In the earlier section we discussed how can we parallelize the crawling\ntask by distributing the urls into multiple queues and assign workers to each\nqueue.<\/p>\n<p>Let&rsquo;s implement this functionality, We can create a new function called <code>Start<\/code>\nfor the crawler struct:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-go\" data-lang=\"go\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">func<\/span> (c <span style=\"color:#af3a03\">*<\/span>Crawler) <span style=\"color:#b57614\">Start<\/span>() {\n<\/span><\/span><span style=\"display:flex;\"><span>    distributedInputs <span style=\"color:#af3a03\">:=<\/span> <span style=\"color:#b57614\">make<\/span>([]<span style=\"color:#af3a03\">chan<\/span> <span style=\"color:#af3a03\">*<\/span>url.URL, c.config.WorkerCount)\n<\/span><\/span><span style=\"display:flex;\"><span>    workersResults <span style=\"color:#af3a03\">:=<\/span> <span style=\"color:#b57614\">make<\/span>([]<span style=\"color:#af3a03\">chan<\/span> CrawlResult, c.config.WorkerCount)\n<\/span><\/span><span style=\"display:flex;\"><span>    done <span style=\"color:#af3a03\">:=<\/span> <span style=\"color:#b57614\">make<\/span>(<span style=\"color:#af3a03\">chan<\/span> <span style=\"color:#af3a03\">struct<\/span>{})\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">for<\/span> i <span style=\"color:#af3a03\">:=<\/span> <span style=\"color:#8f3f71\">0<\/span>; i &lt; c.config.WorkerCount; i<span style=\"color:#af3a03\">++<\/span> {\n<\/span><\/span><span style=\"display:flex;\"><span>        distributedInputs[i] = <span style=\"color:#b57614\">make<\/span>(<span style=\"color:#af3a03\">chan<\/span> <span style=\"color:#af3a03\">*<\/span>url.URL)\n<\/span><\/span><span style=\"display:flex;\"><span>        workersResults[i] = <span style=\"color:#b57614\">make<\/span>(<span style=\"color:#af3a03\">chan<\/span> CrawlResult)\n<\/span><\/span><span style=\"display:flex;\"><span>    }\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">go<\/span> <span style=\"color:#b57614\">distributeUrls<\/span>(c.frontier, distributedInputs)\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">for<\/span> i <span style=\"color:#af3a03\">:=<\/span> <span style=\"color:#8f3f71\">0<\/span>; i &lt; c.config.WorkerCount; i<span style=\"color:#af3a03\">++<\/span> {\n<\/span><\/span><span style=\"display:flex;\"><span>        worker <span style=\"color:#af3a03\">:=<\/span> <span style=\"color:#b57614\">NewWorker<\/span>(distributedInputs[i], workersResults[i], done, i, c.deadLetter)\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">go<\/span> worker.<span style=\"color:#b57614\">Start<\/span>()\n<\/span><\/span><span style=\"display:flex;\"><span>    }\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    mergedResults <span style=\"color:#af3a03\">:=<\/span> <span style=\"color:#b57614\">make<\/span>(<span style=\"color:#af3a03\">chan<\/span> CrawlResult)\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">go<\/span> <span style=\"color:#b57614\">mergeResults<\/span>(workersResults, mergedResults)<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>Here we start by creating an input channel and an output channel for each worker.<\/p>\n<p>There is a done channel here as well. It&rsquo;s a practice in go to use an empty\nchannel to notify the goroutines that the process is done or cancelled.<\/p>\n<p>Then a function will start distributing URLs from frontier to worker channels.<\/p>\n<p>Finally we have a another function that merges results from worker outputs.<\/p>\n<p>Note that these two functions and worker start are executed in a separate goroutine.\nSo they will continuously consume from frontier, add to worker input channel,\nand put merge the result into a single output channel.<\/p>\n<p>How can we implement the distribute and merge mechanisms?\n<a href=\"https:\/\/go.dev\/blog\/pipelines\" rel=\"noopener\" target=\"_blank\">This post<\/a> fully explains the fan-in and fan-out.<\/p>\n<p>Let&rsquo;s create a separate file and implement these two functions.<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-go\" data-lang=\"go\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">func<\/span> <span style=\"color:#b57614\">distributeUrls<\/span>(frontier <span style=\"color:#af3a03\">*<\/span>frontier.Frontier, distributedInputs []<span style=\"color:#af3a03\">chan<\/span> <span style=\"color:#af3a03\">*<\/span>url.URL) {\n<\/span><\/span><span style=\"display:flex;\"><span>    HostToWorker <span style=\"color:#af3a03\">:=<\/span> <span style=\"color:#b57614\">make<\/span>(<span style=\"color:#af3a03\">map<\/span>[<span style=\"color:#b57614\">string<\/span>]<span style=\"color:#b57614\">int<\/span>)\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">for<\/span> url <span style=\"color:#af3a03\">:=<\/span> <span style=\"color:#af3a03\">range<\/span> frontier.<span style=\"color:#b57614\">Get<\/span>() {\n<\/span><\/span><span style=\"display:flex;\"><span>        index <span style=\"color:#af3a03\">:=<\/span> rand.<span style=\"color:#b57614\">Intn<\/span>(<span style=\"color:#b57614\">len<\/span>(distributedInputs))\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">if<\/span> prevIndex, ok <span style=\"color:#af3a03\">:=<\/span> HostToWorker[url.Host]; ok {\n<\/span><\/span><span style=\"display:flex;\"><span>            index = prevIndex\n<\/span><\/span><span style=\"display:flex;\"><span>        } <span style=\"color:#af3a03\">else<\/span> {\n<\/span><\/span><span style=\"display:flex;\"><span>            HostToWorker[url.Host] = index\n<\/span><\/span><span style=\"display:flex;\"><span>        }\n<\/span><\/span><span style=\"display:flex;\"><span>        distributedInputs[index] <span style=\"color:#af3a03\">&lt;-<\/span> url\n<\/span><\/span><span style=\"display:flex;\"><span>    }\n<\/span><\/span><span style=\"display:flex;\"><span>}<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>Here we have a for loop over a channel.\nThis means that our function never exits until the frontier closes the channel.\nFor each url coming into the channel we take it and assign it to a worker input channel.<\/p>\n<p>The assignment algorithm is very simple, we have a list of already assigned hosts.\nIf a host is new we assign it randomly, otherwise we send it to the assigned host.<\/p>\n<p>Now let&rsquo;s implement the merger:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-go\" data-lang=\"go\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">func<\/span> <span style=\"color:#b57614\">mergeResults<\/span>(workersResults []<span style=\"color:#af3a03\">chan<\/span> CrawlResult, out <span style=\"color:#af3a03\">chan<\/span> CrawlResult) {\n<\/span><\/span><span style=\"display:flex;\"><span>    collect <span style=\"color:#af3a03\">:=<\/span> <span style=\"color:#af3a03\">func<\/span>(in <span style=\"color:#af3a03\">chan<\/span> CrawlResult) {\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">for<\/span> result <span style=\"color:#af3a03\">:=<\/span> <span style=\"color:#af3a03\">range<\/span> in {\n<\/span><\/span><span style=\"display:flex;\"><span>            out <span style=\"color:#af3a03\">&lt;-<\/span> result\n<\/span><\/span><span style=\"display:flex;\"><span>        }\n<\/span><\/span><span style=\"display:flex;\"><span>        log.<span style=\"color:#b57614\">Println<\/span>(<span style=\"color:#79740e\">&#34;Worker finished&#34;<\/span>)\n<\/span><\/span><span style=\"display:flex;\"><span>    }\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">for<\/span> i, result <span style=\"color:#af3a03\">:=<\/span> <span style=\"color:#af3a03\">range<\/span> workersResults {\n<\/span><\/span><span style=\"display:flex;\"><span>        log.<span style=\"color:#b57614\">Printf<\/span>(<span style=\"color:#79740e\">&#34;Start collecting results from worker %d&#34;<\/span>, i)\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">go<\/span> <span style=\"color:#b57614\">collect<\/span>(result)\n<\/span><\/span><span style=\"display:flex;\"><span>    }\n<\/span><\/span><span style=\"display:flex;\"><span>}<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>This function might be a bit more complex.\nFirst we created a function named collect that consumes from a single channel.\nThen we loop over the workers and call this function on all the output channels.<\/p>\n<p>So this starts a goroutine per worker that listens to output channel.\nThe result is put into the merged output channel.<\/p>\n<p>Pretty simple yet powerful technique to parallelize a task.<\/p>\n<h3 class=\"heading\" id=\"worker\">\n  Worker\n  <a class=\"anchor\" href=\"#worker\">#<\/a>\n<\/h3>\n<p>To implement the worker we first need to define the struct and crawl result.<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-go\" data-lang=\"go\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">type<\/span> CrawlResult <span style=\"color:#af3a03\">struct<\/span> {\n<\/span><\/span><span style=\"display:flex;\"><span>    Url         <span style=\"color:#af3a03\">*<\/span>url.URL\n<\/span><\/span><span style=\"display:flex;\"><span>    ContentType <span style=\"color:#b57614\">string<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>    Body        []<span style=\"color:#b57614\">byte<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>}\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">type<\/span> Worker <span style=\"color:#af3a03\">struct<\/span> {\n<\/span><\/span><span style=\"display:flex;\"><span>    input      <span style=\"color:#af3a03\">chan<\/span> <span style=\"color:#af3a03\">*<\/span>url.URL\n<\/span><\/span><span style=\"display:flex;\"><span>    deadLetter <span style=\"color:#af3a03\">chan<\/span> <span style=\"color:#af3a03\">*<\/span>url.URL\n<\/span><\/span><span style=\"display:flex;\"><span>    result     <span style=\"color:#af3a03\">chan<\/span> CrawlResult\n<\/span><\/span><span style=\"display:flex;\"><span>    done       <span style=\"color:#af3a03\">chan<\/span> <span style=\"color:#af3a03\">struct<\/span>{}\n<\/span><\/span><span style=\"display:flex;\"><span>    id         <span style=\"color:#b57614\">int<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>    logger     <span style=\"color:#af3a03\">*<\/span>log.Entry\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#928374;font-style:italic\">\/\/ Only contains the host part of the URL<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>    history <span style=\"color:#af3a03\">map<\/span>[<span style=\"color:#b57614\">string<\/span>]time.Time\n<\/span><\/span><span style=\"display:flex;\"><span>}<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>The crawl result represents a successful page load with content and the type.<\/p>\n<p>Let&rsquo;s breakdown what worker has:<\/p>\n<ul>\n<li>input: the channel that worker consumes from<\/li>\n<li>deadLetter: another channel that worker puts in the failed URLs into<\/li>\n<li>result: channel for sending successful crawls<\/li>\n<li>done: the channel that notifies the worker if it has to stop<\/li>\n<li>id: an id assigned to the worker this is useful for marking logs from each worker<\/li>\n<li>logger: a logger with worker context so log messages are distinguishable from others.\n<code>logger := log.WithField(&quot;worker&quot;, id)<\/code><\/li>\n<\/ul>\n<p>The Start method of the worker is a for-select statement to consume\nany message that comes into the input channel, fetch and pass the result.<\/p>\n<p>Before fetching the URL we check for politeness and sleep if needed.\nThere is a downside to this if we have consecutive URLs from one host.\nSince we have to sleep and it slows down.<\/p>\n<p>There are two improvements here I can think of:<\/p>\n<ol>\n<li>Discarding that URL to deadletter and continue until we get another host<\/li>\n<li>Distribute the URLs in worker input channel to reduce the chance of blocking<\/li>\n<\/ol>\n<p>But here we just go with the simple approach<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-go\" data-lang=\"go\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">func<\/span> (w <span style=\"color:#af3a03\">*<\/span>Worker) <span style=\"color:#b57614\">Start<\/span>() {\n<\/span><\/span><span style=\"display:flex;\"><span>    w.logger.<span style=\"color:#b57614\">Debugf<\/span>(<span style=\"color:#79740e\">&#34;Worker %d started&#34;<\/span>, w.id)\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">for<\/span> {\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">select<\/span> {\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">case<\/span> url <span style=\"color:#af3a03\">:=<\/span> <span style=\"color:#af3a03\">&lt;-<\/span>w.input:\n<\/span><\/span><span style=\"display:flex;\"><span>            <span style=\"color:#af3a03\">for<\/span> !w.<span style=\"color:#b57614\">CheckPoliteness<\/span>(url) {\n<\/span><\/span><span style=\"display:flex;\"><span>                time.<span style=\"color:#b57614\">Sleep<\/span>(<span style=\"color:#8f3f71\">2<\/span> <span style=\"color:#af3a03\">*<\/span> time.Second)\n<\/span><\/span><span style=\"display:flex;\"><span>            }\n<\/span><\/span><span style=\"display:flex;\"><span>            content, err <span style=\"color:#af3a03\">:=<\/span> w.<span style=\"color:#b57614\">fetch<\/span>(url)\n<\/span><\/span><span style=\"display:flex;\"><span>            <span style=\"color:#af3a03\">if<\/span> err <span style=\"color:#af3a03\">!=<\/span> <span style=\"color:#af3a03\">nil<\/span> {\n<\/span><\/span><span style=\"display:flex;\"><span>                log.<span style=\"color:#b57614\">Errorf<\/span>(<span style=\"color:#79740e\">&#34;Worker %d error fetching content: %s&#34;<\/span>, w.id, err)\n<\/span><\/span><span style=\"display:flex;\"><span>                w.deadLetter <span style=\"color:#af3a03\">&lt;-<\/span> url\n<\/span><\/span><span style=\"display:flex;\"><span>                <span style=\"color:#af3a03\">continue<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>            }\n<\/span><\/span><span style=\"display:flex;\"><span>            w.history[url.Host] = time.<span style=\"color:#b57614\">Now<\/span>()\n<\/span><\/span><span style=\"display:flex;\"><span>            b. result <span style=\"color:#af3a03\">&lt;-<\/span> content\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">case<\/span> <span style=\"color:#af3a03\">&lt;-<\/span>w.done:\n<\/span><\/span><span style=\"display:flex;\"><span>            <span style=\"color:#af3a03\">return<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>        }\n<\/span><\/span><span style=\"display:flex;\"><span>    }\n<\/span><\/span><span style=\"display:flex;\"><span>}<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>The fetch function does a simple fetch and also determines the content type.<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-go\" data-lang=\"go\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">func<\/span> (w <span style=\"color:#af3a03\">*<\/span>Worker) <span style=\"color:#b57614\">fetch<\/span>(url <span style=\"color:#af3a03\">*<\/span>url.URL) (CrawlResult, <span style=\"color:#b57614\">error<\/span>) {\n<\/span><\/span><span style=\"display:flex;\"><span>    w.logger.<span style=\"color:#b57614\">Debugf<\/span>(<span style=\"color:#79740e\">&#34;Worker %d fetching %s&#34;<\/span>, w.id, url)\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">defer<\/span> w.logger.<span style=\"color:#b57614\">Debugf<\/span>(<span style=\"color:#79740e\">&#34;Worker %d done fetching %s&#34;<\/span>, w.id, url)\n<\/span><\/span><span style=\"display:flex;\"><span>    res, err <span style=\"color:#af3a03\">:=<\/span> http.<span style=\"color:#b57614\">Get<\/span>(url.<span style=\"color:#b57614\">String<\/span>())\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">if<\/span> err <span style=\"color:#af3a03\">!=<\/span> <span style=\"color:#af3a03\">nil<\/span> {\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">return<\/span> CrawlResult{}, err\n<\/span><\/span><span style=\"display:flex;\"><span>    }\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">defer<\/span> res.Body.<span style=\"color:#b57614\">Close<\/span>()\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">if<\/span> res.StatusCode <span style=\"color:#af3a03\">!=<\/span> http.StatusOK {\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">return<\/span> CrawlResult{}, fmt.<span style=\"color:#b57614\">Errorf<\/span>(<span style=\"color:#79740e\">&#34;status code error: %d %s&#34;<\/span>, res.StatusCode, res.Status)\n<\/span><\/span><span style=\"display:flex;\"><span>    }\n<\/span><\/span><span style=\"display:flex;\"><span>    body, err <span style=\"color:#af3a03\">:=<\/span> io.<span style=\"color:#b57614\">ReadAll<\/span>(res.Body)\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">if<\/span> err <span style=\"color:#af3a03\">!=<\/span> <span style=\"color:#af3a03\">nil<\/span> {\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">return<\/span> CrawlResult{}, err\n<\/span><\/span><span style=\"display:flex;\"><span>    }\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">var<\/span> inferredContentType <span style=\"color:#b57614\">string<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>    contentType, ok <span style=\"color:#af3a03\">:=<\/span> res.Header[<span style=\"color:#79740e\">&#34;Content-Type&#34;<\/span>]\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">if<\/span> ok <span style=\"color:#af3a03\">&amp;&amp;<\/span> <span style=\"color:#b57614\">len<\/span>(contentType) &gt; <span style=\"color:#8f3f71\">0<\/span> {\n<\/span><\/span><span style=\"display:flex;\"><span>        inferredContentType = contentType[<span style=\"color:#8f3f71\">0<\/span>]\n<\/span><\/span><span style=\"display:flex;\"><span>    } <span style=\"color:#af3a03\">else<\/span> {\n<\/span><\/span><span style=\"display:flex;\"><span>        inferredContentType = http.<span style=\"color:#b57614\">DetectContentType<\/span>(body)\n<\/span><\/span><span style=\"display:flex;\"><span>    }\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">return<\/span> CrawlResult{\n<\/span><\/span><span style=\"display:flex;\"><span>        Url:         url,\n<\/span><\/span><span style=\"display:flex;\"><span>        ContentType: inferredContentType,\n<\/span><\/span><span style=\"display:flex;\"><span>        Body:        body,\n<\/span><\/span><span style=\"display:flex;\"><span>    }, <span style=\"color:#af3a03\">nil<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>}\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">func<\/span> (w <span style=\"color:#af3a03\">*<\/span>Worker) <span style=\"color:#b57614\">CheckPoliteness<\/span>(url <span style=\"color:#af3a03\">*<\/span>url.URL) <span style=\"color:#b57614\">bool<\/span> {\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">if<\/span> lastFetch, ok <span style=\"color:#af3a03\">:=<\/span> w.history[url.Host]; ok {\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">return<\/span> time.<span style=\"color:#b57614\">Since<\/span>(lastFetch) &gt; <span style=\"color:#8f3f71\">2<\/span><span style=\"color:#af3a03\">*<\/span>time.Second\n<\/span><\/span><span style=\"display:flex;\"><span>    }\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">return<\/span> <span style=\"color:#af3a03\">true<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>}<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<h3 class=\"heading\" id=\"extracting-links\">\n  Extracting Links\n  <a class=\"anchor\" href=\"#extracting-links\">#<\/a>\n<\/h3>\n<p>To extract a link we implement the Processor interface we defined above.<\/p>\n<p>This processor takes in parsers and crawl result then outputs links.<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-go\" data-lang=\"go\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">type<\/span> LinkExtractor <span style=\"color:#af3a03\">struct<\/span> {\n<\/span><\/span><span style=\"display:flex;\"><span>    Parsers []parser.Parser\n<\/span><\/span><span style=\"display:flex;\"><span>    NewUrls <span style=\"color:#af3a03\">chan<\/span> <span style=\"color:#af3a03\">*<\/span>url.URL\n<\/span><\/span><span style=\"display:flex;\"><span>}\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">func<\/span> (e <span style=\"color:#af3a03\">*<\/span>LinkExtractor) <span style=\"color:#b57614\">Process<\/span>(result CrawlResult) <span style=\"color:#b57614\">error<\/span> {\n<\/span><\/span><span style=\"display:flex;\"><span>    foundUrls <span style=\"color:#af3a03\">:=<\/span> <span style=\"color:#b57614\">make<\/span>([]<span style=\"color:#af3a03\">*<\/span>url.URL, <span style=\"color:#8f3f71\">0<\/span>)\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">for<\/span> _, parser <span style=\"color:#af3a03\">:=<\/span> <span style=\"color:#af3a03\">range<\/span> e.Parsers {\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">if<\/span> !parser.<span style=\"color:#b57614\">IsSupportedExtension<\/span>(result.ContentType) {\n<\/span><\/span><span style=\"display:flex;\"><span>            <span style=\"color:#af3a03\">continue<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>        }\n<\/span><\/span><span style=\"display:flex;\"><span>        parsedUrls, err <span style=\"color:#af3a03\">:=<\/span> parser.<span style=\"color:#b57614\">Parse<\/span>(<span style=\"color:#b57614\">string<\/span>(result.Body))\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">if<\/span> err <span style=\"color:#af3a03\">!=<\/span> <span style=\"color:#af3a03\">nil<\/span> {\n<\/span><\/span><span style=\"display:flex;\"><span>            <span style=\"color:#af3a03\">return<\/span> fmt.<span style=\"color:#b57614\">Errorf<\/span>(<span style=\"color:#79740e\">&#34;Error parsing content: %s&#34;<\/span>, err)\n<\/span><\/span><span style=\"display:flex;\"><span>        }\n<\/span><\/span><span style=\"display:flex;\"><span>        log.<span style=\"color:#b57614\">Infof<\/span>(<span style=\"color:#79740e\">&#34;Extracted %d urls&#34;<\/span>, <span style=\"color:#b57614\">len<\/span>(parsedUrls))\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">for<\/span> _, parsedUrl <span style=\"color:#af3a03\">:=<\/span> <span style=\"color:#af3a03\">range<\/span> parsedUrls {\n<\/span><\/span><span style=\"display:flex;\"><span>            newUrl, err <span style=\"color:#af3a03\">:=<\/span> url.<span style=\"color:#b57614\">Parse<\/span>(parsedUrl.Value)\n<\/span><\/span><span style=\"display:flex;\"><span>            <span style=\"color:#af3a03\">if<\/span> err <span style=\"color:#af3a03\">!=<\/span> <span style=\"color:#af3a03\">nil<\/span> {\n<\/span><\/span><span style=\"display:flex;\"><span>                log.<span style=\"color:#b57614\">Debugf<\/span>(<span style=\"color:#79740e\">&#34;Error parsing url: %s&#34;<\/span>, err)\n<\/span><\/span><span style=\"display:flex;\"><span>                <span style=\"color:#af3a03\">continue<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>            }\n<\/span><\/span><span style=\"display:flex;\"><span>            params <span style=\"color:#af3a03\">:=<\/span> newUrl.<span style=\"color:#b57614\">Query<\/span>()\n<\/span><\/span><span style=\"display:flex;\"><span>            <span style=\"color:#af3a03\">for<\/span> param <span style=\"color:#af3a03\">:=<\/span> <span style=\"color:#af3a03\">range<\/span> params {\n<\/span><\/span><span style=\"display:flex;\"><span>                newUrl = <span style=\"color:#b57614\">stripQueryParam<\/span>(newUrl, param)\n<\/span><\/span><span style=\"display:flex;\"><span>            }\n<\/span><\/span><span style=\"display:flex;\"><span>            <span style=\"color:#af3a03\">if<\/span> newUrl.Scheme <span style=\"color:#af3a03\">==<\/span> <span style=\"color:#79740e\">&#34;http&#34;<\/span> <span style=\"color:#af3a03\">||<\/span> newUrl.Scheme <span style=\"color:#af3a03\">==<\/span> <span style=\"color:#79740e\">&#34;https&#34;<\/span> {\n<\/span><\/span><span style=\"display:flex;\"><span>                foundUrls = <span style=\"color:#b57614\">append<\/span>(foundUrls, newUrl)\n<\/span><\/span><span style=\"display:flex;\"><span>            }\n<\/span><\/span><span style=\"display:flex;\"><span>        }\n<\/span><\/span><span style=\"display:flex;\"><span>    }\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">for<\/span> _, foundUrl <span style=\"color:#af3a03\">:=<\/span> <span style=\"color:#af3a03\">range<\/span> foundUrls {\n<\/span><\/span><span style=\"display:flex;\"><span>        e.NewUrls <span style=\"color:#af3a03\">&lt;-<\/span> foundUrl\n<\/span><\/span><span style=\"display:flex;\"><span>    }\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">return<\/span> <span style=\"color:#af3a03\">nil<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>}\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">func<\/span> <span style=\"color:#b57614\">stripQueryParam<\/span>(inputURL <span style=\"color:#af3a03\">*<\/span>url.URL, stripKey <span style=\"color:#b57614\">string<\/span>) <span style=\"color:#af3a03\">*<\/span>url.URL {\n<\/span><\/span><span style=\"display:flex;\"><span>    query <span style=\"color:#af3a03\">:=<\/span> inputURL.<span style=\"color:#b57614\">Query<\/span>()\n<\/span><\/span><span style=\"display:flex;\"><span>    query.<span style=\"color:#b57614\">Del<\/span>(stripKey)\n<\/span><\/span><span style=\"display:flex;\"><span>    inputURL.RawQuery = query.<span style=\"color:#b57614\">Encode<\/span>()\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">return<\/span> inputURL\n<\/span><\/span><span style=\"display:flex;\"><span>}<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>The this struct keeps a list of parsers and has a channel to output links.<\/p>\n<p>The process function takes in a crawl result and matches the type with it&rsquo;s parsers.\nIt&rsquo;s also important to make sure we strip the query params,\nstrings like <code>?sort=foo<\/code>.\nThere might be case that we care about them, but here to simply remove duplicates\nwe do this.<\/p>\n<p>A better approach here is to use the <code>rel=canonical<\/code> HTML attribute to identify if\nURL is identical to current page.<\/p>\n<p>The result from this extractor are put in a new channel.<\/p>\n<p>So in the crawler we can add this processor and get the URLs:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-go\" data-lang=\"go\"><span style=\"display:flex;\"><span>    c.<span style=\"color:#b57614\">AddProcessor<\/span>(<span style=\"color:#af3a03\">&amp;<\/span>LinkExtractor{Parsers: c.contentParsers, NewUrls: newUrls})\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">go<\/span> <span style=\"color:#af3a03\">func<\/span>() {\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">for<\/span> newUrl <span style=\"color:#af3a03\">:=<\/span> <span style=\"color:#af3a03\">range<\/span> newUrls {\n<\/span><\/span><span style=\"display:flex;\"><span>            _ = c.frontier.<span style=\"color:#b57614\">Add<\/span>(newUrl)\n<\/span><\/span><span style=\"display:flex;\"><span>        }\n<\/span><\/span><span style=\"display:flex;\"><span>    }()<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<h3 class=\"heading\" id=\"saving-content\">\n  Saving Content\n  <a class=\"anchor\" href=\"#saving-content\">#<\/a>\n<\/h3>\n<p>To save the content we use another processor.\nThis processor uses the storage backed provided to the crawler to store pages.<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-go\" data-lang=\"go\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">type<\/span> SaveToFile <span style=\"color:#af3a03\">struct<\/span> {\n<\/span><\/span><span style=\"display:flex;\"><span>    storageBackend storage.Storage\n<\/span><\/span><span style=\"display:flex;\"><span>}\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">func<\/span> (s <span style=\"color:#af3a03\">*<\/span>SaveToFile) <span style=\"color:#b57614\">Process<\/span>(result CrawlResult) <span style=\"color:#b57614\">error<\/span> {\n<\/span><\/span><span style=\"display:flex;\"><span>    savePath <span style=\"color:#af3a03\">:=<\/span> <span style=\"color:#b57614\">getSavePath<\/span>(result.Url)\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">switch<\/span> result.ContentType {\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">default<\/span>:\n<\/span><\/span><span style=\"display:flex;\"><span>        savePath = savePath <span style=\"color:#af3a03\">+<\/span> <span style=\"color:#79740e\">&#34;.html&#34;<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>        err <span style=\"color:#af3a03\">:=<\/span> s.storageBackend.<span style=\"color:#b57614\">Set<\/span>(savePath, <span style=\"color:#b57614\">string<\/span>(result.Body))\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">if<\/span> err <span style=\"color:#af3a03\">!=<\/span> <span style=\"color:#af3a03\">nil<\/span> {\n<\/span><\/span><span style=\"display:flex;\"><span>            <span style=\"color:#af3a03\">return<\/span> err\n<\/span><\/span><span style=\"display:flex;\"><span>        }\n<\/span><\/span><span style=\"display:flex;\"><span>    }\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">return<\/span> <span style=\"color:#af3a03\">nil<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>}\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">func<\/span> <span style=\"color:#b57614\">getSavePath<\/span>(url <span style=\"color:#af3a03\">*<\/span>url.URL) <span style=\"color:#b57614\">string<\/span> {\n<\/span><\/span><span style=\"display:flex;\"><span>    fileName <span style=\"color:#af3a03\">:=<\/span> url.Path <span style=\"color:#af3a03\">+<\/span> <span style=\"color:#79740e\">&#34;-page&#34;<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>    savePath <span style=\"color:#af3a03\">:=<\/span> path.<span style=\"color:#b57614\">Join<\/span>(url.Host, fileName)\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">return<\/span> savePath\n<\/span><\/span><span style=\"display:flex;\"><span>}<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>And again we add it easily to the crawler:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-go\" data-lang=\"go\"><span style=\"display:flex;\"><span>    c.<span style=\"color:#b57614\">AddProcessor<\/span>(<span style=\"color:#af3a03\">&amp;<\/span>SaveToFile{storageBackend: c.storage})<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<h3 class=\"heading\" id=\"running-processors\">\n  Running Processors\n  <a class=\"anchor\" href=\"#running-processors\">#<\/a>\n<\/h3>\n<p>The final step in our start method is to run processors on results.<\/p>\n<p>Since the list of processors can be extended and we must not block the\ngoroutine, we execute each of them in a separate goroutine.<\/p>\n<p>This is important because if we can&rsquo;t consume from the merged results fast enough\nthen each worker might wait until the processors are ran so they can send to channel.\nRemember the send to channel blocks until the consumer is ready.<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-go\" data-lang=\"go\"><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">for<\/span> result <span style=\"color:#af3a03\">:=<\/span> <span style=\"color:#af3a03\">range<\/span> mergedResults {\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">for<\/span> _, processor <span style=\"color:#af3a03\">:=<\/span> <span style=\"color:#af3a03\">range<\/span> c.processors {\n<\/span><\/span><span style=\"display:flex;\"><span>            <span style=\"color:#af3a03\">go<\/span> <span style=\"color:#af3a03\">func<\/span>(processor Processor, result CrawlResult) {\n<\/span><\/span><span style=\"display:flex;\"><span>                processErr <span style=\"color:#af3a03\">:=<\/span> processor.<span style=\"color:#b57614\">Process<\/span>(result)\n<\/span><\/span><span style=\"display:flex;\"><span>                <span style=\"color:#af3a03\">if<\/span> processErr <span style=\"color:#af3a03\">!=<\/span> <span style=\"color:#af3a03\">nil<\/span> {\n<\/span><\/span><span style=\"display:flex;\"><span>                    log.<span style=\"color:#b57614\">Error<\/span>(processErr)\n<\/span><\/span><span style=\"display:flex;\"><span>                }\n<\/span><\/span><span style=\"display:flex;\"><span>            }(processor, result)\n<\/span><\/span><span style=\"display:flex;\"><span>        }\n<\/span><\/span><span style=\"display:flex;\"><span>    }<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<h3 class=\"heading\" id=\"failed-urls\">\n  Failed URLs\n  <a class=\"anchor\" href=\"#failed-urls\">#<\/a>\n<\/h3>\n<p>This part is open ended and you can try it as an exercise.\nWe only consume the failed ones and log them to the console.<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-go\" data-lang=\"go\"><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">go<\/span> <span style=\"color:#af3a03\">func<\/span>() {\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">for<\/span> deadUrl <span style=\"color:#af3a03\">:=<\/span> <span style=\"color:#af3a03\">range<\/span> c.deadLetter {\n<\/span><\/span><span style=\"display:flex;\"><span>            log.<span style=\"color:#b57614\">Debugf<\/span>(<span style=\"color:#79740e\">&#34;Dismissed %s&#34;<\/span>, deadUrl)\n<\/span><\/span><span style=\"display:flex;\"><span>        }\n<\/span><\/span><span style=\"display:flex;\"><span>    }()<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<h3 class=\"heading\" id=\"html-parser\">\n  HTML parser\n  <a class=\"anchor\" href=\"#html-parser\">#<\/a>\n<\/h3>\n<p>We need to implement at least 1 parser so our crawler can\nparse HTML pages.<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-go\" data-lang=\"go\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">package<\/span> parser\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">import<\/span> (\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#79740e\">&#34;errors&#34;<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#79740e\">&#34;fmt&#34;<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#79740e\">&#34;io&#34;<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#79740e\">&#34;strings&#34;<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#79740e\">&#34;golang.org\/x\/net\/html&#34;<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>)\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">type<\/span> HtmlParser <span style=\"color:#af3a03\">struct<\/span> {\n<\/span><\/span><span style=\"display:flex;\"><span>}\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">func<\/span> (p <span style=\"color:#af3a03\">*<\/span>HtmlParser) <span style=\"color:#b57614\">getSupportedExtensions<\/span>() []<span style=\"color:#b57614\">string<\/span> {\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">return<\/span> []<span style=\"color:#b57614\">string<\/span>{<span style=\"color:#79740e\">&#34;.html&#34;<\/span>, <span style=\"color:#79740e\">&#34;.htm&#34;<\/span>}\n<\/span><\/span><span style=\"display:flex;\"><span>}\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">func<\/span> (p <span style=\"color:#af3a03\">*<\/span>HtmlParser) <span style=\"color:#b57614\">IsSupportedExtension<\/span>(extension <span style=\"color:#b57614\">string<\/span>) <span style=\"color:#b57614\">bool<\/span> {\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">for<\/span> _, supportedExtension <span style=\"color:#af3a03\">:=<\/span> <span style=\"color:#af3a03\">range<\/span> p.<span style=\"color:#b57614\">getSupportedExtensions<\/span>() {\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">if<\/span> extension <span style=\"color:#af3a03\">==<\/span> supportedExtension {\n<\/span><\/span><span style=\"display:flex;\"><span>            <span style=\"color:#af3a03\">return<\/span> <span style=\"color:#af3a03\">true<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>        }\n<\/span><\/span><span style=\"display:flex;\"><span>    }\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">return<\/span> <span style=\"color:#af3a03\">true<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>}\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">func<\/span> (p <span style=\"color:#af3a03\">*<\/span>HtmlParser) <span style=\"color:#b57614\">Parse<\/span>(content <span style=\"color:#b57614\">string<\/span>) ([]Token, <span style=\"color:#b57614\">error<\/span>) {\n<\/span><\/span><span style=\"display:flex;\"><span>    htmlParser <span style=\"color:#af3a03\">:=<\/span> html.<span style=\"color:#b57614\">NewTokenizer<\/span>(strings.<span style=\"color:#b57614\">NewReader<\/span>(content))\n<\/span><\/span><span style=\"display:flex;\"><span>    tokens <span style=\"color:#af3a03\">:=<\/span> []Token{}\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">for<\/span> {\n<\/span><\/span><span style=\"display:flex;\"><span>        tokenType <span style=\"color:#af3a03\">:=<\/span> htmlParser.<span style=\"color:#b57614\">Next<\/span>()\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">if<\/span> tokenType <span style=\"color:#af3a03\">==<\/span> html.ErrorToken {\n<\/span><\/span><span style=\"display:flex;\"><span>            <span style=\"color:#af3a03\">break<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>        }\n<\/span><\/span><span style=\"display:flex;\"><span>        token <span style=\"color:#af3a03\">:=<\/span> htmlParser.<span style=\"color:#b57614\">Token<\/span>()\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">if<\/span> tokenType <span style=\"color:#af3a03\">==<\/span> html.StartTagToken {\n<\/span><\/span><span style=\"display:flex;\"><span>            <span style=\"color:#af3a03\">switch<\/span> token.Data {\n<\/span><\/span><span style=\"display:flex;\"><span>            <span style=\"color:#af3a03\">case<\/span> <span style=\"color:#79740e\">&#34;a&#34;<\/span>:\n<\/span><\/span><span style=\"display:flex;\"><span>                <span style=\"color:#af3a03\">for<\/span> _, attr <span style=\"color:#af3a03\">:=<\/span> <span style=\"color:#af3a03\">range<\/span> token.Attr {\n<\/span><\/span><span style=\"display:flex;\"><span>                    <span style=\"color:#af3a03\">if<\/span> attr.Key <span style=\"color:#af3a03\">==<\/span> <span style=\"color:#79740e\">&#34;href&#34;<\/span> {\n<\/span><\/span><span style=\"display:flex;\"><span>                        tokens = <span style=\"color:#b57614\">append<\/span>(tokens, Token{Name: <span style=\"color:#79740e\">&#34;link&#34;<\/span>, Value: attr.Val})\n<\/span><\/span><span style=\"display:flex;\"><span>                    }\n<\/span><\/span><span style=\"display:flex;\"><span>                }\n<\/span><\/span><span style=\"display:flex;\"><span>            }\n<\/span><\/span><span style=\"display:flex;\"><span>        }\n<\/span><\/span><span style=\"display:flex;\"><span>    }\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">if<\/span> htmlParser.<span style=\"color:#b57614\">Err<\/span>() <span style=\"color:#af3a03\">!=<\/span> <span style=\"color:#af3a03\">nil<\/span> {\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">if<\/span> !errors.<span style=\"color:#b57614\">Is<\/span>(htmlParser.<span style=\"color:#b57614\">Err<\/span>(), io.EOF) {\n<\/span><\/span><span style=\"display:flex;\"><span>            <span style=\"color:#af3a03\">return<\/span> tokens, fmt.<span style=\"color:#b57614\">Errorf<\/span>(<span style=\"color:#79740e\">&#34;error scanning html: %s&#34;<\/span>, htmlParser.<span style=\"color:#b57614\">Err<\/span>())\n<\/span><\/span><span style=\"display:flex;\"><span>        }\n<\/span><\/span><span style=\"display:flex;\"><span>    }\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">return<\/span> tokens, <span style=\"color:#af3a03\">nil<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>}<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>The parsing process is straight forward, we use a parser package and\nwalk over the elements and extract the ones with <code>a<\/code> tag and <code>href<\/code> attribute.<\/p>\n<h3 class=\"heading\" id=\"putting-it-all-together\">\n  Putting it All Together\n  <a class=\"anchor\" href=\"#putting-it-all-together\">#<\/a>\n<\/h3>\n<p>We finally have everything needed to crawl some pages.<\/p>\n<p>The parser we created is not a program, it&rsquo;s a library.\nThis can be imported and be started within another program.<\/p>\n<p>You can create a CLI using this or use a main function.\nWe&rsquo;ll create a main function to test it out:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-go\" data-lang=\"go\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">func<\/span> <span style=\"color:#b57614\">main<\/span>() {\n<\/span><\/span><span style=\"display:flex;\"><span>    log.<span style=\"color:#b57614\">SetFormatter<\/span>(<span style=\"color:#af3a03\">&amp;<\/span>log.TextFormatter{FullTimestamp: <span style=\"color:#af3a03\">true<\/span>})\n<\/span><\/span><span style=\"display:flex;\"><span>    initialUrls <span style=\"color:#af3a03\">:=<\/span> []url.URL{}\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    myUrl, _ <span style=\"color:#af3a03\">:=<\/span> url.<span style=\"color:#b57614\">Parse<\/span>(<span style=\"color:#79740e\">&#34;https:\/\/glyphack.com&#34;<\/span>)\n<\/span><\/span><span style=\"display:flex;\"><span>    initialUrls = <span style=\"color:#b57614\">append<\/span>(initialUrls, <span style=\"color:#af3a03\">*<\/span>myUrl)\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    contentStorage, err <span style=\"color:#af3a03\">:=<\/span> storage.<span style=\"color:#b57614\">NewFileStorage<\/span>(<span style=\"color:#79740e\">&#34;.\/data&#34;<\/span>)\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">if<\/span> err <span style=\"color:#af3a03\">!=<\/span> <span style=\"color:#af3a03\">nil<\/span> {\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#b57614\">panic<\/span>(err)\n<\/span><\/span><span style=\"display:flex;\"><span>    }\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    contentParsers <span style=\"color:#af3a03\">:=<\/span> []parser.Parser{}\n<\/span><\/span><span style=\"display:flex;\"><span>    contentParsers = <span style=\"color:#b57614\">append<\/span>(contentParsers, <span style=\"color:#af3a03\">&amp;<\/span>JsonParser{})\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    crawler <span style=\"color:#af3a03\">:=<\/span> crawler.<span style=\"color:#b57614\">NewCrawler<\/span>(initialUrls, contentStorage, <span style=\"color:#af3a03\">&amp;<\/span>crawler.Config{\n<\/span><\/span><span style=\"display:flex;\"><span>        MaxRedirects:    <span style=\"color:#8f3f71\">5<\/span>,\n<\/span><\/span><span style=\"display:flex;\"><span>        RevisitDelay:    time.Hour <span style=\"color:#af3a03\">*<\/span> <span style=\"color:#8f3f71\">2<\/span>,\n<\/span><\/span><span style=\"display:flex;\"><span>        WorkerCount:     <span style=\"color:#8f3f71\">100<\/span>,\n<\/span><\/span><span style=\"display:flex;\"><span>        ExcludePatterns: []<span style=\"color:#b57614\">string<\/span>{},\n<\/span><\/span><span style=\"display:flex;\"><span>    })\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    crawler.<span style=\"color:#b57614\">Start<\/span>()\n<\/span><\/span><span style=\"display:flex;\"><span>}<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<h2 class=\"heading\" id=\"conclusion\">\n  Conclusion\n  <a class=\"anchor\" href=\"#conclusion\">#<\/a>\n<\/h2>\n<p>Going through building this crawler and facing many deadlocks taught me a lot\nabout golang.\nAnd writing about this was a good practice to question my design and\nthe way I wrote the code.<\/p>\n<p>I could not explain the problems I faced in this post because I wrote it\nlong after I finished the code itself. But you know enough to not make\nthose mistakes as I did.<\/p>\n"},{"title":"Rate Limiter From Scratch in Python Part 2","link":"https:\/\/glyphack.com\/rate-limiter-python-2\/","pubDate":"Tue, 21 Feb 2023 21:34:49 +0100","guid":"https:\/\/glyphack.com\/rate-limiter-python-2\/","description":"<!--toc:start-->\n<ul>\n<li><a href=\"https:\/\/glyphack.com\/rate-limiter-python-2\/#introduction\">Introduction<\/a><\/li>\n<li><a href=\"https:\/\/glyphack.com\/rate-limiter-python-2\/#new-rate-limiting-algorithms\">New Rate Limiting Algorithms<\/a>\n<ul>\n<li><a href=\"https:\/\/glyphack.com\/rate-limiter-python-2\/#fixed-window\">Fixed Window<\/a>\n<ul>\n<li><a href=\"https:\/\/glyphack.com\/rate-limiter-python-2\/#test\">Test<\/a><\/li>\n<\/ul>\n<\/li>\n<li><a href=\"https:\/\/glyphack.com\/rate-limiter-python-2\/#sliding-window-log\">Sliding Window Log<\/a>\n<ul>\n<li><a href=\"https:\/\/glyphack.com\/rate-limiter-python-2\/#testing\">Testing<\/a><\/li>\n<\/ul>\n<\/li>\n<li><a href=\"https:\/\/glyphack.com\/rate-limiter-python-2\/#sliding-window-count\">Sliding Window Count<\/a>\n<ul>\n<li><a href=\"https:\/\/glyphack.com\/rate-limiter-python-2\/#tests\">Tests<\/a><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<li><a href=\"https:\/\/glyphack.com\/rate-limiter-python-2\/#conclusion\">Conclusion<\/a><\/li>\n<\/ul>\n<!--toc:end-->\n<h2 class=\"heading\" id=\"introduction\">\n  Introduction\n  <a class=\"anchor\" href=\"#introduction\">#<\/a>\n<\/h2>\n<p>In the last <a href=\"https:\/\/glyphack.com\/rate-limiter-python-1\/\">post<\/a>\nI started writing a rate limiter.\nThe project right now supports only 1 rate limiting algorithm(Token Bucket).<\/p>\n<p>In this part we&rsquo;re going to implement the following algorithms:<\/p>\n<ul>\n<li>Fixed window<\/li>\n<li>Sliding window log<\/li>\n<li>Sliding window count<\/li>\n<\/ul>\n<p>We&rsquo;ll see how each algorithm compares to another, and the trade offs.\nAlso after implementing each one we&rsquo;ll see how to abstractions we created\npreviously helped minimizing the implementation for new algorithms.<\/p>\n<p>At the end of this post we&rsquo;ll add Redis as storage backend to our application.<\/p>\n<h2 class=\"heading\" id=\"new-rate-limiting-algorithms\">\n  New Rate Limiting Algorithms\n  <a class=\"anchor\" href=\"#new-rate-limiting-algorithms\">#<\/a>\n<\/h2>\n<p>Before implementing the algorithm we can start by adding them to our rate limiter\nservice.<\/p>\n<p>First we need to update the LimiterStrategy enum:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-python\" data-lang=\"python\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">class<\/span> LimitStrategies(<span style=\"color:#b57614\">str<\/span>, Enum):\n<\/span><\/span><span style=\"display:flex;\"><span>    TOKEN_BUCKET <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#79740e\">&#34;token_bucket&#34;<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>    FIXED_WINDOW <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#79740e\">&#34;fixed_window&#34;<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>    SLIDING_WINDOW_LOG <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#79740e\">&#34;sliding_window_log&#34;<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>    SLIDING_WINDOW_COUNTER <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#79740e\">&#34;sliding_window_counter&#34;<\/span><\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>The code that initialized the limiter strategy objects is in rate limiter service.\nYou can use a empty class(with no implementation) and implement it as we go\nthrough them one by one.<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-python\" data-lang=\"python\"><span style=\"display:flex;\"><span>            <span style=\"color:#af3a03\">for<\/span> descriptor <span style=\"color:#af3a03\">in<\/span> rule<span style=\"color:#af3a03\">.<\/span>descriptors:\n<\/span><\/span><span style=\"display:flex;\"><span>                <span style=\"color:#af3a03\">if<\/span> config<span style=\"color:#af3a03\">.<\/span>limit_strategy <span style=\"color:#af3a03\">==<\/span> LimitStrategies<span style=\"color:#af3a03\">.<\/span>TOKEN_BUCKET:\n<\/span><\/span><span style=\"display:flex;\"><span>                    limits<span style=\"color:#af3a03\">.<\/span>append(\n<\/span><\/span><span style=\"display:flex;\"><span>                        TokenBucket(\n<\/span><\/span><span style=\"display:flex;\"><span>                            storage_backend<span style=\"color:#af3a03\">=<\/span><span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>storage_engine,\n<\/span><\/span><span style=\"display:flex;\"><span>                            rule_descriptor<span style=\"color:#af3a03\">=<\/span>descriptor,\n<\/span><\/span><span style=\"display:flex;\"><span>                        )\n<\/span><\/span><span style=\"display:flex;\"><span>                    )\n<\/span><\/span><span style=\"display:flex;\"><span>                <span style=\"color:#af3a03\">elif<\/span> config<span style=\"color:#af3a03\">.<\/span>limit_strategy <span style=\"color:#af3a03\">==<\/span> LimitStrategies<span style=\"color:#af3a03\">.<\/span>FIXED_WINDOW:\n<\/span><\/span><span style=\"display:flex;\"><span>                    limits<span style=\"color:#af3a03\">.<\/span>append(\n<\/span><\/span><span style=\"display:flex;\"><span>                        TokenBucket(\n<\/span><\/span><span style=\"display:flex;\"><span>                            storage_backend<span style=\"color:#af3a03\">=<\/span><span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>storage_engine,\n<\/span><\/span><span style=\"display:flex;\"><span>                            rule_descriptor<span style=\"color:#af3a03\">=<\/span>descriptor,\n<\/span><\/span><span style=\"display:flex;\"><span>                        )\n<\/span><\/span><span style=\"display:flex;\"><span>                    )\n<\/span><\/span><span style=\"display:flex;\"><span>                <span style=\"color:#af3a03\">elif<\/span> config<span style=\"color:#af3a03\">.<\/span>limit_strategy <span style=\"color:#af3a03\">==<\/span> LimitStrategies<span style=\"color:#af3a03\">.<\/span>SLIDING_WINDOW_LOG: limits<span style=\"color:#af3a03\">.<\/span>append(\n<\/span><\/span><span style=\"display:flex;\"><span>                        TokenBucket(\n<\/span><\/span><span style=\"display:flex;\"><span>                            storage_backend<span style=\"color:#af3a03\">=<\/span><span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>storage_engine,\n<\/span><\/span><span style=\"display:flex;\"><span>                            rule_descriptor<span style=\"color:#af3a03\">=<\/span>descriptor,\n<\/span><\/span><span style=\"display:flex;\"><span>                        )\n<\/span><\/span><span style=\"display:flex;\"><span>                    )\n<\/span><\/span><span style=\"display:flex;\"><span>                <span style=\"color:#af3a03\">elif<\/span> config<span style=\"color:#af3a03\">.<\/span>limit_strategy <span style=\"color:#af3a03\">==<\/span> LimitStrategies<span style=\"color:#af3a03\">.<\/span>SLIDING_WINDOW_COUNTER:\n<\/span><\/span><span style=\"display:flex;\"><span>                    limits<span style=\"color:#af3a03\">.<\/span>append(\n<\/span><\/span><span style=\"display:flex;\"><span>                        TokenBucket(\n<\/span><\/span><span style=\"display:flex;\"><span>                            storage_backend<span style=\"color:#af3a03\">=<\/span><span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>storage_engine,\n<\/span><\/span><span style=\"display:flex;\"><span>                            rule_descriptor<span style=\"color:#af3a03\">=<\/span>descriptor,\n<\/span><\/span><span style=\"display:flex;\"><span>                        )\n<\/span><\/span><span style=\"display:flex;\"><span>                    )\n<\/span><\/span><span style=\"display:flex;\"><span>                <span style=\"color:#af3a03\">else<\/span>:\n<\/span><\/span><span style=\"display:flex;\"><span>                    <span style=\"color:#af3a03\">raise<\/span> <span style=\"color:#fb4934\">ValueError<\/span>(\n<\/span><\/span><span style=\"display:flex;\"><span>                        <span style=\"color:#79740e\">f<\/span><span style=\"color:#79740e\">&#34;Limit strategy <\/span><span style=\"color:#79740e\">{<\/span>config<span style=\"color:#af3a03\">.<\/span>limit_strategy<span style=\"color:#79740e\">}<\/span><span style=\"color:#79740e\"> not supported&#34;<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>                    )<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<h3 class=\"heading\" id=\"fixed-window\">\n  Fixed Window\n  <a class=\"anchor\" href=\"#fixed-window\">#<\/a>\n<\/h3>\n<p>In the fixed window algorithm, we split the time into unit-size buckets.\nEach bucket has a specified capacity and can limit the requests once it&rsquo;s reached.<\/p>\n<p>For example, if our unit is 1 minute, our buckets would be 10:00, 10:01, and 10:02.<\/p>\n<p>Now how can we choose the hash key?\nA hash key like <code>path_1000_&lt;key&gt;_&lt;value&gt;<\/code> is good because\nit puts all requests from a specific entity to a path into the correct bucket.\nSo we can query this key and check the count to determine the request.<\/p>\n<p>But choosing the hour &amp; minute combination to add time to the key is not going to work,\nbecause there might be collisions when the day passes and we reach that time again.<\/p>\n<p>To overcome this problem, we can use <a href=\"https:\/\/www.unixtimestamp.com\/\" rel=\"noopener\" target=\"_blank\">timestmap<\/a>,\nsince each time second has a unique timestamp, we resolve the collision.<\/p>\n<p>Since the timestamp represents the seconds,\nwe can&rsquo;t create a bucket for minute intervals if we use this value directly in the cache.\nWhen the limiting unit is a minute, we need to find the value which\nis the same for every moment in a given minute.<\/p>\n<p>We can do this by dividing the timestamp by our unit:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-python\" data-lang=\"python\"><span style=\"display:flex;\"><span>current_interval <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#b57614\">str<\/span>(<span style=\"color:#b57614\">int<\/span>(datetime<span style=\"color:#af3a03\">.<\/span>now()<span style=\"color:#af3a03\">.<\/span>timestamp() <span style=\"color:#af3a03\">\/<\/span> <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>interval_len_sec))<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>this value will be the same for all moments in the interval.<\/p>\n<p>We can see that based on how this interval is calculated,\nour limiter limits the requests for the window 10:00:00 and 10:01:00.\nBut it does not check the window 10:00:30 and 10:01:30.\nThis is the problem that sliding window algorithm solves,\nby not making the window fixed.<\/p>\n<p>Now that we figured out the hard part let&rsquo;s look at the code:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-python\" data-lang=\"python\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">class<\/span> FixedWindow(AbstractStrategy):\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">__init__<\/span>(\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#b57614\">self<\/span>,\n<\/span><\/span><span style=\"display:flex;\"><span>        storage_backend: AbstractStorage,\n<\/span><\/span><span style=\"display:flex;\"><span>        rule_descriptor: Descriptor,\n<\/span><\/span><span style=\"display:flex;\"><span>    ):\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#b57614\">super<\/span>(FixedWindow, <span style=\"color:#b57614\">self<\/span>)<span style=\"color:#af3a03\">.<\/span><span style=\"color:#b57614\">__init__<\/span>(storage_backend, rule_descriptor)\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>interval_len_sec <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>rule_descriptor<span style=\"color:#af3a03\">.<\/span>unit<span style=\"color:#af3a03\">.<\/span>to_seconds()\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>interval_max <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>rule_descriptor<span style=\"color:#af3a03\">.<\/span>requests_per_unit\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">do_limit<\/span>(<span style=\"color:#b57614\">self<\/span>, request: Request):\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>request <span style=\"color:#af3a03\">=<\/span> request\n<\/span><\/span><span style=\"display:flex;\"><span>        counter_key <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>_get_counter_key()\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">if<\/span> counter_key <span style=\"color:#af3a03\">is<\/span> <span style=\"color:#af3a03\">None<\/span>:\n<\/span><\/span><span style=\"display:flex;\"><span>            <span style=\"color:#af3a03\">return<\/span> <span style=\"color:#af3a03\">False<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">if<\/span> <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>_window_max_reached(counter_key):\n<\/span><\/span><span style=\"display:flex;\"><span>            <span style=\"color:#af3a03\">return<\/span> <span style=\"color:#af3a03\">True<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">return<\/span> <span style=\"color:#af3a03\">False<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">_get_counter_key<\/span>(<span style=\"color:#b57614\">self<\/span>):\n<\/span><\/span><span style=\"display:flex;\"><span>        current_interval <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#b57614\">str<\/span>(<span style=\"color:#b57614\">int<\/span>(datetime<span style=\"color:#af3a03\">.<\/span>now()<span style=\"color:#af3a03\">.<\/span>timestamp() <span style=\"color:#af3a03\">\/<\/span> <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>interval_len_sec))\n<\/span><\/span><span style=\"display:flex;\"><span>        descriptor <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>rule_descriptor\n<\/span><\/span><span style=\"display:flex;\"><span>        path <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>request<span style=\"color:#af3a03\">.<\/span>path\n<\/span><\/span><span style=\"display:flex;\"><span>        key <span style=\"color:#af3a03\">=<\/span> descriptor<span style=\"color:#af3a03\">.<\/span>key\n<\/span><\/span><span style=\"display:flex;\"><span>        value <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>request<span style=\"color:#af3a03\">.<\/span>data[key]\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">if<\/span> descriptor<span style=\"color:#af3a03\">.<\/span>value <span style=\"color:#af3a03\">is<\/span> <span style=\"color:#af3a03\">not<\/span> <span style=\"color:#af3a03\">None<\/span> <span style=\"color:#af3a03\">and<\/span> value <span style=\"color:#af3a03\">!=<\/span> descriptor<span style=\"color:#af3a03\">.<\/span>value:\n<\/span><\/span><span style=\"display:flex;\"><span>            <span style=\"color:#af3a03\">return<\/span> <span style=\"color:#af3a03\">None<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">else<\/span>:\n<\/span><\/span><span style=\"display:flex;\"><span>            <span style=\"color:#af3a03\">return<\/span> path <span style=\"color:#af3a03\">+<\/span> current_interval <span style=\"color:#af3a03\">+<\/span> <span style=\"color:#79740e\">&#34;_&#34;<\/span> <span style=\"color:#af3a03\">+<\/span> key <span style=\"color:#af3a03\">+<\/span> <span style=\"color:#79740e\">&#34;_&#34;<\/span> <span style=\"color:#af3a03\">+<\/span> value\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">_window_max_reached<\/span>(<span style=\"color:#b57614\">self<\/span>, counter_key):\n<\/span><\/span><span style=\"display:flex;\"><span>        counter <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>storage_backend<span style=\"color:#af3a03\">.<\/span>get(counter_key)\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">if<\/span> counter <span style=\"color:#af3a03\">is<\/span> <span style=\"color:#af3a03\">None<\/span>:\n<\/span><\/span><span style=\"display:flex;\"><span>            <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>storage_backend<span style=\"color:#af3a03\">.<\/span>set(counter_key, <span style=\"color:#8f3f71\">1<\/span>, <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>interval_len_sec)\n<\/span><\/span><span style=\"display:flex;\"><span>            <span style=\"color:#af3a03\">return<\/span> <span style=\"color:#af3a03\">False<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">elif<\/span> counter <span style=\"color:#af3a03\">&gt;=<\/span> <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>interval_max:\n<\/span><\/span><span style=\"display:flex;\"><span>            <span style=\"color:#af3a03\">return<\/span> <span style=\"color:#af3a03\">True<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>        counter <span style=\"color:#af3a03\">+=<\/span> <span style=\"color:#8f3f71\">1<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>storage_backend<span style=\"color:#af3a03\">.<\/span>incr(counter_key)\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">return<\/span> <span style=\"color:#af3a03\">False<\/span><\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>Notice that here we are using the <code>incr<\/code> method from the storage.\nWe haven&rsquo;t implemented this functionality yet, but this is a good interface to add.<\/p>\n<p>Since other storages such as redis has support for increment it&rsquo;s better to use it,\nrather than get, increment and set the value approach.<\/p>\n<p>So we add new method to <code>AbstractStorage<\/code>:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-python\" data-lang=\"python\"><span style=\"display:flex;\"><span>@abstractmethod\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">incr<\/span>(<span style=\"color:#b57614\">self<\/span>, key):\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">raise<\/span> <span style=\"color:#fb4934\">NotImplementedError<\/span><\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>And implement it in the memory:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-python\" data-lang=\"python\"><span style=\"display:flex;\"><span>  <span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">incr<\/span>(<span style=\"color:#b57614\">self<\/span>, key: <span style=\"color:#b57614\">str<\/span>):\n<\/span><\/span><span style=\"display:flex;\"><span>      <span style=\"color:#af3a03\">if<\/span> key <span style=\"color:#af3a03\">in<\/span> <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>data:\n<\/span><\/span><span style=\"display:flex;\"><span>          <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>data[key] <span style=\"color:#af3a03\">+=<\/span> <span style=\"color:#8f3f71\">1<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>      <span style=\"color:#af3a03\">else<\/span>:\n<\/span><\/span><span style=\"display:flex;\"><span>          <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>data[key] <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#8f3f71\">1<\/span><\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<h4 class=\"heading\" id=\"test\">\n  Test\n  <a class=\"anchor\" href=\"#test\">#<\/a>\n<\/h4>\n<p>Testing this new strategy is so easy,\nsince all of our strategies have the same interface(input\/output) we can\nuse pytest to <a href=\"https:\/\/docs.pytest.org\/en\/6.2.x\/parametrize.html\" rel=\"noopener\" target=\"_blank\">parameterize<\/a>\nthe strategy that is being tested.<\/p>\n<p>Let&rsquo;s go back to the test we wrote for token bucket and rewrite it in this way:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-python\" data-lang=\"python\"><span style=\"display:flex;\"><span>@pytest.mark.parametrize(\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#79740e\">&#34;limit_strategy&#34;<\/span>,\n<\/span><\/span><span style=\"display:flex;\"><span>    [\n<\/span><\/span><span style=\"display:flex;\"><span>        TokenBucket,\n<\/span><\/span><span style=\"display:flex;\"><span>        FixedWindow,\n<\/span><\/span><span style=\"display:flex;\"><span>    ],\n<\/span><\/span><span style=\"display:flex;\"><span>I)\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">test_apply_limit_per_unit<\/span>(local_storage, limit_strategy):\n<\/span><\/span><span style=\"display:flex;\"><span>    rule_descriptor <span style=\"color:#af3a03\">=<\/span> Descriptor(\n<\/span><\/span><span style=\"display:flex;\"><span>        key<span style=\"color:#af3a03\">=<\/span><span style=\"color:#79740e\">&#34;user_id&#34;<\/span>,\n<\/span><\/span><span style=\"display:flex;\"><span>        requests_per_unit<span style=\"color:#af3a03\">=<\/span><span style=\"color:#8f3f71\">1<\/span>,\n<\/span><\/span><span style=\"display:flex;\"><span>        unit<span style=\"color:#af3a03\">=<\/span>Unit<span style=\"color:#af3a03\">.<\/span>SECOND,\n<\/span><\/span><span style=\"display:flex;\"><span>    )\n<\/span><\/span><span style=\"display:flex;\"><span>    token_bucket <span style=\"color:#af3a03\">=<\/span> limit_strategy(\n<\/span><\/span><span style=\"display:flex;\"><span>        storage_backend<span style=\"color:#af3a03\">=<\/span>local_storage,\n<\/span><\/span><span style=\"display:flex;\"><span>        rule_descriptor<span style=\"color:#af3a03\">=<\/span>rule_descriptor,\n<\/span><\/span><span style=\"display:flex;\"><span>    )\n<\/span><\/span><span style=\"display:flex;\"><span>    request <span style=\"color:#af3a03\">=<\/span> Request(path<span style=\"color:#af3a03\">=<\/span><span style=\"color:#79740e\">&#34;dd&#34;<\/span>, data<span style=\"color:#af3a03\">=<\/span>{<span style=\"color:#79740e\">&#34;user_id&#34;<\/span>: <span style=\"color:#79740e\">&#34;1&#34;<\/span>})\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">assert<\/span> token_bucket<span style=\"color:#af3a03\">.<\/span>do_limit(request) <span style=\"color:#af3a03\">is<\/span> <span style=\"color:#af3a03\">False<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">assert<\/span> token_bucket<span style=\"color:#af3a03\">.<\/span>do_limit(request) <span style=\"color:#af3a03\">is<\/span> <span style=\"color:#af3a03\">True<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    test_now <span style=\"color:#af3a03\">=<\/span> datetime<span style=\"color:#af3a03\">.<\/span>datetime<span style=\"color:#af3a03\">.<\/span>now() <span style=\"color:#af3a03\">+<\/span> datetime<span style=\"color:#af3a03\">.<\/span>timedelta(seconds<span style=\"color:#af3a03\">=<\/span><span style=\"color:#8f3f71\">3<\/span>)\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">with<\/span> freezegun<span style=\"color:#af3a03\">.<\/span>freeze_time(test_now):\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">assert<\/span> token_bucket<span style=\"color:#af3a03\">.<\/span>do_limit(request) <span style=\"color:#af3a03\">is<\/span> <span style=\"color:#af3a03\">False<\/span><\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>the testing strategy is now passed to this test and it only tests the behavior.<\/p>\n<p>Now we can rewrite the remaining tests as well:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-python\" data-lang=\"python\"><span style=\"display:flex;\"><span>@pytest.mark.parametrize(\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#79740e\">&#34;limit_strategy&#34;<\/span>,\n<\/span><\/span><span style=\"display:flex;\"><span>    [\n<\/span><\/span><span style=\"display:flex;\"><span>        TokenBucket,\n<\/span><\/span><span style=\"display:flex;\"><span>        FixedWindow,\n<\/span><\/span><span style=\"display:flex;\"><span>    ],\n<\/span><\/span><span style=\"display:flex;\"><span>I)\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">test_apply_limit_per_value<\/span>(local_storage, limit_strategy):\n<\/span><\/span><span style=\"display:flex;\"><span>    rule_descriptor <span style=\"color:#af3a03\">=<\/span> Descriptor(\n<\/span><\/span><span style=\"display:flex;\"><span>        key<span style=\"color:#af3a03\">=<\/span><span style=\"color:#79740e\">&#34;user_id&#34;<\/span>,\n<\/span><\/span><span style=\"display:flex;\"><span>        requests_per_unit<span style=\"color:#af3a03\">=<\/span><span style=\"color:#8f3f71\">1<\/span>,\n<\/span><\/span><span style=\"display:flex;\"><span>        unit<span style=\"color:#af3a03\">=<\/span>Unit<span style=\"color:#af3a03\">.<\/span>SECOND,\n<\/span><\/span><span style=\"display:flex;\"><span>    )\n<\/span><\/span><span style=\"display:flex;\"><span>    token_bucket <span style=\"color:#af3a03\">=<\/span> limit_strategy(\n<\/span><\/span><span style=\"display:flex;\"><span>        storage_backend<span style=\"color:#af3a03\">=<\/span>local_storage,\n<\/span><\/span><span style=\"display:flex;\"><span>        rule_descriptor<span style=\"color:#af3a03\">=<\/span>rule_descriptor,\n<\/span><\/span><span style=\"display:flex;\"><span>    )\n<\/span><\/span><span style=\"display:flex;\"><span>    user_1_request <span style=\"color:#af3a03\">=<\/span> Request(path<span style=\"color:#af3a03\">=<\/span><span style=\"color:#79740e\">&#34;dd&#34;<\/span>, data<span style=\"color:#af3a03\">=<\/span>{<span style=\"color:#79740e\">&#34;user_id&#34;<\/span>: <span style=\"color:#79740e\">&#34;1&#34;<\/span>})\n<\/span><\/span><span style=\"display:flex;\"><span>    user_2_request <span style=\"color:#af3a03\">=<\/span> Request(path<span style=\"color:#af3a03\">=<\/span><span style=\"color:#79740e\">&#34;dd&#34;<\/span>, data<span style=\"color:#af3a03\">=<\/span>{<span style=\"color:#79740e\">&#34;user_id&#34;<\/span>: <span style=\"color:#79740e\">&#34;2&#34;<\/span>})\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">assert<\/span> token_bucket<span style=\"color:#af3a03\">.<\/span>do_limit(user_1_request) <span style=\"color:#af3a03\">is<\/span> <span style=\"color:#af3a03\">False<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">assert<\/span> token_bucket<span style=\"color:#af3a03\">.<\/span>do_limit(user_2_request) <span style=\"color:#af3a03\">is<\/span> <span style=\"color:#af3a03\">False<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">assert<\/span> token_bucket<span style=\"color:#af3a03\">.<\/span>do_limit(user_1_request) <span style=\"color:#af3a03\">is<\/span> <span style=\"color:#af3a03\">True<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">assert<\/span> token_bucket<span style=\"color:#af3a03\">.<\/span>do_limit(user_2_request) <span style=\"color:#af3a03\">is<\/span> <span style=\"color:#af3a03\">True<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>@pytest.mark.parametrize(\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#79740e\">&#34;limit_strategy&#34;<\/span>,\n<\/span><\/span><span style=\"display:flex;\"><span>    [\n<\/span><\/span><span style=\"display:flex;\"><span>        TokenBucket,\n<\/span><\/span><span style=\"display:flex;\"><span>        FixedWindow,\n<\/span><\/span><span style=\"display:flex;\"><span>    ],\n<\/span><\/span><span style=\"display:flex;\"><span>I)\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">test_apply_limit_specific_value<\/span>(local_storage, limit_strategy):\n<\/span><\/span><span style=\"display:flex;\"><span>    rule_descriptor <span style=\"color:#af3a03\">=<\/span> Descriptor(\n<\/span><\/span><span style=\"display:flex;\"><span>        key<span style=\"color:#af3a03\">=<\/span><span style=\"color:#79740e\">&#34;user_id&#34;<\/span>,\n<\/span><\/span><span style=\"display:flex;\"><span>        value<span style=\"color:#af3a03\">=<\/span><span style=\"color:#79740e\">&#34;1&#34;<\/span>,\n<\/span><\/span><span style=\"display:flex;\"><span>        requests_per_unit<span style=\"color:#af3a03\">=<\/span><span style=\"color:#8f3f71\">1<\/span>,\n<\/span><\/span><span style=\"display:flex;\"><span>        unit<span style=\"color:#af3a03\">=<\/span>Unit<span style=\"color:#af3a03\">.<\/span>MINUTE,\n<\/span><\/span><span style=\"display:flex;\"><span>    )\n<\/span><\/span><span style=\"display:flex;\"><span>    token_bucket <span style=\"color:#af3a03\">=<\/span> limit_strategy(\n<\/span><\/span><span style=\"display:flex;\"><span>        storage_backend<span style=\"color:#af3a03\">=<\/span>local_storage,\n<\/span><\/span><span style=\"display:flex;\"><span>        rule_descriptor<span style=\"color:#af3a03\">=<\/span>rule_descriptor,\n<\/span><\/span><span style=\"display:flex;\"><span>    )\n<\/span><\/span><span style=\"display:flex;\"><span>    user_1_req <span style=\"color:#af3a03\">=<\/span> Request(path<span style=\"color:#af3a03\">=<\/span><span style=\"color:#79740e\">&#34;dd&#34;<\/span>, data<span style=\"color:#af3a03\">=<\/span>{<span style=\"color:#79740e\">&#34;user_id&#34;<\/span>: <span style=\"color:#79740e\">&#34;1&#34;<\/span>})\n<\/span><\/span><span style=\"display:flex;\"><span>    user_2_req <span style=\"color:#af3a03\">=<\/span> Request(path<span style=\"color:#af3a03\">=<\/span><span style=\"color:#79740e\">&#34;dd&#34;<\/span>, data<span style=\"color:#af3a03\">=<\/span>{<span style=\"color:#79740e\">&#34;user_id&#34;<\/span>: <span style=\"color:#79740e\">&#34;2&#34;<\/span>})\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">assert<\/span> token_bucket<span style=\"color:#af3a03\">.<\/span>do_limit(user_1_req) <span style=\"color:#af3a03\">is<\/span> <span style=\"color:#af3a03\">False<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">assert<\/span> token_bucket<span style=\"color:#af3a03\">.<\/span>do_limit(user_2_req) <span style=\"color:#af3a03\">is<\/span> <span style=\"color:#af3a03\">False<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">assert<\/span> token_bucket<span style=\"color:#af3a03\">.<\/span>do_limit(user_1_req) <span style=\"color:#af3a03\">is<\/span> <span style=\"color:#af3a03\">True<\/span><\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<h3 class=\"heading\" id=\"sliding-window-log\">\n  Sliding Window Log\n  <a class=\"anchor\" href=\"#sliding-window-log\">#<\/a>\n<\/h3>\n<p>As discussed earlier, the sliding window log does not take the time window fixed.\nImagine a request comes in at 10:00:30, instead of looking at request count in\nthe window 10:00 to 10:01\nwe check the number of requests in the window of 09:59:30 till that request.<\/p>\n<p>So the steps are:<\/p>\n<ol>\n<li>When a new request comes in save the timestamp into a list<\/li>\n<li>Count all the requests within the time unit of the arrived request<\/li>\n<li>If count more than allowed requests limit the request<\/li>\n<\/ol>\n<p>How this can be done?<\/p>\n<p>We need to save the timestamp when each requests comes in.\nThen when the next request comes we need to query all requests in the previous minute.<\/p>\n<p>Now the question is what data structure should be used here.\nWe need a data structure which can search and return all the values within a range.<\/p>\n<p>Redis provides <a href=\"https:\/\/redis.io\/docs\/latest\/develop\/data-types\/sorted-sets\/\" rel=\"noopener\" target=\"_blank\">sorted sets<\/a>\nwhich can provide an efficient way for finding a range of values in a list.<\/p>\n<p>Although sorted sets are\n<a href=\"https:\/\/github.com\/redis\/redis\/blob\/unstable\/src\/t_zset.c\" rel=\"noopener\" target=\"_blank\">implemented<\/a>\nwith hash map and <a href=\"https:\/\/brilliant.org\/wiki\/skip-lists\" rel=\"noopener\" target=\"_blank\">skip list<\/a>,\nwe are going to use a naive approach for implementing them in local memory storage.\nThis can be a good topic for another post.<\/p>\n<p>Let&rsquo;s start implementing the algorithm.<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-python\" data-lang=\"python\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">class<\/span> SlidingWindowLog(AbstractStrategy):\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">__init__<\/span>(\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#b57614\">self<\/span>,\n<\/span><\/span><span style=\"display:flex;\"><span>        storage_backend: AbstractStorage,\n<\/span><\/span><span style=\"display:flex;\"><span>        rule_descriptor: Descriptor,\n<\/span><\/span><span style=\"display:flex;\"><span>    ):\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#b57614\">super<\/span>(SlidingWindowLog, <span style=\"color:#b57614\">self<\/span>)<span style=\"color:#af3a03\">.<\/span><span style=\"color:#b57614\">__init__<\/span>(storage_backend, rule_descriptor)\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>interval_len_sec <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>rule_descriptor<span style=\"color:#af3a03\">.<\/span>unit<span style=\"color:#af3a03\">.<\/span>to_seconds()\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>interval_max <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>rule_descriptor<span style=\"color:#af3a03\">.<\/span>requests_per_unit\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">do_limit<\/span>(<span style=\"color:#b57614\">self<\/span>, request: Request):\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>request <span style=\"color:#af3a03\">=<\/span> request\n<\/span><\/span><span style=\"display:flex;\"><span>        request_logs_key <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>_get_request_logs_key()\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">if<\/span> request_logs_key <span style=\"color:#af3a03\">is<\/span> <span style=\"color:#af3a03\">None<\/span>:\n<\/span><\/span><span style=\"display:flex;\"><span>            <span style=\"color:#af3a03\">return<\/span> <span style=\"color:#af3a03\">False<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">if<\/span> <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>_window_max_reached(request_logs_key):\n<\/span><\/span><span style=\"display:flex;\"><span>            <span style=\"color:#af3a03\">return<\/span> <span style=\"color:#af3a03\">True<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">return<\/span> <span style=\"color:#af3a03\">False<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">_get_request_logs_key<\/span>(<span style=\"color:#b57614\">self<\/span>):\n<\/span><\/span><span style=\"display:flex;\"><span>        descriptor <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>rule_descriptor\n<\/span><\/span><span style=\"display:flex;\"><span>        path <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>request<span style=\"color:#af3a03\">.<\/span>path\n<\/span><\/span><span style=\"display:flex;\"><span>        key <span style=\"color:#af3a03\">=<\/span> descriptor<span style=\"color:#af3a03\">.<\/span>key\n<\/span><\/span><span style=\"display:flex;\"><span>        value <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>request<span style=\"color:#af3a03\">.<\/span>data[key]\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">if<\/span> descriptor<span style=\"color:#af3a03\">.<\/span>value <span style=\"color:#af3a03\">is<\/span> <span style=\"color:#af3a03\">not<\/span> <span style=\"color:#af3a03\">None<\/span> <span style=\"color:#af3a03\">and<\/span> value <span style=\"color:#af3a03\">!=<\/span> descriptor<span style=\"color:#af3a03\">.<\/span>value:\n<\/span><\/span><span style=\"display:flex;\"><span>            <span style=\"color:#af3a03\">return<\/span> <span style=\"color:#af3a03\">None<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">else<\/span>:\n<\/span><\/span><span style=\"display:flex;\"><span>            <span style=\"color:#af3a03\">return<\/span> path <span style=\"color:#af3a03\">+<\/span> <span style=\"color:#79740e\">&#34;_&#34;<\/span> <span style=\"color:#af3a03\">+<\/span> key <span style=\"color:#af3a03\">+<\/span> <span style=\"color:#79740e\">&#34;_&#34;<\/span> <span style=\"color:#af3a03\">+<\/span> value<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>First we get key of the list holding request logs.\nThen we check if current windows max request count is reached or not.<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-python\" data-lang=\"python\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">_window_max_reached<\/span>(<span style=\"color:#b57614\">self<\/span>, window_key):\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>storage_backend<span style=\"color:#af3a03\">.<\/span>sorted_set_remove(\n<\/span><\/span><span style=\"display:flex;\"><span>        window_key,\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#8f3f71\">0<\/span>,\n<\/span><\/span><span style=\"display:flex;\"><span>        datetime<span style=\"color:#af3a03\">.<\/span>now()<span style=\"color:#af3a03\">.<\/span>timestamp() <span style=\"color:#af3a03\">-<\/span> <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>interval_len_sec,\n<\/span><\/span><span style=\"display:flex;\"><span>    )\n<\/span><\/span><span style=\"display:flex;\"><span>    current_window_req_count <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>storage_backend<span style=\"color:#af3a03\">.<\/span>sorted_set_count(\n<\/span><\/span><span style=\"display:flex;\"><span>        window_key,\n<\/span><\/span><span style=\"display:flex;\"><span>        datetime<span style=\"color:#af3a03\">.<\/span>now()<span style=\"color:#af3a03\">.<\/span>timestamp() <span style=\"color:#af3a03\">-<\/span> <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>interval_len_sec,\n<\/span><\/span><span style=\"display:flex;\"><span>        datetime<span style=\"color:#af3a03\">.<\/span>now()<span style=\"color:#af3a03\">.<\/span>timestamp(),\n<\/span><\/span><span style=\"display:flex;\"><span>    )\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">if<\/span> current_window_req_count <span style=\"color:#af3a03\">is<\/span> <span style=\"color:#af3a03\">None<\/span>:\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>storage_backend<span style=\"color:#af3a03\">.<\/span>sorted_set_add(window_key, datetime<span style=\"color:#af3a03\">.<\/span>now()<span style=\"color:#af3a03\">.<\/span>timestamp())\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">return<\/span> <span style=\"color:#af3a03\">False<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">elif<\/span> current_window_req_count <span style=\"color:#af3a03\">&gt;=<\/span> <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>interval_max:\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">return<\/span> <span style=\"color:#af3a03\">True<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>storage_backend<span style=\"color:#af3a03\">.<\/span>sorted_set_add(window_key, datetime<span style=\"color:#af3a03\">.<\/span>now()<span style=\"color:#af3a03\">.<\/span>timestamp())\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">return<\/span> <span style=\"color:#af3a03\">False<\/span><\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>Before checking the request count,\nwe need to remove all the request that are not in the current window.<\/p>\n<p>Then we count the requests within the time unit and check if\nit&rsquo;s more than the allowed count for the interval.<\/p>\n<h4 class=\"heading\" id=\"testing\">\n  Testing\n  <a class=\"anchor\" href=\"#testing\">#<\/a>\n<\/h4>\n<p>Same as how we tested the previous strategy we\ncan add this new strategy to test parameters:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-python\" data-lang=\"python\"><span style=\"display:flex;\"><span>@pytest.mark.parametrize(\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#79740e\">&#34;limit_strategy&#34;<\/span>,\n<\/span><\/span><span style=\"display:flex;\"><span>    [\n<\/span><\/span><span style=\"display:flex;\"><span>        TokenBucket,\n<\/span><\/span><span style=\"display:flex;\"><span>        FixedWindow,\n<\/span><\/span><span style=\"display:flex;\"><span>        SlidingWindowLog,\n<\/span><\/span><span style=\"display:flex;\"><span>    ],\n<\/span><\/span><span style=\"display:flex;\"><span>)<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>we can add this strategy to previous tests,\nbut there&rsquo;s also a new behavior we can test for this strategy.\nSince the sliding window algorithm does not allow over-limit requests\nat the edge of the time unit (like between 01:50 and 02:10) we can add test it.<\/p>\n<p>So create a new test:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-python\" data-lang=\"python\"><span style=\"display:flex;\"><span>@pytest.mark.parametrize(\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#79740e\">&#34;limit_strategy&#34;<\/span>,\n<\/span><\/span><span style=\"display:flex;\"><span>    [\n<\/span><\/span><span style=\"display:flex;\"><span>        SlidingWindowLog,\n<\/span><\/span><span style=\"display:flex;\"><span>    ],\n<\/span><\/span><span style=\"display:flex;\"><span>)\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">test_sliding_window_does_not_allow_requests_in_unit_edges<\/span>(\n<\/span><\/span><span style=\"display:flex;\"><span>    local_storage, limit_strategy\n<\/span><\/span><span style=\"display:flex;\"><span>):\n<\/span><\/span><span style=\"display:flex;\"><span>    rule_descriptor <span style=\"color:#af3a03\">=<\/span> Descriptor(\n<\/span><\/span><span style=\"display:flex;\"><span>        key<span style=\"color:#af3a03\">=<\/span><span style=\"color:#79740e\">&#34;user_id&#34;<\/span>,\n<\/span><\/span><span style=\"display:flex;\"><span>        requests_per_unit<span style=\"color:#af3a03\">=<\/span><span style=\"color:#8f3f71\">2<\/span>,\n<\/span><\/span><span style=\"display:flex;\"><span>        unit<span style=\"color:#af3a03\">=<\/span>Unit<span style=\"color:#af3a03\">.<\/span>MINUTE,\n<\/span><\/span><span style=\"display:flex;\"><span>    )\n<\/span><\/span><span style=\"display:flex;\"><span>    sliding_window <span style=\"color:#af3a03\">=<\/span> limit_strategy(\n<\/span><\/span><span style=\"display:flex;\"><span>        storage_backend<span style=\"color:#af3a03\">=<\/span>local_storage,\n<\/span><\/span><span style=\"display:flex;\"><span>        rule_descriptor<span style=\"color:#af3a03\">=<\/span>rule_descriptor,\n<\/span><\/span><span style=\"display:flex;\"><span>    )\n<\/span><\/span><span style=\"display:flex;\"><span>    user_1_req <span style=\"color:#af3a03\">=<\/span> Request(path<span style=\"color:#af3a03\">=<\/span><span style=\"color:#79740e\">&#34;dd&#34;<\/span>, data<span style=\"color:#af3a03\">=<\/span>{<span style=\"color:#79740e\">&#34;user_id&#34;<\/span>: <span style=\"color:#79740e\">&#34;1&#34;<\/span>})\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    current_time <span style=\"color:#af3a03\">=<\/span> datetime<span style=\"color:#af3a03\">.<\/span>datetime<span style=\"color:#af3a03\">.<\/span>now()<span style=\"color:#af3a03\">.<\/span>replace(\n<\/span><\/span><span style=\"display:flex;\"><span>        hour<span style=\"color:#af3a03\">=<\/span><span style=\"color:#8f3f71\">0<\/span>, minute<span style=\"color:#af3a03\">=<\/span><span style=\"color:#8f3f71\">0<\/span>, second<span style=\"color:#af3a03\">=<\/span><span style=\"color:#8f3f71\">50<\/span>, microsecond<span style=\"color:#af3a03\">=<\/span><span style=\"color:#8f3f71\">0<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>    )\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">with<\/span> freezegun<span style=\"color:#af3a03\">.<\/span>freeze_time(current_time):\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">assert<\/span> sliding_window<span style=\"color:#af3a03\">.<\/span>do_limit(user_1_req) <span style=\"color:#af3a03\">is<\/span> <span style=\"color:#af3a03\">False<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    test_now <span style=\"color:#af3a03\">=<\/span> current_time <span style=\"color:#af3a03\">+<\/span> datetime<span style=\"color:#af3a03\">.<\/span>timedelta(seconds<span style=\"color:#af3a03\">=<\/span><span style=\"color:#8f3f71\">15<\/span>)\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">with<\/span> freezegun<span style=\"color:#af3a03\">.<\/span>freeze_time(test_now):\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">assert<\/span> sliding_window<span style=\"color:#af3a03\">.<\/span>do_limit(user_1_req) <span style=\"color:#af3a03\">is<\/span> <span style=\"color:#af3a03\">False<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">assert<\/span> sliding_window<span style=\"color:#af3a03\">.<\/span>do_limit(user_1_req) <span style=\"color:#af3a03\">is<\/span> <span style=\"color:#af3a03\">True<\/span><\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>Notice in our test we set the initial time to a time near ending of a minute,\nthen move the time forward to be in the next minute, previous algorithms wouldn&rsquo;t\nblock this.\nBut since the window is not fixed in this limiter it will block the third request,\neven if it&rsquo;s sent in the in the next minute. Nice improvement.<\/p>\n<h3 class=\"heading\" id=\"sliding-window-count\">\n  Sliding Window Count\n  <a class=\"anchor\" href=\"#sliding-window-count\">#<\/a>\n<\/h3>\n<p>The sliding window log solves the problem of allowing over-limit\nrequests at unit edges.<\/p>\n<p>But this algorithm uses more storage since it&rsquo;s storing full timestmap.<\/p>\n<p>To solve this problem, there is another approach to count requests in each window.\nWhen a request comes in calculate the current 1 minute window.\nThis window spans the current minute and probably the previous minute.\nWe can calculate what percentage of the rolling window is in previous window.\nThen we can use that percentage to assign a weight to previous window request count.<\/p>\n<p>so it would be\n<code>total_requests = previous_window_weight * previous_window_count + current_window_count<\/code>.<\/p>\n<p>For the implementation we use the previous way to create keys for each interval.<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-python\" data-lang=\"python\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">class<\/span> SlidingWindowCount(AbstractStrategy):\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">__init__<\/span>(\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#b57614\">self<\/span>,\n<\/span><\/span><span style=\"display:flex;\"><span>        storage_backend: AbstractStorage,\n<\/span><\/span><span style=\"display:flex;\"><span>        rule_descriptor: Descriptor,\n<\/span><\/span><span style=\"display:flex;\"><span>    ):\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#b57614\">super<\/span>(SlidingWindowCount, <span style=\"color:#b57614\">self<\/span>)<span style=\"color:#af3a03\">.<\/span><span style=\"color:#b57614\">__init__<\/span>(storage_backend, rule_descriptor)\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>interval_len_sec <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>rule_descriptor<span style=\"color:#af3a03\">.<\/span>unit<span style=\"color:#af3a03\">.<\/span>to_seconds()\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>interval_max <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>rule_descriptor<span style=\"color:#af3a03\">.<\/span>requests_per_unit\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">do_limit<\/span>(<span style=\"color:#b57614\">self<\/span>, request: Request):\n<\/span><\/span><span style=\"display:flex;\"><span>        current_interval <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#b57614\">str<\/span>(<span style=\"color:#b57614\">int<\/span>(datetime<span style=\"color:#af3a03\">.<\/span>now()<span style=\"color:#af3a03\">.<\/span>timestamp() <span style=\"color:#af3a03\">\/<\/span> <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>interval_len_sec))\n<\/span><\/span><span style=\"display:flex;\"><span>        prev_interval <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#b57614\">str<\/span>(<span style=\"color:#b57614\">int<\/span>(datetime<span style=\"color:#af3a03\">.<\/span>now()<span style=\"color:#af3a03\">.<\/span>timestamp() <span style=\"color:#af3a03\">\/<\/span> <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>interval_len_sec) <span style=\"color:#af3a03\">-<\/span> <span style=\"color:#8f3f71\">1<\/span>)\n<\/span><\/span><span style=\"display:flex;\"><span>        key <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>rule_descriptor<span style=\"color:#af3a03\">.<\/span>key\n<\/span><\/span><span style=\"display:flex;\"><span>        path <span style=\"color:#af3a03\">=<\/span> request<span style=\"color:#af3a03\">.<\/span>path\n<\/span><\/span><span style=\"display:flex;\"><span>        value <span style=\"color:#af3a03\">=<\/span> request<span style=\"color:#af3a03\">.<\/span>data[key]\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>        previous_interval_key <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>_get_counter_key(prev_interval, path, key, value)\n<\/span><\/span><span style=\"display:flex;\"><span>        current_interval_key <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>_get_counter_key(current_interval, path, key, value)\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">if<\/span> previous_interval_key <span style=\"color:#af3a03\">is<\/span> <span style=\"color:#af3a03\">None<\/span> <span style=\"color:#af3a03\">or<\/span> current_interval_key <span style=\"color:#af3a03\">is<\/span> <span style=\"color:#af3a03\">None<\/span>:\n<\/span><\/span><span style=\"display:flex;\"><span>            <span style=\"color:#af3a03\">return<\/span> <span style=\"color:#af3a03\">False<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>storage_backend<span style=\"color:#af3a03\">.<\/span>incr(current_interval_key)\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>        current_interval_counter <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>storage_backend<span style=\"color:#af3a03\">.<\/span>get(current_interval_key) <span style=\"color:#af3a03\">or<\/span> <span style=\"color:#8f3f71\">0<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>        previous_interval_counter <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>storage_backend<span style=\"color:#af3a03\">.<\/span>get(previous_interval_key) <span style=\"color:#af3a03\">or<\/span> <span style=\"color:#8f3f71\">0<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>        percent_of_previous_interval_overlap_current_window <span style=\"color:#af3a03\">=<\/span> (\n<\/span><\/span><span style=\"display:flex;\"><span>            <span style=\"color:#8f3f71\">1<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>            <span style=\"color:#af3a03\">-<\/span> (\n<\/span><\/span><span style=\"display:flex;\"><span>                <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>interval_len_sec\n<\/span><\/span><span style=\"display:flex;\"><span>                <span style=\"color:#af3a03\">-<\/span> datetime<span style=\"color:#af3a03\">.<\/span>now()<span style=\"color:#af3a03\">.<\/span>timestamp() <span style=\"color:#af3a03\">%<\/span> <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>interval_len_sec\n<\/span><\/span><span style=\"display:flex;\"><span>            )\n<\/span><\/span><span style=\"display:flex;\"><span>            <span style=\"color:#af3a03\">\/<\/span> <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>interval_len_sec\n<\/span><\/span><span style=\"display:flex;\"><span>        )\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>        total_requests <span style=\"color:#af3a03\">=<\/span> math<span style=\"color:#af3a03\">.<\/span>ceil(\n<\/span><\/span><span style=\"display:flex;\"><span>            previous_interval_counter\n<\/span><\/span><span style=\"display:flex;\"><span>            <span style=\"color:#af3a03\">*<\/span> percent_of_previous_interval_overlap_current_window\n<\/span><\/span><span style=\"display:flex;\"><span>            <span style=\"color:#af3a03\">+<\/span> current_interval_counter\n<\/span><\/span><span style=\"display:flex;\"><span>        )\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">if<\/span> total_requests <span style=\"color:#af3a03\">&gt;<\/span> <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>interval_max:\n<\/span><\/span><span style=\"display:flex;\"><span>            <span style=\"color:#af3a03\">return<\/span> <span style=\"color:#af3a03\">True<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">return<\/span> <span style=\"color:#af3a03\">False<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">_get_counter_key<\/span>(<span style=\"color:#b57614\">self<\/span>, interval, path, key, value):\n<\/span><\/span><span style=\"display:flex;\"><span>        descriptor <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>rule_descriptor\n<\/span><\/span><span style=\"display:flex;\"><span>        key <span style=\"color:#af3a03\">=<\/span> descriptor<span style=\"color:#af3a03\">.<\/span>key\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">if<\/span> descriptor<span style=\"color:#af3a03\">.<\/span>value <span style=\"color:#af3a03\">is<\/span> <span style=\"color:#af3a03\">not<\/span> <span style=\"color:#af3a03\">None<\/span> <span style=\"color:#af3a03\">and<\/span> value <span style=\"color:#af3a03\">!=<\/span> descriptor<span style=\"color:#af3a03\">.<\/span>value:\n<\/span><\/span><span style=\"display:flex;\"><span>            <span style=\"color:#af3a03\">return<\/span> <span style=\"color:#af3a03\">None<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">else<\/span>:\n<\/span><\/span><span style=\"display:flex;\"><span>            <span style=\"color:#af3a03\">return<\/span> path <span style=\"color:#af3a03\">+<\/span> interval <span style=\"color:#af3a03\">+<\/span> <span style=\"color:#79740e\">&#34;_&#34;<\/span> <span style=\"color:#af3a03\">+<\/span> key <span style=\"color:#af3a03\">+<\/span> <span style=\"color:#79740e\">&#34;_&#34;<\/span> <span style=\"color:#af3a03\">+<\/span> value<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>Most of the code is similar to the sliding window log, except that we use\nboth previous and current interval keys to count the requests.\nThe mysterious formula <code>datetime.now().timestamp() % self.interval_len_sec<\/code>\nalways outputs\nthe number of seconds remaining until the end of interval and diving this by\nthe interval\nlength will give us the percentage of the current window passed. Subtracting\nthis from 1\nwill give how much of the sliding window is in the past interval to calculate\nthe weight.<\/p>\n<p>Also since our calculation can result in a floating point number we can round it\nup or down. Rounding up is chosen in this case.<\/p>\n<h4 class=\"heading\" id=\"tests\">\n  Tests\n  <a class=\"anchor\" href=\"#tests\">#<\/a>\n<\/h4>\n<p>Since this is another implementation for the sliding window, we can add it as a parameter\nto all previous tests and the sliding window test.<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-python\" data-lang=\"python\"><span style=\"display:flex;\"><span>@pytest.mark.parametrize(\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#79740e\">&#34;limit_strategy&#34;<\/span>,\n<\/span><\/span><span style=\"display:flex;\"><span>    [\n<\/span><\/span><span style=\"display:flex;\"><span>        SlidingWindowLog,\n<\/span><\/span><span style=\"display:flex;\"><span>        SlidingWindowCount,\n<\/span><\/span><span style=\"display:flex;\"><span>    ],\n<\/span><\/span><span style=\"display:flex;\"><span>)\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">test_sliding_window_does_not_allow_requests_in_unit_edges<\/span><\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>And finally running all the tests, results in 18 tests for all of our strategies\nwith very minimal test code.\nIt&rsquo;s always good to write less code cause less code is better.<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-python\" data-lang=\"python\"><span style=\"display:flex;\"><span>tests<span style=\"color:#af3a03\">\/<\/span>limit_strategy<span style=\"color:#af3a03\">\/<\/span>test_limit_strategy<span style=\"color:#af3a03\">.<\/span>py <span style=\"color:#af3a03\">..............<\/span>                                                                         [ <span style=\"color:#8f3f71\">77<\/span><span style=\"color:#af3a03\">%<\/span>]\n<\/span><\/span><span style=\"display:flex;\"><span>tests<span style=\"color:#af3a03\">\/<\/span>service<span style=\"color:#af3a03\">\/<\/span>test_limiter<span style=\"color:#af3a03\">.<\/span>py <span style=\"color:#af3a03\">....<\/span>                                                                                                 [<span style=\"color:#8f3f71\">100<\/span><span style=\"color:#af3a03\">%<\/span>]\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">===========================================================<\/span> <span style=\"color:#8f3f71\">18<\/span> passed <span style=\"color:#af3a03\">in<\/span> <span style=\"color:#8f3f71\">0.20<\/span>s <span style=\"color:#af3a03\">===========================================================<\/span><\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<h2 class=\"heading\" id=\"conclusion\">\n  Conclusion\n  <a class=\"anchor\" href=\"#conclusion\">#<\/a>\n<\/h2>\n<p>We now have implemented all different algorithms for our rate limiter.\nThe true power of our abstractions are shown in the less code we have to\nwrite for each limiter, we can test them all with universal test cases,\nthe rate limiter service can use them without knowing what the underlying strategy\nis.<\/p>\n<p>In the next part we can see how to implement another storage backend such as redis,\nwithout having to change any code in rate limiting algorithms.<\/p>\n"},{"title":"Personalize Macos Environment for Your Productivity","link":"https:\/\/glyphack.com\/macos-productivity\/","pubDate":"Sun, 19 Feb 2023 20:58:33 +0100","guid":"https:\/\/glyphack.com\/macos-productivity\/","description":"<p>MacOS is already a polished environment and unlike some other OSes it works out of the box.\nStill, spending investing time to personalize your tools and environment is a smart move.<\/p>\n<p>In the past years using MacOS I found simple tools that helps to make MacOS more ergonomic and fun.<\/p>\n<p>Most of the content here is in <a href=\"https:\/\/github.com\/Glyphack\/dotfiles\" rel=\"noopener\" target=\"_blank\">my dotfiles<\/a>.<\/p>\n<h2 class=\"heading\" id=\"why\">\n  Why?\n  <a class=\"anchor\" href=\"#why\">#<\/a>\n<\/h2>\n<p>If you&rsquo;re already sold to this idea go ahead and start,\notherwise keep reading so I can convince you why you might want to start\nconfiguring your tools.<\/p>\n<p>I first heard the term\n<a href=\"https:\/\/www.youtube.com\/watch?v=QMVIJhC9Veg\" rel=\"noopener\" target=\"_blank\">Personalized Development Environment<\/a>\nin this video about configuring text editors to your liking and the idea stuck with me since then.<\/p>\n<p>I think this approach works well with other tools too.\nThat&rsquo;s exactly people learn touch typing.\nBeing able to use keyboard with the least amount of effort is important.<\/p>\n<p>You can go ahead with default configurations that come out of the box but those\nare not built for you, they are for everyone.<\/p>\n<p>I don&rsquo;t think automation is not primarily here to save your time only.\nAutomation is here to make it easier. Let me give you an example.\nImagine you have to constantly switch between editor and web browser constantly for some task.\nDoing this with the default tools is annoying, requires a lot of keys. But it can be simpler.\nDefault tool for this is to either use the mouse, or ALT+Tab every time to move between windows.\nSpecialized tools can make this simpler by setting a shortcut for each of the windows. Making the toggle just one keybinding.<\/p>\n<h2 class=\"heading\" id=\"remapping-keys\">\n  Remapping Keys\n  <a class=\"anchor\" href=\"#remapping-keys\">#<\/a>\n<\/h2>\n<p><a href=\"https:\/\/karabiner-elements.pqrs.org\/\" rel=\"noopener\" target=\"_blank\">Karabiner<\/a> can be used to remap keyboard.<\/p>\n<p>Try remapping the keys that you don&rsquo;t use often to things that you miss on the\nkeyboard, some examples are:<\/p>\n<ul>\n<li>Caps lock: key can be remapped to <code>Esc<\/code> key when pressed and Hyper Key when held<\/li>\n<li><code>\u00a7<\/code>: which I don&rsquo;t know why is it here in the first place can be mapped to &ldquo;`&rdquo;<\/li>\n<\/ul>\n<p>There are more advanced keybindings that can be done I have <a href=\"https:\/\/glyphack.com\/better-keyboard\/\" rel=\"noopener\" target=\"_blank\">created keybindings<\/a> to write symbols like <code>*<\/code>, <code>-<\/code>, etc. without reaching for shift key and a number.\nThis helped a lot with keeping my hands near the home row and reducing the work my pinky finger has to do.<\/p>\n<h2 class=\"heading\" id=\"text-expanding\">\n  Text Expanding\n  <a class=\"anchor\" href=\"#text-expanding\">#<\/a>\n<\/h2>\n<p>Text expanding is writing a small text and then it expands to a bigger text.<\/p>\n<p>For example instead of typing your mail every time you,\ncan only write <code>:em<\/code> and it expands to your email address.<\/p>\n<p><a href=\"https:\/\/espanso.org\/\" rel=\"noopener\" target=\"_blank\">Espanso<\/a> is the tool I use for this.<\/p>\n<p>Some examples I have are:<\/p>\n<ul>\n<li><code>:date<\/code> to current date like 19\/02\/2023<\/li>\n<\/ul>\n<h2 class=\"heading\" id=\"raycast\">\n  Raycast\n  <a class=\"anchor\" href=\"#raycast\">#<\/a>\n<\/h2>\n<p><a href=\"https:\/\/www.raycast.com\" rel=\"noopener\" target=\"_blank\">Raycast<\/a> is the single best application I have in this list.<\/p>\n<p>What does it do?<\/p>\n<p>It gives a launch bar(like Spotlight) that can open applications,\nfind files and perform actions. I suggest reading through their\n<a href=\"https:\/\/manual.raycast.com\/\" rel=\"noopener\" target=\"_blank\">manual<\/a> to understand all it can do.\nWith Raycast you can integrate the stuff you need while reading\/coding to quickly\npull them off without leaving your current work.<\/p>\n<p>Raycast is easy to extend yourself, every time you find yourself doing something\nover an over or need to open something regularly, take a look at it&rsquo;s\n<a href=\"https:\/\/www.raycast.com\/store\" rel=\"noopener\" target=\"_blank\">store<\/a>\nto check if there&rsquo;s a solution.<\/p>\n<p>For example these are some of the things I&rsquo;m using:\n\n\n\n\n\n\n\n\n<figure class=\"\">\n\n    \n    \n      \n        \n        <div class=\"img-container\" style=\"--w: 1980; --h: 1186;\">\n            <img loading=\"lazy\" alt=\"My Raycast\" src=\"https:\/\/glyphack.com\/macos-productivity\/my-raycast_hu_78a55e76b2d2f55c.png\" width=\"1980\" height=\"1186\">\n        <\/div>\n      \n    \n\n    \n<\/figure>\n<\/p>\n<p><a href=\"https:\/\/www.raycast.com\/raycast\/browser-bookmarks\" rel=\"noopener\" target=\"_blank\">Search bookmarks<\/a>\nStart bookmarking any page you need to visit frequently, for example\nhomepages for your projects.<\/p>\n<p>The reason this is handy is that, first you don&rsquo;t need to open browser to search\nand through the history to open frequently visited pages also you don&rsquo;t have to\nnavigate through the pages to get to where you want.\nImagine navigating through Confluence to update some page you have to do everyday.\nThese days I just bookmark things I want to visit again and don&rsquo;t bother with organizing them.<\/p>\n<p>Clipboard history &amp; edit for traveling trough clipboard and change the content.<\/p>\n<p><a href=\"https:\/\/www.raycast.com\/raycast\/github\" rel=\"noopener\" target=\"_blank\">Github<\/a>\nextension is also useful to check notifications.\n\n\n\n\n\n\n\n\n<figure class=\"\">\n\n    \n    \n      \n        \n        <div class=\"img-container\" style=\"--w: 1426; --h: 656;\">\n            <img loading=\"lazy\" alt=\"Raycast Github\" src=\"https:\/\/glyphack.com\/macos-productivity\/raycast-github-pr_hu_1c91df4383a37d9c.png\" width=\"1426\" height=\"656\">\n        <\/div>\n      \n    \n\n    \n<\/figure>\n<\/p>\n<h2 class=\"heading\" id=\"hammerspoon\">\n  Hammerspoon\n  <a class=\"anchor\" href=\"#hammerspoon\">#<\/a>\n<\/h2>\n<p>This one&rsquo;s the most powerful tool, it&rsquo;s a bridge between MacOS and Lua. You can\n<a href=\"https:\/\/www.hammerspoon.org\/docs\/index.html\" rel=\"noopener\" target=\"_blank\">customize anything<\/a>\nwith it&rsquo;s builtin libraries called spoons.<\/p>\n<p>You can take a look at\n<a href=\"https:\/\/github.com\/Glyphack\/dotfiles\/blob\/master\/hammerspoon\/init.lua\" rel=\"noopener\" target=\"_blank\">my config<\/a>\nfor inspirations.<\/p>\n<p>One useful feature if you do multiple meetings per day(which you probably do)\nis it have a global shortcut to mute\/unmute your mic to don&rsquo;t annoy others with\nnoise in the meeting and quickly unmute. I&rsquo;m doing this with <a href=\"https:\/\/github.com\/cmaahs\/global-mute-spoon\" rel=\"noopener\" target=\"_blank\">global mute spoon<\/a>.<\/p>\n<p>Here&rsquo;s how the configuration looks like<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-lua\" data-lang=\"lua\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">local<\/span> GlobalMute <span style=\"color:#af3a03\">=<\/span> hs.loadSpoon(<span style=\"color:#79740e\">&#34;GlobalMute&#34;<\/span>)\n<\/span><\/span><span style=\"display:flex;\"><span>GlobalMute:bindHotkeys({\n<\/span><\/span><span style=\"display:flex;\"><span>    toggle <span style=\"color:#af3a03\">=<\/span> { hyper, <span style=\"color:#79740e\">&#34;t&#34;<\/span> }\n<\/span><\/span><span style=\"display:flex;\"><span>})<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>For example I have a microphone that is both input and output device.\nWhen I connect this microphone I don&rsquo;t want to set it as my output device but that&rsquo;s what mac does by default.\nIn Hammerspoon I can setup a callback to set my input\/output device when the microphone is connected.<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-lua\" data-lang=\"lua\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">local<\/span> <span style=\"color:#af3a03\">function<\/span> <span style=\"color:#b57614\">audiodeviceCallback<\/span>()\n<\/span><\/span><span style=\"display:flex;\"><span>    current <span style=\"color:#af3a03\">=<\/span> hs.audiodevice.defaultInputDevice():name()\n<\/span><\/span><span style=\"display:flex;\"><span>    print(<span style=\"color:#79740e\">&#34;Current device: &#34;<\/span> <span style=\"color:#af3a03\">..<\/span> current)\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">if<\/span> current <span style=\"color:#af3a03\">==<\/span> <span style=\"color:#79740e\">&#34;External Microphone&#34;<\/span> <span style=\"color:#af3a03\">then<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>      print(<span style=\"color:#79740e\">&#34;Forcing default output to Internal Speakers&#34;<\/span>)\n<\/span><\/span><span style=\"display:flex;\"><span>      hs.audiodevice.findOutputByName(<span style=\"color:#79740e\">&#34;MacBook Pro Speakers&#34;<\/span>):setDefaultOutputDevice()\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">end<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">end<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>hs.audiodevice.watcher.setCallback(audiodeviceCallback)\n<\/span><\/span><span style=\"display:flex;\"><span>hs.audiodevice.watcher.start()<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<h2 class=\"heading\" id=\"switching-windows\">\n  Switching Windows\n  <a class=\"anchor\" href=\"#switching-windows\">#<\/a>\n<\/h2>\n<p>Another Raycast feature is setting keybinding to open application windows.\nThis is useful when you want to have one application on the screen.\nFor example for me it&rsquo;s terminal and browser.\n\n\n\n\n\n\n\n\n<figure class=\"\">\n\n    \n    \n      \n        \n        <div class=\"img-container\" style=\"--w: 1360; --h: 408;\">\n            <img loading=\"lazy\" alt=\"Raycast Apps\" src=\"https:\/\/glyphack.com\/macos-productivity\/raycast-apps_hu_41c8d63890b87e9f.png\" width=\"1360\" height=\"408\">\n        <\/div>\n      \n    \n\n    \n<\/figure>\n<\/p>\n<p>These days I&rsquo;m using <a href=\"https:\/\/glyphack.com\/better-keyboard\/\" rel=\"noopener\" target=\"_blank\">a new solution<\/a> based on Hammerspoon for this.<\/p>\n<h2 class=\"heading\" id=\"terminal\">\n  Terminal\n  <a class=\"anchor\" href=\"#terminal\">#<\/a>\n<\/h2>\n<p>If you made it through here you might as well be a CLI user.\nThis post is not about terminal as it&rsquo;s not related to MacOS but here are a few tips.<\/p>\n<p>Use a fuzzy finder like <a href=\"https:\/\/github.com\/junegunn\/fzf\" rel=\"noopener\" target=\"_blank\">fzf<\/a> for searching directories and history.\nI have the following Fish keybinding to do a fuzzy search in my home directory and select a folder I want to <code>cd<\/code> into.<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-fish\" data-lang=\"fish\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">set<\/span> <span style=\"color:#79740e;font-weight:bold\">-x<\/span> FZF_ALT_C_COMMAND <span style=\"color:#79740e\">&#34;fd -t d . <\/span>$PROGRAMMING_DIR<span style=\"color:#79740e\"> -d 3&#34;<\/span><\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<h3 class=\"heading\" id=\"configure-git\">\n  Configure Git\n  <a class=\"anchor\" href=\"#configure-git\">#<\/a>\n<\/h3>\n<p>A good one to start can be git,\nyou can setup <a href=\"https:\/\/github.com\/Glyphack\/dotfiles\/blob\/master\/gitconf\/.gitignore_global\" rel=\"noopener\" target=\"_blank\">global ignore file<\/a> to ignore files for your specific environment in every project. So you don&rsquo;t need to add your <code>.idea<\/code> folder to every project.\n<a href=\"https:\/\/github.com\/Glyphack\/dotfiles\/blob\/master\/gitconf\/.gitconfig\" rel=\"noopener\" target=\"_blank\">Aliases<\/a> are useful for making shorter commands.<\/p>\n<p>Git work trees are a perfect solution if you want to have access to multiple branches at the same time.\nFor example having the master branch and a feature branch allows you to run benchmarks on both at the same time.<\/p>\n<h3 class=\"heading\" id=\"neovim\">\n  Neovim\n  <a class=\"anchor\" href=\"#neovim\">#<\/a>\n<\/h3>\n<p>There are a lot of guides to configure and work with Neovim,\nI suggest <a href=\"https:\/\/www.youtube.com\/watch?v=w7i4amO_zaE\" rel=\"noopener\" target=\"_blank\">this<\/a>\nand\n<a href=\"https:\/\/www.youtube.com\/watch?v=stqUbv-5u2s\" rel=\"noopener\" target=\"_blank\">this<\/a>.\nYou can also find the plugins\/tools I use or think is interesting to check in my\n<a href=\"https:\/\/github.com\/stars\/Glyphack\/lists\/tools\" rel=\"noopener\" target=\"_blank\">stars list<\/a>.<\/p>\n<h3 class=\"heading\" id=\"learn-your-code-editor\">\n  Learn Your Code Editor\n  <a class=\"anchor\" href=\"#learn-your-code-editor\">#<\/a>\n<\/h3>\n<p>Even if you don&rsquo;t use something like Neovim,\nyour editor supports a lot of customization, and should be customized.\nWhen you find some action requires a lot of effort, try to customize it in your Editor.<\/p>\n<p>For example, most of the time I stage part of the file I&rsquo;m editing for commits.\nIn VSCode you need to open the version control panel,\nscroll through the file and right click to select stage selected.\nThis is super hard if you have to do it 20 times in a productive day.\nI have a config in my vim to directly stage hunks in my editor to commit,\nbut I&rsquo;m sure you can find a VSCode equivalent.<\/p>\n"},{"title":"Rate Limiter From Scratch in Python Part 1","link":"https:\/\/glyphack.com\/rate-limiter-python-1\/","pubDate":"Tue, 14 Feb 2023 23:19:18 +0100","guid":"https:\/\/glyphack.com\/rate-limiter-python-1\/","description":"<h2 class=\"heading\" id=\"introduction\">\n  Introduction\n  <a class=\"anchor\" href=\"#introduction\">#<\/a>\n<\/h2>\n<p>After reading <a href=\"https:\/\/bytebytego.com\/\" rel=\"noopener\" target=\"_blank\">ByteByteGo course<\/a>\nmotivated me to write a rate limiter.\nSo I decided to do it in <a href=\"https:\/\/aosabook.org\/en\/500L\/introduction.html\" rel=\"noopener\" target=\"_blank\">500lines<\/a>\ntheme.<\/p>\n<p>We will focus on how to create the interfaces and components,\nto allow extensibility in the predicted ways.<\/p>\n<p>You can find the complete source code <a href=\"https:\/\/github.com\/Glyphack\/hera-limit\" rel=\"noopener\" target=\"_blank\">here<\/a>.<\/p>\n<h2 class=\"heading\" id=\"what-is-a-rate-limiter\">\n  What is a Rate Limiter?\n  <a class=\"anchor\" href=\"#what-is-a-rate-limiter\">#<\/a>\n<\/h2>\n<p>Well, there are a lot of\n<a href=\"https:\/\/www.cloudflare.com\/en-gb\/learning\/bots\/what-is-rate-limiting\/\" rel=\"noopener\" target=\"_blank\">great explanations<\/a>\non what is a rate limiter, but I&rsquo;ll give a minimal introduction to it for this post.<\/p>\n<p>A rate limiter is software that limits how many times\nsomeone can repeat an action in your software. Take Twitter as an example;\nthey need to specify how often someone can send a tweet per minute; otherwise,\none person can create 1 million tweets in a second and fill up all their server resources.<\/p>\n<h2 class=\"heading\" id=\"high-level-design\">\n  High Level Design\n  <a class=\"anchor\" href=\"#high-level-design\">#<\/a>\n<\/h2>\n<p>First off, what should our rate limiter do?\nThe rate limiter should be a function that takes in a request,\ndecides if the request can go through or not based on the current statistics.<\/p>\n<p>We are going to implement the following features:<\/p>\n<ol>\n<li>Rule-based rate limiting: let the user define rules\n<ul>\n<li>with a simpler version (without nested rules) of <a href=\"https:\/\/github.com\/envoyproxy\/ratelimit#configuration\" rel=\"noopener\" target=\"_blank\">envoy rate limit config<\/a>.<\/li>\n<\/ul>\n<\/li>\n<li>Support both local memory and Redis as storage backend<\/li>\n<li>Support the following rate-limit algorithms:<\/li>\n<li>Token bucket<\/li>\n<li>Fixed window<\/li>\n<li>Sliding window log<\/li>\n<li>Sliding window counter<\/li>\n<li>Distributed deployment model:\n<ul>\n<li>deploying multiple instances with consistent and eventual consistency models.<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n<p>What are the components of our system?<\/p>\n<p>\n\n\n\n\n\n\n\n<figure class=\"\">\n\n    \n    \n      \n        \n        <div class=\"img-container\" style=\"--w: 1113; --h: 747;\">\n            <img loading=\"lazy\" alt=\"System Components\" src=\"https:\/\/glyphack.com\/rate-limiter-python-1\/rate-limiter-components_hu_279cb54092b42dc9.png\" width=\"1113\" height=\"747\">\n        <\/div>\n      \n    \n\n    \n<\/figure>\n<\/p>\n<p>Breaking down the components:<\/p>\n<p><strong>Rules storage<\/strong>:\nit is responsible for loading rules and providing them to the rate limiter.<\/p>\n<p>Separating this component allows the service independent of how the rules\nshould be loaded into the service.\nWe want to start the application with a set of rules saved on disk,\nbut it&rsquo;s helpful to be able to add\/remove rules from an API endpoint\nwhile the application is running.<\/p>\n<p>Although we will not implement that part in this guide, separating this is good\nfor easier testing and future extensibility.<\/p>\n<p><strong>Storage<\/strong> :\nThis component is responsible for holding data used by rate-limiting algorithms.<\/p>\n<p>Creating an interface for storage is helpful because\nwe can ignore the underlying storage implementation in rate limit algorithms.\nAllowing us to use multiple storage backends such as Redis or local memory\nwithout touching the rate limit algorithm code.\nWe can implement operations <code>exists<\/code>, <code>get_value<\/code>, <code>set<\/code>, <code>incr<\/code> for the above algorithms.<\/p>\n<p>Note that we assume that our data store for this software is a key\/value store.\nWe are not creating a general store that every application can use.\nThis data store helps with writing a more straightforward interface\nand implementation for storage.\nFor example, we don&rsquo;t require to support where\/filtering clause.<\/p>\n<p><a href=\"https:\/\/www.youtube.com\/watch?v=tKbV6BpH-C8\" rel=\"noopener\" target=\"_blank\">This video<\/a>\ngives a nice explanation of why sometimes we must do this.<\/p>\n\u2705  In the above abstraction, we are not creating a generic storage but only a key-value store. This abstraction limits the extensibility of the code but makes the work easier. Which makes it a good choice for rate limiter problem scope. Always be careful when creating a very generic abstraction.\n\n<p><strong>Limit Strategy<\/strong>:\nThis component implements the rate limit algorithms without knowing the\nunderlying storage or API implementation.<\/p>\n<p>It should take in storage and a request and provide a result whether it&rsquo;s limited.<\/p>\n<p>The request details are essential to decide whether to limit,\nbut we only need the request data, IP, and path.\nSo it&rsquo;s better only to take in these values and\nnot depend on a particular request type.\nThen in the future, we can write adapters to convert a gRPC request to this\nfunction&rsquo;s input format.<\/p>\n<p>Service: takes care of orchestrating all the components.\nUpon startup, it loads all the rules into memory and creates a list of limit strategies to check.\nThe flow of handling requests:<\/p>\n<ol>\n<li>Run all the rules that apply to the request path<\/li>\n<li>Rules answer whether to allow or deny the request.<\/li>\n<\/ol>\n<p><strong>API<\/strong>: This layer is an interface for other programs to call the rate limiter.<\/p>\n<p>For example, this part can be exposed to the API gateway. We will not implement the API, but we will implement the logic to rate limit requests. Separating the API and rate limit service is helpful as we can expose different interfaces to integrate with other tools, for example:<\/p>\n<ul>\n<li>Importing the rate limiter directly into the app<\/li>\n<li>Making it available as a <a href=\"https:\/\/docs.konghq.com\/gateway\/latest\/plugin-development\/\" rel=\"noopener\" target=\"_blank\">Kong plugin<\/a><\/li>\n<\/ul>\n<p>The rule structure is part of the application interface. Users can define them to rate limit the services,\nand just like an API, we don&rsquo;t change them much.<\/p>\n<p>On the other hand, the LimitStrategy and Storage can be swapped and replaced with different implementations.\nSo a rule can stay the same while the limiter enforcing the rule can be changed to relax the rule or make it more strict.<\/p>\n<h2 class=\"heading\" id=\"implementation\">\n  Implementation\n  <a class=\"anchor\" href=\"#implementation\">#<\/a>\n<\/h2>\n<h3 class=\"heading\" id=\"defining-interfaces\">\n  Defining Interfaces\n  <a class=\"anchor\" href=\"#defining-interfaces\">#<\/a>\n<\/h3>\n<p>Based on the above rule structure we can use the following structure:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-python\" data-lang=\"python\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">from<\/span> <span style=\"color:#79740e\">dataclasses<\/span> <span style=\"color:#af3a03\">import<\/span> dataclass\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">from<\/span> <span style=\"color:#79740e\">enum<\/span> <span style=\"color:#af3a03\">import<\/span> Enum\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">from<\/span> <span style=\"color:#79740e\">typing<\/span> <span style=\"color:#af3a03\">import<\/span> List, Optional\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">class<\/span> Unit(<span style=\"color:#b57614\">str<\/span>, Enum):\n<\/span><\/span><span style=\"display:flex;\"><span>    SECOND <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#79740e\">&#34;second&#34;<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>    MINUTE <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#79740e\">&#34;minute&#34;<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>    HOUR <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#79740e\">&#34;hour&#34;<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">to_seconds<\/span>(<span style=\"color:#b57614\">self<\/span>) <span style=\"color:#af3a03\">-&gt;<\/span> <span style=\"color:#b57614\">int<\/span>:\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">if<\/span> <span style=\"color:#b57614\">self<\/span> <span style=\"color:#af3a03\">==<\/span> Unit<span style=\"color:#af3a03\">.<\/span>SECOND:\n<\/span><\/span><span style=\"display:flex;\"><span>            <span style=\"color:#af3a03\">return<\/span> <span style=\"color:#8f3f71\">1<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">elif<\/span> <span style=\"color:#b57614\">self<\/span> <span style=\"color:#af3a03\">==<\/span> Unit<span style=\"color:#af3a03\">.<\/span>MINUTE:\n<\/span><\/span><span style=\"display:flex;\"><span>            <span style=\"color:#af3a03\">return<\/span> <span style=\"color:#8f3f71\">60<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">elif<\/span> <span style=\"color:#b57614\">self<\/span> <span style=\"color:#af3a03\">==<\/span> Unit<span style=\"color:#af3a03\">.<\/span>HOUR:\n<\/span><\/span><span style=\"display:flex;\"><span>            <span style=\"color:#af3a03\">return<\/span> <span style=\"color:#8f3f71\">3600<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">else<\/span>:\n<\/span><\/span><span style=\"display:flex;\"><span>            <span style=\"color:#af3a03\">raise<\/span> <span style=\"color:#fb4934\">ValueError<\/span>(<span style=\"color:#79740e\">f<\/span><span style=\"color:#79740e\">&#34;Unknown unit: <\/span><span style=\"color:#79740e\">{<\/span><span style=\"color:#b57614\">self<\/span><span style=\"color:#79740e\">}<\/span><span style=\"color:#79740e\">&#34;<\/span>)\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>@dataclass\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">class<\/span> Descriptor:\n<\/span><\/span><span style=\"display:flex;\"><span>    key: <span style=\"color:#b57614\">str<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>    unit: Unit\n<\/span><\/span><span style=\"display:flex;\"><span>    requests_per_unit: <span style=\"color:#b57614\">int<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>    value: Optional[<span style=\"color:#b57614\">str<\/span>] <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#af3a03\">None<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>@dataclass\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">class<\/span> Rule:\n<\/span><\/span><span style=\"display:flex;\"><span>    path: <span style=\"color:#b57614\">str<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>    descriptors: List[Descriptor]\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">match<\/span>(<span style=\"color:#b57614\">self<\/span>, path: <span style=\"color:#b57614\">str<\/span>) <span style=\"color:#af3a03\">-&gt;<\/span> <span style=\"color:#b57614\">bool<\/span>:\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">return<\/span> <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>path <span style=\"color:#af3a03\">==<\/span> path<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p><code>dataclass<\/code> is used here for easier initialization.<\/p>\n<p>Now let&rsquo;s define the storage interface<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-python\" data-lang=\"python\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">from<\/span> <span style=\"color:#79740e\">abc<\/span> <span style=\"color:#af3a03\">import<\/span> ABC, abstractmethod\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">from<\/span> <span style=\"color:#79740e\">enum<\/span> <span style=\"color:#af3a03\">import<\/span> Enum\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">class<\/span> StorageEngines(<span style=\"color:#b57614\">str<\/span>, Enum):\n<\/span><\/span><span style=\"display:flex;\"><span>    REDIS <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#79740e\">&#34;redis&#34;<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>    MEMORY <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#79740e\">&#34;memory&#34;<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">class<\/span> AbstractStorage(ABC):\n<\/span><\/span><span style=\"display:flex;\"><span>    @abstractmethod\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">get<\/span>(<span style=\"color:#b57614\">self<\/span>, key):\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">raise<\/span> <span style=\"color:#fb4934\">NotImplementedError<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    @abstractmethod\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">set<\/span>(<span style=\"color:#b57614\">self<\/span>, key, value, ttl_seconds: <span style=\"color:#b57614\">int<\/span>):\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">raise<\/span> <span style=\"color:#fb4934\">NotImplementedError<\/span><\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p><code>get<\/code> and <code>set<\/code> are the required methods for implementing the above algorithms.\nother operations such as <a href=\"https:\/\/redis.io\/commands\/decr\/\" rel=\"noopener\" target=\"_blank\">decr<\/a>\nare also available which can increase the performance of our code.<\/p>\n<p>Limit strategies only depend on the storage &amp; rule components.\nIt should also have a request type for itself which can be used by\nother components calling it to pass in a request with a generic form.\nSo it does not depend on a specific type of request, but only it&rsquo;s data and path.<\/p>\n<p>The only public function is <code>do_limit<\/code>\nwhich takes in a request and determines if it&rsquo;s limited or not.<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-python\" data-lang=\"python\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">class<\/span> LimitStrategies(<span style=\"color:#b57614\">str<\/span>, Enum):\n<\/span><\/span><span style=\"display:flex;\"><span>    TOKEN_BUCKET <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#79740e\">&#34;token_bucket&#34;<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>@dataclass\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">class<\/span> Request:\n<\/span><\/span><span style=\"display:flex;\"><span>    path: <span style=\"color:#b57614\">str<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>    data: <span style=\"color:#b57614\">dict<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">class<\/span> AbstractStrategy(abc<span style=\"color:#af3a03\">.<\/span>ABC, metaclass<span style=\"color:#af3a03\">=<\/span>abc<span style=\"color:#af3a03\">.<\/span>ABCMeta):\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">__init__<\/span>(\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#b57614\">self<\/span>,\n<\/span><\/span><span style=\"display:flex;\"><span>        storage_backend: AbstractStorage,\n<\/span><\/span><span style=\"display:flex;\"><span>        rule_descriptor: Descriptor,\n<\/span><\/span><span style=\"display:flex;\"><span>    ) <span style=\"color:#af3a03\">-&gt;<\/span> <span style=\"color:#af3a03\">None<\/span>:\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>storage_backend <span style=\"color:#af3a03\">=<\/span> storage_backend\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#b57614\">self<\/span><span style=\"color:#af3a03\">.<\/span>rule_descriptor <span style=\"color:#af3a03\">=<\/span> rule_descriptor\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    @abc.abstractmethod\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">do_limit<\/span>(\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#b57614\">self<\/span>,\n<\/span><\/span><span style=\"display:flex;\"><span>        request: Request,\n<\/span><\/span><span style=\"display:flex;\"><span>    ):\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">raise<\/span> <span style=\"color:#fb4934\">NotImplementedError<\/span><\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<h3 class=\"heading\" id=\"token-bucket\">\n  Token Bucket\n  <a class=\"anchor\" href=\"#token-bucket\">#<\/a>\n<\/h3>\n<p>Now that the interfaces are clear we can start implementing the algorithm.<\/p>\n<p>From the rule structure,\nwe can use the <code>unit<\/code> to to refresh tokens in the bucket.\nand the <code>request_per_unit<\/code> to determine bucket capacity.<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-python\" data-lang=\"python\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">class<\/span> TokenBucket(AbstractStrategy):\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">__init__<\/span>(\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#b57614\">self<\/span>,\n<\/span><\/span><span style=\"display:flex;\"><span>        storage_backend: AbstractStorage,\n<\/span><\/span><span style=\"display:flex;\"><span>        rule_descriptor: Descriptor,\n<\/span><\/span><span style=\"display:flex;\"><span>    ):\n<\/span><\/span><span style=\"display:flex;\"><span>        su<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<h4 class=\"heading\" id=\"test-rate-limiter-service\">\n  Test Rate Limiter Service\n  <a class=\"anchor\" href=\"#test-rate-limiter-service\">#<\/a>\n<\/h4>\n<p>For rate limiter service we need to create two fixtures, config and local storage:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-python\" data-lang=\"python\"><span style=\"display:flex;\"><span>@pytest.fixture\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">local_storage<\/span>():\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">yield<\/span> memory<span style=\"color:#af3a03\">.<\/span>Memory()\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>@pytest.fixture\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">config<\/span>():\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">return<\/span> Config(\n<\/span><\/span><span style=\"display:flex;\"><span>        limit_strategy<span style=\"color:#af3a03\">=<\/span>LimitStrategies<span style=\"color:#af3a03\">.<\/span>TOKEN_BUCKET,\n<\/span><\/span><span style=\"display:flex;\"><span>    )<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>Now what can be tested in the service?\nWith the limit strategy we were testing if the rule descriptor is applied correctly.<\/p>\n<p>Here we should check if the rule is applied correctly,\nthis means we can still test the rule descriptor part,\nbut it&rsquo;s not necessary since if a rule descriptor is not applied correctly\nthen the limit strategy test must throw an error(otherwise we end up with an\nuntested strategy which is a nightmare).<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-python\" data-lang=\"python\"><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">def<\/span> <span style=\"color:#b57614\">test_rate_limit_service_applies_the_rule<\/span>(local_storage: memory<span style=\"color:#af3a03\">.<\/span>Memory, config: Config):\n<\/span><\/span><span style=\"display:flex;\"><span>    rule_descriptor <span style=\"color:#af3a03\">=<\/span> Descriptor(\n<\/span><\/span><span style=\"display:flex;\"><span>        key<span style=\"color:#af3a03\">=<\/span><span style=\"color:#79740e\">&#34;user_id&#34;<\/span>,\n<\/span><\/span><span style=\"display:flex;\"><span>        requests_per_unit<span style=\"color:#af3a03\">=<\/span><span style=\"color:#8f3f71\">1<\/span>,\n<\/span><\/span><span style=\"display:flex;\"><span>        unit<span style=\"color:#af3a03\">=<\/span>Unit<span style=\"color:#af3a03\">.<\/span>SECOND,\n<\/span><\/span><span style=\"display:flex;\"><span>    )\n<\/span><\/span><span style=\"display:flex;\"><span>    rule <span style=\"color:#af3a03\">=<\/span> Rule(path<span style=\"color:#af3a03\">=<\/span><span style=\"color:#79740e\">&#34;\/limited-path&#34;<\/span>, descriptors<span style=\"color:#af3a03\">=<\/span>[rule_descriptor])\n<\/span><\/span><span style=\"display:flex;\"><span>    rate_limit_service <span style=\"color:#af3a03\">=<\/span> RateLimitService(\n<\/span><\/span><span style=\"display:flex;\"><span>        config<span style=\"color:#af3a03\">=<\/span>config, storage_engine<span style=\"color:#af3a03\">=<\/span>local_storage, rules<span style=\"color:#af3a03\">=<\/span>[rule]\n<\/span><\/span><span style=\"display:flex;\"><span>    )\n<\/span><\/span><span style=\"display:flex;\"><span>    request <span style=\"color:#af3a03\">=<\/span> Request(path<span style=\"color:#af3a03\">=<\/span><span style=\"color:#79740e\">&#34;\/limited-path&#34;<\/span>, data<span style=\"color:#af3a03\">=<\/span>{<span style=\"color:#79740e\">&#34;user_id&#34;<\/span>: <span style=\"color:#79740e\">&#34;1&#34;<\/span>})\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">assert<\/span> rate_limit_service<span style=\"color:#af3a03\">.<\/span>do_limit(request) <span style=\"color:#af3a03\">is<\/span> <span style=\"color:#af3a03\">False<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">assert<\/span> rate_limit_service<span style=\"color:#af3a03\">.<\/span>do_limit(request) <span style=\"color:#af3a03\">is<\/span> <span style=\"color:#af3a03\">True<\/span>\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>    time_now <span style=\"color:#af3a03\">=<\/span> datetime<span style=\"color:#af3a03\">.<\/span>datetime<span style=\"color:#af3a03\">.<\/span>now() <span style=\"color:#af3a03\">+<\/span> datetime<span style=\"color:#af3a03\">.<\/span>timedelta(seconds<span style=\"color:#af3a03\">=<\/span><span style=\"color:#8f3f71\">3<\/span>)\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">with<\/span> freezegun<span style=\"color:#af3a03\">.<\/span>freeze_time(time_now):\n<\/span><\/span><span style=\"display:flex;\"><span>        <span style=\"color:#af3a03\">assert<\/span> rate_limit_service<span style=\"color:#af3a03\">.<\/span>do_limit(request) <span style=\"color:#af3a03\">is<\/span> <span style=\"color:#af3a03\">False<\/span><\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<h2 class=\"heading\" id=\"conclusion\">\n  Conclusion\n  <a class=\"anchor\" href=\"#conclusion\">#<\/a>\n<\/h2>\n<p>So far we have a working rate limiter with one implemented rule.\nI think this would be enough for one read,\nIn the next post we will add more rate limiting algorithms and see\nhow the current structure of the program can be extended.<\/p>\n<p>You can find the <a href=\"https:\/\/github.com\/Glyphack\/hera-limit\" rel=\"noopener\" target=\"_blank\">complete source code<\/a>\non my Github.<\/p>\n"},{"title":"Everything you need to know about splitting CDK stacks","link":"https:\/\/glyphack.com\/splitting-cdk-stacks\/","pubDate":"Sun, 13 Nov 2022 09:49:50 +0330","guid":"https:\/\/glyphack.com\/splitting-cdk-stacks\/","description":"<p>AWS CDK is a tool that lets you define your cloud resources\nin a languages such as python.\nAnd like any other project as the codebase grows, it needs to be split into components.<\/p>\n<p>CDK projects can be split into multiple stacks.\nStack is a single deployable unit in CDK.\nwhen you deploy it all the resources inside it will get deployed.<\/p>\n<p>Knowing a few points in the beginning can help to create a good project structure,\nand avoiding common pitfalls like dependency issues.<\/p>\n<p>The questions to ask when considering splitting a stack:<\/p>\n<ul>\n<li>Which components should be extracted?<\/li>\n<li>Should this new code be a stack or a construct?<\/li>\n<li>which components belong to this stack?<\/li>\n<li>What cross-stack dependencies are created?<\/li>\n<\/ul>\n<p>I\u2019m sharing the lessons I learned writing and refactoring multiple cdk projects.<\/p>\n<h2 class=\"heading\" id=\"splitting-stacks\">\n  Splitting stacks\n  <a class=\"anchor\" href=\"#splitting-stacks\">#<\/a>\n<\/h2>\n<p>When components can be deployed separately, it&rsquo;s good to split the stacks.\nSmaller stacks are deployed faster and it&rsquo;s easier to maintain them.<\/p>\n<p>You can extract some parts of a stack into another stack and add it as a dependency.\nKeep in mind that when <code>StackA<\/code> depends on <code>StackB<\/code> then:<\/p>\n<ul>\n<li>To Deploy <code>StackA<\/code> the <code>StackB<\/code> must be deployed.<\/li>\n<li>When stacks are deployed <code>StackB<\/code> cannot be updated without updating <code>StackA<\/code> first.<\/li>\n<\/ul>\n<p>This two way relation can cause problems which we&rsquo;ll talk about it later.<\/p>\n<h3 class=\"heading\" id=\"how-to-decide-the-number-of-stacks\">\n  How to Decide the Number of Stacks?\n  <a class=\"anchor\" href=\"#how-to-decide-the-number-of-stacks\">#<\/a>\n<\/h3>\n<p>A simple rule for separation is based on:<\/p>\n<ol>\n<li>Application domain<\/li>\n<li>Resources life cycle<\/li>\n<\/ol>\n<p>Multiple stacks for apps in different domains is a separation based on domain.\nSeparating based on life cycle can be done for rarely deployed resources like databases.<\/p>\n<p>You can find a good reference on these examples and stacks in\n<a href=\"https:\/\/github.com\/kevinslin\/open-cdk#stacks\" rel=\"noopener\" target=\"_blank\">open-cdk guide<\/a>.<\/p>\n<h2 class=\"heading\" id=\"creating-constructs\">\n  Creating constructs\n  <a class=\"anchor\" href=\"#creating-constructs\">#<\/a>\n<\/h2>\n<p>CDK has another solution for having smaller stacks\nwhich is creating a re-suable component called construct.\nTo decide whether a code should be a stack or a construct we can check:<\/p>\n<ol>\n<li>Can be used in other places: a S3 Bucket with specific options.<\/li>\n<li>A second stack would be always deployed deleted with current stack(highly coupled).<\/li>\n<\/ol>\n<p>Creating constructs is also easier;\nbecause you are not introducing any dependencies between stacks.\nIf you delete an stack all constructs inside it are removed.<\/p>\n<h2 class=\"heading\" id=\"separated-stacks-and-dependencies\">\n  Separated Stacks and Dependencies\n  <a class=\"anchor\" href=\"#separated-stacks-and-dependencies\">#<\/a>\n<\/h2>\n<p>Imagine we have an API with lambda and API gateway, and a route53 hosted zone.\nThis infrastructure has two properties:<\/p>\n<ul>\n<li>Lambda can frequently be changed, but the API endpoint is same.<\/li>\n<li>deploying a resource like a hosted zone takes a lot of time.\nSo it&rsquo;s better to deploy it separately, once.<\/li>\n<li>A hosted zone record must point to API endpoint.<\/li>\n<\/ul>\n<p>By splitting this stack into two:<\/p>\n<ol>\n<li>API stack<\/li>\n<li>Hosted zone stack<\/li>\n<\/ol>\n<p>\n\n\n\n\n\n\n\n<figure class=\"\">\n\n    \n    \n      \n        \n        <div class=\"img-container\" style=\"--w: 2633; --h: 737;\">\n            <img loading=\"lazy\" alt=\"example-cdk-stack\" src=\"https:\/\/glyphack.com\/splitting-cdk-stacks\/example-cdk-stack-deps.excalidraw_hu_e8c9fb71466ed45.png\" width=\"2633\" height=\"737\">\n        <\/div>\n      \n    \n\n    \n<\/figure>\n<\/p>\n<h3 class=\"heading\" id=\"dependency-problem-and-solution\">\n  Dependency problem and Solution\n  <a class=\"anchor\" href=\"#dependency-problem-and-solution\">#<\/a>\n<\/h3>\n<p>What dependencies did we create?\nIf our API endpoint changes then the Hosted zone needs to be updated\nto point to the new endpoint.\nThis dependencies between stack, can cause problems if not considered carefully.\nIf you now try to update the API endpoint(which is part of API stack)\nwhen all the stacks are deployed you will get the following error:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-text\" data-lang=\"text\"><span style=\"display:flex;\"><span>Export ApiStack:ExportsOutputFnGetAtt-******\n<\/span><\/span><span style=\"display:flex;\"><span>cannot be deletedas it is in use by HostedZoneStack<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>The issue is that the Hosted Zone stack is using the referenced endpoint\nfrom ApiStack and without deleting the hosted zone you cannot delete the reference.<\/p>\n<p>You have two solutions to either destroy them both or <a href=\"https:\/\/github.com\/aws\/aws-cdk\/tree\/main\/packages\/aws-cdk-lib#removing-automatic-cross-stack-references\" rel=\"noopener\" target=\"_blank\">do a 2 phase deployment<\/a>.\nAlthough the 2 phase deployment always works it&rsquo;s manual work.\nIt&rsquo;s fine for a database migration but this should not be something to do every week.<\/p>\n<p><em>Solution to prevent the issue<\/em>:\n<strong>Document<\/strong> not changing parts in your CDK app.\nOur assumption is that our API endpoint is not going to change frequently,\notherwise we are just deploying stacks together all the time.<\/p>\n<p>Another solution is to use parameter store to reference these dependencies.\nIn this method you can change anything since CDK does not know about your dependencies.\nThe downside of this method is that you have to manage dependencies now,\nactually you are now the compiler :D.<\/p>\n<p>I would not recommend this method Because you are loosing the type checks,\nlike <code>any<\/code> in a typed language.<\/p>\n<p>It&rsquo;s a solution where you want to share resources between multiple CDK apps where\nyou can&rsquo;t pass the objects.\nFor example if you are referencing DB name in applications as env vars,\nit&rsquo;s better to save the DB name in parameter store and retrieve in runtime.\nBecause that resource can be deployed multiple times but your app does not need\nto be redeployed.<\/p>\n<p>Now Let me give you a <strong>Bad<\/strong> splitting example:<\/p>\n<p>Imagine we have a EC2 with an application load balancer.\nThe EC2 id must be set in load balancer to route the traffic.<\/p>\n<p>If we try to separate these into load balancer and app stacks,\nwhen ever we want to redeploy the EC2 the other stack must be destroyed and redeployed.<\/p>\n<h3 class=\"heading\" id=\"remove-security-group-dependencies\">\n  Remove Security Group dependencies\n  <a class=\"anchor\" href=\"#remove-security-group-dependencies\">#<\/a>\n<\/h3>\n<p>A common dependency between stacks is the security group rules.\nFor example we have a database with and we want our EC2 to access it.\nOne solution could be:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-javascript\" data-lang=\"javascript\"><span style=\"display:flex;\"><span><span style=\"color:#928374;font-style:italic\">\/\/ DB Stack\n<\/span><\/span><\/span><span style=\"display:flex;\"><span>dbInstance <span style=\"color:#af3a03\">=<\/span> ...\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#928374;font-style:italic\">\/\/ App Stack\n<\/span><\/span><\/span><span style=\"display:flex;\"><span>dbInstance.connections.allowFrom(ec2Instance, ec2.Port.tcp(<span style=\"color:#8f3f71\">5432<\/span>));<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>But in this code DB stacks depends on the EC2 instance which can change.\nIs there a way to make our database stack completely independent?<\/p>\n<p>Surprisingly you can use the dependency inversion principle:<\/p>\n<blockquote>\n<p><em>Low level policies should depend upon high level policies.<\/em><\/p>\n<\/blockquote>\n<p>How can we make the application dependent on database in this case?\nWe can just flip the dependency.<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-typescript\" data-lang=\"typescript\"><span style=\"display:flex;\"><span><span style=\"color:#928374;font-style:italic\">\/\/ DB stack\n<\/span><\/span><\/span><span style=\"display:flex;\"><span>dbInstance <span style=\"color:#af3a03\">=<\/span> ...\n<\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#928374;font-style:italic\">\/\/ App stack\n<\/span><\/span><\/span><span style=\"display:flex;\"><span>\n<\/span><\/span><span style=\"display:flex;\"><span>Ec2 <span style=\"color:#af3a03\">=<\/span> <span style=\"color:#af3a03\">new<\/span> ec2.Instance(...)\n<\/span><\/span><span style=\"display:flex;\"><span>Ec2.connections.allowTo(Db, ...)<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>With this method your database is not aware of what resources are connected to it.\nYou made it a high level policy and other services are now aware of how to connect.<\/p>\n<h3 class=\"heading\" id=\"remove-unnecessary-dependencies\">\n  Remove unnecessary dependencies\n  <a class=\"anchor\" href=\"#remove-unnecessary-dependencies\">#<\/a>\n<\/h3>\n<p>Removing redundant dependencies reduces the overhead of managing them between stacks.\nThe security group was an example of how to do this.\nImportant lesson here is to understand how each line of\nCDK is connecting your resources together.<\/p>\n<h2 class=\"heading\" id=\"conclusion\">\n  Conclusion\n  <a class=\"anchor\" href=\"#conclusion\">#<\/a>\n<\/h2>\n<p>Knowing how to split stacks is an art of managing dependencies.\nWhen I first started to do it before knowing these topics, I ran into\nissues I did not except and it helped me see the problem from a different aspect.\nIt&rsquo;s not only about separating code but it&rsquo;s about which resources should be grouped.<\/p>\n<p>I hope knowing this will help you avoiding this common pitfall.<\/p>\n"},{"title":"Planning My Day","link":"https:\/\/glyphack.com\/planning-my-day\/","pubDate":"Wed, 07 Sep 2022 23:22:58 +0430","guid":"https:\/\/glyphack.com\/planning-my-day\/","description":"<p>A while back I started planning my day before starting my day. I did this because I wanted to have a morning routine for myself,\nbut most of the time I found myself starting to pickup something to read or already replying to emails or slack. So by setting this\nrule for myself to write down everything I want to do for the day first, helped to overcome this habit.<\/p>\n<p>Since then I found some improvements that can be done along with writing down the plan for the day. I&rsquo;ll list my observation here, and my\ngoal is to implement these in the <a href=\"https:\/\/github.com\/Glyphack\/koal\" rel=\"noopener\" target=\"_blank\">koal<\/a> app, as a default way of planning the day.<\/p>\n<h2 class=\"heading\" id=\"choosing-what-to-do\">\n  Choosing what to do\n  <a class=\"anchor\" href=\"#choosing-what-to-do\">#<\/a>\n<\/h2>\n<p>I have a long list of things I want to try\/learn\/experiment; and the list continues to expand everyday when I come across new things.\nSo in this situation my mind is full of different things to do, I cannot concentrate on one thing and finish it in a reasonable time.\nSometimes I saw that It&rsquo;s been months that I&rsquo;m working on something without making much progress and that was because I did not spend\ntime continuously to finish it.<\/p>\n<p>One thing that I think is related to this is this quote from <a href=\"https:\/\/jamesclear.com\/atomic-habits\" rel=\"noopener\" target=\"_blank\">atomic habits book<\/a><\/p>\n<blockquote>\n<p>Success is the product of daily habits\u2014not once-in-a-lifetime transformations.<\/p>\n<\/blockquote>\n<p>It tries to say that you don&rsquo;t reach your goals in a single step, you create daily habits based on them.\nI borrowed this definition of success and tried to use it in my planning.<\/p>\n<p>This is the first thing I realized I have to do:\n<strong>If I want to do something I have to do it everyday, no matter how small is my progress is<\/strong><\/p>\n<p>This helps a lot on different aspects of reaching my goal.\nWhen I do my work continuously I&rsquo;m always aware of where I left off,\nI could more easily get in the zone, and needed less time to get focused.\nThis method can be very powerful even if you <a href=\"https:\/\/medium.com\/@alexallain\/ten-minutes-a-day-e2fa1084f924\" rel=\"noopener\" target=\"_blank\">only spend 10 minutes a day on something<\/a>.<\/p>\n<blockquote>\n<p>How, exactly, did I manage to write a book in this short a time? I had one simple rule: I had to work on the book for just ten minutes, every day, no excuses. Ever.<\/p>\n<p>The original reason I tracked my time, in fact, was that I wanted to motivate myself by having a streak of days, and I figured that instead of just tallying check marks, I\u2019d write down exactly how long I spent. It worked \u2014 I never missed a day.<\/p>\n<\/blockquote>\n<p>A challenge that comes with this approach is that I no longer can have so many ongoing work at once,\nI decided to re plan my goals so they are:<\/p>\n<ul>\n<li>more specific: instead of saying I want to learn a tool I said I want to use the tool do make something.<\/li>\n<li>split into approachable goals: an example was that I wanted to get 1800 chess ELO, but this journey is not a single step, in each ELO range you need to focus on something and get better at it. So it split this into two parts of getting to 1400 and the getting to 1800<\/li>\n<\/ul>\n<p>Then I picked only 5 or 6 goals at each point, I had to have enough time for all of them to make continuous progress on each.\nEach day I started by browsing my current work, thinking what is the next small step I can take today and write it down in Koal.<\/p>\n<h2 class=\"heading\" id=\"repetitive-tasks\">\n  Repetitive tasks\n  <a class=\"anchor\" href=\"#repetitive-tasks\">#<\/a>\n<\/h2>\n<p>There are things I want to do everyday, like my morning routine.\nHaving a prepared list of these kind of work saves a lot of time,\nbecause I don&rsquo;t have to remember, and write them down everyday.\nEven some of my goals consists of repetitive tasks,\nlike for chess I wanted to play a game everyday.<\/p>\n<p>Writing down this list also helped me with creating a habit of doing these things, For example when I\nadded meditation to this list, only by seeing this everyday I did not forgot about it.<\/p>\n<p>For me this list is very simple, I just wrote down things I have to do. There are some things that I don&rsquo;t\ndo everyday but I still listed them only to not forget to do them.<\/p>\n<h2 class=\"heading\" id=\"being-focused-on-tasks\">\n  Being focused on tasks\n  <a class=\"anchor\" href=\"#being-focused-on-tasks\">#<\/a>\n<\/h2>\n<p>I continued with my approach for some time and realized there are somethings I need to do to improve my focus.<\/p>\n<p>Sometimes I tried to do multi tasking because I had number of small things to do, So when I was waiting for something to finish\nI started writing down an email, and if it was finished I got back to it and then I lost my focus.<\/p>\n<p>Sometimes I was not even waiting for something but I just jumped to a different thing, because I was not focused.\nSome tasks require you to be focused for the whole time to finish it, like writing a blog post or writing code,\nIn these cases I could not do something that steals my focus like browsing social media while I&rsquo;m waiting.\nBut there are things that do not steal the focus, like pouring a cup of tea or doing the dishes.<\/p>\n<p>I slightly changed my habit to only do these kind of tasks that do not require a context switch,\nwhile I&rsquo;m blocked on my current task.<\/p>\n<p>How to change this habit?<\/p>\n<ul>\n<li>Try to have a list of things you can do while waiting.<\/li>\n<li>Use a pomodoro or a time tracker, this helps to focus because it only requires a short amount of focused work.<\/li>\n<li>Rewarding myself after the focus time, <a href=\"https:\/\/www.frugalconfessions.com\/save-me-money\/reward-yourself\/\" rel=\"noopener\" target=\"_blank\">this list<\/a> is a good one if you want to get started<\/li>\n<\/ul>\n<h2 class=\"heading\" id=\"having-an-uninterrupted-focus-time\">\n  Having an uninterrupted focus time\n  <a class=\"anchor\" href=\"#having-an-uninterrupted-focus-time\">#<\/a>\n<\/h2>\n<p>Not all of the day I have the same productivity and energy, I found that I achieve the most when I&rsquo;m working uninterrupted.\nTo leverage this, I started to mark a time window of 2-3 hours for doing focused and uninterrupted work.\nThis is really important to set right, in different days I have meetings at different hours and some days I have a busy morning,\nand some days I have a busy afternoon.\nI try to find a time window that I can block for myself to focus on important things I have to do.<\/p>\n<p>In the planning I also write down when is this uninterrupted focus time, when the time comes I put my phone away and minimize\nthe chance of loosing focus.<\/p>\n<h2 class=\"heading\" id=\"choosing-one-thing-to-complete-everyday\">\n  Choosing one thing to complete everyday\n  <a class=\"anchor\" href=\"#choosing-one-thing-to-complete-everyday\">#<\/a>\n<\/h2>\n<p>While I have a list of things to do, I know what is the most important one for today to complete.\nThis can be something from my work that I want to get done, or reaching a milestone in one of the goals.\nThis task is very important because it brings me joy and motivation when I finish it.<\/p>\n<p>I mark the important task of the day and try to start that one during my uninterrupted time. This helps me to make sure I can get it done.\nThe other benefit is that every few days I reach an important milestone in my goals or work,\nWhich means that I&rsquo;m one step closer to finish it.\nThis task does not have to be a serious deadline or anything, it&rsquo;s just a thing I decide on.\nFor example if I&rsquo;m working on a new feature the important part is to sending it for review.\nWhen I know today is the day for it to be done I use my most productive time to finish it.<\/p>\n<p>When I started to mark the important task to be done for a day,\nI made a lot more progress in my goals. Because I was finishing these important milestones when it was needed for progress.\nIt&rsquo;s like there are some parts of each goal which require a lot of effort to get it done.\nIf I don&rsquo;t put some serious time on my goals from time to time I cannot make significant progress.<\/p>\n<h2 class=\"heading\" id=\"writing-down-detailed-to-dos\">\n  Writing down detailed to-dos\n  <a class=\"anchor\" href=\"#writing-down-detailed-to-dos\">#<\/a>\n<\/h2>\n<p>While to-do list should be short I found it useful to add what I need for the task to be done in the task.\nWhen I add an entry of reading the article about something I attach the link there.\nIf I want to check the to-do search the google for the article and read it I might encounter\nthings that can steal my focus along the way, like finding another interesting article in the search result.<\/p>\n<h2 class=\"heading\" id=\"leverage-boredom\">\n  Leverage Boredom\n  <a class=\"anchor\" href=\"#leverage-boredom\">#<\/a>\n<\/h2>\n<p>People <a href=\"https:\/\/www.science.org\/content\/article\/people-would-rather-be-electrically-shocked-left-alone-their-thoughts\" rel=\"noopener\" target=\"_blank\">do anything<\/a> to escape boredom.<\/p>\n<p>When I plan my day I try to plan as much as I can so I don&rsquo;t do anything outside the planning, in this way I either have to do the stuff\nI planned for or do nothing. This helps to avoid procrastination.<\/p>\n<p>This is very hard to get write, I sometimes forget some chore tasks or something unplanned happens, but the good part is most of the day\nI&rsquo;m facing my plan and can&rsquo;t do other things before complete the tasks.<\/p>\n<h2 class=\"heading\" id=\"rewarding\">\n  Rewarding\n  <a class=\"anchor\" href=\"#rewarding\">#<\/a>\n<\/h2>\n<p>Most of my motivation to do something comes from either the joy of doing the thing or the joy of finishing it.\nFor some goals(specially the ones that takes more time) it&rsquo;s hard to have these two.\nFor example if your goal is to study for something for 3-4 months it&rsquo;s gonna be very hard to keep the motivation.\nI wish I would know sooner that I have to reward myself for doing my tasks, and this really works well with these kind of tasks.<\/p>\n<p>I try to reward myself after doing a couple of tasks, this can be a video game or watching a YouTube video,\nbut there are also other resources to <a href=\"https:\/\/www.frugalconfessions.com\/save-me-money\/reward-yourself\/\" rel=\"noopener\" target=\"_blank\">reward yourself without money<\/a>.\nOne other effective method is to reward yourself with money after doing something. For example I can put aside 10$ each\ntime I do X and use that money to buy something I like.\nWhen I started streaming my rule was that I won&rsquo;t buy fancy mic\/webcam stuff until I create videos for certain amount of time.<\/p>\n<p>One way to apply this can be by having a list of rewards beside items planned for the day.<\/p>\n<h2 class=\"heading\" id=\"conclusion\">\n  Conclusion\n  <a class=\"anchor\" href=\"#conclusion\">#<\/a>\n<\/h2>\n<p>Using this approach and planning the tasks in the beginning of the day, helped me a lot on being more organized, and following\nmy goals. To summarize the full list into steps for planning the day:<\/p>\n<ol>\n<li>Going through list of current ongoing projects and see list what to do next for each<\/li>\n<li>Going over repetitive tasks and add them to my current list<\/li>\n<li>Setting my uninterrupted focus time for the day<\/li>\n<li>Getting started on the tasks, setting pomodoro timer<\/li>\n<li>After some tasks checking my reward lists and pick up one to enjoy<\/li>\n<li>If anything is left from the work which I know have to pick up tomorrow I write it down in a note.<\/li>\n<\/ol>\n<p>I&rsquo;ll try to update the list if I find more useful tricks but for now this is it. Hope you enjoyed it.<\/p>\n"},{"title":"Stateful Stream Processing","link":"https:\/\/glyphack.com\/stateful-stream-processing\/","pubDate":"Mon, 15 Aug 2022 08:53:57 +0430","guid":"https:\/\/glyphack.com\/stateful-stream-processing\/","description":"<p>A pattern that I have recently seen in a project is data replication through\nreal-time stream processing. This pattern happens when a company has all of\nit&rsquo;s data stored on centralized databases and applications access this data,\nthis means that two example services like &ldquo;search&rdquo; and &ldquo;product catalog&rdquo; are\ndepending on the same data.\nIn this situation some services cannot function by only request the data\nthrough API, they need a snapshot of the data.\nDo address this requirement the real-time data replication solutions come\nhandy.\nWith this approach we need a event driven architecture to handle incoming data\nand apply required transformations.\nAll stream processing frameworks such as Kafka Streams support doing stream\njoins to denormalize the data and while they offer very straightforward\nperformant solutions the join support is limited in some cases.<\/p>\n<h2 class=\"heading\" id=\"streaming-changes-from-database\">\n  Streaming Changes from Database\n  <a class=\"anchor\" href=\"#streaming-changes-from-database\">#<\/a>\n<\/h2>\n<p>We can start by implementing a change data capture system with a tool like\nDebizium and Apache Kafka. An example architecture is the following picture:\n\n\n\n\n\n\n\n\n<figure class=\"\">\n\n    \n    \n      \n        \n        <div class=\"img-container\" style=\"--w: 2656; --h: 1200;\">\n            <img loading=\"lazy\" alt=\"streaming-database-changes\" src=\"https:\/\/glyphack.com\/stateful-stream-processing\/streaming-database-changes.excalidraw_hu_7f45c6e930c8d53d.png\" width=\"2656\" height=\"1200\">\n        <\/div>\n      \n    \n\n    \n<\/figure>\n<\/p>\n<p>Now the downstream services can use the data product topic coming out of Kafka.\nLet&rsquo;s focus on how to implement the stream processing component.<\/p>\n<p>As discussed earlier this use case is very suitable for a tool like Kafka\nStreams but if we have requirements to join data like:<\/p>\n<ul>\n<li>Join on primary and non primary keys<\/li>\n<li>Having no time window on when the join can occur<\/li>\n<li>Support denormalizing database relations<\/li>\n<\/ul>\n<p>Then you cannot utilize full power of Kafka streams.\nBecause Kafka streams joins the data on record key then all records that are\ngoing to be joined need to be published with the same key.\nNow if a stream needs to be joined with multiple streams then it has to bNnne\npublished in multiple topics with different keys.\nThis approach will result in the following diagram<\/p>\n<p>\n\n\n\n\n\n\n\n<figure class=\"\">\n\n    \n    \n      \n        \n        <div class=\"img-container\" style=\"--w: 2471; --h: 560;\">\n            <img loading=\"lazy\" alt=\"reparition-data-join\" src=\"https:\/\/glyphack.com\/stateful-stream-processing\/repartion-data-to-join.excalidraw_hu_dc43f41e4a269653.png\" width=\"2471\" height=\"560\">\n        <\/div>\n      \n    \n\n    \n<\/figure>\n<\/p>\n<p>This can lead to a very expensive solution both in terms of cost and\ndevelopment.<\/p>\n<h2 class=\"heading\" id=\"problem-and-possible-solutions\">\n  Problem and Possible Solutions\n  <a class=\"anchor\" href=\"#problem-and-possible-solutions\">#<\/a>\n<\/h2>\n<p>Now that we know the pattern and the limit let&rsquo;s try to solve it with for an\nexample business.<\/p>\n<p>My goal is to find a solution which is easy to implement not the fastest one\n, but a balance between easy and cost efficient and low latency solutions.\nIt&rsquo;s also possible to come up with a range of different configurations and package them\nas solutions so different users can choose different configurations.<\/p>\n<p>Imagine you are doing this for Github, they want to extract all user data and\nit&rsquo;s interactions with the platform into an easy to use data structure named\n<code>UserProfile<\/code>.<\/p>\n<p>Let&rsquo;s start by giving a very simple example data model:\n\n\n\n\n\n\n\n\n<figure class=\"\">\n\n    \n    \n      \n        \n        <div class=\"img-container\" style=\"--w: 1433; --h: 876;\">\n            <img loading=\"lazy\" alt=\"github-example-data-model\" src=\"https:\/\/glyphack.com\/stateful-stream-processing\/github-example-data-model.excalidraw_hu_97fe87ebcd8c64d7.png\" width=\"1433\" height=\"876\">\n        <\/div>\n      \n    \n\n    \n<\/figure>\n<\/p>\n<p>With following definitions:<\/p>\n<p><strong>User<\/strong><\/p>\n<ul>\n<li>id<\/li>\n<li>email<\/li>\n<li>username<\/li>\n<li>registered_at<\/li>\n<\/ul>\n<p>Pk: email<\/p>\n<p><strong>Repo<\/strong><\/p>\n<ul>\n<li>name<\/li>\n<li>owner<\/li>\n<li>description<\/li>\n<li>star_count<\/li>\n<li>fork_count<\/li>\n<li>created_at<\/li>\n<li>updated_at<\/li>\n<\/ul>\n<p>Pk: (owner,name)<\/p>\n<p><strong>Pull Request<\/strong><\/p>\n<ul>\n<li>title<\/li>\n<li>repo<\/li>\n<li>repo_owner<\/li>\n<li>index<\/li>\n<li>author<\/li>\n<li>status<\/li>\n<li>created_at<\/li>\n<\/ul>\n<p>Pk: (repo,repo_owner,index)<\/p>\n<p><strong>Star<\/strong><\/p>\n<ul>\n<li>username<\/li>\n<li>repo_name<\/li>\n<li>repo_owner<\/li>\n<li>created_at<\/li>\n<\/ul>\n<p>Pk: (username, repo_name, repo_owner)<\/p>\n<p><strong>Payment Info<\/strong><\/p>\n<ul>\n<li>id<\/li>\n<li>user_id<\/li>\n<li>verified<\/li>\n<\/ul>\n<p>Pk: id<\/p>\n<p>Now let&rsquo;s say the <code>UserProfile<\/code> is going to have the structure:<\/p>\n<p><strong>User Profile<\/strong><\/p>\n<ul>\n<li>email<\/li>\n<li>username<\/li>\n<li>starred_repos (many to many rel)\n<ul>\n<li>repo<\/li>\n<li>description<\/li>\n<\/ul>\n<\/li>\n<li>pull_requests (one to many rel)\n<ul>\n<li>repo<\/li>\n<li>index<\/li>\n<li>title<\/li>\n<li>created_at<\/li>\n<li>status<\/li>\n<\/ul>\n<\/li>\n<li>payment_verified<\/li>\n<\/ul>\n<p>In this case when we are consuming events form User, Pull Request and Star\ntable then we need to do join these streams and embed the pull request and\nstar information inside the <code>UserProfile<\/code> output stream.\nIf we start by using message keys as join keys here to join User and Pull\nRequest streams then the Pull Request has to be published with <code>author<\/code> field\nas a key.\nIn case we need to create another data product for pull requests\ndata depends on the repo and index we need to create another stream.<\/p>\n<p>Before that let&rsquo;s see what query join do we need to perform on these streams\nto get the result.<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-SQL\" data-lang=\"SQL\"><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">select<\/span> ...\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">from<\/span> users\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">left<\/span> <span style=\"color:#af3a03\">outer<\/span> <span style=\"color:#af3a03\">join<\/span> star\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">on<\/span> star.username <span style=\"color:#af3a03\">=<\/span> username\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">left<\/span> <span style=\"color:#af3a03\">outer<\/span> <span style=\"color:#af3a03\">join<\/span> pull_request\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">on<\/span> pull_request.author <span style=\"color:#af3a03\">=<\/span> username\n<\/span><\/span><span style=\"display:flex;\"><span><span style=\"color:#af3a03\">left<\/span> <span style=\"color:#af3a03\">outer<\/span> <span style=\"color:#af3a03\">join<\/span> payment_info\n<\/span><\/span><span style=\"display:flex;\"><span>    <span style=\"color:#af3a03\">on<\/span> payment_info.id <span style=\"color:#af3a03\">=<\/span> id<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>Note that in this query we only want to get a single record with <code>UserProfile<\/code>\nstructure.\nThis means to embed all pull requests into record as a list, and add\npayment_verified value as a single value.<\/p>\n<h2 class=\"heading\" id=\"solutions\">\n  Solutions\n  <a class=\"anchor\" href=\"#solutions\">#<\/a>\n<\/h2>\n<p>To give a solution I&rsquo;m focusing on providing something that can work at huge\nscale, imagine we are going to create a hundred data products so system should\nmake it very easy to onboard a new table.<\/p>\n<h3 class=\"heading\" id=\"solution-1-using-rdbms\">\n  Solution 1: Using RDBMS\n  <a class=\"anchor\" href=\"#solution-1-using-rdbms\">#<\/a>\n<\/h3>\n<p>I started with the idea of why not just moving all the computation on the database?\nAfter all the SQL language has nice features to do these transformations.<\/p>\n<p>\n\n\n\n\n\n\n\n<figure class=\"\">\n\n    \n    \n      \n        \n        <div class=\"img-container\" style=\"--w: 3510; --h: 2861;\">\n            <img loading=\"lazy\" alt=\"rdbms-solution\" src=\"https:\/\/glyphack.com\/stateful-stream-processing\/running-transformations-on-db.excalidraw_hu_88f142c6d8b42413.png\" width=\"3510\" height=\"2861\">\n        <\/div>\n      \n    \n\n    \n<\/figure>\n<\/p>\n<p>With this method we can have a generic application that reads streams and place\nthem in database under a table with the topic name.\nSo to onboard a new topic we have to feed this service two values:<\/p>\n<ul>\n<li>table name to insert data into<\/li>\n<li>table schema and mapping of event fields into table fields<\/li>\n<\/ul>\n<p>After we insert the data in the database we\n<a href=\"https:\/\/docs.aws.amazon.com\/AmazonRDS\/latest\/AuroraUserGuide\/AuroraMySQL.Integrating.Lambda.html\" rel=\"noopener\" target=\"_blank\">trigger<\/a> a lambda function to run the SQL query, get the data and embed the joins into a single document and produce message to Kafka.<\/p>\n<p>This lambda function requires these inputs:<\/p>\n<ul>\n<li>a mapping from table names to SQL queries<\/li>\n<\/ul>\n<p>After lambda is triggered it will lookup the mapping and run the SQL query. In our case it will get the user back with all matched pull request.\nNow we need a custom logic in lambda to embed all the matched Pull Request\nentities into <code>UserInfo<\/code> message.<\/p>\n<h4 class=\"heading\" id=\"benefits\">\n  Benefits\n  <a class=\"anchor\" href=\"#benefits\">#<\/a>\n<\/h4>\n<ul>\n<li>Implementation is very generic<\/li>\n<li>We have all the capabilities of SQL to do the processing<\/li>\n<li>Each topic can be joined with any other topic with any valid SQL condition<\/li>\n<\/ul>\n<h4 class=\"heading\" id=\"drawbacks\">\n  Drawbacks\n  <a class=\"anchor\" href=\"#drawbacks\">#<\/a>\n<\/h4>\n<ul>\n<li>The fact that we put all the computation on the database and specially join\nqueries will result in a huge amount of costs<\/li>\n<li>The architecture is not truly real-time, we should utilize indexes on\ndatabase to make sure query returns with low latency<\/li>\n<\/ul>\n<h3 class=\"heading\" id=\"solution-2-join-data-in-stream-processor\">\n  Solution 2: Join Data In Stream Processor\n  <a class=\"anchor\" href=\"#solution-2-join-data-in-stream-processor\">#<\/a>\n<\/h3>\n<p>To attempt to fix the previous solution drawbacks, a reasonable way can be moving the join logic to stream processor application.<\/p>\n<p>In this case the stream processor has a data store named state store that is used to keep track of latests state of records consumed.\nTo get a better understand of how the state store works, Imagine the scenario:<\/p>\n<ul>\n<li>A new pull request is created with information <code>{ title: &quot;PR&quot;, status: &quot;Open&quot;, ...}<\/code><\/li>\n<li>Stream processor consumes the message and insert it into the state store<\/li>\n<li>The pull request is updated and it&rsquo;s closed so another event is emitted with body <code>{ title: &quot;PR&quot;, status: &quot;Closed&quot;, ...}<\/code><\/li>\n<li>Stream processor consumes the new event updates the previous record in the state store using the primary key of the pull request.<\/li>\n<\/ul>\n<p>So at any given point the state store has latest state of an entity.<\/p>\n<p>We can leverage this state store to join events as they are consumed by creating 3 tables: <code>pull_request<\/code>, <code>user<\/code>, <code>user_info<\/code>.<\/p>\n<p>\n\n\n\n\n\n\n\n<figure class=\"\">\n\n    \n    \n      \n        \n        <div class=\"img-container\" style=\"--w: 3206; --h: 2676;\">\n            <img loading=\"lazy\" alt=\"join data in processor\" src=\"https:\/\/glyphack.com\/stateful-stream-processing\/join-in-processor-rdbms.excalidraw_hu_30a77b75267df9c1.png\" width=\"3206\" height=\"2676\">\n        <\/div>\n      \n    \n\n    \n<\/figure>\n\nThis is how the example procedure looks like:<\/p>\n<ol>\n<li>When a pull request event is consumed and inserted into a intermediate table<\/li>\n<li>When a user event is consumed, application queries the pull request table to check if there&rsquo;s any matched records with the join criteria. If there&rsquo;s any it creates the user_info message and saves it into database.<\/li>\n<li>Finally application also saves the <code>user<\/code> message in step 2 into database in case there&rsquo;s other joins for user table.<\/li>\n<\/ol>\n<p>Or in a case that the order of incoming events is not guaranteed when the user event comes in application joins it with the pull request table and creates the user_info message.\nHere also based on the join type we either publish a new <code>user_info<\/code> message or not, in this case this is a left join where the left is user entity:<\/p>\n<ol>\n<li>If user event is consumed even if it do not match any pull request the <code>user_info<\/code> message must be published.<\/li>\n<li>If pull_request event is consumed <code>user_info<\/code> message will only be published when pull request matches a user record, then the application updates the corresponding <code>user_info<\/code> message with the new pull request and publishes the message again.<\/li>\n<\/ol>\n<h4 class=\"heading\" id=\"benefits-1\">\n  Benefits\n  <a class=\"anchor\" href=\"#benefits-1\">#<\/a>\n<\/h4>\n<ul>\n<li>Ability to execute all SQL queries<\/li>\n<li>Can do complex joins<\/li>\n<li>Can index database on the fields that are queried for joins to decrease the processing time<\/li>\n<\/ul>\n<h4 class=\"heading\" id=\"drawbacks-1\">\n  Drawbacks\n  <a class=\"anchor\" href=\"#drawbacks-1\">#<\/a>\n<\/h4>\n<ul>\n<li>The RDBMS database is used for a lot of read\/write operations which can become slow and expensive<\/li>\n<li>The join logic is split in processing events from left and right side of the join.<\/li>\n<\/ul>\n<h3 class=\"heading\" id=\"using-another-database-as-state-store\">\n  Using Another Database as State Store\n  <a class=\"anchor\" href=\"#using-another-database-as-state-store\">#<\/a>\n<\/h3>\n<p>Following the idea of previous solution to move the join logic to application, we can make an adjustment to DB choice.\nThe pattern is inserting a record but retrieving it only by specific fields.\nUsually key value store databases provide fast read and writes with this limitation.<\/p>\n<p>Let&rsquo;s say like the previous method processor consumes events, and saves them inside a KV store like dynamoDB.\nNow it&rsquo;s important to specify a unique key for records so they don&rsquo;t overwrite each other, primary keys are:<\/p>\n<ol>\n<li>User: email<\/li>\n<li>Pull request: (repo,repo_owner,index)<\/li>\n<li>Payment Info: id<\/li>\n<\/ol>\n<p>When the application consumes a user event it&rsquo;s required to join this field with pull requests that have the same username.\nThis is not possible since our hash key is the PK, which is email in this case.<\/p>\n<p>Now we use some auxiliary tables to make this possible.\nWe create a table called <code>join_user_pr<\/code> with key being the join condition value(username) and value be the primary key\nof the user entity with that username, And similarly a table called <code>join_pr_user<\/code> for when joining an incoming PR event\nwith User.<\/p>\n<p>Application can use this table as a lookup table to do joins.\nAn example when <code>user<\/code> event is coming in:<\/p>\n<ol>\n<li>Save user in <code>Users<\/code> table and insert record <code>{user.username: user.email}<\/code> into <code>join_pr_user<\/code><\/li>\n<li>Query the <code>join_user_pr<\/code> with the username to get back primary keys of PRs which has this username.<\/li>\n<li>Query the <code>pull_reqest<\/code> table with primary key of PR and construct the <code>user_info<\/code> message and publish<\/li>\n<\/ol>\n<p>When <code>pull_request<\/code> event comes in:<\/p>\n<ol>\n<li>Save pull_request event in<code>pull_request<\/code> table and insert record <code>{pull_request.pk: pull_request.username}<\/code> into <code>join_user_pr<\/code>.<\/li>\n<li>Query the <code>join_pr_user<\/code> table with pull_request username to get back user email for that PR.<\/li>\n<li>Query user table with email and construct the <code>user_info<\/code> message and publish<\/li>\n<\/ol>\n<p>Since DynamoDB(and other KV stores) provide a fast way to read and write the process time is not going to grow in large\nvolume of data.\nBut we are creating duplicate data to be able to execute these queries, this can also be solved with\nSecondary Indexes in DynamoDB. Secondary Indexes can be fine for small number of joins but if we want to join the User\nwith 20 different tables the cost of User table with 20 SI will be high and more than having the auxiliary tables.<\/p>\n<h4 class=\"heading\" id=\"benefits-2\">\n  Benefits\n  <a class=\"anchor\" href=\"#benefits-2\">#<\/a>\n<\/h4>\n<ul>\n<li>Can support large volume of data<\/li>\n<li>Application can support multiple joins on a single record with low process time.(DynamoDB can provide 10ms read times)<\/li>\n<\/ul>\n<h4 class=\"heading\" id=\"drawbacks-2\">\n  Drawbacks\n  <a class=\"anchor\" href=\"#drawbacks-2\">#<\/a>\n<\/h4>\n<ul>\n<li>Data is duplicated<\/li>\n<li>Coordination of what tables and fields to query makes the application complex, the given example has 6 steps for a single join<\/li>\n<\/ul>\n<h3 class=\"heading\" id=\"leveraging-the-dynamodb-global-secondary-indexes\">\n  Leveraging the DynamoDB Global Secondary Indexes\n  <a class=\"anchor\" href=\"#leveraging-the-dynamodb-global-secondary-indexes\">#<\/a>\n<\/h3>\n<p>The previous solution was really close to achieve a very good result but still the 1 lookup and 1 query per join statement can cause\nlong process time in some cases.\nNow we&rsquo;ll try to exploit another feature of DynamoDB, Global Secondary Indexes, to solve this problem.<\/p>\n<p>Before that let&rsquo;s state what we exactly need from state store:<\/p>\n<ul>\n<li>Being able to get all records of a topic with given join condition, for example all pull_requests with a particular username<\/li>\n<\/ul>\n<p>We can use a single table for all the records to make this join possible and easier.\nImagine we have a table with a composite primary key with following definition:<\/p>\n<ul>\n<li>Partition Key: TABLE#{table_name}<\/li>\n<li>Sort Key: entity PK<\/li>\n<\/ul>\n<p>For example we insert all <code>user<\/code> records in the table with Partition Key of <code>TABLE#users<\/code> and Sort key of <code>user.email<\/code>,\nThe same happens for pull_request as well(we can concatenate multiple columns for the Sort Key).<\/p>\n<p>Now how can we join a user with all of it&rsquo;s pull_requests? Using a Global Secondary Index.\nLet&rsquo;s say we create a column called GSI_1(Global Secondary Index column names can be meaningless and be reused in DynamoDB),\nWe are going to use this column in both <code>user<\/code> and <code>pull_request<\/code> entities.\nFor users we insert the username value in the GSI_1 and for pull requests we insert the username in GSI_1.<\/p>\n<p>When a <code>user<\/code> event comes in the application will:<\/p>\n<ol>\n<li>Insert it in the <code>user<\/code> table and fills the GSI values according to joins specified.<\/li>\n<li>Query the <code>pull_request<\/code> table with the <code>GSI_1=={user.username}<\/code> condition, This will return all the PRs for that username<\/li>\n<li>Create the <code>user_info<\/code> message and publish<\/li>\n<\/ol>\n<p>When a <code>pull_request<\/code> event comes in the application will do the same things again the GSI_1 will be used and PR will be matched\nwith the correct user record.<\/p>\n<p>Now the problem will be managing the GSI fields we create. We need to create GSI fields for every single join that an entity has.\nIn our example if we want to join the user record with payment info we have to create a new GSI_2 field for the user and save the <code>user.id<\/code> in that field.\nBut for the <code>payment_info<\/code> record we can insert the <code>user.id<\/code> in the GSI_1 field because the only join Payment Info has is with the User.\nThis allows us to reuse the GSI field for multiple purpose so we don&rsquo;t hit the 20 GSI limit unless we have a table with 20 different joins.\nAlso we now can get back all pull requests for a user with a single query, this will enable us to perform multiple joins without loosing performance.<\/p>\n<p>To manage the GSI fields and what they are used for in each record we need another storage to save this information.\nWe can have a small PostgreSQL instance that knows for example in User records the GSI_1 field is used to join with Pull Request entity.<\/p>\n"},{"title":"How to Setup 2 Factor Authentication Code Generator on PC","link":"https:\/\/glyphack.com\/2fa-pc\/","pubDate":"Thu, 17 Mar 2022 23:19:14 +0330","guid":"https:\/\/glyphack.com\/2fa-pc\/","description":"<p>I use 2 factor authentication with almost all of my accounts, the only downside of this is that when I need to access something frequently or automate some task I have to manually enter this code from my phone.<\/p>\n<p>So I searched a bit and found these tools to make this process easier, I imported 2fa keys to my laptop and can generate keys with a command so I can copy the code and also the command can be used in automated tasks. Note that in this way anyone with access to your laptop has access to 2fa codes too.<\/p>\n<h2 class=\"heading\" id=\"setup-2fa-code-on-your-machine\">\n  Setup 2FA code on your machine\n  <a class=\"anchor\" href=\"#setup-2fa-code-on-your-machine\">#<\/a>\n<\/h2>\n<p>Here&rsquo;s the process:<\/p>\n<h3 class=\"heading\" id=\"1-export-2fa-accounts-from-google-authenticator\">\n  1. Export 2FA Accounts From Google Authenticator\n  <a class=\"anchor\" href=\"#1-export-2fa-accounts-from-google-authenticator\">#<\/a>\n<\/h3>\n<p>You probably have already setup your accounts with google authenticator. You can use the <a href=\"https:\/\/support.google.com\/accounts\/thread\/107807857\/how-to-export-2fa-codes-from-google-authenticator?hl=en\" rel=\"noopener\" target=\"_blank\">export option<\/a> to export the accounts you need. Export will be a qr code so you need a way to convert it to text. On mac I did it with <a href=\"https:\/\/apps.apple.com\/us\/app\/qr-journal\/id483820530?mt=12\" rel=\"noopener\" target=\"_blank\">qr journal<\/a> .<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-bash\" data-lang=\"bash\"><span style=\"display:flex;\"><span>brew install --cask qr-journal<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<h3 class=\"heading\" id=\"2-extract-account-secret-key\">\n  2. Extract Account Secret Key\n  <a class=\"anchor\" href=\"#2-extract-account-secret-key\">#<\/a>\n<\/h3>\n<p>Once you have the qr code as text you can use <a href=\"https:\/\/github.com\/scito\/extract_otp_secret_keys\" rel=\"noopener\" target=\"_blank\">extract_otp_secret_keys<\/a> to read the text and get the secret strings. save exported qr code in a text file and read it with like this:<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-bash\" data-lang=\"bash\"><span style=\"display:flex;\"><span>gh repo clone scito\/extract_otp_secret_keys\n<\/span><\/span><span style=\"display:flex;\"><span>python extract_otp_secret_keys\/extract_otp_secret_keys.py -p exported.txt<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>This will output the final secret key for accounts.<\/p>\n<h3 class=\"heading\" id=\"3-install-a-2fa-code-generator\">\n  3. Install a 2FA Code Generator\n  <a class=\"anchor\" href=\"#3-install-a-2fa-code-generator\">#<\/a>\n<\/h3>\n<p>I used the <a href=\"https:\/\/github.com\/rsc\/2fa\" rel=\"noopener\" target=\"_blank\">2fa<\/a> app to import accounts in terminal.<\/p>\n<div class=\"code-block\">\n  <div class=\"highlight\"><pre tabindex=\"0\" style=\"color:#3c3836;background-color:#fbf1c7;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"><code class=\"language-bash\" data-lang=\"bash\"><span style=\"display:flex;\"><span>go install rsc.io\/2fa@latest\n<\/span><\/span><span style=\"display:flex;\"><span>2fa -add account_name<\/span><\/span><\/code><\/pre><\/div>\n  <button class=\"copy-code-button\">copy<\/button>\n<\/div>\n<p>Now that you have this you can use the command <code>2fa<\/code> to get all of your accounts 2FA codes or <code>2fa account_name<\/code> to get the 2FA code for a specific account, the latter is useful when writing scripts.<\/p>\n"},{"title":"Fixing One Bug Leads to Another","link":"https:\/\/glyphack.com\/fixing-one-bug-leads-to-another\/","pubDate":"Sat, 05 Feb 2022 09:49:50 +0330","guid":"https:\/\/glyphack.com\/fixing-one-bug-leads-to-another\/","description":"<p>One thing that frequently frustrates me when I&rsquo;m working is sloppy work. I&rsquo;m using the word &ldquo;work&rdquo; here because it can refer to anything, but here I&rsquo;ll talk about sloppy code.<\/p>\n<p>Recently, someone from my previous team asked me to help them fix an issue on a system while the maintainer was not available. At first, I discussed this with my team lead and didn&rsquo;t get involved in the issue, but after a week, the problem was still there, and they asked me for my help again. So I finally decided to give it a try.<\/p>\n<h2 class=\"heading\" id=\"the-story\">\n  The story\n  <a class=\"anchor\" href=\"#the-story\">#<\/a>\n<\/h2>\n<p>The issue was that one of the microservices was consuming messages twice from a Kafka topic. Kafka client consumes a message in this order:<\/p>\n<ol>\n<li>Read a new message<\/li>\n<li>Process the message<\/li>\n<li>Commit the message\nWorking with at least one delivery model where each message is delivered to you at least once, duplicates might occur in your system when a message is delivered twice or more.\nAn example scenario of a duplicate message problem that can occur here is when your application reads the message, updates some business entity in the database, and fails before committing the message. You would see the message coming back again.<\/li>\n<\/ol>\n<p>To overcome this issue, the application has to make the consumer idempotent with <a href=\"https:\/\/chrisrichardson.net\/post\/microservices\/patterns\/2020\/10\/16\/idempotent-consumer.html#:~:text=Specifically%2C%20if%20Apache%20Kafka%20invokes,execute%20the%20database%20transaction%20repeatedly\" rel=\"noopener\" target=\"_blank\">idempotent consumer pattern<\/a><\/p>\n<blockquote>\n<p>Read message\nBegin database transaction\nINSERT into PROCESSED_MESSAGE (subscriberId, ID) VALUES(subscriberId, message.ID)\nUpdate one or more business objects\nCommit transaction\nAcknowledge message\nAfter starting the database transaction, the message handler inserts the message\u2019s ID into the <code>PROCESSED_MESSAGE<\/code> table. Since the <code>(subscriberId, messageID)<\/code> is the <code>PROCESSED_MESSAGE<\/code> table\u2019s primary key the <code>INSERT<\/code> will fail if the message has been processed successfully. The message handler can then abort the transaction and acknowledge the message.<\/p>\n<\/blockquote>\n<p>So I checked if the code was idempotent or not, and hopefully, it had. With several problems:<\/p>\n<p><strong>Sloppy code<\/strong>,\nThe steps were in the wrong order. The insert in <code>PROCESSED_MESSAGE<\/code> was the last step. It should be the first action because you want to assume the message is processed, and if any error occurs, the transaction will fail. The insert will be reverted, so you don&rsquo;t have to manually decide where to mark the message as read in the code process flow.\nAlso, The <code>messageID<\/code> was not unique, so two concurrent inserts of the same message would not cause any issue, while it should. It&rsquo;s always better to crash than to be inconsistent; the latter will take much more time to find out and recover.<\/p>\n<p><strong>Not knowing about the business domain of your application<\/strong>,\nThe third problem; the message ID was generated by the whole body of the message to string, literally a TextField in PostgresSQL for deduplication and searching! While it was inefficient and expensive to use your database for matching full text for deduplication. It&rsquo;s also not working if you change the format of the same message(like updating the schema).\nI can guess that this happens because developers can be far from the business domain, but you should always understand why you are writing a piece of code. By asking questions like what unique values are we looking for in the message? From the product manager, you can use this information to do the deduplication for that use case.<\/p>\n<p>I fixed the algorithm and changed the table schema to have a better deduplication key by discussing this with business people to discover the unique values in this message. Everything was done, and I started testing the code; while I was doing that, I checked the logs and realized that the application was consuming messages from 2 weeks ago. This should not happen when we are committing the messages. I dug more into the issue and found out the duplication bug was part of a more significant issue. Kafka consumer was not committing the messages correctly, so it caused duplicates even after a successful message process. And even worse than that, the application was not failing if the commit failed; it silently ignored the issue. Until that moment, the deduplication logic was not only deduplication but serving as Kafka offset manager!<\/p>\n<p>After realizing that it was not easy to fix and needed proper investigation, I handed it to the original maintainer. But I did investigate why this issue was happening just for my curiosity and ended up with this hypothesis.\nThe issue was that Kafka has a <a href=\"https:\/\/docs.confluent.io\/platform\/current\/installation\/configuration\/consumer-configs.html#consumerconfigs_max.poll.interval.ms\" rel=\"noopener\" target=\"_blank\">max poll interval<\/a> configuration which determines how long it takes for the consumer to poll a message(read a new message from the topic). If a consumer reaches this time limit before polling a message, it will be considered unhealthy and replaced with a new consumer. Meanwhile, the consumer has a timeout to commit all the pending messages to offset, and if it cannot, then those messages are not considered as done and will be consumed again.<\/p>\n<p>The point here is that the second bug was buried under the first issue, and I brought it up by fixing the first one. Kind of a rabbit hole, and who knows, I would not face <a href=\"https:\/\/en.wikipedia.org\/wiki\/Lernaean_Hydra\" rel=\"noopener\" target=\"_blank\">Hydra<\/a> if I continued fixing the problem?\n\n\n\n\n\n\n\n\n<figure class=\"\">\n\n    \n    \n      \n        \n        <div class=\"img-container\" style=\"--w: 1000; --h: 513;\">\n            <img loading=\"lazy\" alt=\"Hydra, a see monster that that as soon as one head was cut off, two more heads would emerge from the fresh wound\" src=\"https:\/\/glyphack.com\/fixing-one-bug-leads-to-another\/Hydra_hu_1dd545634e2870fa.png\" width=\"1000\" height=\"513\">\n        <\/div>\n      \n    \n\n    \n<\/figure>\n<\/p>\n<h2 class=\"heading\" id=\"yak-shaving\">\n  Yak shaving\n  <a class=\"anchor\" href=\"#yak-shaving\">#<\/a>\n<\/h2>\n<p>The more precise term about this situation is <a href=\"https:\/\/seths.blog\/2005\/03\/dont_shave_that\/\" rel=\"noopener\" target=\"_blank\">Yak Shaving<\/a>. It&rsquo;s a situation where you have to fix something else before working on the current issue.\nIt is essential to be ready to go down this path before actually starting. For me, it was always a frustrating experience because You can&rsquo;t get it done, and estimations become incorrect, then you have no clue where you are until you can resolve the final issue.<\/p>\n<h2 class=\"heading\" id=\"what-did-i-learn\">\n  What did I learn\n  <a class=\"anchor\" href=\"#what-did-i-learn\">#<\/a>\n<\/h2>\n<p>Some bug fixes seem easy to solve, like this one. You and the others might have no idea why some issue is happening. When a request like this comes to me from now on, I&rsquo;ll be asking for a time to investigate the issue and how it is happening. It might take more time; in this case, I only had a couple of days to work on it, so I would not have a chance to fix the issue.\nI&rsquo;m not saying all the bug fixes should be like this; if you worked with a system and know why something is wrong and are sure about the system behavior based on experience, go ahead and work on the fix. But for me, there were a lot of changes to the system after I left the team, so the system was not what I used to know.<\/p>\n<p>And also, don&rsquo;t forget the chance of ending up in a Yak Shaving. Investigate enough to see if anything else is wrong except the current issue.\nAs said in pragmatic programmer<\/p>\n<blockquote>\n<p>Is the problem being reported a direct result of the underlying bug, or merely a symptom?<\/p>\n<\/blockquote>\n<p>It&rsquo;s tempting to help others with something you are sure you can. But the bottom line is if someone asks for help, they expect the issue to be resolved, and this kind of help is not helpful anyway.<\/p>\n"},{"title":"How Write and Organize Software Documentation","link":"https:\/\/glyphack.com\/write-software-documentation\/","pubDate":"Sun, 23 Jan 2022 17:30:40 +0330","guid":"https:\/\/glyphack.com\/write-software-documentation\/","description":"<p>Software documents play an important role in software development, everyone pays attention to writing documentation but just like writing code, writing the text is not enough but it has to be readable and understandable for other people, after all the purpose of documenting is to communicate. Recently I was reading documentation of a system and noticed that there&rsquo;s something  strange about it. although all the components are documented, but still it&rsquo;s not easy to find information on something.<\/p>\n<p>Let&rsquo;s review the documentation role in these two situations, onboarding a new person with the system and improving collaboration between a team. This comes down to these two questions:<\/p>\n<ol>\n<li>Can I hand over my technical docs to someone and expect them to have all information they need to make their first commit?<\/li>\n<li>If situation X happens, do my teammates have to call someone to know what is happening?\nSo every day, when one of these happens, think about the answers and see how well your documentation is. For example, when you get similar questions from teammates about a system, think of it as a failure in the documentation system, not all questions can be answered inside docs, but the frequent ones must be answered.<\/li>\n<\/ol>\n<h2 class=\"heading\" id=\"avoid-common-problems-of-technical-software-documentation\">\n  Avoid Common Problems of Technical Software Documentation\n  <a class=\"anchor\" href=\"#avoid-common-problems-of-technical-software-documentation\">#<\/a>\n<\/h2>\n<p>So what are the problems that can be in your technical notes?<\/p>\n<h3 class=\"heading\" id=\"non-existent-documents\">\n  Non existent documents\n  <a class=\"anchor\" href=\"#non-existent-documents\">#<\/a>\n<\/h3>\n<h4 class=\"heading\" id=\"no-entry-point-link-around-a-topic\">\n  No Entry Point Link Around a Topic\n  <a class=\"anchor\" href=\"#no-entry-point-link-around-a-topic\">#<\/a>\n<\/h4>\n<p>There is no primary documentation around a topic; you have to pass multiple links when someone is on boarded or read a single document after cloning the repo.<\/p>\n<h4 class=\"heading\" id=\"cover-all-information-needed-for-a-project\">\n  Cover all information needed for a project\n  <a class=\"anchor\" href=\"#cover-all-information-needed-for-a-project\">#<\/a>\n<\/h4>\n<p>If someone joins your team, they need information on how a system works. They also have to know how to set up their local and sandbox environment and use it.<\/p>\n<h3 class=\"heading\" id=\"hidden-documents\">\n  Hidden Documents\n  <a class=\"anchor\" href=\"#hidden-documents\">#<\/a>\n<\/h3>\n<h4 class=\"heading\" id=\"nesting-overuse\">\n  Nesting Overuse\n  <a class=\"anchor\" href=\"#nesting-overuse\">#<\/a>\n<\/h4>\n<p>Having sub-pages inside sub-pages makes the text to be scattered in different documents. Suppose someone reads a note on compiling a project. They also have to find the testing guide and deployment guide next to that text. It can be another section within that doc or a document next to it. Just make sure you don&rsquo;t need to browse again to find those.\nThis pattern is similar to coupling the code that is related to each other like putting it inside a single module.<\/p>\n<h4 class=\"heading\" id=\"multiple-channels\">\n  Multiple channels\n  <a class=\"anchor\" href=\"#multiple-channels\">#<\/a>\n<\/h4>\n<p>Some materials might be found on the slack channel, others on your documentation tool. Keep all of them in one place as much as possible.<\/p>\n<h3 class=\"heading\" id=\"inaccurate-documents\">\n  Inaccurate Documents\n  <a class=\"anchor\" href=\"#inaccurate-documents\">#<\/a>\n<\/h3>\n<p>If the documents are not updated along with the code, they become inaccurate as you change the software. One possible solution is to link the code and document together(by using Readme file) so more people will see the document while writing code.\nHere you can see that why is it valuable to have a single document for a topic, for example here if you have 4 documents that has to change after changing system codes then it&rsquo;s much harder to update the documents that a single document.<\/p>\n<h3 class=\"heading\" id=\"obsolete-documents\">\n  Obsolete Documents\n  <a class=\"anchor\" href=\"#obsolete-documents\">#<\/a>\n<\/h3>\n<p>If a design is changed and older design decisions don&rsquo;t apply anymore, archive them.<\/p>\n<h3 class=\"heading\" id=\"afterthought-documents\">\n  Afterthought Documents\n  <a class=\"anchor\" href=\"#afterthought-documents\">#<\/a>\n<\/h3>\n<p>This is a problem because if you write your docs after delivering a project, you will end up:<\/p>\n<ol>\n<li>Forgetting to include important notes on the topic because you are in the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Curse_of_knowledge\" rel=\"noopener\" target=\"_blank\">curse of knowledge<\/a>.<\/li>\n<li>Have to explain the project to someone personally if you need to hand it over in the middle.<\/li>\n<\/ol>\n<h2 class=\"heading\" id=\"how-to-make-technical-documentations-better\">\n  How to Make Technical Documentations Better\n  <a class=\"anchor\" href=\"#how-to-make-technical-documentations-better\">#<\/a>\n<\/h2>\n<p>It&rsquo;s essential to organize documents so that it&rsquo;s visible to everyone.<\/p>\n<p><strong>Be Careful with Nesting<\/strong>\nWhen newcomers open the documentation, they should locate all the required information they need to work on the project. This means in your top-level page you should have all the topics you want to explain visible there as links or sub-pages.\nAs an example of a good technical documentation checkout <a href=\"https:\/\/wiki.crdb.io\/wiki\/spaces\/CRDB\/overview\" rel=\"noopener\" target=\"_blank\">CockroachDB documents<\/a>, There are all the things from introduction to deploying the project listed on the first page with only 1 level nesting.\nMy personal suggestion about this is choose a topic and write all the relevant information to that topic in the page. if a sub topic grows over time you can create a sub-page for it later.<\/p>\n<p><strong>Make it like a story<\/strong>\nIt should be easy to follow your documents from beginning to the end. Make sure the topics are in the right order. Keep similar topics in different contexts separate, e.g. keep end-user documents separate from system specification documents but link them because the developer needs to know about the user when writing docs.\nThis means all the journey from finding the repo to clone to deploying a feature can be found inside technical docs.<\/p>\n<p><strong>Make Documentation Writing an Ongoing process<\/strong>\nIt&rsquo;s hard to keep documents that no one reads updated, make sure your documents have users by sending people document links instead of answering questions in Slack.<\/p>\n<p><strong>Writing Software Documentation is a Collaborative Process<\/strong>\nWrite documents with the mindset of other people are going to use them. so you should considering:<\/p>\n<ul>\n<li>Ask your team to review the docs you write<\/li>\n<li>Put new pages you create on draft<\/li>\n<\/ul>\n<h2 class=\"heading\" id=\"other-resources\">\n  Other resources\n  <a class=\"anchor\" href=\"#other-resources\">#<\/a>\n<\/h2>\n<ul>\n<li>\n<p><a href=\"https:\/\/microsoft.github.io\/code-with-engineering-playbook\/documentation\/guidance\/project-and-repositories\/\" rel=\"noopener\" target=\"_blank\">Documenting code repositories<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"https:\/\/blog.prototypr.io\/software-documentation-types-and-best-practices-1726ca595c7f\" rel=\"noopener\" target=\"_blank\">Type of documentations<\/a><\/p>\n<\/li>\n<\/ul>\n"},{"title":"How I Stay Focused and Manage Time","link":"https:\/\/glyphack.com\/how-i-stay-focused-and-manage-time\/","pubDate":"Mon, 03 Jan 2022 19:10:52 +0330","guid":"https:\/\/glyphack.com\/how-i-stay-focused-and-manage-time\/","description":"<p>A few weeks ago, I experienced one of these busy weeks with lots of meetings, Slack messages, working on different tasks. Although I did many things, I didn&rsquo;t feel like getting anything done. So I decided to see how can I improve this.<\/p>\n<p>First of all, the cost of getting interrupted while developing is high. You need time to get back on what you were doing, and if this happens a lot, you can&rsquo;t manage to get many things done.\nThe second thing I noticed was that I have to do better with planning for my day. I have a mix of personal stuff, university, and work. Some days, I&rsquo;m constantly switching between tasks with different contexts, and I get myself out of the zone.<\/p>\n<h1 class=\"heading\" id=\"my-setup\">\n  My setup\n  <a class=\"anchor\" href=\"#my-setup\">#<\/a>\n<\/h1>\n<p>I&rsquo;ve experimented with different productivity tools for focus and time management tools. I was looking for a simple enough task manager with a built-in time tracker that you can decide on something you must do in the day and track your time while doing them for the rest of the day. I could not find such a tool, so I combined different apps to achieve this.<\/p>\n<h3 class=\"heading\" id=\"daily-planning\">\n  Daily Planning\n  <a class=\"anchor\" href=\"#daily-planning\">#<\/a>\n<\/h3>\n<p>I use <a href=\"https:\/\/www.notion.so\/\" rel=\"noopener\" target=\"_blank\">Notion<\/a> as my general notebook to organize projects, learn lists, and other random things that gather here, not to forget them.\nI also started to use it as my time management tool just by creating a table where each row is a date, and inside that, I write my day plan.\nOf course, it does not offer any unique feature here that other apps don&rsquo;t. I prefer to write down my tasks as raw notes instead of task management apps to keep them simple and easy enough to do every day.\nThis way, it helps me two organize my tasks such that:<\/p>\n<ul>\n<li>I know what I want to do upfront, so I can plan to do them appropriately. For example, I can do all of my university stuff together to reduce context switching costs<\/li>\n<li>If I can&rsquo;t manage to do something on that day, it&rsquo;s not lost, and I can move it to the next day<\/li>\n<\/ul>\n<h3 class=\"heading\" id=\"time-tracking\">\n  Time Tracking\n  <a class=\"anchor\" href=\"#time-tracking\">#<\/a>\n<\/h3>\n<p>I use <a href=\"https:\/\/toggl.com\/\" rel=\"noopener\" target=\"_blank\">Toggl<\/a> to track how much time I spend on a task and focus while working.\nToggl has a concept of clients and tasks, I defined my clients as DataChef, University and Personal stuff and tasks are my current work, whenever I decide to work on something I start the timer choose the client and start working. The remarkable feature is that they have a Pomodoro timer too, and I use this technique to focus on my job.\nIf you&rsquo;re not familiar with Pomodoro, here is the procedure from Wikipedia<\/p>\n<blockquote>\n<ol>\n<li>Decide on the task to be done.<\/li>\n<li>Set the Pomodoro timer (typically for 25 minutes).<\/li>\n<li>Work on the task.<\/li>\n<li>End work when the timer rings and take a short break (typically 5\u201310 minutes).<\/li>\n<li>If you have fewer than three Pomodoros, go back to Step 2 and repeat until you go through all three Pomodoros.<\/li>\n<li>After three Pomodoros are done, take the fourth Pomodoro and then take an extended break (traditionally 20 to 30 minutes). Once the long break is finished, return to step 2.<\/li>\n<\/ol>\n<\/blockquote>\n<p>I prefer it over other Pomodoro apps because the features it provides, such as reminders, help me to continuously not lose focus and track time.\nI started using Toggl with 25 min focus 5 min break and increased the time to 45 min focus in just two weeks!<\/p>\n<h1 class=\"heading\" id=\"conclusion\">\n  Conclusion\n  <a class=\"anchor\" href=\"#conclusion\">#<\/a>\n<\/h1>\n<p>I think task management and time tracking are not only for teams, but people can also boost their productivity with these methods.\nI did not go over all the features of the tools I mentioned because it&rsquo;s not the tools that are important but the techniques you put in place to help yourself with focus and time management.<\/p>\n"}]}}