{"@attributes":{"version":"2.0"},"channel":{"title":"Blog","link":"https:\/\/blog.oliviaappleton.com","description":"Olivia Appleton-Crocker's personal blog","copyright":"\u00a9 2020-2026, Olivia Appleton-Crocker","docs":"http:\/\/www.rssboard.org\/rss-specification","generator":"python-feedgen","language":"en","lastBuildDate":"Mon, 09 Mar 2026 22:16:54 +0000","item":[{"title":"Lessons in Grafana - Part Three: Mental Metadata","link":"https:\/\/blog.oliviaappleton.com\/posts\/0009-lessons-in-grafana-03.html","description":"<a rel='me' href='https:\/\/transfem.social\/@LivInTheLookingGlass' role='none' class='invisible'><\/a>\n<script>\n    window.goatcounter = {\n        path: function() {\n            let p = location.pathname;\n            if (p == '\/') {\n                p = '\/index.html';\n            }\n            return '\/blog' + p;\n        }\n    }\n<\/script>\n<script data-goatcounter='https:\/\/livinthelookingglass.goatcounter.com\/count'\n async src='\/\/gc.zgo.at\/count.js'><\/script><section id=\"lessons-in-grafana-part-three-mental-metadata\">\n<h1>Lessons in Grafana - Part Three: Mental Metadata<a class=\"headerlink\" href=\"#lessons-in-grafana-part-three-mental-metadata\" title=\"Link to this heading\">\u00b6<\/a><\/h1>\n<p><span class=\"date\">2026-03-09<\/span><\/p>\n<p>Welcome back to the main part of the series. Today we will be discussing mental health, and the difficulties of collecting data on it. If you\u2019d rather not read that, then come back next week for a discussion of time tracking.<\/p>\n<ul class=\"series-nav-max-3 series-nav simple\">\n<li><p><strong class=\"series-nav-label\">1)<\/strong><span> <\/span><a class=\"reference external\" href=\"\/posts\/0006-lessons-in-grafana-01\">A Vision<\/a><\/p><\/li>\n<li><p><strong class=\"series-nav-label\">2)<\/strong><span> <\/span><a class=\"reference external\" href=\"\/posts\/0007-lessons-in-grafana-02\">Litter Logs<\/a><\/p><\/li>\n<li><p><strong class=\"series-nav-label\">2.5)<\/strong><span> <\/span><a class=\"reference external\" href=\"\/posts\/0008-lessons-in-grafana-02-5\">Achieving Acceptable Averages<\/a><\/p><\/li>\n<li><p><strong class=\"series-nav-label\">3)<\/strong><span> <\/span>Mental Metadata (you are here)<\/p><\/li>\n<li><p><strong class=\"series-nav-label\">4)<\/strong><span> <\/span>Noticing Notes<\/p><\/li>\n<li><p><strong class=\"series-nav-label\">5)<\/strong><span> <\/span>Graphing Goatcounter<\/p><\/li>\n<li><p><strong class=\"series-nav-label\">6)<\/strong><span> <\/span>Bridging Gadgets? Biometrics Galore!<\/p><\/li>\n<li><p><strong class=\"series-nav-label\">7)<\/strong><span> <\/span>Plotting Prospective Plans<\/p><\/li>\n<\/ul>\n<section id=\"prelude-and-inspiration\">\n<h2>Prelude and Inspiration<a class=\"headerlink\" href=\"#prelude-and-inspiration\" title=\"Link to this heading\">\u00b6<\/a><\/h2>\n<p>For a long time, I was frustrated whenever I went to my psychiatrist. They would ask me very reasonable questions like \u201chas this medicine improved things?\u201d, and I could not give an answer with certainty. My therapist would point out that I act as if things haven\u2019t gotten better, when from their perspective there had been clear improvements.<\/p>\n<p>This bias was my initial motivation for this project. I wanted to track depression data over time. I wanted to <em>see<\/em> that things had improved, so I could not ignore that. I wanted to be able to point to a chart and say \u201cyes, doctor, this medication <em>has<\/em> improved things, you can see here where I started and things got better.\u201d Because I am a person who needs side projects to thrive, it spiraled into all <em>this<\/em>. I\u2019m pretty happy about that.<\/p>\n<p>Showing progress on this to friends led to them asking how things might be added. Several wanted to use something like this for their own purposes. In the process of listening to that feedback, I added two things that will be featured very prominently in this post: support for tracking anxiety\/PTSD symptoms, and support for metadata about <a class=\"reference external\" href=\"https:\/\/en.wikipedia.org\/wiki\/Dissociative_identity_disorder\">Dissociative Identity Disorder<\/a> and <a class=\"reference external\" href=\"https:\/\/en.wikipedia.org\/wiki\/Plural_identity\">Plurality<\/a> in general. Because these topics are heavily stigmatized, I once again emphasize: <em>all data in this series is faked and arbitrary<\/em>. They do not correspond to real people and I will not be identifying anyone who utilizes these features.<\/p>\n<\/section>\n<section id=\"depression-tracking\">\n<h2>Depression Tracking<a class=\"headerlink\" href=\"#depression-tracking\" title=\"Link to this heading\">\u00b6<\/a><\/h2>\n<p>Depression is famously difficult to quantify <span id=\"id1\">[<a class=\"reference internal\" href=\"#id120\" title=\"Alex J Mitchell. Why do clinicians have difficulty detecting depression? In Screening for Depression in Clinical Practice: An Evidence-Based Guide. Oxford University Press, 12 2009. URL: https:\/\/doi.org\/10.1093\/oso\/9780195380194.003.0006, arXiv:https:\/\/academic.oup.com\/book\/0\/chapter\/354484514\/chapter-pdf\/43715552\/isbn-9780195380194-book-part-6.pdf, doi:10.1093\/oso\/9780195380194.003.0006.\">Mit09<\/a>, <a class=\"reference internal\" href=\"#id110\" title=\"Nicole Beaulieu Perez, Paula Gordillo Sierra, Brittany Taylor, and Jason Fletcher. Selecting a depression measure for research: a critical examination of five common self-report scales. Journal of Affective Disorders, 394:120542, 2026. URL: https:\/\/www.sciencedirect.com\/science\/article\/pii\/S0165032725019846, doi:https:\/\/doi.org\/10.1016\/j.jad.2025.120542.\">PSTF26<\/a>]<\/span>. Not only is it an inherently subjective experience, it also is good at disguising itself. To anthropomorphize: depression is a shifty liar. It doesn\u2019t just make you think that things are bad, but that they were <em>always<\/em> bad and will <em>never<\/em> get better. Rather than making you feel tired, it can make you feel like you simply have a smaller energy bar in the first place. Measuring something that, by its own nature, tricks you into believing your experience is <em>normal<\/em> requires questions that reference external factors.<\/p>\n<p>Of course, those are only correlates. You can get <em>better<\/em> correlations if you ask just the right questions, but there is an amount of error inherent to the fact that you\u2019re measuring the subjective. This has resulted in a vast quantity of questionnaires. You\u2019ll see variations of them at doctors\u2019 offices, on intake forms, or on population surveys.<\/p>\n<p>Because this project was something I had tried before, years ago, I had some idea what I would need to make this work. I was willing to settle for high margin of error, because that\u2019s better than no data at all. The priority for the collection phase was reducing friction at every possible step. The survey should be sent to me without my input. It should require no more than one button press per question. It should minimize the number of questions asked, and allow any with consistent answers to be skipped and inferred.<\/p>\n<p>Because speed of implementation was the biggest priority, I made two immediate decisions:<\/p>\n<ol class=\"arabic simple\">\n<li><p>I would design my data collection bot such that it didn\u2019t care what survey I was asking, and<\/p><\/li>\n<li><p>I needed to use the shortest survey (after eliminating questions) I could find quickly<\/p><\/li>\n<\/ol>\n<section id=\"what-is-the-phq-9\">\n<h3>What Is The PHQ-9?<a class=\"headerlink\" href=\"#what-is-the-phq-9\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>The PHQ-9 <span id=\"id2\">[<a class=\"reference internal\" href=\"#id134\" title=\"National HIV Curriculum. Patient health questionnaire-9 (phq-9). URL: https:\/\/www.hiv.uw.edu\/page\/mental-health-screening\/phq-9 (visited on 2026-02-21).\">Cur<\/a>]<\/span> is a self-reported questionnaire consisting of 9 questions. Each of them is assigned a value between 0 and 3, then totaled. The questions are as follows:<\/p>\n<div class=\"table-wrapper docutils container\">\n<table class=\"docutils align-default\">\n<thead>\n<tr class=\"row-odd\"><th class=\"head\"><p>Over the last 2 weeks, how often have you been bothered by the following problems?<\/p><\/th>\n<th class=\"head\"><p>Not at all<\/p><\/th>\n<th class=\"head\"><p>Several Days<\/p><\/th>\n<th class=\"head\"><p>More than half the days<\/p><\/th>\n<th class=\"head\"><p>Nearly every day<\/p><\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr class=\"row-even\"><td><p>Little interest or pleasure in doing things<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>+1<\/p><\/td>\n<td><p>+2<\/p><\/td>\n<td><p>+3<\/p><\/td>\n<\/tr>\n<tr class=\"row-odd\"><td><p>Feeling down, depressed, or hopeless<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>+1<\/p><\/td>\n<td><p>+2<\/p><\/td>\n<td><p>+3<\/p><\/td>\n<\/tr>\n<tr class=\"row-even\"><td><p>Trouble falling asleep, staying asleep, or sleeping too much<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>+1<\/p><\/td>\n<td><p>+2<\/p><\/td>\n<td><p>+3<\/p><\/td>\n<\/tr>\n<tr class=\"row-odd\"><td><p>Feeling tired or having little energy<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>+1<\/p><\/td>\n<td><p>+2<\/p><\/td>\n<td><p>+3<\/p><\/td>\n<\/tr>\n<tr class=\"row-even\"><td><p>Poor appetite or overeating<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>+1<\/p><\/td>\n<td><p>+2<\/p><\/td>\n<td><p>+3<\/p><\/td>\n<\/tr>\n<tr class=\"row-odd\"><td><p>Feeling bad about yourself - or that you\u2019re a failure or have let yourself or your family down<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>+1<\/p><\/td>\n<td><p>+2<\/p><\/td>\n<td><p>+3<\/p><\/td>\n<\/tr>\n<tr class=\"row-even\"><td><p>Trouble concentrating on things, such as reading the newspaper or watching television<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>+1<\/p><\/td>\n<td><p>+2<\/p><\/td>\n<td><p>+3<\/p><\/td>\n<\/tr>\n<tr class=\"row-odd\"><td><p>Moving or speaking so slowly that other people could have noticed. Or, the opposite - being so fidgety or restless that you have been moving around a lot more than usual<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>+1<\/p><\/td>\n<td><p>+2<\/p><\/td>\n<td><p>+3<\/p><\/td>\n<\/tr>\n<tr class=\"row-even\"><td><p>Thoughts that you would be better off dead or of hurting yourself in some way<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>+1<\/p><\/td>\n<td><p>+2<\/p><\/td>\n<td><p>+3<\/p><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p>This is quite comprehensive, yet can be answered fairly quickly. But 9 questions? Surely we can skip some of these.<\/p>\n<\/section>\n<section id=\"further-compressing\">\n<h3>Further Compressing<a class=\"headerlink\" href=\"#further-compressing\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>Some of these questions are easy to eliminate. I\u2019ve had sleep inconsistencies my whole life, for example. It just seems to run in the family. That makes it a fairly bad proxy for depression except in the very short term, where there will already be the highest error.<\/p>\n<p>Given that there will be questions like that, let\u2019s write a format that we can store these questionnaires in. We\u2019ll start with the very basics: we want an interval to ask the survey at, and we want a bank of questions.<\/p>\n<div class=\"highlight-javascript notranslate\" id=\"lgbot-questions-phq9-json-partial-1\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"p\">{<\/span>\n<\/span><span data-line=\"2\"><span class=\"w\">  <\/span><span class=\"s2\">&quot;randomIntervalHours&quot;<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">[<\/span><span class=\"mf\">2<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"mf\">12<\/span><span class=\"p\">],<\/span>\n<\/span><span data-line=\"3\"><span class=\"w\">  <\/span><span class=\"s2\">&quot;questions&quot;<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">[]<\/span>\n<\/span><span data-line=\"4\"><span class=\"p\">}<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<p>Good! That solves our problem perfectly. Now, what does it look like when we want a question to be asked? Well, one worry that I had from my last attempt at this was about phrasing. If I make it so that the question is identical each time, then that makes your answers habitual. You want to minimize the friction of asking, but if you aren\u2019t careful you\u2019ll find that process automatic. This means not only do we need different phrasings of each question, we also need to shuffle the answers each time we ask. We want you to be forced to actually read the button you\u2019re pushing.<\/p>\n<p>In addition to that, we need to have an indicator that it should be displayed. Otherwise, how do we know when something is to be skipped? And, of course, we need to hold the answer bank. This approach generalizes well.<\/p>\n<div class=\"highlight-javascript notranslate\" id=\"lgbot-questions-phq9-json-partial-2\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"p\">{<\/span>\n<\/span><span data-line=\"2\"><span class=\"w\">  <\/span><span class=\"s2\">&quot;question&quot;<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">[<\/span>\n<\/span><span data-line=\"3\"><span class=\"w\">    <\/span><span class=\"s2\">&quot;In the last two weeks, how often have you had little interest or pleasure in doing things?&quot;<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"4\"><span class=\"w\">    <\/span><span class=\"s2\">&quot;In the last two weeks, how often have you lacked motivation or enjoyment in your usual daily tasks?&quot;<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"5\"><span class=\"w\">    <\/span><span class=\"s2\">&quot;En las \u00faltimas dos semanas, \u00bfqu\u00e9 tanto has sentido que no quieres hacer nada o que nada te da gusto?&quot;<\/span>\n<\/span><span data-line=\"6\"><span class=\"w\">  <\/span><span class=\"p\">],<\/span>\n<\/span><span data-line=\"7\"><span class=\"w\">  <\/span><span class=\"s2\">&quot;display&quot;<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"kc\">true<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"8\"><span class=\"w\">  <\/span><span class=\"s2\">&quot;answers&quot;<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">{<\/span>\n<\/span><span data-line=\"9\"><span class=\"w\">    <\/span><span class=\"s2\">&quot;Not at all&quot;<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"mf\">0<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"10\"><span class=\"w\">    <\/span><span class=\"s2\">&quot;Several days&quot;<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"mf\">1<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"11\"><span class=\"w\">    <\/span><span class=\"s2\">&quot;More than half the days&quot;<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"mf\">2<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"12\"><span class=\"w\">    <\/span><span class=\"s2\">&quot;Nearly every day&quot;<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"mf\">3<\/span>\n<\/span><span data-line=\"13\"><span class=\"w\">  <\/span><span class=\"p\">}<\/span>\n<\/span><span data-line=\"14\"><span class=\"p\">}<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<p>The other path adds a bit more complication. See, we can\u2019t just <em>skip<\/em> a question. The scoring process for these questionnaires relies on data from them. This means that we need to infer <em>some<\/em> value for them, and it\u2019s intellectually dishonest to just say \u201c0\u201d. So the way I approach that is with a thing I barely remembered from high school: confidence intervals.<\/p>\n<div class=\"highlight-javascript notranslate\" id=\"lgbot-questions-phq9-json-partial-3\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"p\">{<\/span>\n<\/span><span data-line=\"2\"><span class=\"w\">  <\/span><span class=\"s2\">&quot;question&quot;<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">[<\/span><span class=\"w\">   <\/span><span class=\"c1\">\/\/ Why bother with extra phrasings if we&#39;re skipping it?<\/span>\n<\/span><span data-line=\"3\"><span class=\"w\">    <\/span><span class=\"s2\">&quot;In the last two weeks, how often have you had trouble falling or staying asleep, or sleeping too much?&quot;<\/span>\n<\/span><span data-line=\"4\"><span class=\"w\">  <\/span><span class=\"p\">],<\/span>\n<\/span><span data-line=\"5\"><span class=\"w\">  <\/span><span class=\"s2\">&quot;display&quot;<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"kc\">false<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"6\"><span class=\"w\">  <\/span><span class=\"s2\">&quot;floor90&quot;<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"mf\">1<\/span><span class=\"p\">,<\/span><span class=\"w\">   <\/span><span class=\"c1\">\/\/ This says &quot;I&#39;m 90% sure that it&#39;ll be 1 or higher&quot;<\/span>\n<\/span><span data-line=\"7\"><span class=\"w\">  <\/span><span class=\"s2\">&quot;ceil90&quot;<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"mf\">2<\/span><span class=\"p\">,<\/span><span class=\"w\">    <\/span><span class=\"c1\">\/\/ Similarly, 90% of the time it should be 2 or lower.<\/span>\n<\/span><span data-line=\"8\"><span class=\"w\">  <\/span><span class=\"s2\">&quot;floor100&quot;<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"mf\">0<\/span><span class=\"p\">,<\/span><span class=\"w\">  <\/span><span class=\"c1\">\/\/ And you&#39;re completely sure that it will be 0 or higher<\/span>\n<\/span><span data-line=\"9\"><span class=\"w\">  <\/span><span class=\"s2\">&quot;ceil100&quot;<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"mf\">3<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"10\"><span class=\"w\">  <\/span><span class=\"s2\">&quot;answers&quot;<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">{}<\/span><span class=\"w\">   <\/span><span class=\"c1\">\/\/ Answers skipped for brevity<\/span>\n<\/span><span data-line=\"11\"><span class=\"p\">}<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<p>So, we have a way to write our questions down. We know what questions to ask. Now, given that the answers just <em>magically appear<\/em> in our hands, how would we interpret them? (Don\u2019t worry, we cover how they get to you later.)<\/p>\n<\/section>\n<section id=\"interpreting-data\">\n<h3>Interpreting Data<a class=\"headerlink\" href=\"#interpreting-data\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>I want to be fully upfront about this before we continue: the method I use to compute a final score is <em>arbitrary<\/em>. There are absolutely different methods I could use. Some of them might even be better, and I would be happy to talk about any suggestions. With that said, let\u2019s go over how we parse through this data.<\/p>\n<p>For each question, we either get an integer (because we got a concrete answer) or a confidence interval (because we need to infer an answer). Each of the integers can be converted into an infinitely narrow confidence interval. This means that what we\u2019ll get is something like:<\/p>\n<div class=\"table-wrapper docutils container\">\n<table class=\"docutils align-default\">\n<thead>\n<tr class=\"row-odd\"><th class=\"head\"><\/th>\n<th class=\"head\" colspan=\"2\"><p>Floor<\/p><\/th>\n<th class=\"head\" colspan=\"2\"><p>Ceiling<\/p><\/th>\n<\/tr>\n<tr class=\"row-even\"><th class=\"head\"><\/th>\n<th class=\"head\"><p>100%<\/p><\/th>\n<th class=\"head\"><p>90%<\/p><\/th>\n<th class=\"head\"><p>90%<\/p><\/th>\n<th class=\"head\"><p>100%<\/p><\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr class=\"row-odd\"><td><p>1<\/p><\/td>\n<td><p>2<\/p><\/td>\n<td><p>2<\/p><\/td>\n<td><p>2<\/p><\/td>\n<td><p>2<\/p><\/td>\n<\/tr>\n<tr class=\"row-even\"><td><p>2<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>2<\/p><\/td>\n<td><p>3<\/p><\/td>\n<\/tr>\n<tr class=\"row-odd\"><td><p>\u2026<\/p><\/td>\n<td><p>\u2026<\/p><\/td>\n<td><p>\u2026<\/p><\/td>\n<td><p>\u2026<\/p><\/td>\n<td><p>\u2026<\/p><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p>Being able to represent it as a table is <em>wonderful<\/em> because it means we can use the same formula for everything. Now that we have this, we need to get a composite score. That\u2019s going to look like making a composite for each row, then adding them up. Since it seems prudent to weight the narrower range higher, I decided on the following:<\/p>\n<div class=\"math-wrapper docutils container\">\n<div class=\"math notranslate nohighlight\">\n\\[\\dfrac{2(X_{F,90\\%} + X_{C,90\\%}) + X_{F,100\\%} + X_{C,100\\%}}{6}\\]<\/div>\n<\/div>\n<p>Because I want this to serve as a historical record, I save every step of this calculation in the database. This way, if my method of inference changes, I can compare it both by the contemporary method <em>and<\/em> the current method.<\/p>\n<\/section>\n<section id=\"resulting-chart\">\n<h3>Resulting Chart<a class=\"headerlink\" href=\"#resulting-chart\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>This is possibly the longest query in this series, not because it\u2019s complicated, but because we are trying to convey a lot of information. It consists of four subqueries. In the first, we are defining our half life as a constant. In the second, we fetch the raw scores for depression values. In the third, we compute the running averages. In the fourth, we gather the thresholds as defined by the questionnaire. Lastly, we gather all of that together.<\/p>\n<div class=\"highlight-PSQL notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"k\">WITH<\/span><span class=\"w\"> <\/span><span class=\"n\">c<\/span><span class=\"w\"> <\/span><span class=\"k\">AS<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span><span class=\"k\">SELECT<\/span><span class=\"w\"> <\/span><span class=\"nb\">interval<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;2 days&#39;<\/span><span class=\"w\"> <\/span><span class=\"k\">AS<\/span><span class=\"w\"> <\/span><span class=\"n\">half_life<\/span><span class=\"p\">),<\/span>\n<\/span><span data-line=\"2\">\n<\/span><span data-line=\"3\"><span class=\"n\">raw_depression<\/span><span class=\"w\"> <\/span><span class=\"k\">AS<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span>\n<\/span><span data-line=\"4\"><span class=\"w\">  <\/span><span class=\"k\">SELECT<\/span><span class=\"w\"> <\/span><span class=\"n\">respondant<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"s s-Name\">&quot;time&quot;<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"n\">final<\/span>\n<\/span><span data-line=\"5\"><span class=\"w\">  <\/span><span class=\"k\">FROM<\/span><span class=\"w\"> <\/span><span class=\"n\">lgbot<\/span><span class=\"mf\">.<\/span><span class=\"n\">depression<\/span>\n<\/span><span data-line=\"6\"><span class=\"w\">  <\/span><span class=\"k\">WHERE<\/span><span class=\"w\"> <\/span><span class=\"err\">$<\/span><span class=\"n\">__timeFilter<\/span><span class=\"p\">(<\/span><span class=\"s s-Name\">&quot;time&quot;<\/span><span class=\"p\">)<\/span>\n<\/span><span data-line=\"7\"><span class=\"p\">),<\/span>\n<\/span><span data-line=\"8\">\n<\/span><span data-line=\"9\"><span class=\"n\">avg_depression<\/span><span class=\"w\"> <\/span><span class=\"k\">AS<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span>\n<\/span><span data-line=\"10\"><span class=\"w\">  <\/span><span class=\"k\">SELECT<\/span>\n<\/span><span data-line=\"11\"><span class=\"w\">    <\/span><span class=\"n\">respondant<\/span><span class=\"w\"> <\/span><span class=\"o\">||<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39; R.A.&#39;<\/span><span class=\"p\">,<\/span><span class=\"w\">  <\/span><span class=\"c1\">-- concatenation<\/span>\n<\/span><span data-line=\"12\"><span class=\"w\">    <\/span><span class=\"s s-Name\">&quot;time&quot;<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"13\"><span class=\"w\">    <\/span><span class=\"n\">lgbot<\/span><span class=\"mf\">.<\/span><span class=\"n\">ema<\/span><span class=\"p\">(<\/span>\n<\/span><span data-line=\"14\"><span class=\"w\">      <\/span><span class=\"n\">final<\/span><span class=\"o\">::<\/span><span class=\"nb\">numeric<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"15\"><span class=\"w\">      <\/span><span class=\"s s-Name\">&quot;time&quot;<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"16\"><span class=\"w\">      <\/span><span class=\"n\">c<\/span><span class=\"mf\">.<\/span><span class=\"n\">half_life<\/span><span class=\"w\"> <\/span><span class=\"o\">\/<\/span><span class=\"w\"> <\/span><span class=\"n\">ln<\/span><span class=\"p\">(<\/span><span class=\"mf\">2<\/span><span class=\"p\">)<\/span>\n<\/span><span data-line=\"17\"><span class=\"w\">    <\/span><span class=\"p\">)<\/span><span class=\"w\"> <\/span><span class=\"k\">OVER<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span>\n<\/span><span data-line=\"18\"><span class=\"w\">      <\/span><span class=\"k\">PARTITION<\/span><span class=\"w\"> <\/span><span class=\"k\">BY<\/span><span class=\"w\"> <\/span><span class=\"n\">respondant<\/span>\n<\/span><span data-line=\"19\"><span class=\"w\">      <\/span><span class=\"k\">ORDER<\/span><span class=\"w\"> <\/span><span class=\"k\">BY<\/span><span class=\"w\"> <\/span><span class=\"s s-Name\">&quot;time&quot;<\/span>\n<\/span><span data-line=\"20\"><span class=\"w\">    <\/span><span class=\"p\">)<\/span>\n<\/span><span data-line=\"21\"><span class=\"w\">  <\/span><span class=\"k\">FROM<\/span><span class=\"w\"> <\/span><span class=\"n\">c<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"n\">lgbot<\/span><span class=\"mf\">.<\/span><span class=\"n\">depression<\/span>\n<\/span><span data-line=\"22\"><span class=\"w\">  <\/span><span class=\"k\">WHERE<\/span><span class=\"w\"> <\/span><span class=\"s s-Name\">&quot;time&quot;<\/span><span class=\"w\"> <\/span><span class=\"k\">BETWEEN<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span><span class=\"nb\">timestamptz<\/span><span class=\"w\"> <\/span><span class=\"err\">$<\/span><span class=\"n\">__timeFrom<\/span><span class=\"p\">()<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"w\"> <\/span><span class=\"n\">c<\/span><span class=\"mf\">.<\/span><span class=\"n\">half_life<\/span><span class=\"w\"> <\/span><span class=\"o\">*<\/span><span class=\"w\"> <\/span><span class=\"mf\">8<\/span><span class=\"p\">)<\/span><span class=\"w\"> <\/span><span class=\"k\">AND<\/span><span class=\"w\"> <\/span><span class=\"err\">$<\/span><span class=\"n\">__timeTo<\/span><span class=\"p\">()<\/span>\n<\/span><span data-line=\"23\"><span class=\"p\">),<\/span>\n<\/span><span data-line=\"24\">\n<\/span><span data-line=\"25\"><span class=\"n\">thresholds<\/span><span class=\"w\"> <\/span><span class=\"k\">AS<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span>\n<\/span><span data-line=\"26\"><span class=\"w\">  <\/span><span class=\"k\">SELECT<\/span><span class=\"w\"> <\/span><span class=\"n\">x<\/span><span class=\"mf\">.<\/span><span class=\"n\">respondant<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"n\">t<\/span><span class=\"mf\">.<\/span><span class=\"s s-Name\">&quot;time&quot;<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"n\">x<\/span><span class=\"mf\">.<\/span><span class=\"n\">final<\/span>\n<\/span><span data-line=\"27\"><span class=\"w\">  <\/span><span class=\"k\">FROM<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span>\n<\/span><span data-line=\"28\"><span class=\"w\">    <\/span><span class=\"k\">VALUES<\/span>\n<\/span><span data-line=\"29\"><span class=\"w\">  <\/span><span class=\"c1\">--    (&#39;Severe Depression&#39;, 20),<\/span>\n<\/span><span data-line=\"30\"><span class=\"w\">      <\/span><span class=\"p\">(<\/span><span class=\"s1\">&#39;Moderately Severe Depression&#39;<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"mf\">15<\/span><span class=\"p\">),<\/span>\n<\/span><span data-line=\"31\"><span class=\"w\">      <\/span><span class=\"p\">(<\/span><span class=\"s1\">&#39;Moderate Depression&#39;<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"mf\">10<\/span><span class=\"p\">)<\/span>\n<\/span><span data-line=\"32\"><span class=\"w\">  <\/span><span class=\"c1\">--    (&#39;Minor Depression&#39;, 5)<\/span>\n<\/span><span data-line=\"33\"><span class=\"w\">  <\/span><span class=\"p\">)<\/span><span class=\"w\"> <\/span><span class=\"k\">AS<\/span><span class=\"w\"> <\/span><span class=\"n\">x<\/span><span class=\"p\">(<\/span><span class=\"n\">respondant<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"n\">final<\/span><span class=\"p\">)<\/span>\n<\/span><span data-line=\"34\"><span class=\"w\">  <\/span><span class=\"k\">CROSS<\/span><span class=\"w\"> <\/span><span class=\"k\">JOIN<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span>\n<\/span><span data-line=\"35\"><span class=\"w\">    <\/span><span class=\"k\">VALUES<\/span>\n<\/span><span data-line=\"36\"><span class=\"w\">      <\/span><span class=\"p\">(<\/span><span class=\"nb\">timestamptz<\/span><span class=\"w\"> <\/span><span class=\"err\">$<\/span><span class=\"n\">__timeFrom<\/span><span class=\"p\">()<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"w\"> <\/span><span class=\"nb\">interval<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;1 second&#39;<\/span><span class=\"p\">),<\/span>\n<\/span><span data-line=\"37\"><span class=\"w\">      <\/span><span class=\"p\">(<\/span><span class=\"nb\">timestamptz<\/span><span class=\"w\"> <\/span><span class=\"err\">$<\/span><span class=\"n\">__timeTo<\/span><span class=\"p\">()<\/span><span class=\"w\"> <\/span><span class=\"o\">+<\/span><span class=\"w\"> <\/span><span class=\"nb\">interval<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;1 second&#39;<\/span><span class=\"p\">)<\/span>\n<\/span><span data-line=\"38\"><span class=\"w\">  <\/span><span class=\"p\">)<\/span><span class=\"w\"> <\/span><span class=\"k\">AS<\/span><span class=\"w\"> <\/span><span class=\"n\">t<\/span><span class=\"p\">(<\/span><span class=\"s s-Name\">&quot;time&quot;<\/span><span class=\"p\">)<\/span>\n<\/span><span data-line=\"39\"><span class=\"p\">)<\/span>\n<\/span><span data-line=\"40\">\n<\/span><span data-line=\"41\"><span class=\"p\">(<\/span>\n<\/span><span data-line=\"42\"><span class=\"w\">  <\/span><span class=\"k\">SELECT<\/span><span class=\"w\"> <\/span><span class=\"o\">*<\/span><span class=\"w\"> <\/span><span class=\"k\">FROM<\/span><span class=\"w\"> <\/span><span class=\"n\">raw_depression<\/span>\n<\/span><span data-line=\"43\"><span class=\"w\">  <\/span><span class=\"k\">UNION<\/span><span class=\"w\"> <\/span><span class=\"k\">ALL<\/span>\n<\/span><span data-line=\"44\"><span class=\"w\">  <\/span><span class=\"k\">SELECT<\/span><span class=\"w\"> <\/span><span class=\"o\">*<\/span><span class=\"w\"> <\/span><span class=\"k\">FROM<\/span><span class=\"w\"> <\/span><span class=\"n\">avg_depression<\/span>\n<\/span><span data-line=\"45\"><span class=\"w\">  <\/span><span class=\"k\">ORDER<\/span><span class=\"w\"> <\/span><span class=\"k\">BY<\/span><span class=\"w\"> <\/span><span class=\"n\">respondant<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"s s-Name\">&quot;time&quot;<\/span>\n<\/span><span data-line=\"46\"><span class=\"p\">)<\/span>\n<\/span><span data-line=\"47\"><span class=\"k\">UNION<\/span><span class=\"w\"> <\/span><span class=\"k\">ALL<\/span>\n<\/span><span data-line=\"48\"><span class=\"k\">SELECT<\/span><span class=\"w\"> <\/span><span class=\"o\">*<\/span><span class=\"w\"> <\/span><span class=\"k\">FROM<\/span><span class=\"w\"> <\/span><span class=\"n\">thresholds<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<p>I want to take a moment to review the last two subqueries.<\/p>\n<p>In <code class=\"code highlight sql PSQL docutils literal highlight-PSQL\"><span class=\"n\">avg_depression<\/span><\/code> we are gathering averages. The specific method was covered in the bonus entry <a class=\"reference external\" href=\"https:\/\/blog.oliviaappleton.com\/posts\/0008-lessons-in-grafana-02-5\"><em>Achieving Acceptable Averages<\/em><\/a> <span id=\"id3\">[<a class=\"reference internal\" href=\"#id25\" title=\"Olivia Appleton-Crocker. Lessons in grafana - bonus entry: achieving acceptable averages. URL: https:\/\/blog.oliviaappleton.com\/posts\/0008-lessons-in-grafana-02-5.\">ACa<\/a>]<\/span>, but since that was optional I will summarize. The function <code class=\"code highlight sql PSQL docutils literal highlight-PSQL\"><span class=\"n\">lgbot<\/span><span class=\"mf\">.<\/span><span class=\"n\">ema<\/span><\/code> defines an exponential moving average. That means that the farther away a data point is, the less it affects the average value associated with a given measurement. Specifically it does this according to a half life, which we define as a constant. We are applying a less strict time filter to ensure that past values are included in this average. The value chosen here is 8 times the half-life, which means we only exclude values that have a lower weight than <span class=\"math notranslate nohighlight\">\\(2^{-8} \\approx 0.0039\\)<\/span>.<\/p>\n<p>In <code class=\"code highlight sql PSQL docutils literal highlight-PSQL\"><span class=\"n\">thresholds<\/span><\/code> we are using a keyword not yet covered in this series: <code class=\"code highlight sql PSQL docutils literal highlight-PSQL\"><span class=\"k\">CROSS<\/span><span class=\"w\"> <\/span><span class=\"k\">JOIN<\/span><\/code>. You can think of this like Python\u2019s <a class=\"reference external\" href=\"https:\/\/docs.python.org\/3\/library\/itertools.html#itertools.product\" title=\"(in Python v3.14)\"><code class=\"xref py py-func docutils literal notranslate\"><span class=\"pre\">itertools.product()<\/span><\/code><\/a>. In essence, it is a nested for-loop like so:<\/p>\n<div class=\"highlight-python notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"k\">for<\/span> <span class=\"n\">name<\/span><span class=\"p\">,<\/span> <span class=\"n\">value<\/span> <span class=\"ow\">in<\/span> <span class=\"n\">thresholds<\/span><span class=\"p\">:<\/span>\n<\/span><span data-line=\"2\">  <span class=\"k\">for<\/span> <span class=\"n\">ts<\/span> <span class=\"ow\">in<\/span> <span class=\"n\">times<\/span><span class=\"p\">:<\/span>\n<\/span><span data-line=\"3\">    <span class=\"k\">yield<\/span> <span class=\"p\">(<\/span><span class=\"n\">name<\/span><span class=\"p\">,<\/span> <span class=\"n\">value<\/span><span class=\"p\">,<\/span> <span class=\"n\">ts<\/span><span class=\"p\">)<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<p>All of this yields the following chart. I specify in the axis that higher PHQ9 scores are worse because I was frequently causing confusion when not specified. The instinct of \u201chigher number good\u201d is quite strong.<\/p>\n<figure class=\"align-default\" id=\"id139\">\n<img alt=\"A chart noting depression over time, as described in the caption below\" src=\"..\/_images\/depression.png\" \/>\n<figcaption>\n<p><span class=\"caption-text\">This chart shows depression over time. The Y axis measures the inferred value of the PHQ-9 survey, with higher values being worse. Six lines are shown. Two of them note thresholds for degrees of depression, represented with dotted lines. Two of them note the raw scores for two respondents, represented with dashed lines. The last two show running averages for those respondents, shown in bold. Additionally, there are many dashed vertical lines that are used to denote events related to mental health. Not shown, you can click on them to see details of the events. Many of these correspond to spikes or troughs in raw depression levels.<\/span><a class=\"headerlink\" href=\"#id139\" title=\"Link to this image\">\u00b6<\/a><\/p>\n<\/figcaption>\n<\/figure>\n<\/section>\n<\/section>\n<section id=\"survey-delivery-and-responses\">\n<h2>Survey Delivery and Responses<a class=\"headerlink\" href=\"#survey-delivery-and-responses\" title=\"Link to this heading\">\u00b6<\/a><\/h2>\n<p>In <a class=\"reference external\" href=\"https:\/\/blog.oliviaappleton.com\/posts\/0007-lessons-in-grafana-02#prerequisites\"><em>Litter Logs<\/em><\/a> <span id=\"id4\">[<a class=\"reference internal\" href=\"#id26\" title=\"Olivia Appleton-Crocker. Lessons in grafana - part two: litter logs. URL: https:\/\/blog.oliviaappleton.com\/posts\/0007-lessons-in-grafana-02.\">ACc<\/a>]<\/span>, we showed the overall structure of my tracking bot. Let\u2019s keep building on that, simplifying a bit for clarity.<\/p>\n<p>The biggest new piece here is an integration with Telegram.<\/p>\n<section id=\"event-loop\">\n<h3>Event Loop<a class=\"headerlink\" href=\"#event-loop\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>The library I use to interact with Telegram is async-by-default, and comes with an event loop. Everything in this bot is driven by that event loop<a class=\"footnote-reference brackets\" href=\"#id18\" id=\"id5\" role=\"doc-noteref\"><span class=\"fn-bracket\">[<\/span>1<span class=\"fn-bracket\">]<\/span><\/a>. There are four hooks that enable our data collection:<\/p>\n<ol class=\"arabic simple\">\n<li><p>A <code class=\"code highlight python docutils literal highlight-python\"><span class=\"n\">start<\/span><span class=\"p\">()<\/span><\/code> method that sets initial state and handles command callbacks (ex: user sends <code class=\"docutils literal notranslate\"><span class=\"pre\">\/phq9<\/span><\/code>)<\/p><\/li>\n<li><p>A <code class=\"code highlight python docutils literal highlight-python\"><span class=\"n\">do_begin_loop<\/span><span class=\"p\">()<\/span><\/code> method, instantiated for each survey, which sends the first question once it\u2019s the right time<\/p><\/li>\n<li><p>A <code class=\"code highlight python docutils literal highlight-python\"><span class=\"n\">send_next_question<\/span><span class=\"p\">()<\/span><\/code> method that, given a survey, sends the next question in the sequence<\/p><\/li>\n<li><p>A <code class=\"code highlight python docutils literal highlight-python\"><span class=\"n\">questionnaire_callback<\/span><span class=\"p\">()<\/span><\/code> method that determines the survey, records the answer, and kicks off either the next question or answer logging<\/p><\/li>\n<\/ol>\n<\/section>\n<section id=\"kicking-things-off\">\n<h3>Kicking Things Off<a class=\"headerlink\" href=\"#kicking-things-off\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>At the beginning of each run, we call <code class=\"code highlight python docutils literal highlight-python\"><span class=\"n\">start<\/span><span class=\"p\">()<\/span><\/code>. It\u2019s fairly straightforward, and looks like so:<\/p>\n<div class=\"highlight-python notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"k\">async<\/span> <span class=\"k\">def<\/span><span class=\"w\"> <\/span><span class=\"nf\">start<\/span><span class=\"p\">(<\/span>\n<\/span><span data-line=\"2\">  <span class=\"n\">update<\/span><span class=\"p\">:<\/span> <span class=\"n\">Update<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"3\">  <span class=\"n\">context<\/span><span class=\"p\">:<\/span> <span class=\"n\">ContextTypes<\/span><span class=\"o\">.<\/span><span class=\"n\">DEFAULT_TYPE<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"4\">  <span class=\"n\">bank<\/span><span class=\"p\">:<\/span> <span class=\"n\">BankType<\/span>\n<\/span><span data-line=\"5\"><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"kc\">None<\/span><span class=\"p\">:<\/span>\n<\/span><span data-line=\"6\">  <span class=\"k\">async<\/span> <span class=\"k\">with<\/span> <span class=\"n\">SharedState<\/span><span class=\"o\">.<\/span><span class=\"n\">lock<\/span><span class=\"p\">:<\/span>\n<\/span><span data-line=\"7\">    <span class=\"n\">SharedState<\/span><span class=\"o\">.<\/span><span class=\"n\">q_asking<\/span><span class=\"p\">[<\/span><span class=\"n\">bank<\/span><span class=\"p\">]<\/span> <span class=\"o\">=<\/span> <span class=\"kc\">True<\/span>\n<\/span><span data-line=\"8\">    <span class=\"n\">SharedState<\/span><span class=\"o\">.<\/span><span class=\"n\">q_responses<\/span><span class=\"p\">[<\/span><span class=\"n\">bank<\/span><span class=\"p\">]<\/span><span class=\"o\">.<\/span><span class=\"n\">clear<\/span><span class=\"p\">()<\/span>\n<\/span><span data-line=\"9\">    <span class=\"n\">SharedState<\/span><span class=\"o\">.<\/span><span class=\"n\">q_index<\/span><span class=\"p\">[<\/span><span class=\"n\">bank<\/span><span class=\"p\">]<\/span> <span class=\"o\">=<\/span> <span class=\"mi\">0<\/span>\n<\/span><span data-line=\"10\">  <span class=\"k\">await<\/span> <span class=\"n\">send_next_question<\/span><span class=\"p\">(<\/span><span class=\"n\">context<\/span><span class=\"p\">,<\/span> <span class=\"n\">bank<\/span><span class=\"p\">)<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<p>You\u2019ll notice that we are storing three pieces of information. The first piece is just noting that a questionnaire is in progress. This is crucial, because otherwise we might run more than one at the same time. That would be both confusing and annoying. Note that because the <code class=\"code highlight python docutils literal highlight-python\"><span class=\"n\">start<\/span><span class=\"p\">()<\/span><\/code> function is also the command handler, a user <em>can<\/em> manually trigger multiple questionnaires. If they do so, that\u2019s on them.<\/p>\n<p>The next piece of information is the responses received thus far. This is stored as a list, and is fairly straightforward.<\/p>\n<p>Lastly, we store the index of the last question asked. That might seem strange. After all, we have the list of responses. Why not simply go off of that? Well, because you aren\u2019t getting a response to every question. The index keeps track of where we are in the questionnaire <em>including skipped questions<\/em>.<\/p>\n<\/section>\n<section id=\"getting-replies\">\n<h3>Getting Replies<a class=\"headerlink\" href=\"#getting-replies\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>Sending and receiving information through Telegram is relatively easy, but somewhat messy. Because I wanted to minimize friction, we use the on-screen keyboard markup they provide. This means that everything appears as a handy button to click, like so:<\/p>\n<figure class=\"align-center\" id=\"id140\">\n<a class=\"reference internal image-reference\" href=\"..\/_images\/telegram_buttons.png\"><img alt=\"An example of the on-screen keyboard feature in Telegram, as described below\" src=\"..\/_images\/telegram_buttons.png\" style=\"width: 427px;\" \/>\n<\/a>\n<figcaption>\n<p><span class=\"caption-text\">This is an example survey question. In particular, this is the first question of the PHQ-9, as covered above. Note that the answers are scrambled, and appear as a stack of clickable buttons.<\/span><a class=\"headerlink\" href=\"#id140\" title=\"Link to this image\">\u00b6<\/a><\/p>\n<\/figcaption>\n<\/figure>\n<p>In general, we use the following steps:<\/p>\n<ol class=\"arabic simple\">\n<li><p>Load information about the next question (covered in the next section)<\/p><\/li>\n<li><p>Check if we already asked the final question<\/p><\/li>\n<li><p>Construct a keyboard that encodes the questionnaire name and answer label<\/p><\/li>\n<li><p>Send the message<\/p><\/li>\n<li><p>Receive a reply<\/p><\/li>\n<li><p>Decode the questionnaire name and answer label<\/p><\/li>\n<li><p>Fetch the answer value<\/p><\/li>\n<li><p>Increment question index<\/p><\/li>\n<li><p>Delete the message &amp; repeat<\/p><\/li>\n<\/ol>\n<p>This is our message sending method. It handles steps 1-4. The most important thing to pay attention to is the keyboard construction. Note that all of the answers get shuffled using <a class=\"reference external\" href=\"https:\/\/docs.python.org\/3\/library\/random.html#random.sample\" title=\"(in Python v3.14)\"><code class=\"xref py py-func docutils literal notranslate\"><span class=\"pre\">random.sample()<\/span><\/code><\/a>.<\/p>\n<div class=\"highlight-python notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"k\">async<\/span> <span class=\"k\">def<\/span><span class=\"w\"> <\/span><span class=\"nf\">send_next_question<\/span><span class=\"p\">(<\/span>\n<\/span><span data-line=\"2\">  <span class=\"n\">context<\/span><span class=\"p\">:<\/span> <span class=\"n\">ContextTypes<\/span><span class=\"o\">.<\/span><span class=\"n\">DEFAULT_TYPE<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"3\">  <span class=\"n\">bank<\/span><span class=\"p\">:<\/span> <span class=\"n\">BankType<\/span>\n<\/span><span data-line=\"4\"><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"kc\">None<\/span><span class=\"p\">:<\/span>\n<\/span><span data-line=\"5\"><span class=\"w\">  <\/span><span class=\"sd\">&quot;&quot;&quot;This handles the sending of questions to the user&quot;&quot;&quot;<\/span>\n<\/span><span data-line=\"6\">  <span class=\"n\">logger<\/span><span class=\"o\">.<\/span><span class=\"n\">info<\/span><span class=\"p\">(<\/span><span class=\"s2\">&quot;Sending next question for <\/span><span class=\"si\">%s<\/span><span class=\"s2\">&quot;<\/span><span class=\"p\">,<\/span> <span class=\"n\">bank<\/span><span class=\"p\">)<\/span>\n<\/span><span data-line=\"7\">  <span class=\"p\">(<\/span><span class=\"n\">question<\/span><span class=\"p\">,<\/span> <span class=\"n\">answers<\/span><span class=\"p\">),<\/span> <span class=\"n\">num_qs<\/span> <span class=\"o\">=<\/span> <span class=\"k\">await<\/span> <span class=\"n\">gather<\/span><span class=\"p\">(<\/span>\n<\/span><span data-line=\"8\">    <span class=\"n\">select_question<\/span><span class=\"p\">(<\/span><span class=\"n\">bank<\/span><span class=\"p\">),<\/span>\n<\/span><span data-line=\"9\">    <span class=\"n\">len_questions<\/span><span class=\"p\">(<\/span><span class=\"n\">bank<\/span><span class=\"p\">)<\/span>\n<\/span><span data-line=\"10\">  <span class=\"p\">)<\/span>\n<\/span><span data-line=\"11\">\n<\/span><span data-line=\"12\">  <span class=\"k\">async<\/span> <span class=\"k\">with<\/span> <span class=\"n\">SharedState<\/span><span class=\"o\">.<\/span><span class=\"n\">lock<\/span><span class=\"p\">:<\/span>\n<\/span><span data-line=\"13\">    <span class=\"k\">if<\/span> <span class=\"n\">SharedState<\/span><span class=\"o\">.<\/span><span class=\"n\">q_index<\/span><span class=\"p\">[<\/span><span class=\"n\">bank<\/span><span class=\"p\">]<\/span> <span class=\"o\">&gt;<\/span> <span class=\"n\">num_qs<\/span><span class=\"p\">:<\/span>\n<\/span><span data-line=\"14\">      <span class=\"o\">...<\/span>  <span class=\"c1\"># record answers, set up for asking later<\/span>\n<\/span><span data-line=\"15\">      <span class=\"k\">return<\/span>\n<\/span><span data-line=\"16\">    <span class=\"n\">logger<\/span><span class=\"o\">.<\/span><span class=\"n\">info<\/span><span class=\"p\">(<\/span>\n<\/span><span data-line=\"17\">      <span class=\"s2\">&quot;<\/span><span class=\"si\">%s<\/span><span class=\"s2\"> processing question <\/span><span class=\"si\">%i<\/span><span class=\"s2\"> of <\/span><span class=\"si\">%i<\/span><span class=\"s2\">&quot;<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"18\">      <span class=\"n\">bank<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"19\">      <span class=\"n\">SharedState<\/span><span class=\"o\">.<\/span><span class=\"n\">q_index<\/span><span class=\"p\">[<\/span><span class=\"n\">bank<\/span><span class=\"p\">],<\/span>\n<\/span><span data-line=\"20\">      <span class=\"n\">num_qs<\/span>\n<\/span><span data-line=\"21\">    <span class=\"p\">)<\/span>\n<\/span><span data-line=\"22\">\n<\/span><span data-line=\"23\">  <span class=\"n\">reply_markup<\/span> <span class=\"o\">=<\/span> <span class=\"n\">InlineKeyboardMarkup<\/span><span class=\"p\">([<\/span>\n<\/span><span data-line=\"24\">    <span class=\"p\">[<\/span><span class=\"n\">InlineKeyboardButton<\/span><span class=\"p\">(<\/span><span class=\"n\">text<\/span><span class=\"o\">=<\/span><span class=\"n\">label<\/span><span class=\"p\">,<\/span> <span class=\"n\">callback_data<\/span><span class=\"o\">=<\/span><span class=\"sa\">f<\/span><span class=\"s1\">&#39;<\/span><span class=\"si\">{<\/span><span class=\"n\">bank<\/span><span class=\"si\">}<\/span><span class=\"s1\">+<\/span><span class=\"si\">{<\/span><span class=\"n\">label<\/span><span class=\"si\">}<\/span><span class=\"s1\">&#39;<\/span><span class=\"p\">)]<\/span>\n<\/span><span data-line=\"25\">    <span class=\"k\">for<\/span> <span class=\"n\">label<\/span> <span class=\"ow\">in<\/span> <span class=\"n\">sample<\/span><span class=\"p\">((<\/span><span class=\"o\">*<\/span><span class=\"n\">answers<\/span><span class=\"p\">,<\/span> <span class=\"p\">),<\/span> <span class=\"nb\">len<\/span><span class=\"p\">(<\/span><span class=\"n\">answers<\/span><span class=\"p\">))<\/span>\n<\/span><span data-line=\"26\">  <span class=\"p\">])<\/span>\n<\/span><span data-line=\"27\">  <span class=\"k\">await<\/span> <span class=\"n\">context<\/span><span class=\"o\">.<\/span><span class=\"n\">bot<\/span><span class=\"o\">.<\/span><span class=\"n\">send_message<\/span><span class=\"p\">(<\/span>\n<\/span><span data-line=\"28\">    <span class=\"n\">chat_id<\/span><span class=\"o\">=<\/span><span class=\"n\">TELEGRAM_CHAT_ID<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"29\">    <span class=\"n\">text<\/span><span class=\"o\">=<\/span><span class=\"n\">question<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"30\">    <span class=\"n\">reply_markup<\/span><span class=\"o\">=<\/span><span class=\"n\">reply_markup<\/span>\n<\/span><span data-line=\"31\">  <span class=\"p\">)<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<p>This is our message receiver method. It handles steps 5-9, and overall has an easier job. The most important thing it does is handling the question index. If it were handled anywhere else, you could have worrying race conditions around what answer gets processed. Handling it here means that, so long as a user does not initiate two of the same survey at once, it will always be processed correctly.<\/p>\n<div class=\"highlight-python notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"k\">async<\/span> <span class=\"k\">def<\/span><span class=\"w\"> <\/span><span class=\"nf\">questionnaire_callback<\/span><span class=\"p\">(<\/span>\n<\/span><span data-line=\"2\">  <span class=\"n\">update<\/span><span class=\"p\">:<\/span> <span class=\"n\">Update<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"3\">  <span class=\"n\">context<\/span><span class=\"p\">:<\/span> <span class=\"n\">ContextTypes<\/span><span class=\"o\">.<\/span><span class=\"n\">DEFAULT_TYPE<\/span>\n<\/span><span data-line=\"4\"><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"kc\">None<\/span><span class=\"p\">:<\/span>\n<\/span><span data-line=\"5\"><span class=\"w\">  <\/span><span class=\"sd\">&quot;&quot;&quot;When answers are submitted, they are processed here&quot;&quot;&quot;<\/span>\n<\/span><span data-line=\"6\">  <span class=\"n\">query<\/span> <span class=\"o\">=<\/span> <span class=\"n\">update<\/span><span class=\"o\">.<\/span><span class=\"n\">callback_query<\/span>\n<\/span><span data-line=\"7\">  <span class=\"k\">await<\/span> <span class=\"n\">query<\/span><span class=\"o\">.<\/span><span class=\"n\">answer<\/span><span class=\"p\">()<\/span>\n<\/span><span data-line=\"8\">\n<\/span><span data-line=\"9\">  <span class=\"n\">bank<\/span><span class=\"p\">,<\/span> <span class=\"n\">data<\/span> <span class=\"o\">=<\/span> <span class=\"n\">cast<\/span><span class=\"p\">(<\/span><span class=\"n\">Tuple<\/span><span class=\"p\">[<\/span><span class=\"n\">BankType<\/span><span class=\"p\">,<\/span> <span class=\"nb\">str<\/span><span class=\"p\">],<\/span> <span class=\"n\">query<\/span><span class=\"o\">.<\/span><span class=\"n\">data<\/span><span class=\"o\">.<\/span><span class=\"n\">split<\/span><span class=\"p\">(<\/span><span class=\"s1\">&#39;+&#39;<\/span><span class=\"p\">))<\/span>\n<\/span><span data-line=\"10\">  <span class=\"n\">_<\/span><span class=\"p\">,<\/span> <span class=\"n\">answers<\/span> <span class=\"o\">=<\/span> <span class=\"k\">await<\/span> <span class=\"n\">select_question<\/span><span class=\"p\">(<\/span><span class=\"n\">bank<\/span><span class=\"p\">)<\/span>\n<\/span><span data-line=\"11\">  <span class=\"n\">selected<\/span> <span class=\"o\">=<\/span> <span class=\"n\">answers<\/span><span class=\"p\">[<\/span><span class=\"n\">data<\/span><span class=\"p\">]<\/span>\n<\/span><span data-line=\"12\">  <span class=\"k\">async<\/span> <span class=\"k\">with<\/span> <span class=\"n\">SharedState<\/span><span class=\"o\">.<\/span><span class=\"n\">lock<\/span><span class=\"p\">:<\/span>\n<\/span><span data-line=\"13\">    <span class=\"n\">SharedState<\/span><span class=\"o\">.<\/span><span class=\"n\">q_responses<\/span><span class=\"p\">[<\/span><span class=\"n\">bank<\/span><span class=\"p\">]<\/span><span class=\"o\">.<\/span><span class=\"n\">append<\/span><span class=\"p\">(<\/span><span class=\"n\">selected<\/span><span class=\"p\">)<\/span>\n<\/span><span data-line=\"14\">    <span class=\"n\">SharedState<\/span><span class=\"o\">.<\/span><span class=\"n\">q_index<\/span><span class=\"p\">[<\/span><span class=\"n\">bank<\/span><span class=\"p\">]<\/span> <span class=\"o\">+=<\/span> <span class=\"mi\">1<\/span>\n<\/span><span data-line=\"15\">\n<\/span><span data-line=\"16\">  <span class=\"k\">await<\/span> <span class=\"n\">gather<\/span><span class=\"p\">(<\/span>\n<\/span><span data-line=\"17\">    <span class=\"n\">query<\/span><span class=\"o\">.<\/span><span class=\"n\">message<\/span><span class=\"o\">.<\/span><span class=\"n\">delete<\/span><span class=\"p\">(),<\/span>\n<\/span><span data-line=\"18\">    <span class=\"n\">send_next_question<\/span><span class=\"p\">(<\/span><span class=\"n\">context<\/span><span class=\"p\">,<\/span> <span class=\"n\">bank<\/span><span class=\"p\">)<\/span>\n<\/span><span data-line=\"19\">  <span class=\"p\">)<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<\/section>\n<section id=\"the-annoying-part-getting-questions\">\n<h3>The Annoying Part \u2013 Getting Questions<a class=\"headerlink\" href=\"#the-annoying-part-getting-questions\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>Because we are loading questions dynamically and skipping some of them, deciding which to ask next is somewhat complicated. This is further complicated by needing to preface most questionnaires with information for <a class=\"reference external\" href=\"#id8\">plural metadata<\/a>. As a result, this function is pretty messy. It\u2019s definitely not a perfect solution, but is a product of repeatedly making the least modifications possible to accomplish new goals. Rewriting it is one of the things I\u2019d like to do in the near future. With that being said, let\u2019s look at how it works. First I will show code, and then I will attempt to explain the logic in more detail.<\/p>\n<div class=\"highlight-python notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"c1\"># Note: this method may return (&#39;&#39;, {}) in an error; detection is on the caller<\/span>\n<\/span><span data-line=\"2\"><span class=\"k\">async<\/span> <span class=\"k\">def<\/span><span class=\"w\"> <\/span><span class=\"nf\">select_question<\/span><span class=\"p\">(<\/span><span class=\"n\">bank<\/span><span class=\"p\">:<\/span> <span class=\"n\">BankType<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">Tuple<\/span><span class=\"p\">[<\/span><span class=\"nb\">str<\/span><span class=\"p\">,<\/span> <span class=\"n\">Mapping<\/span><span class=\"p\">[<\/span><span class=\"nb\">str<\/span><span class=\"p\">,<\/span> <span class=\"nb\">int<\/span> <span class=\"o\">|<\/span> <span class=\"nb\">str<\/span><span class=\"p\">]]:<\/span>\n<\/span><span data-line=\"3\">  <span class=\"n\">question<\/span><span class=\"p\">:<\/span> <span class=\"nb\">str<\/span> <span class=\"o\">=<\/span> <span class=\"s1\">&#39;&#39;<\/span>\n<\/span><span data-line=\"4\">  <span class=\"n\">answers<\/span><span class=\"p\">:<\/span> <span class=\"n\">Mapping<\/span><span class=\"p\">[<\/span><span class=\"nb\">str<\/span><span class=\"p\">,<\/span> <span class=\"nb\">int<\/span> <span class=\"o\">|<\/span> <span class=\"nb\">str<\/span><span class=\"p\">]<\/span> <span class=\"o\">=<\/span> <span class=\"p\">{}<\/span>\n<\/span><span data-line=\"5\">  <span class=\"k\">async<\/span> <span class=\"k\">with<\/span> <span class=\"n\">SharedState<\/span><span class=\"o\">.<\/span><span class=\"n\">lock<\/span><span class=\"p\">:<\/span>\n<\/span><span data-line=\"6\">    <span class=\"k\">if<\/span> <span class=\"n\">bank<\/span> <span class=\"o\">==<\/span> <span class=\"s1\">&#39;poop&#39;<\/span><span class=\"p\">:<\/span>  <span class=\"c1\"># hack because bodies do what bodies do<\/span>\n<\/span><span data-line=\"7\">      <span class=\"n\">SharedState<\/span><span class=\"o\">.<\/span><span class=\"n\">q_index<\/span><span class=\"p\">[<\/span><span class=\"n\">bank<\/span><span class=\"p\">]<\/span> <span class=\"o\">+=<\/span> <span class=\"mi\">1<\/span>\n<\/span><span data-line=\"8\">\n<\/span><span data-line=\"9\">    <span class=\"k\">if<\/span> <span class=\"n\">SharedState<\/span><span class=\"o\">.<\/span><span class=\"n\">q_index<\/span><span class=\"p\">[<\/span><span class=\"n\">bank<\/span><span class=\"p\">]<\/span> <span class=\"o\">==<\/span> <span class=\"mi\">0<\/span><span class=\"p\">:<\/span>\n<\/span><span data-line=\"10\">      <span class=\"n\">question<\/span> <span class=\"o\">=<\/span> <span class=\"sa\">f<\/span><span class=\"s2\">&quot;Who is answering this questionnaire (<\/span><span class=\"si\">{<\/span><span class=\"n\">bank<\/span><span class=\"si\">}<\/span><span class=\"s2\">)?&quot;<\/span>\n<\/span><span data-line=\"11\">      <span class=\"n\">answers<\/span> <span class=\"o\">=<\/span> <span class=\"p\">{<\/span><span class=\"n\">x<\/span><span class=\"p\">:<\/span> <span class=\"n\">x<\/span> <span class=\"k\">for<\/span> <span class=\"n\">x<\/span> <span class=\"ow\">in<\/span> <span class=\"k\">await<\/span> <span class=\"n\">load_questions<\/span><span class=\"p\">(<\/span><span class=\"s1\">&#39;system&#39;<\/span><span class=\"p\">)}<\/span>\n<\/span><span data-line=\"12\">\n<\/span><span data-line=\"13\">    <span class=\"k\">else<\/span><span class=\"p\">:<\/span>\n<\/span><span data-line=\"14\">      <span class=\"n\">questions<\/span> <span class=\"o\">=<\/span> <span class=\"p\">(<\/span><span class=\"k\">await<\/span> <span class=\"n\">load_questions<\/span><span class=\"p\">(<\/span><span class=\"n\">bank<\/span><span class=\"p\">))[<\/span><span class=\"s1\">&#39;questions&#39;<\/span><span class=\"p\">]<\/span>\n<\/span><span data-line=\"15\">      <span class=\"k\">try<\/span><span class=\"p\">:<\/span>\n<\/span><span data-line=\"16\">        <span class=\"k\">while<\/span> <span class=\"kc\">True<\/span><span class=\"p\">:<\/span>\n<\/span><span data-line=\"17\">          <span class=\"n\">index<\/span> <span class=\"o\">=<\/span> <span class=\"n\">SharedState<\/span><span class=\"o\">.<\/span><span class=\"n\">q_index<\/span><span class=\"p\">[<\/span><span class=\"n\">bank<\/span><span class=\"p\">]<\/span> <span class=\"o\">-<\/span> <span class=\"mi\">1<\/span>\n<\/span><span data-line=\"18\">          <span class=\"k\">if<\/span> <span class=\"n\">questions<\/span><span class=\"p\">[<\/span><span class=\"n\">index<\/span><span class=\"p\">]<\/span><span class=\"o\">.<\/span><span class=\"n\">get<\/span><span class=\"p\">(<\/span><span class=\"s1\">&#39;display&#39;<\/span><span class=\"p\">)<\/span> <span class=\"ow\">is<\/span> <span class=\"kc\">False<\/span><span class=\"p\">:<\/span>\n<\/span><span data-line=\"19\">            <span class=\"n\">logger<\/span><span class=\"o\">.<\/span><span class=\"n\">info<\/span><span class=\"p\">(<\/span><span class=\"s2\">&quot;<\/span><span class=\"si\">%s<\/span><span class=\"s2\"> skipping question <\/span><span class=\"si\">%i<\/span><span class=\"s2\"> as requested&quot;<\/span><span class=\"p\">,<\/span> <span class=\"n\">bank<\/span><span class=\"p\">,<\/span> <span class=\"n\">SharedState<\/span><span class=\"o\">.<\/span><span class=\"n\">q_index<\/span><span class=\"p\">[<\/span><span class=\"n\">bank<\/span><span class=\"p\">])<\/span>\n<\/span><span data-line=\"20\">            <span class=\"n\">SharedState<\/span><span class=\"o\">.<\/span><span class=\"n\">q_index<\/span><span class=\"p\">[<\/span><span class=\"n\">bank<\/span><span class=\"p\">]<\/span> <span class=\"o\">+=<\/span> <span class=\"mi\">1<\/span>\n<\/span><span data-line=\"21\">          <span class=\"k\">else<\/span><span class=\"p\">:<\/span>\n<\/span><span data-line=\"22\">            <span class=\"k\">break<\/span>\n<\/span><span data-line=\"23\">        <span class=\"n\">question<\/span> <span class=\"o\">=<\/span> <span class=\"n\">random<\/span><span class=\"o\">.<\/span><span class=\"n\">choice<\/span><span class=\"p\">(<\/span><span class=\"n\">questions<\/span><span class=\"p\">[<\/span><span class=\"n\">index<\/span><span class=\"p\">][<\/span><span class=\"s1\">&#39;question&#39;<\/span><span class=\"p\">])<\/span>\n<\/span><span data-line=\"24\">        <span class=\"n\">answers<\/span> <span class=\"o\">=<\/span> <span class=\"n\">questions<\/span><span class=\"p\">[<\/span><span class=\"n\">index<\/span><span class=\"p\">][<\/span><span class=\"s1\">&#39;answers&#39;<\/span><span class=\"p\">]<\/span>\n<\/span><span data-line=\"25\">      <span class=\"k\">except<\/span> <span class=\"ne\">IndexError<\/span><span class=\"p\">:<\/span>\n<\/span><span data-line=\"26\">        <span class=\"k\">pass<\/span>\n<\/span><span data-line=\"27\">  <span class=\"k\">return<\/span> <span class=\"n\">question<\/span><span class=\"p\">,<\/span> <span class=\"n\">answers<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<p>As noted above, this function has many jobs. Arguably it has <em>too<\/em> many jobs. Its first task is to identify the respondent. Since this bot is designed to have a single user per instance, this is entirely to allow for plural folks to use this better, as covered towards the end of this entry. If this question would ordinarily be asked, we then need to make an exception for things that are pure bodily functions. After all, that is tracking the body not the mind.<\/p>\n<p>Once this task is out of the way, we need to load the main question bank. We start with the first question, checking each time if the question should be skipped. If it should be, then we advance the question index. This is the <em>only other place<\/em> where that index is advanced. Everywhere else is either reading or clearing it. On the first question that is not marked to skip, we break out of the loop and return. If we do not find such a question, it returns an empty set.<\/p>\n<p>I think that the logic of this is fine, but the implementation details are not my favorite. In my opinion this is the most technically-indebted part of this bot.<\/p>\n<\/section>\n<\/section>\n<section id=\"anxiety-tracking\">\n<h2>Anxiety Tracking<a class=\"headerlink\" href=\"#anxiety-tracking\" title=\"Link to this heading\">\u00b6<\/a><\/h2>\n<p>After some time working on this, I was asked why I hadn\u2019t addressed anxiety. After all, that\u2019s something that most people suffer from to some degree, and it can give context to other things. Since psychiatric medicines affect both, it\u2019s rather foolish to measure one without the other.<\/p>\n<p>Anxiety suffers from the same measurement problems as depression, however. On top of that, anxiety has much more moment-to-moment variation. This makes it a lower priority to track, in my opinion, and so I rely on simple self-reporting. The questionnaire for anxiety looks like this:<\/p>\n<div class=\"highlight-javascript notranslate\" id=\"lgbot-questions-anxiety-json\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"p\">{<\/span>\n<\/span><span data-line=\"2\"><span class=\"w\">  <\/span><span class=\"s2\">&quot;randomIntervalHours&quot;<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">[<\/span><span class=\"mf\">2<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"mf\">12<\/span><span class=\"p\">],<\/span>\n<\/span><span data-line=\"3\"><span class=\"w\">  <\/span><span class=\"s2\">&quot;questions&quot;<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">[{<\/span>\n<\/span><span data-line=\"4\"><span class=\"w\">    <\/span><span class=\"s2\">&quot;question&quot;<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">[<\/span>\n<\/span><span data-line=\"5\"><span class=\"w\">      <\/span><span class=\"s2\">&quot;On a scale of 0 to 5, rate your anxiety&quot;<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"6\"><span class=\"w\">      <\/span><span class=\"s2\">&quot;En una escala del 0 al 5, califique su ansiedad.&quot;<\/span>\n<\/span><span data-line=\"7\"><span class=\"w\">    <\/span><span class=\"p\">],<\/span>\n<\/span><span data-line=\"8\"><span class=\"w\">    <\/span><span class=\"s2\">&quot;index&quot;<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"mf\">1<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"9\"><span class=\"w\">    <\/span><span class=\"s2\">&quot;display&quot;<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"kc\">true<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"10\"><span class=\"w\">    <\/span><span class=\"s2\">&quot;answers&quot;<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">{<\/span>\n<\/span><span data-line=\"11\"><span class=\"w\">      <\/span><span class=\"s2\">&quot;0&quot;<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"mf\">0<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"12\"><span class=\"w\">      <\/span><span class=\"s2\">&quot;1&quot;<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"mf\">1<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"13\"><span class=\"w\">      <\/span><span class=\"s2\">&quot;2&quot;<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"mf\">2<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"14\"><span class=\"w\">      <\/span><span class=\"s2\">&quot;3&quot;<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"mf\">3<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"15\"><span class=\"w\">      <\/span><span class=\"s2\">&quot;4&quot;<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"mf\">4<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"16\"><span class=\"w\">      <\/span><span class=\"s2\">&quot;5&quot;<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"mf\">5<\/span>\n<\/span><span data-line=\"17\"><span class=\"w\">    <\/span><span class=\"p\">}<\/span>\n<\/span><span data-line=\"18\"><span class=\"w\">  <\/span><span class=\"p\">}]<\/span>\n<\/span><span data-line=\"19\"><span class=\"p\">}<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<p>Very simple, very pared down. It just asks for a number. And if more precision is ever needed, I can easily migrate it to a more detailed questionnaire. The query used for this is much the same as the one for depression tracking, without the piece about thresholds. For brevity\u2019s sake I will skip it.<\/p>\n<figure class=\"align-default\" id=\"id141\">\n<img alt=\"A chart noting anxiety over time, as described in the caption below\" src=\"..\/_images\/anxiety.png\" \/>\n<figcaption>\n<p><span class=\"caption-text\">This chart shows anxiety over time. The Y axis measures self-reported anxiety on a scale of 0 to 5, with higher values being worse. Four lines are shown. Two of them note the raw scores for two respondents, represented with dashed lines. The last two show running averages for those respondents, shown in bold. Additionally, there are many dashed vertical lines that are used to denote events related to mental health. Not shown, you can click on them to see details of the events. Many of these correspond to spikes or troughs in raw anxiety levels.<\/span><a class=\"headerlink\" href=\"#id141\" title=\"Link to this image\">\u00b6<\/a><\/p>\n<\/figcaption>\n<\/figure>\n<\/section>\n<section id=\"event-logging\">\n<h2>Event Logging<a class=\"headerlink\" href=\"#event-logging\" title=\"Link to this heading\">\u00b6<\/a><\/h2>\n<p>One of the many things this enables is the logging of discrete events. This is anything where the exact timing is unimportant, but knowing that it happened is quite useful.<\/p>\n<section id=\"toy-example-poops\">\n<h3>Toy Example: Poops<a class=\"headerlink\" href=\"#toy-example-poops\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>The funniest example, and therefore the easiest to talk about, is pooping. Bowel movements. Spending time on the porcelain throne. Not only is this medically useful to track, it also requires the least possible information. With other events, you might need extra information such as duration or severity<a class=\"footnote-reference brackets\" href=\"#p\" id=\"id6\" role=\"doc-noteref\"><span class=\"fn-bracket\">[<\/span>2<span class=\"fn-bracket\">]<\/span><\/a>. So let\u2019s look at how our event handlers treat this.<\/p>\n<p>For one thing, every recorded event is done as a command. In this case, that would be <code class=\"docutils literal notranslate\"><span class=\"pre\">\/poop<\/span><\/code>. This calls <code class=\"code highlight python docutils literal highlight-python\"><span class=\"n\">start<\/span><span class=\"p\">()<\/span><\/code>, which calls <code class=\"code highlight python docutils literal highlight-python\"><span class=\"n\">send_next_question<\/span><span class=\"p\">()<\/span><\/code>. This calls <code class=\"code highlight python docutils literal highlight-python\"><span class=\"n\">select_question<\/span><span class=\"p\">()<\/span><\/code> which, as you saw above, is a bit of a mess. Fortunately, it has a short-circuit for this exact use case: we don\u2019t have any questions for this. It just advances the index past the end of the questionnaire, which triggers the end of the survey. Our bot sees we are past the end, and tries to schedule the next one. Let\u2019s look at the file that defines the <code class=\"docutils literal notranslate\"><span class=\"pre\">\/poop<\/span><\/code> command:<\/p>\n<div class=\"highlight-javascript notranslate\" id=\"lgbot-questions-poop-json\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"p\">{<\/span>\n<\/span><span data-line=\"2\"><span class=\"w\">  <\/span><span class=\"s2\">&quot;randomIntervalHours&quot;<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"kc\">null<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"3\"><span class=\"w\">  <\/span><span class=\"s2\">&quot;questions&quot;<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">[]<\/span>\n<\/span><span data-line=\"4\"><span class=\"p\">}<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<p>Oh my, the random interval is null! This is the nice way to indicate explicitly that we don\u2019t want this to be periodic.<\/p>\n<\/section>\n<section id=\"panic-attacks\">\n<h3>Panic Attacks<a class=\"headerlink\" href=\"#panic-attacks\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>Panic attacks require additional information. The biggest of these are the type and the duration.<a class=\"footnote-reference brackets\" href=\"#p\" id=\"id7\" role=\"doc-noteref\"><span class=\"fn-bracket\">[<\/span>2<span class=\"fn-bracket\">]<\/span><\/a><\/p>\n<p>Because panic attacks often cause a distorted sense of time, the durations we query are intentionally vague. Let\u2019s look at the questionnaire file to see what I mean:<\/p>\n<div class=\"highlight-javascript notranslate\" id=\"lgbot-questions-panic-json\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"p\">{<\/span>\n<\/span><span data-line=\"2\"><span class=\"w\">  <\/span><span class=\"s2\">&quot;randomIntervalHours&quot;<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"kc\">null<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"3\"><span class=\"w\">  <\/span><span class=\"s2\">&quot;questions&quot;<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">[{<\/span>\n<\/span><span data-line=\"4\"><span class=\"w\">    <\/span><span class=\"s2\">&quot;question&quot;<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">[<\/span>\n<\/span><span data-line=\"5\"><span class=\"w\">      <\/span><span class=\"s2\">&quot;How long was this panic attack?&quot;<\/span>\n<\/span><span data-line=\"6\"><span class=\"w\">    <\/span><span class=\"p\">],<\/span>\n<\/span><span data-line=\"7\"><span class=\"w\">    <\/span><span class=\"s2\">&quot;index&quot;<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"mf\">1<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"8\"><span class=\"w\">    <\/span><span class=\"s2\">&quot;display&quot;<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"kc\">true<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"9\"><span class=\"w\">    <\/span><span class=\"s2\">&quot;answers&quot;<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">{<\/span>\n<\/span><span data-line=\"10\"><span class=\"w\">      <\/span><span class=\"s2\">&quot;For less than 5 minutes&quot;<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"mf\">2.5<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"11\"><span class=\"w\">      <\/span><span class=\"s2\">&quot;For 5-10 minutes&quot;<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"mf\">7.5<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"12\"><span class=\"w\">      <\/span><span class=\"s2\">&quot;For 10-20 minutes&quot;<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"mf\">15<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"13\"><span class=\"w\">      <\/span><span class=\"s2\">&quot;For 20-40 minutes&quot;<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"mf\">30<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"14\"><span class=\"w\">      <\/span><span class=\"s2\">&quot;For 40-80 minutes&quot;<\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"mf\">60<\/span>\n<\/span><span data-line=\"15\"><span class=\"w\">    <\/span><span class=\"p\">}<\/span>\n<\/span><span data-line=\"16\"><span class=\"w\">  <\/span><span class=\"p\">}]<\/span>\n<\/span><span data-line=\"17\"><span class=\"p\">}<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<p>By binning the responses like this, we get an idea of severity while minimizing the friction involved in figuring that out. Additionally, we support a second command for this: <code class=\"docutils literal notranslate\"><span class=\"pre\">\/flashback<\/span><\/code>. These two are sometimes difficult to distinguish, but the main difference is in emotional context. Panic is about things happening in the present, in the now. A flashback is a memory rearing its head in some way. Sometimes this is as vivid as feeling physically in that moment. Sometimes it is \u201cmerely\u201d reliving the emotional context, feeling the internal sensations replayed.<\/p>\n<p>The query for this is incredibly simple:<\/p>\n<div class=\"highlight-PSQL notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"k\">SELECT<\/span>\n<\/span><span data-line=\"2\"><span class=\"w\">  <\/span><span class=\"s s-Name\">&quot;time&quot;<\/span><span class=\"w\"> <\/span><span class=\"k\">AS<\/span><span class=\"w\"> <\/span><span class=\"n\">ts<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"3\"><span class=\"w\">  <\/span><span class=\"n\">respondant<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"4\"><span class=\"w\">  <\/span><span class=\"s s-Name\">&quot;data&quot;<\/span><span class=\"w\"> <\/span><span class=\"k\">AS<\/span><span class=\"w\"> <\/span><span class=\"n\">Panic<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"5\"><span class=\"w\">  <\/span><span class=\"k\">NULL<\/span><span class=\"w\"> <\/span><span class=\"k\">AS<\/span><span class=\"w\"> <\/span><span class=\"n\">Flashback<\/span>\n<\/span><span data-line=\"6\"><span class=\"k\">FROM<\/span><span class=\"w\"> <\/span><span class=\"n\">lgbot<\/span><span class=\"mf\">.<\/span><span class=\"n\">panic<\/span>\n<\/span><span data-line=\"7\"><span class=\"k\">WHERE<\/span><span class=\"w\"> <\/span><span class=\"err\">$<\/span><span class=\"n\">__timeFilter<\/span><span class=\"p\">(<\/span><span class=\"s s-Name\">&quot;time&quot;<\/span><span class=\"p\">)<\/span>\n<\/span><span data-line=\"8\"><span class=\"k\">UNION<\/span><span class=\"w\"> <\/span><span class=\"k\">ALL<\/span>\n<\/span><span data-line=\"9\"><span class=\"k\">SELECT<\/span>\n<\/span><span data-line=\"10\"><span class=\"w\">  <\/span><span class=\"s s-Name\">&quot;time&quot;<\/span><span class=\"w\"> <\/span><span class=\"k\">AS<\/span><span class=\"w\"> <\/span><span class=\"n\">ts<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"11\"><span class=\"w\">  <\/span><span class=\"n\">respondant<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"12\"><span class=\"w\">  <\/span><span class=\"k\">NULL<\/span><span class=\"w\"> <\/span><span class=\"k\">AS<\/span><span class=\"w\"> <\/span><span class=\"n\">Panic<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"13\"><span class=\"w\">  <\/span><span class=\"s s-Name\">&quot;data&quot;<\/span><span class=\"w\"> <\/span><span class=\"k\">AS<\/span><span class=\"w\"> <\/span><span class=\"n\">Flashback<\/span>\n<\/span><span data-line=\"14\"><span class=\"k\">FROM<\/span><span class=\"w\"> <\/span><span class=\"n\">lgbot<\/span><span class=\"mf\">.<\/span><span class=\"n\">flashback<\/span>\n<\/span><span data-line=\"15\"><span class=\"k\">WHERE<\/span><span class=\"w\"> <\/span><span class=\"err\">$<\/span><span class=\"n\">__timeFilter<\/span><span class=\"p\">(<\/span><span class=\"s s-Name\">&quot;time&quot;<\/span><span class=\"p\">)<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<figure class=\"align-default\" id=\"id142\">\n<img alt=\"A chart noting panic attacks and flashbacks over time, as described in the caption below\" src=\"..\/_images\/panic.png\" \/>\n<figcaption>\n<p><span class=\"caption-text\">This chart shows panic attacks and flashbacks over time. The Y axis measures rough duration of the events, with higher values being worse. This is rendered as a scatter plot, with a different color representing each type of event. Shown are two panic attacks, with rough durations of 15 and 30 minutes respectively. Additionally, there are many dashed vertical lines that are used to denote events related to mental health. Not shown, you can click on them to see details of the events. Two of these correspond with the panic attacks, providing context.<\/span><a class=\"headerlink\" href=\"#id142\" title=\"Link to this image\">\u00b6<\/a><\/p>\n<\/figcaption>\n<\/figure>\n<\/section>\n<\/section>\n<section id=\"id8\">\n<h2>Plural Metadata<a class=\"headerlink\" href=\"#id8\" title=\"Link to this heading\">\u00b6<\/a><\/h2>\n<p>If you aren\u2019t familiar with the topic, you might not know at all what I mean by plurality. There is a lot of nuance here that I am intentionally skipping over, but <em>in general<\/em> a plural person is someone who experiences multiple identity states<a class=\"footnote-reference brackets\" href=\"#id19\" id=\"id9\" role=\"doc-noteref\"><span class=\"fn-bracket\">[<\/span>3<span class=\"fn-bracket\">]<\/span><\/a><a class=\"footnote-reference brackets\" href=\"#id20\" id=\"id10\" role=\"doc-noteref\"><span class=\"fn-bracket\">[<\/span>4<span class=\"fn-bracket\">]<\/span><\/a>. These are often called alters<a class=\"footnote-reference brackets\" href=\"#id21\" id=\"id11\" role=\"doc-noteref\"><span class=\"fn-bracket\">[<\/span>5<span class=\"fn-bracket\">]<\/span><\/a>, facets, or headmates, and this last shall be used throughout this entry. <em>In general<\/em> there are differences in memory, personality, skill sets, and\/or triggers between headmates. Because of that, you can <em>(in general)<\/em> think of them as having <em>(descriptive)<\/em> roles for the functioning of the whole. These roles can vary from hyper-specific (they clean our ears) to incredibly general (they handle day-to-day functioning), and some reject the notion of even descriptive roles.<\/p>\n<p>The (set of) headmate(s) that are directly influencing the body are said to be \u201cfronting\u201d. The standard analogy for it is to imagine the body as a car, ship, or plane. You can have a single pilot on some aircraft, but others might need a navigator or an engineer as well. If you\u2019re operating the vehicle, you\u2019re in the front seats, hence fronting.<\/p>\n<p>Knowing that much, you can see why it might be useful to track information about the state of your system, your set of headmates. Having some idea of who is fronting during particular times can be vital to figuring out what works best. This is even more important because systems often suffer from memory issues. In my experience from knowing many, that is particularly true of ordering events.<\/p>\n<section id=\"who-is-most-helped-by-this\">\n<h3>Who Is Most Helped By This?<a class=\"headerlink\" href=\"#who-is-most-helped-by-this\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>I think of plurality as having three axes: population <span id=\"id12\">[<a class=\"reference internal\" href=\"#id32\" title=\"The Woodland System. Headcount - pluralpedia. URL: https:\/\/pluralpedia.org\/index.php?title=Headcount&amp;oldid=58294 (visited on 2026-02-08).\">Sys<\/a>]<\/span>, partitionality <span id=\"id13\">[<a class=\"reference internal\" href=\"#id29\" title=\"The Rainy Day System, The Frost System, A Shattered Dragon's Spirit, Pluralanomaly, NeonWebb, Nicolerenee, and LogarithmicLot. Partitionary - pluralpedia. URL: https:\/\/pluralpedia.org\/index.php?title=Partitionary&amp;oldid=65083 (visited on 2026-02-08).\">SSS+<\/a>]<\/span>, and distinctness <span id=\"id14\">[<a class=\"reference internal\" href=\"#id31\" title=\"LogarithmicLot and Ffcad7. Separation - pluralpedia. URL: https:\/\/pluralpedia.org\/index.php?title=Separation&amp;oldid=61671 (visited on 2026-02-08).\">LFfcad7<\/a>]<\/span>. This isn\u2019t a universal framework, but it does help explain the target audience for this sort of tracking.<\/p>\n<dl class=\"simple\">\n<dt>Population<\/dt><dd><p>Your system is highly populous if there are more alters\/headmates\/facets in it. Polyfragmented systems are at the extreme end of this spectrum. Minimal population for the target audience is 2.<\/p>\n<\/dd>\n<dt>Partitionality<\/dt><dd><p>Your system is highly partitioned if amnesia barriers between headmates are high. The less able you are to remember what others did while away from front, the more partitioned you are.<\/p>\n<\/dd>\n<dt>Distinctness<\/dt><dd><p>Your system is highly distinct if you are able to consistently identify who is at front. Each of you is then clearly <em>distinct<\/em> internally.<\/p>\n<\/dd>\n<\/dl>\n<p>The audience that is most likely to benefit from this kind of tracking are systems that are low in population (of fronters), and high in partitionality and distinctness. If they are high in population, then keeping track of all data points is going to be cumbersome. If they are not very partitioned, then this data is less useful because memory may provide a decent substitute. If they are not very distinct, then determining what to mark in the tracking data is very difficult. While this information is most easily collected for this group, it may still be helpful to others, but on a less consistent basis.<\/p>\n<p>With this target audience in mind, let\u2019s look at what data we can easily gather. All of it can be gathered with a standard time tracker. This specific example is from the <a class=\"reference external\" href=\"https:\/\/joplinapp.org\/plugins\/plugin\/joplin.plugin.alondmnt.time-slip\">Time Slip plugin<\/a> for Joplin. I\u2019ll provide more detail on data extraction in the next entry, but the format we get is:<\/p>\n<div class=\"table-wrapper docutils container\">\n<table class=\"docutils align-default\">\n<thead>\n<tr class=\"row-odd\"><th class=\"head\"><p>Project<\/p><\/th>\n<th class=\"head\"><p>Task<\/p><\/th>\n<th class=\"head\"><p>Start<\/p><\/th>\n<th class=\"head\"><p>End<\/p><\/th>\n<th class=\"head\"><p>Duration<\/p><\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr class=\"row-even\"><td><p>text<\/p><\/td>\n<td><p>text<\/p><\/td>\n<td><p>timestamp<\/p><\/td>\n<td><p>timestamp<\/p><\/td>\n<td><p>interval<\/p><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/section>\n<section id=\"current-fronter-s\">\n<h3>Current Fronter(s)<a class=\"headerlink\" href=\"#current-fronter-s\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>The most conceptually simple metric to look at is who is currently fronting. Unfortunately, the query to find this is fairly complex. There are three complicating factors here.<\/p>\n<p>Firstly, data entry isn\u2019t instantaneous. Because it takes time to mark front changes, and because that will almost always involve at least two events (ex: Alice leaves front, Bob swaps in), you have to deal with alignment issues. There has to be some way to ensure that changes close enough together are considered the same change.<\/p>\n<p>Secondly, we aren\u2019t just looking for a single answer. If it was only possible for one headmate to front at a time, this query would be much simpler. You\u2019d just apply the time filter and look at the most recent entry. Boom, there\u2019s the name. Unfortunately, reality disagrees with this simple model. What we actually need to do is find the last change <em>time<\/em> and look for all events that include that time.<\/p>\n<p>Lastly, we aren\u2019t necessarily looking for the set of fronters at the present moment. We want to look at those fronting in the time range displayed. While this is <em>often<\/em> the present, it isn\u2019t always. That means we can\u2019t use any shortcuts to generate this list.<\/p>\n<p>Given all of the above, this query runs in three stages.<\/p>\n<p>In the first stage, we combine the starts and ends into a single subquery. In the next step, we compare that against the list of events, grabbing a list of every headmate who is fronting at a recorded event time. In the last step, we filter out any event that occurs less than 20 seconds before another. The whole time, we are on watch for those who are <em>currently<\/em> fronting, who will have a <code class=\"code highlight sql PSQL docutils literal highlight-PSQL\"><span class=\"k\">NULL<\/span><\/code> stop time. We then output three values: a list of fronters, the time of last update (in the sample range), and a duration.<\/p>\n<div class=\"highlight-PSQL notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"k\">WITH<\/span><span class=\"w\"> <\/span><span class=\"n\">x<\/span><span class=\"w\"> <\/span><span class=\"k\">AS<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span>\n<\/span><span data-line=\"2\"><span class=\"w\">  <\/span><span class=\"k\">SELECT<\/span><span class=\"w\"> <\/span><span class=\"k\">start<\/span><span class=\"w\"> <\/span><span class=\"k\">AS<\/span><span class=\"w\"> <\/span><span class=\"n\">chrono<\/span><span class=\"w\"> <\/span><span class=\"k\">FROM<\/span><span class=\"w\"> <\/span><span class=\"n\">joplin<\/span><span class=\"mf\">.<\/span><span class=\"n\">front_tracking<\/span>\n<\/span><span data-line=\"3\"><span class=\"w\">  <\/span><span class=\"k\">UNION<\/span>\n<\/span><span data-line=\"4\"><span class=\"w\">  <\/span><span class=\"k\">SELECT<\/span><span class=\"w\"> <\/span><span class=\"n\">stop<\/span><span class=\"w\"> <\/span><span class=\"k\">AS<\/span><span class=\"w\"> <\/span><span class=\"n\">chrono<\/span><span class=\"w\"> <\/span><span class=\"k\">FROM<\/span><span class=\"w\"> <\/span><span class=\"n\">joplin<\/span><span class=\"mf\">.<\/span><span class=\"n\">front_tracking<\/span>\n<\/span><span data-line=\"5\"><span class=\"p\">),<\/span>\n<\/span><span data-line=\"6\">\n<\/span><span data-line=\"7\"><span class=\"n\">y<\/span><span class=\"w\"> <\/span><span class=\"k\">AS<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span>\n<\/span><span data-line=\"8\"><span class=\"w\">  <\/span><span class=\"k\">SELECT<\/span>\n<\/span><span data-line=\"9\"><span class=\"w\">    <\/span><span class=\"n\">x<\/span><span class=\"mf\">.<\/span><span class=\"n\">chrono<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"10\"><span class=\"w\">    <\/span><span class=\"n\">string_agg<\/span><span class=\"p\">(<\/span><span class=\"n\">task<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;, &#39;<\/span><span class=\"w\"> <\/span><span class=\"k\">ORDER<\/span><span class=\"w\"> <\/span><span class=\"k\">BY<\/span><span class=\"w\"> <\/span><span class=\"n\">task<\/span><span class=\"p\">)<\/span><span class=\"w\"> <\/span><span class=\"k\">AS<\/span><span class=\"w\"> <\/span><span class=\"n\">fronters<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"11\"><span class=\"w\">    <\/span><span class=\"p\">(<\/span><span class=\"n\">lead<\/span><span class=\"p\">(<\/span><span class=\"n\">chrono<\/span><span class=\"p\">)<\/span><span class=\"w\"> <\/span><span class=\"k\">OVER<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span><span class=\"k\">ORDER<\/span><span class=\"w\"> <\/span><span class=\"k\">BY<\/span><span class=\"w\"> <\/span><span class=\"n\">chrono<\/span><span class=\"p\">))<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"w\"> <\/span><span class=\"n\">chrono<\/span><span class=\"w\"> <\/span><span class=\"k\">AS<\/span><span class=\"w\"> <\/span><span class=\"n\">delta<\/span>\n<\/span><span data-line=\"12\"><span class=\"w\">  <\/span><span class=\"k\">FROM<\/span><span class=\"w\"> <\/span><span class=\"n\">x<\/span><span class=\"w\"> <\/span><span class=\"k\">LEFT<\/span><span class=\"w\"> <\/span><span class=\"k\">OUTER<\/span><span class=\"w\"> <\/span><span class=\"k\">JOIN<\/span><span class=\"w\"> <\/span><span class=\"n\">joplin<\/span><span class=\"mf\">.<\/span><span class=\"n\">front_tracking<\/span>\n<\/span><span data-line=\"13\"><span class=\"w\">    <\/span><span class=\"k\">ON<\/span><span class=\"w\"> <\/span><span class=\"n\">x<\/span><span class=\"mf\">.<\/span><span class=\"n\">chrono<\/span><span class=\"w\"> <\/span><span class=\"o\">&gt;=<\/span><span class=\"w\"> <\/span><span class=\"k\">start<\/span><span class=\"w\"> <\/span><span class=\"k\">AND<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span><span class=\"n\">x<\/span><span class=\"mf\">.<\/span><span class=\"n\">chrono<\/span><span class=\"w\"> <\/span><span class=\"o\">&lt;<\/span><span class=\"w\"> <\/span><span class=\"n\">stop<\/span><span class=\"w\"> <\/span><span class=\"k\">OR<\/span><span class=\"w\"> <\/span><span class=\"n\">stop<\/span><span class=\"w\"> <\/span><span class=\"k\">IS<\/span><span class=\"w\"> <\/span><span class=\"k\">NULL<\/span><span class=\"p\">)<\/span>\n<\/span><span data-line=\"14\"><span class=\"w\">  <\/span><span class=\"k\">WHERE<\/span><span class=\"w\"> <\/span><span class=\"n\">chrono<\/span><span class=\"w\"> <\/span><span class=\"k\">IS<\/span><span class=\"w\"> <\/span><span class=\"k\">NOT<\/span><span class=\"w\"> <\/span><span class=\"k\">NULL<\/span>\n<\/span><span data-line=\"15\"><span class=\"w\">  <\/span><span class=\"k\">GROUP<\/span><span class=\"w\"> <\/span><span class=\"k\">BY<\/span><span class=\"w\"> <\/span><span class=\"mf\">1<\/span><span class=\"w\"> <\/span><span class=\"k\">ORDER<\/span><span class=\"w\"> <\/span><span class=\"k\">BY<\/span><span class=\"w\"> <\/span><span class=\"n\">chrono<\/span><span class=\"w\"> <\/span><span class=\"k\">DESC<\/span>\n<\/span><span data-line=\"16\"><span class=\"p\">)<\/span>\n<\/span><span data-line=\"17\">\n<\/span><span data-line=\"18\"><span class=\"k\">SELECT<\/span>\n<\/span><span data-line=\"19\"><span class=\"w\">  <\/span><span class=\"n\">fronters<\/span><span class=\"w\"> <\/span><span class=\"k\">AS<\/span><span class=\"w\"> <\/span><span class=\"s s-Name\">&quot;Fronter(s)&quot;<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"20\"><span class=\"w\">  <\/span><span class=\"k\">EXTRACT<\/span><span class=\"p\">(<\/span><span class=\"n\">EPOCH<\/span><span class=\"w\"> <\/span><span class=\"k\">FROM<\/span><span class=\"w\"> <\/span><span class=\"k\">CASE<\/span>\n<\/span><span data-line=\"21\"><span class=\"w\">    <\/span><span class=\"k\">WHEN<\/span><span class=\"w\"> <\/span><span class=\"n\">delta<\/span><span class=\"w\"> <\/span><span class=\"k\">IS<\/span><span class=\"w\"> <\/span><span class=\"k\">NOT<\/span><span class=\"w\"> <\/span><span class=\"k\">NULL<\/span>\n<\/span><span data-line=\"22\"><span class=\"w\">    <\/span><span class=\"k\">THEN<\/span><span class=\"w\"> <\/span><span class=\"n\">delta<\/span>\n<\/span><span data-line=\"23\"><span class=\"w\">    <\/span><span class=\"k\">ELSE<\/span><span class=\"w\"> <\/span><span class=\"n\">NOW<\/span><span class=\"p\">()<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"w\"> <\/span><span class=\"n\">chrono<\/span>\n<\/span><span data-line=\"24\"><span class=\"w\">    <\/span><span class=\"k\">END<\/span><span class=\"p\">)<\/span><span class=\"w\"> <\/span><span class=\"k\">AS<\/span><span class=\"w\"> <\/span><span class=\"s s-Name\">&quot;For&quot;<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"25\"><span class=\"w\">  <\/span><span class=\"n\">chrono<\/span><span class=\"w\"> <\/span><span class=\"k\">AS<\/span><span class=\"w\"> <\/span><span class=\"s s-Name\">&quot;Updated At&quot;<\/span>\n<\/span><span data-line=\"26\"><span class=\"k\">FROM<\/span><span class=\"w\"> <\/span><span class=\"n\">y<\/span>\n<\/span><span data-line=\"27\"><span class=\"k\">WHERE<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span><span class=\"n\">delta<\/span><span class=\"w\"> <\/span><span class=\"k\">IS<\/span><span class=\"w\"> <\/span><span class=\"k\">NULL<\/span><span class=\"w\"> <\/span><span class=\"k\">OR<\/span><span class=\"w\"> <\/span><span class=\"n\">delta<\/span><span class=\"w\"> <\/span><span class=\"o\">&gt;<\/span><span class=\"w\"> <\/span><span class=\"nb\">interval<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;20 seconds&#39;<\/span><span class=\"p\">)<\/span>\n<\/span><span data-line=\"28\"><span class=\"w\">  <\/span><span class=\"k\">AND<\/span><span class=\"w\"> <\/span><span class=\"err\">$<\/span><span class=\"n\">__timeFilter<\/span><span class=\"p\">(<\/span><span class=\"n\">chrono<\/span><span class=\"p\">)<\/span>\n<\/span><span data-line=\"29\"><span class=\"k\">LIMIT<\/span><span class=\"w\"> <\/span><span class=\"mf\">1<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<figure class=\"align-default\" id=\"id143\">\n<img alt=\"A panel that displays information about the current group of fronters, as described below\" src=\"..\/_images\/current.png\" \/>\n<figcaption>\n<p><span class=\"caption-text\">This is the resulting panel, holding three values. In the first, we have the group of current fronters. At the moment that is Alice. In the next column you can see how long she been fronting solo for: 5.43 hours. The next column shows the time of the most recent front change.<\/span><a class=\"headerlink\" href=\"#id143\" title=\"Link to this image\">\u00b6<\/a><\/p>\n<\/figcaption>\n<\/figure>\n<\/section>\n<section id=\"front-time-naive-version\">\n<h3>Front Time - Naive Version<a class=\"headerlink\" href=\"#front-time-naive-version\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>The easiest metric after that would be to look at total front time. Naively, we could simply show it as a percentage. This is naive both because it is simple, and because it is flawed. If you are a very partitioned, distinct, small system this may work for you, but if you ever have more than one headmate fronting this will be an erroneous measure. It is excellent for giving a broad picture, but it tells you nothing about shared time, and can often add up to more than 24 hours<a class=\"footnote-reference brackets\" href=\"#id22\" id=\"id15\" role=\"doc-noteref\"><span class=\"fn-bracket\">[<\/span>6<span class=\"fn-bracket\">]<\/span><\/a><a class=\"footnote-reference brackets\" href=\"#id23\" id=\"id16\" role=\"doc-noteref\"><span class=\"fn-bracket\">[<\/span>7<span class=\"fn-bracket\">]<\/span><\/a>.<\/p>\n<p>This dashboard includes two versions: one that includes sleep, and one that does not.<\/p>\n<p>Fortunately, this query <em>actually is<\/em> simple.<\/p>\n<div class=\"highlight-PSQL notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"k\">SELECT<\/span><span class=\"w\"> <\/span><span class=\"k\">DISTINCT<\/span>\n<\/span><span data-line=\"2\"><span class=\"w\">  <\/span><span class=\"n\">task<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"3\"><span class=\"w\">  <\/span><span class=\"k\">EXTRACT<\/span><span class=\"p\">(<\/span><span class=\"n\">EPOCH<\/span><span class=\"w\"> <\/span><span class=\"k\">FROM<\/span><span class=\"w\"> <\/span><span class=\"n\">SUM<\/span><span class=\"p\">(<\/span><span class=\"k\">CASE<\/span>\n<\/span><span data-line=\"4\"><span class=\"w\">    <\/span><span class=\"k\">WHEN<\/span><span class=\"w\"> <\/span><span class=\"n\">duration<\/span><span class=\"w\"> <\/span><span class=\"k\">IS<\/span><span class=\"w\"> <\/span><span class=\"k\">NULL<\/span>\n<\/span><span data-line=\"5\"><span class=\"w\">    <\/span><span class=\"k\">THEN<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span><span class=\"nb\">TIMESTAMP<\/span><span class=\"w\"> <\/span><span class=\"nb\">WITH TIME ZONE<\/span><span class=\"w\"> <\/span><span class=\"err\">$<\/span><span class=\"n\">__timeTo<\/span><span class=\"p\">()<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"w\"> <\/span><span class=\"k\">start<\/span><span class=\"p\">)<\/span>\n<\/span><span data-line=\"6\"><span class=\"w\">    <\/span><span class=\"k\">ELSE<\/span><span class=\"w\"> <\/span><span class=\"n\">duration<\/span>\n<\/span><span data-line=\"7\"><span class=\"w\">  <\/span><span class=\"k\">END<\/span><span class=\"p\">)<\/span><span class=\"w\"> <\/span><span class=\"k\">OVER<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span><span class=\"k\">PARTITION<\/span><span class=\"w\"> <\/span><span class=\"k\">BY<\/span><span class=\"w\"> <\/span><span class=\"n\">task<\/span><span class=\"p\">))<\/span><span class=\"w\"> <\/span><span class=\"k\">AS<\/span><span class=\"w\"> <\/span><span class=\"n\">duration<\/span>\n<\/span><span data-line=\"8\"><span class=\"k\">FROM<\/span><span class=\"w\"> <\/span><span class=\"n\">joplin<\/span><span class=\"mf\">.<\/span><span class=\"n\">front_tracking<\/span>\n<\/span><span data-line=\"9\"><span class=\"k\">WHERE<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span><span class=\"err\">$<\/span><span class=\"n\">__timeFilter<\/span><span class=\"p\">(<\/span><span class=\"n\">stop<\/span><span class=\"p\">)<\/span><span class=\"w\"> <\/span><span class=\"k\">OR<\/span><span class=\"w\"> <\/span><span class=\"n\">stop<\/span><span class=\"w\"> <\/span><span class=\"k\">IS<\/span><span class=\"w\"> <\/span><span class=\"k\">NULL<\/span><span class=\"p\">)<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<figure class=\"align-default\" id=\"id144\">\n<img alt=\"A pair of pie charts displaying relative front times, as described below\" src=\"..\/_images\/naive.png\" \/>\n<figcaption>\n<p><span class=\"caption-text\">This pair of pie charts shows relative front time when computed naively. Each headmate gets a specific color, keeping things consistent. In the upper right corner of each is a legend showing the color, name, value, and percentage for each of them. The one on the left includes sleep, and the one on the right does not.<\/span><a class=\"headerlink\" href=\"#id144\" title=\"Link to this image\">\u00b6<\/a><\/p>\n<\/figcaption>\n<\/figure>\n<\/section>\n<section id=\"front-time-combination-aware\">\n<h3>Front Time - Combination Aware<a class=\"headerlink\" href=\"#front-time-combination-aware\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>A better version of the above would be able to account for those combinations. In this one, our display is modified to list each combination separately, using a canonical ordering to prevent ambiguity. This lets you get much more useful data about system dynamics. For one example, it may be that Carly and Darryl almost always front together.<\/p>\n<p>We can do this by incorporating the concepts we established in the Current Fronters chart. The first two portions are entirely identical, applying the same logic at each step. At the last part, instead of returning a single row we are aggregating the total amount of time for each combination of fronters in the time range.<\/p>\n<div class=\"highlight-PSQL notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"k\">WITH<\/span><span class=\"w\"> <\/span><span class=\"n\">x<\/span><span class=\"w\"> <\/span><span class=\"k\">AS<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span>\n<\/span><span data-line=\"2\"><span class=\"w\">  <\/span><span class=\"k\">SELECT<\/span><span class=\"w\"> <\/span><span class=\"k\">start<\/span><span class=\"w\"> <\/span><span class=\"k\">AS<\/span><span class=\"w\"> <\/span><span class=\"n\">chrono<\/span><span class=\"w\"> <\/span><span class=\"k\">FROM<\/span><span class=\"w\"> <\/span><span class=\"n\">joplin<\/span><span class=\"mf\">.<\/span><span class=\"n\">front_tracking<\/span>\n<\/span><span data-line=\"3\"><span class=\"w\">  <\/span><span class=\"k\">UNION<\/span>\n<\/span><span data-line=\"4\"><span class=\"w\">  <\/span><span class=\"k\">SELECT<\/span><span class=\"w\"> <\/span><span class=\"n\">stop<\/span><span class=\"w\"> <\/span><span class=\"k\">AS<\/span><span class=\"w\"> <\/span><span class=\"n\">chrono<\/span><span class=\"w\"> <\/span><span class=\"k\">FROM<\/span><span class=\"w\"> <\/span><span class=\"n\">joplin<\/span><span class=\"mf\">.<\/span><span class=\"n\">front_tracking<\/span>\n<\/span><span data-line=\"5\"><span class=\"p\">),<\/span>\n<\/span><span data-line=\"6\">\n<\/span><span data-line=\"7\"><span class=\"n\">y<\/span><span class=\"w\"> <\/span><span class=\"k\">AS<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span>\n<\/span><span data-line=\"8\"><span class=\"w\">  <\/span><span class=\"k\">SELECT<\/span>\n<\/span><span data-line=\"9\"><span class=\"w\">    <\/span><span class=\"n\">x<\/span><span class=\"mf\">.<\/span><span class=\"n\">chrono<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"10\"><span class=\"w\">    <\/span><span class=\"n\">string_agg<\/span><span class=\"p\">(<\/span><span class=\"n\">task<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;, &#39;<\/span><span class=\"w\"> <\/span><span class=\"k\">ORDER<\/span><span class=\"w\"> <\/span><span class=\"k\">BY<\/span><span class=\"w\"> <\/span><span class=\"n\">task<\/span><span class=\"p\">)<\/span><span class=\"w\"> <\/span><span class=\"k\">AS<\/span><span class=\"w\"> <\/span><span class=\"n\">fronters<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"11\"><span class=\"w\">    <\/span><span class=\"p\">(<\/span><span class=\"n\">lead<\/span><span class=\"p\">(<\/span><span class=\"n\">chrono<\/span><span class=\"p\">)<\/span><span class=\"w\"> <\/span><span class=\"k\">OVER<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span><span class=\"k\">ORDER<\/span><span class=\"w\"> <\/span><span class=\"k\">BY<\/span><span class=\"w\"> <\/span><span class=\"n\">chrono<\/span><span class=\"p\">))<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"w\"> <\/span><span class=\"n\">chrono<\/span><span class=\"w\"> <\/span><span class=\"k\">AS<\/span><span class=\"w\"> <\/span><span class=\"n\">delta<\/span>\n<\/span><span data-line=\"12\"><span class=\"w\">  <\/span><span class=\"k\">FROM<\/span><span class=\"w\"> <\/span><span class=\"n\">x<\/span><span class=\"w\"> <\/span><span class=\"k\">LEFT<\/span><span class=\"w\"> <\/span><span class=\"k\">OUTER<\/span><span class=\"w\"> <\/span><span class=\"k\">JOIN<\/span><span class=\"w\"> <\/span><span class=\"n\">joplin<\/span><span class=\"mf\">.<\/span><span class=\"n\">front_tracking<\/span>\n<\/span><span data-line=\"13\"><span class=\"w\">    <\/span><span class=\"k\">ON<\/span><span class=\"w\"> <\/span><span class=\"n\">x<\/span><span class=\"mf\">.<\/span><span class=\"n\">chrono<\/span><span class=\"w\"> <\/span><span class=\"o\">&gt;=<\/span><span class=\"w\"> <\/span><span class=\"k\">start<\/span><span class=\"w\"> <\/span><span class=\"k\">AND<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span><span class=\"n\">x<\/span><span class=\"mf\">.<\/span><span class=\"n\">chrono<\/span><span class=\"w\"> <\/span><span class=\"o\">&lt;<\/span><span class=\"w\"> <\/span><span class=\"n\">stop<\/span><span class=\"w\"> <\/span><span class=\"k\">OR<\/span><span class=\"w\"> <\/span><span class=\"n\">stop<\/span><span class=\"w\"> <\/span><span class=\"k\">IS<\/span><span class=\"w\"> <\/span><span class=\"k\">NULL<\/span><span class=\"p\">)<\/span>\n<\/span><span data-line=\"14\"><span class=\"w\">  <\/span><span class=\"k\">WHERE<\/span><span class=\"w\"> <\/span><span class=\"n\">chrono<\/span><span class=\"w\"> <\/span><span class=\"k\">IS<\/span><span class=\"w\"> <\/span><span class=\"k\">NOT<\/span><span class=\"w\"> <\/span><span class=\"k\">NULL<\/span>\n<\/span><span data-line=\"15\"><span class=\"w\">  <\/span><span class=\"k\">GROUP<\/span><span class=\"w\"> <\/span><span class=\"k\">BY<\/span><span class=\"w\"> <\/span><span class=\"mf\">1<\/span><span class=\"w\"> <\/span><span class=\"k\">ORDER<\/span><span class=\"w\"> <\/span><span class=\"k\">BY<\/span><span class=\"w\"> <\/span><span class=\"n\">chrono<\/span><span class=\"w\"> <\/span><span class=\"k\">DESC<\/span>\n<\/span><span data-line=\"16\"><span class=\"p\">)<\/span>\n<\/span><span data-line=\"17\">\n<\/span><span data-line=\"18\"><span class=\"k\">SELECT<\/span>\n<\/span><span data-line=\"19\"><span class=\"w\">  <\/span><span class=\"n\">fronters<\/span><span class=\"w\"> <\/span><span class=\"k\">AS<\/span><span class=\"w\"> <\/span><span class=\"s s-Name\">&quot;Fronter(s)&quot;<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"20\"><span class=\"w\">  <\/span><span class=\"k\">EXTRACT<\/span><span class=\"p\">(<\/span><span class=\"n\">EPOCH<\/span><span class=\"w\"> <\/span><span class=\"k\">FROM<\/span><span class=\"w\"> <\/span><span class=\"n\">SUM<\/span><span class=\"p\">(<\/span><span class=\"k\">CASE<\/span>\n<\/span><span data-line=\"21\"><span class=\"w\">    <\/span><span class=\"k\">WHEN<\/span><span class=\"w\"> <\/span><span class=\"n\">delta<\/span><span class=\"w\"> <\/span><span class=\"k\">IS<\/span><span class=\"w\"> <\/span><span class=\"k\">NOT<\/span><span class=\"w\"> <\/span><span class=\"k\">NULL<\/span>\n<\/span><span data-line=\"22\"><span class=\"w\">    <\/span><span class=\"k\">THEN<\/span><span class=\"w\"> <\/span><span class=\"n\">delta<\/span>\n<\/span><span data-line=\"23\"><span class=\"w\">    <\/span><span class=\"k\">ELSE<\/span><span class=\"w\"> <\/span><span class=\"n\">NOW<\/span><span class=\"p\">()<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"w\"> <\/span><span class=\"n\">chrono<\/span>\n<\/span><span data-line=\"24\"><span class=\"w\">    <\/span><span class=\"k\">END<\/span><span class=\"p\">))<\/span>\n<\/span><span data-line=\"25\"><span class=\"k\">FROM<\/span><span class=\"w\"> <\/span><span class=\"n\">y<\/span>\n<\/span><span data-line=\"26\"><span class=\"k\">WHERE<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span><span class=\"n\">delta<\/span><span class=\"w\"> <\/span><span class=\"k\">IS<\/span><span class=\"w\"> <\/span><span class=\"k\">NULL<\/span><span class=\"w\"> <\/span><span class=\"k\">OR<\/span><span class=\"w\"> <\/span><span class=\"n\">delta<\/span><span class=\"w\"> <\/span><span class=\"o\">&gt;<\/span><span class=\"w\"> <\/span><span class=\"nb\">interval<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;20 seconds&#39;<\/span><span class=\"p\">)<\/span>\n<\/span><span data-line=\"27\"><span class=\"w\">  <\/span><span class=\"k\">AND<\/span><span class=\"w\"> <\/span><span class=\"err\">$<\/span><span class=\"n\">__timeFilter<\/span><span class=\"p\">(<\/span><span class=\"n\">chrono<\/span><span class=\"p\">)<\/span>\n<\/span><span data-line=\"28\"><span class=\"k\">GROUP<\/span><span class=\"w\"> <\/span><span class=\"k\">BY<\/span><span class=\"w\"> <\/span><span class=\"s s-Name\">&quot;Fronter(s)&quot;<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<figure class=\"align-default\" id=\"id145\">\n<img alt=\"A pair of pie charts displaying relative front times, as described below\" src=\"..\/_images\/combos.png\" \/>\n<figcaption>\n<p><span class=\"caption-text\">This pair of pie charts shows relative front time when computed with combinations in mind. Groupings are labelled as a + separated string. Each grouping gets a specific color, keeping things consistent. In the upper right corner of each is a legend showing the color, name, value, and percentage for each of them. The one on the left includes sleep, and the one on the right does not.<\/span><a class=\"headerlink\" href=\"#id145\" title=\"Link to this image\">\u00b6<\/a><\/p>\n<\/figcaption>\n<\/figure>\n<\/section>\n<section id=\"a-timeline\">\n<h3>A Timeline<a class=\"headerlink\" href=\"#a-timeline\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>The other useful view for this data is as a state timeline, as we discussed in <a class=\"reference external\" href=\"https:\/\/blog.oliviaappleton.com\/posts\/0006-lessons-in-grafana-01#state-timelines\">the first entry<\/a> <span id=\"id17\">[<a class=\"reference internal\" href=\"#id27\" title=\"Olivia Appleton-Crocker. Lessons in grafana - part one: a vision. URL: https:\/\/blog.oliviaappleton.com\/posts\/0006-lessons-in-grafana-01.\">ACb<\/a>]<\/span>. Usefully, the time tracking plugin gives us data in precisely the format we need. The only post-processing we need is to apply the standard partition-by-values filter that we\u2019ve used in most of these other panels.<\/p>\n<div class=\"highlight-PSQL notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"k\">SELECT<\/span>\n<\/span><span data-line=\"2\"><span class=\"w\">  <\/span><span class=\"k\">start<\/span><span class=\"w\"> <\/span><span class=\"k\">AS<\/span><span class=\"w\"> <\/span><span class=\"n\">ts<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"3\"><span class=\"w\">  <\/span><span class=\"k\">CASE<\/span>\n<\/span><span data-line=\"4\"><span class=\"w\">    <\/span><span class=\"k\">WHEN<\/span><span class=\"w\"> <\/span><span class=\"n\">stop<\/span><span class=\"w\"> <\/span><span class=\"k\">IS<\/span><span class=\"w\"> <\/span><span class=\"k\">NULL<\/span>\n<\/span><span data-line=\"5\"><span class=\"w\">    <\/span><span class=\"k\">THEN<\/span><span class=\"w\"> <\/span><span class=\"err\">$<\/span><span class=\"n\">__timeTo<\/span><span class=\"p\">()<\/span>\n<\/span><span data-line=\"6\"><span class=\"w\">    <\/span><span class=\"k\">ELSE<\/span><span class=\"w\"> <\/span><span class=\"n\">stop<\/span>\n<\/span><span data-line=\"7\"><span class=\"w\">    <\/span><span class=\"k\">END<\/span><span class=\"w\"> <\/span><span class=\"k\">AS<\/span><span class=\"w\"> <\/span><span class=\"n\">stop<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"8\"><span class=\"w\">  <\/span><span class=\"n\">task<\/span>\n<\/span><span data-line=\"9\"><span class=\"k\">FROM<\/span><span class=\"w\"> <\/span><span class=\"n\">joplin<\/span><span class=\"mf\">.<\/span><span class=\"n\">front_tracking<\/span>\n<\/span><span data-line=\"10\">\n<\/span><span data-line=\"11\"><span class=\"k\">WHERE<\/span><span class=\"w\"> <\/span><span class=\"err\">$<\/span><span class=\"n\">__timeFilter<\/span><span class=\"p\">(<\/span><span class=\"k\">start<\/span><span class=\"p\">)<\/span><span class=\"w\"> <\/span><span class=\"k\">OR<\/span><span class=\"w\"> <\/span><span class=\"err\">$<\/span><span class=\"n\">__timeFilter<\/span><span class=\"p\">(<\/span><span class=\"n\">stop<\/span><span class=\"p\">)<\/span>\n<\/span><span data-line=\"12\"><span class=\"k\">ORDER<\/span><span class=\"w\"> <\/span><span class=\"k\">BY<\/span>\n<\/span><span data-line=\"13\"><span class=\"w\">  <\/span><span class=\"p\">(<\/span><span class=\"n\">task<\/span><span class=\"w\"> <\/span><span class=\"o\">=<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;Sleep (Attempt)&#39;<\/span><span class=\"p\">)<\/span><span class=\"w\"> <\/span><span class=\"k\">DESC<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"14\"><span class=\"w\">  <\/span><span class=\"n\">sum<\/span><span class=\"p\">(<\/span><span class=\"n\">duration<\/span><span class=\"p\">)<\/span><span class=\"w\"> <\/span><span class=\"k\">OVER<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span><span class=\"k\">PARTITION<\/span><span class=\"w\"> <\/span><span class=\"k\">BY<\/span><span class=\"w\"> <\/span><span class=\"n\">task<\/span><span class=\"p\">)<\/span><span class=\"w\"> <\/span><span class=\"k\">DESC<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"15\"><span class=\"w\">  <\/span><span class=\"k\">start<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"16\"><span class=\"w\">  <\/span><span class=\"n\">task<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<p>That <code class=\"code highlight sql PSQL docutils literal highlight-PSQL\"><span class=\"k\">ORDER<\/span><span class=\"w\"> <\/span><span class=\"k\">BY<\/span><\/code> clause might look a bit odd, so let\u2019s go through it step by step. First, we order by a boolean. This is forcing the sleep tracking to the top, since it\u2019s unlike other entries in the timeline. Next, we order by duration. In the unlikely event there is a tie, we order by the start value and then by the name of the fronter.<\/p>\n<figure class=\"align-default\" id=\"id146\">\n<img alt=\"A state timeline showing when different headmates were fronting, as described below\" src=\"..\/_images\/timeline.png\" \/>\n<figcaption>\n<p><span class=\"caption-text\">This is the resulting state timeline. Each headmate (and sleeping time) gets a row to mark their times in. When they are fronting, it is highlighted with an assigned color. The headmates are arranged such that the top name fronts the most, and the bottom name the least. Sleep is always at the top. Like several other panels, this has vertical lines that correspond to mental health events (such as arguments, bad news, etc).<\/span><a class=\"headerlink\" href=\"#id146\" title=\"Link to this image\">\u00b6<\/a><\/p>\n<\/figcaption>\n<\/figure>\n<\/section>\n<section id=\"initial-version-with-pre-processing\">\n<h3>Initial Version with Pre-Processing<a class=\"headerlink\" href=\"#initial-version-with-pre-processing\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>During the prototyping stage, I was initially looking for combinations while extracting the data from Joplin. This meant that every <code class=\"code highlight sql PSQL docutils literal highlight-PSQL\"><span class=\"n\">task<\/span><\/code>, once in the database, would look something like <code class=\"code highlight sql PSQL docutils literal highlight-PSQL\"><span class=\"s1\">&#39;Eustace+Franki+Gina&#39;<\/span><\/code>. Any queries that needed individual names would have to parse that out using regular expressions. This was, of course, horribly inefficient. Worse than that, it was brittle and difficult to debug. Being able to do this processing in the query itself is a significant improvement to the developer experience here.<\/p>\n<\/section>\n<\/section>\n<section id=\"final-product\">\n<h2>Final Product<a class=\"headerlink\" href=\"#final-product\" title=\"Link to this heading\">\u00b6<\/a><\/h2>\n<p>I\u2019m quite happy with how I arranged the final dashboards. It was interesting to find an arrangement that kept related information together while not overwhelming. And because (to some extent) this is a demo, I wanted to be sure that I displayed all the interpretations I could reasonably think of. I think I achieved those goals.<\/p>\n<section id=\"system-specific-dashboard\">\n<h3>System-Specific Dashboard<a class=\"headerlink\" href=\"#system-specific-dashboard\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<figure class=\"align-default\" id=\"id147\">\n<img alt=\"The final dashboard for system data, as described below\" src=\"..\/_images\/system_all.png\" \/>\n<figcaption>\n<p><span class=\"caption-text\">This panel shows the variety of system data we are able to gather. On the top, we have the display of current fronters. Below that is the state timeline which highlights the times when different headmates fronted. Below that is three vertical pairs of pie charts. Each of them is organized such that the version including sleep is on top, and the version excluding sleep is below. On the left are the naively-computed front times. In the middle, and larger than the others, are the combination-aware front times. On the right is a panel displaying combination-aware front time across all historical data.<\/span><a class=\"headerlink\" href=\"#id147\" title=\"Link to this image\">\u00b6<\/a><\/p>\n<\/figcaption>\n<\/figure>\n<\/section>\n<section id=\"general-mental-health-dashboard\">\n<h3>General Mental Health Dashboard<a class=\"headerlink\" href=\"#general-mental-health-dashboard\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<figure class=\"align-default\" id=\"id148\">\n<img alt=\"The final dashboard for general mental health data, as described below\" src=\"..\/_images\/mental_health_all.png\" \/>\n<figcaption>\n<p><span class=\"caption-text\">This is the end result for the general dashboard. There are four rows in total. The top row shows system data. The top left of it has the panel of current fronters. Below that is the state timeline of them, and to the right is the combination-aware pie chart. In the second row we have depression scores as computed by the PHQ-9. Below that is self-reported anxiety scores. And lastly you have the panic attack time series.<\/span><a class=\"headerlink\" href=\"#id148\" title=\"Link to this image\">\u00b6<\/a><\/p>\n<\/figcaption>\n<\/figure>\n<\/section>\n<\/section>\n<section id=\"main-takeaways\">\n<h2>Main Takeaways<a class=\"headerlink\" href=\"#main-takeaways\" title=\"Link to this heading\">\u00b6<\/a><\/h2>\n<p>This was a passion project, but it was also a source of many frustrations. I spent a long time working out how to lay out the data, how to phrase questions, how to interpret things. Initially I had it writing this to a spreadsheet and using the built-in charts there, and have slowly been migrating it towards being database-only. Right now it saves to both.<\/p>\n<p>The biggest challenge here was the evolving requirements. Because I was iterating over minimum-viable products, it made it difficult to form a comprehensive design. The benefit, of course, was that I always had something working. I had the motivation of watching it improve, rather than knowing that there were plans that didn\u2019t function yet.<\/p>\n<p>Overall, I would say that I greatly enjoyed working on this. There are many things I would do differently, knowing what I do now, but I am quite proud of what I do have.<\/p>\n<\/section>\n<section id=\"acknowledgments\">\n<h2>Acknowledgments<a class=\"headerlink\" href=\"#acknowledgments\" title=\"Link to this heading\">\u00b6<\/a><\/h2>\n<p>I greatly appreciate the friends and family who gave feedback for this dashboard along the way. Extra thanks to my friend <a class=\"reference external\" href=\"https:\/\/linktr.ee\/Ultralee0\">&#64;Ultralee0&#64;linktr.ee<\/a>, without whom you would be reading this un-edited. I shudder at the thought. Lastly, I would like to thank the several sensitivity readers who screened this post and offered meaningful suggestions.<\/p>\n<p class=\"rubric\">Footnotes<\/p>\n<aside class=\"footnote-list brackets\">\n<aside class=\"footnote brackets\" id=\"id18\" role=\"doc-footnote\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span><a role=\"doc-backlink\" href=\"#id5\">1<\/a><span class=\"fn-bracket\">]<\/span><\/span>\n<p>And some horribly-hacky workarounds regarding spreadsheet interactions. Did you know that <code class=\"docutils literal notranslate\"><span class=\"pre\">pyexcel_ods3<\/span><\/code> has a memory leak at global scope <em>every time<\/em> you interact with a spreadsheet? Well, I do now, and it means that <em>every time<\/em> I have to spawn a process just for that purpose or consume infinite memory.<\/p>\n<\/aside>\n<aside class=\"footnote brackets\" id=\"p\" role=\"doc-footnote\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span>2<span class=\"fn-bracket\">]<\/span><\/span>\n<span class=\"backrefs\">(<a role=\"doc-backlink\" href=\"#id6\">1<\/a>,<a role=\"doc-backlink\" href=\"#id7\">2<\/a>)<\/span>\n<p>Or for plural folks, who was fronting at the time<\/p>\n<\/aside>\n<aside class=\"footnote brackets\" id=\"id19\" role=\"doc-footnote\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span><a role=\"doc-backlink\" href=\"#id9\">3<\/a><span class=\"fn-bracket\">]<\/span><\/span>\n<p>I personally am not a fan of this phrase, I\u2019m just not aware of a better one to use in its stead.<\/p>\n<\/aside>\n<aside class=\"footnote brackets\" id=\"id20\" role=\"doc-footnote\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span><a role=\"doc-backlink\" href=\"#id10\">4<\/a><span class=\"fn-bracket\">]<\/span><\/span>\n<p>Not everyone who is plural identifies as a system. Some people find it dehumanizing, prefer other terms, or any myriad of other reasons. I will be using the term in this entry because it seems to be the most common one, but note that it is not the <em>only<\/em> one.<\/p>\n<\/aside>\n<aside class=\"footnote brackets\" id=\"id21\" role=\"doc-footnote\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span><a role=\"doc-backlink\" href=\"#id11\">5<\/a><span class=\"fn-bracket\">]<\/span><\/span>\n<p>This term is sometimes frowned upon because of its origins as a shorthand for \u201calternate personality\u201d or \u201calter ego\u201d.<\/p>\n<\/aside>\n<aside class=\"footnote brackets\" id=\"id22\" role=\"doc-footnote\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span><a role=\"doc-backlink\" href=\"#id15\">6<\/a><span class=\"fn-bracket\">]<\/span><\/span>\n<p>In this scheme I include \u201cSleep\u201d as a fronter for ease of processing. That way you can always assume that there is a value for this field.<\/p>\n<\/aside>\n<aside class=\"footnote brackets\" id=\"id23\" role=\"doc-footnote\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span><a role=\"doc-backlink\" href=\"#id16\">7<\/a><span class=\"fn-bracket\">]<\/span><\/span>\n<p>It also can exceed 24 hours because the query used here doesn\u2019t know how to slice events to fit perfectly in the sample window. If an event is included, its total time is considered. This is a place for possible improvement.<\/p>\n<\/aside>\n<\/aside>\n<p class=\"rubric\">Citations<\/p>\n<div class=\"docutils container\" id=\"id24\">\n<div role=\"list\" class=\"citation-list\">\n<div class=\"citation\" id=\"id25\" role=\"doc-biblioentry\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span><a role=\"doc-backlink\" href=\"#id3\">ACa<\/a><span class=\"fn-bracket\">]<\/span><\/span>\n<p>Olivia Appleton-Crocker. Lessons in grafana - bonus entry: achieving acceptable averages. URL: <a class=\"reference external\" href=\"https:\/\/blog.oliviaappleton.com\/posts\/0008-lessons-in-grafana-02-5\">https:\/\/blog.oliviaappleton.com\/posts\/0008-lessons-in-grafana-02-5<\/a>.<\/p>\n<\/div>\n<div class=\"citation\" id=\"id27\" role=\"doc-biblioentry\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span><a role=\"doc-backlink\" href=\"#id17\">ACb<\/a><span class=\"fn-bracket\">]<\/span><\/span>\n<p>Olivia Appleton-Crocker. Lessons in grafana - part one: a vision. URL: <a class=\"reference external\" href=\"https:\/\/blog.oliviaappleton.com\/posts\/0006-lessons-in-grafana-01\">https:\/\/blog.oliviaappleton.com\/posts\/0006-lessons-in-grafana-01<\/a>.<\/p>\n<\/div>\n<div class=\"citation\" id=\"id26\" role=\"doc-biblioentry\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span><a role=\"doc-backlink\" href=\"#id4\">ACc<\/a><span class=\"fn-bracket\">]<\/span><\/span>\n<p>Olivia Appleton-Crocker. Lessons in grafana - part two: litter logs. URL: <a class=\"reference external\" href=\"https:\/\/blog.oliviaappleton.com\/posts\/0007-lessons-in-grafana-02\">https:\/\/blog.oliviaappleton.com\/posts\/0007-lessons-in-grafana-02<\/a>.<\/p>\n<\/div>\n<div class=\"citation\" id=\"id134\" role=\"doc-biblioentry\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span><a role=\"doc-backlink\" href=\"#id2\">Cur<\/a><span class=\"fn-bracket\">]<\/span><\/span>\n<p>National\u00a0HIV Curriculum. Patient health questionnaire-9 (phq-9). URL: <a class=\"reference external\" href=\"https:\/\/www.hiv.uw.edu\/page\/mental-health-screening\/phq-9\">https:\/\/www.hiv.uw.edu\/page\/mental-health-screening\/phq-9<\/a> (visited on 2026-02-21).<\/p>\n<\/div>\n<div class=\"citation\" id=\"id31\" role=\"doc-biblioentry\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span><a role=\"doc-backlink\" href=\"#id14\">LFfcad7<\/a><span class=\"fn-bracket\">]<\/span><\/span>\n<p>LogarithmicLot and Ffcad7. Separation - pluralpedia. URL: <a class=\"reference external\" href=\"https:\/\/pluralpedia.org\/index.php?title=Separation&amp;oldid=61671\">https:\/\/pluralpedia.org\/index.php?title=Separation&amp;oldid=61671<\/a> (visited on 2026-02-08).<\/p>\n<\/div>\n<div class=\"citation\" id=\"id120\" role=\"doc-biblioentry\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span><a role=\"doc-backlink\" href=\"#id1\">Mit09<\/a><span class=\"fn-bracket\">]<\/span><\/span>\n<p>Alex\u00a0J Mitchell. Why do clinicians have difficulty detecting depression? In <em>Screening for Depression in Clinical Practice: An Evidence-Based Guide<\/em>. Oxford University Press, 12 2009. URL: <a class=\"reference external\" href=\"https:\/\/doi.org\/10.1093\/oso\/9780195380194.003.0006\">https:\/\/doi.org\/10.1093\/oso\/9780195380194.003.0006<\/a>, <a class=\"reference external\" href=\"https:\/\/arxiv.org\/abs\/https:\/\/academic.oup.com\/book\/0\/chapter\/354484514\/chapter-pdf\/43715552\/isbn-9780195380194-book-part-6.pdf\">arXiv:https:\/\/academic.oup.com\/book\/0\/chapter\/354484514\/chapter-pdf\/43715552\/isbn-9780195380194-book-part-6.pdf<\/a>, <a class=\"reference external\" href=\"https:\/\/doi.org\/10.1093\/oso\/9780195380194.003.0006\">doi:10.1093\/oso\/9780195380194.003.0006<\/a>.<\/p>\n<\/div>\n<div class=\"citation\" id=\"id110\" role=\"doc-biblioentry\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span><a role=\"doc-backlink\" href=\"#id1\">PSTF26<\/a><span class=\"fn-bracket\">]<\/span><\/span>\n<p>Nicole\u00a0Beaulieu Perez, Paula\u00a0Gordillo Sierra, Brittany Taylor, and Jason Fletcher. Selecting a depression measure for research: a critical examination of five common self-report scales. <em>Journal of Affective Disorders<\/em>, 394:120542, 2026. URL: <a class=\"reference external\" href=\"https:\/\/www.sciencedirect.com\/science\/article\/pii\/S0165032725019846\">https:\/\/www.sciencedirect.com\/science\/article\/pii\/S0165032725019846<\/a>, <a class=\"reference external\" href=\"https:\/\/doi.org\/https:\/\/doi.org\/10.1016\/j.jad.2025.120542\">doi:https:\/\/doi.org\/10.1016\/j.jad.2025.120542<\/a>.<\/p>\n<\/div>\n<div class=\"citation\" id=\"id29\" role=\"doc-biblioentry\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span><a role=\"doc-backlink\" href=\"#id13\">SSS+<\/a><span class=\"fn-bracket\">]<\/span><\/span>\n<p>The Rainy\u00a0Day System, The\u00a0Frost System, A\u00a0Shattered\u00a0Dragon's Spirit, Pluralanomaly, NeonWebb, Nicolerenee, and LogarithmicLot. Partitionary - pluralpedia. URL: <a class=\"reference external\" href=\"https:\/\/pluralpedia.org\/index.php?title=Partitionary&amp;oldid=65083\">https:\/\/pluralpedia.org\/index.php?title=Partitionary&amp;oldid=65083<\/a> (visited on 2026-02-08).<\/p>\n<\/div>\n<div class=\"citation\" id=\"id32\" role=\"doc-biblioentry\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span><a role=\"doc-backlink\" href=\"#id12\">Sys<\/a><span class=\"fn-bracket\">]<\/span><\/span>\n<p>The\u00a0Woodland System. Headcount - pluralpedia. URL: <a class=\"reference external\" href=\"https:\/\/pluralpedia.org\/index.php?title=Headcount&amp;oldid=58294\">https:\/\/pluralpedia.org\/index.php?title=Headcount&amp;oldid=58294<\/a> (visited on 2026-02-08).<\/p>\n<\/div>\n<\/div>\n<\/div>\n<p class=\"rubric\">Cite As<\/p>\n<details><summary>Click here to expand the bibtex entry.<\/summary><div class=\"docutils container\">\n<blockquote>\n<div><div class=\"highlight-bibtex notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"nc\">@online<\/span><span class=\"p\">{<\/span><span class=\"nl\">appleton_blog_0009<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"2\"><span class=\"w\">  <\/span><span class=\"na\">title<\/span><span class=\"w\">    <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ Lessons in Grafana - Part Three: Mental Metadata }<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"3\"><span class=\"w\">  <\/span><span class=\"na\">author<\/span><span class=\"w\">   <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ Olivia Appleton-Crocker }<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"4\"><span class=\"w\">  <\/span><span class=\"na\">editor<\/span><span class=\"w\">   <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ Ultralee0 }<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"5\"><span class=\"w\">  <\/span><span class=\"na\">language<\/span><span class=\"w\"> <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ English }<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"6\"><span class=\"w\">  <\/span><span class=\"na\">version<\/span><span class=\"w\">  <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ 1 }<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"7\"><span class=\"w\">  <\/span><span class=\"na\">date<\/span><span class=\"w\">     <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ 2026-03-09 }<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"8\"><span class=\"w\">  <\/span><span class=\"na\">url<\/span><span class=\"w\">      <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ https:\/\/blog.oliviaappleton.com\/posts\/0009-lessons-in-grafana-03 }<\/span>\n<\/span><span data-line=\"9\"><span class=\"p\">}<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<\/div><\/blockquote>\n<\/div>\n<\/details><p class=\"tags\"><span>Tags: <\/span><a class=\"reference external\" href=\"..\/_tags\/grafana.html\">grafana<\/a><span>, <\/span><a class=\"reference external\" href=\"..\/_tags\/data-collection.html\">data-collection<\/a><span>, <\/span><a class=\"reference external\" href=\"..\/_tags\/data-visualization.html\">data-visualization<\/a><span>, <\/span><a class=\"reference external\" href=\"..\/_tags\/postgres.html\">postgres<\/a><span>, <\/span><a class=\"reference external\" href=\"..\/_tags\/postgresql.html\">postgresQL<\/a><span>, <\/span><a class=\"reference external\" href=\"..\/_tags\/python.html\">python<\/a><span>, <\/span><a class=\"reference external\" href=\"..\/_tags\/surveys.html\">surveys<\/a><span>, <\/span><a class=\"reference external\" href=\"..\/_tags\/mental-health.html\">mental-health<\/a><span>, <\/span><a class=\"reference external\" href=\"..\/_tags\/databases.html\">databases<\/a><\/p>\n\n            <h2>\n                Comments\n                <span class=\"comments-info\">\n                    <img class=\"fedi-icon\" src=\"https:\/\/blog.oliviaappleton.com\/_static\/boost.svg\" alt=\"Boosts\">\n                    <span id=\"global-reblogs\"><\/span>,\n                    <img class=\"fedi-icon\" src=\"https:\/\/blog.oliviaappleton.com\/_static\/like.svg\" alt=\"Likes\">\n                    <span id=\"global-likes\"><\/span>\n                <\/span>\n            <\/h2>\n            <div id=\"comments-section\"><\/div>\n            <script>\n                document.addEventListener(\"DOMContentLoaded\", function () {\n                    setConfig({\n                        boostLink: \"https:\/\/blog.oliviaappleton.com\/_static\/boost.svg\",\n                        allowSensitiveEmoji: false,\n                        allowCustomEmoji: true,\n                        allowMediaAttachments: true,\n                        allowAvatars: true,\n                        delayCommentLoad: true,\n                        defaultReactionEmoji: '\u2764',\n                        retryDelay: 100,\n                    });\n                    fetchComments('misskey', 'transfem.social', 'ajn60lxnpnh003di', 5);\n                });\n            <\/script>\n        <\/section>\n<\/section>\n","guid":"https:\/\/blog.oliviaappleton.com\/posts\/0009-lessons-in-grafana-03.html","category":["grafana","data-collection","data-visualization","postgres","postgresQL","python","surveys","mental-health","databases"],"pubDate":"Mon, 09 Mar 2026 00:00:00 +0000"},{"title":"Lessons in Grafana - Bonus Entry: Achieving Acceptable Averages","link":"https:\/\/blog.oliviaappleton.com\/posts\/0008-lessons-in-grafana-02-5.html","description":"<a rel='me' href='https:\/\/transfem.social\/@LivInTheLookingGlass' role='none' class='invisible'><\/a>\n<script>\n    window.goatcounter = {\n        path: function() {\n            let p = location.pathname;\n            if (p == '\/') {\n                p = '\/index.html';\n            }\n            return '\/blog' + p;\n        }\n    }\n<\/script>\n<script data-goatcounter='https:\/\/livinthelookingglass.goatcounter.com\/count'\n async src='\/\/gc.zgo.at\/count.js'><\/script><section id=\"lessons-in-grafana-bonus-entry-achieving-acceptable-averages\">\n<h1>Lessons in Grafana - Bonus Entry: Achieving Acceptable Averages<a class=\"headerlink\" href=\"#lessons-in-grafana-bonus-entry-achieving-acceptable-averages\" title=\"Link to this heading\">\u00b6<\/a><\/h1>\n<p><span class=\"date\">2026-03-02<\/span><\/p>\n<p>Welcome! This is a bonus entry. If you skip it, the rest of the series will make sense, but I thought that these details were worth sharing. This is in part because I found it difficult to find a well-published way to solve this problem, and in part because I enjoyed the process of learning it. I\u2019m hopeful that you will as well.<\/p>\n<ul class=\"series-nav-max-3 series-nav simple\">\n<li><p><strong class=\"series-nav-label\">1)<\/strong><span> <\/span><a class=\"reference external\" href=\"\/posts\/0006-lessons-in-grafana-01\">A Vision<\/a><\/p><\/li>\n<li><p><strong class=\"series-nav-label\">2)<\/strong><span> <\/span><a class=\"reference external\" href=\"\/posts\/0007-lessons-in-grafana-02\">Litter Logs<\/a><\/p><\/li>\n<li><p><strong class=\"series-nav-label\">2.5)<\/strong><span> <\/span>Achieving Acceptable Averages (you are here)<\/p><\/li>\n<li><p><strong class=\"series-nav-label\">3)<\/strong><span> <\/span><a class=\"reference external\" href=\"\/posts\/0009-lessons-in-grafana-03\">Mental Metadata<\/a><\/p><\/li>\n<li><p><strong class=\"series-nav-label\">4)<\/strong><span> <\/span>Noticing Notes<\/p><\/li>\n<li><p><strong class=\"series-nav-label\">5)<\/strong><span> <\/span>Graphing Goatcounter<\/p><\/li>\n<li><p><strong class=\"series-nav-label\">6)<\/strong><span> <\/span>Bridging Gadgets? Biometrics Galore!<\/p><\/li>\n<li><p><strong class=\"series-nav-label\">7)<\/strong><span> <\/span>Plotting Prospective Plans<\/p><\/li>\n<\/ul>\n<p>As part of the previous and next post, I needed to have a way to implement a rolling average. When dealing with a time series that has inherent error in measurements, this is incredibly useful. It can help ensure that you\u2019re getting closer to the true datapoints you are trying to represent, since values near each other in time should be highly correlated in the cases relevant here. Unfortunately, doing this well was tricky.<\/p>\n<section id=\"why-not-just-use-mean\">\n<h2>Why Not Just Use Mean?<a class=\"headerlink\" href=\"#why-not-just-use-mean\" title=\"Link to this heading\">\u00b6<\/a><\/h2>\n<p>The easy solution here would be to use a rolling mean. You might say something like \u201ctake the average of the last 7 entries\u201d. If you were doing this by hand, that is almost certainly the solution you would use. This, however, runs into several problems.<\/p>\n<section id=\"the-problem-of-window-size\">\n<h3>The Problem of Window Size<a class=\"headerlink\" href=\"#the-problem-of-window-size\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>If you set a fixed number of entries as your window for averaging, you risk being over or under sensitive to recent changes. Having an inflexible window will also give you problems whenever there is an extreme value in your data. As it disappears, it will have a disproportionate influence on the current average.<\/p>\n<\/section>\n<section id=\"the-problem-of-weighting\">\n<h3>The Problem of Weighting<a class=\"headerlink\" href=\"#the-problem-of-weighting\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>It would be a mistake to not apply some weighting to your average. A simple mean would have you believe that a point 1 week ago has the same correlation to now as a point 1 hour ago. This is nonsense.<\/p>\n<p>Applying a weight based on the index can somewhat work around that. The naive, efficient approach would be something like:<\/p>\n<div class=\"highlight-python notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"k\">def<\/span><span class=\"w\"> <\/span><span class=\"nf\">our_mean<\/span><span class=\"p\">(<\/span><span class=\"n\">iterable<\/span><span class=\"p\">):<\/span>\n<\/span><span data-line=\"2\">    <span class=\"n\">total<\/span> <span class=\"o\">=<\/span> <span class=\"mi\">0<\/span>\n<\/span><span data-line=\"3\">    <span class=\"n\">weight<\/span> <span class=\"o\">=<\/span> <span class=\"mi\">0<\/span>\n<\/span><span data-line=\"4\">    <span class=\"k\">for<\/span> <span class=\"n\">idx<\/span><span class=\"p\">,<\/span> <span class=\"n\">val<\/span> <span class=\"ow\">in<\/span> <span class=\"nb\">enumerate<\/span><span class=\"p\">(<\/span><span class=\"n\">iterable<\/span><span class=\"p\">):<\/span>\n<\/span><span data-line=\"5\">        <span class=\"n\">delta<\/span> <span class=\"o\">=<\/span> <span class=\"mi\">1<\/span> <span class=\"o\">\/<\/span> <span class=\"p\">(<\/span><span class=\"n\">idx<\/span> <span class=\"o\">+<\/span> <span class=\"mi\">1<\/span><span class=\"p\">)<\/span>\n<\/span><span data-line=\"6\">        <span class=\"n\">weight<\/span> <span class=\"o\">+=<\/span> <span class=\"n\">delta<\/span>\n<\/span><span data-line=\"7\">        <span class=\"n\">total<\/span> <span class=\"o\">+=<\/span> <span class=\"n\">val<\/span> <span class=\"o\">*<\/span> <span class=\"n\">delta<\/span>\n<\/span><span data-line=\"8\">    <span class=\"k\">return<\/span> <span class=\"n\">total<\/span> <span class=\"o\">\/<\/span> <span class=\"n\">weight<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<p>This is wonderful. It solves the problem at hand, and even has a nice diminishing profile. It\u2019s nearly what we want, but it introduces another problem.<\/p>\n<\/section>\n<section id=\"the-problem-of-inconsistent-frequency\">\n<h3>The Problem of Inconsistent Frequency<a class=\"headerlink\" href=\"#the-problem-of-inconsistent-frequency\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>Who ever said you get data at a constant rate? The use case we describe in the previous entry, pet weights over time, certainly doesn\u2019t. If you want old entries to have proportionally less influence, using the index alone isn\u2019t enough to do that.<\/p>\n<p>Unfortunately, adapting a linearly-weighted mean isn\u2019t going to solve this problem. Anything you do will end up being fairly brittle. The only solution that solves all of these problems is an Exponential Moving Average.<\/p>\n<\/section>\n<\/section>\n<section id=\"the-simple-version\">\n<h2>The Simple Version<a class=\"headerlink\" href=\"#the-simple-version\" title=\"Link to this heading\">\u00b6<\/a><\/h2>\n<p>An exponential average captures our intuition for what a running average should be <em>very<\/em> well. It has two key properties:<\/p>\n<ol class=\"arabic simple\">\n<li><p>All past influences affect the current average<\/p><\/li>\n<li><p>Past measurements affect the current average much less when they are older<\/p><\/li>\n<\/ol>\n<p>In math parlance, it is implemented like so:<\/p>\n<div class=\"math-wrapper docutils container\">\n<div class=\"math notranslate nohighlight\">\n\\[\\begin{split}\\begin{aligned}\n   Y_0 &amp;= X_0 \\\\\n   Y_n &amp;= Y_{n-1} + \u03b1 \\cdot (X_n - Y_{n-1})\n\\end{aligned}\\end{split}\\]<\/div>\n<\/div>\n<p>The <span class=\"math notranslate nohighlight\">\\(\u03b1\\)<\/span> here is an abstract weighting parameter. Higher values of <span class=\"math notranslate nohighlight\">\\(\u03b1\\)<\/span> will give old values a higher influence on the current average; it will reduce the rate of decay.<\/p>\n<\/section>\n<section id=\"adding-complexity\">\n<h2>Adding Complexity<a class=\"headerlink\" href=\"#adding-complexity\" title=\"Link to this heading\">\u00b6<\/a><\/h2>\n<p>Thing is, the implementation above only works for <em>fixed data rates<\/em>. This is pretty easily implemented, and lets you precompute <span class=\"math notranslate nohighlight\">\\(\u03b1\\)<\/span> in a way that runs much more efficiently. But that doesn\u2019t work for any case we are using, as <em>all<\/em> of them have variable sample rates. We have to compute <span class=\"math notranslate nohighlight\">\\(\u03b1\\)<\/span> at each step using the formula it\u2019s derived from <span id=\"id1\">[<a class=\"reference internal\" href=\"#id85\" title=\"Jason Sachs. Answer: exponential moving average sampled at varying times. Author link: https:\/\/stackoverflow.com\/users\/44330\/jason-s and editor link: https:\/\/stackoverflow.com\/users\/107294\/cjs. URL: https:\/\/stackoverflow.com\/a\/1027808 (visited on 2026-02-11).\">Sac<\/a>]<\/span>:<\/p>\n<div class=\"math-wrapper docutils container\">\n<div class=\"math notranslate nohighlight\">\n\\[\\begin{split}\\begin{aligned}\n   Y_0 &amp;= X_0 \\\\\n   Y_n &amp;= Y_{n-1} + \u03b1 \\cdot (X_n - Y_{n-1}) \\\\\n     \u03b1 &amp;= 1 - e^{-\u0394t\/\u03c4}\n\\end{aligned}\\end{split}\\]<\/div>\n<\/div>\n<p>Now it\u2019s revealed that our weighting factor is actually <span class=\"math notranslate nohighlight\">\\(\u03c4\\)<\/span>, which was previously hidden in a precomputed <span class=\"math notranslate nohighlight\">\\(\u03b1\\)<\/span>. In data with a fixed sample rate, you can think of it as analagous to a half-life. Values that are <span class=\"math notranslate nohighlight\">\\(\u03c4\\)<\/span> apart in time will influence the running average a constant factor more than values that are <span class=\"math notranslate nohighlight\">\\(2\u03c4\\)<\/span> away, giving us a lovely exponential curve.<\/p>\n<p>Figuring out what <span class=\"math notranslate nohighlight\">\\(\u03c4\\)<\/span> represented under the hood was <em>not<\/em> obvious to me. The cited StackOverflow post leaves a lot of that to be inferred by the reader, and I wasn\u2019t able to find other resources on this. I am making this blog post primarily to explain this specific variable for those who implement this in the future.<\/p>\n<p>First let\u2019s give it a unit. We are dividing a duration, an <code class=\"code highlight sql PSQL docutils literal highlight-PSQL\"><span class=\"nb\">interval<\/span><\/code>, and getting a real number output. This means that <span class=\"math notranslate nohighlight\">\\(\u03c4\\)<\/span> <em>has<\/em> to be an <code class=\"code highlight sql PSQL docutils literal highlight-PSQL\"><span class=\"nb\">interval<\/span><\/code> as well. The next question is what the exact relationship it has on the function.<\/p>\n<p>Through some playing around, you can find that setting <span class=\"math notranslate nohighlight\">\\(\u03c4' = \\tfrac{\u03c4}{\\ln(2)}\\)<\/span> gives you an exact half-life. This is pretty wonderful to know, and gives you a lot more flexibility in how to define units. No more guess and check, you can directly reason about it.<\/p>\n<\/section>\n<section id=\"final-implementation\">\n<h2>Final Implementation<a class=\"headerlink\" href=\"#final-implementation\" title=\"Link to this heading\">\u00b6<\/a><\/h2>\n<p>Implementing this in PostgresQL requires a good deal of boilerplate, because of how aggregate functions work <span id=\"id2\">[<a class=\"reference internal\" href=\"#id17\" title=\"Redrock Data Services. Postgresql tutorial: custom aggregate functions. URL: https:\/\/www.rockdata.net\/tutorial\/ddl-create-aggregate\/ (visited on 2026-02-11).\">Serb<\/a>]<\/span>. We\u2019ll end up doing it in five steps:<\/p>\n<ol class=\"arabic simple\">\n<li><p>Implement the main function<\/p><\/li>\n<li><p>Add a data type to fit the data format they like<\/p><\/li>\n<li><p>Add a step function for that data type<\/p><\/li>\n<li><p>Add a final function to extract the value we want<\/p><\/li>\n<li><p>Add an aggregate function to work on a series<\/p><\/li>\n<\/ol>\n<p>This will create a control flow that looks as follows:<\/p>\n<section id=\"the-main-function\">\n<h3>The Main Function<a class=\"headerlink\" href=\"#the-main-function\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>This is the main function that implements our average. It takes in two time-value pairs and computes their weighted average. If the earlier pair is <code class=\"code highlight sql PSQL docutils literal highlight-PSQL\"><span class=\"k\">NULL<\/span><\/code>, it returns the newer value. I\u2019ve added as many qualifiers as possible to improve performance with the planner <span id=\"id3\">[<a class=\"reference internal\" href=\"#id115\" title=\"Redrock Data Services. Create function - sql commands. URL: https:\/\/www.rockdata.net\/docs\/15\/sql-createfunction.html (visited on 2026-02-17).\">Sera<\/a>]<\/span>. <code class=\"code highlight sql PSQL docutils literal highlight-PSQL\"><span class=\"k\">IMMUTABLE<\/span><\/code> tells the planner that nothing gets modified as a result of this function. <code class=\"code highlight sql PSQL docutils literal highlight-PSQL\"><span class=\"k\">PARALLEL<\/span><span class=\"w\"> <\/span><span class=\"n\">SAFE<\/span><\/code> tells it that this function can be run in parallel with no restrictions. <code class=\"code highlight sql PSQL docutils literal highlight-PSQL\"><span class=\"k\">LEAKPROOF<\/span><\/code> tells the planner that information only leaves by the return value, not from side effects like logging.<\/p>\n<div class=\"highlight-PSQL notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"k\">CREATE<\/span><span class=\"w\"> <\/span><span class=\"k\">FUNCTION<\/span><span class=\"w\"> <\/span><span class=\"n\">lgbot<\/span><span class=\"mf\">.<\/span><span class=\"n\">_ema_func<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span>\n<\/span><span data-line=\"2\"><span class=\"w\">   <\/span><span class=\"n\">val<\/span><span class=\"w\"> <\/span><span class=\"nb\">NUMERIC<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"3\"><span class=\"w\">   <\/span><span class=\"n\">ts<\/span><span class=\"w\"> <\/span><span class=\"nb\">TIMESTAMP<\/span><span class=\"w\"> <\/span><span class=\"nb\">WITH TIME ZONE<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"4\"><span class=\"w\">   <\/span><span class=\"n\">prev<\/span><span class=\"w\"> <\/span><span class=\"nb\">NUMERIC<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"5\"><span class=\"w\">   <\/span><span class=\"n\">prev_ts<\/span><span class=\"w\"> <\/span><span class=\"nb\">TIMESTAMP<\/span><span class=\"w\"> <\/span><span class=\"nb\">WITH TIME ZONE<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"6\"><span class=\"w\">   <\/span><span class=\"n\">tau<\/span><span class=\"w\"> <\/span><span class=\"nb\">INTERVAL<\/span>\n<\/span><span data-line=\"7\"><span class=\"p\">)<\/span>\n<\/span><span data-line=\"8\"><span class=\"k\">RETURNS<\/span><span class=\"w\"> <\/span><span class=\"nb\">NUMERIC<\/span>\n<\/span><span data-line=\"9\"><span class=\"k\">LANGUAGE<\/span><span class=\"w\"> <\/span><span class=\"k\">SQL<\/span>\n<\/span><span data-line=\"10\"><span class=\"k\">IMMUTABLE<\/span>\n<\/span><span data-line=\"11\"><span class=\"k\">LEAKPROOF<\/span>\n<\/span><span data-line=\"12\"><span class=\"k\">PARALLEL<\/span><span class=\"w\"> <\/span><span class=\"n\">SAFE<\/span>\n<\/span><span data-line=\"13\"><span class=\"k\">AS<\/span><span class=\"w\"> <\/span><span class=\"err\">$$<\/span>\n<\/span><span data-line=\"14\"><span class=\"w\">   <\/span><span class=\"k\">SELECT<\/span><span class=\"w\"> <\/span><span class=\"k\">CASE<\/span>\n<\/span><span data-line=\"15\"><span class=\"w\">      <\/span><span class=\"k\">WHEN<\/span><span class=\"w\"> <\/span><span class=\"n\">prev<\/span><span class=\"w\"> <\/span><span class=\"k\">IS<\/span><span class=\"w\"> <\/span><span class=\"k\">NULL<\/span><span class=\"w\"> <\/span><span class=\"k\">THEN<\/span><span class=\"w\"> <\/span><span class=\"n\">val<\/span>\n<\/span><span data-line=\"16\"><span class=\"w\">      <\/span><span class=\"k\">ELSE<\/span><span class=\"w\"> <\/span><span class=\"n\">prev<\/span><span class=\"w\"> <\/span><span class=\"o\">+<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span><span class=\"mf\">1<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"w\"> <\/span><span class=\"n\">exp<\/span><span class=\"p\">(<\/span>\n<\/span><span data-line=\"17\"><span class=\"w\">               <\/span><span class=\"o\">-<\/span><span class=\"k\">EXTRACT<\/span><span class=\"p\">(<\/span><span class=\"n\">EPOCH<\/span><span class=\"w\"> <\/span><span class=\"k\">FROM<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span><span class=\"n\">ts<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"w\"> <\/span><span class=\"n\">prev_ts<\/span><span class=\"p\">))<\/span><span class=\"w\"> <\/span><span class=\"o\">\/<\/span><span class=\"w\"> <\/span><span class=\"k\">EXTRACT<\/span><span class=\"p\">(<\/span><span class=\"n\">EPOCH<\/span><span class=\"w\"> <\/span><span class=\"k\">FROM<\/span><span class=\"w\"> <\/span><span class=\"n\">tau<\/span><span class=\"p\">)<\/span>\n<\/span><span data-line=\"18\"><span class=\"w\">            <\/span><span class=\"p\">))<\/span><span class=\"w\"> <\/span><span class=\"o\">*<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span><span class=\"n\">val<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"w\"> <\/span><span class=\"n\">prev<\/span><span class=\"p\">)<\/span>\n<\/span><span data-line=\"19\"><span class=\"w\">   <\/span><span class=\"k\">END<\/span><span class=\"p\">;<\/span>\n<\/span><span data-line=\"20\"><span class=\"go\">$$;<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<p>If you aren\u2019t familiar with SQL development, it might throw you to see the <code class=\"code highlight sql PSQL docutils literal highlight-PSQL\"><span class=\"k\">EXTRACT<\/span><span class=\"p\">()<\/span><\/code> function being called. This is happening because, unlike in some other languages, you can\u2019t directly divide intervals. If you try, you will get:<\/p>\n<div class=\"highlight-psql notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"gp\">postgres=#<\/span><span class=\"w\"> <\/span><span class=\"k\">SELECT<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span><span class=\"nb\">interval<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;3 days&#39;<\/span><span class=\"p\">)<\/span><span class=\"w\"> <\/span><span class=\"o\">\/<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span><span class=\"nb\">interval<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;1 day&#39;<\/span><span class=\"p\">);<\/span>\n<\/span><span data-line=\"2\"><span class=\"gs\">ERROR:<\/span><span class=\"gr\">  operator does not exist: interval \/ interval<\/span>\n<\/span><span data-line=\"3\"><span class=\"gs\">LINE 1:<\/span><span class=\"gr\"> SELECT (interval &#39;3 days&#39;) \/ (interval &#39;1 day&#39;);<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<p>The way we get around this is by extracting the seconds-component. This lets us do that division cleanly, with the units canceling out as expected.<\/p>\n<\/section>\n<section id=\"the-data-type\">\n<h3>The Data Type<a class=\"headerlink\" href=\"#the-data-type\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>Because aggregate functions frustratingly require the state to be a single argument, we need to wrap it in a custom type. This is annoying, but tolerable.<\/p>\n<div class=\"highlight-PSQL notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"k\">CREATE<\/span><span class=\"w\"> <\/span><span class=\"k\">TYPE<\/span><span class=\"w\"> <\/span><span class=\"n\">lgbot<\/span><span class=\"mf\">.<\/span><span class=\"n\">_ema_state<\/span><span class=\"w\"> <\/span><span class=\"k\">AS<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span>\n<\/span><span data-line=\"2\"><span class=\"w\">   <\/span><span class=\"n\">val<\/span><span class=\"w\"> <\/span><span class=\"nb\">NUMERIC<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"3\"><span class=\"w\">   <\/span><span class=\"n\">ts<\/span><span class=\"w\"> <\/span><span class=\"nb\">TIMESTAMP<\/span><span class=\"w\"> <\/span><span class=\"nb\">WITH TIME ZONE<\/span>\n<\/span><span data-line=\"4\"><span class=\"p\">);<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<\/section>\n<section id=\"the-step-function\">\n<h3>The Step Function<a class=\"headerlink\" href=\"#the-step-function\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>The purpose of a step function is to take us from the previous state to the next one. Strictly speaking, this could have the main function folded in. It would likely perform faster if it did, even. But when I was writing this, I appreciated having that be cleanly separated. It made it easier to reason about what was going on.<\/p>\n<p>In this case, the step function has two jobs. It needs to guard against a <code class=\"code highlight sql PSQL docutils literal highlight-PSQL\"><span class=\"k\">NULL<\/span><\/code> state, and it needs to translate inputs\/outputs so that the main function and aggregate function understand each other.<\/p>\n<div class=\"highlight-PSQL notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"k\">CREATE<\/span><span class=\"w\"> <\/span><span class=\"k\">FUNCTION<\/span><span class=\"w\"> <\/span><span class=\"n\">lgbot<\/span><span class=\"mf\">.<\/span><span class=\"n\">_ema_step_func<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span>\n<\/span><span data-line=\"2\"><span class=\"w\">   <\/span><span class=\"n\">state<\/span><span class=\"w\"> <\/span><span class=\"n\">lgbot<\/span><span class=\"mf\">.<\/span><span class=\"n\">_ema_state<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"3\"><span class=\"w\">   <\/span><span class=\"n\">val<\/span><span class=\"w\"> <\/span><span class=\"nb\">NUMERIC<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"4\"><span class=\"w\">   <\/span><span class=\"n\">ts<\/span><span class=\"w\"> <\/span><span class=\"nb\">TIMESTAMP<\/span><span class=\"w\"> <\/span><span class=\"nb\">WITH TIME ZONE<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"5\"><span class=\"w\">   <\/span><span class=\"n\">tau<\/span><span class=\"w\"> <\/span><span class=\"nb\">INTERVAL<\/span>\n<\/span><span data-line=\"6\"><span class=\"p\">)<\/span>\n<\/span><span data-line=\"7\"><span class=\"k\">RETURNS<\/span><span class=\"w\"> <\/span><span class=\"n\">lgbot<\/span><span class=\"mf\">.<\/span><span class=\"n\">_ema_state<\/span>\n<\/span><span data-line=\"8\"><span class=\"k\">LANGUAGE<\/span><span class=\"w\"> <\/span><span class=\"k\">SQL<\/span>\n<\/span><span data-line=\"9\"><span class=\"k\">IMMUTABLE<\/span>\n<\/span><span data-line=\"10\"><span class=\"k\">LEAKPROOF<\/span>\n<\/span><span data-line=\"11\"><span class=\"k\">PARALLEL<\/span><span class=\"w\"> <\/span><span class=\"n\">SAFE<\/span>\n<\/span><span data-line=\"12\"><span class=\"k\">AS<\/span><span class=\"w\"> <\/span><span class=\"err\">$$<\/span>\n<\/span><span data-line=\"13\"><span class=\"w\">   <\/span><span class=\"k\">SELECT<\/span><span class=\"w\"> <\/span><span class=\"k\">CASE<\/span>\n<\/span><span data-line=\"14\"><span class=\"w\">      <\/span><span class=\"k\">WHEN<\/span><span class=\"w\"> <\/span><span class=\"n\">state<\/span><span class=\"w\"> <\/span><span class=\"k\">IS<\/span><span class=\"w\"> <\/span><span class=\"k\">NULL<\/span><span class=\"w\"> <\/span><span class=\"k\">THEN<\/span><span class=\"w\"> <\/span><span class=\"k\">ROW<\/span><span class=\"p\">(<\/span><span class=\"n\">val<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"n\">ts<\/span><span class=\"p\">)<\/span><span class=\"o\">::<\/span><span class=\"n\">lgbot<\/span><span class=\"mf\">.<\/span><span class=\"n\">_ema_state<\/span>\n<\/span><span data-line=\"15\"><span class=\"w\">      <\/span><span class=\"k\">ELSE<\/span><span class=\"w\"> <\/span><span class=\"k\">ROW<\/span><span class=\"p\">(<\/span>\n<\/span><span data-line=\"16\"><span class=\"w\">            <\/span><span class=\"n\">lgbot<\/span><span class=\"mf\">.<\/span><span class=\"n\">_ema_func<\/span><span class=\"p\">(<\/span><span class=\"n\">val<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"n\">ts<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"n\">state<\/span><span class=\"mf\">.<\/span><span class=\"n\">val<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"n\">state<\/span><span class=\"mf\">.<\/span><span class=\"n\">ts<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"n\">tau<\/span><span class=\"p\">),<\/span>\n<\/span><span data-line=\"17\"><span class=\"w\">            <\/span><span class=\"n\">ts<\/span>\n<\/span><span data-line=\"18\"><span class=\"w\">      <\/span><span class=\"p\">)<\/span><span class=\"o\">::<\/span><span class=\"n\">lgbot<\/span><span class=\"mf\">.<\/span><span class=\"n\">_ema_state<\/span>\n<\/span><span data-line=\"19\"><span class=\"w\">   <\/span><span class=\"k\">END<\/span><span class=\"p\">;<\/span>\n<\/span><span data-line=\"20\"><span class=\"go\">$$;<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<\/section>\n<section id=\"the-final-function\">\n<h3>The Final Function<a class=\"headerlink\" href=\"#the-final-function\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>This is the last step before we can define the aggregate. It has a very, very simple job: take in a state, and extract the value. This gives us the final output in a way that the end-user doesn\u2019t need to think about.<\/p>\n<div class=\"highlight-PSQL notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"k\">CREATE<\/span><span class=\"w\"> <\/span><span class=\"k\">FUNCTION<\/span><span class=\"w\"> <\/span><span class=\"n\">lgbot<\/span><span class=\"mf\">.<\/span><span class=\"n\">_ema_final<\/span><span class=\"p\">(<\/span><span class=\"n\">state<\/span><span class=\"w\"> <\/span><span class=\"n\">lgbot<\/span><span class=\"mf\">.<\/span><span class=\"n\">_ema_state<\/span><span class=\"p\">)<\/span>\n<\/span><span data-line=\"2\"><span class=\"k\">RETURNS<\/span><span class=\"w\"> <\/span><span class=\"nb\">NUMERIC<\/span>\n<\/span><span data-line=\"3\"><span class=\"k\">LANGUAGE<\/span><span class=\"w\"> <\/span><span class=\"k\">SQL<\/span>\n<\/span><span data-line=\"4\"><span class=\"k\">IMMUTABLE<\/span>\n<\/span><span data-line=\"5\"><span class=\"k\">LEAKPROOF<\/span>\n<\/span><span data-line=\"6\"><span class=\"k\">PARALLEL<\/span><span class=\"w\"> <\/span><span class=\"n\">SAFE<\/span>\n<\/span><span data-line=\"7\"><span class=\"k\">AS<\/span><span class=\"w\"> <\/span><span class=\"err\">$$<\/span>\n<\/span><span data-line=\"8\"><span class=\"w\">   <\/span><span class=\"k\">SELECT<\/span><span class=\"w\"> <\/span><span class=\"n\">state<\/span><span class=\"mf\">.<\/span><span class=\"n\">val<\/span><span class=\"p\">;<\/span>\n<\/span><span data-line=\"9\"><span class=\"go\">$$;<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<\/section>\n<section id=\"the-aggregate-function\">\n<h3>The Aggregate Function<a class=\"headerlink\" href=\"#the-aggregate-function\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>With all of these pieces in place, we can define the aggregate function. It\u2019s actually very clean to read. It needs to know the arguments we\u2019re feeding initially, and a reference to the step function, state type, and final function (optional). Lastly, we indicate that the function can safely run in parallel.<\/p>\n<div class=\"highlight-PSQL notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"k\">CREATE<\/span><span class=\"w\"> <\/span><span class=\"k\">AGGREGATE<\/span><span class=\"w\"> <\/span><span class=\"n\">lgbot<\/span><span class=\"mf\">.<\/span><span class=\"n\">ema<\/span><span class=\"p\">(<\/span>\n<\/span><span data-line=\"2\"><span class=\"w\">   <\/span><span class=\"n\">val<\/span><span class=\"w\"> <\/span><span class=\"nb\">NUMERIC<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"3\"><span class=\"w\">   <\/span><span class=\"n\">ts<\/span><span class=\"w\">  <\/span><span class=\"nb\">TIMESTAMP<\/span><span class=\"w\"> <\/span><span class=\"nb\">WITH TIME ZONE<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"4\"><span class=\"w\">   <\/span><span class=\"n\">tau<\/span><span class=\"w\"> <\/span><span class=\"nb\">INTERVAL<\/span>\n<\/span><span data-line=\"5\"><span class=\"p\">)<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span>\n<\/span><span data-line=\"6\"><span class=\"w\">   <\/span><span class=\"n\">SFUNC<\/span><span class=\"w\">     <\/span><span class=\"o\">=<\/span><span class=\"w\"> <\/span><span class=\"n\">lgbot<\/span><span class=\"mf\">.<\/span><span class=\"n\">_ema_step_func<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"7\"><span class=\"w\">   <\/span><span class=\"n\">STYPE<\/span><span class=\"w\">     <\/span><span class=\"o\">=<\/span><span class=\"w\"> <\/span><span class=\"n\">lgbot<\/span><span class=\"mf\">.<\/span><span class=\"n\">_ema_state<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"8\"><span class=\"w\">   <\/span><span class=\"n\">FINALFUNC<\/span><span class=\"w\"> <\/span><span class=\"o\">=<\/span><span class=\"w\"> <\/span><span class=\"n\">lgbot<\/span><span class=\"mf\">.<\/span><span class=\"n\">_ema_final<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"9\"><span class=\"w\">   <\/span><span class=\"k\">PARALLEL<\/span><span class=\"w\">  <\/span><span class=\"o\">=<\/span><span class=\"w\"> <\/span><span class=\"n\">SAFE<\/span>\n<\/span><span data-line=\"10\"><span class=\"p\">);<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<\/section>\n<\/section>\n<section id=\"main-takeaways\">\n<h2>Main Takeaways<a class=\"headerlink\" href=\"#main-takeaways\" title=\"Link to this heading\">\u00b6<\/a><\/h2>\n<p>This would have been a lot easier if I had settled for an easier-to-implement average function, but I am very glad that I didn\u2019t. I learned a lot as a result of this, and think that I have a better fit to the data I track because of it. I am very hopeful that the next person who runs into this problem will have an easy resource for how to work through it.<\/p>\n<p>If you enjoyed this, stick around for next week\u2019s entry discussing psychological data collection: <a class=\"reference external\" href=\"https:\/\/blog.oliviaappleton.com\/posts\/0009-lessons-in-grafana-03\"><em>Mental Metadata<\/em>.<\/a><\/p>\n<\/section>\n<section id=\"acknowledgments\">\n<h2>Acknowledgments<a class=\"headerlink\" href=\"#acknowledgments\" title=\"Link to this heading\">\u00b6<\/a><\/h2>\n<p>Thank you to Jason Sachs (<a class=\"reference external\" href=\"https:\/\/stackoverflow.com\/users\/44330\/jason-s\">&#64;jason-s&#64;stackoverflow.com<\/a>), who is one of the very few people that have addressed this question online. I have endless gratitude for my friends <a class=\"reference external\" href=\"https:\/\/linktr.ee\/Ultralee0\">&#64;Ultralee0&#64;linktr.ee<\/a> and <a class=\"reference external\" href=\"https:\/\/indiedragon.com\">Ruby<\/a>, who\u2019ve done an exhaustive test reading of this series. Lastly, I\u2019d like to shout out <a class=\"reference external\" href=\"https:\/\/mastodon.mit.edu\/&#64;xy\">&#64;xy&#64;mastodon.mit.edu<\/a> for encouraging me to make this its own post.<\/p>\n<p class=\"rubric\">Citations<\/p>\n<div class=\"docutils container\" id=\"id4\">\n<div role=\"list\" class=\"citation-list\">\n<div class=\"citation\" id=\"id85\" role=\"doc-biblioentry\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span><a role=\"doc-backlink\" href=\"#id1\">Sac<\/a><span class=\"fn-bracket\">]<\/span><\/span>\n<p>Jason Sachs. Answer: exponential moving average sampled at varying times. Author link: <a class=\"reference external\" href=\"https:\/\/stackoverflow.com\/users\/44330\/jason-s\">https:\/\/stackoverflow.com\/users\/44330\/jason-s<\/a> and editor link: <a class=\"reference external\" href=\"https:\/\/stackoverflow.com\/users\/107294\/cjs\">https:\/\/stackoverflow.com\/users\/107294\/cjs<\/a>. URL: <a class=\"reference external\" href=\"https:\/\/stackoverflow.com\/a\/1027808\">https:\/\/stackoverflow.com\/a\/1027808<\/a> (visited on 2026-02-11).<\/p>\n<\/div>\n<div class=\"citation\" id=\"id115\" role=\"doc-biblioentry\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span><a role=\"doc-backlink\" href=\"#id3\">Sera<\/a><span class=\"fn-bracket\">]<\/span><\/span>\n<p>Redrock\u00a0Data Services. Create function - sql commands. URL: <a class=\"reference external\" href=\"https:\/\/www.rockdata.net\/docs\/15\/sql-createfunction.html\">https:\/\/www.rockdata.net\/docs\/15\/sql-createfunction.html<\/a> (visited on 2026-02-17).<\/p>\n<\/div>\n<div class=\"citation\" id=\"id17\" role=\"doc-biblioentry\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span><a role=\"doc-backlink\" href=\"#id2\">Serb<\/a><span class=\"fn-bracket\">]<\/span><\/span>\n<p>Redrock\u00a0Data Services. Postgresql tutorial: custom aggregate functions. URL: <a class=\"reference external\" href=\"https:\/\/www.rockdata.net\/tutorial\/ddl-create-aggregate\/\">https:\/\/www.rockdata.net\/tutorial\/ddl-create-aggregate\/<\/a> (visited on 2026-02-11).<\/p>\n<\/div>\n<\/div>\n<\/div>\n<p class=\"rubric\">Cite As<\/p>\n<details><summary>Click here to expand the bibtex entry.<\/summary><div class=\"docutils container\">\n<blockquote>\n<div><div class=\"highlight-bibtex notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"nc\">@online<\/span><span class=\"p\">{<\/span><span class=\"nl\">appleton_blog_0008<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"2\"><span class=\"w\">  <\/span><span class=\"na\">title<\/span><span class=\"w\">    <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ Lessons in Grafana - Bonus Entry: Achieving Acceptable Averages }<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"3\"><span class=\"w\">  <\/span><span class=\"na\">author<\/span><span class=\"w\">   <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ Olivia Appleton-Crocker }<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"4\"><span class=\"w\">  <\/span><span class=\"na\">editor<\/span><span class=\"w\">   <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ Ultralee0 and Ruby }<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"5\"><span class=\"w\">  <\/span><span class=\"na\">language<\/span><span class=\"w\"> <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ English }<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"6\"><span class=\"w\">  <\/span><span class=\"na\">version<\/span><span class=\"w\">  <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ 1 }<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"7\"><span class=\"w\">  <\/span><span class=\"na\">date<\/span><span class=\"w\">     <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ 2026-03-02 }<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"8\"><span class=\"w\">  <\/span><span class=\"na\">url<\/span><span class=\"w\">      <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ https:\/\/blog.oliviaappleton.com\/posts\/0008-lessons-in-grafana-02-5 }<\/span>\n<\/span><span data-line=\"9\"><span class=\"p\">}<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<\/div><\/blockquote>\n<\/div>\n<\/details><p class=\"tags\"><span>Tags: <\/span><a class=\"reference external\" href=\"..\/_tags\/grafana.html\">grafana<\/a><span>, <\/span><a class=\"reference external\" href=\"..\/_tags\/data-collection.html\">data-collection<\/a><span>, <\/span><a class=\"reference external\" href=\"..\/_tags\/data-visualization.html\">data-visualization<\/a><span>, <\/span><a class=\"reference external\" href=\"..\/_tags\/databases.html\">databases<\/a><span>, <\/span><a class=\"reference external\" href=\"..\/_tags\/scraping.html\">scraping<\/a><span>, <\/span><a class=\"reference external\" href=\"..\/_tags\/postgres.html\">postgres<\/a><span>, <\/span><a class=\"reference external\" href=\"..\/_tags\/postgresql.html\">postgresQL<\/a><span>, <\/span><a class=\"reference external\" href=\"..\/_tags\/python.html\">python<\/a><span>, <\/span><a class=\"reference external\" href=\"..\/_tags\/exponential-moving-average.html\">exponential-moving-average<\/a><span>, <\/span><a class=\"reference external\" href=\"..\/_tags\/ema.html\">ema<\/a><\/p>\n\n            <h2>\n                Comments\n                <span class=\"comments-info\">\n                    <img class=\"fedi-icon\" src=\"https:\/\/blog.oliviaappleton.com\/_static\/boost.svg\" alt=\"Boosts\">\n                    <span id=\"global-reblogs\"><\/span>,\n                    <img class=\"fedi-icon\" src=\"https:\/\/blog.oliviaappleton.com\/_static\/like.svg\" alt=\"Likes\">\n                    <span id=\"global-likes\"><\/span>\n                <\/span>\n            <\/h2>\n            <div id=\"comments-section\"><\/div>\n            <script>\n                document.addEventListener(\"DOMContentLoaded\", function () {\n                    setConfig({\n                        boostLink: \"https:\/\/blog.oliviaappleton.com\/_static\/boost.svg\",\n                        allowSensitiveEmoji: false,\n                        allowCustomEmoji: true,\n                        allowMediaAttachments: true,\n                        allowAvatars: true,\n                        delayCommentLoad: true,\n                        defaultReactionEmoji: '\u2764',\n                        retryDelay: 100,\n                    });\n                    fetchComments('misskey', 'transfem.social', 'ajdczbz14wx00snh', 5);\n                });\n            <\/script>\n        <\/section>\n<\/section>\n","guid":"https:\/\/blog.oliviaappleton.com\/posts\/0008-lessons-in-grafana-02-5.html","category":["grafana","data-collection","data-visualization","databases","scraping","postgres","postgresQL","exponential-moving-average","ema"],"pubDate":"Mon, 02 Mar 2026 00:00:00 +0000"},{"title":"Lessons in Grafana - Part Two: Litter Logs","link":"https:\/\/blog.oliviaappleton.com\/posts\/0007-lessons-in-grafana-02.html","description":"<a rel='me' href='https:\/\/transfem.social\/@LivInTheLookingGlass' role='none' class='invisible'><\/a>\n<script>\n    window.goatcounter = {\n        path: function() {\n            let p = location.pathname;\n            if (p == '\/') {\n                p = '\/index.html';\n            }\n            return '\/blog' + p;\n        }\n    }\n<\/script>\n<script data-goatcounter='https:\/\/livinthelookingglass.goatcounter.com\/count'\n async src='\/\/gc.zgo.at\/count.js'><\/script><section id=\"lessons-in-grafana-part-two-litter-logs\">\n<h1>Lessons in Grafana - Part Two: Litter Logs<a class=\"headerlink\" href=\"#lessons-in-grafana-part-two-litter-logs\" title=\"Link to this heading\">\u00b6<\/a><\/h1>\n<p><span class=\"date\">2026-02-23<\/span><\/p>\n<p>Welcome back! In this series, I am documenting my journey through data collection and visualization. We\u2019ll be posting one entry a week from now on, until the end of the list below. Today we\u2019re going to be going over my efforts to extract data from our cats\u2019 <a class=\"reference external\" href=\"https:\/\/www.litter-robot.com\/litter-robot-4.html\">Litter Robot<\/a>.<\/p>\n<ul class=\"series-nav-max-3 series-nav simple\">\n<li><p><strong class=\"series-nav-label\">1)<\/strong><span> <\/span><a class=\"reference external\" href=\"\/posts\/0006-lessons-in-grafana-01\">A Vision<\/a><\/p><\/li>\n<li><p><strong class=\"series-nav-label\">2)<\/strong><span> <\/span>Litter Logs (you are here)<\/p><\/li>\n<li><p><strong class=\"series-nav-label\">2.5)<\/strong><span> <\/span><a class=\"reference external\" href=\"\/posts\/0008-lessons-in-grafana-02-5\">Achieving Acceptable Averages<\/a><\/p><\/li>\n<li><p><strong class=\"series-nav-label\">3)<\/strong><span> <\/span><a class=\"reference external\" href=\"\/posts\/0009-lessons-in-grafana-03\">Mental Metadata<\/a><\/p><\/li>\n<li><p><strong class=\"series-nav-label\">4)<\/strong><span> <\/span>Noticing Notes<\/p><\/li>\n<li><p><strong class=\"series-nav-label\">5)<\/strong><span> <\/span>Graphing Goatcounter<\/p><\/li>\n<li><p><strong class=\"series-nav-label\">6)<\/strong><span> <\/span>Bridging Gadgets? Biometrics Galore!<\/p><\/li>\n<li><p><strong class=\"series-nav-label\">7)<\/strong><span> <\/span>Plotting Prospective Plans<\/p><\/li>\n<\/ul>\n<section id=\"wait-you-re-getting-data-about-cat-bathrooms\">\n<h2>Wait, You\u2019re Getting Data About Cat Bathrooms?<a class=\"headerlink\" href=\"#wait-you-re-getting-data-about-cat-bathrooms\" title=\"Link to this heading\">\u00b6<\/a><\/h2>\n<p>Yes! Isn\u2019t it great?<\/p>\n<p>We initially got this because it significantly reduces the burden of keeping the litterbox clean. It has a number of side benefits, though. There are many different sensors on it, and these get exposed in interesting and semi-processed ways. With the use of the <code class=\"docutils literal notranslate\"><span class=\"pre\">pylitterbot<\/span><\/code> module <span id=\"id1\">[<a class=\"reference internal\" href=\"#id18\" title=\"Nathan Spencer, Justin Bull, Ian Atha, &#64;jrhe&#64;github.com, Joost Lekkerkerker, Marc Mueller, Ville Skytt\u00e4, and Andreas Billmeier. Pylitterbot. Development ongoing. URL: https:\/\/github.com\/natekspencer\/pylitterbot (visited on 2026-02-05).\">SBA+<\/a>]<\/span>, I have been able to scrape this data.<\/p>\n<p>The biggest benefit of doing so is <em>persistence<\/em>. Their app shows all of it in a clean way, but they only preserve about 1 month of history. Collecting this data myself means that we can have it indefinitely and without fear of their services losing it. It also means that I get to play around with different things to extract from it!<\/p>\n<\/section>\n<section id=\"the-test-subjects\">\n<h2>The Test Subjects<a class=\"headerlink\" href=\"#the-test-subjects\" title=\"Link to this heading\">\u00b6<\/a><\/h2>\n<p>As established by internet law <span id=\"id2\">[<a class=\"reference internal\" href=\"#id24\" title=\"Cat Tax Bureau. What is cat tax? meaning, origin &amp; how to pay. URL: https:\/\/cat-tax.net\/what-is-cat-tax.html (visited on 2026-02-08).\">Bur<\/a>]<\/span>, I am required to pay the cat tax. Here are our test subjects:<\/p>\n<section id=\"ophelia-loaf\">\n<h3>Ophelia (\u201cLoaf\u201d)<a class=\"headerlink\" href=\"#ophelia-loaf\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>Loaf is about 7 years old, and is very people-friendly. If she sees you, she will demand pets. She is the most vocal of our cats by far, and will be very clear if she wants you to follow her. Her favorite hobbies are watching birds and lying between people\u2019s legs.<\/p>\n<figure class=\"align-center\" id=\"id127\">\n<a class=\"reference internal image-reference\" href=\"..\/_images\/loafie.jpg\"><img alt=\"A picture of Loaf, as described below\" src=\"..\/_images\/loafie.jpg\" style=\"width: 80%;\" \/>\n<\/a>\n<figcaption>\n<p><span class=\"caption-text\">Ophelia, a brown\/orange\/black cat is laying on a green duvet. Rainbow lights from a nearby window are projected onto the whole image. She is looking up at the camera in curiosity. More pictures can be found on the fedi hashtag <a class=\"reference external\" href=\"https:\/\/transfem.social\/tags\/OpheTheLoaf\">#OpheTheLoaf<\/a>.<\/span><a class=\"headerlink\" href=\"#id127\" title=\"Link to this image\">\u00b6<\/a><\/p>\n<\/figcaption>\n<\/figure>\n<\/section>\n<section id=\"mayalaran-maya\">\n<h3>Mayalaran (\u201cMaya\u201d)<a class=\"headerlink\" href=\"#mayalaran-maya\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>Maya is about 7 years old, and is quite skittish. She\u2019s gotten a lot more assertive over time, but the starting point was quite low. She is the loudest purrer, and the only one that enjoys belly rubs (in moderation). Her favorite hobbies are judging the neighbors and taking naps.<\/p>\n<figure class=\"align-center\" id=\"id128\">\n<a class=\"reference internal image-reference\" href=\"..\/_images\/maya.webp\"><img alt=\"A picture of Maya, as described below\" src=\"..\/_images\/maya.webp\" style=\"width: 80%;\" \/>\n<\/a>\n<figcaption>\n<p><span class=\"caption-text\">Maya, a gray cat, is laying in \u201cshrimp pose\u201d, where her front legs are tucked in and her back legs splayed out. Her eyes are half closed, and she is purring very loudly. She is on top of a couch. More pictures can be found on the fedi hashtag <a class=\"reference external\" href=\"https:\/\/transfem.social\/tags\/MayalaranTheCat\">#MayalaranTheCat<\/a>.<\/span><a class=\"headerlink\" href=\"#id128\" title=\"Link to this image\">\u00b6<\/a><\/p>\n<\/figcaption>\n<\/figure>\n<p>Because she is a bit of an anxious princess, Maya doesn\u2019t really use the robot. She did for a little while, but around the time we got Rye she stopped. I think that either the moving parts scare her, or she feels trapped. This means she will not be represented in this data, but I include her anyways since she\u2019s cute.<\/p>\n<\/section>\n<section id=\"pumpernickel-rye\">\n<h3>Pumpernickel (\u201cRye\u201d)<a class=\"headerlink\" href=\"#pumpernickel-rye\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>Rye is about 9 months old, and is a bundle of energy. She has <em>all<\/em> the tortitude <span id=\"id3\">[<a class=\"reference internal\" href=\"#id77\" title=\"Dr. Lizzie Youens BSc (Hons) BVSc MRCVS. Tortitude: definition, behavior &amp; how to deal with it. URL: https:\/\/cats.com\/tortitude (visited on 2026-02-15).\">MRC<\/a>]<\/span> and is as distractible as a squirrel after a triple-espresso latte. Her favorite hobbies are wrestling and cuddling.<\/p>\n<figure class=\"align-center\" id=\"id129\">\n<a class=\"reference internal image-reference\" href=\"..\/_images\/rye.jpg\"><img alt=\"A picture of Rye, as described below\" src=\"..\/_images\/rye.jpg\" style=\"width: 80%;\" \/>\n<\/a>\n<figcaption>\n<p><span class=\"caption-text\">This is Rye as a kitten (roughly 5 months ago). She is a mostly black cat, but with a splash of orange covering most of the right side of her face. She is yawning very wide. More pictures can be found on the fedi hashtag <a class=\"reference external\" href=\"https:\/\/transfem.social\/tags\/PumpernickelKitten\">#PumpernickelKitten<\/a>.<\/span><a class=\"headerlink\" href=\"#id129\" title=\"Link to this image\">\u00b6<\/a><\/p>\n<\/figcaption>\n<\/figure>\n<\/section>\n<\/section>\n<section id=\"prerequisites\">\n<h2>Prerequisites<a class=\"headerlink\" href=\"#prerequisites\" title=\"Link to this heading\">\u00b6<\/a><\/h2>\n<p>In the previous entry, I only discussed the Grafana stack itself. We never touched any data collection resources other than entering things by hand. Because of that, there are some things we need to set up. Let\u2019s look at a pared down version of my current data collection workflow, and reason through how it works.<\/p>\n<p>Our first step is to set up database tables. This happens across all of the scrapers I use at the very beginning, and it\u2019s delegated to them to figure things out. In the case of the litterbot scrapers, there are three tables we care about:<\/p>\n<section id=\"cycle-activity\">\n<h3>Cycle Activity<a class=\"headerlink\" href=\"#cycle-activity\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>The simplest table we\u2019ll be collecting is daily cycle activity. The robot has an API endpoint that tells me how many times it ran a cleaning cycle each day. This gives a good general overview of its usage. This data looks like:<\/p>\n<div class=\"highlight-PSQL notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"k\">CREATE<\/span><span class=\"w\"> <\/span><span class=\"k\">TABLE<\/span><span class=\"w\"> <\/span><span class=\"n\">litterbot<\/span><span class=\"mf\">.<\/span><span class=\"n\">cycle_activity<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span>\n<\/span><span data-line=\"2\"><span class=\"w\">   <\/span><span class=\"n\">bot_id<\/span><span class=\"w\"> <\/span><span class=\"nb\">SMALLINT<\/span><span class=\"w\"> <\/span><span class=\"k\">REFERENCES<\/span><span class=\"w\"> <\/span><span class=\"n\">litterbot<\/span><span class=\"mf\">.<\/span><span class=\"n\">bots<\/span><span class=\"p\">(<\/span><span class=\"n\">bot_id<\/span><span class=\"p\">)<\/span><span class=\"w\"> <\/span><span class=\"k\">ON<\/span><span class=\"w\"> <\/span><span class=\"k\">DELETE<\/span><span class=\"w\"> <\/span><span class=\"k\">CASCADE<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"3\"><span class=\"w\">   <\/span><span class=\"k\">day<\/span><span class=\"w\"> <\/span><span class=\"nb\">DATE<\/span><span class=\"w\"> <\/span><span class=\"k\">NOT<\/span><span class=\"w\"> <\/span><span class=\"k\">NULL<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"4\"><span class=\"w\">   <\/span><span class=\"n\">cycles<\/span><span class=\"w\"> <\/span><span class=\"nb\">SMALLINT<\/span><span class=\"w\"> <\/span><span class=\"k\">NOT<\/span><span class=\"w\"> <\/span><span class=\"k\">NULL<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"5\"><span class=\"w\">   <\/span><span class=\"k\">PRIMARY<\/span><span class=\"w\"> <\/span><span class=\"k\">KEY<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span><span class=\"n\">bot_id<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"k\">day<\/span><span class=\"p\">)<\/span>\n<\/span><span data-line=\"6\"><span class=\"p\">)<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<p>That\u2019s a lot of jargon, so let\u2019s break it down. The first part is the name of the table itself. This comes in two parts, the first being a schema (in other words, a grouping), and the second being the table name. Grouping things like this allows you to document your work as you go and prevents confusion and any potential conflicts.<\/p>\n<p>The next part tells you each of the columns in the table. The first looks intimidating, but it\u2019s quite simple. You have the ID of a litterbot, and it\u2019s represented by a small number. When that ID is removed from the database, the <code class=\"code highlight sql PSQL docutils literal highlight-PSQL\"><span class=\"k\">ON<\/span><span class=\"w\"> <\/span><span class=\"k\">DELETE<\/span><span class=\"w\"> <\/span><span class=\"k\">CASCADE<\/span><\/code> tells you that these entries can also be cleared.<\/p>\n<p>The next items are fairly self-explanatory. They give you the information we\u2019re seeking: a day (in UTC) and how many cleaning cycles were executed.<\/p>\n<p>The last piece does something interesting. It tells the database that you will <em>never<\/em> have more than one entry for a given bot on a given day. This allows the database to help us sort out any potential conflicts. An example of this might be if you\u2019re fetching the current day. When you look at 6pm, it says there have been 8 cycles. But when you check an hour later, there are 9. This lets us handle those updates automatically in a neat way, and also speeds up searches.<\/p>\n<\/section>\n<section id=\"pet-weights\">\n<h3>Pet Weights<a class=\"headerlink\" href=\"#pet-weights\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>The most useful table from my perspective is the one that shows our pets\u2019 weights over time. This is especially useful for Rye, as she is supposed to be gaining weight as she grows. It looks like so:<\/p>\n<div class=\"highlight-PSQL notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"k\">CREATE<\/span><span class=\"w\"> <\/span><span class=\"k\">TABLE<\/span><span class=\"w\"> <\/span><span class=\"n\">litterbot<\/span><span class=\"mf\">.<\/span><span class=\"n\">pet_weight<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span>\n<\/span><span data-line=\"2\"><span class=\"w\">   <\/span><span class=\"n\">pet_name<\/span><span class=\"w\"> <\/span><span class=\"nb\">TEXT<\/span><span class=\"w\"> <\/span><span class=\"k\">NOT<\/span><span class=\"w\"> <\/span><span class=\"k\">NULL<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"3\"><span class=\"w\">   <\/span><span class=\"n\">ts<\/span><span class=\"w\"> <\/span><span class=\"nb\">TIMESTAMP<\/span><span class=\"w\"> <\/span><span class=\"nb\">WITH TIME ZONE<\/span><span class=\"w\"> <\/span><span class=\"k\">NOT<\/span><span class=\"w\"> <\/span><span class=\"k\">NULL<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"4\"><span class=\"w\">   <\/span><span class=\"n\">weight<\/span><span class=\"w\"> <\/span><span class=\"nb\">REAL<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"5\"><span class=\"w\">   <\/span><span class=\"n\">estimated_weight<\/span><span class=\"w\"> <\/span><span class=\"nb\">REAL<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"6\"><span class=\"w\">   <\/span><span class=\"n\">last_weight_reading<\/span><span class=\"w\"> <\/span><span class=\"nb\">REAL<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"7\"><span class=\"w\">   <\/span><span class=\"k\">PRIMARY<\/span><span class=\"w\"> <\/span><span class=\"k\">KEY<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span><span class=\"n\">pet_name<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"n\">ts<\/span><span class=\"p\">)<\/span>\n<\/span><span data-line=\"8\"><span class=\"p\">)<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<p>Here there are some new keywords to look at. First, we have the pet\u2019s name. Next, though, is a timestamp. Above we were giving only a date, since we didn\u2019t need precision. That\u2019s not the case here. Weight measurements are associated with specific times, and the precision of it <em>matters<\/em>. If you have five entries in a given day, it\u2019s important to distinguish when they happened because you can\u2019t otherwise build a timeline. Suppose the weight suddenly went up. Well, if it\u2019s around dinner time, that\u2019s okay. If not, maybe she got into the cupboard and we should look for missing things. This precision allows for all sorts of storytelling along those lines.<\/p>\n<p>The next three are different versions of the same thing. Because the Litterbot API is odd, it returns three values for the weight. From my perspective, none of these are obvious. They are almost always identical readings, to the point where I suspect this is something that they <em>intended<\/em> to use, or maybe used on a different product. Functionally you can pretend they are the same thing, and assume we always use the estimated weight value. In this case, note that \u201creal\u201d is used in the math sense. If you think of this as a fraction, that\u2019s close enough.<\/p>\n<p>As before, we set a primary key to help with duplicates. In this case, you\u2019ll never have a pet with two entries at the same time.<\/p>\n<p>This table is <em>by far<\/em> the most useful to someone like the vet\u2019s office. It gives them weight over time and tells you how frequently they\u2019re going to the bathroom. Those are both valuable insights to have.<\/p>\n<\/section>\n<section id=\"bot-snapshots\">\n<h3>Bot Snapshots<a class=\"headerlink\" href=\"#bot-snapshots\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>The biggest table here is the bot status snapshot. Each time we fetch information, we are able to receive the bot\u2019s status. I record everything that seemed halfway useful. Much of this probably isn\u2019t, but it uses so little space that I may as well.<\/p>\n<div class=\"highlight-PSQL notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"k\">CREATE<\/span><span class=\"w\"> <\/span><span class=\"k\">TABLE<\/span><span class=\"w\"> <\/span><span class=\"n\">litterbot<\/span><span class=\"mf\">.<\/span><span class=\"n\">bot_snapshot<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span>\n<\/span><span data-line=\"2\"><span class=\"w\">   <\/span><span class=\"n\">bot_id<\/span><span class=\"w\"> <\/span><span class=\"nb\">SMALLINT<\/span><span class=\"w\"> <\/span><span class=\"k\">REFERENCES<\/span><span class=\"w\"> <\/span><span class=\"n\">litterbot<\/span><span class=\"mf\">.<\/span><span class=\"n\">bots<\/span><span class=\"p\">(<\/span><span class=\"n\">bot_id<\/span><span class=\"p\">)<\/span><span class=\"w\"> <\/span><span class=\"k\">ON<\/span><span class=\"w\"> <\/span><span class=\"k\">DELETE<\/span><span class=\"w\"> <\/span><span class=\"k\">CASCADE<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"3\"><span class=\"w\">   <\/span><span class=\"n\">ts<\/span><span class=\"w\"> <\/span><span class=\"nb\">TIMESTAMP<\/span><span class=\"w\"> <\/span><span class=\"nb\">WITH TIME ZONE<\/span><span class=\"w\"> <\/span><span class=\"k\">NOT<\/span><span class=\"w\"> <\/span><span class=\"k\">NULL<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"4\">\n<\/span><span data-line=\"5\"><span class=\"w\">   <\/span><span class=\"n\">status<\/span><span class=\"w\"> <\/span><span class=\"nb\">TEXT<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"6\"><span class=\"w\">   <\/span><span class=\"n\">power_status<\/span><span class=\"w\"> <\/span><span class=\"nb\">TEXT<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"7\"><span class=\"w\">   <\/span><span class=\"n\">litter_level_state<\/span><span class=\"w\"> <\/span><span class=\"nb\">TEXT<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"8\"><span class=\"w\">   <\/span><span class=\"n\">surface_type<\/span><span class=\"w\"> <\/span><span class=\"nb\">TEXT<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"9\">\n<\/span><span data-line=\"10\"><span class=\"w\">   <\/span><span class=\"n\">last_seen<\/span><span class=\"w\"> <\/span><span class=\"nb\">TIMESTAMP<\/span><span class=\"w\"> <\/span><span class=\"nb\">WITH TIME ZONE<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"11\"><span class=\"w\">   <\/span><span class=\"n\">is_online<\/span><span class=\"w\"> <\/span><span class=\"nb\">BOOLEAN<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"12\"><span class=\"w\">   <\/span><span class=\"n\">is_sleeping<\/span><span class=\"w\"> <\/span><span class=\"nb\">BOOLEAN<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"13\">\n<\/span><span data-line=\"14\"><span class=\"w\">   <\/span><span class=\"n\">cycle_capacity<\/span><span class=\"w\"> <\/span><span class=\"nb\">SMALLINT<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"15\"><span class=\"w\">   <\/span><span class=\"n\">cycle_count<\/span><span class=\"w\"> <\/span><span class=\"nb\">SMALLINT<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"16\"><span class=\"w\">   <\/span><span class=\"n\">cycles_after_drawer_full<\/span><span class=\"w\"> <\/span><span class=\"nb\">SMALLINT<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"17\">\n<\/span><span data-line=\"18\"><span class=\"w\">   <\/span><span class=\"n\">litter_level<\/span><span class=\"w\"> <\/span><span class=\"nb\">SMALLINT<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"19\"><span class=\"w\">   <\/span><span class=\"n\">litter_level_calculated<\/span><span class=\"w\"> <\/span><span class=\"nb\">SMALLINT<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"20\">\n<\/span><span data-line=\"21\"><span class=\"w\">   <\/span><span class=\"n\">waste_drawer_level<\/span><span class=\"w\"> <\/span><span class=\"nb\">SMALLINT<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"22\"><span class=\"w\">   <\/span><span class=\"n\">is_waste_drawer_full<\/span><span class=\"w\"> <\/span><span class=\"nb\">BOOLEAN<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"23\"><span class=\"w\">   <\/span><span class=\"n\">scoops_saved_count<\/span><span class=\"w\"> <\/span><span class=\"nb\">INT<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"24\">\n<\/span><span data-line=\"25\"><span class=\"w\">   <\/span><span class=\"k\">PRIMARY<\/span><span class=\"w\"> <\/span><span class=\"k\">KEY<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span><span class=\"n\">bot_id<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"n\">ts<\/span><span class=\"p\">)<\/span>\n<\/span><span data-line=\"26\"><span class=\"p\">)<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<p>There\u2019s a number of groupings here. We start with a <code class=\"code highlight sql PSQL docutils literal highlight-PSQL\"><span class=\"n\">bot_id<\/span><\/code> and timestamp, as usual. As with the cycle activity, if a bot gets removed from our database, so will all of its corresponding status entries.<\/p>\n<p>The next set is about parsed values. The litterbot API gives a number of indicators for different sensors, and the library I\u2019m using has kindly reverse-engineered the logic of Whisker\u2019s app. The result is four text fields that tell you the most critical issues. <code class=\"code highlight sql PSQL docutils literal highlight-PSQL\"><span class=\"n\">status<\/span><\/code> consolidates all of them into a sort of danger signal. <code class=\"code highlight sql PSQL docutils literal highlight-PSQL\"><span class=\"n\">power_status<\/span><\/code> tells you what it\u2019s drawing power from (usually \u201cAC\u201d). The <code class=\"code highlight sql PSQL docutils literal highlight-PSQL\"><span class=\"n\">litter_level_state<\/span><\/code> tells you whether it\u2019s too full, just right, a bit low, or to \u201cREFILL\u201d right the hell now. Lastly is the <code class=\"code highlight sql PSQL docutils literal highlight-PSQL\"><span class=\"n\">surface_type<\/span><\/code>, which I think is set in the app. This tells the bot what type of flooring it\u2019s dealing with, which can affect weight measurements.<\/p>\n<p>The <code class=\"code highlight sql PSQL docutils literal highlight-PSQL\"><span class=\"n\">last_seen<\/span><\/code> value was, for a time, the bane of my existence. See, none of this is well-documented. You have to figure out what\u2019s going on for yourself, and I, naively, thought that the <code class=\"code highlight sql PSQL docutils literal highlight-PSQL\"><span class=\"n\">last_seen<\/span><\/code> time would <em>update<\/em> while the device is online. \u201cOh, we last contacted it ten minutes ago\u201d, things like that. But no. As it turns out, it shows <em>the last time it was offline<\/em>. If you lose internet at 6pm and gain it back at 6:30pm, that value will stay at 6:30pm until the next time you lose connection. This caused a <em>lot<\/em> of headaches initially, as I tried to use that as an indicator of when the status was last checked. Unfortunately that doesn\u2019t work, so I use the time of request for that purpose instead. The <code class=\"code highlight sql PSQL docutils literal highlight-PSQL\"><span class=\"n\">is_online<\/span><\/code> indicator is tied to that. The <code class=\"code highlight sql PSQL docutils literal highlight-PSQL\"><span class=\"n\">is_sleeping<\/span><\/code> indicator is slightly misleading, and seems to indicate the status of an onboard nightlight, rather than indicate that the device is in low power mode.<\/p>\n<p>The cycle grouping is less functional than I would wish for. You would <em>think<\/em> that <code class=\"code highlight sql PSQL docutils literal highlight-PSQL\"><span class=\"n\">cycle_capacity<\/span><\/code> indicated how many cycles could happen before full. That\u2019s not the case. It is instead counting since the last time you emptied the drawer. That\u2019s\u2026 less useful, but you could in principle use it to figure out how many cycles go between full states. The next is cycle count, which keeps track of the total number of cleaning cycles the machine has executed. Lastly is <code class=\"code highlight sql PSQL docutils literal highlight-PSQL\"><span class=\"n\">cycles_after_drawer_full<\/span><\/code>, which fortunately does exactly what it says. When the robot determines the drawer is full, it will stop cycling. If you force it to do so, it logs that so that it can be upset at you. That value just sits there, silently collecting rage. It doesn\u2019t seem to do anything with it at all.<\/p>\n<p>The litter level grouping is <em>fascinating<\/em>. Under the hood, what it\u2019s returning is data from a <a class=\"reference external\" href=\"https:\/\/en.wikipedia.org\/wiki\/Time_of_flight\">Time of Flight sensor<\/a>. It is bouncing light<a class=\"footnote-reference brackets\" href=\"#id11\" id=\"id4\" role=\"doc-noteref\"><span class=\"fn-bracket\">[<\/span>1<span class=\"fn-bracket\">]<\/span><\/a> off the bottom surface and measuring how long it takes to return. This gives you a fairly precise proxy for how many centimeters of litter you have. Of course, the API doesn\u2019t <em>interpret<\/em> that data, so that\u2019s left up to <code class=\"docutils literal notranslate\"><span class=\"pre\">pylitterbot<\/span><\/code>. The consequence of this is that the litterbot gives you warnings at 50% empty as measured by time of flight. The calculated value gives a percentage, and the plain one gives raw time of flight.<\/p>\n<p>The last grouping is very similar, but about the waste drawer. It has a percentage level for how full it is, an indicator of if it\u2019s considered full, and a running count of how many scoops it thinks it\u2019s saved you from performing.<\/p>\n<\/section>\n<section id=\"these-tables-aren-t-compact-why\">\n<h3>These Tables Aren\u2019t Compact. Why?<a class=\"headerlink\" href=\"#these-tables-aren-t-compact-why\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>If you\u2019re a frequent database user, you\u2019ve likely noticed several places in this code where data could be compacted. An example of doing this is with the <code class=\"code highlight sql PSQL docutils literal highlight-PSQL\"><span class=\"n\">bot_id<\/span><\/code> field, where it references another table. Why are we not doing that with the pet names? Why are we not doing that with the various status elements?<\/p>\n<p>The reason is relatively simple: it\u2019s not worth it. This portion of the database is very small. At time of writing, it occupies less than 1MiB of total space. I could use all sorts of tricks to reduce that size, but what would be the point? I would need to have about 400 years worth of data before it would occupy 1% of my total disk space. The amount of extra reasoning I would need to do in these queries pales in comparison.<\/p>\n<\/section>\n<\/section>\n<section id=\"final-product\">\n<h2>Final Product<a class=\"headerlink\" href=\"#final-product\" title=\"Link to this heading\">\u00b6<\/a><\/h2>\n<p>Let\u2019s examine the different data displays you can get from this data. All of them have some value, and any time they use tables we didn\u2019t specify above I\u2019ll give a brief summary of what they are.<\/p>\n<section id=\"litter-waste-levels\">\n<h3>Litter &amp; Waste Levels<a class=\"headerlink\" href=\"#litter-waste-levels\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>The queries we can use for litter and waste draw levels are <em>delightfully<\/em> simple, and gives us a good initial case to examine. They look quite similar:<\/p>\n<div class=\"highlight-PSQL notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"k\">SELECT<\/span><span class=\"w\"> <\/span><span class=\"n\">litter_level_calculated<\/span>\n<\/span><span data-line=\"2\"><span class=\"k\">FROM<\/span><span class=\"w\"> <\/span><span class=\"n\">litterbot<\/span><span class=\"mf\">.<\/span><span class=\"n\">bot_snapshot<\/span>\n<\/span><span data-line=\"3\"><span class=\"k\">ORDER<\/span><span class=\"w\"> <\/span><span class=\"k\">BY<\/span><span class=\"w\"> <\/span><span class=\"n\">ts<\/span><span class=\"w\"> <\/span><span class=\"k\">DESC<\/span>\n<\/span><span data-line=\"4\"><span class=\"k\">LIMIT<\/span><span class=\"w\"> <\/span><span class=\"mf\">1<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<p>And<\/p>\n<div class=\"highlight-PSQL notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"k\">SELECT<\/span><span class=\"w\"> <\/span><span class=\"n\">waste_drawer_level<\/span>\n<\/span><span data-line=\"2\"><span class=\"k\">FROM<\/span><span class=\"w\"> <\/span><span class=\"n\">litterbot<\/span><span class=\"mf\">.<\/span><span class=\"n\">bot_snapshot<\/span>\n<\/span><span data-line=\"3\"><span class=\"k\">ORDER<\/span><span class=\"w\"> <\/span><span class=\"k\">BY<\/span><span class=\"w\"> <\/span><span class=\"n\">ts<\/span><span class=\"w\"> <\/span><span class=\"k\">DESC<\/span>\n<\/span><span data-line=\"4\"><span class=\"k\">LIMIT<\/span><span class=\"w\"> <\/span><span class=\"mf\">1<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<p>The first part tells us what values we are looking for. Since each of these queries only read a single value, it\u2019s fairly straightforward. The next bit tells us which table to look in, that we should order it by the time recorded (with most recent first), and that we only want a single entry. This lets us get a very pretty gauge panel, as rendered by <span id=\"id5\">[<a class=\"reference internal\" href=\"#id124\" title=\"Grafana gauge visualization. URL: https:\/\/grafana.com\/docs\/grafana\/latest\/visualizations\/panels-visualizations\/visualizations\/guage\/.\">graa<\/a>]<\/span>:<\/p>\n<figure class=\"align-center\" id=\"id130\">\n<a class=\"reference internal image-reference\" href=\"..\/_images\/gauges.png\"><img alt=\"Two gauge charts rendering the above queries, as described below\" src=\"..\/_images\/gauges.png\" style=\"width: 339.0px; height: 349.0px;\" \/>\n<\/a>\n<figcaption>\n<p><span class=\"caption-text\">This shows two gauge charts. The top one shows the current level of the waste drawer, and the bottom the current level of the litter reservoir. There is a solid arc of color on each that sweeps to the current level, leaving the rest unfilled.<\/span><a class=\"headerlink\" href=\"#id130\" title=\"Link to this image\">\u00b6<\/a><\/p>\n<\/figcaption>\n<\/figure>\n<p>You\u2019ll notice the gauges have colored sections. We can set these in the configuration for it. In this case, I have tried to reverse-engineer what the app displays, and add some logic for how critical <em>I<\/em> think it is. For instance, the Whisker app does not distinguish between the fields I label as orange and the fields I label as red.<\/p>\n<\/section>\n<section id=\"current-status\">\n<h3>Current Status<a class=\"headerlink\" href=\"#current-status\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>Our current status query is going to look similar to the ones above, except that it is requesting many more values. We\u2019ll look at this in two steps: first with a simplified query, then a hidden section that shows the actual implementation which colors text correctly.<\/p>\n<p>Our query here is as follows:<\/p>\n<div class=\"highlight-PSQL notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"k\">SELECT<\/span>\n<\/span><span data-line=\"2\"><span class=\"w\">   <\/span><span class=\"n\">power_status<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"3\"><span class=\"w\">   <\/span><span class=\"n\">is_online<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"4\"><span class=\"w\">   <\/span><span class=\"n\">is_sleeping<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"5\"><span class=\"w\">   <\/span><span class=\"n\">initcap<\/span><span class=\"p\">(<\/span><span class=\"n\">status<\/span><span class=\"p\">)<\/span><span class=\"w\"> <\/span><span class=\"k\">AS<\/span><span class=\"w\"> <\/span><span class=\"n\">status<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"6\"><span class=\"w\">   <\/span><span class=\"n\">litter_level_state<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"7\"><span class=\"w\">   <\/span><span class=\"n\">is_waste_drawer_full<\/span>\n<\/span><span data-line=\"8\"><span class=\"k\">FROM<\/span><span class=\"w\"> <\/span><span class=\"n\">litterbot<\/span><span class=\"mf\">.<\/span><span class=\"n\">bot_snapshot<\/span>\n<\/span><span data-line=\"9\"><span class=\"k\">ORDER<\/span><span class=\"w\"> <\/span><span class=\"k\">BY<\/span><span class=\"w\"> <\/span><span class=\"n\">ts<\/span><span class=\"w\"> <\/span><span class=\"k\">DESC<\/span>\n<\/span><span data-line=\"10\"><span class=\"k\">LIMIT<\/span><span class=\"w\"> <\/span><span class=\"mf\">1<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<p>As before, the first part tells us what values we are looking for. This is fairly similar to the above, except for the <code class=\"code highlight sql PSQL docutils literal highlight-PSQL\"><span class=\"n\">status<\/span><\/code> field. This is run through a function <code class=\"code highlight sql PSQL docutils literal highlight-PSQL\"><span class=\"n\">initcap<\/span><span class=\"p\">()<\/span><\/code>, which transforms strings like \u201caBcDEfG\u201d into \u201cAbcdefg\u201d. This is useful because the status returned by <code class=\"docutils literal notranslate\"><span class=\"pre\">pylitterbot<\/span><\/code> is in all caps, which I find aesthetically displeasing. It will say things like \u201cREADY\u201d or \u201cREFILL\u201d, and now in the panel it instead says \u201cReady\u201d or \u201cRefill\u201d.<\/p>\n<details><summary>Color implementation details here.<\/summary><div class=\"docutils container\">\n<p>Unfortunately this isn\u2019t quite enough to get colors working on the display. Grafana lets you set value mappings for these, but because different columns have different rules for what\u2019s good, we have to distinguish them. I accomplished that by giving prefixes. The resulting query looks like:<\/p>\n<div class=\"highlight-PSQL notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"k\">SELECT<\/span>\n<\/span><span data-line=\"2\"><span class=\"w\">   <\/span><span class=\"n\">power_status<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"3\"><span class=\"w\">   <\/span><span class=\"k\">CASE<\/span>\n<\/span><span data-line=\"4\"><span class=\"w\">      <\/span><span class=\"k\">WHEN<\/span><span class=\"w\"> <\/span><span class=\"n\">is_online<\/span><span class=\"w\"> <\/span><span class=\"k\">THEN<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;GYes&#39;<\/span>\n<\/span><span data-line=\"5\"><span class=\"w\">      <\/span><span class=\"k\">ELSE<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;RNo&#39;<\/span>\n<\/span><span data-line=\"6\"><span class=\"w\">   <\/span><span class=\"k\">END<\/span><span class=\"w\"> <\/span><span class=\"k\">AS<\/span><span class=\"w\"> <\/span><span class=\"n\">is_online<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"7\"><span class=\"w\">   <\/span><span class=\"k\">CASE<\/span>\n<\/span><span data-line=\"8\"><span class=\"w\">      <\/span><span class=\"k\">WHEN<\/span><span class=\"w\"> <\/span><span class=\"n\">is_sleeping<\/span><span class=\"w\"> <\/span><span class=\"k\">THEN<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;YYes&#39;<\/span>\n<\/span><span data-line=\"9\"><span class=\"w\">      <\/span><span class=\"k\">ELSE<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;YNo&#39;<\/span>\n<\/span><span data-line=\"10\"><span class=\"w\">   <\/span><span class=\"k\">END<\/span><span class=\"w\"> <\/span><span class=\"k\">AS<\/span><span class=\"w\"> <\/span><span class=\"n\">is_sleeping<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"11\"><span class=\"w\">   <\/span><span class=\"n\">initcap<\/span><span class=\"p\">(<\/span><span class=\"n\">status<\/span><span class=\"p\">)<\/span><span class=\"w\"> <\/span><span class=\"k\">AS<\/span><span class=\"w\"> <\/span><span class=\"n\">status<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"12\"><span class=\"w\">   <\/span><span class=\"n\">litter_level_state<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"13\"><span class=\"w\">   <\/span><span class=\"k\">CASE<\/span>\n<\/span><span data-line=\"14\"><span class=\"w\">      <\/span><span class=\"k\">WHEN<\/span><span class=\"w\"> <\/span><span class=\"n\">is_waste_drawer_full<\/span><span class=\"w\"> <\/span><span class=\"k\">THEN<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;RYes&#39;<\/span>\n<\/span><span data-line=\"15\"><span class=\"w\">      <\/span><span class=\"k\">ELSE<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;GNo&#39;<\/span>\n<\/span><span data-line=\"16\"><span class=\"w\">   <\/span><span class=\"k\">END<\/span><span class=\"w\"> <\/span><span class=\"k\">AS<\/span><span class=\"w\"> <\/span><span class=\"n\">is_waste_drawer_full<\/span>\n<\/span><span data-line=\"17\"><span class=\"k\">from<\/span><span class=\"w\"> <\/span><span class=\"n\">litterbot<\/span><span class=\"mf\">.<\/span><span class=\"n\">bot_snapshot<\/span>\n<\/span><span data-line=\"18\"><span class=\"k\">ORDER<\/span><span class=\"w\"> <\/span><span class=\"k\">BY<\/span><span class=\"w\"> <\/span><span class=\"n\">ts<\/span><span class=\"w\"> <\/span><span class=\"k\">DESC<\/span>\n<\/span><span data-line=\"19\"><span class=\"k\">LIMIT<\/span><span class=\"w\"> <\/span><span class=\"mf\">1<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<p>Syntactically this is very similar. What I\u2019ve mainly done is added a letter prefix indicating the color. G indicates it should be green, R that it should be red, and Y that it should be uncolored. Initially I used that for yellow, but it didn\u2019t end up making sense. Then you make a value mapping that removes the prefix and adds the color.<\/p>\n<\/div>\n<\/details><p>We can render this with Grafana\u2019s Stat plugin <span id=\"id6\">[<a class=\"reference internal\" href=\"#id119\" title=\"Grafana stat visualization. URL: https:\/\/grafana.com\/docs\/grafana\/latest\/visualizations\/panels-visualizations\/visualizations\/stat\/ (visited on 2026-02-23).\">grac<\/a>]<\/span>, which allows us to render individual values. The result looks like this:<\/p>\n<figure class=\"align-center\" id=\"id131\">\n<img alt=\"A stat panel displaying the above query, as described below\" src=\"..\/_images\/current_status.png\" \/>\n<figcaption>\n<p><span class=\"caption-text\">Six status messages are shown side by side. For each of them, there is a smaller title above, with a larger status below. Five of them are colored green, indicating a positive status, and the one labeled \u201cSleeping?\u201d is colored white because it is status-neutral.<\/span><a class=\"headerlink\" href=\"#id131\" title=\"Link to this image\">\u00b6<\/a><\/p>\n<\/figcaption>\n<\/figure>\n<\/section>\n<section id=\"cycles-per-day\">\n<h3>Cycles Per Day<a class=\"headerlink\" href=\"#cycles-per-day\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>Our cycles per day panel is a time series, as we discussed <a class=\"reference external\" href=\"0006-lessons-in-grafana-01#time-series\">in the last entry<\/a> <span id=\"id7\">[<a class=\"reference internal\" href=\"0009-lessons-in-grafana-03.html#id27\" title=\"Olivia Appleton-Crocker. Lessons in grafana - part one: a vision. URL: https:\/\/blog.oliviaappleton.com\/posts\/0006-lessons-in-grafana-01.\">ACb<\/a>]<\/span>. You can look there for information on the relevant data format. This panel introduces a new concept for us: time filtering.<\/p>\n<div class=\"highlight-PSQL notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"k\">SELECT<\/span><span class=\"w\"> <\/span><span class=\"k\">day<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"n\">cycles<\/span>\n<\/span><span data-line=\"2\"><span class=\"k\">FROM<\/span><span class=\"w\"> <\/span><span class=\"n\">litterbot<\/span><span class=\"mf\">.<\/span><span class=\"n\">cycle_activity<\/span>\n<\/span><span data-line=\"3\"><span class=\"k\">WHERE<\/span><span class=\"w\"> <\/span><span class=\"err\">$<\/span><span class=\"n\">__timeFilter<\/span><span class=\"p\">(<\/span><span class=\"k\">day<\/span><span class=\"p\">)<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<p>Grafana provides us with a lovely little function, <code class=\"code highlight sql PSQL docutils literal highlight-PSQL\"><span class=\"gp\">$<\/span>__timeFilter<span class=\"o\">()<\/span><\/code>, which allows us to look only at data that\u2019s being displayed in the current dashboard. This saves us a lot of effort on data lookup, transfer, and processing. Because we are looking at discrete values rather than one thing changing over time, we use a bar chart to represent it:<\/p>\n<figure class=\"align-center\" id=\"id132\">\n<img alt=\"A time series rendering the above query, as described below\" src=\"..\/_images\/cycles_per_day.png\" \/>\n<figcaption>\n<p><span class=\"caption-text\">This panel shows a time series rendered with bars. Each bar represents a day\u2019s worth of usage. It is total usage, not partitioned per-cat.<\/span><a class=\"headerlink\" href=\"#id132\" title=\"Link to this image\">\u00b6<\/a><\/p>\n<\/figcaption>\n<\/figure>\n<\/section>\n<section id=\"device-logs\">\n<h3>Device Logs<a class=\"headerlink\" href=\"#device-logs\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>This panel utilizes a table I didn\u2019t describe above, because it\u2019s mostly useful for debugging purposes. The Litterbot produces a log of actions it\u2019s taken, which can be displayed using the Grafana logs plugin <span id=\"id8\">[<a class=\"reference internal\" href=\"#id116\" title=\"Grafana logs visualization. URL: https:\/\/grafana.com\/docs\/grafana\/latest\/visualizations\/panels-visualizations\/visualizations\/logs\/ (visited on 2026-02-23).\">grab<\/a>]<\/span>. Our query is as follows:<\/p>\n<div class=\"highlight-PSQL notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"k\">SELECT<\/span><span class=\"w\"> <\/span><span class=\"n\">ts<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"k\">action<\/span>\n<\/span><span data-line=\"2\"><span class=\"k\">FROM<\/span><span class=\"w\"> <\/span><span class=\"n\">litterbot<\/span><span class=\"mf\">.<\/span><span class=\"n\">bot_activity<\/span>\n<\/span><span data-line=\"3\"><span class=\"k\">WHERE<\/span><span class=\"w\"> <\/span><span class=\"err\">$<\/span><span class=\"n\">__timeFilter<\/span><span class=\"p\">(<\/span><span class=\"n\">ts<\/span><span class=\"p\">)<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<figure class=\"align-center\" id=\"id133\">\n<a class=\"reference internal image-reference\" href=\"..\/_images\/device_logs.png\"><img alt=\"A log panel rendering the above query, as described below\" src=\"..\/_images\/device_logs.png\" style=\"width: 601.2px; height: 522.0px;\" \/>\n<\/a>\n<figcaption>\n<p><span class=\"caption-text\">This panel shows the logs generated by the litterbot. They are ordered such that the most recent event is at the top. There is a timestamp on the left for each entry. Some of the entries are raw event statuses, like \u201cCAT_DETECTED\u201d, whereas others are clearly formatted strings, like \u201cClean Cycles: 611\u201d.<\/span><a class=\"headerlink\" href=\"#id133\" title=\"Link to this image\">\u00b6<\/a><\/p>\n<\/figcaption>\n<\/figure>\n<p>As you can see, this is a fairly useful debug view. You can tell when cats enter, when a cleaning cycle is interrupted, as well as the exact moment it believes that the waste drawer is full. This is more detail than you typically need, but it helps understand what\u2019s going on if presented with suspect data. That\u2019s quite useful.<\/p>\n<\/section>\n<section id=\"per-pet-usage-history\">\n<h3>Per-Pet Usage History<a class=\"headerlink\" href=\"#per-pet-usage-history\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>This is, in my opinion, the second most medically-useful chart in this dashboard. It gives you a cumulative view of how often each pet uses the litterbox. Now, this isn\u2019t comprehensive. There is a traditional (read: manually cleaned) litterbox next to it, in part because Maya is an anxious princess and in part so two cats can use the bathroom at once. But this does give a rough proxy, and that is medically useful. It can tell you if they are experiencing digestive issues, or if they\u2019re having trouble urinating, or etc.<\/p>\n<p>An example of this is that Rye recently was sick. She threw up for a while, then had a noticeable uptick in litterbox usage that you can see below. There are alternate explanations for that uptick (for example, she is very easily distractible, and will sometimes go back in because she saw it moving), but this fits very well with the standard progression of a short-term stomach bug. It gives us an indicator that her symptoms are mostly passed.<\/p>\n<p>The query for this is a bit more complex than the others covered here. It uses a <code class=\"code highlight sql PSQL docutils literal highlight-PSQL\"><span class=\"k\">UNION<\/span><\/code> statement to combine two data sets. The first part is going to give us a baseline for the time series: everybody starts at 0 times used when the chart begins. The next portion counts how many times a pet appears in the time period we\u2019re looking at.<\/p>\n<div class=\"highlight-PSQL notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"k\">SELECT<\/span><span class=\"w\"> <\/span><span class=\"k\">DISTINCT<\/span>\n<\/span><span data-line=\"2\"><span class=\"w\">   <\/span><span class=\"nb\">TIMESTAMP<\/span><span class=\"w\"> <\/span><span class=\"nb\">WITH TIME ZONE<\/span><span class=\"w\"> <\/span><span class=\"err\">$<\/span><span class=\"n\">__timeFrom<\/span><span class=\"p\">()<\/span><span class=\"w\"> <\/span><span class=\"k\">as<\/span><span class=\"w\"> <\/span><span class=\"n\">ts<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"3\"><span class=\"w\">   <\/span><span class=\"n\">pet_name<\/span><span class=\"w\"> <\/span><span class=\"k\">as<\/span><span class=\"w\"> <\/span><span class=\"n\">pet<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"4\"><span class=\"w\">   <\/span><span class=\"mf\">0<\/span><span class=\"w\"> <\/span><span class=\"k\">as<\/span><span class=\"w\"> <\/span><span class=\"k\">event<\/span>\n<\/span><span data-line=\"5\"><span class=\"k\">FROM<\/span><span class=\"w\"> <\/span><span class=\"n\">litterbot<\/span><span class=\"mf\">.<\/span><span class=\"n\">pet_weight<\/span>\n<\/span><span data-line=\"6\"><span class=\"k\">UNION<\/span><span class=\"w\"> <\/span><span class=\"k\">ALL<\/span>\n<\/span><span data-line=\"7\"><span class=\"k\">SELECT<\/span>\n<\/span><span data-line=\"8\"><span class=\"w\">   <\/span><span class=\"n\">ts<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"9\"><span class=\"w\">   <\/span><span class=\"n\">pet_name<\/span><span class=\"w\"> <\/span><span class=\"k\">as<\/span><span class=\"w\"> <\/span><span class=\"n\">pet<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"10\"><span class=\"w\">   <\/span><span class=\"n\">sum<\/span><span class=\"p\">(<\/span><span class=\"mf\">1<\/span><span class=\"p\">)<\/span><span class=\"w\"> <\/span><span class=\"k\">OVER<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span><span class=\"k\">PARTITION<\/span><span class=\"w\"> <\/span><span class=\"k\">BY<\/span><span class=\"w\"> <\/span><span class=\"n\">pet_name<\/span><span class=\"w\"> <\/span><span class=\"k\">ORDER<\/span><span class=\"w\"> <\/span><span class=\"k\">BY<\/span><span class=\"w\"> <\/span><span class=\"n\">ts<\/span><span class=\"p\">)<\/span><span class=\"w\"> <\/span><span class=\"k\">as<\/span><span class=\"w\"> <\/span><span class=\"k\">event<\/span>\n<\/span><span data-line=\"11\"><span class=\"k\">FROM<\/span><span class=\"w\"> <\/span><span class=\"n\">litterbot<\/span><span class=\"mf\">.<\/span><span class=\"n\">pet_weight<\/span>\n<\/span><span data-line=\"12\"><span class=\"k\">WHERE<\/span><span class=\"w\"> <\/span><span class=\"err\">$<\/span><span class=\"n\">__timeFilter<\/span><span class=\"p\">(<\/span><span class=\"n\">ts<\/span><span class=\"p\">)<\/span>\n<\/span><span data-line=\"13\"><span class=\"k\">ORDER<\/span><span class=\"w\"> <\/span><span class=\"k\">BY<\/span><span class=\"w\"> <\/span><span class=\"n\">pet<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"n\">ts<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<p>Let\u2019s break that down in more detail. The <code class=\"code highlight sql PSQL docutils literal highlight-PSQL\"><span class=\"gp\">$<\/span>__timeFrom<span class=\"o\">()<\/span><\/code> function gives us the beginning of the time series chart as set in Grafana. It\u2019s a macro that\u2019s provided to us, and not available in most SQL contexts. We have to specify a type for it because it\u2019s fed in as a raw value. This is much like how we are able to define an <code class=\"code highlight sql PSQL docutils literal highlight-PSQL\"><span class=\"nb\">interval<\/span><\/code> using a string like <code class=\"code highlight sql PSQL docutils literal highlight-PSQL\"><span class=\"s1\">&#39;7 days&#39;<\/span><\/code>, which you\u2019ll see later in this entry.<\/p>\n<p>In the next <code class=\"code highlight sql PSQL docutils literal highlight-PSQL\"><span class=\"k\">SELECT<\/span><\/code> statement you have something new to us: an aggregate function. This is relatively simple. The <code class=\"code highlight sql PSQL docutils literal highlight-PSQL\"><span class=\"n\">sum<\/span><span class=\"p\">(<\/span><span class=\"mf\">1<\/span><span class=\"p\">)<\/span><\/code> simply counts the number of entries we see. The next bit says what we are counting, which is all values with a matching <code class=\"code highlight sql PSQL docutils literal highlight-PSQL\"><span class=\"n\">pet_name<\/span><\/code> ordered by time. This means that for each event we are counting it and all preceding events in the time window.<\/p>\n<p>This query is run through a couple transformations to give us separate datasets. First we tell Grafana to partition the events by the value of <code class=\"code highlight sql PSQL docutils literal highlight-PSQL\"><span class=\"n\">pet_name<\/span><\/code>, and then we have a quick rename to remove the prefix that adds to each series. This is a very common pattern in our time series charts, and I will likely not mention it in the future.<\/p>\n<figure class=\"align-center\" id=\"id134\">\n<img alt=\"A time series rendering the query above, as described below\" src=\"..\/_images\/cumulative_usage.png\" \/>\n<figcaption>\n<p><span class=\"caption-text\">Each cat starts with a point saying 0 usage at the beginning of the time range. From then on, they have a dotted line that connects each additional usage, climbing monotonically. As usual, the orange represents Rye and the green represents Loaf.<\/span><a class=\"headerlink\" href=\"#id134\" title=\"Link to this image\">\u00b6<\/a><\/p>\n<\/figcaption>\n<\/figure>\n<\/section>\n<section id=\"relative-usage-per-pet\">\n<h3>Relative Usage Per Pet<a class=\"headerlink\" href=\"#relative-usage-per-pet\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>This chart is a simplified view of the one above, packed into a pie chart as discussed <a class=\"reference external\" href=\"0006-lessons-in-grafana-01#pie-charts\">in the last entry<\/a> <span id=\"id9\">[<a class=\"reference internal\" href=\"0009-lessons-in-grafana-03.html#id27\" title=\"Olivia Appleton-Crocker. Lessons in grafana - part one: a vision. URL: https:\/\/blog.oliviaappleton.com\/posts\/0006-lessons-in-grafana-01.\">ACb<\/a>]<\/span>. It gives relative usage between pets that you can see at a glance. In this case, you can see that Rye used the litterbox about 4 times as often as Loaf did in the last week. This is a bit higher than normal (about 3 times more), probably because she had that stomach bug. I\u2019ll be curious to see if that evens out more as Rye gets older. Kids are usually not good at holding it, after all.<\/p>\n<div class=\"highlight-PSQL notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"k\">SELECT<\/span>\n<\/span><span data-line=\"2\"><span class=\"w\">   <\/span><span class=\"n\">pet_name<\/span><span class=\"w\"> <\/span><span class=\"k\">as<\/span><span class=\"w\"> <\/span><span class=\"n\">pet<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"3\"><span class=\"w\">   <\/span><span class=\"n\">sum<\/span><span class=\"p\">(<\/span><span class=\"mf\">1<\/span><span class=\"p\">)<\/span><span class=\"w\"> <\/span><span class=\"k\">OVER<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span><span class=\"k\">PARTITION<\/span><span class=\"w\"> <\/span><span class=\"k\">BY<\/span><span class=\"w\"> <\/span><span class=\"n\">pet_name<\/span><span class=\"w\"> <\/span><span class=\"k\">ORDER<\/span><span class=\"w\"> <\/span><span class=\"k\">BY<\/span><span class=\"w\"> <\/span><span class=\"n\">ts<\/span><span class=\"p\">)<\/span><span class=\"w\"> <\/span><span class=\"k\">as<\/span><span class=\"w\"> <\/span><span class=\"k\">event<\/span>\n<\/span><span data-line=\"4\"><span class=\"k\">FROM<\/span><span class=\"w\"> <\/span><span class=\"n\">litterbot<\/span><span class=\"mf\">.<\/span><span class=\"n\">pet_weight<\/span>\n<\/span><span data-line=\"5\"><span class=\"k\">WHERE<\/span><span class=\"w\"> <\/span><span class=\"err\">$<\/span><span class=\"n\">__timeFilter<\/span><span class=\"p\">(<\/span><span class=\"n\">ts<\/span><span class=\"p\">)<\/span>\n<\/span><span data-line=\"6\"><span class=\"k\">ORDER<\/span><span class=\"w\"> <\/span><span class=\"k\">BY<\/span><span class=\"w\"> <\/span><span class=\"n\">pet<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<figure class=\"align-center\" id=\"id135\">\n<a class=\"reference internal image-reference\" href=\"..\/_images\/relative_usage.png\"><img alt=\"A pie chart that renders the query above, as described below.\" src=\"..\/_images\/relative_usage.png\" style=\"width: 330.0px; height: 234.5px;\" \/>\n<\/a>\n<figcaption>\n<p><span class=\"caption-text\">This pie chart reflects overall litterbox usage over the requested period. The orange portion, corresponding to Rye, is much larger than the green portion, corresponding to Loaf.<\/span><a class=\"headerlink\" href=\"#id135\" title=\"Link to this image\">\u00b6<\/a><\/p>\n<\/figcaption>\n<\/figure>\n<\/section>\n<section id=\"per-pet-weight-history\">\n<h3>Per-Pet Weight History<a class=\"headerlink\" href=\"#per-pet-weight-history\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>This is the most complicated, and most medically-useful, panel in this dashboard. It gives us pet weights over time, including a running average. That average was <em>really<\/em> important for me to get right, and it ended up being enough of a project that I detail it in next week\u2019s bonus entry. Stay tuned for that. The query is in five parts:<\/p>\n<ol class=\"arabic simple\">\n<li><p>Setting constants<\/p><\/li>\n<li><p>Setting up filtering<\/p><\/li>\n<li><p>Fetching raw weights<\/p><\/li>\n<li><p>Calculating the running average<\/p><\/li>\n<li><p>Combining and ordering values<\/p><\/li>\n<\/ol>\n<div class=\"highlight-PSQL notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"k\">WITH<\/span><span class=\"w\"> <\/span><span class=\"n\">constants<\/span><span class=\"w\"> <\/span><span class=\"k\">AS<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span>\n<\/span><span data-line=\"2\"><span class=\"w\">   <\/span><span class=\"k\">SELECT<\/span><span class=\"w\"> <\/span><span class=\"nb\">interval<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;4 days&#39;<\/span><span class=\"w\"> <\/span><span class=\"k\">AS<\/span><span class=\"w\"> <\/span><span class=\"n\">half_life<\/span>\n<\/span><span data-line=\"3\"><span class=\"p\">),<\/span>\n<\/span><span data-line=\"4\"><span class=\"c1\">-- We&#39;re using 4 days as a half-life because it provides a nice smoothing<\/span>\n<\/span><span data-line=\"5\">\n<\/span><span data-line=\"6\"><span class=\"n\">base<\/span><span class=\"w\"> <\/span><span class=\"k\">AS<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span>\n<\/span><span data-line=\"7\"><span class=\"w\">   <\/span><span class=\"k\">SELECT<\/span><span class=\"w\"> <\/span><span class=\"o\">*<\/span>\n<\/span><span data-line=\"8\"><span class=\"w\">   <\/span><span class=\"k\">FROM<\/span><span class=\"w\"> <\/span><span class=\"n\">constants<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"n\">litterbot<\/span><span class=\"mf\">.<\/span><span class=\"n\">pet_weight<\/span>\n<\/span><span data-line=\"9\"><span class=\"w\">   <\/span><span class=\"k\">WHERE<\/span><span class=\"w\"> <\/span><span class=\"n\">ts<\/span><span class=\"w\"> <\/span><span class=\"o\">&lt;<\/span><span class=\"w\"> <\/span><span class=\"err\">$<\/span><span class=\"n\">__timeTo<\/span><span class=\"p\">()<\/span>\n<\/span><span data-line=\"10\"><span class=\"w\">      <\/span><span class=\"k\">AND<\/span><span class=\"w\"> <\/span><span class=\"n\">ts<\/span><span class=\"w\"> <\/span><span class=\"o\">&gt;<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span><span class=\"nb\">TIMESTAMP<\/span><span class=\"w\"> <\/span><span class=\"nb\">WITH TIME ZONE<\/span><span class=\"w\"> <\/span><span class=\"err\">$<\/span><span class=\"n\">__timeFrom<\/span><span class=\"p\">())<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span><span class=\"n\">half_life<\/span><span class=\"w\"> <\/span><span class=\"o\">*<\/span><span class=\"w\"> <\/span><span class=\"mf\">2<\/span><span class=\"p\">)<\/span>\n<\/span><span data-line=\"11\"><span class=\"p\">)<\/span>\n<\/span><span data-line=\"12\"><span class=\"c1\">-- This looks like we aren&#39;t filtering enough, but that&#39;s not true.<\/span>\n<\/span><span data-line=\"13\"><span class=\"c1\">-- We want to ensure that our average has enough initial data to work with.<\/span>\n<\/span><span data-line=\"14\">\n<\/span><span data-line=\"15\"><span class=\"k\">SELECT<\/span><span class=\"w\"> <\/span><span class=\"n\">ts<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"n\">pet_name<\/span><span class=\"w\"> <\/span><span class=\"k\">AS<\/span><span class=\"w\"> <\/span><span class=\"n\">pet<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"n\">weight<\/span>\n<\/span><span data-line=\"16\"><span class=\"k\">FROM<\/span><span class=\"w\"> <\/span><span class=\"n\">base<\/span>\n<\/span><span data-line=\"17\"><span class=\"c1\">-- This gets the raw weights<\/span>\n<\/span><span data-line=\"18\">\n<\/span><span data-line=\"19\"><span class=\"k\">UNION<\/span><span class=\"w\"> <\/span><span class=\"k\">ALL<\/span>\n<\/span><span data-line=\"20\">\n<\/span><span data-line=\"21\"><span class=\"c1\">-- and combines it with the running average<\/span>\n<\/span><span data-line=\"22\"><span class=\"k\">SELECT<\/span>\n<\/span><span data-line=\"23\"><span class=\"w\">   <\/span><span class=\"n\">ts<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"24\"><span class=\"w\">   <\/span><span class=\"n\">pet_name<\/span><span class=\"w\"> <\/span><span class=\"o\">||<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39; R.A.&#39;<\/span><span class=\"w\"> <\/span><span class=\"k\">AS<\/span><span class=\"w\"> <\/span><span class=\"n\">pet<\/span><span class=\"p\">,<\/span><span class=\"w\">  <\/span><span class=\"c1\">-- this is string concatenation<\/span>\n<\/span><span data-line=\"25\"><span class=\"w\">   <\/span><span class=\"n\">lgbot<\/span><span class=\"mf\">.<\/span><span class=\"n\">ema<\/span><span class=\"p\">(<\/span>\n<\/span><span data-line=\"26\"><span class=\"w\">      <\/span><span class=\"n\">weight<\/span><span class=\"o\">::<\/span><span class=\"nb\">numeric<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"27\"><span class=\"w\">      <\/span><span class=\"n\">ts<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"28\"><span class=\"w\">      <\/span><span class=\"n\">half_life<\/span><span class=\"w\"> <\/span><span class=\"o\">\/<\/span><span class=\"w\"> <\/span><span class=\"n\">ln<\/span><span class=\"p\">(<\/span><span class=\"mf\">2<\/span><span class=\"p\">)<\/span>\n<\/span><span data-line=\"29\"><span class=\"c1\">-- this converts from half-life to expected tau values<\/span>\n<\/span><span data-line=\"30\"><span class=\"w\">   <\/span><span class=\"p\">)<\/span><span class=\"w\"> <\/span><span class=\"k\">OVER<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span><span class=\"k\">PARTITION<\/span><span class=\"w\"> <\/span><span class=\"k\">BY<\/span><span class=\"w\"> <\/span><span class=\"n\">pet_name<\/span><span class=\"w\"> <\/span><span class=\"k\">ORDER<\/span><span class=\"w\"> <\/span><span class=\"k\">BY<\/span><span class=\"w\"> <\/span><span class=\"n\">ts<\/span><span class=\"p\">)<\/span>\n<\/span><span data-line=\"31\"><span class=\"c1\">-- and partitioning by pet name means we get that average per-pet<\/span>\n<\/span><span data-line=\"32\"><span class=\"k\">FROM<\/span><span class=\"w\"> <\/span><span class=\"n\">base<\/span>\n<\/span><span data-line=\"33\">\n<\/span><span data-line=\"34\"><span class=\"k\">ORDER<\/span><span class=\"w\"> <\/span><span class=\"k\">BY<\/span><span class=\"w\"> <\/span><span class=\"n\">pet<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"n\">ts<\/span><span class=\"p\">;<\/span>\n<\/span><span data-line=\"35\"><span class=\"go\">-- And then it gets ordered appropriately<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<figure class=\"align-center\" id=\"id136\">\n<img alt=\"A time series built using the query above, as described below.\" src=\"..\/_images\/pet_weights.png\" \/>\n<figcaption>\n<p><span class=\"caption-text\">This chart shows a time series of pet weights. There are dotted lines that represent the raw values, and a solid line that represents the time-weighted average. The scale is in US pounds, as that\u2019s what a veterinarian here will want to see (and what the Litterbot exports). Rye has many more weights logged than Loaf, reflecting her greater frequency of usage.<\/span><a class=\"headerlink\" href=\"#id136\" title=\"Link to this image\">\u00b6<\/a><\/p>\n<div class=\"legend\">\n<p>[Insert jokes about not using the metric system here.]<\/p>\n<\/div>\n<\/figcaption>\n<\/figure>\n<\/section>\n<section id=\"all-together\">\n<h3>All Together<a class=\"headerlink\" href=\"#all-together\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<figure class=\"align-center\" id=\"id137\">\n<img alt=\"A display of the whole dashboard, as described below\" src=\"..\/_images\/whole_dashboard.png\" \/>\n<figcaption>\n<p><span class=\"caption-text\">This shows the whole dashboard, giving all of the panels described here. On the top left you see the current status. Below that are two columns, with the one on the left dedicated to the device logs. To the right of that, occupying the middle of the screen, are the gauges and pie chart. From top to bottom we have: waste drawer level, litter level, and relative usage. On the right side of the dashboard we have our three time series. From top to bottom they are: pet weight over time, litterbox usage, and usage per day.<\/span><a class=\"headerlink\" href=\"#id137\" title=\"Link to this image\">\u00b6<\/a><\/p>\n<\/figcaption>\n<\/figure>\n<\/section>\n<\/section>\n<section id=\"main-takeaways\">\n<h2>Main Takeaways<a class=\"headerlink\" href=\"#main-takeaways\" title=\"Link to this heading\">\u00b6<\/a><\/h2>\n<p>I enjoyed making this a lot! When I was writing it, it had the most complicated query I\u2019d implemented in this project. The data collection itself was fairly trivial once I figured out how <code class=\"docutils literal notranslate\"><span class=\"pre\">pylitterbot<\/span><\/code> represented things, and much of the effort went into finding the optimal layout of the database for my purposes. Each of these panels came about because of a specific need, and the data collection I\u2019m performing allows for more to be displayed later on.<\/p>\n<p>The most interesting thing that came out of this was revamping the way I initialize tables. Before this, I had it running <code class=\"code highlight sql PSQL docutils literal highlight-PSQL\"><span class=\"k\">CREATE<\/span><span class=\"w\"> <\/span><span class=\"k\">TABLE<\/span><span class=\"w\"> <\/span><span class=\"k\">IF<\/span><span class=\"w\"> <\/span><span class=\"k\">NOT<\/span><span class=\"w\"> <\/span><span class=\"k\">EXISTS<\/span><\/code> for every time it ran. As it turns out, that is <em>terrible<\/em> for performance. What I do now is create those tables <em>only on startup<\/em>, and checking that they exist beforehand with a lovely utility function. This was especially useful for the litterbot scraper, as I previously had several indexes and additional tables. The version presented here is <em>much<\/em> simpler than I initially attempted.<\/p>\n<p>I hope to use this as a model for when I eventually try to collect data from our thermostat and a couple other cat devices.<\/p>\n<p>Next week we dive into the weeds of a specific feature, in a bonus entry: <a class=\"reference external\" href=\"https:\/\/blog.oliviaappleton.com\/posts\/0008-lessons-in-grafana-02-5\"><em>Achieving Acceptable Averages<\/em>.<\/a><\/p>\n<\/section>\n<section id=\"acknowledgments\">\n<h2>Acknowledgments<a class=\"headerlink\" href=\"#acknowledgments\" title=\"Link to this heading\">\u00b6<\/a><\/h2>\n<p>My gratitude goes to the authors of <code class=\"docutils literal notranslate\"><span class=\"pre\">pylitterbot<\/span><\/code> <span id=\"id10\">[<a class=\"reference internal\" href=\"#id18\" title=\"Nathan Spencer, Justin Bull, Ian Atha, &#64;jrhe&#64;github.com, Joost Lekkerkerker, Marc Mueller, Ville Skytt\u00e4, and Andreas Billmeier. Pylitterbot. Development ongoing. URL: https:\/\/github.com\/natekspencer\/pylitterbot (visited on 2026-02-05).\">SBA+<\/a>]<\/span>, without whom this work would have been much, <em>much<\/em> more difficult. Much thanks also goes to my friends <a class=\"reference external\" href=\"https:\/\/linktr.ee\/Ultralee0\">&#64;Ultralee0&#64;linktr.ee<\/a> and <a class=\"reference external\" href=\"https:\/\/indiedragon.com\">Ruby<\/a> for their thorough test reading of this series.<\/p>\n<p class=\"rubric\">Footnotes<\/p>\n<aside class=\"footnote-list brackets\">\n<aside class=\"footnote brackets\" id=\"id11\" role=\"doc-footnote\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span><a role=\"doc-backlink\" href=\"#id4\">1<\/a><span class=\"fn-bracket\">]<\/span><\/span>\n<p>Or sound, I suppose. I wasn\u2019t able to find that documented anywhere, but both would work. If it\u2019s sound, then that would be high frequency, which might bother cats. Therefore I assume it\u2019s light.<\/p>\n<\/aside>\n<\/aside>\n<p class=\"rubric\">Citations<\/p>\n<div class=\"docutils container\" id=\"id12\">\n<div role=\"list\" class=\"citation-list\">\n<div class=\"citation\" id=\"id124\" role=\"doc-biblioentry\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span><a role=\"doc-backlink\" href=\"#id5\">graa<\/a><span class=\"fn-bracket\">]<\/span><\/span>\n<p>Grafana gauge visualization. URL: <a class=\"reference external\" href=\"https:\/\/grafana.com\/docs\/grafana\/latest\/visualizations\/panels-visualizations\/visualizations\/guage\/\">https:\/\/grafana.com\/docs\/grafana\/latest\/visualizations\/panels-visualizations\/visualizations\/guage\/<\/a>.<\/p>\n<\/div>\n<div class=\"citation\" id=\"id116\" role=\"doc-biblioentry\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span><a role=\"doc-backlink\" href=\"#id8\">grab<\/a><span class=\"fn-bracket\">]<\/span><\/span>\n<p>Grafana logs visualization. URL: <a class=\"reference external\" href=\"https:\/\/grafana.com\/docs\/grafana\/latest\/visualizations\/panels-visualizations\/visualizations\/logs\/\">https:\/\/grafana.com\/docs\/grafana\/latest\/visualizations\/panels-visualizations\/visualizations\/logs\/<\/a> (visited on 2026-02-23).<\/p>\n<\/div>\n<div class=\"citation\" id=\"id119\" role=\"doc-biblioentry\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span><a role=\"doc-backlink\" href=\"#id6\">grac<\/a><span class=\"fn-bracket\">]<\/span><\/span>\n<p>Grafana stat visualization. URL: <a class=\"reference external\" href=\"https:\/\/grafana.com\/docs\/grafana\/latest\/visualizations\/panels-visualizations\/visualizations\/stat\/\">https:\/\/grafana.com\/docs\/grafana\/latest\/visualizations\/panels-visualizations\/visualizations\/stat\/<\/a> (visited on 2026-02-23).<\/p>\n<\/div>\n<div class=\"citation\" id=\"id15\" role=\"doc-biblioentry\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span>AC<span class=\"fn-bracket\">]<\/span><\/span>\n<span class=\"backrefs\">(<a role=\"doc-backlink\" href=\"#id7\">1<\/a>,<a role=\"doc-backlink\" href=\"#id9\">2<\/a>)<\/span>\n<p>Olivia Appleton-Crocker. Lessons in grafana - part one: a vision. URL: <a class=\"reference external\" href=\"https:\/\/blog.oliviaappleton.com\/posts\/0006-lessons-in-grafana-01\">https:\/\/blog.oliviaappleton.com\/posts\/0006-lessons-in-grafana-01<\/a>.<\/p>\n<\/div>\n<div class=\"citation\" id=\"id24\" role=\"doc-biblioentry\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span><a role=\"doc-backlink\" href=\"#id2\">Bur<\/a><span class=\"fn-bracket\">]<\/span><\/span>\n<p>Cat\u00a0Tax Bureau. What is cat tax? meaning, origin &amp; how to pay. URL: <a class=\"reference external\" href=\"https:\/\/cat-tax.net\/what-is-cat-tax.html\">https:\/\/cat-tax.net\/what-is-cat-tax.html<\/a> (visited on 2026-02-08).<\/p>\n<\/div>\n<div class=\"citation\" id=\"id77\" role=\"doc-biblioentry\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span><a role=\"doc-backlink\" href=\"#id3\">MRC<\/a><span class=\"fn-bracket\">]<\/span><\/span>\n<p>Dr. Lizzie Youens BSc (Hons)\u00a0BVSc MRCVS. Tortitude: definition, behavior &amp; how to deal with it. URL: <a class=\"reference external\" href=\"https:\/\/cats.com\/tortitude\">https:\/\/cats.com\/tortitude<\/a> (visited on 2026-02-15).<\/p>\n<\/div>\n<div class=\"citation\" id=\"id18\" role=\"doc-biblioentry\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span>SBA+<span class=\"fn-bracket\">]<\/span><\/span>\n<span class=\"backrefs\">(<a role=\"doc-backlink\" href=\"#id1\">1<\/a>,<a role=\"doc-backlink\" href=\"#id10\">2<\/a>)<\/span>\n<p>Nathan Spencer, Justin Bull, Ian Atha, &#64;jrhe&#64;github.com, Joost Lekkerkerker, Marc Mueller, Ville Skytt\u00e4, and Andreas Billmeier. Pylitterbot. Development ongoing. URL: <a class=\"reference external\" href=\"https:\/\/github.com\/natekspencer\/pylitterbot\">https:\/\/github.com\/natekspencer\/pylitterbot<\/a> (visited on 2026-02-05).<\/p>\n<\/div>\n<\/div>\n<\/div>\n<p class=\"rubric\">Cite As<\/p>\n<details><summary>Click here to expand the bibtex entry.<\/summary><div class=\"docutils container\">\n<blockquote>\n<div><div class=\"highlight-bibtex notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"nc\">@online<\/span><span class=\"p\">{<\/span><span class=\"nl\">appleton_blog_0007<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"2\"><span class=\"w\">  <\/span><span class=\"na\">title<\/span><span class=\"w\">    <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ Lessons in Grafana - Part Two: Litter Logs }<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"3\"><span class=\"w\">  <\/span><span class=\"na\">author<\/span><span class=\"w\">   <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ Olivia Appleton-Crocker }<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"4\"><span class=\"w\">  <\/span><span class=\"na\">editor<\/span><span class=\"w\">   <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ Ultralee0 and Ruby }<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"5\"><span class=\"w\">  <\/span><span class=\"na\">language<\/span><span class=\"w\"> <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ English }<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"6\"><span class=\"w\">  <\/span><span class=\"na\">version<\/span><span class=\"w\">  <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ 1 }<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"7\"><span class=\"w\">  <\/span><span class=\"na\">date<\/span><span class=\"w\">     <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ 2026-02-23 }<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"8\"><span class=\"w\">  <\/span><span class=\"na\">url<\/span><span class=\"w\">      <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ https:\/\/blog.oliviaappleton.com\/posts\/0007-lessons-in-grafana-02 }<\/span>\n<\/span><span data-line=\"9\"><span class=\"p\">}<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<\/div><\/blockquote>\n<\/div>\n<\/details><p class=\"tags\"><span>Tags: <\/span><a class=\"reference external\" href=\"..\/_tags\/grafana.html\">grafana<\/a><span>, <\/span><a class=\"reference external\" href=\"..\/_tags\/data-collection.html\">data-collection<\/a><span>, <\/span><a class=\"reference external\" href=\"..\/_tags\/data-visualization.html\">data-visualization<\/a><span>, <\/span><a class=\"reference external\" href=\"..\/_tags\/postgres.html\">postgres<\/a><span>, <\/span><a class=\"reference external\" href=\"..\/_tags\/postgresql.html\">postgresQL<\/a><span>, <\/span><a class=\"reference external\" href=\"..\/_tags\/python.html\">python<\/a><span>, <\/span><a class=\"reference external\" href=\"..\/_tags\/scraping.html\">scraping<\/a><span>, <\/span><a class=\"reference external\" href=\"..\/_tags\/databases.html\">databases<\/a><\/p>\n\n            <h2>\n                Comments\n                <span class=\"comments-info\">\n                    <img class=\"fedi-icon\" src=\"https:\/\/blog.oliviaappleton.com\/_static\/boost.svg\" alt=\"Boosts\">\n                    <span id=\"global-reblogs\"><\/span>,\n                    <img class=\"fedi-icon\" src=\"https:\/\/blog.oliviaappleton.com\/_static\/like.svg\" alt=\"Likes\">\n                    <span id=\"global-likes\"><\/span>\n                <\/span>\n            <\/h2>\n            <div id=\"comments-section\"><\/div>\n            <script>\n                document.addEventListener(\"DOMContentLoaded\", function () {\n                    setConfig({\n                        boostLink: \"https:\/\/blog.oliviaappleton.com\/_static\/boost.svg\",\n                        allowSensitiveEmoji: false,\n                        allowCustomEmoji: true,\n                        allowMediaAttachments: true,\n                        allowAvatars: true,\n                        delayCommentLoad: true,\n                        defaultReactionEmoji: '\u2764',\n                        retryDelay: 100,\n                    });\n                    fetchComments('misskey', 'transfem.social', 'aj33yr1lr9l605jo', 5);\n                });\n            <\/script>\n        <\/section>\n<\/section>\n","guid":"https:\/\/blog.oliviaappleton.com\/posts\/0007-lessons-in-grafana-02.html","category":["grafana","data-collection","data-visualization","postgres","postgresQL","python","scraping","databases"],"pubDate":"Mon, 23 Feb 2026 00:00:00 +0000"},{"title":"Lessons in Grafana - Part One: A Vision","link":"https:\/\/blog.oliviaappleton.com\/posts\/0006-lessons-in-grafana-01.html","description":"<a rel='me' href='https:\/\/transfem.social\/@LivInTheLookingGlass' role='none' class='invisible'><\/a>\n<script>\n    window.goatcounter = {\n        path: function() {\n            let p = location.pathname;\n            if (p == '\/') {\n                p = '\/index.html';\n            }\n            return '\/blog' + p;\n        }\n    }\n<\/script>\n<script data-goatcounter='https:\/\/livinthelookingglass.goatcounter.com\/count'\n async src='\/\/gc.zgo.at\/count.js'><\/script><section id=\"lessons-in-grafana-part-one-a-vision\">\n<h1>Lessons in Grafana - Part One: A Vision<a class=\"headerlink\" href=\"#lessons-in-grafana-part-one-a-vision\" title=\"Link to this heading\">\u00b6<\/a><\/h1>\n<p><span class=\"date\">2026-02-23<\/span><\/p>\n<p>A few months ago a friend showed me their data collection in Grafana, a visualization tool. I was deeply impressed! They seemed to collect everything that their self-hosted services were producing and had wonderful displays for them. This is something I had always wanted from a distance, but had never managed to start on. As it turned out, this is because I lacked a motivating use case. Recently, I found one.<\/p>\n<p>This series of blog posts is going to cover my journey through data collection. I will not be telling it in order. Instead, I will try to build up from the simplest task for the lay-person, adding complexity until we reach my works-in-progress. This will come in seven parts posted at one week intervals. As there are many tutorials on how to set up Grafana initially, I will not be including those details here except when discussing future plans. More posts might get added later on as the project develops.<\/p>\n<ul class=\"series-nav-max-3 series-nav simple\">\n<li><p><strong class=\"series-nav-label\">1)<\/strong><span> <\/span>A Vision (you are here)<\/p><\/li>\n<li><p><strong class=\"series-nav-label\">2)<\/strong><span> <\/span><a class=\"reference external\" href=\"\/posts\/0007-lessons-in-grafana-02\">Litter Logs<\/a><\/p><\/li>\n<li><p><strong class=\"series-nav-label\">2.5)<\/strong><span> <\/span><a class=\"reference external\" href=\"\/posts\/0008-lessons-in-grafana-02-5\">Achieving Acceptable Averages<\/a><\/p><\/li>\n<li><p><strong class=\"series-nav-label\">3)<\/strong><span> <\/span><a class=\"reference external\" href=\"\/posts\/0009-lessons-in-grafana-03\">Mental Metadata<\/a><\/p><\/li>\n<li><p><strong class=\"series-nav-label\">4)<\/strong><span> <\/span>Noticing Notes<\/p><\/li>\n<li><p><strong class=\"series-nav-label\">5)<\/strong><span> <\/span>Graphing Goatcounter<\/p><\/li>\n<li><p><strong class=\"series-nav-label\">6)<\/strong><span> <\/span>Bridging Gadgets? Biometrics Galore!<\/p><\/li>\n<li><p><strong class=\"series-nav-label\">7)<\/strong><span> <\/span>Plotting Prospective Plans<\/p><\/li>\n<\/ul>\n<p>As those who follow me know, I\u2019m currently between jobs. My previous position was grant-funded, and the grants ran dry. This means that I have been sending out applications daily, and I needed a way to track them more closely. Before this project, I was doing this in a very ad-hoc way that wasn\u2019t terribly effective. This entry will examine this use case and lay the foundations for future entries.<\/p>\n<section id=\"a-quick-review\">\n<h2>A Quick Review<a class=\"headerlink\" href=\"#a-quick-review\" title=\"Link to this heading\">\u00b6<\/a><\/h2>\n<p>In this series we are going to be looking in depth at data collection and visualization. This means we need to have some fundamentals on what that looks like.<\/p>\n<p>In general, data is formatted as tables. Think of spreadsheets, if that helps, as this entry <em>literally is<\/em> about spreadsheets. Column names are used to determine what data we select, and we can put conditions on what data gets included. You can, for instance, say \u201cI want all workout entries between these two dates\u201d.<\/p>\n<p>This information gets fed into Grafana, which has a number of visualization tools. The ones we will be using most prominently in this series are pie charts, time series, state timelines, and a single Sankey diagram. If you are comfortable with these, feel free to <a class=\"reference external\" href=\"#implementation-challenges\">skip ahead<\/a>.<\/p>\n<section id=\"pie-charts\">\n<h3>Pie Charts<a class=\"headerlink\" href=\"#pie-charts\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>A pie chart is what it says on the tin: it looks like a pie, with different slices being labeled to correspond to different entities. A typical pie chart will look like one of the following:<\/p>\n<figure class=\"align-default\" id=\"id126\">\n<img alt=\"A pie chart, as described in the caption below\" src=\"..\/_images\/Screenshot_20260202_225758.png\" \/>\n<figcaption>\n<p><span class=\"caption-text\">A sample pie chart visualization generated by Grafana\u2019s plugin pie chart plugin <span id=\"id1\">[<a class=\"reference internal\" href=\"#id116\" title=\"Grafana pie chart visualization. URL: https:\/\/grafana.com\/docs\/grafana\/latest\/visualizations\/panels-visualizations\/visualizations\/pie-chart\/ (visited on 2026-02-23).\">graa<\/a>]<\/span>. This shows two charts: a pie chart of sample data shown below, and a <em>donut<\/em> chart (a variation of the pie chart) of the same data.<\/span><a class=\"headerlink\" href=\"#id126\" title=\"Link to this image\">\u00b6<\/a><\/p>\n<\/figcaption>\n<\/figure>\n<p>Pie chart data is structured in two columns: a name and a value<a class=\"footnote-reference brackets\" href=\"#id8\" id=\"id2\" role=\"doc-noteref\"><span class=\"fn-bracket\">[<\/span>1<span class=\"fn-bracket\">]<\/span><\/a>. Such data might look like:<\/p>\n<div class=\"table-wrapper docutils container\">\n<table class=\"docutils align-default\">\n<thead>\n<tr class=\"row-odd\"><th class=\"head\"><p>Name<\/p><\/th>\n<th class=\"head\"><p>Units<\/p><\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr class=\"row-even\"><td><p>Lorem<\/p><\/td>\n<td><p>3<\/p><\/td>\n<\/tr>\n<tr class=\"row-odd\"><td><p>Ipsum<\/p><\/td>\n<td><p>2<\/p><\/td>\n<\/tr>\n<tr class=\"row-even\"><td><p>Dolor<\/p><\/td>\n<td><p>1<\/p><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/section>\n<section id=\"time-series\">\n<h3>Time Series<a class=\"headerlink\" href=\"#time-series\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>A time series shows numeric data over time. You see this kind of visualization a lot in day to day life. When you check the weather for the day, the expected temperature is often represented this way. Stock prices are also represented this way. This means that you will see these quite often in the news.<\/p>\n<figure class=\"align-default\" id=\"id127\">\n<img alt=\"A time series chart, as described in the caption below\" src=\"..\/_images\/Screenshot_20260202_230912.png\" \/>\n<figcaption>\n<p><span class=\"caption-text\">A sample time series visualization generated by Grafana\u2019s time series plugin <span id=\"id3\">[<a class=\"reference internal\" href=\"#id120\" title=\"Grafana time series visualization. URL: https:\/\/grafana.com\/docs\/grafana\/latest\/visualizations\/panels-visualizations\/visualizations\/time-series\/ (visited on 2026-02-23).\">grac<\/a>]<\/span>. This shows the sample data below, but with an interesting twist: one series is represented as bars, but the rest as lines. This demonstrates the ability of time series to take on multiple representation, using lines, bars, or dots. All are the same chart type, and all can appear in one chart.<\/span><a class=\"headerlink\" href=\"#id127\" title=\"Link to this image\">\u00b6<\/a><\/p>\n<\/figcaption>\n<\/figure>\n<p>Time series data is formatted in many columns. One <em>must be<\/em> a timestamp. All other columns are series to be graphed, with the column title as their label. Note that all values must be either numeric or empty.<\/p>\n<div class=\"table-wrapper docutils container\">\n<table class=\"docutils align-default\">\n<thead>\n<tr class=\"row-odd\"><th class=\"head\"><p>Time<\/p><\/th>\n<th class=\"head\"><p>Series A Label<\/p><\/th>\n<th class=\"head\"><p>Series B Label<\/p><\/th>\n<th class=\"head\"><p>Etc.<\/p><\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr class=\"row-even\"><td><p>2026-01-01 12:00:00+00:00<\/p><\/td>\n<td><p>8<\/p><\/td>\n<td><p>9<\/p><\/td>\n<td><p>4<\/p><\/td>\n<\/tr>\n<tr class=\"row-odd\"><td><p>2026-01-01 13:00:00+00:00<\/p><\/td>\n<td><\/td>\n<td><p>10<\/p><\/td>\n<td><p>2<\/p><\/td>\n<\/tr>\n<tr class=\"row-even\"><td><p>2026-01-01 14:00:00+00:00<\/p><\/td>\n<td><p>6<\/p><\/td>\n<td><\/td>\n<td><p>6<\/p><\/td>\n<\/tr>\n<tr class=\"row-odd\"><td><p>2026-01-01 15:00:00+00:00<\/p><\/td>\n<td><p>5<\/p><\/td>\n<td><p>12<\/p><\/td>\n<td><p>1<\/p><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/section>\n<section id=\"state-timelines\">\n<h3>State Timelines<a class=\"headerlink\" href=\"#state-timelines\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>A state timeline is a bit less common than the above. It shows discrete states over time. You might use this if you, for example, wanted to track whether someone is scheduled to work. Another common use case is to show the states of different machines on your network, for example if they are online, offline, booting, overloaded, etc.<\/p>\n<figure class=\"align-default\" id=\"id128\">\n<img alt=\"A state timeline, as described in the caption below\" src=\"..\/_images\/Screenshot_20260202_234111.png\" \/>\n<figcaption>\n<p><span class=\"caption-text\">A sample state timeline visualization generated by Grafana\u2019s state timeline plugin <span id=\"id4\">[<a class=\"reference internal\" href=\"#id119\" title=\"Grafana state timeline visualization. URL: https:\/\/grafana.com\/docs\/grafana\/latest\/visualizations\/panels-visualizations\/visualizations\/state-timeline\/ (visited on 2026-02-23).\">grab<\/a>]<\/span>. This shows data from the example below, looking at worker shifts in a hypothetical restaurant. Hovering over it reveals more information, such as the duration and specific state (if an interpreted value is shown rather than raw).<\/span><a class=\"headerlink\" href=\"#id128\" title=\"Link to this image\">\u00b6<\/a><\/p>\n<\/figcaption>\n<\/figure>\n<p>State timeline data is formatted much like time series data<a class=\"footnote-reference brackets\" href=\"#id9\" id=\"id5\" role=\"doc-noteref\"><span class=\"fn-bracket\">[<\/span>2<span class=\"fn-bracket\">]<\/span><\/a>, but rather than taking on <em>numeric<\/em> values, you can give states any label you would like. Here is an arbitrary example:<\/p>\n<div class=\"table-wrapper docutils container\">\n<table class=\"docutils align-default\">\n<thead>\n<tr class=\"row-odd\"><th class=\"head\"><p>Time<\/p><\/th>\n<th class=\"head\"><p>Worker on Shift<\/p><\/th>\n<th class=\"head\"><p>Current Task<\/p><\/th>\n<th class=\"head\"><p>Difficulty<\/p><\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr class=\"row-even\"><td><p>2026-01-01 12:00:00+00:00<\/p><\/td>\n<td><p>Alice<\/p><\/td>\n<td><p>Food Prep<\/p><\/td>\n<td><p>High<\/p><\/td>\n<\/tr>\n<tr class=\"row-odd\"><td><p>2026-01-01 13:00:00+00:00<\/p><\/td>\n<td><p>Alice<\/p><\/td>\n<td><p>Register<\/p><\/td>\n<td><p>Moderate<\/p><\/td>\n<\/tr>\n<tr class=\"row-even\"><td><p>2026-01-01 14:00:00+00:00<\/p><\/td>\n<td><p>Bob<\/p><\/td>\n<td><p>Dishes<\/p><\/td>\n<td><p>Low<\/p><\/td>\n<\/tr>\n<tr class=\"row-odd\"><td><p>2026-01-01 15:00:00+00:00<\/p><\/td>\n<td><\/td>\n<td><p>Closed<\/p><\/td>\n<td><p>N\/A<\/p><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/section>\n<section id=\"sankey-diagrams\">\n<h3>Sankey Diagrams<a class=\"headerlink\" href=\"#sankey-diagrams\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>Sankey diagrams are definitely the oddest chart format here. In general, a Sankey diagram is used to represent data flow. A common use case for this is finances: one might want to represent the income of two spouses and how it flows into different expenses.<\/p>\n<figure class=\"align-default\" id=\"id129\">\n<img alt=\"A sankey diagram, as described in the caption below\" src=\"..\/_images\/Screenshot_20260202_233252.png\" \/>\n<figcaption>\n<p><span class=\"caption-text\">A sample Sankey diagram visualization generated by Grafana\u2019s Sankey plugin <span id=\"id6\">[<a class=\"reference internal\" href=\"#id117\" title=\"Netsage sankey panel for grafana. URL: https:\/\/grafana.com\/grafana\/plugins\/netsage-sankey-panel\/ (visited on 2026-02-23).\">grad<\/a>]<\/span>. This shows the example we illustrate below, looking at the flow of a two-income couple.<\/span><a class=\"headerlink\" href=\"#id129\" title=\"Link to this image\">\u00b6<\/a><\/p>\n<\/figcaption>\n<\/figure>\n<p>Sankey data is typically represented in two or more columns. In the version specific to Grafana, these columns represent the <em>entire journey<\/em> followed by a value. Other Sankey visualizers support you defining a single source and destination, and it will dynamically figure out the overall journey. To illustrate an example, suppose Alice and Bob are married. They each split their income 50\/50, Alice makes $70k, and Bob makes $40k. Their finances might look like this:<\/p>\n<div class=\"table-wrapper docutils container\">\n<table class=\"docutils align-default\">\n<thead>\n<tr class=\"row-odd\"><th class=\"head\"><p>Start<\/p><\/th>\n<th class=\"head\"><p>Middle<\/p><\/th>\n<th class=\"head\"><p>End<\/p><\/th>\n<th class=\"head\"><p>Count<\/p><\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr class=\"row-even\"><td><p>Alice<\/p><\/td>\n<td><p>Joint Income<\/p><\/td>\n<td><p>Joint Expenses<\/p><\/td>\n<td><p>23500<\/p><\/td>\n<\/tr>\n<tr class=\"row-odd\"><td><p>Bob<\/p><\/td>\n<td><p>Joint Income<\/p><\/td>\n<td><p>Joint Expenses<\/p><\/td>\n<td><p>14500<\/p><\/td>\n<\/tr>\n<tr class=\"row-even\"><td><p>Alice<\/p><\/td>\n<td><p>Alice Income<\/p><\/td>\n<td><p>Alice Expenses<\/p><\/td>\n<td><p>5000<\/p><\/td>\n<\/tr>\n<tr class=\"row-odd\"><td><p>Bob<\/p><\/td>\n<td><p>Bob Income<\/p><\/td>\n<td><p>Indv. Expenses<\/p><\/td>\n<td><p>7500<\/p><\/td>\n<\/tr>\n<tr class=\"row-even\"><td><p>Alice<\/p><\/td>\n<td><p>Joint Income<\/p><\/td>\n<td><p>Joint Savings<\/p><\/td>\n<td><p>11500<\/p><\/td>\n<\/tr>\n<tr class=\"row-odd\"><td><p>Bob<\/p><\/td>\n<td><p>Joint Income<\/p><\/td>\n<td><p>Joint Savings<\/p><\/td>\n<td><p>6500<\/p><\/td>\n<\/tr>\n<tr class=\"row-even\"><td><p>Alice<\/p><\/td>\n<td><p>Alice Income<\/p><\/td>\n<td><p>Alice Savings<\/p><\/td>\n<td><p>30000<\/p><\/td>\n<\/tr>\n<tr class=\"row-odd\"><td><p>Bob<\/p><\/td>\n<td><p>Bob Income<\/p><\/td>\n<td><p>Bob Savings<\/p><\/td>\n<td><p>11500<\/p><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p>It\u2019s important to remember that you can use it to visualize any number of steps. Our example here shows three, but it can handle any number of transitions. You will always have one more column than the number of steps.<\/p>\n<\/section>\n<\/section>\n<section id=\"implementation-challenges\">\n<h2>Implementation Challenges<a class=\"headerlink\" href=\"#implementation-challenges\" title=\"Link to this heading\">\u00b6<\/a><\/h2>\n<p>Before this project, I was tracking job applications in a very ad-hoc way. It wasn\u2019t terribly effective. With the goal of making visualizations and helping to gamify the process, I began working on a tracking dashboard. I had several requirements for this use case:<\/p>\n<ol class=\"arabic simple\">\n<li><p>I want to minimize the friction of adding data. In other words, I should record the minimum information possible, and have as much of it automated as possible.<\/p><\/li>\n<li><p>I want to maximize the number of devices I can add that data from. I should be able to enter this from my phone, laptop, and desktop without worrying about synchronization.<\/p><\/li>\n<li><p>I want to be able to conveniently edit or backdate information. Sometimes things are recorded incorrectly. Whatever tool I am using should allow me to quickly fix such cases.<\/p><\/li>\n<\/ol>\n<p>This leads to the inevitable conclusion of an online spreadsheet. Next, we will talk about the challenges that arise from that data store, as well as alternatives and why they were rejected.<\/p>\n<section id=\"problems-inherent-to-spreadsheets\">\n<h3>Problems Inherent to Spreadsheets<a class=\"headerlink\" href=\"#problems-inherent-to-spreadsheets\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>There are a number of drawbacks inherent to this data format.<\/p>\n<section id=\"dimensionality\">\n<h4>Dimensionality<a class=\"headerlink\" href=\"#dimensionality\" title=\"Link to this heading\">\u00b6<\/a><\/h4>\n<p>Spreadsheets are inherently two-dimensional. This means that if you ever need to encode more than two dimensions in your data, you have to get very creative (ex: using multiple sheets). This is not a problem that exists in databases, where you can have an arbitrary number of dimensions by using tools like arrays, or easily-accessed secondary tables.<\/p>\n<\/section>\n<section id=\"lack-of-query-language\">\n<h4>Lack of Query Language<a class=\"headerlink\" href=\"#lack-of-query-language\" title=\"Link to this heading\">\u00b6<\/a><\/h4>\n<p>Because the data is entered into a spreadsheet, I have no way to do the sorts of powerful queries possible in a database. When querying a spreadsheet, you have exactly four options:<\/p>\n<ol class=\"arabic simple\">\n<li><p>Which set of sheets to look at<\/p><\/li>\n<li><p>Which sheet within that set to look at<\/p><\/li>\n<li><p>How long the data should be cached for<\/p><\/li>\n<li><p>Whether to apply a time filter to the <em>first column only<\/em>.<\/p><\/li>\n<\/ol>\n<p>Everything other than that has to be done with Grafana transformations, which have significant limitations when compared to database queries. They are significantly slower, which means you are limited in the number of them that can be reasonably used.<\/p>\n<\/section>\n<\/section>\n<section id=\"balancing-ease-of-entry-with-ease-of-processing\">\n<h3>Balancing Ease of Entry with Ease of Processing<a class=\"headerlink\" href=\"#balancing-ease-of-entry-with-ease-of-processing\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>In the end, this data is complex enough that I am stuck with two possibilities. I can write an application specific to this task that allows data entry on the fly, or I use a cloud-based spreadsheet. Let\u2019s look at some approaches this application could take, and why I have rejected them.<\/p>\n<section id=\"a-local-spreadsheet\">\n<h4>A Local Spreadsheet<a class=\"headerlink\" href=\"#a-local-spreadsheet\" title=\"Link to this heading\">\u00b6<\/a><\/h4>\n<p>The easiest alternative would be to have a local spreadsheet that tracks it. This would be hosted on my home computer, and possibly synced between other devices such as my laptops. I even have already-existing code that will scan a spreadsheet periodically and dump its contents into a database.<\/p>\n<p>This solution is not ideal, though, because it requires that I be at a computer. While applying for jobs on a mobile device isn\u2019t something I would prefer to do, utilizing this solution would close off that possibility entirely. That\u2019s not acceptable.<\/p>\n<\/section>\n<section id=\"entry-in-joplin\">\n<h4>Entry in Joplin<a class=\"headerlink\" href=\"#entry-in-joplin\" title=\"Link to this heading\">\u00b6<\/a><\/h4>\n<p>Given the constraint above, I could enter this data in Joplin (my note-taking application of choice). There are already easy ways for me to extract CSV (Comma Separated Values) formatted tables from there and dump them to a database. It syncs to mobile devices with their app, meaning I would maintain seamless access.<\/p>\n<p>The problem with this is that CSV is an awful format for humans to work with. It\u2019s wonderful for computers: very simple, easily portable, standardized. But that formatting that makes it easy for computers to parse also means that when working with it <em>as a person<\/em> it is <em>ugly<\/em> and <em>cumbersome<\/em>. As an example, this is what that format looks like for time tracking data:<\/p>\n<div class=\"highlight-default notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"n\">Project<\/span><span class=\"p\">,<\/span><span class=\"n\">Task<\/span><span class=\"p\">,<\/span><span class=\"n\">Start<\/span> <span class=\"n\">date<\/span><span class=\"p\">,<\/span><span class=\"n\">Start<\/span> <span class=\"n\">time<\/span><span class=\"p\">,<\/span><span class=\"n\">End<\/span> <span class=\"n\">date<\/span><span class=\"p\">,<\/span><span class=\"n\">End<\/span> <span class=\"n\">time<\/span><span class=\"p\">,<\/span><span class=\"n\">Duration<\/span>\n<\/span><span data-line=\"2\"><span class=\"n\">Transport<\/span><span class=\"p\">,<\/span><span class=\"n\">Driving<\/span><span class=\"p\">,<\/span><span class=\"mi\">2026<\/span><span class=\"o\">-<\/span><span class=\"mi\">01<\/span><span class=\"o\">-<\/span><span class=\"mi\">17<\/span><span class=\"p\">,<\/span><span class=\"mi\">18<\/span><span class=\"p\">:<\/span><span class=\"mi\">25<\/span><span class=\"p\">:<\/span><span class=\"mi\">46<\/span><span class=\"o\">-<\/span><span class=\"mi\">06<\/span><span class=\"p\">:<\/span><span class=\"mi\">00<\/span><span class=\"p\">,<\/span><span class=\"mi\">2026<\/span><span class=\"o\">-<\/span><span class=\"mi\">01<\/span><span class=\"o\">-<\/span><span class=\"mi\">17<\/span><span class=\"p\">,<\/span><span class=\"mi\">18<\/span><span class=\"p\">:<\/span><span class=\"mi\">59<\/span><span class=\"p\">:<\/span><span class=\"mi\">11<\/span><span class=\"o\">-<\/span><span class=\"mi\">06<\/span><span class=\"p\">:<\/span><span class=\"mi\">00<\/span><span class=\"p\">,<\/span><span class=\"mi\">00<\/span><span class=\"p\">:<\/span><span class=\"mi\">33<\/span><span class=\"p\">:<\/span><span class=\"mi\">25<\/span>\n<\/span><span data-line=\"3\"><span class=\"n\">Transport<\/span><span class=\"p\">,<\/span><span class=\"n\">Driving<\/span><span class=\"p\">,<\/span><span class=\"mi\">2026<\/span><span class=\"o\">-<\/span><span class=\"mi\">01<\/span><span class=\"o\">-<\/span><span class=\"mi\">17<\/span><span class=\"p\">,<\/span><span class=\"mi\">19<\/span><span class=\"p\">:<\/span><span class=\"mi\">51<\/span><span class=\"p\">:<\/span><span class=\"mi\">22<\/span><span class=\"o\">-<\/span><span class=\"mi\">06<\/span><span class=\"p\">:<\/span><span class=\"mi\">00<\/span><span class=\"p\">,<\/span><span class=\"mi\">2026<\/span><span class=\"o\">-<\/span><span class=\"mi\">01<\/span><span class=\"o\">-<\/span><span class=\"mi\">17<\/span><span class=\"p\">,<\/span><span class=\"mi\">20<\/span><span class=\"p\">:<\/span><span class=\"mi\">10<\/span><span class=\"p\">:<\/span><span class=\"mi\">54<\/span><span class=\"o\">-<\/span><span class=\"mi\">06<\/span><span class=\"p\">:<\/span><span class=\"mi\">00<\/span><span class=\"p\">,<\/span><span class=\"mi\">00<\/span><span class=\"p\">:<\/span><span class=\"mi\">19<\/span><span class=\"p\">:<\/span><span class=\"mi\">32<\/span>\n<\/span><span data-line=\"4\"><span class=\"n\">Career<\/span><span class=\"p\">,<\/span><span class=\"n\">Job<\/span> <span class=\"n\">Seeking<\/span><span class=\"p\">,<\/span><span class=\"mi\">2026<\/span><span class=\"o\">-<\/span><span class=\"mi\">01<\/span><span class=\"o\">-<\/span><span class=\"mi\">18<\/span><span class=\"p\">,<\/span><span class=\"mi\">14<\/span><span class=\"p\">:<\/span><span class=\"mi\">00<\/span><span class=\"p\">:<\/span><span class=\"mi\">21<\/span><span class=\"o\">-<\/span><span class=\"mi\">06<\/span><span class=\"p\">:<\/span><span class=\"mi\">00<\/span><span class=\"p\">,<\/span><span class=\"mi\">2026<\/span><span class=\"o\">-<\/span><span class=\"mi\">01<\/span><span class=\"o\">-<\/span><span class=\"mi\">18<\/span><span class=\"p\">,<\/span><span class=\"mi\">14<\/span><span class=\"p\">:<\/span><span class=\"mi\">25<\/span><span class=\"p\">:<\/span><span class=\"mi\">21<\/span><span class=\"o\">-<\/span><span class=\"mi\">06<\/span><span class=\"p\">:<\/span><span class=\"mi\">00<\/span><span class=\"p\">,<\/span><span class=\"mi\">00<\/span><span class=\"p\">:<\/span><span class=\"mi\">25<\/span><span class=\"p\">:<\/span><span class=\"mi\">00<\/span>\n<\/span><span data-line=\"5\"><span class=\"n\">Transport<\/span><span class=\"p\">,<\/span><span class=\"n\">Driving<\/span><span class=\"p\">,<\/span><span class=\"mi\">2026<\/span><span class=\"o\">-<\/span><span class=\"mi\">01<\/span><span class=\"o\">-<\/span><span class=\"mi\">19<\/span><span class=\"p\">,<\/span><span class=\"mi\">10<\/span><span class=\"p\">:<\/span><span class=\"mi\">39<\/span><span class=\"p\">:<\/span><span class=\"mi\">45<\/span><span class=\"o\">-<\/span><span class=\"mi\">06<\/span><span class=\"p\">:<\/span><span class=\"mi\">00<\/span><span class=\"p\">,<\/span><span class=\"mi\">2026<\/span><span class=\"o\">-<\/span><span class=\"mi\">01<\/span><span class=\"o\">-<\/span><span class=\"mi\">19<\/span><span class=\"p\">,<\/span><span class=\"mi\">11<\/span><span class=\"p\">:<\/span><span class=\"mi\">04<\/span><span class=\"p\">:<\/span><span class=\"mi\">14<\/span><span class=\"o\">-<\/span><span class=\"mi\">06<\/span><span class=\"p\">:<\/span><span class=\"mi\">00<\/span><span class=\"p\">,<\/span><span class=\"mi\">00<\/span><span class=\"p\">:<\/span><span class=\"mi\">24<\/span><span class=\"p\">:<\/span><span class=\"mi\">29<\/span>\n<\/span><span data-line=\"6\"><span class=\"n\">Fun<\/span><span class=\"p\">,<\/span><span class=\"n\">RPG<\/span><span class=\"p\">,<\/span><span class=\"mi\">2026<\/span><span class=\"o\">-<\/span><span class=\"mi\">01<\/span><span class=\"o\">-<\/span><span class=\"mi\">19<\/span><span class=\"p\">,<\/span><span class=\"mi\">12<\/span><span class=\"p\">:<\/span><span class=\"mi\">11<\/span><span class=\"p\">:<\/span><span class=\"mi\">37<\/span><span class=\"o\">-<\/span><span class=\"mi\">06<\/span><span class=\"p\">:<\/span><span class=\"mi\">00<\/span><span class=\"p\">,<\/span><span class=\"mi\">2026<\/span><span class=\"o\">-<\/span><span class=\"mi\">01<\/span><span class=\"o\">-<\/span><span class=\"mi\">19<\/span><span class=\"p\">,<\/span><span class=\"mi\">15<\/span><span class=\"p\">:<\/span><span class=\"mi\">05<\/span><span class=\"p\">:<\/span><span class=\"mi\">37<\/span><span class=\"o\">-<\/span><span class=\"mi\">06<\/span><span class=\"p\">:<\/span><span class=\"mi\">00<\/span><span class=\"p\">,<\/span><span class=\"mi\">02<\/span><span class=\"p\">:<\/span><span class=\"mi\">54<\/span><span class=\"p\">:<\/span><span class=\"mi\">00<\/span>\n<\/span><span data-line=\"7\"><span class=\"n\">Transport<\/span><span class=\"p\">,<\/span><span class=\"n\">Driving<\/span><span class=\"p\">,<\/span><span class=\"mi\">2026<\/span><span class=\"o\">-<\/span><span class=\"mi\">01<\/span><span class=\"o\">-<\/span><span class=\"mi\">19<\/span><span class=\"p\">,<\/span><span class=\"mi\">15<\/span><span class=\"p\">:<\/span><span class=\"mi\">19<\/span><span class=\"p\">:<\/span><span class=\"mi\">53<\/span><span class=\"o\">-<\/span><span class=\"mi\">06<\/span><span class=\"p\">:<\/span><span class=\"mi\">00<\/span><span class=\"p\">,<\/span><span class=\"mi\">2026<\/span><span class=\"o\">-<\/span><span class=\"mi\">01<\/span><span class=\"o\">-<\/span><span class=\"mi\">19<\/span><span class=\"p\">,<\/span><span class=\"mi\">15<\/span><span class=\"p\">:<\/span><span class=\"mi\">46<\/span><span class=\"p\">:<\/span><span class=\"mi\">17<\/span><span class=\"o\">-<\/span><span class=\"mi\">06<\/span><span class=\"p\">:<\/span><span class=\"mi\">00<\/span><span class=\"p\">,<\/span><span class=\"mi\">00<\/span><span class=\"p\">:<\/span><span class=\"mi\">26<\/span><span class=\"p\">:<\/span><span class=\"mi\">24<\/span>\n<\/span><span data-line=\"8\"><span class=\"n\">Career<\/span><span class=\"p\">,<\/span><span class=\"n\">Writing<\/span><span class=\"p\">,<\/span><span class=\"mi\">2026<\/span><span class=\"o\">-<\/span><span class=\"mi\">01<\/span><span class=\"o\">-<\/span><span class=\"mi\">19<\/span><span class=\"p\">,<\/span><span class=\"mi\">16<\/span><span class=\"p\">:<\/span><span class=\"mi\">11<\/span><span class=\"p\">:<\/span><span class=\"mi\">24<\/span><span class=\"o\">-<\/span><span class=\"mi\">06<\/span><span class=\"p\">:<\/span><span class=\"mi\">00<\/span><span class=\"p\">,,,<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<p>See how hard that would be to edit? And while there are Joplin plugins that make it easier to edit <em>Markdown<\/em> tables, those plugins do not work for CSV, nor would I expect them to. This solution dramatically increases friction at time of entry.<\/p>\n<\/section>\n<section id=\"extension-to-my-data-tracking-bot\">\n<h4>Extension to my Data Tracking Bot<a class=\"headerlink\" href=\"#extension-to-my-data-tracking-bot\" title=\"Link to this heading\">\u00b6<\/a><\/h4>\n<p>Lastly, I have an existing application that takes in commands from instant messaging and adds entries to a database. This does not suit this purpose because it doesn\u2019t allow me to <em>view<\/em> the data. It\u2019s often difficult to tell which particular application a response is intended for. This means it needs to be easy to view the thing you are editing. This is very cumbersome in an instant messaging context for nearly identical reasons to the above.<\/p>\n<p>Unfortunately, this means a cloud-based spreadsheet is my best solution here. It is possible to get the benefits of both worlds by scraping this data to my postgres database, but that method is somewhat inflexible if the data format changes. Since I don\u2019t know that this is final, I haven\u2019t done so yet. In any case, this would not solve the main drawback of a cloud service: <em>I have no control over where the data is<\/em>, and only limited control for who can read it. For sufficiently private information, storing it plain text in the cloud is a bad security practice. While this is not data that would be devastating to leak, I don\u2019t like having that attack surface in the first place. I try to avoid it whenever possible.<\/p>\n<\/section>\n<\/section>\n<\/section>\n<section id=\"job-application-data-format\">\n<h2>Job Application Data Format<a class=\"headerlink\" href=\"#job-application-data-format\" title=\"Link to this heading\">\u00b6<\/a><\/h2>\n<div class=\"table-wrapper docutils container\">\n<table class=\"docutils align-default\">\n<thead>\n<tr class=\"row-odd\"><th class=\"head\"><p>Company<\/p><\/th>\n<th class=\"head\"><p>Position<\/p><\/th>\n<th class=\"head\"><p>ID?<\/p><\/th>\n<th class=\"head\"><p>Applied<\/p><\/th>\n<th class=\"head\"><p>1st Int.<\/p><\/th>\n<th class=\"head\"><p>2nd Int.<\/p><\/th>\n<th class=\"head\"><p>3rd Int.<\/p><\/th>\n<th class=\"head\"><p>Offer<\/p><\/th>\n<th class=\"head\"><p>Rejected<\/p><\/th>\n<th class=\"head\"><p>Ghosted<\/p><\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr class=\"row-even\"><td><p>Wikimedia<\/p><\/td>\n<td><p>Minister of Coffee<\/p><\/td>\n<td><p>123<\/p><\/td>\n<td><p>Jan 2<\/p><\/td>\n<td><p>Jan 7<\/p><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<td><p>Jan 11<\/p><\/td>\n<td><\/td>\n<\/tr>\n<tr class=\"row-odd\"><td><p>Mapbox<\/p><\/td>\n<td><p>GPS Voice<\/p><\/td>\n<td><p>456<\/p><\/td>\n<td><p>Jan 3<\/p><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<td><p>Jan 17<\/p><\/td>\n<\/tr>\n<tr class=\"row-even\"><td><p>GitLab<\/p><\/td>\n<td><p>Amb. to Antarctica<\/p><\/td>\n<td><p>789<\/p><\/td>\n<td><p>Jan 10<\/p><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p>Each field has a specific purpose, and is put to use either for tracking or for visualizations. Most of these are manually entered, but the Ghosted column is a bit smarter. It\u2019s powered by a formula. If the newest cell leftward of it is more than two weeks old<a class=\"footnote-reference brackets\" href=\"#id10\" id=\"id7\" role=\"doc-noteref\"><span class=\"fn-bracket\">[<\/span>3<span class=\"fn-bracket\">]<\/span><\/a>, it automatically is marked as ghosted. That value is tunable, but 2 weeks is a useful value for most purposes.<\/p>\n<details><summary>If you're curious, click here for the formula<\/summary><div class=\"docutils container\">\n<p>Good to know you\u2019re reading these! There will be quite a few of them in this series.<\/p>\n<div class=\"highlight-default notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\">=MAP(\n<\/span><span data-line=\"2\">    D2:D, E2:E, F2:F, G2:G, H2:H, I2:I,\n<\/span><span data-line=\"3\">    LAMBDA(\n<\/span><span data-line=\"4\">        D, E, F, G, H, I,\n<\/span><span data-line=\"5\">        IF(\n<\/span><span data-line=\"6\">            AND(NOT(H), NOT(I), NOT(NOT(D)), MAX(D, E, F, G) + $M$1 &lt; NOW()),\n<\/span><span data-line=\"7\">            MAX(D, E, F, G) + $M$1,\n<\/span><span data-line=\"8\">            &quot;&quot;\n<\/span><span data-line=\"9\">)))\n<\/span><\/pre><\/div>\n<\/div>\n<\/div>\n<\/details><p>This information is then processed into several views for visualization purposes, which we\u2019ll show below.<\/p>\n<details><summary>The raw data for these charts can be seen by clicking here.<\/summary><div class=\"docutils container\">\n<div class=\"table-wrapper docutils container\">\n<table class=\"docutils align-default\">\n<thead>\n<tr class=\"row-odd\"><th class=\"head\"><p>Company<\/p><\/th>\n<th class=\"head\"><p>Position<\/p><\/th>\n<th class=\"head\"><p>ID?<\/p><\/th>\n<th class=\"head\"><p>Applied<\/p><\/th>\n<th class=\"head\"><p>1st Round<\/p><\/th>\n<th class=\"head\"><p>2nd Round<\/p><\/th>\n<th class=\"head\"><p>3rd Round<\/p><\/th>\n<th class=\"head\"><p>Offer<\/p><\/th>\n<th class=\"head\"><p>Rejected<\/p><\/th>\n<th class=\"head\"><p>Ghosted<\/p><\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr class=\"row-even\"><td><p>Grafana\u2019s Cousin Greg<\/p><\/td>\n<td><p>Staff Backend Engineer \u2013 Loki Ingest &amp; Vibes<\/p><\/td>\n<td><p>GREG-LOKI-001<\/p><\/td>\n<td><p>1\/6\/2026<\/p><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<td><p>1\/7\/2026<\/p><\/td>\n<td><\/td>\n<\/tr>\n<tr class=\"row-odd\"><td><p>Grafana\u2019s Cousin Greg<\/p><\/td>\n<td><p>Senior Backend Engineer \u2013 Observability but Chill<\/p><\/td>\n<td><p>GREG-OBS-002<\/p><\/td>\n<td><p>1\/6\/2026<\/p><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<td><p>1\/20\/2026<\/p><\/td>\n<\/tr>\n<tr class=\"row-even\"><td><p>Wikimedia-ish Foundation<\/p><\/td>\n<td><p>Software Engineer \u2013 Button Alignment Team<\/p><\/td>\n<td><p>WIKI-BTN-404<\/p><\/td>\n<td><p>1\/6\/2026<\/p><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<\/tr>\n<tr class=\"row-odd\"><td><p>Mapbox But Larger<\/p><\/td>\n<td><p>Software Engineer II \u2013 Maps of Maps<\/p><\/td>\n<td><p>MAP-MAP-001<\/p><\/td>\n<td><p>1\/6\/2026<\/p><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<td><p>1\/12\/2026<\/p><\/td>\n<td><\/td>\n<\/tr>\n<tr class=\"row-even\"><td><p>Grafana\u2019s Cousin Greg<\/p><\/td>\n<td><p>Principal Engineer \u2013 Metrics About Metrics<\/p><\/td>\n<td><p>GREG-META-777<\/p><\/td>\n<td><p>1\/7\/2026<\/p><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<td><p>1\/21\/2026<\/p><\/td>\n<\/tr>\n<tr class=\"row-odd\"><td><p>Wikimedia-ish Foundation<\/p><\/td>\n<td><p>Software Engineer \u2013 MediaWiki Interfaces (Again)<\/p><\/td>\n<td><p>WIKI-UI-405<\/p><\/td>\n<td><p>1\/7\/2026<\/p><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<\/tr>\n<tr class=\"row-even\"><td><p>Mapbox But Larger<\/p><\/td>\n<td><p>Software Engineer II \u2013 Search, But Faster<\/p><\/td>\n<td><p>MAP-SRCH-002<\/p><\/td>\n<td><p>1\/7\/2026<\/p><\/td>\n<td><p>1\/12\/2026<\/p><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<\/tr>\n<tr class=\"row-odd\"><td><p>Dropbox (Cardboard Edition)<\/p><\/td>\n<td><p>Backend Software Engineer \u2013 Box Logistics<\/p><\/td>\n<td><p>BOX-ENG-123<\/p><\/td>\n<td><p>1\/7\/2026<\/p><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<\/tr>\n<tr class=\"row-even\"><td><p>Vercel (Edge of the Edge)<\/p><\/td>\n<td><p>Software Engineer \u2013 Edge Compute (Edge of the Edge)<\/p><\/td>\n<td><p>EDGE-EDGE-01<\/p><\/td>\n<td><p>1\/7\/2026<\/p><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<\/tr>\n<tr class=\"row-odd\"><td><p>Wikimedia-ish Foundation<\/p><\/td>\n<td><p>Software Engineer \u2013 MediaWiki Interfaces (Please Notice Me)<\/p><\/td>\n<td><p>WIKI-UI-406<\/p><\/td>\n<td><p>1\/8\/2026<\/p><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<td><p>1\/12\/2026<\/p><\/td>\n<td><\/td>\n<\/tr>\n<tr class=\"row-even\"><td><p>Mapbox But Larger<\/p><\/td>\n<td><p>Software Engineer II \u2013 API (Same Job, New Req)<\/p><\/td>\n<td><p>MAP-API-003<\/p><\/td>\n<td><p>1\/8\/2026<\/p><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<\/tr>\n<tr class=\"row-odd\"><td><p>Figma.fig<\/p><\/td>\n<td><p>Software Engineer \u2013 Distributed Rectangles<\/p><\/td>\n<td><p>FIG-RECT-01<\/p><\/td>\n<td><p>1\/8\/2026<\/p><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<\/tr>\n<tr class=\"row-even\"><td><p>Netflix 2: Direct-to-Cables<\/p><\/td>\n<td><p>Distributed Systems Engineer \u2013 Buffering Prevention<\/p><\/td>\n<td><p>VHS-BUF-198<\/p><\/td>\n<td><p>1\/8\/2026<\/p><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<\/tr>\n<tr class=\"row-odd\"><td><p>Dropbox (Cardboard Edition)<\/p><\/td>\n<td><p>Product Backend Engineer \u2013 Search Platform (Boxes)<\/p><\/td>\n<td><p>BOX-SRCH-124<\/p><\/td>\n<td><p>1\/9\/2026<\/p><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<\/tr>\n<tr class=\"row-even\"><td><p>Figma.fig<\/p><\/td>\n<td><p>Software Engineer \u2013 Multiplayer Cursors<\/p><\/td>\n<td><p>FIG-CURSOR-02<\/p><\/td>\n<td><p>1\/9\/2026<\/p><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<\/tr>\n<tr class=\"row-odd\"><td><p>Netflix 2: Direct-to-Cables<\/p><\/td>\n<td><p>Senior Distributed Systems Engineer \u2013 Open Connect, Closed Feelings<\/p><\/td>\n<td><p>VHS-CDN-199<\/p><\/td>\n<td><p>1\/9\/2026<\/p><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<td><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n<\/details><section id=\"application-statuses-pie-chart\">\n<h3>Application Statuses Pie Chart<a class=\"headerlink\" href=\"#application-statuses-pie-chart\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>The most important chart to be able to view at a glance is a pie chart of current application statuses. With a brief look, you can tell what portion are at each stage of the process. Those statuses are the same as above:<\/p>\n<div class=\"table-wrapper docutils container\">\n<table class=\"docutils align-default\">\n<thead>\n<tr class=\"row-odd\"><th class=\"head\"><p>Status<\/p><\/th>\n<th class=\"head\"><p>Count<\/p><\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr class=\"row-even\"><td><p>Applied<\/p><\/td>\n<td><p>13<\/p><\/td>\n<\/tr>\n<tr class=\"row-odd\"><td><p>1st Round<\/p><\/td>\n<td><p>0<\/p><\/td>\n<\/tr>\n<tr class=\"row-even\"><td><p>2nd Round<\/p><\/td>\n<td><p>0<\/p><\/td>\n<\/tr>\n<tr class=\"row-odd\"><td><p>3rd Round<\/p><\/td>\n<td><p>0<\/p><\/td>\n<\/tr>\n<tr class=\"row-even\"><td><p>Offer<\/p><\/td>\n<td><p>0<\/p><\/td>\n<\/tr>\n<tr class=\"row-odd\"><td><p>Rejected<\/p><\/td>\n<td><p>3<\/p><\/td>\n<\/tr>\n<tr class=\"row-even\"><td><p>Ghosted<\/p><\/td>\n<td><p>2<\/p><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p>This data is then rendered into a pie chart, as seen below.<\/p>\n<figure class=\"align-default\">\n<img alt=\"A pie chart that renders the table above, showing 13 jobs applied, 3 jobs rejected, and 2 jobs ghosted\" src=\"..\/_images\/Screenshot_20260119_170016.png\" \/>\n<\/figure>\n<\/section>\n<section id=\"application-statuses-time-series\">\n<h3>Application Statuses Time Series<a class=\"headerlink\" href=\"#application-statuses-time-series\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>For a more detailed view of the above, you can also break that down into a time series. This view shows the number of applications that entered that status on a given day. For example, if you have an application to Dropbox that goes through two rounds of interviews before being rejected, you would have an entry corresponding to each of those transitions. First there would be one counted towards Applied, then towards 1st Round, then 2nd, then Rejected. The last column shows the goal of how many you intend to apply to that day (on average).<\/p>\n<div class=\"table-wrapper docutils container\">\n<table class=\"docutils align-default\">\n<thead>\n<tr class=\"row-odd\"><th class=\"head\"><p>Date<\/p><\/th>\n<th class=\"head\"><p>Applied<\/p><\/th>\n<th class=\"head\"><p>1st Round<\/p><\/th>\n<th class=\"head\"><p>2nd Round<\/p><\/th>\n<th class=\"head\"><p>3rd Round<\/p><\/th>\n<th class=\"head\"><p>Offer<\/p><\/th>\n<th class=\"head\"><p>Rejected<\/p><\/th>\n<th class=\"head\"><p>Ghosted<\/p><\/th>\n<th class=\"head\"><p>Applied Goal<\/p><\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr class=\"row-even\"><td><p>1\/6\/2026<\/p><\/td>\n<td><p>4<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>1<\/p><\/td>\n<\/tr>\n<tr class=\"row-odd\"><td><p>1\/7\/2026<\/p><\/td>\n<td><p>5<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>1<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>1<\/p><\/td>\n<\/tr>\n<tr class=\"row-even\"><td><p>1\/8\/2026<\/p><\/td>\n<td><p>4<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>1<\/p><\/td>\n<\/tr>\n<tr class=\"row-odd\"><td><p>1\/9\/2026<\/p><\/td>\n<td><p>3<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>1<\/p><\/td>\n<\/tr>\n<tr class=\"row-even\"><td><p>\u2026<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>1<\/p><\/td>\n<\/tr>\n<tr class=\"row-odd\"><td><p>1\/12\/2026<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>1<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>2<\/p><\/td>\n<td><p>1<\/p><\/td>\n<\/tr>\n<tr class=\"row-even\"><td><p>\u2026<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>1<\/p><\/td>\n<\/tr>\n<tr class=\"row-odd\"><td><p>1\/20\/2026<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>1<\/p><\/td>\n<\/tr>\n<tr class=\"row-even\"><td><p>1\/21\/2026<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>0<\/p><\/td>\n<td><p>1<\/p><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p>This gets rendered into the following lovely chart:<\/p>\n<figure class=\"align-default\" id=\"id130\">\n<img alt=\"A time series that renders the table above, as described below.\" src=\"..\/_images\/Screenshot_20260119_170150.png\" \/>\n<figcaption>\n<p><span class=\"caption-text\">A time series that renders the table above. All of the initial applications appear in the first four days. There is one rejection on the second day, and two more on the 12th of January. The applied goal remains consistent at 1. The average applied steadily diminishes to asymptotically approach it.<\/span><a class=\"headerlink\" href=\"#id130\" title=\"Link to this image\">\u00b6<\/a><\/p>\n<\/figcaption>\n<\/figure>\n<p>A second version of this chart is also available, showing the <em>cumulative<\/em> status changes. That is to say, it will show the number of applications that have <em>ever been<\/em> in that status.<\/p>\n<figure class=\"align-default\" id=\"id131\">\n<img alt=\"A time series that renders the table above, as described below.\" src=\"..\/_images\/Screenshot_20260119_170133.png\" \/>\n<figcaption>\n<p><span class=\"caption-text\">A time series that renders the table above, except cumulatively. Each of the points above are rendered instead as permanent increases.<\/span><a class=\"headerlink\" href=\"#id131\" title=\"Link to this image\">\u00b6<\/a><\/p>\n<\/figcaption>\n<\/figure>\n<\/section>\n<section id=\"application-status-journeys\">\n<h3>Application Status Journeys<a class=\"headerlink\" href=\"#application-status-journeys\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>This chart is the most detailed tracker of application statuses. Its goal is to show the journey that all applications took. As a reminder, Sankey diagrams chart out the flow of items through multiple stages. In this case, that means that we are showing the different states that an application can be in, and where they ultimately end up. Given the current hiring market, the most populous entry will likely be that last one of \u201cApplied -&gt; Ghosted\u201d. At the end of this process, this is the chart I am most excited to share. It will demonstrate in a fairly precise way just how much effort goes into job seeking.<\/p>\n<div class=\"table-wrapper docutils container\">\n<table class=\"docutils align-default\">\n<thead>\n<tr class=\"row-odd\"><th class=\"head\"><p>Start<\/p><\/th>\n<th class=\"head\"><p>Stage 2<\/p><\/th>\n<th class=\"head\"><p>Stage 3<\/p><\/th>\n<th class=\"head\"><p>Stage 4<\/p><\/th>\n<th class=\"head\"><p>Final<\/p><\/th>\n<th class=\"head\"><p>Count<\/p><\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr class=\"row-even\"><td><p>Applied<\/p><\/td>\n<td><p>Round One<\/p><\/td>\n<td><p>Round Two<\/p><\/td>\n<td><p>Round Three<\/p><\/td>\n<td><p>Waiting<\/p><\/td>\n<td><p>0<\/p><\/td>\n<\/tr>\n<tr class=\"row-odd\"><td><p>Applied<\/p><\/td>\n<td><p>Round One<\/p><\/td>\n<td><p>Round Two<\/p><\/td>\n<td><p>Round Three<\/p><\/td>\n<td><p>Offer<\/p><\/td>\n<td><p>0<\/p><\/td>\n<\/tr>\n<tr class=\"row-even\"><td><p>Applied<\/p><\/td>\n<td><p>Round One<\/p><\/td>\n<td><p>Round Two<\/p><\/td>\n<td><p>Round Three<\/p><\/td>\n<td><p>Rejected<\/p><\/td>\n<td><p>0<\/p><\/td>\n<\/tr>\n<tr class=\"row-odd\"><td><p>Applied<\/p><\/td>\n<td><p>Round One<\/p><\/td>\n<td><p>Round Two<\/p><\/td>\n<td><p>Round Three<\/p><\/td>\n<td><p>Ghosted<\/p><\/td>\n<td><p>0<\/p><\/td>\n<\/tr>\n<tr class=\"row-even\"><td><p>Applied<\/p><\/td>\n<td><p>Round One<\/p><\/td>\n<td><p>Round Two<\/p><\/td>\n<td><p>Waiting<\/p><\/td>\n<td><p>Waiting<\/p><\/td>\n<td><p>0<\/p><\/td>\n<\/tr>\n<tr class=\"row-odd\"><td><p>Applied<\/p><\/td>\n<td><p>Round One<\/p><\/td>\n<td><p>Round Two<\/p><\/td>\n<td><p>Offer<\/p><\/td>\n<td><p>Offer<\/p><\/td>\n<td><p>0<\/p><\/td>\n<\/tr>\n<tr class=\"row-even\"><td><p>Applied<\/p><\/td>\n<td><p>Round One<\/p><\/td>\n<td><p>Round Two<\/p><\/td>\n<td><p>Rejected<\/p><\/td>\n<td><p>Rejected<\/p><\/td>\n<td><p>0<\/p><\/td>\n<\/tr>\n<tr class=\"row-odd\"><td><p>Applied<\/p><\/td>\n<td><p>Round One<\/p><\/td>\n<td><p>Round Two<\/p><\/td>\n<td><p>Ghosted<\/p><\/td>\n<td><p>Ghosted<\/p><\/td>\n<td><p>0<\/p><\/td>\n<\/tr>\n<tr class=\"row-even\"><td><p>Applied<\/p><\/td>\n<td><p>Round One<\/p><\/td>\n<td><p>Waiting<\/p><\/td>\n<td><p>Waiting<\/p><\/td>\n<td><p>Waiting<\/p><\/td>\n<td><p>0<\/p><\/td>\n<\/tr>\n<tr class=\"row-odd\"><td><p>Applied<\/p><\/td>\n<td><p>Round One<\/p><\/td>\n<td><p>Offer<\/p><\/td>\n<td><p>Offer<\/p><\/td>\n<td><p>Offer<\/p><\/td>\n<td><p>0<\/p><\/td>\n<\/tr>\n<tr class=\"row-even\"><td><p>Applied<\/p><\/td>\n<td><p>Round One<\/p><\/td>\n<td><p>Rejected<\/p><\/td>\n<td><p>Rejected<\/p><\/td>\n<td><p>Rejected<\/p><\/td>\n<td><p>0<\/p><\/td>\n<\/tr>\n<tr class=\"row-odd\"><td><p>Applied<\/p><\/td>\n<td><p>Round One<\/p><\/td>\n<td><p>Ghosted<\/p><\/td>\n<td><p>Ghosted<\/p><\/td>\n<td><p>Ghosted<\/p><\/td>\n<td><p>0<\/p><\/td>\n<\/tr>\n<tr class=\"row-even\"><td><p>Applied<\/p><\/td>\n<td><p>Waiting<\/p><\/td>\n<td><p>Waiting<\/p><\/td>\n<td><p>Waiting<\/p><\/td>\n<td><p>Waiting<\/p><\/td>\n<td><p>13<\/p><\/td>\n<\/tr>\n<tr class=\"row-odd\"><td><p>Applied<\/p><\/td>\n<td><p>Offer<\/p><\/td>\n<td><p>Offer<\/p><\/td>\n<td><p>Offer<\/p><\/td>\n<td><p>Offer<\/p><\/td>\n<td><p>0<\/p><\/td>\n<\/tr>\n<tr class=\"row-even\"><td><p>Applied<\/p><\/td>\n<td><p>Rejected<\/p><\/td>\n<td><p>Rejected<\/p><\/td>\n<td><p>Rejected<\/p><\/td>\n<td><p>Rejected<\/p><\/td>\n<td><p>3<\/p><\/td>\n<\/tr>\n<tr class=\"row-odd\"><td><p>Applied<\/p><\/td>\n<td><p>Ghosted<\/p><\/td>\n<td><p>Ghosted<\/p><\/td>\n<td><p>Ghosted<\/p><\/td>\n<td><p>Ghosted<\/p><\/td>\n<td><p>2<\/p><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p>This table seems pretty long, doesn\u2019t it? Well, that\u2019s because the Sankey plugin for Grafana is non-standard. A more traditional Sankey parser would only have three columns, and therefore be a fair bit more concise. Still, we make due with what we have, and it makes a beautiful flow diagram.<\/p>\n<figure class=\"align-default\">\n<img alt=\"This Sankey diagram shows the job application journeys from the above table. There are five total stages, of which only 2 are utilized in this example. It shows 13 applications that flow from Applied to Waiting, another 3 that flow from Applied to Rejected, and 2 more that flow from Applied to Ghosted.\" src=\"..\/_images\/Screenshot_20260119_170208.png\" \/>\n<\/figure>\n<\/section>\n<section id=\"applications-by-company-pie-chart\">\n<h3>Applications by Company Pie Chart<a class=\"headerlink\" href=\"#applications-by-company-pie-chart\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>This last chart is showing the number of applications sent to each company. This is less interesting to me than the others, but it is a good way to keep yourself cautious. You don\u2019t want to over-invest in one place, as that\u2019s taking on a great deal of risk. Having this chart displayed prevents you from putting all your eggs in one basket.<\/p>\n<div class=\"table-wrapper docutils container\">\n<table class=\"docutils align-default\">\n<thead>\n<tr class=\"row-odd\"><th class=\"head\"><p>Company<\/p><\/th>\n<th class=\"head\"><p>Applied<\/p><\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr class=\"row-even\"><td><p>Dropbox (Cardboard Edition)<\/p><\/td>\n<td><p>3<\/p><\/td>\n<\/tr>\n<tr class=\"row-odd\"><td><p>Figma.fig<\/p><\/td>\n<td><p>2<\/p><\/td>\n<\/tr>\n<tr class=\"row-even\"><td><p>Grafana\u2019s Cousin Greg<\/p><\/td>\n<td><p>3<\/p><\/td>\n<\/tr>\n<tr class=\"row-odd\"><td><p>Mapbox But Larger<\/p><\/td>\n<td><p>3<\/p><\/td>\n<\/tr>\n<tr class=\"row-even\"><td><p>Netflix 2: Direct-to-Cables<\/p><\/td>\n<td><p>2<\/p><\/td>\n<\/tr>\n<tr class=\"row-odd\"><td><p>Vercel (Edge of the Edge)<\/p><\/td>\n<td><p>2<\/p><\/td>\n<\/tr>\n<tr class=\"row-even\"><td><p>Wikimedia-ish Foundation<\/p><\/td>\n<td><p>3<\/p><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p>This table becomes:<\/p>\n<figure class=\"align-default\">\n<img alt=\"A pie chart that renders the table above, 3 applications each to 3 of the companies, and 2 applications to each of the other three\" src=\"..\/_images\/Screenshot_20260119_170040.png\" \/>\n<\/figure>\n<\/section>\n<section id=\"all-together\">\n<h3>All Together<a class=\"headerlink\" href=\"#all-together\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<figure class=\"align-default\" id=\"id132\">\n<img alt=\"A display of the whole dashboard, as described below\" src=\"..\/_images\/all_together.png\" \/>\n<figcaption>\n<p><span class=\"caption-text\">This shows the whole dashboard, giving all five of the panels described here. On the left side are pie charts, first the current application statuses, then below that is applications by company. On the right we have the time series and sankey diagrams. The top shows cumulative status over time. Below that is statuses per day. Below that is the application journey chart.<\/span><a class=\"headerlink\" href=\"#id132\" title=\"Link to this image\">\u00b6<\/a><\/p>\n<\/figcaption>\n<\/figure>\n<\/section>\n<\/section>\n<section id=\"future-steps-for-this-grouping\">\n<h2>Future Steps for This Grouping<a class=\"headerlink\" href=\"#future-steps-for-this-grouping\" title=\"Link to this heading\">\u00b6<\/a><\/h2>\n<section id=\"better-filtering\">\n<h3>Better Filtering<a class=\"headerlink\" href=\"#better-filtering\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>Right now, only the time series charts support filtering by time span in any real capacity. In order to fix this, I would likely need to do most of the processing via Grafana transformations rather than processing in the spreadsheet itself. That\u2019s a significant barrier to entry, as transformations are notoriously difficult to debug when they go wrong. Writing spreadsheet formulas is significantly more flexible and allows for more powerful changes. One way to address this (for ex: with the pie charts) would be to add an additional dimension to the tables to encode time. That is a solution that will not work for the Sankey diagram, meaning that one would <em>certainly<\/em> need to be done via transformations.<\/p>\n<p>I do plan to address this, but as it isn\u2019t a critical need for me yet, it will probably take a good deal of time before I do so.<\/p>\n<\/section>\n<section id=\"time-tracker-integration\">\n<h3>Time Tracker Integration<a class=\"headerlink\" href=\"#time-tracker-integration\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>In future posts I will be describing a time tracker setup that I have been working on. Right now one of the entries tracks job seeking explicitly. I would very much like to display that, but have not yet gotten around to doing so. This would allow me to extract a number of statistics in addition to total time. Some examples include:<\/p>\n<ul class=\"simple\">\n<li><p>Average time per day<\/p><\/li>\n<li><p>Average time per application<\/p><\/li>\n<li><p>Average time per application at a given company (with some error)<\/p><\/li>\n<\/ul>\n<p>These are all valuable insights that would be nice to have. My time tracker view is not yet produced in a standard way, which I would want to do before tackling this. It is especially difficult, as it would involve mixing data sources together (in this case Google Sheets and PostgresQL).<\/p>\n<\/section>\n<\/section>\n<section id=\"main-takeaways\">\n<h2>Main Takeaways<a class=\"headerlink\" href=\"#main-takeaways\" title=\"Link to this heading\">\u00b6<\/a><\/h2>\n<p>The thing I learned with this dashboard specifically is how to maximize ease of entry. Job seeking is a painful process, and minimizing friction at every step of data entry is very important. That framing forced me to use data sources I would normally shy away from, and as a consequence I implemented it in a very different way to the ones you will see in the next entry: <a class=\"reference external\" href=\"https:\/\/blog.oliviaappleton.com\/posts\/0007-lessons-in-grafana-02#prerequisites\"><em>Litter Logs<\/em><\/a>.<\/p>\n<\/section>\n<section id=\"acknowledgments\">\n<h2>Acknowledgments<a class=\"headerlink\" href=\"#acknowledgments\" title=\"Link to this heading\">\u00b6<\/a><\/h2>\n<p>Thank you to my friends <a class=\"reference external\" href=\"https:\/\/linktr.ee\/Ultralee0\">&#64;Ultralee0&#64;linktr.ee<\/a> and <a class=\"reference external\" href=\"https:\/\/indiedragon.com\">Ruby<\/a> for doing extensive test reading of this series.<\/p>\n<p class=\"rubric\">Footnotes<\/p>\n<aside class=\"footnote-list brackets\">\n<aside class=\"footnote brackets\" id=\"id8\" role=\"doc-footnote\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span><a role=\"doc-backlink\" href=\"#id2\">1<\/a><span class=\"fn-bracket\">]<\/span><\/span>\n<p>Note that in this data actually gets transposed under the hood. Grafana expects two <em>rows<\/em> of data formatted like that, but it is much more cleanly represented as two <em>columns<\/em> for the purposes of these posts.<\/p>\n<\/aside>\n<aside class=\"footnote brackets\" id=\"id9\" role=\"doc-footnote\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span><a role=\"doc-backlink\" href=\"#id5\">2<\/a><span class=\"fn-bracket\">]<\/span><\/span>\n<p>Note that in state diagrams you can optionally give an end time to each state, but that\u2019s not necessary for most use cases.<\/p>\n<\/aside>\n<aside class=\"footnote brackets\" id=\"id10\" role=\"doc-footnote\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span><a role=\"doc-backlink\" href=\"#id7\">3<\/a><span class=\"fn-bracket\">]<\/span><\/span>\n<p>This is configurable<\/p>\n<\/aside>\n<\/aside>\n<p class=\"rubric\">Citations<\/p>\n<div class=\"docutils container\" id=\"id11\">\n<div role=\"list\" class=\"citation-list\">\n<div class=\"citation\" id=\"id116\" role=\"doc-biblioentry\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span><a role=\"doc-backlink\" href=\"#id1\">graa<\/a><span class=\"fn-bracket\">]<\/span><\/span>\n<p>Grafana pie chart visualization. URL: <a class=\"reference external\" href=\"https:\/\/grafana.com\/docs\/grafana\/latest\/visualizations\/panels-visualizations\/visualizations\/pie-chart\/\">https:\/\/grafana.com\/docs\/grafana\/latest\/visualizations\/panels-visualizations\/visualizations\/pie-chart\/<\/a> (visited on 2026-02-23).<\/p>\n<\/div>\n<div class=\"citation\" id=\"id119\" role=\"doc-biblioentry\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span><a role=\"doc-backlink\" href=\"#id4\">grab<\/a><span class=\"fn-bracket\">]<\/span><\/span>\n<p>Grafana state timeline visualization. URL: <a class=\"reference external\" href=\"https:\/\/grafana.com\/docs\/grafana\/latest\/visualizations\/panels-visualizations\/visualizations\/state-timeline\/\">https:\/\/grafana.com\/docs\/grafana\/latest\/visualizations\/panels-visualizations\/visualizations\/state-timeline\/<\/a> (visited on 2026-02-23).<\/p>\n<\/div>\n<div class=\"citation\" id=\"id120\" role=\"doc-biblioentry\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span><a role=\"doc-backlink\" href=\"#id3\">grac<\/a><span class=\"fn-bracket\">]<\/span><\/span>\n<p>Grafana time series visualization. URL: <a class=\"reference external\" href=\"https:\/\/grafana.com\/docs\/grafana\/latest\/visualizations\/panels-visualizations\/visualizations\/time-series\/\">https:\/\/grafana.com\/docs\/grafana\/latest\/visualizations\/panels-visualizations\/visualizations\/time-series\/<\/a> (visited on 2026-02-23).<\/p>\n<\/div>\n<div class=\"citation\" id=\"id117\" role=\"doc-biblioentry\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span><a role=\"doc-backlink\" href=\"#id6\">grad<\/a><span class=\"fn-bracket\">]<\/span><\/span>\n<p>Netsage sankey panel for grafana. URL: <a class=\"reference external\" href=\"https:\/\/grafana.com\/grafana\/plugins\/netsage-sankey-panel\/\">https:\/\/grafana.com\/grafana\/plugins\/netsage-sankey-panel\/<\/a> (visited on 2026-02-23).<\/p>\n<\/div>\n<\/div>\n<\/div>\n<p class=\"rubric\">Cite As<\/p>\n<details><summary>Click here to expand the bibtex entry.<\/summary><div class=\"docutils container\">\n<blockquote>\n<div><div class=\"highlight-bibtex notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"nc\">@online<\/span><span class=\"p\">{<\/span><span class=\"nl\">appleton_blog_0006<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"2\"><span class=\"w\">  <\/span><span class=\"na\">title<\/span><span class=\"w\">    <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ Lessons in Grafana - Part One: A Vision }<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"3\"><span class=\"w\">  <\/span><span class=\"na\">author<\/span><span class=\"w\">   <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ Olivia Appleton-Crocker }<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"4\"><span class=\"w\">  <\/span><span class=\"na\">editor<\/span><span class=\"w\">   <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ Ultralee0 and Ruby }<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"5\"><span class=\"w\">  <\/span><span class=\"na\">language<\/span><span class=\"w\"> <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ English }<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"6\"><span class=\"w\">  <\/span><span class=\"na\">version<\/span><span class=\"w\">  <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ 1 }<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"7\"><span class=\"w\">  <\/span><span class=\"na\">date<\/span><span class=\"w\">     <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ 2026-02-23 }<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"8\"><span class=\"w\">  <\/span><span class=\"na\">url<\/span><span class=\"w\">      <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ https:\/\/blog.oliviaappleton.com\/posts\/0006-lessons-in-grafana-01 }<\/span>\n<\/span><span data-line=\"9\"><span class=\"p\">}<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<\/div><\/blockquote>\n<\/div>\n<\/details><p class=\"tags\"><span>Tags: <\/span><a class=\"reference external\" href=\"..\/_tags\/grafana.html\">grafana<\/a><span>, <\/span><a class=\"reference external\" href=\"..\/_tags\/data-collection.html\">data-collection<\/a><span>, <\/span><a class=\"reference external\" href=\"..\/_tags\/data-visualization.html\">data-visualization<\/a><span>, <\/span><a class=\"reference external\" href=\"..\/_tags\/spreadsheets.html\">spreadsheets<\/a><\/p>\n\n            <h2>\n                Comments\n                <span class=\"comments-info\">\n                    <img class=\"fedi-icon\" src=\"https:\/\/blog.oliviaappleton.com\/_static\/boost.svg\" alt=\"Boosts\">\n                    <span id=\"global-reblogs\"><\/span>,\n                    <img class=\"fedi-icon\" src=\"https:\/\/blog.oliviaappleton.com\/_static\/like.svg\" alt=\"Likes\">\n                    <span id=\"global-likes\"><\/span>\n                <\/span>\n            <\/h2>\n            <div id=\"comments-section\"><\/div>\n            <script>\n                document.addEventListener(\"DOMContentLoaded\", function () {\n                    setConfig({\n                        boostLink: \"https:\/\/blog.oliviaappleton.com\/_static\/boost.svg\",\n                        allowSensitiveEmoji: false,\n                        allowCustomEmoji: true,\n                        allowMediaAttachments: true,\n                        allowAvatars: true,\n                        delayCommentLoad: true,\n                        defaultReactionEmoji: '\u2764',\n                        retryDelay: 100,\n                    });\n                    fetchComments('misskey', 'transfem.social', 'aj33wf7l9elr0xk6', 5);\n                });\n            <\/script>\n        <\/section>\n<\/section>\n","guid":"https:\/\/blog.oliviaappleton.com\/posts\/0006-lessons-in-grafana-01.html","category":["grafana","data-collection","data-visualization","spreadsheets"],"pubDate":"Mon, 23 Feb 2026 00:00:00 +0000"},{"title":"Developing <code class=\"docutils literal notranslate\"><span class=\"pre\">sphinx-fediverse<\/span><\/code>","link":"https:\/\/blog.oliviaappleton.com\/posts\/0005-sphinx-fediverse.html","description":"<a rel='me' href='https:\/\/transfem.social\/@LivInTheLookingGlass' role='none' class='invisible'><\/a>\n<script>\n    window.goatcounter = {\n        path: function() {\n            let p = location.pathname;\n            if (p == '\/') {\n                p = '\/index.html';\n            }\n            return '\/blog' + p;\n        }\n    }\n<\/script>\n<script data-goatcounter='https:\/\/livinthelookingglass.goatcounter.com\/count'\n async src='\/\/gc.zgo.at\/count.js'><\/script><section id=\"developing-sphinx-fediverse\">\n<h1>Developing <code class=\"docutils literal notranslate\"><span class=\"pre\">sphinx-fediverse<\/span><\/code><a class=\"headerlink\" href=\"#developing-sphinx-fediverse\" title=\"Link to this heading\">\u00b6<\/a><\/h1>\n<p><span class=\"date\">2025-03-24<\/span><\/p>\n<p>A few months ago, <a class=\"reference external\" href=\"https:\/\/tech.lgbt\/&#64;LivInTheLookingGlass\/113998798864158081\">I had an idea<\/a>. What if I could use\nthe <a class=\"reference external\" href=\"https:\/\/en.wikipedia.org\/wiki\/Fediverse\">Fediverse<\/a> to embed comments in my\n<a class=\"reference external\" href=\"https:\/\/www.sphinx-doc.org\/\">sphinx<\/a>-generated pages?<\/p>\n<p>It turned out I wasn\u2019t the first to think of this. Many others have used this kind of thing to power the comments of\ntheir blogs\n<span id=\"id1\">[<a class=\"reference internal\" href=\"#id22\" title=\"Brian \u201cBeej\u201d Hall. Mastodon comments. URL: https:\/\/beej.us\/blog\/data\/mastodon-comments\/.\">Hal<\/a>, <a class=\"reference internal\" href=\"#id26\" title=\"knz. Mastodon comments. URL: https:\/\/knz.ai\/posts\/2024-08-05-mastodon-comments\/.\">knz<\/a>, <a class=\"reference internal\" href=\"#id23\" title=\"Heart Soul Machine. Mastodon comments. URL: https:\/\/heartsoulmachine.com\/blog\/2025\/01-12-mastodon-comments\/.\">Maca<\/a>, <a class=\"reference internal\" href=\"#id79\" title=\"Graham MacPhee. Mastodon blog comments. URL: https:\/\/grahammacphee.com\/writing\/mastodon-blog-comments.\">Macb<\/a>, <a class=\"reference internal\" href=\"#id78\" title=\"Daniel Pecos. Mastodon as comment system for your static blog. URL: https:\/\/danielpecos.com\/2022\/12\/25\/mastodon-as-comment-system-for-your-static-blog\/.\">Pec<\/a>, <a class=\"reference internal\" href=\"#id27\" title=\"Andreas Scherbaum. Client-side comments with mastodon on a static hugo website. URL: https:\/\/andreas.scherbaum.la\/post\/2024-05-23_client-side-comments-with-mastodon-on-a-static-hugo-website\/.\">Scha<\/a>, <a class=\"reference internal\" href=\"#id81\" title=\"Carl Schwan. Adding comments to your static blog with mastodon. URL: https:\/\/carlschwan.eu\/2020\/12\/29\/adding-comments-to-your-static-blog-with-mastodon\/.\">Schb<\/a>, <a class=\"reference internal\" href=\"#id76\" title=\"Laura L. Scott. Mastodon comments for a hugo static site. URL: https:\/\/lauralisscott.com\/blog\/mastodon-comments-hugo-static-site\/.\">Sco<\/a>, <a class=\"reference internal\" href=\"#id21\" title=\"Winterstein. Static comments for a static website. URL: https:\/\/www.winterstein.biz\/blog\/static-comments-for-static-website\/.\">Win<\/a>]<\/span>.\nEnough so that it was easy to find references on how to implement it.<\/p>\n<p>The main difficulty was the way I wanted to go about it. I wanted it to be a sphinx plugin, and I wanted it to be both\nflexible and configurable. This post covers the challenges I encountered.<\/p>\n<section id=\"the-starting-point\">\n<h2>The Starting Point<a class=\"headerlink\" href=\"#the-starting-point\" title=\"Link to this heading\">\u00b6<\/a><\/h2>\n<p><a class=\"reference external\" href=\"https:\/\/transfem.social\/notes\/a4aewgg8dydl2cq5\">Initially<\/a>, I just wanted to get a proof of concept working. I took\nheavy inspiration from <span id=\"id2\">[<a class=\"reference internal\" href=\"#id79\" title=\"Graham MacPhee. Mastodon blog comments. URL: https:\/\/grahammacphee.com\/writing\/mastodon-blog-comments.\">Macb<\/a>]<\/span> in my first draft. It\u2019s where I first saw the workflow for how to do this, and while I made\na lot of changes, the main workflow is still largely inspired by that post.<\/p>\n<figure class=\"align-default\" id=\"id127\">\n<img alt=\"The first version of sphinx-fediverse shown in the Sphinx Alabaster theme. It's very plain and white. In it you can see a demonstration of several key features: threaded replies and content warnings\" src=\"https:\/\/cdn.transfem.social\/files\/webpublic-681661df-eeb5-4e2a-b0ea-4a0dd73d3a41.png\" \/>\n<figcaption>\n<p><span class=\"caption-text\">The first version of sphinx-fediverse shown in the Sphinx Alabaster theme. In it you can see a demonstration of\nseveral key features: threaded replies and content warnings<\/span><a class=\"headerlink\" href=\"#id127\" title=\"Link to this image\">\u00b6<\/a><\/p>\n<\/figcaption>\n<\/figure>\n<p>Now at this point it only supported <a class=\"reference external\" href=\"https:\/\/en.wikipedia.org\/wiki\/Mastodon_(social_network)\">Mastodon<\/a> and its\nchildren. Sure, you could reply to it from other software types, but you couldn\u2019t host it on them. More than that, a\nlot of the values were hardcoded. It only would work on <a class=\"reference external\" href=\"https:\/\/tech.lgbt\/\">tech.lgbt<\/a>. And it had the comment ID\nhardcoded into the page.<\/p>\n<p>There was a long way to go from here.<\/p>\n<section id=\"the-workflow\">\n<h3>The Workflow<a class=\"headerlink\" href=\"#the-workflow\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>But let\u2019s start with the basics: how does it work?<\/p>\n<section id=\"at-build-time\">\n<h4>At Build Time<a class=\"headerlink\" href=\"#at-build-time\" title=\"Link to this heading\">\u00b6<\/a><\/h4>\n<p>Once you\u2019ve completed the setup and put the right command in your document, the following workflow is executed:<\/p>\n<ol class=\"arabic simple\">\n<li><p>The plugin loads a JSON file containing all of the URL -&gt; comment ID pairs<\/p><\/li>\n<li><p>If it doesn\u2019t find this URL in that mapping and sees the directive, it will create a post. At the time, this was entirely unformatted<\/p><\/li>\n<li><p>It will then save that post to the mapping<\/p><\/li>\n<li><p>In every page on your site, it will add a loader for <code class=\"docutils literal notranslate\"><span class=\"pre\">fedi_script.js<\/span><\/code>,<a class=\"footnote-reference brackets\" href=\"#id8\" id=\"id3\" role=\"doc-noteref\"><span class=\"fn-bracket\">[<\/span>1<span class=\"fn-bracket\">]<\/span><\/a> which will be outlined later<\/p><\/li>\n<li><p>If it sees the directive, it will embed in your page:<\/p>\n<ol class=\"arabic simple\">\n<li><p>The comment ID to look for<\/p><\/li>\n<li><p>The Fedi instance you\u2019re hosting on (in initial versions this was hard-coded)<\/p><\/li>\n<li><p>Which flavor of Fedi instance you\u2019re hosting on<a class=\"footnote-reference brackets\" href=\"#id9\" id=\"id4\" role=\"doc-noteref\"><span class=\"fn-bracket\">[<\/span>2<span class=\"fn-bracket\">]<\/span><\/a><\/p><\/li>\n<li><p>A short script to read the above and activate <code class=\"docutils literal notranslate\"><span class=\"pre\">fedi_script.js<\/span><\/code><\/p><\/li>\n<\/ol>\n<\/li>\n<\/ol>\n<p>This sets everything that the end-user needs into motion.<\/p>\n<\/section>\n<section id=\"on-page-load\">\n<h4>On Page Load<a class=\"headerlink\" href=\"#on-page-load\" title=\"Link to this heading\">\u00b6<\/a><\/h4>\n<p>The workflow that gets executed in browser is a fair bit more complicated, especially these days. I am going to present\nthe earliest working version of it here, and as we continue in the post I will talk about how it has changed. At the\nend of the post I will give a flow chart of the new workflow.<\/p>\n<p>At the time, <code class=\"docutils literal notranslate\"><span class=\"pre\">fedi_script.js<\/span><\/code> contained exactly 6 functions. The entry point is called <code class=\"docutils literal notranslate\"><span class=\"pre\">FetchComments()<\/span><\/code>. Its job\nis to find the first post and process all of its first-generation children. It renders them, then calls\n<code class=\"docutils literal notranslate\"><span class=\"pre\">FetchSubcomments()<\/span><\/code> on each of them. At every point in this, there is a counter that ticks down. If it hits 0, the\npage will stop looking for children. Although it is called <code class=\"docutils literal notranslate\"><span class=\"pre\">maxDepth<\/span><\/code>, that is a bit of a misnomer since a single\ncomment fetch will sometimes return many generations of children. <code class=\"docutils literal notranslate\"><span class=\"pre\">FetchSubcomments()<\/span><\/code> will recursively call itself\nuntil that counter is exhausted.<\/p>\n<p>There is another pair of methods: <code class=\"docutils literal notranslate\"><span class=\"pre\">RenderCommentsBatch()<\/span><\/code> and <code class=\"docutils literal notranslate\"><span class=\"pre\">RenderComment()<\/span><\/code>. These pretty much do what they say\non the tin. <code class=\"docutils literal notranslate\"><span class=\"pre\">RenderCommentsBatch()<\/span><\/code> will sort all of the replies its been handed, then send those off to\n<code class=\"docutils literal notranslate\"><span class=\"pre\">RenderComment()<\/span><\/code> for processing. In turn, <code class=\"docutils literal notranslate\"><span class=\"pre\">RenderComment()<\/span><\/code> will put all of that information into a massive HTML\ntemplate and place it into the body of the page. It was later pointed out to me that this isn\u2019t the most secure way to\ngo about it,<a class=\"footnote-reference brackets\" href=\"#id10\" id=\"id5\" role=\"doc-noteref\"><span class=\"fn-bracket\">[<\/span>3<span class=\"fn-bracket\">]<\/span><\/a> but sanitization is applied to enough fields that it was okay for the moment.<\/p>\n<p>The most annoying part is probably <code class=\"docutils literal notranslate\"><span class=\"pre\">replaceEmoji()<\/span><\/code>, which requires you parse several bits of info and do all sorts\nof string substitutions and CSS tuning. What\u2019s worse is that different Fediverse flavors store it in completely\ndifferent and incompatible ways.<\/p>\n<p>Lastly there is <code class=\"docutils literal notranslate\"><span class=\"pre\">escapeHtml()<\/span><\/code> which does a very basic input sanitization. This describes all the work up to 0.4.2,\nwhich was the first published version.<\/p>\n<\/section>\n<\/section>\n<\/section>\n<section id=\"building-up\">\n<h2>Building Up<a class=\"headerlink\" href=\"#building-up\" title=\"Link to this heading\">\u00b6<\/a><\/h2>\n<p>The next version saw my first attempts at Misskey support. Misskey is a bizarre creature. Its API is quite different,\nwith the biggest problem for me being that it returns <strong>its own flavor of unsanitized markdown<\/strong>. Mastodon, at least in\ntheory, returns sanitized HTML. This created all sorts of parsing problems. Another odd thing is that almost all of its\nAPI is implemented as HTTP POSTs, rather than the more common GET.<\/p>\n<p>In the first versions that supported it, the goal was to make a messy prototype. So far, that\u2019s pretty on-brand for\nthis project.<\/p>\n<p>There were some changes to names such as <code class=\"docutils literal notranslate\"><span class=\"pre\">mastodon-comments<\/span><\/code> becoming <code class=\"docutils literal notranslate\"><span class=\"pre\">fedi-comments<\/span><\/code>. Every function got a new parameter\n<code class=\"docutils literal notranslate\"><span class=\"pre\">fediFlavor<\/span><\/code> to describe which was being processed. And a <em>ton<\/em> of branches had to be added to every part of the\nbrowser-side code. There was a new Python dependency to make the initial posts, as well as different environment\nvariables needed to support the different authentication token scheme. To give an idea of how chaotic this was, I\npresent the render function as of 0.5.2. Just skimming it will give you a good impression of that.<\/p>\n<div class=\"highlight-javascript notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"w\"> <\/span><span class=\"kd\">function<\/span><span class=\"w\"> <\/span><span class=\"nx\">RenderComment<\/span><span class=\"p\">(<\/span><span class=\"nx\">fediFlavor<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"nx\">fediInstance<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"nx\">comment<\/span><span class=\"p\">)<\/span><span class=\"w\"> <\/span><span class=\"p\">{<\/span>\n<\/span><span data-line=\"2\"><span class=\"w\">    <\/span><span class=\"c1\">\/\/ TODO: better input sanitization<\/span>\n<\/span><span data-line=\"3\"><span class=\"w\">    <\/span><span class=\"k\">if<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span><span class=\"nb\">document<\/span><span class=\"p\">.<\/span><span class=\"nx\">getElementById<\/span><span class=\"p\">(<\/span><span class=\"nx\">comment<\/span><span class=\"p\">.<\/span><span class=\"nx\">id<\/span><span class=\"p\">))<\/span><span class=\"w\"> <\/span><span class=\"p\">{<\/span>\n<\/span><span data-line=\"4\"><span class=\"w\">        <\/span><span class=\"k\">return<\/span><span class=\"p\">;<\/span>\n<\/span><span data-line=\"5\"><span class=\"w\">    <\/span><span class=\"p\">}<\/span>\n<\/span><span data-line=\"6\"><span class=\"w\">    <\/span><span class=\"kd\">const<\/span><span class=\"w\"> <\/span><span class=\"nx\">user<\/span><span class=\"w\"> <\/span><span class=\"o\">=<\/span><span class=\"w\"> <\/span><span class=\"nx\">fediFlavor<\/span><span class=\"w\"> <\/span><span class=\"o\">===<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;misskey&#39;<\/span><span class=\"w\"> <\/span><span class=\"o\">?<\/span><span class=\"w\"> <\/span><span class=\"nx\">comment<\/span><span class=\"p\">.<\/span><span class=\"nx\">user<\/span><span class=\"w\"> <\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"nx\">comment<\/span><span class=\"p\">.<\/span><span class=\"nx\">account<\/span><span class=\"p\">;<\/span>\n<\/span><span data-line=\"7\"><span class=\"w\">    <\/span><span class=\"kd\">let<\/span><span class=\"w\"> <\/span><span class=\"nx\">domain<\/span><span class=\"p\">;<\/span>\n<\/span><span data-line=\"8\"><span class=\"w\">    <\/span><span class=\"k\">if<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span><span class=\"nx\">fediFlavor<\/span><span class=\"w\"> <\/span><span class=\"o\">===<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;misskey&#39;<\/span><span class=\"p\">)<\/span><span class=\"w\"> <\/span><span class=\"p\">{<\/span>\n<\/span><span data-line=\"9\"><span class=\"w\">        <\/span><span class=\"nx\">domain<\/span><span class=\"w\"> <\/span><span class=\"o\">=<\/span><span class=\"w\"> <\/span><span class=\"nx\">user<\/span><span class=\"p\">.<\/span><span class=\"nx\">host<\/span><span class=\"w\"> <\/span><span class=\"o\">||<\/span><span class=\"w\"> <\/span><span class=\"nx\">fediInstance<\/span><span class=\"p\">;<\/span>\n<\/span><span data-line=\"10\"><span class=\"w\">    <\/span><span class=\"p\">}<\/span><span class=\"w\"> <\/span><span class=\"k\">else<\/span><span class=\"w\"> <\/span><span class=\"p\">{<\/span>\n<\/span><span data-line=\"11\"><span class=\"w\">        <\/span><span class=\"kd\">const<\/span><span class=\"w\"> <\/span><span class=\"nx\">match<\/span><span class=\"w\"> <\/span><span class=\"o\">=<\/span><span class=\"w\"> <\/span><span class=\"nx\">user<\/span><span class=\"p\">.<\/span><span class=\"nx\">url<\/span><span class=\"p\">.<\/span><span class=\"nx\">match<\/span><span class=\"p\">(<\/span><span class=\"sr\">\/https?:\\\/\\\/([^\\\/]+)\/<\/span><span class=\"p\">);<\/span>\n<\/span><span data-line=\"12\"><span class=\"w\">        <\/span><span class=\"nx\">domain<\/span><span class=\"w\"> <\/span><span class=\"o\">=<\/span><span class=\"w\"> <\/span><span class=\"nx\">match<\/span><span class=\"w\"> <\/span><span class=\"o\">?<\/span><span class=\"w\"> <\/span><span class=\"nx\">match<\/span><span class=\"p\">[<\/span><span class=\"mf\">1<\/span><span class=\"p\">]<\/span><span class=\"w\"> <\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"kc\">null<\/span><span class=\"p\">;<\/span>\n<\/span><span data-line=\"13\"><span class=\"w\">    <\/span><span class=\"p\">}<\/span>\n<\/span><span data-line=\"14\"><span class=\"w\">    <\/span><span class=\"kd\">let<\/span><span class=\"w\"> <\/span><span class=\"nx\">handle<\/span><span class=\"p\">;<\/span>\n<\/span><span data-line=\"15\"><span class=\"w\">    <\/span><span class=\"k\">if<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span><span class=\"o\">!<\/span><span class=\"nx\">domain<\/span><span class=\"p\">)<\/span><span class=\"w\"> <\/span><span class=\"p\">{<\/span>\n<\/span><span data-line=\"16\"><span class=\"w\">        <\/span><span class=\"nx\">console<\/span><span class=\"p\">.<\/span><span class=\"nx\">error<\/span><span class=\"p\">(<\/span><span class=\"s2\">&quot;Could not extract domain name from url: &quot;<\/span><span class=\"w\"> <\/span><span class=\"o\">+<\/span><span class=\"w\"> <\/span><span class=\"nx\">user<\/span><span class=\"p\">.<\/span><span class=\"nx\">url<\/span><span class=\"p\">);<\/span>\n<\/span><span data-line=\"17\"><span class=\"w\">        <\/span><span class=\"nx\">handle<\/span><span class=\"w\"> <\/span><span class=\"o\">=<\/span><span class=\"w\"> <\/span><span class=\"sb\">`@<\/span><span class=\"si\">${<\/span><span class=\"nx\">user<\/span><span class=\"p\">.<\/span><span class=\"nx\">username<\/span><span class=\"si\">}<\/span><span class=\"sb\">`<\/span><span class=\"p\">;<\/span>\n<\/span><span data-line=\"18\"><span class=\"w\">    <\/span><span class=\"p\">}<\/span><span class=\"w\"> <\/span><span class=\"k\">else<\/span><span class=\"w\"> <\/span><span class=\"p\">{<\/span>\n<\/span><span data-line=\"19\"><span class=\"w\">        <\/span><span class=\"nx\">handle<\/span><span class=\"w\"> <\/span><span class=\"o\">=<\/span><span class=\"w\"> <\/span><span class=\"sb\">`@<\/span><span class=\"si\">${<\/span><span class=\"nx\">user<\/span><span class=\"p\">.<\/span><span class=\"nx\">username<\/span><span class=\"si\">}<\/span><span class=\"sb\">@<\/span><span class=\"si\">${<\/span><span class=\"nx\">domain<\/span><span class=\"si\">}<\/span><span class=\"sb\">`<\/span><span class=\"p\">;<\/span>\n<\/span><span data-line=\"20\"><span class=\"w\">    <\/span><span class=\"p\">}<\/span>\n<\/span><span data-line=\"21\"><span class=\"w\">    <\/span><span class=\"kd\">const<\/span><span class=\"w\"> <\/span><span class=\"nx\">commentUrl<\/span><span class=\"w\"> <\/span><span class=\"o\">=<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span><span class=\"nx\">fediFlavor<\/span><span class=\"w\"> <\/span><span class=\"o\">===<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;misskey&#39;<\/span>\n<\/span><span data-line=\"22\"><span class=\"w\">        <\/span><span class=\"o\">?<\/span><span class=\"w\"> <\/span><span class=\"sb\">`https:\/\/<\/span><span class=\"si\">${<\/span><span class=\"nx\">fediInstance<\/span><span class=\"si\">}<\/span><span class=\"sb\">\/notes\/<\/span><span class=\"si\">${<\/span><span class=\"nx\">comment<\/span><span class=\"p\">.<\/span><span class=\"nx\">id<\/span><span class=\"si\">}<\/span><span class=\"sb\">`<\/span>\n<\/span><span data-line=\"23\"><span class=\"w\">        <\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"nx\">comment<\/span><span class=\"p\">.<\/span><span class=\"nx\">url<\/span>\n<\/span><span data-line=\"24\"><span class=\"w\">    <\/span><span class=\"p\">)<\/span>\n<\/span><span data-line=\"25\"><span class=\"w\">    <\/span><span class=\"kd\">const<\/span><span class=\"w\"> <\/span><span class=\"nx\">userUrl<\/span><span class=\"w\"> <\/span><span class=\"o\">=<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span><span class=\"nx\">fediFlavor<\/span><span class=\"w\"> <\/span><span class=\"o\">===<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;misskey&#39;<\/span>\n<\/span><span data-line=\"26\"><span class=\"w\">        <\/span><span class=\"o\">?<\/span><span class=\"w\"> <\/span><span class=\"sb\">`https:\/\/<\/span><span class=\"si\">${<\/span><span class=\"nx\">fediInstance<\/span><span class=\"si\">}<\/span><span class=\"sb\">\/<\/span><span class=\"si\">${<\/span><span class=\"nx\">handle<\/span><span class=\"si\">}<\/span><span class=\"sb\">`<\/span>\n<\/span><span data-line=\"27\"><span class=\"w\">        <\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"nx\">user<\/span><span class=\"p\">.<\/span><span class=\"nx\">url<\/span>\n<\/span><span data-line=\"28\"><span class=\"w\">    <\/span><span class=\"p\">)<\/span>\n<\/span><span data-line=\"29\"><span class=\"w\">    <\/span><span class=\"kd\">let<\/span><span class=\"w\"> <\/span><span class=\"nx\">str<\/span><span class=\"w\"> <\/span><span class=\"o\">=<\/span><span class=\"w\"> <\/span><span class=\"sb\">`&lt;div class=&quot;comment&quot; id=<\/span><span class=\"si\">${<\/span><span class=\"nx\">comment<\/span><span class=\"p\">.<\/span><span class=\"nx\">id<\/span><span class=\"si\">}<\/span><span class=\"sb\">&gt;<\/span>\n<\/span><span data-line=\"30\"><span class=\"sb\">        &lt;div class=&quot;author&quot;&gt;<\/span>\n<\/span><span data-line=\"31\"><span class=\"sb\">            &lt;div class=&quot;avatar&quot;&gt;<\/span>\n<\/span><span data-line=\"32\"><span class=\"sb\">                &lt;img src=&quot;<\/span><span class=\"si\">${<\/span><span class=\"nx\">user<\/span><span class=\"p\">.<\/span><span class=\"nx\">avatar_static<\/span><span class=\"w\"> <\/span><span class=\"o\">||<\/span><span class=\"w\"> <\/span><span class=\"nx\">user<\/span><span class=\"p\">.<\/span><span class=\"nx\">avatarUrl<\/span><span class=\"si\">}<\/span><span class=\"sb\">&quot; height=&quot;30&quot; width=&quot;30&quot; alt=&quot;Avatar for <\/span><span class=\"si\">${<\/span><span class=\"nx\">user<\/span><span class=\"p\">.<\/span><span class=\"nx\">display_name<\/span><span class=\"w\"> <\/span><span class=\"o\">||<\/span><span class=\"w\"> <\/span><span class=\"nx\">user<\/span><span class=\"p\">.<\/span><span class=\"nx\">name<\/span><span class=\"si\">}<\/span><span class=\"sb\">&quot;&gt;<\/span>\n<\/span><span data-line=\"33\"><span class=\"sb\">            &lt;\/div&gt;<\/span>\n<\/span><span data-line=\"34\"><span class=\"sb\">            &lt;a target=&quot;_blank&quot; class=&quot;date&quot; href=&quot;<\/span><span class=\"si\">${<\/span><span class=\"nx\">commentUrl<\/span><span class=\"si\">}<\/span><span class=\"sb\">&quot; rel=&quot;nofollow&quot;&gt;<\/span>\n<\/span><span data-line=\"35\"><span class=\"sb\">                <\/span><span class=\"si\">${<\/span><span class=\"ow\">new<\/span><span class=\"w\"> <\/span><span class=\"nb\">Date<\/span><span class=\"p\">(<\/span><span class=\"nx\">comment<\/span><span class=\"p\">.<\/span><span class=\"nx\">created_at<\/span><span class=\"w\"> <\/span><span class=\"o\">||<\/span><span class=\"w\"> <\/span><span class=\"nx\">comment<\/span><span class=\"p\">.<\/span><span class=\"nx\">createdAt<\/span><span class=\"p\">).<\/span><span class=\"nx\">toLocaleString<\/span><span class=\"p\">()<\/span><span class=\"si\">}<\/span>\n<\/span><span data-line=\"36\"><span class=\"sb\">            &lt;\/a&gt;<\/span>\n<\/span><span data-line=\"37\"><span class=\"sb\">            &lt;a target=&quot;_blank&quot; href=&quot;<\/span><span class=\"si\">${<\/span><span class=\"nx\">userUrl<\/span><span class=\"si\">}<\/span><span class=\"sb\">&quot; rel=&quot;nofollow&quot;&gt;<\/span>\n<\/span><span data-line=\"38\"><span class=\"sb\">                &lt;span class=&quot;username&quot;&gt;<\/span><span class=\"si\">${<\/span><span class=\"nx\">replaceEmoji<\/span><span class=\"p\">(<\/span><span class=\"nx\">escapeHtml<\/span><span class=\"p\">(<\/span><span class=\"nx\">user<\/span><span class=\"p\">.<\/span><span class=\"nx\">display_name<\/span><span class=\"w\"> <\/span><span class=\"o\">||<\/span><span class=\"w\"> <\/span><span class=\"nx\">user<\/span><span class=\"p\">.<\/span><span class=\"nx\">name<\/span><span class=\"p\">),<\/span><span class=\"w\"> <\/span><span class=\"nx\">user<\/span><span class=\"p\">.<\/span><span class=\"nx\">emojis<\/span><span class=\"w\"> <\/span><span class=\"o\">||<\/span><span class=\"w\"> <\/span><span class=\"p\">[])<\/span><span class=\"si\">}<\/span><span class=\"sb\">&lt;\/span&gt; &lt;span class=&quot;handle&quot;&gt;(<\/span><span class=\"si\">${<\/span><span class=\"nx\">handle<\/span><span class=\"si\">}<\/span><span class=\"sb\">)&lt;\/span&gt;<\/span>\n<\/span><span data-line=\"39\"><span class=\"sb\">            &lt;\/a&gt;<\/span>\n<\/span><span data-line=\"40\"><span class=\"sb\">        &lt;\/div&gt;`<\/span><span class=\"p\">;<\/span>\n<\/span><span data-line=\"41\"><span class=\"w\">    <\/span><span class=\"k\">if<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span><span class=\"nx\">comment<\/span><span class=\"p\">.<\/span><span class=\"nx\">sensitive<\/span><span class=\"p\">)<\/span><span class=\"w\"> <\/span><span class=\"p\">{<\/span>\n<\/span><span data-line=\"42\"><span class=\"w\">        <\/span><span class=\"nx\">str<\/span><span class=\"w\"> <\/span><span class=\"o\">+=<\/span><span class=\"w\"> <\/span><span class=\"sb\">`&lt;details&gt;&lt;summary&gt;<\/span><span class=\"si\">${<\/span><span class=\"nx\">comment<\/span><span class=\"p\">.<\/span><span class=\"nx\">spoiler_text<\/span><span class=\"w\"> <\/span><span class=\"o\">||<\/span><span class=\"w\"> <\/span><span class=\"nx\">comment<\/span><span class=\"p\">.<\/span><span class=\"nx\">cw<\/span><span class=\"w\"> <\/span><span class=\"o\">||<\/span><span class=\"w\"> <\/span><span class=\"s2\">&quot;&quot;<\/span><span class=\"si\">}<\/span><span class=\"sb\">&lt;\/summary&gt;`<\/span><span class=\"p\">;<\/span>\n<\/span><span data-line=\"43\"><span class=\"w\">    <\/span><span class=\"p\">}<\/span>\n<\/span><span data-line=\"44\"><span class=\"w\">    <\/span><span class=\"nx\">str<\/span><span class=\"w\"> <\/span><span class=\"o\">+=<\/span><span class=\"w\"> <\/span><span class=\"sb\">`<\/span>\n<\/span><span data-line=\"45\"><span class=\"sb\">        &lt;div class=&quot;content&quot;&gt;<\/span>\n<\/span><span data-line=\"46\"><span class=\"sb\">            &lt;div class=&quot;fedi-comment-content&quot;&gt;<\/span><span class=\"si\">${<\/span><span class=\"nx\">comment<\/span><span class=\"p\">.<\/span><span class=\"nx\">content<\/span><span class=\"w\"> <\/span><span class=\"o\">||<\/span><span class=\"w\"> <\/span><span class=\"nx\">comment<\/span><span class=\"p\">.<\/span><span class=\"nx\">text<\/span><span class=\"si\">}<\/span><span class=\"sb\">&lt;\/div&gt;`<\/span><span class=\"p\">;<\/span>\n<\/span><span data-line=\"47\"><span class=\"w\">    <\/span><span class=\"k\">for<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span><span class=\"kd\">let<\/span><span class=\"w\"> <\/span><span class=\"nx\">attachment<\/span><span class=\"w\"> <\/span><span class=\"k\">of<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span><span class=\"nx\">comment<\/span><span class=\"p\">.<\/span><span class=\"nx\">media_attachments<\/span><span class=\"w\"> <\/span><span class=\"o\">||<\/span><span class=\"w\"> <\/span><span class=\"nx\">comment<\/span><span class=\"p\">.<\/span><span class=\"nx\">files<\/span><span class=\"p\">))<\/span><span class=\"w\"> <\/span><span class=\"p\">{<\/span>\n<\/span><span data-line=\"48\"><span class=\"w\">        <\/span><span class=\"k\">if<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span><span class=\"nx\">attachment<\/span><span class=\"p\">.<\/span><span class=\"nx\">type<\/span><span class=\"w\"> <\/span><span class=\"o\">===<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;image&#39;<\/span><span class=\"p\">)<\/span><span class=\"w\"> <\/span><span class=\"p\">{<\/span>\n<\/span><span data-line=\"49\"><span class=\"w\">            <\/span><span class=\"nx\">str<\/span><span class=\"w\"> <\/span><span class=\"o\">+=<\/span><span class=\"w\"> <\/span><span class=\"sb\">`&lt;img src=&quot;<\/span><span class=\"si\">${<\/span><span class=\"nx\">attachment<\/span><span class=\"p\">.<\/span><span class=\"nx\">remote_url<\/span><span class=\"w\"> <\/span><span class=\"o\">||<\/span><span class=\"w\"> <\/span><span class=\"nx\">attachment<\/span><span class=\"p\">.<\/span><span class=\"nx\">url<\/span><span class=\"si\">}<\/span><span class=\"sb\">&quot; alt=&quot;<\/span><span class=\"si\">${<\/span><span class=\"nx\">attachment<\/span><span class=\"p\">.<\/span><span class=\"nx\">description<\/span><span class=\"si\">}<\/span><span class=\"sb\">&quot; class=&quot;attachment&quot;`<\/span><span class=\"p\">;<\/span>\n<\/span><span data-line=\"50\"><span class=\"w\">        <\/span><span class=\"p\">}<\/span>\n<\/span><span data-line=\"51\"><span class=\"w\">    <\/span><span class=\"p\">}<\/span>\n<\/span><span data-line=\"52\"><span class=\"w\">    <\/span><span class=\"nx\">str<\/span><span class=\"w\"> <\/span><span class=\"o\">+=<\/span><span class=\"w\"> <\/span><span class=\"sb\">`<\/span>\n<\/span><span data-line=\"53\"><span class=\"sb\">        &lt;\/div&gt;<\/span>\n<\/span><span data-line=\"54\"><span class=\"sb\">        <\/span><span class=\"si\">${<\/span><span class=\"p\">(<\/span><span class=\"nx\">comment<\/span><span class=\"p\">.<\/span><span class=\"nx\">sensitive<\/span><span class=\"w\"> <\/span><span class=\"o\">||<\/span><span class=\"w\"> <\/span><span class=\"nx\">comment<\/span><span class=\"p\">.<\/span><span class=\"nx\">cw<\/span><span class=\"p\">)<\/span><span class=\"w\"> <\/span><span class=\"o\">?<\/span><span class=\"w\"> <\/span><span class=\"s2\">&quot;&lt;\/details&gt;&quot;<\/span><span class=\"w\"> <\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">&quot;&quot;<\/span><span class=\"si\">}<\/span>\n<\/span><span data-line=\"55\"><span class=\"sb\">        &lt;div class=&quot;info&quot;&gt;&lt;img src=&quot;_static\/like.svg&quot; alt=&quot;Likes&quot;&gt;<\/span><span class=\"si\">${<\/span><span class=\"nx\">comment<\/span><span class=\"p\">.<\/span><span class=\"nx\">favourites_count<\/span><span class=\"w\"> <\/span><span class=\"o\">||<\/span><span class=\"w\"> <\/span><span class=\"nx\">comment<\/span><span class=\"p\">.<\/span><span class=\"nx\">reactionCount<\/span><span class=\"si\">}<\/span><span class=\"sb\">, &lt;img src=&quot;_static\/boost.svg&quot; alt=&quot;Boosts&quot;&gt;<\/span><span class=\"si\">${<\/span><span class=\"nx\">comment<\/span><span class=\"p\">.<\/span><span class=\"nx\">reblogs_count<\/span><span class=\"w\"> <\/span><span class=\"o\">||<\/span><span class=\"w\"> <\/span><span class=\"nx\">comment<\/span><span class=\"p\">.<\/span><span class=\"nx\">renoteCount<\/span><span class=\"si\">}<\/span><span class=\"sb\">&lt;\/div&gt;<\/span>\n<\/span><span data-line=\"56\"><span class=\"sb\">        &lt;br&gt;<\/span>\n<\/span><span data-line=\"57\"><span class=\"sb\">    &lt;\/div&gt;`<\/span><span class=\"p\">;<\/span>\n<\/span><span data-line=\"58\"><span class=\"w\">    <\/span><span class=\"kd\">const<\/span><span class=\"w\"> <\/span><span class=\"nx\">doc<\/span><span class=\"w\"> <\/span><span class=\"o\">=<\/span><span class=\"w\"> <\/span><span class=\"nx\">parser<\/span><span class=\"p\">.<\/span><span class=\"nx\">parseFromString<\/span><span class=\"p\">(<\/span><span class=\"nx\">replaceEmoji<\/span><span class=\"p\">(<\/span><span class=\"nx\">str<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"nx\">comment<\/span><span class=\"p\">.<\/span><span class=\"nx\">emojis<\/span><span class=\"w\"> <\/span><span class=\"o\">||<\/span><span class=\"w\"> <\/span><span class=\"p\">[]),<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;text\/html&#39;<\/span><span class=\"p\">);<\/span>\n<\/span><span data-line=\"59\"><span class=\"w\">    <\/span><span class=\"kd\">const<\/span><span class=\"w\"> <\/span><span class=\"nx\">fragment<\/span><span class=\"w\"> <\/span><span class=\"o\">=<\/span><span class=\"w\"> <\/span><span class=\"nb\">document<\/span><span class=\"p\">.<\/span><span class=\"nx\">createDocumentFragment<\/span><span class=\"p\">();<\/span>\n<\/span><span data-line=\"60\"><span class=\"w\">    <\/span><span class=\"nb\">Array<\/span><span class=\"p\">.<\/span><span class=\"kr\">from<\/span><span class=\"p\">(<\/span><span class=\"nx\">doc<\/span><span class=\"p\">.<\/span><span class=\"nx\">body<\/span><span class=\"p\">.<\/span><span class=\"nx\">childNodes<\/span><span class=\"p\">).<\/span><span class=\"nx\">forEach<\/span><span class=\"p\">(<\/span><span class=\"nx\">node<\/span><span class=\"w\"> <\/span><span class=\"p\">=&gt;<\/span><span class=\"w\"> <\/span><span class=\"nx\">fragment<\/span><span class=\"p\">.<\/span><span class=\"nx\">appendChild<\/span><span class=\"p\">(<\/span><span class=\"nx\">node<\/span><span class=\"p\">));<\/span>\n<\/span><span data-line=\"61\"><span class=\"w\">    <\/span><span class=\"k\">return<\/span><span class=\"w\"> <\/span><span class=\"nx\">fragment<\/span><span class=\"p\">;<\/span>\n<\/span><span data-line=\"62\"><span class=\"p\">}<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<p>It\u2019s rather terrible. There are conditionals all over the place. More than that, it turned out I was using an insecure\nAPI to render. Only the most trivial attempts at sanitization were happening at all, and in nowhere near enough places.<\/p>\n<p>Fortunately, changes were coming.<\/p>\n<section id=\"a-better-comment-parser\">\n<h3>A Better Comment Parser<a class=\"headerlink\" href=\"#a-better-comment-parser\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>The next versions were all focused on building up a comment parser to try and reduce the complexity of rendering. Each\nFedi implementation got its own <code class=\"docutils literal notranslate\"><span class=\"pre\">extractComment()<\/span><\/code> variation, and there was a common return spec between the two to\nensure consistency. Roughly:<\/p>\n<div class=\"highlight-JSON notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"p\">{<\/span>\n<\/span><span data-line=\"2\"><span class=\"w\">  <\/span><span class=\"nt\">&quot;id&quot;<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">&quot;string&quot;<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"3\"><span class=\"w\">  <\/span><span class=\"nt\">&quot;url&quot;<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">&quot;url&quot;<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"4\"><span class=\"w\">  <\/span><span class=\"nt\">&quot;date&quot;<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">&quot;string&quot;<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"5\"><span class=\"w\">  <\/span><span class=\"nt\">&quot;cw&quot;<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">&quot;null | string&quot;<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"6\"><span class=\"w\">  <\/span><span class=\"nt\">&quot;emoji&quot;<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">{<\/span>\n<\/span><span data-line=\"7\"><span class=\"w\">    <\/span><span class=\"nt\">&quot;name1&quot;<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">&quot;url&quot;<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"8\"><span class=\"w\">    <\/span><span class=\"nt\">&quot;name2&quot;<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">&quot;url&quot;<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"9\"><span class=\"w\">    <\/span><span class=\"nt\">&quot;...&quot;<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">&quot;...&quot;<\/span>\n<\/span><span data-line=\"10\"><span class=\"w\">  <\/span><span class=\"p\">},<\/span>\n<\/span><span data-line=\"11\"><span class=\"w\">  <\/span><span class=\"nt\">&quot;reactionCount&quot;<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">&quot;int&quot;<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"12\"><span class=\"w\">  <\/span><span class=\"nt\">&quot;boostCount&quot;<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">&quot;int&quot;<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"13\"><span class=\"w\">  <\/span><span class=\"nt\">&quot;media&quot;<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">[{<\/span>\n<\/span><span data-line=\"14\"><span class=\"w\">    <\/span><span class=\"nt\">&quot;url&quot;<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">&quot;url&quot;<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"15\"><span class=\"w\">    <\/span><span class=\"nt\">&quot;description&quot;<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">&quot;string&quot;<\/span>\n<\/span><span data-line=\"16\"><span class=\"w\">  <\/span><span class=\"p\">}],<\/span>\n<\/span><span data-line=\"17\"><span class=\"w\">  <\/span><span class=\"nt\">&quot;content&quot;<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">&quot;string&quot;<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"18\"><span class=\"w\">  <\/span><span class=\"nt\">&quot;user&quot;<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">{<\/span>\n<\/span><span data-line=\"19\"><span class=\"w\">    <\/span><span class=\"nt\">&quot;host&quot;<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">&quot;string&quot;<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"20\"><span class=\"w\">    <\/span><span class=\"nt\">&quot;handle&quot;<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">&quot;string&quot;<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"21\"><span class=\"w\">    <\/span><span class=\"nt\">&quot;url&quot;<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">&quot;url&quot;<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"22\"><span class=\"w\">    <\/span><span class=\"nt\">&quot;name&quot;<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">&quot;string&quot;<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"23\"><span class=\"w\">    <\/span><span class=\"nt\">&quot;avatar&quot;<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">&quot;url&quot;<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"24\"><span class=\"w\">    <\/span><span class=\"nt\">&quot;emoji&quot;<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">{<\/span>\n<\/span><span data-line=\"25\"><span class=\"w\">      <\/span><span class=\"nt\">&quot;name1&quot;<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">&quot;url&quot;<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"26\"><span class=\"w\">      <\/span><span class=\"nt\">&quot;name2&quot;<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">&quot;url&quot;<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"27\"><span class=\"w\">      <\/span><span class=\"nt\">&quot;...&quot;<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">&quot;...&quot;<\/span>\n<\/span><span data-line=\"28\"><span class=\"w\">    <\/span><span class=\"p\">}<\/span>\n<\/span><span data-line=\"29\"><span class=\"w\">  <\/span><span class=\"p\">}<\/span>\n<\/span><span data-line=\"30\"><span class=\"p\">}<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<p>This would evolve over time, but it was a <em>lot<\/em> better than parsing this out in render code. That function we showed\nabove was able to shrink from 62 lines down to 38. Now the workflow looked something like:<\/p>\n<p>The next several versions were focused entirely on bug fixes, stabilizing this new API, and fixing CSS problems.<\/p>\n<\/section>\n<section id=\"increasing-misskey-compatibility\">\n<h3>Increasing Misskey Compatibility<a class=\"headerlink\" href=\"#increasing-misskey-compatibility\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<section id=\"parsing-misskey-flavored-markdown\">\n<h4>Parsing Misskey Flavored Markdown<a class=\"headerlink\" href=\"#parsing-misskey-flavored-markdown\" title=\"Link to this heading\">\u00b6<\/a><\/h4>\n<p>The part of Misskey support that I find most frustrating is their markdown syntax. They essentially support a subset of\n<a class=\"reference external\" href=\"https:\/\/en.wikipedia.org\/wiki\/Markdown\">standard Markdown<\/a>, but with a few additions:<\/p>\n<ol class=\"arabic simple\">\n<li><p>hashtags (ex: <a class=\"reference external\" href=\"https:\/\/mastodon.social\/tags\/OpenStreetMap\">#OpenStreetMap<\/a>)<\/p><\/li>\n<li><p>mentions (ex: <a class=\"reference external\" href=\"https:\/\/transfem.social\/&#64;LivInTheLookingGlass\">&#64;LivInTheLookingGlass<\/a>, <a class=\"reference external\" href=\"https:\/\/mathstodon.xyz\/&#64;tao\">&#64;tao&#64;mathstodon.xyz<\/a>)<\/p><\/li>\n<li><p>custom emoji (syntax: <code class=\"docutils literal notranslate\"><span class=\"pre\">:blobhaj:<\/span><\/code> -&gt; <img alt=\"The blobhaj emoji on transfem.social, which is a stylized cartoon of IKEA's blue stuffed shark\" class=\"emoji\" src=\"https:\/\/cdn.transfem.social\/files\/b4a56c87-45f1-4091-b93a-4c59c7039c68.webp\" \/>)<\/p><\/li>\n<li><p><code class=\"docutils literal notranslate\"><span class=\"pre\">&lt;small&gt;<\/span><\/code> (which shrinks text)<\/p><\/li>\n<li><p><code class=\"docutils literal notranslate\"><span class=\"pre\">&lt;plain&gt;<\/span><\/code> (which escapes all formatting <em>without<\/em> wrapping in a code block)<\/p><\/li>\n<li><p><code class=\"docutils literal notranslate\"><span class=\"pre\">&lt;center&gt;<\/span><\/code> (which centers text)<\/p><\/li>\n<li><p><code class=\"docutils literal notranslate\"><span class=\"pre\">&lt;i&gt;<\/span><\/code> (a deprecated HTML tag that is an alias for Markdown\u2019s <code class=\"docutils literal notranslate\"><span class=\"pre\">*text*<\/span><\/code>)<\/p><\/li>\n<li><p>flip (which is a shortcut for CSS transforms)<\/p><\/li>\n<li><p>blur (CSS)<\/p><\/li>\n<li><p>borders (CSS)<\/p><\/li>\n<li><p>foreground and background colors (CSS)<\/p><\/li>\n<li><p>expand (CSS)<\/p><\/li>\n<li><p>position shifts (CSS)<\/p><\/li>\n<li><p>rotations (CSS)<\/p><\/li>\n<li><p><a class=\"reference external\" href=\"https:\/\/en.wikipedia.org\/wiki\/Furigana\">yomigana<\/a>, which adds pronunciation characters above Japanese kanji<\/p><\/li>\n<li><p>search (shows web search results)<\/p><\/li>\n<li><p>several different animations<\/p><\/li>\n<\/ol>\n<p>These are ordered in roughly the order that I am prioritizing them. The first 9 items are currently implemented, and I\nhave plans to at least attempt an implementation of the remaining CSS elements. The last 3 are likely beyond the scope\nof this project.<\/p>\n<p>Given that Misskey Flavored Markdown is notoriously difficult to parse, this project only aims to support a subset of\nit. And the easiest way by far to achieve that was to bring in a new dependency, <a class=\"reference external\" href=\"https:\/\/marked.js.org\/\">marked.js<\/a>.\nThis allows me to set up a nice pipeline of string substitutions:<\/p>\n<ol class=\"arabic simple\">\n<li><p>Original string<\/p><\/li>\n<li><p>Escape the contents of <code class=\"docutils literal notranslate\"><span class=\"pre\">&lt;plain&gt;<\/span><\/code><\/p><\/li>\n<li><p>Replace <code class=\"docutils literal notranslate\"><span class=\"pre\">&lt;center&gt;<\/span><\/code><\/p><\/li>\n<li><p>Replace <code class=\"docutils literal notranslate\"><span class=\"pre\">&lt;i&gt;<\/span><\/code><\/p><\/li>\n<li><p>Replace <code class=\"docutils literal notranslate\"><span class=\"pre\">&lt;small&gt;<\/span><\/code><\/p><\/li>\n<li><p>Replace <code class=\"docutils literal notranslate\"><span class=\"pre\">flip<\/span><\/code><\/p><\/li>\n<li><p>Replace <code class=\"docutils literal notranslate\"><span class=\"pre\">blur<\/span><\/code><\/p><\/li>\n<li><p>Replace hashtags<\/p><\/li>\n<li><p>Replace mentions<\/p><\/li>\n<li><p>Replace the link format that removes previews<\/p><\/li>\n<li><p>Feed to <code class=\"docutils literal notranslate\"><span class=\"pre\">marked.js<\/span><\/code><\/p><\/li>\n<li><p>Raw HTML<\/p><\/li>\n<\/ol>\n<\/section>\n<section id=\"supporting-custom-reactions\">\n<h4>Supporting Custom Reactions<a class=\"headerlink\" href=\"#supporting-custom-reactions\" title=\"Link to this heading\">\u00b6<\/a><\/h4>\n<p>The other big difference between Misskey and (most implementations of) Mastodon is support for emoji-based reactions.\nOn Mastodon, you have two possible interactions: favorite and boost. Misskey expands this to not just the set of emoji\nfound in unicode, but also any custom one implemented on your server. Since at this point I had not implemented support\nfor Misskey emoji, this implementation was restricted to only the unicode ones. In general, if a reaction is unknown or\ngeneric, it is rendered as \u2764. If it is a different reaction that is exactly one character,<a class=\"footnote-reference brackets\" href=\"#id11\" id=\"id6\" role=\"doc-noteref\"><span class=\"fn-bracket\">[<\/span>4<span class=\"fn-bracket\">]<\/span><\/a> that is preserved.\nBoosts were left untouched and rendered as before.<\/p>\n<p>You can still see the previous favorite icon at the top of each comment section.<\/p>\n<\/section>\n<section id=\"other-misc-changes\">\n<h4>Other Misc Changes<a class=\"headerlink\" href=\"#other-misc-changes\" title=\"Link to this heading\">\u00b6<\/a><\/h4>\n<p>In addition to what\u2019s discussed above, this series of changes brought:<\/p>\n<ul class=\"simple\">\n<li><p>Changes to word wrap on handles<\/p><\/li>\n<li><p>Adding the page title to discussion posts<\/p><\/li>\n<\/ul>\n<\/section>\n<\/section>\n<section id=\"improving-the-renderer\">\n<h3>Improving the Renderer<a class=\"headerlink\" href=\"#improving-the-renderer\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>In the 0.7 release sets, the biggest change was switching from an HTML template to using the\n<a class=\"reference external\" href=\"https:\/\/dom.spec.whatwg.org\/\">DOM API<\/a>. There is still one place where raw HTML is being used: <code class=\"docutils literal notranslate\"><span class=\"pre\">replaceEmoji()<\/span><\/code>,\nbut it is reasonably isolated and (in later versions) well-sanitized.<\/p>\n<p>Under the DOM API, you programmatically create elements and append them to existing one. So a flow to make:<\/p>\n<div class=\"highlight-JavaScript notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"k\">for<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span><span class=\"kd\">let<\/span><span class=\"w\"> <\/span><span class=\"nx\">attachment<\/span><span class=\"w\"> <\/span><span class=\"k\">of<\/span><span class=\"w\"> <\/span><span class=\"nx\">comment<\/span><span class=\"p\">.<\/span><span class=\"nx\">media<\/span><span class=\"p\">)<\/span><span class=\"w\"> <\/span><span class=\"p\">{<\/span>\n<\/span><span data-line=\"2\"><span class=\"w\">    <\/span><span class=\"nx\">str<\/span><span class=\"w\"> <\/span><span class=\"o\">+=<\/span><span class=\"w\"> <\/span><span class=\"sb\">`&lt;img src=&quot;<\/span><span class=\"si\">${<\/span><span class=\"nx\">attachment<\/span><span class=\"p\">.<\/span><span class=\"nx\">url<\/span><span class=\"si\">}<\/span><span class=\"sb\">&quot; alt=&quot;<\/span><span class=\"si\">${<\/span><span class=\"nx\">escapeHtml<\/span><span class=\"p\">(<\/span><span class=\"nx\">attachment<\/span><span class=\"p\">.<\/span><span class=\"nx\">description<\/span><span class=\"p\">)<\/span><span class=\"si\">}<\/span><span class=\"sb\">&quot; class=&quot;attachment&quot;&gt;`<\/span><span class=\"p\">;<\/span>\n<\/span><span data-line=\"3\"><span class=\"p\">}<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<p>Might look like:<\/p>\n<div class=\"highlight-JavaScript notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"k\">for<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span><span class=\"kd\">const<\/span><span class=\"w\"> <\/span><span class=\"nx\">attachment<\/span><span class=\"w\"> <\/span><span class=\"k\">of<\/span><span class=\"w\"> <\/span><span class=\"nx\">comment<\/span><span class=\"p\">.<\/span><span class=\"nx\">media<\/span><span class=\"p\">)<\/span><span class=\"w\"> <\/span><span class=\"p\">{<\/span>\n<\/span><span data-line=\"2\"><span class=\"w\">    <\/span><span class=\"kd\">const<\/span><span class=\"w\"> <\/span><span class=\"nx\">attachmentNode<\/span><span class=\"w\"> <\/span><span class=\"o\">=<\/span><span class=\"w\"> <\/span><span class=\"nb\">document<\/span><span class=\"p\">.<\/span><span class=\"nx\">createElement<\/span><span class=\"p\">(<\/span><span class=\"s2\">&quot;img&quot;<\/span><span class=\"p\">);<\/span>\n<\/span><span data-line=\"3\"><span class=\"w\">    <\/span><span class=\"nx\">attachmentNode<\/span><span class=\"p\">.<\/span><span class=\"nx\">setAttribute<\/span><span class=\"p\">(<\/span><span class=\"s2\">&quot;src&quot;<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"nx\">attachment<\/span><span class=\"p\">.<\/span><span class=\"nx\">url<\/span><span class=\"p\">);<\/span>\n<\/span><span data-line=\"4\"><span class=\"w\">    <\/span><span class=\"nx\">attachmentNode<\/span><span class=\"p\">.<\/span><span class=\"nx\">setAttribute<\/span><span class=\"p\">(<\/span><span class=\"s2\">&quot;alt&quot;<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"nx\">attachment<\/span><span class=\"p\">.<\/span><span class=\"nx\">description<\/span><span class=\"p\">);<\/span>\n<\/span><span data-line=\"5\"><span class=\"w\">    <\/span><span class=\"nx\">attachmentNode<\/span><span class=\"p\">.<\/span><span class=\"nx\">classList<\/span><span class=\"p\">.<\/span><span class=\"nx\">add<\/span><span class=\"p\">(<\/span><span class=\"s2\">&quot;attachment&quot;<\/span><span class=\"p\">);<\/span>\n<\/span><span data-line=\"6\"><span class=\"w\">    <\/span><span class=\"nx\">content<\/span><span class=\"p\">.<\/span><span class=\"nx\">appendChild<\/span><span class=\"p\">(<\/span><span class=\"nx\">attachmentNode<\/span><span class=\"p\">);<\/span>\n<\/span><span data-line=\"7\"><span class=\"p\">}<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<p>On the one hand, this is a lot more complicated. But on the other, it means you don\u2019t need to worry about escaping\nstrings and it deals with a lot of possible content injection vectors for you. It actually made the resulting code\nsignificantly longer (43 lines -&gt; 96 lines, +123%), but the benefit is that it\u2019s much more legible.<\/p>\n<\/section>\n<section id=\"supporting-misskey-emoji\">\n<h3>Supporting Misskey Emoji<a class=\"headerlink\" href=\"#supporting-misskey-emoji\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>In versions 0.8.0 and 0.8.1, I began to add support for custom Misskey emoji. Their system significantly differns from\nthat of Mastodon, in that it does not include the URLs in the comment data itself. We have to fetch it individually\nevery single time.<\/p>\n<p>So I needed a way to fetch this. In order to do that efficiently, I also neede to convert <code class=\"docutils literal notranslate\"><span class=\"pre\">extractComment()<\/span><\/code> into an\nasync function so that multiple of these requests can be run at once. In order to reduce network traffic, I also needed\nto keep a cache of previously fetched URLs such that we don\u2019t repeatedly grab that. In later versions I would also add\nrate limit handling to each of these functions.<\/p>\n<section id=\"id7\">\n<h4>Other Misc Changes<a class=\"headerlink\" href=\"#id7\" title=\"Link to this heading\">\u00b6<\/a><\/h4>\n<ul class=\"simple\">\n<li><p>Support for Misskey\u2019s account-wide content warnings<\/p><\/li>\n<li><p>Fix Mastodon invisibility bug<\/p><\/li>\n<li><p>Reactions show up outside content warnings<\/p><\/li>\n<\/ul>\n<\/section>\n<\/section>\n<section id=\"sanitizing-html\">\n<h3>Sanitizing HTML<a class=\"headerlink\" href=\"#sanitizing-html\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>One of the last changes I\u2019ve made is to include code to sanitize HTML. In general, this means that it will block\nelements that can inject harmful content in your page. So if someone carefully crafted a Misskey comment that would\ninclude a <code class=\"docutils literal notranslate\"><span class=\"pre\">&lt;script&gt;<\/span><\/code> tag, and they did so in a way that would evade my simplistic <code class=\"docutils literal notranslate\"><span class=\"pre\">escapeHtml()<\/span><\/code> code, they would\nbe able to execute arbitrary code. This isn\u2019t a problem if it only exists on a demo site, but if it\u2019s going to get a\nstable release it <em>absolutely<\/em> needed to be addressed.<\/p>\n<p>To do this, I include a build of <a class=\"reference external\" href=\"https:\/\/www.npmjs.com\/package\/dompurify\">DOMPurify<\/a>. It is frankly inadvisable to\nwrite your own santiization code. You <em>will<\/em> get it wrong. So using a crowdsourced one is much safer. It will be better\ntested, it will be better examined, and the combination of those means it will be a lot more secure.<\/p>\n<\/section>\n<section id=\"modularizing-the-browser-code\">\n<h3>Modularizing the Browser Code<a class=\"headerlink\" href=\"#modularizing-the-browser-code\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>This is the most recent batch of changes, and in my opinion the coolest. I wanted to make it so that you only need to\ninclude the minimum amount of code needed to interace with your comment host. So I spun off the Mastodon and Misskey\ncode into their own files. I made it so that <code class=\"docutils literal notranslate\"><span class=\"pre\">marked.js<\/span><\/code> was only loaded for Misskey. And I reworked the control flow\nto keep the minimum amount in what I now thing of as the \u201cglue script,\u201d <code class=\"docutils literal notranslate\"><span class=\"pre\">fedi_script.js<\/span><\/code>. While doing this, I renamed\nsome functions to make it more in line with style guides.<\/p>\n<p>The first step was to remove the interface method <code class=\"docutils literal notranslate\"><span class=\"pre\">extractComment()<\/span><\/code> and replace it with something loaded on a\nper-implementation function. So <code class=\"docutils literal notranslate\"><span class=\"pre\">fedi_script_mastodon.js<\/span><\/code> has an implementation of <code class=\"docutils literal notranslate\"><span class=\"pre\">extractComment()<\/span><\/code>, and\n<code class=\"docutils literal notranslate\"><span class=\"pre\">fedi_script_misskey.js<\/span><\/code> has a different one. This way we don\u2019t need to have an intermediate funciton call, and we\ndon\u2019t need to embed a <code class=\"docutils literal notranslate\"><span class=\"pre\">fediFlavor<\/span><\/code> parameter.<\/p>\n<p>Next was to move the <code class=\"docutils literal notranslate\"><span class=\"pre\">fetch*()<\/span><\/code> methods into those implementations, which lets us further remove that <code class=\"docutils literal notranslate\"><span class=\"pre\">fediFlavor<\/span><\/code>\nparameter, and eliminate a large number of conditionals. API implementations are now entirely isolated to their own\nfiles. This reveals something interesting: it\u2019s much easier to support Mastodon. The Misskey file is <em>literally<\/em> twice\nas large. It has twice as many functions. It has about two thirds more lines.<\/p>\n<p>This also triggered a reorganization of the files. The Python and Javascript sources are now in their own\nsubdirectories. They each have their own tests now, as well, bringing code coverage up to 59% (84% for Python + 40% for\nJavaScript). This is a substantial increase in reliability.<\/p>\n<\/section>\n<\/section>\n<section id=\"future-work\">\n<h2>Future Work<a class=\"headerlink\" href=\"#future-work\" title=\"Link to this heading\">\u00b6<\/a><\/h2>\n<section id=\"before-a-stable-release\">\n<h3>Before a Stable Release<a class=\"headerlink\" href=\"#before-a-stable-release\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<section id=\"increasing-configurability\">\n<h4>Increasing Configurability<a class=\"headerlink\" href=\"#increasing-configurability\" title=\"Link to this heading\">\u00b6<\/a><\/h4>\n<p>At the moment, there are very few configuration options. It would be highly desirable to have more options,\n<em>especially<\/em> on a per-instance level. It would be nice to be able to change the post template with more customization.\nEven better would be to optionally translate the <code class=\"docutils literal notranslate\"><span class=\"pre\">:tags:<\/span><\/code> metadata field into a list of hashtags.<\/p>\n<p>I am very open to suggestions on what options should be added. Please provide feedback if you have any thoughts\nsuggestions.<\/p>\n<\/section>\n<section id=\"delaying-comment-loading\">\n<h4>Delaying Comment Loading<a class=\"headerlink\" href=\"#delaying-comment-loading\" title=\"Link to this heading\">\u00b6<\/a><\/h4>\n<p>Another suggestion I have received is to delay the loading of comments until they come into view. This seems feasible\nusing the <a class=\"reference external\" href=\"https:\/\/developer.mozilla.org\/en-US\/docs\/Web\/API\/Intersection_Observer_API\">Intersection Observer API<\/a>\nwith some fallback if that is not implemented (like in older versions of iOS). This means that you would have\nsignificantly lower overhead on initial page load, and if you never see the comments in the first place, you don\u2019t need\nto load them.<\/p>\n<p>This should, of course, be a feature that users can opt out of.<\/p>\n<\/section>\n<section id=\"improving-ease-of-commenting\">\n<h4>Improving Ease of Commenting<a class=\"headerlink\" href=\"#improving-ease-of-commenting\" title=\"Link to this heading\">\u00b6<\/a><\/h4>\n<p>There are two reasonable approaches here, and I would (in the long term) like to implement both. In order of priority:<\/p>\n<ol class=\"arabic simple\">\n<li><p>Add a form that will redirect you to your fedi instance<\/p><\/li>\n<li><p>Add a form that will allow anonymous comment submission through a self-hosted PHP script, with a captcha or <a class=\"reference external\" href=\"https:\/\/fosdem.org\/2025\/schedule\/event\/fosdem-2025-5879-maptcha-the-open-source-captcha-that-improves-openstreetmap\/\">maptcha<\/a> requirement.<\/p><\/li>\n<\/ol>\n<\/section>\n<\/section>\n<section id=\"other-fedi-implementations\">\n<h3>Other Fedi Implementations<a class=\"headerlink\" href=\"#other-fedi-implementations\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<section id=\"plemora-support\">\n<h4>Plemora Support<a class=\"headerlink\" href=\"#plemora-support\" title=\"Link to this heading\">\u00b6<\/a><\/h4>\n<p>Fortunately there is a <a class=\"reference external\" href=\"https:\/\/pypi.org\/project\/akkoma-py\/\">Python wrapper<\/a> for Akkoma, a flork of Plemora. This\nmeans the Python side will be easy to support. Akkoma also supports the Mastodon API (with some minor differences).\nMainline Plemora seems to have an API somewhat similar to Misskey, so it seems to me that much of it could be\nabstracted to share code there.<\/p>\n<\/section>\n<section id=\"lemmy-support\">\n<h4>Lemmy Support<a class=\"headerlink\" href=\"#lemmy-support\" title=\"Link to this heading\">\u00b6<\/a><\/h4>\n<p>Lemmy has a <a class=\"reference external\" href=\"https:\/\/pypi.org\/project\/Lemmy.py\/\">Python wrapper<\/a>, but it\u2019s a 0.0.2 release from 2 years ago. While I\nsuspect that it\u2019s stable enough for this purpose, that\u2019s not great and could break at any moment if Lemmy\u2019s API\nchanges. Their comment API is fairly well documented and shouldn\u2019t be that hard to deal with.<\/p>\n<\/section>\n<section id=\"gnu-social-support\">\n<h4>GNU Social Support<a class=\"headerlink\" href=\"#gnu-social-support\" title=\"Link to this heading\">\u00b6<\/a><\/h4>\n<p>Gnu Social has a <a class=\"reference external\" href=\"https:\/\/pypi.org\/project\/gnusocial\/\">Python wrapper<\/a>, which means post automation should be\ntrivial. That said, I can\u2019t find much information on their API. I assume that it would be not too difficult to parse,\npresuming I can find that.<\/p>\n<\/section>\n<section id=\"gotosocial-support\">\n<h4>GoToSocial Support<a class=\"headerlink\" href=\"#gotosocial-support\" title=\"Link to this heading\">\u00b6<\/a><\/h4>\n<p>While there is no Python wrapper to make posts directly, there <em>is<\/em> a <a class=\"reference external\" href=\"https:\/\/pypi.org\/project\/tootgroup.py\/\">library<\/a> that posts to it. Presumably one could recycle some of their internal\nfunctions to hook into, providing version pinning was enforced. Their reading API is well-documented and doesn\u2019t need\nauthentication, so it should be easy to read comments from that platform.<\/p>\n<\/section>\n<section id=\"friendica-support\">\n<h4>Friendica Support<a class=\"headerlink\" href=\"#friendica-support\" title=\"Link to this heading\">\u00b6<\/a><\/h4>\n<p>Friendica is in a similar situation, and would need to use the exact same library above.<\/p>\n<\/section>\n<section id=\"hubzilla-nextcloud-support\">\n<h4>Hubzilla, Nextcloud Support<a class=\"headerlink\" href=\"#hubzilla-nextcloud-support\" title=\"Link to this heading\">\u00b6<\/a><\/h4>\n<p>For these, I would almost certainly need to implement the Python API usage myself. Additionally, it is difficult to\nfind reliable documentation for their API, meaning it would need to be a very trial-and-error process. I would rather\nnot deal with that. If you have any resources that would be able to help here, it would be deeply appreciated.<\/p>\n<\/section>\n<section id=\"ghost-wordpress-plume-writefreely-etc\">\n<h4>Ghost, WordPress, Plume, WriteFreely, etc.<a class=\"headerlink\" href=\"#ghost-wordpress-plume-writefreely-etc\" title=\"Link to this heading\">\u00b6<\/a><\/h4>\n<p>I am explicitly ruling out the idea of posting to full blogging platforms. It\u2019s one thing to post to a link aggregator\nor microblogging service, but it feels somewhat pointless to make a post to something that could plausibly host your\nwhole article.<\/p>\n<\/section>\n<\/section>\n<section id=\"other-static-site-generators\">\n<h3>Other Static Site Generators<a class=\"headerlink\" href=\"#other-static-site-generators\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>Since most of this is implemented in JavaScript with easy and well-documented hooks, it seems to me that this could\neasily be ported to other static site generators.<\/p>\n<section id=\"mkdocs-pelican-nikola\">\n<h4>MkDocs, Pelican, &amp; Nikola<a class=\"headerlink\" href=\"#mkdocs-pelican-nikola\" title=\"Link to this heading\">\u00b6<\/a><\/h4>\n<p>These generators have the advantage of being written in Python. That means that a lot of the existing code could easily\nbe recycled to them, possibly even using the same package.<\/p>\n<\/section>\n<section id=\"eleventy-metalsmith\">\n<h4>Eleventy &amp; Metalsmith<a class=\"headerlink\" href=\"#eleventy-metalsmith\" title=\"Link to this heading\">\u00b6<\/a><\/h4>\n<p>These generators are written in server-side JavaScript. Given that I\u2019ve already had to come up with ways to import the\nbrowser code for testing purposes, it would be easy to recycle that (presuming good plugin support).<\/p>\n<\/section>\n<section id=\"zola\">\n<h4>Zola<a class=\"headerlink\" href=\"#zola\" title=\"Link to this heading\">\u00b6<\/a><\/h4>\n<p>This generator is written in Rust, which is a language I am slowly falling in love with. If it\u2019s feasible, this seems\nlike a plausible next target.<\/p>\n<p class=\"rubric\">Footnotes<\/p>\n<aside class=\"footnote-list brackets\">\n<aside class=\"footnote brackets\" id=\"id8\" role=\"doc-footnote\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span><a role=\"doc-backlink\" href=\"#id3\">1<\/a><span class=\"fn-bracket\">]<\/span><\/span>\n<p>Recent versions include other scripts as well. Everything includes <code class=\"docutils literal notranslate\"><span class=\"pre\">fedi_script.js<\/span><\/code> and <code class=\"docutils literal notranslate\"><span class=\"pre\">dompurify<\/span><\/code>.\nMastodon-hosted versions include <code class=\"docutils literal notranslate\"><span class=\"pre\">fedi_script_mastodon.js<\/span><\/code>. Misskey-hosted versions include <code class=\"docutils literal notranslate\"><span class=\"pre\">marked.js<\/span><\/code>\nand <code class=\"docutils literal notranslate\"><span class=\"pre\">fedi_script_misskey.js<\/span><\/code><\/p>\n<\/aside>\n<aside class=\"footnote brackets\" id=\"id9\" role=\"doc-footnote\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span><a role=\"doc-backlink\" href=\"#id4\">2<\/a><span class=\"fn-bracket\">]<\/span><\/span>\n<p>In initial versions this was only Mastodon, so it wasn\u2019t included. In the current version we only include the\nparsing code that you actually need, so it is no longer included<\/p>\n<\/aside>\n<aside class=\"footnote brackets\" id=\"id10\" role=\"doc-footnote\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span><a role=\"doc-backlink\" href=\"#id5\">3<\/a><span class=\"fn-bracket\">]<\/span><\/span>\n<p>Special thanks to <a class=\"reference external\" href=\"https:\/\/enby.life\/&#64;hazelnoot\">&#64;hazelnoot&#64;enby.life<\/a>!<\/p>\n<\/aside>\n<aside class=\"footnote brackets\" id=\"id11\" role=\"doc-footnote\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span><a role=\"doc-backlink\" href=\"#id6\">4<\/a><span class=\"fn-bracket\">]<\/span><\/span>\n<p><em>Character<\/em>, not byte. Emoji are multiple bytes, but fortunately JavaScript is smart enough to deal with that<\/p>\n<\/aside>\n<\/aside>\n<p class=\"rubric\">Citations<\/p>\n<div class=\"docutils container\" id=\"id12\">\n<div role=\"list\" class=\"citation-list\">\n<div class=\"citation\" id=\"id22\" role=\"doc-biblioentry\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span><a role=\"doc-backlink\" href=\"#id1\">Hal<\/a><span class=\"fn-bracket\">]<\/span><\/span>\n<p>Brian\u00a0\u201cBeej\u201d Hall. Mastodon comments. URL: <a class=\"reference external\" href=\"https:\/\/beej.us\/blog\/data\/mastodon-comments\/\">https:\/\/beej.us\/blog\/data\/mastodon-comments\/<\/a>.<\/p>\n<\/div>\n<div class=\"citation\" id=\"id26\" role=\"doc-biblioentry\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span><a role=\"doc-backlink\" href=\"#id1\">knz<\/a><span class=\"fn-bracket\">]<\/span><\/span>\n<p>knz. Mastodon comments. URL: <a class=\"reference external\" href=\"https:\/\/knz.ai\/posts\/2024-08-05-mastodon-comments\/\">https:\/\/knz.ai\/posts\/2024-08-05-mastodon-comments\/<\/a>.<\/p>\n<\/div>\n<div class=\"citation\" id=\"id23\" role=\"doc-biblioentry\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span><a role=\"doc-backlink\" href=\"#id1\">Maca<\/a><span class=\"fn-bracket\">]<\/span><\/span>\n<p>Heart\u00a0Soul Machine. Mastodon comments. URL: <a class=\"reference external\" href=\"https:\/\/heartsoulmachine.com\/blog\/2025\/01-12-mastodon-comments\/\">https:\/\/heartsoulmachine.com\/blog\/2025\/01-12-mastodon-comments\/<\/a>.<\/p>\n<\/div>\n<div class=\"citation\" id=\"id79\" role=\"doc-biblioentry\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span>Macb<span class=\"fn-bracket\">]<\/span><\/span>\n<span class=\"backrefs\">(<a role=\"doc-backlink\" href=\"#id1\">1<\/a>,<a role=\"doc-backlink\" href=\"#id2\">2<\/a>)<\/span>\n<p>Graham MacPhee. Mastodon blog comments. URL: <a class=\"reference external\" href=\"https:\/\/grahammacphee.com\/writing\/mastodon-blog-comments\">https:\/\/grahammacphee.com\/writing\/mastodon-blog-comments<\/a>.<\/p>\n<\/div>\n<div class=\"citation\" id=\"id78\" role=\"doc-biblioentry\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span><a role=\"doc-backlink\" href=\"#id1\">Pec<\/a><span class=\"fn-bracket\">]<\/span><\/span>\n<p>Daniel Pecos. Mastodon as comment system for your static blog. URL: <a class=\"reference external\" href=\"https:\/\/danielpecos.com\/2022\/12\/25\/mastodon-as-comment-system-for-your-static-blog\/\">https:\/\/danielpecos.com\/2022\/12\/25\/mastodon-as-comment-system-for-your-static-blog\/<\/a>.<\/p>\n<\/div>\n<div class=\"citation\" id=\"id27\" role=\"doc-biblioentry\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span><a role=\"doc-backlink\" href=\"#id1\">Scha<\/a><span class=\"fn-bracket\">]<\/span><\/span>\n<p>Andreas Scherbaum. Client-side comments with mastodon on a static hugo website. URL: <a class=\"reference external\" href=\"https:\/\/andreas.scherbaum.la\/post\/2024-05-23_client-side-comments-with-mastodon-on-a-static-hugo-website\/\">https:\/\/andreas.scherbaum.la\/post\/2024-05-23_client-side-comments-with-mastodon-on-a-static-hugo-website\/<\/a>.<\/p>\n<\/div>\n<div class=\"citation\" id=\"id81\" role=\"doc-biblioentry\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span><a role=\"doc-backlink\" href=\"#id1\">Schb<\/a><span class=\"fn-bracket\">]<\/span><\/span>\n<p>Carl Schwan. Adding comments to your static blog with mastodon. URL: <a class=\"reference external\" href=\"https:\/\/carlschwan.eu\/2020\/12\/29\/adding-comments-to-your-static-blog-with-mastodon\/\">https:\/\/carlschwan.eu\/2020\/12\/29\/adding-comments-to-your-static-blog-with-mastodon\/<\/a>.<\/p>\n<\/div>\n<div class=\"citation\" id=\"id76\" role=\"doc-biblioentry\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span><a role=\"doc-backlink\" href=\"#id1\">Sco<\/a><span class=\"fn-bracket\">]<\/span><\/span>\n<p>Laura\u00a0L. Scott. Mastodon comments for a hugo static site. URL: <a class=\"reference external\" href=\"https:\/\/lauralisscott.com\/blog\/mastodon-comments-hugo-static-site\/\">https:\/\/lauralisscott.com\/blog\/mastodon-comments-hugo-static-site\/<\/a>.<\/p>\n<\/div>\n<div class=\"citation\" id=\"id21\" role=\"doc-biblioentry\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span><a role=\"doc-backlink\" href=\"#id1\">Win<\/a><span class=\"fn-bracket\">]<\/span><\/span>\n<p>Winterstein. Static comments for a static website. URL: <a class=\"reference external\" href=\"https:\/\/www.winterstein.biz\/blog\/static-comments-for-static-website\/\">https:\/\/www.winterstein.biz\/blog\/static-comments-for-static-website\/<\/a>.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<p class=\"rubric\">Cite As<\/p>\n<details><summary>Click here to expand the bibtex entry.<\/summary><div class=\"docutils container\">\n<blockquote>\n<div><div class=\"highlight-bibtex notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"nc\">@online<\/span><span class=\"p\">{<\/span><span class=\"nl\">appleton_blog_0005<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"2\"><span class=\"w\">  <\/span><span class=\"na\">title<\/span><span class=\"w\">    <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ Developing sphinx-fediverse }<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"3\"><span class=\"w\">  <\/span><span class=\"na\">author<\/span><span class=\"w\">   <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ Olivia Appleton-Crocker }<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"4\"><span class=\"w\">  <\/span><span class=\"na\">language<\/span><span class=\"w\"> <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ English }<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"5\"><span class=\"w\">  <\/span><span class=\"na\">version<\/span><span class=\"w\">  <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ 1.1 }<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"6\"><span class=\"w\">  <\/span><span class=\"na\">date<\/span><span class=\"w\">     <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ 2025-03-24 }<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"7\"><span class=\"w\">  <\/span><span class=\"na\">url<\/span><span class=\"w\">      <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ https:\/\/blog.oliviaappleton.com\/posts\/0004-sphinx-fediverse }<\/span>\n<\/span><span data-line=\"8\"><span class=\"p\">}<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<\/div><\/blockquote>\n<\/div>\n<\/details><p class=\"tags\"><span>Tags: <\/span><a class=\"reference external\" href=\"..\/_tags\/sphinx.html\">sphinx<\/a><span>, <\/span><a class=\"reference external\" href=\"..\/_tags\/fediverse.html\">fediverse<\/a><span>, <\/span><a class=\"reference external\" href=\"..\/_tags\/fedibridge.html\">fedibridge<\/a><span>, <\/span><a class=\"reference external\" href=\"..\/_tags\/open-source.html\">open-source<\/a><span>, <\/span><a class=\"reference external\" href=\"..\/_tags\/project.html\">project<\/a><span>, <\/span><a class=\"reference external\" href=\"..\/_tags\/programming.html\">programming<\/a><span>, <\/span><a class=\"reference external\" href=\"..\/_tags\/python.html\">python<\/a><span>, <\/span><a class=\"reference external\" href=\"..\/_tags\/javascript.html\">javascript<\/a><span>, <\/span><a class=\"reference external\" href=\"..\/_tags\/html.html\">html<\/a><span>, <\/span><a class=\"reference external\" href=\"..\/_tags\/css.html\">css<\/a><\/p>\n\n            <h2>\n                Comments\n                <span class=\"comments-info\">\n                    <img class=\"fedi-icon\" src=\"https:\/\/blog.oliviaappleton.com\/_static\/boost.svg\" alt=\"Boosts\">\n                    <span id=\"global-reblogs\"><\/span>,\n                    <img class=\"fedi-icon\" src=\"https:\/\/blog.oliviaappleton.com\/_static\/like.svg\" alt=\"Likes\">\n                    <span id=\"global-likes\"><\/span>\n                <\/span>\n            <\/h2>\n            <div id=\"comments-section\"><\/div>\n            <script>\n                document.addEventListener(\"DOMContentLoaded\", function () {\n                    setConfig({\n                        boostLink: \"https:\/\/blog.oliviaappleton.com\/_static\/boost.svg\",\n                        allowSensitiveEmoji: false,\n                        allowCustomEmoji: true,\n                        allowMediaAttachments: true,\n                        allowAvatars: true,\n                        delayCommentLoad: true,\n                        defaultReactionEmoji: '\u2764',\n                        retryDelay: 100,\n                    });\n                    fetchComments('misskey', 'transfem.social', 'a5r163xf5g5j0rb0', 5);\n                });\n            <\/script>\n        <\/section>\n<\/section>\n<\/section>\n<\/section>\n","guid":"https:\/\/blog.oliviaappleton.com\/posts\/0005-sphinx-fediverse.html","category":["sphinx","fediverse","fedibridge","open-source","project","programming","python","javascript","html","css"],"pubDate":"Mon, 24 Mar 2025 00:00:00 +0000"},{"title":"Achieving Causality with Physical Clocks \u2013 A Summary","link":"https:\/\/blog.oliviaappleton.com\/posts\/0004-PWC.html","description":"<a rel='me' href='https:\/\/transfem.social\/@LivInTheLookingGlass' role='none' class='invisible'><\/a>\n<script>\n    window.goatcounter = {\n        path: function() {\n            let p = location.pathname;\n            if (p == '\/') {\n                p = '\/index.html';\n            }\n            return '\/blog' + p;\n        }\n    }\n<\/script>\n<script data-goatcounter='https:\/\/livinthelookingglass.goatcounter.com\/count'\n async src='\/\/gc.zgo.at\/count.js'><\/script><section id=\"achieving-causality-with-physical-clocks-a-summary\">\n<h1>Achieving Causality with Physical Clocks \u2013 A Summary<a class=\"headerlink\" href=\"#achieving-causality-with-physical-clocks-a-summary\" title=\"Link to this heading\">\u00b6<\/a><\/h1>\n<p><span class=\"date\">2025-03-17<\/span><\/p>\n<p>During my time at MSU, I had one <a class=\"reference external\" href=\"https:\/\/doi.org\/10.1145\/3491003.3491009\">published paper<\/a> that mattered to me. It\u2019s\na continuation of the HLC paper (<a class=\"reference external\" href=\"\/posts\/0001-HLC-review.html\">my post here<\/a>). If you haven\u2019t read that, I highly\nrecommend doing so before continuing.<\/p>\n<p>As a review, the Hybrid Logical Clock encodes a form of logical time alongside the physical time information. This\nallows consumers that are aware of this encoding to enforce causal guarantees. However, it has limitations that are\nsummarized in the table below, which this paper addresses.<\/p>\n<div class=\"table-wrapper docutils container\">\n<table class=\"docutils align-default\">\n<thead>\n<tr class=\"row-odd\"><th class=\"head\"><p><strong>Guarantees<\/strong><\/p><\/th>\n<th class=\"head\"><p><strong>Hybrid Logical Clock (HLC)<\/strong><\/p><\/th>\n<th class=\"head\"><p><strong>Physical clocks W\/ Causality (PWC)<\/strong><\/p><\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr class=\"row-even\"><td><p>Very near to NTP time<\/p><\/td>\n<td><p>\u2705<\/p><\/td>\n<td><p>\u2705<\/p><\/td>\n<\/tr>\n<tr class=\"row-odd\"><td><p>Uses <span class=\"math notranslate nohighlight\">\\(\\mathcal{O}(1)\\)<\/span> space<\/p><\/td>\n<td><p>\u2705<\/p><\/td>\n<td><p>\u2705<\/p><\/td>\n<\/tr>\n<tr class=\"row-even\"><td><p>Encodes logical time<\/p><\/td>\n<td><p>\u2705<\/p><\/td>\n<td><p>\u2705<\/p><\/td>\n<\/tr>\n<tr class=\"row-odd\"><td><p>Self-stabilizing<\/p><\/td>\n<td><p>\u2705<\/p><\/td>\n<td><p>\u2705<\/p><\/td>\n<\/tr>\n<tr class=\"row-even\"><td><p>Backwards Compatible with NTP<\/p><\/td>\n<td><p>\u2705<\/p><\/td>\n<td><p>\u2705<\/p><\/td>\n<\/tr>\n<tr class=\"row-odd\"><td><p>Tolerates large upticks in clock skew<\/p><\/td>\n<td><p>\u2705<\/p><\/td>\n<td><p>\u2705<\/p><\/td>\n<\/tr>\n<tr class=\"row-even\"><td><p>Allows logical time comparision\nwithout decoding<\/p><\/td>\n<td><p>\u274c<\/p><\/td>\n<td><p>\u2705<\/p><\/td>\n<\/tr>\n<tr class=\"row-odd\"><td><p>Configurable physical precision<\/p><\/td>\n<td><p>\u274c<\/p><\/td>\n<td><p>\u2705<\/p><\/td>\n<\/tr>\n<tr class=\"row-even\"><td><p>On average better physical precision<\/p><\/td>\n<td><p>\u274c<\/p><\/td>\n<td><p>\u2705<\/p><\/td>\n<\/tr>\n<tr class=\"row-odd\"><td><p>Tolerates non-monotonic clocks<\/p><\/td>\n<td><p>\u274c<\/p><\/td>\n<td><p>\u2705<\/p><\/td>\n<\/tr>\n<tr class=\"row-even\"><td><p>Tolerates non-constant logical-part\nsize<\/p><\/td>\n<td><p>\u274c<\/p><\/td>\n<td><p>\u2705<\/p><\/td>\n<\/tr>\n<tr class=\"row-odd\"><td><p>Allows direct comparison of\nwall clock time<\/p><\/td>\n<td><p>\u2705<\/p><\/td>\n<td><p>\u274c <a class=\"footnote-reference brackets\" href=\"#id3\" id=\"id1\" role=\"doc-noteref\"><span class=\"fn-bracket\">[<\/span>1<span class=\"fn-bracket\">]<\/span><\/a><\/p><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<section id=\"the-concept\">\n<h2>The Concept<a class=\"headerlink\" href=\"#the-concept\" title=\"Link to this heading\">\u00b6<\/a><\/h2>\n<p>In the previous paper, they describe a system where the clock maintains three parameters:<\/p>\n<ul class=\"simple\">\n<li><p><span class=\"math notranslate nohighlight\">\\(l\\)<\/span>, a logical clock that\u2019s tied to the physical time<\/p><\/li>\n<li><p><span class=\"math notranslate nohighlight\">\\(l - pt\\)<\/span>, the difference between that and the physical time at the originating node<\/p><\/li>\n<li><p><span class=\"math notranslate nohighlight\">\\(c\\)<\/span>, a counter that acts as a <a class=\"reference external\" href=\"https:\/\/en.wikipedia.org\/wiki\/Lamport_timestamp\">Lamport Clock<\/a><\/p><\/li>\n<\/ul>\n<p>This paper simplifies that significantly by eliminating the <span class=\"math notranslate nohighlight\">\\(l - pt\\)<\/span> parameter. Doing this allows us to dedicate\nsignificantly more memory to the counter portion while also having more precision in the <span class=\"math notranslate nohighlight\">\\(l\\)<\/span> parameter. In\naddition, this lets <span class=\"math notranslate nohighlight\">\\(l\\)<\/span> be significantly closer to the physical time. Note that in this new model there are\ndifferent names assigned to these parameters.<\/p>\n<p>From here on, <span class=\"math notranslate nohighlight\">\\(pwc\\)<\/span> shall refer to the entire encoded timestamp. <span class=\"math notranslate nohighlight\">\\(clpt\\)<\/span> shall refer to the timestamp where\nthe least-significant <span class=\"math notranslate nohighlight\">\\(u\\)<\/span> bits are masked to 0. Under this framing, <span class=\"math notranslate nohighlight\">\\(lpt = pwc - clpt\\)<\/span> is analagous to\n<span class=\"math notranslate nohighlight\">\\(c\\)<\/span> in the above description. When we were writing it, I pronounced that as \u201cclean point\u201d and \u201cel point,\u201d though\nthis terminology was not officially adopted. Additionally we have\n<span class=\"math notranslate nohighlight\">\\(hpt = \\left\\lfloor\\tfrac{pwc}{2^u}\\right\\rfloor\\)<\/span> (pronounced \u201chigh point\u201d).<\/p>\n<p>So to review, we have a structure of either <a class=\"reference external\" href=\"https:\/\/en.wikipedia.org\/wiki\/Network_Time_Protocol#Timestamps\">64 (NTP) or 128 bits (NTPv4)<\/a> that is arranged as:<\/p>\n<div class=\"highlight-default notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"o\">|-------------<\/span> <span class=\"n\">pwc<\/span> <span class=\"o\">------------|<\/span>\n<\/span><span data-line=\"2\"><span class=\"o\">+------------------------------+<\/span>\n<\/span><span data-line=\"3\"><span class=\"o\">|<\/span> <span class=\"n\">hpt<\/span>               <span class=\"o\">|<\/span> <span class=\"n\">lpt<\/span>      <span class=\"o\">|<\/span>\n<\/span><span data-line=\"4\"><span class=\"o\">+-------------------+----------+<\/span>\n<\/span><span data-line=\"5\">                    <span class=\"o\">|-<\/span> <span class=\"n\">u<\/span> <span class=\"n\">bits<\/span> <span class=\"o\">-|<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<\/section>\n<section id=\"the-algorithm\">\n<h2>The Algorithm<a class=\"headerlink\" href=\"#the-algorithm\" title=\"Link to this heading\">\u00b6<\/a><\/h2>\n<p>The algorithm at the heart of this looks very similar to that of the previous paper, except with less burden on\nencoding. If you are generating a local event or sending a message, then:<\/p>\n<div class=\"highlight-python notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"k\">def<\/span><span class=\"w\"> <\/span><span class=\"nf\">on_send<\/span><span class=\"p\">(<\/span><span class=\"n\">pwc<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span><span class=\"p\">,<\/span> <span class=\"n\">u<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">int<\/span><span class=\"p\">:<\/span>\n<\/span><span data-line=\"2\">    <span class=\"n\">mask<\/span> <span class=\"o\">=<\/span> <span class=\"nb\">pow<\/span><span class=\"p\">(<\/span><span class=\"mi\">2<\/span><span class=\"p\">,<\/span> <span class=\"mi\">64<\/span><span class=\"p\">)<\/span> <span class=\"o\">-<\/span> <span class=\"nb\">pow<\/span><span class=\"p\">(<\/span><span class=\"mi\">2<\/span><span class=\"p\">,<\/span> <span class=\"n\">u<\/span><span class=\"p\">)<\/span>\n<\/span><span data-line=\"3\">    <span class=\"n\">clpt<\/span> <span class=\"o\">=<\/span> <span class=\"n\">physical_clock<\/span><span class=\"p\">()<\/span> <span class=\"o\">&amp;<\/span> <span class=\"n\">mask<\/span>\n<\/span><span data-line=\"4\">    <span class=\"k\">return<\/span> <span class=\"nb\">max<\/span><span class=\"p\">((<\/span><span class=\"n\">clpt<\/span><span class=\"p\">,<\/span> <span class=\"n\">pwc<\/span> <span class=\"o\">+<\/span> <span class=\"mi\">1<\/span><span class=\"p\">))<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<p>And on receipt of a message:<\/p>\n<div class=\"highlight-python notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"k\">def<\/span><span class=\"w\"> <\/span><span class=\"nf\">on_receive<\/span><span class=\"p\">(<\/span><span class=\"n\">pwc<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span><span class=\"p\">,<\/span> <span class=\"n\">m_pwc<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span><span class=\"p\">,<\/span> <span class=\"n\">u<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">int<\/span><span class=\"p\">:<\/span>\n<\/span><span data-line=\"2\">    <span class=\"n\">mask<\/span> <span class=\"o\">=<\/span> <span class=\"nb\">pow<\/span><span class=\"p\">(<\/span><span class=\"mi\">2<\/span><span class=\"p\">,<\/span> <span class=\"mi\">64<\/span><span class=\"p\">)<\/span> <span class=\"o\">-<\/span> <span class=\"nb\">pow<\/span><span class=\"p\">(<\/span><span class=\"mi\">2<\/span><span class=\"p\">,<\/span> <span class=\"n\">u<\/span><span class=\"p\">)<\/span>\n<\/span><span data-line=\"3\">    <span class=\"n\">clpt<\/span> <span class=\"o\">=<\/span> <span class=\"n\">physical_clock<\/span><span class=\"p\">()<\/span> <span class=\"o\">&amp;<\/span> <span class=\"n\">mask<\/span>\n<\/span><span data-line=\"4\">    <span class=\"k\">return<\/span> <span class=\"nb\">max<\/span><span class=\"p\">((<\/span><span class=\"n\">clpt<\/span><span class=\"p\">,<\/span> <span class=\"n\">pwc<\/span> <span class=\"o\">+<\/span> <span class=\"mi\">1<\/span><span class=\"p\">,<\/span> <span class=\"n\">m_pwc<\/span> <span class=\"o\">+<\/span> <span class=\"mi\">1<\/span><span class=\"p\">))<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<p>Note just how much simpler this is! The whole thing can be boiled down to a couple of conditionals, bitwise operations,\nand an addition. In fact, it would be trivial to write this in assembly. Clang is able to boil it down to a mere 36\ninstructions (including 1 function call to get the NTP time). By comparison, the HLC receipt algorithm takes 52\ninstructions (44% more). It has <em>half<\/em> the <code class=\"docutils literal notranslate\"><span class=\"pre\">if<\/span><\/code>s that HLC requires. Plus it has the added benefit that the bit\noffsets are configurable, unlike in HLC where they are fixed at compile-time.<\/p>\n<\/section>\n<section id=\"determining-system-parameters\">\n<h2>Determining System Parameters<a class=\"headerlink\" href=\"#determining-system-parameters\" title=\"Link to this heading\">\u00b6<\/a><\/h2>\n<p>Now, something you might have been wondering thus far is \u201chow do I determine the value of <span class=\"math notranslate nohighlight\">\\(u\\)<\/span>?\u201d<\/p>\n<p>As it turns out, this is going to greatly depend on your expected system layout. Mainly it depends on:<\/p>\n<ul class=\"simple\">\n<li><p>max expected clock skew (difference between highest and lowest time)<\/p><\/li>\n<li><p>max expected message delay<\/p><\/li>\n<li><p>max expected event rate<\/p><\/li>\n<li><p>max necessary physical time granularity<\/p><\/li>\n<li><p>network topology<\/p><\/li>\n<\/ul>\n<section id=\"clock-skew\">\n<h3>Clock Skew<a class=\"headerlink\" href=\"#clock-skew\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>In essence, clock skew is the amount of disagreement in physical time on the network. This can be because of hardware\ndifferences (like if your clock ticks at a slightly different rate) or because of configuration differences (like if\nyour clock ticks correctly, but 10ms in the future). <a class=\"footnote-reference brackets\" href=\"#id4\" id=\"id2\" role=\"doc-noteref\"><span class=\"fn-bracket\">[<\/span>2<span class=\"fn-bracket\">]<\/span><\/a><\/p>\n<p>In practice, we measure clock skew by setting the maximum expected difference between two clocks on a network. You want\nthis to be an overestimate, but as conservative of one as you can manage. So for instance, if the average skew is 1ms,\nbut you know that it can be as high as 10ms 0.1% of the time, you should settle on 10ms.<\/p>\n<\/section>\n<section id=\"message-delay\">\n<h3>Message Delay<a class=\"headerlink\" href=\"#message-delay\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>In this paper, the delay between one node sending a message and the other node receiving it is divided into three parts:<\/p>\n<ol class=\"arabic simple\">\n<li><p><span class=\"math notranslate nohighlight\">\\(\\delta_{se}\\)<\/span>, which is the time it takes to get the message onto the networking device<\/p><\/li>\n<li><p><span class=\"math notranslate nohighlight\">\\(\\delta_{tr}\\)<\/span>, which is the time it takes to get the message to the other node\u2019s networking device<\/p><\/li>\n<li><p><span class=\"math notranslate nohighlight\">\\(\\delta_{re}\\)<\/span>, which is the time it takes for the message to get off the network card and into working memory<\/p><\/li>\n<\/ol>\n<p>Each of these is assumed to be greater than 0, and are a significant factor in determining your value of <span class=\"math notranslate nohighlight\">\\(u\\)<\/span>.<\/p>\n<\/section>\n<section id=\"time-granularity\">\n<h3>Time Granularity<a class=\"headerlink\" href=\"#time-granularity\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>The trade-off that you make when you increase <span class=\"math notranslate nohighlight\">\\(u\\)<\/span> is that you are reducing the physical-time granularity of your\nservice. If you absolutely need to discern between events separated by 10ns, for instance, you would need to have at\n<span class=\"math notranslate nohighlight\">\\(u \\le 5\\)<\/span>. Most systems don\u2019t have needs quite that strict, however. Anything that is dealing with ordering\nhuman-generated events will likely need at most 0.5ms granularity, which allows for <span class=\"math notranslate nohighlight\">\\(u \\le 21\\)<\/span>.<\/p>\n<\/section>\n<section id=\"final-equation\">\n<h3>Final Equation<a class=\"headerlink\" href=\"#final-equation\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>My proudest work in this paper was on the simulations, so I am going to be somewhat detailed in the findings from that\nportion. We found that in order to have the maximum time granularity without having wait periods due to the\ntimestamping system, one should set:<\/p>\n<div class=\"math-wrapper docutils container\">\n<div class=\"math notranslate nohighlight\">\n\\[u \\ge \\left\\lceil \\left( \\log_2\\left( \\dfrac{1000 \\cdot S^2}{\\min(\\delta_{re}, \\delta_{se})} \\right) - \\dfrac{\\log_2(\\epsilon)}{\\log_2(S+1)} \\right)K^{-1} \\right\\rceil\\]<\/div>\n<\/div>\n<p>Where:<\/p>\n<ul class=\"simple\">\n<li><p><span class=\"math notranslate nohighlight\">\\(K = 2.9 \\pm 0.1\\)<\/span> is a weighting constant<\/p><\/li>\n<li><p><span class=\"math notranslate nohighlight\">\\(S\\)<\/span> is the send rate in messages per node per millisecond<\/p><\/li>\n<li><p><span class=\"math notranslate nohighlight\">\\(\\epsilon\\)<\/span> is the max clock skew in milliseconds<\/p><\/li>\n<li><p><span class=\"math notranslate nohighlight\">\\(\\delta_{re}, \\delta_{se}\\)<\/span> are the message receive and message send delays in microseconds<\/p><\/li>\n<\/ul>\n<p>Note that this is significantly better than the naive limit (by about a factor of 3 in most cases), which states:<\/p>\n<div class=\"math-wrapper docutils container\">\n<div class=\"math notranslate nohighlight\">\n\\[2^u \\gt \\left\\lceil \\dfrac{\\epsilon}{\\min(\\delta_{loc},\\delta_{re},\\delta_{se})} \\right\\rceil\\]<\/div>\n<\/div>\n<p>It should also be noted that, while this is not explored thoroughly in the paper, this system can handle a <span class=\"math notranslate nohighlight\">\\(u\\)<\/span>\nparameter that changes over time. There just needs to be a mechanism for coordinating that change. And if some nodes\ndon\u2019t get the memo, you\u2019re still left with the same guarantees as the minimum <span class=\"math notranslate nohighlight\">\\(u\\)<\/span> parameter on the network.<\/p>\n<section id=\"effect-of-epsilon-clock-skew\">\n<h4>Effect of <span class=\"math notranslate nohighlight\">\\(\\epsilon\\)<\/span>, clock skew<a class=\"headerlink\" href=\"#effect-of-epsilon-clock-skew\" title=\"Link to this heading\">\u00b6<\/a><\/h4>\n<p>In general, we found that every <span class=\"math notranslate nohighlight\">\\(K\\)<\/span> doublings of clock skew, you need to add 1 bit to <span class=\"math notranslate nohighlight\">\\(u\\)<\/span>.<\/p>\n<\/section>\n<section id=\"effect-of-message-rate\">\n<h4>Effect of message rate<a class=\"headerlink\" href=\"#effect-of-message-rate\" title=\"Link to this heading\">\u00b6<\/a><\/h4>\n<p>An increase of the message rate from a small number (1 \/ node \/ ms) to a moderate amount (4 \/ node \/ ms) represents a\n<em>large<\/em> increase in the needed value of <span class=\"math notranslate nohighlight\">\\(u\\)<\/span>, but this effect diminishes rather quickly. In general, the higher\nthe message rate, the less important <span class=\"math notranslate nohighlight\">\\(\\epsilon\\)<\/span> is.<\/p>\n<\/section>\n<section id=\"effect-of-the-number-of-nodes\">\n<h4>Effect of the number of nodes<a class=\"headerlink\" href=\"#effect-of-the-number-of-nodes\" title=\"Link to this heading\">\u00b6<\/a><\/h4>\n<p>We weren\u2019t able to find any real impact from the number of nodes. To quote the paper:<\/p>\n<blockquote>\n<div><p>This is predicted by Equation 1 which is independent of the number of nodes. This also makes sense, because a node\nshould expect to see 1 in (N-1) messages from other nodes, and there are (N-1) other nodes sending messages, so these\nshould cancel out as observed.<\/p>\n<\/div><\/blockquote>\n<\/section>\n<section id=\"effect-of-network-topology\">\n<h4>Effect of network topology<a class=\"headerlink\" href=\"#effect-of-network-topology\" title=\"Link to this heading\">\u00b6<\/a><\/h4>\n<p>Network topology actually has a large impact not captured by the equations given in this paper. In general, 4\nconfigurations were tested:<\/p>\n<ol class=\"arabic simple\">\n<li><p>A random or mesh network, where nodes pick a destination to send to at random<\/p><\/li>\n<li><p>A hub-and-spoke network, where nodes send to a central hub that distributes messages<\/p><\/li>\n<li><p>A \u201cleader\u201d network, where one clock is arbitrarily ahead of the others, forcing a specific value of <span class=\"math notranslate nohighlight\">\\(\\epsilon\\)<\/span><\/p><\/li>\n<\/ol>\n<p>Mesh networks seem to be more resilient to changes in <span class=\"math notranslate nohighlight\">\\(\\epsilon\\)<\/span>. The leader network has a significantly higher\nbaseline value of <span class=\"math notranslate nohighlight\">\\(\\epsilon\\)<\/span> (as expected) and has a larger variation in minimum-acceptable <span class=\"math notranslate nohighlight\">\\(u\\)<\/span> values.<\/p>\n<p>A similar effect is seen in hub-and-spoke networks, likely because large minimum-acceptable <span class=\"math notranslate nohighlight\">\\(u\\)<\/span> values propogate\nmore readily. In a mesh network, if one node produces messages with high <span class=\"math notranslate nohighlight\">\\(lpt\\)<\/span> values, that will be distributed\nrandomly causing a slow propogation. In a hub-and-spoke network, if the hub node receives such a message it will be\ndistributed very quickly to all other nodes.<\/p>\n<p>In general, this time-keeping schema prefers decentralized networks.<\/p>\n<\/section>\n<\/section>\n<section id=\"rare-events\">\n<h3>Rare Events<a class=\"headerlink\" href=\"#rare-events\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>In the paper, I explicitly call out that some simulations showed quite rare events. In particular I <a class=\"reference external\" href=\"https:\/\/gist.github.com\/AnonymousPaperPwc\/39001b43cda3b5aa3a99783b0b418c74#file-simulation-data-collection-csv-L293\">point to one<\/a>\nwith the system parameters: <span class=\"math notranslate nohighlight\">\\(\\epsilon = 6.25\\text{ms}, N=8, S=64\\)<\/span>. There were about 736,000,000 events in that\nrun. Of those, about 731,000,000 needed 0 bits allocated to their logical portion. They would be just fine using purely\nphysical time. About 2,000,000 events merely needed 1 bit to distinguish them.<\/p>\n<p>In that simulation we were testing against an expectation of 6 bits being needed. It was able to handle more than that,\nof course, but we configured it to alert us to any events that would not fit within that space. 27,852 events did not\nfit, 0.0038% of the total messages.<\/p>\n<p>Only 142 events needed the full 9 bits that we recommend in the abstract. Less than 1 in 1 million.<\/p>\n<p>Below are several charts from the paper detailing these findings.<\/p>\n<a class=\"reference internal image-reference\" href=\"..\/_images\/exp_cum_percentage.png\"><img alt=\"A graph showing the cumulative percentage of messages that can be represented by a size of u, found by simulation. Both scales are linear\" src=\"..\/_images\/exp_cum_percentage.png\" style=\"width: 100%;\" \/>\n<\/a>\n<a class=\"reference internal image-reference\" href=\"..\/_images\/exp_raw_prob.png\"><img alt=\"A graph showing the probability of getting a message which requires u bits, found by simulation. It appears to be flat before dropping quickly, but the y axis is in log scale.\" src=\"..\/_images\/exp_raw_prob.png\" style=\"width: 100%;\" \/>\n<\/a>\n<a class=\"reference internal image-reference\" href=\"..\/_images\/exp_indv_prob.png\"><img alt=\"A graph showing the probability of getting a message with a given lpt value, found by simulation. It appears to drop linearly, but the y axis is in log scale, meaning it gets exponentially less likely as values go up\" src=\"..\/_images\/exp_indv_prob.png\" style=\"width: 100%;\" \/>\n<\/a>\n<\/section>\n<\/section>\n<section id=\"conclusions\">\n<h2>Conclusions<a class=\"headerlink\" href=\"#conclusions\" title=\"Link to this heading\">\u00b6<\/a><\/h2>\n<p>I personally believe that if you are designing a peer-to-peer protocol, this is the design you should be using for\ntime-keeping. Many of the alternatives are not feasible, either because they require specialized hardware or because\nthey incur too much of a performance cost.<\/p>\n<p>This encoding scheme is incredibly simple and preserves most of the benefits of using a physical clock while\nintegrating the benefits of a logical clock. Importantly, because you can use integer comparisons on the resulting\ntimestamps, it is a drop-in replacement for NTP. Your consumer applications do not need to know the details of this\nalgorithm unless they need more granularity than your networking service offers.<\/p>\n<p class=\"rubric\">Footnotes<\/p>\n<aside class=\"footnote-list brackets\">\n<aside class=\"footnote brackets\" id=\"id3\" role=\"doc-footnote\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span><a role=\"doc-backlink\" href=\"#id1\">1<\/a><span class=\"fn-bracket\">]<\/span><\/span>\n<p>Skewed by up to <span class=\"math notranslate nohighlight\">\\(\\epsilon\\)<\/span>, where <span class=\"math notranslate nohighlight\">\\(\\epsilon\\)<\/span> is max expected difference between two clocks on the\nnetwork + message send delay<\/p>\n<\/aside>\n<aside class=\"footnote brackets\" id=\"id4\" role=\"doc-footnote\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span><a role=\"doc-backlink\" href=\"#id2\">2<\/a><span class=\"fn-bracket\">]<\/span><\/span>\n<p>It also can vary based on your gravitational frame, but that\u2019s beyond the scope of this paper. In general,\nclocks will tick at different rates if you are moving very quickly or experience a different amount of gravity. This\nis negligible in most network configurations, and is corrected for by NTP.<\/p>\n<\/aside>\n<\/aside>\n<p class=\"rubric\">Citations<\/p>\n<span class=\"target\" id=\"id5\"><\/span><p class=\"rubric\">Cite As<\/p>\n<details><summary>Click here to expand the bibtex entry.<\/summary><div class=\"docutils container\">\n<blockquote>\n<div><div class=\"highlight-bibtex notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"nc\">@online<\/span><span class=\"p\">{<\/span><span class=\"nl\">appleton_blog_0004<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"2\"><span class=\"w\">  <\/span><span class=\"na\">title<\/span><span class=\"w\">    <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ Achieving Causality with Physical Clocks \u2014 A Summary }<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"3\"><span class=\"w\">  <\/span><span class=\"na\">author<\/span><span class=\"w\">   <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ Olivia Appleton-Crocker }<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"4\"><span class=\"w\">  <\/span><span class=\"na\">language<\/span><span class=\"w\"> <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ English }<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"5\"><span class=\"w\">  <\/span><span class=\"na\">version<\/span><span class=\"w\">  <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ 1.1 }<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"6\"><span class=\"w\">  <\/span><span class=\"na\">date<\/span><span class=\"w\">     <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ 2025-03-17 }<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"7\"><span class=\"w\">  <\/span><span class=\"na\">url<\/span><span class=\"w\">      <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ https:\/\/blog.oliviaappleton.com\/posts\/0004-PWC }<\/span>\n<\/span><span data-line=\"8\"><span class=\"p\">}<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<\/div><\/blockquote>\n<\/div>\n<\/details><p class=\"tags\"><span>Tags: <\/span><a class=\"reference external\" href=\"..\/_tags\/research-review.html\">research-review<\/a><span>, <\/span><a class=\"reference external\" href=\"..\/_tags\/msu.html\">msu<\/a><span>, <\/span><a class=\"reference external\" href=\"..\/_tags\/published-paper.html\">published-paper<\/a><span>, <\/span><a class=\"reference external\" href=\"..\/_tags\/distributed-systems.html\">distributed-systems<\/a><\/p>\n\n            <h2>\n                Comments\n                <span class=\"comments-info\">\n                    <img class=\"fedi-icon\" src=\"https:\/\/blog.oliviaappleton.com\/_static\/boost.svg\" alt=\"Boosts\">\n                    <span id=\"global-reblogs\"><\/span>,\n                    <img class=\"fedi-icon\" src=\"https:\/\/blog.oliviaappleton.com\/_static\/like.svg\" alt=\"Likes\">\n                    <span id=\"global-likes\"><\/span>\n                <\/span>\n            <\/h2>\n            <div id=\"comments-section\"><\/div>\n            <script>\n                document.addEventListener(\"DOMContentLoaded\", function () {\n                    setConfig({\n                        boostLink: \"https:\/\/blog.oliviaappleton.com\/_static\/boost.svg\",\n                        allowSensitiveEmoji: false,\n                        allowCustomEmoji: true,\n                        allowMediaAttachments: true,\n                        allowAvatars: true,\n                        delayCommentLoad: true,\n                        defaultReactionEmoji: '\u2764',\n                        retryDelay: 100,\n                    });\n                    fetchComments('misskey', 'transfem.social', 'a5gw39x43cjn034d', 5);\n                });\n            <\/script>\n        <\/section>\n<\/section>\n","guid":"https:\/\/blog.oliviaappleton.com\/posts\/0004-PWC.html","category":["research-review","msu","published-paper","distributed-systems"],"pubDate":"Mon, 17 Mar 2025 00:00:00 +0000"},{"title":"Kademlia (My Implementation)","link":"https:\/\/blog.oliviaappleton.com\/posts\/0003-Kademlia-implementation.html","description":"<a rel='me' href='https:\/\/transfem.social\/@LivInTheLookingGlass' role='none' class='invisible'><\/a>\n<script>\n    window.goatcounter = {\n        path: function() {\n            let p = location.pathname;\n            if (p == '\/') {\n                p = '\/index.html';\n            }\n            return '\/blog' + p;\n        }\n    }\n<\/script>\n<script data-goatcounter='https:\/\/livinthelookingglass.goatcounter.com\/count'\n async src='\/\/gc.zgo.at\/count.js'><\/script><section id=\"kademlia-my-implementation\">\n<h1>Kademlia (My Implementation)<a class=\"headerlink\" href=\"#kademlia-my-implementation\" title=\"Link to this heading\">\u00b6<\/a><\/h1>\n<p><span class=\"date\">2020-05-21<\/span><\/p>\n<p>If you haven\u2019t yet read it, please see my <a class=\"reference external\" href=\".\/0002-Kademlia-protocol.html\">overview of\nKademlia<\/a> <span id=\"id1\">[<a class=\"reference internal\" href=\"#id74\" title=\"Olivia Appleton-Crocker. Kademlia (the protocol). URL: https:\/\/blog.oliviaappleton.com\/posts\/0002-Kademlia-protocol.\">ACa<\/a>]<\/span>.<\/p>\n<p>Kademlia is obviously an idea that I have a fairly deep interest in at\nthis point, so I spent the last couple weeks writing up another\nimplementation of it. This one managed to solve many of the problems I\nhad in the past by using a slightly different concurrency model, by\nfocusing narrowly on UDP, and by using equivalent alternatives to some\nstructures defined in the Kademlia specification.<\/p>\n<p>What I want to do in this post is walk through how this implementation\nworks, what progress it makes on my larger goal of developing Retrion,\nand what problems it has that remain to be solved.<\/p>\n<p>Code can be found\n<a class=\"reference external\" href=\"https:\/\/github.com\/p2p-today\/retrion-development\">here<\/a> <span id=\"id2\">[<a class=\"reference internal\" href=\"#id73\" title=\"Olivia Appleton-Crocker. P2p-today\/retrion-development: a development repository for a future release of the retrion protocol. URL: https:\/\/github.com\/p2p-today\/retrion-development.\">ACb<\/a>]<\/span>, in the\nfiles that use protocol version 4. Version 5 is my current work to try\nand extend it.<\/p>\n<section id=\"how-it-works\">\n<h2>How It Works<a class=\"headerlink\" href=\"#how-it-works\" title=\"Link to this heading\">\u00b6<\/a><\/h2>\n<section id=\"concurrency-model\">\n<h3>Concurrency Model<a class=\"headerlink\" href=\"#concurrency-model\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>The node\u2019s world is divided into four threads (plus the main thread, but\nthat doesn\u2019t count). Threads 1 and 2 listen for incoming messages on\nUDP\/IPv4 and UDP\/IPv6 respectively. Thread 3, the reaction thread, is\nresponsible for reacting and responding to messages. Thread 4, the\nheartbeat thread, is responsible for scheduled events in the future. It\nimplements a sort of event loop using the\n<a class=\"reference external\" href=\"https:\/\/docs.python.org\/3\/library\/sched.html\">sched<\/a> module.<\/p>\n<p>The listener threads exist primarily out of laziness, since\n<a class=\"reference external\" href=\"https:\/\/docs.python.org\/3\/library\/select.html\">select<\/a> would have\neasily let me consolidate them. It should be noted, however, that having\nmultiple threads like this will allow for future abstraction. If each\nlistening socket gets its own thread, then you don\u2019t need to deal as\nmuch with how those transport methods differ. A websocket, for example,\nwould have a very different interface when compared with a UDP one or a\nBluetooth one.<\/p>\n<\/section>\n<section id=\"serialization\">\n<h3>Serialization<a class=\"headerlink\" href=\"#serialization\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>Serialization was done with\n<a class=\"reference external\" href=\"https:\/\/github.com\/vsergeev\/u-msgpack-python\">u-msgpack-python<\/a>,\nwhich is a particularly nice implementation of the\n<a class=\"reference external\" href=\"https:\/\/github.com\/msgpack\/msgpack\/blob\/master\/spec.md\">MessagePack<\/a>\nserialization scheme. It\u2019s easiest to think of it as \u201cJSON in binary,\nwith some customizability\u201d.<\/p>\n<p>Of particular usefulness, MessagePack allows you to define how custom\nobjects should be serialized and deserialized using their concept of\n\u201cextension objects\u201d. Basically they throw a header onto whatever\nbytestream you hand them and they can use that header to send the stream\nto the correct factory function. I\u2019m going to borrow their notation for\nhow to represent these bytestreams:<\/p>\n<div class=\"highlight-default notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"n\">one<\/span> <span class=\"n\">byte<\/span><span class=\"p\">:<\/span>\n<\/span><span data-line=\"2\"><span class=\"o\">+--------+<\/span>\n<\/span><span data-line=\"3\"><span class=\"o\">|<\/span>        <span class=\"o\">|<\/span>\n<\/span><span data-line=\"4\"><span class=\"o\">+--------+<\/span>\n<\/span><span data-line=\"5\">\n<\/span><span data-line=\"6\"><span class=\"n\">a<\/span> <span class=\"n\">variable<\/span> <span class=\"n\">number<\/span> <span class=\"n\">of<\/span> <span class=\"nb\">bytes<\/span><span class=\"p\">:<\/span>\n<\/span><span data-line=\"7\"><span class=\"o\">+========+<\/span>\n<\/span><span data-line=\"8\"><span class=\"o\">|<\/span>        <span class=\"o\">|<\/span>\n<\/span><span data-line=\"9\"><span class=\"o\">+========+<\/span>\n<\/span><span data-line=\"10\">\n<\/span><span data-line=\"11\"><span class=\"n\">variable<\/span> <span class=\"n\">number<\/span> <span class=\"n\">of<\/span> <span class=\"n\">objects<\/span> <span class=\"n\">stored<\/span> <span class=\"ow\">in<\/span> <span class=\"n\">MessagePack<\/span> <span class=\"nb\">format<\/span><span class=\"p\">:<\/span>\n<\/span><span data-line=\"12\"><span class=\"o\">+~~~~~~~~~~~~~~~~~+<\/span>\n<\/span><span data-line=\"13\"><span class=\"o\">|<\/span>                 <span class=\"o\">|<\/span>\n<\/span><span data-line=\"14\"><span class=\"o\">+~~~~~~~~~~~~~~~~~+<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<p>For an example of how this looks, let\u2019s show the PING message:<\/p>\n<div class=\"highlight-default notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"o\">+========+--------+--------+--------+--------+--------+========+========+<\/span>\n<\/span><span data-line=\"2\"><span class=\"o\">|<\/span> <span class=\"n\">header<\/span> <span class=\"o\">|<\/span>    <span class=\"mi\">0<\/span>   <span class=\"o\">|<\/span>    <span class=\"mi\">4<\/span>   <span class=\"o\">|<\/span><span class=\"n\">compress<\/span><span class=\"o\">|<\/span><span class=\"mi\">10010011<\/span><span class=\"o\">|<\/span>    <span class=\"mi\">1<\/span>   <span class=\"o\">|<\/span><span class=\"n\">sequence<\/span><span class=\"o\">|<\/span><span class=\"n\">senderID<\/span><span class=\"o\">|<\/span>\n<\/span><span data-line=\"3\"><span class=\"o\">+========+--------+--------+--------+--------+--------+========+========+<\/span>\n<\/span><span data-line=\"4\">                                    <span class=\"o\">|-------------<\/span><span class=\"n\">compressed<\/span><span class=\"o\">------------|<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<p>Let\u2019s look at that cell-by-cell. The header is a few bytes that\nMessagePack tacks on to the beginning. It essentially says \u201cI am an\nextension object __ bytes in length.\u201d<\/p>\n<p>The first value we actually give it is that 0, which is the extension\ntypecode I defined for Message objects.<\/p>\n<p>The next cell indicates the compression method used for the remainder of\nthe bytestream. This is negotiated between nodes in a HELLO message. The\nones currently defined are none, ZLIB, GZIP, LZMA, and BZ2, taking their\nvalues in that order.<\/p>\n<p>The next cell is a header for an array. The first four bits indicate\nthat it is an array, and the second four bits indicate the length of the\narray.<\/p>\n<p>The next cell is a 1, which is the message type assigned to PING.<\/p>\n<p>The next cell is the sequence number of the packet. This number should\nmonotonically increase and is assigned by the sending node. It is what\ngets referred to if a response gets sent back.<\/p>\n<p>The next cell is the sender ID. By default this is just 20 random bytes.\nIn future implementations it will likely be a hash of some node\ninformation in a canonical order.<\/p>\n<p>This entire array is unpacked and then handed to a factory function to\nregenerate that message object.<\/p>\n<\/section>\n<section id=\"reactions\">\n<h3>Reactions<a class=\"headerlink\" href=\"#reactions\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>When nodes reconstruct a message, they find that each have a <code class=\"docutils literal notranslate\"><span class=\"pre\">react()<\/span><\/code>\nmethod. For the simplest example, let\u2019s look at the IDENTIFY message:<\/p>\n<div class=\"highlight-python notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"k\">class<\/span><span class=\"w\"> <\/span><span class=\"nc\">Message<\/span><span class=\"p\">(<\/span><span class=\"n\">BaseMessage<\/span><span class=\"p\">):<\/span>\n<\/span><span data-line=\"2\">    <span class=\"o\">...<\/span>\n<\/span><span data-line=\"3\">\n<\/span><span data-line=\"4\">    <span class=\"k\">def<\/span><span class=\"w\"> <\/span><span class=\"nf\">react<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">node<\/span><span class=\"p\">,<\/span> <span class=\"n\">addr<\/span><span class=\"p\">,<\/span> <span class=\"n\">sock<\/span><span class=\"p\">):<\/span>\n<\/span><span data-line=\"5\"><span class=\"w\">        <\/span><span class=\"sd\">&quot;&quot;&quot;Delay sending a PING to the sending node, since the connection is clearly active if you got a message.&quot;&quot;&quot;<\/span>\n<\/span><span data-line=\"6\">        <span class=\"k\">try<\/span><span class=\"p\">:<\/span>\n<\/span><span data-line=\"7\">            <span class=\"n\">event<\/span> <span class=\"o\">=<\/span> <span class=\"n\">node<\/span><span class=\"o\">.<\/span><span class=\"n\">timeouts<\/span><span class=\"o\">.<\/span><span class=\"n\">pop<\/span><span class=\"p\">((<\/span><span class=\"bp\">self<\/span><span class=\"o\">.<\/span><span class=\"n\">sender<\/span><span class=\"p\">,<\/span> <span class=\"p\">))<\/span>\n<\/span><span data-line=\"8\">            <span class=\"n\">node<\/span><span class=\"o\">.<\/span><span class=\"n\">schedule<\/span><span class=\"o\">.<\/span><span class=\"n\">cancel<\/span><span class=\"p\">(<\/span><span class=\"n\">event<\/span><span class=\"p\">)<\/span>\n<\/span><span data-line=\"9\">        <span class=\"k\">except<\/span> <span class=\"p\">(<\/span><span class=\"ne\">KeyError<\/span><span class=\"p\">,<\/span> <span class=\"ne\">ValueError<\/span><span class=\"p\">):<\/span>\n<\/span><span data-line=\"10\">            <span class=\"k\">pass<\/span>\n<\/span><span data-line=\"11\">\n<\/span><span data-line=\"12\">        <span class=\"k\">def<\/span><span class=\"w\"> <\/span><span class=\"nf\">ping<\/span><span class=\"p\">():<\/span>\n<\/span><span data-line=\"13\">            <span class=\"n\">node<\/span><span class=\"o\">.<\/span><span class=\"n\">_send<\/span><span class=\"p\">(<\/span><span class=\"n\">sock<\/span><span class=\"p\">,<\/span> <span class=\"n\">addr<\/span><span class=\"p\">,<\/span> <span class=\"n\">PingMessage<\/span><span class=\"p\">())<\/span>\n<\/span><span data-line=\"14\">\n<\/span><span data-line=\"15\">        <span class=\"n\">node<\/span><span class=\"o\">.<\/span><span class=\"n\">timeouts<\/span><span class=\"p\">[(<\/span><span class=\"bp\">self<\/span><span class=\"o\">.<\/span><span class=\"n\">sender<\/span><span class=\"p\">,<\/span> <span class=\"p\">)]<\/span> <span class=\"o\">=<\/span> <span class=\"n\">node<\/span><span class=\"o\">.<\/span><span class=\"n\">schedule<\/span><span class=\"o\">.<\/span><span class=\"n\">enter<\/span><span class=\"p\">(<\/span><span class=\"mi\">60<\/span><span class=\"p\">,<\/span> <span class=\"mi\">2<\/span><span class=\"p\">,<\/span> <span class=\"n\">ping<\/span><span class=\"p\">)<\/span>\n<\/span><span data-line=\"16\">        <span class=\"c1\"># enter() takes in delay, priority, callable<\/span>\n<\/span><span data-line=\"17\">\n<\/span><span data-line=\"18\"><span class=\"nd\">@Message<\/span><span class=\"o\">.<\/span><span class=\"n\">register<\/span><span class=\"p\">(<\/span><span class=\"nb\">int<\/span><span class=\"p\">(<\/span><span class=\"n\">MessageType<\/span><span class=\"o\">.<\/span><span class=\"n\">IDENTIFY<\/span><span class=\"p\">))<\/span>\n<\/span><span data-line=\"19\"><span class=\"k\">class<\/span><span class=\"w\"> <\/span><span class=\"nc\">IdentifyMessage<\/span><span class=\"p\">(<\/span><span class=\"n\">PingMessage<\/span><span class=\"p\">):<\/span>\n<\/span><span data-line=\"20\">    <span class=\"k\">def<\/span><span class=\"w\"> <\/span><span class=\"fm\">__init__<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">compress<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span> <span class=\"o\">=<\/span> <span class=\"mi\">0<\/span><span class=\"p\">,<\/span> <span class=\"n\">seq<\/span><span class=\"p\">:<\/span> <span class=\"n\">Optional<\/span><span class=\"p\">[<\/span><span class=\"nb\">int<\/span><span class=\"p\">]<\/span> <span class=\"o\">=<\/span> <span class=\"kc\">None<\/span><span class=\"p\">,<\/span> <span class=\"n\">sender<\/span><span class=\"p\">:<\/span> <span class=\"nb\">bytes<\/span> <span class=\"o\">=<\/span> <span class=\"sa\">b<\/span><span class=\"s1\">&#39;&#39;<\/span><span class=\"p\">):<\/span>\n<\/span><span data-line=\"21\">        <span class=\"nb\">super<\/span><span class=\"p\">()<\/span><span class=\"o\">.<\/span><span class=\"fm\">__init__<\/span><span class=\"p\">(<\/span><span class=\"n\">compress<\/span><span class=\"p\">,<\/span> <span class=\"n\">seq<\/span><span class=\"p\">,<\/span> <span class=\"n\">sender<\/span><span class=\"p\">)<\/span>\n<\/span><span data-line=\"22\">        <span class=\"bp\">self<\/span><span class=\"o\">.<\/span><span class=\"n\">message_type<\/span> <span class=\"o\">=<\/span> <span class=\"n\">MessageType<\/span><span class=\"o\">.<\/span><span class=\"n\">IDENTIFY<\/span>\n<\/span><span data-line=\"23\">\n<\/span><span data-line=\"24\">    <span class=\"k\">def<\/span><span class=\"w\"> <\/span><span class=\"nf\">react<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">node<\/span><span class=\"p\">,<\/span> <span class=\"n\">addr<\/span><span class=\"p\">,<\/span> <span class=\"n\">sock<\/span><span class=\"p\">):<\/span>\n<\/span><span data-line=\"25\"><span class=\"w\">        <\/span><span class=\"sd\">&quot;&quot;&quot;Send a HELLO back in leiu of an ACK.&quot;&quot;&quot;<\/span>\n<\/span><span data-line=\"26\">        <span class=\"n\">node<\/span><span class=\"o\">.<\/span><span class=\"n\">logger<\/span><span class=\"o\">.<\/span><span class=\"n\">debug<\/span><span class=\"p\">(<\/span><span class=\"s2\">&quot;Got an IDENTIFY request from <\/span><span class=\"si\">%r<\/span><span class=\"s2\"> (<\/span><span class=\"si\">%r<\/span><span class=\"s2\">)&quot;<\/span><span class=\"p\">,<\/span> <span class=\"n\">addr<\/span><span class=\"p\">,<\/span> <span class=\"bp\">self<\/span><span class=\"p\">)<\/span>\n<\/span><span data-line=\"27\">        <span class=\"n\">node<\/span><span class=\"o\">.<\/span><span class=\"n\">_send_hello<\/span><span class=\"p\">(<\/span><span class=\"n\">sock<\/span><span class=\"p\">,<\/span> <span class=\"n\">addr<\/span><span class=\"p\">)<\/span>\n<\/span><span data-line=\"28\">        <span class=\"n\">Message<\/span><span class=\"o\">.<\/span><span class=\"n\">react<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">node<\/span><span class=\"p\">,<\/span> <span class=\"n\">addr<\/span><span class=\"p\">,<\/span> <span class=\"n\">sock<\/span><span class=\"p\">)<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<p>A node who sends IDENTIFY is saying \u201chey, I have your address but I\ndon\u2019t know who you are. Would you mind introducing yourself?\u201d So this\nevent gets processed in two phases. The first is just the node sending a\nHELLO message. The second part is to modify the node\u2019s schedule. By\ndefault, nodes will send a PING after 60 seconds of dead air to make\nsure that their peer is still alive. If you get a message this timer\nneeds to be reset, which is what happens in <code class=\"docutils literal notranslate\"><span class=\"pre\">Message.react()<\/span><\/code>.<\/p>\n<\/section>\n<section id=\"responses\">\n<h3>Responses<a class=\"headerlink\" href=\"#responses\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>Several messages also have a method called <code class=\"docutils literal notranslate\"><span class=\"pre\">react_response()<\/span><\/code>. The\nidea here is that if you received an ACK, probably that means something\nto some other message. The way this gets triggered looks like:<\/p>\n<div class=\"highlight-python notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"nd\">@Message<\/span><span class=\"o\">.<\/span><span class=\"n\">register<\/span><span class=\"p\">(<\/span><span class=\"nb\">int<\/span><span class=\"p\">(<\/span><span class=\"n\">MessageType<\/span><span class=\"o\">.<\/span><span class=\"n\">ACK<\/span><span class=\"p\">))<\/span>\n<\/span><span data-line=\"2\"><span class=\"k\">class<\/span><span class=\"w\"> <\/span><span class=\"nc\">AckMessage<\/span><span class=\"p\">(<\/span><span class=\"n\">Message<\/span><span class=\"p\">):<\/span>\n<\/span><span data-line=\"3\">    <span class=\"o\">...<\/span>\n<\/span><span data-line=\"4\">\n<\/span><span data-line=\"5\">    <span class=\"k\">def<\/span><span class=\"w\"> <\/span><span class=\"nf\">react<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">node<\/span><span class=\"p\">,<\/span> <span class=\"n\">addr<\/span><span class=\"p\">,<\/span> <span class=\"n\">sock<\/span><span class=\"p\">):<\/span>\n<\/span><span data-line=\"6\"><span class=\"w\">        <\/span><span class=\"sd\">&quot;&quot;&quot;Clear the message from node.awaiting_ack and call the relevant react_response() method.&quot;&quot;&quot;<\/span>\n<\/span><span data-line=\"7\">        <span class=\"n\">node<\/span><span class=\"o\">.<\/span><span class=\"n\">logger<\/span><span class=\"o\">.<\/span><span class=\"n\">debug<\/span><span class=\"p\">(<\/span><span class=\"s2\">&quot;Got an <\/span><span class=\"si\">%s<\/span><span class=\"s2\">ACK from <\/span><span class=\"si\">%r<\/span><span class=\"s2\"> (<\/span><span class=\"si\">%r<\/span><span class=\"s2\">)&quot;<\/span><span class=\"p\">,<\/span> <span class=\"s1\">&#39;N&#39;<\/span> <span class=\"k\">if<\/span> <span class=\"bp\">self<\/span><span class=\"o\">.<\/span><span class=\"n\">status<\/span> <span class=\"k\">else<\/span> <span class=\"s1\">&#39;&#39;<\/span><span class=\"p\">,<\/span> <span class=\"n\">addr<\/span><span class=\"p\">,<\/span> <span class=\"bp\">self<\/span><span class=\"p\">)<\/span>\n<\/span><span data-line=\"8\">        <span class=\"nb\">super<\/span><span class=\"p\">()<\/span><span class=\"o\">.<\/span><span class=\"n\">react<\/span><span class=\"p\">(<\/span><span class=\"n\">node<\/span><span class=\"p\">,<\/span> <span class=\"n\">addr<\/span><span class=\"p\">,<\/span> <span class=\"n\">sock<\/span><span class=\"p\">)<\/span>\n<\/span><span data-line=\"9\">        <span class=\"k\">try<\/span><span class=\"p\">:<\/span>\n<\/span><span data-line=\"10\">            <span class=\"n\">node<\/span><span class=\"o\">.<\/span><span class=\"n\">routing_table<\/span><span class=\"o\">.<\/span><span class=\"n\">member_info<\/span><span class=\"p\">[<\/span><span class=\"bp\">self<\/span><span class=\"o\">.<\/span><span class=\"n\">sender<\/span><span class=\"p\">]<\/span><span class=\"o\">.<\/span><span class=\"n\">local<\/span><span class=\"o\">.<\/span><span class=\"n\">hits<\/span> <span class=\"o\">+=<\/span> <span class=\"mi\">1<\/span>\n<\/span><span data-line=\"11\">        <span class=\"k\">except<\/span> <span class=\"ne\">KeyError<\/span><span class=\"p\">:<\/span>\n<\/span><span data-line=\"12\">            <span class=\"k\">pass<\/span>\n<\/span><span data-line=\"13\">        <span class=\"k\">if<\/span> <span class=\"bp\">self<\/span><span class=\"o\">.<\/span><span class=\"n\">resp_seq<\/span> <span class=\"ow\">in<\/span> <span class=\"n\">node<\/span><span class=\"o\">.<\/span><span class=\"n\">awaiting_ack<\/span><span class=\"p\">:<\/span>\n<\/span><span data-line=\"14\">            <span class=\"n\">node<\/span><span class=\"o\">.<\/span><span class=\"n\">awaiting_ack<\/span><span class=\"p\">[<\/span><span class=\"bp\">self<\/span><span class=\"o\">.<\/span><span class=\"n\">resp_seq<\/span><span class=\"p\">]<\/span><span class=\"o\">.<\/span><span class=\"n\">react_response<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">node<\/span><span class=\"p\">,<\/span> <span class=\"n\">addr<\/span><span class=\"p\">,<\/span> <span class=\"n\">sock<\/span><span class=\"p\">)<\/span>\n<\/span><span data-line=\"15\">            <span class=\"k\">del<\/span> <span class=\"n\">node<\/span><span class=\"o\">.<\/span><span class=\"n\">awaiting_ack<\/span><span class=\"p\">[<\/span><span class=\"bp\">self<\/span><span class=\"o\">.<\/span><span class=\"n\">resp_seq<\/span><span class=\"p\">]<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<p>Let\u2019s break that down.<\/p>\n<p>The first step is just like above, where it resets that PING timer for\nthat peer.<\/p>\n<p>After that, we\u2019d like to record that a message successfully went\nthrough. So, if we have the sender in our routing table, we look up our\nlocal information on that node (as opposed to public data like their ID\nor listening addresses) and record that there was a hit. This\ninformation can be used when pruning peers (which I did not yet\nimplement).<\/p>\n<p>The last step is to see if the node was expecting this acknowledgement\nfor something. If it was, we call the message in question\u2019s\n<code class=\"docutils literal notranslate\"><span class=\"pre\">react_response()<\/span><\/code> method. Currently there are only two messages that\nuse this feature: FIND_NODE and FIND_KEY. Let\u2019s look at the former:<\/p>\n<div class=\"highlight-python notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"nd\">@Message<\/span><span class=\"o\">.<\/span><span class=\"n\">register<\/span><span class=\"p\">(<\/span><span class=\"nb\">int<\/span><span class=\"p\">(<\/span><span class=\"n\">MessageType<\/span><span class=\"o\">.<\/span><span class=\"n\">FIND_NODE<\/span><span class=\"p\">))<\/span>\n<\/span><span data-line=\"2\"><span class=\"k\">class<\/span><span class=\"w\"> <\/span><span class=\"nc\">FindNodeMessage<\/span><span class=\"p\">(<\/span><span class=\"n\">Message<\/span><span class=\"p\">):<\/span>\n<\/span><span data-line=\"3\">    <span class=\"o\">...<\/span>\n<\/span><span data-line=\"4\">\n<\/span><span data-line=\"5\">    <span class=\"k\">def<\/span><span class=\"w\"> <\/span><span class=\"nf\">react_response<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">ack<\/span><span class=\"p\">,<\/span> <span class=\"n\">node<\/span><span class=\"p\">,<\/span> <span class=\"n\">addr<\/span><span class=\"p\">,<\/span> <span class=\"n\">sock<\/span><span class=\"p\">,<\/span> <span class=\"n\">message_constructor<\/span><span class=\"o\">=<\/span><span class=\"n\">PingMessage<\/span><span class=\"p\">):<\/span>\n<\/span><span data-line=\"6\"><span class=\"w\">        <\/span><span class=\"sd\">&quot;&quot;&quot;Attempt to connect to each of the nodes you were told about.&quot;&quot;&quot;<\/span>\n<\/span><span data-line=\"7\">        <span class=\"n\">node<\/span><span class=\"o\">.<\/span><span class=\"n\">logger<\/span><span class=\"o\">.<\/span><span class=\"n\">debug<\/span><span class=\"p\">(<\/span><span class=\"s2\">&quot;Got a response to a FIND_NODE from <\/span><span class=\"si\">%r<\/span><span class=\"s2\"> (<\/span><span class=\"si\">%r<\/span><span class=\"s2\">, <\/span><span class=\"si\">%r<\/span><span class=\"s2\">)&quot;<\/span><span class=\"p\">,<\/span> <span class=\"n\">addr<\/span><span class=\"p\">,<\/span> <span class=\"bp\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">ack<\/span><span class=\"p\">)<\/span>\n<\/span><span data-line=\"8\">        <span class=\"k\">for<\/span> <span class=\"n\">info<\/span> <span class=\"ow\">in<\/span> <span class=\"n\">ack<\/span><span class=\"o\">.<\/span><span class=\"n\">data<\/span><span class=\"p\">:<\/span>\n<\/span><span data-line=\"9\">            <span class=\"n\">name<\/span> <span class=\"o\">=<\/span> <span class=\"n\">info<\/span><span class=\"o\">.<\/span><span class=\"n\">name<\/span>\n<\/span><span data-line=\"10\">            <span class=\"k\">if<\/span> <span class=\"n\">name<\/span> <span class=\"o\">!=<\/span> <span class=\"n\">node<\/span><span class=\"o\">.<\/span><span class=\"n\">id<\/span> <span class=\"ow\">and<\/span> <span class=\"n\">name<\/span> <span class=\"ow\">not<\/span> <span class=\"ow\">in<\/span> <span class=\"n\">node<\/span><span class=\"o\">.<\/span><span class=\"n\">routing_table<\/span><span class=\"p\">:<\/span>\n<\/span><span data-line=\"11\">                <span class=\"k\">for<\/span> <span class=\"n\">address<\/span> <span class=\"ow\">in<\/span> <span class=\"n\">info<\/span><span class=\"o\">.<\/span><span class=\"n\">addresses<\/span><span class=\"p\">:<\/span>\n<\/span><span data-line=\"12\">                    <span class=\"k\">try<\/span><span class=\"p\">:<\/span>\n<\/span><span data-line=\"13\">                        <span class=\"k\">if<\/span> <span class=\"n\">node<\/span><span class=\"o\">.<\/span><span class=\"n\">routing_table<\/span><span class=\"o\">.<\/span><span class=\"n\">add<\/span><span class=\"p\">(<\/span><span class=\"n\">name<\/span><span class=\"p\">,<\/span> <span class=\"n\">address<\/span><span class=\"o\">.<\/span><span class=\"n\">args<\/span><span class=\"p\">,<\/span> <span class=\"n\">address<\/span><span class=\"o\">.<\/span><span class=\"n\">addr_type<\/span><span class=\"p\">):<\/span>\n<\/span><span data-line=\"14\">                            <span class=\"n\">node<\/span><span class=\"o\">.<\/span><span class=\"n\">_send<\/span><span class=\"p\">(<\/span><span class=\"n\">address<\/span><span class=\"o\">.<\/span><span class=\"n\">addr_type<\/span><span class=\"p\">,<\/span> <span class=\"n\">address<\/span><span class=\"o\">.<\/span><span class=\"n\">args<\/span><span class=\"p\">,<\/span> <span class=\"n\">message_constructor<\/span><span class=\"p\">())<\/span>\n<\/span><span data-line=\"15\">                            <span class=\"n\">node<\/span><span class=\"o\">.<\/span><span class=\"n\">routing_table<\/span><span class=\"o\">.<\/span><span class=\"n\">member_info<\/span><span class=\"p\">[<\/span><span class=\"n\">name<\/span><span class=\"p\">]<\/span><span class=\"o\">.<\/span><span class=\"n\">public<\/span> <span class=\"o\">=<\/span> <span class=\"n\">info<\/span>\n<\/span><span data-line=\"16\">                        <span class=\"k\">break<\/span>\n<\/span><span data-line=\"17\">                    <span class=\"k\">except<\/span> <span class=\"ne\">Exception<\/span><span class=\"p\">:<\/span>\n<\/span><span data-line=\"18\">                        <span class=\"k\">continue<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<p>A FIND_NODE message expects to get back an array of <code class=\"docutils literal notranslate\"><span class=\"pre\">GlobalPeerInfo<\/span><\/code>\nobjects, the details of which don\u2019t matter very much here. The only\nimportant things for our purposes is that it contains a list of\nlistening addresses and a list of supported compression methods in order\nof preference.<\/p>\n<p>For each of objects, we check that it\u2019s not someone we already know\nabout or ourself. If not, then we try to send a message to each address\nin sequence, ignoring errors.<\/p>\n<\/section>\n<\/section>\n<section id=\"design-problems\">\n<h2>Design Problems<a class=\"headerlink\" href=\"#design-problems\" title=\"Link to this heading\">\u00b6<\/a><\/h2>\n<section id=\"the-math-on-b\">\n<h3>The math on <span class=\"math notranslate nohighlight\">\\(b\\)<\/span><a class=\"headerlink\" href=\"#the-math-on-b\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>The idea behind <span class=\"math notranslate nohighlight\">\\(b\\)<\/span>, Kademlia\u2019s accelleration parameter, is that\nyou can trade between routing efficiency and routing table size by\nlooking at <span class=\"math notranslate nohighlight\">\\(b\\)<\/span>-bit symbols rather than 1-bit ones. So if you are\nID 101010\u2026, <span class=\"math notranslate nohighlight\">\\(b=1\\)<\/span> would have you making a routing table with\nentries like:<\/p>\n<ul class=\"simple\">\n<li><p>0\u2026<\/p><\/li>\n<li><p>01\u2026<\/p><\/li>\n<li><p>010\u2026<\/p><\/li>\n<li><p>0101\u2026<\/p><\/li>\n<li><p>01010\u2026<\/p><\/li>\n<li><p>010101\u2026<\/p><\/li>\n<li><p>etc.<\/p><\/li>\n<\/ul>\n<p>If <span class=\"math notranslate nohighlight\">\\(b=2\\)<\/span>, however, you would end up with a routing table like:<\/p>\n<ul class=\"simple\">\n<li><p>00\u2026<\/p><\/li>\n<li><p>01\u2026<\/p><\/li>\n<li><p>11\u2026<\/p><\/li>\n<li><p>0100\u2026<\/p><\/li>\n<li><p>0101\u2026<\/p><\/li>\n<li><p>0111\u2026<\/p><\/li>\n<li><p>010100\u2026<\/p><\/li>\n<li><p>010101\u2026<\/p><\/li>\n<li><p>010111\u2026<\/p><\/li>\n<li><p>etc.<\/p><\/li>\n<\/ul>\n<p>The trade off you make here is that the number of queries you need to\nfind a node is <span class=\"math notranslate nohighlight\">\\(\\mathcal{O}(\\log_{2^b}(n))\\)<\/span> whereas routing table size is\n<span class=\"math notranslate nohighlight\">\\(\\mathcal{O}(2^b \\cdot \\log_{2^b}(n))\\)<\/span>. A more precise guess for the\nrouting table is that it would scale as\n<span class=\"math notranslate nohighlight\">\\(\\left\\lceil\\tfrac{(2^b - 1)}{b}\\right\\rceil \\cdot \\log_{2^b}(n)\\)<\/span>, but I\nhaven\u2019t actually sat down to check that math, so take it with a grain of\nsalt.<\/p>\n<p>The problem is, I\u2019m doing the math wrong in my routing table for\n<span class=\"math notranslate nohighlight\">\\(b &gt; 1\\)<\/span>, and I haven\u2019t been able to identify where. All I know is\nthat occasionally I get an <code class=\"docutils literal notranslate\"><span class=\"pre\">IndexError<\/span><\/code> and my routing table has to\nfall back to searching over the set of all known peers instead of a\nparticular bucket.<\/p>\n<\/section>\n<section id=\"the-crappy-broadcast-algorithm\">\n<h3>The Crappy Broadcast Algorithm<a class=\"headerlink\" href=\"#the-crappy-broadcast-algorithm\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>The broadcast method I used in this implementation is naive at best. It\nuses a simple flooding model using the following rules:<\/p>\n<ol class=\"arabic simple\">\n<li><p>If you saw this broadcast already (identified by sender, sequence\npairs), ignore this<\/p><\/li>\n<li><p>For each peer in your routing table, echo this message to them unless<\/p>\n<ol class=\"arabic simple\">\n<li><p>They are the original sender, as identified by the sender<\/p><\/li>\n<li><p>They are the node that sent it to you, as identified by the addr,\nsock pair in <code class=\"docutils literal notranslate\"><span class=\"pre\">react()<\/span><\/code><\/p><\/li>\n<\/ol>\n<\/li>\n<\/ol>\n<p>This ends up with <span class=\"math notranslate nohighlight\">\\(\\mathcal{O}(n^2)\\)<\/span> messages sent, and given the topology\nof a Kademlia network, <span class=\"math notranslate nohighlight\">\\(\\mathcal{O}(log(n))\\)<\/span> hops before reaching the final\nnode.<\/p>\n<p>This is obviously sub-optimal, and is a topic I will explore further\nwhen I manage to get a write-up for <em>Solution for the broadcasting in\nKademlia peer-to-peer overlay<\/em> by Czirkos and Hossz\u00fa.<\/p>\n<\/section>\n<section id=\"what-s-this-alpha-parameter\">\n<h3>What\u2019s This <span class=\"math notranslate nohighlight\">\\(\\alpha\\)<\/span> Parameter?<a class=\"headerlink\" href=\"#what-s-this-alpha-parameter\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>Kademlia defines a concurrency parameter called <span class=\"math notranslate nohighlight\">\\(\\alpha\\)<\/span>, which\nessentially says \u201cthou shalt send <span class=\"math notranslate nohighlight\">\\(\\alpha\\)<\/span> messages per request.\u201d\nConfusingly, some messages use <span class=\"math notranslate nohighlight\">\\(k\\)<\/span> for their concurrency parameter\ninstead. The main culprit there is STORE, but rather than figure out\nwhere to do what, I just assumed that <span class=\"math notranslate nohighlight\">\\(k=\\alpha\\)<\/span>. It\u2019s a\nrelatively small change to fix that, so presumably I will before I do\nanything major with this.<\/p>\n<\/section>\n<\/section>\n<section id=\"retrion-progress\">\n<h2>Retrion Progress<a class=\"headerlink\" href=\"#retrion-progress\" title=\"Link to this heading\">\u00b6<\/a><\/h2>\n<p>Recall the initial goals of Retrion:<\/p>\n<ol class=\"arabic simple\">\n<li><p>An object model and serialization standard that can be used easily\nin most languages<\/p><\/li>\n<li><p>A simple high-level API (along with a fine-grained one)<\/p><\/li>\n<li><p><span class=\"math notranslate nohighlight\">\\(log(n)\\)<\/span> distributed hash table get\/sets<\/p><\/li>\n<li><p>Support for disabling DHT support network-wide<\/p><\/li>\n<li><p>Support for \u201cedge nodes\u201d which do not store DHT data except as a\ncache<\/p><\/li>\n<li><p><span class=\"math notranslate nohighlight\">\\(log(n)\\)<\/span> broadcast delivery time<\/p><\/li>\n<li><p><span class=\"math notranslate nohighlight\">\\(log(n)\\)<\/span> routed message delivery<\/p><\/li>\n<li><p>Support for multicast routed messages<\/p><\/li>\n<li><p>Support for in-protocol group messages<\/p><\/li>\n<li><p>Transparent (to the developer) compression<\/p><\/li>\n<li><p>transparent (to the user) encryption<\/p><\/li>\n<li><p>Support for multiple transport protocols (TCP, UDP, etc.)<\/p><\/li>\n<li><p>Support for \u201cvirtual transports\u201d (using other nodes as a bridge)<\/p><\/li>\n<li><p>A well-defined standard configuration space<\/p><\/li>\n<li><p>Support for extension subprotocols and implementation-specific\nconfigurations<\/p><\/li>\n<\/ol>\n<p>I would say that this implementation likely satisfies 1, 3, 6, 10, 14,\nand 15.<\/p>\n<p>It certainly does not satisfy 4, 5, 7, 8, 9, 11, 12, or 13.<\/p>\n<p>That means we need some kind of roadmap to delivering these other\nproperties and addressing other deficiencies<\/p>\n<section id=\"support-for-disabling-dht-support-network-wide\">\n<h3>4. Support for disabling DHT support network-wide<a class=\"headerlink\" href=\"#support-for-disabling-dht-support-network-wide\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>This is probably the easiest to do, as it just puts you into a mode\nsimilar to how Ethereum already works. Probably it\u2019s as simple as adding\na flag to the NetworkConfiguration object and having done with it.<\/p>\n<\/section>\n<section id=\"support-for-edge-nodes-which-do-not-store-dht-data-except-as-a-cache\">\n<h3>5. Support for \u201cedge nodes\u201d which do not store DHT data except as a cache<a class=\"headerlink\" href=\"#support-for-edge-nodes-which-do-not-store-dht-data-except-as-a-cache\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>This one is harder, as it presents some design challenges. My initial\nthought would be to have edge node status indicated by having the most\nsignificant bit of your ID be 1, but this presents a few problems:<\/p>\n<ol class=\"arabic simple\">\n<li><p>How do you guarantee that every edge node is in someone\u2019s routing\ntable? Otherwise no broadcast support<\/p><\/li>\n<li><p>This means that nodes have to decide that at ID-assignment time,\nunless we allow nodes to make new IDs<\/p><\/li>\n<li><p>The use case for edge nodes is something that runs in a browser or\nphone. That means we need support for<\/p>\n<ol class=\"arabic simple\">\n<li><p>roaming addresses, and<\/p><\/li>\n<li><p>websocket or webrtc connections<\/p><\/li>\n<\/ol>\n<\/li>\n<\/ol>\n<p>For now, I think this is off the table. It definitely should be\nrevisited in the future, but there are too many problems with the\nconcept as it exists now to implement it.<\/p>\n<\/section>\n<section id=\"log-n-routed-message-delivery\">\n<h3>7. <span class=\"math notranslate nohighlight\">\\(log(n)\\)<\/span> routed message delivery<a class=\"headerlink\" href=\"#log-n-routed-message-delivery\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>This one is trivial to implement, I just haven\u2019t gotten to it yet. It\u2019s\nalmost a duplicate of the work I already do in FIND_NODE and STORE\nanyways, I just need to change the side effect.<\/p>\n<p>Probably there are optimizable algorithms that I\u2019ll need to look for.<\/p>\n<\/section>\n<section id=\"support-for-multicast-routed-messages\">\n<h3>8. Support for multicast routed messages<a class=\"headerlink\" href=\"#support-for-multicast-routed-messages\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>An naive implementation of this should be fairly easy, I would just need\nto change the target field in a message like above to hold an array of\ntargets instead of just a target.<\/p>\n<\/section>\n<section id=\"support-for-in-protocol-group-messages\">\n<h3>9. Support for in-protocol group messages<a class=\"headerlink\" href=\"#support-for-in-protocol-group-messages\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>Leveraging the above, it should be possible to make a chatroom context\nor the like in-protocol. The real question is whether it should use a\nname registry at the network level which references some key in the DHT,\nor whether individual nodes should track their own rooms and be able to\nattach a room name to a multicast routed message. I kinda lean towards\nthe latter, but we\u2019ll see.<\/p>\n<\/section>\n<section id=\"transparent-to-the-user-encryption\">\n<h3>11. transparent (to the user) encryption<a class=\"headerlink\" href=\"#transparent-to-the-user-encryption\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>I have no idea how to approach this yet beyond broad concepts. It would\nbe fantastic if you could have a bit in your GlobalPeerInfo object that\nsays \u201chey, use the OWS protocol with me now\u201d, or something similar. The\nonly question is going to be how many methods are easily implementable\nin many languages. Probably the languages I will choose to care about\nhere are C\/C++, JavaScript, Python, and possibly Java or Kotlin.<\/p>\n<\/section>\n<section id=\"support-for-multiple-transport-protocols-tcp-udp-etc\">\n<h3>12. Support for multiple transport protocols (TCP, UDP, etc.)<a class=\"headerlink\" href=\"#support-for-multiple-transport-protocols-tcp-udp-etc\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>No idea how to approach this in terms of limiting its downsides. I\u2019ll\nupdate when I have a confident idea of how to handle the local resource\noverhead (number of sockets the system lets you have, for example) of\nconnection-based transports before this is really approachable. I think\nthe way I\u2019ll approach this is a two-pronged approach.<\/p>\n<p>First I implement it under the assumption of UDPv4 being the minimum\nrequirement to avoid network fragmentation.<\/p>\n<p>In parallel I need to do some kind of formal analysis on network\nfragmentation risk if you allow some nodes to have incompatible lists of\nsupported transports. It would be useful if your sandboxed browser code\ncould interact with a large network of external nodes that communicate\nwith more convenient transports than WebSockets, but I\u2019m fairly worried\nabout how many risks and inefficiencies that would expose.<\/p>\n<\/section>\n<section id=\"support-for-virtual-transports-using-other-nodes-as-a-bridge\">\n<h3>13. Support for \u201cvirtual transports\u201d (using other nodes as a bridge)<a class=\"headerlink\" href=\"#support-for-virtual-transports-using-other-nodes-as-a-bridge\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>This is an idea that I might want to scrap. If routed messages are\nworkable, then virtual transports are probably trivial to implement. The\nonly question is gonna be how inefficient it is without explicitly\nsetting up a bridge route.<\/p>\n<\/section>\n<\/section>\n<section id=\"conclusions\">\n<h2>Conclusions<a class=\"headerlink\" href=\"#conclusions\" title=\"Link to this heading\">\u00b6<\/a><\/h2>\n<p>I\u2019ve got a long road ahead of me, but this is a pretty good start!<\/p>\n<p class=\"rubric\">Citations<\/p>\n<div class=\"docutils container\" id=\"id3\">\n<div role=\"list\" class=\"citation-list\">\n<div class=\"citation\" id=\"id74\" role=\"doc-biblioentry\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span><a role=\"doc-backlink\" href=\"#id1\">ACa<\/a><span class=\"fn-bracket\">]<\/span><\/span>\n<p>Olivia Appleton-Crocker. Kademlia (the protocol). URL: <a class=\"reference external\" href=\"https:\/\/blog.oliviaappleton.com\/posts\/0002-Kademlia-protocol\">https:\/\/blog.oliviaappleton.com\/posts\/0002-Kademlia-protocol<\/a>.<\/p>\n<\/div>\n<div class=\"citation\" id=\"id73\" role=\"doc-biblioentry\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span><a role=\"doc-backlink\" href=\"#id2\">ACb<\/a><span class=\"fn-bracket\">]<\/span><\/span>\n<p>Olivia Appleton-Crocker. P2p-today\/retrion-development: a development repository for a future release of the retrion protocol. URL: <a class=\"reference external\" href=\"https:\/\/github.com\/p2p-today\/retrion-development\">https:\/\/github.com\/p2p-today\/retrion-development<\/a>.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<p class=\"rubric\">Cite As<\/p>\n<details><summary>Click here to expand the bibtex entry.<\/summary><div class=\"docutils container\">\n<blockquote>\n<div><div class=\"highlight-bibtex notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"nc\">@online<\/span><span class=\"p\">{<\/span><span class=\"nl\">appleton_blog_0003<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"2\"><span class=\"w\">  <\/span><span class=\"na\">title<\/span><span class=\"w\">    <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ Kademlia (My Implementation) }<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"3\"><span class=\"w\">  <\/span><span class=\"na\">author<\/span><span class=\"w\">   <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ Olivia Appleton-Crocker }<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"4\"><span class=\"w\">  <\/span><span class=\"na\">language<\/span><span class=\"w\"> <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ English }<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"5\"><span class=\"w\">  <\/span><span class=\"na\">version<\/span><span class=\"w\">  <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ 1.1 }<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"6\"><span class=\"w\">  <\/span><span class=\"na\">date<\/span><span class=\"w\">     <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ 2020-05-21 }<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"7\"><span class=\"w\">  <\/span><span class=\"na\">url<\/span><span class=\"w\">      <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ https:\/\/blog.oliviaappleton.com\/posts\/0002-Kademlia-implementation }<\/span>\n<\/span><span data-line=\"8\"><span class=\"p\">}<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<\/div><\/blockquote>\n<\/div>\n<\/details><p class=\"tags\"><span>Tags: <\/span><a class=\"reference external\" href=\"..\/_tags\/research-review.html\">research-review<\/a><span>, <\/span><a class=\"reference external\" href=\"..\/_tags\/research-progress.html\">research-progress<\/a><span>, <\/span><a class=\"reference external\" href=\"..\/_tags\/hobby-horse.html\">hobby-horse<\/a><span>, <\/span><a class=\"reference external\" href=\"..\/_tags\/retrion.html\">retrion<\/a><span>, <\/span><a class=\"reference external\" href=\"..\/_tags\/distributed-systems.html\">distributed-systems<\/a><\/p>\n\n            <h2>\n                Comments\n                <span class=\"comments-info\">\n                    <img class=\"fedi-icon\" src=\"https:\/\/blog.oliviaappleton.com\/_static\/boost.svg\" alt=\"Boosts\">\n                    <span id=\"global-reblogs\"><\/span>,\n                    <img class=\"fedi-icon\" src=\"https:\/\/blog.oliviaappleton.com\/_static\/like.svg\" alt=\"Likes\">\n                    <span id=\"global-likes\"><\/span>\n                <\/span>\n            <\/h2>\n            <div id=\"comments-section\"><\/div>\n            <script>\n                document.addEventListener(\"DOMContentLoaded\", function () {\n                    setConfig({\n                        boostLink: \"https:\/\/blog.oliviaappleton.com\/_static\/boost.svg\",\n                        allowSensitiveEmoji: false,\n                        allowCustomEmoji: true,\n                        allowMediaAttachments: true,\n                        allowAvatars: true,\n                        delayCommentLoad: true,\n                        defaultReactionEmoji: '\u2764',\n                        retryDelay: 100,\n                    });\n                    fetchComments('misskey', 'transfem.social', 'a59ynf4ageyb01nz', 5);\n                });\n            <\/script>\n        <\/section>\n<\/section>\n","guid":"https:\/\/blog.oliviaappleton.com\/posts\/0003-Kademlia-implementation.html","category":["research-progress","hobby-horse","retrion","distributed-systems"],"pubDate":"Thu, 21 May 2020 00:00:00 +0000"},{"title":"Kademlia (The Protocol)","link":"https:\/\/blog.oliviaappleton.com\/posts\/0002-Kademlia-protocol.html","description":"<a rel='me' href='https:\/\/transfem.social\/@LivInTheLookingGlass' role='none' class='invisible'><\/a>\n<script>\n    window.goatcounter = {\n        path: function() {\n            let p = location.pathname;\n            if (p == '\/') {\n                p = '\/index.html';\n            }\n            return '\/blog' + p;\n        }\n    }\n<\/script>\n<script data-goatcounter='https:\/\/livinthelookingglass.goatcounter.com\/count'\n async src='\/\/gc.zgo.at\/count.js'><\/script><section id=\"kademlia-the-protocol\">\n<h1>Kademlia (The Protocol)<a class=\"headerlink\" href=\"#kademlia-the-protocol\" title=\"Link to this heading\">\u00b6<\/a><\/h1>\n<p><span class=\"date\">2020-05-17<\/span><\/p>\n<p>Kademlia describes a protocol for implementing a <a class=\"reference external\" href=\"https:\/\/en.wikipedia.org\/wiki\/Distributed_hash_table\">Distributed Hash\nTable<\/a> (DHT)\nand is described in\n<a class=\"reference external\" href=\"http:\/\/www.scs.stanford.edu\/~dm\/home\/papers\/kpos.pdf\">these<\/a> two\n<a class=\"reference external\" href=\"https:\/\/pdos.csail.mit.edu\/~petar\/papers\/maymounkov-kademlia-lncs.pdf\">papers<\/a> <span id=\"id1\">[<a class=\"reference internal\" href=\"#id91\" title=\"Petar Maymounkov and David Mazi\u00e8res. Kademlia: a peer-to-peer information system based on the xor metric. Includes additional images and implementation details. URL: http:\/\/www.scs.stanford.edu\/~dm\/home\/papers\/kpos.pdf.\">MMa<\/a>, <a class=\"reference internal\" href=\"#id90\" title=\"Petar Maymounkov and David Mazi\u00e8res. Kademlia: a peer-to-peer information system based on the xor metric. More pared-down version that focuses on theory rather than implementation. URL: https:\/\/pdos.csail.mit.edu\/~petar\/papers\/maymounkov-kademlia-lncs.pdf.\">MMb<\/a>]<\/span>.\nIt is a widely used and widely extended protocol that serves to enable\nor improve many other systems, so it\u2019s worth taking some time to dive\ninto it.<\/p>\n<section id=\"some-background\">\n<h2>Some Background<a class=\"headerlink\" href=\"#some-background\" title=\"Link to this heading\">\u00b6<\/a><\/h2>\n<p>There are many DHTs that came before this, and there will likely be many\nwhich come after. I want to first describe\n<a class=\"reference external\" href=\"https:\/\/en.wikipedia.org\/wiki\/Chord_(peer-to-peer)\">Chord<\/a>\nas background to show why this was a useful idea, and to build some\nintuitions for how DHTs work.<\/p>\n<section id=\"chord\">\n<h3>Chord<a class=\"headerlink\" href=\"#chord\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>Chord is a precursor to Kademlia. What it does is picture its network as\na circle.<\/p>\n<p>Each node in this circle is assigned an ID by some agreed-upon process.\nAll nodes are required to hold a connection with the node before and\nafter it.<\/p>\n<p>To make a request, whether that is to get or set an entry, you pass that\nrequest from one node to the next along that circle until you arrive at\nthe node \u201cnearest\u201d to that entry\u2019s key by some agreed-upon method. To\ntake an example, let\u2019s look at a network that supports a max of 8 nodes\nand uses the last 3 bits of MD5<a class=\"footnote-reference brackets\" href=\"#id4\" id=\"id2\" role=\"doc-noteref\"><span class=\"fn-bracket\">[<\/span>1<span class=\"fn-bracket\">]<\/span><\/a> to decide who owns what. All of these\ndistances are done by \u201cPrice is Right\u201d rules, regardless of method. Distances\nare <em>only<\/em> ever counted going clockwise around the circle.<\/p>\n<figure class=\"align-default\" id=\"id121\">\n<img alt=\"What this toy model might look like\" src=\"..\/_images\/chord-example-1.png\" \/>\n<figcaption>\n<p><span class=\"caption-text\">What this toy model might look like<\/span><a class=\"headerlink\" href=\"#id121\" title=\"Link to this image\">\u00b6<\/a><\/p>\n<\/figcaption>\n<\/figure>\n<p>This model is missing a lot, though. For instance, this setup would\nyield you <span class=\"math notranslate nohighlight\">\\(\\mathcal{O}(n)\\)<\/span> lookup time, which is not at all ideal. Chord\nthen imposes one further rule: each node must keep a \u201cfinger table\u201d\nwhere each node is at most the next power of two away. So if you\u2019re node\n0, you have a connection to the nodes closest to 1, 2, and 4.<\/p>\n<figure class=\"align-default\" id=\"id122\">\n<img alt=\"What this toy model might look like from the perspective of Node 0\" src=\"..\/_images\/chord-example-2.png\" \/>\n<figcaption>\n<p><span class=\"caption-text\">What this toy model might look like from the perspective of Node 0<\/span><a class=\"headerlink\" href=\"#id122\" title=\"Link to this image\">\u00b6<\/a><\/p>\n<\/figcaption>\n<\/figure>\n<p>Note that in the above image, the green arrow corresponds to a\nconnection required by Node 6. All the other connections are required by\nNode 0.<\/p>\n<p>Doing it this way you end up with an <span class=\"math notranslate nohighlight\">\\(\\mathcal{O}(log_2(n))\\)<\/span> lookup time.\nThe worst case is if you are requesting something owned by your direct\npredesessor. Node 0 forwards the request to Node 4, who forwards to Node\n6, who forwards to Node 7.<\/p>\n<p>There are some additional rules for adding nodes, but much of it just\nfollows from trying to hold the above properties, so I won\u2019t talk about\nit here for either Chord or Kademlia.<\/p>\n<\/section>\n<section id=\"bittorrent\">\n<h3>BitTorrent<a class=\"headerlink\" href=\"#bittorrent\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>Most BitTorrent clients <a class=\"reference external\" href=\"https:\/\/en.wikipedia.org\/wiki\/BitTorrent#Distributed_trackers\">use\nKademlia<\/a>\nto find peers, starting since about 2005.<\/p>\n<\/section>\n<section id=\"ethereum\">\n<h3>Ethereum<a class=\"headerlink\" href=\"#ethereum\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>The Ethereum network <a class=\"reference external\" href=\"https:\/\/github.com\/ethereum\/wiki\/wiki\/Kademlia-Peer-Selection\">uses\nKademlia<\/a>\nfor peer selection, rather than supporting its whole system. This is\nuseful because it allows you to make assumptions in performance analysis\nthat you would not be able to make in an unstructured network, and\nbecause it provides some assurances about Denial of Service resistance.<\/p>\n<\/section>\n<\/section>\n<section id=\"the-main-ideas\">\n<h2>The Main Ideas<a class=\"headerlink\" href=\"#the-main-ideas\" title=\"Link to this heading\">\u00b6<\/a><\/h2>\n<p>Kademlia largely brought together innovations present in other projects.\nExponential hopping came from Chord, and XOR distance somewhat comes\nfrom Pastry.<a class=\"footnote-reference brackets\" href=\"#id5\" id=\"id3\" role=\"doc-noteref\"><span class=\"fn-bracket\">[<\/span>2<span class=\"fn-bracket\">]<\/span><\/a> The main benefits they got came from\nputting a much larger emphasis on parallelism and from making a logical\nleap that the Pastry authors did not.<\/p>\n<section id=\"xor-distance\">\n<h3>XOR Distance<a class=\"headerlink\" href=\"#xor-distance\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>At first glance, Kademlia looks a lot like Chord. All nodes are assigned\nan ID using an agreed method, all requests are routed towards nodes that\nare closer to the target.<\/p>\n<p>The biggest difference here is that instead of using a circular distance\nmetric, Kademlia uses the XOR function. This seems like a strange\ndecision at first, or even an invalid one, but XOR is a perfectly valid\ndistance metric. In fact, it has more useful properties than Chord\u2019s\ncircular one.<\/p>\n<p>A general distance metric must satisfy the following four properties:<\/p>\n<ol class=\"arabic simple\">\n<li><p><span class=\"math notranslate nohighlight\">\\(d(x, x) = 0\\)<\/span><\/p><\/li>\n<li><p><span class=\"math notranslate nohighlight\">\\(d(x, y) &gt; 0\\)<\/span> where <span class=\"math notranslate nohighlight\">\\(x \\ne y\\)<\/span><\/p><\/li>\n<li><p><span class=\"math notranslate nohighlight\">\\(d(x, y) = d(y, x)\\)<\/span><\/p><\/li>\n<li><p>It satisfies the triangle inequality, so\n<span class=\"math notranslate nohighlight\">\\(d(x, z) \\le d(x, y) + d(y, z)\\)<\/span><\/p><\/li>\n<\/ol>\n<p>The Chord metric violates rule 3 (in our example <span class=\"math notranslate nohighlight\">\\(d(1, 2) = 1\\)<\/span> but\n<span class=\"math notranslate nohighlight\">\\(d(2, 1) = 7\\)<\/span>), whereas XOR satisfies all of them.<\/p>\n<\/section>\n<section id=\"exponential-hopping\">\n<h3>Exponential Hopping<a class=\"headerlink\" href=\"#exponential-hopping\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>Using XOR as a metric allows you to have exponential hopping in a much\nmore intuitive way. This part is where Kademlia looks most like Pastry,\nusing a prefix-based routing scheme.<\/p>\n<p>All Kademlia nodes maintain a routing table organized into \u201ck-buckets\u201d\nof <span class=\"math notranslate nohighlight\">\\(\\le k\\)<\/span> nodes where their IDs share the first <span class=\"math notranslate nohighlight\">\\(x\\)<\/span> bits of\nyour ID, or alternately, where the first bit they agree on is the\n<span class=\"math notranslate nohighlight\">\\(x\\)<\/span>th. When trying to decide on a node to send your request to,\nyou pick from the group with the same prefix as the key you\u2019re\nrequesting. This leads to exponential hopping in a way that is both more\nflexible than and feels less artificial than Chord\u2019s structure.<\/p>\n<\/section>\n<section id=\"emphasis-on-parallelism-and-replication\">\n<h3>Emphasis on Parallelism and Replication<a class=\"headerlink\" href=\"#emphasis-on-parallelism-and-replication\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>Because Kademlia\u2019s routing table is organized into groups, they are able\nto use this productively. Many implementations essentially say \u201cscrew\ntrying to find the optimal node, let\u2019s send the request to everyone in\nthat bucket.\u201d Indeed, this is a strategy actually pointed out by the\npaper.<\/p>\n<blockquote>\n<div><p>Kademlia [\u2026] can send a query to any node within an interval,\nallowing it to select routes based on latency or even send parallel,\nasynchronous queries to equally appropriate nodes.<\/p>\n<\/div><\/blockquote>\n<p>They even give a system parameter, <span class=\"math notranslate nohighlight\">\\(\\alpha\\)<\/span>, to describe how many\nnodes get sent non-STORE messages.<\/p>\n<\/section>\n<\/section>\n<section id=\"the-protocol\">\n<h2>The Protocol<a class=\"headerlink\" href=\"#the-protocol\" title=\"Link to this heading\">\u00b6<\/a><\/h2>\n<p>Kademlia consists entirely of 4 RPCs. All of them share some base\nproperties. For instance, all of these messages share the property that\nif the sender doesn\u2019t receive a reply then that is noted in the routing\ntable. If a node misses too many it will get replaced when another\nsuitable peer is found.<\/p>\n<section id=\"ping\">\n<h3>PING<a class=\"headerlink\" href=\"#ping\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>This probes a node to see if it\u2019s online. Fairly self explanatory.<\/p>\n<\/section>\n<section id=\"find-node\">\n<h3>FIND_NODE<a class=\"headerlink\" href=\"#find-node\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>This RPC is the heart of the Kademlia protocol. It takes a node ID as an\nargument, and the node which receives it will send back information for\nthe <span class=\"math notranslate nohighlight\">\\(k\\)<\/span> closest nodes it knows of. For a well-saturated node, this\nwill usually come from a single bucket in its routing table. If a node\ndoesn\u2019t know <span class=\"math notranslate nohighlight\">\\(k\\)<\/span> nodes yet, it sends as many as it does know of.<\/p>\n<p>Using this, Kademlia nodes can implement what they call a <em>node lookup<\/em>.\nThe gist of it is:<\/p>\n<ol class=\"arabic simple\">\n<li><p>Pick the <span class=\"math notranslate nohighlight\">\\(\\alpha\\)<\/span> nodes closest to your target<\/p><\/li>\n<li><p>Send each of them a FIND_NODE for your target<\/p><\/li>\n<li><p>If you receive a reply that includes your target, break<\/p><\/li>\n<li><p>If this round doesn\u2019t get you closer to your target, break<\/p><\/li>\n<li><p>Otherwise, go to 1<\/p><\/li>\n<\/ol>\n<\/section>\n<section id=\"find-key\">\n<h3>FIND_KEY<a class=\"headerlink\" href=\"#find-key\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>Retrieving a key is very similar to finding a node. When receiving a\nFIND_KEY message, Kademlia nodes may respond in one of two ways: If they\ndon\u2019t know of <span class=\"math notranslate nohighlight\">\\(k\\)<\/span> closer nodes to the target or they have that key\nin their local cache, they simply return the answer. Otherwise, they\nrespond exactly as if they had gotten a FIND_NODE message, and the peer\nwill use this to send further FIND_KEY requests.<\/p>\n<p>Kademlia calls out one additional step that I want to highlight\nverbatim:<\/p>\n<blockquote>\n<div><p>For caching purposes, once a lookup succeeds, the requesting node\nstores the (key, value) pair at the closest node it observed to the\nkey that did not return the value.<\/p>\n<\/div><\/blockquote>\n<\/section>\n<section id=\"store\">\n<h3>STORE<a class=\"headerlink\" href=\"#store\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>To store a value, one first looks up the closest <span class=\"math notranslate nohighlight\">\\(k\\)<\/span> nodes to that\nkey. Once they are found, send a STORE message to each of them with the\n(key, value) pair. It is important that you get the closest <span class=\"math notranslate nohighlight\">\\(k\\)<\/span>\nnodes, because all others will drop it from their local storage at a\nrate inversely proportional to their distance to the key.<\/p>\n<p>Those nodes that deem themselves responsible for a key will republish it\nas needed to keep it alive. Usually this includes the one who set it and\nthe <span class=\"math notranslate nohighlight\">\\(k\\)<\/span> closest nodes, but there could be other interested parties\nas well.<\/p>\n<\/section>\n<\/section>\n<section id=\"why-it-works\">\n<h2>Why it works<a class=\"headerlink\" href=\"#why-it-works\" title=\"Link to this heading\">\u00b6<\/a><\/h2>\n<p>Structurally, this works for a lot of the same reasons Chord does. It\ncan guarantee logarithmic lookup times with its hopping mechanism. In\nfact, it can do even better because it implements this in parallel.<\/p>\n<p>Persistance of keys relies on some assumptions, though, so I want to\noutline those here. The only way for a key to die would be if the\n<span class=\"math notranslate nohighlight\">\\(k\\)<\/span> closest nodes to all die or leave the network with no lookups\nor republishing between them, and for this value to not have been cached\nin any other node on the search path. Thus, the assumption is that this\nvalue <span class=\"math notranslate nohighlight\">\\(k\\)<\/span> is somewhat tuned to how many nodes you expect to leave\nin the longest expected refresh periods.<\/p>\n<p>The authors include in the longer version of the paper some analysis\nthey did on Gnutella networks to justify their belief that this is a\nreasonable assumption. The gist of it is that the longer any particular\nnode has been online, the more likely they will be online an hour from\nnow. They also use this to provide some Denial of Service resistance by\nrequiring that nodes have a preference for older peers over younger ones\nwhen selecting peers for their routing tables.<\/p>\n<\/section>\n<section id=\"my-thoughts\">\n<h2>My Thoughts<a class=\"headerlink\" href=\"#my-thoughts\" title=\"Link to this heading\">\u00b6<\/a><\/h2>\n<p>This protocol is super interesting to me, especially considering how\nsmall it is. It seems to offer a <em>lot<\/em> of room for extension into more\ngeneral-use protocols, which I\u2019ll be trying to do this summer. In the\nnext week, expect an update on an implementaion of Kademlia that I\u2019ve\nbeen working on.<\/p>\n<p class=\"rubric\">Footnotes<\/p>\n<aside class=\"footnote-list brackets\">\n<aside class=\"footnote brackets\" id=\"id4\" role=\"doc-footnote\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span><a role=\"doc-backlink\" href=\"#id2\">1<\/a><span class=\"fn-bracket\">]<\/span><\/span>\n<p>Yes, this is bad practice, but it\u2019s a toy model.<\/p>\n<\/aside>\n<aside class=\"footnote brackets\" id=\"id5\" role=\"doc-footnote\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span><a role=\"doc-backlink\" href=\"#id3\">2<\/a><span class=\"fn-bracket\">]<\/span><\/span>\n<p>The most major difference being that Pastry uses two routing functions,\nfalling back to numeric distance for last-mile deliveries.<\/p>\n<\/aside>\n<\/aside>\n<p class=\"rubric\">Citations<\/p>\n<div class=\"docutils container\" id=\"id6\">\n<div role=\"list\" class=\"citation-list\">\n<div class=\"citation\" id=\"id91\" role=\"doc-biblioentry\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span><a role=\"doc-backlink\" href=\"#id1\">MMa<\/a><span class=\"fn-bracket\">]<\/span><\/span>\n<p>Petar Maymounkov and David Mazi\u00e8res. Kademlia: a peer-to-peer information system based on the xor metric. Includes additional images and implementation details. URL: <a class=\"reference external\" href=\"http:\/\/www.scs.stanford.edu\/~dm\/home\/papers\/kpos.pdf\">http:\/\/www.scs.stanford.edu\/~dm\/home\/papers\/kpos.pdf<\/a>.<\/p>\n<\/div>\n<div class=\"citation\" id=\"id90\" role=\"doc-biblioentry\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span><a role=\"doc-backlink\" href=\"#id1\">MMb<\/a><span class=\"fn-bracket\">]<\/span><\/span>\n<p>Petar Maymounkov and David Mazi\u00e8res. Kademlia: a peer-to-peer information system based on the xor metric. More pared-down version that focuses on theory rather than implementation. URL: <a class=\"reference external\" href=\"https:\/\/pdos.csail.mit.edu\/~petar\/papers\/maymounkov-kademlia-lncs.pdf\">https:\/\/pdos.csail.mit.edu\/~petar\/papers\/maymounkov-kademlia-lncs.pdf<\/a>.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<p class=\"rubric\">Cite As<\/p>\n<details><summary>Click here to expand the bibtex entry.<\/summary><div class=\"docutils container\">\n<blockquote>\n<div><div class=\"highlight-bibtex notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"nc\">@online<\/span><span class=\"p\">{<\/span><span class=\"nl\">appleton_blog_0002<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"2\"><span class=\"w\">  <\/span><span class=\"na\">title<\/span><span class=\"w\">    <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ Kademlia (The Protocol) }<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"3\"><span class=\"w\">  <\/span><span class=\"na\">author<\/span><span class=\"w\">   <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ Olivia Appleton-Crocker }<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"4\"><span class=\"w\">  <\/span><span class=\"na\">language<\/span><span class=\"w\"> <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ English }<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"5\"><span class=\"w\">  <\/span><span class=\"na\">version<\/span><span class=\"w\">  <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ 1.1 }<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"6\"><span class=\"w\">  <\/span><span class=\"na\">date<\/span><span class=\"w\">     <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ 2020-05-17 }<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"7\"><span class=\"w\">  <\/span><span class=\"na\">url<\/span><span class=\"w\">      <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ https:\/\/blog.oliviaappleton.com\/posts\/0002-Kademlia-protocol }<\/span>\n<\/span><span data-line=\"8\"><span class=\"p\">}<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<\/div><\/blockquote>\n<\/div>\n<\/details><p class=\"tags\"><span>Tags: <\/span><a class=\"reference external\" href=\"..\/_tags\/research-review.html\">research-review<\/a><span>, <\/span><a class=\"reference external\" href=\"..\/_tags\/research-progress.html\">research-progress<\/a><span>, <\/span><a class=\"reference external\" href=\"..\/_tags\/hobby-horse.html\">hobby-horse<\/a><span>, <\/span><a class=\"reference external\" href=\"..\/_tags\/retrion.html\">retrion<\/a><span>, <\/span><a class=\"reference external\" href=\"..\/_tags\/distributed-systems.html\">distributed-systems<\/a><\/p>\n\n            <h2>\n                Comments\n                <span class=\"comments-info\">\n                    <img class=\"fedi-icon\" src=\"https:\/\/blog.oliviaappleton.com\/_static\/boost.svg\" alt=\"Boosts\">\n                    <span id=\"global-reblogs\"><\/span>,\n                    <img class=\"fedi-icon\" src=\"https:\/\/blog.oliviaappleton.com\/_static\/like.svg\" alt=\"Likes\">\n                    <span id=\"global-likes\"><\/span>\n                <\/span>\n            <\/h2>\n            <div id=\"comments-section\"><\/div>\n            <script>\n                document.addEventListener(\"DOMContentLoaded\", function () {\n                    setConfig({\n                        boostLink: \"https:\/\/blog.oliviaappleton.com\/_static\/boost.svg\",\n                        allowSensitiveEmoji: false,\n                        allowCustomEmoji: true,\n                        allowMediaAttachments: true,\n                        allowAvatars: true,\n                        delayCommentLoad: true,\n                        defaultReactionEmoji: '\u2764',\n                        retryDelay: 100,\n                    });\n                    fetchComments('misskey', 'transfem.social', 'a59ynew60v3c00vu', 5);\n                });\n            <\/script>\n        <\/section>\n<\/section>\n","guid":"https:\/\/blog.oliviaappleton.com\/posts\/0002-Kademlia-protocol.html","category":["research-review","research-progress","hobby-horse","retrion","distributed-systems"],"pubDate":"Sun, 17 May 2020 00:00:00 +0000"},{"title":"Hybrid Logical Clocks: Summary","link":"https:\/\/blog.oliviaappleton.com\/posts\/0001-HLC-review.html","description":"<a rel='me' href='https:\/\/transfem.social\/@LivInTheLookingGlass' role='none' class='invisible'><\/a>\n<script>\n    window.goatcounter = {\n        path: function() {\n            let p = location.pathname;\n            if (p == '\/') {\n                p = '\/index.html';\n            }\n            return '\/blog' + p;\n        }\n    }\n<\/script>\n<script data-goatcounter='https:\/\/livinthelookingglass.goatcounter.com\/count'\n async src='\/\/gc.zgo.at\/count.js'><\/script><section id=\"hybrid-logical-clocks-summary\">\n<h1>Hybrid Logical Clocks: Summary<a class=\"headerlink\" href=\"#hybrid-logical-clocks-summary\" title=\"Link to this heading\">\u00b6<\/a><\/h1>\n<p><span class=\"date\">2020-03-02<\/span><\/p>\n<p>This <a class=\"reference external\" href=\"https:\/\/cse.buffalo.edu\/tech-reports\/2014-04.pdf\">paper<\/a> <span id=\"id1\">[<a class=\"reference internal\" href=\"#id98\" title=\"Sandeep S. Kulkarni, Murat Demirbas, Deepak Madappa, Bharadwaj Avva, and Marcelo Leone. Logical physical clocks. In Marcos K. Aguilera, Leonardo Querzoni, and Marc Shapiro, editors, Principles of Distributed Systems, 17\u201332. Cham, 2014. Springer International Publishing.\">KDM+14<\/a>]<\/span>\nfocuses on the idea of time synchronization. They compare several\ndifferent methods, and ultimately introduce a new one which they term\n\u201cHybrid Logical Clocks\u201d. While the original title of the paper is\n<strong>\u201cLogical Physical Clocks and Consistent Snapshots in Globally\nDistributed Databases\u201d<\/strong>, I find this a bit too wordy to say more than\nonce.<\/p>\n<section id=\"some-background\">\n<h2>Some Background<a class=\"headerlink\" href=\"#some-background\" title=\"Link to this heading\">\u00b6<\/a><\/h2>\n<section id=\"physical-clocks-and-ntp\">\n<h3>Physical Clocks and NTP<a class=\"headerlink\" href=\"#physical-clocks-and-ntp\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>Many distributed systems developers might be tempted to just use the\nphysical clock on your local host. However, as anyone who has actually\ntried this knows, there is often a significant drift between the clocks\nof any two systems, <em>especially<\/em> as they increase in distance. The\n<a class=\"reference external\" href=\"https:\/\/en.wikipedia.org\/wiki\/Network_Time_Protocol\">Network Time\nProtocol<\/a> (NTP)\ncan usually keep nodes within ~10ms of each other, but that still leaves\nlarge windows of uncertainty when ordering events for, say, a video\ngame. You also have to worry about things like leap seconds, or\nsysadmins changing the time on you.<\/p>\n<\/section>\n<section id=\"truetime\">\n<h3>TrueTime<a class=\"headerlink\" href=\"#truetime\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>To try to address these problems, Google recently proposed a protocol\ncalled\n<a class=\"reference external\" href=\"https:\/\/storage.googleapis.com\/pub-tools-public-publication-data\/pdf\/45855.pdf\">TrueTime<\/a> <span id=\"id2\">[<a class=\"reference internal\" href=\"#id78\" title=\"Eric Brewer. Spanner, truetime and the cap theorem.\">Bre<\/a>]<\/span>\n(see page 5). What this allows is for you to get a strict bounding on clock error\nby using a combination of GPS and local atomic clocks to synchronize a\n\u201cglobal clock\u201d. The problem with this, of course, is that you need to\nhave your own personal atomic clock. That is highly problematic for the\naverage user. An alternative might be to have a receiver for some of the\npublicly-broadcasting atomic clocks and just store some offset to it,\nbut this is also problematic because it both requires extra hardware and\nwould reduce TrueTime\u2019s error guarantees.<\/p>\n<p>So while this is helpful, it is not <em>generally<\/em> useful.<\/p>\n<\/section>\n<section id=\"lamport-clocks\">\n<h3>Lamport Clocks<a class=\"headerlink\" href=\"#lamport-clocks\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>One of the earliest methods to address the problems of using physical\ntime was the <a class=\"reference external\" href=\"https:\/\/en.wikipedia.org\/wiki\/Lamport_timestamps\">Lamport\nClock<\/a> (LC). The\nidea here is to make a simple counter on each node. Start at 0, and with\neach event that node can see (purely local or otherwise), increment the\ncounter. If it receives a message, that message will always come with LC\ntimestamp from the originating node, and so you take max(yours, theirs)\n+ 1 as your new LC time.<\/p>\n<p>This makes two guarantees:<\/p>\n<ul class=\"simple\">\n<li><p>Event <span class=\"math notranslate nohighlight\">\\(X\\)<\/span> <strong>happens before<\/strong> event <span class=\"math notranslate nohighlight\">\\(Y\\)<\/span> <span class=\"math notranslate nohighlight\">\\(\\implies\\)<\/span>\n<span class=\"math notranslate nohighlight\">\\(X.lc &lt; Y.lc\\)<\/span><\/p><\/li>\n<li><p>If <span class=\"math notranslate nohighlight\">\\(Y\\)<\/span> is directly related to event <span class=\"math notranslate nohighlight\">\\(X\\)<\/span>,\n<span class=\"math notranslate nohighlight\">\\(X.lc &lt; Y.lc\\)<\/span> <span class=\"math notranslate nohighlight\">\\(\\implies\\)<\/span> <span class=\"math notranslate nohighlight\">\\(X\\)<\/span> <strong>happens before<\/strong>\n<span class=\"math notranslate nohighlight\">\\(Y\\)<\/span><\/p>\n<ul>\n<li><p><span class=\"math notranslate nohighlight\">\\(Y\\)<\/span> is directly related to <span class=\"math notranslate nohighlight\">\\(X\\)<\/span> if they are a\nsend\/receive pair, or<\/p><\/li>\n<li><p><span class=\"math notranslate nohighlight\">\\(Y\\)<\/span> is directly related to <span class=\"math notranslate nohighlight\">\\(X\\)<\/span> if they are directly\nnext to each other on some node\u2019s timeline<\/p><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>Example: <img alt=\"image1\" src=\"https:\/\/www.cs.rutgers.edu\/~pxk\/rutgers\/notes\/clocks\/images\/clocks-lamport.png\" \/><\/p>\n<\/section>\n<section id=\"vector-clocks\">\n<h3>Vector Clocks<a class=\"headerlink\" href=\"#vector-clocks\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p><a class=\"reference external\" href=\"https:\/\/en.wikipedia.org\/wiki\/Vector_clock\">Vector Clocks<\/a> are an\nextension of Lamport Clocks which allows you to make strong guarantees\nabout ordering.<\/p>\n<p>The idea behind it is that timestamps, rather than being a single\nnumber, are a vector of numbers where each item corresponds to the last\nknown Lamport time of some other process.<\/p>\n<p>The major benefit here is that event <span class=\"math notranslate nohighlight\">\\(X\\)<\/span> <strong>happens before<\/strong> event\n<span class=\"math notranslate nohighlight\">\\(Y\\)<\/span> <span class=\"math notranslate nohighlight\">\\(\\iff\\)<\/span> <span class=\"math notranslate nohighlight\">\\(X.vc &lt; Y.vc\\)<\/span>, so long as you define\ncomparisons by the rules given by their algorithm. They would specify:\n<span class=\"math notranslate nohighlight\">\\(X.vc &lt; Y.vc\\)<\/span> <span class=\"math notranslate nohighlight\">\\(\\iff\\)<\/span>\n<span class=\"math notranslate nohighlight\">\\([(\\forall i \\; X.vc_i \\le Y.vc_i)\\)<\/span> <span class=\"math notranslate nohighlight\">\\(\\land\\)<\/span>\n<span class=\"math notranslate nohighlight\">\\((\\exists i \\; X.vc_i &lt; Y.vc_i)]\\)<\/span>.<\/p>\n<img alt=\"..\/_images\/vc-rule1.png\" src=\"..\/_images\/vc-rule1.png\" \/>\n<img alt=\"..\/_images\/vc-rule2.png\" src=\"..\/_images\/vc-rule2.png\" \/>\n<img alt=\"..\/_images\/vc-rule3.png\" src=\"..\/_images\/vc-rule3.png\" \/>\n<p>This, however, comes with a number of drawbacks. The biggest of these is\nthat you need to know how many nodes your network will have before\nruntime. A more minor one is that the size of timestamps grows\nproportional to the number of nodes in your network.<\/p>\n<\/section>\n<section id=\"hybridtime\">\n<h3>HybridTime<a class=\"headerlink\" href=\"#hybridtime\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>This is a method which combines Vector Clocks with a set of\noptimizations based on loose synchronization of physical clocks. I can\u2019t\nfind any info on this, except that I think they are referencing <a class=\"reference external\" href=\"https:\/\/dl.acm.org\/doi\/10.5555\/647272.722012\">this\narticle<\/a> <span id=\"id3\">[<a class=\"reference internal\" href=\"#id102\" title=\"Sandeep S. Kulkarni and Ravikant. Stabilizing causal deterministic merge. In Proceedings of the 5th International Workshop on Self-Stabilizing Systems, WSS '01, 183\u2013199. Berlin, Heidelberg, 2001. Springer-Verlag.\">KR01<\/a>]<\/span> that I can\u2019t\nget access to. I\u2019ll ask my advisor if I remember to. They reference this\nin the paper, but not for much more than to say \u201chey, people have tried\nthis\u201d.<\/p>\n<\/section>\n<\/section>\n<section id=\"the-main-ideas\">\n<h2>The Main Ideas<a class=\"headerlink\" href=\"#the-main-ideas\" title=\"Link to this heading\">\u00b6<\/a><\/h2>\n<p>By the time we\u2019re done, the authors will have shown us an algorithm\nthat:<\/p>\n<ol class=\"arabic simple\">\n<li><p>Guarantees <span class=\"math notranslate nohighlight\">\\(X\\)<\/span> <strong>h.b.<\/strong> <span class=\"math notranslate nohighlight\">\\(Y\\)<\/span> <span class=\"math notranslate nohighlight\">\\(\\implies\\)<\/span>\n<span class=\"math notranslate nohighlight\">\\(X.hlc &lt; Y.hlc\\)<\/span><\/p><\/li>\n<li><p>Can be used in a backwards-compatible way with NTP timestamps<\/p><\/li>\n<li><p>Uses <span class=\"math notranslate nohighlight\">\\(\\mathcal{O}(1)\\)<\/span> memory<\/p><\/li>\n<li><p>Is resistant both to \u201cstragglers\u201d and \u201crushers\u201d who are significantly\nbehind or ahead of the rest of the network<\/p><\/li>\n<li><p>Is directly useful in identifying distributed snapshots<\/p><\/li>\n<\/ol>\n<\/section>\n<section id=\"the-algorithm\">\n<h2>The Algorithm<a class=\"headerlink\" href=\"#the-algorithm\" title=\"Link to this heading\">\u00b6<\/a><\/h2>\n<section id=\"the-components\">\n<h3>The Components<a class=\"headerlink\" href=\"#the-components\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>A Hybrid Logical Clock consists of three elements:<\/p>\n<ul class=\"simple\">\n<li><p><span class=\"math notranslate nohighlight\">\\(l\\)<\/span>, which represents the highest physical time this node has\nseen, truncated to <span class=\"math notranslate nohighlight\">\\(2^{-16}\\)<\/span> second precision<\/p><\/li>\n<li><p><span class=\"math notranslate nohighlight\">\\(c\\)<\/span>, which is a Lamport Clock that resets whenever <span class=\"math notranslate nohighlight\">\\(l\\)<\/span>\nincreases<\/p><\/li>\n<li><p><span class=\"math notranslate nohighlight\">\\(pt\\)<\/span>, which is a local property representing the given node\u2019s\nphysical time<\/p><\/li>\n<\/ul>\n<\/section>\n<section id=\"on-a-send-or-local-event\">\n<h3>On a Send or Local Event<a class=\"headerlink\" href=\"#on-a-send-or-local-event\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>When a node has a local event or sends a message, they should do the\nfollowing:<\/p>\n<div class=\"highlight-python notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"k\">def<\/span><span class=\"w\"> <\/span><span class=\"nf\">on_local<\/span><span class=\"p\">(<\/span><span class=\"n\">l<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span><span class=\"p\">,<\/span> <span class=\"n\">c<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span><span class=\"p\">):<\/span>\n<\/span><span data-line=\"2\">    <span class=\"n\">pt<\/span> <span class=\"o\">=<\/span> <span class=\"n\">physical_clock<\/span><span class=\"p\">()<\/span>\n<\/span><span data-line=\"3\">    <span class=\"n\">new_l<\/span> <span class=\"o\">=<\/span> <span class=\"nb\">max<\/span><span class=\"p\">((<\/span><span class=\"n\">l<\/span><span class=\"p\">,<\/span> <span class=\"n\">truncate<\/span><span class=\"p\">(<\/span><span class=\"n\">pt<\/span><span class=\"p\">)))<\/span>\n<\/span><span data-line=\"4\">    <span class=\"k\">if<\/span> <span class=\"n\">new_l<\/span> <span class=\"o\">==<\/span> <span class=\"n\">l<\/span><span class=\"p\">:<\/span>\n<\/span><span data-line=\"5\">        <span class=\"n\">logical<\/span> <span class=\"o\">=<\/span> <span class=\"n\">c<\/span> <span class=\"o\">+<\/span> <span class=\"mi\">1<\/span>\n<\/span><span data-line=\"6\">    <span class=\"k\">else<\/span><span class=\"p\">:<\/span>\n<\/span><span data-line=\"7\">        <span class=\"n\">logical<\/span> <span class=\"o\">=<\/span> <span class=\"mi\">0<\/span>\n<\/span><span data-line=\"8\">    <span class=\"k\">return<\/span> <span class=\"n\">HybridLogicalClock<\/span><span class=\"p\">(<\/span><span class=\"n\">new_l<\/span><span class=\"p\">,<\/span> <span class=\"n\">new_l<\/span> <span class=\"o\">-<\/span> <span class=\"n\">pt<\/span><span class=\"p\">,<\/span> <span class=\"n\">logical<\/span><span class=\"p\">)<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<\/section>\n<section id=\"on-receiving-a-message\">\n<h3>On Receiving a Message<a class=\"headerlink\" href=\"#on-receiving-a-message\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>When a node receives a message, they should do the following:<\/p>\n<div class=\"highlight-python notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"k\">def<\/span><span class=\"w\"> <\/span><span class=\"nf\">on_receive<\/span><span class=\"p\">(<\/span><span class=\"n\">l<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span><span class=\"p\">,<\/span> <span class=\"n\">c<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span><span class=\"p\">,<\/span> <span class=\"n\">m_l<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span><span class=\"p\">,<\/span> <span class=\"n\">m_c<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span><span class=\"p\">):<\/span>\n<\/span><span data-line=\"2\">    <span class=\"n\">pt<\/span> <span class=\"o\">=<\/span> <span class=\"n\">physical_clock<\/span><span class=\"p\">()<\/span>\n<\/span><span data-line=\"3\">    <span class=\"n\">new_l<\/span> <span class=\"o\">=<\/span> <span class=\"nb\">max<\/span><span class=\"p\">((<\/span><span class=\"n\">l<\/span><span class=\"p\">,<\/span> <span class=\"n\">m_l<\/span><span class=\"p\">,<\/span> <span class=\"n\">truncate<\/span><span class=\"p\">(<\/span><span class=\"n\">pt<\/span><span class=\"p\">)))<\/span>\n<\/span><span data-line=\"4\">    <span class=\"k\">if<\/span> <span class=\"n\">new_l<\/span> <span class=\"ow\">in<\/span> <span class=\"p\">(<\/span><span class=\"n\">l<\/span><span class=\"p\">,<\/span> <span class=\"n\">m_l<\/span><span class=\"p\">):<\/span>\n<\/span><span data-line=\"5\">        <span class=\"n\">new_c<\/span> <span class=\"o\">=<\/span> <span class=\"nb\">max<\/span><span class=\"p\">([(<\/span><span class=\"n\">l<\/span><span class=\"p\">,<\/span> <span class=\"n\">c<\/span><span class=\"p\">),<\/span> <span class=\"p\">(<\/span><span class=\"n\">m_l<\/span><span class=\"p\">,<\/span> <span class=\"n\">m_c<\/span><span class=\"p\">)])[<\/span><span class=\"mi\">1<\/span><span class=\"p\">]<\/span> <span class=\"o\">+<\/span> <span class=\"mi\">1<\/span>\n<\/span><span data-line=\"6\">    <span class=\"k\">else<\/span><span class=\"p\">:<\/span>\n<\/span><span data-line=\"7\">        <span class=\"n\">new_c<\/span> <span class=\"o\">=<\/span> <span class=\"mi\">0<\/span>\n<\/span><span data-line=\"8\">    <span class=\"k\">return<\/span> <span class=\"n\">HybridLogicalClock<\/span><span class=\"p\">(<\/span><span class=\"n\">new_l<\/span><span class=\"p\">,<\/span> <span class=\"n\">new_l<\/span> <span class=\"o\">-<\/span> <span class=\"n\">pt<\/span><span class=\"p\">,<\/span> <span class=\"n\">new_c<\/span><span class=\"p\">)<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<\/section>\n<section id=\"example-trace\">\n<h3>Example Trace<a class=\"headerlink\" href=\"#example-trace\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<img alt=\"..\/_images\/hlc_clock.png\" src=\"..\/_images\/hlc_clock.png\" \/>\n<p>Note that the paper has a typo in this image. The cell which reads as\n\u201c3, 13\u201d should read as \u201c3, 10, 3\u201d. This is because they recycled another\nfigure from their paper but forgot to re-annotate that cell.<\/p>\n<\/section>\n<\/section>\n<section id=\"why-it-works\">\n<h2>Why it works<a class=\"headerlink\" href=\"#why-it-works\" title=\"Link to this heading\">\u00b6<\/a><\/h2>\n<section id=\"frozen-pt-lamport-clock\">\n<h3>Frozen PT = Lamport Clock<a class=\"headerlink\" href=\"#frozen-pt-lamport-clock\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>Let\u2019s suppose for a moment that <span class=\"math notranslate nohighlight\">\\(pt\\)<\/span> never increments for any\nnode. In such a scenario, this will act exactly like a Lamport Clock.\nFortunately, most systems <em>don\u2019t<\/em> have permanently frozen clocks.<\/p>\n<\/section>\n<section id=\"pt-and-l-are-tightly-coupled\">\n<h3>PT and L are Tightly Coupled<a class=\"headerlink\" href=\"#pt-and-l-are-tightly-coupled\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>If <span class=\"math notranslate nohighlight\">\\(pt\\)<\/span> ever gets past <span class=\"math notranslate nohighlight\">\\(l\\)<\/span>, then <span class=\"math notranslate nohighlight\">\\(l\\)<\/span> will change and\n<span class=\"math notranslate nohighlight\">\\(c\\)<\/span> will reset. So the way we compare two timestamps is just by\nusing tuple comparisons of the form\n<span class=\"math notranslate nohighlight\">\\(\\left\\langle X.l, X.c\\right\\rangle\\)<\/span>. For review, the way that\nworks is that you first compare the first element. If they are not\nequal, return the result of the comparison. If they are equal, move on\nto the next element and repeat until the end. If you get past the end,\nthey are equal (and in this case, explicitly concurrent). From this\nfollows a very easy implication: <span class=\"math notranslate nohighlight\">\\(X\\)<\/span> <strong>h.b.<\/strong> <span class=\"math notranslate nohighlight\">\\(Y\\)<\/span>\n<span class=\"math notranslate nohighlight\">\\(\\implies\\)<\/span> <span class=\"math notranslate nohighlight\">\\(\\left\\langle X.l, X.c\\right\\rangle\\)<\/span> <span class=\"math notranslate nohighlight\">\\(&lt;\\)<\/span>\n<span class=\"math notranslate nohighlight\">\\(\\left\\langle Y.l, Y.c\\right\\rangle\\)<\/span>.<\/p>\n<p>This paper also shows that the gap between <span class=\"math notranslate nohighlight\">\\(pt\\)<\/span> and <span class=\"math notranslate nohighlight\">\\(l\\)<\/span> is\nbounded in any system without adversarial sysadmins. To get there we\nhave to go through another theorem though. <span class=\"math notranslate nohighlight\">\\(X.l &gt; X.pt\\)<\/span>\n<span class=\"math notranslate nohighlight\">\\(\\implies\\)<\/span> <span class=\"math notranslate nohighlight\">\\((\\exists Y : Y\\)<\/span> <strong>h.b.<\/strong> <span class=\"math notranslate nohighlight\">\\(X\\)<\/span> <span class=\"math notranslate nohighlight\">\\(\\land\\)<\/span>\n<span class=\"math notranslate nohighlight\">\\(Y.pt = X.l)\\)<\/span>. What they do next is introduce a system parameter\n<span class=\"math notranslate nohighlight\">\\(\\epsilon\\)<\/span>, which is defined as the maximum clock drift in the\nsystem (the difference between the smallest and largest <span class=\"math notranslate nohighlight\">\\(pt\\)<\/span>s on\nthe network).<\/p>\n<p>Given that, it is impossible to find some pair of events such that\n<span class=\"math notranslate nohighlight\">\\(X\\)<\/span> <strong>h.b.<\/strong> <span class=\"math notranslate nohighlight\">\\(Y\\)<\/span> and <span class=\"math notranslate nohighlight\">\\(X.pt &gt; Y.pt + \\epsilon\\)<\/span>. Because\nof that, it must follow that\n<span class=\"math notranslate nohighlight\">\\((\\forall X)(|X.l - X.pt| \\le \\epsilon)\\)<\/span>.<\/p>\n<\/section>\n<section id=\"c-has-a-maximum-value\">\n<h3>C has a Maximum Value<a class=\"headerlink\" href=\"#c-has-a-maximum-value\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>This also means that we can put bounds on the value of <span class=\"math notranslate nohighlight\">\\(c\\)<\/span>\n(assuming a system where sysadmins don\u2019t intervene). If we assume that\nthe minimum gap between events is some timestep <span class=\"math notranslate nohighlight\">\\(d\\)<\/span>, then\n<span class=\"math notranslate nohighlight\">\\((\\forall X)(X.c \\le \\tfrac{\\epsilon}{d} + 1)\\)<\/span>.<\/p>\n<\/section>\n<section id=\"encoding-as-an-ntp-timestamp\">\n<h3>Encoding as an NTP Timestamp<a class=\"headerlink\" href=\"#encoding-as-an-ntp-timestamp\" title=\"Link to this heading\">\u00b6<\/a><\/h3>\n<p>So this ends up creating a system with the causal guarantees of a\nLamport Clock, but can we encode it as a traditional timestamp? It turns\nout the answer is yes, but mostly because NTP provides some <em>absurd<\/em>\nprecision in its timestamps.<\/p>\n<p>NTP timestamps are 64-bits, divided into two 32-bit sections. The first\nportion is for a whole number of Unix-time seconds. The second portion\nis for fractional seconds. To be clear here, that means that NTP allows\nfor <span class=\"math notranslate nohighlight\">\\(2^{-32}\\)<\/span> second precision. This ends up being\n~<span class=\"math notranslate nohighlight\">\\(\\tfrac{1}{4}\\)<\/span>ns precision, which is just silly. It\u2019s a\nlittle easier to show the change they made when expressed as a C struct:<\/p>\n<div class=\"highlight-c notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"k\">struct<\/span><span class=\"w\"> <\/span><span class=\"nc\">NTP_stamp<\/span><span class=\"w\"> <\/span><span class=\"p\">{<\/span><span class=\"w\">  <\/span><span class=\"c1\">\/\/ note: NTP is big-endian<\/span>\n<\/span><span data-line=\"2\"><span class=\"w\">    <\/span><span class=\"kt\">uint32_t<\/span><span class=\"w\"> <\/span><span class=\"n\">seconds<\/span><span class=\"p\">;<\/span>\n<\/span><span data-line=\"3\"><span class=\"w\">    <\/span><span class=\"kt\">uint32_t<\/span><span class=\"w\"> <\/span><span class=\"n\">frac_seconds<\/span><span class=\"p\">;<\/span>\n<\/span><span data-line=\"4\"><span class=\"p\">};<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<p>What this paper does is instead allow for <span class=\"math notranslate nohighlight\">\\(2^{-16}\\)<\/span> second\nprecision in the physical time portion by saying:<\/p>\n<div class=\"highlight-c notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"k\">struct<\/span><span class=\"w\"> <\/span><span class=\"nc\">HLC_stamp<\/span><span class=\"w\"> <\/span><span class=\"p\">{<\/span><span class=\"w\">  <\/span><span class=\"c1\">\/\/ still big-endian<\/span>\n<\/span><span data-line=\"2\"><span class=\"w\">    <\/span><span class=\"kt\">uint32_t<\/span><span class=\"w\"> <\/span><span class=\"n\">seconds<\/span><span class=\"p\">;<\/span><span class=\"w\">       <\/span><span class=\"c1\">\/\/ corresponds to l<\/span>\n<\/span><span data-line=\"3\"><span class=\"w\">    <\/span><span class=\"kt\">uint16_t<\/span><span class=\"w\"> <\/span><span class=\"n\">frac_seconds<\/span><span class=\"p\">;<\/span><span class=\"w\">  <\/span><span class=\"c1\">\/\/ corresponds to l<\/span>\n<\/span><span data-line=\"4\"><span class=\"w\">    <\/span><span class=\"kt\">uint16_t<\/span><span class=\"w\"> <\/span><span class=\"n\">l_pt<\/span><span class=\"w\"> <\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"mi\">12<\/span><span class=\"p\">;<\/span><span class=\"w\">     <\/span><span class=\"c1\">\/\/ corresponds to l - pt<\/span>\n<\/span><span data-line=\"5\"><span class=\"w\">    <\/span><span class=\"kt\">uint8_t<\/span><span class=\"w\"> <\/span><span class=\"n\">c<\/span><span class=\"w\"> <\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"mi\">4<\/span><span class=\"p\">;<\/span>\n<\/span><span data-line=\"6\"><span class=\"p\">};<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<p>Or, making use of the bare union extension:<\/p>\n<div class=\"highlight-c notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"k\">struct<\/span><span class=\"w\"> <\/span><span class=\"nc\">HLC_stamp<\/span><span class=\"w\"> <\/span><span class=\"p\">{<\/span><span class=\"w\">  <\/span><span class=\"c1\">\/\/ still big-endian<\/span>\n<\/span><span data-line=\"2\"><span class=\"w\">    <\/span><span class=\"k\">union<\/span><span class=\"w\"> <\/span><span class=\"p\">{<\/span>\n<\/span><span data-line=\"3\"><span class=\"w\">        <\/span><span class=\"k\">struct<\/span><span class=\"w\"> <\/span><span class=\"p\">{<\/span>\n<\/span><span data-line=\"4\"><span class=\"w\">            <\/span><span class=\"kt\">uint64_t<\/span><span class=\"w\"> <\/span><span class=\"n\">l<\/span><span class=\"w\"> <\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"mi\">48<\/span><span class=\"p\">;<\/span>\n<\/span><span data-line=\"5\"><span class=\"w\">            <\/span><span class=\"kt\">uint16_t<\/span><span class=\"w\"> <\/span><span class=\"n\">l_pt<\/span><span class=\"w\"> <\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"mi\">12<\/span><span class=\"p\">;<\/span>\n<\/span><span data-line=\"6\"><span class=\"w\">            <\/span><span class=\"kt\">uint8_t<\/span><span class=\"w\"> <\/span><span class=\"n\">c<\/span><span class=\"w\"> <\/span><span class=\"o\">:<\/span><span class=\"w\"> <\/span><span class=\"mi\">4<\/span><span class=\"p\">;<\/span>\n<\/span><span data-line=\"7\"><span class=\"w\">        <\/span><span class=\"p\">};<\/span>\n<\/span><span data-line=\"8\"><span class=\"w\">        <\/span><span class=\"k\">struct<\/span><span class=\"w\"> <\/span><span class=\"p\">{<\/span>\n<\/span><span data-line=\"9\"><span class=\"w\">            <\/span><span class=\"kt\">uint32_t<\/span><span class=\"w\"> <\/span><span class=\"n\">seconds<\/span><span class=\"p\">;<\/span>\n<\/span><span data-line=\"10\"><span class=\"w\">            <\/span><span class=\"kt\">uint32_t<\/span><span class=\"w\"> <\/span><span class=\"n\">frac_seconds<\/span><span class=\"p\">;<\/span><span class=\"w\">  <\/span><span class=\"c1\">\/\/ always assign this *before* c, never after<\/span>\n<\/span><span data-line=\"11\"><span class=\"w\">        <\/span><span class=\"p\">};<\/span>\n<\/span><span data-line=\"12\"><span class=\"w\">    <\/span><span class=\"p\">};<\/span>\n<\/span><span data-line=\"13\"><span class=\"p\">};<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<p>Note that the latter version is more faithful to the original paper,\nsince <span class=\"math notranslate nohighlight\">\\(c\\)<\/span> is fully incorporated into the fractional seconds field.\nThis, in effect, means that the fractional seconds field has an <em>actual<\/em>\nprecision of <span class=\"math notranslate nohighlight\">\\(2^{-16}\\)<\/span>, but a <em>perceived<\/em> precision of\n<span class=\"math notranslate nohighlight\">\\(2^{-32}\\)<\/span>, with the least significant bits corresponding entirely\nto the <span class=\"math notranslate nohighlight\">\\(l - pt\\)<\/span> and <span class=\"math notranslate nohighlight\">\\(c\\)<\/span> values.<\/p>\n<p>From here they perform some experiments to show that there is\nessentially no overhead to doing this, as compared to physical time\nsynchronization.<\/p>\n<\/section>\n<\/section>\n<section id=\"my-thoughts\">\n<h2>My Thoughts<a class=\"headerlink\" href=\"#my-thoughts\" title=\"Link to this heading\">\u00b6<\/a><\/h2>\n<p>This is a very interesting way to go about things. Coming out of it, my\nbiggest question is whether, given enough additional information, one\ncould use this to replace Vector Clocks. As it currently stands, that\nwould be a poor decision in small systems, but this stands as a very\npromising addition to peer-to-peer protocols.<\/p>\n<p>This is a topic that my group in CSE 812 will be incorporating into our\nresearch. Sometime in the next few days I hope to have a post up about\nthe other paper we are basing things on. It is significantly longer, but\nalso can be condensed more easily if I approach things right.<\/p>\n<p>Let me know what you think below!<\/p>\n<p class=\"rubric\">Citations<\/p>\n<div class=\"docutils container\" id=\"id4\">\n<div role=\"list\" class=\"citation-list\">\n<div class=\"citation\" id=\"id78\" role=\"doc-biblioentry\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span><a role=\"doc-backlink\" href=\"#id2\">Bre<\/a><span class=\"fn-bracket\">]<\/span><\/span>\n<p>Eric Brewer. Spanner, truetime and the cap theorem.<\/p>\n<\/div>\n<div class=\"citation\" id=\"id98\" role=\"doc-biblioentry\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span><a role=\"doc-backlink\" href=\"#id1\">KDM+14<\/a><span class=\"fn-bracket\">]<\/span><\/span>\n<p>Sandeep\u00a0S. Kulkarni, Murat Demirbas, Deepak Madappa, Bharadwaj Avva, and Marcelo Leone. Logical physical clocks. In Marcos\u00a0K. Aguilera, Leonardo Querzoni, and Marc Shapiro, editors, <em>Principles of Distributed Systems<\/em>, 17\u201332. Cham, 2014. Springer International Publishing.<\/p>\n<\/div>\n<div class=\"citation\" id=\"id102\" role=\"doc-biblioentry\">\n<span class=\"label\"><span class=\"fn-bracket\">[<\/span><a role=\"doc-backlink\" href=\"#id3\">KR01<\/a><span class=\"fn-bracket\">]<\/span><\/span>\n<p>Sandeep\u00a0S. Kulkarni and Ravikant. Stabilizing causal deterministic merge. In <em>Proceedings of the 5th International Workshop on Self-Stabilizing Systems<\/em>, WSS '01, 183\u2013199. Berlin, Heidelberg, 2001. Springer-Verlag.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<p class=\"rubric\">Cite As<\/p>\n<details><summary>Click here to expand the bibtex entry.<\/summary><div class=\"docutils container\">\n<blockquote>\n<div><div class=\"highlight-bibtex notranslate\"><div class=\"highlight\"><pre><span><\/span><span data-line=\"1\"><span class=\"nc\">@online<\/span><span class=\"p\">{<\/span><span class=\"nl\">appleton_blog_0001<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"2\"><span class=\"w\">  <\/span><span class=\"na\">title<\/span><span class=\"w\">    <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ Hybrid Logical Clocks: Summary }<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"3\"><span class=\"w\">  <\/span><span class=\"na\">author<\/span><span class=\"w\">   <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ Olivia Appleton-Crocker }<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"4\"><span class=\"w\">  <\/span><span class=\"na\">language<\/span><span class=\"w\"> <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ English }<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"5\"><span class=\"w\">  <\/span><span class=\"na\">version<\/span><span class=\"w\">  <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ 1.1 }<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"6\"><span class=\"w\">  <\/span><span class=\"na\">date<\/span><span class=\"w\">     <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ 2020-03-02 }<\/span><span class=\"p\">,<\/span>\n<\/span><span data-line=\"7\"><span class=\"w\">  <\/span><span class=\"na\">url<\/span><span class=\"w\">      <\/span><span class=\"p\">=<\/span><span class=\"w\"> <\/span><span class=\"s\">{ https:\/\/blog.oliviaappleton.com\/posts\/0001-HLC-review }<\/span>\n<\/span><span data-line=\"8\"><span class=\"p\">}<\/span>\n<\/span><\/pre><\/div>\n<\/div>\n<\/div><\/blockquote>\n<\/div>\n<\/details><p class=\"tags\"><span>Tags: <\/span><a class=\"reference external\" href=\"..\/_tags\/research-review.html\">research-review<\/a><span>, <\/span><a class=\"reference external\" href=\"..\/_tags\/cse-812.html\">cse-812<\/a><span>, <\/span><a class=\"reference external\" href=\"..\/_tags\/advisee-group.html\">advisee-group<\/a><span>, <\/span><a class=\"reference external\" href=\"..\/_tags\/msu.html\">msu<\/a><span>, <\/span><a class=\"reference external\" href=\"..\/_tags\/distributed-systems.html\">distributed-systems<\/a><\/p>\n\n            <h2>\n                Comments\n                <span class=\"comments-info\">\n                    <img class=\"fedi-icon\" src=\"https:\/\/blog.oliviaappleton.com\/_static\/boost.svg\" alt=\"Boosts\">\n                    <span id=\"global-reblogs\"><\/span>,\n                    <img class=\"fedi-icon\" src=\"https:\/\/blog.oliviaappleton.com\/_static\/like.svg\" alt=\"Likes\">\n                    <span id=\"global-likes\"><\/span>\n                <\/span>\n            <\/h2>\n            <div id=\"comments-section\"><\/div>\n            <script>\n                document.addEventListener(\"DOMContentLoaded\", function () {\n                    setConfig({\n                        boostLink: \"https:\/\/blog.oliviaappleton.com\/_static\/boost.svg\",\n                        allowSensitiveEmoji: false,\n                        allowCustomEmoji: true,\n                        allowMediaAttachments: true,\n                        allowAvatars: true,\n                        delayCommentLoad: true,\n                        defaultReactionEmoji: '\u2764',\n                        retryDelay: 100,\n                    });\n                    fetchComments('misskey', 'transfem.social', 'a59ynep3wiiy00w2', 5);\n                });\n            <\/script>\n        <\/section>\n<\/section>\n","guid":"https:\/\/blog.oliviaappleton.com\/posts\/0001-HLC-review.html","category":["research-review","cse-812","advisee-group","msu","distributed-systems"],"pubDate":"Mon, 02 Mar 2020 00:00:00 +0000"}]}}