{"@attributes":{"version":"2.0"},"channel":{"title":"Debian on Website of Jamie McClelland","link":"https:\/\/current.workingdirectory.net\/tags\/debian\/","description":"Recent content in Debian on Website of Jamie McClelland","generator":"Hugo -- gohugo.io","language":"en","lastBuildDate":"Fri, 10 Apr 2026 08:27:10 -0400","item":[{"title":"AI Hacking the Planet","link":"https:\/\/current.workingdirectory.net\/posts\/2026\/ai-hacking-the-planet\/","pubDate":"Fri, 10 Apr 2026 08:27:10 -0400","guid":"https:\/\/current.workingdirectory.net\/posts\/2026\/ai-hacking-the-planet\/","description":"<p>A colleague asked me if we should move all our money to our pillow cases after\nreading the latest AI editorial from <a href=\"https:\/\/www.nytimes.com\/2026\/04\/07\/opinion\/anthropic-ai-claude-mythos.html\">Thomas\nFriedman<\/a>.\nThe article reads like a press release from Anthropic, repeating the claim that\ntheir latest AI model is so good at finding software vulnerabilities that it is\na danger to the world.<\/p>\n<p>I think I now know what it&rsquo;s like to be a doctor who is forced to watch Gray&rsquo;s\nAnatomy.<\/p>\n<p>By now every journalist should be able to recognize the AI publicity playbook:<\/p>\n<p><strong>Step 1:<\/strong> Start with a wildly unsubstantiated claim about how dangerous your\nproduct is:<\/p>\n<p><del><em>AI will cause human extinction before we have a chance to colonize mars<\/em><\/del>\n(remember that one? Even Kim Stanley Robinson, author of perhaps the most\ncompelling science fiction on colonizing mars <a href=\"https:\/\/www.newscientist.com\/article\/2520312-author-of-red-mars-calls-bullshit-on-emigrating-to-the-planet\/\">calls bull\nshit<\/a>\non it).<\/p>\n<p><del><em>AI will eliminate all of our jobs<\/em><\/del> (this one was extremely effective at\nproviding cover for software companies laying off staff but it has quickly\ndawned on people that the companies that did this are living in chaos not\nhumming along happily with functional robots)<\/p>\n<p><em>AI will discover massive software vulnerabilities allowing bad actors to &ldquo;hack\npretty much every major software system in the world&rdquo;.<\/em> (Did Friedman pull that\ndirectly from Anthropic&rsquo;s press release or was that his contribution?)<\/p>\n<p><strong>Step 2:<\/strong> To help stave off human collapse, only release the new version to a\nvetted group of software companies and developers, preferably ones with big\nsocial media followings<\/p>\n<p><strong>Step 3:<\/strong> Wait for the limited release developers to spew unbridled\nenthusiasm and shocking examples that seem to suggest this new AI produce is\ntruly unbelievable<\/p>\n<p><strong>Step 4:<\/strong> Watch stock prices and valuations soar<\/p>\n<p><strong>Step 5:<\/strong> Release to the world, and experience a steady stream of mockery as\npeople discover how wrong you are<\/p>\n<p><strong>Step 6:<\/strong> Start over<\/p>\n<p>Even if Friedman missed the text book example of the playbook, I have to ask:\nif you think bad actors compromising software resulting in massive loss of\nprivate data, major outages and wasted resources needs to be reported on, then\nwhere have you been for the last 10 years? This literally happens <a href=\"https:\/\/www.bleepingcomputer.com\/news\/security\/\">on a daily\nbasis<\/a> due to the\nfundamentally flawed way capitalism has been writing software even before the\ninvention of AI. A small part of me wonders - maybe AI writing software is not\nso bad, because how could it be any worse than it is now?<\/p>\n<p>Also, let&rsquo;s keep in mind that AI&rsquo;s super ability at finding vulnerable software\ndepends on having access to the software&rsquo;s source code, which most companies\nkeep locked up tight. That means the owners of the software can use AI to find\nvulnerabilities and fix them but bad actors can&rsquo;t.<\/p>\n<p>Oh, but wait, what if a company is so incompetent that they <a href=\"https:\/\/www.cnbc.com\/2026\/03\/31\/anthropic-leak-claude-code-internal-source.html\">accidentally\nrelease their proprietary software to the\nInternet<\/a>?<\/p>\n<p>Surely that would allow AI bots to discover their vulnerabilities and destroy\nthe company right? I&rsquo;m not sure if anyone has discovered world ending\nvulnerabilities in Anthropic&rsquo;s Claude code since it was accidentally released,\nbut it is fun to watch people <a href=\"https:\/\/neuromatch.social\/@jonny\/116325668039992121\">mock\nsoftware<\/a> that is clearly\nwritten by AI (and spoiler alert, it seems way worse that software written\nnow).<\/p>\n<p>Well&hellip; we probably should all be keeping our money in a pillow case anyway.<\/p>\n"},{"title":"Mailman3 has 2 databases. Whoops.","link":"https:\/\/current.workingdirectory.net\/posts\/2026\/mailman3-migration\/","pubDate":"Mon, 30 Mar 2026 08:27:10 -0400","guid":"https:\/\/current.workingdirectory.net\/posts\/2026\/mailman3-migration\/","description":"<p>At <a href=\"https:\/\/mayfirst.coop\/\">May First<\/a> we have been carefully planning our\nmigration of about 1200 lists from mailman2 to mailman3 for almost six months\nnow. We did a lot of user communications, had several months of beta testing\nwith a handful of lists ported over, and everything was looking good. So we\nkicked off the migration!<\/p>\n<p>But, about 15% of the way through I started seeing sqlite lock errors. Wait,\nwhat? I carefully re-configured mailman3 to use postgres, not sqlite. Well,\nyes, but apparently that was for the database managing the email list\nconfiguration, not the database powering the django web app, which,\nincidentally, also includes hundresds of gigabytes of archives. In other words,\nthe one we <em>really<\/em> need in postgres, not sqlite.<\/p>\n<h2 id=\"moving-from-sqlite-to-postgres\">Moving from sqlite to postgres<\/h2>\n<p>Well that sucks. We immediately stopped the migration to deal with this.<\/p>\n<p>I noticed that the web is full of useful django instructions on how to migrate\nyour database from one database to antoher. However, if you read the fine\nprint, those convenient looking &ldquo;<code>dumpdata<\/code> <code>loaddata<\/code>&rdquo; workflows are designed\nto move the table definitions and a small amount of data. In our case, even\nafter just 15% of our lists moved, our sqlite database was about 30GB.<\/p>\n<p>I considered some of the hacks to manage memory and try to run this via django,\nbut eventually decided that <a href=\"https:\/\/pgloader.io\/\">pgloader<\/a> was a more robust\noption. This option also allowed me to more easily test things out on a copy of\nour sqlite database (made while mailman was turned off). This way I could\nmigrate and re-migrate the sqlite database over and over without impacting our\nlive installation until I was satisfied it was all working.<\/p>\n<p>My first decision was to opt out of pgloader&rsquo;s schema creation. I used django&rsquo;s\nschema creation tool by:<\/p>\n<ul>\n<li>Turning off mailman3 and mailman3-web and changing the mailman web\nconfiguration to use the new postgresql database.<\/li>\n<li>Running <code>mailman-web migrate<\/code><\/li>\n<li>Changing the mailman web configuration back to sqlite and starting\neverything again.<\/li>\n<\/ul>\n<p>Note: I tried just adding new database settings in the mailman web\nconfiguration indexed to &rsquo;new&rsquo; - django has the ability to define different\ndatabases by name, then you can run <code>mailman-web migrate --database new<\/code>. But,\nduring the migration, I caught django querying the sqlite database for some\nmigrations that required referencing existing fields (specifically hyperkitty&rsquo;s\n<code>0003_thread_starting_email<\/code>). I didn&rsquo;t want any of these steps to touch the\nlive database so I opted for the cleaner approach.<\/p>\n<p>Once I had a clean postgres schema, I dumped it so I could easily return to\nthis spot.<\/p>\n<p>Next I started working on our <code>pgloader<\/code> load file. After a lot of trial and\nerror, I ended with:<\/p>\n<div class=\"highlight\"><pre tabindex=\"0\" class=\"chroma\"><code class=\"language-gdscript3\" data-lang=\"gdscript3\"><span class=\"line\"><span class=\"cl\"><span class=\"n\">LOAD<\/span> <span class=\"n\">DATABASE<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">    <span class=\"n\">FROM<\/span> <span class=\"n\">sqlite<\/span><span class=\"p\">:<\/span><span class=\"o\">\/\/\/<\/span><span class=\"k\">var<\/span><span class=\"o\">\/<\/span><span class=\"n\">lib<\/span><span class=\"o\">\/<\/span><span class=\"n\">mailman3<\/span><span class=\"o\">\/<\/span><span class=\"n\">sqlite<\/span><span class=\"o\">-<\/span><span class=\"n\">postgres<\/span><span class=\"o\">-<\/span><span class=\"n\">migration<\/span><span class=\"o\">\/<\/span><span class=\"n\">mailman3web<\/span><span class=\"o\">.<\/span><span class=\"n\">clean<\/span><span class=\"o\">.<\/span><span class=\"n\">backup<\/span><span class=\"o\">.<\/span><span class=\"n\">db<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">    <span class=\"n\">INTO<\/span> <span class=\"n\">postgresql<\/span><span class=\"p\">:<\/span><span class=\"o\">\/\/<\/span><span class=\"n\">mailmanweb<\/span><span class=\"p\">:<\/span><span class=\"n\">xxxxxxxxxxx<\/span><span class=\"err\">@<\/span><span class=\"n\">localhost<\/span><span class=\"p\">:<\/span><span class=\"mi\">5432<\/span><span class=\"o\">\/<\/span><span class=\"n\">mailmanweb<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"n\">WITH<\/span> <span class=\"n\">data<\/span> <span class=\"n\">only<\/span><span class=\"p\">,<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">    <span class=\"n\">reset<\/span> <span class=\"n\">sequences<\/span><span class=\"p\">,<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">    <span class=\"n\">include<\/span> <span class=\"n\">no<\/span> <span class=\"n\">drop<\/span><span class=\"p\">,<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">    <span class=\"n\">disable<\/span> <span class=\"n\">triggers<\/span><span class=\"p\">,<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">    <span class=\"n\">create<\/span> <span class=\"n\">no<\/span> <span class=\"n\">tables<\/span><span class=\"p\">,<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">    <span class=\"n\">batch<\/span> <span class=\"n\">size<\/span> <span class=\"o\">=<\/span> <span class=\"mi\">5<\/span><span class=\"n\">MB<\/span><span class=\"p\">,<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">    <span class=\"n\">batch<\/span> <span class=\"n\">rows<\/span> <span class=\"o\">=<\/span> <span class=\"mi\">500<\/span><span class=\"p\">,<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">    <span class=\"n\">prefetch<\/span> <span class=\"n\">rows<\/span> <span class=\"o\">=<\/span> <span class=\"mi\">50<\/span><span class=\"p\">,<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">    <span class=\"n\">workers<\/span> <span class=\"o\">=<\/span> <span class=\"mi\">2<\/span><span class=\"p\">,<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">    <span class=\"n\">concurrency<\/span> <span class=\"o\">=<\/span> <span class=\"mi\">1<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"n\">SET<\/span> <span class=\"n\">work_mem<\/span> <span class=\"n\">to<\/span> <span class=\"s1\">&#39;64MB&#39;<\/span><span class=\"p\">,<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">    <span class=\"n\">maintenance_work_mem<\/span> <span class=\"n\">to<\/span> <span class=\"s1\">&#39;512MB&#39;<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"n\">CAST<\/span> <span class=\"n\">type<\/span> <span class=\"n\">datetime<\/span> <span class=\"n\">to<\/span> <span class=\"n\">timestamptz<\/span> <span class=\"n\">drop<\/span> <span class=\"n\">default<\/span> <span class=\"n\">drop<\/span> <span class=\"ow\">not<\/span> <span class=\"n\">null<\/span><span class=\"p\">,<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">    <span class=\"n\">type<\/span> <span class=\"n\">date<\/span> <span class=\"n\">to<\/span> <span class=\"n\">date<\/span> <span class=\"n\">drop<\/span> <span class=\"n\">default<\/span> <span class=\"n\">drop<\/span> <span class=\"ow\">not<\/span> <span class=\"n\">null<\/span><span class=\"p\">,<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">    <span class=\"n\">type<\/span> <span class=\"ne\">int<\/span> <span class=\"n\">when<\/span> <span class=\"p\">(<\/span><span class=\"o\">=<\/span> <span class=\"n\">precision<\/span> <span class=\"mi\">1<\/span><span class=\"p\">)<\/span> <span class=\"n\">to<\/span> <span class=\"n\">boolean<\/span> <span class=\"n\">using<\/span> <span class=\"n\">tinyint<\/span><span class=\"o\">-<\/span><span class=\"n\">to<\/span><span class=\"o\">-<\/span><span class=\"n\">boolean<\/span><span class=\"p\">,<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">    <span class=\"n\">type<\/span> <span class=\"n\">text<\/span> <span class=\"n\">to<\/span> <span class=\"n\">varchar<\/span> <span class=\"n\">using<\/span> <span class=\"n\">remove<\/span><span class=\"o\">-<\/span><span class=\"n\">null<\/span><span class=\"o\">-<\/span><span class=\"n\">characters<\/span><span class=\"p\">;<\/span>\n<\/span><\/span><\/code><\/pre><\/div><p>The batch, prefetch, workers and concurreny settings are all there to ensure\nmemory doesn&rsquo;t blow up.<\/p>\n<p>I also discovered that I had to make some changes to the schema before loading\ndata. Mostly truncating tables that the django migrate command populated to\navoid duplicate key errors:<\/p>\n<div class=\"highlight\"><pre tabindex=\"0\" class=\"chroma\"><code class=\"language-zed\" data-lang=\"zed\"><span class=\"line\"><span class=\"cl\"><span class=\"n\">TRUNCATE<\/span><span class=\"w\"> <\/span><span class=\"n\">TABLE<\/span><span class=\"w\"> <\/span><span class=\"n\">django_migrations<\/span><span class=\"w\"> <\/span><span class=\"n\">CASCADE<\/span><span class=\"p\">;<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"w\"><\/span><span class=\"n\">TRUNCATE<\/span><span class=\"w\"> <\/span><span class=\"n\">TABLE<\/span><span class=\"w\"> <\/span><span class=\"n\">django_content_type<\/span><span class=\"w\"> <\/span><span class=\"n\">CASCADE<\/span><span class=\"p\">;<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"w\"><\/span><span class=\"n\">TRUNCATE<\/span><span class=\"w\"> <\/span><span class=\"n\">TABLE<\/span><span class=\"w\"> <\/span><span class=\"n\">auth_permission<\/span><span class=\"w\"> <\/span><span class=\"n\">CASCADE<\/span><span class=\"p\">;<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"w\"><\/span><span class=\"n\">TRUNCATE<\/span><span class=\"w\"> <\/span><span class=\"n\">TABLE<\/span><span class=\"w\"> <\/span><span class=\"n\">django_site<\/span><span class=\"w\"> <\/span><span class=\"n\">CASCADE<\/span><span class=\"p\">;<\/span><span class=\"w\">\n<\/span><\/span><\/span><\/code><\/pre><\/div><p>And also, I had to change a column type. Apparently the mailman import process\nallowed an attachment file name that exceeds the limit for postgres, but was\nallowed into sqlite:<\/p>\n<div class=\"highlight\"><pre tabindex=\"0\" class=\"chroma\"><code class=\"language-fallback\" data-lang=\"fallback\"><span class=\"line\"><span class=\"cl\">ALTER TABLE hyperkitty_attachment ALTER COLUMN name TYPE text\n<\/span><\/span><\/code><\/pre><\/div><p>When <code>pgloader<\/code> runs, we still get a lot of warnings from pgloader, which wants\nto cast columns differently than django does. These are harmless (I was able to\nimport the data without a problem).<\/p>\n<p>And there are still a lot of warnings along the lines of:<\/p>\n<blockquote>\n<p>2026-03-30T14:08:01.691990Z WARNING PostgreSQL warning: constraint &ldquo;hyperkitty_vote_email_id_73a50f4d_fk_hyperkitty_email_id&rdquo; of relation &ldquo;hyperkitty_vote&rdquo; does not exist, skipping<\/p><\/blockquote>\n<p>These are harmless as well. They appear because <code>disable triggers<\/code> disables\nforeign key constraints. Without it, we wouldn&rsquo;t be able to load tables that\nrequire values in tables that have not yet been populated.<\/p>\n<p>After all the tweaking, the import of our 30GB sqlite database took about 40\nminutes.<\/p>\n<h2 id=\"final-steps\">Final Steps<\/h2>\n<p>I think the <code>reset sequences<\/code> from <code>pgloader<\/code> should take care of this, but just in case:<\/p>\n<div class=\"highlight\"><pre tabindex=\"0\" class=\"chroma\"><code class=\"language-fallback\" data-lang=\"fallback\"><span class=\"line\"><span class=\"cl\">mailman-web sqlsequencereset hyperkitty mailman_django auth | mailman-web dbshell\n<\/span><\/span><\/code><\/pre><\/div><p>And, just to ensure postgres is optimized, run this in the psql shell:<\/p>\n<div class=\"highlight\"><pre tabindex=\"0\" class=\"chroma\"><code class=\"language-fallback\" data-lang=\"fallback\"><span class=\"line\"><span class=\"cl\">ANALYZE VERBOSE;\n<\/span><\/span><\/code><\/pre><\/div><h2 id=\"last-thoughts\">Last thoughts<\/h2>\n<p>I understand very well all the decisions the mailman3 devs made in designing\nthe next version of mailman, and if I was in the same place I may have made\nthem the same ones. For example, separating the code running the mailing list\nfrom the code managing the archives and the web interface makes perfectly good\nsense - many people might want to run just the mailing list part without a web\ninterface. And building the web interface in django makes a lot of sense as\nwell - why re-invent the wheel? I&rsquo;m sure a lot of time and effort was saved by\nsimply using the built in features you get for free with django.<\/p>\n<p>But the unfortunate consequence of these decisions is that sys admins have a\nmuch harder time. Almost everyone wants the email lists along with the web\ninterface and the archives. But nobody wants two different configuration files\nwith different syntaxes and logic, not to mention two different command lines\nto use for maintenance and configuration with completely different APIs. Trying\nto understand how to change a default template or set list defaults requires a\nlot of research and usually you have to write a python script to do it.<\/p>\n<p>I have finally come to the conclusion that mailman2 is designed for sys admins,\nwhile mailman3 is designed for developers.<\/p>\n<p>Despite these short comings, I am impressed with the community and their quick\nand friendly responses to the questions of a confused sys admin. That might be\nmore valuable than anything else.<\/p>\n"},{"title":"Avoiding Apache Max Request Workers Errors","link":"https:\/\/current.workingdirectory.net\/posts\/2025\/avoiding-max-request-workers-errors\/","pubDate":"Fri, 11 Jul 2025 08:27:10 -0400","guid":"https:\/\/current.workingdirectory.net\/posts\/2025\/avoiding-max-request-workers-errors\/","description":"<p><em>[Update 2025-10-09&hellip; turns out my first great solution doesn&rsquo;t work after\nall, updated to reflect new options.]<\/em><\/p>\n<p>Wow, I hate this error:<\/p>\n<blockquote>\n<p>AH00484: server reached MaxRequestWorkers setting, consider raising the\nMaxRequestWorkers setting<\/p><\/blockquote>\n<p>Or, the variation:<\/p>\n<blockquote>\n<p>AH03490: scoreboard is full, not at MaxRequestWorkers. Increase ServerLimit.<\/p><\/blockquote>\n<p>For starters, it means I have to relearn how <code>MaxRequestWorkers<\/code> functions in Apache:<\/p>\n<blockquote>\n<p>For threaded and hybrid servers (e.g. event or worker), MaxRequestWorkers\nrestricts the total number of threads that will be available to serve\nclients. For hybrid MPMs, the default value is 16 (ServerLimit) multiplied by\nthe value of 25 (ThreadsPerChild). Therefore, to increase MaxRequestWorkers\nto a value that requires more than 16 processes, you must also raise\nServerLimit.<\/p><\/blockquote>\n<p>Ok&hellip; remind me what <code>ServerLimit<\/code> refers to?<\/p>\n<blockquote>\n<p>For the prefork MPM, this directive sets the maximum configured value for\nMaxRequestWorkers for the lifetime of the Apache httpd process. For the\nworker and event MPMs, this directive in combination with ThreadLimit sets\nthe maximum configured value for MaxRequestWorkers for the lifetime of the\nApache httpd process. For the event MPM, this directive also defines how many\nold server processes may keep running and finish processing open connections.\nAny attempts to change this directive during a restart will be ignored, but\nMaxRequestWorkers can be modified during a restart. Special care must be\ntaken when using this directive. If ServerLimit is set to a value much higher\nthan necessary, extra, unused shared memory will be allocated. If both\nServerLimit and MaxRequestWorkers are set to values higher than the system\ncan handle, Apache httpd may not start or the system may become unstable.\nWith the prefork MPM, use this directive only if you need to set\nMaxRequestWorkers higher than 256 (default). Do not set the value of this\ndirective any higher than what you might want to set MaxRequestWorkers to.\nWith worker, use this directive only if your MaxRequestWorkers and\nThreadsPerChild settings require more than 16 server processes (default). Do\nnot set the value of this directive any higher than the number of server\nprocesses required by what you may want for MaxRequestWorkers and\nThreadsPerChild. With event, increase this directive if the process number\ndefined by your MaxRequestWorkers and ThreadsPerChild settings, plus the\nnumber of gracefully shutting down processes, is more than 16 server\nprocesses (default).<\/p><\/blockquote>\n<p>Got it? In other words, you can &ldquo;consider&rdquo; raising the <code>MaxRequestWorkers<\/code>\nsetting all you want, but you can&rsquo;t just change that setting, you have to read\nabout several other compliated settings, do some math, and spend a lot of time\nwondering if you are going to remember what you just did and how to undo it if\nyou blow up your server.<\/p>\n<p>On the plus side, typically, nobody should increase this limit - because if the\nserver runs out of connections, it usually means something else is wrong.<\/p>\n<p>In our case, on a shared web server running Apache2 and PHP-FPM, it&rsquo;s usually\nbecause a single web site has gone out of control.<\/p>\n<p><em>But wait! How can that happen, we are using PHP-FPM&rsquo;s <code>max_children<\/code> setting to prevent\na single PHP web site from taking down the server?<\/em><\/p>\n<p>After years of struggling with this problem I have finally made some headway.<\/p>\n<p>Our PHP pool configuration typically looks like this:<\/p>\n<div class=\"highlight\"><pre tabindex=\"0\" class=\"chroma\"><code class=\"language-fallback\" data-lang=\"fallback\"><span class=\"line\"><span class=\"cl\">user = site342999writer\n<\/span><\/span><span class=\"line\"><span class=\"cl\">group = site342999writer\n<\/span><\/span><span class=\"line\"><span class=\"cl\">listen = \/run\/php\/8.1-site342999.sock\n<\/span><\/span><span class=\"line\"><span class=\"cl\">listen.owner = www-data\n<\/span><\/span><span class=\"line\"><span class=\"cl\">listen.group = www-data\n<\/span><\/span><span class=\"line\"><span class=\"cl\">pm = ondemand\n<\/span><\/span><span class=\"line\"><span class=\"cl\">pm.max_children = 12\n<\/span><\/span><span class=\"line\"><span class=\"cl\">pm.max_requests = 500\n<\/span><\/span><span class=\"line\"><span class=\"cl\">php_admin_value[memory_limit] = 256M\n<\/span><\/span><\/code><\/pre><\/div><p>And we invoke PHP-FPM via this apache snippet:<\/p>\n<div class=\"highlight\"><pre tabindex=\"0\" class=\"chroma\"><code class=\"language-gdscript3\" data-lang=\"gdscript3\"><span class=\"line\"><span class=\"cl\"><span class=\"o\">&lt;<\/span><span class=\"n\">FilesMatch<\/span> \\<span class=\"o\">.<\/span><span class=\"n\">php<\/span><span class=\"o\">$&gt;<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">        <span class=\"n\">SetHandler<\/span> <span class=\"s2\">&#34;proxy:unix:\/var\/run\/php\/8.1-site342999.sock|fcgi:\/\/localhost&#34;<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"o\">&lt;\/<\/span><span class=\"n\">FilesMatch<\/span><span class=\"o\">&gt;<\/span>\n<\/span><\/span><\/code><\/pre><\/div><p>With these settings in place, what happens when we use up all 12 <code>max_children<\/code>?<\/p>\n<p>According to <a href=\"https:\/\/httpd.apache.org\/docs\/2.4\/mod\/mod_proxy.html#proxypass\">the docs<\/a>:<\/p>\n<blockquote>\n<p>By default, <code>mod_proxy<\/code> will allow and retain the maximum number of connections\nthat could be used simultaneously by that web server child process. Use the\nmax parameter to reduce the number from the default. The pool of connections\nis maintained per web server child process, and max and other settings are\nnot coordinated among all child processes, except when only one child process\nis allowed by configuration or MPM design.<\/p><\/blockquote>\n<p>The <code>max<\/code> parameter seems to default to the <code>ThreadsPerChild<\/code>, so it seems that\nthe default here is to allow any web site to consume <code>ThreadsPerChild<\/code> (25) x\n<code>ServerLimit<\/code> (16), which is also the max number of over all connections. Not\ngreat.<\/p>\n<p>To make matter worse, there is another setting available which is mysteriously\ncalled <code>acquire<\/code>:<\/p>\n<blockquote>\n<p>If set, this will be the maximum time to wait for a free connection in the\nconnection pool, in milliseconds. If there are no free connections in the\npool, the Apache httpd will return <code>SERVER_BUSY<\/code> status to the client.<\/p><\/blockquote>\n<p>By default this is not set which seems to suggest Apache will just hang on to\nconnections forever until a free PHP process becomes available (or some other\ntime out happens).<\/p>\n<p>So, let&rsquo;s try something different:<\/p>\n<div class=\"highlight\"><pre tabindex=\"0\" class=\"chroma\"><code class=\"language-fallback\" data-lang=\"fallback\"><span class=\"line\"><span class=\"cl\"> &lt;Proxy &#34;fcgi:\/\/localhost&#34;&gt;\n<\/span><\/span><span class=\"line\"><span class=\"cl\">    ProxySet acquire=1 max=12\n<\/span><\/span><span class=\"line\"><span class=\"cl\">  &lt;\/proxy&gt;\n<\/span><\/span><\/code><\/pre><\/div><p>This snippet seems to be the way you can configure the proxy configuration we\nsetup in the <code>SetHandler<\/code> statement above. It&rsquo;s documented on the <a href=\"https:\/\/httpd.apache.org\/docs\/2.4\/mod\/mod_proxy.html#proxypass\">Apache\nmod_proxy\npage<\/a>.<\/p>\n<p>Unfortunately, this does not work. I tried all kinds of different combinations\nbut my best guess is that the use of <code>max<\/code> and <code>acquire<\/code> are reserved for tcp\nconnections not unix socket connections, so the only way to achieve this would\nbe to switch our PHP FPM configuration to work over <code>127.0.0.1<\/code> instead of unix\nsockets, which would bring it&rsquo;s own problems.<\/p>\n<p><strong>Now what?<\/strong><\/p>\n<p>I can see two options:<\/p>\n<ol>\n<li>We already pass all traffic via an nginx proxy before it even hits one of\nour apache back end servers. So, rather than configure just one nginx\n<a href=\"https:\/\/nginx.org\/en\/docs\/http\/ngx_http_upstream_module.html#upstream\">upstream<\/a>,\nwe can assign each site their very own upstream with their very own\n<code>max_conn<\/code> settings. It feels ugly and wasteful to have one upstream per\nsite on a shared server, but it works.<\/li>\n<li>Install an unsupported apache module. I found\n<a href=\"https:\/\/github.com\/IvnSoft\/mod_vhost_limit\">mod_vhost_limit<\/a>, whose very\nexistence seems to confirm my failed struggle at getting this to work. It\nwas written for Redhat and hasn&rsquo;t been touched in 5 years, but I managed to\nget it to work on Debian Trixie <a href=\"https:\/\/github.com\/IvnSoft\/mod_vhost_limit\/issues\/2#issuecomment-3386234000\">without much\neffort<\/a>.\nAnd when I tested, it worked on the first try.<\/li>\n<\/ol>\n"},{"title":"AI's Actual Impact","link":"https:\/\/current.workingdirectory.net\/posts\/2025\/ai-changes\/","pubDate":"Fri, 21 Mar 2025 08:27:10 -0400","guid":"https:\/\/current.workingdirectory.net\/posts\/2025\/ai-changes\/","description":"<p>Two years after OpenAI launched ChatGPT 3.5, humanity is not on the cusp of\nextinction and Elon Musk seems more responsible for job loss than any AI agent.<\/p>\n<p>However, ask any web administrator and you will learn that <a href=\"https:\/\/thelibre.news\/foss-infrastructure-is-under-attack-by-ai-companies\/\">large language\nmodels are having a significant impact on the world wide\nweb<\/a>\n(or, for a less technical account, see <a href=\"https:\/\/www.forbes.com\/sites\/emmawoollacott\/2024\/04\/16\/yes-the-bots-really-are-taking-over-the-internet\/\">Forbes articles on\nbots<\/a>).\nAt <a href=\"https:\/\/mayfirst.coop\/\">May First<\/a>, a membership organization that has been\nsupporting thousands of web site for over 20 years, we have never seen anything\nlike this before.<\/p>\n<h2 id=\"wow-my-site-is-getting-really-popular\">Wow, my site is getting really popular<\/h2>\n<p>It started in 2023. Web sites that performed quite well with a steady\nviewership started having traffic spikes. These were relatively easy to\ndiagnose, since most of the spikes came from visitors that properly identified\nthemselves as bots, allowing us to see that the big players - OpenAI, Bing,\nGoogle, Facebook - were increasing their efforts to scrape as much content from\nweb sites as possible.<\/p>\n<p>Small brochure sites were mostly unaffected because they could be scraped in a\nmatter of minutes. But large sites with an archive of high quality human\nwritten content were getting hammered. Any web site with a search feature or a\ncalendar or any interface that generated exponential hits that could be\nfollowed were particularly vulnerable.<\/p>\n<p>But hey, that&rsquo;s what <a href=\"http:\/\/www.robotstxt.org\/\">robots.txt<\/a> is for, right? To\ntell robots to back off if you don&rsquo;t want them scraping your site?<\/p>\n<p>Eventually, the cracks began to show. Bots were ignoring robots.txt (did they\never pay that much attention to it in the first place?). Furthermore, rate\nlimiting requests by user agent also began to fail. When you post a link on\nFacebook, a bot identifying itself as &ldquo;facebooketernalhit&rdquo; is invoked to\npreview the page so it can show a picture and other meta data. We don&rsquo;t want to\nrate limit that bot, right? Except, Facebook is also using this bot to scrape\nyour site, often bringing your site to its knees. And don&rsquo;t get me started on\nTwitterBot.<\/p>\n<p>Eventually, it became clear that the majority of the armies of bots scraping\nour sites have completely given up on identifying themselves as bots and are\ninstead using user agents indistinguishable from regular browsers. By using\nthousands of different IP addresses, it has become really hard to separate the\nreal humans from the bots.<\/p>\n<h2 id=\"now-what\">Now what?<\/h2>\n<p>So, no, unfortunately, your web site is not suddenly getting really popular.\nAnd, you are blessed with a whole new set of strategic decisions.<\/p>\n<p>Fortunately, May First has undergone a major infrastructure transition,\nresulting in centralized logging of all web sites and a fleet of web proxy\nservers that intercept all web traffic. Centralized logging means we can\nanalyze traffic and identify bots more easily, and a web proxy fleet allows us\nto more easily implement rules across all web sites.<\/p>\n<p>However, even with all of our latest changes and hours upon hours of work to\nkeep out the bots, our members are facing some hard decisions about maintaining\nan open web.<\/p>\n<p>One member of May First provides Google translations of their web site to every\nlanguage available. But wow, that is now a disaster because instead of having\nevery bot under the sun scrapping all 843 (a made up number) pieces of unique\ncontent on their site, the same bots are scraping 843 * (number of available\nlanguages) pieces of content on their site. Should they stop providing this\ntranslation service in order to ensure people can access their site in the\nsite&rsquo;s primary language?<\/p>\n<p>Should web sites turn off their search features that include drop down options\nof categories to prevent bots from systematically refreshing the search page\nwith every possible combination of search terms?<\/p>\n<p>Do we need to alter our calendar software to avoid providing endless links into\nthe future (ok, that is an easy one)?<\/p>\n<h2 id=\"whats-next\">What&rsquo;s next?<\/h2>\n<p>Something has to change.<\/p>\n<ul>\n<li>\n<p><strong>Lock down web 2.0.<\/strong> Web 2.0 brought us wonderful dynamic web sites, which\nDrupal and WordPress and many other pieces of amazing software have\nsupported for over a decade. This is the software that is getting bogged\ndown by bots. Maybe we need to figure out a way to lock down the dynamic\naspects of this software to logged in users and provide static content for\neveryone else?<\/p>\n<\/li>\n<li>\n<p><strong>Paywalls and accounts everywhere.<\/strong> There&rsquo;s always been an amazing\nnon-financial reward to providing a web site with high quality movement\noriented content for free. It populates the search engines, provides links\nto inspiring and useful content in moments of crises, and can galvanize\nmovements. But these moments of triumph happen between long periods of hard\nlabor that now seems to mostly feed capitalist AI scumbags. If we add a new\nset of expenses and labor to keep the sites running for this purpose, how\nsustainable is that? Will our treasure of free movement content have to move\nbehind paywalls or logins? If we provide logins, will that keep the bots out\nor just create a small hurdle for them to automate the account creation\nprocess? What happens when we can&rsquo;t search for this kind of content via\nsearch engines?<\/p>\n<\/li>\n<li>\n<p><strong>Cutting deals.<\/strong> What if our movement content providers are forced to cut\ndeals with the AI entrepreneurs to allow the paying scumbags to fund the content\ncreation. Eww. Enough said.<\/p>\n<\/li>\n<li>\n<p><strong>Bot detection.<\/strong> Maybe we just need to get better at bot detection? This\nwill surely be an arms race, but would have some good benefits. Bots have\nalso been filling out our forms and populating our databases with spam,\ntesting credit cards against our donation pages, conducting denial of\nservice attacks and all kinds of other irritating acts of vandalism. If we\nwere better at stopping bots automatically it would have a lot of benefits.\nBut what impact would it have on our web sites and the experience of using\nthem? What about &ldquo;good&rdquo; bots (RSS feed readers, payment processors,\nweb hooks, uptime detectors)? Will we cut the legs off any developer trying\nto automate something?<\/p>\n<\/li>\n<\/ul>\n<p>I&rsquo;m not really sure where this is going, but it seems that the world wide web is\nabout to head in a new direction.<\/p>\n"},{"title":"How do I warm up an IP Address?","link":"https:\/\/current.workingdirectory.net\/posts\/2024\/how-do-i-warm-up-an-ip\/","pubDate":"Sat, 21 Sep 2024 08:27:10 -0400","guid":"https:\/\/current.workingdirectory.net\/posts\/2024\/how-do-i-warm-up-an-ip\/","description":"<p>After years on the waiting list, <a href=\"https:\/\/mayfirst.coop\/\">May First<\/a> was just\ngiven a \/24 block of IP addresses. Excellent.<\/p>\n<p>Now we want to start using them for, among other things, sending email.<\/p>\n<p>I haven&rsquo;t added a new IP address to our mail relays in a while and things seems\nto change regularly in the world of email so I&rsquo;m curious: what&rsquo;s the best 2024\nway to warm up IP addresses, particularly using postfix?<\/p>\n<p><a href=\"https:\/\/sendgrid.com\/en-us\/resource\/email-guide-ip-warm-up\">Sendergrid has a nice page on the\ntopic<\/a>. It\nestablishes the number of messages to send per day. But I&rsquo;m not entirely sure\nhow to fit messages per day into our setup.<\/p>\n<p>We use round robin DNS to direct email to one of several dozen email relay\nservers using postfix. And unfortunately our DNS software\n(<a href=\"https:\/\/sendgrid.com\/en-us\/resource\/email-guide-ip-warm-up\">knot<\/a>) doesn&rsquo;t\nhave a way to add weights to ensure some IPs show up more often than others\n(much less limit the specific number of messages a given relay should get).<\/p>\n<p>Postfix has some nice knobs for rate limiting, particularly:\n<a href=\"https:\/\/www.postfix.org\/postconf.5.html#default_destination_recipient_limit\"><code>default_destination_recipient_limit<\/code><\/a>\nand\n<a href=\"https:\/\/www.postfix.org\/postconf.5.html#default_destination_rate_delay\"><code>default_destination_rate_delay<\/code><\/a><\/p>\n<p>If <code>default_destination_recipient_limit<\/code> is over 1, then\n<code>default_destination_rate_delay<\/code> is equal to the minimum delay between sending\nemail to the same domain.<\/p>\n<p>So, I&rsquo;m staring our IP addresses out at 30m - which prevents any single domain\nfrom receiving more than 2 messages per hour. Sadly, there are a lot of\ndifferent domain names that deliver to the same set of popular corporate MX\nservers, so I am not sure I can accurately control how many messages a given\nprovider sees coming from a given IP address. But it&rsquo;s a start.<\/p>\n<p>A bigger problem is that messages that exceed the limit hang out in the\n<em>active<\/em> queue until they can be sent without violating the rate limit. Since I\ncan&rsquo;t fully control the number of messages a given queue receives (due to my\ninability to control the DNS round robin weights), a lot of messages are going\nto be severely delayed, especially ones with an <code>@gmail.com<\/code> domain name.<\/p>\n<p>I know I can temporarily set <code>relayhost<\/code> to a different queue and flush\n<em>deferred<\/em> messages, however, as far as I can tell, it doesn&rsquo;t work with\n<em>active<\/em> messages.<\/p>\n<p>To help mitigate the problem I&rsquo;m only using our bulk mail queue to warm up IPs,\nbut really, this is not ideal.<\/p>\n<p>Suggestions welcome!<\/p>\n<h2 id=\"update-1\">Update #1<\/h2>\n<p>If you are running postfix in a multi-instance setup and you have instances\nthat are already warmed up, you can move active messages between queues with\nthese steps:<\/p>\n<div class=\"highlight\"><pre tabindex=\"0\" class=\"chroma\"><code class=\"language-gdscript3\" data-lang=\"gdscript3\"><span class=\"line\"><span class=\"cl\"><span class=\"c1\"># Put the message on hold in the warming up instance<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"n\">postsuper<\/span> <span class=\"o\">-<\/span><span class=\"n\">c<\/span> <span class=\"o\">\/<\/span><span class=\"n\">etc<\/span><span class=\"o\">\/<\/span><span class=\"n\">postfix<\/span><span class=\"o\">-<\/span><span class=\"n\">warmingup<\/span> <span class=\"o\">-<\/span><span class=\"n\">h<\/span> <span class=\"o\">$<\/span><span class=\"n\">queueid<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"c1\"># Copy to a warmed up instance<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"n\">cp<\/span> <span class=\"o\">--<\/span><span class=\"n\">preserve<\/span><span class=\"o\">=<\/span><span class=\"n\">mode<\/span><span class=\"p\">,<\/span><span class=\"n\">ownership<\/span><span class=\"p\">,<\/span><span class=\"n\">timestamp<\/span> <span class=\"o\">\/<\/span><span class=\"k\">var<\/span><span class=\"o\">\/<\/span><span class=\"n\">spool<\/span><span class=\"o\">\/<\/span><span class=\"n\">postfix<\/span><span class=\"o\">-<\/span><span class=\"n\">warmingup<\/span><span class=\"o\">\/<\/span><span class=\"n\">hold<\/span><span class=\"o\">\/$<\/span><span class=\"n\">queueid<\/span> <span class=\"o\">\/<\/span><span class=\"k\">var<\/span><span class=\"o\">\/<\/span><span class=\"n\">spool<\/span><span class=\"o\">\/<\/span><span class=\"n\">postfix<\/span><span class=\"o\">-<\/span><span class=\"n\">warmedup<\/span><span class=\"o\">\/<\/span><span class=\"n\">incoming<\/span><span class=\"o\">\/<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"c1\"># Queue the message<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"n\">postqueue<\/span> <span class=\"o\">-<\/span><span class=\"n\">c<\/span> <span class=\"o\">\/<\/span><span class=\"n\">etc<\/span><span class=\"o\">\/<\/span><span class=\"n\">postfix<\/span><span class=\"o\">-<\/span><span class=\"n\">warmedup<\/span> <span class=\"o\">-<\/span><span class=\"n\">i<\/span> <span class=\"o\">$<\/span><span class=\"n\">queueid<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"c1\"># Delete from the original queue.<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"n\">postsuper<\/span> <span class=\"o\">-<\/span><span class=\"n\">c<\/span> <span class=\"o\">\/<\/span><span class=\"n\">etc<\/span><span class=\"o\">\/<\/span><span class=\"n\">postfix<\/span><span class=\"o\">-<\/span><span class=\"n\">warmingup<\/span> <span class=\"o\">-<\/span><span class=\"n\">d<\/span> <span class=\"o\">$<\/span><span class=\"n\">queueid<\/span>\n<\/span><\/span><\/code><\/pre><\/div><p>After just 12 hours we had thousands of messages piling up. This warm up method\nwas never going to work without the ability to move them to a faster queue.<\/p>\n<p>[Additional update: be sure to reload the postfix instance after flushing the queue so\nmessages are drained from the active queue on the correct schedule. See update #4.]<\/p>\n<h2 id=\"update-2\">Update #2<\/h2>\n<p>After 24 hours, most email is being accepted as far as I can tell. I am still\ngetting a small percentage of email deferred by Yahoo with:<\/p>\n<blockquote>\n<p>421 4.7.0 [TSS04] Messages from 204.19.241.9 temporarily deferred due to unexpected volume or user complaints - 4.16.55.1; see <a href=\"https:\/\/postmaster.yahooinc.com\/error-codes\">https:\/\/postmaster.yahooinc.com\/error-codes<\/a> (in reply<\/p><\/blockquote>\n<p>So I will keep it as 30m for another 24 hours or so and then move to 15m. Now\nthat I can flush the backlog of active messages I am in less of a hurry.<\/p>\n<h2 id=\"update-3\">Update #3<\/h2>\n<p>Well, this doesn&rsquo;t seem to be working the way I want it to.<\/p>\n<p>When a message arrives faster than the designated rate limit, it remains in the active queue.<\/p>\n<p>I&rsquo;m entirely sure how the timing is supposed to work, but at this point I&rsquo;m\ndown to a 5m rate delay, and the active messages are just hanging out for a lot\nlonger than 5m. I tried flushing the queue, but that only seems to affect the\ndeferred messages. I finally got them re-tried with <code>systemctl reload<\/code>. I\nwonder if there is a setting to control this retry? Or better yet, why can&rsquo;t\nthese messages that exceed the rate delayed be deferred instead?<\/p>\n<h1 id=\"update-4\">Update #4<\/h1>\n<p>I think I see why I was confused in Update #3 about the timing. I suspect that\nwhen I move messages out of the active queue it screws up the timer. Reloading\nthe instance resets the timer. Every time you muck with active messages, you\nshould reload.<\/p>\n"},{"title":"Gmail vs Tor vs Privacy","link":"https:\/\/current.workingdirectory.net\/posts\/2024\/gmail-vs-tor-vs-privacy\/","pubDate":"Wed, 18 Sep 2024 08:27:10 -0400","guid":"https:\/\/current.workingdirectory.net\/posts\/2024\/gmail-vs-tor-vs-privacy\/","description":"<p>A legit email went to spam. Here are the redacted, relevant headers:<\/p>\n<div class=\"highlight\"><pre tabindex=\"0\" class=\"chroma\"><code class=\"language-fallback\" data-lang=\"fallback\"><span class=\"line\"><span class=\"cl\">[redacted]\n<\/span><\/span><span class=\"line\"><span class=\"cl\">X-Spam-Flag: YES\n<\/span><\/span><span class=\"line\"><span class=\"cl\">X-Spam-Level: ******\n<\/span><\/span><span class=\"line\"><span class=\"cl\">X-Spam-Status: Yes, score=6.3 required=5.0 tests=DKIM_SIGNED,DKIM_VALID,\n<\/span><\/span><span class=\"line\"><span class=\"cl\">[redacted]\n<\/span><\/span><span class=\"line\"><span class=\"cl\">\t*  1.0 RCVD_IN_XBL RBL: Received via a relay in Spamhaus XBL\n<\/span><\/span><span class=\"line\"><span class=\"cl\">\t*      [185.220.101.64 listed in xxxxxxxxxxxxx.zen.dq.spamhaus.net]\n<\/span><\/span><span class=\"line\"><span class=\"cl\">\t*  3.0 RCVD_IN_SBL_CSS Received via a relay in Spamhaus SBL-CSS\n<\/span><\/span><span class=\"line\"><span class=\"cl\">\t*  2.5 RCVD_IN_AUTHBL Received via a relay in Spamhaus AuthBL\n<\/span><\/span><span class=\"line\"><span class=\"cl\">\t*  0.0 RCVD_IN_PBL Received via a relay in Spamhaus PBL\n<\/span><\/span><span class=\"line\"><span class=\"cl\">[redacted]\n<\/span><\/span><span class=\"line\"><span class=\"cl\">[very first received line follows...]\n<\/span><\/span><span class=\"line\"><span class=\"cl\">Received: from [10.137.0.13] ([185.220.101.64])\n<\/span><\/span><span class=\"line\"><span class=\"cl\">        by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-378956d2ee6sm12487760f8f.83.2024.09.11.15.05.52\n<\/span><\/span><span class=\"line\"><span class=\"cl\">        for &lt;xxxxx@mayfirst.org&gt;\n<\/span><\/span><span class=\"line\"><span class=\"cl\">        (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128\/128);\n<\/span><\/span><span class=\"line\"><span class=\"cl\">        Wed, 11 Sep 2024 15:05:53 -0700 (PDT)\n<\/span><\/span><\/code><\/pre><\/div><p>At first I though a Gmail IP address was listed in spamhaus - I even opened a\nticket. But then I realized it wasn&rsquo;t the last hop that Spamaus is complaining\nabout, it&rsquo;s the first hop, specifically the ip <code>185.220.101.64<\/code> which appears\nto be a Tor exit node.<\/p>\n<p>The sender is using their own client to relay email directly to Gmail. Like any\nsane person, they don&rsquo;t trust Gmail to protect their privacy, so they are\nsending via Tor. But WTF, Gmail is not stripping the sending IP address from\nthe header.<\/p>\n<p>I&rsquo;m a big fan of harm reduction and have always considered using your own\nclient to relay email with Gmail as a nice way to avoid some of the\nsurveillance tax Google imposes.<\/p>\n<p>However, it <em>seems<\/em> that if you pursue this option you have two unpleasant\nchoices:<\/p>\n<ul>\n<li>Embed your IP address in every email message or<\/li>\n<li>Use Tor and have your email messages go to spam<\/li>\n<\/ul>\n<p>I supposed you could also use a VPN, but I doubt the IP reputation of most VPN\nexit nodes are going to be more reliable than Tor.<\/p>\n"},{"title":"MariaDB mystery","link":"https:\/\/current.workingdirectory.net\/posts\/2024\/mariadb-mystery\/","pubDate":"Wed, 11 Sep 2024 08:27:10 -0400","guid":"https:\/\/current.workingdirectory.net\/posts\/2024\/mariadb-mystery\/","description":"<p>I keep getting an error in our backup logs:<\/p>\n<div class=\"highlight\"><pre tabindex=\"0\" class=\"chroma\"><code class=\"language-fallback\" data-lang=\"fallback\"><span class=\"line\"><span class=\"cl\">Sep 11 05:08:03 Warning: mysqldump: Error 2013: Lost connection to server during query when dumping table `1C4Uonkwhe_options` at row: 1402\n<\/span><\/span><span class=\"line\"><span class=\"cl\">Sep 11 05:08:03 Warning: Failed to dump mysql databases ic_wp\n<\/span><\/span><\/code><\/pre><\/div><p>It&rsquo;s a WordPress database having trouble dumping the options table.<\/p>\n<p>The error log has a corresponding message:<\/p>\n<div class=\"highlight\"><pre tabindex=\"0\" class=\"chroma\"><code class=\"language-fallback\" data-lang=\"fallback\"><span class=\"line\"><span class=\"cl\">Sep 11 13:50:11 mysql007 mariadbd[580]: 2024-09-11 13:50:11 69577 [Warning] Aborted connection 69577 to db: &#39;ic_wp&#39; user: &#39;root&#39; host: &#39;localhost&#39; (Got an error writing communication packets)\n<\/span><\/span><\/code><\/pre><\/div><p>The Internet is full of suggestions, almost all of which either focus on the\nnetwork connection between the client and the server or the FEDERATED plugin.\nWe aren&rsquo;t using the federated plugin and this error happens when conneting via\nthe socket.<\/p>\n<p>Check it out - what is better than a consistently reproducible problem!<\/p>\n<p>It happens if I try to select all the values in the table:<\/p>\n<div class=\"highlight\"><pre tabindex=\"0\" class=\"chroma\"><code class=\"language-fallback\" data-lang=\"fallback\"><span class=\"line\"><span class=\"cl\">root@mysql007:~# mysql --protocol=socket -e &#39;select * from 1C4Uonkwhe_options&#39; ic_wp &gt; \/dev\/null\n<\/span><\/span><span class=\"line\"><span class=\"cl\">ERROR 2013 (HY000) at line 1: Lost connection to server during query\n<\/span><\/span><span class=\"line\"><span class=\"cl\">root@mysql007:~#\n<\/span><\/span><\/code><\/pre><\/div><p>It happens when I specifiy one specific offset:<\/p>\n<div class=\"highlight\"><pre tabindex=\"0\" class=\"chroma\"><code class=\"language-fallback\" data-lang=\"fallback\"><span class=\"line\"><span class=\"cl\">root@mysql007:~# mysql --protocol=socket -e &#39;select * from 1C4Uonkwhe_options limit 1 offset 1402&#39; ic_wp\n<\/span><\/span><span class=\"line\"><span class=\"cl\">ERROR 2013 (HY000) at line 1: Lost connection to server during query\n<\/span><\/span><span class=\"line\"><span class=\"cl\">root@mysql007:~#\n<\/span><\/span><\/code><\/pre><\/div><p>It happens if I specify the field name explicitly:<\/p>\n<div class=\"highlight\"><pre tabindex=\"0\" class=\"chroma\"><code class=\"language-gdscript3\" data-lang=\"gdscript3\"><span class=\"line\"><span class=\"cl\"><span class=\"n\">root<\/span><span class=\"err\">@<\/span><span class=\"n\">mysql007<\/span><span class=\"p\">:<\/span><span class=\"o\">~<\/span><span class=\"c1\"># mysql --protocol=socket -e &#39;select option_id,option_name,option_value,autoload from 1C4Uonkwhe_options limit 1 offset 1402&#39; ic_wp<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"n\">ERROR<\/span> <span class=\"mi\">2013<\/span> <span class=\"p\">(<\/span><span class=\"n\">HY000<\/span><span class=\"p\">)<\/span> <span class=\"n\">at<\/span> <span class=\"n\">line<\/span> <span class=\"mi\">1<\/span><span class=\"p\">:<\/span> <span class=\"n\">Lost<\/span> <span class=\"n\">connection<\/span> <span class=\"n\">to<\/span> <span class=\"n\">server<\/span> <span class=\"n\">during<\/span> <span class=\"n\">query<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"n\">root<\/span><span class=\"err\">@<\/span><span class=\"n\">mysql007<\/span><span class=\"p\">:<\/span><span class=\"o\">~<\/span><span class=\"c1\">#<\/span>\n<\/span><\/span><\/code><\/pre><\/div><p>It <em>doesn&rsquo;t<\/em> happen if I specify the key field:<\/p>\n<div class=\"highlight\"><pre tabindex=\"0\" class=\"chroma\"><code class=\"language-fallback\" data-lang=\"fallback\"><span class=\"line\"><span class=\"cl\">root@mysql007:~# mysql --protocol=socket -e &#39;select option_id from 1C4Uonkwhe_options limit 1 offset 1402&#39; ic_wp\n<\/span><\/span><span class=\"line\"><span class=\"cl\">+-----------+\n<\/span><\/span><span class=\"line\"><span class=\"cl\">| option_id |\n<\/span><\/span><span class=\"line\"><span class=\"cl\">+-----------+\n<\/span><\/span><span class=\"line\"><span class=\"cl\">|  16296351 |\n<\/span><\/span><span class=\"line\"><span class=\"cl\">+-----------+\n<\/span><\/span><span class=\"line\"><span class=\"cl\">root@mysql007:~#\n<\/span><\/span><\/code><\/pre><\/div><p>It <em>does<\/em> happen if I specify the value field:<\/p>\n<div class=\"highlight\"><pre tabindex=\"0\" class=\"chroma\"><code class=\"language-fallback\" data-lang=\"fallback\"><span class=\"line\"><span class=\"cl\">root@mysql007:~# mysql --protocol=socket -e &#39;select option_value from 1C4Uonkwhe_options limit 1 offset 1402&#39; ic_wp\n<\/span><\/span><span class=\"line\"><span class=\"cl\">ERROR 2013 (HY000) at line 1: Lost connection to server during query\n<\/span><\/span><span class=\"line\"><span class=\"cl\">root@mysql007:~#\n<\/span><\/span><\/code><\/pre><\/div><p>It doesn&rsquo;t happen if I query the specific row by key field:<\/p>\n<div class=\"highlight\"><pre tabindex=\"0\" class=\"chroma\"><code class=\"language-gdscript3\" data-lang=\"gdscript3\"><span class=\"line\"><span class=\"cl\"><span class=\"n\">root<\/span><span class=\"err\">@<\/span><span class=\"n\">mysql007<\/span><span class=\"p\">:<\/span><span class=\"o\">~<\/span><span class=\"c1\"># mysql --protocol=socket -e &#39;select * from 1C4Uonkwhe_options where option_id = 16296351&#39; ic_wp<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"o\">+-----------+----------------------+--------------+----------+<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"o\">|<\/span> <span class=\"n\">option_id<\/span> <span class=\"o\">|<\/span> <span class=\"n\">option_name<\/span>          <span class=\"o\">|<\/span> <span class=\"n\">option_value<\/span> <span class=\"o\">|<\/span> <span class=\"n\">autoload<\/span> <span class=\"o\">|<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"o\">+-----------+----------------------+--------------+----------+<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"o\">|<\/span>  <span class=\"mi\">16296351<\/span> <span class=\"o\">|<\/span> <span class=\"n\">z_taxonomy_image8905<\/span> <span class=\"o\">|<\/span>              <span class=\"o\">|<\/span> <span class=\"n\">yes<\/span>      <span class=\"o\">|<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"o\">+-----------+----------------------+--------------+----------+<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"n\">root<\/span><span class=\"err\">@<\/span><span class=\"n\">mysql007<\/span><span class=\"p\">:<\/span><span class=\"o\">~<\/span><span class=\"c1\">#<\/span>\n<\/span><\/span><\/code><\/pre><\/div><p>Hm. Surely there is some funky non-printing character in that <code>option_value<\/code> right?<\/p>\n<div class=\"highlight\"><pre tabindex=\"0\" class=\"chroma\"><code class=\"language-fallback\" data-lang=\"fallback\"><span class=\"line\"><span class=\"cl\">root@mysql007:~# mysql --protocol=socket -e &#39;select CHAR_LENGTH(option_value) from 1C4Uonkwhe_options where option_id = 16296351&#39; ic_wp\n<\/span><\/span><span class=\"line\"><span class=\"cl\">+---------------------------+\n<\/span><\/span><span class=\"line\"><span class=\"cl\">| CHAR_LENGTH(option_value) |\n<\/span><\/span><span class=\"line\"><span class=\"cl\">+---------------------------+\n<\/span><\/span><span class=\"line\"><span class=\"cl\">|                         0 |\n<\/span><\/span><span class=\"line\"><span class=\"cl\">+---------------------------+\n<\/span><\/span><span class=\"line\"><span class=\"cl\">root@mysql007:~# mysql --protocol=socket -e &#39;select HEX(option_value) from 1C4Uonkwhe_options where option_id = 16296351&#39; ic_wp\n<\/span><\/span><span class=\"line\"><span class=\"cl\">+-------------------+\n<\/span><\/span><span class=\"line\"><span class=\"cl\">| HEX(option_value) |\n<\/span><\/span><span class=\"line\"><span class=\"cl\">+-------------------+\n<\/span><\/span><span class=\"line\"><span class=\"cl\">|                   |\n<\/span><\/span><span class=\"line\"><span class=\"cl\">+-------------------+\n<\/span><\/span><span class=\"line\"><span class=\"cl\">root@mysql007:~#\n<\/span><\/span><\/code><\/pre><\/div><p>Resetting the value to an empty value doesn&rsquo;t make a difference:<\/p>\n<div class=\"highlight\"><pre tabindex=\"0\" class=\"chroma\"><code class=\"language-fallback\" data-lang=\"fallback\"><span class=\"line\"><span class=\"cl\">root@mysql007:~# mysql --protocol=socket -e &#39;update 1C4Uonkwhe_options set option_value = &#34;&#34; where option_id = 16296351&#39; ic_wp\n<\/span><\/span><span class=\"line\"><span class=\"cl\">root@mysql007:~# mysql --protocol=socket -e &#39;select * from 1C4Uonkwhe_options&#39; ic_wp &gt; \/dev\/null\n<\/span><\/span><span class=\"line\"><span class=\"cl\">ERROR 2013 (HY000) at line 1: Lost connection to server during query\n<\/span><\/span><span class=\"line\"><span class=\"cl\">root@mysql007:~#\n<\/span><\/span><\/code><\/pre><\/div><p>Deleting the row in question causes the error to specify a new offset:<\/p>\n<div class=\"highlight\"><pre tabindex=\"0\" class=\"chroma\"><code class=\"language-fallback\" data-lang=\"fallback\"><span class=\"line\"><span class=\"cl\">root@mysql007:~# mysql --protocol=socket -e &#39;delete from 1C4Uonkwhe_options where option_id = 16296351&#39; ic_wp\n<\/span><\/span><span class=\"line\"><span class=\"cl\">root@mysql007:~# mysql --protocol=socket -e &#39;select * from 1C4Uonkwhe_options&#39; ic_wp &gt; \/dev\/null\n<\/span><\/span><span class=\"line\"><span class=\"cl\">ERROR 2013 (HY000) at line 1: Lost connection to server during query\n<\/span><\/span><span class=\"line\"><span class=\"cl\">root@mysql007:~# mysqldump ic_wp &gt; \/dev\/null\n<\/span><\/span><span class=\"line\"><span class=\"cl\">mysqldump: Error 2013: Lost connection to server during query when dumping table `1C4Uonkwhe_options` at row: 1401\n<\/span><\/span><span class=\"line\"><span class=\"cl\">root@mysql007:~#\n<\/span><\/span><\/code><\/pre><\/div><p>If I put the record I deleted back in, we return to the old offset:<\/p>\n<div class=\"highlight\"><pre tabindex=\"0\" class=\"chroma\"><code class=\"language-fallback\" data-lang=\"fallback\"><span class=\"line\"><span class=\"cl\">root@mysql007:~# mysql --protocol=socket -e &#39;insert into 1C4Uonkwhe_options VALUES(16296351,&#34;z_taxonomy_image8905&#34;,&#34;&#34;,&#34;yes&#34;);&#39; ic_wp \n<\/span><\/span><span class=\"line\"><span class=\"cl\">root@mysql007:~# mysqldump ic_wp &gt; \/dev\/null\n<\/span><\/span><span class=\"line\"><span class=\"cl\">mysqldump: Error 2013: Lost connection to server during query when dumping table `1C4Uonkwhe_options` at row: 1402\n<\/span><\/span><span class=\"line\"><span class=\"cl\">root@mysql007:~#\n<\/span><\/span><\/code><\/pre><\/div><p>I&rsquo;m losing my little mind. Let&rsquo;s get drastic and create a whole new table, copy over the data delicately working around\nthe deadly offset:<\/p>\n<div class=\"highlight\"><pre tabindex=\"0\" class=\"chroma\"><code class=\"language-fallback\" data-lang=\"fallback\"><span class=\"line\"><span class=\"cl\">oot@mysql007:~# mysql --protocol=socket -e &#39;create table 1C4Uonkwhe_new_options like 1C4Uonkwhe_options;&#39; ic_wp \n<\/span><\/span><span class=\"line\"><span class=\"cl\">root@mysql007:~# mysql --protocol=socket -e &#39;insert into 1C4Uonkwhe_new_options select * from 1C4Uonkwhe_options limit 1402 offset 0;&#39; ic_wp \n<\/span><\/span><span class=\"line\"><span class=\"cl\">--- There is only 33 more records, not sure how to specify unlimited limit but 100 does the trick.\n<\/span><\/span><span class=\"line\"><span class=\"cl\">root@mysql007:~# mysql --protocol=socket -e &#39;insert into 1C4Uonkwhe_new_options select * from 1C4Uonkwhe_options limit 100 offset 1403;&#39; ic_wp \n<\/span><\/span><\/code><\/pre><\/div><p>Now let&rsquo;s make sure all is working properly:<\/p>\n<div class=\"highlight\"><pre tabindex=\"0\" class=\"chroma\"><code class=\"language-fallback\" data-lang=\"fallback\"><span class=\"line\"><span class=\"cl\">root@mysql007:~# mysql --protocol=socket -e &#39;select * from 1C4Uonkwhe_new_options&#39; ic_wp &gt;\/dev\/null;\n<\/span><\/span><\/code><\/pre><\/div><p>Now let&rsquo;s examine which row we are missing:<\/p>\n<div class=\"highlight\"><pre tabindex=\"0\" class=\"chroma\"><code class=\"language-fallback\" data-lang=\"fallback\"><span class=\"line\"><span class=\"cl\">root@mysql007:~# mysql --protocol=socket -e &#39;select option_id from 1C4Uonkwhe_options where option_id not in (select option_id from 1C4Uonkwhe_new_options) ;&#39; ic_wp \n<\/span><\/span><span class=\"line\"><span class=\"cl\">+-----------+\n<\/span><\/span><span class=\"line\"><span class=\"cl\">| option_id |\n<\/span><\/span><span class=\"line\"><span class=\"cl\">+-----------+\n<\/span><\/span><span class=\"line\"><span class=\"cl\">|  18405297 |\n<\/span><\/span><span class=\"line\"><span class=\"cl\">+-----------+\n<\/span><\/span><span class=\"line\"><span class=\"cl\">root@mysql007:~#\n<\/span><\/span><\/code><\/pre><\/div><p><strong>Wait, what? I was expecting option_id 16296351.<\/strong><\/p>\n<p>Oh, now we are getting somewhere. And I see my mistake: when using offsets, you need to use <code>ORDER BY<\/code> or you won&rsquo;t get consistent results.<\/p>\n<div class=\"highlight\"><pre tabindex=\"0\" class=\"chroma\"><code class=\"language-fallback\" data-lang=\"fallback\"><span class=\"line\"><span class=\"cl\">root@mysql007:~# mysql --protocol=socket -e &#39;select option_id from 1C4Uonkwhe_options order by option_id limit 1 offset 1402&#39; ic_wp ;\n<\/span><\/span><span class=\"line\"><span class=\"cl\">+-----------+\n<\/span><\/span><span class=\"line\"><span class=\"cl\">| option_id |\n<\/span><\/span><span class=\"line\"><span class=\"cl\">+-----------+\n<\/span><\/span><span class=\"line\"><span class=\"cl\">|  18405297 |\n<\/span><\/span><span class=\"line\"><span class=\"cl\">+-----------+\n<\/span><\/span><span class=\"line\"><span class=\"cl\">root@mysql007:~#\n<\/span><\/span><\/code><\/pre><\/div><p>Now that I have the correct row&hellip; what is in it:<\/p>\n<div class=\"highlight\"><pre tabindex=\"0\" class=\"chroma\"><code class=\"language-fallback\" data-lang=\"fallback\"><span class=\"line\"><span class=\"cl\">root@mysql007:~# mysql --protocol=socket -e &#39;select * from 1C4Uonkwhe_options where option_id = 18405297&#39; ic_wp ;\n<\/span><\/span><span class=\"line\"><span class=\"cl\">ERROR 2013 (HY000) at line 1: Lost connection to server during query\n<\/span><\/span><span class=\"line\"><span class=\"cl\">root@mysql007:~#\n<\/span><\/span><\/code><\/pre><\/div><p>Well, that makes a lot more sense. Let&rsquo;s start over with examining the value:<\/p>\n<div class=\"highlight\"><pre tabindex=\"0\" class=\"chroma\"><code class=\"language-fallback\" data-lang=\"fallback\"><span class=\"line\"><span class=\"cl\">root@mysql007:~# mysql --protocol=socket -e &#39;select CHAR_LENGTH(option_value) from 1C4Uonkwhe_options where option_id = 18405297&#39; ic_wp ;\n<\/span><\/span><span class=\"line\"><span class=\"cl\">+---------------------------+\n<\/span><\/span><span class=\"line\"><span class=\"cl\">| CHAR_LENGTH(option_value) |\n<\/span><\/span><span class=\"line\"><span class=\"cl\">+---------------------------+\n<\/span><\/span><span class=\"line\"><span class=\"cl\">|                  50814767 |\n<\/span><\/span><span class=\"line\"><span class=\"cl\">+---------------------------+\n<\/span><\/span><span class=\"line\"><span class=\"cl\">root@mysql007:~#\n<\/span><\/span><\/code><\/pre><\/div><p>Wow, that&rsquo;s a lot of characters. If it were a book, it would be 35,000 pages\nlong (I just discovered <a href=\"https:\/\/thecharcounter.com\/characters-to-words\/\">this\nsite<\/a>). It&rsquo;s a <code>LONGTEXT<\/code>\nfield so it should be able to handle it. But now I have a better idea of what\ncould be going wrong. The name of the option is &ldquo;rewrite_rules&rdquo; so it seems\nlike something is going wrong with the generation of that option.<\/p>\n<p>I imagine there is some tweak I can make to allow MariaDB to cough up the value\n(<code>read_buffer_size<\/code>? <code>tmp_table_size<\/code>?). But I&rsquo;ll start with checking in with\nthe database owner because I don&rsquo;t think 35,000 pages of rewrite rules is\nappropriate for any site.<\/p>\n"},{"title":"Who ate my RAM?","link":"https:\/\/current.workingdirectory.net\/posts\/2024\/who-ate-my-ram\/","pubDate":"Wed, 07 Aug 2024 08:27:10 -0400","guid":"https:\/\/current.workingdirectory.net\/posts\/2024\/who-ate-my-ram\/","description":"<p>One of our newest servers, with a hefty 256GB of RAM, recently began killing\nprocesses via the oomkiller.<\/p>\n<p>According to <code>free<\/code>, only half of the RAM was in use (125GB). About 4GB was\nfree, with the remainer used by the file cache.<\/p>\n<p>I&rsquo;m used to seeing unexpected &ldquo;free RAM&rdquo; numbers like this and have been\nassured that the kernel is simply not wasting RAM. If it&rsquo;s not needed, use it\nto cache files to save on disk I\/O. That make sense.<\/p>\n<p>However&hellip; why is the oomkiller being called instead of flushing the file\ncache?<\/p>\n<p>I came up with all kinds of amazing and wrong theories: maybe the RAM is\nfragmented (is that even a thing?!?), maybe there is a spike in RAM and the\nkernel can&rsquo;t flush the cache quickly enough (I really don&rsquo;t think that&rsquo;s a\nthing). Maybe our kvm-manager has a weird bug (nope, but that didn&rsquo;t stop me\nfrom opening a <a href=\"https:\/\/0xacab.org\/dkg\/kvm-manager\/-\/issues\/12\">spurious bug\nreport<\/a>).<\/p>\n<p>I learned lots of cool things, like the oomkiller report includes a\ntable of the memory in use by each process (via the <code>rss<\/code> column) - and you\nhave to muliply that number by 4096 because it&rsquo;s in 4K pages.<\/p>\n<p>That&rsquo;s how I discovered that the oomkiller was killing off processes with only\nhalf the memory in use.<\/p>\n<p>I also learned that <code>lsof<\/code> sometimes lists the same open file multiple times,\nwhich made me think a bunch of files were being opened repeatedly causing a\nmemory problem, but really it amounted to nothing.<\/p>\n<p>That last thing I learned, courtesy of <a href=\"https:\/\/askubuntu.com\/questions\/762717\/high-shmem-memory-usage\">an askubuntu\npost<\/a> is that\nthe <code>\/dev<\/code> filesystem is allocated by default exactly half the RAM on the\nsystem. What a coincidence! That is exactly how much RAM is useable on the\nserver.<\/p>\n<p>And, on the server in question, that filesystem is full. What?!? Normally, that\nfilesystem should be using 0 bytes because it&rsquo;s not a real filesystem. But in\nour case a process created a 127GB file there - it was only stopped because the\nfile system filled up.<\/p>\n"},{"title":"Users without passwords","link":"https:\/\/current.workingdirectory.net\/posts\/2023\/users-without-passwords\/","pubDate":"Sun, 22 Oct 2023 08:27:10 -0400","guid":"https:\/\/current.workingdirectory.net\/posts\/2023\/users-without-passwords\/","description":"<p>About fifteen years ago, while debugging a database probem, I was horrified to\ndiscover that we had two root users - one with the password I had been using\nand one without a password. Nooo!<\/p>\n<p>So, I wrote a simple maintenance script that searched for and deleted any user\nin our database without a password. I even made it part of our puppet recipe -\nsince the database server was in use by users and I didn&rsquo;t want anyone using\nSQL statements to change their password to an empty value.<\/p>\n<p>Then I forgot about it.<\/p>\n<p>Recently, I upgraded our MariaDB databases to Debian bullseye, which inserted\nthe <code>mariadb.sys<\/code> user which&hellip;. doesn&rsquo;t have a password set. It seems to be\nlocked down in other ways, but my dumb script didn&rsquo;t know about that and\nhappily deleted the user.<\/p>\n<p>Who needs that <code>mariadb.sys<\/code> user anyway?<\/p>\n<p>Apparently we all do. On one server, I can&rsquo;t login as root anymore. On another\nserver I can login as root, but if I try to list users I get an error:<\/p>\n<blockquote>\n<p>ERROR 1449 (HY000): The user specified as a definer (&lsquo;mariadb.sys&rsquo;@&rsquo;localhost&rsquo;) does not exist<\/p><\/blockquote>\n<p>The Internt is full of useless advice. The most common is to simply insert that user. Except&hellip;<\/p>\n<pre><code>MariaDB [mysql]&gt; CREATE USER `mariadb.sys`@`localhost` ACCOUNT LOCK PASSWORD EXPIRE;\nERROR 1396 (HY000): Operation CREATE USER failed for 'mariadb.sys'@'localhost'\nMariaDB [mysql]&gt; \n<\/code><\/pre>\n<p>Yeah, that&rsquo;s not going to work.<\/p>\n<p>It seems like we are dealing with two changes. One, the old <code>mysql.user<\/code> table\n<a href=\"https:\/\/mariadb.com\/kb\/en\/mysqlglobal_priv-table\/\">was replaced by the <code>global_priv<\/code>\ntable<\/a> and then turned into\na view for backwards compatibility.<\/p>\n<p>And two, for <a href=\"https:\/\/jira.mariadb.org\/browse\/MDEV-19650\">sensible reasons<\/a> the\ndefault definer for this view has been changed from the root user to a user that,\nahem, is unlikely to be changed or deleted.<\/p>\n<p>Apparently I can&rsquo;t add the <code>mariadb.sys<\/code> user because it would alter the <code>user<\/code>\nview which has a definer that doesn&rsquo;t exist. Although not sure if this really is\nthe reason?<\/p>\n<p>Fortunately, I found an <a href=\"https:\/\/stackoverflow.com\/a\/19707173\">excellent\nsuggestion<\/a> for changing the definer of a\nview. My modified version of the answer is, run the following command which\nwill generate a SQL statement:<\/p>\n<pre><code>SELECT CONCAT(&quot;ALTER DEFINER=root@localhost VIEW &quot;, table_name, &quot; AS &quot;, view_definition, &quot;;&quot;) FROM information_schema.views WHERE table_schema='mysql' AND definer = 'mariadb.sys@localhost';\n<\/code><\/pre>\n<p>Then, execute the statement.<\/p>\n<p>And then also update the <code>mysql.proc<\/code> table:<\/p>\n<pre><code>UPDATE mysql.proc SET definer = 'root@localhost' WHERE definer = 'mariadb.sys@localhost';\n<\/code><\/pre>\n<p>And lastly, I had to run:<\/p>\n<pre><code>DELETE FROM tables_priv WHERE User = 'mariadb.sys';\nFLUSH privileges;\n<\/code><\/pre>\n<p>Wait, was the <code>tables_priv<\/code> entry the whole problem all along? Not sure. But now I can run:<\/p>\n<pre><code>CREATE USER `mariadb.sys`@`localhost` ACCOUNT LOCK PASSWORD EXPIRE;\nGRANT SELECT, DELETE ON `mysql`.`global_priv` TO `mariadb.sys`@`localhost`;\n<\/code><\/pre>\n<p>And reverse the other statements:<\/p>\n<pre><code>SELECT CONCAT(&quot;ALTER DEFINER=`mariadb.sys`@localhost VIEW &quot;, table_name, &quot; AS &quot;, view_definition, &quot;;&quot;) FROM information_schema.views WHERE table_schema='mysql' AND definer = 'root@localhost';\n<\/code><\/pre>\n<p>[Execute the output.]<\/p>\n<pre><code>UPDATE mysql.proc SET definer = 'mariadb.sys@localhost' WHERE definer = 'root@localhost';\n<\/code><\/pre>\n<p>And while we&rsquo;re on the topic of borked MariaDB authentication, here are the\nsteps to change the root password and restore all root privielges if you can&rsquo;t\nget in at all or your root user is missing the GRANT OPTION (you can change\n&ldquo;ALTER&rdquo; to &ldquo;CREATE&rdquo; if the root user does not even exist):<\/p>\n<pre><code>systemctl stop mariadb\nmariadbd-safe --skip-grant-tables --skip-networking &amp;\nmysql -u root\n[mysql]&gt; FLUSH PRIVILEGES\n[mysql]&gt; ALTER USER `root`@`localhost` IDENTIFIED VIA mysql_native_password USING PASSWORD('your-secret-password') OR unix_socket; \n[mysql]&gt; GRANT ALL PRIVILEGES ON *.* to 'root'@'localhost' WITH GRANT OPTION;\nmariadbd-admin shutdown\nsystemctl start mariadb\n<\/code><\/pre>\n"},{"title":"What am I missing about AI?","link":"https:\/\/current.workingdirectory.net\/posts\/2023\/what-am-i-missing\/","pubDate":"Tue, 18 Jul 2023 08:27:10 -0400","guid":"https:\/\/current.workingdirectory.net\/posts\/2023\/what-am-i-missing\/","description":"<p><a href=\"https:\/\/current.workingdirectory.net\/posts\/2023\/enough-about-ai\">Last month I blogged<\/a> about how the mainstream\nmedia is focusing on the wrong parts of the Artificial Intelligence\/ChatGPT\nstory.<\/p>\n<p>One of the comments left on the post was:<\/p>\n<pre><code>I encourage you to dig a little deeper. If LLM\u2019s were just probability\nmachines, no one would be raising any flags.\n\nHinton, Bengio, Tegmark and many others are not simpletons. It is the fact that\nthe architecture and specific training (deep NN, back prop \/ gradient descend)\nproduces a system with emergent properties, beyond just a probability machine,\nwhen the system size reaches some thresholds, that has them spooked.\n\nThey do understand mathematics and stats and probabilities, i assure you. It is\njust that you may have only read the layman\u2019s articles and not the scientific\nones\n<\/code><\/pre>\n<p>I confess: I haven&rsquo;t made much progress in this regard. I gave <a href=\"https:\/\/raw.githubusercontent.com\/veekaybee\/what_are_embeddings\/main\/embeddings.pdf?utm_source=pocket_saves\">Vicky Boykis'\nEmbeddings<\/a>\na go, and started to get a handle on the math, but honestly had a hard time\nfollowing it. I&rsquo;m open to suggestions from anyone with a few good\nrecommendations for scientific papers accessible to non-math professionals,\nparticularly ones that explain the &ldquo;emergent&rdquo; properties and what that means.<\/p>\n<p>Meanwhile, regardless of the scientific truths or falsehoods around chat GPT,\nthe mainstream media continues to miserably fail in helping the rest of us\nunderstand the implications of this technology.<\/p>\n<p>Most recently, I listend to <a href=\"https:\/\/www.thisamericanlife.org\/803\/greetings-people-of-earth\">This American Life&rsquo;s &ldquo;First Contact&rdquo; (part of\ntheir &ldquo;Greetings People of Earth&rdquo;\nshow<\/a>).<\/p>\n<p>They interviewed several Microsft AI researchers who first experimented with\nChatGPT 4 prior to it&rsquo;s big release.<\/p>\n<p>The focus of the researchers was: can we demonstrate chat GPT&rsquo;s general\nintelligence ability by presenting it with logic problems it could not possibly\nhave encountered before? And the answer: YES!<\/p>\n<p>The two examples were:<\/p>\n<ol>\n<li>\n<p>Stacking: the researcher asked chat GPT how to stack a number of odd objects\nin a stable way (a book, a dozen eggs, a nail, etc) and chat GPT gave both\nthe correct answer and a reasonable explanation of why.<\/p>\n<\/li>\n<li>\n<p>Hidden state: the researcher described two people in a room with a cat. One\nperson put the cat in a basket and left. The other moved the cat to a box\nad left. And, remarkably, chat GPT could explain that when they returned,\nthe first person would think the cat is in the basket and the second person\nwould know it&rsquo;s in the box.<\/p>\n<\/li>\n<\/ol>\n<p>I thought this was pretty cool. So I fired up chat GPT (and even ponied up for\nchat GPT version 4). I asked it my own stacking question and, hm, chat GPT\nthought a plate should be placed on top of a can of soda instead of beneath it.\nSo, well, mostly right but I&rsquo;m pretty sure any reasonable human would put the\ncan of soda on the plate not the other way around (chat GPT 3.5 wanted the can\nof soda to be balanced on the tip of the nail).<\/p>\n<p>I then asked it my own simple version of the cat problem and it got it right.\nVery good. But when I asked it a much more complicated and weird version of the\ncat problem (involving beetles in a mansion with a movie theater and changing\nmovies and a butler with a big mustache) it got the answer flat out wrong.<\/p>\n<p>Did anyone at This American Life try this? Really? It seems like a basic\nresponsibility of journalism to fact check the experts. Maybe the scientists\nwould have had a convincing response? Or maybe scientists are just like\neveryone else and can get caught up in the excitement and make mistakes?<\/p>\n<p>I am amazed and awed by what chat GPT can do - it truly is remarkable. And, I\nthink that a lot of <em>human<\/em> intelligence is synthesizing what we&rsquo;ve seen and\nsimply regurgitating it in a different context - a task that chat GPT is way\nbetter at doing than we are.<\/p>\n<p>But the overriding message of most mainstream media stories is that chat GPT is\nsomehow going beyond word synthesis and probability and magically tapping into\na form of logic. If the scientific papers <em>are<\/em> demonstrating this remarkable\nfeat, I think the media needs to do a way better job reporting it.<\/p>\n"},{"title":"Enough about the AI Apocalypse Already","link":"https:\/\/current.workingdirectory.net\/posts\/2023\/enough-about-ai\/","pubDate":"Thu, 01 Jun 2023 08:27:10 -0400","guid":"https:\/\/current.workingdirectory.net\/posts\/2023\/enough-about-ai\/","description":"<p>After watching <a href=\"https:\/\/democracynow.org\">Democracy Now&rsquo;s<\/a> <a href=\"https:\/\/www.democracynow.org\/2023\/6\/1\/ai_bengio_petty_tegmark\">segment on\nartificial\nintelligence<\/a> I\nstarted to wonder - am I out of step on this topic?<\/p>\n<p>When people claim artificial intelligence will surpass human intelligence and\nthus threaten humanity with extinction, they seem to be referring specifically\nto advances made with large language models.<\/p>\n<p>As I understand them, large language models are probability machines that have\ningested massive amounts of text scraped from the Internet. They answer\nquestions based on the probability of one series of words (their answer)\nfollowing another series of words (the question).<\/p>\n<p>It seems like a stretch to call this intelligence, but if we accept that\ndefinition then it follows that this kind of intelligence is nothing remotely\nlike human intelligence, which makes the claim that it will surpass human\nintelligence confusing. Hasn&rsquo;t this kind of machine learning surpassed us\ndecades ago?<\/p>\n<p>Or when we say &ldquo;surpass&rdquo; does that simply refer to fooling people into thinking\nan AI machine is a human via conversation? That is an important milestone, but\nI&rsquo;m not ready to accept the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Turing_test\">turing\ntest<\/a> as proof of equal\nintelligence.<\/p>\n<p>Furthermore, large language models &ldquo;hallucinate&rdquo; and also reflect the biases of\ntheir training data. The word &ldquo;hallucinate&rdquo; seems like a euphemism, as if it\ncould be corrected with the right medication when in fact it seems hard to\navoid when your strategy is to correlate words based on probability. But even\nif you could solve the &ldquo;here is a completely wrong answer presented with\nsociopathic confidence&rdquo; problem, reflecting the biases of your data sources\nseems fairly intractable. <em>In what world would a system with built-in bias be\nconsidered on the brink of surpassing human intelligence?<\/em><\/p>\n<p>The danger from LLMs seems to be their ability to convince people that their\nanswers are correct, including their patently wrong and\/or biased answers.<\/p>\n<p>Why do people think they are giving correct answers? Oh right&hellip;\nterrifying right wing billionaires (with <a href=\"https:\/\/www.longtermism-hub.com\/\">terrifying\nagendas<\/a> have been <a href=\"https:\/\/futureoflife.org\/open-letter\/pause-giant-ai-experiments\/\">claiming AI will exceed\nhuman intelligence and threaten\nhumanity<\/a> and\nevery time they <a href=\"https:\/\/www.safe.ai\/statement-on-ai-risk#open-letter\">sign a hyperbolic\nstatement<\/a> they get front\npage mainstream coverage. And even <a href=\"https:\/\/www.democracynow.org\/2023\/6\/1\/ai_bengio_petty_tegmark\">progressive news outlets are spreading this\nnarrative<\/a> with\nminimal space for contrary opinions (thank you Tawana Petty from the\n<a href=\"https:\/\/www.ajl.org\/\">Algorithmic Justice League<\/a> for providing the only\nglimpse of reason in the segment).<\/p>\n<p>The belief that artificial intelligence is or will soon become omnipotent has\nreal world harms today: specifically it creates the misperception that current\nLLMs are accurate, which paves the way for greater adoption among police\nforces, social service agencies, medical facilities and other places where\nracial and economic biases have life and death consequences.<\/p>\n<p>When the CEO of OpenAI calls the technology dangerous and in need of\nregulation, he gets both free advertising promoting the power and supposed\naccuracy of his product and the possibility of freezing further developments in\nthe field that might challenge OpenAI&rsquo;s current dominance.<\/p>\n<p>The real threat to humanity is not AI, it&rsquo;s massive inequality and the use of\ntactics ranging from mundane bureaucracy to deadly force and incarceration to\nsegregate the affluent from the growing number of people unable to make ends\nmeet. We have spent decades training bureaucrats, judges and cops to\nrobotically follow biased laws to maintain this order without compassion or\nempathy. Replacing them with AI would be make things worse and should be\nstopped. But, let&rsquo;s be clear, the narrative that AI is poised to surpass human\nintelligence and make humanity extinct is a dangerous distraction that runs\ncounter to <a href=\"https:\/\/www.dair-institute.org\/blog\/letter-statement-March2023\">a much more important story about &ldquo;the very real and very present\nexploitative practices of the [companies building AI], who are rapidly\ncentralizing power and increasing social\ninequities.&rdquo;<\/a>.<\/p>\n<p>Maybe we should talk about <em>that<\/em> instead?<\/p>\n"},{"title":"Cranky old timers should know perl","link":"https:\/\/current.workingdirectory.net\/posts\/2023\/cranky-old-timers-should-know-perl\/","pubDate":"Wed, 17 May 2023 08:27:10 -0400","guid":"https:\/\/current.workingdirectory.net\/posts\/2023\/cranky-old-timers-should-know-perl\/","description":"<p>I act like an old timer (I&rsquo;ve been around linux for 25 years and I&rsquo;m cranky\nabout new tech that is not easily maintained and upgraded) yet somehow I don&rsquo;t\nknow perl. How did that happen?<\/p>\n<p>I discovered this state when I decided to move from the heroically packaged yet\n<a href=\"https:\/\/github.com\/trusteddomainproject\/OpenDMARC\/issues\/240\">seemingly upstream\nun-maintained<\/a>\n<a href=\"https:\/\/github.com\/trusteddomainproject\/OpenDMARC\">opendmarc<\/a> package to\n<a href=\"https:\/\/github.com\/fastmail\/authentication_milter\/\">authentication_milter<\/a>.<\/p>\n<p>It&rsquo;s written in perl. And, alas, <a href=\"https:\/\/bugs.debian.org\/cgi-bin\/bugreport.cgi?bug=1036235\">not in\ndebian<\/a>.<\/p>\n<p>How hard could this be?<\/p>\n<p>The instructions for installing seemed pretty straight forward: <code>cpanm Mail::Milter::Authentication<\/code>.<\/p>\n<p>Wah. I&rsquo;m glad I tried this out on a test virtual machine. It took forever! It\nran tests! It compiled things! And, it installed a bunch of perl modules\nalready packaged in Debian.<\/p>\n<p>I don&rsquo;t think I want to add this command to my ansible playbook.<\/p>\n<p>Next I spent an inordinate amount of time trying to figure out how to list the\ndependencies of a given CPAN module. I was looking for something like <code>cpanm --list-dependencies Mail::Milter::Authentication<\/code> but eventually ended up <a href=\"https:\/\/metacpan.org\/pod\/CPAN::FindDependencies\">writing\na perl script<\/a> that output\nperl code, inserting a &ldquo;use &quot; before each dependency and a semicolon and line\nbreak after them. Then, I could execute that script on a clean debian\ninstallation and see which perl modules I needed. For each error, I checked for\nthe module in Debian (and installed it) or kept a list of modules I would have\nto build (and commented out the line).<\/p>\n<p>Once I had a list of modules to build, I used the handy <code>cpan2deb<\/code> command. It\ntook some creative ordering but eventually I got it right. Since I will surely\nforget how to do this when it&rsquo;s time to upgrade, I <a href=\"https:\/\/code.mayfirst.org\/mfmt\/seed\/-\/blob\/master\/roles\/tests\/files\/build-authentication-milter-debian-packages\">wrote a\nscript<\/a>.<\/p>\n<p>In total it took me several days to figure this all out, so I once again find\nmyself very appreciative of all the debian packagers out there - particularly\nthe perl ones!!<\/p>\n<p>And also&hellip; if I did this all wrong and there is an easier way I would love to\nhear about it in the comments.<\/p>\n"},{"title":"Electron doesn't like negative layout coordinates","link":"https:\/\/current.workingdirectory.net\/posts\/2023\/electron-doesnt-like-negative-numbers\/","pubDate":"Thu, 20 Apr 2023 08:27:10 -0400","guid":"https:\/\/current.workingdirectory.net\/posts\/2023\/electron-doesnt-like-negative-numbers\/","description":"<p>I got a second external monitor. Overkill? Probably, but I like having a\ndedicated space to instant messaging (now left monitor) and also a dedicated\nspace for a web browser (right monitor).<\/p>\n<p>But, when I moved signal-desktop to the left monitor, clicks stopped working. I\nmoved it back to my laptop screen, clicks started working. Other apps (like\ngajim) worked fine. A real mystery.<\/p>\n<p>I spent a lot of time on the wrong thing. I turned this monitor into portrait\nmode. Maybe signal doesn&rsquo;t like portrait mode? Nope.<\/p>\n<p>Maybe signal doesn&rsquo;t think it has enough horizontal space? Nope.<\/p>\n<p>Maybe signal suddently doesn&rsquo;t like being on an external monitor? Nope.<\/p>\n<p>Maybe signal will output something useful if I start it via the terminal? Nope\n(but that was a real distraction).<\/p>\n<p>Then, I discovered that mattermost desktop behaves the same way. A clue! So,\nnow I know it&rsquo;s an electron app limitation, not a signal limitation.<\/p>\n<p>Finally I hit on the problem: Via my sway desktop, I set my laptop screen to\nthe x,y coordinates: 0,0 (since it&rsquo;s in the middle). I set the left monitor to\nnegative coordinates so it would appear on the left. Well, electron does not\nlike that, not sure why.<\/p>\n<p>Now, my left monitor occupies 0,0 and the rest are adjusted to that center.<\/p>\n<p>I originally used negative coordinates so when I unplugged my monitors, my\nlaptop would still be 0,0 and display all desktops properly. Fortunately, sway\nmagically figures that all out even when the &ldquo;center&rdquo; shifts by unplugging the\nmonitors. Hooray for sway.<\/p>\n"},{"title":"Doing whatever Gmail says","link":"https:\/\/current.workingdirectory.net\/posts\/2023\/doing-whatever-gmail-says\/","pubDate":"Wed, 12 Apr 2023 08:27:10 -0400","guid":"https:\/\/current.workingdirectory.net\/posts\/2023\/doing-whatever-gmail-says\/","description":"<p>As we slowly move our members to our new email infrastructure, an unexpected\ntwist turned up: One member reported getting the Gmail warning:<\/p>\n<blockquote>\n<p>Be careful with this message The sender hasn&rsquo;t authenticated this message so Gmail can&rsquo;t verify that it actually came from them.<\/p><\/blockquote>\n<p>They have their email delivered to May First, but have configured Gmail to pull\nin that email using the &ldquo;Check mail from other accounts&rdquo; feature. It worked\nfine on our old infrastructure, but started giving this message when we\ntransitioned.<\/p>\n<p>A further twist: he only receives this message from email sent by other\npeople in his organization - in other words email sent via May First gets\nflagged, email sent from other people does not.<\/p>\n<p>These Gmail messages typically warn users about email that has failed (or\nlacks) <em>both<\/em> SPF and DKIM. However, before diving into the technical details,\nmy first thought was: why is Gmail giving a warning on a message that wasn&rsquo;t\neven delivered to them?  It&rsquo;s always nice to get <a href=\"https:\/\/serverfault.com\/a\/1088593\/477557\">confirmation from\nothers<\/a> that this is totally wrong\nbehavior. Unfortunately, when it&rsquo;s Gmail, it doesn&rsquo;t matter if they are wrong.\nWe all have to deal with it.<\/p>\n<p>So next, I decided to investigate why this message failed both DKIM (digital\nsignature) and SPF (ensuring the message was sent from an authorized server).<\/p>\n<p>Examining the headers immediately turned up the SPF failure:<\/p>\n<div class=\"highlight\"><pre tabindex=\"0\" class=\"chroma\"><code class=\"language-fallback\" data-lang=\"fallback\"><span class=\"line\"><span class=\"cl\">Authentication-Results: mx.google.com;\n<\/span><\/span><span class=\"line\"><span class=\"cl\">       spf=fail (google.com: domain of xxx@xxx.org does not designate n.n.n.n as permitted sender) smtp.mailfrom=xxx@xxx.org\n<\/span><\/span><\/code><\/pre><\/div><p>The IP address Google checked to ensure the message was sent by an authorized\nserver is the IP address of our internal mail filter server in our new email\ninfrastructure. That&rsquo;s the last hop before delivery to the user&rsquo;s mailbox, so\nthat&rsquo;s the last hop Gmail sees. This is why Gmail is totally wrong to run this\ncheck: all email messages retreived via their mail fecthing service are going\nto fail the SPF test because Gmail has no way of knowing what the actual last\nhop is.<\/p>\n<p>So why is this problem only showing up after we transitioned to our new\ninfrastructure? Because our old infrastructure had only one mail server for\nevery user. The one mail server was the MX server and the relay server, so it\nwas included in their SPF record.<\/p>\n<p>And why does this only affect mail sent via May First and not other domains?<\/p>\n<p>Because we add our DKIM signature to <em>outgoing<\/em> email, not to email delivered\ninternally. Therefore, these messages both fail the SPF check and also don&rsquo;t\nhave a DKIM signature. Other messages have a DKIM signature.<\/p>\n<p>Ugggg. So what do we do <em>now<\/em>? Clearly, something dumb and simple is in order:\nI added the IP addresses of our internal filter servers to our global SPF\nrecord.<\/p>\n<p>Someday, years from now, after Gmail is long gone (or has fixed this dumb\nbehavior), when I&rsquo;m doing whatever retired people like me do, someone will\nnotice that our internal filter server IPs are included in our SPF record.\nHopefully they will fix the problem, but instead they&rsquo;ll probably think: no\nidea why these are here - something will probably break if I remove them.<\/p>\n"},{"title":"Web caching is hard","link":"https:\/\/current.workingdirectory.net\/posts\/2022\/web-caching-is-hard\/","pubDate":"Sun, 14 Aug 2022 08:27:10 -0400","guid":"https:\/\/current.workingdirectory.net\/posts\/2022\/web-caching-is-hard\/","description":"<p>Web caching is hard. And also, maybe I&rsquo;m not that good under pressure? In any\nevent, I made the following mistakes while trying to debug a web site using our\nnginx cache that bit the dust under heavy load today:<\/p>\n<p><strong>Action:<\/strong> I ran <code>curl -I https:\/\/website.org\/<\/code> and it hung.<\/p>\n<p><em>Wrong assumption:<\/em> Something is wrong with nginx. Why else would it just hang?<\/p>\n<p><em>Reconsidered conclusion<\/em>: The resource (the home page) is a MISS, so nginx has\nto retrieve it from the origin, but the origin is over-loaded and timing out,\nso my request is also timing out. Maybe something is wrong with the nginx\ncaching configuration since the home page really should be a HIT&hellip; but that&rsquo;s\nanother problem.<\/p>\n<p><strong>Action:<\/strong> I changed the configuration from our normal caching set of\ndirectives to our aggressive caching set of directives, reloaded nginx and\n<code>curl -I https:\/\/website.org\/<\/code> still hung.<\/p>\n<p><em>Wrong assumption:<\/em> aggressive caching isn&rsquo;t working and I need a different\nconfiguration.<\/p>\n<p><em>Reconsidered conclusion:<\/em> The home page still has failed to be loaded from the\norigin, so every request for it is going to be a MISS, and is going to hang,\nuntil nginx is able to fill the cache with it. The configuration change might\nbe the right change; we just need the origin to calm down before we will know.<\/p>\n<p><strong>Action:<\/strong> I restarted PHP on the origin to free up PHP processes so my home\npage request can fill the cache &hellip; and <em>still<\/em> <code>curl -I https:\/\/website.org\/<\/code>\nhangs.<\/p>\n<p><em>Wrong assumption:<\/em> WTF! The world is ending!<\/p>\n<p><em>Reconsidered conclusion:<\/em> The regular traffic which is accessing other pages\n(not the home page) consumed all the available PHP processes on the origin\nbefore my request for the home page could complete, so nginx is <em>still<\/em> unable\nto fill the cache with the home page.<\/p>\n<p><strong>Action:<\/strong> Once we got things under control, I changed the caching level from\naggressive back down to normal. I ran <code>curl -I https:\/\/website.org\/<\/code> and it was\nHIT&rsquo;ing. I concluded that we don&rsquo;t need the aggressive cache after all. Got\nsome coffee, came back later and ran it again and it consistently showed MISS.<\/p>\n<p><em>Wrong assumption:<\/em> What?!? Did something change on the origin to stop the\ncache from working??<\/p>\n<p><em>Reconsidered conclusion:<\/em> The aggressive cache set the cache for 5 minutes.\nEven after changing to normal caching, the home page was <em>still cached<\/em> so it\nwas served from the cache. After 5 minutes, the cache expired. Now, the normal\ncache setting are in play to determine whether the request would be cached or\nnot. In other words, you have to wait for the cache to expire (or bust the\ncache) before you can effectively know if the new cache settings are working.<\/p>\n"},{"title":"Fine tuning Thunderbird's end-to-end encryption","link":"https:\/\/current.workingdirectory.net\/posts\/2022\/thunderbird-and-openpgp\/","pubDate":"Thu, 04 Aug 2022 18:27:10 -0400","guid":"https:\/\/current.workingdirectory.net\/posts\/2022\/thunderbird-and-openpgp\/","description":"<p>I love that Thunderbird really tackled OpenPGP head on and incorporated it\ndirectly into the client. I know it&rsquo;s been a bit rough for some users, but I\nthink it&rsquo;s a good long term investment.<\/p>\n<p>And to demonstrate I&rsquo;ll now complain about a minor issue :).<\/p>\n<p>I replied to an encrypted message but couldn&rsquo;t send the response using\nencryption. I got an error message indicating that &ldquo;End-to-end encryption\nrequires resolving certificate issues for&rdquo; and it listed the recipient\nemail address.<\/p>\n<p><img src=\"https:\/\/current.workingdirectory.net\/posts\/2022\/thunderbird-and-openpgp\/resolving-certificate-issues.png\" alt=\"Screen shot of error message saying: End-to-end encryption requires resolving certificate issues for\"  \/>\n<\/p>\n<p>I spent an enormous amount of time examining the recipient&rsquo;s OpenPGP key. I\nmade sure it was not expired. I made sure it was actually in my Thunderbird key\nstore not just in my OpenPGP keychain. I made sure I had indicated that I trust it\nenough to use. I re-downloaded it.<\/p>\n<p>I eventually gave up and didn&rsquo;t send the email. Then I responded to another\nencrypted email and it worked. What!?!?<\/p>\n<p>I spent more time comparing the recipients before I realized the problem was\nthe sending address, not the recipient address.<\/p>\n<p>I have an OpenPGP key that lists several identities. I have a Thunderbird\nAccount that uses the Identities feature to add several from addresses. And, it\nturns out that in Thunderbird, you need to indicate which OpenPGP key to use\nfor your main account&hellip; but also for each identity. When you drill down to\nManage Identities for your account, you are able to indicate which OpenPGP key\nyou want to use for each identity. Once I indicated that each identity should\nuse my OpenPGP key, the issue was resolved.<\/p>\n<p>And here&rsquo;s my <a href=\"https:\/\/bugzilla.mozilla.org\/show_bug.cgi?id=1783424\">Thunderbird bug asking for an error message pointing to the\nsender address, not the recipient\naddress<\/a>.<\/p>\n"},{"title":"Deleting an app won't bring back Roe v Wade","link":"https:\/\/current.workingdirectory.net\/posts\/2022\/dont-panic-organize\/","pubDate":"Sat, 25 Jun 2022 18:27:10 -0400","guid":"https:\/\/current.workingdirectory.net\/posts\/2022\/dont-panic-organize\/","description":"<p>In some ways it feels like 2016 all over again.<\/p>\n<p>I&rsquo;m seeing panic-stricken calls for everyone to delete their period apps, close\ntheir Facebook accounts, de-Google their cell phones and, generally speaking,\nturn their entire online lives upside down to avoid the techno-surveillance\ndragnet unleashed by the overturning of Roe v. Wade.<\/p>\n<p>I&rsquo;m sympathetic and generally agree that many of us should do most of those\nthings on any given day. But, there is a serious problem with this cycle of\nrepression and panic: it&rsquo;s very bad for organizing.<\/p>\n<p><strong>In our rush to give people concrete steps they can take to feel safer, we&rsquo;re\nfueling a frenzy of panic and fear, which seriously inhibits activism.<\/strong><\/p>\n<p>Now is the time to remind people that, over the last 20 years, <em>a growing\nmovement of organizers and technologists have been building user-driven,\nprivacy-respecting, consentful technology platforms as well as organizations\nand communities to develop them.<\/em><\/p>\n<p>We have an entire eco system of:<\/p>\n<ul>\n<li><a href=\"https:\/\/platform.coop\/\">technology platform cooperatives<\/a>,<\/li>\n<li>movement aligned Internet providers that pre-date the founding of Twitter\nand are still going strong (<a href=\"https:\/\/mayfirst.coop\">May First<\/a>,\n<a href=\"https:\/\/riseup.net\">Riseup<\/a>, and <a href=\"https:\/\/autistici.org\/\">Autistici<\/a> just\nto name a few),<\/li>\n<li>the <a href=\"https:\/\/fediverse.party\/en\/fediverse\/\">fediverse<\/a>, a well developed,\nde-centralized alterntiave to corporate social media (try\n<a href=\"https:\/\/wiki.social.coop\/home.html\">Social.coop<\/a> if you want to get started),<\/li>\n<li>powerful open source, privacy respecting software geared for organizing and movement providers\nhosting it (see <a href=\"https:\/\/civicrm.org\/\">CiviCRM<\/a> and <a href=\"https:\/\/progressivetech.org\">Progressive Technology\nProject<\/a>),<\/li>\n<li>multi-year campaigns targeting poor tech practices of corporate technology\ngiants (see <a href=\"https:\/\/mijente.org\">Mijente&rsquo;s<\/a> <a href=\"https:\/\/notechforice.com\/\">No Tech for\nICE<\/a>),<\/li>\n<li>so many more examples, far too numerous to name.<\/li>\n<\/ul>\n<p>All of these projects need our love and support over the long haul.  Please\nhelp spread the word - rather then just deleting an app, let&rsquo;s encourage people\nto join an organziation or try out a new kind of technology that will serve us\ndown the road when we may need it even more then today.<\/p>\n"},{"title":"A very liberal spam assassin rule","link":"https:\/\/current.workingdirectory.net\/posts\/2022\/liberal-spam-rule\/","pubDate":"Mon, 20 Jun 2022 08:27:10 -0400","guid":"https:\/\/current.workingdirectory.net\/posts\/2022\/liberal-spam-rule\/","description":"<p>I just sent myself a test message via <a href=\"https:\/\/ourpowerbase.net\/\">Powerbase<\/a> (a\nhosted <a href=\"https:\/\/civicrm.org\">CiviCRM<\/a> project for community organizers) and it\ndidn&rsquo;t arrive. Wait, nope, there it is in my junk folder with a spam score of\n6!<\/p>\n<div class=\"highlight\"><pre tabindex=\"0\" class=\"chroma\"><code class=\"language-fallback\" data-lang=\"fallback\"><span class=\"line\"><span class=\"cl\">X-Spam-Status: Yes, score=6.093 tagged_above=-999 required=5\n<\/span><\/span><span class=\"line\"><span class=\"cl\">\ttests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1,\n<\/span><\/span><span class=\"line\"><span class=\"cl\">\tDKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, DMARC_MISSING=0.1,\n<\/span><\/span><span class=\"line\"><span class=\"cl\">\tHTML_MESSAGE=0.001, KAM_WEBINAR=3.5, KAM_WEBINAR2=3.5,\n<\/span><\/span><span class=\"line\"><span class=\"cl\">\tNO_DNS_FOR_FROM=0.001, SPF_HELO_NONE=0.001, ST_KGM_DEALS_SUB_11=1.1,\n<\/span><\/span><span class=\"line\"><span class=\"cl\">\tT_SCC_BODY_TEXT_LINE=-0.01] autolearn=no autolearn_force=no\n<\/span><\/span><\/code><\/pre><\/div><p>What just happened?<\/p>\n<p>A careful look at the scores suggest that the <code>KAM_WEBINAR<\/code> and <code>KAM_WEBINAR2<\/code>\nrules killed me. I&rsquo;ve never heard of them (this email came through a system I&rsquo;m\nnot administering). So, I did some searching and <a href=\"https:\/\/github.com\/NethServer\/nethserver-mail-filter\/blob\/master\/root\/etc\/mail\/spamassassin\/KAM.cf\">found a page with the\nrules<\/a>:<\/p>\n<div class=\"highlight\"><pre tabindex=\"0\" class=\"chroma\"><code class=\"language-fallback\" data-lang=\"fallback\"><span class=\"line\"><span class=\"cl\"># SEMINARS AND WORKSHOPS SPAM\n<\/span><\/span><span class=\"line\"><span class=\"cl\">header   __KAM_WEBINAR1 From =~ \/education|career|manage|learning|webinar|project|efolder\/i\n<\/span><\/span><span class=\"line\"><span class=\"cl\">header   __KAM_WEBINAR2 Subject =~ \/last chance|increase productivity|workplace morale|payroll dept|trauma.training|case.study|issues|follow.up|service.desk|vip.(lunch|breakfast)|manage.your|private.business|professional.checklist|customers.safer|great.timesaver|prep.course|crash.course|hunger.to.learn|(keys|tips).(to|for).smarter\/i\n<\/span><\/span><span class=\"line\"><span class=\"cl\">header   __KAM_WEBINAR3 Subject =~ \/webinar|strateg|seminar|owners.meeting|webcast|our.\\d.new|sales.video\/i\n<\/span><\/span><span class=\"line\"><span class=\"cl\">body     __KAM_WEBINAR4 \/executive.education|contactid|register now|\\d+.minute webinar|management.position|supervising.skills|discover.tips|register.early|take.control|marketing.capabilit|drive.more.sales|leveraging.cloud|solution.provider|have.a.handle|plan.to.divest|being.informed|upcoming.webinar|spearfishing.email|increase.revenue|industry.podcast|\\d+.in.depth.tips|early.bird.offer|pmp.certified|lunch.briefing\/i\n<\/span><\/span><span class=\"line\"><span class=\"cl\">\n<\/span><\/span><span class=\"line\"><span class=\"cl\">meta     KAM_WEBINAR (__KAM_WEBINAR1 + __KAM_WEBINAR2 + __KAM_WEBINAR3 + __KAM_WEBINAR4 &gt;= 3)\n<\/span><\/span><span class=\"line\"><span class=\"cl\">describe KAM_WEBINAR Spam for webinars\n<\/span><\/span><span class=\"line\"><span class=\"cl\">score    KAM_WEBINAR 3.5\n<\/span><\/span><span class=\"line\"><span class=\"cl\">\n<\/span><\/span><span class=\"line\"><span class=\"cl\">meta     KAM_WEBINAR2 (__KAM_WEBINAR1 + __KAM_WEBINAR2 + __KAM_WEBINAR3 + __KAM_WEBINAR4 &gt;= 4)\n<\/span><\/span><span class=\"line\"><span class=\"cl\">describe KAM_WEBINAR2 Spam for webinars\n<\/span><\/span><span class=\"line\"><span class=\"cl\">score    KAM_WEBINAR2 3.5\n<\/span><\/span><\/code><\/pre><\/div><p>For those of you who don&rsquo;t care to parse those regular expressions, here&rsquo;s a summary:<\/p>\n<ul>\n<li>There are four tests. If you fail 3 or more, you get 3.5 points, if you fail\n4 you get another 3.5 points (my email failed all 4).<\/li>\n<li>Here is how I failed them:\n<ul>\n<li>The <em>from address<\/em> can&rsquo;t have a bunch of words, including &ldquo;<em>project.<\/em>&rdquo; My from address includes my organization&rsquo;s name: The Progressive Technology Project.<\/li>\n<li>The <em>subject line<\/em> cannot include a number of strings, including &ldquo;<em>last chance<\/em>.&rdquo; My subject line was &ldquo;Last change to register for our webinar.&rdquo;<\/li>\n<li>The <em>subject line<\/em> cannot include a number of other strings, including &ldquo;<em>webinar<\/em>&rdquo; (and also webcast and even strategy). My subject line was &ldquo;Last chance to register for our webinar.&rdquo;<\/li>\n<li>The <em>body<\/em> of the message cannot include a bunch of strings, including &ldquo;<em>register now.<\/em>&rdquo; Well, you won&rsquo;t be suprised to know that my email contained the string &ldquo;Register now.&rdquo;<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>Hm. I&rsquo;m glad I can now fix our email, but this doesn&rsquo;t work so well for people\nwith a name that includes &ldquo;project&rdquo; that like to organize webinars for which you\nhave to register.<\/p>\n"},{"title":"LVM Cache Surprises","link":"https:\/\/current.workingdirectory.net\/posts\/2022\/lvm-cache\/","pubDate":"Thu, 03 Mar 2022 08:27:10 -0400","guid":"https:\/\/current.workingdirectory.net\/posts\/2022\/lvm-cache\/","description":"<p>By far the biggest LVM Cache surprise is just how well it works.<\/p>\n<p>Between 2010 and 2020, my single, biggest and most consistent headache managing\nservers at <a href=\"https:\/\/mayfirst.coop\">May First<\/a> has been disk i\/o. We run a\nnumber of physical hosts with encrypted disks, with each providing a dozen or\nso sundry KVM guests. And they consume a lot of disk i\/o.<\/p>\n<p>This problem kept me awake at night and made me want to put my head on the\ntable and cry during the day as I monitored the output of <code>vmstat 1<\/code> and\nwatched each disk i\/o death spiral unfold.<\/p>\n<p>We tried everything. Turned off fsck&rsquo;s, turned off RAID monthly checks.\nSwitched to less intensive backup systems. Added solid state drives and tried\nto stragically distribute them to our database partitions and other read\/write\nheavy services. Added tmpfs file systems where it was possible.<\/p>\n<p>But, the sad truth was: we simply did not have the resources to pay for the\ninfrastructure that could support the disk i\/o our services demanded.<\/p>\n<p>Then, we discovered LVM caching (cue Hallelujah). We starting provisioning SSD\npartitions to back up our busiest spinning disk logical volumes and presto. Ten\nyears of agony gone like a poof of smoke!<\/p>\n<p>I don&rsquo;t know which individuals are responsible for writing the LVM caching code\nbut if you see this: THANK YOU! Your contributions to the world are noticed,\nappreciated and have had an enormous impact on at least one individual.<\/p>\n<h2 id=\"some-surprises\">Some surprises<\/h2>\n<h3 id=\"filters\">Filters<\/h3>\n<p>For the last two years, with the exception of <a href=\"https:\/\/current.workingdirectory.net\/posts\/2020\/lvm-cache\/\">one little heart\nattack<\/a>, LVM caches have gone very smoothly.<\/p>\n<p>Then, last week we upgraded 13 physical servers straight through from stretch\nto bullseye.<\/p>\n<p>It went relatively smoothly for the first half of our servers (the old ones\nhosting fewer resources). But, after rebooting our first server with lvm\ncaching going on, we noticed that the cached disk wasn&rsquo;t accessible.<\/p>\n<p>No problem, we reasoned. We&rsquo;ll just uncache it. Except that didn&rsquo;t work either.\nWe tried every argument we could find on the Internet but <code>lvm<\/code> insisted that\nthe block device from the SSD volume group (that provides the caching device)\nwas not available. Running <code>pvs<\/code> showed an &ldquo;unknown&rdquo; device and <code>vgs<\/code> reported\nsimilar errors. Now I started to panic a bit. There was a clean shutdown of the\nserver, so surely all the data had been flushed to the disk. But, how can we\nget that data? We started a restore from backup process because we really\nthought that data was gone for ever.<\/p>\n<p>Then we had a really great theory: the caching logical volume comes from the\nSSD volume group, which gets decrypted <em>after<\/em> the spinning disk volume group.<\/p>\n<p>Maybe there&rsquo;s a timing issue? When the spinning disk volume group comes online,\nthe caching logical volume is not yet available.<\/p>\n<p>So, we booted into busybox, and manually decrypted the SSD volume first,\nfollowed by the spinning disk volume. Alas, no dice.<\/p>\n<p>Now that we were fully desperate, we decided to restore the lvm configuration\nfile for the entire spinning disk volume group. This felt kinda risky since we\nmight be damaging all the currently working logical volumes, but it seemed like\nthe only option we had.<\/p>\n<p>The main problem was that busybox didn&rsquo;t seem to have the lvm config tool we\nneeded to restore the configuration from our backup (I think it might be there\nbut it was late and we couldn&rsquo;t figure it out). And, our only readily available\nlive install media was a Debian stretch disk via\n<a href=\"https:\/\/tracker.debian.org\/pkg\/debirf\">debirf<\/a>.<\/p>\n<p>Debian stretch is pretty old and we really would have preferred to have the\nmost modern tools available, but we decided to go with what we had.<\/p>\n<p>And, that was a good thing, because as soon as we booted into stretch and\ndecrypted the disks, the lvm volume suddenly appeared, happy as ever. We\nuncached it and booted into the host system and there it was.<\/p>\n<p>We went to bed confused but relieved.<\/p>\n<p>The next morning my co-worker figured it out: filtering.<\/p>\n<p>During the stretch days we occassionally ran into an annoying problem: the\nlogical volumes from guests would suddenly pop up on the host. This was mostly\nannoying but also it made possible some serious mistakes if you accidentally\ntook a volume from a guest and used it on the host.<\/p>\n<p>The LVM folks seemed to have noticed this problem and introduced a new default\nfilter that tries to only show you the devices that you should be seeing.<\/p>\n<p>Unfortunately for us, this new filter removed logical volumes from the list of\navailable physical volumes. That does make sense for most people. But, not for\nus. It sounds a bit weird, but our setup looks like this:<\/p>\n<ul>\n<li>One volume group derived from the spinning disks<\/li>\n<li>One volume group derived from the SSD disks<\/li>\n<\/ul>\n<p>Then we carve out logical volumes from each for each guest.<\/p>\n<p>Once we discovered LVM caching, we carved out SSD logical volumes to be used as\ncaches for the spinning logical volumes.<\/p>\n<p>In restrospect, if we could start over, we would probably do it differently.<\/p>\n<p>In any event, once we discovered the problem, we used the handy configuration\noptions in <code>lvm.conf<\/code> to tweak the filters to include our cache disks and once\nagain, everything is back to working.<\/p>\n<h3 id=\"saturated-ssds\">Saturated SSDs<\/h3>\n<p>The other surprise seems unrelated to the upgrade. We have a phsyical server\nthat has been suffering from disk i\/o problems despite our use of LVM caching.<\/p>\n<p>Our answer, of course, was to add more LVM caches to the spinning logical\nvolumes that seemed to be suffering.<\/p>\n<p>But somehow this was making things even worse.<\/p>\n<p>Then, we finally just removed the LVM caches from all the spinning disks and\npresto, disk i\/o problems seemed to go away. What? Isn&rsquo;t that the opposite of\nwhat&rsquo;s supposed to happen?<\/p>\n<p>We&rsquo;re still trying to figure this one out, but it seems that our SSDs are\nsaturated, in which case adding them as a caching volume really <em>is<\/em> going to\nmake things worse.<\/p>\n<p>We&rsquo;re still not sure why they are saturated when none of the SSDs on our other\nhosts are saturated, but a few theories include:<\/p>\n<ul>\n<li>\n<p>They are doing more writing and\/or it&rsquo;s a different kind of writing. I&rsquo;m\nstill not sure I quite have the right tool to compare this host with other\nhosts. And, this host is our only MySQL network database server, hosting\nhundreds of GBs of database - all writing\/reading direclty onto the SSDs.<\/p>\n<\/li>\n<li>\n<p>They are broken or substanard SSDs (<code>smartctl<\/code> doesn&rsquo;t uncover any problems\nbut maybe it&rsquo;s a bad model?)<\/p>\n<\/li>\n<\/ul>\n<p>I&rsquo;ll update this post as we learn more but welcome any suggestions in the\ncomments.<\/p>\n<p><strong>Update: 2022-03-07<\/strong><\/p>\n<p>Two more possible causes:<\/p>\n<ul>\n<li>\n<p>Our use of the write back feature: LVM cache has a nice feature that caches\nwrites to smooth out writes to the underlying disk. Maybe our disks are\nsimply writing more then can be handled and not using write back is our\nsolution. This server supports a guest with an unusually large disk.<\/p>\n<\/li>\n<li>\n<p>Maybe we haven&rsquo;t allocated a big enough LVM cache for the given volume so\nthe contents are constantly being ejected?<\/p>\n<\/li>\n<\/ul>\n<p><strong>Update: 2022-06-20<\/strong><\/p>\n<p>We figured it out shortly after my last update, but I never came back to update\nthis post. After turning off lvm caching entirely, everything went back to\nnormal during regular working hours (phew!).<\/p>\n<p>But&hellip; the backup took close to 9 hours to complete. Comparable servers take a\ncouple hours to backup. So, we started a process of backing up only half the\ntop level directories. If the backup went through normally, we added back in\nhalf the directories we previously omitted.<\/p>\n<p>Over the course of about 7 days we narrowed it down to just one top level\ndirectory and, after perusing that directory I found a 20GB file. That&rsquo;s not\nterribly unusual and certainly should not be causing this level of crisis (we\nuse an incremental backup system so only the parts of that file that have\nchanged should get backed up).<\/p>\n<p>But&hellip; this was no ordinary 20GB file. It was the file written to by a\nWordPress site with <a href=\"https:\/\/wordpress.org\/support\/article\/debugging-in-wordpress\/\">debug\nenabled<\/a>. And,\nmost importantly, it had been writing debug errors to this file for close to\nfive years. That means two important things for us:<\/p>\n<ul>\n<li>It most likely was fragmented all over the disk<\/li>\n<li>It changed every day<\/li>\n<\/ul>\n<p>From the backup perspective, it means every time the backup ran, the entire\nfile had to be read to see what changed, which required pulling 20GB in the\nmost inefficient way possible causing a major spike in disk i\/o.<\/p>\n<p>From the LVM cache perspective, it means that every time this file was written\nto (i.e. every time this lousy WordPress site logged a debug message), the file\nhad to be read from the spinning disk into the ssd cache disk. I imagine that\nlvm cache doesn&rsquo;t have a &ldquo;only load what&rsquo;s changed&rdquo; feature and instead simply\nre-reads the entire file everytime it changes.<\/p>\n<p>Mystery solved. LVM cache still rocks. And now we have a new DDOS vector for\nvery patient people.<\/p>\n"},{"title":"Putty Problems","link":"https:\/\/current.workingdirectory.net\/posts\/2021\/putty-problems\/","pubDate":"Mon, 20 Sep 2021 08:27:10 -0400","guid":"https:\/\/current.workingdirectory.net\/posts\/2021\/putty-problems\/","description":"<p>I upgraded my first servers from buster to bullseye over the weekend and it\nwent very smoothly, so <em>big<\/em> thank you to all the debian developers who\ncontributed your labor to the bullseye release!<\/p>\n<p>This morning, however, I hit a snag when the first windows users tried to login.\nIt seems like a putty bug (see update below).<\/p>\n<p>First, the user received an error related to algorithm selection. I didn&rsquo;t\nrecord the exact error and simply suggested that the user upgrade.<\/p>\n<p>Once the user was running the latest version of putty (0.76), they received a new error:<\/p>\n<pre><code>Server refused public-key signature despite accepting key!\n<\/code><\/pre>\n<p>I turned up debugging on the server and recorded:<\/p>\n<pre><code>Sep 20 13:10:32 container001 sshd[1647842]: Accepted key RSA SHA256:t3DVS5wZmO7DVwqFc41AvwgS5gx1jDWnR89apGmFpf4 found at \/home\/XXXXXXXXX\/.ssh\/authorized_keys:6\nSep 20 13:10:32 container001 sshd[1647842]: debug1: restore_uid: 0\/0\nSep 20 13:10:32 container001 sshd[1647842]: Postponed publickey for XXXXXXXXX from xxx.xxx.xxx.xxx port 63579 ssh2 [preauth]\nSep 20 13:10:33 container001 sshd[1647842]: debug1: userauth-request for user XXXXXXXXX service ssh-connection method publickey [preauth]\nSep 20 13:10:33 container001 sshd[1647842]: debug1: attempt 2 failures 0 [preauth]\nSep 20 13:10:33 container001 sshd[1647842]: debug1: temporarily_use_uid: 1000\/1000 (e=0\/0)\nSep 20 13:10:33 container001 sshd[1647842]: debug1: trying public key file \/home\/XXXXXXXXX\/.ssh\/authorized_keys\nSep 20 13:10:33 container001 sshd[1647842]: debug1: fd 5 clearing O_NONBLOCK\nSep 20 13:10:33 container001 sshd[1647842]: debug1: \/home\/XXXXXXXXX\/.ssh\/authorized_keys:6: matching key found: RSA SHA256:t3DVS5wZmO7DVwqFc41AvwgS5gx1jDWnR89apGmFpf4\nSep 20 13:10:33 container001 sshd[1647842]: debug1: \/home\/XXXXXXXXX\/.ssh\/authorized_keys:6: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding\nSep 20 13:10:33 container001 sshd[1647842]: Accepted key RSA SHA256:t3DVS5wZmO7DVwqFc41AvwgS5gx1jDWnR89apGmFpf4 found at \/home\/XXXXXXXXX\/.ssh\/authorized_keys:6\nSep 20 13:10:33 container001 sshd[1647842]: debug1: restore_uid: 0\/0\nSep 20 13:10:33 container001 sshd[1647842]: debug1: auth_activate_options: setting new authentication options\nSep 20 13:10:33 container001 sshd[1647842]: Failed publickey for XXXXXXXXX from xxx.xxx.xxx.xxx port 63579 ssh2: RSA SHA256:t3DVS5wZmO7DVwqFc41AvwgS5gx1jDWnR89apGmFpf4\nSep 20 13:10:39 container001 sshd[1647514]: debug1: Forked child 1648153.\nSep 20 13:10:39 container001 sshd[1648153]: debug1: Set \/proc\/self\/oom_score_adj to 0\nSep 20 13:10:39 container001 sshd[1648153]: debug1: rexec start in 5 out 5 newsock 5 pipe 8 sock 9\nSep 20 13:10:39 container001 sshd[1648153]: debug1: inetd sockets after dupping: 4, 4\n<\/code><\/pre>\n<p>The server log seems to agree with the client returned message: first the key\nwas accepted, then it was refused.<\/p>\n<p>We re-generated a new key. We turned off the windows firewall. We deleted all\nthe putty settings via the windows registry and re-set them from scratch.<\/p>\n<p>Nothing seemed to work. Then, another windows user reported no problem (and\nthat user was running putty version 0.74). So the first user downgraded to 0.74\nand everything worked fine.<\/p>\n<h2 id=\"update\">Update<\/h2>\n<p>Wow, very impressed with the responsiveness of putty devs!<\/p>\n<p>And, who knew that putty is available in debian??<\/p>\n<p>Long story short: putty version 0.76 works on linux and, from what I can tell,\nworks for everyone except my one user. Maybe it&rsquo;s their provider doing some\nfiltering?  Maybe a nuance to their version of Windows?<\/p>\n"},{"title":"Anyone still using gitweb?","link":"https:\/\/current.workingdirectory.net\/posts\/2021\/gitweb\/","pubDate":"Wed, 18 Aug 2021 08:27:10 -0400","guid":"https:\/\/current.workingdirectory.net\/posts\/2021\/gitweb\/","description":"<p>It seems like the self-hosting git world has all moved to\n<a href=\"https:\/\/about.gitlab.com\/install\/\">gitlab<\/a> or <a href=\"https:\/\/gitea.com\/\">gitea<\/a>.<\/p>\n<p>For a number of reasons not worth enumerating, I&rsquo;m still running\n<a href=\"https:\/\/packages.debian.org\/search?keywords=gitolite3\">gitolite<\/a> and recently decided I\nwanted to checkout my code via https using\n<a href=\"https:\/\/packages.debian.org\/search?keywords=gitweb\">gitweb<\/a>.<\/p>\n<p>I got through most of the installation and configuration without trouble (I\ncould browse via the web and see all my repositories). But, when I tried to\n<code>git clone<\/code> using the https address I got a fatal &ldquo;not found&rdquo; error.<\/p>\n<p>It seems that gitweb, out of the box, allows for easy web-browsing of git\nrepositories but needs some extra work if you want to clone over https.\nSpecifically, you need to use <code>git-http-backend<\/code>.<\/p>\n<p>The <code>git-http-backend<\/code> man page is very useful, but assumes you are accessing\nyour repos via <code>https:\/\/example.org\/git<\/code> instead of simply\n<code>https:\/\/git.exmple.org<\/code>.<\/p>\n<p>These lines are my variation to the suggested apache configuration lines provided by\n<code>man git-http-backend<\/code>.<\/p>\n<p>They differ by:<\/p>\n<ul>\n<li>allowing for web access without specifying a subdirectory<\/li>\n<li>using the debian <code>\/usr\/lib\/git-core<\/code> path instead of <code>\/usr\/libexec\/git-core<\/code><\/li>\n<li>removing <code>git-receive-pack<\/code> since I only plan to clone and don&rsquo;t plan to push\nback to this repo.<\/li>\n<\/ul>\n<div class=\"highlight\"><pre tabindex=\"0\" class=\"chroma\"><code class=\"language-gdscript3\" data-lang=\"gdscript3\"><span class=\"line\"><span class=\"cl\"><span class=\"n\">DocumentRoot<\/span> <span class=\"o\">\/<\/span><span class=\"n\">usr<\/span><span class=\"o\">\/<\/span><span class=\"n\">share<\/span><span class=\"o\">\/<\/span><span class=\"n\">gitweb<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"o\">&lt;<\/span><span class=\"ne\">Directory<\/span> <span class=\"o\">\/<\/span><span class=\"n\">usr<\/span><span class=\"o\">\/<\/span><span class=\"n\">share<\/span><span class=\"o\">\/<\/span><span class=\"n\">gitweb<\/span><span class=\"o\">&gt;<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">  <span class=\"n\">Options<\/span> <span class=\"o\">+<\/span><span class=\"n\">FollowSymLinks<\/span> <span class=\"o\">+<\/span><span class=\"n\">ExecCGI<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">  <span class=\"n\">AddHandler<\/span> <span class=\"n\">cgi<\/span><span class=\"o\">-<\/span><span class=\"n\">script<\/span> <span class=\"o\">.<\/span><span class=\"n\">cgi<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">  <span class=\"n\">Require<\/span> <span class=\"n\">all<\/span> <span class=\"n\">granted<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"o\">&lt;\/<\/span><span class=\"ne\">Directory<\/span><span class=\"o\">&gt;<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"o\">&lt;<\/span><span class=\"ne\">Directory<\/span> <span class=\"o\">\/<\/span><span class=\"n\">usr<\/span><span class=\"o\">\/<\/span><span class=\"n\">lib<\/span><span class=\"o\">\/<\/span><span class=\"n\">git<\/span><span class=\"o\">-<\/span><span class=\"n\">core<\/span><span class=\"o\">&gt;<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">  <span class=\"n\">Require<\/span> <span class=\"n\">all<\/span> <span class=\"n\">granted<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"o\">&lt;\/<\/span><span class=\"ne\">Directory<\/span><span class=\"o\">&gt;<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"n\">SetEnv<\/span> <span class=\"n\">GIT_PROJECT_ROOT<\/span> <span class=\"o\">\/<\/span><span class=\"k\">var<\/span><span class=\"o\">\/<\/span><span class=\"n\">lib<\/span><span class=\"o\">\/<\/span><span class=\"n\">git<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"n\">AliasMatch<\/span> <span class=\"o\">^\/<\/span><span class=\"p\">(<\/span><span class=\"o\">.*\/<\/span><span class=\"n\">objects<\/span><span class=\"o\">\/<\/span><span class=\"p\">[<\/span><span class=\"mi\">0<\/span><span class=\"o\">-<\/span><span class=\"mi\">9<\/span><span class=\"n\">a<\/span><span class=\"o\">-<\/span><span class=\"n\">f<\/span><span class=\"p\">]{<\/span><span class=\"mi\">2<\/span><span class=\"p\">}<\/span><span class=\"o\">\/<\/span><span class=\"p\">[<\/span><span class=\"mi\">0<\/span><span class=\"o\">-<\/span><span class=\"mi\">9<\/span><span class=\"n\">a<\/span><span class=\"o\">-<\/span><span class=\"n\">f<\/span><span class=\"p\">]{<\/span><span class=\"mi\">38<\/span><span class=\"p\">})<\/span><span class=\"o\">$<\/span>          <span class=\"o\">\/<\/span><span class=\"k\">var<\/span><span class=\"o\">\/<\/span><span class=\"n\">lib<\/span><span class=\"o\">\/<\/span><span class=\"n\">git<\/span><span class=\"o\">\/$<\/span><span class=\"mi\">1<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"n\">AliasMatch<\/span> <span class=\"o\">^\/<\/span><span class=\"p\">(<\/span><span class=\"o\">.*\/<\/span><span class=\"n\">objects<\/span><span class=\"o\">\/<\/span><span class=\"n\">pack<\/span><span class=\"o\">\/<\/span><span class=\"n\">pack<\/span><span class=\"o\">-<\/span><span class=\"p\">[<\/span><span class=\"mi\">0<\/span><span class=\"o\">-<\/span><span class=\"mi\">9<\/span><span class=\"n\">a<\/span><span class=\"o\">-<\/span><span class=\"n\">f<\/span><span class=\"p\">]{<\/span><span class=\"mi\">40<\/span><span class=\"p\">}<\/span><span class=\"o\">.<\/span><span class=\"p\">(<\/span><span class=\"n\">pack<\/span><span class=\"o\">|<\/span><span class=\"n\">idx<\/span><span class=\"p\">))<\/span><span class=\"o\">$<\/span> <span class=\"o\">\/<\/span><span class=\"k\">var<\/span><span class=\"o\">\/<\/span><span class=\"n\">lib<\/span><span class=\"o\">\/<\/span><span class=\"n\">git<\/span><span class=\"o\">\/$<\/span><span class=\"mi\">1<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"n\">Alias<\/span> <span class=\"o\">\/<\/span><span class=\"k\">static<\/span> <span class=\"o\">\/<\/span><span class=\"n\">usr<\/span><span class=\"o\">\/<\/span><span class=\"n\">share<\/span><span class=\"o\">\/<\/span><span class=\"n\">gitweb<\/span><span class=\"o\">\/<\/span><span class=\"k\">static<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"n\">ScriptAliasMatch<\/span> \\\n<\/span><\/span><span class=\"line\"><span class=\"cl\">  <span class=\"s2\">&#34;(?x)^\/(.*\/(HEAD | <\/span><span class=\"se\">\\\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"se\"><\/span><span class=\"s2\">    info\/refs | <\/span><span class=\"se\">\\\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"se\"><\/span><span class=\"s2\">    objects\/info\/[^\/]+ | <\/span><span class=\"se\">\\\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"se\"><\/span><span class=\"s2\">    git-upload-pack))$&#34;<\/span> \\\n<\/span><\/span><span class=\"line\"><span class=\"cl\">  <span class=\"o\">\/<\/span><span class=\"n\">usr<\/span><span class=\"o\">\/<\/span><span class=\"n\">lib<\/span><span class=\"o\">\/<\/span><span class=\"n\">git<\/span><span class=\"o\">-<\/span><span class=\"n\">core<\/span><span class=\"o\">\/<\/span><span class=\"n\">git<\/span><span class=\"o\">-<\/span><span class=\"n\">http<\/span><span class=\"o\">-<\/span><span class=\"n\">backend<\/span><span class=\"o\">\/$<\/span><span class=\"mi\">1<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"n\">ScriptAlias<\/span> <span class=\"o\">\/<\/span> <span class=\"o\">\/<\/span><span class=\"n\">usr<\/span><span class=\"o\">\/<\/span><span class=\"n\">share<\/span><span class=\"o\">\/<\/span><span class=\"n\">gitweb<\/span><span class=\"o\">\/<\/span><span class=\"n\">gitweb<\/span><span class=\"o\">.<\/span><span class=\"n\">cgi<\/span><span class=\"o\">\/<\/span>\n<\/span><\/span><\/code><\/pre><\/div><p>The main trick is to direct some requests to <code>apache2<\/code>, some requests to\n<code>\/usr\/lib\/git-core\/git-http-backend<\/code>, and everything else to <code>gitweb.cgi<\/code>.<\/p>\n"},{"title":"Fixing old PHP code","link":"https:\/\/current.workingdirectory.net\/posts\/2021\/old-php\/","pubDate":"Sat, 31 Jul 2021 11:11:05 -0400","guid":"https:\/\/current.workingdirectory.net\/posts\/2021\/old-php\/","description":"<p>I wrote a control panel in 2005 using PHP, without any framework. Who could\nhave guessed it would still be in production now?<\/p>\n<p>We&rsquo;ve recently decided to put off replacing it for a few years, so I have to\nfix all the deprecation warnings, which are almost all due to:<\/p>\n<pre><code>while(list($k, $v) = each($array)) {\n<\/code><\/pre>\n<p>At some point, early in my PHP coding life, someone told me <code>foreach($array as $k =&gt; $v) {<\/code> was bad. I don&rsquo;t even remember why. But it stuck, so my code is\nlittered with the while\/list\/each approach. If I ever wrote malware in PHP you\ncould definitely fingerprint me with this one.<\/p>\n<p>I&rsquo;m working on some sed magic to fix them, starting with:<\/p>\n<pre><code>find . -name '*.php' -exec sed -E -i 's#while\\(list\\((\\$[a-z_]*), ?(\\$[a-z_]*)\\) = each\\((\\$[a-z_&gt;-]+)\\)\\) \\{#foreach(\\3 as \\1 =&gt; \\2) {#g' '{}' \\;\n<\/code><\/pre>\n<p>But&hellip; it misses this variation:<\/p>\n<pre><code>while(list(, $v) = each($array)) {\n<\/code><\/pre>\n<p>So I also ran:<\/p>\n<pre><code>find . -name '*.php' -exec sed -E -i 's#while\\(list\\(,(\\$[a-z_]*)\\) = each\\((\\$[a-z_&gt;-]+)\\)\\) \\{#foreach(\\2 as \\1) {#g' '{}' \\;\n<\/code><\/pre>\n<p>I ended up with 10 replacments I had to do by hand (<code>while(list($k) = each($array))<\/code> and a few others with unusual spacing).<\/p>\n"},{"title":"Google and Bitly","link":"https:\/\/current.workingdirectory.net\/posts\/2021\/google-and-bitly\/","pubDate":"Sun, 18 Jul 2021 13:25:59 -0400","guid":"https:\/\/current.workingdirectory.net\/posts\/2021\/google-and-bitly\/","description":"<p>It seems I&rsquo;m the only person on the Internet who didn&rsquo;t know <a href=\"https:\/\/duckduckgo.com\/?q=google+bit.ly+deliverability&amp;t=h_&amp;ia=web\">sending email to\nGoogle with bit.ly links will tank your\ndeliverability<\/a>.\nTo my credit, I&rsquo;ve been answering deliverability support questions for 16 years\nand this has never come up.<\/p>\n<p>Until last week.<\/p>\n<p>For some reason, at <a href=\"https:\/\/mayfirst.coop\/\">May First<\/a> we suddenly had about\nthree percent of our email to Google deferred with the ominous sounding:<\/p>\n<blockquote>\n<p>&ldquo;Our system has detected that this message is 421-4.7.0 suspicious due to the\nnature of the content and\/or the links within.&rdquo;<\/p><\/blockquote>\n<p>The quantity of email that accounts for just three percent of mail to Google is\nhigh, and caused all kinds of monitoring alarms to go off, putting us into a\nbit of panic.<\/p>\n<p>Eventually we realized all but one of the email messages had bit.ly links.<\/p>\n<p>I&rsquo;m still not sure whether this issue was caused by a weird and coincidental\nspike in users sending bit.ly links to Google. Or whether some subtle change in\nthe Google algorithm is responsible. Or some change in our IP address\nreputation placed greater emphasis on bit.ly links.<\/p>\n<p>In the end it doesn&rsquo;t really matter - the real point is that until we disrupt\nthis growing monopoly we will all be at the mercy of Google and their\nalgorithms for email deliverability (and much, much more).<\/p>\n"},{"title":"From Ikiwiki to Hugo","link":"https:\/\/current.workingdirectory.net\/posts\/2021\/hugo\/","pubDate":"Fri, 16 Jul 2021 08:27:10 -0400","guid":"https:\/\/current.workingdirectory.net\/posts\/2021\/hugo\/","description":"<p>Back in the days of Etch, I converted this blog from Drupal to\n<a href=\"https:\/\/ikiwiki.info\">ikiwiki<\/a>. I remember being very excited about this brand\nnew concept of static web sites derived from content stored in a version\ncontrol system.<\/p>\n<p>And now over a decade later I&rsquo;ve moved to <a href=\"https:\/\/gohugo.io\/\">hugo<\/a>.<\/p>\n<p>I feel some loyalty to ikiwiki and Joey Hess for opening my eyes to the static\nweb site concept. But ultimately I grew tired of splitting my time and energy\nbetween learning ikiwiki and hugo, which has been my tool of choice for new\nprojects. When I started getting strange emails that I suspect had something to\ndo with spammers filling out ikiwiki&rsquo;s commenting registration system, I choose\nto invest my time in switching to hugo over debugging and really understanding\nhow ikiwiki handles user registration.<\/p>\n<p>I carefully reviewed <a href=\"https:\/\/anarc.at\/services\/wiki\/ikiwiki-hugo-conversion\/\">anarcat&rsquo;s blog on converting from ikiwiki to\nhugo<\/a> and learned\nabout a lot of ikiwiki features I am not using. Wow, it&rsquo;s times like these that\nI&rsquo;m glad I keep it really simple. Based on the various ikiwiki2hugo python\nscripts I studied, I eventually wrote a <a href=\"https:\/\/gitlab.com\/jamie\/current-working-directory\/-\/tree\/main\/tools\">far simpler\none<\/a>\ntailored to my needs.<\/p>\n<p>Also, in what could only be called a desperate act of procrastination combined\nwith a touch of self-hatred (it&rsquo;s been a rough week) I rejected all the\ncommenting options available to me and choose to <a href=\"https:\/\/gitlab.com\/jamie\/commenter\">implement my own in\nPHP<\/a>.<\/p>\n<p>What?!?!  Why would anyone do such a thing?<\/p>\n<p>I refer you to my previous sentence about desperate procrastination. And\nalso&hellip; I know it&rsquo;s fashionable to hate PHP, but honestly as the first\nprogramming language I learned, there is something comforting and familiar\nabout it. And, on a more objective level, I can deploy it easily to just about\nany hosting provider in the world. I don&rsquo;t have to maintain a unicorn service\nor a nodejs service and make special configuration entries in my web\nconfiguration. All I have to do is upload the php files and I&rsquo;m done.<\/p>\n<p>Well, I&rsquo;m sure I&rsquo;ll regret this decision.<\/p>\n<p>Special thanks to <a href=\"https:\/\/github.com\/lxndrblz\">Alexander Bilz<\/a> for the\n<a href=\"https:\/\/github.com\/lxndrblz\/anatole\/\">anatole hugo theme<\/a>. I choose it via a\nnearly random click to avoid the rabbit hole of choosing a theme. And, by luck,\nit has turned out quite well. I only had to override the commento partial theme\npage to hijack it for my own commenting system&rsquo;s use.<\/p>\n"},{"title":"How to Meet Online with Simultaneous Interpretation","link":"https:\/\/current.workingdirectory.net\/posts\/2021\/jitsi-and-language-justice\/","pubDate":"Tue, 22 Jun 2021 09:14:50 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2021\/jitsi-and-language-justice\/","description":"<p><a href=\"https:\/\/mayfirst.coop\/\">May First Movement Technology<\/a> has been running a\npublic <a href=\"https:\/\/jitsi.org\/jitsi-meet\/\">Jitsi Meet<\/a> instance since well before\nthe pandemic to support Internet-based, video meetings for folks who don&rsquo;t want\nto rely on corporate and proprietary infrastructure.<\/p>\n<p>However (until this week - see below), we haven&rsquo;t been using it for our own\nmeetings for one main reason: simultaneous interpretation. We&rsquo;re an\ninternational organization with roots in the US and Mexico and we are committed\nto building a bi-national leadership with a movement strategy that recongizes\nthe symbolic and practical disaster of the US\/Mexico border.<\/p>\n<p>As a result, we simply can&rsquo;t hold a meeting without simultaneous interpretation\nbetween english and spanish.<\/p>\n<p>Up to now, we&rsquo;ve worked out <a href=\"https:\/\/support.mayfirst.org\/wiki\/mumble-interpreter-setup\">a creative way to have mumble meetings with\nsimultaneous\ninterpretation<\/a>. In\nshort: we have a room for interpretation. If you move into the interpretation\nroom, you hear the interpreter. If you move into the main room, you hear the\nlive voices of the participants. You can switch between rooms as needed. This\napproach is rock solid, and we benefit from mumble&rsquo;s excellent performance in\nlow bandwidth situations and the availability of mumble clients on both Android\nand iPhones.<\/p>\n<p>However, there are limitations, which include:<\/p>\n<ul>\n<li>\n<p>You can&rsquo;t hear both the live voices and the interpretation at the same time:\nit&rsquo;s one or the other. If you are in a face-to-face meeting and receiving\ninterpretation via headphones, you can see the person talking and even\nremove the headphones from one ear to get a sense of the tone and emotions\nof the speaker. Not with mumble. In fact, you can&rsquo;t even tell who is\nspeaking.<\/p>\n<\/li>\n<li>\n<p>Two chat rooms: If you chat to the live group, it&rsquo;s only seen by the live\ngroup. If you chat with the interpretation group, it&rsquo;s only seen by the\ninterpretation group.<\/p>\n<\/li>\n<li>\n<p>No video in mumble: well, some people consider this a positive. I&rsquo;ll leave\nit at that.<\/p>\n<\/li>\n<\/ul>\n<p>After years of reviewing the many dead-end threads and issue requests around\nsimultaneous interpretation on the Jitsi boards (and the Big Blue Button boards\nfor that matter) I finally came across <a href=\"https:\/\/community.jitsi.org\/t\/adjust-volume-levels-via-javascript\/100701\">the\nthread<\/a>\nthat led to the <a href=\"https:\/\/github.com\/jitsi\/jitsi-meet\/pull\/9322\">pull request<\/a>\nthat changed everything.<\/p>\n<p>With the ability to control local volume via the Jitsi Meet API, I was able to\npull together a very small amount of code to produce <a href=\"https:\/\/gitlab.com\/mfmt\/jsi\">Jitsi Simultaneous\nInterpretation (JSI)<\/a> - a way to run your Jitsi\nMeet server with an interpretation slider at the top allowing you to set the\nvolume of the interpreter at any time during the meeting.<\/p>\n<p>It&rsquo;s still not perfect - the main problem is that you can&rsquo;t use any of the\nJitsi Meet apps - so it runs well on most desktops, but when it comes to cell\nphones, it only runs (in browser) on modern android phones.<\/p>\n"},{"title":"From openbox to sway","link":"https:\/\/current.workingdirectory.net\/posts\/2021\/sway\/","pubDate":"Tue, 11 May 2021 08:33:26 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2021\/sway\/","description":"<p>I&rsquo;ve been running the <a href=\"http:\/\/openbox.org\/wiki\/Main_Page\">Openbox window\nmanager<\/a> since 2005. That&rsquo;s longer then I&rsquo;ve\nlived in any one apartment in my entire life!<\/p>\n<p>However, over the years I&rsquo;ve been bracing for a change.<\/p>\n<p>It seems clear the Wayland is the future, although when that future is supposed\nto begin is much more hazy.<\/p>\n<p>Really, I&rsquo;ve felt a bit like a ping pong ball, from panicking over whether\n<a href=\"https:\/\/www.phoronix.com\/scan.php?page=news_item&amp;px=XServer-Abandonware\">Xorg is\nabandoned<\/a> (with a follow up from a <a href=\"https:\/\/ajaxnwnk.blogspot.com\/2020\/10\/on-abandoning-x-server.html\">X server maintainer<\/a>)\nto anxiously wondering if <a href=\"https:\/\/gist.github.com\/probonopd\/9feb7c20257af5dd915e3a9f2d1f2277\">literally everything will break the moment I switch\nto\nWayland<\/a>.<\/p>\n<p>In fact, I started this blog post over a year ago when I first decided to\nswitch from the Openbox to <a href=\"https:\/\/swaywm.org\/\">Sway<\/a>.<\/p>\n<p>This is my third major attempt to make the change and I <em>think<\/em> it will finally\nstick this time.<\/p>\n<p>In retrospect, it would have been more sensible to first switch from openbox to\n<a href=\"https:\/\/i3wm.org\/\">i3<\/a> (which is a huge transition) and then from i3 to sway,\nbut I decided to dive into the deep end with both changes. Note: thanks to a\nhelpful comment on this blog, I learned that there is\n<a href=\"https:\/\/github.com\/wizbright\/waybox\">waybox<\/a>, an openbox clone for wayland,\nwhich would have been another version of a less drastic change.<\/p>\n<p>So&hellip; I&rsquo;m on debian bullseye so I installed sway and friends (from sid).<\/p>\n<p>Then I copied \/etc\/sway\/config to ~\/.config\/sway\/config.<\/p>\n<p>I start openbox after logging in with <code>exec startx<\/code> so after rebooting, I ran\n<code>exec sway<\/code> and to my astonishment sway started. Hooray!<\/p>\n<p>However, I found that <code>ssh-agent<\/code> wasn&rsquo;t running so I couldn&rsquo;t <code>ssh<\/code>  into\nany servers. That&rsquo;s kinda a problem.<\/p>\n<p>Launching <code>ssh-agent<\/code> under openbox was buried deep in\n<code>\/etc\/X11\/Xsession.d\/90x11-common_ssh-agent<\/code> and clearly was not going to\nhappen via wayland.<\/p>\n<p>Since programs using <code>ssh-agent<\/code> depend on the environment variables\n<code>SSH_AUTH_SOCK<\/code> and <code>SSH_AGENT_PID<\/code> being globally available I thought I could\nsimply run <code>$(eval ssh-agent)<\/code> via my tty terminal before running <code>exec sway<\/code>.<\/p>\n<p>And, that <em>would have worked<\/em>. Except&hellip; I like to add my keys via <code>ssh-add -c<\/code>\nso that everytime my key is being used I get a ssh-askpass prompt to confirm\nthe use.<\/p>\n<p>It seems that since <code>ssh-add<\/code> is started before a window manager is running, it\ncan&rsquo;t run the prompt.<\/p>\n<p>Ok, we can fix this. After searching the web, I came upon a solution of running\nssh-agent via <code>systemctl --user<\/code>:<\/p>\n<pre><code># This service myst be started manually after sway\n# starts.\n[Unit]\n\nDescription=OpenSSH private key agent\nIgnoreOnIsolate=true\n\n[Service]\nType=forking\nEnvironment=SSH_AUTH_SOCK=%t\/ssh-agent.socket\nExecStart=\/usr\/bin\/ssh-agent -a $SSH_AUTH_SOCK\n<\/code><\/pre>\n<p>Then, in my <code>~\/.bashrc<\/code> file I have:<\/p>\n<pre><code>if [ -n WAYLAND_DISPLAY ]; then\n  export SSH_AUTH_SOCK=\/run\/user\/1000\/ssh-agent.socket\nfi\n<\/code><\/pre>\n<p>I think <code>$SSH_AGENT_PID<\/code> is only used by <code>ssh-agent<\/code> to kill itself. Now that\nis running via <code>systemd<\/code> - killing it should be do-able without a global\nenvironment variable.\nDone? Hardly.<\/p>\n<p>I&rsquo;ve been using <code>impass<\/code> (nee <code>assword<\/code>) happily for years but alas it is\ntightly integrated with <code>xdo<\/code> and <code>xclip<\/code>.<\/p>\n<p>So&hellip; I&rsquo;ve switched to <code>keepassxc<\/code> which works out of the box with wayland.<\/p>\n<p>My next challenge was the status bar. Farewell faithful\n<a href=\"https:\/\/gitlab.com\/o9000\/tint2\">tint2<\/a>. One of the reasons I failed on my\nfirst two attempts to switch to Sway was the difficulty of getting the swaybar\nto work how I wanted, particularly with nm-applet. Two things allowed me to move forward:<\/p>\n<ul>\n<li><a href=\"https:\/\/packages.debian.org\/bullseye\/waybar\">waybar<\/a> was added to Debian.\nThank you Debian waybar maintainers!<\/li>\n<li>I gave up on having nm-applet work the way I&rsquo;m used to working and resigned\nmyself to using <code>nmtui<\/code>. Sigh.<\/li>\n<\/ul>\n<p>Next up: the waybar clock module doesn&rsquo;t work, but that is <a href=\"https:\/\/github.com\/Alexays\/Waybar\/issues\/977\">easy enough to work\naround<\/a>.<\/p>\n<p>Replacing my uses of <code>xclip<\/code> with\n<a href=\"https:\/\/github.com\/bugaevc\/wl-clipboard\">wl-clipboard<\/a> was a little tedious\nbut really not that difficult.<\/p>\n<p>Getting my screen shot and screen recorder functionality was a bit harder. I\ndid a lot of searching before I finally found and compiled both <a href=\"https:\/\/github.com\/jtheoof\/swappy\">swappy, screen\nshot<\/a> and\n<a href=\"https:\/\/github.com\/ammen99\/wf-recorder\">wf-recorder<\/a>.<\/p>\n<p>In the course of all my adventures, I came across the following helpful tips:<\/p>\n<ul>\n<li><a href=\"https:\/\/github.com\/natpen\/awesome-wayland#screencasting\">awesome wayland<\/a><\/li>\n<li><a href=\"https:\/\/www.autodidacts.io\/switching-to-sway-wayland-from-i3-x11-ubuntu\/\">autodidacts blog<\/a><\/li>\n<li><a href=\"https:\/\/shibumi.dev\/posts\/wayland-in-2021\/\">wayland in 20201<\/a><\/li>\n<li><a href=\"https:\/\/www.jacobo.org\/sway-en-debian-bullseye\/\">Jacobo&rsquo;s experiences (in spanish)<\/a><\/li>\n<\/ul>\n<hr>\n<p>Updates<\/p>\n<ol>\n<li>I&rsquo;ve installed <code>libreoffice-gtk3<\/code> to ensure libre office runs under wayland<\/li>\n<li>I&rsquo;ve installed the <a href=\"https:\/\/www.gimp.org\/news\/2020\/11\/06\/gimp-2-99-2-released\/\">latest Gimp via flatpak to get proper wayland support<\/a>. Update: as of 2022-03-29, it seems that Gimp 2.10.30 in Debian Bullseye works fine with Sway.<\/li>\n<li>I&rsquo;ve exported <code>MOZ_ENABLE_WAYLAND<\/code> to ensure firefox works properly.<\/li>\n<li>I&rsquo;ve found that passing -c to my <code>ssh-add<\/code> command to ensure I am prompted for each use of my key seems to cause sway to crash intermittently. Update: This is no longer the case. Not sure why.<\/li>\n<li>I learned about a <a href=\"https:\/\/hugo.barrera.io\/journal\/2020\/06\/14\/zoom-screensharing-on-archlinux\/\">work around to get screen sharing to work in zoom<\/a>. Somehow this actually works. Amazing. Unfortunately, though, sharing your screen in the context of a Zoom meeting pins your screen for all participants. So, sharing your desktop through your camera really doesn&rsquo;t cut it. I finally landed an an obvious work around: Install chromium (which runs under X11); Install the <a href=\"https:\/\/chrome.google.com\/webstore\/detail\/zoom-redirector\/fmaeeiocbalinknpdkjjfogehkdcbkcd\">the chrome zoom redirector extension<\/a> (update: This extension is no longer available); Open zoom links in chromium; You can now share other chromium windows. Not the full desktop or any wayland window, but if you only need to share a web browser window, you are set; For the record, Zoom links normally are in the format: <a href=\"https:\/\/us04web.zoom.us\/j\/123456\">https:\/\/us04web.zoom.us\/j\/123456<\/a>. If you want to force the use of the web client, just change the &ldquo;j&rdquo; to &ldquo;wc&rdquo;: <a href=\"https:\/\/us04web.zoom.us\/wc\/123456\">https:\/\/us04web.zoom.us\/wc\/123456<\/a>.<\/li>\n<li>Speaking of screen sharing - when using Firefox, I can only share Xwayland screens. Firefox is running under wayland so I can&rsquo;t share it. Chromium is running under xwayland, so I have to use Chromium when screen sharing.<\/li>\n<li>Wait, scratch that about screen sharing in Firefox. I&rsquo;ve installed <a href=\"https:\/\/github.com\/emersion\/xdg-desktop-portal-wlr\">xdg-desktop-portal-wlr<\/a>, added <code>export XDG_CURRENT_DESKTOP=sway<\/code> and <code>export XDG_SESSION_TYPE=wayland<\/code> to my <code>.bashrc<\/code>, and after hours of frustration, realize that I needed to <a href=\"https:\/\/github.com\/netblue30\/firejail\/issues\/3872\">configured firejail to allow it<\/a> so that I can share my entire screen in Firefox. It doesn&rsquo;t yet support sharing a specific window, so I still have to keep chromium around for that (and Chromium can only share xwayland windows). Sigh. Oh, one more thing about Firefox: the option to choose what to share doesn&rsquo;t have &ldquo;Entire Screen&rdquo; as an option, you are just supposed to know that you should choose &ldquo;Use operating system settings&rdquo;.<\/li>\n<li>I still am getting weekly crashes. Some of them I&rsquo;ve fixed by switching to wayland friendly versions (e.g. Libre Office and Gimp) but others I haven&rsquo;t yet tracked down. Update: No longer getting weekly crashes. Very stable.<\/li>\n<li>My keyboard does not have an altgr key, so even though I have selected the &ldquo;English (US) - English (intl., with AltGr dead keys)&rdquo; I can&rsquo;t get accent marks. I went down a rabbit hole of trying to re-map the Alt key to the right of my space bar but it all seemed too complicated. So - I found a way easier approach. In my <code>~\/.config\/sway\/config<\/code> file I have: <code>bindsym Mod4+e exec wtype &quot;\u00e9&quot;<\/code>. I have repeated that line for the main accent marks I need.<\/li>\n<li>Due to a <a href=\"https:\/\/bugzilla.mozilla.org\/show_bug.cgi?id=1590661\">Firefox Bug<\/a>, when I share my desktop or mic or camera, the sharing indicator expands like a normal tiling window instead of remaining a tiny little box on each desktop reminding me that I&rsquo;m sharing something. I&rsquo;d prefer to have it be a tiny little box, but since I can&rsquo;t figure that out, I&rsquo;ve disabled it by typing <code>about:config<\/code> in the Firefox location window, searching for <code>privacy.webrtc.legacyGlobalIndicator<\/code> and setting it to <strong>False<\/strong>. The <a href=\"https:\/\/www.reddit.com\/r\/swaywm\/comments\/iva00z\/hi_help_needed_why_is_mic_indicator_appearing_as\/\">reddit thread<\/a> also suggested finding <code>privacy.webrtc.hideGlobalIndicator<\/code> and setting it to True, but that setting doesn&rsquo;t seem to be available and setting the first one alone seems to do the trick.<\/li>\n<li>Oh, one more environment variable to set: <code>GDK_BACKEND=wayland,x11<\/code>. First I just set it to wayland to get gtk3 apps to use wayland (like gajim). But that broke electron apps (like signal) which notice that variable but don&rsquo;t have a way to display via wayland (at least <a href=\"https:\/\/github.com\/signalapp\/Signal-Desktop\/issues\/3411\">not yet<\/a>). Setting it to &ldquo;wayland,x11&rdquo; shows the priority. Thank you <a href=\"https:\/\/discourse.ubuntu.com\/t\/gtk-backend-selection-or-why-gtk-cannot-open-display-0\/17657\">ubuntu community<\/a>.<\/li>\n<li>I&rsquo;ve also finally consolidated where my environment variables go. I&rsquo;ve added them all to <code>~\/.config\/sway\/env<\/code>. That seems like an official sway place to put them, but sway doesn&rsquo;t pay any attention to them. So I start sway via my own bash script which sources that file via <code>[ -f &quot;$HOME\/.config\/sway\/env&quot; ] &amp;&amp; . &quot;$HOME\/.config\/sway\/env&quot;<\/code> before <code>exec<\/code>&lsquo;ing sway.<\/li>\n<\/ol>\n"},{"title":"The problem with Richard Stallman is not about free speech","link":"https:\/\/current.workingdirectory.net\/posts\/2021\/stallman\/","pubDate":"Mon, 29 Mar 2021 08:50:40 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2021\/stallman\/","description":"<p>Free speech and censorship are critically important issues. And, using them to\ndefend Richard Stallman&rsquo;s return to the Free Software Foundation (FSF)\nboard is just plain wrong.<\/p>\n<p>Richard Stallman resigned from the Board in 2019 after he sent an email in\ndefense of Marvin Minsky (Minsky is accused of raping one of Jeffreys Epstein&rsquo;s\nvictims).<\/p>\n<p>Stallman&rsquo;s fateful email, however, is just <em>one piece of the reason for why he\nshould not be on the board.<\/em> The <a href=\"https:\/\/selamjie.medium.com\/remove-richard-stallman-appendix-a-a7e41e784f88\">full\nstory<\/a>\nis about his history of abuse toward women and is extensive.<\/p>\n<p>On March 21st, 2021, Stallman announced <a href=\"https:\/\/www.theregister.com\/2021\/03\/22\/richard_stallman_back_on_fsf_board\/\">he is back on the\nboard<\/a>.<\/p>\n<p>There are profound reasons why any movement interested in equitable and open\nparticipation would want to publicly distance themselves from Stallman.\nHowever, the long form defenses of Stallman, including <a href=\"https:\/\/www.wetheweb.org\/post\/cancel-we-the-web\">a note from Nadine\nStrossen, the former executive director of the ACLU, quoted in this\ndefense<\/a>, persist.<\/p>\n<p>Many of the arguments defending Richard Stallman (including the one from\nStrossen) are grounded in a belief that Stallman is being punished for his\nunpopular political views, which deserve to be defended on the grounds of\nfreedom of expression.<\/p>\n<p>That&rsquo;s wrong.<\/p>\n<p>Stallman should be kicked off the board because he has a long history of\nabusing his position to hit on women, which, when combined with his public\nopinions on under-age sex and his defense of Minsky, send a strong signal that\nthe FSF does not care about the participation of women.<\/p>\n<p>Being on a board of directors is a privilege, not a right. Being removed from a\nboard is not a punnishment. And being criticized and removed from a board\nbecause your behavior and public statements are an obstacle to building an\ninclusive and equitable movement is what every board should strive to do.<\/p>\n<p>If we are going to make this an issue about free expression, it should be about\nall the political expression lost to the free software movement because\nStallman&rsquo;s unequal behavior toward women excluded an enormous number of\ntalented individuals.<\/p>\n"},{"title":"So... is Signal good or bad?","link":"https:\/\/current.workingdirectory.net\/posts\/2021\/signal\/","pubDate":"Fri, 29 Jan 2021 15:37:11 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2021\/signal\/","description":"<p>After <a href=\"https:\/\/arstechnica.com\/tech-policy\/2021\/01\/whatsapp-users-must-share-their-data-with-facebook-or-stop-using-the-app\/\">Facebook updated their Whatsapp privacy\npolicy<\/a>,\nand a certain rich capitalist who doesn&rsquo;t like Facebook for reasons different\nthen mine told the world to use Signal, Signal&rsquo;s downloads <a href=\"https:\/\/www.businessinsider.com\/whatsapp-facebook-data-signal-download-telegram-encrypted-messaging-2021-1?op=1\">went up by\n4,200%<\/a>.<\/p>\n<p>As often happens when something becomes popular, the criticisms start to fly!<\/p>\n<p>For the record, I currently think promoting <a href=\"https:\/\/signal.org\/\">Signal<\/a> is\nan important tactical strategy for the left. [I also think we should promote\nand install federated chat apps like <a href=\"https:\/\/conversations.im\/\">conversations<\/a>\nand <a href=\"https:\/\/element.io\/\">element<\/a> and <a href=\"https:\/\/delta.chat\/en\/\">delta chat<\/a>\nwhereever it is possible.]<\/p>\n<p>Here are some of the main criticisms I hear that I think are a distraction:<\/p>\n<ul>\n<li>\n<p><strong>Signal forces you to use the Google Play store and Google Services<\/strong>: This\nisn&rsquo;t true any more. You can download <a href=\"https:\/\/signal.org\/android\/apk\/\">the apk\ndirectly<\/a> on a phone without any Google\nservices and it works great. The app will alert you to new versions.<\/p>\n<p>Don&rsquo;t get me wrong: the Signal network still depends on Google services.\nAnd, we <em>should<\/em> be avoiding corporate technology and building our own\ninfrastructure. However, in practice, Signal is an alternative to Whatsapp\nand Telegram - which not only use the same corporate services but are\nproprietary technology that is fully owned by powerful tech giants. Signal\nis still a non-profit organization with a vastly different mission.<\/p>\n<\/li>\n<li>\n<p><strong>Signal&rsquo;s approach to privacy isn&rsquo;t perfect<\/strong> (the most common variation on\nthis theme is that a state actor could monitor your outgoing communications\nand the incoming communications of the person you are communicating with and\nprove that you are communicating with each other).<\/p>\n<p>This criticism missed what makes Signal so important. The beauty of Signal\nis that it addresses the &ldquo;woops!&rdquo; moment most privacy activists had when\nSnowden&rsquo;s data trove become public: it provides <em>mass<\/em> privacy to stop\n<em>mass<\/em> surveillance. Prior to 2013, most tech\/privacy activists were focused\non the &ldquo;targeted&rdquo; individual approach to privacy, working hard to make sure\nour tools were as absolutely perfect as possible for the tiny percentage of\npeople who know they are under surveillance. Very little effort went into\ngetting them adopted on a mass scale.<\/p>\n<p>Criticizing Signal for not providing perfect privacy misses that fact that\nthese things often are trade offs.<\/p>\n<p>This trade-off also applies to the first point - dependency on Google\nservices makes installation far easier for suporting millions of people.<\/p>\n<\/li>\n<\/ul>\n<p>Here are some criticisms that I think are nuanced:<\/p>\n<ul>\n<li>\n<p><strong>Signal is a centralized app<\/strong>: this criticism often includes examples of\nMoxie (Signal&rsquo;s founder) refusing and actively discouraging attempts by\nothers to write software that interacts with Signal.<\/p>\n<p>Signal is free software, which is a major improvement over most corporate\ntechnology. But since it&rsquo;s entirely controlled by one entity, it can be\nshutdown in a heart beat. And, if Signal changes direction, we cannot easily\ntake the work we have all invested in learning signal and create our own\nversion that reflects our values.<\/p>\n<p>This problem is in contrast to federated systems like email - where anyone\ncan run their own email server and apply their own policies. If one email\nserveer is shutdown, you can move to another.<\/p>\n<p>I agree with this critique, but I think it&rsquo;s nuanced because of the trade\noffs. Having full control over the entire network and all the software\nprovides a level of reliability and consistency that would not be possible\nwith a federated protocol. And, we already have three different, fully\nviable federated chat protocols (see above). I&rsquo;d rather have Signal be\nSignal and invest our energy on a federated chat system via the existing,\nwell-developed alternatives.<\/p>\n<p>This opinion is tactical - and could change at any moment. I think there\nwill come a time when we are going to tell the world to move from Signal to\nthe best available federated protocol. But I&rsquo;m not convinced we have a\nrobust enough federated chat infrastructure to support that move.<\/p>\n<\/li>\n<li>\n<p><strong>Signal forces you to use your phone number as an identifier<\/strong>: You can&rsquo;t\nget a Signal account without a phone number. And you generally can&rsquo;t get a\nphone number without revealing some aspect of your identity. That makes\nstaying anoymous very difficult. There are reports of a new Signal feature\nmaking it possible to avoid revealing your phone number when communicating\nwith others, but you would still need a phone number to get an account\nbecause a SMS or phone call confirmation is required.<\/p>\n<\/li>\n<li>\n<p><strong>Signal isn&rsquo;t getting ahead of the curve on abuse<\/strong>: There&rsquo;s an interesting\n<a href=\"https:\/\/www.platformer.news\/p\/-the-battle-inside-signal\">piece informed by former Signal staff\npeople<\/a> about the\nmanagement&rsquo;s resistance to getting ahead of the curve when it comes to\nabuse. How would signal respond to reports of harrassment? What would signal\ndo if it recognized facsists movements organizing on its platform? Any mass\nplatform that is not planning for abuse is going to be in big trouble very\nsoon.<\/p>\n<\/li>\n<\/ul>\n<p>These last two are not exactly two sides of the same coin, but they are\nrelated. How Signal manages to balance privacy and protection from abuse will\nbe the real test as to whether promoting Signal continues to be a useful\nstrategy for the left.<\/p>\n"},{"title":"Being your own Certificate Authority","link":"https:\/\/current.workingdirectory.net\/posts\/2020\/cacert\/","pubDate":"Sun, 15 Nov 2020 13:48:45 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2020\/cacert\/","description":"<p>There are many blogs and tutorials with nice shortcuts providing the necessary\n<code>openssl<\/code> commands to create and sign x509 certficates.<\/p>\n<p>However, there is precious few instructions for how to easily create your own\ncertificate authority.<\/p>\n<p>You probably never want to do this in a production environment, but in a\ndevelopment environment it will make your life signficantly easier.<\/p>\n<h2 id=\"create-the-certificate-authority\">Create the certificate authority<\/h2>\n<h3 id=\"prepare-your-ca-directories\">Prepare your CA directories<\/h3>\n<p>Pick a directory to store your keys and certificates in.<\/p>\n<p>Then make a subdirectory for the certficiate authority and some required directories:<\/p>\n<pre><code>mkdir ca\nmkdir ca\/private\ntouch index.txt\n<\/code><\/pre>\n<h3 id=\"create-the-key-and-certificate\">Create the key and certificate<\/h3>\n<p>Then, make your certificate authority key and certificate:<\/p>\n<pre><code>openssl genrsa -out ca\/private\/cakey.pem 2048\nopenssl req -x509 -new -nodes -key ca\/private\/cakey.pem -sha256 -days 1024 -out ca\/cacert.pem\n<\/code><\/pre>\n<p>Some tips:<\/p>\n<ul>\n<li>You will be prompted to enter some information about your certificate\nauthoirty. Provide the minimum information - i.e., only overwrite the\ndefaults. So, provide a value for <code>Country<\/code>, <code>State or Province<\/code>, and\n<code>Organization Name<\/code> and leave the rest blank.<\/li>\n<li>You probably want to leave the password blank if this is a\ndevelopment\/testing environment.<\/li>\n<\/ul>\n<p>Want to review what you created?<\/p>\n<pre><code>openssl x509 -text -noout -in ca\/cacert.pem \n<\/code><\/pre>\n<h2 id=\"prepare-your-opensslcnf-file\">Prepare your openssl.cnf file<\/h2>\n<p>With Debian Trixie, the default openssl configuration file seems to be\n<code>\/usr\/lib\/ssl\/openssl.cnf<\/code>. Prior versions used <code>\/etc\/ssl\/openssl.cnf<\/code>.<\/p>\n<p>In this document we will explicitly use <code>\/etc\/ssl\/openssl.cnf<\/code> to get the Debiand defaults.<\/p>\n<p>But, there are two bits missing from the Debian configuration that have to be added.<\/p>\n<ol>\n<li>At the very top add: <code>subjectAltName          = ''<\/code>.<\/li>\n<li>In the <code>[user_cert]<\/code> section, add: <code>subjectAltName=$ENV::subjectAltName<\/code><\/li>\n<\/ol>\n<h2 id=\"create-a-your-first-key-and-ceritificate-signing-request\">Create a your first key and ceritificate signing request<\/h2>\n<p>First, pick your domain names (aka &ldquo;common&rdquo; names). For example, <code>example.org<\/code>\nand <code>www.example.org<\/code>.<\/p>\n<p>Set those values in an environment variable. If you just have one:<\/p>\n<pre><code>export subjectAltName=DNS:example.org\n<\/code><\/pre>\n<p>If you have more then one:<\/p>\n<pre><code>export subjectAltName=DNS:example.org,DNS:www.example.org\n<\/code><\/pre>\n<p>If you have a wild card domain:<\/p>\n<pre><code>export subjectAltName='DNS:*.example.org'\n<\/code><\/pre>\n<p>Next, create a key and a certificate signing request:<\/p>\n<pre><code>openssl req -config \/etc\/ssl\/openssl.cnf -new -nodes -addext &quot;$subjectAltName&quot; -out new.csr -keyout new.key\n<\/code><\/pre>\n<p>Again, you will be prompted for some values (country, state, etc) - be sure to\nchoose the same values you used with your certficiate authority! I honestly\ndon&rsquo;t understand why this is necessary (when I set different values I get an\nerror on the signing request step below). Maybe someone can add a comment to\nthis post explaining why these values have to match?<\/p>\n<p>Also, you must provide a common name for your certificate - you can choose the\nsame name as the altSubjectNames value you set above (but just one domain).<\/p>\n<p>Want to review what you created?<\/p>\n<pre><code>openssl req -config \/etc\/ssl\/openssl.cnf -in new.csr -text -noout \n<\/code><\/pre>\n<h2 id=\"sign-it\">Sign it!<\/h2>\n<p>At last the momenet we have been waiting for:<\/p>\n<pre><code>openssl ca -config \/etc\/ssl\/openssl.cnf -keyfile ca\/private\/cakey.pem -cert ca\/cacert.pem -out new.crt -outdir . -rand_serial -infiles new.csr\n<\/code><\/pre>\n<p>Now you have a new.crt and new.csr that you can install via your web browser,\nmail server, etc specification.<\/p>\n<h2 id=\"smoke-test\">Smoke Test<\/h2>\n<p>This command will confirm that the certificate is trusted by your certificate\nauthority.<\/p>\n<pre><code>openssl verify -config \/etc\/ssl\/openssl.cnf -no-CApath -CAfile ca\/cacert.pem new.crt \n<\/code><\/pre>\n<h2 id=\"but-wait-theres-still-a-question-of-trust\">But wait, there&rsquo;s still a question of trust<\/h2>\n<p>You probably want to tell your computer or browser that you want to trust your\ncertificate signing authority.<\/p>\n<h3 id=\"command-line-tools\">Command line tools<\/h3>\n<p>Most tools in linux by default will trust all the certificates in\n<code>\/etc\/ssl\/certs\/ca-certificates.crt<\/code>. (If that file doesn&rsquo;t exist, try\ninstalling the <code>ca-certificates<\/code> package). If you want to add your certificate\nto that file:<\/p>\n<pre><code>cp cacert.pem \/usr\/local\/share\/ca-certificates\/cacert.crt\nsudo dpkg-reconfigure ca-certificates\n<\/code><\/pre>\n<p>Want to know what&rsquo;s funny? Ok, not really funny. If the certificate name ends\nwith <code>.pem<\/code> the command above won&rsquo;t work. Seriously.<\/p>\n<p>Once your certificate is installed with your web server you can now test to\nmake sure it&rsquo;s all working with:<\/p>\n<pre><code>gnutls-cli --print-cert $domainName\n<\/code><\/pre>\n<p>Want a second opinion?<\/p>\n<pre><code>curl https:\/\/$domainName\nwget https:\/\/$domainName -O-\n<\/code><\/pre>\n<p>Both will report errors if the certificate can&rsquo;t be verified by a system\ncertificate.<\/p>\n<p>If you really want to narrow down the cause of error (maybe reconfiguring\nca-certificates didn&rsquo;t work)?<\/p>\n<pre><code>curl --cacert \/path\/to\/your\/cacert.pem --capath \/tmp\n<\/code><\/pre>\n<p>Those arguments tell curl to use your certificate authority file and not to\nload any other certificate authority files (well, unless you have some\ninstalled in the temp directory).<\/p>\n<h3 id=\"web-browsers\">Web browsers<\/h3>\n<p>Firefox and Chrome have their own store of trusted certificates - you&rsquo;ll have\nto import your cacert.pem file into each browser that you want to trust your\nkey.<\/p>\n<h2 id=\"renewing\">Renewing<\/h2>\n<p>In the first step, you created a certificate signing authority key with an\nexpiration of 1,024 days.<\/p>\n<p>With luck, you&rsquo;ll still be using it after 3 years which means you&rsquo;ll need to\nrenew it.<\/p>\n<p>Start by changing into the <code>ca<\/code> directory.<\/p>\n<p>Then, create a new certificate signing request:<\/p>\n<pre><code>openssl x509 -x509toreq -in ca\/cacert.pem -signkey ca\/private\/cakey.pem -out new-server.csr\n<\/code><\/pre>\n<p>That command creates the file <code>new-server.csr<\/code> - a certificate signing request.<\/p>\n<p>Now, simply sign it:<\/p>\n<pre><code>openssl x509 -req -days 1024 -in new-server.csr -signkey private\/cakey.pem -out new-cacert.pem\n<\/code><\/pre>\n<p>This command generates your brand new <code>cacert.pem<\/code> file, but with the new name <code>new-cacert.pem<\/code>.<\/p>\n<p>Now, you simply use the new file to replace your old <code>cacert.pem<\/code> file.<\/p>\n"},{"title":"LVM Cache woops","link":"https:\/\/current.workingdirectory.net\/posts\/2020\/lvm-cache\/","pubDate":"Thu, 23 Apr 2020 13:51:25 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2020\/lvm-cache\/","description":"<p>At <a href=\"https:\/\/mayfirst.coo\/\">May First<\/a>, disk i\/o has been our most serious bottle neck for many years. We have plenty of RAM, disk space and even CPU.<\/p>\n<p>But when too much data is being written to our spinning disks everyting grinds to a halt.<\/p>\n<p>As we have been adding SSD disks to our servers, we&rsquo;ve recently begun experimenting with adding SSD-backed lvm caches. This approach has had a tremendous impact - resolving most of our disk i\/o problems.<\/p>\n<p>However, this morning we rebooted one of those virtual guests and I almost had a heart attack:<\/p>\n<pre><code>0 claudette:~# mount \/home\nmount: special device \/dev\/mapper\/vg_claudette0-home does not exist\n32 claudette:~# lvs vg_claudette0\/home\n  LV   VG            Attr       LSize   Pool             Origin       Data%  Meta%  Move Log Cpy%Sync Convert\n  home vg_claudette0 Cwi---C--- 309.00g [home_cachepool] [home_corig]                                        \n0 claudette:~# ls \/dev\/mapper\/\ncontrol             vg_claudette0-swap_1  vg_claudette0-var\nvg_claudette0-root  vg_claudette0-tmp     vg_claudette0-var+lib+mysql\n0 claudette:~# \n<\/code><\/pre>\n<p>Wah! lvm ate our data!<\/p>\n<p>Let&rsquo;s remove the cache and return to the way it was:<\/p>\n<pre><code>0 claudette:~# lvconvert --uncache vg_claudette0\/home\n  \/usr\/sbin\/cache_check: execvp failed: No such file or directory\n  Check of pool vg_claudette0\/home_cachepool failed (status:2). Manual repair required!\n  Failed to active cache locally vg_claudette0\/home.\n5 claudette:~#\n<\/code><\/pre>\n<p>Wah! That doesn&rsquo;t work either!<\/p>\n<p>Let&rsquo;s repair:<\/p>\n<pre><code>0 claudette:~# lvconvert --repair vg_claudette0\/home_cachepool\n  Using default stripesize 64.00 KiB.\n  Operation not permitted on cache pool LV vg_claudette0\/home_cachepool.\n  Operations permitted on a cache pool LV are:\n  --splitcache    (operates on cache LV)\n\n5 claudette:~#\n<\/code><\/pre>\n<p>What is happening!!<\/p>\n<p>I booted into a live rescue disk with a more modern version of lvm that really should support the <code>--repair<\/code> option:<\/p>\n<pre><code>0 debirf-rescue:~# lvconvert --repair vg_claudette0\/home_cachepool\n  \/dev\/vg_claudette0\/lvol1: not found: device not cleared\n  Aborting. Failed to wipe start of new LV.\n  WARNING: If everything works, remove vg_claudette0\/home_cachepool_meta0 volume.\n  WARNING: Use pvmove command to move vg_claudette0\/home_cachepool_cmeta on the best fitting PV.\n0 debirf-rescue:~#\n<\/code><\/pre>\n<p>Help! Help!<\/p>\n<p>Wait&hellip; thanks to a cool headed colleauge, it turns out the the <em>only<\/em> problem\nwas that <code>thin-provisioning-tools<\/code> was not installed on the host.<\/p>\n<p>Feel free to <a href=\"https:\/\/support.mayfirst.org\/ticket\/15613\">review the whole fiasco as it unfolded<\/a>.<\/p>\n"},{"title":"Free software in the age of Corona Virus","link":"https:\/\/current.workingdirectory.net\/posts\/2020\/corona-software\/","pubDate":"Fri, 27 Mar 2020 15:51:00 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2020\/corona-software\/","description":"<p>There are the printed remarks I made during the <a href=\"https:\/\/mayfirst.coop\/\">May\nFirst<\/a> webinar on free software during the Corona Virus\npandemic, which can be <a href=\"https:\/\/mayfirst.coop\/en\/audio\/free-software-during-corona-virus-pandemic\/\">heard via the\nrecording<\/a>.<\/p>\n<p>Hi everyone, so glad to be here with all of you during these unprecedented times.<\/p>\n<p>I&rsquo;d like to first make clear that I&rsquo;m proud of the tools and services that May\nFirst and our movement in general <a href=\"https:\/\/mayfirst.coop\/en\/post\/2020\/content-alternatives-live-meetings\/\">has made\navailable<\/a>\nto everyone seeking a way to continue our important organizing work online. And\neveryone on this call is a testament to our movement&rsquo;s ability to choose to\nbuild online meeting software that respects our privacy and invests in movement\nowned technology.<\/p>\n<p>However, I&rsquo;d like to address something we can&rsquo;t avoid: capitalism has created\nonline, full video Internet conferences with as many people as you want where\nyou can see everyone&rsquo;s faces and share any screen.<\/p>\n<p>Our movement has not figured out how to provide that service in a way that\nprotects our privacy and ensures such a vital tool remains in the control of\nthe movement and not in the hands of the corporate world we are fighting so\nhard to stop.<\/p>\n<p>As a result of this shortcoming, our movement is, as we speak, investing\nheavily - both in terms of money and in terms of learning new tools - to ensure\nthat these corporate solutions become the default and entrenched methods for\nmeeting online.<\/p>\n<p>I&rsquo;d like to address a couple questions: How did we get here? And what can we do\nabout it?<\/p>\n<p>We could fill an entire conference on the techno\/political reasons why video on\nthe Internet is hard, so I&rsquo;ll just skim the surface: The success of the\nInternet is largely due to it&rsquo;s open nature: there is a standard for how\ninformation can be shared, and anyone who builds a program that uses that\nstandard can inter-operate with everyone else.<\/p>\n<p>From the birth of the corporate Internet, capitalists have been trying to\ncontrol the Internet in a way that ensures they will profit the most from it.\nAnd video, which had no standards defined for it when things really took off\nback in 1994, has been one of the biggest prizes. As a result, the last 20\nyears have witnessed battles over what standards will govern how video is\ntransmitted over the Internet and, despite some recent victories, the ones\npromoting free and open tools have largely lost (those of you old enough to\nremember that abomination called &ldquo;flash&rdquo; will know what I&rsquo;m talking about).<\/p>\n<p>The result is a fractured Internet when it comes to video. While things have\nbeen recently changing, for most of the last 20 years it has been impossible to\ncome up with a way to transmit video that will work on all computers and all\ncell phones. As a result, only organizations with a lot of capital have been\nable to afford the engineering teams needed to write the software for all the\ndevices.<\/p>\n<p>However there&rsquo;s also a second reason, one that might be even more important:\nthe movement has not prioritized it. Why? We certainly know how to prioritize\nInternet development.<\/p>\n<p>In the late 1980&rsquo;s the left wing Internet organization IGC organized a\nleft-wing forum system for earlier Internet users.  The Zapatistas prioritized\nthe use of the Internet in the mid 1990&rsquo;s before anyone had figured it out. The\nGlobal Justice movement build the global Indymedia Network when nobody knew how\nto publish our own news. In the early 2000&rsquo;s CiviCRM created a database system\nthat the movement needed and PTP has further developed it for the Movement as\nPowerbase.<\/p>\n<p>However, when it comes to video conferencing we seem to have ceded this work to\nthe corporatate world.<\/p>\n<p>The next question is: how do we turn this around? That&rsquo;s for us to figure out\ntogether, but I&rsquo;ll start with some ideas:<\/p>\n<ol>\n<li>\n<p>Let&rsquo;s design campaigns that demonstrate to the movement why this is\nimportant. <a href=\"https:\/\/notechforice.com\/\">No Tech for Ice<\/a> is a great example -\nit shows how big tech has a real and direct impact on our issues. Same with\nanti-surveillance campaigns.<\/p>\n<\/li>\n<li>\n<p>Let&rsquo;s remember our values: it&rsquo;s not just about video, it&rsquo;s about ensuring\nthat everyone has access, even in low bandwidth environments, where people\nwho can&rsquo;t make an international phone call can still connect, where people\ncan speak the language they want and get interpreted in real time.<\/p>\n<\/li>\n<li>\n<p>Let&rsquo;s use our imaginations to change our culture: we can&rsquo;t just move an &ldquo;in\nperson&rdquo; meeting to an online format. In fact, maybe our in person formats\nweren&rsquo;t working so well anyway! How do we build a movement culture that\nsupports unifying our disperate issue focused groups into a powerful force\nfor systemic change? And now, how does the Internet support that agenda?<\/p>\n<\/li>\n<\/ol>\n"},{"title":"Programming with 6 year olds","link":"https:\/\/current.workingdirectory.net\/posts\/2020\/programming\/","pubDate":"Fri, 27 Mar 2020 15:45:31 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2020\/programming\/","description":"<p>I know there is a pedagogically appropriate way to teach 6 year olds how to program and this is not it.<\/p>\n<p>But, it was still fun. And maybe we programmers could use the bark sound more often?<\/p>\n<pre><code>#!\/usr\/bin\/python3\n\nimport random\nimport time\nimport subprocess\n\nwhile 1:\n    answers = [\n            &quot;What is your question?&quot;,\n            &quot;There is no answer.&quot;,\n            &quot;53&quot;,\n            &quot;The answer to your question lies within your ear.&quot;,\n            &quot;Blue berries!&quot;,\n            &quot;The grasshopper is 20 feet tall.&quot;,\n            &quot;Elephants are the size of bacteria.&quot;,\n            &quot;True&quot;,\n            &quot;False&quot;,\n            &quot;Yes, no, well, maybe.&quot;,\n            &quot;I have no idea.&quot;,\n            &quot;I do not know WHAT you are talking about.&quot;,\n            &quot;That's rude.&quot;,\n            &quot;Ask your dad.&quot;,\n            &quot;I'm sick, please ask someone else.&quot;,\n            &quot;Maybe you should ask yourself that question.&quot;,\n            &quot;Could you ask a different question?&quot;,\n            &quot;You could probably find out by asking the Internet.&quot;,\n            &quot;Is that question really so important to you?&quot;,\n            &quot;Can I have a turn asking a question?&quot;,\n            &quot;How come you get to ask all the questions?&quot;,\n            &quot;Absolutely not.&quot;,\n            &quot;You are asking so many questions and haven't even introduced yourself to me.&quot;,\n            &quot;What other question do you have?&quot;,\n            &quot;What are you talking about?&quot;,\n            &quot;I couldn't possibly answer that question.&quot;,\n            &quot;Can we play a different game?&quot;,\n            &quot;That makes my brain hurt.&quot;,\n            &quot;Why are you asking me a question you know the answer to?&quot;,\n            &quot;Ha ha ha ha ha ha ha ha.&quot;,\n            &quot;Boo boo butt&quot;,\n            &quot;Really, ask someone else, I am sick!&quot;,\n            &quot;Computer, stop talking - you are interrupting me&quot;,\n            &quot;You are really bad at typing&quot;,\n            &quot;Why do you keep asking me questions?&quot;,\n            &quot;Ask someone else already.&quot;,\n            &quot;The answer is.&quot;,\n            &quot;You have a bug in your programming.&quot;\n    ]\n    subprocess.call([&quot;paplay&quot;, &quot;\/usr\/share\/sounds\/gnome\/default\/alerts\/bark.ogg&quot;])\n    question = &quot;What is your question? &quot;\n    subprocess.call([&quot;espeak&quot;, question])\n    user_question = input(question)\n    subprocess.call([&quot;espeak&quot;, &quot;Your question is ... &quot; + user_question])\n    subprocess.call([&quot;paplay&quot;, &quot;\/usr\/share\/sounds\/gnome\/default\/alerts\/bark.ogg&quot;])\n\n    answer = random.choice(answers)\n\n    print(&quot;And the answer is...&quot;)\n    subprocess.call([&quot;paplay&quot;, &quot;\/usr\/share\/sounds\/gnome\/default\/alerts\/bark.ogg&quot;])\n    subprocess.call([&quot;espeak&quot;, &quot;And the answer is&quot;])\n    subprocess.call([&quot;paplay&quot;, &quot;\/usr\/share\/sounds\/gnome\/default\/alerts\/bark.ogg&quot;])\n\n    time.sleep(2)\n\n    subprocess.call([&quot;paplay&quot;, &quot;\/usr\/share\/sounds\/gnome\/default\/alerts\/bark.ogg&quot;])\n    print(&quot;... the answer is....&quot;)\n    subprocess.call([&quot;paplay&quot;, &quot;\/usr\/share\/sounds\/gnome\/default\/alerts\/bark.ogg&quot;])\n    subprocess.call([&quot;espeak&quot;, &quot;the answer is&quot;])\n\n    time.sleep(2)\n\n    subprocess.call([&quot;paplay&quot;, &quot;\/usr\/share\/sounds\/gnome\/default\/alerts\/bark.ogg&quot;])\n    subprocess.call([&quot;paplay&quot;, &quot;\/usr\/share\/sounds\/gnome\/default\/alerts\/bark.ogg&quot;])\n    subprocess.call([&quot;paplay&quot;, &quot;\/usr\/share\/sounds\/gnome\/default\/alerts\/bark.ogg&quot;])\n    subprocess.call([&quot;paplay&quot;, &quot;\/usr\/share\/sounds\/gnome\/default\/alerts\/bark.ogg&quot;])\n    print(answer)\n    subprocess.call([&quot;espeak&quot;, answer])\n    subprocess.call([&quot;paplay&quot;, &quot;\/usr\/share\/sounds\/gnome\/default\/alerts\/bark.ogg&quot;])\n<\/code><\/pre>\n"},{"title":"Editing video without a GUI? Really?","link":"https:\/\/current.workingdirectory.net\/posts\/2019\/melt\/","pubDate":"Tue, 08 Oct 2019 09:21:39 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2019\/melt\/","description":"<p>It seems counter intuitive - if ever there was a program in need of a graphical\nuser interface, it&rsquo;s a non-linear video editing program.<\/p>\n<p>However, as part of the <a href=\"https:\/\/mayfirst.coop\/\">May First<\/a> board elections, I\ndiscovered otherwise.<\/p>\n<p>We asked each board candidate to submit a 1 - 2 minute video introduction about\nwhy they want to be on the board.  My job was to connect them all into a single\nvideo.<\/p>\n<p>I had an unrealistic thought that I could find some simple tool that could\nconcatenate them all together (like <code>mkvmerge<\/code>) but I soon realized that this\napproach requires that everyone use the exact same format, codec, bit rate,\nsample rate and blah blah blah.<\/p>\n<p>I soon realized that I needed to actually <em>make<\/em> a video, not compile one. I\ncreate videos so infrequently, that I often forget the name of the video\nediting software I used last time so it takes some searching. This time I found\nthat I had <a href=\"https:\/\/tracker.debian.org\/pkg\/openshot-qt\">openshot-qt<\/a> installed\nbut when I tried to run it, I got a back trace (which <a href=\"https:\/\/bugs.debian.org\/cgi-bin\/bugreport.cgi?bug=940839\">someone else has already\nreported<\/a>).<\/p>\n<p>I considered looking for another GUI editor, but I wasn&rsquo;t that interested in\nlearning what might be a complicated user interface when what I need is so\nsimple.<\/p>\n<p>So I kept searching and found <a href=\"https:\/\/tracker.debian.org\/pkg\/mlt\">melt<\/a>. Wow.<\/p>\n<p>I ran:<\/p>\n<pre><code>melt originals\/* -consumer avformat:all.webm acodec=libopus vcodec=libvpx\n<\/code><\/pre>\n<p>And a while later I had a video. Impressive. It handled people who submitted\ntheir videos in portrait mode on their cell phones in mp4 as well as web cam\nsubmissions using webm\/vp9 on landscape mode.<\/p>\n<p>Thank you melt developers!<\/p>\n"},{"title":"Welcome to the Lazy Bookkeeper","link":"https:\/\/current.workingdirectory.net\/posts\/2019\/lazy-bookkeeper\/","pubDate":"Thu, 04 Apr 2019 12:18:47 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2019\/lazy-bookkeeper\/","description":"<p>Welcome to the first (and given my laziness, possibly last) installment of the\nlazy bookkeeper. I&rsquo;m actually not really a bookkeeper at all, so if you are\nlooking for a legal and sound way to keep your books, don&rsquo;t read this post.<\/p>\n<p>I am lazy though.<\/p>\n<p>I&rsquo;m tasked with keeping the books on a US organization with a budget of about\n$150,000. I hate doing bookkeeping so I&rsquo;ve naturally tried to automate\neverything I possibly can. I&rsquo;ll explain how I do it in this blog.<\/p>\n<p>The first two (and most important) steps are to use\n<a href=\"https:\/\/gnucash.org\/\">GnuCash<\/a> and the <a href=\"https:\/\/wiki.gnucash.org\/wiki\/Python_Bindings\">GnuCash Python\nbindings<\/a>. Last I checked about\na year ago, the Python bindings were not working with the Python3 version of\nGnuCash so you may need to hold back on that version.<\/p>\n<h2 id=\"income\">Income<\/h2>\n<p>Income is easier then expenses, perhaps because we have so little of it :).<\/p>\n<p>The only source of income for the organization are:<\/p>\n<ul>\n<li>Membership dues (accounting for almost everything)<\/li>\n<li>Donations<\/li>\n<li>Program related income (e.g. contracts)<\/li>\n<li>Interest (like $10 every year)<\/li>\n<\/ul>\n<p>We get income via credit card payments (both Stripe and Paypal) and checks.<\/p>\n<p>We have two external systems that handle Membership dues (our custom control\npanel software) and Donations (CiviCRM).<\/p>\n<p>The custom control panel software automatically generates new membership\ninvoices and sends reminders to people to pay them.<\/p>\n<p>When I get checks, I:<\/p>\n<ul>\n<li>Take a picture of the check with my phone<\/li>\n<li>Copy it to a Nextcloud account in a folder called: Checks to be processed<\/li>\n<li>Enter the checks manually in either our control panel or CiviCRM databases\nso the sender can be properly credited (these are our single sources of\ntruth for such things)<\/li>\n<li>Deposit the checks and take a picture of the receipt (and share it with the\nsame next Nextcloud folder as the checks)<\/li>\n<\/ul>\n<p>At this point in the process, I had to struggle against my lazy instincts to\ntake an additional step to solve two problems:<\/p>\n<ol>\n<li>The checks are named based on date\/time I took them picture. Not that\nhelpful if I&rsquo;m looking at a list of them<\/li>\n<li>Entering the checks manually is a source of typos and mistakes - which is a\nsubstantial cause of misery when bookkeeping.<\/li>\n<\/ol>\n<p>So, I&rsquo;ve written a simple script that reads all the pictures of all the checks\nand then displays them one by one.<\/p>\n<p>For each check that is displayed, I am prompted to enter the check number. The\nscript then queries the control panel for that check number and displays the\nmember it was coded against and the amount. I check these details with the\ncheck itself and confirm it.<\/p>\n<p>Then, the script moves the check from the &ldquo;checks to be processed&rdquo; folder to a\nnew folder (in a YYYY\/MM format) and renames it to include the check number,\nshort version of the organization it is applied against and the amount in the\nname itself.<\/p>\n<p>The last step is that every few weeks I run a script to import data from both\nthe control panel and the CiviCRM database.<\/p>\n<p>The script finds control panel invoices and payments and donations in CiviCRM\nand automatically creates the records in GnuCash.<\/p>\n<h2 id=\"expenses\">Expenses<\/h2>\n<p>Expenses have been a much harder nut to crack. For many years I simply put all\nreceipts in a giant folder and then, after waiting until the last possible\nmoment, I would go through them all an enter them. I made a Python script to\nfacilitate it - it asks how much the expense is for, what category it should be\ncoded to, etc. However, this task was particularly tedious because I usually\ncouldn&rsquo;t remember what any expense was for.<\/p>\n<p>So, this year I&rsquo;ve instituted a new system. I have a set of folders named after\neach expense account. Now, when I get a receipt, I name it in the format:\ninvoice-number_brief-description_YYYY-MM-DD_00.00_bank.pdf and I save it in the\nfolder matching the expense account.<\/p>\n<p>In short, I am doing the sorting every time a receipt comes in rather than\nwaiting until the end of the year.<\/p>\n<p>Then, I have a script that validates all of my names (in case I made any typos)\nand automatically imports everything in to GnuCash.<\/p>\n<p>The only other expense to deal with is payroll, so I&rsquo;ve naturally scripted that\nas well. I do have to run it for every payroll and enter the variations in\nrandom state taxes and such, but by using a terminal script I don&rsquo;t have to\nclick around the GnuCash interface.<\/p>\n<h1 id=\"reconciliation\">Reconciliation<\/h1>\n<p>The only part I can&rsquo;t automate is the reconciliation. However, by minimizing\ntypos it is much easier. Basically consisting of:<\/p>\n<ol>\n<li>Entering interest and bank fees<\/li>\n<li>Tracking down expenses I didn&rsquo;t get receipts for<\/li>\n<\/ol>\n<h1 id=\"the-scripts\">The scripts<\/h1>\n<p>These scripts can&rsquo;t be directly used by anyone else and will be useless if you\nthink of them as general purpose scripts. However, if you want to do something\nsimilar for your own purposes, you may find them helpful as a way to see how\ncertain items can be imported into GnuCash.<\/p>\n<p><a href=\"https:\/\/code.mayfirst.org\/mfmt\/gnucash-import-scripts\">https:\/\/code.mayfirst.org\/mfmt\/gnucash-import-scripts<\/a><\/p>\n"},{"title":"I didn't know what ibus was one day ago, now I love it","link":"https:\/\/current.workingdirectory.net\/posts\/2019\/ibuswtf\/","pubDate":"Mon, 11 Feb 2019 15:50:56 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2019\/ibuswtf\/","description":"<p>[See update below.]<\/p>\n<p>After over a decade using mutt as my email client, I finally gave up pretending\nI didn&rsquo;t want to see pretty pictures in my email and switched to Thunderbird.<\/p>\n<p>Since I don&rsquo;t write email in spanish often, I didn&rsquo;t immediately notice that my\nold <a href=\"https:\/\/en.wikipedia.org\/wiki\/Dead_key\">dead key<\/a> trick for typing spanish\naccent characters didn&rsquo;t work in Thunderbird like it does in vim or any\nterminal program.<\/p>\n<p>I learned many years ago that I could set a special key via my openbox\nautostart script with <code>xmodmap -e 'keysym Alt_R = Multi_key'<\/code>. Then, if I\nwanted to type \u00e9, I would press and hold my right alt key while I press the\napostrophe key, let go, and press the e key. I could get an \u00f1 using the same\ntrick but press the tilde key instead of the apostrophe key. Pretty easy.<\/p>\n<p>When I tried that trick in Thunderbird I got an upside down e. WTF.<\/p>\n<p>I spent about 30 minutes clawing my way through search results on several\noccassions over the course of many months before I finally found someone say:\n&ldquo;I installed the ibus package, rebooted and it all worked.&rdquo; (Sorry Internet, I\ncan&rsquo;t find that page now!)<\/p>\n<p>ibus? <code>apt-cache show ibus<\/code> states:<\/p>\n<pre><code>IBus is an Intelligent Input Bus. It is a new input framework for the Linux\nOS. It provides full featured and user friendly input method user\ninterface. It also may help developers to develop input method easily.\n<\/code><\/pre>\n<p>Well, that&rsquo;s succinct. I still had no idea what ibus was, but it sounded like\nit might work. I followed those directions and suddenly in my system tray area,\nthere was a new icon. If I clicked on it, it listed by default my English\nkeyboard.<\/p>\n<p>I could right click, hit preferences and add a new keyboard:<\/p>\n<p>English - English (US, International with dead keys)<\/p>\n<p>Now, when I select that new option, I simply press my right alt key and e (no\nneed for the apostrophe) and I get my \u00e9. Same with \u00f1. Hooray!<\/p>\n<p>My only complaint is that while using this keyboard, I can&rsquo;t using regular\napostrophes or ~&rsquo;s. Not sure why, but it&rsquo;s not that hard to switch.<\/p>\n<p>As far as I can tell,\n<a href=\"https:\/\/en.wikipedia.org\/wiki\/Intelligent_Input_Bus\">ibus<\/a> tries to abstract\nsome of the difficulties around input methods so it&rsquo;s easier on GUI developers.<\/p>\n<p><strong>Update 2019-02-11<\/strong><\/p>\n<p>Thanks, Internet, particularly for the comment from Alex about how I was\nchoosing the wrong International keyboard. Of course, my keyboard does not have\ndead keys, so I need to choose the one called &ldquo;English (intl., with AltGr dead\nkeys).&rdquo;<\/p>\n<p>Now everything works perfectly. No need for ibus at all. I can get \u00e9 with my\nright alt key followed by e. It works in my unicode terminal, thunderbird, and\neverywhere else that I&rsquo;ve tried.<\/p>\n"},{"title":"Identifying resource hogs","link":"https:\/\/current.workingdirectory.net\/posts\/2018\/resource-hogs\/","pubDate":"Wed, 24 Oct 2018 11:22:33 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2018\/resource-hogs\/","description":"<p>Suppose you manage a multi-user server with wily and unpredictable users\nrunning compromisable web sites and email accounts with loads of email.<\/p>\n<p>Suppose your server blows up at 3:03 pm.<\/p>\n<p>Suppose you ask: Was there a single compromised user account that caused this\nmess and if so, which user?<\/p>\n<p>How would you figure it out?<\/p>\n<h2 id=\"sysstat\">sysstat<\/h2>\n<p>If you are lucky enough to be logged into the server when it is blowing up and\nlucky enough that the server is responsive enough to run commands for you, then\nyou have some options.<\/p>\n<p><code>vmstat 1<\/code> nicely shows you whether you are cpu bound or disk i\/o bound. The\nfirst two columns show number of processes waiting to run due to either not\nenough CPU (r) or not enough disk i\/o (b).<\/p>\n<p><code>pidstat -d 1<\/code> shows which processes are writing and reading to disk and when\nyou ctrl-c cancel that command it summarizes what you have been watching\nallowing you to more easily pin point who is doing the most writing.<\/p>\n<p>And good ole <code>top<\/code> can show you which users are consuming the most CPU.<\/p>\n<p>But, these tools are often misleading. When your system is under heavy\ncontention, all kinds of processes get backed up and these tools often just\nshow a mess of processes desperately trying to run amidst a giant resource\nshortage. It&rsquo;s hard to pinpoint the user that may have started the problem.<\/p>\n<p>Also, these tools are useless if you can&rsquo;t login to the server or if you arrive\nat the scene after the storm has passed.<\/p>\n<h2 id=\"munin-and-sar\">munin and sar<\/h2>\n<p>Both munin and sar (provided by sysstat) can record a history of usage. And,\nboth can tell you, for example, whether your system was CPU bound or disk i\/o\nbound and exactly when the problem started.<\/p>\n<p>However, I can&rsquo;t seem to convince either (out of the box at least) how to track\nsuch usage on a per user basis.<\/p>\n<h2 id=\"gnu-accounting-utilities\">Gnu Accounting utilities<\/h2>\n<p>Now we are getting somewhere. The <code>acct<\/code> package is specifically designed to\nrecord usage information on a per user basis.<\/p>\n<p>However, it suffers from a few problems considering our use case:<\/p>\n<ul>\n<li>\n<p>It has a subtley different goal: <code>acct<\/code> wants to account for total resource\nusage at the end of the day. I want to measure per user resource usage at an\nexact point and time.<\/p>\n<p>The <code>acct<\/code>package works in a elegant fashion. It enables a feature of the\nkernel that causes the kernel to add data to a file every time a process\nends. The data includes the pid, uid, total cpu usage and average memory\nconsumption and the date and time the process began.<\/p>\n<p>This approach means you don&rsquo;t have to poll running processes and you always\nget accurate information.<\/p>\n<p>For the purposes of pin-pointing who is consuming resources when, this works\ngreat for short running processes.<\/p>\n<p>But for long running processes, if you chart it by the date\/time provided\n(which is the time the process started), you get a giant jump in resource\nusage when the process starts. If a process runs for 30 minutes and consumes\nmassive resources during the last minute of it&rsquo;s life, that resource usage\nwill get reported 29 minutes before it happened.<\/p>\n<\/li>\n<li>\n<p>Confusing to munin. The date\/time on the data is when the process started,\nhowever, it is reported to the kernel file when the process ends. For the\npurposes of munin graphing, we would have to record it when the process ends\nor else we would be reporting times in the past.<\/p>\n<\/li>\n<li>\n<p>A much bigger problem is lack of disk i\/o. Although the spec seems to\ninclude disk i\/o, reporting disk i\/o does not seem to be available on linux\nand sadly disk i\/o is almost always the caues of our resource problems. The\n&ldquo;io&rdquo; column in dump-acct is always 0.<\/p>\n<\/li>\n<li>\n<p>The information is reported in binary form making the raw file a bit hard to\nread. And the tools that come with the <code>acct<\/code> package interpret that file\n(thanks!) but do so in a way that is hard to parse (in particular, dates are\nhuman readable and only include when the process began, not when it started,\nso you can&rsquo;t effecitvely limit output by date range).<\/p>\n<\/li>\n<\/ul>\n<h2 id=\"alternatives\">Alternatives<\/h2>\n<p>I&rsquo;ve spent years writing various polling programs that either use <code>pidstat<\/code> or\nit&rsquo;s ilk (or directly access pid statistics via \/proc) in cron jobs or\nconstantly running processes and collect and record it&rsquo;s output. However, all\nthese scripts suffer from either being inaccurate because they depend on\npolling running processes or overly resource consuming themselves because they\nare in a constant loop measuring things.<\/p>\n"},{"title":"Which is faster, rsync or rdiff-backup?","link":"https:\/\/current.workingdirectory.net\/posts\/2018\/rsyncvsrdiff\/","pubDate":"Mon, 13 Aug 2018 12:22:41 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2018\/rsyncvsrdiff\/","description":"<p>Surprise: rdiff-backup (given our particular constraints).<\/p>\n<p>As our data grows (and some filesystems balloon to over 800GBs, with many small\nfiles) we have started seeing our night time backups continue through the\nmorning, causing serious disk i\/o problems as our users wake up and regular\nusage rises.<\/p>\n<p>For years we have implemented a conservative backup policy - each server runs\nthe backup twice: once via rdiff-backup to the onsite server with 10 days of\nincrements kept. A second is an rsync to our offsite backup servers for\ndisaster recovery.<\/p>\n<p>Simple, I thought. I will change the rdiff-backup to the onsite server to use\nthe ultra fast and simple rsync. Then, I&rsquo;ll use borgbackup to create an\nincremental backup <em>from the onsite backup server<\/em> to our off site backup\nservers.  Piece of cake. And with each server only running one backup instead\nof two, they should complete in record time.<\/p>\n<p>Except, some how the rsync backup to the onsite backup server was taking almost\nas long as the original rdiff-backup to the onsite server and rsync backup to\nthe offsite server combined. What? I thought nothing was faster than the\nawesome simplicity of rsync, especially compared to the ancient python-based\nrdiff-backup, which hasn&rsquo;t had an upstream release since 2009.<\/p>\n<p>Turns out that rsync is not faster if disk i\/o on the target server is your\nbottle neck.<\/p>\n<p>By default, rsync determines if a file needs to be updated by comparing the\ntime stamp and size of the files on both the source and the target server. That\nmeans rsync has to read the meta data on every single file on the source <em>and<\/em>\nevery single file on the target. At first glance, this would seem faster than\nrdiff-backup, which compares sha1 checksums (it has to read the entire file,\nnot just the metadata). And, this is definitely the case the first time\nrdiff-backup runs. However, rdiff-backup has a trick up its sleave: the\nrdiff-backup-data\/mirror_metadata file.<\/p>\n<p>As rdiff-backup runs, it keeps track of the sha1 checksum of every file it\nbackups up in the mirror_medata file on the target. It seems that the next time\nit runs, it simply compares the sha1 on the source with the sha1 in this file,\nmeaning it doesn&rsquo;t have to read each file on the target. The result:\nsignificantly less disk i\/o on the target for faster backups (there is more\ndisk i\/o on the source, though, since rdiff-backup has to calculate the sha1\nchecksum instead of just collecting the size and last modified time stamp).<\/p>\n<p>rdiff-backup also wins by saving all metadata (file ownership and permissions).\nSince we backup to a non-privileged user on the backup server, this data is\nlost with rsync. And, for reasons of simplicity, I appreciate having the backup\nfiles via a plain filesystem (unlike borgbackup which requires special commands\njust to get a listing of the files).<\/p>\n<p>For the long term, filesystem-based backup tools seem like a losing proposition\ncompared with block-based backups (like\n<a href=\"https:\/\/en.wikipedia.org\/wiki\/Distributed_Replicated_Block_Device\">drbd<\/a>).\nHowever, until we can re-organize our data to take advantage of drdb, we will\nbe sticking with rdiff-backup.<\/p>\n"},{"title":"Diversity doesn't help the bottom line","link":"https:\/\/current.workingdirectory.net\/posts\/2017\/diversity\/","pubDate":"Mon, 30 Apr 2018 09:54:09 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2017\/diversity\/","description":"<p>A Google software engineer&rsquo;s <a href=\"http:\/\/gizmodo.com\/exclusive-heres-the-full-10-page-anti-diversity-screed-1797564320\">sexist screed against diversity<\/a> has been making the rounds lately.<\/p>\n<p>Most notable are the offensive and mis-guided statements about gender\nessentialism, which honestly make the thing hard to read at all.<\/p>\n<p>What seems lost in the hype, however, is that his primary point seems quite\naccurate. In short: If Google successfully diversified it&rsquo;s workforce, racial\nand gender tensions would <em>increase<\/em> not decrease,  divisiveness would spread\nand, with all liklihood, Google could be damaged.<\/p>\n<p>Imagine what would happen if the thousands of existing, mostly male, white and\nAsian engineers, the majority of whom are convinced that they play no part in\nracism and sexism, were confronted with thousands of smart and ambitious women,\nAfrican Americans and Latinos who were becoming their bosses, telling them to\nwork in different ways, and taking &ldquo;their&rdquo; promotions.<\/p>\n<p>It would be a revolution! I&rsquo;d love to see it. Google&rsquo;s bosses definitely do\nnot.<\/p>\n<p>That&rsquo;s why none of the diversity programs at Google or any other major tech\ncompany are having any impact - because they are not designed to have an\nimpact. They are designed to boost morale and make their existing engineers\nfeel good about what they do.<\/p>\n<p>Google has one goal: to make money. And one strategy: to design software\nthat people want to use. One of their tactics that is highly effective is\nbuilding tightly knit groups of programmers who work well together. If the\ncreation of hostile, racist and sexist environments is a by-product - well,\nit&rsquo;s not one that affects their bottom line.<\/p>\n<p>Would Google make better software with a more diverse group of engineers?\nDefinitely! For one, if African American engineers were working on their facial\nrecognition software, it&rsquo;s doubtful <a href=\"https:\/\/www.usatoday.com\/story\/tech\/2015\/07\/01\/google-apologizes-after-photos-identify-black-people-as-gorillas\/29567465\/\">it would have mistaken people with black\nfaces for gorillas<\/a>.<\/p>\n<p>However, if the perceived improvement in software outweighed the risks of\ndiversification, then Google would not waste any time on feel-good programs and\ntrainings - they would simply build a jobs pipeline and change their job\noutreach programs to recruit substantially more female, African Americans and\nLatino candidates.<\/p>\n<p>In the end, this risk avoidance and failure to perceive the limitations of\nhomogeneity is the achiles heel of corporate software design.<\/p>\n<p>Our challenge is to see what we can build outside the confines of corporate\nculture that prioritizes profits, production efficiency, and stability. What\ncan we do with teams that are willing to embrace racial and gender tension,\nrisk diviseveness and be willing to see benefits beyond releasing version 1.0?<\/p>\n"},{"title":"Procrastinating by tweaking my desktop with devilspie2","link":"https:\/\/current.workingdirectory.net\/posts\/2018\/devilspie2\/","pubDate":"Mon, 30 Apr 2018 09:48:05 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2018\/devilspie2\/","description":"<p>Tweaking my desktop seems to be my preferred form of procrastination. So, a blog like this is a sure sign I have too much work on my plate.<\/p>\n<p>I have a laptop. I carry it to work and plug it into a large monitor - where I like to keep all my instant or near-instant communications displayed at all times while I switch between workspaces on my smaller laptop screen as I move from email (workspace one), to shell (workspace two), to web (workspace three), etc.<\/p>\n<p>When I&rsquo;m not at the office, I only have my laptop screen - which has to accomdate everything.<\/p>\n<p>I soon got tired of dragging things around everytime I plugged or unplugged the monitor and starting accumulating a mess of bash scripts running <a href=\"https:\/\/packages.debian.org\/wmctrl\">wmctrl<\/a> and even calling my own <a href=\"https:\/\/packages.debian.org\/python-wnck\">python-wnck<\/a> script. (At first I couldn&rsquo;t get <code>wmctrl<\/code> to pin a window but I lived with it. But when <a href=\"https:\/\/packages.debian.org\/gajim\">gajim<\/a> switched to gtk3 and my openbox window decorations disappeared, then I couldn&rsquo;t even pin my window manually. NOTE: This behavior has changed - I now have my openbox decorations back on my gajim windows.)<\/p>\n<p>Now I have the following simpler setup.<\/p>\n<h2 id=\"manage-hot-plugging-of-my-monitor\">Manage hot plugging of my monitor.<\/h2>\n<p>Symlink to my monitor status device:<\/p>\n<div class=\"highlight\"><pre tabindex=\"0\" class=\"chroma\"><code class=\"language-fallback\" data-lang=\"fallback\"><span class=\"line\"><span class=\"cl\">0 jamie@turkey:~$ ls -l ~\/.config\/turkey\/monitor.status \n<\/span><\/span><span class=\"line\"><span class=\"cl\">lrwxrwxrwx 1 jamie jamie 64 Jan 15 15:26 \/home\/jamie\/.config\/turkey\/monitor.status -&gt; \/sys\/devices\/pci0000:00\/0000:00:02.0\/drm\/card0\/card0-DP-1\/status\n<\/span><\/span><span class=\"line\"><span class=\"cl\">0 jamie@turkey:~$ \n<\/span><\/span><\/code><\/pre><\/div><p>Create a udev rule to handle things when the monitor is unplugged (for some reason, automating the detection of the monitor being plugged in was too unreliable).<\/p>\n<div class=\"highlight\"><pre tabindex=\"0\" class=\"chroma\"><code class=\"language-fallback\" data-lang=\"fallback\"><span class=\"line\"><span class=\"cl\">0 jamie@turkey:~$ cat \/etc\/udev\/rules.d\/90-vga.rules \n<\/span><\/span><span class=\"line\"><span class=\"cl\"># When a monitor is plugged in, adjust my display to take advantage of it\n<\/span><\/span><span class=\"line\"><span class=\"cl\">ACTION==&#34;change&#34;, SUBSYSTEM==&#34;drm&#34;, ENV{HOTPLUG}==&#34;1&#34;, RUN+=&#34;\/etc\/udev\/scripts\/vga-adjust&#34;\n<\/span><\/span><span class=\"line\"><span class=\"cl\">0 jamie@turkey:~$ \n<\/span><\/span><\/code><\/pre><\/div><p>And here is the udev script:<\/p>\n<div class=\"highlight\"><pre tabindex=\"0\" class=\"chroma\"><code class=\"language-bash\" data-lang=\"bash\"><span class=\"line\"><span class=\"cl\"><span class=\"m\">0<\/span> jamie@turkey:~$ cat \/etc\/udev\/scripts\/vga-adjust \n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"c1\">#!\/bin\/bash<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">\n<\/span><\/span><span class=\"line\"><span class=\"cl\">logger -t <span class=\"s2\">&#34;jamie-udev&#34;<\/span> <span class=\"s2\">&#34;Monitor event detected, waiting 1 second for \\\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"s2\">system to detect change.&#34;<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">sleep <span class=\"m\">1<\/span> \n<\/span><\/span><span class=\"line\"><span class=\"cl\">\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"c1\"># This process seems to be unreliable when alerting the X11 system, so I&#39;m<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"c1\"># only configuring it to tell the X windows system when the monitor is unplugged.<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"c1\">#<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"c1\"># The X11 system adjusts for a monitor being plugged in via the systemd service<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"c1\"># devilspie2 (~\/.config\/systemd\/user\/devilspie2.service) which executes a-vga-auto<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"c1\"># via a PreExec command.<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"c1\">#<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"c1\"># We don&#39;t know whether the VGA monitor is being plugged in or unplugged so we<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"c1\"># have to autodetect first. And,it takes a few seconds to assess whether the<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"c1\"># monitor is there or not, so sleep for 1 second.<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"nv\">monitor_status<\/span><span class=\"o\">=<\/span><span class=\"s2\">&#34;\/home\/jamie\/.config\/turkey\/monitor.status&#34;<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"nv\">status<\/span><span class=\"o\">=<\/span><span class=\"k\">$(<\/span>cat <span class=\"s2\">&#34;<\/span><span class=\"nv\">$monitor_status<\/span><span class=\"s2\">&#34;<\/span><span class=\"k\">)<\/span>  \n<\/span><\/span><span class=\"line\"><span class=\"cl\">\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"nb\">export<\/span> <span class=\"nv\">XAUTHORITY<\/span><span class=\"o\">=<\/span>\/home\/jamie\/.Xauthority\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"nb\">export<\/span> <span class=\"nv\">DISPLAY<\/span><span class=\"o\">=<\/span>:0\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"k\">if<\/span> <span class=\"o\">[<\/span> <span class=\"s2\">&#34;<\/span><span class=\"nv\">$status<\/span><span class=\"s2\">&#34;<\/span> <span class=\"o\">=<\/span> <span class=\"s2\">&#34;disconnected&#34;<\/span> <span class=\"o\">]<\/span><span class=\"p\">;<\/span> <span class=\"k\">then<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">  <span class=\"c1\"># The monitor is not plugged in\t<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">  logger -t <span class=\"s2\">&#34;jamie-udev&#34;<\/span> <span class=\"s2\">&#34;Monitor is being unplugged&#34;<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">  \/usr\/bin\/xrandr --output DP-1 --off\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"k\">fi<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"m\">0<\/span> jamie@turkey:~$\n<\/span><\/span><\/code><\/pre><\/div><h2 id=\"move-windows-into-place\">Move windows into place.<\/h2>\n<p>So far, this handles ensuring the monitor is activated and placed in the right position. But, nothing has changed in my workspace.<\/p>\n<p>Here&rsquo;s where the devilspie2 configuration comes in:<\/p>\n<div class=\"highlight\"><pre tabindex=\"0\" class=\"chroma\"><code class=\"language-gdscript3\" data-lang=\"gdscript3\"><span class=\"line\"><span class=\"cl\"><span class=\"o\">==&gt;<\/span> <span class=\"o\">\/<\/span><span class=\"n\">home<\/span><span class=\"o\">\/<\/span><span class=\"n\">jamie<\/span><span class=\"o\">\/.<\/span><span class=\"n\">config<\/span><span class=\"o\">\/<\/span><span class=\"n\">devilspie2<\/span><span class=\"o\">\/<\/span><span class=\"mi\">00<\/span><span class=\"o\">-<\/span><span class=\"n\">globals<\/span><span class=\"o\">.<\/span><span class=\"n\">lua<\/span> <span class=\"o\">&lt;==<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"o\">--<\/span> <span class=\"n\">Collect<\/span> <span class=\"n\">some<\/span> <span class=\"n\">global<\/span> <span class=\"n\">varibles<\/span> <span class=\"n\">to<\/span> <span class=\"n\">be<\/span> <span class=\"n\">used<\/span> <span class=\"n\">throughout<\/span><span class=\"o\">.<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"n\">name<\/span> <span class=\"o\">=<\/span> <span class=\"n\">get_window_name<\/span><span class=\"p\">();<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"n\">app<\/span> <span class=\"o\">=<\/span> <span class=\"n\">get_application_name<\/span><span class=\"p\">();<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"n\">instance<\/span> <span class=\"o\">=<\/span> <span class=\"n\">get_class_instance_name<\/span><span class=\"p\">();<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"o\">--<\/span> <span class=\"n\">See<\/span> <span class=\"k\">if<\/span> <span class=\"n\">the<\/span> <span class=\"n\">monitor<\/span> <span class=\"n\">is<\/span> <span class=\"n\">plugged<\/span> <span class=\"ow\">in<\/span> <span class=\"ow\">or<\/span> <span class=\"ow\">not<\/span><span class=\"o\">.<\/span> <span class=\"n\">If<\/span> <span class=\"n\">monitor<\/span> <span class=\"n\">is<\/span> <span class=\"bp\">true<\/span><span class=\"p\">,<\/span> <span class=\"n\">it<\/span> <span class=\"n\">is<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"o\">--<\/span> <span class=\"n\">plugged<\/span> <span class=\"ow\">in<\/span><span class=\"p\">,<\/span> <span class=\"k\">if<\/span> <span class=\"n\">it<\/span> <span class=\"n\">is<\/span> <span class=\"bp\">false<\/span><span class=\"p\">,<\/span> <span class=\"n\">it<\/span> <span class=\"n\">is<\/span> <span class=\"ow\">not<\/span> <span class=\"n\">plugged<\/span> <span class=\"ow\">in<\/span><span class=\"o\">.<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"n\">monitor<\/span> <span class=\"o\">=<\/span> <span class=\"bp\">false<\/span><span class=\"p\">;<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"n\">device<\/span> <span class=\"o\">=<\/span> <span class=\"s2\">&#34;\/home\/jamie\/.config\/turkey\/monitor.status&#34;<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"n\">f<\/span> <span class=\"o\">=<\/span> <span class=\"n\">io<\/span><span class=\"o\">.<\/span><span class=\"n\">open<\/span><span class=\"p\">(<\/span><span class=\"n\">device<\/span><span class=\"p\">,<\/span> <span class=\"s2\">&#34;rb&#34;<\/span><span class=\"p\">)<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"k\">if<\/span> <span class=\"n\">f<\/span> <span class=\"n\">then<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">  <span class=\"o\">--<\/span> <span class=\"n\">Read<\/span> <span class=\"n\">the<\/span> <span class=\"n\">contents<\/span><span class=\"p\">,<\/span> <span class=\"n\">remove<\/span> <span class=\"n\">the<\/span> <span class=\"n\">trailing<\/span> <span class=\"n\">line<\/span> <span class=\"k\">break<\/span><span class=\"o\">.<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">  <span class=\"n\">content<\/span> <span class=\"o\">=<\/span> <span class=\"n\">string<\/span><span class=\"o\">.<\/span><span class=\"n\">gsub<\/span><span class=\"p\">(<\/span><span class=\"n\">f<\/span><span class=\"p\">:<\/span><span class=\"n\">read<\/span> <span class=\"s2\">&#34;*all&#34;<\/span><span class=\"p\">,<\/span> <span class=\"s2\">&#34;<\/span><span class=\"se\">\\n<\/span><span class=\"s2\">&#34;<\/span><span class=\"p\">,<\/span> <span class=\"s2\">&#34;&#34;<\/span><span class=\"p\">);<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">  <span class=\"k\">if<\/span> <span class=\"n\">content<\/span> <span class=\"o\">==<\/span> <span class=\"s2\">&#34;connected&#34;<\/span> <span class=\"n\">then<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">    <span class=\"n\">monitor<\/span> <span class=\"o\">=<\/span> <span class=\"bp\">true<\/span><span class=\"p\">;<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">  <span class=\"n\">end<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"n\">end<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">\n<\/span><\/span><span class=\"line\"><span class=\"cl\">\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"o\">==&gt;<\/span> <span class=\"o\">\/<\/span><span class=\"n\">home<\/span><span class=\"o\">\/<\/span><span class=\"n\">jamie<\/span><span class=\"o\">\/.<\/span><span class=\"n\">config<\/span><span class=\"o\">\/<\/span><span class=\"n\">devilspie2<\/span><span class=\"o\">\/<\/span><span class=\"n\">gajim<\/span><span class=\"o\">.<\/span><span class=\"n\">lua<\/span> <span class=\"o\">&lt;==<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"o\">--<\/span> <span class=\"n\">Look<\/span> <span class=\"k\">for<\/span> <span class=\"n\">my<\/span> <span class=\"n\">gajim<\/span> <span class=\"n\">message<\/span> <span class=\"n\">window<\/span><span class=\"o\">.<\/span> <span class=\"n\">Pin<\/span> <span class=\"n\">it<\/span> <span class=\"k\">if<\/span> <span class=\"n\">we<\/span> <span class=\"n\">have<\/span> <span class=\"n\">the<\/span> <span class=\"n\">monitor<\/span><span class=\"o\">.<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"k\">if<\/span> <span class=\"n\">string<\/span><span class=\"o\">.<\/span><span class=\"n\">find<\/span><span class=\"p\">(<\/span><span class=\"n\">name<\/span><span class=\"p\">,<\/span> <span class=\"s2\">&#34;Gajim: conversations.im&#34;<\/span><span class=\"p\">)<\/span> <span class=\"n\">then<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">  <span class=\"k\">if<\/span> <span class=\"n\">monitor<\/span> <span class=\"n\">then<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">    <span class=\"n\">set_window_geometry<\/span><span class=\"p\">(<\/span><span class=\"mi\">1931<\/span><span class=\"p\">,<\/span><span class=\"mi\">31<\/span><span class=\"p\">,<\/span><span class=\"mi\">590<\/span><span class=\"p\">,<\/span><span class=\"mi\">1025<\/span><span class=\"p\">);<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">    <span class=\"n\">pin_window<\/span><span class=\"p\">();<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">  <span class=\"k\">else<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">    <span class=\"n\">set_window_workspace<\/span><span class=\"p\">(<\/span><span class=\"mi\">4<\/span><span class=\"p\">);<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">    <span class=\"n\">set_window_geometry<\/span><span class=\"p\">(<\/span><span class=\"mi\">676<\/span><span class=\"p\">,<\/span><span class=\"mi\">31<\/span><span class=\"p\">,<\/span><span class=\"mi\">676<\/span><span class=\"p\">,<\/span><span class=\"mi\">725<\/span><span class=\"p\">);<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">    <span class=\"n\">unpin_window<\/span><span class=\"p\">();<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">  <span class=\"n\">end<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"n\">end<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"o\">==&gt;<\/span> <span class=\"o\">\/<\/span><span class=\"n\">home<\/span><span class=\"o\">\/<\/span><span class=\"n\">jamie<\/span><span class=\"o\">\/.<\/span><span class=\"n\">config<\/span><span class=\"o\">\/<\/span><span class=\"n\">devilspie2<\/span><span class=\"o\">\/<\/span><span class=\"n\">grunt<\/span><span class=\"o\">.<\/span><span class=\"n\">lua<\/span> <span class=\"o\">&lt;==<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"o\">--<\/span> <span class=\"n\">grunt<\/span> <span class=\"n\">is<\/span> <span class=\"n\">the<\/span> <span class=\"n\">window<\/span> <span class=\"n\">I<\/span> <span class=\"n\">use<\/span> <span class=\"n\">to<\/span> <span class=\"n\">connect<\/span> <span class=\"n\">via<\/span> <span class=\"n\">irc<\/span><span class=\"o\">.<\/span> <span class=\"n\">I<\/span> <span class=\"n\">typically<\/span> <span class=\"n\">connect<\/span> <span class=\"n\">to<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"o\">--<\/span> <span class=\"n\">grunt<\/span> <span class=\"n\">via<\/span> <span class=\"n\">a<\/span> <span class=\"n\">terminal<\/span> <span class=\"n\">called<\/span> <span class=\"n\">spade<\/span><span class=\"p\">,<\/span> <span class=\"n\">which<\/span> <span class=\"n\">is<\/span> <span class=\"n\">opened<\/span> <span class=\"n\">using<\/span> <span class=\"n\">a<\/span><span class=\"o\">-<\/span><span class=\"n\">terminal<\/span><span class=\"o\">-<\/span><span class=\"n\">yoohoo<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"o\">--<\/span> <span class=\"n\">so<\/span> <span class=\"n\">that<\/span> <span class=\"n\">bell<\/span> <span class=\"n\">actions<\/span> <span class=\"n\">cause<\/span> <span class=\"n\">a<\/span> <span class=\"n\">notification<\/span><span class=\"o\">.<\/span> <span class=\"n\">The<\/span> <span class=\"n\">window<\/span> <span class=\"n\">is<\/span> <span class=\"n\">called<\/span> <span class=\"n\">spade<\/span> <span class=\"k\">if<\/span> <span class=\"n\">I<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"o\">--<\/span> <span class=\"n\">just<\/span> <span class=\"n\">opened<\/span> <span class=\"n\">it<\/span> <span class=\"n\">but<\/span> <span class=\"n\">usually<\/span> <span class=\"n\">changes<\/span> <span class=\"n\">names<\/span> <span class=\"n\">to<\/span> <span class=\"n\">grunt<\/span> <span class=\"n\">after<\/span> <span class=\"n\">I<\/span> <span class=\"n\">connect<\/span> <span class=\"n\">via<\/span> <span class=\"n\">autossh<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"o\">--<\/span> <span class=\"n\">to<\/span> <span class=\"n\">grunt<\/span><span class=\"o\">.<\/span> \n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"o\">--<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"o\">--<\/span> <span class=\"n\">If<\/span> <span class=\"n\">no<\/span> <span class=\"n\">monitor<\/span><span class=\"p\">,<\/span> <span class=\"n\">put<\/span> <span class=\"n\">spade<\/span> <span class=\"ow\">in<\/span> <span class=\"n\">workspace<\/span> <span class=\"mi\">2<\/span><span class=\"p\">,<\/span> <span class=\"k\">if<\/span> <span class=\"n\">monitor<\/span><span class=\"p\">,<\/span> <span class=\"n\">then<\/span> <span class=\"n\">pin<\/span> <span class=\"n\">it<\/span> <span class=\"n\">to<\/span> <span class=\"n\">all<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"o\">--<\/span> <span class=\"n\">workspaces<\/span> <span class=\"ow\">and<\/span> <span class=\"n\">maximize<\/span> <span class=\"n\">it<\/span> <span class=\"n\">vertically<\/span><span class=\"o\">.<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"k\">if<\/span> <span class=\"n\">instance<\/span> <span class=\"o\">==<\/span> <span class=\"s2\">&#34;urxvt&#34;<\/span> <span class=\"n\">then<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">  <span class=\"o\">--<\/span> <span class=\"n\">When<\/span> <span class=\"n\">we<\/span> <span class=\"n\">launch<\/span><span class=\"p\">,<\/span> <span class=\"n\">the<\/span> <span class=\"n\">terminal<\/span> <span class=\"n\">is<\/span> <span class=\"n\">called<\/span> <span class=\"n\">spade<\/span><span class=\"p\">,<\/span> <span class=\"n\">after<\/span> <span class=\"n\">we<\/span> <span class=\"n\">connect<\/span> <span class=\"n\">it<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">  <span class=\"o\">--<\/span> <span class=\"n\">seems<\/span> <span class=\"n\">to<\/span> <span class=\"n\">get<\/span> <span class=\"n\">changed<\/span> <span class=\"n\">to<\/span> <span class=\"n\">jamie<\/span><span class=\"err\">@<\/span><span class=\"n\">grunt<\/span> <span class=\"ow\">or<\/span> <span class=\"n\">something<\/span> <span class=\"n\">like<\/span> <span class=\"n\">that<\/span><span class=\"o\">.<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">  <span class=\"k\">if<\/span> <span class=\"n\">name<\/span> <span class=\"o\">==<\/span> <span class=\"s2\">&#34;spade&#34;<\/span> <span class=\"ow\">or<\/span> <span class=\"n\">string<\/span><span class=\"o\">.<\/span><span class=\"n\">find<\/span><span class=\"p\">(<\/span><span class=\"n\">name<\/span><span class=\"p\">,<\/span> <span class=\"s2\">&#34;grunt:&#34;<\/span><span class=\"p\">)<\/span> <span class=\"n\">then<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">    <span class=\"k\">if<\/span> <span class=\"n\">monitor<\/span> <span class=\"n\">then<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">      <span class=\"n\">set_window_geometry<\/span><span class=\"p\">(<\/span><span class=\"mi\">1365<\/span><span class=\"p\">,<\/span><span class=\"mi\">10<\/span><span class=\"p\">,<\/span><span class=\"mi\">570<\/span><span class=\"p\">,<\/span><span class=\"mi\">1025<\/span><span class=\"p\">);<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">      <span class=\"n\">set_window_workspace<\/span><span class=\"p\">(<\/span><span class=\"mi\">3<\/span><span class=\"p\">);<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">      <span class=\"o\">--<\/span> <span class=\"n\">maximize_vertically<\/span><span class=\"p\">();<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">      <span class=\"n\">pin_window<\/span><span class=\"p\">();<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">    <span class=\"k\">else<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">      <span class=\"n\">set_window_geometry<\/span><span class=\"p\">(<\/span><span class=\"mi\">677<\/span><span class=\"p\">,<\/span><span class=\"mi\">10<\/span><span class=\"p\">,<\/span><span class=\"mi\">676<\/span><span class=\"p\">,<\/span><span class=\"mi\">375<\/span><span class=\"p\">);<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">      <span class=\"n\">set_window_workspace<\/span><span class=\"p\">(<\/span><span class=\"mi\">2<\/span><span class=\"p\">);<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">      <span class=\"n\">unpin_window<\/span><span class=\"p\">();<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">    <span class=\"n\">end<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">  <span class=\"n\">end<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"n\">end<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"o\">==&gt;<\/span> <span class=\"o\">\/<\/span><span class=\"n\">home<\/span><span class=\"o\">\/<\/span><span class=\"n\">jamie<\/span><span class=\"o\">\/.<\/span><span class=\"n\">config<\/span><span class=\"o\">\/<\/span><span class=\"n\">devilspie2<\/span><span class=\"o\">\/<\/span><span class=\"n\">terminals<\/span><span class=\"o\">.<\/span><span class=\"n\">lua<\/span> <span class=\"o\">&lt;==<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"o\">--<\/span> <span class=\"n\">Note<\/span> <span class=\"o\">-<\/span> <span class=\"n\">these<\/span> <span class=\"n\">will<\/span> <span class=\"n\">typically<\/span> <span class=\"n\">only<\/span> <span class=\"n\">work<\/span> <span class=\"n\">after<\/span> <span class=\"n\">I<\/span> <span class=\"n\">start<\/span> <span class=\"n\">the<\/span> <span class=\"n\">terminals<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"o\">--<\/span> <span class=\"k\">for<\/span> <span class=\"n\">the<\/span> <span class=\"n\">first<\/span> <span class=\"n\">time<\/span> <span class=\"n\">because<\/span> <span class=\"n\">their<\/span> <span class=\"n\">names<\/span> <span class=\"n\">seem<\/span> <span class=\"n\">to<\/span> <span class=\"n\">change<\/span><span class=\"o\">.<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"k\">if<\/span> <span class=\"n\">instance<\/span> <span class=\"o\">==<\/span> <span class=\"s2\">&#34;urxvt&#34;<\/span> <span class=\"n\">then<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">  <span class=\"k\">if<\/span> <span class=\"n\">name<\/span> <span class=\"o\">==<\/span> <span class=\"s2\">&#34;heart&#34;<\/span> <span class=\"n\">then<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">    <span class=\"n\">set_window_geometry<\/span><span class=\"p\">(<\/span><span class=\"mi\">0<\/span><span class=\"p\">,<\/span><span class=\"mi\">10<\/span><span class=\"p\">,<\/span><span class=\"mi\">676<\/span><span class=\"p\">,<\/span><span class=\"mi\">375<\/span><span class=\"p\">);<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">  <span class=\"n\">elseif<\/span> <span class=\"n\">name<\/span> <span class=\"o\">==<\/span> <span class=\"s2\">&#34;spade&#34;<\/span> <span class=\"n\">then<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">    <span class=\"n\">set_window_geometry<\/span><span class=\"p\">(<\/span><span class=\"mi\">677<\/span><span class=\"p\">,<\/span><span class=\"mi\">10<\/span><span class=\"p\">,<\/span><span class=\"mi\">676<\/span><span class=\"p\">,<\/span><span class=\"mi\">375<\/span><span class=\"p\">);<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">  <span class=\"n\">elseif<\/span> <span class=\"n\">name<\/span> <span class=\"o\">==<\/span> <span class=\"s2\">&#34;diamond&#34;<\/span> <span class=\"n\">then<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">    <span class=\"n\">set_window_geometry<\/span><span class=\"p\">(<\/span><span class=\"mi\">0<\/span><span class=\"p\">,<\/span><span class=\"mi\">376<\/span><span class=\"p\">,<\/span><span class=\"mi\">676<\/span><span class=\"p\">,<\/span><span class=\"mi\">375<\/span><span class=\"p\">);<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">  <span class=\"n\">elseif<\/span> <span class=\"n\">name<\/span> <span class=\"o\">==<\/span> <span class=\"s2\">&#34;clover&#34;<\/span> <span class=\"n\">then<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">    <span class=\"n\">set_window_geometry<\/span><span class=\"p\">(<\/span><span class=\"mi\">677<\/span><span class=\"p\">,<\/span><span class=\"mi\">376<\/span><span class=\"p\">,<\/span><span class=\"mi\">676<\/span><span class=\"p\">,<\/span><span class=\"mi\">375<\/span><span class=\"p\">);<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">  <span class=\"n\">end<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"n\">end<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"o\">==&gt;<\/span> <span class=\"o\">\/<\/span><span class=\"n\">home<\/span><span class=\"o\">\/<\/span><span class=\"n\">jamie<\/span><span class=\"o\">\/.<\/span><span class=\"n\">config<\/span><span class=\"o\">\/<\/span><span class=\"n\">devilspie2<\/span><span class=\"o\">\/<\/span><span class=\"n\">zimbra<\/span><span class=\"o\">.<\/span><span class=\"n\">lua<\/span> <span class=\"o\">&lt;==<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"o\">--<\/span> <span class=\"n\">Look<\/span> <span class=\"k\">for<\/span> <span class=\"n\">my<\/span> <span class=\"n\">zimbra<\/span> <span class=\"n\">firefox<\/span> <span class=\"n\">window<\/span><span class=\"o\">.<\/span> <span class=\"n\">Shows<\/span> <span class=\"n\">support<\/span> <span class=\"n\">queue<\/span><span class=\"o\">.<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"k\">if<\/span> <span class=\"n\">string<\/span><span class=\"o\">.<\/span><span class=\"n\">find<\/span><span class=\"p\">(<\/span><span class=\"n\">name<\/span><span class=\"p\">,<\/span> <span class=\"s2\">&#34;Zimbra&#34;<\/span><span class=\"p\">)<\/span> <span class=\"n\">then<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">  <span class=\"k\">if<\/span> <span class=\"n\">monitor<\/span> <span class=\"n\">then<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">    <span class=\"n\">unmaximize<\/span><span class=\"p\">();<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">    <span class=\"n\">set_window_geometry<\/span><span class=\"p\">(<\/span><span class=\"mi\">2520<\/span><span class=\"p\">,<\/span><span class=\"mi\">10<\/span><span class=\"p\">,<\/span><span class=\"mi\">760<\/span><span class=\"p\">,<\/span><span class=\"mi\">1022<\/span><span class=\"p\">);<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">    <span class=\"n\">pin_window<\/span><span class=\"p\">();<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">  <span class=\"k\">else<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">    <span class=\"n\">set_window_workspace<\/span><span class=\"p\">(<\/span><span class=\"mi\">5<\/span><span class=\"p\">);<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">    <span class=\"n\">set_window_geometry<\/span><span class=\"p\">(<\/span><span class=\"mi\">0<\/span><span class=\"p\">,<\/span><span class=\"mi\">10<\/span><span class=\"p\">,<\/span><span class=\"mi\">676<\/span><span class=\"p\">,<\/span><span class=\"mi\">375<\/span><span class=\"p\">);<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">    <span class=\"o\">--<\/span> <span class=\"n\">Zimbra<\/span> <span class=\"n\">can<\/span> <span class=\"n\">take<\/span> <span class=\"n\">up<\/span> <span class=\"n\">the<\/span> <span class=\"n\">whole<\/span> <span class=\"n\">window<\/span> <span class=\"n\">on<\/span> <span class=\"n\">this<\/span> <span class=\"n\">workspace<\/span><span class=\"o\">.<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">    <span class=\"n\">maximize<\/span><span class=\"p\">();<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">    <span class=\"n\">unpin_window<\/span><span class=\"p\">();<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">  <span class=\"n\">end<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"n\">end<\/span>\n<\/span><\/span><\/code><\/pre><\/div><p>It is started (and restartd) with:<\/p>\n<div class=\"highlight\"><pre tabindex=\"0\" class=\"chroma\"><code class=\"language-fallback\" data-lang=\"fallback\"><span class=\"line\"><span class=\"cl\">0 jamie@turkey:~$ cat ~\/.config\/systemd\/user\/devilspie2.service \n<\/span><\/span><span class=\"line\"><span class=\"cl\">[Unit]\n<\/span><\/span><span class=\"line\"><span class=\"cl\">Description=Start devilspie2, program to place windows in the right locations.\n<\/span><\/span><span class=\"line\"><span class=\"cl\">\n<\/span><\/span><span class=\"line\"><span class=\"cl\">[Service]\n<\/span><\/span><span class=\"line\"><span class=\"cl\">ExecStart=\/usr\/bin\/devilspie2\n<\/span><\/span><span class=\"line\"><span class=\"cl\">ExecStartPre=\/home\/jamie\/bin\/a-vga-auto\n<\/span><\/span><span class=\"line\"><span class=\"cl\">\n<\/span><\/span><span class=\"line\"><span class=\"cl\">[Install]\n<\/span><\/span><span class=\"line\"><span class=\"cl\">WantedBy=multi-user.target\n<\/span><\/span><span class=\"line\"><span class=\"cl\">0 jamie@turkey:~$ \n<\/span><\/span><\/code><\/pre><\/div><p>Which I have configured via a key combination that I hit everytime I plug in or unplug my monitor.<\/p>\n<p>And lastly, the ExecStartPre:<\/p>\n<div class=\"highlight\"><pre tabindex=\"0\" class=\"chroma\"><code class=\"language-bash\" data-lang=\"bash\"><span class=\"line\"><span class=\"cl\"><span class=\"cp\">#!\/bin\/bash\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"cp\"><\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"c1\"># If the monitor is plugged in, we have to tell the X window system about it<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"c1\"># so we can see our desktop. If it&#39;s unplugged, we have tell the X window system<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"c1\"># to re-adjust. This scripts checks whether the monitor is plugged in and<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"c1\"># whether the x window system realizes it. It auto adjusts the X window setting.<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"c1\">#<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"c1\"># This script is invoked by the devilspie2 systemd service <\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"c1\"># (~\/.config\/systemd\/user\/devilspie2.service). The devilspie2 systemd server<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"c1\"># is invoked with W-d as configured in ~\/.config\/openbox\/rc.xml.<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"nv\">monitor_status_path<\/span><span class=\"o\">=<\/span><span class=\"s2\">&#34;\/home\/jamie\/.config\/turkey\/monitor.status&#34;<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"nv\">monitor_status<\/span><span class=\"o\">=<\/span><span class=\"k\">$(<\/span>cat <span class=\"s2\">&#34;<\/span><span class=\"nv\">$monitor_status_path<\/span><span class=\"s2\">&#34;<\/span><span class=\"k\">)<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"nv\">xrandr_status<\/span><span class=\"o\">=<\/span><span class=\"s2\">&#34;disconnected&#34;<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"k\">if<\/span> \/usr\/bin\/xrandr <span class=\"p\">|<\/span> grep <span class=\"s2\">&#34;DP-1 connected 1920x1080+1366+0&#34;<\/span> &gt;\/dev\/null<span class=\"p\">;<\/span> <span class=\"k\">then<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">  <span class=\"nv\">xrandr_status<\/span><span class=\"o\">=<\/span><span class=\"s2\">&#34;connected&#34;<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"k\">fi<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"k\">if<\/span> <span class=\"o\">[<\/span> <span class=\"s2\">&#34;<\/span><span class=\"nv\">$monitor_status<\/span><span class=\"s2\">&#34;<\/span> <span class=\"o\">=<\/span> <span class=\"s2\">&#34;disconnected&#34;<\/span> -a <span class=\"s2\">&#34;<\/span><span class=\"nv\">$xrandr_status<\/span><span class=\"s2\">&#34;<\/span> <span class=\"o\">=<\/span> <span class=\"s2\">&#34;connected&#34;<\/span> <span class=\"o\">]<\/span><span class=\"p\">;<\/span> <span class=\"k\">then<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">  \/usr\/bin\/xrandr --output DP-1 --off\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"k\">elif<\/span> <span class=\"o\">[<\/span> <span class=\"s2\">&#34;<\/span><span class=\"nv\">$monitor_status<\/span><span class=\"s2\">&#34;<\/span> <span class=\"o\">=<\/span> <span class=\"s2\">&#34;connected&#34;<\/span> -a <span class=\"s2\">&#34;<\/span><span class=\"nv\">$xrandr_status<\/span><span class=\"s2\">&#34;<\/span> <span class=\"o\">=<\/span> <span class=\"s2\">&#34;disconnected&#34;<\/span> <span class=\"o\">]<\/span><span class=\"p\">;<\/span> <span class=\"k\">then<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">  \/usr\/bin\/xrandr --output DP-1 --right-of eDP-1 --auto\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"k\">fi<\/span>\n<\/span><\/span><\/code><\/pre><\/div>"},{"title":"The pain of installing custom ROMs on Android phones","link":"https:\/\/current.workingdirectory.net\/posts\/2018\/android-upgrades\/","pubDate":"Mon, 30 Apr 2018 09:46:16 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2018\/android-upgrades\/","description":"<p>A while back I bought a Nexus 5x. During a <a href=\"https:\/\/current.workingdirectory.net\/posts\/2016\/trusted-mobile-device\/\">three-day ordeal I finally got Omnirom installed<\/a> - with full disk encryption, root access and some stitched together fake Google Play code that allowed me to run Signal without actually letting Google into my computer.<\/p>\n<p>A short while later, Open Whisper Systems released a version of <a href=\"https:\/\/signal.org\">Signal<\/a> that uses Web Sockets when Google Play services is not installed (and allows for installation via a web page without the need for the Google Play store). Dang. Should have waited.<\/p>\n<p>Now, post Meltdown\/Spectre, I worked up the courage to go through this process again. In the <a href=\"https:\/\/current.workingdirectory.net\/posts\/2016\/trusted-mobile-device\/#comment-ca5ce67c58646175148ae37f3ecbc724\">comments<\/a> of my Omnirom post, I received a few suggestions about not really needing root. Hm - why didn&rsquo;t I think of that? Who needs root anyway? Combining root with full disk encryption was the real pain point in my previous install, so perhaps I can make things much easier. Also, not needing any of the fake Google software would be a definite plus.<\/p>\n<p>This time around I decided to go with <a href=\"https:\/\/lineageos.org\/\">LineageOS<\/a> since it seems to be the most mainstream of the custom ROMs. I found <a href=\"https:\/\/wiki.lineageos.org\/devices\/bullhead\/install\">perfectly reasonable sounding instructions<\/a>.<\/p>\n<p>My first mistake was skipping the initial steps (since I already had TWRP recovery installed I didn&rsquo;t think I needed to follow them). I went straight to the step of installing LineageOS (including wiping the Cache, System and Data partitions). Unfortunately, when it came time to flash the ROM, I got the error that the ROM is for bullhead, but the hardware I was using is &quot;&quot; (yes, empty string there).<\/p>\n<p>After some Internet searching I learned that the problem is an out-dated version of TWRP. Great, let&rsquo;s upgrade TWRP. I went back and started from the beginning of the LineageOS install instructions. But when it came to the <code>fastboot flashing unlock<\/code> step, I got the message explaining that my phone did not have OEM unlocking enabled. There are plenty of posts in the Internet demonstrating how to boot into your phone and <a href=\"https:\/\/www.androidsage.com\/2017\/06\/14\/what-is-oem-and-how-to-enable-oem-unlock-on-any-android-device\/\">flip the switch to allow OEM unlocking from the Developer section of your System tools<\/a>. Great, except that I could no longer boot into my phone thanks to the various deletion steps I had already taken. Dang. Did just brick my phone?<\/p>\n<p>I started thinking through how to buy a new phone.<\/p>\n<p>Then, I did more searching on the Internet and learned that I can flash a new version of TWRP the same way you flash anything else. Phew! New TWRP flashed and new LineageOS ROM installed! And, my first question: what is the point of locking your phone if you can still flash recovery images and even new ROMs?<\/p>\n<p>However, when I booted, I got an &ldquo;OS vendor mismatch error&rdquo;. WTF. Ok, now my phone is really bricked.<\/p>\n<p>Fortunately, someone not only identified this problem but contributed an <a href=\"https:\/\/gist.github.com\/MacKentoch\/48ad6b91613213ee9774c138267e2ed4\">exceptionally well-written step-by-step set of directions to fix the problem<\/a>. The post, in combination with some comments on it, explains that you have to <a href=\"https:\/\/developers.google.com\/android\/images\">download the Google firmware<\/a> that corresponds to the error code in your message (in case that post ever goes away: unzip the file you download, then cd into the directory created and unzip the file that starts with image-bullhead. Then, minimally flash the vendor.img to the vendor partition in TWRP).<\/p>\n<p>In other words, the LineageOS ROM depends on having the right Google firmware installed.<\/p>\n<p>All of these steps were possible without unlocking the phone. However, when I tried to update the Radio and Bootloader using:<\/p>\n<pre><code>fastboot flash bootloader bootloader-bullhead-bhz11l.img\nfastboot flash radio radio-bullhead-m8994f-2.6.37.2.21.img\n<\/code><\/pre>\n<p>It failed, so I booted into my now working install, enabled OEM unlock, unlocked the phone (which wiped everything so I had to start over) and then it all worked.<\/p>\n<p>And, kudos to LineageOS for the simple setup process and ease of getting full disk encryption.<\/p>\n<p>Now that I&rsquo;m done, I am asking myself a few questions:<\/p>\n<p>I have my own custom ROM and I am not trusting Google with everything anymore. Hooray! So &hellip; who am I trusting? This question I know the answer to (I think&hellip;):<\/p>\n<ul>\n<li><a href=\"https:\/\/twrp.me\/\">Team Win<\/a>, which provides the TWRP recovery software has total control of everything. Geez, I hope these people aren&rsquo;t assholes.<\/li>\n<li><a href=\"https:\/\/google.com\">Google<\/a>, since I blindly install their firmware vendor image, bootloader image and radio image. I guess they still can control my phone.<\/li>\n<li><a href=\"https:\/\/f-droid.org\/\">Fdroid<\/a>, I hope they vette their packages, because I blindly install them from their default archives.<\/li>\n<li><a href=\"https:\/\/guardianproject.info\/\">Guardian Project<\/a>, since I enable their fdroid repo too - but hey at least I have met a few of these people and they are <a href=\"https:\/\/mayfirst.org\/\">May First\/People Link<\/a> members.<\/li>\n<li><a href=\"https:\/\/mozilla.org\">Firefox<\/a>, I download firefox directly from Mozilla since fdroid doesn&rsquo;t seem to really support them.<\/li>\n<li><a href=\"https:\/\/signal.org\">Signal<\/a>, since I download that APK directly as well.<\/li>\n<li>And the https certificate system (which pretty much means <a href=\"https:\/\/letsencrypt.org\/\">Let&rsquo;s Encrypt<\/a> nowadays - since nearly everything depends on the integrity of the packages I am downloading over https.<\/li>\n<\/ul>\n<p>But I&rsquo;m still not sure about one more question:<\/p>\n<p>Should I lock my phone? Given what I just accomplished without locking it, it seems that locking the phone could make my life harder the next time I upgrade and doesn&rsquo;t really stop someone else from replacing key components of my operating system without me knowing it.<\/p>\n"},{"title":"Docker in Debian","link":"https:\/\/current.workingdirectory.net\/posts\/2017\/docker-in-debian\/","pubDate":"Tue, 10 Oct 2017 12:10:04 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2017\/docker-in-debian\/","description":"<p>It&rsquo;s not easy getting Docker to work in Debian.<\/p>\n<p>It&rsquo;s not in stable at all:<\/p>\n<pre><code>0 jamie@turkey:~$ rmadison docker.io\ndocker.io  | 1.6.2~dfsg1-1~bpo8+1 | jessie-backports | source, amd64, armel, armhf, i386\ndocker.io  | 1.11.2~ds1-5         | unstable         | source, arm64\ndocker.io  | 1.11.2~ds1-5         | unstable-debug   | source\ndocker.io  | 1.11.2~ds1-6         | unstable         | source, armel, armhf, i386, ppc64el\ndocker.io  | 1.11.2~ds1-6         | unstable-debug   | source\ndocker.io  | 1.13.1~ds1-2         | unstable         | source, amd64\ndocker.io  | 1.13.1~ds1-2         | unstable-debug   | source\n0 jamie@turkey:~$ \n<\/code><\/pre>\n<p>And a <a href=\"https:\/\/bugs.debian.org\/cgi-bin\/bugreport.cgi?bug=877329\">problem with runc<\/a> makes it really hard to get it working on Debian unstable.<\/p>\n<p>These are the steps I took to get it running today (2017-10-10).<\/p>\n<p>Remove runc (allow it to remove containerd and docker.io):<\/p>\n<pre><code>sudo apt-get remove runc\n<\/code><\/pre>\n<p>Install docker-runc (now in testing)<\/p>\n<pre><code>sudo apt-get install docker-runc\n<\/code><\/pre>\n<p>Fix containerd package to depend on docker-runc instead of runc:<\/p>\n<pre><code>mkdir containerd\ncd containerd\napt-get download containerd \nar x containerd_0.2.3+git20170126.85.aa8187d~ds1-2_amd64.deb\ntar -xzf control.tar.gz\nsed -i s\/runc\/docker-runc\/g control\ntar -c md5sums control | gzip -c &gt; control.tar.gz\nar rcs new-containerd.deb debian-binary control.tar.gz data.tar.xz\nsudo dpkg -i new-containerd.deb\n<\/code><\/pre>\n<p>Fix docker.io package to depend on docker-runc instead of runc.<\/p>\n<pre><code>mkdir docker\ncd docker\napt-get download docker.io\nar x docker.io_1.13.1~ds1-2_amd64.deb\ntar -xzf control.tar.gz\nsed -i s\/runc\/docker-runc\/g control\ntar -c {post,pre}{inst,rm} md5sums control | gzip -c &gt; control.tar.gz\nar rcs new-docker.io.deb debian-binary control.tar.gz data.tar.xz\nsudo dpkg -i new-docker.io.deb\n<\/code><\/pre>\n<p>Symlink <code>docker-runc<\/code> =&gt; <code>runc<\/code><\/p>\n<pre><code>sudo ln -s \/usr\/sbin\/docker-runc \/usr\/sbin\/runc\n<\/code><\/pre>\n<p>Keep <code>apt-get<\/code> from upgrading until this bug is fixed:<\/p>\n<pre><code>printf &quot;# Remove when docker.io and containerd depend on docker-runc\n# instead of normal runc\n# https:\/\/bugs.debian.org\/cgi-bin\/bugreport.cgi?bug=877329\nPackage: runc \nPin: release * \nPin-Priority: -1 \n\nPackage: containderd \nPin: release * \nPin-Priority: -1 \n\nPackage: docker.io\nPin: release * \nPin-Priority: -1&quot; | sudo tee \/etc\/apt\/preferences.d\/docker.pref\n<\/code><\/pre>\n<p>Thanks to <a href=\"https:\/\/coderwall.com\/p\/hes3ha\/change-the-dependencies-of-a-deb-package\">coderwall<\/a> for tips on manipulating deb files.<\/p>\n"},{"title":"Late to the Raspberry Pi party","link":"https:\/\/current.workingdirectory.net\/posts\/2017\/raspbian\/","pubDate":"Thu, 18 May 2017 23:37:06 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2017\/raspbian\/","description":"<p>I finally bought my first raspberry pi to setup as a router and wifi access point.<\/p>\n<p>It wasn&rsquo;t easy.<\/p>\n<p>I first had to figure out what to buy. I think that was the hardest part.<\/p>\n<p>I ended up with:<\/p>\n<ul>\n<li>Raspberry PI 3 Model B A1.2GHz 64-bit quad-core ARMv8 CPU, 1GB RAM (Model number: RASPBERRYPI3-MODB-1GB)<\/li>\n<li>Transcend USB 3.0 SDHC \/ SDXC \/ microSDHC \/ SDXC Card Reader, TS-RDF5K (Black). I only needed this because I don&rsquo;t have one already and I will need a way to copy a raspbian image from my laptop to a micro SD card.<\/li>\n<li>Centon Electronics Micro SD Card 16 GB (S1-MSDHC4-16G). This is the micro sd card.<\/li>\n<li>Smraza Clear case for Raspberry Pi 3 2 Model B with Power Supply,2pcs Heatsinks and Micro USB with On\/Off Switch. And this is the box to put it all in.<\/li>\n<\/ul>\n<p>I already have a cable matters USB to ethernet device, which will provide the second ethernet connection so this device can actually work as a router.<\/p>\n<p>I studiously <a href=\"https:\/\/www.raspberrypi.org\/documentation\/installation\/installing-images\/\">followed the directions to download the raspbian image and copy it to my micro sd card<\/a>. I also <a href=\"https:\/\/www.raspberrypi.org\/documentation\/remote-access\/ssh\/\">touched a file on the boot partition called ssh<\/a> so ssh would start automatically. Note: I first touched the ssh file on the root partition (sdb2) before realizing it belonged on the boot partition (sdb1). And, despite ambiguous directions found on the Internet, lowercase &lsquo;ssh&rsquo; for the filename seems to do the trick.<\/p>\n<p>Then, I found the IP address with the help of NMAP (<code>sudo nmap -sn 192.168.69.*<\/code>) and tried to ssh in but alas&hellip;<\/p>\n<pre><code>Connection reset by 192.168.69.116 port 22\n<\/code><\/pre>\n<p>No dice.<\/p>\n<p>So, I re-mounted the sdb2 partition of the micro sd card and looked in var\/log\/auth.log and found:<\/p>\n<pre><code>May  5 19:23:00 raspberrypi sshd[760]: error: Could not load host key: \/etc\/ssh\/ssh_host_ed25519_key\nMay  5 19:23:00 raspberrypi sshd[760]: fatal: No supported key exchange algorithms [preauth]\nMay  5 19:23:07 raspberrypi sshd[762]: error: key_load_public: invalid format\nMay  5 19:23:07 raspberrypi sshd[762]: error: Could not load host key: \/etc\/ssh\/ssh_host_rsa_key\nMay  5 19:23:07 raspberrypi sshd[762]: error: key_load_public: invalid format\nMay  5 19:23:07 raspberrypi sshd[762]: error: Could not load host key: \/etc\/ssh\/ssh_host_dsa_key\nMay  5 19:23:07 raspberrypi sshd[762]: error: key_load_public: invalid format\nMay  5 19:23:07 raspberrypi sshd[762]: error: Could not load host key: \/etc\/ssh\/ssh_host_ecdsa_key\nMay  5 19:23:07 raspberrypi sshd[762]: error: key_load_public: invalid format\n<\/code><\/pre>\n<p>How did that happen? And wait a minute&hellip;<\/p>\n<pre><code>0 jamie@turkey:~$ ls -l \/mnt\/etc\/ssh\/ssh_host_ecdsa_key\n-rw------- 1 root root 0 Apr 10 05:58 \/mnt\/etc\/ssh\/ssh_host_ecdsa_key\n0 jamie@turkey:~$ date\nFri May  5 15:44:15 EDT 2017\n0 jamie@turkey:~$\n<\/code><\/pre>\n<p>Are the keys embedded in the image? Isn&rsquo;t that wrong?<\/p>\n<p>I fixed with:<\/p>\n<pre><code>0 jamie@turkey:mnt$ sudo rm \/mnt\/etc\/ssh\/ssh_host_*\n0 jamie@turkey:mnt$ sudo ssh-keygen -q -f \/mnt\/etc\/ssh\/ssh_host_rsa_key -N '' -t rsa\n0 jamie@turkey:mnt$ sudo ssh-keygen -q -f \/mnt\/etc\/ssh\/ssh_host_dsa_key -N '' -t dsa\n0 jamie@turkey:mnt$ sudo ssh-keygen -q -f \/mnt\/etc\/ssh\/ssh_host_ecdsa_key -N '' -t ecdsa\n0 jamie@turkey:mnt$ sudo ssh-keygen -q -f \/mnt\/etc\/ssh\/ssh_host_ed25519_key -N '' -t ed25519\n0 jamie@turkey:mnt$\n<\/code><\/pre>\n<p>NOTE: I just did a second installation and this didn&rsquo;t happen. Maybe something went wrong as I experiment with SSH vs ssh on the boot partition?<\/p>\n<p>Then I could ssh in. I removed the <code>pi<\/code> user account and added my ssh key to \/root\/.ssh\/authorized_keys and put a new name &ldquo;mondragon&rdquo; in the \/etc\/hostname file.<\/p>\n<p>And&hellip; I upgraded to Debian stretch and rebooted.<\/p>\n<p>Then, I followed <a href=\"https:\/\/raspberrypi.stackexchange.com\/questions\/53245\/pi-3-raspbian-stretch-testing-disables-wifi\">these instructions for fixing the wifi<\/a> (replacing the firmware does still work for me).<\/p>\n<p>I plugged my cable matters USB\/Ethernet adapter into the device so it would be recognized, but left it dis-connected.<\/p>\n<p>Next I started to configure the device to be a wifi access point <a href=\"https:\/\/frillip.com\/using-your-raspberry-pi-3-as-a-wifi-access-point-with-hostapd\/\">using this excellend tutorial<\/a>, but decided I wanted to setup my networks using systemd-networkd instead.<\/p>\n<p>Since \/etc\/network\/interaces already had eth0 set to manual (because apparently it is controlled by dhcpcd instead), I didn&rsquo;t need any modifications there.<\/p>\n<p>However, I wanted to use the dhcp client built-in to systemd-networkd, so to prevent dhcpcd from obtaining an IP address, I purged dhcpcd:<\/p>\n<pre><code>apt-get purge dhcpcd5\n<\/code><\/pre>\n<p>I was planning to also use systemd-networkd to name the devices (<a href=\"https:\/\/www.freedesktop.org\/software\/systemd\/man\/systemd.link.html\">using *.link files<\/a>) but nothing I could do could convince systemd to rename them, so I gave up and added \/etc\/udev\/rules.d\/70-persistent-net.rules:<\/p>\n<pre><code>\tSUBSYSTEM==&quot;net&quot;, ACTION==&quot;add&quot;, DRIVERS==&quot;?*&quot;, ATTR{address}==&quot;b8:27:eb:ce:b5:c3&quot;, ATTR{dev_id}==&quot;0x0&quot;, ATTR{type}==&quot;1&quot;, NAME:=&quot;wan&quot;\n\tSUBSYSTEM==&quot;net&quot;, ACTION==&quot;add&quot;, DRIVERS==&quot;?*&quot;, ATTR{address}==&quot;a0:ce:c8:01:20:7d&quot;, ATTR{dev_id}==&quot;0x0&quot;, ATTR{type}==&quot;1&quot;, NAME:=&quot;lan&quot;\n\tSUBSYSTEM==&quot;net&quot;, ACTION==&quot;add&quot;, DRIVERS==&quot;?*&quot;, ATTR{address}==&quot;b8:27:eb:9b:e0:96&quot;, ATTR{dev_id}==&quot;0x0&quot;, ATTR{type}==&quot;1&quot;, NAME:=&quot;wlan&quot;\n<\/code><\/pre>\n<p>(If you are copying and pasting the mac addresses will have to change.)<\/p>\n<p>Then I added the following files:<\/p>\n<pre><code>root@mondragon:~# head \/etc\/systemd\/network\/*\n==&gt; \/etc\/systemd\/network\/50-lan.network &lt;==\n[Match]\nName=lan\n\n[Network]\nAddress=192.168.69.1\/24\n\n==&gt; \/etc\/systemd\/network\/55-wlan.network &lt;==\n[Match]\nName=wlan\n\n[Network]\nAddress=10.0.69.1\/24\n\n==&gt; \/etc\/systemd\/network\/60-wan.network &lt;==\n[Match]\nName=wan\n\n[Network]\nDHCP=v4\nIPForward=yes\nIPMasquerade=yes\nroot@mondragon:~#\n<\/code><\/pre>\n<p>Sadly, IPMasquerade doesn&rsquo;t seem to work either for some reason, so&hellip;<\/p>\n<pre><code>root@mondragon:~# cat \/etc\/systemd\/system\/masquerade.service \n[Unit]\nDescription=Start masquerading because Masquerade=yes not working in wan.network.\n\n[Service]\nType=oneshot\nExecStart=\/sbin\/iptables -t nat -A POSTROUTING -o wan -j MASQUERADE\n\n[Install]\nWantedBy=network.target\nroot@mondragon:~#\n<\/code><\/pre>\n<p>And, systemd DHCPServer worked, but then it didn&rsquo;t and I couldn&rsquo;t figure out how to debug, so&hellip;<\/p>\n<pre><code>apt-get install dnsmasq\n<\/code><\/pre>\n<p>Followed by:<\/p>\n<pre><code>root@mondragon:~# cat \/etc\/dnsmasq.d\/mondragon.conf \n# Don't provide DNS services (unbound does that).\nport=0\n\ninterface=lan\ninterface=wlan\n\n# Only provide dhcp services since systemd-networkd dhcpserver seems\n# flakey.\ndhcp-range=set:cable,192.168.69.100,192.168.69.150,255.255.255.0,4h\ndhcp-option=tag:cable,option:dns-server,192.168.69.1\ndhcp-option=tag:cable,option:router,192.168.69.1\n\ndhcp-range=set:wifi,10.0.69.100,10.0.69.150,255.255.255.0,4h\ndhcp-option=tag:wifi,option:dns-server,10.0.69.1\ndhcp-option=tag:wifi,option:router,10.0.69.1\n\nroot@mondragon:~#\n<\/code><\/pre>\n<p>It would probably be simpler to have dnsmasq provide DNS service also, but I happen to like unbound:<\/p>\n<pre><code>apt-get install unbound\n<\/code><\/pre>\n<p>And&hellip;<\/p>\n<pre><code>root@mondragon:~# cat \/etc\/unbound\/unbound.conf.d\/server.conf \nserver:\n    interface: 127.0.0.1\n    interface: 192.168.69.1\n    interface: 10.0.69.1\n\n    access-control: 192.168.69.0\/24 allow\n    access-control: 10.0.69.0\/24 allow\n\n    # We do query localhost for our stub zone: loc.cx\n    do-not-query-localhost: no\n\n    # Up this level when debugging.\n    log-queries: no\n    logfile: &quot;&quot;\n    #verbosity: 1\n\n    # Settings to work better with systemcd\n    do-daemonize: no\n    pidfile: &quot;&quot;\nroot@mondragon:~# \n<\/code><\/pre>\n<p>Now on to the wifi access point.<\/p>\n<pre><code>apt-get install hostapd\n<\/code><\/pre>\n<p>And the configuration file:<\/p>\n<pre><code>root@mondragon:~# cat \/etc\/hostapd\/hostapd.conf\n# This is the name of the WiFi interface we configured above\ninterface=wlan\n\n# Use the nl80211 driver with the brcmfmac driver\ndriver=nl80211\n\n# This is the name of the network\nssid=peacock\n\n# Use the 2.4GHz band\nhw_mode=g\n\n# Use channel 6\nchannel=6\n\n# Enable 802.11n\nieee80211n=1\n\n# Enable WMM\nwmm_enabled=1\n\n# Enable 40MHz channels with 20ns guard interval\nht_capab=[HT40][SHORT-GI-20][DSSS_CCK-40]\n\n# Accept all MAC addresses\nmacaddr_acl=0\n\n# Use WPA authentication\nauth_algs=1\n\n# Require clients to know the network name\nignore_broadcast_ssid=0\n\n# Use WPA2\nwpa=2\n\n# Use a pre-shared key\nwpa_key_mgmt=WPA-PSK\n\n# The network passphrase\nwpa_passphrase=xxxxxxxxxxxx\n\n# Use AES, instead of TKIP\nrsn_pairwise=CCMP\nroot@mondragon:~#\n<\/code><\/pre>\n<p>The hostapd package doesn&rsquo;t have a systemd start up file so I added one:<\/p>\n<pre><code>root@mondragon:~# cat \/etc\/systemd\/system\/hostapd.service \n[Unit]\nDescription=Hostapd IEEE 802.11 AP, IEEE 802.1X\/WPA\/WPA2\/EAP\/RADIUS Authenticator\nWants=network.target\nBefore=network.target\nBefore=network.service\n\n[Service]\nExecStart=\/usr\/sbin\/hostapd \/etc\/hostapd\/hostapd.conf\n\n[Install]\nWantedBy=multi-user.target\nroot@mondragon:~#\n<\/code><\/pre>\n<p>My last step was to modify \/etc\/ssh\/sshd_config so it only listens on the lan and wlan interfaces (listening on wlan is a bit of a risk, but also useful when mucking with the lan network settings to ensure I don&rsquo;t get locked out).<\/p>\n"},{"title":"Re-thinking Web App Security","link":"https:\/\/current.workingdirectory.net\/posts\/2017\/new-security-paradigm\/","pubDate":"Wed, 15 Feb 2017 11:56:30 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2017\/new-security-paradigm\/","description":"<p>An organizer friend interested in activating a rapid response network to counter Trump-era ICE raids on immigrants asked me about any existing simple and easy tools that could send out emergency SMS\/text message alerts.<\/p>\n<p>I thought about it and ended up writing my first <a href=\"https:\/\/pouchdb.com\/\">pouchdb<\/a> web application to accomplish the task. For the curious, you can see it <a href=\"https:\/\/sms.workingdirectory.net\/\">in action<\/a> and <a href=\"https:\/\/gitlab.com\/jamie\/basenotify\">browse the source code<\/a>. To use it to send SMS, you have to <a href=\"https:\/\/www.twilio.com\/try-twilio\">register for a Twilio account<\/a> - you can get a free account that has very restricted SMS sending capability or pay for full functionality.<\/p>\n<p>The project is unlike anything I have done before.<\/p>\n<p>I chose pouchdb because it stores all of your contacts <em>in your browser<\/em> not on a server somewhere in the so-called cloud. (You can also choose to sync to a server, a feature I have not yet implemented.)<\/p>\n<p>The implications of storing your data locally are quite profound.<\/p>\n<h2 id=\"classic-web-app\">Classic Web App<\/h2>\n<p>Let&rsquo;s first consider the more common web application: You visit a web site (the same web site that your colleagues visit, or in the case of a massive application like gmail.com, the same web site that everyone in the world visits). Then, you login with your own unique username and password, which grants you access to the portion the database that you are suppose to have access to.<\/p>\n<p>For most use-cases, this model is fairly ideal:<\/p>\n<ul>\n<li>If you have a technically competent host, your data is backed up regularly and the database is available nearly 100% of the time<\/li>\n<li>If you have a politically trust-worthy host, your host will notify you and put up a fight before turning any of your data over to a government agent<\/li>\n<li>If you drop your phone in the toilet you can always login from another computer to access your data<\/li>\n<li>If you save your password in your browser and your laptop is stolen, you can always change your password to prevent the thief from accessing your data<\/li>\n<li>You can easily share your data with others by creating new usernames and passwords<\/li>\n<\/ul>\n<p>However, there are some downsides:<\/p>\n<ul>\n<li>If your host is not technically competent or polically trust-worthy, you could lose all of your data to a hard drive crash or subpoena<\/li>\n<li>Even if your host <em>is<\/em> competent, all of your data is one previously undiscovered vulnerability away from being hacked<\/li>\n<li>Even if your host <em>is<\/em> politically trust-worthy, you cannot always stop a subpoena, particularly given the legal escalations of tools like national security letters<\/li>\n<\/ul>\n<h2 id=\"pouchdb-no-sync\">pouchdb no sync<\/h2>\n<p>Assuming you are accessing your database on a device with an encrypted disk <em>and<\/em> you manage your own backups, pouchdb without synchoronizing provides the most privacy and autonomy. You have complete control of your data and you are not dependent on any server operator.<\/p>\n<p>However, the trade-offs are harsh:<\/p>\n<ul>\n<li>Availability: if you lose your device, you would need to restore from backup - which is much more difficult than simply logging in from another device<\/li>\n<li>Collaboration: you simply can&rsquo;t share this data with anyone else<\/li>\n<\/ul>\n<p>It seems this model is fairly useless except in very tight corner cases.<\/p>\n<h2 id=\"pouchdb-that-synchronizes-to-a-server\">pouchdb that synchronizes to a server<\/h2>\n<p>With this model, you avoid the trade-offs of the no sync model (hooray!). However, you also lose all of the privacy benefits, and it&rsquo;s even <em>worse<\/em>: your data can be compromised either via a server breach or via a compromise of any of the devices you are using. If any of these devices lack encrypted disks, then it&rsquo;s borderline reckless.<\/p>\n<p>On the other hand, you gain a huge benefit in terms of reliability. If the server goes down, loses your data, fails to backup or is taken offline by a legal order, you can still function perfectly well and can optionally choose to sync to a different host.<\/p>\n<h2 id=\"conclusions\">Conclusions<\/h2>\n<p>Ultimately, we need to better evaluate the trade-offs between privacy and availability for each given use of a database and try to make the best decision.<\/p>\n<p>And&hellip; keep working on new models. For example, it seems an ideal middle ground would be to sync in a peer-to-peer fashion with our colleagues (see <a href=\"https:\/\/github.com\/natevw\/PeerPouch\">PeerPouch<\/a>) or sync to a server under your control in your office.<\/p>\n"},{"title":"What's Up with WhatsApp?","link":"https:\/\/current.workingdirectory.net\/posts\/2017\/whats-app-whats-up\/","pubDate":"Fri, 13 Jan 2017 21:02:48 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2017\/whats-app-whats-up\/","description":"<p>Despite my jaded feelings about corporate Internet services in general, I was suprised to learn that <a href=\"https:\/\/www.theguardian.com\/technology\/2017\/jan\/13\/whatsapp-backdoor-allows-snooping-on-encrypted-messages\">WhatsApp&rsquo;s end-to-end encryption was a lie<\/a>. In short, it is possible to send an encrypted message to a user that is intercepted and effectively de-crypted without the sender&rsquo;s knowledge.<\/p>\n<p>However, I was even more surprised to read <a href=\"https:\/\/whispersystems.org\/blog\/there-is-no-whatsapp-backdoor\/\">Open Whisper Systems critique of the original story<\/a>, claiming that it is not a backdoor because the WhatsApp sender&rsquo;s client is always notified when a message is de-crypted.<\/p>\n<p>The Open Whisper Systems post acknowledges that the WhatsApp sender can choose to disable these notifications, but claims that is not such a big deal because the WhatsApp server has no way to know which clients have this feature enabled and which do not, so intercepting a message is risky because it could result in the sender realizing it.<\/p>\n<p>However, there is a fairly important piece of information missing, namely: as far as I can tell, the setting to notify users about key changes is disabled by default.<\/p>\n<p>So, using the default installation, your end-to-end encrypted message could be intercepted and decrypted without you or the party you are communicating with knowing it. How is this not a back door? And yes, if the interceptor can&rsquo;t tell whether or not the sender has these notifications turned on, the interceptor runs the risk of someone knowing they have intercepted the message. Great. That&rsquo;s better than nothing. Except that there is strong evidence that many powerful governments on this planet routinely risk exposure in their pursuit of compromising our ability to communicate securely. And&hellip; not to mention non-governmental (or governmental) adversaries for whom exposure is not a big deal.<\/p>\n<p>Furthermore a critical reason for end-to-end encrption is so that your provider does not have the technical capacity to intercept your communications. That&rsquo;s simply not true of WhatsApp. It <em>is<\/em> true of Signal and OMEMO, which requires the active participation of the sender to compromise the communication.<\/p>\n<p>Why in the world would you distribute a client that not only has the ability to surpress such warnings, but has it enabled by default?<\/p>\n<p>Some may argue that users regularly dismiss notifications like &ldquo;fingerprint has changed&rdquo; and that this problem is the achilles heal of secure communications. I agree. But&hellip; there is still a monumental difference between a user absent-mindedly dismissing an important security warning and never seeing the warning in the first place.<\/p>\n<p>This flaw in WhatsApp is a critical reminder that secure communications doesn&rsquo;t just depend on a good protocol or technology, but on trust in the people who design and maintain our systems.<\/p>\n"},{"title":"End-to-End Encrypted group chats via XMPP","link":"https:\/\/current.workingdirectory.net\/posts\/2017\/encrypted-mucs\/","pubDate":"Thu, 05 Jan 2017 17:16:49 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2017\/encrypted-mucs\/","description":"<p>It&rsquo;s been over a year since my colleagues and I at the <a href=\"https:\/\/progressivetech.org\/\">Progressive Technology Project<\/a> abandoned Skype, first for IRC and soon after for XMPP. Thanks to the talented folks maintaining <a href=\"https:\/\/account.conversations.im\/\">conversations.im<\/a> it&rsquo;s been a breeze to get everyone setup with accounts (8 Euros\/year is quite worth it) and a group chat going.<\/p>\n<p>However, our group chats have not been using end-to-end encryption&hellip; until now. It wasn&rsquo;t exactly painless, so I&rsquo;m sharing some tips and tricks.<\/p>\n<ul>\n<li>Use either Conversations for Android (<a href=\"https:\/\/f-droid.org\/repository\/browse\/?fdid=eu.siacs.conversations\">f-droid<\/a> or <a href=\"https:\/\/play.google.com\/store\/apps\/details?id=eu.siacs.conversations\">Play<\/a>) or <a href=\"https:\/\/gajim.org\/\">Gajim<\/a> for Windows or Linux. At the time of this writing, these are the only two applications I know of that support <a href=\"https:\/\/conversations.im\/omemo\/\">OMEMO<\/a>, the XMPP extension that supports end-to-end encryption. Chat Secure for iOS, however, is <a href=\"https:\/\/github.com\/ChatSecure\/ChatSecure-iOS\/issues\/376\">just a release away<\/a>. We managed to get things working with most of us using <em>both<\/em> Gajim and Conversations. It would probably have been much easier and smoother if everyone were <em>only<\/em> using Conversations because OMEMO is built-in to core, rather than Gajim, where OMEMO support is provided <a href=\"https:\/\/dev.gajim.org\/gajim\/gajim-plugins\">via an extension<\/a>.<\/li>\n<li>If you are using Gajim&hellip; After installing the OMEMO plugin in Gajim, fully restart Gajim. Similarly, if you add or remove a contact from your group, it seems you have to fully restart Gajim. Not sure why. If something is not working in Gajim, try restarting it.<\/li>\n<li>Ensure that everyone in your group has added everyone else in the group to their roster. This was the single biggest and most confusing part of the process. If you are missing just one contact in your roster, then messages you type into the group chat will not show up <a href=\"https:\/\/dev.gajim.org\/gajim\/gajim-plugins\/issues\/173\">without any indication as to what happened or why<\/a> (on Gajim). Take this step first or prepare for confusing failures. Remember: <em>everyone<\/em> has to have <em>everyone else<\/em> in their roster.<\/li>\n<li>Create the group in the android Conversations app, not in Gajim. There are strict requirements for how the group needs to be setup (private, members only and non-anonymous). I tried creating the group in Gajim and followed the directions but couldn&rsquo;t get it to work. Creating the group in Conversations worked right away. Remember: don&rsquo;t add members to the group unless everyone has them in their roster!<\/li>\n<li>You can give your group a easy to remember name in your Gajim bookmarks, but under the hood, it will be assigned a random name. Conversations will show you the random name via &ldquo;Conference Details&rdquo; and Gajim will show it under the tab in the Messages window. When inviting people to the group you may need to select the random name.<\/li>\n<li>Trust on First Use. In our experiment, we created a group for four people and we were all on a video and voice chat while we set things up. Three out of the four of us had both Gajim and Conversations in play. That meant 4 different people had to verify between 5 and 6 fingerprints each. We decided to use Trust on First Use rather than go through the process of reading out all the fingerprints (for the record, it still took us an hour and 15 minutes to get it all working). See Daniel Gultsch&rsquo;s interesting article on <a href=\"https:\/\/gultsch.de\/trust.html\">Trust on First Use<\/a>.<\/li>\n<li>If you get an error &ldquo;This is not a group chat&rdquo; it may be because you accidentally added the group as a contact to your roster. Click View -&gt; Offline contacts. And if you see your group listed, delete it and close the tab in your Messages window (if one is open for it). You may also need to restart Gajim. Repeat until it no longer shows up in your roster.<\/li>\n<\/ul>\n<p>Anyone interested in secure XMPP may also find the <a href=\"https:\/\/we.riseup.net\/riseup\/xmpp\">Riseup XMPP page<\/a> useful.<\/p>\n"},{"title":"Should we be pushing OpenPGP?","link":"https:\/\/current.workingdirectory.net\/posts\/2016\/openpgp\/","pubDate":"Thu, 15 Dec 2016 08:53:16 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2016\/openpgp\/","description":"<p>Bjarni R\u00fanar, the author of <a href=\"https:\/\/www.mailpile.is\">Mailpile<\/a> released a <a href=\"https:\/\/www.mailpile.is\/blog\/2016-12-13_Too_Cool_for_PGP.html\">blog about recent blogs disparaging OpenPGP<\/a>. It&rsquo;s a good read.<\/p>\n<p>There&rsquo;s one reason to support OpenPGP missing from the blog: OpenPGP protects you if your mail server is hacked. I&rsquo;m sure that <a href=\"https:\/\/en.wikipedia.org\/wiki\/2016_Democratic_National_Committee_email_leak\">Debbie Wasserman Schultz wishes she had been using OpenPGP<\/a>.<\/p>\n<p>Having said all of this&hellip; OpenPGP didn&rsquo;t make <a href=\"https:\/\/network.progressivetech.org\/online-protection\">my recent list of security tips<\/a>. That ommission is for two reasons:<\/p>\n<ul>\n<li>I&rsquo;ve never trusted my phone enough to store my OpenPGP keys on it. However, now that I am encrypting my data partition on the phone, should I re-consider? I use the K-9 email client which has had OpenPGP support for years, should I recommend that other people use K-9 and upload their keys to their phones? Suggesting that people use OpenPGP without the ability to use it on your phone seems like an empty suggestion. What about OpenPGP on the iPhone?<\/li>\n<li>I&rsquo;m waiting for Mailiple 1.0 to be released so I have a viable suggestion for how people can start using encryption now on their desktops. The complexity of using Thunderbird with Enigmail (and the <a href=\"http:\/\/kb.mozillazine.org\/Future_of_Thunderbird\">uncertain future of Thunderbird<\/a>) make it a hard sell. Should I re-consider? What about <a href=\"https:\/\/www.mailvelope.com\/\">Mailvelope<\/a>? Should I be encouraging people to use Mailvelope with their Gmail, etc. accounts?<\/li>\n<\/ul>\n"},{"title":"Wait... is that how you are supposed to configure your SSD card?","link":"https:\/\/current.workingdirectory.net\/posts\/2016\/ssd-discard\/","pubDate":"Thu, 08 Sep 2016 13:43:46 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2016\/ssd-discard\/","description":"<p>I bought a laptop with only SSD drives a while ago and based on a limited amount of reading, added the &ldquo;discard&rdquo; option to my \/etc\/fstab file for all partitions and happily went on my way expecting to avoid the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Trim_(computing)\">performance degradation problems that happen on SSD cards without this setting<\/a>.<\/p>\n<p>Yesterday, after a <a href=\"https:\/\/support.mayfirst.org\/ticket\/11870\">several month ordeal<\/a>, I finally installed SSD drives in one of <a href=\"https:\/\/mayfirst.org\/\">May First\/People Link&rsquo;s<\/a> servers and started doing more research to find the best way to set things up.<\/p>\n<p>I was quite surprised to <a href=\"http:\/\/blog.neutrino.es\/2013\/howto-properly-activate-trim-for-your-ssd-on-linux-fstrim-lvm-and-dmcrypt\/\">learn that my change in \/etc\/fstab accomplished nothing<\/a>. Well, not entirely true, my \/boot partition was still getting empty sectors reported to the SSD card.<\/p>\n<p>Since my filesystem is on top of LVM and LVM is on top of an encrypted disk, those messages from the files system to the disk were not getting through. I learned that when I tried to run the <code>fstrim<\/code> command on one of the partitions and received the message that the disk didn&rsquo;t support it. Since my \/boot partition is not in LVM or encrypted, it worked on \/boot.<\/p>\n<p>I then made the necessary changes to \/etc\/lvm\/lvm.conf and \/etc\/crypttab, restarted and&hellip; same result. Then I ran <code>update-initramfs -u<\/code> and rebooted and now fstrim works. I decided to remove the discard option from \/etc\/fstab and will set a cron job to run fstrim periodically.<\/p>\n<p>Also, I learned of some <a href=\"http:\/\/asalor.blogspot.de\/2011\/08\/trim-dm-crypt-problems.html\">security implications of using trim on an encrypted disk<\/a> which don&rsquo;t seem to outweigh the benefits.<\/p>\n"},{"title":"Trusted Mobile Device: How hard could it be?","link":"https:\/\/current.workingdirectory.net\/posts\/2016\/trusted-mobile-device\/","pubDate":"Wed, 31 Aug 2016 23:39:41 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2016\/trusted-mobile-device\/","description":"<p>I bought a new phone. After my experiences with <a href=\"https:\/\/current.workingdirectory.net\/posts\/2016\/signal\/\">signal<\/a> and the helpful comments readers gave regarding the ability to run android and signal without Google Play using <a href=\"https:\/\/microg.org\/\">microg<\/a> I thought I would give it a shot.<\/p>\n<p>Since microg reports that <a href=\"https:\/\/github.com\/microg\/android_packages_apps_GmsCore\/wiki\/Signature-Spoofing\">signature spoofing<\/a> is required and comes out-of-the-box with <a href=\"http:\/\/omnirom.org\/\">omnirom<\/a> I thought I&rsquo;d aim for installing omnirom&rsquo;s version of Android 6 (marshmallow) after years of using <a href=\"http:\/\/www.cyanogenmod.org\/\">cyanomgenmod<\/a>&rsquo;s version of Android.<\/p>\n<p>The Nexus line of phones seemed well-supported by omnirom in particular (and the alternative ROM community in general) so I bought a Nexus 5x.<\/p>\n<p>I carefully followed the <a href=\"https:\/\/docs.omnirom.org\/Installing_Omni_on_your_device\">directions for installing omnirom<\/a> however when it came time to boot into omnirom, I just got the boot sequence animation over and over again.<\/p>\n<p>Frustrated, I decided to go back to cyanogenmod and see if I could <a href=\"https:\/\/github.com\/microg\/android_packages_apps_GmsCore\/wiki\/Signature-Spoofing\">use one of the microg recommended methods for getting signature spoofing to work<\/a>. The easiest seemed to be <a href=\"https:\/\/github.com\/moosd\/Needle\">Needle by moosd<\/a> but alas <a href=\"https:\/\/github.com\/moosd\/Needle\/issues\/16\">no such luck with Marshmallow<\/a>. Someone else <a href=\"https:\/\/github.com\/ale5000-git\/tingle\/issues\/2\">forked the code and might fix it one day<\/a>. I then spent too much time trying to understand what <a href=\"http:\/\/repo.xposed.info\/\">xposed is<\/a> before I gave up understanding it and just tried to install it (<a href=\"http:\/\/repo.xposed.info\/module\/de.robv.android.xposed.installer\">woops, looks like the installer page is out of date<\/a> so instead I followed <a href=\"http:\/\/forum.xda-developers.com\/showthread.php?t=3034811\">sketchy instructions from a forum thread<\/a>). Well, to make a long story short it resulted in a boot loop.<\/p>\n<p>So, I decided to return to omnirom. After reading some vague references to omnirom and supersu, I decided to flash both of them together and voila, it worked!<\/p>\n<p>Next, I decided to enable full disk encryption. Not so fast. After clicking through the screens and hitting the final confirmation, my phone rebooted and spent the next 5 hours showing me the omnirom boot animation. Somehow, powering down and starting again resulted in a working machine, but no disk encryption.<\/p>\n<p>After much web searching, guessing and trial and error, I fixed the problem by clicking on the SuperSU option to &ldquo;Full unroot&rdquo; the device (I pressed &ldquo;no&rdquo; when prompted to attempt to restore stock image). Then I rebooted and followed the directions to encrypt the device. And it worked! Hooray!<\/p>\n<p>I had to reboot and re-flash the supersu to regain su privileges.<\/p>\n<p>All was great.<\/p>\n<p>The first root action I decided to take was to install the <a href=\"https:\/\/f-droid.org\/repository\/browse\/?fdfilter=cryptfs&amp;fdid=org.nick.cryptfs.passwdmanager\">cryptfs<\/a> program from <a href=\"https:\/\/f-droid.org\/\">f-droid<\/a> because using the same password to decrypt your device as you use to unlock the screen seems either tedious or insecure.<\/p>\n<p>That process didn&rsquo;t work so well. I got a message saying: use this command from a root shell before you reboot: <code>vdc cryptfs changepw &lt;password&gt;<\/code>. I followed the advice, carefully typing in my 12 character password which includes numbers and letters.<\/p>\n<p>Then, I happily did what I expected to be my last reboot when, to my horror, I was prompted to decrypt my disk with &hellip; a numeric-only keypad.<\/p>\n<p>That wasn&rsquo;t going to work. At this point I had already spent 5 days and about 8 hours on this project. Sigh. So, I started over.<\/p>\n<p>Guess what? It only took me 25 minutes but, it seems that cryptfs is broken. Even with a numeric password it fails. Ok, I guess I need a long pin to unlock my phone. This time it only took my 15 minutes to wipe and re-install everything.<\/p>\n<p>There are only two positive things I can think of:<\/p>\n<ul>\n<li><a href=\"https:\/\/twrp.me\/\">TWRP<\/a>, which provides the recovery image, is really great. Everytime something went wrong I booted into the TWRP recovery image and could fix anything.<\/li>\n<li>I&rsquo;m starting to get used to the error on startup warning me that &ldquo;Your device is corrupt. It can&rsquo;t be trusted and may not work properly.&rdquo; It&rsquo;s a good thing to remember about all digital devices.<\/li>\n<\/ul>\n<p>p.s. I haven&rsquo;t even tried to install microg yet&hellip; which was the whole point.<\/p>\n"},{"title":"Networking in 2016","link":"https:\/\/current.workingdirectory.net\/posts\/2016\/networking-in-2016\/","pubDate":"Sun, 21 Aug 2016 10:20:03 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2016\/networking-in-2016\/","description":"<p>So many options, so little time.<\/p>\n<p>Many years ago I handled all my network connections via \/etc\/network\/interfaces.<\/p>\n<p>Then, I was in desperate need of a Internet connection and all I had was a borrowed USB cell phone modem. My friends running <a href=\"https:\/\/wiki.gnome.org\/Projects\/NetworkManager\">NetworkManager<\/a> just plugged the stick in and were online. I was left with the task of figuring out how to manually configure this piece of hardware without being online. I ended up borrowing my friend&rsquo;s computer. Then, when I got home, I installed NetworkManager.<\/p>\n<p>Once I had NetworkManager installed, I decided it was easier to find, connect to and manage passwords of wireless networks using a graphical tool rather than digging through the copious output of commands run from my terminal and trying to keep track of the passwords. So long wireless.<\/p>\n<p>Then I had to help a colleague get on our Virtual Private Network. Wow. There&rsquo;s a NetworkManager GUI for that too. If I&rsquo;m going to support my colleauge with this tool&hellip; I guess I should use it as well. I also managed to write a dispatcher script in \/etc\/NetworkManager\/dispatcher.d that calls <code>su -c &quot;\/usr\/bin\/smbnetfs \/media\/smbnetfs&quot; -l jamie<\/code> when it receives and action of &ldquo;vpn-up&rdquo; and <code>umount \/media\/smbnetfs 2&gt;\/dev\/null<\/code> on &ldquo;vpn-down.&rdquo; Now I can mount the samba share by simply connecting to the VPN via NetworkManager.<\/p>\n<p>My cable connections are almost always configured using DHCP. Almost everything else is in NetworkManager, why not move enp1s0f2 as well?<\/p>\n<p>What&rsquo;s left? My final piece is my bridge. I still <a href=\"https:\/\/current.workingdirectory.net\/posts\/2012\/managing-kvm\/\">run and manage my own KVM guests<\/a> and I have a bridge to handle that traffic. I first decided to move this functionality to systemd.network because systemd can not only handle the bridge, but can also handle IP Forwarding, DHCP service, and best of all, IP Masquerading. Well, almost&hellip; not IP Masquerading after all, <a href=\"https:\/\/bugs.debian.org\/cgi-bin\/bugreport.cgi?bug=787480\">at least not yet<\/a>.<\/p>\n<p>Without IP masquerading, I figured I&rsquo;d go with NetworkManager. Having all networking in the same place gives me an illusion of control at best and at worst makes it easier to debug. So, I setup a crufty script in \/etc\/NetworkManager\/dispatcher.d that configures masquerading via <code>iptables<\/code> everytime either my wireless or wired network goes up or down, which I&rsquo;m not crazy about. Maybe when <a href=\"https:\/\/bugs.debian.org\/cgi-bin\/bugreport.cgi?bug=787480\">#787480<\/a> is fixed I&rsquo;ll got back to systemd. I also edited \/etc\/sysctl.conf to enable #net.ipv4.ip_forward=1. Then I changed it back and added my own file in \/etc\/sysctl.d to do the same thing. Then I deleted that file and added <code>sysctl net.ipv4.ip_forward=1<\/code> and <code>sysctl net.ipv4.ip_forward=0<\/code> to my crufty dispatcher script. I decided to do without DHCP - I can manually configure the few KVM instances that I run.<\/p>\n<p>Now \/etc\/network\/interfaces is so lonely.<\/p>\n<h2 id=\"update-2017-08-17\">Update 2017-08-17<\/h2>\n<p>For some reason placing <code>su -c &quot;\/usr\/bin\/smbnetfs \/media\/smbnetfs&quot; -l jamie<\/code> in my <code>\/etc\/NetworkManager\/dispatcher.d\/<\/code> script stopped working. According to <code>journalctl<\/code>:<\/p>\n<pre><code>Aug 21 10:01:13 turkey systemd[1]: session-c18.scope: Killing process 27385 (smbnetfs) with signal SIGTERM.\nAug 21 10:01:13 turkey systemd[1]: session-c18.scope: Killing process 27390 (smbnetfs) with signal SIGTERM.\nAug 21 10:01:13 turkey systemd[1]: session-c18.scope: Killing process 27391 (smbnetfs) with signal SIGTERM.\n<\/code><\/pre>\n<p>I have no idea why and wasn&rsquo;t really sure how to debug it (running the command <code>su -c &quot;\/usr\/bin\/smbnetfs \/media\/smbnetfs&quot; -l jamie<\/code> via a root terminal was successful).<\/p>\n<p>So I decided to create a system service:<\/p>\n<pre><code>0 jamie@turkey:~$ cat \/etc\/systemd\/system\/mount-borges.service\n[Unit]\nDescription=Mount PTP's SMB file server borges\n\n[Service]\nUser=jamie\nExecStart=\/usr\/bin\/smbnetfs \/media\/smbnetfs -f\nExecStop=\/bin\/umount \/media\/smbnetfs\n0 jamie@turkey:~$\n<\/code><\/pre>\n<p>And now instead of calling <code>su -c &quot;\/usr\/bin\/smbnetfs \/media\/smbnetfs&quot; -l jamie<\/code> on vpn-up and <code>umount \/media\/smbnetfs 2&gt;\/dev\/null<\/code> on vpn-down I call <code>systemctl start mount-borges<\/code> and <code>systemctl stop mount-borges<\/code> instead.<\/p>\n"},{"title":"Nice Work Apertium","link":"https:\/\/current.workingdirectory.net\/posts\/2016\/nice-work-apertium\/","pubDate":"Wed, 17 Aug 2016 10:07:36 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2016\/nice-work-apertium\/","description":"<p>For the last few years I have been periodically testing out <a href=\"https:\/\/www.apertium.org\/\">apertium<\/a> and today I did again and was pleasantly surprised with the quality of the english-spanish and spanish-english translations (and also their nifty web site translator).<\/p>\n<p>So, I dusted off some of my geeky code to make it easier to use and continue testing.<\/p>\n<p>For starters&hellip;<\/p>\n<pre><code>\tsudo apt-get install apertium-en-es xclip coreutils\n<\/code><\/pre>\n<p>Then, I added the following to my .muttrc file:<\/p>\n<pre><code>\tmacro pager &lt;F2&gt; &quot;&lt;enter-command&gt;set pipe_decode&lt;enter&gt;&lt;pipe-entry&gt; sed '1,\/^$\/d' | apertium es-en | less&lt;enter&gt;&lt;enter-command&gt;unset pipe_decode&lt;enter&gt;&quot; &quot;translate from spanish&quot;\n<\/code><\/pre>\n<p>If you press F2 while reading a message in spanish it will print out the English translation.<\/p>\n<p>If you use vim, you can create ~\/.vim\/plugins\/apertium.vim with:<\/p>\n<pre><code>\tfunction s:Translate()\n\t\tsilent !clear\n\t\texecute &quot;! apertium en-es &quot; . bufname(&quot;%&quot;) . &quot; | tee &gt;(xclip)&quot;\n\tendfunction\n\tcommand Translate :call &lt;SID&gt;Translate()\n<\/code><\/pre>\n<p>Then, you can type the command:<\/p>\n<p>:Translate<\/p>\n<p>And it will display the English to Spanish translation of the file you are editing and copy the translation into your clip board so you can paste it into your document.<\/p>\n"},{"title":"Noam use Gnome","link":"https:\/\/current.workingdirectory.net\/posts\/2016\/noam-uses-gnome\/","pubDate":"Sat, 13 Aug 2016 21:04:54 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2016\/noam-uses-gnome\/","description":"<p>I don&rsquo;t quite remember when I read <a href=\"http:\/\/changelog.complete.org\">John Goerzen<\/a>&rsquo;s post about <a href=\"http:\/\/changelog.complete.org\/archives\/6915-a-4-year-old-linux-command-line-and-microphone\">teaching a 4 year old to use the linux command line with audio<\/a> on <a href=\"http:\/\/planet.debian.org\/\">planet Debian<\/a>. According to the byline it was published nearly 2 years before Noam was born, but I seem to remember reading it in the weeks after his birth when I was both thrilled at the prospect of teaching my kid to use the command line and, in my sleepless stupor, not entirely convinced he would ever be old enough.<\/p>\n<p>Well, the time came this morning. He found an old USB key board and discovered that a green light came on when he plugged it in. He was happily hitting the keys when Meredith suggested we turn on the monitor and open a program so he could see the letters appear on the screen and try to spell his name.<\/p>\n<p>After 10 minutes in Libre Office I remembered John&rsquo;s blog and was inspired to start writing a bash script in my head (I would have to stop the fun with Libre Office to write it so the pressure was on&hellip;). In the end it was only a few minutes and I came up with:<\/p>\n<pre><code>#!\/bin\/bash\n\nwhile [ 1 ]; do\n  read -p &quot;What shall I say? &quot;\n  espeak &quot;$REPLY&quot;\ndone\n<\/code><\/pre>\n<p>It was a hit. He said what he wanted to hear and hit the keys, my job was to spell for him.<\/p>\n<p><img src=\"https:\/\/current.workingdirectory.net\/posts\/2016\/noam-uses-gnome\/noam-uses-gnome-terminal.png\" alt=\"Noam uses Gnome Terminal to talk\"  \/>\n<\/p>\n<p>Oh, also: he discovered key combinations that did things that were unsurprising to me (like taking the screen grab above) and also things that I&rsquo;m still scratching my head about (like causing a prompt on the screen that said: &ldquo;Downloading shockwave plugin.&rdquo; No thanks. And, how did he do that?<\/p>\n"},{"title":"Monitoring Deflect","link":"https:\/\/current.workingdirectory.net\/posts\/2016\/monitoring-deflect\/","pubDate":"Tue, 19 Jul 2016 13:26:13 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2016\/monitoring-deflect\/","description":"<p><a href=\"https:\/\/mayfirst.org\/\">May First\/People Link<\/a> has several members that are targets of politically motivated denial of service attacks (mostly groups that support reproductive justice for women and palestinian rights). To fight off the attacks, we work closely with <a href=\"https:\/\/deflect.ca\/\">Deflect<\/a> - a non-governmental organization based in Canada that fights against this kind of censorship.<\/p>\n<p>When a site is down, it&rsquo;s not always easy to understand why. Deflect runs as many as 5 edge servers, any of them could be down. And, of course, the origin server could also be down.<\/p>\n<p>I tried using a commericial\/free as in beer service for monitoring up time, but when it reported the site being down, I had no idea which part was down.<\/p>\n<p>Here are few of my tips.<\/p>\n<p>First, you usually want to lookup the IP addresses reported by DNS to you can get a list of your edge servers:<\/p>\n<pre><code>dig +short mayfirst.org\n<\/code><\/pre>\n<p>Then, you can use httping to test each one, e.g.:<\/p>\n<pre><code> httping -l --divert-connect 31.24.34.89 https:\/\/mayfirst.org\/\n<\/code><\/pre>\n<p>The -l says to use https (seems redundant if you specify https as the target, but if you don&rsquo;t put -l, httping will complain).<\/p>\n<p>The &ndash;divert-connect option is particularly useful with your origin IP - because you can compare response times and ensure that your origin is up.<\/p>\n<p>This is all fine and good for generally testing your edges, but sometimes it&rsquo;s a particular page that is causing the problem. We&rsquo;ve found that edges sometimes cache error pages, even after the error has gone away. But&hellip; how can you prove it?<\/p>\n<pre><code>curl --resolve mayfirst.org:443:31.24.34.89 --head https:\/\/mayfirst.org\/en\/contact\/\n<\/code><\/pre>\n<p>The &ndash;resolve argument is the key here - it allows you to specify how a domain should be resolved (with this option, curl does not perform a DNS lookup). Replace the IP address with the origin IP address and you have a nice comparison between the edge and the origin. The caching server should respond with a Via header to confirms it is caching. And, of course, the first line shows you the http response which should be the same on the origin and caching server.<\/p>\n<p>I&rsquo;ve written the script below to help debug sites that may not be working properly. It combines both the httping and curl options.<\/p>\n<pre><code>a-reverse-proxy-test domain.org [origin-ip]\n<\/code><\/pre>\n<p>Here&rsquo;s the code:<\/p>\n<pre><code>#!\/bin\/bash\n\n# Test all given edges using httping and curl.  Pass the full protocol, domain\n# and page you want to test as the first argument. Optionally pass the origin\n# IP address as the second argument (so you can tell if your origin is down).\n# For example:\n# \n# a-reverse-proxy-test https:\/domain.org\/contact 1.2.3.4\n#\n\nurl=&quot;$1&quot;\norigin=&quot;$2&quot;\n\nif [ -z &quot;$url&quot; ]; then\n  printf &quot;Please pass the url as first argument.\\n&quot;\n  exit 1\nfi\nproto=$(echo &quot;$url&quot; | sed -E &quot;s#(https?).*#\\1#&quot;)\n\n\nhost=$(echo &quot;$url&quot; | sed -E &quot;s#https?:\/\/([^\/]+)\/?.*#\\1#&quot;)\n\nif [ -z &quot;$proto&quot; ]; then\n  printf &quot;Please include http or https as part of the url.\\n&quot;\n  exit 1\nfi\n\nif [ -z &quot;$host&quot; ]; then\n  printf &quot;Failed to extract the host. If passing just host, ensure it ends in a \/, e.g. https:\/\/mayfirst.org\/.\\n&quot;\n  exit 1\nfi\n\nif ! ping -c 1 4.2.2.1 &gt;\/dev\/null; then\n  # printf &quot;We are offline. Not running.\\n&quot;\n  exit 1\nfi\n\nips=$(dig +short &quot;$host&quot;)\nif [ &quot;$?&quot; -ne &quot;0&quot; ]; then\n  # printf &quot;DNS lookup failure. Not running.\\n&quot;\n  exit 1\nfi\nif [ -n &quot;$origin&quot; ]; then\n  ips=&quot;$ips $origin&quot;\nfi\n\nl=\nport=80\nif [ &quot;$proto&quot; = &quot;https&quot; ]; then\n  l=-l\n  port=443\nfi\n\nfor ip in $ips; do\n  printf &quot;Testing IP: %s\\n&quot; &quot;$ip&quot;\n  printf &quot;httping response: %s\\n&quot; &quot;$out&quot;\n  httping $l -m -t 5 -c 1 --divert-connect &quot;$ip&quot; &quot;$url&quot;\n  printf &quot;curl HTTP and Via headers:\\n&quot;\n  curl --resolve &quot;$host:$port:$ip&quot; --head &quot;$url&quot; 2&gt;&amp;1 | egrep &quot;^(HTTP|Via)&quot;\n  printf &quot;\\n&quot;\ndone\n<\/code><\/pre>\n<p>To help with regular monitoring (using <code>httping<\/code>) I wrote a script that runs via a cron job and outputs the results to a log file.<\/p>\n<pre><code>\t#!\/bin\/bash\n\n\t# Test all given edges \n\n\tdomain=&quot;$1&quot;\n\torigin=&quot;$2&quot;\n\tproto=http\n\tif [ -n &quot;$3&quot; ]; then\n\t\tproto=&quot;$3&quot;\n\tfi\n\n\tif [ -z &quot;$domain&quot; ]; then\n\t\tprintf &quot;Please pass the domain as first argument.\\n&quot;\n\t\texit 1\n\tfi\n\n\tif ! ping -c 1 4.2.2.1 &gt;\/dev\/null; then\n\t\t# printf &quot;We are offline. Not running.\\n&quot;\n\t\texit 1\n\tfi\n\n\tips=$(dig +short &quot;$domain&quot;)\n\tif [ &quot;$?&quot; -ne &quot;0&quot; ]; then\n\t\t# printf &quot;DNS lookup failure. Not running.\\n&quot;\n\t\texit 1\n\tfi\n\tif [ -n &quot;$origin&quot; ]; then\n\t\tips=&quot;$ips $origin&quot;\n\tfi\n\n\tl=\n\tif [ &quot;$proto&quot; = &quot;https&quot; ]; then\n\t\tl=-l\n\tfi\n\n\tfor ip in $ips; do\n\t\tdate=$(date +%Y.%m.%d-%H:%M)\n\t\tfor i in 1 2 3; do\n\t\t\tout=$(httping $l -m -t 5 -c 1 --divert-connect &quot;$ip&quot; &quot;$proto:\/\/$domain&quot;)\n\t\t\t[ -z &quot;$out&quot; ] &amp;&amp; out=1\n\t\t\tprintf &quot;%s %s %s\\n&quot; &quot;$date&quot; &quot;$ip&quot; &quot;$out&quot;\n\t\tdone\n\tdone\n<\/code><\/pre>\n"},{"title":"Signal and Mobile XMPP Update","link":"https:\/\/current.workingdirectory.net\/posts\/2016\/signal-mobile-update\/","pubDate":"Sat, 04 Jun 2016 23:03:27 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2016\/signal-mobile-update\/","description":"<p>First, many thanks to <a href=\"http:\/\/planet.debian.org\">Planet Debian<\/a> readers for your thoughtful and constructive feedback to my <a href=\"https:\/\/current.workingdirectory.net\/posts\/2016\/signal\/\">Signal<\/a> and <a href=\"https:\/\/current.workingdirectory.net\/posts\/2016\/mobile-instant-messaging\/\">Mobile Instant Messaging<\/a> blogs. I learned a lot.<\/p>\n<p>Particularly useful was the comment directing me to <a href=\"https:\/\/gultsch.de\/xmpp_2016.html\">Daniel Gultsch&rsquo;s The State of Mobile in 2016<\/a> post.<\/p>\n<p>I had previously listed the outstanding technical challenges as:<\/p>\n<ul>\n<li>Implement end-to-end encryption<\/li>\n<li>Receive messages the moment they are sent without draining the battery<\/li>\n<\/ul>\n<p>I am now fairly convined that both problems are well-solved on Android via the <a href=\"https:\/\/conversations.im\/\">Conversations<\/a> app and a <a href=\"https:\/\/support.mayfirst.org\/ticket\/11822\">well-tuned XMPP server<\/a> (I had no idea how easy it was to <a href=\"https:\/\/prosody.im\/doc\/installing_modules\">install your own Prosody modulues<\/a> &ndash; the client state indicator module is only 22 lines of lua code!)<\/p>\n<p>I think the current technical challenges could be better summarized as: adding iOS (iPhone) support. Both end-to-end encryption and receiving messages consistently seem to be hurdles. However, it seems that Chris Ballinger and the Chat Secure team are well on their way toward solving the <a href=\"https:\/\/chatsecure.org\/blog\/chatsecure-v32-push\/\">push issue<\/a> and facing <a href=\"https:\/\/github.com\/ChatSecure\/ChatSecure-iOS\/issues\/376#issuecomment-218902284\">funder skittishness<\/a> on the encryption front. Nonetheless, but <em>seem<\/em> to be progressing.<\/p>\n<p>With the obvious technical hurdles in progress, we have the luxury of talking about the less obvious ones - particularly the ones requiring trade-offs.<\/p>\n<p>In particular: Signal replaces your SMS client. It looks and feels like an SMS client and automatically sends un-encrypted messages to everyone your address book that is not on signal and sends encrypted messages to those that are on signal.<\/p>\n<p>The significance of this feature is hard to over-state. It differentiates tools by and for technically minded people and those designed for a mass audience.<\/p>\n<p>When I convince people to use Conversations, in contrast, I have to teach them to:<\/p>\n<ul>\n<li>Create an entirely new address book by entering addresses for your friends that you don&rsquo;t already have<\/li>\n<li>Use a new and different app for sending encrypted messages<\/li>\n<\/ul>\n<p>For most people who don&rsquo;t (yet) have their friends XMPP addresses or for people who don&rsquo;t have any friends who use XMPP, it means that they will install it, send me a few messages and then never use it again.<\/p>\n<p>The price Signal pays for this convenience is steep: Signal seems to synchronize your entire address book to their servers so they can keep a map of cell phone numbers to signal users. It&rsquo;s not only creepy (I get a text message everytime someone in my address book joins Signal) but it&rsquo;s flies in the face of expectations for a privacy-minded application.<\/p>\n<p>How could we take advantage of this feature, without the privacy problems?<\/p>\n<p>What if&hellip;<\/p>\n<ul>\n<li>Our app could send both XMPP messages and SMS messages<\/li>\n<li>Everytime you added a new XMPP contact, it added the contact to your address book with a new XMPP field<\/li>\n<li>Anytime you send a message to a contact with an XMPP field filled in, it would send via XMPP and otherwise it would send a normal SMS message<\/li>\n<\/ul>\n<p>The main downside (which Signal faces as well) is that you have to contend with the complexities of sending SMS messages on top of the work needed to write a well-functioning XMPP client. As I mentioned in my <a href=\"https:\/\/current.workingdirectory.net\/posts\/2016\/signal\/\">Signal<\/a> blog, there are no shortage of <a href=\"https:\/\/github.com\/WhisperSystems\/Signal-Android\/issues?q=is%3Aissue+is%3Aopen+mms+label%3Amms\">MMS bugs against Signal<\/a>. Nobody wants that head-ache.<\/p>\n<p>Additinally, we would still lose one Signal feature: with Signal, when a user joins, everyone automatically sends them encrypted messages. With this proposed app, each user would have to manually add the XMPP address and have no way of knowing when one of their friends gets an XMPP address.<\/p>\n<p>Any other ideas?<\/p>\n"},{"title":"Mobile Instant Messaging","link":"https:\/\/current.workingdirectory.net\/posts\/2016\/mobile-instant-messaging\/","pubDate":"Wed, 01 Jun 2016 21:04:22 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2016\/mobile-instant-messaging\/","description":"<p>Now that I&rsquo;ve <a href=\"https:\/\/current.workingdirectory.net\/posts\/2016\/signal\/\">gone down the signal road<\/a>, when can I get off of it?<\/p>\n<p>The two contenders I&rsquo;ve found for more politically conscious mobile-friendly instant messaging are <a href=\"https:\/\/tox.chat\/\">Tox<\/a> and <a href=\"https:\/\/en.wikipedia.org\/wiki\/XMPP\">XMPP<\/a>.<\/p>\n<p>And the problems to solve are:<\/p>\n<ul>\n<li>Implement end-to-end encryption<\/li>\n<li>Receive messages the moment they are sent without draining the battery<\/li>\n<\/ul>\n<p><a href=\"https:\/\/tox.chat\">The Tox Project<\/a> has a great design (peer-to-peer) and <a href=\"https:\/\/tox.chat\/faq.html#how-tox-privacy\">handles end-to-end encryption<\/a> as part of its core design.<\/p>\n<p>However, <a href=\"https:\/\/github.com\/Antox\/Antox\/issues\/159\">the Antox mobile client has not only failed to solve the battery drain problem but seems to also have a serious bandwidth issue as well<\/a>. Also, what is up with this <a href=\"https:\/\/blog.tox.im\/2016\/04\/01\/litigation\/\">drama<\/a>?<\/p>\n<p>How about XMPP?<\/p>\n<p>For years, XMPP clients have been using <a href=\"https:\/\/en.wikipedia.org\/wiki\/Off-the-Record_Messaging\">OTR<\/a> - and many instant messaging applications support it, including <a href=\"https:\/\/chatsecure.org\/\">ChatSecure<\/a>. And now, there seems to be quite a bit of excitement around implementing a <a href=\"https:\/\/en.wikipedia.org\/wiki\/OMEMO\">better protocol called OMEMO<\/a>. Incidentally, the OMEMO protocol uses the same <a href=\"https:\/\/en.wikipedia.org\/wiki\/Double_Ratchet_Algorithm\">Double Ratchet (previously referred to as the Axolotl Ratchet)<\/a> protocol that Signal uses.<\/p>\n<p>While there are some <a href=\"https:\/\/github.com\/anurodhp\/Monal\/issues\/9#issuecomment-208063040\">irritating hiccups<\/a> around using the available GPL library for Double Ratchet on iPhone apps, it seems like these issues are being sorted out and soon we&rsquo;ll have some kind of standard way for XMPP clients to exchange end-to-end encrypted messages.<\/p>\n<p>Let&rsquo;s move on to notifications and battery drain.<\/p>\n<p>If you believe the author of the Android Conversations App <a href=\"https:\/\/conversations.im\/#optimizations\">there doesn&rsquo;t seem to be a problem<\/a>. However, <a href=\"https:\/\/xmpp.org\/extensions\/xep-0352.html\">client state indication<\/a> seems to be the only issue related to battery drainage he addresses and <a href=\"https:\/\/support.mayfirst.org\/ticket\/11822#comment:1\">I couldn&rsquo;t figure out how to easily install that extension on our Debian Jessie prosody instance<\/a> since it isn&rsquo;t included in the <a href=\"https:\/\/packages.debian.org\/jessie-backports\/prosody-modules\">prosody-modules package in Debian<\/a> and it requires <a href=\"https:\/\/modules.prosody.im\/mod_csi.html\">additional modules for it to work<\/a>.<\/p>\n<p>Chris Ballinger, the author of ChatSecure (iOS and Android XMPP app), is <a href=\"https:\/\/chatsecure.org\/blog\/fixing-the-xmpp-push-problem\/\">less optimistic<\/a> about this problem in general. In short, he thinks we need a proper &ldquo;push&rdquo; mechanism and has even <a href=\"https:\/\/chatsecure.org\/blog\/chatsecure-v32-push\/\">started to implement one<\/a>. At the same time, a new (and different) XMPP standard called <a href=\"https:\/\/xmpp.org\/extensions\/xep-0357.html\">Push Notifications - XEP-0357<\/a> has been released and it is <a href=\"https:\/\/modules.prosody.im\/mod_cloud_notify.html\">even implemented in Prosody<\/a> (although not yet available in Debian).<\/p>\n<p>So the future seems bright, right? Well, not exactly. All of this &ldquo;push&rdquo; activity seems to solve this problem: A federated\/decentralized application cannot properly use <a href=\"https:\/\/en.wikipedia.org\/wiki\/Apple_Push_Notification_Service\">Apple&rsquo;s APNs<\/a> or Google&rsquo;s <a href=\"https:\/\/en.wikipedia.org\/wiki\/Google_Cloud_Messaging\">GCM<\/a>. In the <a href=\"https:\/\/chatsecure.org\/blog\/fixing-the-xmpp-push-problem\/\">words<\/a> of Chris Ballinger:<\/p>\n<blockquote>\n<p>The biggest problem is that there is no easy way to send push messages\nbetween my app and your app. For me to send a push to one of your app\u2019s users\non iOS, I must first obtain an APNs SSL certificate\/key pair from you, and\none of your user\u2019s \u2018push token\u2019 that uniquely identifies their device to\nApple. These push tokens are potentially sensitive information because they\nallow Apple to locate your device (in order to send it a push).<\/p><\/blockquote>\n<p>So, even if we manage to get one of these two standards for push notifications up and running, we have only succeeded in solving Signal&rsquo;s centralization problem, not the dependence on Google Play Services and Apple Push Network (in fact it&rsquo;s quite mysterious to me <a href=\"https:\/\/hg.prosody.im\/prosody-modules\/file\/218a3d3f7f97\/mod_cloud_notify\/README.markdown\">how you could even use the Prosody implementation of Push Notifications with GCM or APN<\/a>).<\/p>\n<p>So&hellip; what we really would need would be to figure out how to implement one of these two push standards and then get it to work with an alternative to GCM and APN (perhaps <a href=\"http:\/\/mqtt.org\/\">MQTT<\/a>)? Which, I think <a href=\"https:\/\/ollieparsley.com\/2013\/05\/20\/using-mqtt-as-a-gcm-replacement-for-android-push-notifications\/\">would require changes to the XMPP client<\/a>.<\/p>\n<p>Geez. I may be on Signal longer than I planned.<\/p>\n"},{"title":"Signal and Google Cloud Services","link":"https:\/\/current.workingdirectory.net\/posts\/2016\/signal\/","pubDate":"Wed, 01 Jun 2016 19:08:13 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2016\/signal\/","description":"<p>I just installed <a href=\"https:\/\/whispersystems.org\/blog\/signal\/\">Signal<\/a> on my Android phone.<\/p>\n<p>It wasn&rsquo;t an easy decision. I have been running <a href=\"http:\/\/www.cyanogenmod.org\/\">Cyanogenmod<\/a>, a Google-free version of Android, and installing apps from <a href=\"https:\/\/f-droid.org\/\">F-Droid<\/a>, a repository of free software android apps, for several years now. This setup allows me to run all the applications I need without Google accessing any of my cell phone data. It has been a remarkably successful experiment leaving me with all the phone software I need. And it&rsquo;s consistent with my belief that Google&rsquo;s size, reach and goals are a menace to the left&rsquo;s ability to develop the autonomous communications systems on the Internet that we need to achieve any meaningful political change.<\/p>\n<p>However, if I want to install Signal, I <a href=\"https:\/\/github.com\/WhisperSystems\/Signal-Android\/issues\/560\">have to<\/a> install Google Play, and the only way to install Google Play is to install Google&rsquo;s base line set of apps and to connect your cell phone to a Gmail account.<\/p>\n<p>Why in the world would Signal require Google Play? There is plenty of discussion of the technical debate on this topic, but politically it boils down to this: security is about trade-offs, and the trade-offs you find important are based on your politics. While I consider Signal to be on the same team in the big picture, I think Signal&rsquo;s winning a short term victory for more massively adopted end-to-end encryption at the expense of a longer term and more important struggle for autonomous communication systems specifically for communities fighting corporate power (of which Google itself is an important target) and fighting US hegemony on a global scale.<\/p>\n<p>Furthermore, Signal&rsquo;s lead developer&rsquo;s <a href=\"https:\/\/github.com\/LibreSignal\/LibreSignal\/issues\/37#issuecomment-217211165\">outright hostility to alternate clients connecting to the centralized signal servers<\/a> demonstrates another political decision that favors central control over any confidence in a broader movement control over what could have become a power new protocol for secure communication. And his <a href=\"https:\/\/github.com\/anurodhp\/Monal\/issues\/9#issuecomment-208063040\">refusal to grant an exemption for other developers to use just the encryption algorythm<\/a> is frustrating to say the least.<\/p>\n<p>Given this reasoning, why install Signal? The main reason is because I have yet to convince anyone to remove Google Apps from their phone and Signal, right now, represents a dramatic improvement over most people&rsquo;s current communications habit. And, when it comes down to it, I need to run what I recommend.<\/p>\n<p>I&rsquo;m still running both <a href=\"https:\/\/conversations.im\/\">conversations<\/a> and <a href=\"https:\/\/tox.chat\/\">Antox<\/a> which are far better alternatives in the long run. However until they gain more widespread adoption, I&rsquo;ll be experimenting with Signal.<\/p>\n<p>Technical Details<\/p>\n<p>Oh, and by the way, installing Google Apps nearly bricked my phone.<\/p>\n<p>Cyanogenmod&rsquo;s web site conveniently <a href=\"https:\/\/wiki.cyanogenmod.org\/w\/Google_Apps\">provides instructions<\/a> for installing Google Apps. However, confusingly, they provide two different links to choose from, one is from <a href=\"http:\/\/opengapps.org\">OpenGapps<\/a> which provided my a link to download the zip file to flash on a non-https page (the link itself was over https). The other link was to an https-enabled page <a href=\"https:\/\/www.androidfilehost.com\/\">on androidfilehost.com<\/a> that offered a non-https download (but did provide a md5 checksum). I am now sure why people offering software downloads don&rsquo;t enable https from start to finish (well, maybe I do - I haven&rsquo;t yet enabled https on this site&hellip;).<\/p>\n<p>However, more confusing is that both links were to different files. The OpenGapps one seemed to be a daily build and the androidfilehost was to a file with the date 20140606 in it&rsquo;s name, suggesting it was built nearly 2 years ago.<\/p>\n<p>I went with the daily build.<\/p>\n<p>When I restarted, I got the error &ldquo;Unfortunately, Android Keyboard (Aosp) Has Stopped.&rdquo; If you search the web for this error you will see loads of people getting it. However, none of them seem to be using an encrypted disk. Yes, that is a bigger problem since you can&rsquo;t enter your encrypted disk passphrase if your keyboard app has crashed and you can&rsquo;t boot your phone if you can&rsquo;t even hit enter at the passphrase prompt. If you can&rsquo;t boot, you can&rsquo;t clear the keyboard app cache or most of the suggestions. In fact, when you press and hold the power key you don&rsquo;t even get the option to reboot into recovery mode. And, if you connect your device to your USB cable and run the <code>adb<\/code> tool on your computer, the tool reports that you are not authorized to connect your device.<\/p>\n<p>Oh damn. Did I just brick my phone?<\/p>\n<p>Fortunately, you can still boot into recovery mode on a Samsung S4 by powering it off. Then, press and hold the up volume button while turning it on.<\/p>\n<p>In recovery mode, I as able to convince the adb tool to connect to my device and I copied over the other Gapps zip file from androidfilehost.com and flashing that one seems to have fixed the problem.<\/p>\n<p>Once I booted, I ran Google Play and opted to create a new Google Account. I chose the option to not sync my data. Then, I checked in Settings -&gt; Accounts I saw that a Google Account was there and was synchronizing. Great. What was it synchronizing? I clicked the account, then clicked &ldquo;Accounts and Privacy&rdquo; and ensured that everything was turned off. Let&rsquo;s hope that works.<\/p>\n<p>[Update]<\/p>\n<p>Signal&rsquo;s option to take over as your default SMS client and send un-encrypted normal SMS messages while sending encrypted messages to other Signal users is a very good way to smooth adoption. Unfortunately I had some problems with MMS message for which I <a href=\"https:\/\/github.com\/WhisperSystems\/Signal-Android\/issues\/4878#issuecomment-221449778\">found a work-around<\/a>. But sheesh, <a href=\"https:\/\/github.com\/WhisperSystems\/Signal-Android\/issues?q=is%3Aissue+is%3Aopen+mms+label%3Amms\">lots of MMS problems<\/a> at the moment.<\/p>\n"},{"title":"Docker: evil spawn or useful tool","link":"https:\/\/current.workingdirectory.net\/posts\/2015\/docker-evil\/","pubDate":"Sat, 30 Jan 2016 23:44:07 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2015\/docker-evil\/","description":"<p>There are plenty of criticisms of <a href=\"https:\/\/docker.com\">docker<\/a>, the system for building a <a href=\"https:\/\/en.wikipedia.org\/wiki\/Operating-system-level_virtualization\">container<\/a>-based virtual machine running just a single application. I&rsquo;ve read many of them have have consistently been either in agreement or at least amused.<\/p>\n<p>The most relevant criticism is about the basic approach of building single-application virtual machines. To understand this criticism, let&rsquo;s remember - in traditional application deployment there are at least three distinct jobs:<\/p>\n<ul>\n<li>Distrubtion developers - the team the integrates all the various packages available into a coherent and well-functioning system and, importantly, monitors the upstream of all the packages to ensure that security bugs are properly packaged and made easily available for installation. These people don&rsquo;t give a shit about your PHP parse error and don&rsquo;t really care if you can&rsquo;t figure out the command to create a new Postgres Database.<\/li>\n<li>System administrators - these are the people that ensure that the security updates provided by the distribution developers get installed on a regular basis and if something breaks during this process, they are the ones that fix it. They know how to create your new postgres database. They don&rsquo;t really care about your PHP parse error either, but may get roped in to tell you it&rsquo;s a PHP parse error.<\/li>\n<li>Application developers - these are the people that care about the PHP parse error. They also know how to create beautiful things that end users can interact with.<\/li>\n<\/ul>\n<p>These three groups have lived in happy tension for years.<\/p>\n<p>Now we have containers.<\/p>\n<p>The problem with containers is that suddenly the system administrators are out entirely and the distribution developers&rsquo; role may have been dramatically minimized or circumvented altogether. Meanwhile, the application developer is both liberated from any constraints imposed by either the system administrator or the distribution developers (hooray!) but is also saddled with the enormous responsibilities of these two roles - which may not be apparent to the application developer at first.<\/p>\n<p>I have practically no distribution developer experience and have considerable experience as a system administrator and an application developer. And, it took me months to sort through how to properly develop and deploy an application using docker in a way that I thought was responsible and secure.<\/p>\n<p>I started by learning about how docker wants you to deploy images - by downloading them from their <a href=\"https:\/\/registry.hub.docker.com\/\">shared registry<\/a>. As I mentioned, I have very little experience in the realm of distribution development, but I at least know enough about Debian to know that a lot of time and thought has gone into cryptographically verifying packages that I install, which apparently is <a href=\"https:\/\/lwn.net\/Articles\/628343\/\">not done at all with Docker images<\/a>. Is that obvious to everyone using Docker??<\/p>\n<p>Fortunately, you can work around this problem by <a href=\"https:\/\/docs.docker.com\/articles\/baseimages\/\">creating your own base image<\/a> which is trivially easy. So, now I build all images, from scratch, locally. That helps put Debian developers back into the mix.<\/p>\n<p>Next, I started looking at Docker Files that I could use to construct my images and discovered something else troubling. Take, for example, the official <a href=\"https:\/\/registry.hub.docker.com\/_\/nginx\/\">nginx<\/a> Docker image. It is based on Debian Jessie (hooray - our distribution developers are in the mix!). However, it then proceeds to install nginx from the nginx repository, not the debian repository. Well, I guess if you are nginx you want to have full control, but still, the Debian developers version of nginx has been vetted to ensure it works with Debian Jessie, so you are really losing something here.<\/p>\n<p>So&hellip; my next break from Docker convention is to use all Debian packages in my Docker files.<\/p>\n<p>Once my images were built and my application was tested and running, I was done, right?<\/p>\n<p>Wrong. Well, I would have been done if I didn&rsquo;t care about upgrading my systems, backing up my data or running cron jobs. Remember those things? Those are things that distribution developers and system administrators have been perfecting for decades. And, they are not trivial.<\/p>\n<p>I then built a series of scripts to help alert me when a image I am using has updates (you can&rsquo;t just use cron-apt any more since cron isn&rsquo;t running in your container) and help me update the image and deploy it to all my applications (which involves restarting the application). Backing up data is a whole different can of worms - sometimes involving interacting with your container (if it&rsquo;s a database container, you have to launch a database dump) or simply copying files from the host, assuming you got the right <a href=\"https:\/\/docs.docker.com\/userguide\/dockervolumes\/\">Docker volume strategy<\/a> (which took me days to fully understand). Lastly, I had to run a cron job from the host that then runs whatever commands are needed on each of my containers.<\/p>\n<p>This was complicated.<\/p>\n<p>In the end, was it worth it? Yes, I think so. However, not because it was simple, which seems to be the Docker mantra. I think it&rsquo;s worth it because I can run 100 instances of my application using significantly less resources than when I was using full virtualization and because I can more easily and flexibly adjust the resource allocation. However, check in with me in a year and I&rsquo;ll probably have a different opinion.<\/p>\n"},{"title":"Agents, the kinds that work for us.","link":"https:\/\/current.workingdirectory.net\/posts\/2015\/agent-startup\/","pubDate":"Mon, 03 Aug 2015 10:37:36 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2015\/agent-startup\/","description":"<p>I recently decided I wanted to adjust how long my gpg password cache lasts and realized I had no idea how either my ssh-agent or gpg-agent was being launched or even which programs were being used as the agent for which protocol. While I appreciate that everthing just works based on blind faith, I decided to investigate more closely.<\/p>\n<p>I started by reading the man pages for ssh-agent and gpg-agent and discovered that gpg-agent could be used for providing ssh-agent.<\/p>\n<p>Since I run openbox, I decided to dig through my ~\/.config\/openbox\/autostart file and discovered that I was also launching gnome-keyring-daemon, which provides both an ssh-agent and a gpg-agent. This is getting confusing.<\/p>\n<p>I then examined the output of <code>ps -eFH<\/code> and found this:<\/p>\n<pre><code>jamie    28499 28489  0  1082  1516   1 10:06 tty1     00:00:00     \/bin\/sh \/usr\/bin\/startx\njamie    28542 28499  0  3987  1912   1 10:06 tty1     00:00:00       xinit \/etc\/X11\/xinit\/xinitrc -- \/etc\/X11\/xinit\/xserverrc :0 vt1 -auth \/tmp\/serverauth.gn3C5pYbPc\nroot     28543 28542  0 58065 25036   3 10:06 tty1     00:00:05         \/usr\/bin\/X -nolisten tcp :0 vt1 -auth \/tmp\/serverauth.gn3C5pYbPc\njamie    28548 28542  0 69563 19980   0 10:06 tty1     00:00:00         \/usr\/bin\/openbox --startup \/usr\/lib\/x86_64-linux-gnu\/openbox-autostart OPENBOX\njamie    28604 28548  0  2774   348   1 10:06 ?        00:00:00           \/usr\/bin\/ssh-agent \/usr\/bin\/gpg-agent --daemon --sh --write-env-file=\/home\/jamie\/.gnupg\/gpg-agent-info-animal \/usr\/bin\/dbus-launch --exit-with-session \/usr\/bin\/monkeysphere-validation-agent \/usr\/bin\/im-launch \/home\/jamie\/.xsession\njamie    28605 28548  0  4915   248   0 10:06 ?        00:00:00           \/usr\/bin\/gpg-agent --daemon --sh --write-env-file=\/home\/jamie\/.gnupg\/gpg-agent-info-animal \/usr\/bin\/dbus-launch --exit-with-session \/usr\/bin\/monkeysphere-validation-agent \/usr\/bin\/im-launch \/home\/jamie\/.xsession\njamie    28610 28548  0 22402 34456   0 10:06 tty1     00:00:00           \/usr\/bin\/perl -wT \/usr\/bin\/monkeysphere-validation-agent \/usr\/bin\/im-launch \/home\/jamie\/.xsession\n<\/code><\/pre>\n<p>Further down (and not nested under the xinit process), I found my gnome-keyring-daemon process.<\/p>\n<p>I started by removing my own call to gnome-keyring-daemon and realized that everything still worked fine (both gpg and ssh). I must have added that line before things were configured system-wide to run automatically.<\/p>\n<p>Then, I saw that the gpg-agent was being launched without the <code>--enable-ssh-support<\/code> option (and ssh-agent was running), so looks like ssh is handled by ssh-agent and gpg is handled by gpg-agent. So far so good.<\/p>\n<p>Now, how do they get launched??<\/p>\n<p>I start my graphical session by logging in at a console and running <code>exec startx<\/code>. In my home directory, I have ~\/.xsession which contains the line <code>exec \/usr\/bin\/openbox-session<\/code>. Therefore, I started by scouring all things openbox on my system and couldn&rsquo;t find any reference to these agents.<\/p>\n<p>Next, I turned my attention to X.  Based on my <code>ps -eFH<\/code> output, I started with \/etc\/X11\/xinit\/xinitrc, which led me to \/etc\/X11\/Xsession which led me to \/etc\/X11\/Xsession.d and presto, I found my answer: a series of shell scripts that modify a global variable STARTUP which is executed by X.<\/p>\n<p>Thanks Debian developers for making it all work out of the box!<\/p>\n"},{"title":"Docker networking... private range or not?","link":"https:\/\/current.workingdirectory.net\/posts\/2015\/docker-networking\/","pubDate":"Tue, 19 May 2015 12:07:23 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2015\/docker-networking\/","description":"<p>Am I missing something?<\/p>\n<p>I installed docker and noticed that it created a virtual interface named docker0 with the IP address 172.17.42.1. This behavior is consistent with the <a href=\"https:\/\/docs.docker.com\/articles\/networking\/\">Docker networking documentation<\/a>. However, I was confused by this statement:<\/p>\n<pre><code>It randomly chooses an address and subnet from the private range defined by RFC 1918 that are not in use on the host machine, and assigns it to docker0. Docker made the choice 172.17.42.1\/16 when I started it a few minutes ago...\n<\/code><\/pre>\n<p>It seems like <a href=\"https:\/\/tools.ietf.org\/html\/rfc1918\">RFC 1918<\/a> defines:<\/p>\n<ul>\n<li>10.0.0.0        -   10.255.255.255  (10\/8 prefix)<\/li>\n<li>172.16.0.0      -   172.31.255.255  (172.16\/12 prefix)<\/li>\n<li>192.168.0.0     -   192.168.255.255 (192.168\/16 prefix)<\/li>\n<\/ul>\n<p>How is 172.17.42.1\/16 from the private ranges listed above? Is 172.17.42.1 a potentially public IP address?<\/p>\n"},{"title":"GnuCash with python bindings","link":"https:\/\/current.workingdirectory.net\/posts\/2011\/gnucash-python-bindings\/","pubDate":"Wed, 06 May 2015 09:13:58 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2011\/gnucash-python-bindings\/","description":"<p>Hats off to the <a href=\"http:\/\/www.gnucash.org\/\">GnuCash<\/a> developers and the <a href=\"http:\/\/parit.ca\/\">Parit Worker Collective<\/a> for the python bindings to GnuCash. If there is any doubt that free software makes life easier it&rsquo;s being able to write your own accounting import scripts while reading the original source code.<\/p>\n<p>It&rsquo;s 2015-05-06 and I&rsquo;m updating this blog post that I wrote back in 2011 since you no longer have to rebuild gnucash on Debian to get the python bindings. Thanks Debian!<\/p>\n<p>I&rsquo;ve also published May First\/People Link&rsquo;s python import scripts via git:\ngit clone git:\/\/git.mayfirst.org\/mfpl\/gnucash-import.git<\/p>\n<p>I&rsquo;m happy to report that the import script now does the following:<\/p>\n<ul>\n<li>Imports <a href=\"https:\/\/mayfirst.org\/\">May First\/People Link<\/a> members as GnuCash &ldquo;Customers&rdquo;<\/li>\n<li>Imports membership dues invoices as GnuCash Invoices<\/li>\n<li>Imports payments and applies them against the Invoices<\/li>\n<li>Imports deleted invoices and puts them in our Unrecoverable A\/R account<\/li>\n<\/ul>\n<p>At this point we have eliminated all double data entry between our membership database and our accounting system. I can&rsquo;t say that I enjoy bookkeeping yet, but it&rsquo;s a lot better than before.<\/p>\n"},{"title":"So long email, it's been good to know yuh","link":"https:\/\/current.workingdirectory.net\/posts\/2015\/email\/","pubDate":"Thu, 30 Apr 2015 10:00:52 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2015\/email\/","description":"<p>Yesterday I permanently deleted 15 years of email.<\/p>\n<p>It wasn&rsquo;t because I didn&rsquo;t have enough hard disk space to store it. It&rsquo;s because I decided, after 15 years, that the benefits of keeping all this email did not outweigh the risks. Although I have never had my email subpoenaed, I have had many <a href=\"https:\/\/support.mayfirst.org\/wiki\/legal\">legal interactions<\/a> due to my involvement with <a href=\"https:\/\/mayfirst.org\">May First\/People Link<\/a>, some of which were about finding the real identities of May First\/People Link members. I&rsquo;d rather not risk compromising anyone or needlessly exposing my networks. Now I have an Inbox, Sent Box, Trash Box and Detected Spam Box. The Inbox I empty manually and the other boxes are automatically purged on a scheduled basis.<\/p>\n<p>In this age of surveillance it&rsquo;s sad to see data evaluated based on risk of exposure.<\/p>\n"},{"title":"Getting to know systemd","link":"https:\/\/current.workingdirectory.net\/posts\/2014\/getting-to-know-systemd\/","pubDate":"Wed, 20 Aug 2014 10:17:39 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2014\/getting-to-know-systemd\/","description":"<p>Update 2014-08-20: apcid needs tweaking. See update section below.<\/p>\n<p>Somehow both my regular work laptop and home entertainment computers (both running Debian Jessie) were switched to <a href=\"https:\/\/en.wikipedia.org\/wiki\/Systemd\">systemd<\/a> without me noticing. Judging from by dpkg.log it may have happened quite a while ago. I&rsquo;m pretty sure that&rsquo;s a compliment to the backwards compatibility efforts made by the systemd developers and a criticism of me (I should be paying closer attention to what&rsquo;s happening on my own machines!).<\/p>\n<p>In any event, I&rsquo;ve started trying to pay more attention now - particularly learning how to take advantage of this new software. I&rsquo;ll try to keep this blog updated as I learn. For now, I have made a few changes and discoveries.<\/p>\n<p>First - I have a convenient bash wrapper I use that both starts my OpenVPN client to a samba server and then mounts the drive. I only connect when I need to and rarely do I properly disconnect (the OpenVPN connection does not start automatically). So, I&rsquo;ve written the script to carefully check if my openvpn connection is present and either restart or start depending on the results.<\/p>\n<p>I had something like this:<\/p>\n<pre><code>if ps -eFH | egrep [o]penvpn; then\n  sudo \/etc\/init.d\/openvpn restart\nelse\n  sudo \/etc\/init.d\/openvpn start\nfi\n<\/code><\/pre>\n<p>One of the advertised advantages of systemd is the ability to more accurately detect if a service is running. So, first I changed this to:<\/p>\n<pre><code>if systemctl -q is-active openvpn.service; then\n  sudo systemctl restart openvpn.service\nelse\n  sudo systemctl start openvpn.service\nfi\n<\/code><\/pre>\n<p>However, after reviewing the man page I realized I can shorten if further to simply:<\/p>\n<pre><code>  sudo systemctl restart openvpn.service\n<\/code><\/pre>\n<p>According to the man page, restart means:<\/p>\n<pre><code>Restart one or more units specified on the command line. If the units are not\nrunning yet, they will be started.\n<\/code><\/pre>\n<p>After discovering this meaning for &ldquo;restart&rdquo; in systemd, I tested and realized that &ldquo;restart&rdquo; works the same way for openvpn using the old sysv style init system. Oh well. At least there&rsquo;s a man page and a stronger guarantee that it will work with <em>all<\/em> services, not just the ones that happen to respect that convention in their init.d scripts.<\/p>\n<p>The next step was to disable openvpn on start up. I confess, I never bothered to really learn update-rc.d. Everytime I read the manpage I ended up throwing up my hands and renaming symlinks by hand. In the case of openvpn I had previously edited \/etc\/default\/openvpn to indicate that &ldquo;none&rdquo; of the virtual private networks should be started.<\/p>\n<p>Now, I&rsquo;ve returned that file to the default configuration and instead I ran:<\/p>\n<pre><code>systemctl disable openvpn.service\n<\/code><\/pre>\n<p>UPDATES<\/p>\n<p>2014-08-20: I&rsquo;ve recently noticed strange behavior when I wake my laptop. Seems to sometimes go right back to sleep. After doing some digging I traced the problem to some customizations I have made to my laptop&rsquo;s acpid behavior combined with systemd taking over some apci events.<\/p>\n<p>Up to now, I have created my own \/etc\/acpi files so I have full control over the acpi events. In particular, I don&rsquo;t want my laptop to suspend when I close the lid. I only want it to suspend when I press the suspend key. And, when it does suspend, I want it to run my own personal suspend script so I can do things like lock the screen and restart tint2.<\/p>\n<p>I&rsquo;ve found that systemd launches it&rsquo;s own acpi event monitoring that ignores my \/etc\/acpid rules (the systemd &ldquo;unit&rdquo; that monitors events is called acpid.socket which exists in addition to acpid.service). The systemd reaction to events is controlled by the systemd-logind.service which has a configuration file: \/etc\/systemd\/logind.conf. By default, systemd-logind.service will put my laptop to sleep when the lid is closed and when the suspend button is pushed. systemd seems to get the signal first, putting the laptop to sleep. After I wake it up, acpid gets the signal - so it goes right back to sleep.<\/p>\n<p>Reading <code>man logind.conf<\/code> helps. I was able to restore my desired behavior by adding these lines to \/etc\/systemd\/logind.conf:<\/p>\n<pre><code>HandleSuspendKey=ignore\nHandleLidSwitch=ignore\n<\/code><\/pre>\n<p>Then: <code>sudo systemctl restart systemd-logind.service<\/code>.<\/p>\n"},{"title":"New OpenPGP key","link":"https:\/\/current.workingdirectory.net\/posts\/2014\/new-gpg-key\/","pubDate":"Thu, 09 Jan 2014 09:09:11 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2014\/new-gpg-key\/","description":"<p>After playing around with <a href=\"https:\/\/howsecureismypassword.net\/\">How secure is my password<\/a>, I dediced to start changing the handful of passwords I have committed to memory to being very long phrases rather than shorter, concentrated collections of random-seeming characters (another interesting blog post would be to understand the algorythm behind that site but for now I&rsquo;ve just taken it at face value).<\/p>\n<p>Despite my usual practice of introducing new passphrases by changing my laptop login (which can always be reset if necessary), I went ahead and changed by GPG key instead to a new 50 character phrase. Fifteen minutes passed before I needed access to the key and&hellip; I couldn&rsquo;t repeat the passphrase.<\/p>\n<p>I tried to be clever. I opened a text file and typed in this passphrase over and over again, ultimately generating 49 unique typo-ridden versions of the passphrase.<\/p>\n<p>Then, I wrote a simple script that took each of these versions and generated additional versions by:<\/p>\n<ul>\n<li>Changing one character (from a preselected character list of letters, numbers and punctuation within key range of the actual letters in the passphrase).<\/li>\n<li>Deleting one character<\/li>\n<li>Adding one character<\/li>\n<li>Transposing all characters<\/li>\n<\/ul>\n<p>In the end I had about 350,000 versions of my passphrase.<\/p>\n<p>None of them worked.<\/p>\n<p>What in the world did I type??<\/p>\n<p>Could I just ask the NSA?<\/p>\n<p>In any event, my new OpenPGP key fingerprint is:<\/p>\n<pre><code>1F9C30CB3CFC5DA9987FA035A014C05A607B7535\n<\/code><\/pre>\n<p>Please note: I&rsquo;ve changed the &ldquo;real name&rdquo; part of my User Id from &ldquo;Jamie McClelland&rdquo; to &ldquo;James McClelland&rdquo; becuase James is on my government issued id.<\/p>\n<p>I&rsquo;ll be working on collecting new signatures for the rest of my life.<\/p>\n<p>Some interesting information on how this made my life difficult:<\/p>\n<ul>\n<li>I use the <a href=\"http:\/\/monkeysphere.info\/\">Monkeysphere<\/a> for accessing the servers I maintain using by OpenPGP key. Fortunately, the key was loaded into my ssh agent, so I have still been able to access the servers. If my computer crashes or gets restarted I&rsquo;ll be locked out until I get my new key in place. I&rsquo;ll need to ask another <a href=\"https:\/\/mayfirst.org\/\">May First\/People Link<\/a> administrator to sign a puppet release with my new fingerprint.<\/li>\n<li>May First\/People Link uses <a href=\"https:\/\/keyringer.pw\/\">keyringer<\/a> to store shared passwords, encrypted to a small collection of administrator OpenPGP keys, including my old one. I&rsquo;ll need to ask another admin to update the list of keys.<\/li>\n<li>I have hundreds of (mostly) web-site passwords stored in <a href=\"http:\/\/finestructure.net\/assword\/\">assword<\/a> and encrypted to my old key. They are all gone. I&rsquo;ll be doing lots of password resets now.<\/li>\n<li>I OpenPGP sign my emails. That has been turned off between the time I lost the passphrase and generated a new key.<\/li>\n<li>I am subscribed to a few email lists run via <a href=\"http:\/\/schleuder2.nadir.org\/\">Schleuder<\/a>. I can&rsquo;t access those emails now. I&rsquo;ll need to send my new key to the list.<\/li>\n<li>I&rsquo;ve lost access to every encrypted message that has been sent to my old key.<\/li>\n<\/ul>\n<hr>\n<p>Postscript: Thanks to Jordan Uggla, I was directed to <a href=\"http:\/\/c0decstuff.blogspot.com\/2011\/01\/in-memory-extraction-of-ssl-private.html\">passe-partout<\/a>, a tool that can extract an RSA private key out of memory when ssh-agent is running with the key loaded. Since I am using the <a href=\"http:\/\/monkeysphere.info\/\">Monkeysphere<\/a>, my OpenPGP key has an authentication-capable subkey, which is fed to an ssh-agent and used to grant me access to remote computers using ssh. After losing my passphrase, I was not able to access that subkey.  With the help of pass-partout, and the fact that the subkey was loaded into ssh-agent before I lost the passphrase, I was able to extract the private RSA key from my ssh-agent and save it to a file. Then, I could restart my ssh-agent with my &lsquo;&rsquo;new&rsquo;&rsquo; authentication-capable OpenPGP subkey, thus allowing me to access servers that have granted me access via my new key and the handful of servers that still only grant me access via my old key.<\/p>\n"},{"title":"Administering CUPS from the command line","link":"https:\/\/current.workingdirectory.net\/posts\/2013\/cups-cli-admin\/","pubDate":"Tue, 08 Jan 2013 09:32:44 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2013\/cups-cli-admin\/","description":"<p>I usually try to avoid administering printers whenever possible. As a result I end of flailing around the CUPS web interface before I figure out how to re-enable a printer. And, when I get a call to help debug a printer, I can&rsquo;t easily tell people what to do.<\/p>\n<p>When I try to do what I need via the command line, I end up spending at least 10 or 15 minutes re-reading man pages before I piece together the steps.<\/p>\n<p>Here&rsquo;s my attempt to document the steps so I don&rsquo;t have to re-read man pages.<\/p>\n<h2 id=\"setup\">Setup<\/h2>\n<p>The cups commands in these examples can be run as a non-root user if that user is in the lpadmin group.<\/p>\n<p>Type:<\/p>\n<pre><code>groups\n<\/code><\/pre>\n<p>To see if lpadmin is listed. If not:<\/p>\n<pre><code>sudo adduser &lt;your-user-name&gt; lpadmin\n<\/code><\/pre>\n<p>Then, to gain access to the new group without logging out and logging in again:<\/p>\n<pre><code>newgrp lpadmin\n<\/code><\/pre>\n<h2 id=\"finding-printers\">Finding printers.<\/h2>\n<p>In these examples, the printer name in question is: <code>stability<\/code> and it is a network printer, with local DNS that properly resolves the hostname stability to an IP address.<\/p>\n<p>You can list all detected printers with: <code>lpstat -v<\/code><\/p>\n<p>Note that the resolved IP address is listed if it&rsquo;s a network\/wifi connected printer.<\/p>\n<h2 id=\"network-access\">Network access<\/h2>\n<p>First, try to ping the printer:<\/p>\n<pre><code>ping stability\n<\/code><\/pre>\n<p>If this fails, restart the printer and\/or check network cables. No point in doing anything else until it responds to pings.<\/p>\n<h2 id=\"cant-submit-new-jobs-to-the-printer\">Can&rsquo;t submit new jobs to the printer<\/h2>\n<p>Next, if the problem is that the printer is greyed out when you try to print a document or your application tells you that the printer is rejecting jobs, confirm this status with:<\/p>\n<pre><code>lpstat -a stability\n<\/code><\/pre>\n<p>It will either output:<\/p>\n<pre><code>stability accepting requests since Mon 20 May 2013 10:28:57 AM EDT\n<\/code><\/pre>\n<p>Or<\/p>\n<pre><code>stability not accepting requests since Mon 20 May 2013 10:28:57 AM EDT -\n  Rejecting Jobs\n<\/code><\/pre>\n<p>If it is rejecting jobs, try:<\/p>\n<pre><code>\/usr\/sbin\/cupsaccept stability\n<\/code><\/pre>\n<h2 id=\"accepts-new-jobs-but-just-doesnt-print\">Accepts new jobs, but just doesn&rsquo;t print<\/h2>\n<p>On the other hand, if the printer is accepting jobs, but the jobs are not printing, find out if the printer is enabled with:<\/p>\n<pre><code>lpstat -p stability\n<\/code><\/pre>\n<p>You should get either:<\/p>\n<pre><code>printer stability is idle.  enabled since Mon 20 May 2013 10:28:57 AM EDT\n<\/code><\/pre>\n<p>Or:<\/p>\n<pre><code>printer stability disabled since Mon 20 May 2013 10:35:10 AM EDT -\n  Paused\n<\/code><\/pre>\n<p>If it is disabled, you should first see what queued jobs there are:<\/p>\n<pre><code>sudo rlpq -P &lt;printer&gt; -H &lt;printer&gt;\n<\/code><\/pre>\n<p>(If this fails, try replacing the <code>-H<\/code> option with the IP of the printer - remember, run <code>lpstat -v<\/code> to see the IP address of your printer.)<\/p>\n<p>If you have a list of duplicate pending jobs, be sure to delete the duplicates to avoid having your print job come out multiple times.<\/p>\n<p>To delete a queued job, type the following (n should be the number in the Job column of the lpq output):<\/p>\n<pre><code>cancel &lt;n&gt;\n<\/code><\/pre>\n<p>After you have deleted duplicate jobs, try &ldquo;enabling&rdquo; it:<\/p>\n<pre><code>\/usr\/sbin\/cupsenable stability\n<\/code><\/pre>\n<p>Then, re-rerun the lpq command and see if it&rsquo;s now &ldquo;ready.&rdquo; At this point, the jobs should start printing.<\/p>\n<h2 id=\"submit-a-job-via-the-command-line\">Submit a job via the command line<\/h2>\n<pre><code>lp -h &lt;printer-host&gt; \/path\/to\/file\nlp -d &lt;printer-name&gt; \/path\/to\/file\n<\/code><\/pre>\n<p>If a pdf doesn&rsquo;t print, try &ldquo;fixing&rdquo; the PDF with:<\/p>\n<pre><code>mutool clean input.pdf output.pdf\n<\/code><\/pre>\n<h2 id=\"review-of-concepts\">Review of concepts<\/h2>\n<p>For review&hellip; a few important concepts:<\/p>\n<ul>\n<li>cupsaccept\/cupsreject: controls whether a printer will accept or reject <em>new<\/em> jobs. It doesn&rsquo;t matter whether the printer is enabled or disabled.<\/li>\n<li>cupsenable\/cupsdisable: controls whether a printer will print existing jobs. It doesn&rsquo;t matter whether the print is accepting or rejecting new jobs.<\/li>\n<\/ul>\n"},{"title":"If you don't know what a quine is, consider yourself lucky","link":"https:\/\/current.workingdirectory.net\/posts\/2013\/quine\/","pubDate":"Tue, 08 Jan 2013 09:32:44 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2013\/quine\/","description":"<p>rhatto just asked me to approve a change in <a href=\"https:\/\/keyringer.sarava.org\/\">keygringer&rsquo;s<\/a> license from AGPLv3+ to GPLv3+ citing a <a href=\"https:\/\/lists.debian.org\/debian-devel\/2013\/07\/msg00031.html\">discussion about a Berkeley DB&rsquo;s switch to AGPLv3<\/a>. For some reason, a reference to a <a href=\"https:\/\/lists.debian.org\/debian-devel\/2013\/07\/msg00057.html\">quine<\/a> caught my eye.<\/p>\n<p>If you don&rsquo;t know what a quine is, I suggest you remain ignorant. I won&rsquo;t even provide the Wikipedia link.<\/p>\n<p>A quick web search suggests that it&rsquo;s quite possible to write one in bash. I hope to be productive again some day.<\/p>\n"},{"title":"Sharing screen with remote Linux user behind a firewall","link":"https:\/\/current.workingdirectory.net\/posts\/2013\/sharing-screen-with-remote-linux-user\/","pubDate":"Tue, 08 Jan 2013 09:32:44 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2013\/sharing-screen-with-remote-linux-user\/","description":"<p>Intentionally helping someone poke a hole through their firewall to allow any users on a remote machine access to their computer is generally a bad idea&hellip; unless they want you to help them with their computer.<\/p>\n<p>In that case, I find it really useful.<\/p>\n<p>Toward this end, I&rsquo;ve setup a dedicated user account (<a href=\"mailto:jamie-share@chavez.mayfirst.org\">jamie-share@chavez.mayfirst.org<\/a> in this example). I granted myself shell access to this user and generated an ssh key pair:<\/p>\n<pre><code>ssh-keygen -t rsa\n<\/code><\/pre>\n<p>Then, as a convenience I move my public key into the home directory:<\/p>\n<pre><code>cp ~\/.ssh\/id_rsa.pub ~\/\n<\/code><\/pre>\n<p>Next, I give the user I want to help ssh access to this shell account on an Internet connected computer (in this example: <a href=\"mailto:jamie-share@chavez.mayfirst.org\">jamie-share@chavez.mayfirst.org<\/a>). Either I share the password with them or if they have an ssh key or monkeysphere identity I use that.<\/p>\n<p>Then, I ask them to grant teh jamie-share user access to their user account by running (which downloads the key I just created):<\/p>\n<pre><code>mkdir -p ~\/.ssh\nssh jamie-share@chavez.mayfirst.org &quot;cat id_rsa.pub&quot; &gt;&gt; ~\/.ssh\/authorized_keys\n<\/code><\/pre>\n<p>Next, ask them to install openssh-server on their local computer:<\/p>\n<pre><code>sudo apt-get install openssh-server screen\n<\/code><\/pre>\n<p>Next, they run the following command on their local computer:<\/p>\n<pre><code>ssh -R 2222:localhost:22 jamie-share@chavez.mayfirst.org\n<\/code><\/pre>\n<p>This command says: forward port 2222 on chavez.mayfirst.org to port 22 on your local computer.<\/p>\n<p>Lastly, I log in to <a href=\"mailto:jamie-share@chavez.mayfirst.org\">jamie-share@chavez.mayfirst.org<\/a> and run, replacing <their-username> with their local username.<\/p>\n<pre><code>ssh &lt;their-username&gt;localhost -p 2222\n<\/code><\/pre>\n<p>You can do whatever you want now, but running screen is a good way to share a session so the person you are working with can see what you are doing as you do it.<\/p>\n<p>p.s. Thanks Ross for the tips!<\/p>\n"},{"title":"Creating virtual machines","link":"https:\/\/current.workingdirectory.net\/posts\/2012\/grml\/","pubDate":"Sun, 08 Jan 2012 09:32:44 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2012\/grml\/","description":"<p>I recently setup an Alix board from <a href=\"http:\/\/pcengines.ch\">PC Engines<\/a>. I installed Debian onto a Compact Flash card using grml for the first time.<\/p>\n<p>After using parted to create a single partition, I ran:<\/p>\n<pre><code>grml-debootstrap --arch i386 --filesystem ext4 --grub \/dev\/sdb --hostname yaqeen --mirror http:\/\/http.us.debian.org\/debian \\ \n--nopassword  --release squeeze --target \/dev\/sdb1\n<\/code><\/pre>\n<p>I had to manually mount the partition and add console redirection for grub (and \/etc\/inittab).<\/p>\n<p>And then, when booting, I had to manually edit the grub configuration because it was trying to find the root filesystem on \/dev\/sdb instead of \/dev\/sda. I&rsquo;m sure there is some tweaking I can do via \/etc\/deboostrap\/scripts to fix those needs.<\/p>\n<p>Overall, I was very happy with the results. Thanks grml developers!<\/p>\n<p>Update 2012-06-19<\/p>\n<p>When installing a server I prefer keeping things a bit more manual.<\/p>\n<p>I start by doing a pxeboot into a <a href=\"http:\/\/cmrg.fifthhorseman.net\/wiki\/debirf\">debirf image<\/a>. Then, I run through all the <a href=\"https:\/\/support.mayfirst.org\/wiki\/install_debian#DrivepartioningDisksetup\">disk partitioning\/encrypting\/etc steps listed on the May First\/People Link installation page<\/a>.<\/p>\n<p>Then, I mount each partition in \/mnt and run debootstrap:<\/p>\n<pre><code>debootstrap squeeze \/mnt\n<\/code><\/pre>\n<p>Copy over any non-free firmware packages, e.g.:<\/p>\n<pre><code>cp \/media\/usb\/firmware-bnx2_0.28+squeeze1_all.deb \/mnt\/root\/\n<\/code><\/pre>\n<p>Next, mount the system directories and chroot:<\/p>\n<pre><code>for dir in sys dev proc; do mkdir -p \/mnt\/$dir; mount -o bind \/$dir \/mnt\/$dir; done\nchroot \/mnt\n<\/code><\/pre>\n<p>Install non-free firmware packages, e.g.:<\/p>\n<pre><code>dpkg -i \/root\/firmware-bnx2_0.28+squeeze1_all.deb\n<\/code><\/pre>\n<p>Add \/etc\/fstab, \/etc\/network\/interfaces, \/etc\/crypttab files<\/p>\n<p>Edit \/etc\/inittab to enable output to the serial console<\/p>\n<p>Set the root password.<\/p>\n<p>Set the hostname (edit \/etc\/hostname)<\/p>\n<p>Install necessary packages that debootstrap won&rsquo;t install for you:<\/p>\n<pre><code>apt-get update\napt-get install linux-image-2.6-amd64 lvm2 cryptsetup mdadm grub-pc\n<\/code><\/pre>\n<p>When prompted by grub for Linux command line options, enter:<\/p>\n<pre><code>console=ttyS0,115200n8\n<\/code><\/pre>\n<p>Edit \/etc\/default\/grub adding to the bottom:<\/p>\n<pre><code>GRUB_TERMINAL=serial\nGRUB_SERIAL_COMMAND=&quot;serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1&quot;\n<\/code><\/pre>\n<p>Edit \/boot\/grub\/device.map to remove any USB devices<\/p>\n<p>And run update-grub<\/p>\n"},{"title":"Key Mapping for console redirection","link":"https:\/\/current.workingdirectory.net\/posts\/2012\/console-bios-key-map\/","pubDate":"Sun, 08 Jan 2012 09:32:44 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2012\/console-bios-key-map\/","description":"<p>This always flashes by too quickly for me to read&hellip;<\/p>\n<p>I&rsquo;m preserving it here for next time.<\/p>\n<pre><code>Press the spacebar to pause...\n\nKEY MAPPING FOR CONSOLE REDIRECTION:\n\nUse the &lt;ESC&gt;&lt;0&gt; key sequence for &lt;F10&gt;\nUse the &lt;ESC&gt;&lt;!&gt; key sequence for &lt;F11&gt;\nUse the &lt;ESC&gt;&lt;@&gt; key sequence for &lt;F12&gt;\n\nUse the &lt;ESC&gt;&lt;Ctrl&gt;&lt;M&gt; key sequence for &lt;Ctrl&gt;&lt;M&gt;\nUse the &lt;ESC&gt;&lt;Ctrl&gt;&lt;H&gt; key sequence for &lt;Ctrl&gt;&lt;H&gt;\nUse the &lt;ESC&gt;&lt;Ctrl&gt;&lt;I&gt; key sequence for &lt;Ctrl&gt;&lt;I&gt;\nUse the &lt;ESC&gt;&lt;Ctrl&gt;&lt;J&gt; key sequence for &lt;Ctrl&gt;&lt;J&gt;\n\nUse the &lt;ESC&gt;&lt;X&gt;&lt;X&gt; key sequence for &lt;Alt&gt;&lt;x&gt;, where x is any letter\nkey, and X is the upper case of that key\n\nUse the &lt;ESC&gt;&lt;R&gt;&lt;ESC&gt;&lt;r&gt;&lt;ESC&gt;&lt;R&gt; key sequence for &lt;Ctrl&gt;&lt;Alt&gt;&lt;Del&gt;\n<\/code><\/pre>\n"},{"title":"Killing MySQL","link":"https:\/\/current.workingdirectory.net\/posts\/2012\/killing-mysql\/","pubDate":"Sun, 08 Jan 2012 09:32:44 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2012\/killing-mysql\/","description":"<p>MySQL is supposed to allow one + the max_connections from a super user so you can stop\/restartmysql even when it is reporting too many connections. Unfortunately, for reasons I&rsquo;m not entirely sure of, sometimes that connection seems to be in use.<\/p>\n<p>I&rsquo;ve encountered this problem with web apps and now follow these steps:<\/p>\n<ul>\n<li>Shutdown apache to prevent new connections<\/li>\n<li>Search for and kill any backup processes (this can cause table locking which can be\nthe cause of the problem)<\/li>\n<li>Try a clean shutdown again<\/li>\n<li>search for and kill mysqld_safe safe process on host. This step should kill one\nconnection allowing you to cleanly stop mysql with:<\/li>\n<\/ul>\n<p>\/etc\/init.d\/\/mysql stop<\/p>\n"},{"title":"Live video streaming using vp8enc and webm on Debian","link":"https:\/\/current.workingdirectory.net\/posts\/2012\/vp8-and-debian\/","pubDate":"Sun, 08 Jan 2012 09:32:44 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2012\/vp8-and-debian\/","description":"<p>The vp8 video codec and webm container format (which are alternates for theora\/ogg and h264\/flv) are getting lots of attention, especially as more web browsers are supporting html5 - which includes a new video tag that plays video directly in a web page, rather than requiring a plugin. With Google fully behind these new formats (and <a href=\"http:\/\/news4geeks.net\/2011\/04\/25\/youtube-transcodes-all-its-videos-to-webm-vp8-h-264-still-supported\/\">converting YouTube videos<\/a>) and even <a href=\"http:\/\/gigaom.com\/video\/skype-vp8-video-conferencing\/\">skype using vp8<\/a>, it seems as though it could supplant h264\/flash as the default video codec and container.<\/p>\n<p>After years under the boot of the proprietary flash video, this change could usher in a lot of exciting developments.<\/p>\n<p>While the shift in formats is also significant for on-demand video, I&rsquo;m focusing on live streaming in this blog.<\/p>\n<p>There are a lot of tools needed to successfully stream a live video. Enough of them support vp8 and webm to create a live video stream, however, not all of the right versions of these tools have landed in Debian.<\/p>\n<p>Without any patches or changes, if you try to send a live video stream to icecast using vp8 encoding and webm, you&rsquo;ll get something like this:<\/p>\n<pre><code>0 jamie@animal:~$ gst-launch v4l2src ! video\/x-raw,width=320,height=240 ! vp8enc ! webmmux ! shout2send ip=icecast.server port=8000 password=secret mount=\/test.webm\nWARNING: erroneous pipeline: could not link webmmux0 to shout2send0\n1 jamie@animal:~$ \n<\/code><\/pre>\n<p>I&rsquo;ve documented the steps for patching\/upgrading gstreamer and icecast2 so you can get a live video stream using vp8 encoding and the webmc container that originates on a Debian wheezy machine using gstreamer and streams to a Debian squeeze server running icecast2.<\/p>\n<p>On the workstation, you&rsquo;ll need libshout3 version 2.3.0 or higher, which is at the moment available in experimental and installable without bringing in any dependencies. In addition, you will need to bring in gstreamer1.0 from sid and you will need to rebuild gstreamer1.0-plugins-good after libshout-dev has been installed from experimental.<\/p>\n<p>So, begin by adding experimental and sid to your sources list by creating the files \/etc\/apt\/sources.list.d\/experimental.list and \/etc\/apt\/sources\/lists.d\/sid.list with the respective contents:<\/p>\n<pre><code>deb http:\/\/mirror.cc.columbia.edu\/debian experimental main \n<\/code><\/pre>\n<p>and<\/p>\n<pre><code>deb http:\/\/mirror.cc.columbia.edu\/debian sid main \ndeb-src http:\/\/mirror.cc.columbia.edu\/debian sid main \n<\/code><\/pre>\n<p>In order to avoid upgrading your entire machine to sid (experimental packages won&rsquo;t upgrade by default), you&rsquo;ll also need to add the file \/etc\/apt\/preferences.d\/sid:<\/p>\n<pre><code>Package: *\nPin: release n=sid\nPin-Priority: 200\n<\/code><\/pre>\n<p>Then update and install:<\/p>\n<pre><code>apt-get update\napt-get install -t experimental libshout3\napt-get install gir1.2-clutter-gst-1.0 gir1.2-gst-plugins-base-1.0 gir1.2-gstreamer-1.0 gstreamer0.10-ffmpeg  gstreamer1.0-libav:amd64 gstreamer1.0-plugins-bad:amd64 gstreamer1.0-plugins-base:amd64 gstreamer1.0-plugins-base-apps gstreamer1.0-plugins-good:amd64 gstreamer1.0-pulseaudio:amd64 gstreamer1.0-tools gstreamer1.0-x:amd64 libclutter-gst-1.0-0:amd64 libgstreamer-plugins-bad1.0-0:amd64 libgstreamer-plugins-base1.0-0:amd64 libgstreamer-plugins-base1.0 libgstreamer1.0-0:amd64 libgstreamer1.0\n<\/code><\/pre>\n<p>Fortunately, the version of gstreamer1.0-plugins-good in sid includes a fix for <a href=\"https:\/\/bugzilla.gnome.org\/show_bug.cgi?id=689336\">a bug<\/a> patched on November 31, 2012 that allows the use of the webm container. However, for it to work properly gstreamer1.0-plugins-good has to be built against libshout version 2.3.0 or higher (which is in experimental, not sid). Therefore, you have install libshout-dev from experimental and then rebuild gstreamer1.0-plugins-good.<\/p>\n<pre><code>sudo apt-get build-dep gstreamer1.0-plugins-good\nsudo apt-get -t experimental install libshout-dev\napt-get source gstreamer1.0-plugins-good\ncd gst-plugins-good1.0-1.0.4\ndch -v 1.0.4-1+webmmux\nfakeroot debian\/rules binary\nsudo dpkg -i ..\/gstreamer1.0-plugins-good_1.0.4-1+webmmux_amd64.deb\n<\/code><\/pre>\n<p>Now if you try the command, things should work on the client side. When streaming to an icecast2 server running squeeze, I don&rsquo;t get any errors, however, when opening the streaming page via the icecast URL, there is a play button, which can be clicked, but no picture is shown. That&rsquo;s because the version of Icecast2 in squeeze doesn&rsquo;t support either vp8 or webm.<\/p>\n<p>On the server side, you will need icecast2 version 2.4beta or higher. This version hasn&rsquo;t landed anywhere in Debian as of this writing.<\/p>\n<p>However, you can build it from source by following these steps (the patches I&rsquo;m removing have all been applied to the new upstream version).<\/p>\n<pre><code>apt-get source icecast2\nwget http:\/\/downloads.xiph.org\/releases\/icecast\/icecast-2.4-beta.tar.gz\ntar -xzvf icecast-2.4-beta.tar.gz\ncp -r icecast2-2.3.2\/debian icecast-2.3.99.0\/\ncd icecast-2.3.99.0\/\nrm debian\/patches\/{1001_autotools_avoid_debian-subdir.patch,1002_add_missing_xspf_file,1003_fix_memory_leak.patch,series}\ndch -v 2.3.99.0-1\nfakeroot debian\/rules binary\n<\/code><\/pre>\n<p>When you install it, be sure to replace the icecast2.xml configuration file and update it with your server-specific customizations.<\/p>\n<p>Hopefully this version of icecast2 will make it to experimental soon.<\/p>\n<p>And now&hellip; you can successfully stream via:<\/p>\n<pre><code>gst-launch-1.0 v4l2src ! video\/x-raw,width=320,height=240 ! vp8enc ! webmmux !  shout2send ip=icecast.server port=8000 password=secret mount=\/test.webm\n<\/code><\/pre>\n<p>If you want full audio and video, here&rsquo;s a more complete pipeline:<\/p>\n<p>gst-launch-1.0 v4l2src ! queue ! video\/x-raw,width=320,height=240 ! tee name=tscreen ! queue ! autovideosink tscreen. ! queue ! videorate ! video\/x-raw,framerate=25\/2 ! queue ! vp8enc ! queue ! webmmux name=mux pulsesrc ! queue ! audioconvert ! vorbisenc ! queue ! queue ! mux. mux. ! queue ! shout2send ip=hobo port=8000 mount=jamie.webm password=secret<\/p>\n<p>Unfortunately, since I&rsquo;m not sure how to deal with the python2.7 vs python3 dependecies, my elegant <a href=\"http:\/\/publish.mayfirst.org\/icecast\/mobi\">mobi<\/a> python script that creates this pipeline for you using python-gtk as been reduced to a crude <a href=\"http:\/\/publish.mayfirst.org\/icecast\/mobi.sh\">bash script<\/a>.<\/p>\n<p>Another item on the todo list is opus audio encoding. gstreamer doesn&rsquo;t seem able to use opus in the webm container (although it seems to work with ogg).<\/p>\n<p>In the course of this research I found <a href=\"http:\/\/jderose.blogspot.com\/2012\/07\/how-to-run-gstreamer-uninstalled.html\">directions for getting gstreamer-1.0 installed on debian<\/a> independent of packages installed on the system. If you follow the directions above, these steps should not be necessary. I&rsquo;m including in case you need the latest sources for another reason.<\/p>\n<p>Extra steps I needed were:<\/p>\n<ul>\n<li>Edit gst-rebuild, comment out gst-plugins-ugly from plugins var (line 14) and sources arg (line 14) - these caused errors for me.<\/li>\n<li>manually install the following packages from experimental (apt-get build-dep gets confused about how to find them):<\/li>\n<li>apt-get install -t experimental libavcodec-dev=6:9~beta3-1<\/li>\n<li>apt-get install -t experimental libavformat-dev=6:9~beta3-1<\/li>\n<li>apt-get install -t experimental libswscale-dev=6:9<del>beta3-1 libswscale2=6:9<\/del>beta3-1<\/li>\n<li>Then follow directions as advertised<\/li>\n<\/ul>\n"},{"title":"Managing KVM instances","link":"https:\/\/current.workingdirectory.net\/posts\/2012\/managing-kvm\/","pubDate":"Sun, 08 Jan 2012 09:32:44 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2012\/managing-kvm\/","description":"<p>At <a href=\"https:\/\/mayfirst.org\">May First\/People Link<\/a> we have been using <a href=\"https:\/\/en.wikipedia.org\/wiki\/Kernel-based_Virtual_Machine\">KVM<\/a> for several years now and recently I have been running KVM instances on my local laptop.<\/p>\n<p>I&rsquo;m pleased to see all the work that has gone into <a href=\"http:\/\/packages.debian.org\/search?keywords=libvirt\">libvirt<\/a>, which seems like a robust and full-featured suite of tools for managing many virtualization technologies, including KVM. However, we don&rsquo;t use it at May First\/People Link for a number of reasons. The most pressing is that it runs all virtual guests as the same user, but also because it offers far more features than we need (such as graphical access to virtual server, which we don&rsquo;t need since none of our guest servers run X).<\/p>\n<p>On May First\/People Link hosts, we are using a relatively simple set of bash scripts (accessible via git at git:\/\/lair.fifthhorseman.net\/~dkg\/kvm-manager). These scripts re-use many tools we are already familiar with to build and launch kvm guests. Each guests runs as a dedicated non-provileged user, with a console available using screen, and the kvm process is managed using runit. Since our admins are familiar with these tools already, the learning curve involved is much less steep.<\/p>\n<p>Despite the relative simplicity of kvm-manager, it was still more complicated and involved than I wanted on my laptop. Additionally, I wanted to fully understand every piece of the puzzle and separating out user privileges wasn&rsquo;t important to me.<\/p>\n<p>So - I wrote the a bash script to launch virtual servers. It assumes you are using logical volume manager.<\/p>\n<p>Some editing required if you want to re-use it. You can presuse it below or <a href=\"https:\/\/current.workingdirectory.net\/downloads\/misc\/vlaunch\">download it<\/a>.<\/p>\n<pre><code># !\/bin\/bash\n# Manage a virtual server\n\n# This script assumes you are using Logical Volume Manager (LVM)\n#\n# There are changes to your system that you need to make once to get this\n# system working. Once you have made these changes you are done.  There are\n# other steps you have to take everytime you add a new guest to your\n# system.\n#\n# ONE TIME CHANGES\n#\n# Install necessary packages \n#\n# sudo apt-get install qemu-kvm screen bridge-utils dnsmasq\n#\n# For networking to properly work, your kernel must allow packet forward.\n#\n# You can enable packet forwarding by adding the file \/etc\/sysctl.d\/local.conf\n# with the contents:\n#\n# # used for networking kvm instance\n# net.ipv4.ip_forward=1\n#\n# When you restart your computer, this change will effect. Or, you can run the\n# following command to get it to take effect right away:\n#\n# sudo -i    # to become root\n# echo 1 &gt; \/proc\/sys\/net\/ipv4\/ip_forward\n# exit    # to return to being a normal user\n#\n# You must modify your \/etc\/network\/interfaces file and add the following\n# stanza:\n#\n# auto virbr0 \n# iface virbr0 inet static \n#  address 10.11.13.1 \n#  netmask 255.255.255.0 \n#  pre-up brctl addbr virbr0 \n#  post-down brctl delbr virbr0\n#\n# Then, bring up the interface with:\n# \n# sudo ifup virbr0\n# \n# Next configure dnsmasq by creating the file: \/etc\/dnsmasq.d\/local with the following\n# content:\n#\n# interface=virbr0 \n# dhcp-range=10.11.13.2,10.11.13.100,1h\n#\n# Restart dnsmasq for the changes to take place:\n#\n# sudo service dnsmasq restart\n#\n# In order to run your virtual server as a non-privileged user (e.g. your\n# normal user) you will need to make a change to your system so that your\n# newly created logical volume (and all future logical volumes) will be\n# owned by your user. \n# \n# Add a file called \/etc\/udev\/rules.d\/92-kvm.rules with the following line\n# (change &quot;jamie&quot; to the group you are running as and vg_kermit0 to the\n# name of your volume group). If you are not sure the name of your volume\n# group type: sudo vgs.\n#\n# ACTION==&quot;change&quot;, SUBSYSTEM==&quot;block&quot;, ATTR{dm\/name}==&quot;vg_kermit0-*_root&quot;, GROUP=&quot;jamie&quot;\n#\n# Lastly, you will need to download a Debian installer ISO to your file\n# system.\n# \n# Example command to download the full installer (CD 1):\n# wget http:\/\/cdimage.debian.org\/debian-cd\/6.0.5\/amd64\/iso-cd\/debian-6.0.5-amd64-CD-1.iso\n# \n# Example command to download the net installer (smaller download, will use \n# the Internet to download needed packages):\n#\n# wget http:\/\/cdimage.debian.org\/debian-cd\/6.0.5\/amd64\/iso-cd\/debian-6.0.5-amd64-netinst.iso\n#\n# REPEAT ONCE FOR EACH GUEST\n#\n# Now, create a new logical volume for your virtual server.  You need to\n# repeat this step for every virtual server you create. This example\n# assumes that you are creating a virtual server named gonzo and that your\n# logical volume group name is vg_kermit0 (if you are not sure what your\n# logical volume group is named, try typing the command: sudo vgs).\n#\n# sudo lvcreate --name gonzo_root --size 15GB vg_kermit0\n#\n# Finally, assuming your ISO is stored in \/home\/jamie\/ISOs\/debian.iso,\n# type: \n#\n# .\/vlaunch gonzo start \/home\/jamie\/ISOs\/debian.iso \n#\n# and you are ready to go. \n# \n# This form of the command will start your virtual server with the Debian\n# installer passed to it and you should be prompted through the\n# installation.\n#\n# After you have installed Debian, you can start it with simply:\n#\n# vlaunch gonzo\n#\n# If you want to clean up the networking devices created (after you have\n# shutdown your virtual server) you can do that with:\n#\n# vlaunch gonzo cleanup\n#\n# Be sure to edit the variables below to match your system:\n\nbridge=virbr0\nuser=jamie\nvg=vg_animal0\n# Change graphic to 0 if you want to launch this via screen\ngraphic=1\n# Change configure_nat to 0 if you want to handle your nat creation\n# on your own (e.g. via \/etc\/network\/ifup.d\/\nconfigure_nat=1\n\n# Modify server memory here. Depending on how much memory you have for your entire\n# system you may want to raise or lower this number\nmem=512\n\n# Create a function that will echo a variable passed in and then exit the script\ndie () {\n  printf &quot;$1\\n&quot;\n  exit 1\n}\n\n# This is the function that will be called if we are starting a virtual server\nfunction start() {\n  if [ &quot;$configure_nat&quot; -eq 1 ]; then\n    # Get the name of the current network device\n    dev=$(ip route | grep ^default | grep -oE &quot;dev [a-z0-9]+&quot; | sed &quot;s\/dev \/\/&quot;)\n\n    if [ -n &quot;$dev&quot; ]; then\n      # Flush the nat table to avoid duplicates\n      sudo iptables --table nat -F\n\n      # Create a NAT (network address translation rule)\n      sudo iptables --table nat -A POSTROUTING ! -d 127.0.0.1\/8 --out-interface &quot;$dev&quot; -j MASQUERADE\n    else\n      printf &quot;I could not determine your network device. Not configuring NAT.\\n&quot;\n    fi\n  fi\n  lvname=&quot;${vg}-${server}_root&quot;\n\n  # Trigger udev to ensure we have proper ownership of the block device.\n  sudo udevadm trigger --subsystem-match=block --attr-match=dm\/name=&quot;$lvname&quot;\n\n  lv=&quot;\/dev\/mapper\/$lvname&quot;\n  [ ! -e &quot;$lv&quot; ] &amp;&amp; die &quot;Can't find $lv&quot;\n\n  sudo modprobe -v tun || die &quot;Failed to modprobe tun module&quot;\n\n  # Create network device if it doesn't already exist.\n  ip tuntap | grep &quot;$tap&quot; &gt;\/dev\/null || sudo ip tuntap add dev &quot;$tap&quot; mode tap user &quot;$user&quot; || die &quot;Failed to create device $tap&quot;\n  # Bring up device if it's not already up.\n  ip link | grep &quot; $tap &quot; &gt;\/dev\/null || sudo ip link set &quot;$tap&quot; up || die &quot;Failed to set $tap to up&quot;\n  # Add the device to the bridge so it get use the upstream network connections.\n  \/sbin\/brctl show | grep &quot;$tap&quot; &gt; \/dev\/null || sudo brctl addif &quot;$bridge&quot; &quot;$tap&quot; || die &quot;Failed to add tap to bridge&quot;\n\n  # Launch kvm.\n  screen=\n  nographic=\n  if [ &quot;$graphic&quot; -eq 0 ]; then\n    # Launch with -nographic in a screen session\n    screen=&quot;screen -S $server&quot;\n    nographic=&quot;-nographic&quot;\n  fi\n\n  if [ &quot;$command&quot; = &quot;show&quot; ]; then\n    printf &quot;Here is the command that would be executed:\\n&quot;\n    echo kvm -drive &quot;file=$lv,if=virtio,id=hda,format=raw&quot; -m &quot;$mem&quot; -device &quot;virtio-net-pci,vlan=1,id=net0,mac=$mac,bus=pci.0&quot; -net &quot;tap,ifname=$tap,script=no,downscript=no,vlan=1,name=hostnet0&quot; $cdarg $nographic \n  else\n    $screen kvm -drive &quot;file=$lv,if=virtio,id=hda,format=raw&quot; -m &quot;$mem&quot; -device &quot;virtio-net-pci,vlan=1,id=net0,mac=$mac,bus=pci.0&quot; -net &quot;tap,ifname=$tap,script=no,downscript=no,vlan=1,name=hostnet0&quot; $cdarg $nographic || die &quot;Failed to start kvm&quot; \n  fi\n}\n\n# This is the function we will call to cleanup.\nfunction cleanup() {\n  read -p &quot;Please shutdown the host first then hit any key to continue...&quot;\n  sudo brctl delif &quot;$bridge&quot; &quot;$tap&quot;\n  sudo ip link set &quot;$tap&quot; down\n  sudo ip tuntap del mode tap dev &quot;$tap&quot;\n}\n\n# Start the main program logic.\n# The first argument passed to the script is $1 - resave as the variable $server\nserver=&quot;$1&quot;\n\n# Second argument is the command\ncommand=&quot;$2&quot;\n# If no command is passed, assume we are starting.\nif [ -z &quot;$command&quot; ]; then\n  command=start\nfi\n\n# By default cdarg variable is left empty\ncdarg=\n\n# If a third variable is passed, it means they are passing an ISO image.\nif [ -n &quot;$3&quot; ]; then\n  # Make sure it exists.\n  [ ! -f &quot;$3&quot; ] &amp;&amp; die &quot;Third argument should be path to cd iso. Can't find that path.&quot;\n  cdarg=&quot;-cdrom $3&quot;\nfi\n\n\n# Generate reproducible mac address.\nmac=&quot;$(printf &quot;02:%s&quot; &quot;$(printf &quot;%s\\0%s&quot; &quot;$(hostname)&quot; &quot;${server}&quot; | sha256sum | sed 's\/\\(..\\)\/\\1:\/g' | cut -f1-5 -d:)&quot; )&quot;\ntap=&quot;${server}0&quot;\n\nif [ &quot;$command&quot; = &quot;start&quot; ] || [ &quot;$command&quot; = &quot;show&quot; ]; then\n  start &quot;$command&quot;\nelif [ &quot;$command&quot; = &quot;cleanup&quot; ]; then\n  cleanup\nelse\n  die &quot;Please pass start or cleanup as first argument. You passed: $command&quot;\nfi\n<\/code><\/pre>\n"},{"title":"MySQL throwing the wrong error","link":"https:\/\/current.workingdirectory.net\/posts\/2012\/mysql-incorrect-error-message\/","pubDate":"Sun, 08 Jan 2012 09:32:44 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2012\/mysql-incorrect-error-message\/","description":"<p>I was running a routine <a href=\"http:\/\/civicrm.org\">CiviCRM<\/a> installation when I was stopped by the error:<\/p>\n<pre><code>ERROR 1071 (42000) at line 2415: Specified key was too long; max key length is 1000 bytes\n<\/code><\/pre>\n<p>The line referred to a table that created two indexes. Each index combined a 512 byte varchar field with a 4 byte integer field (well below 1000 bytes).<\/p>\n<p>I then tried downloading the sql file to my local computer and it imported without a problem.<\/p>\n<p>So&hellip; I started the painstaking process of comparing the configurations on each computer. Eventually, I came to this setting that was causing the problem:<\/p>\n<pre><code>innodb_log_file_size = 268440000\n<\/code><\/pre>\n<p>In fact, it wasn&rsquo;t the setting causing the error. The error was caused because I added that setting without properly shutting down MySQL, removing \/var\/lib\/mysql\/ib_logfile0 and \/var\/lib\/mysql\/ib_logfile1 files and then restarting MySQL.<\/p>\n"},{"title":"New Privacy Arguments are Needed","link":"https:\/\/current.workingdirectory.net\/posts\/2012\/new-privacy-arguments-needed\/","pubDate":"Sun, 08 Jan 2012 09:32:44 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2012\/new-privacy-arguments-needed\/","description":"<p>Another <a href=\"https:\/\/www.nytimes.com\/2012\/11\/18\/technology\/your-online-attention-bought-in-an-instant-by-advertisers.html\">decent article on online privacy<\/a> was published by the NYT this morning. The article documents the collection and centralized aggregation of consumer data online. It ends with a strong message about consumers becoming the &ldquo;product&rdquo; - which echoes what Noam Chomsky and Ben Bagdikian have been saying for decades.<\/p>\n<p>The problem with this critique is that it doesn&rsquo;t go far enough. Chomsky and Bagdikian didn&rsquo;t say readers are the product of modern journalism and that&rsquo;s bad because nobody wants to be a product. They critiqued the model because it produces terrible journalism and democracy suffers as a result.<\/p>\n<p>Similarly, the online trading of audiences and the invasion of our privacy that results has more important implications than &ldquo;don&rsquo;t be a sucker&rdquo; which seems to be the gist of the privacy argument these days. Honestly, I don&rsquo;t stay awake at night worrying that Nike knows I&rsquo;m looking for a pair of sneakers, or even that they know my income.<\/p>\n<p>I do stay awake at night worrying that people are tagging my photo on Facebook, which could allow the New York Police Dept to submit a photo of protesters to Facebook and get a list of names and addresses of the people in the photo. Or it could allow the police to track my movements via existing networks of surveillance cameras by matching my image to my name. Would that require a search warrant? How would that impact my trust in my government to know that my movements are being tracked? Or worse, to know they might be tracked but I&rsquo;ll never know if they are or aren&rsquo;t?<\/p>\n<p>Many governments (including the US) have sordid histories of infiltrating political organizations and intentionally sowing dissent. Having access to activists&rsquo; purchasing habits provides a treasure trove of material for this purpose that previously was labor intensive for cops to collect. It also provides leverage during interrogation, a time of heightened emotional vulnerability where the mere mention of a private detail of one&rsquo;s life can have a powerful impact. How can we prevent law enforcement from having access to these details of our lives?<\/p>\n<p>And then there&rsquo;s the dragnet. A central database of consumer habits provides a tantalizing collection of information that could be searched for people who fit a profile of a crime. Is solving the crime at hand worth the invasion of privacy that results from people being investigating only based on their consumer habits? How would we measure the impact an investigation like this would have on our confidence in democracy and government?<\/p>\n<p>The corporate advertisers promoting this kind of data collection argue that these problems are the price of progress. However, the ideology behind these technology developments is capitalism, not some intrinsic aspect of the Internet or network communications. Historically, the Internet has developed based on a drive toward de-centralization. Email and web sites, the bedrock of Internet activity, are based on these principles: anyone can add their own email server or web server and everyone can seamlessly access it. In contrast, the logic of capitalism as it relates to the Internet, is one of centralization and aggregating data. Facebook and Google want you to use their platforms for all your online activities so they have as complete a picture as possible of what you like and do. As the New York Times article describes, companies like Rubicon want to track all your movements and aggregate that data for profit.<\/p>\n<p>We have a choice. We can carefully evaluate our use of corporate services and consider the implications it has not only on our lives but on society as a whole. And, we can choose to use and support services that promote open standards and interoperability that will respect our personal privacy values.<\/p>\n<p>Want a practical example? <a href=\"http:\/\/friendica.com\/\">Friendica<\/a> is software designed to allow you to post status alerts, photos, videos and other information about yourself in a way similar to Facebook. However, it has one big difference: you can host your account on any server running the software anywhere in the world. You can still &ldquo;friend&rdquo; people on different servers, but we don&rsquo;t have to all trust the same organization to host it. Furthermore, Friendica allows you to post once and automatically cross-post to your Twitter, Facebook or other similar services. Sound good? You are welcome to try out an account on the May First\/People Link install: <a href=\"https:\/\/friends.mayfirst.org\/\">https:\/\/friends.mayfirst.org\/<\/a>.<\/p>\n"},{"title":"Posting to identi.ca via the command line","link":"https:\/\/current.workingdirectory.net\/posts\/2012\/identi.ca-via-command-line\/","pubDate":"Sun, 08 Jan 2012 09:32:44 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2012\/identi.ca-via-command-line\/","description":"<p>I set aside 15 minutes to find a tool that would allow me to easily post to my <a href=\"https:\/\/identi.ca\/jamiemcclelland\">identi.ca account<\/a> via the command line. I&rsquo;m two hours in and I finally sent my first remote post. Hopefully nobody else will have to spend this much time!<\/p>\n<p>There are a lot of tools to help you interact with Twitter via the command line and identi.ca supports the twitter API, however, convincing these tools to use identi.ca was harder than I expected and is woefully under-documented. This hardship is largely self-inflicted since I chose to authenticate via oauth, however, it&rsquo;s not clear to me if identi.ca supports basic auth and if so for how much longer. In any event, oauth seems like a much preferable authetication approach because I don&rsquo;t have to store my identi.ca password in plain text.<\/p>\n<p>I decided to use <a href=\"http:\/\/packages.debian.org\/search?keywords=tweepy\">tweepy<\/a> since it&rsquo;s packaged for Debian and it&rsquo;s written in python.<\/p>\n<p>If you are not familiar with <a href=\"https:\/\/en.wikipedia.org\/wiki\/Oauth\">OAuth<\/a>, it&rsquo;s worth reading up on. In short:<\/p>\n<ul>\n<li>register your application via identi.ca web interface<\/li>\n<li>retrieve consumer token and consumer secret from identi.ca (via the web)<\/li>\n<li>using this token and secret, request from identi.ca an application key and secret (you must provide your identi.ca user login credentials before you get these strings)<\/li>\n<li>and now, configure your application to use the applicaiton key and secret everytime it connects<\/li>\n<\/ul>\n<p>tweepy has <a href=\"http:\/\/packages.python.org\/tweepy\/html\/auth_tutorial.html#oauth-authentication\">documentation on using setting up oauth<\/a>, however, I found it hard to follow, especially since it goes back and forth between using it for a web app and using it for a desktop app. Since I&rsquo;m interested in a desktop app (shell), the web app business was just clutter.<\/p>\n<p>I wrote the script below to initialize my app and retrieve the application key and secret. If you are writing a desktop app, this needs to be run once for each user of the app:<\/p>\n<pre><code>#!\/usr\/bin\/python\n\n&quot;&quot;&quot; \nThis script should be used once to initialize your desktop\/command line\napplication with a oauth access key and access secret. \n\nYour first step is to login, via the web, to your identi.ca account\nand authorize your application. Click Edit next to your username\nto edit your profile settings. Then click Connections on the left\nside bar. Then, in the right sidebar, click \n&quot;Register an OAuth client application&quot;.\n\nWhen you are done, you will see a page listing your token and secret.\nFill in the consumer_token and consumer_secret variables below with those\nvalues.\n&quot;&quot;&quot;\n&quot;&quot;&quot; Fill in these values with values provided in the identi.ca web app when you register your app! &quot;&quot;&quot;\nconsumer_token = &quot;&quot;\nconsumer_secret = &quot;&quot;\n\nimport tweepy\n\nhost = 'identi.ca'\napi_root = '\/api\/'\noauth_root = api_root + 'oauth\/'\n\nauth = tweepy.OAuthHandler(consumer_token, consumer_secret, 'oob')\n\nauth.OAUTH_HOST = host\nauth.OAUTH_ROOT = oauth_root\nauth.secure = True\n\ntry:\n  redirect_url = auth.get_authorization_url()\nexcept tweepy.TweepError:\n  print 'Error! Failed to get request token.'\n  quit()\n\nreq_key = auth.request_token.key\nreq_secret = auth.request_token.secret\n\nprint &quot;you don't need these values... just fyi...&quot;\nprint &quot;auth request key is: &quot; + req_key\nprint &quot;auth request secret is: &quot; + req_secret\n\nprint &quot;Go to this URL for verify code: &quot; + redirect_url\n\nprint &quot;enter the verify code from the URL above&quot;\nverifier = raw_input('Verify code: ')\n\nauth = tweepy.OAuthHandler(consumer_token, consumer_secret, 'oob')\nauth.set_request_token(req_key, req_secret)\n\nauth.OAUTH_HOST = host\nauth.OAUTH_ROOT = oauth_root\nauth.secure = True\n\ntry:\n  auth.get_access_token(verifier)\nexcept tweepy.TweepError:\n  print 'Error! Failed to get access token.'\n\nprint &quot;Store these values in your application. You will re-use them&quot;\nprint &quot;auth access key is: &quot; + auth.access_token.key \nprint &quot;auth access secret is: &quot; + auth.access_token.secret \nprint &quot;done with initialization&quot;\n<\/code><\/pre>\n<p>Once you have your key and secret, the following simpler application will post for you:<\/p>\n<pre><code>#!\/usr\/bin\/python\nimport tweepy\n\npost = &quot;I microblog from bash&quot;\nconsumer_token = &quot;&quot;\nconsumer_secret = &quot;&quot;\naccess_key = &quot;&quot;\naccess_secret = &quot;&quot;\n\nhost = 'identi.ca'\napi_root = '\/api\/'\noauth_root = api_root + 'oauth\/'\n\nauth = tweepy.OAuthHandler(consumer_token, consumer_secret, 'oob')\nauth.OAUTH_HOST = host \nauth.OAUTH_ROOT = oauth_root \nauth.secure = True \nauth.set_access_token(access_key, access_secret)\n\napi = tweepy.API(auth, host = host, api_root = api_root)\napi.update_status(post)\n<\/code><\/pre>\n"},{"title":"Property is Theft","link":"https:\/\/current.workingdirectory.net\/posts\/2012\/property-is-theft\/","pubDate":"Sun, 08 Jan 2012 09:32:44 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2012\/property-is-theft\/","description":"<p>I&rsquo;ve recently been making my way through the Proudhon Reader, a collection of writings by Pierre-Joseph Proudhon recently published by AK Press. Proudhon, known as the father of Anarchism is most famous for his declaration: &ldquo;Property is theft.&rdquo;<\/p>\n<p>The simple declaration is based on pages of explanation which, simplistically, can be boiled down to the idea that workers should all equally share in the fruits of their labor. While Proudhon&rsquo;s influence on Marx is hotly debated, this idea is fundamental to Marx&rsquo;s idea around the alienation of labor. If we are working to produce a product that is then owned by the proprietor, rather than the worker, we are alienated from our labor.<\/p>\n<p>Many May First\/People Link members, such as <a href=\"http:\/\/palantetech.org\">Palante Tech<\/a>, <a href=\"http:\/\/openflows.com\">Openflows<\/a>, <a href=\"http:\/\/unionweb.org\">Union Web Services<\/a>, and <a href=\"http:\/\/agaricdesign.com\/\">Agaric Design<\/a> are all committed to fighting this alienation by organizing as worker-run cooperatives. Nobody is the boss, all workers share in all profits.<\/p>\n<p>However, Proudhon got me thinking about free software shops that aren&rsquo;t worker-run cooperatives. If you are employed by a for-profit corporation as a worker and your only job is to write code that is released under a free software license, are you alienated from your labor in the Marxist sense? The fruits of your labor might be <strong>owned<\/strong> by your company, but they are freely licensed to the world (which, of course, includes you).<\/p>\n<p>There is still surplus value accrued to your company (if you do good work for your company, you will be contributing to their reputation from which they will profit). However, if you are primarily writing code, the vast majority of your labor <strong>is<\/strong> producing a product that you are fundamentally not alienated from.<\/p>\n<p>Proudhon, while sprinkling the word revolution throughout his writings, was notably not particularly revolutionary in his calls for action. He was reformist. Rather than calling for a direct confrontation with capitalism, he called for worker-run collectives to be formed throughout the world to make capitalism irrelevant. While many people point to corporate use of free software as an argument for why free software and capitalism are perfectly compatible, the truth may be more complicated. Fundamentally, and in a non-confrontational way, free software seems to undermine one of the basic tenants of capitalist: worker alienation.<\/p>\n<p>Now we just need to work on all the other ways corporate work places are alienating&hellip;<\/p>\n"},{"title":"Removing the user list from gdm","link":"https:\/\/current.workingdirectory.net\/posts\/2012\/remove-users-from-gdm\/","pubDate":"Sun, 08 Jan 2012 09:32:44 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2012\/remove-users-from-gdm\/","description":"<p>UPDATE<\/p>\n<p>Nothing lasts for ever. I just restarted gdm3 (July 29, 2012) and my list of users again popped up.<\/p>\n<p>None of the tips in my original blog or the ones suggested in the comments made any difference.<\/p>\n<p>With help from <a href=\"http:\/\/www.debian-administration.org\/users\/dkg\">dkg<\/a>&rsquo;s grepping &hellip; we found in \/etc\/gdm3\/greeter.gsettings:<\/p>\n<pre><code>#disable-user-list=true\n<\/code><\/pre>\n<p>If you don&rsquo;t have that setting, you can add it under the [org.gnome.login-screen] section. If you do have it, just un-comment it.<\/p>\n<p>No need to dpkg-reconfigure, just restart gdm3.<\/p>\n<p>Also, don&rsquo;t be fooled by IncludeAll or Include in \/etc\/gdm3\/daemon.conf. Those settings don&rsquo;t seem to have any affect.<\/p>\n<p>INFO BELOW IS OUTDATED<\/p>\n<p>I don&rsquo;t want to advertise the list of available logins when gdm starts and presents a login screen on my laptop.<\/p>\n<p>Seems harder to figure out how to do this than it should&hellip;<\/p>\n<p>Thanks to <a href=\"http:\/\/www.hackido.com\/2010\/03\/quick-tip-configure-gdm-to-hide.html\">a helpful blog<\/a> I&rsquo;ve figured out two ways to accomplish the task:<\/p>\n<pre><code>sudo gconftool-2 --direct --config-source xml:readwrite:\/etc\/gconf\/gconf.xml.mandatory --type Boolean --set \/apps\/gdm\/simple-greeter\/disable_user_list True\n<\/code><\/pre>\n<p>or edit:<\/p>\n<p>\/etc\/gconf\/gconf.xml.mandatory\/%gconf-tree.xml<\/p>\n<p>And add:<\/p>\n<pre><code>&lt;?xml version=&quot;1.0&quot;?&gt;\n&lt;gconf&gt;\n  &lt;dir name=&quot;apps&quot;&gt;\n    &lt;dir name=&quot;gdm&quot;&gt;\n      &lt;dir name=&quot;simple-greeter&quot;&gt;\n        &lt;entry name=&quot;disable_user_list&quot; mtime=&quot;1338752251&quot; type=&quot;bool&quot; value=&quot;true&quot;\/&gt;\n      &lt;\/dir&gt;\n    &lt;\/dir&gt;\n  &lt;\/dir&gt;\n&lt;\/gconf&gt;\n<\/code><\/pre>\n"},{"title":"Servers4All... unless someone complains","link":"https:\/\/current.workingdirectory.net\/posts\/2012\/server4all\/","pubDate":"Sun, 08 Jan 2012 09:32:44 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2012\/server4all\/","description":"<p>On Wednesday, February 1, a new virtual server <a href=\"https:\/\/mayfirst.org\/\">May First\/People Link<\/a> recently rented went offline.  We contracted the virtual server through <a href=\"http:\/\/server4all.org\">Server4All<\/a> because we need their un-metered 100Mbit connection to help us handle the bandwidth for <a href=\"http:\/\/saharareporters.com\/\">Sahara Reporters<\/a>, one of the most important independent news sources for Saharan Africa. With the server offline, the web site was down as well.<\/p>\n<p>We scrambled to setup alternative caching servers to handle the bandwidth.<\/p>\n<p>When I logged into our control panel, I saw the message: This virtual server has been suspended by the administrator. Please contact support.<\/p>\n<p>I immediately contacted support and then received the message:<\/p>\n<pre><code>Hello\n\nWe have received the following complaint associated with your server\/service.\n\nIP: 76.73.121.164\n\nTo prevent any further abuse we have suspended this service. In order to\nresume, we request you to cooperate with our investigation as promptly as\npossible. Please respond to us with the following details:\n\n(1) What has caused the complaint\n(2) What is the server used for. Purpose?\n(3) How can you resolve the complaint and make sure it will not be repeated.\n\nDepending on the nature of the complaint and your response, we will put back\nthe server online.  Please note, this has violated our Terms Of Service. We\nexpect your response within 24 hours, otherwise your account will be\nterminated permanently.  Thank you\n<\/code><\/pre>\n<p>What?? What complaint?? I followed up but had to wait til the next day to get the response.<\/p>\n<pre><code>Here is the full log,\n\nAn email advertizing the Domain Name: saharareporters.com\nhas been sent to the blacklist.woody.ch spamtrap.\n\nThis Domain does resolve to IP addresses one of which your are responsible:\n76.73.121.164\n\nPlease investigate why this Domain has been advertized.\n\nAttached you find the headers and reports in ARF for automatic processing.\nFeedback is appreciated.\n\nActual listing periods:\n\nBounce: 1 Hour in DNS.\nWhitelisted IP: Not lised in DNS.\nSpam: 24 hours in DNS.\n\nEvery Hit: 14 days in evidence DB.\n\nFor any questions or Feedback, contact abuse@woody.ch\n\nFrom: is intentionally set to a bit-bucket.\n\nKind regards\n-Benoit Panizzon- \n<\/code><\/pre>\n<p>There is no attachment. I went to woody.ch and it was in German. Then tried blacklist.woody.ch, but no luck. Finally I found the <a href=\"http:\/\/blacklist.woody.ch\/rblcheck.php3\">Woody&rsquo;s World Blacklist Page<\/a>. I plugged in our IP address into their checker and I got:<\/p>\n<pre><code>Output from the Check, if empty the IP is not listed.\n164.121.73.76.[name of the blacklist] being tested.\n\nHost 164.121.73.76.blacklist.woody.ch not found: 3(NXDOMAIN)\nHost 164.121.73.76.blacklist.woody.ch not found: 3(NXDOMAIN)\n\nHost 164.121.73.76.rbl.maps.vix.com not found: 3(NXDOMAIN)\nHost 164.121.73.76.rbl.maps.vix.com not found: 3(NXDOMAIN)\n\nHost 164.121.73.76.relays.mail-abuse.org not found: 3(NXDOMAIN)\nHost 164.121.73.76.relays.mail-abuse.org not found: 3(NXDOMAIN)\n\n;; connection timed out; no servers could be reached\n;; connection timed out; no servers could be reached\n\n;; connection timed out; no servers could be reached\n;; connection timed out; no servers could be reached\n\nHost 164.121.73.76.relays.ordb.org not found: 3(NXDOMAIN)\nHost 164.121.73.76.relays.ordb.org not found: 3(NXDOMAIN)\n\nHost 164.121.73.76.dev.null.dk not found: 3(NXDOMAIN)\nHost 164.121.73.76.dev.null.dk not found: 3(NXDOMAIN)\n\nHost 164.121.73.76.blackholes.five-ten-sg.com not found: 3(NXDOMAIN)\nHost 164.121.73.76.blackholes.five-ten-sg.com not found: 3(NXDOMAIN)\n\nHost 164.121.73.76.bl.spamcop.net not found: 3(NXDOMAIN)\nHost 164.121.73.76.bl.spamcop.net not found: 3(NXDOMAIN)\n\nHost 164.121.73.76.relays.visi.com not found: 3(NXDOMAIN)\nHost 164.121.73.76.relays.visi.com not found: 3(NXDOMAIN)\n\n164.121.73.76.blacklist.spambag.org has address 208.91.197.182\n164.121.73.76.blacklist.spambag.org descriptive text &quot;v=spf1 -all&quot;\n<\/code><\/pre>\n<p>So, Woody&rsquo;s World thinks we are listed in spambag.org. I went to <a href=\"http:\/\/spambag.org\">spambag.org<\/a> and learned that the domain is for sale. I then tried <a href=\"http:\/\/blacklist.spambag.org\">blacklist.spambag.org<\/a> and got the same page. This page has many links all pointing to advertisements. The &ldquo;RBL List&rdquo; link takes me to a page advertising &ldquo;5 foods you must not eat.&rdquo;<\/p>\n<p>Next, out of curiousity, I tried <a href=\"http:\/\/www.mxtoolbox.com\/SuperTool.aspx\">Mxtoolbox<\/a>. I got one hit from Barricuda. Barricuda says the reputation of the IP address is &ldquo;poor&rdquo;. Why? According to Barricuda, the reasons could be:<\/p>\n<ul>\n<li>Your email server contains a virus and has been sending out spam.<\/li>\n<li>Your email server may be improperly configured.<\/li>\n<li>Your PC may be infected with a virus or botnet software program.<\/li>\n<li>Someone in your organization may have an infected PC with a virus or botnet program.<\/li>\n<li>You may be using a dynamic IP address which was previously used by a known spammer.<\/li>\n<li>Your marketing department may be sending out bulk emails that do not comply with the CAN-SPAM Act.<\/li>\n<li>You may have an insecure wireless network which is allowing unknown users to use your network to send spam.<\/li>\n<li>In some rare cases, your recipient&rsquo;s Barracuda Spam Firewall may be improperly configured.<\/li>\n<\/ul>\n<p>Keep in mind, this IP address is not used for sending email. It&rsquo;s just the web site.<\/p>\n<p>I then took a step back and re-read the complaint and noticed that it says that the domain name saharareporters.com was listed in a spam email. Hm. More concerted searching for the terms &ldquo;woody spamtrap blacklist&rdquo; and I found a pattern in URLs that suggested I plug in the following:<\/p>\n<p><a href=\"http:\/\/news.scoutnet.org\/rblhostlist.php?id=saharareporters.com.uri\">http:\/\/news.scoutnet.org\/rblhostlist.php?id=saharareporters.com.uri<\/a><\/p>\n<p>And sure enough, there was a result. In short, it was a classic Nigerian Oil scam in which the person claims to be &ldquo;JAMES IBORI ex-governor of DELTA STATE oil city.&rdquo; The scammer acknowledges that he has been arrested, but promises lots of cash to the person who can help him. In an effort to boost their credibility, the scammer included a link to a Sahara Reporters article about the real <a href=\"http:\/\/saharareporters.com\/news-page\/uk-money-laundering-trial-james-ibori-suffers-various-reverses\">James Ibori<\/a>.<\/p>\n<p>And that, my friends, is enough to have one of the most prominent independent African news organization taken offline.<\/p>\n<p><del>But, Sahara Reporters shouldn&rsquo;t necessarily feel singled out.<\/del> Sahara Reporters should feel singled out. They exist to illuminate news from Africa. They were taken offline because a series of individuals don&rsquo;t know the difference between a prominent independent African news service and a criminal scammer. To most of the Internet, Nigeria and email scams are synonymous. That has to change.<\/p>\n<p>Although Sahara Reporters is particularly vulnerable, any site hosted with Server4All can potentially be taken down. All you have to do is write a fake spam\/scam email, including a link to the web site you want to be taken offline, and then send that email to: <a href=\"mailto:listme@blacklist.woody.ch\">listme@blacklist.woody.ch<\/a>.<\/p>\n<p>I&rsquo;m currently following up with both Woody&rsquo;s World and Server4All. However, once this particular issue is resolved, we&rsquo;re left with a much bigger and ominous problem. If your hosting provider (or their upstream provider) takes your site offline when it receives a complaint first and then asks questions second, you have a big problem.<\/p>\n<p>All of our legal fights over our rights to keep content online are moot if our providers, without any legal pressure to do so, still take down our services based on spurious complaints.<\/p>\n"},{"title":"Sparkleshare","link":"https:\/\/current.workingdirectory.net\/posts\/2012\/sparkleshare\/","pubDate":"Sun, 08 Jan 2012 09:32:44 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2012\/sparkleshare\/","description":"<p><a href=\"http:\/\/sparkleshare.org\/\">Sparkleshare<\/a> is a file sharing utility that keeps folders in sync on multiple computers in a similar fashion to the commercial and proprietary Dropbox. Sparkleshare recently released a Windows client, finally allowing me to start switching my co-workers at the <a href=\"http:\/\/progressivetech.org\/\">Progressive Technology Project<\/a> away from Dropbox.<\/p>\n<p>Overall, I&rsquo;m very impressed. In particular, I appreciate the Sparkleshare authors&rsquo; decision to build on top of existing tools (git for storage and revision control, ssh for transport, and ssh public key infrastructure for authorization and authentication). That means I don&rsquo;t have to learn new tools and protocols to debug and it means Sparkleshare can focus on the file sharing pieces.<\/p>\n<p>Despite my overall enthusiasm, I do have some serious concerns.<\/p>\n<h2 id=\"protecting-your-credentials\">Protecting your credentials<\/h2>\n<p>A researcher found a <a href=\"http:\/\/dereknewton.com\/2011\/04\/dropbox-authentication-static-host-ids\/\">startling security flaw in Dropbox<\/a> - if you can copy a particular file from a user&rsquo;s computer to your own computer, you can impersonate them flawlessly, accessing all their Dropbox files without having to know the user&rsquo;s password. Unfortunately, Sparkleshare suffers from this same vulnerability if you use the default configuration.<\/p>\n<p>When you first install Sparkleshare, it creates a password-less ssh private\/public key pair and then makes the public part easily accessible to you so you can add it to your server (or gitorious account, etc). Very convenient. But it also means that all an attacker needs to do is copy your private key (and your Sparkleshare configuration file) and they get complete control over your files.<\/p>\n<p>This problem is easily avoided. If you are running Linux or Mac OS X and you have your own key loaded in your ssh agent, Sparkleshare will happily use that key. So - simply by using your existing (presumably password-protected key) on your remote Sparkleshare servers, you can mitigate this problem. Sparkleshare will still load your Sparkleshare generated key, but if you don&rsquo;t provide that key with access to anything, no harm is done.<\/p>\n<h2 id=\"confirming-each-use-of-your-key\">Confirming each use of your key<\/h2>\n<p>However&hellip; that leads to a new problem. If you are like me, your ssh agent is configured to ask for a confirmation every time your key is used. And, Sparkleshare regularly polls the remote git repository for changes. At best repeatedly clicking to confirm is tedious. At worst, it prevents you from intelligently rejecting malicious requests, thus defeating the whole purpose of the check.<\/p>\n<p>It is possible to launch Sparkleshare via ssh-agent in an environment in which you are not requiring the confirmation when Sparkleshare uses your key, but still requiring it for all other uses. However, given the trade offs, I&rsquo;ve decided to add a password to my Sparkleshare provided ssh key rather than using my existing key:<\/p>\n<p>ssh-keygen -p -f ~\/.config\/sparkleshare\/sparkleshare.jamie@progressivetech.org.key<\/p>\n<p>Now, I am prompted to enter my passphrase when I start Sparkleshare and don&rsquo;t have to confirm every use of the key. And, I continue to confirm each use of my main key.<\/p>\n<h2 id=\"git-was-designed-to-store-code-not-documents\">git was designed to store code, not documents<\/h2>\n<p>Just because something is designed for one purpose doesn&rsquo;t mean it can&rsquo;t be used for another. However, there are a few limitations.<\/p>\n<p>Large files are one. git can handle files over 100MB, but may run into memory problems. I had to alter the <a href=\"http:\/\/stackoverflow.com\/questions\/4826639\/repack-of-git-repository-fails\">git windowMemory setting<\/a>, raising it higher than the size of the largest file.<\/p>\n<p>Another problem is disk space. Since git keeps full revision history on every machine, you have to download more data than just the files that are checked out. The more edits you make to a repository, the more disk space beyond what is checked out is needed (and deleting files won&rsquo;t help).<\/p>\n<p>Lastly, you can&rsquo;t use file modification times. With git, the file modification time will depend on when you checkout the files. With some fancy hook writing, you could tweak things so that the file modification date is the same as the commit date, but that still won&rsquo;t help you if you add an existing directory to Sparkleshare because all the files will have the same commit time.<\/p>\n<p>This &ldquo;bug&rdquo; has been reported to git and it has been rejected because mucking with the modification time of files can have bad results when you are using make to compile code. As Linus <a href=\"http:\/\/kerneltrap.org\/mailarchive\/git\/2007\/3\/5\/240536\">colorfully put it<\/a>:<\/p>\n<pre><code>I'm sorry. If you don't see how it's WRONG to set a datestamp back to something that will make a simple &quot;make&quot; miscompile your source tree, \nI don't know what definition of &quot;wrong&quot; you are talking about.\nIt's WRONG.\nIt's STUPID.\nAnd it's totally INFEASIBLE to implement.\n<\/code><\/pre>\n<p>Well, did I mention that git was designed for source code?<\/p>\n<h2 id=\"no-server-validation\">No server validation<\/h2>\n<p>Of all the problems I encountered, this one is by far the most serious.<\/p>\n<p>Perusing ~\/.config\/sparkleshare\/debug.log is very informative. On a default installation, after you have added a project, you&rsquo;ll see:<\/p>\n<pre><code>19:32:27 [Fetcher][\/home\/jamie\/SparkleShare\/.tmp\/bar] Fetching folder: ssh:\/\/foo@bar.org\/foo\/bar\n19:32:27 [Fetcher] Disabled host key checking for bar.org\n<\/code><\/pre>\n<p>Wah.<\/p>\n<p>Once the project is added, whatever key fingerprint is given is stuffed into your ~\/.ssh\/known_hosts file. Host key checking is only disabled when you initially create the project, so <em>if<\/em> you connect to the proper server the first time, you are protected from subsequent man-in-the-middle attacks. However&hellip; if the initial fingerprint is wrong, your personal ssh configuration is now poisoned. This seems like a very bad idea. I&rsquo;ve opened a <a href=\"http:\/\/bugs.debian.org\/cgi-bin\/bugreport.cgi?bug=671864\">debian bug<\/a> to address it.<\/p>\n"},{"title":"Capitalism Is Crashed","link":"https:\/\/current.workingdirectory.net\/posts\/2011\/capitalism-is-crashed\/","pubDate":"Sat, 08 Jan 2011 09:32:44 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2011\/capitalism-is-crashed\/","description":"<p>The Occupy Wall Street protests taking place around the world are incredibly moving.<\/p>\n<p>And, at <a href=\"https:\/\/mayfirst.org\/\">May First\/People Link<\/a>, the joint us jumping with new projects. <a href=\"http:\/\/occupytogether.org\">Occupy Together<\/a> was the first to join, followed by the <a href=\"http:\/\/occupiedmedia.com\">Occupy Wall Street Journal<\/a>. We then <a href=\"https:\/\/mayfirst.org\/lowdown\/action\/mfpl-offering-free-hosting-occupy-movement-organizations\">launched boggs.mayfirst.org as our dedicated Occupy Wall Street shared server<\/a> and the radical tech work keeps on coming (<a href=\"http:\/\/occupytechnology.org\/\">OccupyTechnoloyg<\/a>, <a href=\"http:\/\/occupyprovidence.com\">OccupyProvidence<\/a>, <a href=\"http:\/\/occupycolumbusga.org\/\">OccupyColumbus GA<\/a> just to name a few).<\/p>\n<p>It&rsquo;s also great to see free software and <a href=\"https:\/\/mayfirst.org\/lowdown\/guest-article\/progressive-technology-principles-and-software-service\">responsible software as a service<\/a> making it&rsquo;s way into the conversation. I&rsquo;ve heard many reports about the politics of technology taking it&rsquo;s rightful place in the realm of political decisions being made.<\/p>\n<p>I don&rsquo;t have the context for this photo from the blog <a href=\"http:\/\/www.nontistavocercando.it\/\">non ti stavo cercando<\/a>, but it sure sums up the connection :<\/p>\n<p><img src=\"http:\/\/www.nontistavocercando.it\/wp-content\/uploads\/2011\/10\/DSC_0137r-500x332.jpg\" alt=\"Capitalism is Crashed! Install news system?\"  title=\"YES! apt-get install anarchism\" \/>\n<\/p>\n"},{"title":"Privilege Separation","link":"https:\/\/current.workingdirectory.net\/posts\/2011\/privilege-separation\/","pubDate":"Sat, 08 Jan 2011 09:32:44 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2011\/privilege-separation\/","description":"<p>What are the biggest security threats to my laptop? Almost all the software I install is vetted and signed by a member of the Debian team. I run reliable virus software on our mail server. My disk is encrypted and xscreensaver locks my screen after a few minutes of inactivity. What else?<\/p>\n<p>The two biggest threats I&rsquo;ve recently considered are: web browsing and non-free software or software that doesn&rsquo;t come from Debian (I regularly have to use skype and zimbra, for example).<\/p>\n<p>To mitigate these risks, I&rsquo;ve configured these programs to run as their own users, thus adding a layer of separation between the programs and my primary user account, which has access to my gpg\/ssh keys and email.<\/p>\n<p>With a program like zimbra it&rsquo;s fairly easy. I created a zimbra user, added my primary account&rsquo;s public ssh key to \/home\/zimbra\/.ssh\/authorized_keys, and add the following stanza to ~\/.ssh\/config:<\/p>\n<pre><code>Host zimbra \nHostname localhost \nUser zimbra \nForwardX11 yes\n<\/code><\/pre>\n<p>Now I can start zimbra with:<\/p>\n<pre><code>ssh zimbra .\/zdesktop-launch\n<\/code><\/pre>\n<p>Skype was a little harder, since the skype client has to access the audio system. With <a href=\"http:\/\/packages.debian.org\/search?keywords=pulseaudio\">pulseaudio<\/a>, though, it&rsquo;s a snap. I copied \/etc\/pulse\/default.pa to ~\/.pulse\/ and added the line:<\/p>\n<pre><code>load-module module-native-protocol-tcp auth-ip-acl=127.0.0.1\n<\/code><\/pre>\n<p>Then, I added \/home\/skype\/.pulse\/client.conf with the contents:<\/p>\n<pre><code>default-server = 127.0.0.1\n<\/code><\/pre>\n<p>[Note: Jamie Rollins and dkg have pointed out that this arrangement allows any user on my laptop to send arbitrary data to pulseaudio, running as my primay account. They suggested configuring pulseaudio to listen on a unix-domain socket, and then configuring pulseaudio to only permit access to users in a particular group.]<\/p>\n<p>iceweasel is the most complicated. In addition to the pulseaudio trick, I had to make two other allowances.<\/p>\n<p>First, there are a lot of processes that launch a web browser in a number of different ways (sometimes asking for a new session, other times adding a tab to an existing one, sometimes passing an URL as an argument, etc).<\/p>\n<p>The one that I got the most stuck on was mutt. Sometimes I want to see how an HTML message looks in iceweasel. Via mailcap, mutt creates a temp file with the html content and then launches a web browser to view the file. As can be reasonably expected, this temp file is owned by my primary user, and only read-able by the owner. That means my iceweasel user can&rsquo;t read it.<\/p>\n<p>Eventually, I decided the easiest way to deal with these various scenarios was to write a simple bash script to launch my web browser (see below). I registered it via update-alternatives, so most reasonable programs that want to launch a web browser will use it.<\/p>\n<p>The second issue is that I use the <a href=\"http:\/\/packages.debian.org\/search?keywords=xul-ext-monkeysphere\">monkeysphere xul plugin<\/a> to verify TLS certificates, which requires iceweasel to communicate with my monkeysphere validation agent. My agent runs as my primary user and by default only responds to queries from my primary user.<\/p>\n<p>Fortunately, monkeysphere is well-designed and can handle this situation. As you can see from my web launcher script, I pass MONKEYSPHERE_VALIDATION_AGENT_SOCKET=$MONKEYSPHERE_VALIDATION_AGENT_SOCKET when calling iceweasel. In addition, I added the following before I exec monkeysphere-validation-agent:<\/p>\n<pre><code>export MSVA_ALLOWED_USERS=&quot;iceweasel jamie&quot;\n<\/code><\/pre>\n<p>With this simple infrastructure setup, it&rsquo;s possible to easily isolate future programs as well.<\/p>\n<p>Lastly&hellip; here&rsquo;s the script for launching iceweasel.<\/p>\n<pre><code>#!\/bin\/bash\n\nstdin=\nnew_session=no\nurl=\n\nfor arg in &quot;$@&quot;; do\n  if [[ &quot;$arg&quot; =~ ^-- ]]; then \n    if [ &quot;$arg&quot; = &quot;--new-session&quot; ]; then\n      new_session=yes\n    elif [ &quot;$arg&quot; = &quot;--from-stdin&quot; ]; then\n      stdin=yes\n    fi\n  else\n    url=&quot;$1&quot;\n  fi\ndone\n\nif [ &quot;$stdin&quot; = &quot;yes&quot; ]; then\n  temp=$(mktemp)\n  while read line; do\n    echo &quot;$line&quot; &gt;&gt; &quot;$temp&quot;\n  done\n  # it must be readable by the iceweasel user\n  chmod 755 &quot;$temp&quot;\n  url=&quot;file:\/\/\/$temp&quot;\nfi\n\nargs=\nif [ &quot;$new_session&quot; = &quot;yes&quot; ]; then\n  args=&quot;--no-remote -ProfileManager&quot;\nfi\n\nif [ -n &quot;$url&quot; ]; then\n  args=&quot;$args '$url'&quot;\nfi\n\nssh iceweasel &quot;MONKEYSPHERE_VALIDATION_AGENT_SOCKET=$MONKEYSPHERE_VALIDATION_AGENT_SOCKET iceweasel $args&quot; &amp;\n\n[ -f &quot;$temp&quot; ] &amp;&amp; sleep 5 &amp;&amp; rm &quot;$temp&quot;\n<\/code><\/pre>\n<p>Comments by email&hellip;<\/p>\n<p>Hi,<\/p>\n<p><a href=\"http:\/\/current.workingdirectory.net\/posts\/2011\/privilege-separation\/\">http:\/\/current.workingdirectory.net\/posts\/2011\/privilege-separation\/<\/a> has good intentions but afaik it does not improve security much. X applications can sniff your passwords and inject commands to your terminal emulators.<\/p>\n<p>I personally use xpra to get a similar solution without the hazards of X. I&rsquo;ve been using it for two months now both at work and home. There are still bugs but svn version is getting better all the time.<\/p>\n<p>See for example my reply to<\/p>\n<p><a href=\"https:\/\/grepular.com\/Protecting_a_Laptop_from_Simple_and_Sophisticated_Attacks\">https:\/\/grepular.com\/Protecting_a_Laptop_from_Simple_and_Sophisticated_Attacks<\/a><\/p>\n<p>Timo<\/p>\n"},{"title":"Puppet Without Masters","link":"https:\/\/current.workingdirectory.net\/posts\/2011\/puppet-without-masters\/","pubDate":"Sat, 08 Jan 2011 09:32:44 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2011\/puppet-without-masters\/","description":"<p><a href=\"http:\/\/puppetlabs.com\">Puppet labs<\/a>&rsquo;s use of the term puppetmaster is rather clever (in contrast to other <a href=\"https:\/\/current.workingdirectory.net\/posts\/2011\/master-slave\">un-necessarily offensive uses of &ldquo;master&rdquo; in the software world<\/a>).<\/p>\n<p>While I appreciate the clever name, I&rsquo;m less impressed with the concept.<\/p>\n<p>At <a href=\"http:\/\/mayfirst.org\">May First\/People Link<\/a> we&rsquo;ve spent the last several years (including the last couple months in earnest) working to transition management of our 90-some servers from a collection of hand-written bash scripts to puppet.<\/p>\n<p>Over the years, we&rsquo;ve worked hard to keep our servers as secure as possible. We have a team of about a half dozen people who all have root access on all servers. It&rsquo;s all key-based access. To help mitigate a disaster if one person&rsquo;s keys were compromised, we&rsquo;ve implemented <a href=\"http:\/\/monkeysphere.info\/\">monkeysphere<\/a> on all servers, allowing us to easily revoke access.<\/p>\n<p>After spending so much time thinking through our root-access strategy and fully implementing the monkeysphere to reduce our exposure to a single point of vulnerability, I was disappointed by puppet&rsquo;s use of a puppet master. For those less familiar with puppet, it goes something like this:<\/p>\n<p>One server (or god forbid multiple servers), run an externally accessible daemon. Each and every server on your network runs a daemon <strong>as root<\/strong> that periodically communicates with the puppet master, receives new instructions, and then (again, as root) executes these instructions.<\/p>\n<p>In other words, if your puppet master is compromised, I&rsquo;m not sure exactly what you would need to do, short of rebuilding every server in your network.<\/p>\n<p>To make matters worse, it seems as though some users generate and store all server ssh keys (private and public) on the puppet master and then push the private keys to their respective nodes. That means an intruder doesn&rsquo;t need to write to the puppet master, just reading these keys would be enough to compromise all servers in your network.<\/p>\n<p>There must be a better way.<\/p>\n<h2 id=\"puppet-without-masters\">Puppet without masters<\/h2>\n<p>After some web-searching, I found a <a href=\"http:\/\/www.mailinglistarchive.com\/html\/puppet-users@googlegroups.com\/2011-02\/msg00162.html\">promising thread on the puppet list asking what&rsquo;s lost without a puppet master<\/a>. This thread lead to a <a href=\"http:\/\/semicomplete.com\/presentations\/puppet-at-loggly\/puppet-at-loggly.pdf.html\">couple<\/a> <a href=\"http:\/\/bitfieldconsulting.com\/scaling-puppet-with-distributed-version-control\">other<\/a> blogs by people who have worked out a system for using puppet without a master.<\/p>\n<p>It turns out that there are two distinct points of centralization with puppet. One is the puppet master (as described above). In addition, there is a concept called <a href=\"http:\/\/projects.puppetlabs.com\/projects\/1\/wiki\/Using_Stored_Configuration\">storeconfigs<\/a> - which allows each node in the network to store information in a central database. For example, one server can store a request for an account to be setup on a backup server. The next time the backup server runs, it checks the storeconfigs, finds the request, and creates the user.<\/p>\n<p>It&rsquo;s possible to run puppet <em><strong>with<\/strong><\/em> storeconfigs but <em><strong>without<\/strong><\/em> running a puppet master (that avoids the hassle of running the puppet daemons, while providing the convenience of centralization). For our purposes, however, we decided to forego both the puppet master and storeconfigs. We did not want any form of centralization that would provide an additional point of vulnerability.<\/p>\n<p>As is common with puppet, we are storing our puppet recipes in a git repository. And, we are publishing to a single, canonical git repository on the Internet. On each node, we have two git repositories - one is a bare repo (that we can push to) and the other is a checked out repo (in \/etc\/puppet) that is read by puppet. The bare repo has a post-update hook the changes into the \/etc\/puppet directory, pulls in the changes from the bare repository, and runs puppet against the newly checked out files. Therefore, we can apply new puppet recipes to any server on the network with<\/p>\n<pre><code>git push &lt;server&gt;\n<\/code><\/pre>\n<p>No daemons: neither a master daemon nor a puppet daemon running on the node using up memory or providing a potential security hole. The git push happens over an ssh connection - since all system administrators already have root-level ssh access on every server - there is no need to grant any additional access above what we already have.<\/p>\n<p>Pushing works great - but with 90 nodes we don&rsquo;t want to have to push to 90 servers  everytime we want a change made. That&rsquo;s where the canonical git repository comes in. A cron job runs a script on each node once an hour that runs <code>git remote update<\/code> from \/etc\/puppet. The script then checks the time stamp on the most recent gpg-signed tag and compares it with the time stamp of the current commit. If the most recent gpg-signed tag is newer, it verifies that the tag came from a list of authorized gpg keys (the very same gpg keys used by the monkeysphere to grant root level ssh access). If the gpg signature of the tag can be properly verified, then the changes are merged and puppet is run on the new recipes.<\/p>\n<h2 id=\"what-about-privacy\">What about privacy?<\/h2>\n<p>One of the benefits of a puppet master setup is that nodes get configuration details on a need-to-know basis. The puppet master doesn&rsquo;t share the entire puppet repo - only the compiled manifest for the node with which it&rsquo;s communicating.<\/p>\n<p>Our solution to this problem was to go screaming in the other direction. As you might notice from our <a href=\"https:\/\/support.mayfirst.org\/\">support wiki and ticket system<\/a>, we generally favor transparency. Since we are <a href=\"https:\/\/git.mayfirst.org\/?p=mfpl\/puppet.git;a=summary\">publishing our entire puppet git repo<\/a> publicly, there seems little point in trying to hide one node&rsquo;s configuration details from another node.<\/p>\n<p>That also means each node carries around about 4Mb of extra weight in the form of disk space for the git repo. That seems like a small price to pay for the resource savings of not running a puppetd process all the time.<\/p>\n<h2 id=\"more-differences\">More differences<\/h2>\n<p>As I&rsquo;ve read the puppet lists, faqs and documentation, I&rsquo;ve found yet more ways our use of puppet diverges from the norm.<\/p>\n<p>The first is a little thing really - most people seem to store all their node configurations in a single nodes.pp file. I&rsquo;m not sure why. Fortunately, puppet&rsquo;s include syntax allows globbing, so we&rsquo;ve created a directory and gave each server it&rsquo;s own .pp file. This arrangement makes it much easier to parse the configuration with tools other than puppet (like, Q. How many servers do we have? A. <code>ls | wc -l<\/code>).<\/p>\n<h2 id=\"backup-and-nagios-monitoring-without-storeconfigs\">Backup and Nagios monitoring without storeconfigs<\/h2>\n<p>More significantly - there are some things we can&rsquo;t do since we are not using storeconfigs. While many puppet users add a variable, like <code>$nagios = true<\/code> before including their sshd class (which then causes the sshd class to store a configuration for the nagios server to monitor ssh on the node in question), we were forced to come up with alternatives.<\/p>\n<p>My first solution was to simply list all the servers that needed to be monitored in the server node configuration file for the nagios server. Ditto for the backup servers. This approach, however, proved cumbersome and error prone. When adding a new node, I now have to edit three files instead of one. And, how can I easily tell if all nodes have their nagios and\/or backup configurations set? The solution was rather simple - there&rsquo;s more than one way to store a config for another node. Our nagios server is called jojobe.mayfirst.org and our backup server is luisa.mayfirst.org. A typical node.pp file looks like this:<\/p>\n<pre><code>node pietri.mayfirst.org {\n  # node config goes here\n\n}\nif ( $fqdn == &quot;jojobe.mayfirst.org ) {\n  nagios_montior { &quot;pietri&quot;: }\n}\nif ( $fqdn == &quot;luisa.mayfirst.org ) {\n  backup_access { &quot;pietri&quot;: }\n}\n<\/code><\/pre>\n<p>This way all configuration related to pietri stays in a single file.<\/p>\n<h2 id=\"host-keys-and-granting-access-between-servers\">Host keys and granting access between servers<\/h2>\n<p>storeconfigs is commonly used to distribute host ssh keys. Every node that is added to puppet has it&rsquo;s ssh host key stored centrally and then re-distributed to every other node. That way, you can ssh from node to node without ever getting the ssh fingerprint verification. Avoiding that prompt is particularly important when backing up from one server to another via automated scripts. storeconfigs can additionally be used to copy user&rsquo;s public ssh keys - thus granting user access between servers.<\/p>\n<p>Our solution to this problem: <a href=\"http:\/\/monkeysphere.info\">monkeysphere<\/a>. Rather than maintaining our own private data store of keys, we publish (and sign) our ssh keys via the web of trust. In addition to server keys, each one of our servers&rsquo; root user has an ssh-enabled gpg key (also publicly signed by us). By configuring each server to trust our system administrators&rsquo; gpg keys for verifying other keys, we can both avoid the ssh fingerprint manual verification step and we can grant a root user on one server access to another server by simply dropping <code>root@$server.mayfirst.org<\/code> into an authorized_user_ids file on the target server.<\/p>\n<p>There&rsquo;s no question - the setup was rather tedious (we&rsquo;re using runit to maintain an ssh-agent for each root user), however, now that&rsquo;s in place (and configured via puppet), it&rsquo;s a breeze to add new servers. The only extra step we have to take is to confirm and sign each new server&rsquo;s keys. This &ldquo;extra&rdquo; step not only allows our servers to verify each other, but also allows our users to verify the servers as well, so it&rsquo;s hardly an extra step at all.<\/p>\n<h2 id=\"shared-modules\">Shared modules<\/h2>\n<p>There&rsquo;s a vibrant community of third party module developers for puppet. Rather than figure out the intricacies of having puppet configure sshd, for example, you can install a contributed sshd module and then you simply add:<\/p>\n<pre><code>include sshd\n<\/code><\/pre>\n<p>And you get a default sshd setup. Many of these modules are fairly well developed, offering the ability to easily customize your setup in a number of different ways.<\/p>\n<p>Unfortunately, most of the modules assume you are using storeconfigs and if you are not, they will either fail to work right or you will get noisy errors. At first, this seemed like a problem. However, as I built our puppet recipes, I found myself increasingly frustrated with the third party modules that we could use.<\/p>\n<p>Configuring servers is hard - and requires constant debugging and trouble shooting. puppet already provides a layer of abstraction between you and the server you are setting up. Given the benefits of puppet, I&rsquo;m willing to spend the time learning the puppet syntax and asking the rest of our system administrators to do the same. This layer of abstraction is further compounded by our use of git to store the configurations (not a problem if you are git hero - but most of us are already struggling to get a handle on using git). Again, all seems worth it for the pay off.<\/p>\n<p>Now enter the puppet module. In addition to learning puppet syntax (and struggling with git) you now need to understand how the third party module works. With software programming, I typically don&rsquo;t need or want to learn how a library or class does what it does - that&rsquo;s the beauty of object-oriented programming: it hides the complexity. But when it comes to configuring the servers that I will be responsible for debugging and maintaining, I really need to know exactly what is happening.<\/p>\n<p>To further compound the problem, I found myself wading through third party module code designed to work on Debian, Ubuntu, CentOS, Redhat, gentoo&hellip; and more. We run entirely on Debian - we don&rsquo;t need any of this extra code. And, once I got rid of all the other operating systems, I was still left with a complex module that allows you to configure software in ways we&rsquo;ll never need.<\/p>\n<p>In the end, we tore out most of these third party modules and replaced them with file and exec puppet resources that did exactly what we needed them to do. Our code base is now much smaller and simpler.<\/p>\n<h2 id=\"not-just-a-whiner\">Not just a whiner<\/h2>\n<p>I have a lot more to whine about (like why native resources for things like nagios that are so easily handled with the file resource?).<\/p>\n<p>However - the remarkable thing about puppet is that it&rsquo;s flexible. Despite some fairly substantial problems with the &ldquo;typical&rdquo; use of puppet, the program provides enough flexibility for us to use it in a way that fully meets our needs. After having built my own bash-based set of configuration scripts and deeply exploring puppet, I have a great appreciation for the difficulty of building system configuration software (we considered and rejected cf-engine and chef as not being any better).<\/p>\n<p>And, if you are still not convinced that puppet will work fo you &hellip; you might consider a package I learned about <strong>after<\/strong> going down the puppet route: <a href=\"https:\/\/code.google.com\/p\/slack\/\">slack<\/a>.<\/p>\n"},{"title":"Testing x509 Certificates","link":"https:\/\/current.workingdirectory.net\/posts\/2011\/testing-x509-certs\/","pubDate":"Sat, 08 Jan 2011 09:32:44 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2011\/testing-x509-certs\/","description":"<p>Navigating the openssl suite of subcommands is time consuming.<\/p>\n<p>Here&rsquo;s my list of frequently used commands for easy reference:<\/p>\n<p>Examining the certificates being offered by a host (replace imap\/https with any service, -starttls imap can be eliminated if you are checking a service that provides tls from the start):<\/p>\n<pre><code>openssl s_client -starttls imap -showcerts -connect chavez.mayfirst.org:imap\nopenssl s_client -showcerts -connect chavez.mayfirst.org:https\n<\/code><\/pre>\n<p>Examine a certificate signing request:<\/p>\n<pre><code>openssl req -text -verify -noout -in \/path\/to\/file.csr \n<\/code><\/pre>\n<p>Examine a x509 certificate:<\/p>\n<pre><code>openssl x509  -noout -text -purpose -in path\/to\/file.crt\n<\/code><\/pre>\n"},{"title":"We need better metaphors","link":"https:\/\/current.workingdirectory.net\/posts\/2011\/master-slave\/","pubDate":"Sat, 08 Jan 2011 09:32:44 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2011\/master-slave\/","description":"<p>Most of us developers use metaphors to help convey an understanding of how our software works. Sadly, many developers choose the &ldquo;master&rdquo; and &ldquo;slave&rdquo; metaphor.\nWhy would we voluntarily choose such an ugly human human interaction with a real and present history in many of our lives to describe software that we&rsquo;ve written?<\/p>\n<p>I wouldn&rsquo;t rule out the possibility that someone will (or already has) invented software for which the master\/slave metaphor aptly conveys its function, however, I&rsquo;d prefer not to use it. Also, many people <strong>intentionally<\/strong> play with master\/slave relationships because of the history and connotations it provokes.<\/p>\n<p>However, use of this metaphor in software like <a href=\"http:\/\/dev.mysql.com\/doc\/refman\/5.0\/en\/replication.html\">MySQL databases<\/a> or <a href=\"http:\/\/www.zytrax.com\/books\/dns\/ch4\/#master\">Bind Name servers<\/a>  seems pointless, offensive and mis-leading.<\/p>\n<p>I&rsquo;ve never seen the use of the master\/slave metaphor in software that conveyed more than the idea that one piece of the software essentially controls, wholly or in part, another piece of software. So, why master\/slave? There are many metaphors to choose from - parent\/infant, boss\/worker, guard\/prisoner, king\/subject, landowner\/serf, top\/bottom, etc.<\/p>\n<p>What does master\/slave convey that the others don&rsquo;t convey?<\/p>\n<p>Or, maybe they all convey the same thing, so it doesn&rsquo;t make a difference which one you choose (after all, it&rsquo;s just a metaphor). In that case, how about replacing master\/slave with the metaphor husband\/wife? Whoops. Now that&rsquo;s offensive (if you don&rsquo;t think that&rsquo;s offensive you can stop reading here and please don&rsquo;t post any comments - I won&rsquo;t change your mind).<\/p>\n<p>But, that&rsquo;s different right? At one point in history, the role of the wife was to do as her husband says, but it&rsquo;s 2011, and we&rsquo;ve evolved beyond that, so by using husband\/wife we&rsquo;d be re-enforcing that ugly, sexist stereotype.<\/p>\n<p>However, how is that different from master\/slave? It <strong>is<\/strong> 2011 after all. Isn&rsquo;t the role of the master to abdicate and the role of the slave to revolt?<\/p>\n<p>Maybe that&rsquo;s why Bind has so many security problems&hellip;<\/p>\n"},{"title":"Benchmarking PHP","link":"https:\/\/current.workingdirectory.net\/posts\/2010\/benchmarking-php\/","pubDate":"Fri, 08 Jan 2010 09:32:44 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2010\/benchmarking-php\/","description":"<p>What&rsquo;s the best way to run PHP in Debian in a shared environment?<\/p>\n<p>Years ago <a href=\"http:\/\/mayfirst.org\">May First\/People Link<\/a> switched from the old\nreliable mod_php to suPHP because mod_php ran all php scripts as the same user,\nmeaning (among other things) that any site could access the database\nconfigurations of any other site.<\/p>\n<p>suPHP was great, however, after several years we&rsquo;ve decided to replace it because:<\/p>\n<ul>\n<li>As our members increasingly use <a href=\"http:\/\/drupal.org\">Drupal<\/a> and Drupal sites grow increasingly more complex, suPHP&rsquo;s performance has become a problem.<\/li>\n<li>suPHP executes with the permissions of the owner of the file it is executing. From a zero-conf perspective, this features makes suPHP very easy to setup. However, from a security perspective, it forces you to run your php script with the permission to delete itself. Not optimal.<\/li>\n<\/ul>\n<p>We first turned to\n<a href=\"http:\/\/packages.debian.org\/search?keywords=libapache2-mod-fcgid\">fcgid<\/a>. We\ntried it out on a couple individual web sites (fcgid can run alongside suPHP),\nand, after a week without much problems, we implemented it on an entire server.<\/p>\n<p>Out of the box, on a shared server with several dozen web sites, the most\nnotable resource utilization difference between fcgid and suPHP is the\n<em>increase<\/em> in memory usage. Unlike suPHP which dies after each requests, fcgid\ngains a performance boost by sticking around (with either 2 or 3 processes\nliving on) to server future requests. If you have several dozen web sites that\naren&rsquo;t high traffic sites, it means they may each have up to the three\nprocesses staking out memory that would otherwise be free with suPHP.<\/p>\n<p>We addressed the memory problem by adding the following to\n\/etc\/apache2\/mods-available\/fcgid.conf:<\/p>\n<pre><code>DefaultMinClassProcessCount 0\nIdleTimeout 60\nIdleScanInterval 60\n<\/code><\/pre>\n<p>These directions tell fcgid to kill processes that have been idle for more than\n60 seconds. That means we still get the performance gain from a site getting\nmore than one page view per minute, but otherwise we re-claim the memory.<\/p>\n<p>Before switching our remaining servers we decided to do more extensive testing\nto ensure we would be getting the performance and resource gains we were\nhoping. Also, we decided to consider\n<a href=\"http:\/\/packages.debian.org\/search?keywords=apache2-mpm-itk\">mpm-itk<\/a>, which is\na variation of mpm-prefork that runs each virtual host with a per-vhost\nconfigured user and group (like suexec, but for the entire virtual host). Since\nit&rsquo;s a variation of mpm-prefork, it&rsquo;s safe to run it with the more resource\nfriendly mod_php.<\/p>\n<p>We used the <a href=\"http:\/\/httpd.apache.org\/docs\/2.3\/programs\/ab.html\">apache benchmarking tool\nab<\/a>. And we tested on a\nvirtual machine (running on my laptop) with 1GB of RAM. The test web site we\nused was a copy of the <a href=\"http:\/\/mayfirst.org\">May First\/People Link site<\/a> (we\nused the home page for the test), which is running Drupal 6. It&rsquo;s a relatively\nsimple Drupal site, however, we turned Drupal caching off to better simulate\nPHP processing usage likely to be found on our members&rsquo; sites.<\/p>\n<p>We ran tests against the following configurations:<\/p>\n<ul>\n<li>Plain mpm-prefork with mod_php and apc (as a standard to strive for)<\/li>\n<li>Both mpm-worker with suPHP and mpm-prefork with suPHP (how much of a difference does worker vs prefork make with a Drupal site?)<\/li>\n<li>mpm-worker\/fcgid\/suexec<\/li>\n<li>mpm-itk with mod_php<\/li>\n<\/ul>\n<p>The tests were run with:<\/p>\n<ul>\n<li>1 request (in the results, this test is labelled 1:1)<\/li>\n<li>20 requests, 5 concurrent (20:5)<\/li>\n<li>50 requests, 10 concurrent (50:10)<\/li>\n<li>70 requests, 10 concurrent (70:10)<\/li>\n<\/ul>\n<p>We originally tried, for the last test, 100 requests\/25 concurrent but it\ncaused ab to time out on the mpm-worker\/fcgid\/suexec configuration. We then\nlowered it to 75\/15. Still timed out (load on our test server went way up).<\/p>\n<p>The tests paused 120 seconds between testing each environment to give the\nserver load a chance to settle down. And we paused 30 seconds between each test\n(not enough time for the fcgid processes to be killed, but enough time for all\nthe processes to hopefully complete their requests).<\/p>\n<p>The tests are <a href=\"php-benchmark\">scripted<\/a> and the <a href=\"report.txt\">full\nresults<\/a> are available. You can also see\nthe various vhost configurations\n(<a href=\"test.mouse\">test.mouse<\/a>,\n<a href=\"test.mouse.itk\">test.mouse.itk<\/a>,\n<a href=\"test.mouse.suexec.fcgid\">test.mouse.suexec.fcgid<\/a>),\nand the <a href=\"fcgid.conf\">fcgid.conf<\/a>.<\/p>\n<p>To summarize the report &hellip; if you are running one web site on a server,\nthere&rsquo;s little argument for using anything other than mod_php and mpm-prefork -\nthe performance of this combination far exceeds any other combination. I&rsquo;m\nscratching my head over the the <a href=\"http:\/\/2bits.com\/articles\/apache-fcgid-acceptable-performance-and-better-resource-utilization.html\">vastly different\nconclusions<\/a>\nreached by 2bits. They didn&rsquo;t publish their tests so I&rsquo;m not sure exactly how\nto compare. Maybe they have a lot more static content? Or - maybe there&rsquo;s a\nmistake in our tests? Open to ideas.<\/p>\n<p>suPHP seems to be left in the dust compared with either fcgid\/suexec or\nmpm-itk. Given that it&rsquo;s also less flexible, there&rsquo;s not much going for it.<\/p>\n<p>And lastly&hellip; fcgid\/suexec and mpm-itk seem comparable. That was a surprise. We\nwere expecting mpm-itk to perform better because it&rsquo;s running mod_php. Given\nthe similarity in performance, fcgid\/mpm-worker seems like a much more\nwell-tested and well-supported approach for a shared server.<\/p>\n<p>One side note&hellip; originally, out of laziness, we tested against the default\nDrupal install.php page. Less static content and much less PHP processing. In\nthese tests, with low loads, the\n<a href=\"report.install.txt\">results<\/a> provide different\nnumbers, but the comparisons between environment were similar &hellip; except with\nthe last test.  With the 70 requests\/10 concurrent test, fcgid\/suexec\ndramatically out-performed everyone (by a factor or more than 2). That&rsquo;s\nprobably because no new cgi processes had to be spawned <em>and<\/em> the effect of\nmod_php using APC was reduced because there was very little php code to\nprocess.<\/p>\n"},{"title":"From vservers to KVM","link":"https:\/\/current.workingdirectory.net\/posts\/2010\/vservers-to-kvm\/","pubDate":"Fri, 08 Jan 2010 09:32:44 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2010\/vservers-to-kvm\/","description":"<p>Given the impending\n<a href=\"http:\/\/bugs.debian.org\/cgi-bin\/bugreport.cgi?bug=574529\">deprecation<\/a> of\n<a href=\"http:\/\/linux-vserver.org\/Welcome_to_Linux-VServer.org\">vservers<\/a>, I&rsquo;ve decided\nto make the switch to <a href=\"http:\/\/www.linux-kvm.org\/page\/Main_Page\">KVM<\/a> on my\nlaptop. Although <a href=\"http:\/\/en.wikipedia.org\/wiki\/Lxc_Linux_Containers\">lxc<\/a> is a\ncloser approximation to vservers, I decided to go with KVM due to it&rsquo;s support\nin <a href=\"http:\/\/virt-manager.et.redhat.com\/\">Virtual Machine Manager<\/a>.<\/p>\n<p>My first step was to confirm that my CPU would support kvm:<\/p>\n<pre><code>egrep -o &quot;svm|vmx&quot; \/proc\/cpuinfo\n<\/code><\/pre>\n<p>If that command outputs either svm or vmx (depending on whether you have Intel\nor AMD hardware) then your CPU supports virtualization.<\/p>\n<p>I&rsquo;m working on a host machine called chicken, which has a logical volume called\nvg_chicken0. All vservers on chicken operate on a root filesystem that is\nbacked by their own logical volume.<\/p>\n<p>In this post, I&rsquo;ll describe the steps to convert the vserver hobo (which\noperates on a filesystem mounted on the host in \/var\/lib\/vservers\/hobo and\nis backed by the logical volume called vg_chicken0-hobo_root).<\/p>\n<p>Both chicken and hobo are running debian squeeze.<\/p>\n<p>vservers don&rsquo;t have a kernel installed or grub. KVM virtual servers need both.<\/p>\n<p>I was hoping I could simply enter the vserver, install both a kernal and grub\nand be ready to go. However, grub installation will fail miserably because grub\ncan&rsquo;t figure out how to install on the underlying disk (which is hidden from the\nvserver).<\/p>\n<p>Next, I tried launching a kvm instance, passing a\n<a href=\"http:\/\/cmrg.fifthhorseman.net\/wiki\/debirf\">debirf<\/a> generated ISO with the -c\n(cdrom) option. However, grub recognized that it was being installed onto a\ndevice that did not have a partition table (the logical volume was directly\nformatted with a file system).<\/p>\n<p>So, since I had disk space to spare, I created a new logical volume:<\/p>\n<pre><code>lvcreate --size 5GB --name hobo_root_new vg_chicken0\n<\/code><\/pre>\n<p>I then added a <a href=\"http:\/\/en.wikipedia.org\/wiki\/GUID_Partition_Table\">gpt<\/a>\npartition table (why not prepare for the coming 2TB disks?) and created two\npartitions. One partition for grub2 and one for everything else:<\/p>\n<pre><code>parted \/dev\/mapper\/vg_chicken0-hobo_root_new mklabel gpt\nparted \/dev\/mapper\/vg_chicken0-hobo_root_new unit s mkpart biosboot 2048 4095 \nparted \/dev\/mapper\/vg_chicken0-hobo_root_new set 1 bios_grub on \nparted \/dev\/mapper\/vg_chicken0-hobo_root_new unit s mkpart primary 4096 \n<\/code><\/pre>\n<p>When prompted for the end of the last partition, choose: -1 and accept the adjustment.<\/p>\n<p>I had to eyeball cat \/proc\/partitions to figure out which dm device was the\nsecond partition (dm-19).<\/p>\n<p>I then created a file system:<\/p>\n<pre><code>mkfs -t ext3 \/dev\/dm-19\n<\/code><\/pre>\n<p>Mounted it:<\/p>\n<pre><code>mount \/dev\/dm-19 \/mnt\n<\/code><\/pre>\n<p>And rsync&rsquo;ed the data:<\/p>\n<pre><code>rsync -a \/var\/lib\/vservers\/hobo\/ \/mnt\/\n<\/code><\/pre>\n<p>With the data in place, I chroot&rsquo;ed and installed the packages I needed. When prompted, I chose not to install grub to the disk, because I wanted to wait until I had an environment in which the proper disk would be available to grub as it will when the virtual server boots (see below):<\/p>\n<pre><code>chroot \/mnt\nmount \/proc\naptitude install linux-image-2.6-amd64 grub2\numount \/proc\nexit\n<\/code><\/pre>\n<p>Then, I cleaned up:<\/p>\n<pre><code>umount \/mnt\numount \/var\/lib\/vservers\/hobo\nlvremove vg_chicken0\/hobo_root\nlvrename vg_chicken0\/hobo_root_new hobo_root\ndmsetup remove \/dev\/mapper\/vg_chicken0-hobo_root_newp1\ndmsetup remove \/dev\/mapper\/vg_chicken0-hobo_root_newp2\nkpartx -d \/dev\/mapper\/vg_chicken0-hobo_root\n<\/code><\/pre>\n<p>And I removed it from \/etc\/fstab.<\/p>\n<p>Next, I created a new kvm virtual server, using the disk \/dev\/mapper\/vg_chicken0-hobo_root and passing a debirf cd image with -c:<\/p>\n<p>virt-install &ndash;name hobo &ndash;ram 512 &ndash;disk \/dev\/mapper\/vg_chicken0-hobo_root -c \/usr\/local\/share\/debian\/ISOs\/debirf-rescue_squeeze_2.6.32-5-vserver-amd64.iso<\/p>\n<p>After logging in, I installed grub2 (aptitude update; aptitude install grub2) and then I installed grub:<\/p>\n<pre><code>mount \/dev\/sda2 \/mnt\/\ngrub-install --no-floppy --root-directory=\/mnt \/dev\/sda\n<\/code><\/pre>\n<p>After running grub-install, edit \/mnt\/boot\/grub\/device.map so it reads:<\/p>\n<pre><code>(hd0) \/dev\/sda\n<\/code><\/pre>\n<p>Then, rerun grub-install command.<\/p>\n<p>I tried generating the grub.cfg file, but got an error message indicating that\ngrub-probe would not detect the device providing \/ (because I was running on a\nram file system from debirf).<\/p>\n<p>I added the following to \/mnt\/etc\/fstab:<\/p>\n<pre><code>\/dev\/sda2  \/     ext3 errors=remount-ro 0 1\nproc       \/proc proc defaults          0 0\n<\/code><\/pre>\n<p>And then re-generate the initrd image:<\/p>\n<pre><code>chroot \/mnt\nmount \/proc\nupdate-initramfs -u\n<\/code><\/pre>\n<p>So, I rebooted the virtual machine by typing:<\/p>\n<pre><code>exit\nreboot \n<\/code><\/pre>\n<p>This dropped me into a grub shell. I manually typed:<\/p>\n<pre><code>root (hd0,gpt2)\nlinux \/vmlinuz root=\/dev\/sda2 ro\ninitrd \/initrd.img\nboot\n<\/code><\/pre>\n<p>Once booted, I logged in a completed the task with:<\/p>\n<pre><code>update-grub\n<\/code><\/pre>\n"},{"title":"Growing disks: upgrading from 1 TB to 2 TB disks","link":"https:\/\/current.workingdirectory.net\/posts\/2010\/growing-disks\/","pubDate":"Fri, 08 Jan 2010 09:32:44 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2010\/growing-disks\/","description":"<p>We&rsquo;ve got a fairly layered set of software that runs beneath our file systems: RAID1 followed by cryptsetup, followed by lvm. It works remarkably well.<\/p>\n<p>However, when it&rsquo;s time to do something like enlarge a partition or disk, it gets complicated.<\/p>\n<p>We have a server (hubert) that was running two 1 TB disks in a RAID1. We typically create two partitions: one small boot partition and one large partition the uses the rest of the available disk (with 2 TB disks <a href=\"https:\/\/current.workingdirectory.net\/posts\/2010\/two-tb-disks\">we&rsquo;ve added a third partition for grub<\/a>).<\/p>\n<p>Last month I replaced one of the disks with a 2 TB disk, partitioned the new disk with similar a partition scheme (except that the &ldquo;rest of the disk&rdquo; partition was twice the size as the original one), and then added the new partitions to the RAID (mdadm \/dev\/md0 &ndash;add \/dev\/sda1; mdadm \/dev\/md1 &ndash;add \/dev\/sda2).<\/p>\n<p>Yesterday, I replaced the 2nd 2 TB disks, and again added the new partitions.<\/p>\n<p>The result was a fairly painless transition from 1 to 2 TB disks with minimal downtime.<\/p>\n<p>The problem - though - is that we&rsquo;re not using all the disk space available, since md1, the RAID array that uses the &ldquo;rest of the disk&rdquo; partition, is only about 1 TB when it should be just under 2 TB.<\/p>\n<p>That problem was easy enough to solve by growing the RAID device with:<\/p>\n<pre><code>mdadm -G \/dev\/md1 -z max\n<\/code><\/pre>\n<p>It took a while, but eventually, running:<\/p>\n<pre><code>cat \/proc\/mdstat\n<\/code><\/pre>\n<p>And<\/p>\n<pre><code>cat \/proc\/partitions\n<\/code><\/pre>\n<p>Reported the right sizes for md1.<\/p>\n<p>After RAID, comes cryptsetup (\/dev\/md1 is the source device for md1_crypt). So, the next command was:<\/p>\n<pre><code>cryptsetup resize md1_crypt\n<\/code><\/pre>\n<p>In the output of:<\/p>\n<pre><code>cat \/proc\/partitions\n<\/code><\/pre>\n<p>I saw the change reflected in the dm-0 device.<\/p>\n<p>Since md1_crypt is used as the physical volume for a volume group, the next step was to resize the physical volume:<\/p>\n<pre><code>pvresize \/dev\/dm-0\n<\/code><\/pre>\n<p>This change was reflected with:<\/p>\n<pre><code>pvs\n<\/code><\/pre>\n<p>Now, the command:<\/p>\n<pre><code>vgs\n<\/code><\/pre>\n<p>Displayed a full 931 GB available that wasn&rsquo;t there before.<\/p>\n<p>Next up: resizing the logical volume:<\/p>\n<pre><code>lvresize --size 1700GB vg_hubert0\/blanco_members\n<\/code><\/pre>\n<p>And, last but not least, was the file system:<\/p>\n<pre><code> resize2fs \/dev\/mapper\/vg_hubert0-blanco_members\n<\/code><\/pre>\n<p>See also dkg&rsquo;s <a href=\"http:\/\/www.debian-administration.org\/article\/Resizing_Encrypted_Filesystems\">article<\/a> on resizing crypt partitions.<\/p>\n"},{"title":"rm -rf at ~ 500 Mbytes\/second","link":"https:\/\/current.workingdirectory.net\/posts\/2010\/rm-at-500-mbs\/","pubDate":"Fri, 08 Jan 2010 09:32:44 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2010\/rm-at-500-mbs\/","description":"<p>If you ever have the misfortune of accidentally passing the path to a directory\ncontaining 177GB of data to the <code>rm -rf<\/code> command, I&rsquo;ll start by suggesting that\nyou hit ctl-c. The sooner the better.<\/p>\n<p>Next, assuming you have some sort of backup, you&rsquo;ll be staring at two\nmonumentally large data sets, wondering exactly what was deleted from the\noriginal.<\/p>\n<p>With help from <a href=\"http:\/\/www.debian-administration.org\/users\/dkg\/weblog\">dkg<\/a>, I\nlearned somethings about <code>rm -rf<\/code>. For one, it deletes one top level directory\nat a time. So - a comparison of the top level directory listings of the\noriginal and backup is a good place to start.<\/p>\n<p>Top level directories that are entirely missing in the original are easy to\nrestore. However, the presence of a top level directory in the original doesn&rsquo;t\nmean it was un-touched.<\/p>\n<p>Next, you&rsquo;ll want to figure out which top level directory <code>rm<\/code> was operating on\nwhen you hit ctl-c.  dkg discovered that <code>ls -UR<\/code> will provide a listing in the\nsame order that <code>rm -rf<\/code> uses. The -U means do not sort.  Note - the unsorted\nlisting of the backup directory might not be the same as the unsorted listing\nof the original, so <code>ls -UR<\/code> is only really helpful on the original directory.<\/p>\n<p>After selecting the first top level directory, <code>rm -rf<\/code> seems to delete all\nfiles in that directory first (presumably in the same order that ls -UR will\nlist them), then it enters the first sub directory (as returned by ls -UR) and\nrepeats the process.<\/p>\n<p>With a careful comparison of <code>ls -UR<\/code> on the original with the directory\nlistings on the backup, you should be able to pinpoint the exact sub\ndirectories affected, allowing you to restore only the files and directories\nthat you deleted.<\/p>\n<p>Thanks to dkg for technical and blog title suggestions.<\/p>\n"},{"title":"Two Terabyte disks: prepare for major changes","link":"https:\/\/current.workingdirectory.net\/posts\/2010\/two-tb-disks\/","pubDate":"Fri, 08 Jan 2010 09:32:44 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2010\/two-tb-disks\/","description":"<p>When we bought our first 2 TiB disk we had no idea what was in store.<\/p>\n<p>Over the last several months we have been tearing our hair out over painfully\nslow performance caused by I\/O bottlenecks compounded by our desperate attempts\nto remedy the situation by moving terabytes of data from slow performing disks\nto better performing disks.<\/p>\n<p>Over time, we&rsquo;ve started to piece together the underlying issues and figure out\na strategy for properly using these disks -  causing yet more slow downs during\nnew server installations.<\/p>\n<p>Below is an attempt to explain at a high level why previously routine tasks,\nsuch as installing Debian on new machines, or replacing dead hard disks has\nbecome significantly more complicated and taken exponentially longer than\nbefore.<\/p>\n<h2 id=\"partition-tables\">Partition tables<\/h2>\n<p>Up to now, most of us have created partition tables (the information stored on\na disk that instructs an OS or bios what partitions exists, where to find them, etc.) in the <a href=\"http:\/\/en.wikipedia.org\/wiki\/Master_Boot_Record\">Master Boot\nRecord<\/a> or MBR. That&rsquo;s the\nfirst 512-byte sector of the disk. When you run the debian installer or just\nabout any disk partition utility, it stores the partition info in the MBR.<\/p>\n<p>The MBR approach, however, has a limitation: it <a href=\"http:\/\/en.wikipedia.org\/wiki\/Master_Boot_Record#Disk_partitioning\">can&rsquo;t handle disk partitions\nlarger than 2 TiB<\/a> (well&hellip; technically that means you could still use the MBR\nwith a 2 TiB disk because no single partition would be larger than 2 TiB&hellip;\nhowever, given the rate of growth in disk size, it seems like now is the time\nto tackle this problem).<\/p>\n<p>Fortunately, a new partition table layout has been created: <a href=\"http:\/\/en.wikipedia.org\/wiki\/GUID_Partition_Table\">GPT or GUID Partition Tables<\/a>.<\/p>\n<p>The bad news is that not all of our favorite tools can handle GPT partitioned\ndisks, and some will fail spectacularly.<\/p>\n<h2 id=\"grub\">Grub<\/h2>\n<p>Although legacy Grub (Grub 1) <a href=\"http:\/\/en.wikipedia.org\/wiki\/GUID_Partition_Table#OS_support_of_GPT\">supposedly supports GPT\ntables<\/a>,\nwhen making this change, we&rsquo;ve opted to switch to Grub 2, which their <a href=\"http:\/\/www.gnu.org\/software\/grub\/grub-2.en.html\">web site\nproudly proclaims has been re-written from\nscratch<\/a>. Ug. Although legacy\nGrub users will recognize a few bits and pieces from Grub 1, it&rsquo;s a steep\nlearning curve.<\/p>\n<p>When combined with GPT, that learning curve includes a significant departure\nfrom how we previously installed Grub on a disk. If you want to use Grub 2 with\nGPT, you need to create a small partition for Grub on the disk (in addition to\nyour regular \/boot partition) and add a flag on that partition called\nbios_grub. Note: Grub 2 only needs this partition if you are using a\nGPT-partitioned disk.<\/p>\n<p>Then, when you run install-grub, it will be installed into that partition in a\nway that will properly boot your operating system.<\/p>\n<h2 id=\"partition-alignment\">Partition alignment<\/h2>\n<p>If all of this wasn&rsquo;t enough&hellip;<\/p>\n<p>With the introduction of 2 TiB disks, disk manufacturers are beginning to\nchange the way they are writing data.<\/p>\n<p>Previously, disks wrote data in 512-byte sectors. As a result, all disk\nutilities of the recent past have religiously created partitions and all other\nforms of dividing up a disk on 512-byte boundaries.<\/p>\n<p>Some manufacturers of 2 TiB disks, however, are writing data in 4096-byte\nboundaries. That means if you create a partition that overlaps a 4096-byte\nboundary you are essentially screwed.<\/p>\n<p>Consider a disk in which the following pipes represent 512-byte boundaries and\n[Pn ] represent partitions properly aligned along those boundaries:<\/p>\n<pre><code>|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | \n[P1     ][P2        ][P3                        ]\n<\/code><\/pre>\n<p>All partitions neatly start at the beginning of a 512-byte block. Every time\nthe disk wants to write, it can easily fit the data into the sectors.<\/p>\n<p>A 2 TiB disk that uses 4096-byte sectors, however, needs to be divided along\n4096-byte boundaries, displayed below with the middle row of pipes. As you can\nsee, your beautifully aligned partitions are now a mis-aligned mess:<\/p>\n<pre><code>|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | \n|                       |                       |\n[P1     ][P2        ][P3                        ]\n<\/code><\/pre>\n<h2 id=\"what-you-can-do\">What you can do<\/h2>\n<p>We haven&rsquo;t yet fully tested mdadm, cryptsetup and lvm to ensure that they\ncreate data on 4096-byte boundaries. Initial poking around suggests that they\ndo - but more work is needed to be certain.<\/p>\n<p>The version parted we&rsquo;re using (squeeze), on the other hand, will not attempt\nto align your partitions on 4096-byte boundaries for you. You need to do that\nyourself by specifying the exact, properly aligned boundaries.<\/p>\n<p>We have a <a href=\"https:\/\/support.mayfirst.org\/wiki\/install_debian\">write up<\/a> with the\nnew steps for creating a Debian server using 2 TiB disks.<\/p>\n<p>The summary is: when partitioning your disk using parted:<\/p>\n<ul>\n<li>switch the unit to sectors (unit s)<\/li>\n<li>ensure that the starting sector is divisible by both 8 and 512<\/li>\n<li>ensure that the ending sector + 1 is divisible by both 8 and 512 (so that the next sector start point is properly aligned)<\/li>\n<li>ensure that the size is divisible by both 8 and 512.<\/li>\n<\/ul>\n<p>For the math challenged, here&rsquo;s a functional layout of a GPT partitioned disk:<\/p>\n<pre><code>~ # parted \/dev\/sda unit s p\nModel: ATA WDC WD20EADS-00R (scsi)\nDisk \/dev\/sda: 3907029168s\nSector size (logical\/physical): 512B\/512B\nPartition Table: gpt\n\nNumber  Start     End          Size         File system  Name      Flags    \n 1      2048s     4095s        2048s                     biosboot  bios_grub\n 2      4096s     1052671s     1048576s                  boot      raid     \n 3      1052672s  3905974271s  3904921600s               pv        raid  \n<\/code><\/pre>\n<p>You can get there with these commands:<\/p>\n<pre><code>parted \/dev\/sda mklabel gpt\n\nparted \/dev\/sda unit s mkpart biosboot 2048 4095 \nparted \/dev\/sda set 1 bios_grub on \n\nparted \/dev\/sda unit s mkpart boot 4096 1052671 \nparted \/dev\/sda set 2 raid on \n\nparted \/dev\/sda unit s mkpart pv 1052672 3905974271\nparted \/dev\/sda set 3 raid on \n<\/code><\/pre>\n<p>I&rsquo;m not sure what the disk size limitations of GPT are&hellip; but I hope we don&rsquo;t\nreach them any time soon.<\/p>\n"},{"title":"video4linux and audio","link":"https:\/\/current.workingdirectory.net\/posts\/2010\/video4linux-and-audio\/","pubDate":"Fri, 08 Jan 2010 09:32:44 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2010\/video4linux-and-audio\/","description":"<p>With <a href=\"http:\/\/www.cc2010.mx\/swb\/\">cop16<\/a> coming up in just a few months, we&rsquo;ve been working at MFPL on organizing people from around the world to share live video of the COP16 related events: protests, panels, performances, etc. The goal is to use live video as an alternative to expensive and environmentally destructive travel - a goal well-suited to the environmental focus of COP16.<\/p>\n<p>I&rsquo;ve spent the better part of the last three weekends figuring out how to do this on Debian Squeeze using all free software and codecs.<\/p>\n<p>The debconf organizers have done an amazing job <a href=\"http:\/\/dvswitch.alioth.debian.org\/wiki\/component_interaction\/\">developing and documenting<\/a> how to broadcast live video from a conference using DV via firewire input. It&rsquo;s impressive and, based on my experiences, works quite well.<\/p>\n<p>The problem is the firewire port. Computers aren&rsquo;t made with firewire cards any more and even if they were, I don&rsquo;t have a video camera with firewire out. I do, however, have a laptop with a USB camera and a mini audio plug that will take a cheap microphone. And, there are millions of others in this position.<\/p>\n<p>The other draw back to the debconf approach is that it assumes all video cameras are attached to the same local network. We are interested in having people contribute video content from all over the world.<\/p>\n<h2 id=\"the-main-pieces\">The main pieces<\/h2>\n<h3 id=\"video\">Video<\/h3>\n<p><a href=\"http:\/\/en.wikipedia.org\/wiki\/Video4linux\">video4linux<\/a> is a programming interface that makes this all possible. Most video-based applications available on linux support it as do most web cameras that I&rsquo;ve experimented with.<\/p>\n<h3 id=\"audio\">Audio<\/h3>\n<p>Audio proved to be the most difficult piece - way harder than video. Many of my problems seemed to have been largely specific to my computer, but not all of them.<\/p>\n<p>Part of the problem with audio is that, on linux, it&rsquo;s a real mess. There are a half dozen methods in use for accessing your audio card. Furthermore, in my case, my audio card is not well supported in linux. More on that below. The important piece is that, whenever possible, I tried to use <a href=\"http:\/\/en.wikipedia.org\/wiki\/Advanced_Linux_Sound_Architecture\">alsa<\/a>.<\/p>\n<h3 id=\"codecs\">Codecs<\/h3>\n<p>I decided to only focus on free and open source codecs. Google seems to have made <a href=\"http:\/\/en.wikipedia.org\/wiki\/VP8\">VP8<\/a> free, which means we may be moving away from theora encoded ogg files toward VP8 encoded .webm files, however, for now all the Debian tools in squeeze work with <a href=\"http:\/\/en.wikipedia.org\/wiki\/Theora\">theora<\/a>, <a href=\"http:\/\/en.wikipedia.org\/wiki\/Vorbis\">vorbis<\/a>, and the <a href=\"http:\/\/en.wikipedia.org\/wiki\/Ogg\">ogg container<\/a>.<\/p>\n<h3 id=\"media-server\">Media Server<\/h3>\n<p><a href=\"http:\/\/packages.debian.org\/search?keywords=icecast2\">icecast<\/a> is a streaming media server. The goal is to get my laptop to send an audio\/video stream to our icecast server, which will then be responsible for re-distributing it over the Internet.<\/p>\n<h3 id=\"media-player\">Media player<\/h3>\n<p>I&rsquo;m only testing with the <a href=\"http:\/\/en.wikipedia.org\/wiki\/Html5\">HTML5<\/a> video tag. With Firefox 3.5 and up, video can be displayed directly in a web browser without any extra software.<\/p>\n<h2 id=\"the-pain\">The pain<\/h2>\n<h3 id=\"ffmpeg2theora\">ffmpeg2theora<\/h3>\n<p>I started off thinking I could do everything with a simple <a href=\"http:\/\/packages.debian.org\/search?keywords=ffmpeg2theora\">ffmpeg2theora<\/a> command piped to <a href=\"http:\/\/packages.debian.org\/search?keywords=oggfwd\">oggfwd<\/a>. Something like:<\/p>\n<pre><code>ffmpeg2theora \/dev\/video0 -f video4linux2 -o \/dev\/stdout | oggfwd icecast.server 8000 secret \/test.ogg\n<\/code><\/pre>\n<p>That is an elegant command and it worked perfectly the first time I ran it. With one problem: there&rsquo;s no audio. ffmpeg2theora will gladly add audio to it&rsquo;s output, provided your input has audio. However, \/dev\/video0 provides just video and there is no way to specify a video input and an audio input using ffmpeg2theora. Sigh.<\/p>\n<p>I was so loathe to give up on such an elegant command, that I started working on sending two streams to our icecast server: one using ffmpeg2theora for video and one using <a href=\"http:\/\/packages.debian.org\/search?keywords=darkice\">darkice<\/a> for audio. I wouldn&rsquo;t recommend this approach - there&rsquo;s no way to keep the audio and video in sync. However, I couldn&rsquo;t even get that much to work due to some kind of strange bug. Seems that when I run darkice, I get a nice consistent audio stream to my icecast server. If, during this stream, I start a video4linux device (seems to affect any video4linux device, even cheese), it causes darkice to crap out and my audio input stops working. I <a href=\"http:\/\/bugs.debian.org\/cgi-bin\/bugreport.cgi?bug=590215\">opened the bug against cheese<\/a> - who knows where it really belongs.<\/p>\n<h3 id=\"ffmpeg\">ffmpeg<\/h3>\n<p>Next, I moved on to ffmpeg - which does allow for both a video <em>and<\/em> audio input.<\/p>\n<p>I could successfully get ffmpeg to record audio with the command, using the alsa-oss compatible driver:<\/p>\n<pre><code> ffmpeg -f oss -ar 48000 -i \/dev\/audio -acodec pcm_s16le out.wav\n<\/code><\/pre>\n<p>It even works with alsa directly (note the need for -ac 2 - alsa fails with the default 1 channel):<\/p>\n<pre><code> ffmpeg -f alsa -ac 2 -ar 48000 -i hw:0,0 -acodec pcm_s16le out.wav\n<\/code><\/pre>\n<p>However, when I add my video4linux2 device, I lose the sound:<\/p>\n<pre><code>ffmpeg -f alsa -ac 2 -ar 48000 -i hw:0,0 -acodec pcm_s16le -f video4linux2 -s 320x240 -i \/dev\/video0 out.mpg\n<\/code><\/pre>\n<p>The video plays back fine, but the audio is silent. At this point, I moved on to vlc&hellip; however, I later discovered the problem (which I will describe here, out of chronological order).<\/p>\n<p>Turns out, my system is not playing pulse audio properly, or at least mplayer is not. The audio really is there. I just needed to test using mplayer with -ao alsa:<\/p>\n<pre><code>mplayer -ao also out.mpg\n<\/code><\/pre>\n<p>After much haggling with options, I finally got this train wreck to run without an error:<\/p>\n<pre><code> ffmpeg -f alsa -ac 2 -ar 48000 -i hw:0,0 -f video4linux2 -s 320x240 -i \/dev\/video0 -f yuv4mpegpipe -pix_fmt yuv444p - | \\\n  ffmpeg2theora -o - - | oggfwd icecast.server 8000 secret \/test.ogg\n<\/code><\/pre>\n<p>However, the video ran at a crawl, I never did hear any audio, and the process died after a 3 minutes. ffmpeg was not going to be an elegant solution.<\/p>\n<h3 id=\"vlc\">vlc<\/h3>\n<p>vlc seemed like a perfect option, given that it runs on linux, Mac and Windows. If I could get it to work on linux, providing directions for other operating systems would be a breeze.. Beginning with the graphical user interface, I selected Media -&gt; Convert \/ Save&hellip;. Then I clicked the Capture Device tab, to indcate I wanted to convert\/save something I was capturing. I hit the Convert\/Save button (leaving all settings at their defaults). The next stream suggested &ldquo;Video - H.264 + ACC (TS)&rdquo; as the profile. I left it alone. Then, I entered the path to the file I wanted to save to (ending in .mpg), and lastly clicked Start.<\/p>\n<p>And&hellip; I got this error:<\/p>\n<pre><code>Streaming \/ Transcoding failed:\nIt seems your FFMPEG (libavcodec) installation lacks the following encoder:\nH264 - MPEG-4 AVC (part 10).\nIf you don't know how to fix this, ask for support from your distribution.\n\nThis is not an error inside VLC media player.\nDo not contact the VideoLAN project about this issue.\n<\/code><\/pre>\n<p>So, I repeated, but this time, for profile, selected &ldquo;Video - Theora + Vorbis (OGG)&rdquo;. This time it recorded. But, when I played back using vlc, it played back at twice the recorded speed, and there was no audio (no audio playing back in vlc or in mplayer using -ao alsa).<\/p>\n<p>At this point, I saw that vlc 1.10 was available in unstable. In case I was experiencing fixed vlc bugs&hellip;<\/p>\n<pre><code>sudo aptitude install vlc\/unstable vlc-data\/unstable vlc-nox\/unstable\n<\/code><\/pre>\n<p>Sadly, no difference :(. The theora\/vorbis file still played back without audio and about twice the speed at which it was recorded.<\/p>\n<p>Not one who gives up easily, I researched online tips for using command line vlc (cvlc) and came up with:<\/p>\n<pre><code>cvlc v4l2:\/\/ :v4l2-vdev=&quot;\/dev\/video0&quot; :v4l2-adev=&quot;\/dev\/audio&quot; --sout \\\n &quot;#transcode{vcodec=theo,vb=800,scale=1,acodec=vorb,ab=128,channels=2,samplerate=44100}:std{access=shout,mux=ogg,dst=source:secret@icecast.server:8000\/test.ogg}&quot;\n<\/code><\/pre>\n<p>Works for video (barely - it&rsquo;s pretty choppy) but still no audio :(.<\/p>\n<p>So much for vlc.<\/p>\n<h2 id=\"gisstv-and-a-ray-of-hope\">giss.tv and a ray of hope<\/h2>\n<p>At this point&hellip; frustration with vlc set in and, after some browsing, I came across <a href=\"http:\/\/giss.tv\">giss.tv<\/a> and their <a href=\"http:\/\/giss.tv\/wiki\/index.php\/Sa'habuntu_Live_CD\">docs page<\/a>.<\/p>\n<p>The <a href=\"http:\/\/giss.tv\/wiki\/images\/1\/19\/Webcamstream-v4l2.pys\">Webcamstream-v4l2.pys python script<\/a> was the first to catch my eye.<\/p>\n<p>After downloading, I tried to run it with:<\/p>\n<pre><code>python Webcamstream-v4l2.pys\n<\/code><\/pre>\n<p>But, got the error:<\/p>\n<pre><code>Error: Could not initialize supporting library. gstautovideosink.c(367): gst_auto_video_sink_detect (): \n \/GstPipeline:pipeline0\/GstAutoVideoSink:autovideosink0:\nFailed to set target pad\n<\/code><\/pre>\n<p>One of the helpful giss.tv folks suggested I try to run xvinfo, which returned:<\/p>\n<pre><code>0 jamie@chicken:~$ xvinfo \nX-Video Extension version 2.2\nscreen #0\nno adaptors present\n1 jamie@chicken:~$\n<\/code><\/pre>\n<p>After considerable searching and debugging (I have a Toshiba Satellite with a Radeon HD 3200 Graphics card), I finally discovered <a href=\"http:\/\/bugs.debian.org\/cgi-bin\/bugreport.cgi?bug=579918\">debian bug 579918<\/a> which helped me realize I needed the ^%<em>$<\/em> propriety firmware-linux package installed. After re-booting, xvinfo reported a lot of information (and I discovered that my machine would wake from suspend successfully again).<\/p>\n<p>Next problem when trying to run the python Webcamstream script:<\/p>\n<pre><code>0 jamie@chicken:~$ python Webcamstream-v4l2.pys\nThe program 'Webcamstream-v4l2.pys' received an X Window System error.\nThis probably reflects a bug in the program.\nThe error was 'BadIDChoice (invalid resource ID chosen for this connection)'.\n  (Details: serial 836 error_code 14 request_code 1 minor_code 0)\n  (Note to programmers: normally, X errors are reported asynchronously;\n   that is, you will receive the error a while after causing it.\n   To debug your program, run it with the --sync command line\n   option to change this behavior. You can then get a meaningful\n   backtrace from your debugger if you break on the gdk_x_error() function.)\n1 jamie@chicken:~$\n<\/code><\/pre>\n<p>Time to give up (for now)&hellip;<\/p>\n<p>More browsing of giss.tv led me to yet another program&hellip; <a href=\"http:\/\/gollum.artefacte.org\/tss\/\">Theora Streaming Studio - TSS<\/a>.<\/p>\n<p>A Debian squeeze deb was not available, so I downloaded the lenny version. Ug. It required libraw1394-8, which is not available in squeeze (libraw1394-11 is).<\/p>\n<p>So&hellip; downloaded the source. It relies on automake version 1.10. 1.11 is in squeeze, so I had to run:<\/p>\n<pre><code>sudo ln -s \/usr\/share\/automake-1.11 \/usr\/share\/automake-1.10\n<\/code><\/pre>\n<p>Then installed tss with the standard:<\/p>\n<pre><code>.\/configure\nmake\nsudo make install\n<\/code><\/pre>\n<p>It seems to have a lot of potential - but it did not seem to work for me and didn&rsquo;t provide any output to let me know what was wrong.<\/p>\n<p>So&hellip; I returned to the Webcamstream-v4l2.pys script. The script relies on <a href=\"http:\/\/www.gstreamer.net\/\">gstreamer<\/a> for handling all of the heavy duty video and audio work. Even though I couldn&rsquo;t get the script to work (for X11 reasons), gstreamer seemed very impressive.<\/p>\n<p>Debian squeeze ships gst-launch-0.10, a developers command line tool for testing various things that the gstreamer library can do. I created an alias in my .bashrc file so I could simply type gst-launch to invoke the problem.  After reading through some man pages and a few helpful examples on the web and from the giss.tv script &hellip;<\/p>\n<p>Working audio recording:<\/p>\n<pre><code>gst-launch alsasrc ! audioconvert ! vorbisenc ! oggmux ! filesink location=input.ogg\n<\/code><\/pre>\n<p>Working audio streaming:<\/p>\n<pre><code>gst-launch alsasrc ! audioconvert ! vorbisenc ! oggmux \\\n ! shout2send ip=icecast.server port=8000 password=secure mount=\/test.ogg\n<\/code><\/pre>\n<p>Working video recording:<\/p>\n<pre><code>gst-launch v4l2src ! ffmpegcolorspace ! videoscale ! video\/x-raw-yuv,width=320,height=240 \\\n ! theoraenc quality=16 ! oggmux !  filesink location=input.ogg\n<\/code><\/pre>\n<p>Working video streaming:<\/p>\n<pre><code>gst-launch v4l2src ! ffmpegcolorspace ! videoscale ! video\/x-raw-yuv,width=320,height=240 \\\n ! theoraenc quality=16 ! oggmux !  shout2send ip=icecast.server port=8000 password=secret mount=\/test.ogg\n<\/code><\/pre>\n<p>Working combo recording:<\/p>\n<pre><code>gst-launch v4l2src ! queue ! ffmpegcolorspace ! videoscale ! video\/x-raw-yuv,width=320,height=240 \\\n ! theoraenc quality=16 ! queue ! oggmux name=mux alsasrc  ! queue !  audioconvert ! vorbisenc \\\n ! queue ! mux. mux. ! queue ! filesink location=input.ogg\n<\/code><\/pre>\n<p>And, finally (!!!)&hellip;. working combo streaming:<\/p>\n<pre><code>gst-launch v4l2src ! queue ! ffmpegcolorspace ! videoscale ! video\/x-raw-yuv,width=320,height=240 \\\n ! theoraenc quality=16 ! queue ! oggmux name=mux alsasrc  ! queue !  audioconvert ! vorbisenc ! queue  \\\n ! mux. mux. ! queue ! shout2send ip=icecast.server port=8000 password=secret mount=\/test.ogg\n<\/code><\/pre>\n<p>Success!! My first live video and audio stream with acceptable quality.<\/p>\n<p><strong>\\0\/ \\0\/ \\0\/ \\0\/ \\0\/ \\0\/ \\0\/ \\0\/<\/strong><\/p>\n<p>I did some more tweaking and came up with the following, which, in addition to streaming to an icecast server, displays the video and saves it to a local file:<\/p>\n<pre><code>gst-launch v4l2src ! queue ! ffmpegcolorspace ! videoscale ! video\/x-raw-yuv,width=320,height=240 \\\n ! tee name=tscreen ! queue ! autovideosink tscreen. ! queue ! videorate ! video\/x-raw-yuv,framerate=25\/2 \\\n !  queue ! theoraenc quality=16 ! queue ! oggmux name=mux alsasrc ! queue ! audioconvert ! vorbisenc quality=0.2 \\\n ! queue ! queue ! mux. mux. ! queue ! tee name=tfile ! queue ! filesink location=stream.ogg tfile. ! queue \\\n ! shout2send ip=icecast.server port=8000 mount=test.ogg password=secret\n<\/code><\/pre>\n<h2 id=\"flumotion\">Flumotion<\/h2>\n<p>All of this gstreamer business eventually led me to <a href=\"http:\/\/www.flumotion.net\/\">flumotion<\/a>, an elegant collection of programs that use gstreamer and python&rsquo;s <a href=\"http:\/\/twistedmatrix.com\/trac\/\">twisted<\/a> library to create a full featured streaming studio. The program is GUI driven to make it easy for newbies, while at the same time, it is dreamily composed of many separate and discreet parts providing a level of flexibility that is really useful.<\/p>\n<p>Getting flumotion to live stream video and audio on Debian squeeze did take some work and help from the flumotion developers via IRC.<\/p>\n<p>For starters, I had to add the flumotion user to the video and audio group (and then restart flumotion). In addition, I needed the python-gi package.<\/p>\n<p>I could then run flumotion-admin and work through all the default options in the wizard except overlay&hellip; which produced the following error:<\/p>\n<pre><code>gst-stream-error-quark: 1\ngstbasesrc.c(2543): gst_base_src_loop (): \/GstPipeline:pipeline-overlay-video\/GstAppSrc:source:\nstreaming task paused, reason not-negotiated (-4)\n<\/code><\/pre>\n<p>I never did figure it out - I simply unchecked overlay in the wizard.<\/p>\n<p>The default options, however, used a test video and test audio source - not my webcam and audio card.<\/p>\n<p>When I tried to stream using my hardware capture devices, flumotion insisted that another program was using my sound card. I was sure pulseaudio was turned off and nothing else should have been accessing it. Finally, on the suggestion of one of the developers, I applied a <a href=\"https:\/\/code.fluendo.com\/flumotion\/trac\/changeset\/8341#file1\">patch<\/a> to the audio.py file that is scheduled for the next release and it all worked!<\/p>\n<h2 id=\"next-steps\">Next steps<\/h2>\n<p>gstreamer definitely seems to be the best tool for the task. While flumotion is the best general purpose tool, the Webcamstream-4vl2.pys gave me a lot of ideas on how to create a simple program that just streams live video and audio to an icecast server. Given the <a href=\"http:\/\/oggconvert.tristanb.net\/download\/windows\/\">work done with oggconvert to get gstreamer and python bindings functional on Windows<\/a>, it even seems possible to make something that would run on Windows.<\/p>\n<p>However, the biggest next step to really meet our goals will be to get a live streaming option for the android phone. Seems like <a href=\"http:\/\/guij.emont.org\/blog\/2010\/03\/13\/playing-with-the-android-ndk-and-gstreamer\/\">at least someone<\/a> is working on getting gstreamer to work on android. Hopefully that will progress!<\/p>\n"},{"title":"Booting from a USB stick into Grub","link":"https:\/\/current.workingdirectory.net\/posts\/2009\/grub-on-usb\/","pubDate":"Wed, 24 Jun 2009 17:41:17 +0000","guid":"https:\/\/current.workingdirectory.net\/posts\/2009\/grub-on-usb\/","description":"<p>Often when replacing a failed disk in a RAID1 setup (or in many other cases)\ngrub fails to load. Usually it&rsquo;s because the bios is expecting to find a boot\nloader and the various files on which the boot loader depends on the first\ndisk in the machine. If that&rsquo;s the disk that was replaced, then loading grub\nwill fail.<\/p>\n<p>One way of recovering is to boot the server from a USB stick that has grub\ninstalled and then manually tell grub how to boot from the disks that are\navailable.<\/p>\n<p>If you are reading this <em>before<\/em> you replace a disk - be sure to copy the\n\/boot\/grub\/menu.lst file from the computer before turning it off. That will\nsave you a lot of pain.<\/p>\n<p>There are a lot of complicated tutorials on how to do this - however, if you\ndon&rsquo;t care about the contents of your USB stick, it&rsquo;s really quite simple.<\/p>\n<ul>\n<li>\n<p>Put your USB disk into working linux  computer (don&rsquo;t mount it). On my\ncomputer it is typically recognized as \/dev\/sdb. If your USB stick is\nrecognized differently (and\/or your hard disk is \/dev\/sdb) then you <em>must<\/em>\nreplace all the instances below of \/dev\/sdb with whatever your actual USB\nstick is being recognized. This is, well, kinda important.<\/p>\n<\/li>\n<li>\n<p>Create a single partition. I typically do this with:<\/p>\n<pre><code>  sudo cfdisk \/dev\/sdb\n<\/code><\/pre>\n<p>Feel free to create a FAT32 partition if you want to use this stick on\ndifferent machines with different operating systems (including Windows).<\/p>\n<\/li>\n<li>\n<p>Mount the partition you just created:<\/p>\n<pre><code>  sudo mount \/dev\/sdb1 \/mnt\n<\/code><\/pre>\n<\/li>\n<li>\n<p>Install GRUB:<\/p>\n<pre><code>  sudo grub-install --no-floppy --root-directory=\/mnt \/dev\/sdb\n<\/code><\/pre>\n<p>That should create a \/mnt\/boot\/grub directory.<\/p>\n<p>It will also create \/mnt\/boot\/grub\/device.map, which is a list of disks on\nthe computer it is running. This list will include not only the USB stick\nbut your computer&rsquo;s hard disk as well. For example:<\/p>\n<pre><code>  (hd0)\t\/dev\/hda\n  (hd1)\t\/dev\/sdb\n<\/code><\/pre>\n<p>Change to:<\/p>\n<pre><code>  (hd0)\t\/dev\/sdb\n<\/code><\/pre>\n<p>So that the only device showing is your USB stick. Then re-run:<\/p>\n<pre><code>  sudo grub-install --root-directory=\/mnt \/dev\/sdb\n<\/code><\/pre>\n<\/li>\n<li>\n<p>Now comes the hard part. You need a grub configuration that will make sense\nfor the server you are booting.<\/p>\n<p>If you are using grub1&hellip;<\/p>\n<p>Hopefully you have a copy of the menu.lst file from the server. In that case,\nsimply copy it to \/mnt\/boot\/grub\/ and you are ready to go. Otherwise, you&rsquo;ll\nneed to craft one. Below is a sample.<\/p>\n<pre><code>  # uncomment these lines if you want to send grub to a serial console\n  #serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1\n  #terminal serial\n  default\t\t0\n  timeout\t\t5\n  color cyan\/blue white\/blue\n\n  # simple setup\n  title           Debian GNU\/Linux, kernel 2.6.26-2-686\n  root            (hd0,5)\n  kernel          \/boot\/vmlinuz-2.6.26-2-686 root=\/dev\/hda6 ro\n  initrd          \/boot\/initrd.img-2.6.26-2-686\n\n  # here's a more complicated one\n  title\t\tDebian GNU\/Linux, kernel 2.6.26-2-vserver-amd64\n  root\t\t(hd0,0)\n  kernel\t\t\/vmlinuz-2.6.26-2-vserver-amd64 root=\/dev\/mapper\/vg_pianeta0-root \\\n  ro console=ttyS0,115200n8 cryptopts=target=md1_crypt,source=\/dev\/md1 \\\n  cryptopts=target=md2_crypt,source=\/dev\/md2,lvm=vg_pianeta0-root \n  initrd\t\t\/initrd.img-2.6.26-2-vserver-amd64\n<\/code><\/pre>\n<p>If you are using grub2&hellip;<\/p>\n<pre><code>  grub-mkconfig -o \/mnt\/boot\/grub\/grub.cfg\n<\/code><\/pre>\n<p>Then edit. You might want something like:<\/p>\n<pre><code>  serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1\n  terminal_input serial\n  terminal_output serial\n  insmod raid\n  insmod mdraid\n  insmod part_gpt\n  set default=0\n  set timeout=5\n\n  menuentry &quot;Debian GNU\/Linux, with Linux 2.6.32-trunk-vserver-amd64&quot; --class debian --class gnu-linux --class gnu --class os {\n  \tset root='(hd0,1)'\n  \tsearch --no-floppy --fs-uuid --set 7682a24c-b06f-456b-b3d4-bcb7294d81e2\n  \techo\tLoading Linux 2.6.32-trunk-vserver-amd64 ...\n  \tlinux\t\/vmlinuz-2.6.32-trunk-vserver-amd64 root=\/dev\/mapper\/vg_chicken0-root ro quiet\n  \techo\tLoading initial ramdisk ...\n  \tinitrd\t\/initrd.img-2.6.32-trunk-vserver-amd64\n  }\n<\/code><\/pre>\n<\/li>\n<li>\n<p>Put the USB stick into the target computer, configure it to boot from the\nUSB stick via bios, and then you should see the GRUB menu come up.<\/p>\n<\/li>\n<li>\n<p>It&rsquo;s possible that your computer will just boot with your menu.lst file. In\nthat case - congrats! See the last step below to figure out how to ensure it\ncan boot without your USB stick. On the other hand, if it fails, you&rsquo;ll need\nto experimentally figure out which disk has which partitions and which\nkernels. Fortunately grub supports tab completion which makes this job\neasier:<\/p>\n<ul>\n<li>\n<p>When the grub menu comes up, pick from the menu list the most likely candidate and press &rsquo;e&rsquo; for edit.<\/p>\n<\/li>\n<li>\n<p>You should see the various lines from the stanzas for the list item you\npicked (i.e. a root, kernel, and initrd stanza). You may use the up\/down\narrows to select a line. If that doesn&rsquo;t work, look for hints on the\nscreen for how to get around. Going left and right on a given line may\nrequire Ctl-b and Ctl-f for back and forward. You also may need to use\nthe delete key (not backspace) to delete characters<\/p>\n<\/li>\n<li>\n<p>Select a line, delete the characters from the end of the line, and then\ntry tab completion with various options. For example, on the root line\ntry typing simply:<\/p>\n<p>root (<\/p>\n<\/li>\n<\/ul>\n<p>And then tab. You should be presented with the available disks (numbered 0\nand up). Try typing one of the disks and hitting tab and you should be\npresented with the available partitions. Continue this process until you\nfind the one that seems right.<\/p>\n<\/li>\n<li>\n<p>When you are done and you have successfully booted, you can ensure that\nboots will work without the usb key by installing grub on all available\ndisks:<\/p>\n<pre><code>  grub-install \/dev\/sda\n  grub-install \/dev\/sdb\n<\/code><\/pre>\n<\/li>\n<\/ul>\n<hr>\n<p>###For more information<\/p>\n<ul>\n<li><a href=\"http:\/\/www.linuxjournal.com\/article\/4622\">Boot with GRUB<\/a><\/li>\n<\/ul>\n"}]}}