{"title":"Georg's Log - linux","link":[{"@attributes":{"href":"https:\/\/gms.tf\/","rel":"alternate"}},{"@attributes":{"href":"https:\/\/gms.tf\/feeds\/linux.atom.xml","rel":"self"}}],"id":"https:\/\/gms.tf\/","updated":"2026-03-29T18:00:00+02:00","entry":[{"title":"Better Datamash Build Story with Meson","link":{"@attributes":{"href":"https:\/\/gms.tf\/datamash-meson.html","rel":"alternate"}},"published":"2026-03-29T18:00:00+02:00","updated":"2026-03-29T18:00:00+02:00","author":{"name":"Georg Sauthoff"},"id":"tag:gms.tf,2026-03-29:\/datamash-meson.html","summary":"<p>This article is a small case study on introducing the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Meson_(software)\">Meson<\/a> build\ntool into a legacy <a href=\"https:\/\/en.wikipedia.org\/wiki\/GNU_Autotools\">Autotools<\/a> and <a href=\"https:\/\/en.wikipedia.org\/wiki\/Gnulib\">Gnulib<\/a> centric code base, i.e.\nthe <a href=\"https:\/\/www.gnu.org\/software\/datamash\/\">GNU Datamash<\/a> project.\nWith Meson, the <a href=\"https:\/\/opencommit.eu\/gms\/datamash\/commits\/branch\/main\">result<\/a> builds twice as fast and configures one order of magnitude faster\nwhile only requiring a few hundred lines \u2026<\/p>","content":"<p>This article is a small case study on introducing the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Meson_(software)\">Meson<\/a> build\ntool into a legacy <a href=\"https:\/\/en.wikipedia.org\/wiki\/GNU_Autotools\">Autotools<\/a> and <a href=\"https:\/\/en.wikipedia.org\/wiki\/Gnulib\">Gnulib<\/a> centric code base, i.e.\nthe <a href=\"https:\/\/www.gnu.org\/software\/datamash\/\">GNU Datamash<\/a> project.\nWith Meson, the <a href=\"https:\/\/opencommit.eu\/gms\/datamash\/commits\/branch\/main\">result<\/a> builds twice as fast and configures one order of magnitude faster\nwhile only requiring a few hundred lines of Meson build files.\nAs as side-effect of this experiment the size of the executables is halved.<\/p>\n<div class=\"toc\">\n<ul>\n<li><a href=\"#on-datamash\">On Datamash<\/a><\/li>\n<li><a href=\"#on-meson\">On Meson<\/a><\/li>\n<li><a href=\"#motivation\">Motivation<\/a><\/li>\n<li><a href=\"#configure\">Configure<\/a><\/li>\n<li><a href=\"#build\">Build<\/a><\/li>\n<li><a href=\"#code-size\">Code Size<\/a><\/li>\n<li><a href=\"#on-gnulib\">On Gnulib<\/a><\/li>\n<li><a href=\"#source-overhead\">Source Overhead<\/a><\/li>\n<li><a href=\"#test-suite\">Test Suite<\/a><\/li>\n<li><a href=\"#side-effects\">Side Effects<\/a><\/li>\n<li><a href=\"#see-also\">See also<\/a><\/li>\n<li><a href=\"#conclusion\">Conclusion<\/a><\/li>\n<\/ul>\n<\/div>\n<h2 id=\"on-datamash\">On Datamash<a class=\"headerlink\" href=\"#on-datamash\" title=\"Permanent link\">&para;<\/a><\/h2>\n<p><a href=\"https:\/\/www.gnu.org\/software\/datamash\/\">Datamash<\/a> is a command line program for running various statistics\non tabular data, such as computing the count, average or\nmedian of a column.<\/p>\n<p>Its sweet-spot seems to be ad-hoc analyses on the command line,\non systems where installing a more heavy-weight alternative such as <a href=\"https:\/\/en.wikipedia.org\/wiki\/R_(programming_language)\">R<\/a> or <a href=\"https:\/\/en.wikipedia.org\/wiki\/DuckDB\">DuckDB<\/a> would be too inconvenient.<\/p>\n<p>In the following comparisons, I compare Datamash 1.9, released in the first half of 2025,\nagainst my <a href=\"https:\/\/opencommit.eu\/gms\/datamash\/commits\/branch\/main\">datamash 'main' branch<\/a> which has Meson support and a few other fixes and improvements added,\nand is based on the end of 2025 upstream 'master' branch, i.e. release 1.9 plus just a few minor upstream commits.<\/p>\n<h2 id=\"on-meson\">On Meson<a class=\"headerlink\" href=\"#on-meson\" title=\"Permanent link\">&para;<\/a><\/h2>\n<p><a href=\"https:\/\/mesonbuild.com\/\">Meson<\/a> is modern build tool that is an alternative to autotools\nand CMake. See also my 2021 <a href=\"the-rise-of-meson.html#about-meson\">summary of meson<\/a>.<\/p>\n<h2 id=\"motivation\">Motivation<a class=\"headerlink\" href=\"#motivation\" title=\"Permanent link\">&para;<\/a><\/h2>\n<p>The <a href=\"https:\/\/en.wikipedia.org\/wiki\/Meson_(software)\">Meson<\/a> design improves on pain points many users often\nhave with <a href=\"https:\/\/en.wikipedia.org\/wiki\/GNU_Autotools\">Autotools<\/a>, such as:<\/p>\n<ul>\n<li>configure wasting time checking for tons of things that are true\n  everywhere since decades (and doing so single-threaded)<\/li>\n<li>build wasting time with inefficient auto-generated recursive\n  makefiles<\/li>\n<li>layers of <a href=\"https:\/\/en.wikipedia.org\/wiki\/M4_(computer_language)\">m4<\/a> and shell scripting are involved which results in a\n  <a href=\"https:\/\/en.wikipedia.org\/wiki\/Spaghetti_code#Big_ball_of_mud\">big ball of mud<\/a>\n  that easily breaks and is hard to fix or work on, in general<\/li>\n<li>all the Autotools generated project specific code being hard to\n  review and being an invitation for <a href=\"https:\/\/en.wikipedia.org\/wiki\/Supply_chain_attack\">supply chain attackers<\/a> such as <a href=\"https:\/\/en.wikipedia.org\/wiki\/XZ_Utils_backdoor\">Jia Tan<\/a><\/li>\n<\/ul>\n<p>The idea with Meson is that it eliminates all these issues,\nin particular that it speeds up the build, makes the project specific build files\neasier to review, reduces maintenance efforts and simplifies\nworking with the build system, in general.<\/p>\n<h2 id=\"configure\">Configure<a class=\"headerlink\" href=\"#configure\" title=\"Permanent link\">&para;<\/a><\/h2>\n<p>Using the Datamash 1.9 release, running <code>.\/configure<\/code> takes <strong>13 s<\/strong> or\nso on a modern laptop (12th Gen Intel CPU, NVMe storage, Fedora 43).\nThat configure triggers 488 checks which seems excessive.<\/p>\n<p>Some things are checked redundantly which <a href=\"https:\/\/en.wikipedia.org\/wiki\/Autoconf\">autoconf<\/a> tries to\nmitigate via caching the results, such as checks for <code>round()<\/code>:<\/p>\n<div class=\"m-code\"><pre><span><\/span><code>checking whether round is declared... yes\nchecking whether round works... yes\nchecking whether roundl is declared... yes\nchecking whether round is declared... (cached) yes\nchecking whether round works... (cached) yes\nchecking whether roundl is declared... (cached) yes\n<\/code><\/pre><\/div>\n\n<p>For some reason, other checks are invoked up to two times:<\/p>\n<div class=\"m-code\"><pre><span><\/span><code>checking for stdint.h... yes\nchecking whether stdint.h conforms to C99... yes\nchecking whether stdint.h works without ISO C predefines... yes\nchecking whether stdint.h has UINTMAX_WIDTH etc.... yes\nchecking for stdint.h... yes\nchecking for stdint.h... (cached) yes\nchecking for stdint.h... (cached) yes\n<\/code><\/pre><\/div>\n\n<p>In total there are 30 cached checks.<\/p>\n<p>Some more examples of surprising checks:<\/p>\n<div class=\"m-code\"><pre><span><\/span><code><span class=\"go\">checking whether strdup is declared... yes<\/span>\n<span class=\"go\">checking whether strnlen is declared... yes<\/span>\n<span class=\"go\">checking for strnlen... yes<\/span>\n<span class=\"go\">checking for working strnlen... yes<\/span>\n<span class=\"go\">checking for strtold... yes<\/span>\n<span class=\"go\">checking for strtoumax... yes<\/span>\n<span class=\"go\">checking whether strtoumax is declared... yes<\/span>\n<span class=\"go\">checking whether strtod obeys C99... yes<\/span>\n<span class=\"go\">checking whether strtold obeys POSIX... yes<\/span>\n<span class=\"go\">checking for strtoll... yes<\/span>\n<span class=\"go\">checking whether strtoll works... yes<\/span>\n<span class=\"go\">checking for strtoull... yes<\/span>\n<span class=\"go\">checking whether strtoull works... yes<\/span>\n<\/code><\/pre><\/div>\n\n<p>These are covered since either <a href=\"https:\/\/en.wikipedia.org\/wiki\/ANSI_C#C89\">C89<\/a>, <a href=\"https:\/\/en.wikipedia.org\/wiki\/ANSI_C#C99\">C99<\/a>, <a href=\"https:\/\/en.wikipedia.org\/wiki\/POSIX#POSIX.1-2001\">POSIX.1.-2001<\/a> or <a href=\"https:\/\/en.wikipedia.org\/wiki\/POSIX#POSIX.1-2008\">POSIX.1-2008<\/a>.\nI mean, how realistic is it that there are actually\nDatamash users who want to install the latests greatest datamash,\nbut run such a broken and vulnerable operating system that lacks support\nfor theses decades old standards.<\/p>\n<p>Sure, <a href=\"https:\/\/en.wikipedia.org\/wiki\/Retrocomputing\">retrocomputing<\/a> can be fun, but isn't the point of\nretrocomputing to run <em>old<\/em> software?<\/p>\n<hr>\n<p>In comparison, the Meson checks concentrate on the essentials\nsuch that the setup finishes in <strong>0.5 s<\/strong> or so!\nMeaning an order of magnitude faster.\nAlso, the output volume is lower and thus more useful:<\/p>\n<div class=\"m-code\"><pre><span><\/span><code><span class=\"gp\">$ <\/span>meson<span class=\"w\"> <\/span>setup<span class=\"w\"> <\/span>--buildtype<span class=\"o\">=<\/span>debugoptimized<span class=\"w\"> <\/span>..\n<span class=\"go\">The Meson build system<\/span>\n<span class=\"go\">Version: 1.8.5<\/span>\n<span class=\"go\">Source dir: \/home\/juser\/del\/datamash<\/span>\n<span class=\"go\">Build dir: \/home\/juser\/del\/datamash\/build<\/span>\n<span class=\"go\">Build type: native build<\/span>\n<span class=\"go\">Project name: datamash<\/span>\n<span class=\"go\">Project version: undefined<\/span>\n<span class=\"go\">C compiler for the host machine: cc (gcc 15.2.1 &quot;cc (GCC) 15.2.1 20260123 (Red Hat 15.2.1-7)&quot;)<\/span>\n<span class=\"go\">C linker for the host machine: cc ld.bfd 2.45.1-1<\/span>\n<span class=\"go\">Host machine cpu family: x86_64<\/span>\n<span class=\"go\">Host machine cpu: x86_64<\/span>\n<span class=\"go\">Library m found: YES<\/span>\n<span class=\"go\">Found pkg-config: YES (\/usr\/bin\/pkg-config) 2.3.0<\/span>\n<span class=\"go\">Run-time dependency nettle found: YES 3.10.1<\/span>\n<span class=\"go\">Compiler for C supports function attribute alloc_size: YES<\/span>\n<span class=\"go\">Compiler for C supports function attribute cold: YES<\/span>\n<span class=\"go\">Compiler for C supports function attribute const: YES<\/span>\n<span class=\"go\">Fetching value of define &quot;__has_c_attribute(fallthrough)&quot; : 202311<\/span>\n<span class=\"go\">Compiler for C supports function attribute format: YES<\/span>\n<span class=\"go\">Compiler for C supports function attribute malloc: YES<\/span>\n<span class=\"go\">Compiler for C supports function attribute warn_unused_result: YES<\/span>\n<span class=\"go\">Compiler for C supports function attribute pure: YES<\/span>\n<span class=\"go\">Compiler for C supports function attribute returns_nonnull: YES<\/span>\n<span class=\"go\">Compiler for C supports function attribute sentinel: YES<\/span>\n<span class=\"go\">Checking for function &quot;strtoumax&quot; : YES<\/span>\n<span class=\"go\">Configuring config.h using configuration<\/span>\n<span class=\"go\">Program .\/tests\/datamash-tests.pl found: YES (\/home\/juser\/del\/datamash\/.\/tests\/datamash-tests.pl)<\/span>\n<span class=\"go\">[..]<\/span>\n<span class=\"go\">Program .\/tests\/decorate-sort-tests.pl found: YES (\/usr\/bin\/env perl \/home\/juser\/del\/datamash\/.\/tests\/decorate-sort-tests.pl)<\/span>\n<span class=\"go\">Program perl found: YES (\/usr\/bin\/perl)<\/span>\n<span class=\"go\">Program sh found: YES (\/usr\/bin\/sh)<\/span>\n<span class=\"go\">Program msgfmt found: YES (\/usr\/bin\/msgfmt)<\/span>\n<span class=\"go\">Program msginit found: YES (\/usr\/bin\/msginit)<\/span>\n<span class=\"go\">Program msgmerge found: YES (\/usr\/bin\/msgmerge)<\/span>\n<span class=\"go\">Program xgettext found: YES (\/usr\/bin\/xgettext)<\/span>\n<span class=\"go\">Program help2man found: YES (\/usr\/bin\/help2man)<\/span>\n<span class=\"go\">Configuring version.texi using configuration<\/span>\n<span class=\"go\">Program makeinfo found: YES (\/usr\/bin\/makeinfo)<\/span>\n<span class=\"go\">Build targets in project: 22<\/span>\n<\/code><\/pre><\/div>\n\n<p>The check for <code>strtoumax<\/code> is included as an example how easy it's\nto implement such a check in Meson, if you really need it.<\/p>\n<h2 id=\"build\">Build<a class=\"headerlink\" href=\"#build\" title=\"Permanent link\">&para;<\/a><\/h2>\n<p>Allowing the make to run on 2 cores in parallel, the datamash\nautotools build finishes in <strong>4 s<\/strong> or so.<\/p>\n<p>Looking at the build messages, there are a few oddities:<\/p>\n<div class=\"m-code\"><pre><span><\/span><code><span class=\"n\">gcc<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"n\">DLOCALEDIR<\/span><span class=\"o\">=<\/span>\\<span class=\"s2\">&quot;\/home\/juser\/local\/datamash-1.9\/share\/locale<\/span><span class=\"se\">\\&quot;<\/span><span class=\"s2\"> -DHAVE_CONFIG_H -I.<\/span>\n<span class=\"w\">    <\/span><span class=\"o\">-<\/span><span class=\"n\">Ilib<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"n\">I<\/span><span class=\"o\">.\/<\/span><span class=\"n\">lib<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"n\">Isrc<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"n\">I<\/span><span class=\"o\">.\/<\/span><span class=\"n\">src<\/span><span class=\"w\">  <\/span><span class=\"o\">-<\/span><span class=\"n\">Wall<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"n\">Wextra<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"n\">Wformat<\/span><span class=\"o\">-<\/span><span class=\"n\">security<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"n\">Wswitch<\/span><span class=\"o\">-<\/span><span class=\"k\">enum<\/span>\n<span class=\"w\">    <\/span><span class=\"o\">-<\/span><span class=\"n\">Wswitch<\/span><span class=\"o\">-<\/span><span class=\"n\">default<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"n\">Wunused<\/span><span class=\"o\">-<\/span><span class=\"n\">parameter<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"n\">Wfloat<\/span><span class=\"o\">-<\/span><span class=\"n\">equal<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"n\">fdiagnostics<\/span><span class=\"o\">-<\/span><span class=\"n\">show<\/span><span class=\"o\">-<\/span><span class=\"n\">option<\/span>\n<span class=\"w\">    <\/span><span class=\"o\">-<\/span><span class=\"n\">funit<\/span><span class=\"o\">-<\/span><span class=\"n\">at<\/span><span class=\"o\">-<\/span><span class=\"n\">a<\/span><span class=\"o\">-<\/span><span class=\"n\">time<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"n\">Wmissing<\/span><span class=\"o\">-<\/span><span class=\"n\">format<\/span><span class=\"o\">-<\/span><span class=\"n\">attribute<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"n\">Wstrict<\/span><span class=\"o\">-<\/span><span class=\"n\">overflow<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"n\">Wsuggest<\/span><span class=\"o\">-<\/span><span class=\"n\">attribute<\/span><span class=\"o\">=<\/span><span class=\"k\">const<\/span>\n<span class=\"w\">    <\/span><span class=\"o\">-<\/span><span class=\"n\">Wsuggest<\/span><span class=\"o\">-<\/span><span class=\"n\">attribute<\/span><span class=\"o\">=<\/span><span class=\"n\">pure<\/span><span class=\"w\">   <\/span>\n<span class=\"w\">    <\/span><span class=\"o\">-<\/span><span class=\"n\">g<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"n\">O2<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"n\">MT<\/span><span class=\"w\"> <\/span><span class=\"n\">src<\/span><span class=\"o\">\/<\/span><span class=\"n\">datamash<\/span><span class=\"o\">-<\/span><span class=\"n\">column<\/span><span class=\"o\">-<\/span><span class=\"n\">headers<\/span><span class=\"o\">.<\/span><span class=\"n\">o<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"n\">MD<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"n\">MP<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"n\">MF<\/span><span class=\"w\"> <\/span><span class=\"n\">src<\/span><span class=\"o\">\/.<\/span><span class=\"n\">deps<\/span><span class=\"o\">\/<\/span><span class=\"n\">datamash<\/span><span class=\"o\">-<\/span><span class=\"n\">column<\/span><span class=\"o\">-<\/span><span class=\"n\">headers<\/span><span class=\"o\">.<\/span><span class=\"n\">Tpo<\/span>\n<span class=\"w\">    <\/span><span class=\"o\">-<\/span><span class=\"n\">c<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"n\">o<\/span><span class=\"w\"> <\/span><span class=\"n\">src<\/span><span class=\"o\">\/<\/span><span class=\"n\">datamash<\/span><span class=\"o\">-<\/span><span class=\"n\">column<\/span><span class=\"o\">-<\/span><span class=\"n\">headers<\/span><span class=\"o\">.<\/span><span class=\"n\">o<\/span><span class=\"w\"> <\/span><span class=\"err\">`<\/span><span class=\"n\">test<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"n\">f<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;src\/column-headers.c&#39;<\/span><span class=\"w\"> <\/span><span class=\"o\">||<\/span><span class=\"w\"> <\/span><span class=\"n\">echo<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;.\/&#39;<\/span><span class=\"err\">`<\/span><span class=\"n\">src<\/span><span class=\"o\">\/<\/span><span class=\"n\">column<\/span><span class=\"o\">-<\/span><span class=\"n\">headers<\/span><span class=\"o\">.<\/span><span class=\"n\">c<\/span>\n<\/code><\/pre><\/div>\n\n<p>There is an extra shell invocation for each translation unit,\nwhich seems to be pointless, as you either end up with<\/p>\n<div class=\"m-code\"><pre><span><\/span><code>gcc ... -c -o src\/foo.o src\/foo.c\n<\/code><\/pre><\/div>\n\n<p>or<\/p>\n<div class=\"m-code\"><pre><span><\/span><code>gcc ... -c -o src\/foo.o .\/src\/foo.c\n<\/code><\/pre><\/div>\n\n<p>There is also some redundancy in linking to <code>-lm<\/code>:<\/p>\n<div class=\"m-code\"><pre><span><\/span><code><span class=\"n\">gcc<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"n\">Wall<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"n\">Wextra<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"n\">Wformat<\/span><span class=\"o\">-<\/span><span class=\"n\">security<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"n\">Wswitch<\/span><span class=\"o\">-<\/span><span class=\"k\">enum<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"n\">Wswitch<\/span><span class=\"o\">-<\/span><span class=\"n\">default<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"n\">Wunused<\/span><span class=\"o\">-<\/span><span class=\"n\">parameter<\/span>\n<span class=\"w\">    <\/span><span class=\"o\">-<\/span><span class=\"n\">Wfloat<\/span><span class=\"o\">-<\/span><span class=\"n\">equal<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"n\">fdiagnostics<\/span><span class=\"o\">-<\/span><span class=\"n\">show<\/span><span class=\"o\">-<\/span><span class=\"n\">option<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"n\">funit<\/span><span class=\"o\">-<\/span><span class=\"n\">at<\/span><span class=\"o\">-<\/span><span class=\"n\">a<\/span><span class=\"o\">-<\/span><span class=\"n\">time<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"n\">Wmissing<\/span><span class=\"o\">-<\/span><span class=\"n\">format<\/span><span class=\"o\">-<\/span><span class=\"n\">attribute<\/span>\n<span class=\"w\">    <\/span><span class=\"o\">-<\/span><span class=\"n\">Wstrict<\/span><span class=\"o\">-<\/span><span class=\"n\">overflow<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"n\">Wsuggest<\/span><span class=\"o\">-<\/span><span class=\"n\">attribute<\/span><span class=\"o\">=<\/span><span class=\"k\">const<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"n\">Wsuggest<\/span><span class=\"o\">-<\/span><span class=\"n\">attribute<\/span><span class=\"o\">=<\/span><span class=\"n\">pure<\/span>\n<span class=\"w\">    <\/span><span class=\"o\">-<\/span><span class=\"n\">g<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"n\">O2<\/span><span class=\"w\">   <\/span><span class=\"o\">-<\/span><span class=\"n\">o<\/span><span class=\"w\"> <\/span><span class=\"n\">datamash<\/span>\n<span class=\"w\">    <\/span><span class=\"n\">src<\/span><span class=\"o\">\/<\/span><span class=\"n\">datamash<\/span><span class=\"o\">-<\/span><span class=\"n\">text<\/span><span class=\"o\">-<\/span><span class=\"n\">options<\/span><span class=\"o\">.<\/span><span class=\"n\">o<\/span><span class=\"w\"> <\/span><span class=\"n\">src<\/span><span class=\"o\">\/<\/span><span class=\"n\">datamash<\/span><span class=\"o\">-<\/span><span class=\"n\">utils<\/span><span class=\"o\">.<\/span><span class=\"n\">o<\/span><span class=\"w\"> <\/span><span class=\"n\">src<\/span><span class=\"o\">\/<\/span><span class=\"n\">datamash<\/span><span class=\"o\">-<\/span><span class=\"n\">randutils<\/span><span class=\"o\">.<\/span><span class=\"n\">o<\/span><span class=\"w\"> <\/span><span class=\"n\">src<\/span><span class=\"o\">\/<\/span><span class=\"n\">datamash<\/span><span class=\"o\">-<\/span><span class=\"n\">text<\/span><span class=\"o\">-<\/span><span class=\"n\">lines<\/span><span class=\"o\">.<\/span><span class=\"n\">o<\/span>\n<span class=\"w\">    <\/span><span class=\"n\">src<\/span><span class=\"o\">\/<\/span><span class=\"n\">datamash<\/span><span class=\"o\">-<\/span><span class=\"n\">column<\/span><span class=\"o\">-<\/span><span class=\"n\">headers<\/span><span class=\"o\">.<\/span><span class=\"n\">o<\/span><span class=\"w\"> <\/span><span class=\"n\">src<\/span><span class=\"o\">\/<\/span><span class=\"n\">datamash<\/span><span class=\"o\">-<\/span><span class=\"n\">op<\/span><span class=\"o\">-<\/span><span class=\"n\">defs<\/span><span class=\"o\">.<\/span><span class=\"n\">o<\/span><span class=\"w\"> <\/span><span class=\"n\">src<\/span><span class=\"o\">\/<\/span><span class=\"n\">datamash<\/span><span class=\"o\">-<\/span><span class=\"n\">op<\/span><span class=\"o\">-<\/span><span class=\"n\">scanner<\/span><span class=\"o\">.<\/span><span class=\"n\">o<\/span>\n<span class=\"w\">    <\/span><span class=\"n\">src<\/span><span class=\"o\">\/<\/span><span class=\"n\">datamash<\/span><span class=\"o\">-<\/span><span class=\"n\">op<\/span><span class=\"o\">-<\/span><span class=\"n\">parser<\/span><span class=\"o\">.<\/span><span class=\"n\">o<\/span><span class=\"w\"> <\/span><span class=\"n\">src<\/span><span class=\"o\">\/<\/span><span class=\"n\">datamash<\/span><span class=\"o\">-<\/span><span class=\"n\">field<\/span><span class=\"o\">-<\/span><span class=\"n\">ops<\/span><span class=\"o\">.<\/span><span class=\"n\">o<\/span><span class=\"w\"> <\/span><span class=\"n\">src<\/span><span class=\"o\">\/<\/span><span class=\"n\">datamash<\/span><span class=\"o\">-<\/span><span class=\"n\">crosstab<\/span><span class=\"o\">.<\/span><span class=\"n\">o<\/span>\n<span class=\"w\">    <\/span><span class=\"n\">src<\/span><span class=\"o\">\/<\/span><span class=\"n\">datamash<\/span><span class=\"o\">-<\/span><span class=\"n\">double<\/span><span class=\"o\">-<\/span><span class=\"n\">format<\/span><span class=\"o\">.<\/span><span class=\"n\">o<\/span><span class=\"w\"> <\/span><span class=\"n\">src<\/span><span class=\"o\">\/<\/span><span class=\"n\">datamash<\/span><span class=\"o\">-<\/span><span class=\"n\">datamash<\/span><span class=\"o\">.<\/span><span class=\"n\">o<\/span><span class=\"w\"> <\/span><span class=\"n\">lib<\/span><span class=\"o\">\/<\/span><span class=\"n\">libdatamash<\/span><span class=\"o\">.<\/span><span class=\"n\">a<\/span>\n<span class=\"w\">    <\/span><span class=\"o\">-<\/span><span class=\"n\">lm<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"n\">lm<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"n\">lm<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"n\">lm<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"n\">lm<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"n\">lm<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"n\">lm<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"n\">lm<\/span><span class=\"w\">              <\/span><span class=\"o\">-<\/span><span class=\"n\">lm<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"n\">lm<\/span><span class=\"w\">   <\/span><span class=\"o\">-<\/span><span class=\"n\">lm<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"n\">lm<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"n\">lm<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"n\">lm<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"n\">lm<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"n\">lm<\/span>\n<\/code><\/pre><\/div>\n\n<p>Meaning for unknown reasons Autotools links 16 times to libm.\nArguably, this is a good example of how even Autotools fans\nstruggle with its complexity and are overwhelmed by it.<\/p>\n<hr>\n<p>In comparison, Meson builds everything in <strong>1.5 s<\/strong> or so (i.e. more\nthan twice as fast) and its default output is more reasonable.\nThat means by default the user isn't spammed with overly long and\ncryptic low-level command invocations, but useful high-level\nprogress:<\/p>\n<div class=\"m-code\"><pre><span><\/span><code><span class=\"gp\">$ <\/span>ninja<span class=\"w\"> <\/span>-j2<span class=\"w\"> <\/span>\n<span class=\"go\">[1\/50] Compiling C object datamash.p\/src_column-headers.c.o<\/span>\n<span class=\"go\">[2\/50] Compiling C object datamash.p\/src_crosstab.c.o<\/span>\n<span class=\"go\">[..]<\/span>\n<span class=\"go\">[15\/50] Compiling C object datamash.p\/_usr_share_gnulib_lib_hashcode-mem.c.o<\/span>\n<span class=\"go\">[16\/50] Compiling C object datamash.p\/_usr_share_gnulib_lib_hashcode-string2.c.o<\/span>\n<span class=\"go\">[..]<\/span>\n<span class=\"go\">[31\/50] Linking target datamash<\/span>\n<span class=\"go\">[32\/50] Building translation po\/da\/LC_MESSAGES\/datamash-da.mo<\/span>\n<span class=\"go\">[33\/50] Building translation po\/de\/LC_MESSAGES\/datamash-de.mo<\/span>\n<span class=\"go\">[34\/50] Compiling C object decorate.p\/_usr_share_gnulib_lib_xmalloc.c.o<\/span>\n<span class=\"go\">[35\/50] Building translation po\/eo\/LC_MESSAGES\/datamash-eo.mo<\/span>\n<span class=\"go\">[..]<\/span>\n<span class=\"go\">[40\/50] Linking target decorate<\/span>\n<span class=\"go\">[48\/50] Generating datamash manual with a custom command (wrapped by meson to set env)<\/span>\n<span class=\"go\">[49\/50] Generating decorate manual with a custom command (wrapped by meson to set env)<\/span>\n<span class=\"go\">[50\/50] Generating gen-info with a custom command<\/span>\n<\/code><\/pre><\/div>\n\n<p>Of course, when a command errors out the full command line and\nerror message are presented to the user and alternatively, one\ncan run the build in verbose mode by adding <code>-v<\/code> to the ninja\ncommand.<\/p>\n<p>The verbose mode doesn't slow down the build noticeable.\nEnabling it shows that meson doesn't have the same issues with\nlinking libm:<\/p>\n<div class=\"m-code\"><pre><span><\/span><code>cc  -o datamash\n    datamash.p\/src_column-headers.c.o datamash.p\/src_crosstab.c.o datamash.p\/src_datamash.c.o\n    datamash.p\/src_double-format.c.o datamash.p\/src_field-ops.c.o datamash.p\/src_op-defs.c.o\n    datamash.p\/src_op-parser.c.o datamash.p\/src_op-scanner.c.o datamash.p\/src_randutils.c.o\n    datamash.p\/src_text-lines.c.o datamash.p\/src_text-options.c.o datamash.p\/src_utils.c.o\n    datamash.p\/modern_system.c.o datamash.p\/_usr_share_gnulib_lib_exitfail.c.o\n    datamash.p\/_usr_share_gnulib_lib_hash.c.o datamash.p\/_usr_share_gnulib_lib_hashcode-mem.c.o\n    datamash.p\/_usr_share_gnulib_lib_hashcode-string2.c.o datamash.p\/_usr_share_gnulib_lib_linebuffer.c.o\n    datamash.p\/_usr_share_gnulib_lib_next-prime.c.o datamash.p\/_usr_share_gnulib_lib_version-etc.c.o\n    datamash.p\/_usr_share_gnulib_lib_xalloc-die.c.o datamash.p\/_usr_share_gnulib_lib_xmalloc.c.o\n    -Wl,--as-needed -Wl,--no-undefined -Wl,--start-group -lm \/usr\/lib64\/libnettle.so -Wl,--end-group\n<\/code><\/pre><\/div>\n\n<p>In contrast to the Autotools status quo, I decided against creating a static library of the Gnulib parts,\nsince the effort doesn't pay off, because it's only used by two executables.\nInstead I link only those Gnulib translation units directly which are essential for each binary.<\/p>\n<h2 id=\"code-size\">Code Size<a class=\"headerlink\" href=\"#code-size\" title=\"Permanent link\">&para;<\/a><\/h2>\n<p>Comparing the sizes of executables in the Datamash 1.9 vs. Meson branch shows substantial\nsavings with Meson (all sizes in bytes, via <code>size -G<\/code>):<\/p>\n<table class=\"m-table\">\n<thead>\n<tr>\n<th>branch<\/th>\n<th>filename<\/th>\n<th><a href=\"https:\/\/en.wikipedia.org\/wiki\/Code_segment\">text<\/a><\/th>\n<th><a href=\"https:\/\/en.wikipedia.org\/wiki\/Data_segment\">data<\/a><\/th>\n<th><a href=\"https:\/\/en.wikipedia.org\/wiki\/.bss\">bss<\/a><\/th>\n<th>total<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>1.9<\/td>\n<td>datamash<\/td>\n<td>97370<\/td>\n<td>42120<\/td>\n<td>968<\/td>\n<td>140458<\/td>\n<\/tr>\n<tr>\n<td>meson<\/td>\n<td>datamash<\/td>\n<td>46736<\/td>\n<td>35756<\/td>\n<td>640<\/td>\n<td>83132<\/td>\n<\/tr>\n<tr>\n<td>1.9<\/td>\n<td>decorate<\/td>\n<td>27348<\/td>\n<td>22068<\/td>\n<td>1576<\/td>\n<td>50992<\/td>\n<\/tr>\n<tr>\n<td>meson<\/td>\n<td>decorate<\/td>\n<td>15520<\/td>\n<td>16224<\/td>\n<td>1248<\/td>\n<td>32992<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>That means that the code size of the datamash executable is reduced by over <strong>50 percent<\/strong>.<\/p>\n<p>These space savings are due to only linking those parts of <a href=\"https:\/\/en.wikipedia.org\/wiki\/Gnulib\">Gnulib<\/a> that are essential,\ni.e. Datamash 1.9 links the following 120 Gnulib translation units:<\/p>\n<div class=\"m-code\"><pre><span><\/span><code><span class=\"go\">af_alg.c                        hash-pjw-bare.c                 stat-time.c<\/span>\n<span class=\"go\">arpa_inet.c                     hash-pjw.c                      stdlib.c<\/span>\n<span class=\"go\">asnprintf.c                     ialloc.c                        striconv.c<\/span>\n<span class=\"go\">base64.c                        imaxtostr.c                     stripslash.c<\/span>\n<span class=\"go\">basename.c                      inttostr.c                      strnlen1.c<\/span>\n<span class=\"go\">basename-lgpl.c                 linebuffer.c                    sys_socket.c<\/span>\n<span class=\"go\">bitrotate.c                     localcharset.c                  trim.c<\/span>\n<span class=\"go\">c32isalnum.c                    localename.c                    u64.c<\/span>\n<span class=\"go\">c32isalpha.c                    localename-environ.c            uinttostr.c<\/span>\n<span class=\"go\">c32isblank.c                    localename-table.c              umaxtostr.c<\/span>\n<span class=\"go\">c32iscntrl.c                    localename-unsafe.c             unicase\/tolower.c<\/span>\n<span class=\"go\">c32isdigit.c                    malloca.c                       unictype\/ctype_alnum.c<\/span>\n<span class=\"go\">c32isgraph.c                    math.c                          unictype\/ctype_alpha.c<\/span>\n<span class=\"go\">c32islower.c                    mbchar.c                        unictype\/ctype_blank.c<\/span>\n<span class=\"go\">c32isprint.c                    mbrtoc32.c                      unictype\/ctype_cntrl.c<\/span>\n<span class=\"go\">c32ispunct.c                    mbrtowc.c                       unictype\/ctype_digit.c<\/span>\n<span class=\"go\">c32isspace.c                    mbslen.c                        unictype\/ctype_graph.c<\/span>\n<span class=\"go\">c32isupper.c                    mbsstr.c                        unictype\/ctype_lower.c<\/span>\n<span class=\"go\">c32isxdigit.c                   mbszero.c                       unictype\/ctype_print.c<\/span>\n<span class=\"go\">c32tolower.c                    mbuiter.c                       unictype\/ctype_punct.c<\/span>\n<span class=\"go\">c32width.c                      mbuiterf.c                      unictype\/ctype_space.c<\/span>\n<span class=\"go\">c-ctype.c                       md5.c                           unictype\/ctype_upper.c<\/span>\n<span class=\"go\">cloexec.c                       md5-stream.c                    unictype\/ctype_xdigit.c<\/span>\n<span class=\"go\">closeout.c                      offtostr.c                      unistd.c<\/span>\n<span class=\"go\">close-stream.c                  printf-args.c                   unistr\/u8-mbtoucr.c<\/span>\n<span class=\"go\">c-strcasecmp.c                  printf-parse.c                  unistr\/u8-uctomb-aux.c<\/span>\n<span class=\"go\">dirname.c                       progname.c                      unistr\/u8-uctomb.c<\/span>\n<span class=\"go\">dirname-lgpl.c                  propername.c                    uniwidth\/width.c<\/span>\n<span class=\"go\">exitfail.c                      quotearg.c                      vasnprintf.c<\/span>\n<span class=\"go\">fcntl.c                         reallocarray.c                  version-etc.c<\/span>\n<span class=\"go\">fd-hook.c                       realloc.c                       vsnzprintf.c<\/span>\n<span class=\"go\">fpurge.c                        setlocale_null.c                wctype-h.c<\/span>\n<span class=\"go\">freading.c                      setlocale_null-unlocked.c       xalloc-die.c<\/span>\n<span class=\"go\">getlocalename_l-unsafe.c        sha1.c                          xmalloc.c<\/span>\n<span class=\"go\">getprogname.c                   sha1-stream.c                   xsize.c<\/span>\n<span class=\"go\">glthread\/lock.c                 sha256.c                        xstriconv.c<\/span>\n<span class=\"go\">glthread\/once.c                 sha256-stream.c                 xstrtol.c<\/span>\n<span class=\"go\">glthread\/threadlib.c            sha512.c                        xstrtol-error.c<\/span>\n<span class=\"go\">hard-locale.c                   sha512-stream.c                 xstrtoul.c<\/span>\n<span class=\"go\">hash.c                          sh-quote.c                      xstrtoumax.c<\/span>\n<\/code><\/pre><\/div>\n\n<p>While the Meson branch only links the following 9 Gnulib translation units:<\/p>\n<div class=\"m-code\"><pre><span><\/span><code><span class=\"go\">exitfail.c<\/span>\n<span class=\"go\">hash.c<\/span>\n<span class=\"go\">hashcode-mem.c<\/span>\n<span class=\"go\">hashcode-string2.c<\/span>\n<span class=\"go\">linebuffer.c<\/span>\n<span class=\"go\">next-prime.c<\/span>\n<span class=\"go\">version-etc.c<\/span>\n<span class=\"go\">xalloc-die.c<\/span>\n<span class=\"go\">xmalloc.c<\/span>\n<\/code><\/pre><\/div>\n\n<p>The following measures make this reduction possible:<\/p>\n<ol>\n<li>Identifying Gnulib functionality that is widely available in the standard library\n   or in quasi-standard libraries and use that instead of Gnulib.<\/li>\n<li>Removal of accidental code bloat.<\/li>\n<\/ol>\n<p>For example, after <a href=\"https:\/\/opencommit.eu\/gms\/datamash\/commit\/9a7b5e271e6d000cf9c9db2587c6fb7355726a6d\">fixing the percentile computation<\/a>, I was able to replace the separate <a href=\"https:\/\/opencommit.eu\/gms\/datamash\/commit\/27ee8d1995a8032bb579454140225612636afd5c\">median implementation<\/a> with an alias that saves 200 bytes or so in code size.<\/p>\n<p>Since <a href=\"https:\/\/en.wikipedia.org\/wiki\/Nettle_(cryptographic_library)\">libnettle<\/a> is widely available and often pre-installed as core system dependency,\nit suggests itself for the cryptographic hash and base64 computations in datamash instead of <a href=\"https:\/\/en.wiktionary.org\/wiki\/vendor#Verb\">vendoring<\/a> in the Gnulib versions.\n<a href=\"https:\/\/opencommit.eu\/gms\/datamash\/commit\/63f234ec7ff33d81af82f3772f09e4ef8a9d7598\">Switching to libnettle<\/a>\nis a moderate source code change but saves several kilobytes in the executable.<\/p>\n<p>Another source of Gnulib over-usage is quoting, i.e. after <a href=\"https:\/\/opencommit.eu\/gms\/datamash\/commit\/df9e598b89c00483d0e8221ca866717e33195355\">replacing a popen<\/a> invocation with an idiomatic direct fork and exec sequence, we can get rid of Gnulib's <code>sh-quote.c<\/code> translation unit and its dependency hell.\nSimilarly, to me a <a href=\"https:\/\/opencommit.eu\/gms\/datamash\/commit\/b4a30543833c89bc70099cf003001b598be57b37\">minimal quoting implementation<\/a> in the datamash decorate command is preferable as it's more size efficient and easier to review.<\/p>\n<p>One more code bloat source is how Datamash 1.9 quotes trivial arguments in error messages, i.e. via a Gnulib function that quotes in a localized fashion. I don't think that this is worth it and thus switched the branch to a <a href=\"https:\/\/opencommit.eu\/gms\/datamash\/src\/commit\/1fc869a607497c7a366af9d8a18abf5d3d8ddd87\/modern\/system.c\">more minimal version<\/a>.<\/p>\n<p>In another example, I changed the code into invoking the standardized <a href=\"https:\/\/opencommit.eu\/gms\/datamash\/commit\/36768af613dca2c92b2b24e78062c5991d44d980\">strtoumax<\/a> instead of using the closely named but non-conforming Gnulib version.<\/p>\n<p>Besides code sizes, Gnulib versions are always suspect to being less battle-tested and optimized,\nand when bugs are fixed in them, it's easy to miss updates, i.e. in contrast to a shared system library,\nas Gnulib is designed to be <a href=\"https:\/\/en.wiktionary.org\/wiki\/vendor#Verb\">vendored<\/a>.<\/p>\n<p>Of course, the code changes aren't really Meson specific,\nbut I argue that the way I integrated the Gnulib dependencies into to Meson build file simplified\nthose changes vs. having to hack the autotools Gnulib integration to link only the essential translation units.<\/p>\n<h2 id=\"on-gnulib\">On Gnulib<a class=\"headerlink\" href=\"#on-gnulib\" title=\"Permanent link\">&para;<\/a><\/h2>\n<p><a href=\"https:\/\/en.wikipedia.org\/wiki\/Gnulib\">Gnulib<\/a> describes itself as source-code library and a user is supposed to <a href=\"https:\/\/en.wiktionary.org\/wiki\/vendor#Verb\">vendor<\/a> the gnulib parts one\nneeds into a project (i.e. copy and bundle gnulib source code files with his\/her project).<\/p>\n<p>Although the Gnulib manual has a few paragraphs on its philosophy and design, it doesn't give <em>any<\/em> rationale for <em>that<\/em> odd design decision. It <a href=\"https:\/\/www.gnu.org\/software\/gnulib\/manual\/gnulib.html#Library-vs_002e-Reusable-Code\">just states<\/a>:<\/p>\n<blockquote>\n<p>Classical libraries are installed as binary object code. Gnulib is different: It is used as a source code library. Each package that uses Gnulib thus ships with part of the Gnulib source code. The used portion of Gnulib is tailored to the package: A build tool, called gnulib-tool, is provided that copies a tailored subset of Gnulib into the package.<\/p>\n<\/blockquote>\n<p>In my opinion Gnulib is different here for now good reason.\nThere is no reason why Gnulib couldn't provide the same functionality, i.e. the portability wrapper and glue code functions\nin a secure, reliable and structured way, as a normal shared library, like - say - <a href=\"https:\/\/libbsd.freedesktop.org\/\">libbsd<\/a>.<\/p>\n<p>Even worse, this introduces several disadvantages:<\/p>\n<ol>\n<li>Similar to all the generated build file cruft Autotools projects come with,\n   Gnulib code from some arbitrary version dumped into a <code>lib\/<\/code> sub-directory is\n   just another great opportunity for a <a href=\"https:\/\/en.wikipedia.org\/wiki\/Supply_chain_attack\">supply chain attacker<\/a> to hide malicious code.<\/li>\n<li>As with static linking, and in contrast to shared linking, updates to library\n   require rebuilding all users.<\/li>\n<li>Worse than with static linking, since the code is bundled, the dependency isn't obvious,\n   and thus easy to miss. For example, it cannot be searched for with a distribution's package manager.<\/li>\n<li>Bundling Gnulib multiplies reviewing efforts, since with each copy it has to be checked\n   whether it can be traced to the upstream Gnulib repository or contains malicious modifications.<\/li>\n<\/ol>\n<p>Curiously, when it comes to <a href=\"https:\/\/www.gnu.org\/software\/gnulib\/manual\/gnulib.html#Comparison-with-other-source-code-libraries\">supply chain security Gnulib tries to talk down\nother language package ecosystems<\/a>, while completely ignoring it's own fundamental issues:<\/p>\n<blockquote>\n<p>Many programming languages nowadays have\nan ecosystem of reusable source code packages, available through a central site,\ntogether with a tool that downloads dependencies from this central site. \n[.. Python example ..]\nMost of them are vulnerable to supply chain attacks.\n[..]\nWhile some mitigations exist, they are often cumbersome to put in place.\n[..]\nGnulib is not vulnerable to such attacks, because all of its code is managed in a single repository, with a limited set of committers and with established code review practices.<\/p>\n<\/blockquote>\n<p>This misses the point very hard.<\/p>\n<p>If you take their Python example, the stuff Gnulib provides <em>is<\/em> part of the Python standard library, since it <em>exactly<\/em> fits Python's <a href=\"https:\/\/en.wikipedia.org\/wiki\/Batteries_Included\">batteries included<\/a> philosophy.\nThus, there is really no need to install some random package via pip for basic functionality,\nas the Gnulib team is trying to insinuate here.<\/p>\n<p>Having a limited set of comitters and 'established code review practices' transports almost zero information.\nMany projects could claim that.\nEven the <a href=\"https:\/\/en.wikipedia.org\/wiki\/XZ_Utils_backdoor\">xz project<\/a> had a limited set of comitters.<\/p>\n<p>The main <a href=\"https:\/\/en.wikipedia.org\/wiki\/Supply_chain_attack\">supply chain<\/a> risk the Gnulib team should worry about is the <a href=\"https:\/\/en.wiktionary.org\/wiki\/vendor#Verb\">vendoring<\/a> of their 'library'.\nInstead of encouraging developers to bundle their questionable\nwrappers, where better standardized wrappers or common libraries\nexist for decades, Gnulib could look into releasing a proper shared library.<\/p>\n<h2 id=\"source-overhead\">Source Overhead<a class=\"headerlink\" href=\"#source-overhead\" title=\"Permanent link\">&para;<\/a><\/h2>\n<p>Another important metric is the size of the build system in a project.\nThe more files and more lines of code the higher the efforts for review, maintenance and debugging when something goes wrong.<\/p>\n<p>With Datamash 1.9 the <a href=\"https:\/\/en.wikipedia.org\/wiki\/GNU_Autotools\">Autotools<\/a> build system is quite large.\nThere are at least:<\/p>\n<div class=\"m-code\"><pre><span><\/span><code>$<span class=\"w\"> <\/span>wc<span class=\"w\"> <\/span>-l<span class=\"w\"> <\/span>aclocal.m4<span class=\"w\"> <\/span>cfg.mk<span class=\"w\"> <\/span>config.in<span class=\"w\">  <\/span>configure<span class=\"w\">  <\/span>configure.ac<span class=\"w\"> <\/span>GNUmakefile<span class=\"w\"> <\/span>init.cfg<span class=\"w\"> <\/span>maint.mk<span class=\"w\"> <\/span>Makefile.am<span class=\"w\"> <\/span>Makefile.in<span class=\"w\">    <\/span>\n<span class=\"w\">   <\/span><span class=\"m\">1926<\/span><span class=\"w\"> <\/span>aclocal.m4\n<span class=\"w\">    <\/span><span class=\"m\">193<\/span><span class=\"w\"> <\/span>cfg.mk\n<span class=\"w\">   <\/span><span class=\"m\">3047<\/span><span class=\"w\"> <\/span>config.in\n<span class=\"w\">  <\/span><span class=\"m\">56440<\/span><span class=\"w\"> <\/span>configure\n<span class=\"w\">    <\/span><span class=\"m\">229<\/span><span class=\"w\"> <\/span>configure.ac\n<span class=\"w\">    <\/span><span class=\"m\">130<\/span><span class=\"w\"> <\/span>GNUmakefile\n<span class=\"w\">     <\/span><span class=\"m\">79<\/span><span class=\"w\"> <\/span>init.cfg\n<span class=\"w\">   <\/span><span class=\"m\">1950<\/span><span class=\"w\"> <\/span>maint.mk\n<span class=\"w\">    <\/span><span class=\"m\">347<\/span><span class=\"w\"> <\/span>Makefile.am\n<span class=\"w\">  <\/span><span class=\"m\">12206<\/span><span class=\"w\"> <\/span>Makefile.in\n<span class=\"w\">  <\/span><span class=\"m\">76547<\/span><span class=\"w\"> <\/span>total\n$<span class=\"w\"> <\/span>du<span class=\"w\"> <\/span>-ch<span class=\"w\"> <\/span>aclocal.m4<span class=\"w\"> <\/span>cfg.mk<span class=\"w\"> <\/span>config.in<span class=\"w\">  <\/span>configure<span class=\"w\">  <\/span>configure.ac<span class=\"w\"> <\/span>GNUmakefile<span class=\"w\"> <\/span>init.cfg<span class=\"w\"> <\/span>maint.mk<span class=\"w\"> <\/span>Makefile.am<span class=\"w\"> <\/span>Makefile.in<span class=\"w\">  <\/span><span class=\"p\">|<\/span><span class=\"w\"> <\/span>tail<span class=\"w\"> <\/span>-1\n<span class=\"m\">2<\/span>.5M<span class=\"w\">    <\/span>total\n<\/code><\/pre><\/div>\n\n<p>The <code>configure<\/code> is auto-generated from other files and part of the release source archive\nto allow users compilation without having to install <a href=\"https:\/\/en.wikipedia.org\/wiki\/Autoconf\">autoconf<\/a>.<\/p>\n<p>After running <code>.\/configure<\/code> another large Makefile is generated:<\/p>\n<div class=\"m-code\"><pre><span><\/span><code>$<span class=\"w\"> <\/span>wc<span class=\"w\"> <\/span>-l<span class=\"w\"> <\/span>Makefile\n<span class=\"m\">12206<\/span><span class=\"w\"> <\/span>Makefile\n$<span class=\"w\"> <\/span>du<span class=\"w\"> <\/span>-h<span class=\"w\"> <\/span>Makefile\n800K<span class=\"w\">    <\/span>Makefile\n<\/code><\/pre><\/div>\n\n<p>For comparison, the actual Datamash Source Code is less than 400 kilobytes large:<\/p>\n<div class=\"m-code\"><pre><span><\/span><code>$<span class=\"w\"> <\/span>du<span class=\"w\"> <\/span>-h<span class=\"w\"> <\/span>src\n316K<span class=\"w\">    <\/span>src\n<\/code><\/pre><\/div>\n\n<p>In addition, the <a href=\"https:\/\/en.wiktionary.org\/wiki\/vendor#Verb\">vendored<\/a> Gnulib sources come with over 30 thousand lines (or over 1 megabyte) of <a href=\"https:\/\/en.wikipedia.org\/wiki\/M4_(computer_language)\">m4<\/a> macros and over 4 MiB of bundled C source code:<\/p>\n<div class=\"m-code\"><pre><span><\/span><code>$<span class=\"w\"> <\/span>wc<span class=\"w\"> <\/span>-l<span class=\"w\"> <\/span>m4\/*<span class=\"w\"> <\/span><span class=\"p\">|<\/span><span class=\"w\"> <\/span>tail<span class=\"w\"> <\/span>-1\n<span class=\"w\">  <\/span><span class=\"m\">33272<\/span><span class=\"w\"> <\/span>total\n$<span class=\"w\"> <\/span>du<span class=\"w\"> <\/span>-h<span class=\"w\"> <\/span>m4\n<span class=\"m\">1<\/span>.7M<span class=\"w\">    <\/span>m4\n$<span class=\"w\"> <\/span>du<span class=\"w\"> <\/span>-h<span class=\"w\"> <\/span>lib\n<span class=\"m\">4<\/span>.3M<span class=\"w\">    <\/span>lib\n<\/code><\/pre><\/div>\n\n<p>When building from the datamash git repository, one also has to consider\nat least just another largish shell script:<\/p>\n<div class=\"m-code\"><pre><span><\/span><code>$<span class=\"w\"> <\/span>wc<span class=\"w\"> <\/span>-l<span class=\"w\"> <\/span>bootstrap*\n<span class=\"w\"> <\/span><span class=\"m\">1087<\/span><span class=\"w\"> <\/span>bootstrap\n<span class=\"w\">  <\/span><span class=\"m\">177<\/span><span class=\"w\"> <\/span>bootstrap.conf\n<span class=\"w\"> <\/span><span class=\"m\">1264<\/span><span class=\"w\"> <\/span>total\n<\/code><\/pre><\/div>\n\n<p>This is for generating Autotools files which aren't part of the repository,\nsuch as the large <code>configure<\/code> script.\nIt also updates the Gnulib bundle and downloads translation files.\nAgain this is another good target for a <a href=\"https:\/\/en.wikipedia.org\/wiki\/Supply_chain_attack\">supply chain attacker<\/a> to hide malicious code.\nOften such <a href=\"https:\/\/en.wikipedia.org\/wiki\/Boilerplate_code\">boilerplate code<\/a> is just copy and pasted from some upstream and doesn't need to be customized,\nbut having to review that, having to check whether it was modified, if it was whether the modification is legitimate,\nis quite wasteful, i.e. busy work that isn't necessary when using a better build tool.<\/p>\n<p>Having all those bits and pieces bundled also increases the risk only updating some of it and thus ending\nup with a collection of moving parts and state that is very likely to be untested.<\/p>\n<hr>\n<p>In comparison, with the Meson branch there is just:<\/p>\n<div class=\"m-code\"><pre><span><\/span><code>$<span class=\"w\"> <\/span>wc<span class=\"w\"> <\/span>-l<span class=\"w\"> <\/span>meson.build<span class=\"w\"> <\/span>meson.options<span class=\"w\"> <\/span>po\/meson.build<span class=\"w\"> <\/span>\n<span class=\"w\">  <\/span><span class=\"m\">404<\/span><span class=\"w\"> <\/span>meson.build\n<span class=\"w\">    <\/span><span class=\"m\">6<\/span><span class=\"w\"> <\/span>meson.options\n<span class=\"w\">    <\/span><span class=\"m\">5<\/span><span class=\"w\"> <\/span>po\/meson.build\n<span class=\"w\">  <\/span><span class=\"m\">415<\/span><span class=\"w\"> <\/span>total\n$<span class=\"w\"> <\/span>du<span class=\"w\"> <\/span>-ch<span class=\"w\"> <\/span>meson.build<span class=\"w\"> <\/span>meson.options<span class=\"w\"> <\/span>po\/meson.build<span class=\"w\"> <\/span><span class=\"p\">|<\/span><span class=\"w\"> <\/span>tail<span class=\"w\"> <\/span>-1\n28K<span class=\"w\">     <\/span>total\n<\/code><\/pre><\/div>\n\n<p>The 404 lines meson.build is structured like this:<\/p>\n<table class=\"m-table\">\n<thead>\n<tr>\n<th>#lines<\/th>\n<th>category<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>56<\/td>\n<td>comments<\/td>\n<\/tr>\n<tr>\n<td>55<\/td>\n<td>empty lines<\/td>\n<\/tr>\n<tr>\n<td>91<\/td>\n<td>configuration data macro settings<\/td>\n<\/tr>\n<tr>\n<td>53<\/td>\n<td>test suite definitions<\/td>\n<\/tr>\n<tr>\n<td>149<\/td>\n<td>main build target definitions<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>With Meson the megabytes of Autotools bloat listed above can be dropped.\nSince the Meson build file uses a system-wide installed Gnulib, a few more megabytes can be dropped,\nremoving the bundled copy.\nRemoving those files would reduce the size of the release\n<code>.tar.gz<\/code> archive by almost 50 %, i.e. from 2.6 MiB to 1.4 MiB or\nso.<\/p>\n<h2 id=\"test-suite\">Test Suite<a class=\"headerlink\" href=\"#test-suite\" title=\"Permanent link\">&para;<\/a><\/h2>\n<p><a href=\"https:\/\/www.gnu.org\/software\/datamash\/\">Datamash<\/a> comes with a test suite, i.e. a bunch of (mostly Perl) scripts that invoke the datamash\nexecutables with various test inputs.<\/p>\n<p>With <a href=\"https:\/\/en.wikipedia.org\/wiki\/GNU_Autotools\">Autotools<\/a> a test suite invocation runs like this:<\/p>\n<div class=\"m-code\"><pre><span><\/span><code>$<span class=\"w\"> <\/span>make<span class=\"w\"> <\/span>check\n<span class=\"k\">if<\/span><span class=\"w\"> <\/span><span class=\"nb\">test<\/span><span class=\"w\"> <\/span>-d<span class=\"w\"> <\/span>.\/.git<span class=\"w\">                               <\/span><span class=\"se\">\\<\/span>\n<span class=\"w\">        <\/span><span class=\"o\">&amp;&amp;<\/span><span class=\"w\"> <\/span>git<span class=\"w\"> <\/span>--version<span class=\"w\"> <\/span>&gt;\/dev\/null<span class=\"w\"> <\/span><span class=\"m\">2<\/span>&gt;<span class=\"p\">&amp;<\/span><span class=\"m\">1<\/span><span class=\"p\">;<\/span><span class=\"w\"> <\/span><span class=\"k\">then<\/span><span class=\"w\">                  <\/span><span class=\"se\">\\<\/span>\n<span class=\"w\">  <\/span><span class=\"nb\">cd<\/span><span class=\"w\"> <\/span>.<span class=\"w\"> <\/span><span class=\"o\">&amp;&amp;<\/span><span class=\"w\">                                               <\/span><span class=\"se\">\\<\/span>\n<span class=\"w\">  <\/span>git<span class=\"w\"> <\/span>submodule<span class=\"w\"> <\/span>--quiet<span class=\"w\"> <\/span>foreach<span class=\"w\">                                 <\/span><span class=\"se\">\\<\/span>\n<span class=\"w\">      <\/span><span class=\"s1\">&#39;test &quot;$(git rev-parse &quot;$sha1&quot;)&quot;                  \\<\/span>\n<span class=\"s1\">          = &quot;$(git merge-base origin &quot;$sha1&quot;)&quot;&#39;<\/span><span class=\"w\">         <\/span><span class=\"se\">\\<\/span>\n<span class=\"w\">    <\/span><span class=\"o\">||<\/span><span class=\"w\"> <\/span><span class=\"o\">{<\/span><span class=\"w\"> <\/span><span class=\"nb\">echo<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;maint.mk: found non-public submodule commit&#39;<\/span><span class=\"w\"> <\/span>&gt;<span class=\"p\">&amp;<\/span><span class=\"m\">2<\/span><span class=\"p\">;<\/span><span class=\"w\">        <\/span><span class=\"se\">\\<\/span>\n<span class=\"w\">         <\/span><span class=\"nb\">exit<\/span><span class=\"w\"> <\/span><span class=\"m\">1<\/span><span class=\"p\">;<\/span><span class=\"w\"> <\/span><span class=\"o\">}<\/span><span class=\"p\">;<\/span><span class=\"w\">                                             <\/span><span class=\"se\">\\<\/span>\n<span class=\"k\">else<\/span><span class=\"w\">                                                            <\/span><span class=\"se\">\\<\/span>\n<span class=\"w\">  <\/span>:<span class=\"w\"> <\/span><span class=\"p\">;<\/span><span class=\"w\">                                                           <\/span><span class=\"se\">\\<\/span>\n<span class=\"k\">fi<\/span>\nmake<span class=\"w\">  <\/span>check-recursive\nmake<span class=\"o\">[<\/span><span class=\"m\">1<\/span><span class=\"o\">]<\/span>:<span class=\"w\"> <\/span>Entering<span class=\"w\"> <\/span>directory<span class=\"w\"> <\/span><span class=\"s1\">&#39;\/home\/juser\/del\/datamash-1.9&#39;<\/span>\nMaking<span class=\"w\"> <\/span>check<span class=\"w\"> <\/span><span class=\"k\">in<\/span><span class=\"w\"> <\/span>po\nmake<span class=\"o\">[<\/span><span class=\"m\">2<\/span><span class=\"o\">]<\/span>:<span class=\"w\"> <\/span>Entering<span class=\"w\"> <\/span>directory<span class=\"w\"> <\/span><span class=\"s1\">&#39;\/home\/juser\/del\/datamash-1.9\/po&#39;<\/span>\nmake<span class=\"o\">[<\/span><span class=\"m\">2<\/span><span class=\"o\">]<\/span>:<span class=\"w\"> <\/span>Nothing<span class=\"w\"> <\/span>to<span class=\"w\"> <\/span>be<span class=\"w\"> <\/span><span class=\"k\">done<\/span><span class=\"w\"> <\/span><span class=\"k\">for<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;check&#39;<\/span>.\nmake<span class=\"o\">[<\/span><span class=\"m\">2<\/span><span class=\"o\">]<\/span>:<span class=\"w\"> <\/span>Leaving<span class=\"w\"> <\/span>directory<span class=\"w\"> <\/span><span class=\"s1\">&#39;\/home\/juser\/del\/datamash-1.9\/po&#39;<\/span>\nmake<span class=\"o\">[<\/span><span class=\"m\">2<\/span><span class=\"o\">]<\/span>:<span class=\"w\"> <\/span>Entering<span class=\"w\"> <\/span>directory<span class=\"w\"> <\/span><span class=\"s1\">&#39;\/home\/juser\/del\/datamash-1.9&#39;<\/span>\nmake<span class=\"w\">  <\/span>check-TESTS\nmake<span class=\"o\">[<\/span><span class=\"m\">3<\/span><span class=\"o\">]<\/span>:<span class=\"w\"> <\/span>Entering<span class=\"w\"> <\/span>directory<span class=\"w\"> <\/span><span class=\"s1\">&#39;\/home\/juser\/del\/datamash-1.9&#39;<\/span>\nmake<span class=\"o\">[<\/span><span class=\"m\">4<\/span><span class=\"o\">]<\/span>:<span class=\"w\"> <\/span>Entering<span class=\"w\"> <\/span>directory<span class=\"w\"> <\/span><span class=\"s1\">&#39;\/home\/juser\/del\/datamash-1.9&#39;<\/span>\nPASS:<span class=\"w\"> <\/span>tests\/datamash-show-env.sh\nPASS:<span class=\"w\"> <\/span>tests\/datamash-tests.pl\nPASS:<span class=\"w\"> <\/span>tests\/datamash-tests-deprecated.pl\nPASS:<span class=\"w\"> <\/span>tests\/datamash-tests-2.pl\n<span class=\"o\">[<\/span>..<span class=\"o\">]<\/span>\nPASS:<span class=\"w\"> <\/span>tests\/decorate-sort-tests.pl\n<span class=\"o\">============================================================================<\/span>\nTestsuite<span class=\"w\"> <\/span>summary<span class=\"w\"> <\/span><span class=\"k\">for<\/span><span class=\"w\"> <\/span>GNU<span class=\"w\"> <\/span>datamash<span class=\"w\"> <\/span><span class=\"m\">1<\/span>.9\n<span class=\"o\">============================================================================<\/span>\n<span class=\"c1\"># TOTAL: 30<\/span>\n<span class=\"c1\"># PASS:  28<\/span>\n<span class=\"c1\"># SKIP:  2<\/span>\n<span class=\"c1\"># XFAIL: 0<\/span>\n<span class=\"c1\"># FAIL:  0<\/span>\n<span class=\"c1\"># XPASS: 0<\/span>\n<span class=\"c1\"># ERROR: 0<\/span>\n<span class=\"o\">============================================================================<\/span>\nmake<span class=\"o\">[<\/span><span class=\"m\">4<\/span><span class=\"o\">]<\/span>:<span class=\"w\"> <\/span>Leaving<span class=\"w\"> <\/span>directory<span class=\"w\"> <\/span><span class=\"s1\">&#39;\/home\/juser\/del\/datamash-1.9&#39;<\/span>\nmake<span class=\"o\">[<\/span><span class=\"m\">3<\/span><span class=\"o\">]<\/span>:<span class=\"w\"> <\/span>Leaving<span class=\"w\"> <\/span>directory<span class=\"w\"> <\/span><span class=\"s1\">&#39;\/home\/juser\/del\/datamash-1.9&#39;<\/span>\nmake<span class=\"o\">[<\/span><span class=\"m\">2<\/span><span class=\"o\">]<\/span>:<span class=\"w\"> <\/span>Leaving<span class=\"w\"> <\/span>directory<span class=\"w\"> <\/span><span class=\"s1\">&#39;\/home\/juser\/del\/datamash-1.9&#39;<\/span>\nmake<span class=\"o\">[<\/span><span class=\"m\">1<\/span><span class=\"o\">]<\/span>:<span class=\"w\"> <\/span>Leaving<span class=\"w\"> <\/span>directory<span class=\"w\"> <\/span><span class=\"s1\">&#39;\/home\/juser\/del\/datamash-1.9&#39;<\/span>\n<\/code><\/pre><\/div>\n\n<p>As it's common with Autotools, there is a lot of extra noise and echoing that distracts from the important information.<\/p>\n<hr>\n<p>In comparison the Meson output is more to the point and useful:<\/p>\n<div class=\"m-code\"><pre><span><\/span><code>$<span class=\"w\"> <\/span>meson<span class=\"w\"> <\/span><span class=\"nb\">test<\/span><span class=\"w\">                                                 <\/span>\nninja:<span class=\"w\"> <\/span>Entering<span class=\"w\"> <\/span>directory<span class=\"w\"> <\/span><span class=\"sb\">`<\/span>\/home\/juser\/del\/datamash\/build<span class=\"err\">&#39;<\/span>\nninja:<span class=\"w\"> <\/span>no<span class=\"w\"> <\/span>work<span class=\"w\"> <\/span>to<span class=\"w\"> <\/span><span class=\"k\">do<\/span>.\n<span class=\"w\"> <\/span><span class=\"m\">1<\/span>\/25<span class=\"w\"> <\/span>deprecated_test<span class=\"w\">                     <\/span>OK<span class=\"w\">              <\/span><span class=\"m\">0<\/span>.08s\n<span class=\"w\"> <\/span><span class=\"m\">2<\/span>\/25<span class=\"w\"> <\/span>tests2<span class=\"w\">                              <\/span>OK<span class=\"w\">              <\/span><span class=\"m\">0<\/span>.40s\n<span class=\"w\"> <\/span><span class=\"m\">3<\/span>\/25<span class=\"w\"> <\/span>deprecated2_test<span class=\"w\">                    <\/span>OK<span class=\"w\">              <\/span><span class=\"m\">0<\/span>.07s\n<span class=\"w\"> <\/span><span class=\"m\">4<\/span>\/25<span class=\"w\"> <\/span>tests<span class=\"w\">                               <\/span>OK<span class=\"w\">              <\/span><span class=\"m\">0<\/span>.66s\n<span class=\"w\"> <\/span><span class=\"m\">5<\/span>\/25<span class=\"w\"> <\/span>parser_test<span class=\"w\">                         <\/span>OK<span class=\"w\">              <\/span><span class=\"m\">0<\/span>.20s\n<span class=\"w\"> <\/span><span class=\"m\">6<\/span>\/25<span class=\"w\"> <\/span>md5_test<span class=\"w\">                            <\/span>OK<span class=\"w\">              <\/span><span class=\"m\">0<\/span>.04s\n<span class=\"o\">[<\/span>..<span class=\"o\">]<\/span>\n<span class=\"m\">25<\/span>\/25<span class=\"w\"> <\/span>deco_sort<span class=\"w\">                           <\/span>OK<span class=\"w\">              <\/span><span class=\"m\">0<\/span>.15s\n\nOk:<span class=\"w\">                <\/span><span class=\"m\">25<\/span><span class=\"w\">  <\/span>\nFail:<span class=\"w\">              <\/span><span class=\"m\">0<\/span><span class=\"w\">   <\/span>\n\nFull<span class=\"w\"> <\/span>log<span class=\"w\"> <\/span>written<span class=\"w\"> <\/span>to<span class=\"w\"> <\/span>\/home\/juser\/del\/datamash\/build\/meson-logs\/testlog.txt\n<\/code><\/pre><\/div>\n\n<p><br\/><\/p>\n<h2 id=\"side-effects\">Side Effects<a class=\"headerlink\" href=\"#side-effects\" title=\"Permanent link\">&para;<\/a><\/h2>\n<p>As a side effect of introducing <a href=\"https:\/\/en.wikipedia.org\/wiki\/Meson_(software)\">Meson<\/a>, a few other things are now possible or much easier than with <a href=\"https:\/\/en.wikipedia.org\/wiki\/GNU_Autotools\">Autotools<\/a>.<\/p>\n<p>In particular, the first-class support of out-of-tree builds simplifies having multiple build trees around\nfor different target configurations, e.g. one with release optimization enabled, one with special debugging flags etc.<\/p>\n<p>Also, Meson supports out-of-the-box complicated build modes such as <a href=\"https:\/\/en.wikipedia.org\/wiki\/Interprocedural_optimization\">LTO<\/a> and instrumentalization via\nsimple setup flags.<\/p>\n<p>Nowadays Autotools supports out-of-tree builds, as well, but usually this isn't documented well and for each project that uses Autotools  there is a risk that it doesn't support this mode fully. Similarly, passing LTO and instrumentalization flags is also possible with Autotools, but arguably this is more tedious and error-prone.<\/p>\n<h2 id=\"see-also\">See also<a class=\"headerlink\" href=\"#see-also\" title=\"Permanent link\">&para;<\/a><\/h2>\n<p>My <a href=\"https:\/\/opencommit.eu\/gms\/datamash\/commits\/branch\/main\">Meson branch<\/a> (named <code>main<\/code>) contains all the changes\nthat are discussed in this article. My changes are the first commits in 2026 and that series contains 9 consecutive commits or so.<\/p>\n<p>Direct links:<\/p>\n<ul>\n<li><a href=\"https:\/\/opencommit.eu\/gms\/datamash\/src\/branch\/main\/meson.build\">main meson.build<\/a> file<\/li>\n<li><a href=\"https:\/\/opencommit.eu\/gms\/datamash\/src\/branch\/main\/meson.options\">meson.options<\/a> - custom options that can be set when initializing the meson build directory<\/li>\n<\/ul>\n<h2 id=\"conclusion\">Conclusion<a class=\"headerlink\" href=\"#conclusion\" title=\"Permanent link\">&para;<\/a><\/h2>\n<p>Porting <a href=\"https:\/\/www.gnu.org\/software\/datamash\/\">Datamash<\/a> from <a href=\"https:\/\/en.wikipedia.org\/wiki\/GNU_Autotools\">Autotools<\/a> to <a href=\"https:\/\/en.wikipedia.org\/wiki\/Meson_(software)\">Meson<\/a> required relatively little effort\nthat quickly pays off, since build setup is sped up over ten times, build time is sped up two times\nand the executable sizes are halved,\nwhile the Meson build configuration is simpler and more compact and thus easier to review and maintain.<\/p>\n<p>It shows that Meson already was adopted by many open source projects,\nbecause Meson support for configuration macros and tests is good.\nLast but not least, it helps that Meson is better documented than Autotools.<\/p>","category":[{"@attributes":{"term":"linux"}},{"@attributes":{"term":"meson"}},{"@attributes":{"term":"build"}},{"@attributes":{"term":"linux"}}]},{"title":"The curious case of nanosleep vs. hr_sleep","link":{"@attributes":{"href":"https:\/\/gms.tf\/hr_sleep.html","rel":"alternate"}},"published":"2024-07-09T19:00:00+02:00","updated":"2024-07-09T19:00:00+02:00","author":{"name":"Georg Sauthoff"},"id":"tag:gms.tf,2024-07-09:\/hr_sleep.html","summary":"<p>This weekend, I finally got around reading a bookmarked paper that proposes an improved nanosleep:<\/p>\n<p><a href=\"https:\/\/www.researchgate.net\/publication\/350371596_Metronome_adaptive_and_precise_intermittent_packet_retrieval_in_DPDK\">Faltelli, Marco &amp; Belocchi, Giacomo &amp; Quaglia, Francesco &amp; Pontarelli, Salvatore &amp; Bianchi, Giuseppe. (2021). Metronome: adaptive and precise intermittent packet retrieval in DPDK.<\/a><\/p>\n<p>In the following I document a few notes.<\/p>\n<h2 id=\"the-papers\">The Papers<a class=\"headerlink\" href=\"#the-papers\" title=\"Permanent link\">&para;<\/a><\/h2>\n<p>At page 4 I'm asking \u2026<\/p>","content":"<p>This weekend, I finally got around reading a bookmarked paper that proposes an improved nanosleep:<\/p>\n<p><a href=\"https:\/\/www.researchgate.net\/publication\/350371596_Metronome_adaptive_and_precise_intermittent_packet_retrieval_in_DPDK\">Faltelli, Marco &amp; Belocchi, Giacomo &amp; Quaglia, Francesco &amp; Pontarelli, Salvatore &amp; Bianchi, Giuseppe. (2021). Metronome: adaptive and precise intermittent packet retrieval in DPDK.<\/a><\/p>\n<p>In the following I document a few notes.<\/p>\n<h2 id=\"the-papers\">The Papers<a class=\"headerlink\" href=\"#the-papers\" title=\"Permanent link\">&para;<\/a><\/h2>\n<p>At page 4 I'm asking myself whether the authors ever posted their code to the LKML.\nThey did and the paper receives devastating criticism from <a href=\"https:\/\/de.wikipedia.org\/wiki\/Thomas_Gleixner\">Thomas\nGleixner<\/a>, the Linux kernel\nmaintainer of the affected subsystem and a well-known expert in realtime computing in general and high resolution timers in particular:<\/p>\n<p><a href=\"https:\/\/lore.kernel.org\/lkml\/87eefl4s7f.ffs@nanos.tec.linutronix.de\/\">Thomas Gleixner. Re: [PATCH] kernel\/time: Feedback reply for hr_sleep syscall, a fine-grained sleep service (2021-04-08)<\/a>\n(<a href=\"https:\/\/lkml.org\/lkml\/2021\/4\/7\/1614\">mirror<\/a>)<\/p>\n<p>Turns out I was reading the <a href=\"https:\/\/arxiv.org\/abs\/2103.13263v1\">first version<\/a> of a <a href=\"https:\/\/arxiv.org\/abs\/2103.13263\">preprint<\/a>\nand meanwhile <a href=\"https:\/\/arxiv.org\/pdf\/2103.13263\">version 3<\/a> was released after that LKML post.\nThat version apparently corrects a few of the serious flaws pointed out by Thomas Gleixner, but:<\/p>\n<ul>\n<li>the paper's authors don't reply on the LKML to the posted criticism<\/li>\n<li>they don't acknowledge Thomas Gleixner for his review in the acknowledgements\n  section in the new version of their paper, although they certainly profited\n  from his explanations and independent tests<\/li>\n<li>from the revised preprint (3rd version) the authors reference\n  their old and identically named <a href=\"https:\/\/dl.acm.org\/doi\/10.1145\/3386367.3432730\">conference paper<\/a> for more\n  details, although it also contains the criticized flaws that were\n  removed from the 3rd preprint revision<\/li>\n<li>the ACM conference paper page doesn't mention any corrections<\/li>\n<li>the paper's accompanying <a href=\"https:\/\/github.com\/marcofaltelli\/Metronome\">github repository<\/a> doesn't reference any of the criticism<\/li>\n<\/ul>\n<p>Also, in the 3rd preprint revision, as of 2024-07-09 the latest version, Section 3 A now has some new issues:<\/p>\n<ul>\n<li>'This factor can be controlled using the prctl() system call, putting it to the minimal value of 1.'<br\/>\n  Thus, the reader is left wondering what unit the minimal value might have ... (yes, it's nanoseconds).<\/li>\n<li>'These data have been collected by running the thread issuing\n  the sleep request as a classical SCHED_OTHER (normal) priority\n  thread and\u2014as hinted before\u2014with the timer slack of nanosleep()\n  set to 1\u00b5s.'<br\/>\n  Either they mixed up the units and they actually used the minimal value of 1 ns or they used an unnecessarily high\n  <a href=\"https:\/\/manpath.be\/f38\/2\/prctl#L1108\">timer slack<\/a> value to make their proposed improvement look a bit better.<\/li>\n<\/ul>\n<h2 id=\"the-good\">The Good<a class=\"headerlink\" href=\"#the-good\" title=\"Permanent link\">&para;<\/a><\/h2>\n<p>Scrutinizing the API of syscalls that are parametrized with\ntimespec struct is certainly a valuable contribution.<\/p>\n<p>Passing a user pointer into a syscall that needs to be\ndereferenced in the kernel (over the user space and kernel\naddress space boundary) has <em>some<\/em>\n<a href=\"https:\/\/elixir.bootlin.com\/linux\/v6.9.8\/source\/kernel\/time\/time.c#L871\">overhead<\/a>.<\/p>\n<p>Also, such an indirection makes tracing it and debugging\nsomewhat harder.<\/p>\n<p>Looking at the <a href=\"https:\/\/manpath.be\/f39\/2\/nanosleep#L14\">Linux sycall calling\nconventions<\/a>, it's clear\nthat there are more than enough call argument registers available\nto pass all of struct timespec's fields directly via registers.\nAlso, in the case of nanosleep, which optionally returns the\nremaining waiting time via a second argument pointer, at least some\narchitectures such as x86-64 would allow to return these values\nvia registers, as well. (on x86-64, syscall return values are put into\nregister rax and optionally register rdx which are both 64 bit\nwide).<\/p>\n<p>Hence, it isn't even necessary to limit the sleep time\nspecification to a single nanoseconds argument like <code>hr_sleep()<\/code> does.\nFWIW, on 64 architectures, the Linux kernel has <a href=\"https:\/\/elixir.bootlin.com\/linux\/v6.9.8\/source\/include\/linux\/ktime.h#L36\">no problem to\nconvert<\/a>\nup to <code>2^63-1<\/code> nanoseconds into its internal <code>ktime_t<\/code> and for all\npractical purposes a maximum sleep time of up to 292 years or so\nseems to be sufficient.\nAlso FWIW, when using POSIX API such as <code>nanosleep()<\/code> the valid\nrange of <code>timespec::tv_nsec<\/code> is 'just' <code>[0, 999999999]<\/code>.<\/p>\n<p>However, measuring the likely tiny overhead of copying a timespec\nstruct from user to kernel space requires careful and rigorous\nwork.<\/p>\n<p>Of course, it's highly questionable whether eliminating the\nexpected timespec user to kernel copy overhead really would\njustify adding another syscall.<\/p>\n<h2 id=\"addendum\">Addendum<a class=\"headerlink\" href=\"#addendum\" title=\"Permanent link\">&para;<\/a><\/h2>\n<p>Apparently, the <code>hr_sleep()<\/code> authors <a href=\"https:\/\/lore.kernel.org\/lkml\/20210115180733.5663-1-marco.faltelli@uniroma2.it\/\">initially\ncontacted<\/a> the\nkernel community only after their <a href=\"https:\/\/dl.acm.org\/doi\/10.1145\/3386367.3432730\">conference paper<\/a> was published.\n(i.e. first LKML post on 2021-01-15, conference paper was\npublished 2020-11-24 and presented at the conference in the first\nweek of December, 2020)\nAt that time it received some\n<a href=\"https:\/\/lore.kernel.org\/lkml\/CALCETrWfnL=3M3nmmHs-a3si5JptSCtF6cEtHVtsDNwA5mHnRg@mail.gmail.com\/\">feedback<\/a> from another kernel\ndeveloper, Andy Lutomirski. Like with the second posting, the\nhr_sleep authors didn't bother to reply directly to any of the\nissues mentioned in the review.<\/p>\n<p>It seems that the paper was submitted to a proper conference and\nhence it was peer reviewed, but perhaps the academic reviewers\nweren't sufficiently familiar with the relevant parts of the\nLinux kernel and thus missed the issues that were discussed on\nthe LKML.<\/p>\n<hr>\n<p>On 2024-07-07, I <a href=\"https:\/\/github.com\/marcofaltelli\/Metronome\/issues\/5\">opened an issue<\/a> (<a href=\"https:\/\/web.archive.org\/web\/20240707222020\/https:\/\/github.com\/marcofaltelli\/Metronome\/issues\/5\">archive<\/a>) on the hr_sleep's github\n<a href=\"https:\/\/github.com\/marcofaltelli\/Metronome\">repository<\/a>, asking for clarification.<\/p>\n<p>It was deleted quickly without any comment:<\/p>\n<blockquote>\n<p>This issue has been deleted.<\/p>\n<\/blockquote>\n<p>For the sake of completeness I'm reproducing my github post below:<\/p>\n<blockquote>\n<p>Clarifications nanosleep vs. hr_sleep measurements #5<\/p>\n<p>Reading through Section 3 A of your <a href=\"https:\/\/arxiv.org\/abs\/2103.13263\">preprint (3rd revision)<\/a> I noticed a few issues:<\/p>\n<ol>\n<li>'This factor can be controlled using the prctl() system call, putting it to the minimal value of 1.'\n    Perhaps you want to add the unit of that minimal value.<\/li>\n<li>'These data have been collected by running the thread issuing the sleep request as a classical SCHED_OTHER (normal) priority thread and\u2014as hinted before\u2014with the timer slack of nanosleep() set to 1\u00b5s.'\n    Please clarify, did you really set the timer slack to 1 \u00b5s? Or did you set it to its minimal value, i.e. 1 <em>ns<\/em>?<\/li>\n<li>'We remand the reader to [14] for an extended evaluation of this implementation.In [..]'\n    Firstly, there is a space missing after the full stop.\n    Secondly,  why do you continue to refer to an evaluation whose <a href=\"https:\/\/lkml.org\/lkml\/2021\/4\/7\/1614\">flaws were pointed out to you by the maintainer of the relevant Linux subsystems<\/a> while apparently you tried to address some of them in that 3rd version of your preprint?<\/li>\n<li>'Figure 1'\n    You don't mention any details regarding the system you measured those latencies on.\n    Thus, the results are hard to reproduce.\n    The text states 'The tests have been conducted on an isolated NUMA node equipped with Intel Xeon Silver 2.1 GHz cores. The server is running Linux kernel 5.4.'\n    But why don't you mention the exact CPU model and exact kernel version?\n    Also relevant but completely missing: whether relevant kernel parameters where supplied, such as turning some\/all kernel CPU bug mitigations off,  whether Hyperthreading was enabled, whether frequency scaling, energy saving modes, turbo boost, etc. etc. where enabled ...<\/li>\n<\/ol>\n<p>BTW, google searches still return the <a href=\"https:\/\/www.researchgate.net\/publication\/350371596_Metronome_adaptive_and_precise_intermittent_packet_retrieval_in_DPDK\">ResearchGate page<\/a> for the first preprint version at relatively high rank for me - perhaps you want to remove it from ResearchGate to avoid further confusion.<\/p>\n<p>Also, the <a href=\"https:\/\/dl.acm.org\/doi\/10.1145\/3386367.3432730\">ACM page<\/a> of the conference paper version doesn't contain any hints regarding regarding published corrections.<\/p>\n<p>Similarly, in the README of this repository you still link to your conference paper without mentioning that the analysis of hr_sleep vs. nanosleep therein is deeply flawed.<\/p>\n<p>Finally, since you profited from the LKML review of your paper and apparently that review prompted a significant rework of Section 3 A in the 3rd revision of your preprint, why didn't you acknowledge that Linux kernel maintainer in the Acknowledgement Section in your 3rd revision?<\/p>\n<\/blockquote>","category":[{"@attributes":{"term":"linux"}},{"@attributes":{"term":"kernel"}},{"@attributes":{"term":"time"}},{"@attributes":{"term":"dpdk"}}]},{"title":"On the Costs of Syscalls","link":{"@attributes":{"href":"https:\/\/gms.tf\/on-the-costs-of-syscalls.html","rel":"alternate"}},"published":"2021-08-30T15:00:00+02:00","updated":"2021-08-30T15:00:00+02:00","author":{"name":"Georg Sauthoff"},"id":"tag:gms.tf,2021-08-30:\/on-the-costs-of-syscalls.html","summary":"<p>It's well known that <a href=\"https:\/\/en.wikipedia.org\/wiki\/System_call\">syscalls<\/a> are expensive. And that\nsoftware mitigations against <a href=\"https:\/\/en.wikipedia.org\/wiki\/Transient_execution_CPU_vulnerability\">CPU bugs<\/a> (such as Meltdown) even\nhave made them more expensive. But how expensive are they really?\nTo begin to answer this question I wrote a small\n<a href=\"https:\/\/github.com\/gsauthof\/osjitter\/blob\/master\/bench_syscalls.cc\">micro-benchmark<\/a> in order to measure the minimal costs of a\nsyscall \u2026<\/p>","content":"<p>It's well known that <a href=\"https:\/\/en.wikipedia.org\/wiki\/System_call\">syscalls<\/a> are expensive. And that\nsoftware mitigations against <a href=\"https:\/\/en.wikipedia.org\/wiki\/Transient_execution_CPU_vulnerability\">CPU bugs<\/a> (such as Meltdown) even\nhave made them more expensive. But how expensive are they really?\nTo begin to answer this question I wrote a small\n<a href=\"https:\/\/github.com\/gsauthof\/osjitter\/blob\/master\/bench_syscalls.cc\">micro-benchmark<\/a> in order to measure the minimal costs of a\nsyscall. Meaning the cost of syscalls one always has to pay\nwhether a context-switch happens or not, even when the work in\nthe kernel is minuscule,  i.e. the costs of switching from\n<a href=\"https:\/\/en.wikipedia.org\/wiki\/Context_switch#User_and_kernel_mode_switching\">user-mode to kernel-mode<\/a> and back.<\/p>\n<h2 id=\"methods\">Methods<a class=\"headerlink\" href=\"#methods\" title=\"Permanent link\">&para;<\/a><\/h2>\n<p>The user-kernel mode-switch <a href=\"https:\/\/en.wikipedia.org\/wiki\/Benchmark_(computing)#Types_of_benchmark\">micro-benchmark<\/a> uses Google's\n<a href=\"https:\/\/github.com\/google\/benchmark\">benchmark library<\/a> for the measurements and is <a href=\"https:\/\/github.com\/gsauthof\/osjitter\/blob\/master\/bench_syscalls.cc\">available in\na git repository<\/a>. The repository also contains some helper\nscripts, e.g. a <a href=\"https:\/\/github.com\/gsauthof\/osjitter\/blob\/master\/helper\/bench_playbook.py\">playbook<\/a> for distributing it to and executing\nit on a bunch of hosts. The benchmark library repeats each case\nuntil the result is considered stable and the playbook allows for\nrepeated executions of the test cases. In the following sections\nthe median value of 100 repetitions is reported (real time in\nnanoseconds).<\/p>\n<p>For the benchmark a bunch of syscalls is called that are expected\nto be very cheap, such as getting the user id (UID), the program id\n(PID), closing an invalid file descriptor, calling an\nnon-existent syscall etc. Thus, a measurement should really\njust include two mode switches. As controls, a few cases don't call\na syscall but do other cheap stuff.<\/p>\n<p>I ran the benchmark on a heterogeneous set of hosts, i.e.\ndifferent kernels, operating systems and configurations. For more\ndetails see also the Hosts Section.<\/p>\n<h2 id=\"results-and-discussion\">Results and Discussion<a class=\"headerlink\" href=\"#results-and-discussion\" title=\"Permanent link\">&para;<\/a><\/h2>\n<p>The following table shows the real time (ns) for the different\ncases:<\/p>\n<style  type=\"text\/css\" >\n    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072 {\n            margin-bottom: 1em\n        }\n<\/style>\n<style  type=\"text\/css\" >\n    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row0_col0 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row0_col1 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row0_col2 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row0_col3 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row0_col4 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row0_col5 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row0_col6 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row0_col7 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row0_col8 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row0_col9 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row0_col10 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row0_col11 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row0_col12 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row0_col13 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row0_col14 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row1_col0 {\n            background-color:  #ead4c8;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row1_col1 {\n            background-color:  #cdd9ec;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row1_col2 {\n            background-color:  #b40426;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row1_col3 {\n            background-color:  #aac7fd;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row1_col4 {\n            background-color:  #6f92f3;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row1_col5 {\n            background-color:  #6f92f3;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row1_col6 {\n            background-color:  #82a6fb;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row1_col7 {\n            background-color:  #4b64d5;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row1_col8 {\n            background-color:  #cdd9ec;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row1_col9 {\n            background-color:  #4b64d5;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row1_col10 {\n            background-color:  #f3c8b2;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row1_col11 {\n            background-color:  #4b64d5;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row1_col12 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row1_col13 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row1_col14 {\n            background-color:  #f3c8b2;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row2_col0 {\n            background-color:  #e3d9d3;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row2_col1 {\n            background-color:  #d6dce4;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row2_col2 {\n            background-color:  #b40426;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row2_col3 {\n            background-color:  #6c8ff1;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row2_col4 {\n            background-color:  #6c8ff1;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row2_col5 {\n            background-color:  #6c8ff1;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row2_col6 {\n            background-color:  #7ea1fa;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row2_col7 {\n            background-color:  #4a63d3;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row2_col8 {\n            background-color:  #c6d6f1;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row2_col9 {\n            background-color:  #5a78e4;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row2_col10 {\n            background-color:  #efcfbf;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row2_col11 {\n            background-color:  #4a63d3;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row2_col12 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row2_col13 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row2_col14 {\n            background-color:  #efcfbf;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row3_col0 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row3_col1 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row3_col2 {\n            background-color:  #3e51c5;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row3_col3 {\n            background-color:  #f7ac8e;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row3_col4 {\n            background-color:  #f7aa8c;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row3_col5 {\n            background-color:  #cdd9ec;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row3_col6 {\n            background-color:  #f39778;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row3_col7 {\n            background-color:  #c6d6f1;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row3_col8 {\n            background-color:  #b40426;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row3_col9 {\n            background-color:  #e16751;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row3_col10 {\n            background-color:  #e9d5cb;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row3_col11 {\n            background-color:  #adc9fd;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row3_col12 {\n            background-color:  #6485ec;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row3_col13 {\n            background-color:  #b3cdfb;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row3_col14 {\n            background-color:  #92b4fe;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row4_col0 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row4_col1 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row4_col2 {\n            background-color:  #3e51c5;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row4_col3 {\n            background-color:  #f7ac8e;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row4_col4 {\n            background-color:  #f7a98b;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row4_col5 {\n            background-color:  #cedaeb;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row4_col6 {\n            background-color:  #f39475;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row4_col7 {\n            background-color:  #c6d6f1;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row4_col8 {\n            background-color:  #b40426;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row4_col9 {\n            background-color:  #e16751;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row4_col10 {\n            background-color:  #e9d5cb;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row4_col11 {\n            background-color:  #adc9fd;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row4_col12 {\n            background-color:  #6485ec;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row4_col13 {\n            background-color:  #b5cdfa;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row4_col14 {\n            background-color:  #93b5fe;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row5_col0 {\n            background-color:  #e9785d;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row5_col1 {\n            background-color:  #9dbdff;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row5_col2 {\n            background-color:  #a5c3fe;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row5_col3 {\n            background-color:  #f7b396;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row5_col4 {\n            background-color:  #f7ad90;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row5_col5 {\n            background-color:  #a9c6fd;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row5_col6 {\n            background-color:  #f5a081;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row5_col7 {\n            background-color:  #a5c3fe;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row5_col8 {\n            background-color:  #b40426;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row5_col9 {\n            background-color:  #d65244;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row5_col10 {\n            background-color:  #d2dbe8;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row5_col11 {\n            background-color:  #90b2fe;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row5_col12 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row5_col13 {\n            background-color:  #9abbff;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row5_col14 {\n            background-color:  #5673e0;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row6_col0 {\n            background-color:  #b40426;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row6_col1 {\n            background-color:  #d2dbe8;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row6_col2 {\n            background-color:  #d1dae9;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row6_col3 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row6_col4 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row6_col5 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row6_col6 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row6_col7 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row6_col8 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row6_col9 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row6_col10 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row6_col11 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row6_col12 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row6_col13 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row6_col14 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row7_col0 {\n            background-color:  #e46e56;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row7_col1 {\n            background-color:  #93b5fe;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row7_col2 {\n            background-color:  #a5c3fe;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row7_col3 {\n            background-color:  #f7b497;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row7_col4 {\n            background-color:  #f7af91;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row7_col5 {\n            background-color:  #aec9fc;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row7_col6 {\n            background-color:  #f49a7b;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row7_col7 {\n            background-color:  #a5c3fe;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row7_col8 {\n            background-color:  #b40426;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row7_col9 {\n            background-color:  #d65244;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row7_col10 {\n            background-color:  #d5dbe5;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row7_col11 {\n            background-color:  #8fb1fe;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row7_col12 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row7_col13 {\n            background-color:  #98b9ff;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row7_col14 {\n            background-color:  #5a78e4;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row8_col0 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row8_col1 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row8_col2 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row8_col3 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row8_col4 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row8_col5 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row8_col6 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row8_col7 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row8_col8 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row8_col9 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row8_col10 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row8_col11 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row8_col12 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row8_col13 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row8_col14 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row9_col0 {\n            background-color:  #4f69d9;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row9_col1 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row9_col2 {\n            background-color:  #93b5fe;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row9_col3 {\n            background-color:  #ee8468;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row9_col4 {\n            background-color:  #f5a081;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row9_col5 {\n            background-color:  #f7b89c;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row9_col6 {\n            background-color:  #ee8468;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row9_col7 {\n            background-color:  #f2cbb7;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row9_col8 {\n            background-color:  #e0654f;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row9_col9 {\n            background-color:  #e5d8d1;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row9_col10 {\n            background-color:  #b40426;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row9_col11 {\n            background-color:  #e5d8d1;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row9_col12 {\n            background-color:  #e5d8d1;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row9_col13 {\n            background-color:  #e5d8d1;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row9_col14 {\n            background-color:  #b40426;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row10_col0 {\n            background-color:  #d75445;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row10_col1 {\n            background-color:  #bbd1f8;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row10_col2 {\n            background-color:  #d5dbe5;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row10_col3 {\n            background-color:  #f7b194;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row10_col4 {\n            background-color:  #f7b79b;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row10_col5 {\n            background-color:  #cdd9ec;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row10_col6 {\n            background-color:  #f08a6c;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row10_col7 {\n            background-color:  #adc9fd;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row10_col8 {\n            background-color:  #b40426;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row10_col9 {\n            background-color:  #dd5f4b;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row10_col10 {\n            background-color:  #dedcdb;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row10_col11 {\n            background-color:  #88abfd;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row10_col12 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row10_col13 {\n            background-color:  #93b5fe;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row10_col14 {\n            background-color:  #6e90f2;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row11_col0 {\n            background-color:  #b2ccfb;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row11_col1 {\n            background-color:  #506bda;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row11_col2 {\n            background-color:  #b40426;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row11_col3 {\n            background-color:  #80a3fa;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row11_col4 {\n            background-color:  #80a3fa;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row11_col5 {\n            background-color:  #80a3fa;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row11_col6 {\n            background-color:  #80a3fa;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row11_col7 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row11_col8 {\n            background-color:  #c9d7f0;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row11_col9 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row11_col10 {\n            background-color:  #6788ee;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row11_col11 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row11_col12 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row11_col13 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row11_col14 {\n            background-color:  #6788ee;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row12_col0 {\n            background-color:  #6b8df0;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row12_col1 {\n            background-color:  #6b8df0;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row12_col2 {\n            background-color:  #b40426;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row12_col3 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row12_col4 {\n            background-color:  #516ddb;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row12_col5 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row12_col6 {\n            background-color:  #516ddb;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row12_col7 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row12_col8 {\n            background-color:  #516ddb;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row12_col9 {\n            background-color:  #516ddb;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row12_col10 {\n            background-color:  #86a9fc;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row12_col11 {\n            background-color:  #516ddb;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row12_col12 {\n            background-color:  #516ddb;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row12_col13 {\n            background-color:  #516ddb;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row12_col14 {\n            background-color:  #86a9fc;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row13_col0 {\n            background-color:  #d95847;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row13_col1 {\n            background-color:  #a6c4fe;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row13_col2 {\n            background-color:  #aec9fc;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row13_col3 {\n            background-color:  #f7b79b;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row13_col4 {\n            background-color:  #f7aa8c;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row13_col5 {\n            background-color:  #b1cbfc;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row13_col6 {\n            background-color:  #f18f71;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row13_col7 {\n            background-color:  #a1c0ff;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row13_col8 {\n            background-color:  #b40426;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row13_col9 {\n            background-color:  #cf453c;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row13_col10 {\n            background-color:  #dbdcde;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row13_col11 {\n            background-color:  #93b5fe;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row13_col12 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row13_col13 {\n            background-color:  #9abbff;\n            color:  #000000;\n        }    #T_dde6e7fa_0989_11ec_9e32_f8cab850d072row13_col14 {\n            background-color:  #5d7ce6;\n            color:  #000000;\n        }<\/style>\n<table id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072\" ><thead>    <tr>        <th class=\"index_name level0\" >host<\/th>        <th class=\"col_heading level0 col0\" >5i4250u<\/th>        <th class=\"col_heading level0 col1\" >7i6600u<\/th>        <th class=\"col_heading level0 col2\" >ac3758<\/th>        <th class=\"col_heading level0 col3\" >x2643<\/th>        <th class=\"col_heading level0 col4\" >x2667h<\/th>        <th class=\"col_heading level0 col5\" >x2667s<\/th>        <th class=\"col_heading level0 col6\" >x2687w<\/th>        <th class=\"col_heading level0 col7\" >x2689<\/th>        <th class=\"col_heading level0 col8\" >x2690<\/th>        <th class=\"col_heading level0 col9\" >xg6144<\/th>        <th class=\"col_heading level0 col10\" >xg6148<\/th>        <th class=\"col_heading level0 col11\" >xg6246<\/th>        <th class=\"col_heading level0 col12\" >xg6256<\/th>        <th class=\"col_heading level0 col13\" >xg6256b<\/th>        <th class=\"col_heading level0 col14\" >xs4110<\/th>    <\/tr>    <tr>        <th class=\"index_name level0\" >name<\/th>        <th class=\"blank\" ><\/th>        <th class=\"blank\" ><\/th>        <th class=\"blank\" ><\/th>        <th class=\"blank\" ><\/th>        <th class=\"blank\" ><\/th>        <th class=\"blank\" ><\/th>        <th class=\"blank\" ><\/th>        <th class=\"blank\" ><\/th>        <th class=\"blank\" ><\/th>        <th class=\"blank\" ><\/th>        <th class=\"blank\" ><\/th>        <th class=\"blank\" ><\/th>        <th class=\"blank\" ><\/th>        <th class=\"blank\" ><\/th>        <th class=\"blank\" ><\/th>    <\/tr><\/thead><tbody>\n                <tr>\n                        <th id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072level0_row0\" class=\"row_heading level0 row0\" >assign<\/th>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row0_col0\" class=\"data row0 col0\" >0<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row0_col1\" class=\"data row0 col1\" >0<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row0_col2\" class=\"data row0 col2\" >0<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row0_col3\" class=\"data row0 col3\" >0<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row0_col4\" class=\"data row0 col4\" >0<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row0_col5\" class=\"data row0 col5\" >0<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row0_col6\" class=\"data row0 col6\" >0<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row0_col7\" class=\"data row0 col7\" >0<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row0_col8\" class=\"data row0 col8\" >0<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row0_col9\" class=\"data row0 col9\" >0<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row0_col10\" class=\"data row0 col10\" >0<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row0_col11\" class=\"data row0 col11\" >0<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row0_col12\" class=\"data row0 col12\" >0<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row0_col13\" class=\"data row0 col13\" >0<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row0_col14\" class=\"data row0 col14\" >0<\/td>\n            <\/tr>\n            <tr>\n                        <th id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072level0_row1\" class=\"row_heading level0 row1\" >clock_gettime<\/th>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row1_col0\" class=\"data row1 col0\" >23<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row1_col1\" class=\"data row1 col1\" >21<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row1_col2\" class=\"data row1 col2\" >31<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row1_col3\" class=\"data row1 col3\" >19<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row1_col4\" class=\"data row1 col4\" >16<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row1_col5\" class=\"data row1 col5\" >16<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row1_col6\" class=\"data row1 col6\" >17<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row1_col7\" class=\"data row1 col7\" >14<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row1_col8\" class=\"data row1 col8\" >21<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row1_col9\" class=\"data row1 col9\" >14<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row1_col10\" class=\"data row1 col10\" >24<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row1_col11\" class=\"data row1 col11\" >14<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row1_col12\" class=\"data row1 col12\" >13<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row1_col13\" class=\"data row1 col13\" >13<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row1_col14\" class=\"data row1 col14\" >24<\/td>\n            <\/tr>\n            <tr>\n                        <th id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072level0_row2\" class=\"row_heading level0 row2\" >clock_gettime_mono<\/th>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row2_col0\" class=\"data row2 col0\" >23<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row2_col1\" class=\"data row2 col1\" >22<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row2_col2\" class=\"data row2 col2\" >32<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row2_col3\" class=\"data row2 col3\" >16<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row2_col4\" class=\"data row2 col4\" >16<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row2_col5\" class=\"data row2 col5\" >16<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row2_col6\" class=\"data row2 col6\" >17<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row2_col7\" class=\"data row2 col7\" >14<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row2_col8\" class=\"data row2 col8\" >21<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row2_col9\" class=\"data row2 col9\" >15<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row2_col10\" class=\"data row2 col10\" >24<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row2_col11\" class=\"data row2 col11\" >14<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row2_col12\" class=\"data row2 col12\" >13<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row2_col13\" class=\"data row2 col13\" >13<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row2_col14\" class=\"data row2 col14\" >24<\/td>\n            <\/tr>\n            <tr>\n                        <th id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072level0_row3\" class=\"row_heading level0 row3\" >clock_gettime_mono_raw<\/th>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row3_col0\" class=\"data row3 col0\" >23<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row3_col1\" class=\"data row3 col1\" >22<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row3_col2\" class=\"data row3 col2\" >33<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row3_col3\" class=\"data row3 col3\" >542<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row3_col4\" class=\"data row3 col4\" >544<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row3_col5\" class=\"data row3 col5\" >350<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row3_col6\" class=\"data row3 col6\" >582<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row3_col7\" class=\"data row3 col7\" >332<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row3_col8\" class=\"data row3 col8\" >762<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row3_col9\" class=\"data row3 col9\" >660<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row3_col10\" class=\"data row3 col10\" >427<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row3_col11\" class=\"data row3 col11\" >274<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row3_col12\" class=\"data row3 col12\" >122<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row3_col13\" class=\"data row3 col13\" >290<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row3_col14\" class=\"data row3 col14\" >218<\/td>\n            <\/tr>\n            <tr>\n                        <th id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072level0_row4\" class=\"row_heading level0 row4\" >clock_gettime_tai<\/th>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row4_col0\" class=\"data row4 col0\" >23<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row4_col1\" class=\"data row4 col1\" >21<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row4_col2\" class=\"data row4 col2\" >32<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row4_col3\" class=\"data row4 col3\" >542<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row4_col4\" class=\"data row4 col4\" >546<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row4_col5\" class=\"data row4 col5\" >352<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row4_col6\" class=\"data row4 col6\" >587<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row4_col7\" class=\"data row4 col7\" >333<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row4_col8\" class=\"data row4 col8\" >762<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row4_col9\" class=\"data row4 col9\" >660<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row4_col10\" class=\"data row4 col10\" >427<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row4_col11\" class=\"data row4 col11\" >274<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row4_col12\" class=\"data row4 col12\" >122<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row4_col13\" class=\"data row4 col13\" >292<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row4_col14\" class=\"data row4 col14\" >218<\/td>\n            <\/tr>\n            <tr>\n                        <th id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072level0_row5\" class=\"row_heading level0 row5\" >close<\/th>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row5_col0\" class=\"data row5 col0\" >568<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row5_col1\" class=\"data row5 col1\" >262<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row5_col2\" class=\"data row5 col2\" >275<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row5_col3\" class=\"data row5 col3\" >484<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row5_col4\" class=\"data row5 col4\" >495<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row5_col5\" class=\"data row5 col5\" >283<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row5_col6\" class=\"data row5 col6\" >514<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row5_col7\" class=\"data row5 col7\" >277<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row5_col8\" class=\"data row5 col8\" >668<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row5_col9\" class=\"data row5 col9\" >610<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row5_col10\" class=\"data row5 col10\" >356<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row5_col11\" class=\"data row5 col11\" >243<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row5_col12\" class=\"data row5 col12\" >93<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row5_col13\" class=\"data row5 col13\" >257<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row5_col14\" class=\"data row5 col14\" >145<\/td>\n            <\/tr>\n            <tr>\n                        <th id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072level0_row6\" class=\"row_heading level0 row6\" >getpid<\/th>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row6_col0\" class=\"data row6 col0\" >558<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row6_col1\" class=\"data row6 col1\" >257<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row6_col2\" class=\"data row6 col2\" >255<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row6_col3\" class=\"data row6 col3\" >2<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row6_col4\" class=\"data row6 col4\" >1<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row6_col5\" class=\"data row6 col5\" >1<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row6_col6\" class=\"data row6 col6\" >2<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row6_col7\" class=\"data row6 col7\" >1<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row6_col8\" class=\"data row6 col8\" >2<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row6_col9\" class=\"data row6 col9\" >1<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row6_col10\" class=\"data row6 col10\" >2<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row6_col11\" class=\"data row6 col11\" >1<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row6_col12\" class=\"data row6 col12\" >1<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row6_col13\" class=\"data row6 col13\" >1<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row6_col14\" class=\"data row6 col14\" >2<\/td>\n            <\/tr>\n            <tr>\n                        <th id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072level0_row7\" class=\"row_heading level0 row7\" >getuid<\/th>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row7_col0\" class=\"data row7 col0\" >560<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row7_col1\" class=\"data row7 col1\" >231<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row7_col2\" class=\"data row7 col2\" >259<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row7_col3\" class=\"data row7 col3\" >464<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row7_col4\" class=\"data row7 col4\" >473<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row7_col5\" class=\"data row7 col5\" >276<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row7_col6\" class=\"data row7 col6\" >505<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row7_col7\" class=\"data row7 col7\" >259<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row7_col8\" class=\"data row7 col8\" >649<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row7_col9\" class=\"data row7 col9\" >592<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row7_col10\" class=\"data row7 col10\" >347<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row7_col11\" class=\"data row7 col11\" >224<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row7_col12\" class=\"data row7 col12\" >78<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row7_col13\" class=\"data row7 col13\" >239<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row7_col14\" class=\"data row7 col14\" >137<\/td>\n            <\/tr>\n            <tr>\n                        <th id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072level0_row8\" class=\"row_heading level0 row8\" >nothing<\/th>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row8_col0\" class=\"data row8 col0\" >0<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row8_col1\" class=\"data row8 col1\" >0<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row8_col2\" class=\"data row8 col2\" >0<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row8_col3\" class=\"data row8 col3\" >0<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row8_col4\" class=\"data row8 col4\" >0<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row8_col5\" class=\"data row8 col5\" >0<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row8_col6\" class=\"data row8 col6\" >0<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row8_col7\" class=\"data row8 col7\" >0<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row8_col8\" class=\"data row8 col8\" >0<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row8_col9\" class=\"data row8 col9\" >0<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row8_col10\" class=\"data row8 col10\" >0<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row8_col11\" class=\"data row8 col11\" >0<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row8_col12\" class=\"data row8 col12\" >0<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row8_col13\" class=\"data row8 col13\" >0<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row8_col14\" class=\"data row8 col14\" >0<\/td>\n            <\/tr>\n            <tr>\n                        <th id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072level0_row9\" class=\"row_heading level0 row9\" >pthread_cond_signal<\/th>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row9_col0\" class=\"data row9 col0\" >3<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row9_col1\" class=\"data row9 col1\" >2<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row9_col2\" class=\"data row9 col2\" >6<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row9_col3\" class=\"data row9 col3\" >14<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row9_col4\" class=\"data row9 col4\" >13<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row9_col5\" class=\"data row9 col5\" >12<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row9_col6\" class=\"data row9 col6\" >14<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row9_col7\" class=\"data row9 col7\" >11<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row9_col8\" class=\"data row9 col8\" >15<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row9_col9\" class=\"data row9 col9\" >10<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row9_col10\" class=\"data row9 col10\" >17<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row9_col11\" class=\"data row9 col11\" >10<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row9_col12\" class=\"data row9 col12\" >10<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row9_col13\" class=\"data row9 col13\" >10<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row9_col14\" class=\"data row9 col14\" >17<\/td>\n            <\/tr>\n            <tr>\n                        <th id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072level0_row10\" class=\"row_heading level0 row10\" >sched_yield<\/th>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row10_col0\" class=\"data row10 col0\" >706<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row10_col1\" class=\"data row10 col1\" >374<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row10_col2\" class=\"data row10 col2\" >430<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row10_col3\" class=\"data row10 col3\" >569<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row10_col4\" class=\"data row10 col4\" >560<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row10_col5\" class=\"data row10 col5\" >414<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row10_col6\" class=\"data row10 col6\" >634<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row10_col7\" class=\"data row10 col7\" >346<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row10_col8\" class=\"data row10 col8\" >773<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row10_col9\" class=\"data row10 col9\" >694<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row10_col10\" class=\"data row10 col10\" >454<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row10_col11\" class=\"data row10 col11\" >280<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row10_col12\" class=\"data row10 col12\" >126<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row10_col13\" class=\"data row10 col13\" >300<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row10_col14\" class=\"data row10 col14\" >232<\/td>\n            <\/tr>\n            <tr>\n                        <th id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072level0_row11\" class=\"row_heading level0 row11\" >sqrt<\/th>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row11_col0\" class=\"data row11 col0\" >6<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row11_col1\" class=\"data row11 col1\" >2<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row11_col2\" class=\"data row11 col2\" >15<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row11_col3\" class=\"data row11 col3\" >4<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row11_col4\" class=\"data row11 col4\" >4<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row11_col5\" class=\"data row11 col5\" >4<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row11_col6\" class=\"data row11 col6\" >4<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row11_col7\" class=\"data row11 col7\" >1<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row11_col8\" class=\"data row11 col8\" >7<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row11_col9\" class=\"data row11 col9\" >1<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row11_col10\" class=\"data row11 col10\" >3<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row11_col11\" class=\"data row11 col11\" >1<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row11_col12\" class=\"data row11 col12\" >1<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row11_col13\" class=\"data row11 col13\" >1<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row11_col14\" class=\"data row11 col14\" >3<\/td>\n            <\/tr>\n            <tr>\n                        <th id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072level0_row12\" class=\"row_heading level0 row12\" >sqrtrec<\/th>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row12_col0\" class=\"data row12 col0\" >4<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row12_col1\" class=\"data row12 col1\" >4<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row12_col2\" class=\"data row12 col2\" >15<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row12_col3\" class=\"data row12 col3\" >2<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row12_col4\" class=\"data row12 col4\" >3<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row12_col5\" class=\"data row12 col5\" >2<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row12_col6\" class=\"data row12 col6\" >3<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row12_col7\" class=\"data row12 col7\" >2<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row12_col8\" class=\"data row12 col8\" >3<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row12_col9\" class=\"data row12 col9\" >3<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row12_col10\" class=\"data row12 col10\" >5<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row12_col11\" class=\"data row12 col11\" >3<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row12_col12\" class=\"data row12 col12\" >3<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row12_col13\" class=\"data row12 col13\" >3<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row12_col14\" class=\"data row12 col14\" >5<\/td>\n            <\/tr>\n            <tr>\n                        <th id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072level0_row13\" class=\"row_heading level0 row13\" >syscall<\/th>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row13_col0\" class=\"data row13 col0\" >560<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row13_col1\" class=\"data row13 col1\" >252<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row13_col2\" class=\"data row13 col2\" >265<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row13_col3\" class=\"data row13 col3\" >440<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row13_col4\" class=\"data row13 col4\" >460<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row13_col5\" class=\"data row13 col5\" >269<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row13_col6\" class=\"data row13 col6\" >497<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row13_col7\" class=\"data row13 col7\" >243<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row13_col8\" class=\"data row13 col8\" >620<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row13_col9\" class=\"data row13 col9\" >579<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row13_col10\" class=\"data row13 col10\" >345<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row13_col11\" class=\"data row13 col11\" >221<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row13_col12\" class=\"data row13 col12\" >76<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row13_col13\" class=\"data row13 col13\" >233<\/td>\n                        <td id=\"T_dde6e7fa_0989_11ec_9e32_f8cab850d072row13_col14\" class=\"data row13 col14\" >136<\/td>\n            <\/tr>\n    <\/tbody><\/table>\n\n<h3 id=\"controls\">Controls<a class=\"headerlink\" href=\"#controls\" title=\"Permanent link\">&para;<\/a><\/h3>\n<p>The cases used as controls are 'nothing' which literally does\nnothing, 'assign' which just assigns to a variable, <code>sqrt<\/code> which\ncomputes the square root of a small constant and <code>sqrtrec<\/code> which\nstacks a bunch of sqrt calls. The results for these are\nplausible, i.e. doing nothing is really measured as <code>10**-7<\/code> ns or\nso, the assignment costs 0.5 ns or so and computing the square\nroot takes only a few ns. Perhaps the most remarkable result is,\nthat computing the square root on a Atom CPU (ac3758) is pretty\nconstant over the 2 cases, whereas on the other hosts its runtime\ndepends on its argument.<\/p>\n<h3 id=\"clock-gettime\">Clock Gettime<a class=\"headerlink\" href=\"#clock-gettime\" title=\"Permanent link\">&para;<\/a><\/h3>\n<p>Looking at the syscalls, one relation that holds true on all\nhosts is that the <code>clock_gettime(CLOCK_REALTIME)<\/code> syscall is much\nfaster than <code>getuid()<\/code> or <code>close()<\/code>. This can be explained by the\nfact that on Linux, <code>clock_gettime(CLOCK_REALTIME)<\/code> and a <a href=\"https:\/\/manpath.be\/f34\/7\/vdso#L334\">few\nother syscalls<\/a> are implemented via the efficient <a href=\"https:\/\/en.wikipedia.org\/wiki\/VDSO\">vDSO<\/a> mechanism.\nMeaning when they are called no mode switch happens!<\/p>\n<p><code>clock_gettime()<\/code> supports different clocks and not all of them\nare vDSO optimized on all kernels. The table shows that on RHEL 7 querying\n<code>CLOCK_MONOTIC_RAW<\/code> and <code>CLOCK_TAI<\/code> invokes a real syscall while\non Fedora 33 kernels (5.12\/5.13) these clock readings are also\nimplemented as vDSO.<\/p>\n<h3 id=\"dummy-signaling\">Dummy Signaling<a class=\"headerlink\" href=\"#dummy-signaling\" title=\"Permanent link\">&para;<\/a><\/h3>\n<p>Similarly, the dummy <code>pthread_cond_signal()<\/code> case, which signals\nwithout anybody listening, is much cheaper than a real syscall - \nsince the C library doesn't have to call a real syscall but can\njust invoke some relatively cheap atomic operation.<\/p>\n<h3 id=\"getpid\">Getpid<a class=\"headerlink\" href=\"#getpid\" title=\"Permanent link\">&para;<\/a><\/h3>\n<p>The <code>getpid()<\/code> syscall is surprisingly fast on RHEL 7. It turns\nout that RHEL 7 ships an <a href=\"https:\/\/manpath.be\/f34\/2\/getpid#L43\">older glibc version which caches the ID\nof a process<\/a>! Which arguably is a curious optimization, since,\nwhat's the point? I mean how often to you have to call\n<code>getpid()<\/code> in a program, really? At some point (around Fedora 26)\nthis <a href=\"https:\/\/bugzilla.redhat.com\/show_bug.cgi?id=1469670\">feature was removed<\/a> since it apparently caused more\n<a href=\"https:\/\/yarchive.net\/comp\/linux\/getpid_caching.html\">trouble<\/a> than it's worth. Perhaps unsurprisingly, that\n<a href=\"https:\/\/bugzilla.redhat.com\/show_bug.cgi?id=1469670\">removal<\/a> even <a href=\"https:\/\/xkcd.com\/1172\/\">broke somebodies workflow<\/a>.<\/p>\n<h3 id=\"real-syscalls\">Real Syscalls<a class=\"headerlink\" href=\"#real-syscalls\" title=\"Permanent link\">&para;<\/a><\/h3>\n<p>So looking at the real syscalls, the user-kernel mode switches\ncost in the order of a few hundred nanoseconds, on all hosts. The\nhigher costs on some hosts can be explained by CPU bug\nmitigations being enabled (they are enabled by default) and\/or\nkind of older\/lower-end hardware. See also the Hosts Section for\nsome details.<\/p>\n<p>The fastest host is <code>xg6256<\/code> which manages to switch modes in\nless than 100 ns. It has a fast CPU with good\nsingle core performance (Xeon Gold 6256), has frequency scaling\ndisabled and runs at a constant 4.1 GHz frequency above it's base\nfrequency (i.e. a frequency between the base and turbo\nfrequency).<\/p>\n<h3 id=\"sched-yield\">Sched Yield<a class=\"headerlink\" href=\"#sched-yield\" title=\"Permanent link\">&para;<\/a><\/h3>\n<p>The <code>sched_yield()<\/code> syscall could be considered as a minimal work\nsyscall, e.g. when there is nothing to yield to. Also, the\nbenchmark process is running under the standard scheduling policy\nand on Linux <a href=\"https:\/\/manpath.be\/f34\/2\/sched_yield#L43\"><code>sched_yield()<\/code> is described as<\/a>:<\/p>\n<blockquote>\n<p><code>sched_yield()<\/code> is  intended for use with real-time scheduling policies (i.e., <code>SCHED_FIFO<\/code> or <code>SCHED_RR<\/code>).\nUse of <code>sched_yield()<\/code> with nondeterministic scheduling policies such as <code>SCHED_OTHER<\/code> is unspecified  and\nvery likely means your application design is broken.<\/p>\n<\/blockquote>\n<p>So unspecified could mean that the syscall just bails out after\nthe process' scheduling policy compares equal to <code>SCHED_OTHER<\/code>.<\/p>\n<p>On most hosts sched_yield 150 ns or so more expensive than a\nreally minimal syscall such as <code>getuid()<\/code> - which indicates some\nmore overhead, but not necessarily a context switch.<\/p>\n<h3 id=\"nanosleep\">Nanosleep<a class=\"headerlink\" href=\"#nanosleep\" title=\"Permanent link\">&para;<\/a><\/h3>\n<p>A bit out of the competition is the <code>nanosleep()<\/code> syscall:<\/p>\n<style  type=\"text\/css\" >\n    #T_07822f0e_0988_11ec_9e32_f8cab850d072 {\n            margin-bottom: 1em\n        }\n<\/style>\n<style  type=\"text\/css\" >\n    #T_07822f0e_0988_11ec_9e32_f8cab850d072row0_col0 {\n            background-color:  #e7d7ce;\n            color:  #000000;\n        }    #T_07822f0e_0988_11ec_9e32_f8cab850d072row0_col1 {\n            background-color:  #5875e1;\n            color:  #000000;\n        }    #T_07822f0e_0988_11ec_9e32_f8cab850d072row0_col2 {\n            background-color:  #e6d7cf;\n            color:  #000000;\n        }    #T_07822f0e_0988_11ec_9e32_f8cab850d072row0_col3 {\n            background-color:  #5f7fe8;\n            color:  #000000;\n        }    #T_07822f0e_0988_11ec_9e32_f8cab850d072row0_col4 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_07822f0e_0988_11ec_9e32_f8cab850d072row0_col5 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_07822f0e_0988_11ec_9e32_f8cab850d072row0_col6 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_07822f0e_0988_11ec_9e32_f8cab850d072row0_col7 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_07822f0e_0988_11ec_9e32_f8cab850d072row0_col8 {\n            background-color:  #4e68d8;\n            color:  #000000;\n        }    #T_07822f0e_0988_11ec_9e32_f8cab850d072row0_col9 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_07822f0e_0988_11ec_9e32_f8cab850d072row0_col10 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_07822f0e_0988_11ec_9e32_f8cab850d072row0_col11 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_07822f0e_0988_11ec_9e32_f8cab850d072row0_col12 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_07822f0e_0988_11ec_9e32_f8cab850d072row0_col13 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_07822f0e_0988_11ec_9e32_f8cab850d072row0_col14 {\n            background-color:  #b40426;\n            color:  #f1f1f1;\n        }    #T_07822f0e_0988_11ec_9e32_f8cab850d072row1_col0 {\n            background-color:  #d4dbe6;\n            color:  #000000;\n        }    #T_07822f0e_0988_11ec_9e32_f8cab850d072row1_col1 {\n            background-color:  #7093f3;\n            color:  #000000;\n        }    #T_07822f0e_0988_11ec_9e32_f8cab850d072row1_col2 {\n            background-color:  #b40426;\n            color:  #f1f1f1;\n        }    #T_07822f0e_0988_11ec_9e32_f8cab850d072row1_col3 {\n            background-color:  #8caffe;\n            color:  #000000;\n        }    #T_07822f0e_0988_11ec_9e32_f8cab850d072row1_col4 {\n            background-color:  #6a8bef;\n            color:  #000000;\n        }    #T_07822f0e_0988_11ec_9e32_f8cab850d072row1_col5 {\n            background-color:  #5977e3;\n            color:  #000000;\n        }    #T_07822f0e_0988_11ec_9e32_f8cab850d072row1_col6 {\n            background-color:  #7093f3;\n            color:  #000000;\n        }    #T_07822f0e_0988_11ec_9e32_f8cab850d072row1_col7 {\n            background-color:  #bad0f8;\n            color:  #000000;\n        }    #T_07822f0e_0988_11ec_9e32_f8cab850d072row1_col8 {\n            background-color:  #97b8ff;\n            color:  #000000;\n        }    #T_07822f0e_0988_11ec_9e32_f8cab850d072row1_col9 {\n            background-color:  #6b8df0;\n            color:  #000000;\n        }    #T_07822f0e_0988_11ec_9e32_f8cab850d072row1_col10 {\n            background-color:  #7295f4;\n            color:  #000000;\n        }    #T_07822f0e_0988_11ec_9e32_f8cab850d072row1_col11 {\n            background-color:  #5673e0;\n            color:  #000000;\n        }    #T_07822f0e_0988_11ec_9e32_f8cab850d072row1_col12 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_07822f0e_0988_11ec_9e32_f8cab850d072row1_col13 {\n            background-color:  #4a63d3;\n            color:  #f1f1f1;\n        }    #T_07822f0e_0988_11ec_9e32_f8cab850d072row1_col14 {\n            background-color:  #b5cdfa;\n            color:  #000000;\n        }    #T_07822f0e_0988_11ec_9e32_f8cab850d072row2_col0 {\n            background-color:  #d3dbe7;\n            color:  #000000;\n        }    #T_07822f0e_0988_11ec_9e32_f8cab850d072row2_col1 {\n            background-color:  #7093f3;\n            color:  #000000;\n        }    #T_07822f0e_0988_11ec_9e32_f8cab850d072row2_col2 {\n            background-color:  #b40426;\n            color:  #f1f1f1;\n        }    #T_07822f0e_0988_11ec_9e32_f8cab850d072row2_col3 {\n            background-color:  #8caffe;\n            color:  #000000;\n        }    #T_07822f0e_0988_11ec_9e32_f8cab850d072row2_col4 {\n            background-color:  #6a8bef;\n            color:  #000000;\n        }    #T_07822f0e_0988_11ec_9e32_f8cab850d072row2_col5 {\n            background-color:  #5977e3;\n            color:  #000000;\n        }    #T_07822f0e_0988_11ec_9e32_f8cab850d072row2_col6 {\n            background-color:  #6f92f3;\n            color:  #000000;\n        }    #T_07822f0e_0988_11ec_9e32_f8cab850d072row2_col7 {\n            background-color:  #b9d0f9;\n            color:  #000000;\n        }    #T_07822f0e_0988_11ec_9e32_f8cab850d072row2_col8 {\n            background-color:  #97b8ff;\n            color:  #000000;\n        }    #T_07822f0e_0988_11ec_9e32_f8cab850d072row2_col9 {\n            background-color:  #6b8df0;\n            color:  #000000;\n        }    #T_07822f0e_0988_11ec_9e32_f8cab850d072row2_col10 {\n            background-color:  #7295f4;\n            color:  #000000;\n        }    #T_07822f0e_0988_11ec_9e32_f8cab850d072row2_col11 {\n            background-color:  #5673e0;\n            color:  #000000;\n        }    #T_07822f0e_0988_11ec_9e32_f8cab850d072row2_col12 {\n            background-color:  #3b4cc0;\n            color:  #f1f1f1;\n        }    #T_07822f0e_0988_11ec_9e32_f8cab850d072row2_col13 {\n            background-color:  #4a63d3;\n            color:  #f1f1f1;\n        }    #T_07822f0e_0988_11ec_9e32_f8cab850d072row2_col14 {\n            background-color:  #b3cdfb;\n            color:  #000000;\n        }<\/style>\n<table id=\"T_07822f0e_0988_11ec_9e32_f8cab850d072\"><thead>    <tr>        <th class=\"index_name level0\" >host<\/th>        <th class=\"col_heading level0 col0\" >5i4250u<\/th>        <th class=\"col_heading level0 col1\" >7i6600u<\/th>        <th class=\"col_heading level0 col2\" >ac3758<\/th>        <th class=\"col_heading level0 col3\" >x2643<\/th>        <th class=\"col_heading level0 col4\" >x2667h<\/th>        <th class=\"col_heading level0 col5\" >x2667s<\/th>        <th class=\"col_heading level0 col6\" >x2687w<\/th>        <th class=\"col_heading level0 col7\" >x2689<\/th>        <th class=\"col_heading level0 col8\" >x2690<\/th>        <th class=\"col_heading level0 col9\" >xg6144<\/th>        <th class=\"col_heading level0 col10\" >xg6148<\/th>        <th class=\"col_heading level0 col11\" >xg6246<\/th>        <th class=\"col_heading level0 col12\" >xg6256<\/th>        <th class=\"col_heading level0 col13\" >xg6256b<\/th>        <th class=\"col_heading level0 col14\" >xs4110<\/th>    <\/tr>    <tr>        <th class=\"index_name level0\" >name<\/th>        <th class=\"blank\" ><\/th>        <th class=\"blank\" ><\/th>        <th class=\"blank\" ><\/th>        <th class=\"blank\" ><\/th>        <th class=\"blank\" ><\/th>        <th class=\"blank\" ><\/th>        <th class=\"blank\" ><\/th>        <th class=\"blank\" ><\/th>        <th class=\"blank\" ><\/th>        <th class=\"blank\" ><\/th>        <th class=\"blank\" ><\/th>        <th class=\"blank\" ><\/th>        <th class=\"blank\" ><\/th>        <th class=\"blank\" ><\/th>        <th class=\"blank\" ><\/th>    <\/tr><\/thead><tbody>\n                <tr>\n                        <th id=\"T_07822f0e_0988_11ec_9e32_f8cab850d072level0_row0\" class=\"row_heading level0 row0\" >nanosleep0<\/th>\n                        <td id=\"T_07822f0e_0988_11ec_9e32_f8cab850d072row0_col0\" class=\"data row0 col0\" >52632<\/td>\n                        <td id=\"T_07822f0e_0988_11ec_9e32_f8cab850d072row0_col1\" class=\"data row0 col1\" >50474<\/td>\n                        <td id=\"T_07822f0e_0988_11ec_9e32_f8cab850d072row0_col2\" class=\"data row0 col2\" >52620<\/td>\n                        <td id=\"T_07822f0e_0988_11ec_9e32_f8cab850d072row0_col3\" class=\"data row0 col3\" >50588<\/td>\n                        <td id=\"T_07822f0e_0988_11ec_9e32_f8cab850d072row0_col4\" class=\"data row0 col4\" >50003<\/td>\n                        <td id=\"T_07822f0e_0988_11ec_9e32_f8cab850d072row0_col5\" class=\"data row0 col5\" >50011<\/td>\n                        <td id=\"T_07822f0e_0988_11ec_9e32_f8cab850d072row0_col6\" class=\"data row0 col6\" >50000<\/td>\n                        <td id=\"T_07822f0e_0988_11ec_9e32_f8cab850d072row0_col7\" class=\"data row0 col7\" >50014<\/td>\n                        <td id=\"T_07822f0e_0988_11ec_9e32_f8cab850d072row0_col8\" class=\"data row0 col8\" >50312<\/td>\n                        <td id=\"T_07822f0e_0988_11ec_9e32_f8cab850d072row0_col9\" class=\"data row0 col9\" >50018<\/td>\n                        <td id=\"T_07822f0e_0988_11ec_9e32_f8cab850d072row0_col10\" class=\"data row0 col10\" >50000<\/td>\n                        <td id=\"T_07822f0e_0988_11ec_9e32_f8cab850d072row0_col11\" class=\"data row0 col11\" >50014<\/td>\n                        <td id=\"T_07822f0e_0988_11ec_9e32_f8cab850d072row0_col12\" class=\"data row0 col12\" >50000<\/td>\n                        <td id=\"T_07822f0e_0988_11ec_9e32_f8cab850d072row0_col13\" class=\"data row0 col13\" >50000<\/td>\n                        <td id=\"T_07822f0e_0988_11ec_9e32_f8cab850d072row0_col14\" class=\"data row0 col14\" >54866<\/td>\n            <\/tr>\n            <tr>\n                        <th id=\"T_07822f0e_0988_11ec_9e32_f8cab850d072level0_row1\" class=\"row_heading level0 row1\" >nanosleep0_slack1<\/th>\n                        <td id=\"T_07822f0e_0988_11ec_9e32_f8cab850d072row1_col0\" class=\"data row1 col0\" >4355<\/td>\n                        <td id=\"T_07822f0e_0988_11ec_9e32_f8cab850d072row1_col1\" class=\"data row1 col1\" >2836<\/td>\n                        <td id=\"T_07822f0e_0988_11ec_9e32_f8cab850d072row1_col2\" class=\"data row1 col2\" >7076<\/td>\n                        <td id=\"T_07822f0e_0988_11ec_9e32_f8cab850d072row1_col3\" class=\"data row1 col3\" >3247<\/td>\n                        <td id=\"T_07822f0e_0988_11ec_9e32_f8cab850d072row1_col4\" class=\"data row1 col4\" >2736<\/td>\n                        <td id=\"T_07822f0e_0988_11ec_9e32_f8cab850d072row1_col5\" class=\"data row1 col5\" >2483<\/td>\n                        <td id=\"T_07822f0e_0988_11ec_9e32_f8cab850d072row1_col6\" class=\"data row1 col6\" >2835<\/td>\n                        <td id=\"T_07822f0e_0988_11ec_9e32_f8cab850d072row1_col7\" class=\"data row1 col7\" >3908<\/td>\n                        <td id=\"T_07822f0e_0988_11ec_9e32_f8cab850d072row1_col8\" class=\"data row1 col8\" >3401<\/td>\n                        <td id=\"T_07822f0e_0988_11ec_9e32_f8cab850d072row1_col9\" class=\"data row1 col9\" >2762<\/td>\n                        <td id=\"T_07822f0e_0988_11ec_9e32_f8cab850d072row1_col10\" class=\"data row1 col10\" >2870<\/td>\n                        <td id=\"T_07822f0e_0988_11ec_9e32_f8cab850d072row1_col11\" class=\"data row1 col11\" >2446<\/td>\n                        <td id=\"T_07822f0e_0988_11ec_9e32_f8cab850d072row1_col12\" class=\"data row1 col12\" >1974<\/td>\n                        <td id=\"T_07822f0e_0988_11ec_9e32_f8cab850d072row1_col13\" class=\"data row1 col13\" >2248<\/td>\n                        <td id=\"T_07822f0e_0988_11ec_9e32_f8cab850d072row1_col14\" class=\"data row1 col14\" >3837<\/td>\n            <\/tr>\n            <tr>\n                        <th id=\"T_07822f0e_0988_11ec_9e32_f8cab850d072level0_row2\" class=\"row_heading level0 row2\" >nanosleep1_slack1<\/th>\n                        <td id=\"T_07822f0e_0988_11ec_9e32_f8cab850d072row2_col0\" class=\"data row2 col0\" >4348<\/td>\n                        <td id=\"T_07822f0e_0988_11ec_9e32_f8cab850d072row2_col1\" class=\"data row2 col1\" >2840<\/td>\n                        <td id=\"T_07822f0e_0988_11ec_9e32_f8cab850d072row2_col2\" class=\"data row2 col2\" >7102<\/td>\n                        <td id=\"T_07822f0e_0988_11ec_9e32_f8cab850d072row2_col3\" class=\"data row2 col3\" >3252<\/td>\n                        <td id=\"T_07822f0e_0988_11ec_9e32_f8cab850d072row2_col4\" class=\"data row2 col4\" >2736<\/td>\n                        <td id=\"T_07822f0e_0988_11ec_9e32_f8cab850d072row2_col5\" class=\"data row2 col5\" >2486<\/td>\n                        <td id=\"T_07822f0e_0988_11ec_9e32_f8cab850d072row2_col6\" class=\"data row2 col6\" >2834<\/td>\n                        <td id=\"T_07822f0e_0988_11ec_9e32_f8cab850d072row2_col7\" class=\"data row2 col7\" >3908<\/td>\n                        <td id=\"T_07822f0e_0988_11ec_9e32_f8cab850d072row2_col8\" class=\"data row2 col8\" >3410<\/td>\n                        <td id=\"T_07822f0e_0988_11ec_9e32_f8cab850d072row2_col9\" class=\"data row2 col9\" >2767<\/td>\n                        <td id=\"T_07822f0e_0988_11ec_9e32_f8cab850d072row2_col10\" class=\"data row2 col10\" >2871<\/td>\n                        <td id=\"T_07822f0e_0988_11ec_9e32_f8cab850d072row2_col11\" class=\"data row2 col11\" >2446<\/td>\n                        <td id=\"T_07822f0e_0988_11ec_9e32_f8cab850d072row2_col12\" class=\"data row2 col12\" >1975<\/td>\n                        <td id=\"T_07822f0e_0988_11ec_9e32_f8cab850d072row2_col13\" class=\"data row2 col13\" >2246<\/td>\n                        <td id=\"T_07822f0e_0988_11ec_9e32_f8cab850d072row2_col14\" class=\"data row2 col14\" >3836<\/td>\n            <\/tr>\n    <\/tbody><\/table>\n\n<p>Calling <code>nanosleep()<\/code> to sleep for 0 ns or 1 ns seemingly also is\na very cheap syscall or even a null operation.<\/p>\n<p>However, in the first case it takes 50 \u00b5s an all hosts.\nIncidentally, 50 \u00b5s is also the <a href=\"https:\/\/manpath.be\/f34\/2\/prctl#L1052\">default timer slack value<\/a>\nfor a normally scheduled process on Linux. The <a href=\"https:\/\/lwn.net\/Articles\/588086\/\">timer slack\nmechanism<\/a> extends timer expirations up to the slack value in\norder to group multiple timers since this reduces wake ups and\nthus saves energy. And since <code>nanosleep()<\/code> creates a timer, it's\nalso affected by this mechanism.<\/p>\n<p>Thus, the other nanosleep cases <a href=\"https:\/\/github.com\/gsauthof\/osjitter\/blob\/f1a4ca9cbf7516efc61c3bab2fe06ffe83cfb43c\/bench_syscalls.cc#L117\">set a minimal timer slack<\/a> of 1\nns which reduces the runtime, as expected. However, it's still\nmuch more expensive than the other syscalls. Of course, a timer\nexpiration has a limited accuracy. However, with 0 ns or 1 ns no\ntimer has to be expired, really. It turns out that calling\n<code>nanosleep()<\/code> unconditionally yields a (voluntary) <a href=\"https:\/\/en.wikipedia.org\/wiki\/Context_switch\">context\nswitch<\/a>.  Even on isolated cores, where the scheduler happily\nswitches to the swapper kernel thread. Thus, the last two\nnanonsleep cases really measure the context switch costs which\nare more expensive than a simple mode switch.<\/p>\n<p>The costs of a context switch match what <a href=\"https:\/\/eli.thegreenplace.net\/2018\/measuring-context-switching-and-memory-overheads-for-linux-threads\/#how-expensive-are-context-switches\">others are\nmeasuring<\/a> (modulo division by two).<\/p>\n<h2 id=\"hosts\">Hosts<a class=\"headerlink\" href=\"#hosts\" title=\"Permanent link\">&para;<\/a><\/h2>\n<p>The following table shows the hosts under benchmark:<\/p>\n<table class=\"m-table\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th><\/th>\n      <th>host<\/th>\n      <th>CPU<\/th>\n      <th>mitigations<\/th>\n      <th>poll<\/th>\n      <th>os<\/th>\n      <th>kernel<\/th>\n    <\/tr>\n  <\/thead>\n  <tbody>\n    <tr>\n      <th>0<\/th>\n      <td>5i4250u<\/td>\n      <td>Core i5-4250U<\/td>\n      <td>yes<\/td>\n      <td>yes<\/td>\n      <td>Fedora 33<\/td>\n      <td>5.12<\/td>\n    <\/tr>\n    <tr>\n      <th>1<\/th>\n      <td>7i6600u<\/td>\n      <td>Core i7-6600U<\/td>\n      <td>yes<\/td>\n      <td>yes<\/td>\n      <td>Fedora 33<\/td>\n      <td>5.13<\/td>\n    <\/tr>\n    <tr>\n      <th>2<\/th>\n      <td>ac3758<\/td>\n      <td>Atom C3758<\/td>\n      <td>no<\/td>\n      <td>yes<\/td>\n      <td>Fedora 33<\/td>\n      <td>5.13<\/td>\n    <\/tr>\n    <tr>\n      <th>3<\/th>\n      <td>x2643<\/td>\n      <td>Xeon E5-2643 v2<\/td>\n      <td>yes<\/td>\n      <td>no<\/td>\n      <td>RHEL 7<\/td>\n      <td>3.10<\/td>\n    <\/tr>\n    <tr>\n      <th>4<\/th>\n      <td>x2667h<\/td>\n      <td>Xeon E5-2667 v3<\/td>\n      <td>yes<\/td>\n      <td>yes<\/td>\n      <td>RHEL 7<\/td>\n      <td>3.10<\/td>\n    <\/tr>\n    <tr>\n      <th>5<\/th>\n      <td>x2667s<\/td>\n      <td>Xeon E5-2667 v3<\/td>\n      <td>yes<\/td>\n      <td>yes<\/td>\n      <td>RHEL 7<\/td>\n      <td>3.10<\/td>\n    <\/tr>\n    <tr>\n      <th>6<\/th>\n      <td>x2687w<\/td>\n      <td>Xeon E5-2687W v3<\/td>\n      <td>yes<\/td>\n      <td>yes<\/td>\n      <td>RHEL 7<\/td>\n      <td>3.10<\/td>\n    <\/tr>\n    <tr>\n      <th>7<\/th>\n      <td>x2689<\/td>\n      <td>Xeon E5-2689 v4<\/td>\n      <td>yes<\/td>\n      <td>yes<\/td>\n      <td>RHEL 7<\/td>\n      <td>3.10<\/td>\n    <\/tr>\n    <tr>\n      <th>8<\/th>\n      <td>x2690<\/td>\n      <td>Xeon E5-2690 0<\/td>\n      <td>yes<\/td>\n      <td>yes<\/td>\n      <td>RHEL 7<\/td>\n      <td>3.10<\/td>\n    <\/tr>\n    <tr>\n      <th>9<\/th>\n      <td>xg6144<\/td>\n      <td>Xeon Gold 6144<\/td>\n      <td>yes<\/td>\n      <td>yes<\/td>\n      <td>RHEL 7<\/td>\n      <td>3.10<\/td>\n    <\/tr>\n    <tr>\n      <th>10<\/th>\n      <td>xg6148<\/td>\n      <td>Xeon Gold 6148<\/td>\n      <td>no<\/td>\n      <td>yes<\/td>\n      <td>RHEL 7<\/td>\n      <td>3.10<\/td>\n    <\/tr>\n    <tr>\n      <th>11<\/th>\n      <td>xg6246<\/td>\n      <td>Xeon Gold 6246<\/td>\n      <td>yes<\/td>\n      <td>yes<\/td>\n      <td>RHEL 7<\/td>\n      <td>3.10<\/td>\n    <\/tr>\n    <tr>\n      <th>12<\/th>\n      <td>xg6256<\/td>\n      <td>Xeon Gold 6256<\/td>\n      <td>no<\/td>\n      <td>yes<\/td>\n      <td>RHEL 7<\/td>\n      <td>3.10<\/td>\n    <\/tr>\n    <tr>\n      <th>13<\/th>\n      <td>xg6256b<\/td>\n      <td>Xeon Gold 6256<\/td>\n      <td>yes<\/td>\n      <td>yes<\/td>\n      <td>RHEL 7<\/td>\n      <td>3.10<\/td>\n    <\/tr>\n    <tr>\n      <th>14<\/th>\n      <td>xs4110<\/td>\n      <td>Xeon Silver 4110<\/td>\n      <td>no<\/td>\n      <td>yes<\/td>\n      <td>RHEL 7<\/td>\n      <td>3.10<\/td>\n    <\/tr>\n  <\/tbody>\n<\/table>\n\n<p>Notes:<\/p>\n<ul>\n<li>the kernels are the ones packaged by the distributions<\/li>\n<li>most RHEL hosts are on RHEL 7.9<\/li>\n<li>CPU mitigations are disabled via the <code>mitigations=off<\/code> kernel\n  parameter or similar parameters<\/li>\n<li>polling means that CPU frequency scaling and power saving is\n  disabled via kernel parameters and tuned PM QoS settings<\/li>\n<li>so the host's CPU runs on a fixed frequency; where possible this\n  frequency is set slightly above the base frequency, e.g. on the\n  Xeon Gold 6256 CPU it's set to 4.1 GHz<\/li>\n<li>the Atom CPU doesn't support Hyperthreading and Hyperthreading\n  is disabled on all Xeon hosts<\/li>\n<li>all hosts have SELinux and\/or Auditing enabled (on Fedora\/RHEL these features are enabled by default) which adds some syscall overhead to some degree<\/li>\n<\/ul>\n<h2 id=\"terminology\">Terminology<a class=\"headerlink\" href=\"#terminology\" title=\"Permanent link\">&para;<\/a><\/h2>\n<p>There are basically two important separate terms to distinguish\nin the above discussion:<\/p>\n<ol>\n<li><a href=\"https:\/\/en.wikipedia.org\/wiki\/Context_switch#User_and_kernel_mode_switching\">Mode Switch<\/a> (or Mode Transition)<\/li>\n<li><a href=\"https:\/\/en.wikipedia.org\/wiki\/Context_switch\">Context Switch<\/a><\/li>\n<\/ol>\n<p>The definitions of these terms might vary in different literature\nand different operating systems. Also, in other contexts (no pun\nintended!) one might describe different modes as different\ncontexts. However, the definitions given in the linked Wikipedia\narticles are widely used and apply to Linux.<\/p>\n<p>Basically, a mode transition denotes the switch between user mode and\nkernel mode (or between user space and kernel space) whereas a context switch\ndenotes a switch between different tasks, which is facilitated by\nthe kernel. A context switch requires more work than a mode\nswitch and thus is more expensive.<\/p>","category":[{"@attributes":{"term":"linux"}},{"@attributes":{"term":"kernel"}},{"@attributes":{"term":"linux"}},{"@attributes":{"term":"latency"}}]},{"title":"The Rise of Meson","link":{"@attributes":{"href":"https:\/\/gms.tf\/the-rise-of-meson.html","rel":"alternate"}},"published":"2021-08-07T13:00:00+02:00","updated":"2021-08-07T13:00:00+02:00","author":{"name":"Georg Sauthoff"},"id":"tag:gms.tf,2021-08-07:\/the-rise-of-meson.html","summary":"<p><a href=\"https:\/\/en.wikipedia.org\/wiki\/Meson_(software)#Adopters\">More and more<\/a> open source projects seem to switch to the <a href=\"https:\/\/mesonbuild.com\/\">Meson<\/a> build tool, in the last years.\nIn an attempt to quantify this development I looked at the build tools usage by <a href=\"https:\/\/en.wikipedia.org\/wiki\/Fedora_(operating_system)\">Fedora<\/a> packages, during the last 14 years.<\/p>\n<h2 id=\"about-meson\">About Meson<a class=\"headerlink\" href=\"#about-meson\" title=\"Permanent link\">&para;<\/a><\/h2>\n<p><a href=\"https:\/\/en.wikipedia.org\/wiki\/Meson_(software)\">Meson<\/a> is a build tool with good C \u2026<\/p>","content":"<p><a href=\"https:\/\/en.wikipedia.org\/wiki\/Meson_(software)#Adopters\">More and more<\/a> open source projects seem to switch to the <a href=\"https:\/\/mesonbuild.com\/\">Meson<\/a> build tool, in the last years.\nIn an attempt to quantify this development I looked at the build tools usage by <a href=\"https:\/\/en.wikipedia.org\/wiki\/Fedora_(operating_system)\">Fedora<\/a> packages, during the last 14 years.<\/p>\n<h2 id=\"about-meson\">About Meson<a class=\"headerlink\" href=\"#about-meson\" title=\"Permanent link\">&para;<\/a><\/h2>\n<p><a href=\"https:\/\/en.wikipedia.org\/wiki\/Meson_(software)\">Meson<\/a> is a build tool with good C\/C++ support. It's\nwritten in Python and generates <a href=\"https:\/\/en.wikipedia.org\/wiki\/Ninja_(build_system)\">Ninja<\/a> build files. Since it was\ncreated after <a href=\"https:\/\/en.wikipedia.org\/wiki\/CMake\">CMake<\/a> (which was initially released in\n2000), it had the chance to learn from CMake's\nmistakes. In comparison to CMake, Meson has better <a href=\"https:\/\/mesonbuild.com\/Manual.html\">documentation<\/a>,\nit doesn't support generating classic makefiles, it only supports\nout of source tree builds, and its domain specific language is\narguably much better designed.<\/p>\n<p>So, from the start, Meson arguably is also more opinionated in its\ndesign, but in a useful way. Usually, simple stuff is easy to solve\nwith Meson and there is one kind of obvious way to do it. Whereas\nwith CMake, there are often several ways to implement common\nbuild tasks (cf. <a href=\"https:\/\/en.wikipedia.org\/wiki\/There%27s_more_than_one_way_to_do_it\">TMTOWTDI<\/a>), including some legacy ones. With Meson, build files\nare often much shorter than equivalent CMake ones.<\/p>\n<p>CMake evolved over the years and it's possible to write -\nsay - a modern style of CMake build files that are less awful,\nand use it in a modern way to some degree, but it feels\nthat the <a href=\"https:\/\/cmake.org\/cmake\/help\/v3.21\/\">official documentation<\/a> doesn't help much in that regard,\nand that one would have to <a href=\"https:\/\/pabloariasal.github.io\/2018\/02\/19\/its-time-to-do-cmake-right\/\">study<\/a> several <a href=\"https:\/\/cliutils.gitlab.io\/modern-cmake\/#other-sources\">books<\/a> on modern CMake in\norder to arrive there.<\/p>\n<h2 id=\"fedora-data\">Fedora Data<a class=\"headerlink\" href=\"#fedora-data\" title=\"Permanent link\">&para;<\/a><\/h2>\n<p>The first packages that required Meson to build arrived in Fedora 22,\ni.e. 6 years ago (in 2015, Fedora runs on a half-year release schedule and\nMeson was initially released in 2013, 8 years ago).\nSince then the number of packages that\nbuild-require Meson grew to almost 600 hundred:<\/p>\n<p><img src=\"https:\/\/gms.tf\/image\/meson-et-al-fedora.svg\" alt=\"Number of packages that build-require CMake, Meson, Scons\" width=\"100%\"\/><\/p>\n<p>So this graph shows the number of packages that require a notable\nbuild tool at build time, for each Fedora release.<\/p>\n<p><a href=\"https:\/\/en.wikipedia.org\/wiki\/SCons\">Scons<\/a> never really took off and nowadays only a handful of\npackages still require it. In contrast, the Meson usage shows\nsome serious growth.<\/p>\n<p>When looking at the Meson growth in relation to the packages that\neither require CMake, Scons or Meson - Meson is now required by\n30 % or so of the packages:<\/p>\n<p><img src=\"https:\/\/gms.tf\/image\/meson-rise.svg\" alt=\"Relative Meson Build Requirements growth\" width=\"100%\"\/><\/p>\n<p>The growth slowed a bit down in the last Fedora releases, it\nseems.<\/p>\n<h2 id=\"other-tools\">Other Tools<a class=\"headerlink\" href=\"#other-tools\" title=\"Permanent link\">&para;<\/a><\/h2>\n<p>So what about <a href=\"https:\/\/en.wikipedia.org\/wiki\/Autoconf\">Autoconf<\/a>, <a href=\"https:\/\/en.wikipedia.org\/wiki\/Automake\">Automake<\/a>, etc.\n(a.k.a. <a href=\"https:\/\/en.wikipedia.org\/wiki\/GNU_Autotools\">Autotools<\/a>) and other build tools?\nSince most project that use Autotools release their tar archive\nfiles with already generated configure scripts\/make files, the\nresulting packages don't depend on those tools. Plus, <a href=\"https:\/\/fedoraproject.org\/wiki\/Changes\/Remove_make_from_BuildRoot\">before\nFedora 34<\/a>, there isn't even a dependency on GNU make,\nsince it was part of the standard build root environment. This\nchanged with Fedora 34, where 9000 packages or so depend on GNU\nmake.  Subtracting the packages that also require CMake, there\nremain 8000 packages or so that might use Autotools. But how many\nof those use plain old Makefiles can't be easily derived by just\nlooking at the package dependencies.<\/p>\n<p>In any case, using Autotools for a modern C\/C++ project in 2021\nis like using CVS for source code version control in 2021: there\nare better tools available and thus it isn't very interesting to\nstill consider the legacy solutions.<\/p>\n<p>The <a href=\"https:\/\/en.wikipedia.org\/wiki\/Boost_(C%2B%2B_libraries)\">Boost<\/a> project has its own build system: <a href=\"https:\/\/en.wikipedia.org\/wiki\/Perforce_Jam#Boost.Jam\">BJam<\/a>. There\nis some <a href=\"https:\/\/www.boost.org\/build\/doc\/html\/index.html\">documentation<\/a>, but it's kind of minimal and\ncryptic. As of Fedora 34, the is no package that build-requires\nBJam. (Boost itself doesn't need to explicitly require it since\nit's included in the Boost release tar archive.) Incidentally,\nthe Boost steering committee <a href=\"https:\/\/lists.boost.org\/boost-interest\/2017\/07\/0162.php\">announced<\/a> in 2017 its\ndesire to migrate to CMake, however, as of 2021, the current\nBoost documentation still just mentions BJam\/b2 for building\nBoost.<\/p>\n<h2 id=\"methods\">Methods<a class=\"headerlink\" href=\"#methods\" title=\"Permanent link\">&para;<\/a><\/h2>\n<p>The build tool usage stats were obtained with:<\/p>\n<div class=\"m-code\"><pre><span><\/span><code><span class=\"p\">{<\/span>\n<span class=\"w\">    <\/span><span class=\"n\">echo<\/span><span class=\"w\"> <\/span><span class=\"s2\">&quot;fedora_release,tool,users&quot;<\/span>\n<span class=\"w\">    <\/span><span class=\"k\">for<\/span><span class=\"w\"> <\/span><span class=\"p\">((<\/span><span class=\"n\">i<\/span><span class=\"o\">=<\/span><span class=\"mi\">7<\/span><span class=\"p\">;<\/span><span class=\"w\"> <\/span><span class=\"n\">i<\/span><span class=\"o\">&lt;<\/span><span class=\"mi\">35<\/span><span class=\"p\">;<\/span><span class=\"w\"> <\/span><span class=\"o\">++<\/span><span class=\"n\">i<\/span><span class=\"p\">));<\/span><span class=\"w\"> <\/span><span class=\"n\">do<\/span>\n<span class=\"w\">        <\/span><span class=\"k\">for<\/span><span class=\"w\"> <\/span><span class=\"n\">k<\/span><span class=\"w\"> <\/span><span class=\"ow\">in<\/span><span class=\"w\"> <\/span><span class=\"n\">meson<\/span><span class=\"w\"> <\/span><span class=\"n\">cmake<\/span><span class=\"w\"> <\/span><span class=\"n\">scons<\/span><span class=\"p\">;<\/span><span class=\"w\"> <\/span><span class=\"n\">do<\/span>\n<span class=\"w\">            <\/span><span class=\"n\">n<\/span><span class=\"o\">=$<\/span><span class=\"p\">(<\/span><span class=\"n\">dnf<\/span><span class=\"w\"> <\/span><span class=\"n\">repoquery<\/span><span class=\"w\"> <\/span><span class=\"o\">--<\/span><span class=\"n\">releasever<\/span><span class=\"o\">=$<\/span><span class=\"n\">i<\/span><span class=\"w\"> <\/span><span class=\"o\">--<\/span><span class=\"n\">disablerepo<\/span><span class=\"o\">=<\/span><span class=\"s1\">&#39;*&#39;<\/span><span class=\"w\"> <\/span><span class=\"o\">--<\/span><span class=\"n\">enablerepo<\/span><span class=\"o\">=<\/span><span class=\"s1\">&#39;*-source&#39;<\/span><span class=\"w\"> <\/span>\\\n<span class=\"w\">                      <\/span><span class=\"o\">--<\/span><span class=\"n\">arch<\/span><span class=\"o\">=<\/span><span class=\"n\">src<\/span><span class=\"w\"> <\/span><span class=\"o\">--<\/span><span class=\"n\">whatrequires<\/span><span class=\"w\"> <\/span><span class=\"o\">$<\/span><span class=\"n\">k<\/span><span class=\"w\"> <\/span><span class=\"o\">|<\/span><span class=\"w\"> <\/span><span class=\"n\">sed<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;s\/-[0-9]:[0-9].*$\/\/&#39;<\/span><span class=\"w\"> <\/span>\\\n<span class=\"w\">                    <\/span><span class=\"o\">|<\/span><span class=\"w\"> <\/span><span class=\"n\">sort<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"n\">u<\/span><span class=\"w\">  <\/span><span class=\"o\">|<\/span><span class=\"w\"> <\/span><span class=\"n\">wc<\/span><span class=\"w\"> <\/span><span class=\"o\">-<\/span><span class=\"n\">l<\/span><span class=\"p\">)<\/span>\n<span class=\"w\">            <\/span><span class=\"n\">echo<\/span><span class=\"w\"> <\/span><span class=\"s2\">&quot;$i,$k,$n&quot;<\/span>\n<span class=\"w\">        <\/span><span class=\"n\">done<\/span>\n<span class=\"w\">    <\/span><span class=\"n\">done<\/span>\n<span class=\"p\">}<\/span><span class=\"w\"> <\/span><span class=\"o\">|<\/span><span class=\"w\"> <\/span><span class=\"n\">tee<\/span><span class=\"w\"> <\/span><span class=\"n\">build<\/span><span class=\"o\">-<\/span><span class=\"n\">stat<\/span><span class=\"o\">.<\/span><span class=\"n\">csv<\/span>\n<\/code><\/pre><\/div>\n\n<p>And for GNU make, on Fedora 34:<\/p>\n<div class=\"m-code\"><pre><span><\/span><code>dnf repoquery --disablerepo=&#39;*&#39; --enablerepo=&#39;*-source&#39; --arch=src \\\n      --whatrequires make | sed &#39;s\/-[0-9]:[0-9].*$\/\/&#39; \\\n    | sort -u  | wc -l &gt; f34-require-make.log\n<\/code><\/pre><\/div>","category":[{"@attributes":{"term":"linux"}},{"@attributes":{"term":"meson"}},{"@attributes":{"term":"build"}},{"@attributes":{"term":"linux"}}]},{"title":"Deploying Full Disk Encryption in the Cloud with Ansible","link":{"@attributes":{"href":"https:\/\/gms.tf\/fedora-initramfs.html","rel":"alternate"}},"published":"2020-11-17T12:00:00+01:00","updated":"2020-11-17T12:00:00+01:00","author":{"name":"Georg Sauthoff"},"id":"tag:gms.tf,2020-11-17:\/fedora-initramfs.html","summary":"<p>Deploying a new virtual machine 'in the cloud' is simple enough\nand fast. As long as one doesn't need to setup an encrypted root\nfilesystem. This article shows how to automate the setup of a\nFedora system with an encrypted root filesystem remotely - 'in\nthe cloud' - on e.g. a \u2026<\/p>","content":"<p>Deploying a new virtual machine 'in the cloud' is simple enough\nand fast. As long as one doesn't need to setup an encrypted root\nfilesystem. This article shows how to automate the setup of a\nFedora system with an encrypted root filesystem remotely - 'in\nthe cloud' - on e.g. a DigitalOcean, Hetzner, Linode, etc. VM -\nwith Ansible and a special <a href=\"https:\/\/github.com\/gsauthof\/playbook\/tree\/master\/fedora\/initramfs\">Fedora rescue system<\/a> that runs purely\nfrom an <a href=\"https:\/\/en.wikipedia.org\/wiki\/Initial_ramdisk\">initramfs<\/a> archive.<\/p>\n<h2 id=\"motivation\">Motivation<a class=\"headerlink\" href=\"#motivation\" title=\"Permanent link\">&para;<\/a><\/h2>\n<p>Having an encrypted root filesystem gives you <a href=\"https:\/\/en.wikipedia.org\/wiki\/Data_at_rest\">data at rest\nprotection<\/a>. At the end of the day, your virtual disk\ndevice inside your virtual machine (VM) is just a file on a\nstorage solution where disks are getting replaced and reallocated\nfor other things. After - say - hardware replacements perhaps\nsuch a disk even ends up on eBay without being properly deleted\nand thus some random person now has access to your data. With\nyour root filesystem being encrypted you are protected against\nsuch scenarios.<\/p>\n<p>If you think that such a scenario is far fetched, see also for\nexample <a href=\"https:\/\/www.heise.de\/ct\/artikel\/SSD-mit-Daten-von-Jugendamt-und-Zulassungsstelle-bei-eBay-gefunden-4615144.html\">this 2019 computer magazine article (in German)<\/a>\nwhich describes in detail how some SSDs with very sensitive\npersonal data ended up on eBay.<\/p>\n<h2 id=\"the-problem\">The Problem<a class=\"headerlink\" href=\"#the-problem\" title=\"Permanent link\">&para;<\/a><\/h2>\n<p>Cloud providers such as DigitalOcean, Linode or Hetzner provide a\nset of pre-built disk images for different operating systems.\nUsers also have the option to add custom images to their account,\nusually. When deploying a new virtual machine (e.g. a Droplet\nwhen using DigitalOcean) the selected disk image is extracted to\nthe new VM guest, the image has to run some first-boot\ninitialization script which fetches the network configuration,\ncreates sshd host keys and stuff like that.<\/p>\n<p>This process is very fast because the image is pre-built and just\nneeds to be copied while the first-boot initialization doesn't\nhave much to do. Compare this with a traditional automated Linux\ninstall method such as <a href=\"https:\/\/en.wikipedia.org\/wiki\/Kickstart_(Linux)\">Kickstart<\/a> where RPMs are installed one\nafter the other. This takes much longer.<\/p>\n<p>The disadvantage is a loss in flexibility. Especially, this makes\nsetting up the root filesystem on a <a href=\"https:\/\/en.wikipedia.org\/wiki\/Linux_Unified_Key_Setup\">LUKS encrypted<\/a> device\nchallenging.<\/p>\n<h2 id=\"non-solutions\">Non-Solutions<a class=\"headerlink\" href=\"#non-solutions\" title=\"Permanent link\">&para;<\/a><\/h2>\n<p>A naive approach to a remote system with an encrypted root\nfilesystem is to locally install a Linux distribution with disk\nencryption enabled and upload the resulting image. The problem\nwith this is that if you want to deploy multiple VMs all then\nhave the same LUKS encryption key. Certainly you can't openly\nshare this image with others and you would upload a new image for\neach new VM deployment. Obviously this is tedious and doesn't\nscale.<\/p>\n<p>After the VM is deployed with an unencrypted root filesystem,\nencrypting it on the fly isn't possible, because the root\nfilesystem is in use.<\/p>\n<h2 id=\"a-solution\">A Solution<a class=\"headerlink\" href=\"#a-solution\" title=\"Permanent link\">&para;<\/a><\/h2>\n<p>For some time Linux has the interesting feature to boot a new\nkernel image from a running Linux system. That means the\ncurrently running kernel can be replaced with a new one using the\n<a href=\"https:\/\/en.wikipedia.org\/wiki\/Kexec\">kexec system call<\/a>.<\/p>\n<p>A standard part of the Linux boot process is to extract an early\nboot root filesystem <a href=\"https:\/\/en.wikipedia.org\/wiki\/Initial_ramdisk\">image into a ram disk<\/a>, execute an init\nprocess from it that basically mounts the real root filesystem\nand switches to it. Of course, the final root filesystem switch\nis optional.<\/p>\n<p>In combination these features, i.e. kexec and an early-boot\nenvironment, can be used to <a href=\"https:\/\/en.wikipedia.org\/wiki\/Bootstrapping\">bootstrap<\/a> a fresh Linux installation\nfrom scratch - from a running system.<\/p>\n<p>The basic idea here is to kexec a new kernel with an\n<a href=\"https:\/\/en.wikipedia.org\/wiki\/Initial_ramdisk\">initramfs<\/a> image that contains a small Linux system, i.e.\none that includes enough software to provide network connectivity\nincluding SSH and to drive the Fedora install process.<\/p>\n<p>Recently, I implemented this approach, i.e. I thus wrote a\n<a href=\"https:\/\/github.com\/gsauthof\/playbook\/tree\/master\/fedora\/initramfs\">script that creates such initramfs images<\/a>. That means it\ninstalls a minimal Fedora system into an initramfs.<\/p>\n<p>I also created an <a href=\"https:\/\/github.com\/gsauthof\/playbook\/tree\/master\/fedora\/initramfs\/ansible\">Ansible playbook that drives the whole\nprocess<\/a>.  That means it downloads the initramfs image to the VM,\ncreates VM specific configuration, kexecs into the rescue system\nand finally installs the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Linux_Unified_Key_Setup\">LUKS-encrypted<\/a> target system from there.\nIt even supports creating a DigitalOcean Droplet, first.<\/p>\n<p>Without considering the VM creation, the playbook consists of 30\nsteps or so. After the playbook finishes, the target VM has an\nencrypted root filesystem with a fresh Fedora system on top of\nit.<\/p>\n<p>So far, I successfully used this method and playbook on\nDigitalOcean, Linode and Hetzner VMs and even on some old-school\nbare-metal root-server (rented from Manitu).<\/p>","category":[{"@attributes":{"term":"linux"}},{"@attributes":{"term":"initramfs"}},{"@attributes":{"term":"ansible"}},{"@attributes":{"term":"fedora"}}]},{"title":"Deploying SSH Host Keys with Digital Ocean Droplets","link":{"@attributes":{"href":"https:\/\/gms.tf\/ssh-host-key-deployment.html","rel":"alternate"}},"published":"2020-06-05T22:00:00+02:00","updated":"2020-06-05T22:00:00+02:00","author":{"name":"Georg Sauthoff"},"id":"tag:gms.tf,2020-06-05:\/ssh-host-key-deployment.html","summary":"<p>When deploying a VM in a remote location one might end up in an\nimpossible situation: a VM base image doesn't contain <a href=\"https:\/\/en.wikipedia.org\/wiki\/Secure_Shell\">SSH<\/a>\nhost keys as otherwise the image couldn't be shared between\nmultiple hosts. But if SSH host keys are auto-generated during\nthe first boot one doesn't know their \u2026<\/p>","content":"<p>When deploying a VM in a remote location one might end up in an\nimpossible situation: a VM base image doesn't contain <a href=\"https:\/\/en.wikipedia.org\/wiki\/Secure_Shell\">SSH<\/a>\nhost keys as otherwise the image couldn't be shared between\nmultiple hosts. But if SSH host keys are auto-generated during\nthe first boot one doesn't know their fingerprints and thus they\ncan't be trusted on first connect. This article describes how to\nsecurely deploy SSH host keys when deploying a Digital Ocean\nDroplet (i.e. a VM).<\/p>\n<h2 id=\"problem-description\">Problem Description<a class=\"headerlink\" href=\"#problem-description\" title=\"Permanent link\">&para;<\/a><\/h2>\n<p>Pre-build disk images usually don't contain any SSH host keys. At\nleast they shouldn't contain any host keys if they are used for\nmultiple hosts, but sometimes people get this wrong, e.g.\n<a href=\"https:\/\/wiki.hetzner.de\/index.php\/Ed25519\/en#Security_Notice_for_Ed25519_SSH_host_keys\">Hetzner (a web hoster) in 2015<\/a>, <a href=\"https:\/\/tools.cisco.com\/security\/center\/content\/CiscoSecurityAdvisory\/cisco-sa-20150625-ironport\">Cisco (a network equipment\ncompany) also in 2015<\/a> and <a href=\"https:\/\/blog.shodan.io\/duplicate-ssh-keys-everywhere\/\">many others<\/a>.<\/p>\n<p>One common approach is to generate the SSH host keys during first\nboot.<\/p>\n<p>The problem is then: how can we trust this new machine during the\nfirst SSH login? We don't know the fingerprint of any SSH host\nkey because they are newly generated, remotely. Thus, we can't\nreally trust it that way.<\/p>\n<p>One approach is to just accept any first encountered host key and\nstore its fingerprint for later verifications (<a href=\"https:\/\/en.wikipedia.org\/wiki\/Trust_on_first_use\">trust on first\nuse<\/a>). This is better then <a href=\"https:\/\/github.com\/devopsgroup-io\/vagrant-digitalocean\/issues\/289\">ignoring<\/a> the problem\ncompletely but this still makes you vulnerable against\n<a href=\"https:\/\/en.wikipedia.org\/wiki\/Man-in-the-middle_attack\">man-in-the-middle attacks<\/a>.<\/p>\n<p>Generating a new disk image with new host keys for each new host\nand transferring it to a cloud provider doesn't scale.<\/p>\n<h2 id=\"a-solution-for-digital-ocean-droplets\">A Solution for Digital Ocean Droplets<a class=\"headerlink\" href=\"#a-solution-for-digital-ocean-droplets\" title=\"Permanent link\">&para;<\/a><\/h2>\n<p>DigitalOcean (like some other cloud providers) uses\n<a href=\"https:\/\/cloudinit.readthedocs.io\/en\/latest\/\">cloud-init<\/a> for the first-boot configuration. And cloud-init\nalso supports supplying SSH host keys via the so called\n<a href=\"https:\/\/cloudinit.readthedocs.io\/en\/20.2\/topics\/format.html#cloud-config-data\">cloud-config<\/a>. Thus, we can supply a cloud-config (with\nlocally generated host keys) via the <a href=\"https:\/\/developers.digitalocean.com\/documentation\/v2\/#create-a-new-droplet\">user-data parameter<\/a>\nwhen creating a new VM via the DigitalOcean API.<\/p>\n<p>SSH host key problem solved. Almost, since it turns out that the\nuser-data cloud-config is served over a <a href=\"https:\/\/developers.digitalocean.com\/documentation\/metadata\/\">link-local address<\/a>\nfor the lifetime of the VM. Since this turns the SSH host keys\ninto world-readable information (to processes running on that VM)\nwe have to consider these cloud-init SSH host keys as ephemeral\nand have to replace them with another set (of locally generated\nSSH host keys) as part of our deployment.<\/p>\n<p>See also an example Ansible playbook of mine that <a href=\"https:\/\/github.com\/gsauthof\/playbook\/blob\/1a792ae163276b5f831225ac61c95f6b131d9c01\/fedora\/initramfs\/ansible\/digital-ocean.yml#L15\">creates a\nDigitalOcean Droplet<\/a> and <a href=\"https:\/\/github.com\/gsauthof\/playbook\/blob\/master\/fedora\/initramfs\/ansible\/digital-ocean.yml#L109\">automates the SSH host key\ndeployment<\/a> in the described way.<\/p>\n<p>In that way, <a href=\"https:\/\/en.wikipedia.org\/wiki\/Man-in-the-middle_attack\">man-in-the-middle attacks<\/a> can be successfully\naverted (and detected) during deployment as the DigitalOcean API\nis called over HTTPS (where the server's certificate is signed by\na trusted CA) and the first ephemeral SSH host key is thus\ntransferred over that secure connection. The second and final SSH\nhost key is transferred over SSH where the (ephemeral) host key\ncan be authenticated by it's known fingerprint.<\/p>","category":[{"@attributes":{"term":"linux"}},{"@attributes":{"term":"ansible"}},{"@attributes":{"term":"digitalocean"}},{"@attributes":{"term":"ssh"}}]},{"title":"NFS VM Cached Read Illustrated","link":{"@attributes":{"href":"https:\/\/gms.tf\/nfs-vm-cached-read-illustrated.html","rel":"alternate"}},"published":"2017-08-12T18:12:00+02:00","updated":"2017-08-12T18:12:00+02:00","author":{"name":"Georg Sauthoff"},"id":"tag:gms.tf,2017-08-12:\/nfs-vm-cached-read-illustrated.html","summary":"<p>This article illustrates how the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Virtual_memory\">virtual memory (VM)<\/a> subsystem of\nthe <a href=\"https:\/\/en.wikipedia.org\/wiki\/Linux\">Linux kernel<\/a> is able to cache files that are located on\nan <a href=\"https:\/\/en.wikipedia.org\/wiki\/Network_File_System\">NFS<\/a> server.<\/p>\n<div class=\"toc\">\n<ul>\n<li><a href=\"#environment\">Environment<\/a><\/li>\n<li><a href=\"#high-level-view\">High-Level View<\/a><\/li>\n<li><a href=\"#dstat\">Dstat<\/a><\/li>\n<li><a href=\"#nfs-traffic\">NFS Traffic<\/a><\/li>\n<li><a href=\"#closing-remarks\">Closing Remarks<\/a><\/li>\n<\/ul>\n<\/div>\n<h2 id=\"environment\">Environment<a class=\"headerlink\" href=\"#environment\" title=\"Permanent link\">&para;<\/a><\/h2>\n<p>In this small experiment we just need two Linux machines on a\nlocal network, where one \u2026<\/p>","content":"<p>This article illustrates how the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Virtual_memory\">virtual memory (VM)<\/a> subsystem of\nthe <a href=\"https:\/\/en.wikipedia.org\/wiki\/Linux\">Linux kernel<\/a> is able to cache files that are located on\nan <a href=\"https:\/\/en.wikipedia.org\/wiki\/Network_File_System\">NFS<\/a> server.<\/p>\n<div class=\"toc\">\n<ul>\n<li><a href=\"#environment\">Environment<\/a><\/li>\n<li><a href=\"#high-level-view\">High-Level View<\/a><\/li>\n<li><a href=\"#dstat\">Dstat<\/a><\/li>\n<li><a href=\"#nfs-traffic\">NFS Traffic<\/a><\/li>\n<li><a href=\"#closing-remarks\">Closing Remarks<\/a><\/li>\n<\/ul>\n<\/div>\n<h2 id=\"environment\">Environment<a class=\"headerlink\" href=\"#environment\" title=\"Permanent link\">&para;<\/a><\/h2>\n<p>In this small experiment we just need two Linux machines on a\nlocal network, where one acts as <a href=\"https:\/\/en.wikipedia.org\/wiki\/Network_File_System\">NFS<\/a> server and the other\none as NFS client. With a current Linux distribution like\n<a href=\"https:\/\/en.wikipedia.org\/wiki\/Fedora_(operating_system)\">Fedora<\/a> 26 you get <a href=\"https:\/\/en.wikipedia.org\/wiki\/Network_File_System#NFSv4\">NFSv4<\/a> if you don't do\nanything special.<\/p>\n<p>The following results are from a <a href=\"https:\/\/en.wikipedia.org\/wiki\/Gigabit_Ethernet#1000BASE-T\">1 GBit ethernet<\/a> network where the\nNFS client has 16 GB RAM and both machines run Fedora 26.<\/p>\n<h2 id=\"high-level-view\">High-Level View<a class=\"headerlink\" href=\"#high-level-view\" title=\"Permanent link\">&para;<\/a><\/h2>\n<p>A simple way to read over the network from an NFS filesystem\nwithout yielding disk IO on the client is:<\/p>\n<ol>\n<li>empty the VM cache - e.g. rebooting the client and server\n  or via executing <code>free &amp;&amp; sync &amp;&amp; echo 3 &gt; \/proc\/sys\/vm\/drop_caches &amp;&amp;\n  free<\/code> on both machines<\/li>\n<li>on the client: <code>cat<\/code> a relatively large file from a NFS\n   filesystem to <code>\/dev\/null<\/code><\/li>\n<\/ol>\n<p>The file should be large enough such that we can easily measure\nsome metrics but small enough to fit into RAM.<\/p>\n<p>In our environment the 4.2 GB backup copy of the <a href=\"https:\/\/en.wikipedia.org\/wiki\/John_Woo\">John Woo<\/a> movie\n<a href=\"https:\/\/en.wikipedia.org\/wiki\/The_Killer_(1989_film)\">'The Killer'<\/a> is well suited:<\/p>\n<div class=\"m-code\"><pre><span><\/span><code><span class=\"gp\">$ <\/span><span class=\"nb\">time<\/span><span class=\"w\"> <\/span>cat<span class=\"w\"> <\/span>The_Killer.iso<span class=\"w\"> <\/span>&gt;<span class=\"w\"> <\/span>\/dev\/null\n<\/code><\/pre><\/div>\n\n<p>The <a href=\"https:\/\/en.wikipedia.org\/wiki\/Z_shell\">zsh<\/a> time builtin reports:<\/p>\n<div class=\"m-code\"><pre><span><\/span><code><span class=\"mf\">0.00<\/span><span class=\"n\">s<\/span><span class=\"w\"> <\/span><span class=\"n\">user<\/span><span class=\"w\"> <\/span><span class=\"mf\">1.08<\/span><span class=\"n\">s<\/span><span class=\"w\"> <\/span><span class=\"kr\">sys<\/span><span class=\"n\">tem<\/span><span class=\"w\"> <\/span><span class=\"mf\">2<\/span><span class=\"err\">%<\/span><span class=\"w\"> <\/span><span class=\"n\">cpu<\/span><span class=\"w\"> <\/span><span class=\"mf\">38.373<\/span><span class=\"w\"> <\/span><span class=\"kr\">to<\/span><span class=\"n\">tal<\/span>\n<\/code><\/pre><\/div>\n\n<p>That means it took 38 seconds to transfer that 4.2 GiB file. This is\nplausible because the resulting read rate of 113 MiB\/s is close\nto the wire speed (<code>10^9\/8\/1024\/1024<\/code>) minus the expected protocol overhead.<\/p>\n<p>Executing the read command<\/p>\n<div class=\"m-code\"><pre><span><\/span><code><span class=\"gp\">$ <\/span><span class=\"nb\">time<\/span><span class=\"w\"> <\/span>cat<span class=\"w\"> <\/span>The_Killer.iso<span class=\"w\"> <\/span>&gt;<span class=\"w\"> <\/span>\/dev\/null\n<\/code><\/pre><\/div>\n\n<p>a second and third time is way faster:<\/p>\n<div class=\"m-code\"><pre><span><\/span><code><span class=\"mf\">0.00<\/span><span class=\"n\">s<\/span><span class=\"w\"> <\/span><span class=\"n\">user<\/span><span class=\"w\"> <\/span><span class=\"mf\">0.57<\/span><span class=\"n\">s<\/span><span class=\"w\"> <\/span><span class=\"kr\">sys<\/span><span class=\"n\">tem<\/span><span class=\"w\"> <\/span><span class=\"mf\">99<\/span><span class=\"err\">%<\/span><span class=\"w\"> <\/span><span class=\"n\">cpu<\/span><span class=\"w\"> <\/span><span class=\"mf\">0.578<\/span><span class=\"w\"> <\/span><span class=\"kr\">to<\/span><span class=\"n\">tal<\/span>\n<span class=\"mf\">0.00<\/span><span class=\"n\">s<\/span><span class=\"w\"> <\/span><span class=\"n\">user<\/span><span class=\"w\"> <\/span><span class=\"mf\">0.44<\/span><span class=\"n\">s<\/span><span class=\"w\"> <\/span><span class=\"kr\">sys<\/span><span class=\"n\">tem<\/span><span class=\"w\"> <\/span><span class=\"mf\">98<\/span><span class=\"err\">%<\/span><span class=\"w\"> <\/span><span class=\"n\">cpu<\/span><span class=\"w\"> <\/span><span class=\"mf\">0.447<\/span><span class=\"w\"> <\/span><span class=\"kr\">to<\/span><span class=\"n\">tal<\/span>\n<\/code><\/pre><\/div>\n\n<p>Just half a second.<\/p>\n<p>After clearing the VM cache with<\/p>\n<div class=\"m-code\"><pre><span><\/span><code><span class=\"gh\">#<\/span> free &amp;&amp; sync &amp;&amp; echo 3 &gt; \/proc\/sys\/vm\/drop_caches &amp;&amp; free\n<\/code><\/pre><\/div>\n\n<p>where the final free reports<\/p>\n<div class=\"m-code\"><pre><span><\/span><code>buff\/cache\n1871568\n<\/code><\/pre><\/div>\n\n<p>and reading the file just another time we are back at the initial\ntiming:<\/p>\n<div class=\"m-code\"><pre><span><\/span><code><span class=\"mf\">0.02<\/span><span class=\"n\">s<\/span><span class=\"w\"> <\/span><span class=\"n\">user<\/span><span class=\"w\"> <\/span><span class=\"mf\">1.26<\/span><span class=\"n\">s<\/span><span class=\"w\"> <\/span><span class=\"kr\">sys<\/span><span class=\"n\">tem<\/span><span class=\"w\"> <\/span><span class=\"mf\">3<\/span><span class=\"err\">%<\/span><span class=\"w\"> <\/span><span class=\"n\">cpu<\/span><span class=\"w\"> <\/span><span class=\"mf\">38.551<\/span><span class=\"w\"> <\/span><span class=\"kr\">to<\/span><span class=\"n\">tal<\/span>\n<\/code><\/pre><\/div>\n\n<p>A <code>free<\/code> then reports<\/p>\n<div class=\"m-code\"><pre><span><\/span><code>buff\/cache\n6267676\n<\/code><\/pre><\/div>\n\n<p>This is plausible as well, i.e. <code>(6267676-1871568)\/1024\/1024<\/code>\nis close to the 4.2 GB file size.<\/p>\n<p>If we have a rough idea how the VM subsystem works with a local\nfilsystem in a common OS kernel (like Linux) all this isn't\nsurprising. One just might wonder how the kernel decides that it\nis safe to reuse the VM cache since the file could have changed\non the server (e.g. locally or by another NFS client).<\/p>\n<p>We can verify that the kernel is able to detect file changes by\nother parties via touching the file on the server, e.g.:<\/p>\n<div class=\"m-code\"><pre><span><\/span><code><span class=\"gp\">$ <\/span>touch<span class=\"w\"> <\/span>The_Killer.iso\n<\/code><\/pre><\/div>\n\n<p>After that, reading the file again on the client takes the initial\namount of time (39 seconds).<\/p>\n<h2 id=\"dstat\">Dstat<a class=\"headerlink\" href=\"#dstat\" title=\"Permanent link\">&para;<\/a><\/h2>\n<p>When running the <a href=\"http:\/\/dag.wiee.rs\/home-made\/dstat\/\"><code>dstat<\/code> command<\/a> on the client we can obtain a\nmore detailed picture. For example:<\/p>\n<div class=\"m-code\"><pre><span><\/span><code><span class=\"gp\">$ <\/span>dstat<span class=\"w\"> <\/span>-d<span class=\"w\"> <\/span>-D<span class=\"w\"> <\/span>total<span class=\"w\"> <\/span>-g<span class=\"w\"> <\/span>-n<span class=\"w\"> <\/span>-N<span class=\"w\"> <\/span>enp0s31f6<span class=\"w\"> <\/span>-r<span class=\"w\"> <\/span>-c<span class=\"w\"> <\/span>--vm\n<\/code><\/pre><\/div>\n\n<p>By default, <code>dstat<\/code> repeatedly averages over 1 second, i.e. it prints every\nsecond a new line.<\/p>\n<p>The dstat output during the first file read:<\/p>\n<p><img alt=\"Dstat output during first read\" src=\"https:\/\/gms.tf\/image\/dstat-first.png\"><\/p>\n<p>We can observe a few things in the listing:<\/p>\n<ul>\n<li>the network receive rate matches the computed read rate (modulo\n  protocol overhead) and since reads need to be acknowledged\n  the outgoing network traffic increases as well<\/li>\n<li>directly after starting the read the VM heavily allocates and\n  frees caches pages (in the order of <code>60 * 10^3<\/code> pages or so)<\/li>\n<\/ul>\n<p>The work the VM subsystem is doing is plausible. The system has a\npage size of 4 KiB and thus the amount of cache pages is\nsufficient to deal with the read rate.<\/p>\n<p>The listing for the second read is much shorter:<\/p>\n<p><img alt=\"Dstat output during second read\" src=\"https:\/\/gms.tf\/image\/dstat-cached.png\"><\/p>\n<p>Here we don't see much network traffic, but we see a clear\nincrease in traffic that could be explained by some NFS\ncommunication where the client inquires the server whether the\nfile is outdated or not.<\/p>\n<p>Also, as expected, the VM subsystem isn't busy with allocations\nand frees as before.<\/p>\n<h2 id=\"nfs-traffic\">NFS Traffic<a class=\"headerlink\" href=\"#nfs-traffic\" title=\"Permanent link\">&para;<\/a><\/h2>\n<p>To get an idea how the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Network_File_System\">NFS<\/a> client detects file changes we can\ncapture the network traffic and inspect with an analyzer like\n<a href=\"https:\/\/en.wikipedia.org\/wiki\/Wireshark\">Wireshark<\/a>.<\/p>\n<p>In case directly capturing with Wireshark isn't an option, a\nsimple alternative is to capture with <a href=\"https:\/\/en.wikipedia.org\/wiki\/Tcpdump\">tcpdump<\/a> to a file and open\nthat file later with wireshark, e.g.:<\/p>\n<div class=\"m-code\"><pre><span><\/span><code><span class=\"gp\"># <\/span>tcpdump<span class=\"w\"> <\/span>-i<span class=\"w\"> <\/span>enp0s31f6<span class=\"w\"> <\/span>-s0<span class=\"w\"> <\/span>-w<span class=\"w\"> <\/span>\/var\/tmp\/nfs.dump\n<\/code><\/pre><\/div>\n\n<p>The NFS flow for the second file read is quite compact in the\nwireshark overview:<\/p>\n<p><img alt=\"Wireshark NFS overview\" src=\"https:\/\/gms.tf\/image\/wireshark-nfs-overview.png\"><\/p>\n<p>As expected, it is <a href=\"https:\/\/en.wikipedia.org\/wiki\/Network_File_System#NFSv4\">NFS version 4<\/a> and there are just a few query\noperations.<\/p>\n<p>In detail the second <code>GETATTR<\/code> reply:<\/p>\n<p><img alt=\"Wireshark NFS details\" src=\"https:\/\/gms.tf\/image\/wireshark-nfs-detail.png\"><\/p>\n<p>We observe that the server replies that the file is a regular\nfile (<code>NF4REG<\/code>) and reports its change-ID and file-ID.<\/p>\n<p>The file-ID is conceptually similar to an <a href=\"https:\/\/en.wikipedia.org\/wiki\/Inode\">inode<\/a> and thus the NFS\nclient can compare the change-ID with the one obtained from last\ntime to decide if it can reuse the pages from the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Virtual_memory\">VM cache<\/a> or if\nit has to retrieve the change file from the server.<\/p>\n<p>Sure enough, after touching the file the server sends a new\nchange-ID, i.e. 17002 after the first touch and then 17003 after\nthe second touch. As expected, the file-ID doesn't change.<\/p>\n<p>Why does the client issues two <code>GETTATTR<\/code> requests? The first is for\nthe directory the file is located in (<code>NF4DIR<\/code>).<\/p>\n<h2 id=\"closing-remarks\">Closing Remarks<a class=\"headerlink\" href=\"#closing-remarks\" title=\"Permanent link\">&para;<\/a><\/h2>\n<p>So when does the VM caching matter? Or, how can accidentally work\nagainst this mechanism when using NFS?<\/p>\n<p>Say, entirely hypothetical, you have a <a href=\"https:\/\/en.wikipedia.org\/wiki\/Computer_cluster\">compute cluster<\/a> where\ntypical compute jobs are actually program chains where the output\nof one program is the input of the next one.  Also assume that\nthe working directories are shared via NFS for flexibility and\nredundancy. For example, when one cluster node crashes in the\nmiddle of a chain then the chain can be completed on another node\nwithout having to recompute everything.<\/p>\n<p>In that scenario, the best case would be: the complete chain is\nexecuted on one cluster node. A <a href=\"https:\/\/en.wikipedia.org\/wiki\/Job_scheduler\">job-scheduler<\/a> only schedules the\ncomplete chain to a cluster node. It doesn't re-schedule\nsingle programs of a chain between cluster nodes. Then the\ninput for the program next in chain comes from VM subsystem and\ndoesn't need to be transmitted over the network.<\/p>\n<p>In contrast to that, since the work directories are shared, it is\nalso possible to have finer job scheduling granularity, i.e. for\neach program of a chain. But this would increase the load on the\nNFS server.  In the worst-case the NFS server's outgoing network\ntraffic would multiply because the result of one chain program\nexecuted on cluster node A would have to be retrieved from the\nNFS server by the next chain program executed on cluster node B.<\/p>\n<p>Similar reasoning applies to other network filesystems (and other\noperating systems with a VM subsystem) and thus\nit makes sense to build a chain-to-node-affinity mechanism into the job\nscheduler such that the VM cache re-use is maximized in such a\nchain-job environment.<\/p>","category":[{"@attributes":{"term":"linux"}},{"@attributes":{"term":"nfs"}}]},{"title":"Check CPU Microcode Version on Linux","link":{"@attributes":{"href":"https:\/\/gms.tf\/check-cpu-microcode-version-on-linux.html","rel":"alternate"}},"published":"2017-07-01T10:10:00+02:00","updated":"2017-07-01T10:10:00+02:00","author":{"name":"Georg Sauthoff"},"id":"tag:gms.tf,2017-07-01:\/check-cpu-microcode-version-on-linux.html","summary":"<p>Debian recently released a <a href=\"https:\/\/lists.debian.org\/debian-devel\/2017\/06\/msg00308.html\">warning advisory regarding a hardware\nbug in the hyperthreading implementation on certain Skylake\/Kaby\nLake Intel CPUs<\/a>. It can be mitigated by a <a href=\"https:\/\/en.wikipedia.org\/wiki\/Microcode\">microcode<\/a> firmware\nupdate - at least on some models. This article is about checking\na non-Debian system for this issue and verifying if the \u2026<\/p>","content":"<p>Debian recently released a <a href=\"https:\/\/lists.debian.org\/debian-devel\/2017\/06\/msg00308.html\">warning advisory regarding a hardware\nbug in the hyperthreading implementation on certain Skylake\/Kaby\nLake Intel CPUs<\/a>. It can be mitigated by a <a href=\"https:\/\/en.wikipedia.org\/wiki\/Microcode\">microcode<\/a> firmware\nupdate - at least on some models. This article is about checking\na non-Debian system for this issue and verifying if the right\nmicrocode is in place.<\/p>\n<p>The below console snippets are from a Dell notebook with Skylake\nCPU that runs Fedora 25. It turns out that Fedora, in contrast to\nDebian, installs some firmware packages, by default, including\nIntel microcode images (cf. <code>microcode_ctl<\/code>). Thus, a Fedora\nsystem on Skylake should already run with the microcode version\nthat includes the fix.<\/p>\n<p>Check for certain CPU features - e.g. if hypterthreading is\nenabled:<\/p>\n<div class=\"m-code\"><pre><span><\/span><code><span class=\"gp\">$ <\/span>grep<span class=\"w\"> <\/span><span class=\"s1\">&#39;\\&lt;ht\\&gt;&#39;<\/span><span class=\"w\"> <\/span>\/proc\/cpuinfo<span class=\"w\"> <\/span>-q<span class=\"w\"> <\/span><span class=\"o\">&amp;&amp;<\/span><span class=\"w\"> <\/span><span class=\"nb\">echo<\/span><span class=\"w\"> <\/span><span class=\"m\">1<\/span>\n<span class=\"go\">1<\/span>\n<\/code><\/pre><\/div>\n\n<p>Display CPU model, stepping and active microcode version:<\/p>\n<div class=\"m-code\"><pre><span><\/span><code><span class=\"gp\">$ <\/span>grep<span class=\"w\"> <\/span><span class=\"s1\">&#39;stepping\\|model\\|microcode&#39;<\/span><span class=\"w\"> <\/span>\/proc\/cpuinfo\n<span class=\"go\">model       : 78<\/span>\n<span class=\"go\">model name  : Intel(R) Core(TM) i7-6600U CPU @ 2.60GHz<\/span>\n<span class=\"go\">stepping    : 3<\/span>\n<span class=\"go\">microcode   : 0xba<\/span>\n<span class=\"go\">[..]<\/span>\n<\/code><\/pre><\/div>\n\n<p>The advisory states that Skylake model 78, stepping 3 is affected\nand that microcode version <code>0xba<\/code> includes the fix.<\/p>\n<p>Investigate whether the microcode was loaded by the BIOS or by the kernel:<\/p>\n<div class=\"m-code\"><pre><span><\/span><code><span class=\"gp\"># <\/span>journalctl<span class=\"w\"> <\/span>--no-hostname<span class=\"w\"> <\/span>-o<span class=\"w\"> <\/span>short-monotonic<span class=\"w\"> <\/span>--boot<span class=\"w\"> <\/span>-0<span class=\"w\"> <\/span><span class=\"se\">\\<\/span>\n<span class=\"w\">  <\/span><span class=\"p\">|<\/span><span class=\"w\"> <\/span>sed<span class=\"w\"> <\/span>-n<span class=\"w\"> <\/span><span class=\"s1\">&#39;1,\/PM: Preparing system for sleep\/p&#39;<\/span><span class=\"w\"> <\/span><span class=\"p\">|<\/span><span class=\"w\"> <\/span>grep<span class=\"w\"> <\/span><span class=\"s1\">&#39;microcode\\|smp&#39;<\/span>\n<span class=\"go\">[    0.000000] microcode: microcode updated early to revision 0xba, \\<\/span>\n<span class=\"go\">                   date = 2017-04-09<\/span>\n<span class=\"go\">[    0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs<\/span>\n<span class=\"go\">[    0.040087] smpboot: Max logical packages: 2<\/span>\n<span class=\"go\">[    0.056402] smpboot: CPU0: Intel(R) Core(TM) i7-6600U CPU @ 2.60GHz \\<\/span>\n<span class=\"go\">                   (family: 0x6, model: 0x4e, stepping: 0x3)<\/span>\n<span class=\"go\">[    0.056975] smp: Bringing up secondary CPUs ...<\/span>\n<span class=\"go\">[    0.239023] smp: Brought up 1 node, 4 CPUs<\/span>\n<span class=\"go\">[    0.239023] smpboot: Total of 4 processors activated (22476.05 BogoMIPS)<\/span>\n<span class=\"go\">[    0.693589] microcode: sig=0x406e3, pf=0x80, revision=0xba<\/span>\n<span class=\"go\">[    0.693671] microcode: Microcode Update Driver: v2.2.<\/span>\n<\/code><\/pre><\/div>\n\n<p>That means the microcode image in the BIOS is older, thus, the\nKernel loaded a newer version (the microcode date also matches\nthe one mentioned in the advisory).<\/p>\n<p>After a BIOS update, check that the BIOS loads the microcode\nbefore kernel start:<\/p>\n<div class=\"m-code\"><pre><span><\/span><code><span class=\"gp\"># <\/span>journalctl<span class=\"w\"> <\/span>--no-hostname<span class=\"w\"> <\/span>-o<span class=\"w\"> <\/span>short-monotonic<span class=\"w\"> <\/span>--boot<span class=\"w\"> <\/span>-0<span class=\"w\"> <\/span><span class=\"se\">\\<\/span>\n<span class=\"w\">    <\/span><span class=\"p\">|<\/span><span class=\"w\"> <\/span>sed<span class=\"w\"> <\/span>-n<span class=\"w\"> <\/span><span class=\"s1\">&#39;1,\/PM: Preparing system for sleep\/p&#39;<\/span><span class=\"w\"> <\/span><span class=\"p\">|<\/span><span class=\"w\"> <\/span>grep<span class=\"w\"> <\/span><span class=\"s1\">&#39;microcode\\|smp&#39;<\/span>\n<span class=\"go\">[    0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs<\/span>\n<span class=\"go\">[    0.039057] smpboot: Max logical packages: 2<\/span>\n<span class=\"go\">[    0.054709] smpboot: CPU0: Intel(R) Core(TM) i7-6600U CPU @ 2.60GHz \\<\/span>\n<span class=\"go\">                   (family: 0x6, model: 0x4e, stepping: 0x3)<\/span>\n<span class=\"go\">[    0.055000] smp: Bringing up secondary CPUs ...<\/span>\n<span class=\"go\">[    0.235034] smp: Brought up 1 node, 4 CPUs<\/span>\n<span class=\"go\">[    0.235034] smpboot: Total of 4 processors activated (22560.93 BogoMIPS)<\/span>\n<span class=\"go\">[    0.711458] microcode: sig=0x406e3, pf=0x80, revision=0xba<\/span>\n<span class=\"go\">[    0.711662] microcode: Microcode Update Driver: v2.2.<\/span>\n<\/code><\/pre><\/div>\n\n<p>(there is no <code>microcode udpated early<\/code> message, anymore)<\/p>\n<p>Modern CPU <a href=\"https:\/\/en.wikipedia.org\/wiki\/Microcode\">microcode<\/a>, like used by Intel CPUs, isn't persistent,\ni.e. it must be loaded at each boot, either by the BIOS or by the\nkernel. Apparently, it even has to be reloaded during resume\nafter a suspend-to-RAM (cf. the kernel messages).<\/p>","category":[{"@attributes":{"term":"linux"}},{"@attributes":{"term":"firmware"}},{"@attributes":{"term":"dell"}}]},{"title":"Fedora as Grml replacement","link":{"@attributes":{"href":"https:\/\/gms.tf\/fedora-as-grml-replacement.html","rel":"alternate"}},"published":"2017-03-12T21:00:00+01:00","updated":"2017-03-12T21:00:00+01:00","author":{"name":"Georg Sauthoff"},"id":"tag:gms.tf,2017-03-12:\/fedora-as-grml-replacement.html","summary":"<p><a href=\"https:\/\/grml.org\/\">Grml<\/a> is a Linux Live CD distribution geared towards system\nadministration tasks and other console work. Unfortunately, its\ndevelopment has slowed down in the past years, i.e. the latest\nstable release is from 2014. In 2017, when dealing with modern\nhardware, modern features like Btrfs, SELinux etc. this is \u2026<\/p>","content":"<p><a href=\"https:\/\/grml.org\/\">Grml<\/a> is a Linux Live CD distribution geared towards system\nadministration tasks and other console work. Unfortunately, its\ndevelopment has slowed down in the past years, i.e. the latest\nstable release is from 2014. In 2017, when dealing with modern\nhardware, modern features like Btrfs, SELinux etc. this is a deal\nbreaker. It turns out that <a href=\"https:\/\/en.wikipedia.org\/wiki\/Fedora_(operating_system)\">Fedora<\/a>, when installed on a USB mass\nstorage device, is a good substitute for traditional Linux Live\nCDs like Grml.<\/p>\n<h2 id=\"how-it-works\">How it works<a class=\"headerlink\" href=\"#how-it-works\" title=\"Permanent link\">&para;<\/a><\/h2>\n<p>A very fast and convenient method to install Fedora on a USB\nstick is to use <a href=\"http:\/\/libguestfs.org\/virt-builder.1.html\"><code>virt-builder<\/code><\/a>, e.g.:<\/p>\n<div class=\"m-code\"><pre><span><\/span><code>$<span class=\"w\"> <\/span>virt-builder<span class=\"w\"> <\/span>fedora-25<span class=\"w\"> <\/span>--hostname<span class=\"w\"> <\/span>rescue.example.org<span class=\"w\">  <\/span><span class=\"se\">\\<\/span>\n--ssh-inject<span class=\"w\"> <\/span>root:file:<span class=\"s2\">&quot;<\/span><span class=\"nv\">$pubkey<\/span><span class=\"s2\">&quot;<\/span><span class=\"w\"> <\/span>--root-password<span class=\"w\"> <\/span>file:<span class=\"s2\">&quot;<\/span><span class=\"nv\">$pwfile<\/span><span class=\"s2\">&quot;<\/span><span class=\"w\"> <\/span><span class=\"se\">\\<\/span>\n--update<span class=\"w\"> <\/span>--install<span class=\"w\"> <\/span><span class=\"k\">$(<\/span>paste<span class=\"w\"> <\/span>-d,<span class=\"w\"> <\/span>-s<span class=\"w\"> <\/span>package.list<span class=\"k\">)<\/span><span class=\"w\"> <\/span><span class=\"se\">\\<\/span>\n--run<span class=\"w\"> <\/span>guest-setup.sh<span class=\"w\"> <\/span>--selinux-relabel<span class=\"w\"> <\/span><span class=\"se\">\\<\/span>\n-o<span class=\"w\"> <\/span>\/dev\/sdz\n<\/code><\/pre><\/div>\n\n<p>The <a href=\"http:\/\/libguestfs.org\/virt-builder.1.html\"><code>virt-builder<\/code> command<\/a> is probably mostly used for fast\ngeneration of virtual machine images but it works as well for\nbare-metal installations. It is so fast because it starts from\nfilesystem-templates.<\/p>\n<p>In the above example, the shell script <code>guest-setup.sh<\/code> applies\nsome defaults and other post-installation customizations in the\ntarget system.<\/p>\n<p>The result can be easily tested with <a href=\"http:\/\/wiki.qemu-project.org\/Features\/KVM\">QEMU\/KVM<\/a>, e.g.:<\/p>\n<div class=\"m-code\"><pre><span><\/span><code>$<span class=\"w\"> <\/span>qemu-system-x86_64<span class=\"w\"> <\/span>-enable-kvm<span class=\"w\"> <\/span>-drive<span class=\"w\"> <\/span><span class=\"nv\">file<\/span><span class=\"o\">=<\/span>\/dev\/sdz,if<span class=\"o\">=<\/span>virtio,format<span class=\"o\">=<\/span>raw<span class=\"w\"> <\/span><span class=\"se\">\\<\/span>\n<span class=\"w\">  <\/span>-m<span class=\"w\"> <\/span><span class=\"m\">2048<\/span><span class=\"w\"> <\/span>-netdev<span class=\"w\"> <\/span>bridge,id<span class=\"o\">=<\/span>nd0,name<span class=\"o\">=<\/span>tap0,br<span class=\"o\">=<\/span>virbr0<span class=\"w\"> <\/span><span class=\"se\">\\<\/span>\n<span class=\"w\">  <\/span>-device<span class=\"w\"> <\/span>e1000,netdev<span class=\"o\">=<\/span>nd0,id<span class=\"o\">=<\/span>d0<span class=\"w\">  <\/span>-display<span class=\"w\"> <\/span>curses\n<\/code><\/pre><\/div>\n\n<h2 id=\"the-code\">The Code<a class=\"headerlink\" href=\"#the-code\" title=\"Permanent link\">&para;<\/a><\/h2>\n<p>I've <a href=\"https:\/\/github.com\/gsauthof\/playbook\/tree\/master\/fedora\/rescue-stick\">published some scripts to generate such a Fedora based USB\nstick in a Git repository<\/a>. The repository also contains some\nmore details in a README including a detailed comparison with\nGrml.<\/p>\n<h2 id=\"addendum\">Addendum<a class=\"headerlink\" href=\"#addendum\" title=\"Permanent link\">&para;<\/a><\/h2>\n<p>Also, nowadays, the traditional Linux Live CD approach is a\nlittle bit outdated. Many systems don't even have a CDROM drive\nanymore for good reason. Sophisticated distributions like Grml\nare hybrid, i.e. they can also be booted from a USB stick, but\nthey don't make use of the extra space or the fact that the stick\nis writable, by default. Grml even supports customizations like\nadding a writable filesystem but such changes are kind of\ntedious.<\/p>\n<p>Grml doesn't just come with good defaults (e.g. zsh login shell),\nthe team also has a talent choosing great release names (e.g.\n<a href=\"https:\/\/grml.org\/changelogs\/README-grml-2011.12\/\">Knecht Rootrecht<\/a>).<\/p>","category":[{"@attributes":{"term":"linux"}},{"@attributes":{"term":"rescue"}},{"@attributes":{"term":"playbook"}},{"@attributes":{"term":"virt-builder"}}]},{"title":"Btrfs requires noatime","link":{"@attributes":{"href":"https:\/\/gms.tf\/btrfs-requires-noatime.html","rel":"alternate"}},"published":"2017-02-08T21:10:00+01:00","updated":"2017-02-08T21:10:00+01:00","author":{"name":"Georg Sauthoff"},"id":"tag:gms.tf,2017-02-08:\/btrfs-requires-noatime.html","summary":"<p>Traditionally, UNIX filesystems also maintain the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Stat_(system_call)\">access time\n(atime)<\/a> of a file. This is very much an anti-feature because it\nyields a write operation for each read operation. Which is\nobviously bad for performance. Since <a href=\"https:\/\/en.wikipedia.org\/wiki\/Btrfs\">Btrfs<\/a> is a <a href=\"https:\/\/en.wikipedia.org\/wiki\/Copy-on-write\">copy-on-write\n(COW)<\/a> filesystem maintaining atimes is even more painful. Thus, a\nsensible \u2026<\/p>","content":"<p>Traditionally, UNIX filesystems also maintain the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Stat_(system_call)\">access time\n(atime)<\/a> of a file. This is very much an anti-feature because it\nyields a write operation for each read operation. Which is\nobviously bad for performance. Since <a href=\"https:\/\/en.wikipedia.org\/wiki\/Btrfs\">Btrfs<\/a> is a <a href=\"https:\/\/en.wikipedia.org\/wiki\/Copy-on-write\">copy-on-write\n(COW)<\/a> filesystem maintaining atimes is even more painful. Thus, a\nsensible recommendation is to make sure that every filesystem is\nmounted with the <code>noatime<\/code> option.  Especially, if it is a Btrfs\nfilesystem.<\/p>\n<h2 id=\"btrfs-case-study\">Btrfs Case Study<a class=\"headerlink\" href=\"#btrfs-case-study\" title=\"Permanent link\">&para;<\/a><\/h2>\n<p>A typical example of how writing the atime can significantly\naffect performance.<\/p>\n<p>The symptoms: an <a href=\"https:\/\/en.wikipedia.org\/wiki\/Rsync\">rsync<\/a> job from a Btrfs filesystem\n(located on an SSD) to an external USB 3 disk drive with XFS runs\nwith ~ 17 MiB\/s while <code>iotop<\/code> and <code>dstat<\/code> show significantly\nhigher IO values for the source device, e.g. ~ 60 MiB\/s.<\/p>\n<p>Reasons: This Btrfs filesystem is mounted with default options,\nthat means that instead of <code>noatime<\/code> the <code>relatime<\/code> is active,\nresulting in atime updates for effectively each file.<\/p>\n<p>In addition to that, the filesystem contains some snapshots, i.e.\nfor all subvolumes there is also one or more snapshots. Creating\na filesystem snapshot is very cheap on <a href=\"https:\/\/en.wikipedia.org\/wiki\/Btrfs\">Btrfs<\/a> because of\nits <a href=\"https:\/\/en.wikipedia.org\/wiki\/Copy-on-write\">COW<\/a> design. An atime update of a snapshotted file only\nupdates the atime of one copy. Thus, in the likely case that the\nfilesystem sector with the atime is still shared it has to be\ncopied on that atime write. Meaning 2 or more write operations as\na result of one read.<\/p>\n<p>Resolution: After canceling the rsync command,\nremounting the Btrfs filesystem with <code>noatime<\/code> and restarting\nthe rsync job it sure enough performs much better, i.e.\nthe numbers reported by rsync match the dstat ones, as expected.<\/p>\n<p>Note that the number of files included in a snapshot is\nmost relevant for this issue, not necessarily the number of\nsnapshots.<\/p>\n<p>If you are really unlucky, running a backup job (or even just a\ngrep) on a Btrfs filesystem mounted without noatime might even\nyield out-of-space errors in case all the copy-on-write atime\nupdates exceed the available free space.<\/p>\n<h2 id=\"relatime\">Relatime<a class=\"headerlink\" href=\"#relatime\" title=\"Permanent link\">&para;<\/a><\/h2>\n<p>Since 2009 or so Linux kernels by default mount filesystems with\nthe <code>relatime<\/code> option. With this options, the atime is updated\nsomewhat less frequently. That means under the presence of reads,\nit is updated at least once a day:<\/p>\n<blockquote>\n<div class=\"m-code\"><pre><span><\/span><code>relatime\n   Update  inode access times relative to modify or change\n   time.  Access time is only updated if the previous\n   access time was earlier than the current modify or\n   change time.  (Similar to noatime, but it doesn&#39;t break\n   mutt or other applications that need to know  if  a\n   file has been read since the last time it was modified.)\n\n   Since  Linux  2.6.30,  the  kernel defaults to the\n   behavior provided by this option (unless noatime was\n   specified), and the strictatime option is required to\n   obtain traditional semantics.  In addition, since Linux\n   2.6.30, the file&#39;s last access time is always updated\n   if it is more than 1 day old.\n<\/code><\/pre><\/div>\n\n<\/blockquote>\n<p>(<a href=\"https:\/\/www.freebsd.org\/cgi\/man.cgi?query=mount&amp;apropos=0&amp;sektion=8&amp;manpath=CentOS+7.1&amp;arch=default&amp;format=html\">mount(8)<\/a>)<\/p>\n<p>Thus, <code>relatime<\/code> doesn't help much with the principal cause for\nthe described performance issue. You still likely get massive\natime update induced writes during bulk read-only activity like\nbackup or search jobs. You only have to experience a timespan\ngreater than 24h were most files aren't read. A pretty standard\nscenario.<\/p>\n<p>Unfortunately, switching the system default from <code>relatime<\/code> to <code>noatime<\/code> isn't\npossible (as of 2017-02). Thus, one has to remember to always specify\n<code>noatime<\/code>. Think <code>mount -o noatime ...<\/code> and add it to each <code>\/etc\/fstab<\/code>\nentry.<\/p>\n<h2 id=\"testing\">Testing<a class=\"headerlink\" href=\"#testing\" title=\"Permanent link\">&para;<\/a><\/h2>\n<p>Testing the effect of atime uptime in the presence of snapshots\nis easier when the Btrfs filesystem is mounted with\n<code>strictatime<\/code>. This overrides the <code>relatime<\/code> default option.\nThe time attributes of a file or directory can be displayed with\n<code>ls<\/code> but the GNU utility <code>stat<\/code> is more convenient for printing\nall timestamps.<\/p>\n<h2 id=\"ps\">P.S.<a class=\"headerlink\" href=\"#ps\" title=\"Permanent link\">&para;<\/a><\/h2>\n<p>There are use-cases for an actively maintained file atime.\nExamples are measuring the usage of installed binaries (cf.\nDebian's popcontest) or detecting unread mails. But there are\nbetter alternatives to implement such tasks and thus those aren't\nvery convincing arguments for enabling atime writing, by default\n(be it relative or even strict). In conclusion, writing the\naccess time means much pain with little gain.<\/p>\n<h2 id=\"see-also\">See also<a class=\"headerlink\" href=\"#see-also\" title=\"Permanent link\">&para;<\/a><\/h2>\n<ul>\n<li><a href=\"https:\/\/lwn.net\/Articles\/499293\/\">Atime and btrfs: a bad combination?<\/a> - this is also an\n  counter-example to <a href=\"https:\/\/en.wikipedia.org\/wiki\/Betteridge's_law_of_headlines\">Betteridge's law of headlines<\/a><\/li>\n<\/ul>","category":[{"@attributes":{"term":"linux"}},{"@attributes":{"term":"btrfs"}}]}]}