The init framework — which bootstraps the userspace environment and manages all the background "daemon" processes that are needed in a healthy GNU/Linux system — is based on the s6 suite of programs by Laurent Bercot. This uses a bunch of tiny little programs to manage the basic userspace of the system, rather than a few really big ones — looking at the process table on a basic LBL system, there are 38 s6-related processes running, adding up to a total of 6725 pages of RAM.^[1] In contrast, looking at a systemd-managed system (booted into single-user mode so that an absolutely minimal set of programs is running), there are only seven processes running — but they occupy a total of 23,480 pages of memory.

Rather than maintaining a central database of installed packages, and providing specialized package-management tools to query that database to discover things like what files are part of a package or what package a specific file belongs to, LBL simply creates a separate user account for each package. The files installed by each package are owned by the package-specific user, so standard system utilities can be used to determine package information. To find out what package is responsible for a file, for example, you can just use ls -l; or you can find / -user …​ to list all the files and directories owned by a package.

I have, for years, had the habit of maintaining a version control repository of configuration files for the programs I use a lot — especially programs like bash and vim and tmux, whose configurations I have customized, in some cases extensively. A lot of people I know do this — it’s very handy to be able to start with a fresh-out-of-the-box operating system installation, and configure everything the way you like it just by checking out your normal configuration files from version control.

After I’d been doing that for a while, it occurred to me that exactly the same considerations apply for system configuration files — things like the configuration files for sshd and sudo and other such programs. Why not put all of those configuration files in a version control repository as well, so you have a record of what files changed, and when, and for what reason? On servers managed by more than one person, it can also be helpful to know who made those changes.

That’s part of what all Little Blue Linux systems have: a git repository for tracking changes to system configuration and policy files, accessed using a simple wrapper script.

Sometimes, when I’ve been interested in using a program but am using a distribution like Debian or CentOS, I have run up against version constraints on dependencies. The current version of QEMU, say, has a feature I’d like to use, so I try to install it; but then it turns out it needs four or five libraries with a version later than what is provided in the distribution package repositories, and building modern versions of those libraries reveals other updates that they need — it’s frustrating!

Little Blue Linux is not always completely up-to-date, since new versions of packages are released all the time, but it’s reasonably close. A couple of times a year, I go through the packages that make up the base LBL system and update the blueprints to use the latest stable version of everything. So I’ve never run into a situation where I have to upgrade a bunch of other packages to be able to use a modern version of something else.

(To be completely honest, this isn’t as much of a selling point as it could be — because, although there are not ancient or obsolete versions of any package in LBL, there are a huge number of packages for which CBL simply has no blueprints, and you’ll have some work to do to use them at all. But it suits my purposes just fine!)

As I mentioned in the Introduction and Overview, I want to be sure that every part of the final system was built from source code, and that no binary code was simply copied from the original host system to the final system. The best way I’ve thought of to make sure of this is to make sure that none of the code from the host system will actually work on the eventual LB Linux system that we are building here. We do that by building everything for a different kind of computer than the one we start the build on. The very first part of the CBL process constructs a cross-toolchain: a compiler and related tools that run on one type of computer — conventionally, this is referred to as the "host" system — but create programs and libraries that will work on some completely different type of computer architecture: the "target" system.

The CBL process itself breaks down naturally into two different parts: the part that you run on the host system, and the part that you run on the target system. I sometimes refer to those as the two "sides" of the CBL process, the "host side" and the "target side."

The idea of using a cross-toolchain, to start with one kind of computer and use it to build a system that works on a different kind of computer, is so fundamental to CBL that I named the process after it. Within that constraint, though, there is a lot of flexibility in how to perform the CBL build. The host can be a physical machine, like an Intel x86-architecture notebook computer or ARM-architecture chromebook, or it can be a virtual machine emulated by a program like QEMU. There are advantages and disadvantages to both options. The target, similarly, can be a physical computer or a QEMU virtual machine; and, again, there are benefits and drawbacks to both. Any of those combinations can work — you can use a physical computer as the host system, then move all the pieces you built there to a different physical computer to finish the build, or you can use a virtual machine as the host system and a different virtual machine as the target system (either or both of which can be emulated computers running a different architecture than the actual physical computer you’re using), or anything else you can think of.

The main benefits of using QEMU — for the host or target or both — are, first, that you don’t need to have a real computer with the architecture you want to use for that side of the CBL process; and, second, that it’s easy to automate the entire build process, since you don’t have to move a physical storage device from one computer to another, or press any actual power buttons, or anything like that.

The main disadvantages of using QEMU are, first, that emulated systems are a lot slower than real computer systems, so the build process takes a long time; and, second, QEMU is sometimes not as stable as a real computer. When running an ARM64-target emulator on a 64-bit Intel notebook computer, QEMU sometimes crashes during the final system glib build, for example. In many cases this appears to be caused by the limited system resources (especially memory) available on the emulated system. When using QEMU to emulate a computer (as opposed to using it to run a virtual machine of the same architecture as the host system), I primarily emulate ARM systems, because QEMU for ARM can emulate a computer with any amount of RAM by using the virt machine type.

In theory, you can follow the CBL process with any kind of computer as the host or target platform, as long as they are supported by the GNU toolchain programs and the Linux kernel, but it seems as though every different CPU architecture presents idiosyncrasies that require additional work to support. That means that if you go outside the host/target pairings that I use for developing and testing CBL, you will probably need to do some additional work.

The physical computer systems that I use are 64-bit x86 (aka Intel- or AMD-architecture) computers, 32- and 64-bit ARM systems, and 32-bit MIPS systems — because those are the types of computers I have handy.

The canonical form of Cross-Building Linux is a set of "blueprint" files that are available in a publicly-accessible git repository.^[2] If you’re reading this as a book or web page, it was produced from those blueprint files by the litbuild program,^[3] from repository commit 5c12463f9fb55511cc0c03be193a372e345dbbf6.

Every time litbuild produces the CBL book, it configures it for a specific type of build, with a particular kind of host system and target system. It also includes instructions on how to launch the target-side build in QEMU if those are relevant, and so on. If this book does not describe the type of build you want to do, all you need to do is obtain the CBL blueprint files and the litbuild program, set some environment variables as described in the Configuration Parameters and Default Values section, and use litbuild to generate a new version of the book.

For this CBL build, the host system is aarch64-unknown-linux-gnu and the target system is x86_64-cbl-linux-gnu. The final system will be called autointel.lblinux.org. Most of the work will be done in the /home/lbl/work/build directory, so the storage device where that directory lives should have enough free space for the build. Fifty gigabytes is probably enough; a hundred definitely is.

If any of that looks wrong — or if any of the other parameters in Configuration Parameters and Default Values are not set the way you want them to be — change the configuration and generate a new book!

Building an entire operating system from source code is a lot of work, so it’s reasonable to wonder how much time you’ll be investing in the project. As with so much else in CBL, it depends. Emulators are much slower than real computers; slow CPUs are (obviously) slower than fast ones.

The default build — which uses a real 64-bit x86 computer for the host build and an emulated 64-bit ARM computer (running on the same physical computer) for the target build takes less than an hour for the host side and almost three days for the target side.

More commonly, I use the qemu-to-qemu build style,^[4] which does more work (it executes the Rebuild the Packages Whose Tests Were Skipped appendix after completing the base build), and the entire process takes about three and a half days. Repeating the same qemu-to-qemu build in the other direction, with an emulated ARM computer as the host system and a virtualized (but not emulated!) x86 computer as the target, the host side takes about twenty hours and the target side takes a little less than four; the full process including rebuilding packages with tests takes about twenty seven hours. (Those timings might be unnecessarily long, because I’m typically doing both builds at the same time, and using the computer for other things while they are running.)

If you want to get things to run more quickly, the best thing to do is to get a second computer with a different CPU architecture, so you can avoid emulation entirely. I have an Orange Pi 5 Plus with 32 GiB of RAM and an NVMe storage device;^[5] when I use it as the host system for an aarch64-to-x86_64 build, the entire qemu-to-qemu process takes about eight and a half hours, and when I use it as the target system for an x86_64-to-aarch64 build, the same process takes about sixteen hours. Excluding the extra work done by the qemu-to-qemu build, and just considering the basic CBL process, those take six and twelve hours, respectively. To me that seems pretty good, considering that we’re building a complete operating system from source code!

The CBL build can be adjusted and tuned in a variety of ways using the configuration parameters described in this section. To override any parameter from its default value, you can set an environment variable with its name to a new value (or simply modify this blueprint).

The default values here are appropriate for a build taking place on a 64-bit Intel or AMD computer, building a 64-bit ARM target system, and using a QEMU-emulated virtual machine for the target system. (The defaults work regardless of whether the host system is a physical computer or QEMU virtual machine.)

Each sub-section below discusses a related set of configuration parameters.

These parameters determine what type of build is done and what options are used to control aspects of that build.

As described in the overview, the CBL build process always uses a cross-toolchain, and produces a build of Little Blue Linux that will run on a different kind of computer than the initial host system. The HOST parameter should be set to the "triplet" of the system where the host side of CBL (that is, the cross-toolchain itself and the target-system "scaffolding" programs built using it) will be constructed. You can read more about "triplets" in the Constructing a GNU Cross-Toolchain section or in the documentation of GNU Autoconf (as of Autoconf 2.71, triplets are described in section 14).

The easiest way to get the correct value here is simply to run the script config.guess, found in the sources for GCC or Autoconf, on whatever computer or virtual machine you’re going to use for the host side of the CBL build.

TARGET is the other parameter used to control cross-compilation. It should be set to the triplet of the target system, whatever computer or virtual machine will be booted into using the scaffolding. The sample configuration files provided as part of CBL may be helpful in figuring out what you should set this to. We conventionally set the second component of the triplet to cbl when following the CBL process.

When configuring GCC for the target system, it may be useful or necessary to specify some set of flags. You can consult the GCC installation instructions and the gcc section for more details about the options available to you.

It’s sometimes useful or necessary to specify some set of configuration flags when configuring GCC for the host system (that is, for the GCC build done in Trustworthy Host-System Programs, if those are being built). This works just like TARGET_GCC_CONFIG, but for the initial native GCC.

Similarly to GCC, the GMP library may need to have some extra configuration flags specified — so that it knows what ABI to build for, for example.

Different packages or programs have different ways of referring to CPU architectures. As mentioned earlier, the GNU toolchain refers to "triplets," which you can read about in Constructing a GNU Cross-Toolchain; the CPU architecture is the first component of the triplet. The Linux kernel has its own naming convention for CPU architectures, which in many (but not all) cases is the same as the CPU field in the triplet.

The default value for the KERNEL_ARCH parameter is an example where the naming convention differs. The GNU toolchain refers to the 64-bit ARM architecture as aarch64 (for "ARM Architecture, 64-bit words"), but the Linux kernel calls it arm64.

Another example is MIPS. The Linux kernel has a single architecture, mips, that is used for all MIPS variants (big-endian and little-endian, with both 32-bit and 64-bit word lengths), but each variant has a different value for the CPU component of the triplet: "mips," "mipsel", "mips64," and "mips64el."

To verify that the cross-toolchain is working as expected, CBL compiles a simple program and then inspects the resulting binary to see whether it is built for the correct kind of machine. If the binary doesn’t indicate that it’s built for the machine type specified by this parameter, the build will halt so you can inspect the situation and see what’s going on.

The Linux kernel has approximately a jillion different configuration elements. These determine the hardware devices and features that will be supported by the linux kernel produced by the build. Starting from scratch isn’t necessarily a good idea; luckily, we don’t have to do that, because the kernel is distributed with a set of default configuration files for every supported CPU architecture. This parameter sets the default configuration file that will be used as a starting point for the CBL build.

The linux build process is sometimes inconsistent between CPU architectures. One place this manifests is the name of the kernel file that is produced by the build. It is sometimes vmlinux, sometimes vmlinuz, sometimes bzImage, sometimes Image.gz — As far as I can tell, it’s left to the discretion of whoever is maintaining that architecture within the kernel, and of course everyone has their own preferences.

If the target-side build has a storage device available for virtual memory, it can be specified as TARGET_SWAP_DEVICE and will be used if such a device is found. If the device doesn’t exist, it won’t cause any problems, though; and if the target does exist but has a filesystem or partition table or anything like that on it, it won’t be touched.

When building the programs and libraries that will comprise the final system, it is generally desirable to set the CFLAGS (for C) and CXXFLAGS (for C++) environment variables to a common value so that optimization flags are used consistently across the entire system. TARGET_SYSTEM_CFLAGS provides a way to do that.

The presence of the -fno-omit-frame-pointer flag deserves some additional comment. A frame pointer is a pointer to a stack frame; if GCC is told to store frame pointers, it uses a specific CPU register to store a pointer to the stack frame when making function calls. That register is then unavailable for other purposes, which can make code larger and less efficient, but facilitates "unwinding" the stack. This can be useful when trying to diagnose exceptions. The default behavior of GCC was to include frame pointers until GCC 8, and then changed to omit frame pointers.

Since the default build style for CBL targets 64-bit ARM architecture, it is important to set the default CFLAGS to include frame pointers: The AArch64 ABI was designed with the presumption that frame pointers would always be present. See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=84521 for some detail on this.

If you’re doing a CBL build for a different target, and you don’t plan to debug programs using gdb or something similar, you might want to omit -fno-omit-frame-pointer (and possibly add -fomit-frame-pointer) so that the resulting programs and libraries are a little bit smaller and faster.

I normally include an option like -mcpu=native so that everything on the final system is built with optimizations appropriate for the machine where the build is running. With GCC 14, though, I’ve seen that lead to issues in the final system GCC build, so I exclude that flag from the default value here.^[6]

The make program is used by most projects to automate their build and installation processes. Among many other things, make supports parallelism in builds: if you are using a computer with multiple CPUs and sufficient memory to support several simultaneous compiler processes, you can run make with a -jN command line option, or set the environment variable MAKEFLAGS to include such an option, and make will run up to N processes concurrently. This can speed up builds enormously!

For CBL, the degree of parallelism in build processes should not necessarily be the same on the host system and the target systems, because they may have very different hardware resources available. So to configure the number of concurrent build processes on the host system, simply set MAKEFLAGS as normal when running the host-side scripts. To configure the number of concurrent build processes on the target system, use this parameter!

For target builds that will run in a QEMU emulator, you may want to check whether QEMU supports multi-threaded emulation for the relevant guest architecture — this is the MTTCG feature that you can read about at https://wiki.qemu.org/Features/tcg-multithread. Unless MTTCG is supported by QEMU for that emulation target, there’s no point in specifying any parallelism higher than -j1, because QEMU won’t actually run multiple emulated CPUs simultaneously.

This parameter should generally align to the TARGET_QEMU_CPUCOUNT parameter, if the target will be a QEMU virtual machine. Because virtual machines can be run with any number of virtual CPUs, the default value for this parameter is simply $(nproc), which uses the nproc program from GNU coreutils to determine the current number of CPUs.

The default presumptions made by CBL are that the target system has a wired ethernet interface, that there is a DHCP server available, and that networking should be enabled when the target system is booted. If any of those isn’t the case, set ENABLE_TARGET_NETWORK to any value other than true — in that case, you’ll need to set up networking yourself after the build completes.

Similarly, the default presumption for CBL systems is that there is some source of cryptographically-strong random numbers that can be used by the rngd program to keep the kernel’s entropy pool full. (See rng-tools if you don’t know what any of that means.) If that’s not the case, you can set ENABLE_TARGET_RNGD to some non-true value.

There’s more than one way to get from the host side of the CBL build process to the target side. Each of these is defined in a blueprint named target-bridge-NAME (where NAME can be whatever you like); the one that is used for a particular CBL build is the one named in this parameter.

The default CBL process uses the real host computer system for the first part of the build, and an emulated QEMU virtual machine for the target. That’s what the qemu target bridge does. Another option is to use QEMU virtual machines for both the host and target systems; the qemu-to-qemu-build blueprint describes one way to do that.

Under most circumstances, nothing extraordinary or unusual needs to be done to support particular target devices. However, some single-board computers like the Raspberry Pi require extensive modifications to the Linux kernel to work properly. To support those computers without unnecessarily modifying the kernel for other systems, the blueprint for the Linux kernel has blocks that can be enabled based on the TARGET_MACHINE parameter.

If you want to build LB Linux for one of those devices, set this parameter appropriately.

Different parameters are required depending on whether the target system is a real computer or a virtual machine. The parameters in this section are used whenever the target system is a QEMU virtual machine.

If you’re not using a QEMU virtual machine for the target, most of these are irrelevant — with one exception, TARGET_QEMU_ARCH, as noted in its description.

The most fundamental of the QEMU-related parameters is QEMU’s name for the target CPU architecture. Like the Linux kernel, the way that QEMU refers to machine architectures is often the same as the CPU field in the triplet — this is the case for 64-bit ARM machines, which both the GNU toolchain and QEMU call "aarch64" (for "ARM Architecture, 64-bit").

Even when the target system is going to be a physical computer rather than a virtual one, it’s important to specify the correct value for TARGET_QEMU_ARCH — QEMU is used to validate that the cross-toolchain works properly, even if it’s not used for the target-side build.

For most architectures, QEMU can emulate a variety of different machines. This parameter lets you select from those. This selection relates to the kernel configuration you start with, which is specified with the KERNEL_CONFIG parameter.

The best documentation for what machine types are supported for different architectures is on the QEMU wiki: https://wiki.qemu.org/Documentation/Platforms is the top-level URL as of October 2023. You can also run the QEMU full-system-emulator program (qemu-system-x86_64 or qemu-system-aarch64 or whatever) with the argument -machine help to get a list of the machines it supports.

Similarly to the machine argument, QEMU can emulate a variety of CPUs; and you can get a list of the options here with -cpu help. In many cases you don’t really need to specify a CPU because there will be a default value that works fine, but this is not the case with the virt machine that is used in the default configuration.

If the target QEMU system will be run as a native virtual machine rather than an emulator — that is, if the actual computer is an x86_64 machine, and you’re doing a build in an x86_64 virtual machine so you can use the computer for other things while the build is running — you can specify -cpu host to tell QEMU not to emulate a processor at all, and simply act as a hypervisor. You’ll probably also need to specify some type of acceleration, like -accel kvm, in that case.

The QEMU full-system emulators can provide multiple CPU cores to the guest virtual machine. This may or may not actually be helpful in terms of performance — as mentioned earlier, when talking about TARGET_SYSTEM_MAKEFLAGS, this has historically only been helpful when QEMU is running as a hypervisor, not emulating a different machine architecture, because the TCG code generator that converts guest CPU instructions into host system instructions only ever operated in a single thread. Recent versions of QEMU have supported multi-threaded code generation (the "MTTCG" feature) for some architectures, which provides real parallelism within emulated machines. This speeds up builds dramatically for some packages, up to the limit of parallelism that QEMU emulated machines will accept.

Obviously, it’s a bad idea to set this parameter higher than the number of CPU cores that the system actually has!

QEMU allows you to define the amount of RAM that will be made available to a virtual machine. The CBL process puts a fair amount of stress on the target system, and the amount of memory available to the compiler — especially for C++ builds — has a huge impact on the reliability of the process as a whole. The default value of 8 GiB works pretty well, but when the host system has more memory, I always give the target as much as I can, up to about 24 GiB.

The virt machine type, available when using the ARM emulators, allows as much memory as you’d like to allocate.

The emulated hardware in QEMU virtual machines is not the same for all architectures and machines. Depending on what hardware is emulated, storage devices might show up as sd (SCSI) devices, hd (IDE or ATAPI) devices, vd (virtual) devices, or possibly something else entirely. This driver-defined prefix must be used when launching QEMU so that the Linux kernel can find the root filesystem, and is also used in QEMU builds when creating partition tables and filesystems.

The way drive prefixes are used in CBL correspond to a historical convention for the way that device files have been named: SCSI disks, for example, are referred to as /dev/sda, /dev/sdb, and so on (appending a letter to the drive prefix to refer to the whole volume); the device files for partitions on a disk simply add a partition number to the whole-volume block file, like /dev/sda1, /dev/sda2, and so on, for /dev/sda. This convention is not always used, though; the convention for NVMe storage is for the devices to be named /dev/nvme0n1, /dev/nvme1n1, and so on; and for partitions on those devices to be /dev/nvme0n1p1, /dev/nvem0n1p2, and so on.

That means that if you’re doing a CBL build using NVMe storage, or some other type of storage that uses a different naming convention than the historical one, you’ll need to tweak the blueprints that refer to the TARGET_QEMU_DRIVE_PREFIX parameters.

When building the target system in a QEMU virtual machine, the normal graphics console provided by QEMU is not used. Instead, we take advantage of QEMU’s ability to map the standard input and standard output of the virtual machine process to a simulated serial device. The first serial device on most Linux systems is /dev/ttyS0, but for ARM computers it might show up as /dev/ttyAMA0 instead.

Similarly to TARGET_BRIDGE, there’s more than one way to make a GNU/Linux system bootable, and so there are multiple blueprints for doing that. These parameters are used to select which blueprint to use, and (for those that need additional parameters) configure how it should work.

The BOOTLOADER parameter selects the blueprint that will be used to set up the boot loader for the target system. The actual blueprint that will be used for this is named setup-bootloader- followed by this parameter; in the case of this specific build, for example, the blueprint is setup-bootloader-grub.

As with the target bridge, the default option here, manual, means you’re on your own when it comes to making the target system bootable — the setup-bootloader-manual blueprint does not do anything.

If a boot loader is being installed, it’s a good idea to set BOOT_DEVICE to the name of the block special device that will be used by the boot loader. For example, the GRUB boot loader is typically installed on the first sector (also known as the "boot sector") of a storage device, where it can be found and loaded by the BIOS.

A non-root user account is always created during the CBL process. These parameters control the details that will be used for that account. It’s almost certainly a good idea to override them with values you prefer!

This parameter controls the login name for the non-root user. It’s a good idea to change this to your preferred login name. (The parameter name USER would be more idiomatic, but litbuild uses environment variables to override the default value for configuration parameters, and the bash shell always sets USER to the current user name — so using USER here would conflict with that.)

The UNIX user database has a "comment" field that, for accounts used by actual human users, is conventionally used for the full name of the user. Again, it’s a good idea to change this to your own full name.

A domain name — preferably one that you control, defaulted here to example.org — will be used in various places throughout system setup and configuration.

The final target system will set its hostname to whatever is specified by this parameter (in the DOMAIN_NAME domain).

The rest of the parameters govern where different parts of the build artifacts will reside. You can set these however you like.

When targeting a virtual machine, the disk image files used by the QEMU emulator will be created in this directory.

This sets the directory into which the cross-toolchain will be installed.

This sets the directory into which the Trustworthy Host-System Programs will be installed, if they are being built. If these are not needed, this can be left at the default value of /usr — or, if the host system has QEMU installed in a different location, whatever location that is.

The "sysroot" framework is used for the cross-toolchain, and will contain the root filesystem that will be used by the target system. You can read much more about this in the various GCC sections. Initially, during the host stage of the CBL build, this will only contain a single subdirectory, /scaffolding. Absolutely everything else will be created in the target stage of the build.

The source code for the software packages that make up the CBL system is distributed in files created by the tar program. tar stands for "Tape ARchive" — a term left over from bygone days, when magnetic tapes were the primary mechanism for moving large amounts of data from one system to another. Even though tapes aren’t commonly used any more, tar files, generally compressed, are still used as the primary distribution format for source code on UNIX-ish systems.

This parameter sets the location where CBL will look for all of the source package archives needed during the build.

Sometimes, the source code package that is distributed by a project needs to be modified or adjusted before it is built. This is generally done using the patch utility, and the files that specify the modifications that need to be made are called "patch files."

This parameter sets the location where CBL will look for all of the patch files needed during the build.

When building software from source code, you need to unpack the source code somewhere and then configure, build, and (sometimes) test it before installing it to its final destination directory. The WORK_SITE parameter specifies where all that activity will be done; the name comes from the construction metaphor that litbuild uses. It should be considered a transient or temporary directory, and can be deleted after the build is complete. The full CBL process can use around fifty gigabytes of storage space, so to be on the safe side, define WORK_SITE as some location with at least that much free space available.

Everything printed to standard output and standard error throughout the CBL build process will be written to log files; if something goes wrong, the main way to figure out what happened is to look at the log files.

This parameter specifies the location where log files are written during the host side of the CBL process. Log files written during the target side of the build will be on the target system, of course.

SCRIPT_DIR specifies the location where litbuild will write the bash scripts that automate the build story — the "tangle" side of the literate build system.

DOCUMENT_DIR specifies the location where litbuild will write an AsciiDoc document that tells the build story — the "weave" side of the literate build system.

Many of the packages that constitute the basic Little Blue system — including almost all of the most fundamental components, like the C standard library and software construction tools — are part of the GNU system created and maintained by the Free Software Foundation. These packages are generally designed to be constructed using the GNU Build System, which includes the autotools for configuration and make for running all the commands that actually compile, test, and install the package.

So many of the packages built during the CBL process use this build system that it forms the default sequence of steps used to set up software packages. If you look at the source blueprints that define the CBL process, you may notice that package blueprints often omit some or all of the commands used to set up the package. That’s because the default sequence of steps can be used for them:

It’s common for packages to be set up using one or two commands that differ from the defaults, so sometimes you’ll see that there’s an explicit definition for the configuration commands, or the test commands, or something like that.

CBL is designed and intended for automation — if you take the source blueprints for CBL, set environment variables for all the configuration parameters you want to override, and run the litbuild program on them, you’ll wind up with a shell script you can run to kick off at least the host side of the build; depending on which "target bridge" you use, the conclusion of that process might automatically kick off the entire second half of the build as well.

The reason for the focus on automated builds is simple: people are, generally, bad at being consistent. That means that any time you want to be able to repeat a process several times consistently and without mistakes, automation is a practical necessity — and I definitely want to be able to repeat the CBL process many times! I sometimes run through the CBL process multiple times in a week.

On the other hand…​ you may have a different preference. If, for whatever reason, you prefer to type all the commands yourself, or copy and paste them from a PDF or web browser, you can do that. This section has a few tips for that approach.

Several parts of CBL set environment variables that control or influence how things are built. Those enviroment variables are scoped by the section structure of the process, so (for example) when you start the Constructing a GNU Cross-Toolchain section, you should set all the environment variables defined in that section (and explicitly unset any variables that say they should not be defined at all) before you start building any of the packages in that section. But when you’ve finished that part of the process, you should start a new shell process without those variables set.

Aside from those environment variables, every section in CBL either has some commands to run — which are hopefully pretty straightforward — or sets up a package. If you’re following the process manually, here’s how to set up each package:

If anything goes wrong at any stage…​ well, that will give you a taste of why it’s a lot of work to create and maintain a GNU/Linux distribution. Operating systems are complicated! Things break all the time, and you have to spend time and effort figuring out whether it’s a real issue that has to be addressed or an ephemeral problem that will go away if you just restart whatever process failed.

In some cases, blueprints in CBL define files that should be written to the system at some specific location. These are sometimes written piecemeal throughout the text of a section, interspersed with a narrative explaining why the file contains what it does, but at the end of each such section, you’ll always find the complete text of these files. These should just be copied verbatim to the system, before running any of the commands defined in that section.

This whole process is exactly what the scripts produced by litbuild do automatically, up until the package-users framework is installed. At that point, the process you should use to build packages manually will also change — and we’ll talk more about that when we get there!

Sometimes, packages just don’t work the way that we would like them to. For example, when cross-compiling the GNU binutils for some host/target pairs, GCC issues a string truncation warning when compiling gas/config/tc-i386.c. This causes the binutils build to crash, since it’s set to treat all warnings as errors. Modifying a snprintf call to use the correct formatting spec (%hhx rather than %x) resolves the problem, but the binutils maintainers don’t want to simply make that change because of concerns that it might have a negative impact on some of the (many) architectures that binutils supports.

In this kind of situation, CBL applies a patch to make the necessary change.

The Linux kernel has to be configured to include support for whatever features and hardware device drivers are needed, and omit support for features and device drivers that are not desired. This is a pretty complicated process and can be hard to automate.

The configuration system provides a way to start with a named configuration rather than the default settings for an architecture; if this feature is used, a file from arch/*/configs is used to override the default settings.

CBL provides kernel configurations for some specific cases — for example, a configuration for kernels intended to run on EC2 instances in the Amazon Web Services cloud. These default configuration files are added to the Linux source tree via patches.

All of this is described more thoroughly in the Linux sections, so you can consult those for more information.

The GNU project maintains a repository of source code that is intended to be used in other software packages. This repository is called "Gnulib." However, unlike other packages referred to as "libraries," like libxml2 or libgcrypt, Gnulib is not compiled into a library of functions that are then dynamically linked into other programs at runtime. Rather, the expectation is that files from the Gnulib repository will be copied into the source tree of other projects.

This sometimes presents challenges.

An example of this is release 2.28 of the glibc package. This release of glibc removes some obsolete and deprecated header files — an example is libio.h — that are not a part of the C standard library but were part of earlier glibc releases. These header files are not referenced by Gnulib — but until sometime in 2018, they were. Since other packages, like m4 and gzip, still include those old versions of the Gnulib files, those packages won’t build on systems that use glibc 2.28 or later. At least, not until new versions of those packages, using modern versions of the Gnulib components, are released! And some packages are not released very often: as I write this, for example, it’s October of 2023, and the most recent version of m4 is 1.4.19, released in May of 2021 — over two years ago.

When I encounter this situation, my practice is simply to compose a patch by copying in the newest version of whatever Gnulib source files have compilation issues.

The most common — and least objectionable — type of patch used in CBL is the "branch-update" patch.

All large and complicated software packages, like GCC and the GNU C library, have bugs. Some of those bugs are, inevitably, severe. A common practice for project teams is to maintain bugfix branches in their source repositories for at least the most recent few releases of their package(s); as bugs are found and fixed, these changes are back-ported from the current main line of development to these bugfix branches.

It’s my practice to pull in all the updates from these bugfix branches from time to time, and apply those as patches so that the CBL system is as stable and bug-free as possible.

Any time you see a package with a patch called branch-update- along with a year-month-day datestamp in its name, that’s what it is: a compilation of all changes from the upstream project’s bugfix branch, as of whatever date is indicated by the patch file name.

Unlike other CBL patches, these branch-update patches are not tracked in the cbl-patches git repository, although they are present in the CBL file repository. The rationale for this is that it’s trivial to reproduce these patches from the upstream project’s version control repository; all you have to do is obtain that repository and then use a command like git diff glibc-2.28..remotes/origin/glibc-2.28/master to produce a current branch-update patch for glibc 2.28.

Almost all packages depend on other packages in one way or another. The most obvious dependency for packages on GNU/Linux systems is the C standard library (in most cases glibc), which is needed by almost every C program. But most packages written in C require other packages in addition to the standard library. As examples, gcc depends on binutils, and cmake depends on curl, expat, zlib, bzip2, and xz.

In the C and C++ language ecosystem, there is no automated way to download and manage other packages, so there is a general tendency to avoid adding dependencies unless there is a compelling reason to do so. That is very convenient for the CBL approach of having a separate blueprint for each package, and explicitly naming dependencies in each blueprint so that litbuild can figure out the complete set of scripts (and documentation sections) it should write based on the blueprint named as its target.

The tools provided with many programming languages — golang, perl, python, and ruby all leap to mind — make it substantially easier to manage dependencies. For example, in a ruby program, it’s common practice to include a file called Gemfile that specifies all of the ruby packages needed by that program; the program bundler (which, conveniently, comes bundled with the ruby language) can be used to download and install all of those pacakges, along with any packages those dependencies rely on, as long as an Internet connection is available. As you might imagine, when a language makes it very easy to manage external dependencies, this sometimes results in projects that have a great many dependencies. In some cases, projects written in these languages depend on over a hundred other modules. This does not fit well with the CBL way of doing things: writing a blueprint for a new package that is actually desired or needed is not much of a burden, but it does not seem like a good use of time or energy to write a hundred or more additional blueprints for all of the other packages it needs. There are a number of alternative approaches that can be used in situations like this. Here, we’re going to discuss one of those approaches, which leverages the idea of dependency "vendoring."

Easily adding, downloading and installing dependencies is a great convenience when developing software or extending it to have new features. It becomes less convenient when integrating multiple programs — each of which may depend on some common subset of external packages, but possibly on different versions of those packages. It can also be frustrating (I speak from personal experience on this!) when automated systems expect to be able to download and install dependency packages at the time that software is being deployed, since the upstream source for those packages may happen to be unavailable at the precise time you want to deploy a new version. To work around these problems, languages that support this kind of easy dependency management generally provide a mechanism for copying the correct version of all external packages directly into a subdirectory of the project code. A common convention is to copy these external packages into a subdirectory called "vendor" (since the contents of that subdirectory were not produced locally, but rather obtained from external "vendors"), and the practice of doing this is referred to as "vendoring" dependencies.

For packages that have so many dependencies that it’s simply not reasonable to write blueprints for them all, a practice that I sometimes use is to vendor those dependencies so that a copy of them is present in the project source directory structure, then create a single "vendored-dependencies" patch for that modification. These patches are generally quite large! For example, the source code archive for version 5.30.0 of the image package is less than 500 kilobytes in size, compressed. A patch that adds all of its dependencies (of which there are a hundred and seventy one in total) is about five and a half megabytes — over eleven times larger than the package source code per se. This approach is not by any means elegant, but it is the least-bad approach I’ve found.

First, and most importantly, it’s important to make sure you have all the tools you need on the host system. Many GNU/Linux systems are missing one or more of the programs necessary for CBL, or provide a version of those programs that won’t work properly for one reason or another. So, although it’s not an intrinsic part of the build process per se, CBL includes the Trustworthy Host-System Programs appendix for building the programs that we’ve found to cause problems later on.

The basic set of requirements from the host system include modern versions of the GNU toolchain (GCC, binutils) and build system (make, the autotools, and so on), the QEMU emulator, and the lzip compression program. The file program is also necessary, and must be the same version that is used in the CBL process. If you’re unsure of whether you have everything you need, please do check the Trustworthy Host-System Programs appendix and make sure you have the things built there.

If you’re following the CBL process on a LB Linux system, this is probably unnecessary — all of the packages built in the host-prerequisites appendix are also built in the CBL process and are part of the basic LB Linux system. The only gotcha in that case is the file package, which can only be cross-compiled on a system that has the same version of file installed already. So if you’re using an LB Linux system but the version of file installed there is older than the one that the CBL process currently uses, you’ll need to upgrade that package before you can proceed.

The whole first part of the CBL process — the part that runs on the host system — will need to find the trusted host system programs (if they were built) and the cross-toolchain programs, so we should make sure they are on the PATH. We also need to ensure that shared libraries installed as part of those packages can be found by those programs — if you’re not clear on what that last part means and don’t feel like being patient, you can skip ahead to A Word About The Dynamic Linker, where shared libraries are discussed.

Litbuild provides a feature that allows the scripts it generates to be re-run if the build crashes partway through, by recording which scripts have completed successfully and skipping them in future runs. To activate this feature, we define an environment variable LITBUILDDBDIR.

Now, at last, we’re ready to start the CBL build process.

Building a GNU/Linux system from the ground up is like constructing a building. At least, that’s the metaphor I had in mind while I was designing the CBL process and writing this book.

When constructing a building, it’s sometimes useful to start by assembling a scaffolding around the build site. Then you can climb onto the scaffolding and use it as a support and framework while you construct the building you actually want. Once the final building is complete, you can tear down and discard the scaffolding — it’s not important in and of itself, only as a means to an end.

That’s what we do in the CBL process: we use the cross-toolchain to construct a set of programs and libraries that we can boot into, as an ephemeral "scaffolding" framework from which we can build the actual target system. We build that scaffolding in such a way that it sits alongside the final system as we build it; the scaffolding components won’t conflict with anything that will eventually form part of the final Little Blue Linux system, because everything related to those components will be self-contained within a top-level /scaffolding directory. After the build is complete, we’ll delete /scaffolding.

A "toolchain" is the set of all of the programs and libraries required to transform source code into executable programs that you can actually run. It’s called a "chain" because there are multiple programs involved: each program takes some kind of input file and produces some kind of output file, which then becomes the input for the next program in the chain. Each program is like a link in the chain. (When you think about it that way, it’s not really the best metaphor. It’s really a lot more like an assembly line! But "toolassemblyline" sounds awful.)

In this case, we are building a C and C++ toolchain: it will be able to construct executable programs from C and C++ source code.

The toolchain being built here consists of: a preprocessor, which handles include directives and macro calls and things like that; a compiler, which takes C or C++ source code and translates it into assembly language code; an assembler, which takes that assembly code and translates it into binary object code; and a linker, which combines object code produced by the assembler with additional object code contained in libraries and produces executable programs. The process also requires an implementation of the C and C++ standard libraries, which contain a large collection of functions that the linker uses when producing programs. In many cases, the implementation of those functions involves making system calls, which are basically functions provided by the operating system kernel; because the standard libraries make use of system calls, the process of building a toolchain also requires the kernel header files that specify what system calls are available and how to invoke them.

In the GNU toolchain, the preprocessor is cpp and the compilers are gcc (for C source code) and g++ (for C++ source code). All of these — along with the standard C++ library — are contained in the GNU Compiler Collection (gcc) distribution. The assembler and linker are as and ld, and are provided by the GNU binutils ("binary utilities") distribution. The C standard library is distributed separately as the glibc (short for "GNU libc") package, and the kernel header files are part of the Linux kernel distribution.

Since glibc is a rather large library — over a hundred megabytes of source code, producing shared library files that are in the megabytes — it is not very well-suited to building programs that must fit in a compact space such as the flash chip that holds the firmware in wireless routers. For those programs, an alternative C library, such as musl or uClibc, may be more appropriate. Toolchains using those C libraries can be produced using a variant of these instructions.

A "cross-toolchain" is much like a normal, or "native," toolchain. The difference is that a cross-toolchain runs on a computer of one type (the "host" system) but builds programs that will run on a different type (the "target" system). For the CBL project, for example, we use common Intel-processor computers to build software that will run on an ARM-architecture CPU. That way we can be reasonably certain that the final system really is entirely built from source, and no binary code was simply copied from the original build system to the target system: code from the build system simply can’t run on the target system, so if any binary code from the host system winds up on the target system, it won’t work at all.

To build a GNU cross-toolchain, we pass --host, --target, and --build options to the configure scripts (produced using the autotools programs of the GNU build system) of the toolchain components.

As you may recall from Configuration Parameters and Default Values, the values assigned to these options are called "target triplets," a term that also comes from the GNU build system. A triplet is a string with multiple components that are separated by hyphens. Historically, the triplet has had fields for CPU, manufacturer, and operating system (e.g., mipsel-pc-gnu for a little-endian MIPS CPU, PC hardware, and the GNU operating system); more commonly, these days, the OS field is subdivided into two fields, "kernel" and "system," so for all intents and purposes the target triplet winds up having four components: cpu-manufacturer-kernel-os (e.g., mipsel-pc-linux-gnu). That means that the term "target triplet" is kind of obsolete and misleading, and it would make more sense to refer to them as "target quadruplets." But sometimes history wins over accuracy and clarity.

The manufacturer field is basically free-form; in many cases it’s just set as pc for IBM PC-architecture systems, or left as unknown or none. In CBL we set the manufacturer to cbl, because it’s shorter than unknown and is distinctive: any time you see a triplet like arm-cbl-linux-gnu, you can be fairly confident it was produced using the CBL procedure.

The manufacturer field can be omitted entirely, but that makes the whole situation much more complex and ambiguous: for example, in the triplet arm-linux-gnu, is linux the manufacturer or is it part of the os field? The GNU build system includes a script called config.sub specifically to take triplet strings and figure out what they mean. My advice is: always specify triplets as cpu-manufacturer-kernel-os.

The other components of the triplet — cpu, kernel, and OS — sometimes trigger specific behavior, especially during GCC builds. It’s a good idea to review the "Host/Target specific installation notes for GCC" section of the GCC build instructions when choosing a triplet for your system.

The GNU build system includes a script, config.guess, that tries to figure out what the host triplet is. Generally, this should be used as the HOST configuration parameter in CBL. You can always find an up-to-date copy of config.guess, and the related script config.sub mentioned above, in the GCC source code distribution.

Different combinations of those configure directives, host, target, and build, are used for different toolchain components and at different points in the CBL process. What they mean is:

When build, host, and target are all different, it’s called a "Canadian Cross." We’re not sure why. In CBL, we don’t build any Canadian Crosses. During the CBL process, we:

Once the cross-toolchain is built, most packages built using it will be configured with build set to the host computer and host set to the target computer.

The cross-toolchain built here uses the sysroot framework. The idea behind a sysroot toolchain is simple: a directory on the host system (the "sysroot" directory) is set up to contain a subset of what will eventually become the root filesystem of the target system: /bin, /lib, /usr/lib, /etc, that sort of thing. Header files and libraries will be used only from the sysroot location, not from the normal host system locations.

This is fine for most cross-compiling purposes, but it’s not quite perfect for our purposes in CBL.

Remember that the whole point of this cross-toolchain is to build just enough of a GNU/Linux system that it can be booted — the kernel, and a minimal set of userspace programs that we’ll then use to construct the final system. Those components, the "scaffolding," won’t be used any longer than is necessary. This is because we don’t want to rely on the programs produced by the cross-toolchain. By preference, for any program that is distributed with a test suite, we want to run the test suite before we trust it works properly! When you’re cross-compiling programs, you can’t easily run the tests.

So once we have the target system booted, we only use the scaffolding to build the first parts of the final system. As we build those, we install them into the canonical filesystem locations where they belong: the standard directories /bin, /lib, /usr, and so on. To make that process as straightforward as possible, and avoid any interference between the scaffolding and the final system components, we want all the scaffolding to wind up in a /scaffolding subdirectory of the root filesystem. If all of the ephemeral stuff is self-contained within /scaffolding, then obviously it won’t conflict with the final system programs as we build and install them.

Setting up everything so that it’s self-contained within a non-standard location also makes it easier to ensure that our final system doesn’t still rely on any of the components there: once the full system build is complete, we can simply delete the /scaffolding directory and we’ll be left with just the Little Blue Linux system.

When building a toolchain, it’s important to simplify the execution environment as much as possible; any unnecessary compiler or linker flags can cause things to break. Don’t worry about optimizing anything for this stage, either: remember, everything we’re building at this point is ephemeral, and we’ll be throwing it away as soon as we can.

This is as good a point as any to discuss the dynamic linker and the way it works.

In most cases, executable programs on Linux systems are linked against shared libraries rather than static libraries. That means that programs don’t contain a copy of the binary code for library functions they invoke; instead, programs have references to those library functions, and have a list of the shared libraries that are expected to contain the implementation for those functions.

Whenever a program is executed, those references to library functions obviously must be resolved — that is, the shared libraries are searched for all the functions needed by the program (and, since those functions might call other library functions as well, libraries are searched for those functions as well, and so on recursively), and linked together so that all of the necessary functions are available. This resolution is done by the dynamic linker, also known as the dynamic loader and, sometimes, as the program interpreter. This is a program included with the GNU C library — or whatever other C library is being used, like musl — and is conventionally installed at /lib/ld.so or /lib/ld-linux.so.

The way that the dynamic linker finds shared library files is a bit complicated, and that complexity is at the root of a lot of the issues that can come up during the CBL build process, so let’s talk about it a bit!

For an overview of gcc, see gcc.

Now that we have a C library installed, we can finally do a full GCC build. So now we’ll enable multi-threaded code and some of the runtime libraries we turned off previously.

Fairly early in the build process, a cross-compiler version of gcc is built and installed as xgcc, for use in later build steps. Then later on, when building libgcc.so and various other libraries that are part of GCC, xgcc runs the cross-binutils ld. Unfortunately, xgcc doesn’t know to tell ld to look in the /scaffolding directory to find the startup files (crti.o and so on). xgcc also insists on looking for libraries and startup files in the variant lib directories used by the multilib scheme, even though we configure GCC with --disable-multilib; it doesn’t appear that there’s any way to coerce the GCC build system not to use those multilib directories.

A workaround that sometimes helps is to add the correct directory using the LDFLAGS_FOR_TARGET and CFLAGS_FOR_TARGET options. That’s no good in this case, though: many of the libraries in gcc use the libtool script to do all their compilation and linking, and libtool ignores the LDFLAGS and CFLAGS we set.

So to get this build completed, we use a kludge: we create symlinks from /home/lbl/work/sysroot/scaffolding/lib to all the locations where xgcc might expect to find the libraries.

If the build crashes and gets restarted, the ln commands will fail. That doesn’t matter, so we temporarily tell the shell to proceed rather than terminating if an error occurs.

To maintain a hard line of separation between the scaffolding and the final system components, we’re building all of this stuff so that it installs into a directory called /scaffolding. When we set up the root filesystem for the target, it’s only going to have that one top-level directory! Then, as the scaffolding programs are used to construct the final system components, those final-system programs will be installed to normal system directories — /bin, /usr, and so on — and will be used in preference to the scaffolding programs that they replace.

In most cases, the scaffolding components use the GNU build system. That means they can be configured to expect that they will live in /scaffolding, but then be installed with a DESTDIR of the sysroot directory. That way, they actually get installed to /home/lbl/work/sysroot/scaffolding (which is exactly where they need to be on the host system), but think they’re installed to a top-level /scaffolding directory — which is where they actually will live once we boot into the target device.

Building everything with a very simple environment is still a good idea.

Some of the scaffolding pieces install libraries and headers. We want those to be visible to the rest of the scaffolding, so CFLAGS and LDFLAGS are not as empty as they have previously been.

To use the cross-toolchain for these builds, we need to define a bunch of additional environment variables. Many of the scaffolding programs use the GNU build system, and therefore consult these environment variables to determine how to invoke toolchain programs.

We’re also going to use the cross-toolchain options build, host, and target for most of the components we’re building in this section. This time, --build is going to be the host system; --host and --target are going to refer to the target system.

Some target architectures have a "multilib" feature, and the installation process for many packages insists on installing library files into a variety of different directories (like lib32 and lib64) to support this feature — even when multilib is disabled, as we try to do throughout the CBL process. The issue with this is, not all of the multilib directories are on the default library path known by the dynamic loader; this leads to errors in some package builds.

To ensure that all library files wind up in the lib directory per se rather than a multilib variant directory, we use a ugly but simple and effective kludge: we create symbolic links to ensure that all library files are placed directly into /scaffolding/lib. (In the target-side build, we’ll do something similar for the /lib and /usr/lib directories: this is done in Write the Scaffolding Init Script.)

The set of symbolic links needed here might expand as additional targets are added to the set that CBL can handle.