Introducing balenaOS 6

We recently released balenaOS version 6, so in this post we’ll take a look at what has changed from version 5.x. Balena uses semantic versioning for our OS releases, therefore in this case the jump from 5 to 6 can be considered a major change. BalenaOS major changes are not typically breaking changes for most users, but rather changes to internal libraries or components.

This version of balenaOS introduces module compression by default across all device types. This trades a small amount of compute power when loading modules for a large space savings, and in some rare cases can require a minor application change to remain compatible.

Benefits

Modules are now zstandard compressed, which shrinks the disk usage of kernel modules to approximately one quarter of the uncompressed size. Module decompression is very fast, the zstd codec is close to LZ4 in decompression speed while comparing favorably to LZMA in compression ratio. (For a bit of fun, run zstd -b on your device to see how fast it is!)

The additional headroom in the root filesystem enables us to continue bringing new features and hardware support to existing device types more easily. Host OS images will be slightly smaller, and host OS upgrades will have less to write to disk. Devices with slower disks, such as micro SD cards, may even see a slight decrease in the amount of time to load modules, as there’s less to read from disk.

Do I need to make any changes?

Only users loading kernel modules shipped with balenaOS from inside a container, such as by using the io.balena.features.kernel-modules label to bind mount the host OS modules directory into a container followed by an explicit file path load (such as using insmod) are potentially affected.

Users loading kernel modules in this manner in balenaOS v6 are required to use a build of kmod that supports optional zstandard compression, added in kmod v28. Some distros, notably Debian Bullseye, ship a build of kmod that has zstd disabled. You can check if your container image ships kmod with the required compression codec from the shell. If your version of kmod supports zstd, no changes are necessary.

root@bullseye:/# grep VERSION= /etc/os-release
VERSION="11 (bullseye)"
root@bullseye:/# kmod -V
kmod version 28
-ZSTD +XZ -ZLIB +LIBCRYPTO -EXPERIMENTAL

In this Debian Bullseye container, we see -ZSTD, indicating the feature was disabled at build time.

root@bookworm:/# grep VERSION= /etc/os-release
VERSION="12 (bookworm)"
root@bookworm:/# kmod -V
kmod version 30
+ZSTD +XZ -ZLIB +LIBCRYPTO -EXPERIMENTAL

In this Debian Bookworm container, zstandard support is enabled, and kmod is capable of loading compressed modules from balenaOS.

Workarounds

If you load modules from balenaOS in your container, and your container image ships a version of kmod that doesn’t support the required compression codec, you’ll see an Invalid module format message, like below.

root@bullseye:/# insmod /lib/modules/$(uname -r)/kernel/drivers/block/loop.ko.zst
insmod: ERROR: could not insert module /lib/modules/6.6.23/kernel/drivers/block/loop.ko.zst: Invalid module format

Don’t fret. There are several solutions.

1. Upgrade to a base image with kmod +ZSTD
Check to see if a simple version bump brings in a build of kmod with ZSTD support. In the case of an application with a container based on debian:bullseye, as noted above, the only modification necessary is changing the base image to debian:bookworm or newer.

2. Manually decompress modules before loading
If your image includes the zstd utility, or it’s available in your distro’s repos, you can manually decompress a module before explicitly loading it. For instance, in our bullseye container:

root@bullseye:/# unzstd /lib/modules/$(uname -r)/kernel/drivers/block/loop.ko.zst -o loop.ko \
> && insmod loop.ko \
> && dmesg | tail -1
/lib/modules/6.6.23/kernel/drivers/block/loop.ko.zst: 85719 bytes
[1732066.399405] loop: module loaded

Unfortunately, this approach has complications when a module depends on another module.

root@bullseye:/# unzstd /lib/modules/$(uname -r)/kernel/drivers/usb/usbip/usbip-host.ko.zst -o usbip-host.ko \
> && insmod usbip-host.ko
/lib/modules/6.6.23/kernel/drivers/usb/usbip/usbip-host.ko.zst: 67145 bytes
insmod: ERROR: could not insert module usbip-host.ko: Unknown symbol in module

This module requires another module to be loaded first, which is typically handled by kmod, but because we’re bypassing it, we need to perform this step manually.

Module dependencies are encoded in modules.dep.bin, and the optional human-readable modules.dep. This can be parsed to load dependencies before the desired module.

root@bullseye:/# grep usbip-host.ko "/lib/modules/$(uname -r)/modules.dep"
kernel/drivers/usb/usbip/usbip-host.ko.zst: kernel/drivers/usb/usbip/usbip-core.ko.zst
root@bullseye:/# for mod in usbip-core.ko usbip-host.ko; do
> unzstd "$(find "/lib/modules/$(uname -r)/kernel" -name "${mod}.zst")" -o "$mod"
> insmod "$mod"
> done
/lib/modules/6.6.23/kernel/drivers/usb/usbip/usbip-core.ko.zst: 65745 bytes
/lib/modules/6.6.23/kernel/drivers/usb/usbip/usbip-host.ko.zst: 67145 bytes
root@bullseye:/# dmesg | tail -1
[ 2990.047329] usbcore: registered new device driver usbip-host

3. Build kmod from source with ZSTD enabled

The last option is building kmod from source, which is a rather standard autotools and make build. This can be completed as another stage of your existing container image build process.

FROM debian:bullseye AS kmod

RUN apt-get update \
        && apt-get install -y \
                build-essential \
                pkgconf \
                libzstd-dev

WORKDIR /usr/src
ARG pkgver=32
ADD https://www.kernel.org/pub/linux/utils/kernel/kmod/kmod-$pkgver.tar.xz .
RUN tar xvf kmod-$pkgver.tar.xz
RUN kmod-$pkgver/configure --with-zstd --prefix=/usr/src/kmod-output \
        && make install

FROM debian:bullseye AS run

COPY --from=kmod /usr/src/kmod-output /

Building from source requires installing the development package for libzstd, and running the configure script with --with-zstd.

root@bullseye:/# kmod -V
kmod version 32
+ZSTD -XZ -ZLIB -LIBCRYPTO
root@bullseye:/# insmod /lib/modules/$(uname -r)/kernel/drivers/block/loop.ko.zst \
> && dmesg | tail -1
[1732066.399405] loop: module loaded

Additional resources

Hopefully the workarounds described above will solve any issues you may experience with the latest version of balenaOS. If that’s not the case, feel free to reach out to us on support, in our forums or in the comments below. Also note that if you would like more detailed information about the changes in balenaOS, check out our changelog here.