drums-criu

The Drums fork of CRIU: a memory template server and restore-fidelity patches that let drumsd fork and reset Linux desktops in milliseconds.

Website · Docs · Benchmarks · Console · Upstream CRIU

---

This file has two parts. The first describes the fork. The second, from "CRIU -- A project to implement checkpoint/restore functionality for Linux" on, is upstream's README, kept as it is.

What this fork is

Drums runs computer-use agents in isolated Linux desktops that all come from one running operating system. drumsd, the Drums host daemon, keeps one checkpoint per desktop image and restores it once per environment, forks a running environment into N copies of the same state, and resets a used environment back to its checkpoint. Every one of those operations is a CRIU restore, and the time each takes is the floor under the cost of a rollout.

This repository is upstream CRIU with the patches that make that work: a template server that holds one checkpoint's memory and forks a restore from it instead of reading the images again, fixes for the state a full desktop carries (file locks, nested namespaces, protection keys, interrupted syscalls, resource limits), and a way to give every restored copy its own range of host uids. It is not a general-purpose CRIU release. For CRIU itself, its documentation and its bug tracker, go to criu.org and checkpoint-restore/criu. Report problems with the Drums patches here.

Upstream base	master at 2b7ac603c, criu: Version 4.3, tag v4.3
Fork branch	drums-main (the default branch); git log v4.3..drums-main is the whole difference
Drums patches	128 commits, listed below
Licence	unchanged: GPL-2.0 (COPYING), LGPL-2.1 for lib/, Expat (MIT) for images/

The Drums patches

Every change is a commit on top of v4.3. In groups, with why each exists:

Series	Commits	What it does	Why Drums needs it
Memory template server (criu restore --template-server SOCK, criu/pageshare.c)	11	One server process reads a checkpoint once, lays out every task's private anonymous memory, and forks a child per restore request; the restored tree starts with its pages shared copy-on-write with the template. Later commits fill the template with parallel reads, keep VMA boundaries and soft-dirty bits as a normal restore would, reserve premap slots, take the client's oom_score_adj, and serve incremental images through their parent chain.	Siblings restored from one checkpoint share their private anonymous pages copy-on-write until they write them, instead of each reading a private copy from the images. This is what makes creating, forking and resetting many environments from one checkpoint fast and memory-cheap.
Incremental dumps of a restored tree (--prev-images-uptime, --parent-template-only)	1	A tree that was itself restored from a base image can be dumped incrementally against that base, with tasks started after the restore dumped in full.	Capturing a reached state writes only what changed since the checkpoint it was restored from.
Nested pid, user, net, uts, ipc and mount namespaces (--nested-pidns)	62	Dump and restore trees whose tasks sit in their own pid and user namespaces below the root task, with the net, uts, ipc and mount namespaces those user namespaces own, zombies, threads, seccomp, sockets and mount propagation included. New crit x userns.	Chrome's sandbox and other sandboxed programs create exactly these trees; upstream CRIU refuses to dump them.
Chrome sandbox fidelity	11	Groups as seen from the top user namespace, a gid the parent maps, the LSM label through thread-self, the dumpable flag across call_in_child_process(), an AppArmor profile in the test. Test: zdtm/static/userns_chrome01.	Each commit removes one cause of a restored Chrome renderer crashing or exiting after the restore.
--userns-remap [u:|g:]FROM:TO:COUNT	5	On restore, the root user namespace gets a different range of host uids and gids. Also an RPC field (userns_remap in images/rpc.proto).	Every tree restored from one checkpoint would otherwise run in the host id range of the tree that was dumped; drumsd gives each restored copy its own range.
Lock fidelity	1	POSIX locks are set after the VMA file descriptors are closed.	POSIX locks restored before the VMA descriptors were closed did not match the dumped state.
Overlay index for fsnotify	8	One upper-layer index per overlay mount per dump, a watch resolved from its file handle before any walk, an optional persistent index cache (CRIU_OVL_INDEX_CACHE) for a read-only lower layer.	A GNOME desktop on an overlay root has 316 inotify marks whose handle cannot be opened; one walk per mark was 130 s of a 130 s dump, with the tasks frozen.
RLIMIT_FSIZE and ghost files	5	RLIMIT_FSIZE is set after the restorer's last log write; no SIGXFSZ is left behind on dump; a shared reg_file_info path is restored before remap_open_lock is released.	rtkit-daemon sets RLIMIT_FSIZE to 0 and took a SIGXFSZ from the restorer's own log writes (3.6 % of concurrent restores of a GNOME session failed); many tasks holding the same renamed-over dconf files raced on a shared path.
--restart-interrupted-syscalls	2	poll(), nanosleep() and futex waits with a timeout that the dump interrupts start again instead of failing with EINTR.	VS Code's file watcher treats EINTR from poll() as an error and was replaced after every dump. x86 only.
Restore and dump robustness	10	A restorer BUG() before the creds stage fails the restore instead of hanging; the pid namespace daemon stops before a killed init is waited for; SIGCHLD is blocked around the creds helper children; the parasite logs through a pipe; a dump unlocks only the network lock it took. Two new fault-injection points.	Each came from a hang or a leaked lock seen while running many concurrent restores on one host.
Memory protection keys	6	pkey_mprotect, pkey_alloc, pkey_free in compel on x86; a VMA's protection key and an mm's allocated keys are dumped and restored.	Chrome keeps memory under protection key 1 (PartitionAlloc thread isolation); a restore that dropped the key made Chrome crash with SIGILL.
Fork housekeeping	6	This README, SECURITY.md, the fork's CI, and three one-line fixes that CI found (a lint rule, a musl build of a zdtm test, the unit-test link).

The complete list, oldest first, as of this README (git log --oneline --no-merges --reverse v4.3..drums-main is always current):

All commits

• 7b4628484 lock fidelity patch: set POSIX locks after VMA fd close
• 7465bed8f fsnotify: index each overlay mount once per dump instead of walking it per watch
• 8173f44dd restore: share private anonymous memory between restores of one checkpoint
• 52b450a40 pageshare: leave VMA boundaries and soft-dirty flags as a normal restore does
• 16313610d restorer: unmap the gaps a memory template leaves in the premap area
• 627686b7a restore: set RLIMIT_FSIZE after the restorer has written its last log line
• 869335679 zdtm: restore a task with several threads and RLIMIT_FSIZE 0
• 956026786 files-reg: restore the shared reg_file_info path before releasing remap_open_lock
• f25393594 zdtm: many tasks hold their own open and mapping of two renamed-over files (ghost_open_race)
• d19dc86ad pageshare: reserve premap slots in the template for VMAs a restore must premap
• ea79ef5ff fsnotify: optional persistent overlay index (CRIU_OVL_INDEX_CACHE) for a read-only lower layer
• 5ee6286fc images: pstree: add the ids of a task at every pid namespace level
• 2f1874e91 util: add parse_ns_ids() for the NSpid, NSpgid and NSsid lines
• 87d6d9893 unittest: check parse_ns_ids() against a nested pid namespace
• e8265cef5 proc_parse: keep every pid namespace level of NSpid, NSpgid and NSsid
• 893efb588 seize: keep the pid namespace levels of every task and thread
• 621b28ed4 pstree: dump the ids of tasks at every pid namespace level
• ee76537c3 crit: show the pid namespace levels in 'crit x ps'
• 3c7ee37cf criu: add the --nested-pidns option
• d1b500742 images: describe nested pid namespaces
• da4db5ded dump: allow nested pid namespaces with --nested-pidns
• fb96ec716 clone-noasan: add clone3_with_pids_noasan()
• cd39c6d0c pstree: read the pid namespace levels of tasks on restore
• a16c23f51 restore: create tasks with their pids at every pid namespace level
• cf0bb8dc6 zdtm: add pidns_nested02, pidns_nested03 and pidns_nested04
• 40d7fc088 compel: add the unshare syscall
• ecc28baf2 images: add the pid namespace for children of a task
• ae873ae13 dump: keep the pid namespace for children of tasks
• 9df20a8a3 pstree: read and check the pid namespace for children on restore
• f3406071a restore: give tasks their pid namespace for children
• 80daaa730 zdtm: add pidns_unshare01 and pidns_unshare02
• 21a718813 images: describe the tree of user namespaces
• f3bc62d4f dump: collect the tree of user namespaces with --nested-pidns
• 64472da45 crit: add 'crit x userns'
• 7ba56fac0 test/others: add userns-tree
• 6545160ca namespaces: read the tree of user namespaces on restore
• aa1dd82a1 restore: create the nested user namespaces
• e80b7d96c restore: create pid namespaces owned by a nested user namespace
• 5298d1029 restore: let tasks enter their nested user namespace
• 234106ab2 restore: create the threads of tasks in nested user namespaces
• e7935fd38 net: restore net namespaces owned by a nested user namespace
• cd590b95b dump: write nested net namespaces if the root task has none of its own
• dad7835a3 dump: allow nested user namespaces with --nested-pidns
• 2ecb44fc8 zdtm: add userns_nested01, userns_nested02 and userns_netns01
• 757dfa4b2 test/others: userns-tree: restore the tree with nested user namespaces
• 8571520b0 namespaces: restore nested user namespaces that have no maps
• 325686a9f dump: keep the user namespace and the credentials of zombies
• baf5b5998 restore: give zombies their credentials and user namespace
• b4f8a728a zdtm: add userns_zombie01
• 562ee6cb8 zdtm: add userns_seccomp01
• 0b64561ab sockets: restore sockets of nested net namespaces in the one of criu
• 56db0d764 zdtm: add userns_sk01
• 6d25c9049 image: share the ids of page images with the namespace dumpers
• 4c66bc93f ipc_ns: keep the shm pages of nested ipc namespaces apart
• 02249462e restore: create nested uts and ipc namespaces in their owner
• b8d455d5d dump: allow nested uts and ipc namespaces with --nested-pidns
• 2dde9b4f2 zdtm: add userns_utsipc01
• 6ccc845e9 util: add cr_system_userns_fd()
• 1cf07b966 dump: keep the pid namespace of proc mounts and the owner of tmpfs
• 2d9516ebb mount-v2: restore mount namespaces owned by a nested user namespace
• fac36dbf2 zdtm: add userns_mntns01
• e1671e1c6 mount: export mnt_build_tree() and mnt_fsmount()
• e8200ab89 images: describe copied mount namespaces
• a3f99eb92 mount: restore nested mount namespaces of a root task in the one of criu
• 13fcb3d2c zdtm: add userns_container01
• 3a82d6fb3 mount: restore copied mount namespaces with mount-v2 too
• 085bfc8ce zdtm: check the locks of mounts in copied mount namespaces
• ba050e763 Documentation: describe the nested namespaces restored with --nested-pidns
• 0502b29f4 mount: fix the propagation of mounts of copied mount namespaces
• 8fc1cd258 zdtm: userns_mntns01: add a peer group and a shared slave
• 78df183b2 zdtm: userns_sk01, userns_mntns01: cover more of the restore
• 3ef92e13f zdtm: add userns_chrome01, the namespaces of the Chrome sandbox
• 2233ed189 namespaces: rewind the images dir before looking for userns images
• 983a96f6e test/others: add pageshare-userns, a template restore of nested user namespaces
• 9fd0dcc16 namespaces: create a nested user namespace with a gid its parent maps
• 040144ddf zdtm: userns_chrome01: map only uid and gid 1000, as Chrome does
• c798c41bd dump, restore: keep the groups of tasks of nested user namespaces as seen from the top
• 956754a2d zdtm: userns_chrome01: give the tasks groups their user namespaces do not map
• 5979ff1b9 restorer: set the lsm label through thread-self
• e3183a780 zdtm: userns_chrome01: run R1 and R2 in an AppArmor profile, as Chrome is
• 7cd971e94 compel, dump: add --restart-interrupted-syscalls
• 72dd49a54 zdtm: add restartblock01, poll() and a futex wait with a timeout across C/R
• cede7a11a fault: FI_RESTORE_CREDS_BUG makes the restorer fail restore_creds() and BUG()
• a245251b6 restore: fail at once when a traced thread faults before the creds stage finishes
• 08d0f8154 test: fault 9 (a restorer BUG in a thread before the creds stage) must fail the restore, not hang
• ade3f3da8 zdtm: the nested-namespace tree checks compare R, S, D and I as one state
• ee9118f93 pageshare: the forked restore takes the client's oom_score_adj
• 118c95813 test/others: add pageshare-oom, the oom_score_adj of a template restore
• 8cd7f8437 test/others/overlayfs: add run-ovl-index, overlay watches resolved without the full walk
• 2b5ee7a56 fsnotify: put the overlay mountpoint itself in the upper index
• b2ea456e7 fsnotify: keep one upper index per overlay mount
• 35a5957db fsnotify: resolve an overlay watch from its file handle before any walk
• bc393bbad test/others/overlayfs: run-ovl-index: cover the lookup by file handle
• 02afd0209 fsnotify: write the overlay index cache file at most once per dump
• fd87d43ac fault injection 10: a creds helper child of criu exits before criu waits for it
• 7a2f7f138 dump: block SIGCHLD from fork to waitpid of the creds helper children
• 514b8e2ec zdtm: dump a task with RLIMIT_FSIZE 0/0 and check that no SIGXFSZ is left
• e10195158 compel: give the parasite a pipe for its log instead of the log file
• 5018e7d1f fault injection 11: the restorer can't enter the pid namespace for children of its task
• 208b8a517 restore: stop the pid namespace daemon before waiting for a killed init
• 358380f66 restore: open the pid namespace for children before the task enters its user namespace
• fbbd7c20a net: a dump unlocks only the network lock it took
• 549eedbc4 dump: read the groups of a nested task from the user namespace of the root task, by its real pid
• 9ab683f62 zdtm: run userns_chrome01 in the uns flavor too (the root task in its own user namespace)
• 7cb53173f util: keep the dumpable flag of the caller of call_in_child_process()
• a17b78241 zdtm: userns_chrome01: a /proc dir of the zygote kept open, and the oom_score_adj of a renderer
• dfb7ae3fa images: the protection key of a VMA and the keys an mm has allocated
• ad88c44d2 compel: add pkey_mprotect, pkey_alloc and pkey_free on x86
• f100f93c0 dump: keep memory protection keys
• 52ac59456 restore: give back memory protection keys
• b8859f65e zdtm: add pkey01, memory protection keys across C/R
• b4fbfddf1 zdtm: run pkey01 in the uns flavor too (the root task in its own user namespace, as a hardened slot)
• c35422e0f restore: --userns-remap FROM:TO:COUNT gives the root user namespace other host ids
• 4849fad3e rpc: userns_remap, the --userns-remap ranges of a restore request
• 70cb8672a Documentation: --userns-remap
• 06ec7210f zdtm: add userns_remap01, the root user namespace restored with other host ids (uns)
• bfdd3e062 test/others: add userns-remap, a slot-like tree restored into another host id range
• 9bc7812a9 pageshare: fill the template with parallel preads
• faac1aad7 log: per-VMA and per-pagemap lines at debug level
• d4a05d1e1 pageshare: madvise a template region only when its mark changes, and unmap a forked subtree's regions in the parent
• 344fd0a2a pageshare: fill the template from an incremental image's parent chain (planned preads at the parents' offsets)
• 6600614ba dump, restore: --prev-images-uptime for a base the tree was restored from, --parent-template-only, and template restores of incremental images
• 18f1aca3b README: describe the Drums fork; drop upstream scheduled workflows that cannot run here
• 51113ff3e compel: report the parasite log pipe error with strerror(), not %m
• b29bd6dc2 zdtm: userns_nested01: include limits.h for PATH_MAX
• e79e45147 unittest: add a stub for add_userns_remap()
• 1a6491e4d ci: a fork CI that can pass here; upstream's full matrix on demand
• this commit: README, SECURITY.md, logo

Changes that are useful outside Drums are meant to go upstream. Nothing here is an upstream feature until it lands in checkpoint-restore/criu.

How drumsd uses it

drumsd does not link CRIU. It builds this tree into one binary and runs it as a separate process, the same way the stock criu command is used:

1. Install. drumsd's installer clones this repository at the commit pinned in its install/lock.json, runs make -j criu, installs the binary as /opt/drums/criu/bin/criu, runs criu check, and records the source commit and the binary's sha256. drumsd's configuration names that binary ("criu": "/opt/drums/criu/bin/criu" in drumsd.json).
2. One template server per checkpoint. For every checkpoint drumsd keeps a warm criu restore --template-server SOCK -D IMAGES -o LOG -v4 --pidfile PIDFILE in its own cgroup, protected from the OOM killer. It survives a drumsd restart and is adopted again from its pidfile.
3. One restore per create, fork or reset. A short-lived helper joins the environment's cgroup, unshares a private mount namespace, mounts the environment's overlay, and sends the restore command line with its stdio to the template server over the socket. The template server forks and that child runs the restore. The command line is the ordinary one:

criu restore -D IMAGES --root CROOT --restore-detached --file-locks --action-script HOOK
     --shell-job --tcp-close --manage-cgroups=ignore --nested-pidns -v4
     --userns-remap FROM:TO:65536 --pidfile PIDFILE -o LOG

--action-script points at drumsd's own hook binary, which runs CRIU's action hooks (the lock census before resume, the cgroup move) without a Python process. 4. Captures. A reached state is dumped with criu dump --track-mem --prev-images-dir BASE --prev-images-uptime USEC ..., so the new images hold only what changed since the checkpoint, and a template server can serve them through the parent chain. 5. Gates. After every restore drumsd checks the result (locks applied, cgroup, mount table, netns, state hash, processes alive after resume) and quarantines the environment if a check fails. Those checks live in drumsd, not here.

Build

The same steps as upstream. On Ubuntu 24.04 (what drumsd's installer runs on):

sudo apt-get install -y build-essential pkg-config git libprotobuf-dev libprotobuf-c-dev \
  protobuf-c-compiler protobuf-compiler python3-protobuf libnl-3-dev libnet1-dev libcap-dev \
  libbsd-dev libgnutls28-dev libnftables-dev libaio-dev libdrm-dev uuid-dev libnl-route-3-dev

git clone https://github.com/Drums-Labs/drums-criu.git
cd drums-criu
make -j"$(nproc)" criu
./criu/criu --version
sudo ./criu/criu check

make without a target also builds crit, libcriu and the compel plugins; see upstream's INSTALL.md for other distributions and for make install. CRIU builds are not byte-reproducible: two builds of the same commit have different hashes, which is why drumsd pins and checks the source commit, not a binary hash.

Tests

make -j"$(nproc)" -C test/zdtm
sudo ./test/zdtm.py run -t zdtm/static/userns_nested01 -t zdtm/static/userns_remap01 -t zdtm/static/pkey01

The zdtm tests the Drums patches add are pidns_nested02..04, pidns_unshare01..02, userns_nested01..02, userns_netns01, userns_zombie01, userns_seccomp01, userns_sk01, userns_utsipc01, userns_mntns01, userns_container01, userns_chrome01, userns_remap01, pkey01, restartblock01, rlimits02 and ghost_open_race. Larger scenarios are scripts under test/others/: pageshare-userns (a template restore of nested user namespaces), pageshare-oom, userns-tree, userns-remap, overlayfs/run-ovl-index and netlock-dump-fail. sudo ./test/zdtm.py run -a runs upstream's whole suite.

CI on this fork (ci.yml) builds with gcc and clang, runs make unittest, and runs the Drums zdtm tests above on a GitHub-hosted runner; lint.yml is upstream's linter (make lint, make indent). Upstream's full test matrix is kept as upstream-ci.yml and runs on demand.

Syncing with upstream

Upstream develops on criu-dev and releases on master. This fork tracks master releases:

git remote add upstream https://github.com/checkpoint-restore/criu.git
git fetch upstream --tags
git merge v4.4

After a merge: build, make unittest, run the Drums zdtm tests, and update the base commit at the top of this file and in the table above.

Security

To report a problem in the Drums patches, use GitHub's private vulnerability reporting on this repository (see SECURITY.md). Problems in CRIU itself go to upstream's process. Note that CRIU runs as root; the desktop processes it restores for Drums run as an unprivileged user inside their own namespaces, and nothing in this fork changes that.

Licence

GPL-2.0, as upstream: COPYING. Files under lib/ are LGPL-2.1 and files under images/ are Expat (MIT), see images/LICENSE. Drums distributes this source so that anyone who runs a drumsd host can build the exact CRIU it uses.

Status

Beta. The template server, --nested-pidns, --userns-remap and the fidelity fixes are what drumsd runs in production on Ubuntu 24.04 x86_64 hosts. Other architectures build but are not tested by this fork's CI. Upstream's --lazy-pages and --exec-cmd cannot be combined with --template-server.

---

CRIU -- A project to implement checkpoint/restore functionality for Linux

CRIU (stands for Checkpoint and Restore in Userspace) is a utility to checkpoint/restore Linux tasks.

Using this tool, you can freeze a running application (or part of it) and checkpoint it to a hard drive as a collection of files. You can then use the files to restore and run the application from the point it was frozen at. The distinctive feature of the CRIU project is that it is mainly implemented in user space. There are some more projects doing C/R for Linux, and so far CRIU appears to be the most feature-rich and up-to-date with the kernel.

CRIU project is (almost) the never-ending story, because we have to always keep up with the Linux kernel supporting checkpoint and restore for all the features it provides. Thus we're looking for contributors of all kinds -- feedback, bug reports, testing, coding, writing, etc. Please refer to CONTRIBUTING.md if you would like to get involved.

The project started as the way to do live migration for OpenVZ Linux containers, but later grew to more sophisticated and flexible tool. It is currently used by (integrated into) OpenVZ, LXC/LXD, Docker, and other software, project gets tremendous help from the community, and its packages are included into many Linux distributions.

The project home is at http://criu.org. This wiki contains all the knowledge base for CRIU we have. Pages worth starting with are:

• Installation instructions
• A simple example of usage
• Examples of more advanced usage
• Troubleshooting can be hard, some help can be found here, here and here

Checkpoint and restore of simple loop process

Advanced features

As main usage for CRIU is live migration, there's a library for it called P.Haul. Also the project exposes two cool core features as standalone libraries. These are libcompel for parasite code injection and libsoccr for TCP connections checkpoint-restore.

Live migration

True live migration using CRIU is possible, but doing all the steps by hands might be complicated. The phaul sub-project provides a Go library that encapsulates most of the complexity. This library and the Go bindings for CRIU are stored in the go-criu repository.

Parasite code injection

In order to get state of the running process CRIU needs to make this process execute some code, that would fetch the required information. To make this happen without killing the application itself, CRIU uses the parasite code injection technique, which is also available as a standalone library called libcompel.

TCP sockets checkpoint-restore

One of the CRIU features is the ability to save and restore state of a TCP socket without breaking the connection. This functionality is considered to be useful by itself, and we have it available as the libsoccr library.

Licence

The project is licensed under GPLv2 (though files sitting in the lib/ directory are LGPLv2.1).

All files in the images/ directory are licensed under the Expat license (so-called MIT). See the images/LICENSE file.