nexcage 0.17.0

One binary, both backends. The release's binary manages Proxmox LXC
containers and is the OCI runtime a container engine drives, with the crun
backend built in. The .deb installs it with the libraries it needs.

A pod can run under an isolation profile. The node's administrator names,
in the configuration file, what a container may not exceed. An engine's
RuntimeClass picks a profile by the handler's program name,
nexcage@<profile>. On a k3s node, two pods under two profiles were checked
from inside each pod.

Read "Upgrading" before you install. The configuration file now fails
closed, so a file that loaded under 0.16.0 can be refused.

One release: one binary, one .deb (#380)

From 0.11.2 to 0.16.0 a release carried two binaries:

• nexcage-<version>-amd64, with Proxmox LXC only, which the .deb packed;
• nexcage-<version>-amd64-crun, with the crun backend too.

So an engine configuration or an isolation profile next to the .deb pointed
at a backend the installed binary did not have.

Now there is one binary with both backends, and one .deb that packs
it:

• Its Depends pulls libjson-c5, libseccomp2, libcap2 and
  libsystemd0, so apt install ./nexcage-0.17.0-amd64.deb is the whole
  setup, for the command line and for a container engine alike.
• The package ships config.oci.example.json next to config.json.
• The node configurations in deploy/kubernetes/node/ name
  /usr/bin/nexcage, where the package puts the binary.

Before it publishes, the release installs the .deb in a clean Debian 13
container and asks the crun backend for features. crun_build.yml does the
same on every pull request, and k8s_e2e.yml installs the .deb on the E2E
node to run its pods. The .deb's copyright file names the licenses of
libcrun (LGPL-2.1-or-later) and libocispec, which the binary carries.

Thanks to @IhorTelepenko, whose #301 started this.

Isolation profiles

{ "profiles": {
    "hardened": { "runtime": "crun", "crun": {
      "user_namespace": "require",
      "seccomp": "require",
      "capabilities": { "drop": ["CAP_NET_RAW", "CAP_MKNOD", "CAP_SYS_CHROOT"] },
      "limits": { "memory": "1G", "pids": 1024 } } } } }

create applies a profile to the bundle before libcrun sees it.

• Requirements refuse a bundle that lacks them:
  • a user namespace with uid and gid mappings (hostUsers: false on the pod);
  • a seccomp filter (seccompProfile.type: RuntimeDefault).
• Capabilities are dropped from every set of the process. That includes
  exec: containerd builds kubectl exec's process from its own copy of the
  spec, from before the profile narrowed it. An exec'd process is cut to the
  container's bounding set.
• Memory and pids limits are lowered to the profile's, and set where the
  bundle has none. The bundle's swap allowance is kept: containerd's
  swap = limit, which means no swap, still means no swap.

A profile only takes away: it never adds a capability or raises a limit.

Naming a profile.

• An engine names one by the program name, nexcage@<profile>: a symlink used
  as containerd's BinaryName or CRI-O's runtime_path.
• A person uses --profile <name>.
• Only create reads a profile. Every later command finds the container
  without it.
• state reports the profile as the annotation io.cageforge.nexcage.profile.

Refusals. These exit 2 and create nothing:

• a profile the file does not define;
• --profile and the program name naming different profiles;
• a --runtime that disagrees with the profile;
• run given a profile;
• a bundle the profile refuses;
• a bundle that carries the profile annotation itself, with a profile or
  without.

packaging/config/config.oci.example.json now defines two profiles,
hardened and small.
INSTALL.md
shows the symlink, the k3s handler and the RuntimeClass for each, and
CLI_REFERENCE.md
the keys.

Checked from inside the container, not from nexcage's log

tests/k8s/pod_on_node.sh on the E2E node (k3s v1.36.5) runs one pod under
each of two classes:

A third pod under nexcage-hardened without hostUsers: false is refused,
and its events say "requires a user namespace".

tests/crun/profile.sh runs in crun_build.yml on every push. It checks the
same on a bundle, through exec --process and through cgroup v2's
memory.swap.max.

The configuration file fails closed (#371)

A key nexcage does not read, a value of the wrong type, and a runtime or log
level it does not know all refuse the file. The message names the file and the
key, nothing runs, and the exit status is 1.

Under 0.16.0:

• a misspelt runtime in a routing rule sent containers to LXC without a word;
• {"runtime": 5} crashed every command, --help included;
• "security": {"seccomp": true} turned nothing on.

The keys that remain are the README's table, plus profiles.

A container's backend comes from the container (#372)

Every command used to choose its backend from the routing rules again, so a
rule changed after create sent state, kill or delete to the other
backend. Routing now decides only for create and run. Every other command
goes to the backend that has the container. --runtime lxc for a container on
the crun backend exits 2 instead of being obeyed.

Fixed

• --config, --root and --runtime were read after --. That made
  nexcage exec web-1 -- grep --config x load x as nexcage's configuration.
  Words after -- belong to the command exec runs.
• exec's temporary process file has a predictable name in /tmp. It was
  opened with truncate, so on a host with fs.protected_symlinks off, a
  symlink left there would have been followed. It is now created exclusively,
  mode 0600.

Not in this release

A pod as a Proxmox LXC container (#316).
ADR-006
records what a probe on the E2E node (Proxmox VE 9.2.20) showed. It can be
done:

• lxc-execute passes its stdio to the container and exits with its status;
• --share-net joins CNI's namespace;
• pct list shows the container.

Its costs fall on every pod:

• the sandbox stays on crun;
• every create goes through Proxmox's perl internals and pmxcfs;
• the container is privileged;
• its cgroups are outside the kubelet's.

All it would add is visibility in pct list. A profile with
"runtime": "lxc" keeps refusing the file.

Upgrading from 0.16.0

1. Check the configuration before you replace the binary. Run the new
   binary against your file:

   ./nexcage-0.17.0-amd64 --config /etc/nexcage/config.json version

   • It prints the version if the file loads.
   • If it does not, it names the file and the key to remove or fix, and exits
     1.
   • The keys 0.16.0 parsed and never used are refused like misspelt ones:
     runtime_type, default_runtime, runtime.root_path, data_dir,
     cache_dir, temp_dir, network.ip, network.gateway, security,
     resources, container_config.default_container_type, and pct_path,
     node and legacy_api under proxmox.
   • container_config.crun_name_patterns is refused, with
     runtime.routing named as its replacement.

2. There is no -crun asset. A script that downloads
   nexcage-<version>-amd64-crun has to drop the suffix: nexcage-0.17.0-amd64
   is that build now.

3. Install the .deb: apt install ./nexcage-0.17.0-amd64.deb.

4. A node set up from 0.16.0's instructions has the -crun binary at
   /usr/local/bin/nexcage, and its engine names that path. The .deb puts
   nexcage at /usr/bin/nexcage. Either change BinaryName (or CRI-O's
   runtime_path) to /usr/bin/nexcage and restart the engine, then remove
   /usr/local/bin/nexcage, or leave the engine as it is and replace
   /usr/local/bin/nexcage with the new binary. Do not leave two different
   versions installed.

5. Profiles are optional. A file without profiles behaves as before.

nexcage 0.16.0

The OCI runtime is measured by the suites that define it, a Kubernetes node on
Proxmox VE is installed from what a release ships, and a private registry needs
one skopeo login per node.

Nothing a script or an engine sends is read differently. Upgrading is
installing.

Conformance, on record

The first three run in crun_build.yml on every push and pull request to
main, against the -crun build routed to crun. runtime-tools and critest
run again against crun itself, for reference. A failure not on a suite's
known-failures list turns CI red, and so does a listed one that passes.

critest's first run found a defect the engines had not: containerd could not
stop a pod whose container had just exited (#361). With --log <file>, which
every engine passes, nexcage wrote libcrun's reason to the file only.
containerd's shim reads "no such process" from the command's output to know a
kill came too late. It got operation failed instead, and StopPodSandbox
failed with it. An error now goes to stderr as well, as runc and crun print
theirs.

A Kubernetes node on Proxmox VE

docs/INSTALL.md has the steps in order. The files they install are in the
release's tag:

They are examples, and nothing enables them for you. A pod runs on nexcage
only when it names the class; the node's default runtime is unchanged.
tests/k8s/pod_on_node.sh installs these same files, so k8s_e2e.yml checks
what an administrator is told to install. The -crun build stays a bare
binary. A package for it belongs to the APT repository (#319).

Images from a private registry

Proxmox's pull takes no credentials, but it runs skopeo copy as root, and
skopeo reads root's auth file on the node that pulls. On the E2E node that was
checked through pvesh, and through pvesh with the environment emptied as
pvedaemon's is, which is how a pull with --node runs on the other node.
The API path itself was not run. So:

skopeo login --authfile /root/.config/containers/auth.json registry.example.com

on each node that pulls. With --node titan, that is titan. Give
--authfile: skopeo's default for root is under /run, which a reboot
empties. The file holds the credentials encoded, not encrypted, so use a token
that can only read.

The same check found that pvesh exits 0 when its pull task fails.
nexcage reported every such failure as "reported success but … holds no new
template". It now gives skopeo's reason. When the registry refused a login, it
also gives the skopeo login line for the node that pulls:

$ nexcage pull ghcr.io/acme/private:1
[1791499398] ERROR nexcage: pulling ghcr.io/acme/private:1 into local on pve1 failed: initializing source docker://ghcr.io/acme/private:1: unable to retrieve auth token: invalid username/password: unauthorized: incorrect username or password
[1791499398] ERROR nexcage: for a registry that needs a login, run as root on pve1: skopeo login --authfile /root/.config/containers/auth.json ghcr.io (docs/INSTALL.md)
nexcage: pull: operation failed

The Proxmox E2E puts wrong credentials for docker.io in that file on every run
and requires the pull to be refused with that reason and that line. That
proves the file is read without a secret in CI. A pull from a real private
registry has not been run in CI. It needs a registry and a credential owned by
the project.

Security

The self-hosted jobs no longer run a pull request from a fork (#362).
proxmox_e2e.yml and buildagent.yml ran any pull request's code, and on the
E2E node the runner user has sudo for everything. A fork's pull request now
skips them. k8s_e2e.yml was written that way from the start.

Not in this release

events. The 0.16.0 plan took it only once an engine sends it. None has:
podman, ctr, containerd's CRI, CRI-O, critest and a kubelet ran without
asking for it.

Upgrading from 0.15.0

• Install and carry on.
• A pull that fails now says why. If you parsed "holds no new template" out
  of stderr, the message is skopeo's reason now.

nexcage 0.15.0

create takes what pct create is usually given, and an option nexcage does
not know is refused by name instead of skipped.

Read this first if you script nexcage

A misspelt or unsupported option now fails with exit 2. Until this release
an option no command took was skipped without a word, and its value became the
next positional word:

$ nexcage create --name typo-1 --memroy 2G local:vztmpl/debian-12-standard_12.7-1_amd64.tar.zst
nexcage: unknown option '--memroy' for 'create'; see 'nexcage create --help'
nexcage: invalid input

0.14.1 answered the same line with "OCI bundle '2G' must be an absolute path".
An unknown option without a value was worse: kill --al web-1 signalled the
init alone and exited 0. If a script passed an option nexcage never
implemented, it now stops there.

Container engines are not affected. Every option containerd, CRI-O and podman
were seen sending already had its own branch, and the CI's pod through
containerd's CRI runs unchanged. runc options nexcage does not implement
(--no-pivot, --preserve-fds, ...) are refused rather than dropped, since
dropping one would run the container differently from what was asked.

create --image <image>, the form create --help shows, had only worked
because --image was skipped and its value taken for the image. It is parsed
now.

create takes what pct create is usually given

nexcage create --name db-1 --memory 4G --cores 2 --ip 10.0.0.5/24 --gw 10.0.0.1 \
  --vlan 20 --onboot --mp local-lvm:20,mp=/var/lib/postgresql \
  local:vztmpl/debian-12-standard_12.7-1_amd64.tar.zst

Before this release, everything past the name and image took a pct set by
VMID afterwards, outside nexcage. Now:

They go through pct create for a container here and through the node's API
for one made with --node. The Proxmox E2E creates a container with every one
of them and reads each back from pct config.

Refused rather than half-done:

• a limit Proxmox has no setting for, as update refuses it;
• an address containing , or =, which would add a net0 key;
• a VLAN outside 1-4094;
• --mp with an OCI bundle, whose mounts take the mp entries;
• all of them on the crun backend, which takes limits, network and mounts from
  the bundle's config.json.

Not in this release

Private registries (#309) moved to 0.16.0. Proxmox's oci-registry-pull
still takes no credentials. But it pulls with skopeo copy as root, and skopeo
reads the containers auth file, so a skopeo login on each node may be all it
takes. That is to be checked on a real node before anything is built. The
issue has what was found and how to check it.

Upgrading from 0.14.1

• Install and carry on.
• A script passing an option nexcage does not take now exits 2, naming the
  option. Remove the option, or spell it as --help shows.

nexcage 0.14.1

exec --user 1000 on Proxmox LXC ran the command as root. This release
refuses that and every other flag run and exec accepted there and then
dropped. It also has three smaller fixes: an explicit --log-level info,
health's idea of the configuration, and how CI installs Zig.

Refused rather than ignored

nexcage's rule since 0.10.0 is that a flag it cannot honour is refused by
name: a caller that is told no can do something else, and one that is ignored
believes something false. 0.14.0's documentation pass found two commands that
broke it on the Proxmox LXC backend.

exec passed --user, --cwd, --console-socket and --pid-file to
the crun backend only. On Proxmox LXC it runs pct exec, which takes a
command and nothing else. The command runs as root, in a directory nexcage
does not choose, and no pty or pid comes back. So a caller asking for uid 1000
got root without a word. These flags now exit 1, naming the flag, as
--process and --detach already did:

$ nexcage exec --user 1000 web-1 id; echo "exit $?"
[1791452028] ERROR nexcage: --user is not possible on the Proxmox LXC backend: pct exec takes a command and nothing else, and runs it as root. Use --runtime crun
nexcage: exec: not supported on this host or by this build
exit 1

--tty is honoured when nexcage runs on a terminal. pct exec runs
lxc-attach, which gives the command a terminal whenever one of its standard
descriptors is one. When none is, nothing can give the command a terminal, so
--tty is refused.

run dropped --node, --console-socket and --pid-file. run --node titan made the container on this host. All three are refused now, before
anything is created. To make a container on another node, use create --node titan, then start.

If a script relied on the old silence, it now sees the refusal. On this
backend the flags never did what they said.

Fixed

• --log-level info and NEXCAGE_LOG_LEVEL=info now override the
  configuration file. An explicit info used to be taken for "not set", so
  a file saying debug could not be turned back down for one command. A level
  that is named wins over the file's, and takes the [DEBUG] lines the file's
  debug turned on with it. --debug and NEXCAGE_DEBUG still keep them.
• health reports the configuration file the commands read: the one
  --config names, else the first of ./config.json,
  /etc/nexcage/config.json and /etc/nexcage/nexcage.json. It used to look
  at a pair of files of its own, whatever --config said. It no longer runs
  nslookup google.com: nexcage resolves no names itself, and a root tool on
  an air-gapped cluster should not query an outside host to warn about
  nothing.

CI

Zig comes from mlugg/setup-zig, pinned by commit, instead of
goto-bus-stop/setup-zig, whose README calls it unmaintained. The Proxmox E2E
on main failed in that step before any test ran. On the self-hosted runners
Zig now stays in the runner's tool cache. The release workflow builds without
the Zig cache, from the tagged tree alone.

Upgrading from 0.14.0

Install and carry on. A script that passes --user, --cwd,
--console-socket or --pid-file to exec, or --node to run, on the
Proxmox LXC backend now gets exit 1 instead of a command that did something
else.

nexcage 0.14.0

Snapshots, create --node on a stock Proxmox VE host, and a project that says
true things: the Proxmox VM stub and two build options that could not work are
gone, the documentation was checked line by line against the code, and the
ownership and review rules describe what is practised.

Read this first if your configuration names vm or proxmox

Such a file is now refused by every command. The Proxmox VM backend is
removed (below), and unlike runc in 0.13.0 there is no backend to send such a
container to instead: the default one would make a container where a virtual
machine was asked for. The message says what to name instead — lxc or
crun. --runtime vm and --runtime qemu exit 2, saying the backend was
removed. A configuration that names neither is unaffected.

Snapshots

$ nexcage snapshot web-1 before-upgrade --description "pre 2.4"
$ nexcage snapshots web-1
NAME	CREATED	DESCRIPTION
before-upgrade	2026-10-02T10:31:07Z	pre 2.4
$ nexcage rollback web-1 before-upgrade --start
$ nexcage delsnapshot web-1 before-upgrade

snapshot, snapshots, rollback and delsnapshot, with pct's names, for
the Proxmox LXC backend. Proxmox takes the snapshot, so each is one pct call
for a container on this node and one call to its node's API for a container
elsewhere; nexcage adds the container by name on any node, a plain line when
the storage cannot snapshot, and a tab-separated list, or --format json with
each snapshot's parent as well. rollback of a
running container stops it, as Proxmox does, and --start starts it again.
The crun backend refuses them: libcrun has no storage of its own to snapshot.
The E2E suite exercises them on a storage that can snapshot and on one that
cannot.

Fixed

• create --node was refused on every stock Proxmox VE host — "cannot be
  used with a ZFS rootfs" — because the guard asked whether the zfs tools were
  installed, and Proxmox VE installs them. It now asks what it meant to: whether
  nexcage would make a dataset itself, which no configuration key does. A
  rootfs on a ZFS storage goes through; Proxmox creates it on the node that owns
  the container. The simulator's fake zfs had answered "not installed", which
  is a host Proxmox VE does not ship.
• state reports ociVersion 1.3.0, the runtime-spec baseline since
  0.7.4, rather than a hand-written 1.0.0.
• The documentation and every command's --help, checked statement by
  statement against the code. Among what was wrong: a bundle had to sit under
  two directories (any absolute path has been accepted since 0.10.0); the crun
  backend had no state or exec (it has both); --log-file was said to
  receive the log (it holds the start and completion lines, never an error);
  list was said to cover every backend (Proxmox LXC only); the man page and
  the completion lacked a dozen commands.
• CODEOWNERS named the organization and an account that is not the
  maintainer's, so GitHub never requested a review from anyone.
  GOVERNANCE.md and MAINTAINERS.md asked for two reviewers with one active
  maintainer; they now say how changes are reviewed today, and when the
  two-reviewer rule starts.

Removed

• The Proxmox VM backend and -Denable-backend-proxmox-vm. It was compiled
  out by default and built by no workflow; the router answered every operation
  routed to it as not implemented, and the driver imported a module the build
  does not define. qm can come back with a test behind it and someone who
  needs it.
• -Denable-backend-proxmox-lxc. Setting it to false never compiled. The
  Proxmox LXC backend is always built; zig build rejects the option.
• crun_vendor_sync.yml, crun_headers_generate.yml and
  scripts/sync_crun_vendor.sh, which changed files inside the deps/crun
  submodule and so never had a pull request to open.

Proxmox VE 8 is no longer claimed

Earlier releases ran on it, but no test ever did, and Proxmox ended its own
support for 8.x in August 2026. nexcage does not refuse an 8.x host; it no
longer promises anything there. Proxmox VE 9 is what the E2E suite runs on.

For contributors

scripts/dev.sh (make doctor, e2e, act, local-ci, perf,
dev-shell) runs the unit tests, the simulator, the crun backend, a pod
through containerd's CRI and the GitHub-hosted CI jobs on a workstation, with
rootless podman or Docker. tests/perf/ times every command and counts the
Proxmox tool runs each makes; scripts/dev.sh perf --against <ref> fails on a
regression. ROADMAP.md says what comes next.

Upgrading from 0.13.0

• Plain binary, .deb and -crun binary: install and carry on.
• A configuration naming vm or proxmox as a runtime: name lxc or crun
  — see the top.
• Building from source: drop -Denable-backend-proxmox-vm and
  -Denable-backend-proxmox-lxc from your flags.

Still missing

Flags that run and exec accept on Proxmox LXC and then drop — exec --user
runs the command as root — are refused in 0.14.1, the next release, with the
other bugs the documentation pass found. See ROADMAP.md.

nexcage 0.13.0

update — the last runtime-spec verb that was missing — vendored crun moves
from 1.24 to 1.30.1, and two things are removed: the runc backend, which
nothing ever built or ran, and crun_name_patterns.

Read this first if you install the -crun binary

It needs libjson-c5 now, not libyajl2. crun 1.28 replaced YAJL with
json-c, libocispec included, and the vendored crun is 1.30.1 in this release.

apt install libjson-c5 libseccomp2 libcap2

A Proxmox VE 9 host has all three already — it never had libyajl2, so this
is one library fewer to install than before. On a host without it the binary
does not start: error while loading shared libraries: libjson-c.so.5. The
plain binary needs none of this, as before.

update

$ nexcage update --memory 768M --cpu-quota 50000 --cpu-period 100000 web-1
$ pct config 100 | grep -E '^(memory|cpulimit):'
cpulimit: 0.5
memory: 768
$ cat /sys/fs/cgroup/lxc/100/memory.max
805306368

A running container's resource limits, as runc update and crun update do,
with their flag names: --memory, --memory-swap, --cpu-quota,
--cpu-period, --cpu-share, --pids-limit and the rest. --resources <file> takes a runtime-spec linux.resources document, and - reads it from
stdin — which is how containerd sends an in-place pod resize
(update --resources=- <id>). The CRI test asks for one now and reads the
container's cgroup afterwards.

On the crun backend everything reaches libcrun. On Proxmox LXC the settings
are said to pct set in its own terms:

That last row is deliberate: a limit asked for and silently not applied is the
worst answer an update can give. A container on another node is updated
through its node's API. Proxmox keeps the change in the config, so it survives
a restart — a runtime's cgroup write does not.

The runc backend is gone

It shelled out to a runc binary, no workflow ever built it, and it had never
been run; the crun backend answers the same OCI interface through libcrun and
is what every engine was verified against. Two things a configuration might
still say:

• a routing rule with "runtime": "runc" routes to crun — the container
  it describes is an OCI container either way — and the log says the file
  should say so;
• --runtime runc on the command line is refused with the replacement
  named: use --runtime crun. Exit 2.

container_config.crun_name_patterns is removed as well. It routed names
matching a glob to crun before runtime.routing existed; a glob under
routing is the same matcher, so each entry is one rule:

{ "runtime": { "routing": [ { "pattern": "kube-ovn-*", "runtime": "crun" } ] } }

A file that still carries the old key gets a warning naming the replacement,
because a container it used to send to crun now goes to the default backend,
and that should not happen without a word.

Fixed

create from a registry image called the pull endpoint every time and then
guessed the volid from the reference — a guess about how Proxmox normalises a
file name — onto local whatever was asked. It now reuses an image that is
already on the storage, what a container engine does, and otherwise pulls the
way pull does, reading the volid back from the storage. create --storage
and run --storage say which storage. On the real host the evidence is
Proxmox's own task record: a second create from the same reference adds no
ociregistrypull task.

ADR-001 says what ships

The architecture decision from 2024 chose crun as the primary runtime with
runc as an automatic fallback, and assumed nexcage would call a runtime
binary. It now records the reality: Proxmox LXC is the default backend, crun
is libcrun linked in and what an engine's host routes to, there is no fallback
between backends, and there is no runc. docs/architecture/BACKENDS.md had two
rows wrong too (kill via pct exec; name lookup via pct list only), and
says what the code does.

Vendored crun 1.30.1

The pin was a fork commit on 1.24 carrying a revert of two intelrdt commits;
it is now upstream 1.30.1 plus the .upstream_tag marker the header script
reads, at CageForge/crun. The revert was
not needed to build. The features ABI check passes against the new header
unchanged. How to do the next bump is written down in
docs/DEVELOPMENT_WORKFLOW.md.

Upgrading from 0.12.0

• Plain binary and .deb: install and carry on.
• -crun binary: libjson-c5 instead of libyajl2 — see the top.
• A config naming runc or carrying crun_name_patterns keeps working with
  a warning; change it at your convenience.

Still missing

events, a metrics stream — nothing has asked for it. The kubelet test on a
node (tests/k8s/pod_on_node.sh) is still run by hand.

nexcage 0.12.0

If you run nexcage on a Proxmox cluster, upgrade. Cluster support shipped in
0.11.0 and never worked on a real host: the query it makes was rejected by the
Proxmox API on every call. 0.11.0, 0.11.1 and 0.11.2 all carry the defect.

Also new: pause and resume.

The cluster lookup asked for something that does not exist

$ nexcage start web-2          # web-2 is on titan
ERROR container 'web-2' not found

Under that answer, on every 0.11.x:

$ pvesh get /cluster/resources --type lxc
400 Parameter verification failed.
type: value 'lxc' does not have a value in the enumeration 'vm, storage, node, sdn'

The API has no lxc resource type. Containers come back under vm, carrying
their own type field:

[ { "vmid": 200, "type": "lxc", "node": "titan", "status": "running" },
  { "vmid": 300, "type": "qemu", "node": "titan", "status": "running" } ]

So the cluster query failed every time and nexcage fell back to pct list,
which is correct for a container on this host and answers "not found" for one
anywhere else — the exact thing cluster support was added for. list showed
only local containers, start/stop/delete/state on a container elsewhere
said it did not exist, and --node on images/rmi had no cluster to check
against.

The simulator's fake pvesh accepted --type lxc, so 208 checks agreed
with the mistake. It now refuses it with Proxmox's own message, which is how the
fix is tested rather than merely asserted.

Two more things a real cluster showed

/cluster/resources is a cache. pvestatd refreshes it every few seconds,
so this failed:

nexcage create --name c1 local:vztmpl/debian-13-standard_13.6-1_amd64.tar.zst
nexcage start c1        # ERROR container 'c1' not found

The container existed; the cache had not caught up. A miss in the cluster now
means "keep looking on this host", and only both sources coming up empty is a
container that does not exist.

And the cache was preferred over pct. For a container on this host the
cached status overrode the live one, so state reported stopped for several
seconds after a successful start — while list said running, from the same
binary. For a container here pct is the current answer now; the cluster keeps
the one thing only it knows, which node a container elsewhere is on.

All three are in code that 0.11.0, 0.11.1 and 0.11.2 shipped.

pause and resume

$ nexcage pause web-1
$ nexcage state web-1 | jq -r .status
paused
$ nexcage resume web-1

The cgroup freezer: every process in the container stops where it is and stays
in memory. On the crun backend through libcrun; on Proxmox LXC by writing
/sys/fs/cgroup/lxc/<vmid>/cgroup.freeze directly.

state reads the freezer itself, because pct status cannot answer this —
it keeps saying running, as Proxmox has no notion of a frozen container. That
is worth knowing if you pause one and then look for it with Proxmox's own tools.

This is deliberately not pct suspend. That runs lxc-checkpoint -s, which
dumps the processes through CRIU and takes the container down — a different
operation, and on the Proxmox VE 9.2 host this was measured on it simply failed.
The freezer is what the OCI runtime-spec describes, and nexcage never calls
pct suspend.

list says paused for the same container. It read pct's answer only while
this was being written, so the same binary called one container running and
paused depending on which command you asked — the same fault as the stale
status above, and caught the same way.

Pausing a container on another node is refused, naming it: the cgroup filesystem
there is not this host's to write.

The E2E suite runs all of this on a real Proxmox host, and takes the kernel's
word rather than its own: cgroup.events has to say frozen 1, not just
cgroup.freeze holding what nexcage wrote, and a command sent into the
container while it is frozen has to be held there — and to run once it is
thawed. Every defect fixed in this release was in code the
simulator's fakes were perfectly happy with.

Upgrading from 0.11.x

Install and carry on — no configuration or command-line changes. A container
that was running stays running.

One behaviour change to be aware of if you script against it: state for a
container on this host now reports what pct says at that moment rather than
what the cluster cache last recorded. If something in your scripts was
accidentally relying on the stale answer, it will now see the current one.

Still missing

• update and events. update is next: libcrun exposes it, and the LXC
  side maps onto pct set. Nothing has asked for events in a single run
  against podman, ctr, containerd, CRI-O or a kubelet.
• The kubelet test on a node (tests/k8s/pod_on_node.sh) is still run by hand.
  Making it a CI check needs the crun build as a workflow artifact; a release
  carries it as an asset since 0.11.2, which is not the same thing.

nexcage 0.11.2

A release now carries the binary a container engine drives.

VERSION=0.11.2
wget https://github.com/CageForge/nexcage/releases/download/v$VERSION/nexcage-$VERSION-amd64-crun
wget https://github.com/CageForge/nexcage/releases/download/v$VERSION/checksums.txt
sha256sum --ignore-missing -c checksums.txt
install -m 0755 nexcage-$VERSION-amd64-crun /usr/local/bin/nexcage

apt install libyajl2 libseccomp2 libcap2

Until now the runtime backend that podman, containerd, CRI-O and a kubelet drive
had to be built from source, through the Dockerfile, because vendored libcrun
needs the submodules and two generated header sets. That build is published.

Which binary you want

libyajl2 is not on a Proxmox VE host by default. Without it the -crun
binary does not start at all:

error while loading shared libraries: libyajl.so.2

A container engine never passes --runtime, so the configuration has to send
containers to that backend:

{ "runtime": { "routing": [ { "pattern": "*", "runtime": "crun" } ] } }

A routing pattern is a regular expression only when it starts with ^ or ends
with $. ".*" is read as a wildcard — a literal dot followed by anything —
and matches nothing. Use "*".

Both binaries are built with -Dcpu=baseline and checked on a Xeon E5-2697 v2
without AVX2 before the release is announced.

The README stopped being wrong

It had been saying "Status as of version 0.9.0", with containerd/CRI
integration and containers on other cluster nodes listed under "Not yet" —
months after both were done. A project's own front page is the worst place for a
claim like that.

It now says what nexcage is: a command line for LXC containers on a Proxmox VE
cluster, and an OCI runtime that podman, ctr, containerd's CRI, CRI-O and a
kubelet drive. With a status table that names what is not there, install
instructions for both binaries, and the sections an open-source project carries.

docs/index.md, the site description and docs/architecture/OVERVIEW.md carried
the same stale framing — the last described nexcage as turning CLI commands into
pct calls, when it is called from two directions — and now match.

Nothing else changed

No runtime behaviour, no command line. Upgrading from 0.11.1 is replacing the
binary, and the plain one behaves exactly as it did.

nexcage 0.11.1

A small release. Nothing new to use; images, pull and rmi shipped in
0.11.0 having met only the simulator's fakes, and they are checked against a
real Proxmox host now.

The one thing to notice

images --node on a node the cluster does not have used to print a header and
exit 0:

$ nexcage images --node definitely-not-a-node
NODE  STORAGE  SHARED  SIZE  TEMPLATE
$ echo $?
0

That reads as "that node has no templates", which is a different statement and
sends someone looking in the wrong place. It is an error now, naming the nodes
the cluster does have:

$ nexcage images --node definitely-not-a-node
ERROR no node called 'definitely-not-a-node' in this cluster; it has: prox-home, titan
$ echo $?
1

Why this release exists

Three of the failures while writing the simulator's fakes for those commands
were the fake and not the code: a shared storage modelled as separate files per
node, a pulled template that appeared in the wrong list, a version the fake
reported as 8.4.1. A storage listing is exactly the kind of thing a fake gets
subtly wrong, so the commands are now exercised where nothing is pretending:

[E2E] images lists what this node has, and agrees with pvesm
NODE          STORAGE  SHARED  SIZE  TEMPLATE
nexcage-e2e   local    no      123M  local:vztmpl/debian-13-standard_13.6-1_amd64.tar.zst
[E2E] a node the cluster does not have is an error, not an empty list
[E2E] pull, and the volid it prints is the one the storage holds
[E2E] pulled local:vztmpl/alpine_3.20.tar
[E2E] rmi removes it from the storage, not just from the listing
local:vztmpl/alpine_3.20.tar removed from nexcage-e2e
[E2E] and removing it twice is an error, not a second success
[E2E] templates path passed

The step frees anything it pulled on the way out, failure included. images
runs on any Proxmox; pull and rmi need 9.1 or later, and the step says so
and stops rather than failing there.

Also fixed

The E2E job uploaded only e2e_lxc.log. The registry step has been writing
e2e_registry.log since it was added and nothing ever collected it, so a
failure there left nothing to read afterwards. All of them are collected now.

Upgrading from 0.11.0

Nothing to do. The only behaviour change is the one above, and it affects a
command that was answering wrongly.

nexcage 0.11.0

A Proxmox VE cluster is more than one node, and nexcage now sees all of them.

$ nexcage list
ID    IMAGE     COMMAND  CREATED   STATUS   BACKEND      NODE        NAMES
100   unknown   pct      unknown   running  proxmox-lxc  prox-home   web-1
200   unknown   pct      unknown   running  proxmox-lxc  titan       web-2

$ nexcage start web-2        # on titan, from here

pct only ever answers for the host it runs on. Until now that meant list
showed a fraction of what was there without saying so, and every other
command answered "not found" for a container on another node — the same answer it
gives for a name that does not exist anywhere.

Read this if you parse list

list has a NODE column between BACKEND and NAMES. A script taking the
container name from the seventh tab-separated field wants the eighth now:

# before
nexcage list | awk -F'\t' '{print $7}'
# now
nexcage list | awk -F'\t' '{print $8}'

That is the only incompatibility in this release.

What works across the cluster

list, state, start, stop and delete find a container wherever it lives.
A container on this host still goes through pct; one elsewhere goes through the
owning node's API.

state for a container on another node carries where it is:

  "pid": 0,
  "annotations": { "io.cageforge.nexcage.node": "titan" }

Three things stay local, and are refused with the node's name rather than
answered wrongly:

Creating on another node

nexcage create --name web-2 --node titan local:vztmpl/debian-13-standard_13.0-1_amd64.tar.zst

The template is checked on that node, before a VMID is taken. This is the
trap the check exists for: a storage called local is a different directory on
every node unless it is shared, so a volid that exists here may simply not be
there.

Three things --node refuses rather than half-does, each because something in
the path is local by construction: an OCI bundle (its rootfs is packed into a
template on this host's storage), a registry image (the pull lands here), and a
ZFS rootfs (the dataset would be created in this host's pool).

--node naming the host nexcage runs on is not "another node" — it takes the
ordinary local path.

Templates

$ nexcage images
NODE        STORAGE      SHARED  SIZE   TEMPLATE
prox-home   local        no      129M   local:vztmpl/ubuntu-22.04-standard_22.04-1_amd64.tar.zst
prox-home   shared-rdma  yes     31M    shared-rdma:vztmpl/redis_7.tar
titan       local        no      98M    local:vztmpl/alpine-3.22-default_20250617_amd64.tar.xz

$ nexcage pull docker.io/library/redis:7 --storage shared-rdma
shared-rdma:vztmpl/redis_7.tar
$ nexcage create --name r1 --node titan shared-rdma:vztmpl/redis_7.tar

$ nexcage rmi shared-rdma:vztmpl/redis_7.tar --node titan
shared-rdma:vztmpl/redis_7.tar removed from the shared storage; it is gone from every node

That middle pair is why pull exists on its own: create pulls too, but only
onto this host's local storage, which the other node cannot read. Pulling
onto a shared storage is what makes create --node usable.

A storage marked SHARED carries the same files on every node and is listed
once rather than once per node. rmi on such a storage removes the file for
every node at once, whichever node was named, and says so.

Two limits worth knowing before you plan around them:

• Pulling needs Proxmox VE 9.1 or later — oci-registry-pull does not exist
  before it, and nexcage says which version it needs rather than failing
  obscurely.
• That endpoint takes no credentials, so a private registry cannot be
  authenticated through it. This is a property of the Proxmox API, not something
  nexcage left out.

Fixed

• state reported "pid": 0 for a running container whenever the PID could not
  be read, which a caller cannot tell apart from a container that has none. It
  is an error now — except for a container on another node, where there is no
  PID on this host and the annotation says which node to ask.

Nothing here touches the runtime side

containerd, CRI-O and a kubelet drive nexcage as an OCI runtime, and that is
unchanged from 0.10.0. For Kubernetes, more than one host was never a runtime
question: a kubelet runs on each node and calls the binary there, the way runc is
called. The same goes for images — containerd and CRI-O implement CRI's
ImageService themselves and hand the runtime a bundle whose rootfs is already
unpacked.

Still missing

• pause, resume, update, events. Nothing has asked for any of them in
  a single run against podman, ctr, containerd, CRI-O or a kubelet.
• The kubelet test on a node (tests/k8s/pod_on_node.sh) is run by hand. Making
  it a CI check needs a crun-enabled build published as an artifact; today that
  build exists only inside a Docker image.