lxcfs process leaked after snap refresh

Please confirm

I have searched existing issues to check if an issue already exists for the bug I encountered.

Distribution

Ubuntu

Distribution version

26.04

Output of `snap list --all lxd core20 core22 core24 core26 snapd`

root@snap-test:~# snap list --all lxd core20 core22 core24 core26 snapd
Name    Version           Rev    Tracking       Publisher   Notes
core26  20260629          462    latest/stable  canonical✓  base
lxd     0+git.dd11726c4a  x3     latest/edge    -           disabled
lxd     0+git.dd11726c4a  x4     latest/edge    -           -
snapd   2.76.2            27710  latest/stable  canonical✓  snapd

System info

Instance log

Expected behavior

The oldest lxcfs process (pid=3529) should be reaped and resources reclaimed after snap GC.

Actual behavior

The oldest lxcfs process (pid=3529) still alive and resources not reclaimed.

Steps to reproduce

repro_lxcfs_leak.sh

The script does 2 refreshes. The refresh.retain value is by default 2 (assumed). Actual value is not set.

refresh.retain=2 means snap will keep 2 versions and GC the 3rd one (oldest one).

root@snap-test:~# snap get -d system 
{
	"cloud": {
		"name": "lxd"
	},
	"pki": {
		"certs": {}
	},
	"refresh": {},
	"seed": {
		"loaded": true
	},
	"system": {
		"hostname": "snap-test",
		"motd": "",
		"network": {},
		"timezone": "UTC"
	}
}

#!/bin/bash
# Minimal repro: a snap refresh leaves an LXCFS process behind, and nothing ever
# reaps it -- not even once the container it was serving is stopped and deleted,
# and not once snapd garbage-collects the revision it is executing from.
#
# Run as root on a THROWAWAY machine with snapd. Installs LXD from latest/stable
# and refreshes it along latest/candidate -> latest/edge, so these are ordinary
# store refreshes across real version boundaries.
#
# Two refreshes are used rather than one because refresh.retain defaults to 2:
# the third revision is what makes snapd evict the first, which is when the
# leaked process ends up executing a binary that is no longer installed.
# (latest/beta is not in the list -- it is "^", i.e. the same revision as
# candidate, so refreshing to it would only switch tracking.)
#
#   sudo ./repro_lxcfs_leak.sh

set -uo pipefail

IMAGE="${IMAGE:-ubuntu-minimal-daily:24.04}"
POOL_BACKEND="${POOL_BACKEND:-dir}"
FROM_CHANNEL="${FROM_CHANNEL:-latest/stable}"
REFRESH_TO="${REFRESH_TO:-latest/candidate latest/edge}"

[ "$(id -u)" = 0 ] || { echo "must run as root"; exit 1; }

say() { echo; echo "=== $* ==="; }
fuse_dev() { nsenter -t "$1" -m -- awk '/fuse\.lxcfs/{print $3; exit}' /proc/self/mountinfo 2>/dev/null; }
ctn_dev()  { lxc exec "$1" -- awk '$5=="/proc/meminfo"{print $3; exit}' /proc/self/mountinfo 2>/dev/null; }
snap_ver() { snap list lxd | awk 'NR==2{print $2" (rev "$3")"}'; }
show_lxcfs() {
    for p in $(pgrep -x lxcfs | sort -n); do
        echo "    pid=$p ns=$(readlink /proc/$p/ns/mnt) mount=$(fuse_dev "$p") exe=$(readlink /proc/$p/exe)"
    done
}

say "0. install LXD from ${FROM_CHANNEL}"
snap install lxd --channel="${FROM_CHANNEL}" 2>&1 | tail -1
lxd waitready --timeout=180
lxc storage show default >/dev/null 2>&1 || lxd init --auto --storage-backend "$POOL_BACKEND" >/dev/null
echo "  installed: $(snap_ver)"

say "1. start a container, before any refresh"
lxc launch "$IMAGE" c1 >/dev/null 2>&1 || { echo "launch failed"; exit 1; }
sleep 8
OLD_PID="$(cat /var/snap/lxd/common/lxcfs.pid)"
OLD_NS="$(readlink /proc/${OLD_PID}/ns/mnt)"
OLD_EXE="$(readlink /proc/${OLD_PID}/exe)"
OLD_VER="$(snap_ver)"
echo "  lxcfs pid=${OLD_PID}  ns=${OLD_NS}"
echo "  lxcfs exe=${OLD_EXE}"
echo "  c1 is served by      : $(ctn_dev c1)  (lxcfs mount $(fuse_dev "$OLD_PID"))"

for ch in ${REFRESH_TO}; do
    say "2. snap refresh lxd --channel=${ch}, leaving c1 running"
    snap refresh lxd --channel="${ch}" 2>&1 | tail -1
    lxd waitready --timeout=180
    sleep 5
    echo "  now: $(snap_ver)"
    echo "  LXCFS processes:"
    show_lxcfs
done

say "3. stop and delete c1 -- the only container the old LXCFS was serving"
lxc stop c1 --timeout 30 >/dev/null 2>&1
lxc delete c1 >/dev/null 2>&1
sleep 10
echo "  containers remaining: $(lxc list -c n --format csv | tr '\n' ' ')"
echo "  LXCFS processes now:"
show_lxcfs

say "snap revisions and their backing files"
snap list --all lxd
losetup -a 2>/dev/null | grep -i lxd || true

say "RESULT"
if [ -d "/proc/${OLD_PID}" ]; then
    NOW_EXE="$(readlink /proc/${OLD_PID}/exe)"
    echo "  LEAKED: lxcfs ${OLD_PID}, started under ${OLD_VER} in namespace ${OLD_NS},"
    echo "          is still running with no container left to serve."
    echo "          exe at start : ${OLD_EXE}"
    echo "          exe now      : ${NOW_EXE}"
    echo "          ppid=$(awk '{print $4}' /proc/${OLD_PID}/stat)  cgroup=$(cat /proc/${OLD_PID}/cgroup)"
    echo
    echo "          Nothing reaps it: daemon.stop only kills the pid recorded in"
    echo "          lxcfs.pid, which each refresh replaces, and the process sits"
    echo "          outside the service cgroup so systemd's KillMode=control-group"
    echo "          does not reach it either. It persists until the host reboots."
else
    echo "  lxcfs ${OLD_PID} is gone -- did not reproduce."
fi

Repro Output

=== 0. install LXD from latest/stable ===
lxd 6.9-ab8fad2 from Canonical** installed
  installed: 6.9-ab8fad2 (rev 40424)
=== 1. start a container, before any refresh ===
  lxcfs pid=3529  ns=mnt:[4026532781]
  lxcfs exe=/snap/lxd/40424/bin/lxcfs
  c1 is served by      : 0:67  (lxcfs mount 0:67)
=== 2. snap refresh lxd --channel=latest/candidate, leaving c1 running ===
lxd (candidate) 6.9-029ed75 from Canonical** refreshed
  now: 6.9-029ed75 (rev 40638)
  LXCFS processes:
    pid=3529 ns=mnt:[4026532781] mount=0:67 exe=/snap/lxd/40424/bin/lxcfs
    pid=5498 ns=mnt:[4026532910] mount=0:94 exe=/snap/lxd/40638/bin/lxcfs
=== 2. snap refresh lxd --channel=latest/edge, leaving c1 running ===
lxd (edge) git-0a6bca2 from Canonical** refreshed
  now: git-0a6bca2 (rev 40657)
  LXCFS processes:
    pid=3529 ns=mnt:[4026532781] mount=0:67 exe=/bin/lxcfs
    pid=5498 ns=mnt:[4026532910] mount=0:94 exe=/snap/lxd/40638/bin/lxcfs
    pid=7109 ns=mnt:[4026532906] mount=0:100 exe=/snap/lxd/40657/bin/lxcfs
=== 3. stop and delete c1 -- the only container the old LXCFS was serving ===
  containers remaining:
  LXCFS processes now:
    pid=3529 ns=mnt:[4026532781] mount=0:67 exe=/bin/lxcfs
    pid=5498 ns=mnt:[4026532910] mount=0:94 exe=/snap/lxd/40638/bin/lxcfs
    pid=7109 ns=mnt:[4026532906] mount=0:100 exe=/snap/lxd/40657/bin/lxcfs
=== snap revisions and their backing files ===
Name  Version      Rev    Tracking     Publisher    Notes
lxd   6.9-029ed75  40638  latest/edge  canonical**  disabled
lxd   git-0a6bca2  40657  latest/edge  canonical**  -
/dev/loop4: [2049]:102123 (/var/lib/snapd/snaps/lxd_40657.snap)
/dev/loop2: [2049]:102124 (/var/lib/snapd/snaps/lxd_40424.snap (deleted))
/dev/loop3: [2049]:102152 (/var/lib/snapd/snaps/lxd_40638.snap)
=== RESULT ===
  LEAKED: lxcfs 3529, started under 6.9-ab8fad2 (rev 40424) in namespace mnt:[4026532781],
          is still running with no container left to serve.
          exe at start : /snap/lxd/40424/bin/lxcfs
          exe now      : /bin/lxcfs
          ppid=1  cgroup=0::/

Information to attach

Any relevant kernel output (dmesg)
Instance configuration (lxc config show <instance> --expanded)
Main daemon log (at /var/log/lxd/lxd.log or /var/snap/lxd/common/lxd/logs/lxd.log)
If a lxc command fails, output of the command with --debug
Output of the daemon with --debug (or use lxc monitor while reproducing the issue)

LXCFS reuse path in `daemon.start` is unreachable: every snap refresh leaks an LXCFS process and its mount namespace

RCA Summary

snapcraft/commands/daemon.start has a branch meant to reuse the running LXCFS across a snap refresh, reloading it in place with kill -USR1 instead of starting a second one:

if [ -e "${SNAP_COMMON}/var/lib/lxcfs/cgroup" ]; then
    echo "=> Re-using existing LXCFS"
    ...
    kill -USR1 "${CURRENT_PID}" || true
else
    ...
    echo "=> Starting LXCFS"

That branch is never taken. The gate is evaluated in the new revision's mount namespace, where ${SNAP_COMMON}/var/lib/lxcfs is an empty directory, so it is false on every host regardless of cgroup layout or LXCFS version.

Every daemon start therefore spawns a fresh LXCFS while the previous one is deliberately left running by daemon.stop. Nothing ever reaps the old one — not when its last container goes away, and not when snapd garbage-collects the revision it is executing from.

Why the gate is false

On a refresh snapd gives the daemon a new mount namespace. The surviving LXCFS's FUSE mount is in the old one and cannot reach the new one:

The shmounts tmpfs propagates one way only. The master lives in the namespace setup-shmounts pins at /var/snap/lxd/common/ns/shmounts (shared:N); every revision's daemon namespace holds that same superblock as a pure slave (master:N). Mounts made in the master propagate down into every revision — that is how the persistent tmpfs is shared — but mounts made inside a revision, which is where lxcfs creates its FUSE mount, propagate nowhere.
daemon.start does not re-establish it either. Its setup block only mkdir -ps ${SNAP_COMMON}/shmounts/lxcfs and re-creates the symlink; the FUSE mount is created solely by the lxcfs process itself, in the else branch further down.

daemon.start reports this itself. The message is guarded by a mountpoint test:

if ! mountpoint -q "${SNAP_COMMON}/var/lib/lxcfs"; then
    echo "====> Making LXCFS use the persistent path"

and it prints on every start, including every refresh — the daemon observing that nothing is mounted at the gate path each time.

Secondary, and not the binding reason: even if the mount were visible, /cgroup would not be there on a current snap. lxcfs 6.0 made cgroup emulation opt-in behind --enable-cgroup, default off (lxc/lxcfs@000b539f1b35), and daemon.start never passes it; lxc/lxcfs@8eb40435d31e removed the option entirely in 7.0.0. On a cgroup2 host --enable-cgroup produces an empty cgroup/ directory, so it is not a workaround worth taking. The namespace reason binds regardless of LXCFS version.

Consequences

One LXCFS process and one mount namespace leaked per refresh, permanently. Each leaked namespace is a full stale copy of that revision's mount table.
Superseded revisions do not free their disk. snapd garbage-collects the revision, but the leaked LXCFS is still executing from that squashfs, so its loop device stays attached to a deleted blob (~128 MB each).
LXCFS is never upgraded for running containers. The reuse branch's kill -USR1 is LXCFS's in-place reload (do_reload() → dlclose/dlopen("liblxcfs.so")), which keeps the same FUSE mount so every container bound to it picks up the new library. Because that never runs, containers stay pinned to whichever LXCFS existed when they started: in a separate run a container living across a refresh kept FUSE mount 0:66 while one started immediately afterwards got 0:93. An LXCFS fix shipped in a refresh therefore reaches new containers only, and one host ends up serving different containers from different LXCFS versions.
The "restart system to upgrade LXCFS" warnings never fire for the base-snap-changed and libfuse-soname-changed cases, so an administrator is never told that running containers are on an old LXCFS.

Reproducing

repro_lxcfs_leak.sh (attached) runs as root on a throwaway VM. It installs LXD from latest/stable, starts a container, refreshes along latest/candidate → latest/edge, then stops and deletes that container. These are ordinary store refreshes; nothing uses --dangerous.

Two refreshes rather than one because refresh.retain defaults to 2 — the third revision is what makes snapd evict the first, which is when the leaked process ends up executing a binary that is no longer installed. (latest/beta is omitted: it is ^, the same revision as candidate, so refreshing to it would only switch tracking.)

Verbatim output, Ubuntu 24.04 VM, cgroup2-only:

=== 0. install LXD from latest/stable ===
lxd 6.9-ab8fad2 from Canonical** installed
  installed: 6.9-ab8fad2 (rev 40424)

=== 1. start a container, before any refresh ===
  lxcfs pid=3529  ns=mnt:[4026532781]
  lxcfs exe=/snap/lxd/40424/bin/lxcfs
  c1 is served by      : 0:67  (lxcfs mount 0:67)

=== 2. snap refresh lxd --channel=latest/candidate, leaving c1 running ===
lxd (candidate) 6.9-029ed75 from Canonical** refreshed
  now: 6.9-029ed75 (rev 40638)
  LXCFS processes:
    pid=3529 ns=mnt:[4026532781] mount=0:67 exe=/snap/lxd/40424/bin/lxcfs
    pid=5498 ns=mnt:[4026532910] mount=0:94 exe=/snap/lxd/40638/bin/lxcfs

=== 2. snap refresh lxd --channel=latest/edge, leaving c1 running ===
lxd (edge) git-0a6bca2 from Canonical** refreshed
  now: git-0a6bca2 (rev 40657)
  LXCFS processes:
    pid=3529 ns=mnt:[4026532781] mount=0:67 exe=/bin/lxcfs
    pid=5498 ns=mnt:[4026532910] mount=0:94 exe=/snap/lxd/40638/bin/lxcfs
    pid=7109 ns=mnt:[4026532906] mount=0:100 exe=/snap/lxd/40657/bin/lxcfs

=== 3. stop and delete c1 -- the only container the old LXCFS was serving ===
  containers remaining:
  LXCFS processes now:
    pid=3529 ns=mnt:[4026532781] mount=0:67 exe=/bin/lxcfs
    pid=5498 ns=mnt:[4026532910] mount=0:94 exe=/snap/lxd/40638/bin/lxcfs
    pid=7109 ns=mnt:[4026532906] mount=0:100 exe=/snap/lxd/40657/bin/lxcfs

=== snap revisions and their backing files ===
Name  Version      Rev    Tracking     Publisher    Notes
lxd   6.9-029ed75  40638  latest/edge  canonical**  disabled
lxd   git-0a6bca2  40657  latest/edge  canonical**  -
/dev/loop4: [2049]:102123 (/var/lib/snapd/snaps/lxd_40657.snap)
/dev/loop2: [2049]:102124 (/var/lib/snapd/snaps/lxd_40424.snap (deleted))
/dev/loop3: [2049]:102152 (/var/lib/snapd/snaps/lxd_40638.snap)

=== RESULT ===
  LEAKED: lxcfs 3529, started under 6.9-ab8fad2 (rev 40424) in namespace mnt:[4026532781],
          is still running with no container left to serve.
          exe at start : /snap/lxd/40424/bin/lxcfs
          exe now      : /bin/lxcfs
          ppid=1  cgroup=0::/

Three points from that:

Two ordinary refreshes leave three LXCFS in three mount namespaces, with zero containers on the host. Stopping and deleting c1 does not reap the LXCFS that was serving it.
exe for pid 3529 changes from /snap/lxd/40424/bin/lxcfs to /bin/lxcfs across the second refresh: snapd evicted rev 40424 out from under a running process, which now executes a binary that is not installed anywhere.
The evicted revision's disk is not reclaimed. snap list --all no longer lists 40424, but /dev/loop2 is still attached to lxd_40424.snap (deleted) because pid 3529 holds it.

Nothing reaps these: daemon.stop only kills the pid recorded in lxcfs.pid, which each refresh replaces, and the processes sit outside the service cgroup (ppid=1, cgroup=0::/) so systemd's KillMode=control-group does not reach them either. They persist until the host reboots.

Suggested fix

The gate expression is not really the bug. Rewriting it to something version-independent — say, checking that lxcfs.pid names a live lxcfs process — would still leave it false after a refresh, because the mount genuinely is not there, and taking the reuse branch in that state would leave the new daemon with no LXCFS mount at all.

A real fix has to get the surviving LXCFS's mount into the new namespace: either make shmounts a shared peer group so submounts propagate between revisions, or have daemon.start re-attach the existing FUSE mount rather than only re-creating the directory and symlink. Only then does flipping the gate become safe.

If per-revision LXCFS is considered acceptable, the alternative is to reap the previous process once its last user is gone rather than reviving the reuse branch. Either closes the leak; the current behaviour does neither.

Found while investigating #18470, whose primary symptom (dataset is busy when deleting an instance after a refresh) is caused by the same surviving-LXCFS-pins-the-old-namespace mechanism. That is addressed separately in #18894; this issue is the reuse path itself, which that PR deliberately does not change.

@yinbin-dev

@yinbin-dev please can you do some git sleuthing to identify what caused this regression. I think its important to understand why it has occurred. Also which LXD series does this affect? Ta

#18933 (comment)

Since which commit

59018f512e — "Use external shared mount to survive core updates" (Stéphane Graber, 2018-07-30, closes #18).

Before that commit, ${SNAP_COMMON}/var/lib/lxcfs was a plain mkdir -p directory living
directly under SNAP_COMMON — no namespace trickery, so a still-running LXCFS's FUSE mount
there was visible to any later invocation of daemon.start, and the reuse check worked as
written:

if [ -e "${SNAP_COMMON}/var/lib/lxcfs/cgroup" ]; then
    echo "=> Re-using existing LXCFS"
    ...
    kill -USR1 "$(cat "${SNAP_COMMON}/lxcfs.pid")" || true

That commit turned ${SNAP_COMMON}/var/lib/lxcfs into a symlink onto a new persistent
${SNAP_COMMON}/shmounts/lxcfs tree, added for a different purpose entirely — surviving
core/base snap updates (issue #18), not LXCFS reuse. shmounts/setup-shmounts.c explicitly
marks that tree MS_SHARED and pins the master end at /var/snap/lxd/common/ns/shmounts;
every revision's daemon mount namespace only ever holds it as a slave.

Shared-subtree propagation is one-way: mounts made in the master propagate down to every
slave, but a mount made inside a slave — which is exactly where lxcfs runs and creates
its FUSE mount — propagates nowhere else. daemon.start never re-establishes the mount from
the master side either; its setup block only mkdir -ps ${SNAP_COMMON}/shmounts/lxcfs and
recreates the symlink.

So from that commit onward, [ -e "${SNAP_COMMON}/var/lib/lxcfs/cgroup" ] is evaluated in a
mount namespace where the path is structurally guaranteed to be an empty directory on every
refresh, regardless of cgroup layout or LXCFS version — the else branch (spawn a brand-new
LXCFS) always wins.

I checked the predicate itself across the full history of snapcraft/commands/daemon.start:
it has never been touched since — only cosmetic refactors nearby (helper-function extraction,
ldd-parsing rewrites, base-snap-name extraction).

This isn't really a "regression" in the usual sense of something that broke later — the reuse
branch appears to have never worked again after mid-2018. It went unnoticed for ~8 years
because a lingering LXCFS process/mount-namespace didn't visibly break anything until
deletion/GC paths started colliding with it (#18470, and now #18933).

Which LXD series are affected

Diffing snapcraft/commands/daemon.start across canonical/lxd branches shows an identical
shmounts/lxcfs symlink setup and an identical, equally dead var/lib/lxcfs/cgroup reuse
check in:

stable-4.0
stable-5.0
stable-5.21
main (→ 6.x)

Every currently maintained series is affected — this is not a main-only regression.

One real difference between series, worth noting but not the reason the path is dead:
the pinned LXCFS version in snapcraft.yaml differs.

stable-5.0 pins lxcfs-5.0.4 — cgroup emulation on by default.
stable-5.21 / main pin v6.0.6 — cgroup emulation opt-in since
lxc/lxcfs@000b539f1b35, off by default, and daemon.start never passes
--enable-cgroup.

That changes what /cgroup would contain if the mount were visible at all, but it doesn't
change the verdict: the mount-namespace propagation break makes the check unreachable on every
series regardless of LXCFS version — the namespace reason binds even where the cgroup file
would otherwise exist.

Which commit first introduced the reuse logic, and why

f9b247c57a — "Better handle snap refreshes" (Stéphane Graber, 2017-06-29).

This is the commit that introduced the whole snap-refresh handling pattern still in use
today, on both sides:

daemon.stop: added reason detection (snap refresh vs crashed vs
host shutdown) and, for the snap refresh case, made it stop only LXD, then
exit 0 — deliberately not touching LXCFS.

daemon.start: added the reuse check itself:

if [ -e "${SNAP_COMMON}/var/lib/lxcfs/cgroup" ]; then
    echo "=> Re-using existing LXCFS"
    kill -USR1 "$(cat "${SNAP_COMMON}/lxcfs.pid")" || true
else
    ...
    echo "=> Starting LXCFS"
    ( lxcfs "${SNAP_COMMON}/var/lib/lxcfs" -p "${SNAP_COMMON}/lxcfs.pid" & )
fi

Rationale (inferred from the paired diff — no PR description or issue reference in the
commit message): before this commit, daemon.stop had no refresh-aware path at all —
every stop, including a snap refresh, killed LXD and implicitly tore down LXCFS with it,
so every refresh meant a fresh LXCFS and, more importantly, briefly no /proc overlay for
any running container while LXD was down. The real fix was on the daemon.stop side: leave
LXCFS running across a refresh so containers keep their /proc overlays uninterrupted. The
daemon.start reuse branch is the necessary counterpart — since LXCFS is now still alive
when the new revision starts, it has to be signaled (kill -USR1, LXCFS's config-reload
signal) in place instead of blindly starting a second instance on top of it.

So the reuse logic was correctly designed for the world it was born into:
${SNAP_COMMON}/var/lib/lxcfs was a plain, unnamespaced directory at the time, so the FUSE
mount left behind by the still-running LXCFS was visible to the next daemon.start
invocation. It only became unreachable a year later, once 59018f512e (2018-07-30) rerouted
that same path through the one-way shared/slave shmounts mount tree described above.

My gut feeling is that re-activating the lxcfs reuse branch that wasn't used for so long might be more dangerous than just properly reaping the old lxcfs process and always spinning up a new one. This might be a good thing in a few aspects

static stability. Running containers aren't immediately broken because they are still using the old lxcfs, a dlclose/dlopen might break them otherwise, i.e. if a non backward compatible change is made in lxcfs but somehow versioned incorrectly as ABI compatible, the new lxcfs always approach would tolerate that. The current behavior basically treats every lxcfs update as non ABI compatible.
potentially it opens door to a cleaner 'rollback' story if we ever want to support that.

didn't visibly break anything until
deletion/GC paths started colliding with it (#18470, and now #18933).

Do we know why these started showing up recently?

But i think its worth fixing so we dont leave lingering mounts, but only on the main branch.

didn't visibly break anything until
deletion/GC paths started colliding with it (#18470, and now #18933).

Do we know why these started showing up recently?

So this leakage is not reported by any issue, it's a drive-by discovery when I was working on the #18470 .
It's also not manifesting itself in any violent way, it's just a silent leaked process there which a reboot will fix. With the direct fixes for the 3 snap-refresh-related issues, nothing is visibly broken at the moment.

But i think its worth fixing so we dont leave lingering mounts, but only on the main branch.

To be faithful to the original intention, the fix would be to reuse the lxcfs process that lxd intentionally left untouched during refresh. It would hot reload liblxcfs.so when it's ABI compatible. The treatment of non ABI compatible liblxcfs.so is however a bit lackadaisical in the original implementation (just a warning message)

lxd/snapcraft/commands/daemon.start

Lines 664 to 667 in e1a269f

    
           if [ "${CURRENT_BASE}" != "${NEW_BASE}" ]; then 
        
               echo "==> snap base has changed, restart system to upgrade LXCFS" 
        
           elif [ "${CURRENT_FUSE}" != "${NEW_FUSE}" ]; then 
        
               echo "==> FUSE version mismatch, restart system to upgrade LXCFS"

I think a better approach in the non hot reloadable case, it might make sense to spin up a new lxcfs process for new containers like we do in the else branch. That means we also need to fix the reaping problem independently when the existing containers are all gone (restarted or deleted).

From what I can tell, implementing the 'reaping' is probably a bigger effort (I'll time-box the research on this). A compromise would be to let it leak if it's truly a non compatible upgrade for liblxcfs which arguably is a less common case. And in that scenario, we would rely on a host reboot to fix.

This looks similar to an issue Ben found in Workshop: canonical/workshop#1062. It's only transient so not really a big deal, but would be nice to get this fixed at some point.

I have an alternative reproducer here: canonical/workshop#1062 (comment)

Also some AI context:

a full stop deletes ${SNAP_COMMON}/ns on the host.

lxd/snap/local/commands/daemon.stop

Lines 173 to 175 in 02ae75e

    
           ## Flush our shared namespace from the host 
        
           echo "=> Cleaning up namespaces" 
        
           nsenter -t 1 -m umount -l "${SNAP_COMMON}/ns" 2>/dev/null || true

ns/mntns is only recreated when setup-shmounts runs, and the next start skips it because the snap's namespace survived.

lxd/shmounts/setup-shmounts.c

Lines 448 to 465 in 02ae75e

    
           // Save our current mntns 
        
           fd = open("/var/snap/lxd/common/ns/mntns", O_CREAT | O_RDWR, 0600); 
        
           if (fd < 0) { 
        
           	fprintf(stderr, "Failed to create new mntns mountpoint: %s\n", strerror(errno)); 
        
           	return -1; 
        
           } 
        
           close(fd); 
        
           // Make sure that the ns path is still private 
        
           if (mount("none", "/var/snap/lxd/common/ns", NULL, MS_PRIVATE, NULL) < 0) { 
        
           	fprintf(stderr, "Failed to mark mount as private: %s\n", strerror(errno)); 
        
           	return -1; 
        
           } 
        
           if (mount("/run/snapd/ns/lxd.mnt", "/var/snap/lxd/common/ns/mntns", NULL, MS_BIND, NULL) < 0) { 
        
           	fprintf(stderr, "Failed to mount new mntns: %s\n", strerror(errno)); 
        
           	return -1; 
        
           }

Without it, the mount-moving code has nothing to work from.

lxd/shmounts/setup-shmounts.c

Lines 336 to 337 in 02ae75e

    
           nsfd_old = open("/var/snap/lxd/common/ns/mntns", O_RDONLY); 
        
           if (nsfd_old >= 0) {

the -e var/lib/lxcfs/cgroup check is always false on cgroup v2.

lxd/snap/local/commands/daemon.start

Line 663 in 02ae75e

if [ -e "${SNAP_COMMON}/var/lib/lxcfs/cgroup" ]; then

So a second lxcfs is started, and the cleanup never reaches the old one.

lxd/snap/local/commands/daemon.start

Lines 749 to 763 in 02ae75e

    
           if [ -n "${PREV_MNT_NS}" ] && [ "$(readlink "/proc/self/ns/mnt")" != "${PREV_MNT_NS}" ]; then 
        
               echo "==> Cleaning up existing LXCFS namespace" 
        
               # Read mountinfo directly, extracting the mount point (5th column) 
        
               while read -r _ _ _ _ mp _; do 
        
                   # Prefix match using case and glob 
        
                   case "${mp}" in 
        
                       "${SNAP_COMMON}/shmounts/storage-pools/"* | "${SNAP_COMMON}/lxd/storage-pools/"*) 
        
                           nsenter -t "${PID}" -m umount -l "${mp}" || true 
        
                           ;; 
        
                       *) 
        
                           ;; 
        
                   esac 
        
               done < "/proc/${PID}/mountinfo" || true 
        
           fi

Navigation Menu

Repository navigation

lxcfs process leaked after snap refresh #18933

Please confirm

Distribution

Distribution version

Output of `snap list --all lxd core20 core22 core24 core26 snapd`

System info

Instance log

Expected behavior

Actual behavior

Steps to reproduce

Repro Output

Information to attach

LXCFS reuse path in `daemon.start` is unreachable: every snap refresh leaks an LXCFS process and its mount namespace

RCA Summary

Why the gate is false

Consequences

Reproducing

Suggested fix

Related

Since which commit

Which LXD series are affected

Which commit first introduced the reuse logic, and why

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	if [ "${CURRENT_BASE}" != "${NEW_BASE}" ]; then
	echo "==> snap base has changed, restart system to upgrade LXCFS"
	elif [ "${CURRENT_FUSE}" != "${NEW_FUSE}" ]; then
	echo "==> FUSE version mismatch, restart system to upgrade LXCFS"

	## Flush our shared namespace from the host
	echo "=> Cleaning up namespaces"
	nsenter -t 1 -m umount -l "${SNAP_COMMON}/ns" 2>/dev/null \|\| true

	// Save our current mntns
	fd = open("/var/snap/lxd/common/ns/mntns", O_CREAT \| O_RDWR, 0600);
	if (fd < 0) {
	fprintf(stderr, "Failed to create new mntns mountpoint: %s\n", strerror(errno));
	return -1;
	}
	close(fd);

	// Make sure that the ns path is still private
	if (mount("none", "/var/snap/lxd/common/ns", NULL, MS_PRIVATE, NULL) < 0) {
	fprintf(stderr, "Failed to mark mount as private: %s\n", strerror(errno));
	return -1;
	}

	if (mount("/run/snapd/ns/lxd.mnt", "/var/snap/lxd/common/ns/mntns", NULL, MS_BIND, NULL) < 0) {
	fprintf(stderr, "Failed to mount new mntns: %s\n", strerror(errno));
	return -1;
	}

	nsfd_old = open("/var/snap/lxd/common/ns/mntns", O_RDONLY);
	if (nsfd_old >= 0) {

	if [ -n "${PREV_MNT_NS}" ] && [ "$(readlink "/proc/self/ns/mnt")" != "${PREV_MNT_NS}" ]; then
	echo "==> Cleaning up existing LXCFS namespace"

	# Read mountinfo directly, extracting the mount point (5th column)
	while read -r _ _ _ _ mp _; do
	# Prefix match using case and glob
	case "${mp}" in
	"${SNAP_COMMON}/shmounts/storage-pools/"* \| "${SNAP_COMMON}/lxd/storage-pools/"*)
	nsenter -t "${PID}" -m umount -l "${mp}" \|\| true
	;;
	*)
	;;
	esac
	done < "/proc/${PID}/mountinfo" \|\| true
	fi

lxcfs process leaked after snap refresh #18933

Description

Please confirm

Distribution

Distribution version

Output of snap list --all lxd core20 core22 core24 core26 snapd

System info

Instance log

Expected behavior

Actual behavior

Steps to reproduce

Repro Output

Information to attach

Activity

yinbin-dev commented on Aug 28, 2026

LXCFS reuse path in daemon.start is unreachable: every snap refresh leaks an LXCFS process and its mount namespace

RCA Summary

Why the gate is false

Consequences

Reproducing

Suggested fix

Related

tomponline commented on Aug 28, 2026

yinbin-dev commented on Aug 28, 2026

Since which commit

Which LXD series are affected

Which commit first introduced the reuse logic, and why

yinbin-dev commented on Aug 28, 2026

tomponline commented on Aug 28, 2026

tomponline commented on Aug 28, 2026

yinbin-dev commented on Aug 28, 2026

yinbin-dev commented on Aug 28, 2026

jonathan-conder commented on Sep 30, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Output of `snap list --all lxd core20 core22 core24 core26 snapd`

LXCFS reuse path in `daemon.start` is unreachable: every snap refresh leaks an LXCFS process and its mount namespace