That's pretty odd. The only thing I can really think of is that we may have some logic on Update() which attempts to activate/deactivate the instance volume and is somehow causing the unmount, but that's pretty weird.

So, the problem is indeed in Update:

I think because I’m deleting the directory and recreating it, the bindmount still points to the deleted entry.

So, not sure if I should continue doing my initialization in initLXC, and if so, if I can really rely on cConfig to tell me that the initialization has already been done (because the reverter in Update resets it)…

At least now it works. I don’t know how that would play with live-migration, but I also don’t think I should really bother, as live-migrating a systemd container is a very bad idea…

I’m not sure all those failing tests are related to the PR

I think they are, they're all failing in the same way and the only stuff that's passing is cluster tests where no mount is involved (dir/btrfs).

Hmmm ok, I’ll run the test suite locally then.

Ok there’s something:

root@incus-dev:~/incus/test# grep -r .incus-systemd-credentials tmp.oJD/SuiwKDXIw
tmp.oJD/SuiwKDXIw/run/c1/lxc.conf:lxc.environment = CREDENTIALS_DIRECTORY=/dev/.incus-systemd-credentials
tmp.oJD/SuiwKDXIw/run/c1/lxc.conf:lxc.mount.entry = /root/incus/test/tmp.oJD/SuiwKDXIw/containers/c1/credentials dev/.incus-systemd-credentials none bind,ro,create=dir 0 0
tmp.oJD/SuiwKDXIw/logs/c1/lxc.log:lxc c1 20251007192936.694 ERROR    utils - ../src/lxc/utils.c:safe_mount:1330 - No such file or directory - Failed to mount "/root/incus/test/tmp.oJD/SuiwKDXIw/containers/c1/credentials" onto "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/.incus-systemd-credentials"
tmp.oJD/SuiwKDXIw/logs/c1/lxc.log:lxc c1 20251007192936.694 ERROR    conf - ../src/lxc/conf.c:mount_entry:2217 - No such file or directory - Failed to mount "/root/incus/test/tmp.oJD/SuiwKDXIw/containers/c1/credentials" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/.incus-systemd-credentials"

But

root@incus-dev:~/incus/test# ls -al /root/incus/test/tmp.oJD/SuiwKDXIw/containers/c1/credentials
total 0
d--x------ 2 1000000 1000000 40 Oct  7 19:29 .
d--x------ 3 root    root    60 Oct  7 19:29 ..

This part however, /usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/.incus-systemd-credentials is very intriguing (well, to me at least).

This part however, /usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/.incus-systemd-credentials is very intriguing (well, to me at least).

That's normal, /usr/lib/x86_64-linux-gnu/lxc/rootfs/ is the temporary mount point of the instance's root disk before pivot_root and such.

The error could happen on either side of the mount, so it could be because the source of the mount is somehow unavailable at the time the mount is attempted or because the mount target doesn't exist.

Now create=dir should in theory take care of the target part, making the source more likely to be an issue though I'm not too sure what may be going on there.

,optional seems to help quite a bit, but to be honest, I don’t really understand what I’m doing on that mount line…
I’m looking at the new errors.

Definitely shouldn't need optional as that would make it skip the missing source path which isn't really what we want here, we always want that mount.

Did you figure out a simple reproducer for the failure? Then I can take a look at it.

Ok I’ll revert the recent changes and I’ll try to produce a MWE. This is probably a rather trivial oversight; I’m quite new to the LXC driver codebase, and I’m still figuring out its general structure.

Edit: looks like it fails with tests using ZFS

I can reproduce the problem on lvm and zfs drivers, but not dir or btrfs:

root@incus-dev:~/incus# incus launch testimage test-1 --profile dir
Launching test-1
root@incus-dev:~/incus# incus launch testimage test-2 --profile zfs
Launching test-2
Error: Failed to run: /root/go/bin/incusd forkstart test-2 /var/lib/incus/containers /run/incus/test-2/lxc.conf: exit status 1

So it looks like something is mounted too late when working with logical volume managers, but I’m having some trouble understanding the whole LXC driver dance for now.

Okay, I should be able to easily test it here on ZFS then.

Nice. Please use c87d666, as it doesn’t include my desperate attempts :)

I found the problem:

root@dakara:~# ls /var/lib/incus/containers/d13/
backup.yaml  metadata.yaml  rootfs  templates
root@dakara:~# umount /var/lib/incus/containers/d13/
root@dakara:~# ls /var/lib/incus/containers/d13/
credentials
root@dakara:~# 

So basically the credentials folder was created prior to the instance's volume being mounted.
So when the instance volume does get mounted, it hides the credentials folder, causing the error.

Oh, makes perfect sense! Thanks a lot, I’ll review that tomorrow.

Aaaaah it looks good now.
I’m not 100% sure about the idmap, and I don’t fully grasp the implications of live changes of the idmap on that mounted directory. Any pointers?
I’ll write a few tests in the meantime.

The tests are showing some rather strange race conditions on update (or maybe setupCredentials not getting called at all) where the DB gets updated but not the FS files

Edit: the tests have shown something that I completely overlooked: if a user sets both systemd.credential.foo and systemd.credential-binary.foo.

Great, should be ready to review now