system.etc.overlay.enable crashes systemd when doing switch-to-configuration test #287261
Description
Describe the bug
Steps To Reproduce
Steps to reproduce the behavior:
- Set
system.etc.overlay.enable = true; - Run /run/current-system/bin/switch-to-configuration test
Expected behavior
Does not crash system/systemd
Screenshots
vm-test-run-sops-user-passwords-sysusers> machine # activating the configuration...
vm-test-run-sops-user-passwords-sysusers> machine # mount: /tmp/tmp.6nsMFjSpvc: /dev/loop0 already mounted or mount point busy.
vm-test-run-sops-user-passwords-sysusers> machine # dmesg(1) may have more information after failed mount system call.
vm-test-run-sops-user-passwords-sysusers> machine # [ 11.430221] overlayfs: upperdir is in-use as upperdir/workdir of another mount, accessing files from both mounts will result in undefined behavior.
vm-test-run-sops-user-passwords-sysusers> machine # [ 11.431890] overlayfs: workdir is in-use as upperdir/workdir of another mount, accessing files from both mounts will result in undefined behavior.
vm-test-run-sops-user-passwords-sysusers> machine # Moving mount
vm-test-run-sops-user-passwords-sysusers> machine # Mounting beneath top mount
vm-test-run-sops-user-passwords-sysusers> machine # [ 11.472760] systemd[1]: Reloading requested from client PID 876 ('systemctl') (unit backdoor.service)...
vm-test-run-sops-user-passwords-sysusers> machine # [ 11.476360] systemd[1]: Reloading...
vm-test-run-sops-user-passwords-sysusers> machine # [ 11.630551] systemd[1]: Assertion 'close_nointr(fd) != -EBADF' failed at src/basic/fd-util.c:75, function safe_close(). Aborting.
vm-test-run-sops-user-passwords-sysusers> machine # [ 11.633418] systemd[1]: Caught <ABRT>, from our own process.
vm-test-run-sops-user-passwords-sysusers> machine # [ 11.653222] systemd-coredump[898]: Due to PID 1 having crashed coredump collection will now be turned off.
vm-test-run-sops-user-passwords-sysusers> machine # [ 11.739104] systemd-coredump[898]: Process 897 (systemd) of user 0 dumped core.
vm-test-run-sops-user-passwords-sysusers> machine #
vm-test-run-sops-user-passwords-sysusers> machine # Module libbz2.so.1 without build-id.
vm-test-run-sops-user-passwords-sysusers> machine # Module libz.so.1 without build-id.
vm-test-run-sops-user-passwords-sysusers> machine # Module libbpf.so.1 without build-id.
vm-test-run-sops-user-passwords-sysusers> machine # Module libgpg-error.so.0 without build-id.
vm-test-run-sops-user-passwords-sysusers> machine # Module libattr.so.1 without build-id.
vm-test-run-sops-user-passwords-sysusers> machine # Module libzstd.so.1 without build-id.
vm-test-run-sops-user-passwords-sysusers> machine # Module liblzma.so.5 without build-id.
vm-test-run-sops-user-passwords-sysusers> machine # Module libpam.so.0 without build-id.
vm-test-run-sops-user-passwords-sysusers> machine # Module liblz4.so.1 without build-id.
vm-test-run-sops-user-passwords-sysusers> machine # Module libgcrypt.so.20 without build-id.
vm-test-run-sops-user-passwords-sysusers> machine # Module libcrypt.so.2 without build-id.
vm-test-run-sops-user-passwords-sysusers> machine # Module libcap.so.2 without build-id.
vm-test-run-sops-user-passwords-sysusers> machine # Module libacl.so.1 without build-id.
vm-test-run-sops-user-passwords-sysusers> machine # Module libseccomp.so.2 without build-id.
vm-test-run-sops-user-passwords-sysusers> machine # Module libkmod.so.2 without build-id.
vm-test-run-sops-user-passwords-sysusers> machine # Module libaudit.so.1 without build-id.
vm-test-run-sops-user-passwords-sysusers> machine # Module libapparmor.so.1 without build-id.
vm-test-run-sops-user-passwords-sysusers> machine # Module libsystemd-shared-255.so without build-id.
vm-test-run-sops-user-passwords-sysusers> machine # Module libsystemd-core-255.so without build-id.
vm-test-run-sops-user-passwords-sysusers> machine # Module systemd without build-id.
vm-test-run-sops-user-passwords-sysusers> machine # Stack trace of thread 897:
vm-test-run-sops-user-passwords-sysusers> machine # #0 0x00007fd90d5aef4d syscall (libc.so.6 + 0x10bf4d)
vm-test-run-sops-user-passwords-sysusers> machine # #1 0x00007fd90d8a6e81 propagate_signal (libsystemd-shared-255.so + 0x21ae81)
vm-test-run-sops-user-passwords-sysusers> machine # #2 0x000056345928fcd6 crash (systemd + 0x10cd6)
vm-test-run-sops-user-passwords-sysusers> machine # #3 0x00007fd90d4e0eb0 __restore_rt (libc.so.6 + 0x3deb0)
vm-test-run-sops-user-passwords-sysusers> machine # #4 0x00007fd90d53002c __pthread_kill_implementation (libc.so.6 + 0x8d02c)
vm-test-run-sops-user-passwords-sysusers> machine # #5 0x00007fd90d4e0e06 raise (libc.so.6 + 0x3de06)
vm-test-run-sops-user-passwords-sysusers> machine # #6 0x00007fd90d4c98f5 abort (libc.so.6 + 0x268f5)
vm-test-run-sops-user-passwords-sysusers> machine # #7 0x00007fd90d717d1b log_assert_failed.cold (libsystemd-shared-255.so + 0x8bd1b)
vm-test-run-sops-user-passwords-sysusers> machine # #8 0x00007fd90d86e90f safe_close (libsystemd-shared-255.so + 0x1e290f)
vm-test-run-sops-user-passwords-sysusers> machine # #9 0x00007fd90db96fde service_deserialize_item (libsystemd-core-255.so + 0x121fde)
vm-test-run-sops-user-passwords-sysusers> machine # #10 0x00007fd90dbba604 unit_deserialize_state (libsystemd-core-255.so + 0x145604)
vm-test-run-sops-user-passwords-sysusers> machine # #11 0x00007fd90db5f483 manager_deserialize (libsystemd-core-255.so + 0xea483)
vm-test-run-sops-user-passwords-sysusers> machine # #12 0x00007fd90db68b5c manager_reload (libsystemd-core-255.so + 0xf3b5c)
vm-test-run-sops-user-passwords-sysusers> machine # #13 0x0000563459289646 main (systemd + 0xa646)
vm-test-run-sops-user-passwords-sysusers> machine # #14 0x00007fd90d4cb0ce __libc_start_call_main (libc.so.6 + 0x280ce)
vm-test-run-sops-user-passwords-sysusers> machine # #15 0x00007fd90d4cb189 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x28189)
vm-test-run-sops-user-passwords-sysusers> machine # #16 0x00005634592898a5 _start (systemd + 0xa8a5)
vm-test-run-sops-user-passwords-sysusers> machine # ELF object binary architecture: AMD x86-64
vm-test-run-sops-user-passwords-sysusers> machine #
vm-test-run-sops-user-passwords-sysusers> machine # Reload daemon failed: Connection reset by peer
vm-test-run-sops-user-passwords-sysusers> machine # [ 11.781448] systemd[1]: Caught <ABRT>, dumped core as pid 897.
vm-test-run-sops-user-passwords-sysusers> machine # [ 11.786629] systemd[1]: Freezing execution.
vm-test-run-sops-user-passwords-sysusers> machine # restarting sysinit-reactivation.target
vm-test-run-sops-user-passwords-sysusers> machine # [ 12.659552] systemd-oomd[383]: Failed to connect to /run/systemd/io.systemd.ManagedOOM: Connection refused
vm-test-run-sops-user-passwords-sysusers> machine # [ 12.662089] systemd-oomd[383]: Failed to acquire varlink connection: Connection refused
vm-test-run-sops-user-passwords-sysusers> machine # [ 12.664328] systemd-oomd[383]: Event loop failed: Connection refused
vm-test-run-sops-user-passwords-sysusers> machine # [ 14.420117] systemd-udevd[348]: Failed to send readiness notification, ignoring: Connection refused
Additional context
This can be reproduced using this test in sops-nix:
$ git clone https://github.com/Mic92/sops-nix
$ git checkout sysusers
Edit pkgs/sops-install-secrets/nixos-test.nix to have this change
diff --git a/pkgs/sops-install-secrets/nixos-test.nix b/pkgs/sops-install-secrets/nixos-test.nix
index f44f790..4430bcd 100644
--- a/pkgs/sops-install-secrets/nixos-test.nix
+++ b/pkgs/sops-install-secrets/nixos-test.nix
@@ -32,7 +32,6 @@ let
machine.succeed("cat /run/secrets-for-users/test_key | grep -q 'test_value'") # the user password still exists
# BUG in nixos's overlayfs... systemd crashes on switch-to-configuration test
- '' + pkgs.lib.optionalString (!(extraConfig ? system.etc.overlay.enable)) ''
machine.succeed("/run/current-system/bin/switch-to-configuration test")
machine.succeed("cat /run/secrets/nested/test/file | grep -q 'another value'") # the regular secrets still work after a switch
machine.succeed("cat /run/secrets-for-users/test_key | grep -q 'test_value'") # the user password is still present after a switch$ nix build .#checks.x86_64-linux.user-passwords-sysuser -L