libvirtd socket activation not working since 253.5

systemd version the issue has been seen with

253.5

Used distribution

Arch Linux (rolling)

Linux kernel version used

6.3.5-arch1-1

CPU architectures issue was seen on

x86_64

Component

systemd

Expected behaviour you didn't see

In its default configuration, libvirtd.service starts a libvirtd process that times out after 120 seconds of inactivity (no running domains and no connected clients). The service remains active and libvirtd is respawned on-demand via systemd socket activation (libvirtd-ro.socket, libvirtd.socket and libvirtd-admin.socket correspond to /var/lib/libvirt/{libvirt-sock-ro,libvirt-sock,libvirt-admin-sock}` respectively by default).

Unexpected behaviour you saw

After updating from systemd 253.4 to 253.5, socket activation stopped working. Attempting to connect to libvirtd via a socket after it times out yields nothing. Client software (I'm using virsh and virt-manager on the same host) halts indefinitely as soon as a libvirtd-dependent command is issued, and no errors are reported or logged before or after they are terminated (e.g. with Ctrl+C), including by systemd.

If systemctl restart libvirtd.service is issued in a separate terminal, libvirtd starts as normal and any processes trying to connect via a socket succeed immediately (there is no need to cancel the attempt and try again).

After rolling back systemd, libvirt and other updates installed at the same time in various combinations, the culprit appears to be systemd 253.5. Downgrading to 253.4 restores the desired behaviour.

Steps to reproduce the problem

Install a libvirt package with stock systemd unit files
Start libvirtd and/or enable its sockets as needed
Wait for the libvirtd process to terminate after 2 minutes
Run sudo virsh list or similar
Wait indefinitely

Additional program output to the terminal or log subsystem illustrating the issue

Here's output from strace, in the absence of anything appearing in a log entry:

https://gist.github.com/lkrms/5bb9ef2e7ad8aa5d89e164b6e3f4d4a6#file-gistfile1-txt

This is not a useful bug report. Always provide debug logs that show what's going on, i.e. "systemd-analyze log-level debug" and then reproduce the issue and provide what the service manager (i.e. pid1) logged as result of the activation request.

what makes you think this is a systemd issue? where's this tracked at libvirt project?

After rolling back systemd, libvirt and other updates installed at the same time in various combinations, the culprit appears to be systemd 253.5. Downgrading to 253.4 restores the desired behaviour.

Uh, stable dot releases are not tracked in this repo, that's a systemd-stable git repo thing.

Can you reproduce this on current git main?

Bisected to ff32060 (Confirmed. Reverting this single commit makes it work.)
Not sure why this causes it... I haven't tried current git main yet...

Upstream libvirt issue here [1], but I believe it's barking up the wrong tree and contains misinformation (including from me).

libvirt daemons are documented here [2]. I am using the libvirt monolithic daemon.

[1] https://gitlab.com/libvirt/libvirt/-/issues/483
[2] https://libvirt.org/daemons.html

BTW, when libvirtd is working properly, this is often found in the journal which could well be related (it's been like this for years IIRC):

systemd[1]: libvirtd.service: Found left-over process 1125 (dnsmasq) in control group while starting unit. Ignoring.
systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
systemd[1]: libvirtd.service: Found left-over process 1126 (dnsmasq) in control group while starting unit. Ignoring.
systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.

the dnsmasq processes are definitely still running when the libvirtd.service shuts down...

the dnsmasq processes are definitely still running when the libvirtd.service shuts down...

Libvirt intentionally leaves various things running when it shuts down, because it needs to be able to stop & start (for software upgrades) without disrupting running guests. Hence it sets KillMode=process (as opposd to cgroup)

if libvirt watns to leave stuff running, it should probably move that stuff to a distinct scope (or service), so that it is unaffected by libvirt restarts.

@msekletar

Fedora Rawhide also impacted:

https://bugzilla.redhat.com/show_bug.cgi?id=2213660

/cc @msekletar

What is the intended behavior of systemd? When is systemd supposed to change the status of a service from "Active" to "Inactive"?
Is it
A: when the process with the Main PID exits?
or
B: when all processes in the CGroup have exited?

The commit ff32060 seems to flip the behavior from A to B. As libvirt depends on behavior A it worked so far and now as the behavior has changed to B libvirt fails. So the question for me here is: Is behavior B intended? If yes, then libvirt needs to be adapted. If not, it's a systemd bug.

Hi guys!! To temporarily fix the issue follow the guide that I have mentioned in the arch forum - https://bbs.archlinux.org/viewtopic.php?pid=2104316#p2104316

What is the intended behavior of systemd? When is systemd supposed to change the status of a service from "Active" to "Inactive"? Is it A: when the process with the Main PID exits? or B: when all processes in the CGroup have exited?

This depends on ExitType=. It should be when the main process goes down (ExitType=main, the deault), or when the last process in the service's cgroup goes down (ExitType=cgroup).

The commit ff32060 seems to flip the behavior from A to B. As libvirt depends on behavior A it worked so far and now as the behavior has changed to B libvirt fails. So the question for me here is: Is behavior B intended? If yes, then libvirt needs to be adapted. If not, it's a systemd bug.

I don't follow here. Can you explain why reseting the PID known state will make this behave like ExitType=cgroup?

@Toolybird

Thanks for the explanation (I'm not too familiar with systemd). Good that it can be configured with ExitType= and by default it should do what libvirt expects. But why doesn't it do it all of a sudden? And no, I cannot explain why this one commit changed this behavior, I went with the bisection search result of @Toolybird . I can only wildly guess that when the main pid is set to unknown, determining whether the service is still active based on the main pid is not possible anymore, but this would need to be confirmed with a dive into the source code.

Maybe we should add check for exit type in service_good(). Hence we wouldn't make decisions about state transitions based on cgroup state for services which have ExitType=main. Also, we might want to rename service_good() to something more descriptive.

@poettering

I don't follow here. Can you explain why reseting the PID known state will make this behave like ExitType=cgroup?

@poettering Setting main_pid_known to false will cause main_pid_good to return -EAGAIN:

systemd/src/core/service.c

Lines 1847 to 1861 in 768fcd7

    
                   if (s->main_pid_known) { 
        
                           /* If it's an alien child let's check if it is still 
        
                            * alive ... */ 
        
                           if (s->main_pid_alien && s->main_pid > 0) 
        
                                   return pid_is_alive(s->main_pid); 
        
                           /* .. otherwise assume we'll get a SIGCHLD for it, 
        
                            * which we really should wait for to collect exit 
        
                            * status and code */ 
        
                           return s->main_pid > 0; 
        
                   } 
        
                   /* We don't know the pid */ 
        
                   return -EAGAIN;

which causes service_good to erroneously fall back to cgroup_good:

systemd/src/core/service.c

Lines 2223 to 2233 in 768fcd7

    
                   main_pid_ok = main_pid_good(s); 
        
                   if (main_pid_ok > 0) /* It's alive */ 
        
                           return true; 
        
                   if (main_pid_ok == 0) /* It's dead */ 
        
                           return false; 
        
                   /* OK, we don't know anything about the main PID, maybe 
        
                    * because there is none. Let's check the control group 
        
                    * instead. */ 
        
                   return cgroup_good(s) != 0;

effectively causing it to behave like ExitType=cgroup instead.

As can be seen in main_pid_good it seems like the way it is intended to be is for the main_pid to be set to 0 when SIGCHLD is received while main_pid_known remains true.

core/service: don't make state transitions based on cgroup state when…

Just tested PR #28035 and can confirm it makes libvirt happy again. Thanks!

a simple test case would be this:

[Unit]
Description=test bug

[Service]
Type=oneshot
ExecStart=bash -c "nohup sleep infinity & exit"

● test.service - test bug
     Loaded: loaded (/etc/systemd/system/test.service; static)
     Active: active (running) since Fri 2023-06-16 18:20:20 CEST; 2s ago
    Process: 36430 ExecStart=bash -c nohup sleep infinity & exit (code=exited, status=0/SUCCESS)
   Main PID: 36430 (code=exited, status=0/SUCCESS)
      Tasks: 1 (limit: 4656)
     Memory: 200.0K
        CPU: 3ms
     CGroup: /system.slice/test.service
             └─36431 sleep infinity

[root@arch-testing system]# systemctl show -p KillMode test.service
KillMode=control-group
[root@arch-testing system]# systemctl show -p ExitType test.service
ExitType=main

after the main bash process exits with 0, the background child is not killed.

ps.

bash -c "nohup sleep infinity & exit 1" on the other hand, fails the service AND kills the child processes too.

if libvirt watns to leave stuff running, it should probably move that stuff to a distinct scope (or service), so that it is unaffected by libvirt restarts.

Yes, I don't disagree there - it would be a good idea if libvirt got its on-demand launched dnsmasq to be a (transient) systemd unit. We just had never considered that a priority thing, as we mostly thought the ExitType=main setting was intended to be compatible with leaving the process running when switching over from sysv-init originally & it had worked until now.

Add workaround for libvirt stuck issue for CRC

test: add coverage for systemd#27953

Merge pull request #28835 from mrc0mmand/more-tests

I'm useing ubuntu 24.04 desktop, and I'm trying to start libvirtd, but it auto turn inactive after 2 minutes. I'v run journalctl -u libvirtd.service -b and got logs like this:

Mar 15 22:34:20 right systemd[1]: libvirtd.service: Found left-over process 2362 (dnsmasq) in control group while starting unit. Ignoring.
Mar 15 22:34:20 right systemd[1]: libvirtd.service: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Mar 15 22:34:20 right systemd[1]: Starting libvirtd.service - libvirt legacy monolithic daemon...
Mar 15 22:34:21 right systemd[1]: Started libvirtd.service - libvirt legacy monolithic daemon.
Mar 15 22:34:21 right dnsmasq[2362]: read /etc/hosts - 101 names
Mar 15 22:34:21 right dnsmasq[2362]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 2 names
Mar 15 22:36:21 right systemd[1]: libvirtd.service: Deactivated successfully.
Mar 15 22:36:21 right systemd[1]: libvirtd.service: Unit process 2362 (dnsmasq) remains running after unit stopped.

I don't know what can I do now, please help me. TKS!

@iamwo0

@iamwo0, That looks like expected behavior. by default libvirtd process times out (exit) after 120 seconds of inactivity (assuming there are no running domains and no connected clients), and this causes libvirtd.service to change state to inactive. libvirtd.service is reactivated automatically by the socket (e.g. by running virsh list).

Are you facing any problem due to libvirtd.service changing to inactive?

@iamwo0

@iamwo0 if you currently have problems with libvirtd please file them on libvirt's bug tracker (https://gitlab.com/libvirt/libvirt/-/issues/) rather than systemd's, as it is highly unlikely to be related to this old systemd issue. Further, as a general rule it is better to file new bugs, instead of commenting on bugs that were closed a long time ago.

Navigation Menu

Repository navigation

libvirtd socket activation not working since 253.5 #27953

systemd version the issue has been seen with

Used distribution

Linux kernel version used

CPU architectures issue was seen on

Component

Expected behaviour you didn't see

Unexpected behaviour you saw

Steps to reproduce the problem

Additional program output to the terminal or log subsystem illustrating the issue

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	if (s->main_pid_known) {

	/* If it's an alien child let's check if it is still
	* alive ... */
	if (s->main_pid_alien && s->main_pid > 0)
	return pid_is_alive(s->main_pid);

	/* .. otherwise assume we'll get a SIGCHLD for it,
	* which we really should wait for to collect exit
	* status and code */
	return s->main_pid > 0;
	}

	/* We don't know the pid */
	return -EAGAIN;

	main_pid_ok = main_pid_good(s);
	if (main_pid_ok > 0) /* It's alive */
	return true;
	if (main_pid_ok == 0) /* It's dead */
	return false;

	/* OK, we don't know anything about the main PID, maybe
	* because there is none. Let's check the control group
	* instead. */

	return cgroup_good(s) != 0;

libvirtd socket activation not working since 253.5 #27953

Description

systemd version the issue has been seen with

Used distribution

Linux kernel version used

CPU architectures issue was seen on

Component

Expected behaviour you didn't see

Unexpected behaviour you saw

Steps to reproduce the problem

Additional program output to the terminal or log subsystem illustrating the issue

Activity

poettering commented on Jun 7, 2023

poettering commented on Jun 7, 2023

Toolybird commented on Jun 7, 2023

Toolybird commented on Jun 7, 2023

berrange commented on Jun 8, 2023

poettering commented on Jun 8, 2023

Toolybird commented on Jun 9, 2023

MichaelRiss commented on Jun 9, 2023

MrDisguised commented on Jun 10, 2023

poettering commented on Jun 12, 2023

MichaelRiss commented on Jun 12, 2023

msekletar commented on Jun 13, 2023

msizanoen1 commented on Jun 13, 2023

Toolybird commented on Jun 14, 2023

gdamjan commented on Jun 16, 2023

berrange commented on Jun 19, 2023

iamwo0 commented on Mar 15, 2025

jcpvdm commented on Mar 15, 2025

berrange commented on Mar 17, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions