Skip to content

About

systemd-analyze blame for shutdown: which units slowed down the last shutdown or reboot, read from the journal

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

1 Commit

Folders and files

Repository files navigation

shutdown-blame

systemd-analyze blame for shutdown. After a slow shutdown or reboot, shutdown-blame reads the journal of the previous boot and lists every unit that systemd stopped during the shutdown, sorted by how long the stop took. Units that ran into their stop timeout and the processes systemd then killed are named.

Output for the end-to-end test container from tests/e2e, stopped with docker stop:

$ shutdown-blame -D journal/ -b all
Shutdown of boot bcc6c0a68390 began 2026-11-01 05:06:51 UTC (Received SIGRTMIN+3.).
The journal stopped 4.353s later; anything after that (final unmounts, killing leftover processes, firmware) is not recorded.

     TOOK    STARTED  UNIT                           NOTES
   4.216s    +0.014s  ignores-term.service           timed out in stop-sigterm; SIGKILL: sh, sleep
   3.009s    +0.021s  slow-stop.service
   2.220s    +0.023s  [email protected]
   2.207s    +0.024s  user 1500: user-slow.service   timed out in stop-sigterm; SIGKILL: sh, sleep
   0.024s    +0.022s  systemd-logind.service
   0.024s    +2.311s  [email protected]
   ...

TOOK is the time from systemd's "Stopping ..." message to "Stopped ..." for that stop job. STARTED is when the stop job began, relative to the start of the shutdown. Stop jobs run in parallel, so the longest one is the first place to look; a unit that started late (large STARTED) waited for other units ordered after it. Units of user service managers (user 1000: ...) are included, which covers the common "A stop job is running for User Manager for UID 1000" case.

Why

systemd can show what slowed down the boot (systemd-analyze blame, critical-chain), but there is no equivalent for shutdown. The usual advice is to boot with a debug shell or a shutdown-log hook and read the raw log by hand.

  • systemd/systemd#3371 "Provide systemd-analyze blame equivalent for system shutdown": open since May 2016, 58 reactions. The only workaround in the thread (2024) is a shell loop over journalctl calls.
  • "A stop job is running for ..." questions: unix.SE 273876 (score 83, about 158k views), unix.SE 310737 (58, about 148k views), Ask Ubuntu 760952 (71, about 93k views), unix.SE 539215 "Analyze 'a stop job is running' for 1 minute 30 seconds during shutdown".
  • The one related project I found, systemd-shutdown-diagnose (Arch package, last change 2020), records process exits with ftrace and has to be installed before the shutdown you want to look at.

shutdown-blame needs no preparation: systemd already logs every stop job with a job ID and a timestamp, so the data for the last shutdown is in the journal, as long as the journal is persistent.

Install

Python 3.8 or newer and journalctl; no other dependencies.

pipx install git+https://github.com/Jwrede/shutdown-blame

or copy the single file:

curl -fsSLo ~/.local/bin/shutdown-blame https://raw.githubusercontent.com/Jwrede/shutdown-blame/main/shutdown_blame.py
chmod +x ~/.local/bin/shutdown-blame

Reading the system journal needs root or membership in the systemd-journal or adm group.

Usage

shutdown-blame                  # the shutdown that ended the previous boot
shutdown-blame -b -2            # the one before that (any journalctl -b value)
shutdown-blame --min 1          # hide units that stopped in under a second
shutdown-blame --all            # every stop job, not only the first 20
shutdown-blame --json           # machine-readable report
shutdown-blame -D /mnt/var/log/journal -b all   # journal files copied from another machine
shutdown-blame --since 2026-10-30T17:50:01      # set the start of the shutdown by hand

How the start of the shutdown is found, in this order, using the last match in the boot:

  1. systemd-logind's "System is powering down." / "System is rebooting." entry (message ID 98268866d1d54a499c4e98921d93bc40), written for systemctl poweroff, reboot, the power button and desktop shutdown menus;
  2. PID 1's "Received SIGRTMIN+3." (or +4, +5, +6, SIGINT) and "Activating special unit ...", written when PID 1 is told to shut down by signal, for example by docker stop or machinectl poweroff;
  3. otherwise the first stop job after PID 1 last started an ordinary unit, if a shutdown target follows. The report says when the start was inferred this way.

Exit status: 0 when a shutdown was found, 1 when the boot has no shutdown (power loss, crash, volatile journal), 2 when journalctl failed.

Limitations

  • The journal must be persistent: /var/log/journal must exist (or Storage=persistent be set in journald.conf). Where it is kept only in memory, sudo mkdir -p /var/log/journal and a reboot make the next shutdown visible.
  • journald itself stops near the end of the shutdown. What happens after "Journal stopped" (the final unmount loop, systemd-shutdown killing leftover processes, storage and firmware) is not in the journal, so a delay there shows up only as the gap between the last unit and the real power-off. For that phase the systemd debugging guide describes a /usr/lib/systemd/system-shutdown/ hook.
  • A stop job that was still running when the journal stopped is shown as >N s with the note "still stopping when the journal ended".
  • Timestamps are wall-clock times from the journal. A clock change during the shutdown would distort durations.
  • Tested with systemd 252 (Debian 12), 255 (Ubuntu 24.04) and 257 (Fedora 42), in containers that are shut down with docker stop and with systemctl poweroff. Not tested on a physical machine's reboot by the CI; the journal entries it reads are the same.

Tests

python3 -m unittest discover -s tests            # unit tests, including a journal captured from a real shutdown
docker build -t sb-e2e:ubuntu tests/e2e && tests/e2e/run.sh sb-e2e:ubuntu

The end-to-end test boots systemd in a privileged container with three test services (one stops at once, one has a 3-second ExecStop, one ignores SIGTERM and has TimeoutStopSec=4) and a lingering user whose user service ignores SIGTERM. It shuts the container down twice, once with docker stop and once with systemctl poweroff, and checks the report against the expected durations, timeouts and killed processes. CI runs it for Ubuntu 24.04, Debian 12 and Fedora 42.

License

MIT

About

systemd-analyze blame for shutdown: which units slowed down the last shutdown or reboot, read from the journal

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages