sentinel auto-update cannot write its download under the generated unit (ProtectSystem=strict, binary dir not in ReadWritePaths) #3596

Problem

The systemd unit that containarium service writes hardens the daemon with ProtectSystem=strict, and its ReadWritePaths does not include the directory the daemon binary lives in. The sentinel auto-updater, however, downloads the new binary to /usr/local/bin/containariumd.new (a path hardcoded in internal/server/dual_server.go:3857) and renames it into place. On a unit generated by this template the download can therefore never be written, so sentinel auto-update cannot work, while docs/UPGRADE-AND-ROLLBACK.md says it does (line 22: the daemon "checks the sentinel every 5 minutes … and installs a changed binary"; line 106: a daemon with a sentinel URL "installs whatever the sentinel serves within about 5 minutes").

All references are to origin/main at e904c99a0 (v0.132.1).

What the code does (verified by reading)

• internal/cmd/service.go (the unit template): ExecStart=/usr/local/bin/containariumd daemon … (line 96), ProtectSystem=strict (line 152), and
  ReadWritePaths=-/var/lib/containarium /var/lib/incus /etc/containarium /etc /home /var/lock /run/lock -/opt/containarium /var/log -/mnt/incus-data (line 154). /usr/local/bin is not listed, and nothing in the template adds the binary's directory.
• internal/server/autoupdate.go: tmpPath := u.binaryPath + ".new" (line 179) followed by os.Rename(binaryPath → .old) and os.Rename(.new → binaryPath) (lines 216-224). Under ProtectSystem=strict the first write to /usr/local/bin/containariumd.new fails with EROFS.
• History: the template has carried ProtectSystem=strict since the first service-install commit (2026-02-23); the updater has written .new since 2026-04-05. scripts/containarium.service has the same ProtectSystem=strict and a ReadWritePaths that also omits /usr/local/bin. I did not check every installer or deploy script, so hosts whose unit was written some other way may not have the restriction.

Observed

On a host whose unit matches this template and which has a sentinel URL configured, the daemon logged, on the first poll after the sentinel started serving a different binary:

[auto-update] new version detected (local=43345f53287b..., remote=fd28622c8227...)
[auto-update] check failed: download: open /usr/local/bin/containariumd.new: read-only file system

The host kept running its old binary and retried on the 5-minute tick.

Why it stayed hidden: the updater only writes when the sentinel serves a different binary, so a host whose binary already matches never touches the filesystem. The first real update is the first time the failure can appear, and it appears only as a log line at the same level as an ordinary failed poll.

Impact

• Operators who follow the documented model ("uploading a binary to the sentinel is a fleet-wide rollout") will find hosts that silently never update, and nothing alerts.
• The accidental safety (a host does not upgrade itself) is not a designed property, and anyone who "fixes" the failure by loosening ProtectSystem globally weakens the hardening for the whole daemon.

Proposal (pick one deliberately)

1. Make the updater work under the hardened unit: add the directory of the running binary to ReadWritePaths when generating the unit (the smallest grant compatible with a rename-based swap; it makes every binary in that directory writable by the daemon, so the staging-plus-helper alternative below is the narrower one), and keep ProtectSystem=strict; or stage the download under /var/lib/containarium and swap with a small privileged helper, so the daemon never needs write access to its own binary directory.
2. Or declare auto-update unsupported on the hardened unit: refuse --sentinel-url auto-update at startup with a clear message, and correct the docs.
3. Either way, make the failure loud: log it at warning level with the cause, count consecutive failures, expose a metric, and add a containarium doctor check that flags "auto-update enabled but the binary directory is not writable" (the doctor already probes writability for useradd, but, as its own comment in internal/cmd/doctor.go says, it only gives a true answer when run under the unit's constraints; a check for this should read systemctl show -p ProtectSystem,ReadWritePaths or run inside the daemon, because from a plain root shell it would pass falsely).

Acceptance

\1- [ ] The same holds for the GitHub-tag upgrade path (internal/server/github_upgrade.go:113, which also writes <binary>.new).\n- [ ] On a host with the generated unit and a sentinel serving a changed binary, the daemon installs it (or refuses to start auto-update with a clear error, if option 2 is chosen).

• docs/UPGRADE-AND-ROLLBACK.md matches whatever is chosen.