zfs-backup (setup-gpu-host.sh): replication can wedge permanently when a child dataset is missing on the backup pool, and the failure is silent (exit 0) #3595

Problem

The ZFS backup script that scripts/setup-gpu-host.sh installs (/usr/local/bin/containarium-zfs-backup, run daily by /etc/cron.d/containarium-zfs-backup) can get permanently stuck replicating to the incus-backup pool, and when it does it reports success. On a host that hit this, replication failed on essentially every daily run for several months while the local snapshots kept being taken and pruned, so nothing looked wrong. The backup pool ended up holding a small fraction of one percent of the live data, with whole container datasets missing from it. Meanwhile the 7 retained snapshots per dataset on the live pool accumulated hundreds of GiB and pushed that pool above 90% usage.

All line numbers refer to scripts/setup-gpu-host.sh at e904c99a0 (v0.132.1).

What the code does (verified by reading)

1. The "can we go incremental?" check only looks at the root dataset (line 589): zfs list "${BACKUP_POOL}/${MAIN_POOL}@${PREV_TAG}". A child dataset that exists on the live pool (and so has @PREV there) but was never replicated has no @PREV on the backup pool, yet the check passes.
2. The incremental zfs send -R -i ... | zfs receive -F then fails for that child (line 592). A representative failure from a real run:

   cannot receive incremental stream: destination '<backup>/<main>/containers/containers/<name>' does not exist
   

3. The "repair" fallback does not repair anything (lines 600-605). The comment says "Easier: destroy backup and do full re-send. This is heavy but reliable." but nothing is destroyed or rolled back; it just runs a full zfs send -R | zfs receive -F into a root dataset that already has snapshots, which fails with:

   cannot receive new filesystem stream: destination has snapshots (eg. <backup>/<main>@backup-...)
   

4. Every later run repeats exactly 1-3 (the new base snapshot never lands), so the state never self-heals.
5. A failed replication does not change the exit status (line 630): it prints WARNING: Replication failed — local snapshot kept, but backup pool is out of date and falls through to the EXIT-trap prune. Cron therefore sees exit 0, and the only trace is a line in /var/log/containarium-zfs-backup.log.

Minor: the comment at line 492 says pipefail is intentionally NOT set, but line 494 is set -uo pipefail, which enables it. Harmless here, but the comment is wrong.

I have not reproduced this on a scratch pool, so the exact trigger (a child dataset created after the last successful full send) is inferred from the log and the code, not proven.

Impact

• Silent loss of the off-pool copy for any container created after the last good replication, for as long as nobody reads the log.
• The only restore points are same-pool snapshots, which do not survive loss of the pool.
• The daily snapshots grow the live pool and can push it toward ZFS's full-pool slowdown and failure territory.

Proposal

1. Replicate per dataset (or compare the destination's snapshot set per dataset) instead of one -R stream over the whole tree, so one dataset cannot block the others; send a full stream for datasets that are missing at the destination.
2. Make the fallback do what its comment says, or remove the comment: bring the destination back to a common snapshot (rollback or destroy of the destination dataset) before a full re-send, and only where that is safe.
3. Fail loudly: exit non-zero (after the prune, which should still run) when replication fails, and write a small status file (last success time, failing dataset) that monitoring can read.
4. Warn when the live pool is above a threshold (e.g. 80%) at the end of each run.
5. Fix the pipefail comment.

Acceptance

• A test with two file-backed pools covers: first full replication; a child dataset created afterwards; the next run replicates it (and exits 0); a forced receive failure makes the script exit non-zero.
• After a failed run followed by a fixed condition, the next run recovers without manual zfs destroy.
• --prune behaviour and the 7-snapshot default are unchanged.