fix(upgrade): stop swapoff OOM from aborting run_upgrade - #3172
Merged
Conversation
`swapoff --all` faults all used swap back into RAM at once. On a memory-tight board it runs at peak memory — right after the apt dist-upgrade and with the previous Docker stack still resident — and the OOM-killer reaps it (rc -9), aborting the whole upgrade (issue 3165). - install.sh: stop the container stack before the Ansible play so swapoff has the headroom to fault swap back in; no-op on fresh install - misc.yml: only attempt swapoff when RAM can absorb the used swap; retry and never hard-fail; remove the swapfile only once swap is off Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|
There was a problem hiding this comment.
Pull request overview
This PR prevents Anthias upgrades from aborting when swapoff --all gets SIGKILL’d by the OOM killer on low-memory devices (Issue #3165), by freeing memory earlier and making swap disabling best-effort under memory pressure.
Changes:
- Add a pre-Ansible
stop_docker_stack()step inbin/install.shto bring down the existing Docker Compose stack before the dist-upgrade/swapoff phase. - Harden the Ansible “Disable swap” step with a MemAvailable/SwapUsed guard, retries, and non-fatal behavior; gate swapfile removal on successful
swapoff.
Reviewed changes
Copilot reviewed 1 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
bin/install.sh |
Stops the running Docker Compose stack before the Ansible play to reclaim RAM and reduce OOM risk during swapoff/dist-upgrade. |
ansible/roles/system/tasks/misc.yml |
Adds a memory-based guard and retry/tolerance logic around swapoff, and prevents removing /var/swap unless swapoff actually succeeded. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #3165
Problem
run_upgrade.shaborts at the Ansible tasksystem : Disable swap(
ansible/roles/system/tasks/misc.yml) with:rc: -9is a SIGKILL from the OOM-killer, not an error return.swapoff --allfaults every used swap page back into RAM at once. Itruns at the peak-memory moment of the whole upgrade: right after the apt
dist-upgrade and while the previous-version Docker stack is still
resident (the stack isn't torn down/rebuilt until
upgrade_docker_containers,which runs after the entire Ansible play). On a memory-tight board
(here a 1 GB Pi 4 on Trixie, whose dynamically-sized swap is larger than
the old fixed 100 MB) there isn't enough headroom, so the kernel kills
swapoffand the upgrade aborts. The task itself is long-standing; theTrixie upgrade path is what now reliably tips these boards over.
Fix
bin/install.sh— newstop_docker_stack(), called beforerun_ansible_playbook, brings the running stack down first soswapoff(and the dist-upgrade) have the RAM headroom they need. No-op on a fresh
install; best-effort so a stale compose file can't abort the upgrade.
upgrade_docker_containersbrings everything back up afterwards, and theinstall ends in a reboot regardless.
ansible/roles/system/tasks/misc.yml— hardenedDisable swap:swapoffwhenSwapUsed < MemAvailable - 64 MiBretries: 3+until rc == 0+failed_when: false: transientpressure retries; a persistent OOM leaves swap on rather than aborting
rc == 0— never unlink an active swapfileDisabling swap is best-effort host hygiene, so skipping/leaving it on
under memory pressure is safe; it must never gate the whole upgrade.
Validation (real 1 GB Pi 4,
arm64, zram swap)rc -9SIGKILL reproduceddocker compose stopswapoffruns and succeeds (rc 0), removal gated on successshellcheck,ansible-lint(production profile) andansible-playbook --syntax-checkall pass.🤖 Generated with Claude Code