fix(ansible): install systemd-resolved on x86 to fix cold-boot DNS race - #3231
Merged
vpetersson merged 2 commits intoAug 5, 2026
Merged
Conversation
URL/web assets could fail to load on x86 until a manual `docker restart` (GH issue 3230). On a headless x86 box the DHCP client writes upstream nameservers directly into /etc/resolv.conf, and that file can still be empty or hold a not-yet-reachable server at the moment dockerd first creates the Anthias bridge network. Docker bakes the host resolver config into its embedded DNS (127.0.0.11) at network creation and never re-derives it, so containers stay stuck with a dead upstream for the life of that network. The asset revalidation loop then marks every URL asset unreachable and the viewer skips them; a `docker restart` recreates the network after DNS is up, which is why the manual workaround appears to fix it. Installing and enabling systemd-resolved removes the race: it makes /etc/resolv.conf a static symlink to the 127.0.0.53 stub that exists from install time and never goes empty, while resolved tracks the live upstream behind that fixed address. Docker forwards to the real upstreams from the resolved run directory. The ordering is already correct in Docker's packaged unit (After network-online.target and nss-lookup.target) paired with resolved's Before nss-lookup.target, so enabling resolved makes dockerd genuinely wait for name resolution. Scoped to x86, where the issue is reported; the Pi images have a different, working network/boot stack. Validated on the x86 testbed: after a cold reboot with resolved enabled, host and in-container name resolution and a real HTTPS asset fetch all succeed. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01HJ3ucEkn62cbgPoisAZ5LQ
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #3231 +/- ##
=========================================
Coverage ? 90.82%
=========================================
Files ? 76
Lines ? 8361
Branches ? 885
=========================================
Hits ? 7594
Misses ? 549
Partials ? 218 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
vpetersson
previously approved these changes
Aug 3, 2026
There was a problem hiding this comment.
Pull request overview
This PR addresses an x86-only cold-boot DNS race where Docker can snapshot an empty/stale host resolver configuration, leaving Anthias containers unable to resolve external URLs until Docker/containers are restarted. The fix ensures a stable local resolver is present before Docker is installed/started on x86 hosts.
Changes:
- Add an x86-gated system task to configure DNS stability prior to Docker setup.
- Install/enable
systemd-resolvedon x86 and repoint/etc/resolv.confto the resolved stub.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
ansible/roles/system/tasks/main.yml |
Adds an x86-only include to run DNS setup before Docker tasks. |
ansible/roles/system/tasks/dns.yml |
New task file that installs/enables systemd-resolved and symlinks /etc/resolv.conf to the stub resolver. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
The comment claimed the stub resolv.conf 'exists from install time and never goes empty', but the symlink target (/run/systemd/resolve/stub-resolv.conf) is generated by resolved at boot and is briefly absent until the service starts — as the task's own comment notes. Reword to state the actual invariant: the nameserver is a fixed loopback address (127.0.0.53) and unit ordering (resolved Before=nss-lookup.target, dockerd After=nss-lookup.target) guarantees the stub is answering before dockerd creates the network. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01HJ3ucEkn62cbgPoisAZ5LQ
|
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.
Suppressed comments (2)
ansible/roles/system/tasks/dns.yml:59
- The comment says "expected and harmless. force overwrites…"; the sentence starts mid-word and reads like a typo. Capitalizing it improves readability without changing meaning.
# The stub file is created by resolved at runtime, so the link is
# briefly dangling on first boot before the service starts; that is
# expected and harmless. force overwrites a static resolv.conf left
# behind by the DHCP client / installer.
ansible/roles/system/tasks/dns.yml:55
- Linking /etc/resolv.conf to a file under /run means it will be a dangling symlink early in boot (before systemd-resolved has started and created stub-resolv.conf). That can cause DNS lookups to fail for any unit that starts before resolved. Consider linking to the static stub file shipped by systemd-resolved (commonly /usr/lib/systemd/resolv.conf), or otherwise ensuring resolved is started before any DNS-consuming services on boot.
- name: Point /etc/resolv.conf at the systemd-resolved stub
ansible.builtin.file:
src: /run/systemd/resolve/stub-resolv.conf
dest: /etc/resolv.conf
state: link
vpetersson
approved these changes
Aug 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #3230
Problem
On x86, URL / web-page assets can fail to load after a cold boot and only recover after a manual
docker restart.Root cause
dockerdreads the host resolver config when it creates a network and bakes those nameservers into the container-side embedded DNS (127.0.0.11); it never re-derives them, so if it captures an empty/stale/etc/resolv.confthe containers stay stuck with a dead upstream for the life of that network. Adocker restartrecreates the network after DNS is up, which is why that workaround appears to fix it.Docker is ordered
After=network-online.target, so in principle it should not start until the network (and DNS) is ready. The problem is that on the stock x86 image nothing actually holdsnetwork-online.targetback until DNS is written:allow-hotplug enp2s0(brought up by dhcpcd).NetworkManager-wait-online.servicereturns immediately, because NM does not manage that interface (ifupdown owns it — NM reports itunmanaged).networking.serviceonly blocks onautointerfaces;allow-hotplugones come up asynchronously via udev, so the service completes without waiting for DHCP.So
network-online.targetis declared reached while DHCP/DNS is still pending,dockerdstarts against an empty/etc/resolv.conf, and the race fires.(For contrast, the Raspberry Pi images use NetworkManager to manage the link, so
NetworkManager-wait-onlinegenuinely blocksnetwork-online.targetuntil NM has written the nameserver — Docker then reads a populatedresolv.conf. That is why this is x86-specific.)Fix
Install and enable
systemd-resolvedon x86. This removes the dependence on boot ordering entirely:/etc/resolv.confbecomes a static symlink to the127.0.0.53stub, which exists from install time and never goes empty — so Docker can never capture an empty resolver list.systemd-resolvedtracks the live upstream behind that fixed address, fed by whatever manages the link. This does not require NetworkManager: resolved ships aresolvconfcompatibility shim (/sbin/resolvconf→resolvectl), so dhcpcds20-resolv.confhook (and dhclient, and systemd-networkd natively) push the leased DNS servers into resolved automatically.resolv.conf, forwards to the real upstreams from/run/systemd/resolve/resolv.conf.Scoped to x86, where the issue is reported and where the boot ordering demonstrably fails to gate Docker on DNS.
Validation
Reproduced and validated on the x86 testbed (Debian 13, ifupdown + dhcpcd, NetworkManager present but not managing the NIC — i.e. the no-NM DNS path). After a cold reboot with
systemd-resolvedenabled:/etc/resolv.conf→127.0.0.53stub; resolved active with the correct upstream, populated via dhcpcd → theresolvconf/resolvectlshim (no NetworkManager involved);getent hosts github.com);127.0.0.11embedded DNS), and a real HTTPS asset fetch returns200.🤖 Generated with Claude Code
https://claude.ai/code/session_01HJ3ucEkn62cbgPoisAZ5LQ