Proxy is not propagated to Ansible, Docker daemon, anthias-host-agent, or containers during installation behind an HTTP proxy #3239

Description

On a host that only has internet access through an HTTP proxy (no direct outbound access), installing Anthias requires manually fixing proxy propagation at nearly every layer of the install, one at a time, because none of them share environment/configuration with each other.

Environment

• Device: PC (x86)
• OS: Debian/Ubuntu-based
• Anthias Version: latest; Release 2026.07.4
• Installation Method: Manual Installation — started via install.sh (Release/Git checkout), switched to invoking ansible-playbook directly after install.sh's repo-reset behavior made iterating on the proxy fix impossible while using the script

1. /etc/environment proxy variables not seen by the install

Proxy vars were exported to /etc/environment. This file is only read by PAM at login time, so it populated interactive login shells but was not inherited by:

• non-login shells,
• sudo-spawned processes (unless PAM re-reads it in the sudo stack, which is not default),
• Ansible's local connection plugin (forks Python directly).

Workaround: export proxy vars directly in the shell that invokes the installer, and use sudo -E to preserve them across privilege escalation.

2. ansible.builtin.get_url fails with "network is unreachable" on the Docker apt key task

Even after exporting proxy vars, the Add Docker apt key task (ansible/roles/system/tasks/docker.yml) failed with a network-unreachable error while fetching https://download.docker.com/linux/debian/gpg.

Root cause found by reading get_url's source: it is not a network-skip when dest already exists and force: false (the default) — it still performs a conditional (If-Modified-Since) request to the remote URL to check for changes. This means the task always requires network/proxy access, even if the destination file is already present and correct. This does not match the documented "will only be downloaded if the destination does not exist" behavior and matches previously reported confusing/inconsistent behavior in the get_url module itself (e.g. ansible/ansible#64016 ), a regression report describing repeated downloads despite force: no).

Manually placing the key file via curl -x <proxy> ... beforehand did not prevent Ansible from re-attempting (and failing) the network call.

3. Local edits to ansible/site.yml were silently discarded on every run of install.sh

While still trying to use install.sh as intended, I added an environment: block to ansible/site.yml to carry proxy vars into the play, and re-ran install.sh. The edit was silently gone on the next run. Root cause: install.sh's clone_repo() function runs git -C "${ANTHIAS_REPO_DIR}" reset --hard "${RESET_REF}" on every invocation, discarding any local modifications to tracked files — with no warning that this is about to happen.

Workaround: stopped using install.sh entirely. Cloned the repo once, then ran ansible-playbook directly from that persistent clone (sudo -E ansible-playbook site.ym), so local edits survive.

4. ansible-playbook/Ansible not present outside the installer's disposable venv

install.sh creates a temporary venv (via uv) for Ansible and deletes it on exit; there is no standalone ansible-playbook on the system otherwise. Installed ansible-core manually via apt to run the playbook directly.

Also had to invoke Ansible via its full resolved path (sudo -E $(which ansible-playbook) ...) because sudo's secure_path strips a user-local $PATH even with -E.

(Note: after switching to a direct ansible-playbook invocation, site.yml's pre_tasks assertion on USER/ANTHIAS_BRANCH/DEVICE_TYPE/etc. also required manually exporting all of those vars myself each session, since install.sh normally sets them interactively/automatically — expected once bypassing the script, not really a bug on its own, though clearer documentation of what site.yml expects would help. Similarly, needing docker group membership plus a reboot for it to take effect is expected/known behavior, not a bug — install.sh does prompt for a reboot at the end for this reason.)

5. sudo -E does not preserve USER

Even with -E, sudo resets USER/LOGNAME/HOME to the target (root) user by default on Debian-based sudoers configs. Because site.yml uses {{ lookup('env', 'USER') }} for anthias_user (which is later used to render paths and systemd units), this caused the Anthias host-agent systemd unit to be rendered with ExecStart=/home/root/installer_venv/bin/python -m anthias_host_agent — a path that does not exist — resulting in status=203/EXEC.

Workaround: explicitly force USER back after sudo -E, e.g.:

sudo -E env USER="$(whoami)" $(which ansible-playbook) site.yml --ask-become-pass -e "anthias_user=$(whoami)"

6. x86 hardware executed Raspberry Pi–only tasks

This only surfaced because of bypassing install.sh (see 3) — the script normally passes --skip-tags raspberry-pi automatically for non-Pi DEVICE_TYPEs. Without it, the Render config.txt task in roles/system/tasks/boot.yml failed with Destination directory /boot/firmware does not exist, since that directory only exists on Raspberry Pi OS.

Worth flagging as a robustness point regardless of the script: the task itself has no when: guard against non-Pi hardware/ansible_architecture, so it relies entirely on the wrapper script remembering to skip it — anyone invoking the playbook directly (or a future wrapper that forgets the flag) hits this.

7. Docker daemon does not inherit shell/user proxy env

After the playbook succeeded, ./bin/upgrade_containers.sh failed to pull images from ghcr.io with i/o timeout, because dockerd runs as its own systemd service with its own isolated environment — proxy vars exported in a user shell are never seen by the daemon.

Workaround: added a systemd drop-in:

/etc/systemd/system/docker.service.d/http-proxy.conf
[Service]
Environment="HTTP_PROXY=http://<proxy>"
Environment="HTTPS_PROXY=http://<proxy>"
Environment="NO_PROXY=localhost,127.0.0.1"

followed by daemon-reload + restart docker.

8. anthias-host-agent systemd service crash loop, then a ConnectTimeout to 1.1.1.1:443

After fixing 5, the service started, but its Python (requests) HTTP calls (apparently an internet-connectivity check against 1.1.1.1:443, i.e. Cloudflare) timed out, because — same root cause as 7 — a systemd service does not inherit shell-exported proxy vars.

Workaround: added another systemd drop-in for this unit:

/etc/systemd/system/anthias-host-agent.service.d/http-proxy.conf
[Service]
Environment="HTTP_PROXY=http://<proxy>"
Environment="HTTPS_PROXY=http://<proxy>"
Environment="NO_PROXY=localhost,127.0.0.1"

Separately from the proxy issue: is hardcoding an outbound connectivity check against a specific third-party IP (Cloudflare's 1.1.1.1) the intended approach here? In restricted/corporate/proxy-only environments this adds an external dependency (and, depending on policy, a potential allowlisting/privacy concern) that isn't obviously necessary just to determine local network status. Worth reconsidering, or at least making configurable.

9. Containers need their own proxy, but a naive NO_PROXY breaks internal container-to-container calls, and this raises a data-privacy question

With the above workarounds in place, the Ansible playbook completes successfully (ok=52 changed=22 failed=0), the web UI is reachable, and anthias-host-agent runs without crashing. However:

1. Before adding any proxy configuration to the Docker containers themselves, offline/local content (e.g. static screenshots/images) displayed correctly. Only web-asset content requiring an external URL fetch failed — the anthias-viewer container logged repeated Asset ... at URL ... is not available, skipping. for that kind of asset, presumably for the same "no proxy in this process's environment" reason as 7/8.
2. After adding HTTP_PROXY/HTTPS_PROXY/NO_PROXY=localhost,127.0.0.1 to the anthias-server and anthias-viewer services (via a docker-compose.override.yml) and rebooting, the situation got worse: the viewer stopped rendering anything at all, including a basic screenshot that had worked before.

The suspected cause is that anthias-viewer and anthias-server communicate with each other over Docker's internal bridge network, and since NO_PROXY only excluded localhost,127.0.0.1, that internal container-to-container traffic started being routed through the external proxy as well — which cannot reach Docker's internal network, breaking internal API calls between the two containers.

Expected Behavior

A single proxy configuration step during install (ideally: install.sh prompting for HTTP_PROXY/HTTPS_PROXY/NO_PROXY the same way it already prompts for DEVICE_TYPE) should be sufficient for the entire stack — Ansible tasks, the Docker daemon, the host-agent service, and the server/viewer containers — to install and run correctly behind a proxy, without needing to discover and patch each layer's isolated environment individually. Container-internal communication should keep working regardless of proxy configuration for external-asset fetches.

Suggested fixes for the project

• Have install.sh explicitly prompt for proxy settings (HTTP_PROXY/HTTPS_PROXY/NO_PROXY), the same way it already prompts for DEVICE_TYPE/MANAGE_NETWORK/etc., and then propagate that single input to every layer that currently needs it separately and silently fails otherwise:

  • inject it into the Ansible play's environment: (for get_url/apt tasks),
  • write the Docker daemon systemd drop-in automatically,
  • write the anthias-host-agent systemd drop-in automatically,
  • inject it into the rendered docker-compose.yml for the server/viewer services, with NO_PROXY pre-populated to include the Compose network's own bridge subnet/service hostnames by default.

  This would turn most of the issues above into one prompt, instead of requiring users to discover and fix each layer manually.

• Document proxy usage explicitly for self-hosted/offline/corporate-network installs (currently undocumented), including the data-privacy implication noted in 9.

• install.sh's git reset --hard should probably warn if there are uncommitted local changes before discarding them, rather than doing so silently.

• Consider explicitly setting force: false behavior to truly skip network access when dest already exists in get_url tasks, or add a stat+when guard so environments with pre-provisioned keys/artifacts don't require any network call at all.

• The Ansible role for the host-agent systemd unit should not rely solely on lookup('env', 'USER'), since this is fragile under sudo/become (resolves to root in several common invocation patterns) — consider using ansible_user_id/an explicit anthias_user var consistently, with clear precedence documented.

• Reconsider (or make configurable) the hardcoded connectivity check against 1.1.1.1 in anthias-host-agent (see 8).