Skip to content

ci: point the smoke container's apt at the Azure mirror - #114

Merged
nickjalbert merged 1 commit into
laude-institute:mainfrom
MaxFreedomPollard:ci-install-smoke-mirror
Sep 9, 2026
Merged

nickjalbert merged 1 commit into
laude-institute:mainfrom
MaxFreedomPollard:ci-install-smoke-mirror

Conversation

@MaxFreedomPollard

Copy link
Copy Markdown
Contributor

What

install.sh smoke (ubuntu:24.04) is the job that timed out twice on #110 before the merge. It is not flaky in the test sense. Across the last 120 runs of ci.yml (2026-08-29 to 2026-09-05) the job ran 98 times, passed 92, and never once failed: every bad outcome is cancelled. Two of those were cancel-in-progress doing its job when a newer push to main superseded a run. The rest sit in one window on 2026-09-05, 01:04 to 02:41 UTC, plus one at 2026-09-04 10:08 UTC.

Why

This is the one job in the file that runs inside a container, and the bare ubuntu:24.04 image reads apt from the public hosts:

$ grep '^URIs' /etc/apt/sources.list.d/ubuntu.sources      # inside ubuntu:24.04
URIs: http://archive.ubuntu.com/ubuntu/
URIs: http://security.ubuntu.com/ubuntu/

Every other Ubuntu job runs on the runner host, and the host does not read from there. Its image routes apt through a mirror list with Azure's own mirror first and the public hosts as fallbacks (images/ubuntu/scripts/build/configure-apt-sources.sh in actions/runner-images):

http://azure.archive.ubuntu.com/ubuntu/	priority:1
https://archive.ubuntu.com/ubuntu/	priority:2
https://security.ubuntu.com/ubuntu/	priority:3

So when the public host crawls, the host jobs do not notice and this one does. That is exactly what the window looks like. The container's first apt step, normally 10 to 43 s:

run (UTC) Azure region apt step job
2026-09-04 10:08 westus2 762 s cut at 15 min
2026-09-05 01:04 westus3 99 s, then 404 s inside the smoke test passed
01:07 eastus 409 s passed
01:24 westus3 843 s cut at 15 min
01:43, #110 attempt 2 northcentralus 454 s, then 7 min inside the smoke test cut at 15 min
01:55 westus over 896 s, no output cut at 15 min
02:02, #110 attempt 3 westus over 897 s, no output cut at 15 min
02:36, the merge of #110 westus3 304 s passed
02:44 onward all 10 to 22 s passed

On those same runs bash tests (which apt-gets uuid-runtime on the host) and cloc (which apt-gets cloc on the host) ran at their usual speed, checkout from github.com inside the container took 2 s, and the image pull about a second. Only the container-to-archive.ubuntu.com path was slow; five regions, every run in the window. githubstatus.com has no incident for it. The log says nothing because apt-get update -qq prints nothing until it finishes, so the job sits silent until timeout-minutes ends it as cancelled, which in the run list looks the same as a superseded run.

Neither #110 attempt ran anything from the PR. Attempt 3 never reached checkout. Attempt 2 checked out, printed the smoke script's first section header, and then spent seven minutes inside install.sh's first run, which begins with _require_deps fetching jq and curl through apt-get from the same host.

The change

Give the container the host's mirror list before its first apt-get, so it reads from where the host reads and falls back to the public hosts if the Azure mirror is the one having a bad day. install.sh's own apt-get path, which the smoke test runs twice, inherits it. http rather than https for the fallbacks: the bare image has no ca-certificates until the next step installs them, and apt verifies by signature anyway, which is how the image ships. The step greps the result into the log and fails if the two URIs: lines did not both get rewritten, so a future change in the image's layout shows up as a clear failure rather than a silent return to the public host.

apt-get update goes from -qq to -q, so the Get:/Hit: lines name the mirror each index came from, which is the one thing the log lacked while this stalled.

Not in this change

An Acquire::http::Timeout. apt's default is 120 s and none of these runs tripped it, so the transfers were alive and slow rather than dead, and a shorter timeout would not have fired either. The job's own 15-minute limit stays as the backstop for a stall no list can route around.

Tests

Pre-flight on my fork, two jobs on real runners in the same minute (2026-09-07 11:57 UTC). Untreated, main at 5458fee (job, westus): apt step 13 s, smoke test 8 s, 95 passed, 0 failed. Treated, this branch (job, centralus): mirror step under a second, apt step 12 s, smoke test 6 s, 95 passed, 0 failed, and the log now says where the indexes came from:

Get:1 file:/etc/apt/apt-mirrors.txt Mirrorlist [142 B]
Get:2 http://azure.archive.ubuntu.com/ubuntu noble InRelease [256 kB]
Get:3 http://azure.archive.ubuntu.com/ubuntu noble-updates InRelease [126 kB]
Get:4 http://azure.archive.ubuntu.com/ubuntu noble-backports InRelease [126 kB]
Get:5 http://azure.archive.ubuntu.com/ubuntu noble-security InRelease [126 kB]
...
Fetched 32.6 MB in 2s (19.2 MB/s)

All twenty indexes, noble-security included, came from the Azure mirror.

The step's sed and grep were also run locally under dash -e against a copy of the image's ubuntu.sources: both URIs: lines rewrite and the count check passes. dash -n parses both run blocks, since the container runs them under sh -e. Nothing outside the install-smoke job changes.

The install-smoke job runs in a bare ubuntu:24.04 container, whose apt
reads from the public archive.ubuntu.com. Every other job runs on the
runner host, whose image routes apt through azure.archive.ubuntu.com
with the public hosts as fallbacks. On 2026-09-05, 01:04 to 02:41 UTC,
the public host crawled from every Azure region the runners landed in:
the host-side apt-get calls kept their usual speed while this job's
took 99 s to over 15 minutes per round trip, hit timeout-minutes, and
was reported as cancelled, with nothing in the log. Give the container
the host's mirror list before its first apt-get, so install.sh's own
apt-get path inherits it too, and log which mirror served each index.
@nickjalbert

Copy link
Copy Markdown
Contributor

LGTM! Thanks!

@nickjalbert
nickjalbert merged commit 3e02620 into laude-institute:main Sep 9, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants