Repository navigation
ci: point the smoke container's apt at the Azure mirror - #114
Merged
nickjalbert merged 1 commit intoSep 9, 2026
Merged
Conversation
The install-smoke job runs in a bare ubuntu:24.04 container, whose apt reads from the public archive.ubuntu.com. Every other job runs on the runner host, whose image routes apt through azure.archive.ubuntu.com with the public hosts as fallbacks. On 2026-09-05, 01:04 to 02:41 UTC, the public host crawled from every Azure region the runners landed in: the host-side apt-get calls kept their usual speed while this job's took 99 s to over 15 minutes per round trip, hit timeout-minutes, and was reported as cancelled, with nothing in the log. Give the container the host's mirror list before its first apt-get, so install.sh's own apt-get path inherits it too, and log which mirror served each index.
Contributor
|
LGTM! Thanks! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
install.sh smoke (ubuntu:24.04)is the job that timed out twice on #110 before the merge. It is not flaky in the test sense. Across the last 120 runs ofci.yml(2026-08-29 to 2026-09-05) the job ran 98 times, passed 92, and never once failed: every bad outcome iscancelled. Two of those werecancel-in-progressdoing its job when a newer push tomainsuperseded a run. The rest sit in one window on 2026-09-05, 01:04 to 02:41 UTC, plus one at 2026-09-04 10:08 UTC.Why
This is the one job in the file that runs inside a container, and the bare
ubuntu:24.04image reads apt from the public hosts:Every other Ubuntu job runs on the runner host, and the host does not read from there. Its image routes apt through a mirror list with Azure's own mirror first and the public hosts as fallbacks (
images/ubuntu/scripts/build/configure-apt-sources.shin actions/runner-images):So when the public host crawls, the host jobs do not notice and this one does. That is exactly what the window looks like. The container's first apt step, normally 10 to 43 s:
On those same runs
bash tests(which apt-getsuuid-runtimeon the host) andcloc(which apt-getsclocon the host) ran at their usual speed, checkout from github.com inside the container took 2 s, and the image pull about a second. Only the container-to-archive.ubuntu.compath was slow; five regions, every run in the window. githubstatus.com has no incident for it. The log says nothing becauseapt-get update -qqprints nothing until it finishes, so the job sits silent untiltimeout-minutesends it ascancelled, which in the run list looks the same as a superseded run.Neither #110 attempt ran anything from the PR. Attempt 3 never reached checkout. Attempt 2 checked out, printed the smoke script's first section header, and then spent seven minutes inside install.sh's first run, which begins with
_require_depsfetching jq and curl throughapt-getfrom the same host.The change
Give the container the host's mirror list before its first
apt-get, so it reads from where the host reads and falls back to the public hosts if the Azure mirror is the one having a bad day. install.sh's own apt-get path, which the smoke test runs twice, inherits it.httprather thanhttpsfor the fallbacks: the bare image has noca-certificatesuntil the next step installs them, and apt verifies by signature anyway, which is how the image ships. The step greps the result into the log and fails if the twoURIs:lines did not both get rewritten, so a future change in the image's layout shows up as a clear failure rather than a silent return to the public host.apt-get updategoes from-qqto-q, so theGet:/Hit:lines name the mirror each index came from, which is the one thing the log lacked while this stalled.Not in this change
An
Acquire::http::Timeout. apt's default is 120 s and none of these runs tripped it, so the transfers were alive and slow rather than dead, and a shorter timeout would not have fired either. The job's own 15-minute limit stays as the backstop for a stall no list can route around.Tests
Pre-flight on my fork, two jobs on real runners in the same minute (2026-09-07 11:57 UTC). Untreated,
mainat 5458fee (job, westus): apt step 13 s, smoke test 8 s, 95 passed, 0 failed. Treated, this branch (job, centralus): mirror step under a second, apt step 12 s, smoke test 6 s, 95 passed, 0 failed, and the log now says where the indexes came from:All twenty indexes,
noble-securityincluded, came from the Azure mirror.The step's
sedandgrepwere also run locally underdash -eagainst a copy of the image'subuntu.sources: bothURIs:lines rewrite and the count check passes.dash -nparses both run blocks, since the container runs them undersh -e. Nothing outside theinstall-smokejob changes.