Skip to content

Commit 7e96e60

Browse files
ci: bound the ffmpeg apt fetch so a stalled mirror costs a retry, not the job (#3356)
* ci: bound the ffmpeg apt fetch so a stalled mirror costs a retry, not the job Hosted runners intermittently stall on an apt mirror, and an unbounded apt-get inherits the whole job budget. The producer integration lane normally finishes in ~11 minutes against a 20 minute cap; on a stalled fetch it ran to the cap and failed. Same step, same shape, reproduces on main's tip — it is not specific to any one PR. The cost is not one red check. On the run that prompted this, four went red off that single step: the two jobs that install ffmpeg, plus a Test gate and a preview-regression gate that both fail closed when their dependency does not succeed. So a mirror stall reads as a producer defect and a preview defect. Each attempt is now bounded and retried three times, and the five workflows that installed ffmpeg share one action instead of five copies of the command. Deliberately still apt: caching the binary would strip it from the shared libraries it links against, and switching to a static build would change the ffmpeg under the producer's output comparisons. Neither belongs in a fix for a network stall. * ci: drop the stray version echo left in the player-perf ffmpeg step Converting the step to the shared action left the trailing `ffmpeg -version` line behind, and YAML folded it into the `uses:` value — so the runner looked for an action at a path with the command appended and failed all four perf shards. It parsed cleanly, which is why validating with a YAML load did not catch it: `uses: ./path\n ffmpeg -version` is a legal folded scalar. The check that does catch it asserts every local `uses:` resolves to a directory containing an action file, which is now what I ran. The action prints the version itself. * ci: bound the ffmpeg fetch at the connection, not with a wall-clock kill The first version wrapped apt in `timeout` and retried. A passing run showed why that is the wrong shape: the mirror is slow rather than hung — the install spent ~15 minutes pulling packages from azure.archive.ubuntu.com and finished successfully. Killing it at 300s discarded a download that was making progress and started over, so the retry turned a slow mirror into a slower one, and the worst case of three attempts exceeded the job's own 20 minute cap. Bound the connection instead. Acquire::Retries re-fetches the one package whose connection stalled while keeping everything already downloaded, and Acquire::http::Timeout caps how long any single connection may sit idle. That addresses the stall the original report described without punishing the slow case that is far more common.
1 parent d464f60 commit 7e96e60

5 files changed

Lines changed: 39 additions & 12 deletions

File tree

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
name: Install ffmpeg (Linux)
2+
description: >-
3+
Install ffmpeg from apt, resilient to a slow or stalling mirror.
4+
5+
Hosted runners hit `azure.archive.ubuntu.com` at wildly varying speeds: the
6+
same install has taken 40 seconds and 15 minutes on consecutive runs, and a
7+
stalled connection used to hang until the job's own timeout killed it. That
8+
cost more than one red check, because the gates downstream fail closed when a
9+
dependency does not succeed — a single slow fetch showed up as a producer
10+
defect and a preview defect at once.
11+
12+
The bound belongs on the connection, not on the command. `Acquire::Retries`
13+
re-fetches the individual package that stalled while keeping everything
14+
already downloaded, and `Acquire::http::Timeout` caps how long any one
15+
connection may sit idle. Wrapping the whole command in a wall-clock kill
16+
looks similar but behaves worse: it discards a download that was making
17+
progress and starts it over, which turns a slow mirror into a slower one.
18+
19+
runs:
20+
using: composite
21+
steps:
22+
- name: Install ffmpeg
23+
shell: bash
24+
env:
25+
# Idle-connection cap, and per-package retries that keep prior progress.
26+
APT_RESILIENCE: >-
27+
-o Acquire::Retries=3
28+
-o Acquire::http::Timeout=30
29+
-o Acquire::https::Timeout=30
30+
run: |
31+
set -euo pipefail
32+
sudo apt-get $APT_RESILIENCE update -qq
33+
sudo apt-get $APT_RESILIENCE install -y --no-install-recommends ffmpeg
34+
ffmpeg -version | head -1

‎.github/workflows/catalog-previews.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -66,7 +66,7 @@ jobs:
6666
# file copy), so the job never needed it and ubuntu-latest does not ship
6767
# it.
6868
- name: Install ffmpeg
69-
run: sudo apt-get update -qq && sudo apt-get install -y --no-install-recommends ffmpeg
69+
uses: ./.github/actions/install-ffmpeg-linux
7070

7171
- name: Render changed block/component previews
7272
env:

‎.github/workflows/ci.yml‎

Lines changed: 2 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -322,9 +322,7 @@ jobs:
322322
node-version: 22
323323
- name: Install FFmpeg for integration tests
324324
if: matrix.lane == 'integration'
325-
run: |
326-
sudo apt-get update -qq
327-
sudo apt-get install -y --no-install-recommends ffmpeg
325+
uses: ./.github/actions/install-ffmpeg-linux
328326
- uses: ./.github/actions/prepare-ffmpeg-bin
329327
- run: bash scripts/ci/install-workspace-dependencies.sh
330328
- run: bun run --filter '@hyperframes/{parsers,lint,studio-server}' build
@@ -733,9 +731,7 @@ jobs:
733731
with:
734732
node-version: 22
735733
- name: Install FFmpeg
736-
run: |
737-
sudo apt-get update
738-
sudo apt-get install -y ffmpeg
734+
uses: ./.github/actions/install-ffmpeg-linux
739735
- uses: ./.github/actions/prepare-ffmpeg-bin
740736
- name: Install dependencies
741737
run: bash scripts/ci/install-workspace-dependencies.sh

‎.github/workflows/player-perf.yml‎

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -105,10 +105,7 @@ jobs:
105105
# to just install it here so the shard never trips on infra.
106106
- name: Install ffmpeg (parity shard only)
107107
if: matrix.shard == 'parity'
108-
run: |
109-
sudo apt-get update
110-
sudo apt-get install -y --no-install-recommends ffmpeg
111-
ffmpeg -version | head -n 1
108+
uses: ./.github/actions/install-ffmpeg-linux
112109

113110
- name: Run player perf — ${{ matrix.shard }} (measure mode)
114111
working-directory: packages/player

‎.github/workflows/preview-regression.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -93,7 +93,7 @@ jobs:
9393
run: bun run --cwd packages/producer parity:fixtures:ci
9494

9595
- name: Install ffmpeg
96-
run: sudo apt-get update -qq && sudo apt-get install -y --no-install-recommends ffmpeg
96+
uses: ./.github/actions/install-ffmpeg-linux
9797

9898
- name: Set up Chrome
9999
id: setup-chrome

0 commit comments

Comments
 (0)