Skip to content

Commit a714805

Browse files
juenglinjuenglin
committed
ci: route all artifact downloads through download-artifact-retry
Follow-on to #3037. A job testing against the prior branch recently hit: HTTP 503: Egress is over the account limit (productionresultssa17.blob.core.windows.net) The failing step used raw 'gh run download' with no retry. Some sibling steps were already protected by the download-artifact-retry composite action introduced in #3037, but that action only accepted an exact artifact name, and several steps bypassed it entirely. This commit: * Extends the action with 'pattern' and 'merge-multiple' inputs so it can replace glob-pattern gh run download calls. Adds mutual- exclusion validation (exactly one of 'name' or 'pattern' must be supplied). Adds a third attempt with twice the base delay (15s -> 30s) to give egress throttling more time to clear. * Splits the backport lookup-then-download steps in test-wheel-linux.yml, test-wheel-windows.yml, and build-wheel.yml into two steps each: a lookup step that emits run-id and artifact-pattern as step outputs, then an action download step. Adds a verify step in the test-wheel workflows to confirm exactly one bindings wheel arrived (previously guaranteed by compgen -G). * Repoints all existing pinned @f20d661 callers in the test-wheel workflows to $/ so they pick up the new third attempt. * Swaps every direct actions/download-artifact call in build-wheel.yml, build-docs.yml, and coverage.yml for the action. * Adds an internal retry loop (5 attempts, 15s/30s/60s/120s/120s delays) to ci/tools/download-wheels so the release path is also covered. * Adds build-docs.yml to the actionlint.yaml $/ suppression list.
1 parent 03b4153 commit a714805

8 files changed

Lines changed: 205 additions & 97 deletions

File tree

‎.github/actionlint.yaml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,9 @@ paths:
1515
".github/workflows/build-docs.yml":
1616
ignore:
1717
- 'unexpected key "queue" for "concurrency" section'
18+
# GitHub supports self-repository ($/) action references, but the latest
19+
# actionlint release does not yet recognize them (rhysd/actionlint#711).
20+
- 'specifying action "\$/[^"]+" in invalid format because ref is missing'
1821
".github/workflows/cleanup-pr-previews.yml":
1922
ignore:
2023
- 'unexpected key "queue" for "concurrency" section'

‎.github/actions/download-artifact-retry/action.yml‎

Lines changed: 68 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -5,16 +5,33 @@
55
name: Download artifact (with retry)
66

77
description: >-
8-
Wrapper around actions/download-artifact that retries once after a short
9-
delay when the first attempt fails. Designed to absorb transient
10-
infrastructure flakes such as "other side closed" without failing the job.
8+
Wrapper around actions/download-artifact that retries up to three times with
9+
exponential backoff when a download fails. Handles both transient TCP errors
10+
such as "other side closed" and storage-layer throttling such as
11+
"HTTP 503: Egress is over the account limit". Accepts either an exact
12+
artifact name or a glob pattern; supply exactly one.
1113
1214
inputs:
1315
name:
14-
description: "Name of the artifact to download"
15-
required: true
16+
description: >-
17+
Exact name of the artifact to download. Mutually exclusive with
18+
'pattern'; supply exactly one of the two.
19+
required: false
20+
default: ""
21+
pattern:
22+
description: >-
23+
Glob pattern matching one or more artifact names. Mutually exclusive
24+
with 'name'; supply exactly one of the two.
25+
required: false
26+
default: ""
27+
merge-multiple:
28+
description: >-
29+
When true and 'pattern' is set, merge files from all matched artifacts
30+
directly into 'path' instead of placing each in its own subdirectory.
31+
required: false
32+
default: "false"
1633
path:
17-
description: "Destination path for the downloaded artifact"
34+
description: "Destination path for the downloaded artifact(s)"
1835
required: true
1936
run-id:
2037
description: >-
@@ -32,39 +49,80 @@ inputs:
3249
required: false
3350
default: ""
3451
retry-delay-seconds:
35-
description: "Seconds to wait between the first failure and the retry"
52+
description: "Seconds to wait between the first failure and the second attempt"
3653
required: false
3754
default: "15"
3855

3956
runs:
4057
using: composite
4158
steps:
59+
- name: Validate inputs
60+
shell: bash --noprofile --norc -euo pipefail {0}
61+
env:
62+
INPUT_NAME: ${{ inputs.name }}
63+
INPUT_PATTERN: ${{ inputs.pattern }}
64+
run: |
65+
if [[ -n "${INPUT_NAME}" && -n "${INPUT_PATTERN}" ]]; then
66+
echo "::error::Specify exactly one of 'name' or 'pattern', not both"
67+
exit 1
68+
fi
69+
if [[ -z "${INPUT_NAME}" && -z "${INPUT_PATTERN}" ]]; then
70+
echo "::error::Specify exactly one of 'name' or 'pattern'"
71+
exit 1
72+
fi
73+
4274
- name: Download artifact (attempt 1)
4375
id: attempt1
4476
continue-on-error: true
4577
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
4678
with:
4779
name: ${{ inputs.name }}
80+
pattern: ${{ inputs.pattern }}
81+
merge-multiple: ${{ inputs.merge-multiple }}
4882
path: ${{ inputs.path }}
4983
run-id: ${{ inputs.run-id }}
5084
github-token: ${{ inputs.github-token }}
5185

52-
- name: Wait before retry
86+
- name: Wait before retry (attempt 2)
5387
if: ${{ steps.attempt1.outcome == 'failure' }}
5488
shell: bash --noprofile --norc -euo pipefail {0}
5589
env:
56-
# Pass through env instead of interpolating the input into the shell
57-
# script, so a non-numeric value cannot inject commands.
5890
RETRY_DELAY_SECONDS: ${{ inputs.retry-delay-seconds }}
5991
run: |
6092
echo "::warning::Artifact download failed (attempt 1); retrying in ${RETRY_DELAY_SECONDS}s"
6193
sleep "$RETRY_DELAY_SECONDS"
6294
6395
- name: Download artifact (attempt 2)
96+
id: attempt2
6497
if: ${{ steps.attempt1.outcome == 'failure' }}
98+
continue-on-error: true
99+
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
100+
with:
101+
name: ${{ inputs.name }}
102+
pattern: ${{ inputs.pattern }}
103+
merge-multiple: ${{ inputs.merge-multiple }}
104+
path: ${{ inputs.path }}
105+
run-id: ${{ inputs.run-id }}
106+
github-token: ${{ inputs.github-token }}
107+
108+
- name: Wait before retry (attempt 3)
109+
if: ${{ steps.attempt1.outcome == 'failure' && steps.attempt2.outcome == 'failure' }}
110+
shell: bash --noprofile --norc -euo pipefail {0}
111+
env:
112+
RETRY_DELAY_SECONDS: ${{ inputs.retry-delay-seconds }}
113+
run: |
114+
# Double the delay for the second retry to give egress throttling more time to clear.
115+
RETRY_DELAY_SECONDS=$(( RETRY_DELAY_SECONDS * 2 ))
116+
echo "::warning::Artifact download failed (attempt 2); retrying in ${RETRY_DELAY_SECONDS}s"
117+
sleep "$RETRY_DELAY_SECONDS"
118+
119+
- name: Download artifact (attempt 3)
120+
if: ${{ steps.attempt1.outcome == 'failure' && steps.attempt2.outcome == 'failure' }}
65121
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
66122
with:
67123
name: ${{ inputs.name }}
124+
pattern: ${{ inputs.pattern }}
125+
merge-multiple: ${{ inputs.merge-multiple }}
68126
path: ${{ inputs.path }}
69127
run-id: ${{ inputs.run-id }}
70128
github-token: ${{ inputs.github-token }}

‎.github/workflows/build-docs.yml‎

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -144,7 +144,7 @@ jobs:
144144
echo "CUDA_BINDINGS_ARTIFACTS_DIR=$(realpath "$REPO_DIR/$CUDA_BINDINGS_SOURCE_DIR/dist")" >> $GITHUB_ENV
145145
146146
- name: Download cuda-python build artifacts
147-
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
147+
uses: $/.github/actions/download-artifact-retry
148148
with:
149149
name: cuda-python-wheel
150150
path: .
@@ -157,7 +157,7 @@ jobs:
157157
ls -lahR .
158158
159159
- name: Download cuda-pathfinder build artifacts
160-
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
160+
uses: $/.github/actions/download-artifact-retry
161161
with:
162162
name: cuda-pathfinder-wheel
163163
path: ./cuda_pathfinder
@@ -171,17 +171,17 @@ jobs:
171171
172172
- name: Download cuda.bindings build artifacts
173173
if: ${{ !inputs.is-release }}
174-
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
174+
uses: $/.github/actions/download-artifact-retry
175175
with:
176176
name: ${{ env.CUDA_BINDINGS_ARTIFACT_NAME }}
177177
path: ${{ env.CUDA_BINDINGS_ARTIFACTS_DIR }}
178178

179179
- name: Download cuda.bindings build artifacts
180180
if: ${{ inputs.is-release }}
181-
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
181+
uses: $/.github/actions/download-artifact-retry
182182
with:
183183
pattern: ${{ env.CUDA_BINDINGS_ARTIFACT_NAME }}
184-
merge-multiple: true
184+
merge-multiple: "true"
185185
path: ${{ env.CUDA_BINDINGS_ARTIFACTS_DIR }}
186186
run-id: ${{ inputs.run-id }}
187187
github-token: ${{ github.token }}
@@ -193,17 +193,17 @@ jobs:
193193
194194
- name: Download cuda.core build artifacts
195195
if: ${{ !inputs.is-release }}
196-
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
196+
uses: $/.github/actions/download-artifact-retry
197197
with:
198198
name: ${{ env.CUDA_CORE_ARTIFACT_NAME }}
199199
path: ${{ env.CUDA_CORE_ARTIFACTS_DIR }}
200200

201201
- name: Download cuda.core build artifacts
202202
if: ${{ inputs.is-release }}
203-
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
203+
uses: $/.github/actions/download-artifact-retry
204204
with:
205205
pattern: ${{ env.CUDA_CORE_ARTIFACT_NAME }}
206-
merge-multiple: true
206+
merge-multiple: "true"
207207
path: ${{ env.CUDA_CORE_ARTIFACTS_DIR }}
208208
run-id: ${{ inputs.run-id }}
209209
github-token: ${{ github.token }}
@@ -366,7 +366,7 @@ jobs:
366366
ref: ${{ inputs.git-tag }}
367367

368368
- name: Download docs deployment payload
369-
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
369+
uses: $/.github/actions/download-artifact-retry
370370
with:
371371
name: ${{ needs.build.outputs.deployment-artifact-name }}
372372
path: artifacts/docs/

‎.github/workflows/build-wheel.yml‎

Lines changed: 19 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -170,7 +170,7 @@ jobs:
170170
171171
- name: Download reusable cuda.pathfinder wheel
172172
if: ${{ env.BUILD_PATHFINDER != 'true' }}
173-
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
173+
uses: $/.github/actions/download-artifact-retry
174174
with:
175175
name: cuda-pathfinder-wheel
176176
path: cuda_pathfinder
@@ -271,7 +271,7 @@ jobs:
271271

272272
- name: Download reusable cuda.bindings wheel
273273
if: ${{ env.BUILD_BINDINGS != 'true' }}
274-
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
274+
uses: $/.github/actions/download-artifact-retry
275275
with:
276276
name: ${{ env.CUDA_BINDINGS_ARTIFACT_BASENAME }}-${{ env.BASELINE_SHA }}
277277
path: ${{ env.CUDA_BINDINGS_ARTIFACTS_DIR }}
@@ -395,7 +395,7 @@ jobs:
395395
396396
- name: Download reusable cuda.core wheel
397397
if: ${{ env.BUILD_CORE != 'true' }}
398-
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
398+
uses: $/.github/actions/download-artifact-retry
399399
with:
400400
name: ${{ env.CUDA_CORE_ARTIFACT_BASENAME }}-${{ env.BASELINE_SHA }}
401401
path: ${{ env.CUDA_CORE_ARTIFACTS_DIR }}
@@ -413,7 +413,7 @@ jobs:
413413
414414
- name: Download reusable cuda-python wheel
415415
if: ${{ env.BUILD_PYTHON != 'true' && strategy.job-index == 0 && inputs.host-platform == 'linux-64' }}
416-
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
416+
uses: $/.github/actions/download-artifact-retry
417417
with:
418418
name: cuda-python-wheel
419419
path: cuda_python
@@ -542,14 +542,15 @@ jobs:
542542
${{ env.CUDA_CORE_TEST_BINARIES_DIR }}/*.lib
543543
if-no-files-found: error
544544

545-
- name: Download cuda.bindings build artifacts from the prior branch
545+
- name: Look up prior-branch cuda.bindings run ID
546+
id: prior-bindings
546547
if: ${{ !inputs.single-cuda-major && env.BUILD_CORE == 'true' }}
547548
env:
548549
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
549550
run: |
550551
if ! (command -v gh 2>&1 >/dev/null); then
551552
# See https://github.com/cli/cli/blob/trunk/docs/install_linux.md#debian-ubuntu-linux-raspberry-pi-os-apt.
552-
# gh is needed for artifact fetching.
553+
# gh is needed for lookup-run-id, which calls gh api.
553554
mkdir -p -m 755 /etc/apt/keyrings \
554555
&& out=$(mktemp) && wget -nv -O$out https://cli.github.com/packages/githubcli-archive-keyring.gpg \
555556
&& cat $out | tee /etc/apt/keyrings/githubcli-archive-keyring.gpg > /dev/null \
@@ -565,18 +566,18 @@ jobs:
565566
--branch "${OLD_BRANCH}" \
566567
--artifact "${OLD_ARTIFACT_PATTERN}" \
567568
NVIDIA/cuda-python "CI")
568-
PREV_BINDINGS_DIR="cuda_bindings_13/dist-prev"
569-
570-
gh run download \
571-
"${LATEST_PRIOR_RUN_ID}" \
572-
-p "${OLD_ARTIFACT_PATTERN}" \
573-
-R NVIDIA/cuda-python
574-
OLD_ARTIFACT_DIR=$(compgen -G "${OLD_ARTIFACT_PATTERN}")
575-
test -d "${OLD_ARTIFACT_DIR}"
576-
ls -al "${OLD_ARTIFACT_DIR}"
577-
mkdir -p "${PREV_BINDINGS_DIR}"
578-
mv "${OLD_ARTIFACT_DIR}"/*.whl "${PREV_BINDINGS_DIR}"
579-
rmdir "${OLD_ARTIFACT_DIR}"
569+
echo "run-id=${LATEST_PRIOR_RUN_ID}" >> "${GITHUB_OUTPUT}"
570+
echo "artifact-pattern=${OLD_ARTIFACT_PATTERN}" >> "${GITHUB_OUTPUT}"
571+
572+
- name: Download prior-branch cuda.bindings build artifacts
573+
if: ${{ !inputs.single-cuda-major && env.BUILD_CORE == 'true' }}
574+
uses: $/.github/actions/download-artifact-retry
575+
with:
576+
pattern: ${{ steps.prior-bindings.outputs.artifact-pattern }}
577+
merge-multiple: "true"
578+
path: cuda_bindings_13/dist-prev
579+
run-id: ${{ steps.prior-bindings.outputs.run-id }}
580+
github-token: ${{ secrets.GITHUB_TOKEN }}
580581

581582
- name: Constrain previous cuda.core to the downloaded cuda.bindings wheel
582583
if: ${{ !inputs.single-cuda-major && env.BUILD_CORE == 'true' }}

‎.github/workflows/coverage.yml‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -386,7 +386,7 @@ jobs:
386386
cuda-version: ${{ env.CUDA_VER }}
387387

388388
- name: Download Windows wheel artifacts
389-
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
389+
uses: $/.github/actions/download-artifact-retry
390390
with:
391391
name: coverage-windows-wheels
392392
path: ./wheels/
@@ -485,19 +485,19 @@ jobs:
485485
pip install coverage[toml] Cython
486486
487487
- name: Download Linux coverage data
488-
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
488+
uses: $/.github/actions/download-artifact-retry
489489
with:
490490
name: coverage-data-linux
491491
path: ./
492492

493493
- name: Download cuda source code
494-
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
494+
uses: $/.github/actions/download-artifact-retry
495495
with:
496496
name: cuda-source-linux
497497
path: ./cuda/
498498

499499
- name: Download Windows coverage data
500-
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
500+
uses: $/.github/actions/download-artifact-retry
501501
with:
502502
name: coverage-data-windows
503503
path: ./
@@ -583,7 +583,7 @@ jobs:
583583
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
584584

585585
- name: Download combined coverage results
586-
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
586+
uses: $/.github/actions/download-artifact-retry
587587
with:
588588
name: coverage-combined
589589
path: docs/coverage/

0 commit comments

Comments
 (0)