Skip to content

Commit d220522

Browse files
authored
Upload release assets one at a time to avoid the secondary rate limit (#1503)
<!-- Thank you for your contribution! --> ## What do these changes do? The v6.9.0 deploy job ([run](https://github.com/aio-libs/multidict/actions/runs/35338595754/job/105598615903)) failed on the last step: `softprops/action-gh-release` uploaded 513 files to the GitHub Release in parallel and GitHub rejected the burst with a secondary rate limit error after 499 of them. PyPI publishing and Sigstore signing had already succeeded. The step is replaced with a shell loop that uploads one file at a time with `gh release upload`, skips assets already attached to the release, and retries each upload up to five times with an increasing pause. A re-run of the job therefore only fills in what a previous attempt missed. The set of files attached to the release is unchanged. The 14 wheels missing from v6.9.0 were uploaded by hand from the run artifacts after checking their SHA256 against the Sigstore bundles on the release and against PyPI; the release now has all 513 assets. ## Are there changes in behavior for the user? No. ## Is it a substantial burden for the maintainers to support this? No, it is a short shell step with no third party action. ## Related issue number N/A ## Checklist - [x] I think the code is well written - [ ] Unit tests for the changes exist N/A, release workflow only - [ ] Documentation reflects the changes N/A - [ ] If you provide code modification, please add yourself to `CONTRIBUTORS.txt` N/A - [x] Add a new news fragment into the `CHANGES/` folder - [x] `make doc-spelling` passes and any new technical words are added to `docs/spelling_wordlist.txt` <details> <summary>Agent run details (optional, for reviewers)</summary> Lint: `pre-commit run --files .github/workflows/ci-cd.yml` passed (actionlint, timeout-minutes check, yamllint). Spelling: `make doc-spelling` passed. Dry run: the skip logic was run against the live v6.9.0 release with the upload replaced by `echo`; it selected exactly the 14 missing wheels and skipped files already attached. The real upload path only runs on a release tag and was not exercised end to end. </details> Drafted with Claude Opus 5 (Claude Code); reviewed by @asvetlov.
1 parent 30cd596 commit d220522

2 files changed

Lines changed: 32 additions & 9 deletions

File tree

‎.github/workflows/ci-cd.yml‎

Lines changed: 27 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -944,7 +944,9 @@ jobs:
944944
- build-wheels-for-tested-arches
945945
- pre-setup # transitive, for accessing settings
946946
runs-on: ubuntu-latest
947-
timeout-minutes: 15
947+
# Sequential release asset uploads plus their retry pauses need
948+
# well over the 15 minutes the rest of the job takes.
949+
timeout-minutes: 45
948950

949951
permissions:
950952
contents: write # IMPORTANT: mandatory for making GitHub Releases
@@ -1028,13 +1030,29 @@ jobs:
10281030
./dist/*.whl
10291031
10301032
- name: Upload artifact signatures to GitHub Release
1031-
# Confusingly, this action also supports updating releases, not
1032-
# just creating them. This is what we want here, since we've manually
1033-
# created the release above.
1034-
uses: softprops/action-gh-release@v3
1035-
with:
1036-
# dist/ contains the built packages, which smoketest-artifacts/
1037-
# contains the signatures and certificates.
1038-
files: dist/**
1033+
# Upload one file at a time: a parallel burst of 500+ assets trips
1034+
# GitHub's secondary rate limit. Assets already on the release are
1035+
# skipped so a re-run only fills in what a previous attempt missed.
1036+
env:
1037+
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
1038+
TAG: ${{ github.ref_name }}
1039+
run: |
1040+
gh release view "${TAG}" --repo "${GITHUB_REPOSITORY}" \
1041+
--json assets --jq '.assets[].name' > "${RUNNER_TEMP}/existing"
1042+
for file in dist/*; do
1043+
if grep -qxF "$(basename "${file}")" "${RUNNER_TEMP}/existing"; then
1044+
continue
1045+
fi
1046+
for attempt in 1 2 3 4 5; do
1047+
if gh release upload "${TAG}" "${file}" \
1048+
--repo "${GITHUB_REPOSITORY}"; then
1049+
break
1050+
fi
1051+
if [ "${attempt}" -eq 5 ]; then
1052+
exit 1
1053+
fi
1054+
sleep $((attempt * 60))
1055+
done
1056+
done
10391057
10401058
...

‎CHANGES/1503.contrib.rst‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
The release job was changed to upload distributions and their signatures
2+
to the GitHub Release one file at a time, skipping assets that were
3+
already attached and retrying after a pause, so that a parallel upload
4+
burst no longer tripped the GitHub secondary rate limit
5+
-- by :user:`asvetlov`.

0 commit comments

Comments
 (0)