Retrieve upstream tarballs from GitHub release assets#36435
Retrieve upstream tarballs from GitHub release assets#36435vbraun merged 8 commits intosagemath:developfrom
Conversation
…irror to mirror_list
…ces in SAGE_ROOT/.upstream.d
| continue | ||
| line = line.replace('${SAGE_SERVER}',) | ||
| line = line.strip() | ||
| if line.startswith('#'): |
There was a problem hiding this comment.
Speed probably doesn't matter too much, but wouldn't it be a little faster to move if not line and if line.startswith('#') earlier in this block?
src/bin/sage-update-version
Outdated
| ) > "$SAGE_ROOT/.upstream.d/20-github.com-sagemath-sage-releases" | ||
| rm -f "$SAGE_ROOT/.upstream.d/20-github.com-sagemath-sage-releases.tmp" | ||
|
|
||
| exit |
There was a problem hiding this comment.
Why exit? Doesn't that end the script before executing the rest of it? Also, do we need to git commit the new file?
There was a problem hiding this comment.
Whoops, that was debugging code. Fixed in 371be50
|
In principle it looks okay to me. I'm guessing that github will be a reliable place to store tarballs. Are there any downsides? |
|
I'm not aware of any downsides of this |
|
Documentation preview for this PR (built with commit 423e48a; changes) is ready! 🎉 |
|
Looks good, let's merge it. |
|
Thank you! |
|
I'm marking it as a blocker so that it takes effect in current CI runs, which currently can get stuck because of the missing tarballs (see https://github.com/sagemath/sage/actions/runs/6454925852/job/17521487027#step:8:14136) |
|
why is 10.1 hardcoded in .upstream.d/20-github.com-sagemath-sage-releases ? |
|
As the first two lines of the file say: |
sagemathgh-36435: Retrieve upstream tarballs from GitHub release assets <!-- ^^^^^ Please provide a concise, informative and self-explanatory title. Don't put issue numbers in there, do this in the PR body below. For example, instead of "Fixes sagemath#1234" use "Introduce new method to calculate 1+1" --> <!-- Describe your changes here in detail --> We propose to add GitHub release assets as a source from which upstream tarballs can be retrieved. This would take priority over trying the Sage mirrors, thus reducing the reliance of the project on self-maintained file servers. https://docs.github.com/en/repositories/releasing- projects-on-github/about-releases#storage-and-bandwidth-quotas For the purpose of testing this PR, one tarball missing on the mirrors (sagemath#36381 (comment)) has been uploaded to https://github.com/sagemath/sage/releases/tag/10.1 Creating GitHub releases and depositing the upstream tarballs can later be automated in a GH Actions workflow, e.g., using https://github.com/softprops/action-gh-release. As we don't make GitHub releases for beta versions, the previous stable release can be used. <!-- Why is this change required? What problem does it solve? --> <!-- If this PR resolves an open issue, please link to it here. For example "Fixes sagemath#12345". --> <!-- If your change requires a documentation PR, please link it appropriately. --> ### 📝 Checklist <!-- Put an `x` in all the boxes that apply. --> <!-- If your change requires a documentation PR, please link it appropriately --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> <!-- Feel free to remove irrelevant items. --> - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [ ] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies <!-- List all open PRs that this PR logically depends on - sagemath#12345: short description why this is a dependency - sagemath#34567: ... --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> URL: sagemath#36435 Reported by: Matthias Köppe Reviewer(s): Dima Pasechnik, John H. Palmieri, Matthias Köppe
sagemathgh-36438: .github/workflows/dist.yml: Create release on release tag <!-- ^^^^^ Please provide a concise, informative and self-explanatory title. Don't put issue numbers in there, do this in the PR body below. For example, instead of "Fixes sagemath#1234" use "Introduce new method to calculate 1+1" --> <!-- Describe your changes here in detail --> We extend our workflow dist.yml, which gets run on every release tag, to do the following: - For stable releases (no beta, no rc), when run on sagemath/sage, it automatically creates a Release. (This saves a few clicks by maintainers on each release.) - It uploads the sdist (the 1.2GB file which contains upstream tarballs for all standard packages) as a Release Asset. - It uploads the upstream tarballs as Release Assets. They will be used by the Sage distribution after sagemath#36435 Example of the run: https://github.com/mkoeppe/sage/actions/runs/6464035205/job/17549159925 (where I changed the condition from sagemath/sage to mkoeppe/sage), leading to the example release https://github.com/mkoeppe/sage/releases/tag/10.2.beta007 <!-- Why is this change required? What problem does it solve? --> <!-- If this PR resolves an open issue, please link to it here. For example "Fixes sagemath#12345". --> <!-- If your change requires a documentation PR, please link it appropriately. --> ### 📝 Checklist <!-- Put an `x` in all the boxes that apply. --> <!-- If your change requires a documentation PR, please link it appropriately --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> <!-- Feel free to remove irrelevant items. --> - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies <!-- List all open PRs that this PR logically depends on - sagemath#12345: short description why this is a dependency - sagemath#34567: ... --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> URL: sagemath#36438 Reported by: Matthias Köppe Reviewer(s): Dima Pasechnik
sagemathgh-36435: Retrieve upstream tarballs from GitHub release assets <!-- ^^^^^ Please provide a concise, informative and self-explanatory title. Don't put issue numbers in there, do this in the PR body below. For example, instead of "Fixes sagemath#1234" use "Introduce new method to calculate 1+1" --> <!-- Describe your changes here in detail --> We propose to add GitHub release assets as a source from which upstream tarballs can be retrieved. This would take priority over trying the Sage mirrors, thus reducing the reliance of the project on self-maintained file servers. https://docs.github.com/en/repositories/releasing- projects-on-github/about-releases#storage-and-bandwidth-quotas For the purpose of testing this PR, one tarball missing on the mirrors (sagemath#36381 (comment)) has been uploaded to https://github.com/sagemath/sage/releases/tag/10.1 Creating GitHub releases and depositing the upstream tarballs can later be automated in a GH Actions workflow, e.g., using https://github.com/softprops/action-gh-release. As we don't make GitHub releases for beta versions, the previous stable release can be used. <!-- Why is this change required? What problem does it solve? --> <!-- If this PR resolves an open issue, please link to it here. For example "Fixes sagemath#12345". --> <!-- If your change requires a documentation PR, please link it appropriately. --> ### 📝 Checklist <!-- Put an `x` in all the boxes that apply. --> <!-- If your change requires a documentation PR, please link it appropriately --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> <!-- Feel free to remove irrelevant items. --> - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [ ] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies <!-- List all open PRs that this PR logically depends on - sagemath#12345: short description why this is a dependency - sagemath#34567: ... --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> URL: sagemath#36435 Reported by: Matthias Köppe Reviewer(s): Dima Pasechnik, John H. Palmieri, Matthias Köppe
sagemathgh-36438: .github/workflows/dist.yml: Create release on release tag <!-- ^^^^^ Please provide a concise, informative and self-explanatory title. Don't put issue numbers in there, do this in the PR body below. For example, instead of "Fixes sagemath#1234" use "Introduce new method to calculate 1+1" --> <!-- Describe your changes here in detail --> We extend our workflow dist.yml, which gets run on every release tag, to do the following: - For stable releases (no beta, no rc), when run on sagemath/sage, it automatically creates a Release. (This saves a few clicks by maintainers on each release.) - It uploads the sdist (the 1.2GB file which contains upstream tarballs for all standard packages) as a Release Asset. - It uploads the upstream tarballs as Release Assets. They will be used by the Sage distribution after sagemath#36435 Example of the run: https://github.com/mkoeppe/sage/actions/runs/6464035205/job/17549159925 (where I changed the condition from sagemath/sage to mkoeppe/sage), leading to the example release https://github.com/mkoeppe/sage/releases/tag/10.2.beta007 <!-- Why is this change required? What problem does it solve? --> <!-- If this PR resolves an open issue, please link to it here. For example "Fixes sagemath#12345". --> <!-- If your change requires a documentation PR, please link it appropriately. --> ### 📝 Checklist <!-- Put an `x` in all the boxes that apply. --> <!-- If your change requires a documentation PR, please link it appropriately --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> <!-- Feel free to remove irrelevant items. --> - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies <!-- List all open PRs that this PR logically depends on - sagemath#12345: short description why this is a dependency - sagemath#34567: ... --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> URL: sagemath#36438 Reported by: Matthias Köppe Reviewer(s): Dima Pasechnik
sagemathgh-36438: .github/workflows/dist.yml: Create release on release tag <!-- ^^^^^ Please provide a concise, informative and self-explanatory title. Don't put issue numbers in there, do this in the PR body below. For example, instead of "Fixes sagemath#1234" use "Introduce new method to calculate 1+1" --> <!-- Describe your changes here in detail --> We extend our workflow dist.yml, which gets run on every release tag, to do the following: - For stable releases (no beta, no rc), when run on sagemath/sage, it automatically creates a Release. (This saves a few clicks by maintainers on each release.) - It uploads the sdist (the 1.2GB file which contains upstream tarballs for all standard packages) as a Release Asset. - It uploads the upstream tarballs as Release Assets. They will be used by the Sage distribution after sagemath#36435 Example of the run: https://github.com/mkoeppe/sage/actions/runs/6464035205/job/17549159925 (where I changed the condition from sagemath/sage to mkoeppe/sage), leading to the example release https://github.com/mkoeppe/sage/releases/tag/10.2.beta007 <!-- Why is this change required? What problem does it solve? --> <!-- If this PR resolves an open issue, please link to it here. For example "Fixes sagemath#12345". --> <!-- If your change requires a documentation PR, please link it appropriately. --> ### 📝 Checklist <!-- Put an `x` in all the boxes that apply. --> <!-- If your change requires a documentation PR, please link it appropriately --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> <!-- Feel free to remove irrelevant items. --> - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies <!-- List all open PRs that this PR logically depends on - sagemath#12345: short description why this is a dependency - sagemath#34567: ... --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> URL: sagemath#36438 Reported by: Matthias Köppe Reviewer(s): Dima Pasechnik
…ve `gentoo-python3.12` for now, use `conda-forge` with `-python3.11` <!-- ^^^^^ Please provide a concise, informative and self-explanatory title. Don't put issue numbers in there, do this in the PR body below. For example, instead of "Fixes sagemath#1234" use "Introduce new method to calculate 1+1" --> <!-- Describe your changes here in detail --> As part of the changes in sagemath#36435, we switched from http to https for downloading mirror_list. This broke centos-7; here we fix it. gentoo-python3.12 was activated a little bit too early. We remove it to clean up the portability CI workflow. conda-forge defaults to 3.12 already, and we're not ready for it. <!-- Why is this change required? What problem does it solve? --> <!-- If this PR resolves an open issue, please link to it here. For example "Fixes sagemath#12345". --> <!-- If your change requires a documentation PR, please link it appropriately. --> ### 📝 Checklist <!-- Put an `x` in all the boxes that apply. --> <!-- If your change requires a documentation PR, please link it appropriately --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> <!-- Feel free to remove irrelevant items. --> - [x] The title is concise, informative, and self-explanatory. - [ ] The description explains in detail what this PR is about. - [ ] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies <!-- List all open PRs that this PR logically depends on - sagemath#12345: short description why this is a dependency - sagemath#34567: ... --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> URL: sagemath#36534 Reported by: Matthias Köppe Reviewer(s): Kwankyu Lee
sagemathgh-36534: CI Linux: Fix `centos-7` after sagemath#36435, remove `gentoo-python3.12` for now, use `conda-forge` with `-python3.11` <!-- ^^^^^ Please provide a concise, informative and self-explanatory title. Don't put issue numbers in there, do this in the PR body below. For example, instead of "Fixes sagemath#1234" use "Introduce new method to calculate 1+1" --> <!-- Describe your changes here in detail --> As part of the changes in sagemath#36435, we switched from http to https for downloading mirror_list. This broke centos-7; here we fix it. gentoo-python3.12 was activated a little bit too early. We remove it to clean up the portability CI workflow. conda-forge defaults to 3.12 already, and we're not ready for it. <!-- Why is this change required? What problem does it solve? --> <!-- If this PR resolves an open issue, please link to it here. For example "Fixes sagemath#12345". --> <!-- If your change requires a documentation PR, please link it appropriately. --> ### 📝 Checklist <!-- Put an `x` in all the boxes that apply. --> <!-- If your change requires a documentation PR, please link it appropriately --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> <!-- Feel free to remove irrelevant items. --> - [x] The title is concise, informative, and self-explanatory. - [ ] The description explains in detail what this PR is about. - [ ] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies <!-- List all open PRs that this PR logically depends on - sagemath#12345: short description why this is a dependency - sagemath#34567: ... --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> URL: sagemath#36534 Reported by: Matthias Köppe Reviewer(s): Kwankyu Lee
sagemathgh-36533: `pkgs/sage-conf_pypi`: Repair after sagemath#36400, sagemath#36435 <!-- ^^^^^ Please provide a concise, informative and self-explanatory title. Don't put issue numbers in there, do this in the PR body below. For example, instead of "Fixes sagemath#1234" use "Introduce new method to calculate 1+1" --> <!-- Describe your changes here in detail --> This repairs the installation method described in https://github.com/sagemath/sage#alternative-installation-using-pypi broken by changes in sagemath#36400, sagemath#36435. We also expand the tests for this distribution in its tox.ini. Run `(cd pkgs/sage-conf_pypi && tox -v -v -e py311)` <!-- Why is this change required? What problem does it solve? --> <!-- If this PR resolves an open issue, please link to it here. For example "Fixes sagemath#12345". --> <!-- If your change requires a documentation PR, please link it appropriately. --> ### 📝 Checklist <!-- Put an `x` in all the boxes that apply. --> <!-- If your change requires a documentation PR, please link it appropriately --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> <!-- Feel free to remove irrelevant items. --> - [x] The title is concise, informative, and self-explanatory. - [ ] The description explains in detail what this PR is about. - [ ] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies <!-- List all open PRs that this PR logically depends on - sagemath#12345: short description why this is a dependency - sagemath#34567: ... --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> URL: sagemath#36533 Reported by: Matthias Köppe Reviewer(s): Kwankyu Lee
sagemathgh-36534: CI Linux: Fix `centos-7` after sagemath#36435, remove `gentoo-python3.12` for now, use `conda-forge` with `-python3.11` <!-- ^^^^^ Please provide a concise, informative and self-explanatory title. Don't put issue numbers in there, do this in the PR body below. For example, instead of "Fixes sagemath#1234" use "Introduce new method to calculate 1+1" --> <!-- Describe your changes here in detail --> As part of the changes in sagemath#36435, we switched from http to https for downloading mirror_list. This broke centos-7; here we fix it. gentoo-python3.12 was activated a little bit too early. We remove it to clean up the portability CI workflow. conda-forge defaults to 3.12 already, and we're not ready for it. <!-- Why is this change required? What problem does it solve? --> <!-- If this PR resolves an open issue, please link to it here. For example "Fixes sagemath#12345". --> <!-- If your change requires a documentation PR, please link it appropriately. --> ### 📝 Checklist <!-- Put an `x` in all the boxes that apply. --> <!-- If your change requires a documentation PR, please link it appropriately --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> <!-- Feel free to remove irrelevant items. --> - [x] The title is concise, informative, and self-explanatory. - [ ] The description explains in detail what this PR is about. - [ ] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies <!-- List all open PRs that this PR logically depends on - sagemath#12345: short description why this is a dependency - sagemath#34567: ... --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> URL: sagemath#36534 Reported by: Matthias Köppe Reviewer(s): Kwankyu Lee
We propose to add GitHub release assets as a source from which upstream tarballs can be retrieved. This would take priority over trying the Sage mirrors, thus reducing the reliance of the project on self-maintained file servers. https://docs.github.com/en/repositories/releasing-projects-on-github/about-releases#storage-and-bandwidth-quotas
For the purpose of testing this PR, one tarball missing on the mirrors (#36381 (comment)) has been uploaded to https://github.com/sagemath/sage/releases/tag/10.1
Creating GitHub releases and depositing the upstream tarballs can later be automated in a GH Actions workflow, e.g., using https://github.com/softprops/action-gh-release. As we don't make GitHub releases for beta versions, the previous stable release can be used.
📝 Checklist
⌛ Dependencies