Fetching from URLs falls back to mirrors if they exist by alalazo · Pull Request #13881 · spack/spack

alalazo · 2019-11-25T13:22:05Z

fixes #5613
fixes #4212
fixes #1090
refers to #13874

This PR permits to list mirrors of the main url in packages. If the main url is not available the url in the mirrors will be tried in order. It also defines a GNUMirrorPackage mixin to deal with the special case of GNU packages and modifies all the GNU packages in the built-in repository.

Note that the feature is implemented starting from 5b377a0 while previous commits are just trivial refactors that should reduce a bit the complexity of fetchers. If they are considered noise for a review I can split this PR into two.

lib/spack/spack/fetch_strategy.py

adamjstewart · 2019-11-25T18:23:35Z

This feature has been in demand for quite some time! I think it will be less valuable once Spack has its own source mirror on AWS, but still worth adding in my opinion!

alalazo · 2019-11-26T10:55:02Z

Considering the number of GNU packages that we support (latest count 55), if people agree I can go one step further and create a GNUFetcher mixin. At that point we could simplify a bit packages:

class Autoconf(AutotoolsPackage, GNUFetcher):
    gnu_path = 'autoconf/autoconf-2.69.tar.gz'

and have main url and mirrors setup automatically. This will give a single central place to change if we need to add further mirrors for GNU packages or switch the main url to something else. Waiting to receive feedback on this before proceeding.

Once the feature set of this PR is decided I'll also add docs.

ax3l · 2019-11-26T18:00:43Z

Looks good so far and GNU fetcher abstraction sounds good to me!

alalazo · 2019-11-28T16:40:46Z

Can people have a look at f510531 and let me know what they think? If there's agreement on a solution like that I'll go on and modify the remaining 54 GNU packages.

alalazo · 2019-11-28T16:41:06Z

And add docs too 🙂

nazavode · 2019-11-28T16:56:47Z

Just tried this patch on 008fe37 and spack fetch autoconf finally works. The certificate of the gnu mirror that is usually assigned to southern europe (https://mirror.kumi.systems/gnu/) expired yesterday. Today has been a bloodbath, had to sed my way to make GNU stuff work again.

alalazo · 2019-11-29T11:56:19Z

This PR is ready for a first round of reviews

ax3l

This is really cool, thank you so much!

I wonder if we should also re-try with the next mirror if the downloaded package archive hash does not match what we expected. This can be the case in captive portals but also if some pages moved and are wildcard redirected, or if index/404 pages are downloaded instead of the expected source.

tgamblin

@alalazo: this looks awesome -- I have one minor renaming request and one request for DSL simplification.

lib/spack/docs/packaging_guide.rst

lib/spack/spack/build_systems/gnu.py

alalazo · 2019-12-10T05:59:02Z

@tgamblin I think all the points raised in the review have been addressed. I have in my list further refactors involving fetchers, but if you agree I would save them for later PRs. Same for @ax3l request to try the next mirror on a failed checksum.

tgamblin

@alalazo: a few more change requests but this looks really good to me.

lib/spack/spack/build_systems/gnu.py

lib/spack/spack/fetch_strategy.py

lib/spack/spack/package.py

lib/spack/spack/fetch_strategy.py

var/spack/repos/builtin/packages/bash/package.py

…tly)

Previously the needed to define 'url' and optionally 'mirrors'

alalazo · 2019-12-20T14:49:54Z

@tgamblin Let me know if there are further changes needed in this PR

alalazo added the fetching label Nov 25, 2019

alalazo requested review from adamjstewart, opadron, scheibelp and tldahlgren November 25, 2019 13:22

alalazo force-pushed the refactor/fetchers branch 2 times, most recently from 973cda2 to 66138d2 Compare November 25, 2019 15:01

alalazo commented Nov 25, 2019

View reviewed changes

lib/spack/spack/fetch_strategy.py Outdated Show resolved Hide resolved

alalazo requested a review from ax3l November 26, 2019 10:55

alalazo force-pushed the refactor/fetchers branch from 8162844 to f510531 Compare November 28, 2019 16:31

alalazo marked this pull request as ready for review November 29, 2019 11:06

ax3l approved these changes Dec 3, 2019

View reviewed changes

tgamblin requested changes Dec 5, 2019

View reviewed changes

lib/spack/docs/packaging_guide.rst Show resolved Hide resolved

lib/spack/spack/build_systems/gnu.py Outdated Show resolved Hide resolved

alalazo force-pushed the refactor/fetchers branch from 0af5ae7 to 591c36e Compare December 9, 2019 16:00

ax3l approved these changes Dec 10, 2019

View reviewed changes

This was referenced Dec 10, 2019

Bugfix/mirror symlinks #13908

Merged

Installation issue: cp2k #12701

Closed

scheibelp mentioned this pull request Dec 13, 2019

Mirrors: add option to exclude packages from "mirror create" #14154

Merged

tgamblin requested changes Dec 15, 2019

View reviewed changes

alalazo added 3 commits December 17, 2019 16:37

fetch_strategy: substituted a metaclass with a decorator

442238c

fetch_strategy: import statements are in lexicographical order

99f3dfe

fetch_strategy: removed unused property in favor of an attribute

f0803f9

alalazo added 10 commits December 17, 2019 16:37

fetch_strategy: removed method to set an attribute (just set it direc…

d9cd790

…tly)

URLFetchStrategy can use multiple mirrors

8891cad

URLFetchStrategy: pass mirrors through pattern substitution

fb81289

Factored GNU specific URLs and mirrors into a mixin

a679d95

Modified all the GNU packages we have to inherit from GNUPackage

2b8c9bc

Added documentation for the feature

9006f13

Package needs to define either 'url' or 'urls'

27ce873

Previously the needed to define 'url' and optionally 'mirrors'

Renamed gnu_path -> gnu_mirror_path

8c08cac

Restored whitespaces in fetch_strategy.py

d201f17

TODO to extract a function for version substitution on urls

a075753

alalazo force-pushed the refactor/fetchers branch from b9b7939 to a075753 Compare December 17, 2019 18:57

tgamblin approved these changes Dec 20, 2019

View reviewed changes

tgamblin merged commit 497fddf into spack:develop Dec 20, 2019

alalazo deleted the refactor/fetchers branch December 21, 2019 07:25

scheibelp mentioned this pull request Jan 24, 2020

Force non-interactive git fetch #14616

Closed

2 tasks

alalazo mentioned this pull request Feb 7, 2020

Feature request: support multiple url in case websites are down #11030

Closed

adamjstewart mentioned this pull request Feb 16, 2020

Installation issue: mpfr #14901

Closed

mwkrentel mentioned this pull request Mar 11, 2020

spack info errors out with multiple urls #15449

Closed

Conversation

alalazo commented Nov 25, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

adamjstewart commented Nov 25, 2019

Uh oh!

alalazo commented Nov 26, 2019

Uh oh!

ax3l commented Nov 26, 2019

Uh oh!

alalazo commented Nov 28, 2019

Uh oh!

alalazo commented Nov 28, 2019

Uh oh!

nazavode commented Nov 28, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alalazo commented Nov 29, 2019

Uh oh!

ax3l left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tgamblin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

alalazo commented Dec 10, 2019

Uh oh!

tgamblin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alalazo commented Dec 20, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

alalazo commented Nov 25, 2019 •

edited

Loading

nazavode commented Nov 28, 2019 •

edited

Loading

ax3l left a comment •

edited

Loading