so when using this patch on release-0.14.2, it seems to work but also cause a few warnings on the way, of the form curl: (22) The requested URL returned error: 404 Not Found, when trying to download a *spec.yaml file.

[spack] repr@StateOfTheArt:~/install-cw-software$ spack install m4
[+] /home/repr/cw-software-test/.spack/opt/spack/linux-debian10-x86_64/gcc-8.3.0/libsigsegv-2.12-w3hx4owtzlqntc5dhwliwxrdzxgcebuf
==> 8872: Installing m4
==> Finding buildcaches at https://spack-llnl-mirror.s3-us-west-2.amazonaws.com/build_cache
==> Fetching https://spack-llnl-mirror.s3-us-west-2.amazonaws.com/build_cache/linux-debian10-x86_64-gcc-8.3.0-m4-1.4.18-vjsnod5vuyykzvkclq4yqja2l3r3u45t.spec.yaml
-=O=#   #     #      #                                                                                                                    
curl: (22) The requested URL returned error: 404 Not Found
==> Failed to fetch file from URL: https://spack-llnl-mirror.s3-us-west-2.amazonaws.com/build_cache/linux-debian10-x86_64-gcc-8.3.0-m4-1.4.18-vjsnod5vuyykzvkclq4yqja2l3r3u45t.spec.yaml
    URL https://spack-llnl-mirror.s3-us-west-2.amazonaws.com/build_cache/linux-debian10-x86_64-gcc-8.3.0-m4-1.4.18-vjsnod5vuyykzvkclq4yqja2l3r3u45t.spec.yaml was not found!
==> Fetching from https://spack-llnl-mirror.s3-us-west-2.amazonaws.com/build_cache/linux-debian10-x86_64-gcc-8.3.0-m4-1.4.18-vjsnod5vuyykzvkclq4yqja2l3r3u45t.spec.yaml failed.
==> Fetching https://spack-llnl-mirror.s3-us-west-2.amazonaws.com/m4/m4-1.4.18.tar.gz

Not sure what to make of this, is that because there are some other changes I'm missing on 0.14.2?

Another thing I've noticed that the documentation on spack build caches here:
https://spack.readthedocs.io/en/latest/binary_caches.html#list-of-popular-build-caches
already mentions a mirror here: https://oaciss.uoregon.edu/e4s/inventory.html

So there may be some duplication here, and I wonder if adding one 'hardcoded' to spack and mentioning the other in the documentation might be confusing?

Interestingly, when using this mirror I'm also getting curl failure warnings:

Failed to fetch file from URL: https://cache.e4s.io/e4s/build_cache/linux-debian10-x86_64-gcc-8.3.0-openblas-0.3.8-aj2bc5mmjoprdugg2yqpribnufaucvqp.spec.yaml

so this seems to be a problem with using mirrors in 0.14.2 in general.

Regarding #17077 (comment): those messages are about missing files for binary caches, this particular mirror only stores source archives vs. archives of packages built by Spack (mirrors can store both).

a mirror storing the latest (as of now) version of each package

Wuhu! Does this include pulled patches from URLs? Got quite some downtimes in those recently for fundamental packages.

Hi @scheibelp, is there a way to disable the fetching on spack-public by default? I tried to do spack mirror rm spack-public but got ==> Error: No mirror with name spack-public

@victorapm: The public mirror is a spack default the way we did it here:

$ spack config blame mirrors
---                                                          mirrors:
/Users/gamblin2/src/spack/etc/spack/defaults/mirrors.yaml:2    spack-public: https://spack-llnl-mirror.s3-us-west-2.amazonaws.com/

at the moment you need to edit ~/.spack/mirrors.yaml and make it look like this:

mirrors:: {}

The :: says "override all other configs instead of merging", and it overrides the defaults with an empty dictionary.

We usually assume that users won't edit the defaults but maybe we need a way to get rid of the mirror.

Question: why do you want to?

Thanks, @tgamblin!

In Hypre, we have a regression test that builds it through spack and redirects error messages to a file. The test fails if the error file is not empty. After this merge, I believe, we started to see curl: (22) The requested URL returned error: 404 Not Found. in the error file causing the test to fail. This error message comes from the fetching procedure and to fix this I was thinking to tell spack to fetch packages in the old way, without searching the spack-public mirror.

Do you think this fix is reasonable? Thanks for the help!

@victorapm: Thanks!

I think the fix is reasonable. I suspect you will not need to add it soon -- the first curl failure will go away once @scheibelp rebuilds the mirror to be indexed by sha256 archive names (it's currently indexed by the older human-readable archive name, which is not fetched first by Spack).

Does this include pulled patches from URLs?

@ax3l yes it does