Before this was done only conditionally (no args given) because it can be extremely expensive (e.g. qt which does a lot of spidering). What's the runtime of spack checksum [email protected] on this branch?

It is expensive, but it's also the only way to get correct behavior for packages where the URLs generated by their packages may or may not be correct. The direct answer to your question is that [email protected] takes 3 minutes and 43 seconds.

As far as I can think right now, the only way to deal with this better is to make it illegal for a package to generate broken links, but then R packages can't work. We could work around that by changing the contract for generating URLs from a version to produce a list of URLs, at least one of which must be valid if the version is valid, but that would be a breaking change.

I suppose one intermediate step might be that we can do a check on the provided url and see if we get a 404, and only then fetch remote versions if we hit a failure from the package generated url?

Yuck. Does it take that long to start downloading a valid version, such as 5.15.2? I know the qt "list URLs" is a bit of an outlier but it's going to be ugly if a new version comes out but takes minutes to verify the URL, even if we know it exists.

For spack checksum yes it does, for spack fetch it does not (spack fetch [email protected] takes 16 seconds total with download time), this only gets called if the first URL fetch tries fails. We could do basically the same thing for checksum to shortcut it for valid URLs, it's not quite that simple but it is possible.

Yes, it would be ideal if you could make pkg.fetch_remote_versions a lazy evaluation... it's something of a nuclear option, and because it uses the URL spidering can be unstable -- best if the checksum command only calls it if really necessary.

Ok, so actually it should use the urls attribute rather than the url attribute, that will let both be specified and both will be tried. The way it is now, with one as list_url will also work after this change.

To be more complete about it, the code used to assume that the package could always provide a single url that could be determined to be valid without doing a test fetch. That doesn't really work in the general case, this makes the fallback cases a whole lot more explicit, and uses them in more places. The fetch strategy already did some of this, but checksum and similar did not.

The urls attribute currently fails for fetching the older versions of packages listed in the package file. Interestingly, checksum works for any versions not listed in the package file. Is that something you are trying to fix here as well?

Sorry, I should have pulled the latest changes before posting. With the most recent changes in this PR, spack checksum now works with the urls attribute. Fetching/staging still fails though.

Ok now that's interesting. The change I had but walked back would fix that, I expected current behavior would as well. I'm tempted to say keep it list_url for now, and we can open something to discuss whether the fetch strategy should use my new logic for using fetch checks by default. Honestly I think it should, but there's a unit test that walks every version in every package, and generates fetch strategies for them. That is a recipe for disaster so I walked it back to use just the new url listing stuff, but it should check all the mirrors rather than just dying... that seems like a separate issue though since it dies currently too I guess.