create (via find_versions_of_archive()) does wrong thing for Go projects on GitHub with GoPackage #13940
Description
I'm working on a GoPackage build system (see #13023) in a Spack tree at commit 94c8bf032.
TL;DR:
I'm coincidentally/unfortunately prototyping with projects (e.g. lazygit, lazydocker) that have uploaded a number of "assets" for a release (so many "releases/download" URLs are discovered), but for which the source itself is at an "archive" URL. Spack's heuristic of preferring "download" URLs causes it to get the wrong tarball for a release, which causes the package type guesser to fail and/or to insert an incorrect checksum for a version.
Perhaps there should there be an option to spack create that tells it to just use the provided URL and not try to guess additional versions.
I'd like spack create to Do The Right Thing for Go packages on GitHub, e.g.
spack create https://github.com/jesseduffield/lazygit/archive/v0.11.3.tar.gz
should use the GoPackage template that I've created. Instead of offering to use the URL that I provided it offers me a series of ".../releases/download/" URLS that point to the binaries that have been attached to the releases. It does, eventually, offers a couple of older "archive" tarballs.
I believe that for the recent versions, it's choosing one of the binaries that have been attached to the release (with a .../releases/download/... URL) in preference to the archive URL that I provided, driven by code following this comment in spack/spack/util/web.py. The older releases don't have any attached "assets" and so the Right Thing happens.
I believe that the intent is to prefer maintainer uploaded release source tarballs (which e.g. may already have been autoconfed) in preference to the automagically generated tarball of the repo at that particular tag. In this/my case that turns out to be The Wrong Thing.
Those binary-containing release tarballs don't contain the source tree, so the build system guesser can't do anything useful and I end up with a default package.
My workaround is to
(alice)[11:27:30]spack>>spack create -t go https://github.com/jesseduffield/lazygit/archive/v0.11.3.tar.gz
==> This looks like a URL for lazygit
==> Found 10 versions of lazygit:
0.11.3 https://github.com/jesseduffield/lazygit/releases/download/v0.11.3/lazygit_0.11.3_freebsd_x86_64.tar.gz
0.11.2 https://github.com/jesseduffield/lazygit/releases/download/v0.11.2/lazygit_0.11.2_freebsd_x86_64.tar.gz
0.11.1 https://github.com/jesseduffield/lazygit/releases/download/v0.11.1/lazygit_0.11.1_freebsd_x86_64.tar.gz
0.10.6 https://github.com/jesseduffield/lazygit/releases/download/v0.10.6/lazygit_0.10.6_freebsd_x86_64.tar.gz
0.10.5 https://github.com/jesseduffield/lazygit/releases/download/v0.10.5/lazygit_0.10.5_freebsd_x86_64.tar.gz
0.10.4 https://github.com/jesseduffield/lazygit/releases/download/v0.10.4/lazygit_0.10.4_freebsd_x86_64.tar.gz
0.10.3 https://github.com/jesseduffield/lazygit/releases/download/v0.10.3/lazygit_0.10.3_freebsd_x86_64.tar.gz
0.10.2 https://github.com/jesseduffield/lazygit/releases/download/v0.10.2/lazygit_0.10.2_freebsd_x86_64.tar.gz
0.10.1 https://github.com/jesseduffield/lazygit/archive/v0.10.1.tar.gz
0.10 https://github.com/jesseduffield/lazygit/archive/v0.10.tar.gz
==> How many would you like to checksum? (default is 1, q to abort) 1
==> Downloading...
==> Fetching https://github.com/jesseduffield/lazygit/releases/download/v0.11.3/lazygit_0.11.3_freebsd_x86_64.tar.gz
######################################################################## 100.0%
==> Checksummed 1 version of lazygit:
==> Using specified package template: 'go'
==> Created template for lazygit package
==> Created package file: /Users/hartzell/tmp/spack/var/spack/repos/builtin/packages/lazygit/package.py
(alice)[11:28:00]spack>>
which results in the package.py I expect albeit with an inappropriate sha256 checksum for the tarball (because it calculated the signature on a different file than it's fetching and unpacking).
Then, either by hand or after the build fails, I update the checksum.
(alice)[11:28:42]spack>>spack install lazygit
==> Warning: [email protected] cannot build optimized binaries for "haswell". Using best target possible: "x86_64"
==> libsigsegv is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/libsigsegv-2.12-5rgsrc5khc7wfva6oqq6dnchzcy23c44
==> m4 is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/m4-1.4.18-5qz6n7qmauidmhx6qfm7yqjof6n5pxek
==> pkgconf is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/pkgconf-1.6.3-emiquv3wq32427k4htpjastyyxcibey2
==> ncurses is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/ncurses-6.1-3zvai2464mpmryqtefjdkzn3udofiwux
==> readline is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/readline-8.0-hwrhaf6j4ixq5tf5m5tixatg3llazzpo
==> gdbm is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/gdbm-1.18.1-atkqjart2dfjnori2tgjznimitqqphwu
==> perl is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/perl-5.30.0-wmlonvwujujfe3wvknnbikfwl4nqf3zx
==> autoconf is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/autoconf-2.69-7zvd6iq5hqj6biefv3etpp2ojxqoora7
==> automake is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/automake-1.16.1-jz53ixq46xbx2creygmha2kew4pkpctl
==> zlib is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/zlib-1.2.11-caosjr5fcruvzyocttnqqkpackpwyzpz
==> curl is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/curl-7.63.0-wewrogc25man2k5nu2klgjk6w3qkkyue
==> expat is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/expat-2.2.9-ou2ln2zskxs2jlevzifqijhbj4gz64t5
==> libiconv is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/libiconv-1.16-3cgb7j3myogfqhkfiwubqfoqea5g6j6i
==> diffutils is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/diffutils-3.7-btjr4gj77c3eiyzfirrxiw5mgmr3hm6l
==> bzip2 is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/bzip2-1.0.8-rlvwpbm5a5faacq255eko6kz5kdbcl5h
==> xz is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/xz-5.2.4-n6uufvkzz3jrhiyjylh3ggu3vujnlbto
==> libxml2 is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/libxml2-2.9.9-bh3u7tmbkhzgveodvcrztnkxbmhfw2nr
==> tar is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/tar-1.32-eexlhwrs7ipvnxzvplytgs2aytoxknzv
==> gettext is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/gettext-0.20.1-r6ksjrlxvr2xuezholiysmpcguhpuaxm
==> libtool is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/libtool-2.4.6-75s5xqeyblajfff44vqgkcz73nogfsx4
==> openssl is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/openssl-1.1.1d-zdnx7cdmqrclhe7qvozjsgvyrt24nldd
==> pcre is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/pcre-8.42-oxcmvw6kmk4n5k7dc6jkcj2gkamoe2g3
==> git is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/git-2.21.0-44ozdyhctqq7viiudvmzvxlnm6ryaisd
==> go-bootstrap is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/go-bootstrap-1.4-bootstrap-20171003-t6yornwalmj7lw4ysglchwg7x2lkhcuw
==> go is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/go-1.13.4-w3obiflkwgwd5usciec7qyujpm4556ez
==> Installing lazygit
==> Searching for binary cache of lazygit
==> Warning: No Spack mirrors are currently configured
==> No binary for lazygit found: installing from source
==> Warning: microarchitecture specific optimizations are not supported yet on mixed compiler toolchains [check [email protected] for further details]
==> Fetching https://github.com/jesseduffield/lazygit/archive/v0.11.3.tar.gz
######################################################################## 100.0%
==> Error: ChecksumError: sha256 checksum failed for /var/folders/f7/fcb363190nz0jbypndp3nj_m0000gn/T/hartzell/spack-stage/spack-stage-lazygit-0.11.3-nsecxd6wynzqxlx4m7kcvmqeqj2bjku5/v0.11.3.tar.gz
Expected 5580ece2191fb9a52555e7f5dba38e62451749697889e440002705ecb015c9ec but got b3c503de6b34fd4284fd70655e7f42feafc07f090e7f7cc00db261f56c583c46
/Users/hartzell/tmp/spack/lib/spack/spack/package.py:1083, in do_fetch:
1080 self.stage.fetch(mirror_only)
1081 self._fetch_time = time.time() - start_time
1082
>> 1083 if checksum and self.version in self.versions:
1084 self.stage.check()
1085
1086 self.stage.cache_local()
(alice)[11:30:42]spack>>
I'm wondering if there's a better way around this. Perhaps the fix, if it's useful enough, is an option to spack create that avoids the fancy guesswork and just tells it to do what it's told.
Steps to reproduce the issue
(alice)[10:44:03]spack>>spack -d create https://github.com/jesseduffield/lazygit/archive/v0.11.3.tar.gz
==> [2019-11-30-10:44:26.091159] Imported create from built-in commands
==> [2019-11-30-10:44:26.092401] Imported create from built-in commands
==> [2019-11-30-10:44:26.094375] Reading config file /Users/hartzell/tmp/spack/etc/spack/defaults/modules.yaml
==> [2019-11-30-10:44:26.104312] Reading config file /Users/hartzell/tmp/spack/etc/spack/defaults/darwin/modules.yaml
==> [2019-11-30-10:44:26.107297] Reading config file /Users/hartzell/tmp/spack/etc/spack/modules.yaml
==> [2019-11-30-10:44:26.113038] Reading config file /Users/hartzell/.spack/darwin/modules.yaml
==> [2019-11-30-10:44:26.118919] Reading config file /Users/hartzell/tmp/spack/etc/spack/defaults/config.yaml
==> [2019-11-30-10:44:26.153879] This looks like a URL for lazygit
==> [2019-11-30-10:44:26.419630] HTTPError: HTTP Error 404: Not Found
==> [2019-11-30-10:44:26.772969] HTTPError: HTTP Error 404: Not Found
==> [2019-11-30-10:44:29.634677] Found 10 versions of lazygit:
0.11.3 https://github.com/jesseduffield/lazygit/releases/download/v0.11.3/lazygit_0.11.3_freebsd_x86_64.tar.gz
0.11.2 https://github.com/jesseduffield/lazygit/releases/download/v0.11.2/lazygit_0.11.2_freebsd_x86_64.tar.gz
0.11.1 https://github.com/jesseduffield/lazygit/releases/download/v0.11.1/lazygit_0.11.1_freebsd_x86_64.tar.gz
0.10.6 https://github.com/jesseduffield/lazygit/releases/download/v0.10.6/lazygit_0.10.6_freebsd_x86_64.tar.gz
0.10.5 https://github.com/jesseduffield/lazygit/releases/download/v0.10.5/lazygit_0.10.5_freebsd_x86_64.tar.gz
0.10.4 https://github.com/jesseduffield/lazygit/releases/download/v0.10.4/lazygit_0.10.4_freebsd_x86_64.tar.gz
0.10.3 https://github.com/jesseduffield/lazygit/releases/download/v0.10.3/lazygit_0.10.3_freebsd_x86_64.tar.gz
0.10.2 https://github.com/jesseduffield/lazygit/releases/download/v0.10.2/lazygit_0.10.2_freebsd_x86_64.tar.gz
0.10.1 https://github.com/jesseduffield/lazygit/archive/v0.10.1.tar.gz
0.10 https://github.com/jesseduffield/lazygit/archive/v0.10.tar.gz
==> [2019-11-30-10:44:29.634843] How many would you like to checksum? (default is 1, q to abort) q
==> [2019-11-30-10:45:50.387245] Error: Aborted.and notice that it offers "releases/download" links in preference to the link you provided.