Ignore GPG/PGP signatures/checksums when finding available downloads by adamjstewart · Pull Request #2028 · spack/spack

adamjstewart · 2016-10-15T22:12:57Z

Fixes #282.

Previously, Spack was picking up GPG/PGP signatures/checksums when finding available downloads. This affected spack versions:

$ spack versions emacs
==> Safe versions (already checksummed):
  24.5
==> Remote versions (not yet checksummed):
  25.1.tar.gz.sig  24.4.tar.gz.sig  24.3             24.1.tar.gz.sig  23.4              23.2b.tar.gz.sig  23.1             22.2.tar.gz.sig  22.1
  25.1             24.4             24.2.tar.gz.sig  24.1             23.3b.tar.gz.sig  23.2b             22.3.tar.gz.sig  22.2             21.4a.tar.gz.sig
  24.5.tar.gz.sig  24.3.tar.gz.sig  24.2             23.4.tar.gz.sig  23.3b             23.1.tar.gz.sig   22.3             22.1.tar.gz.sig  21.4a

It affected spack checksum:

$ spack checksum emacs
==> Found 28 versions of emacs
  25.1.tar.gz.sighttp://ftp.gnu.org/gnu/emacs/emacs-25.1.tar.gz.sig
  25.1      http://ftp.gnu.org/gnu/emacs/emacs-25.1.tar.gz
  24.5.tar.gz.sighttp://ftp.gnu.org/gnu/emacs/emacs-24.5.tar.gz.sig
  24.5      http://ftp.gnu.org/gnu/emacs/emacs-24.5.tar.gz
  24.4.tar.gz.sighttp://ftp.gnu.org/gnu/emacs/emacs-24.4.tar.gz.sig
  24.4      http://ftp.gnu.org/gnu/emacs/emacs-24.4.tar.gz
  24.3.tar.gz.sighttp://ftp.gnu.org/gnu/emacs/emacs-24.3.tar.gz.sig
  24.3      http://ftp.gnu.org/gnu/emacs/emacs-24.3.tar.gz
  24.2.tar.gz.sighttp://ftp.gnu.org/gnu/emacs/emacs-24.2.tar.gz.sig
  ...
  21.4a     http://ftp.gnu.org/gnu/emacs/emacs-21.4a.tar.gz

And most annoyingly, it affected spack create:

$ spack create http://ftp.gnu.org/gnu/emacs/emacs-25.1.tar.gz
==> This looks like a URL for emacs version 25.1
==> Creating template for package emacs
==> Found 28 versions of emacs:
  25.1.tar.gz.sighttp://ftp.gnu.org/gnu/emacs/emacs-25.1.tar.gz.sig
  25.1      http://ftp.gnu.org/gnu/emacs/emacs-25.1.tar.gz
  24.5.tar.gz.sighttp://ftp.gnu.org/gnu/emacs/emacs-24.5.tar.gz.sig
  24.5      http://ftp.gnu.org/gnu/emacs/emacs-24.5.tar.gz
  24.4.tar.gz.sighttp://ftp.gnu.org/gnu/emacs/emacs-24.4.tar.gz.sig
  24.4      http://ftp.gnu.org/gnu/emacs/emacs-24.4.tar.gz
  24.3.tar.gz.sighttp://ftp.gnu.org/gnu/emacs/emacs-24.3.tar.gz.sig
  24.3      http://ftp.gnu.org/gnu/emacs/emacs-24.3.tar.gz
  24.2.tar.gz.sighttp://ftp.gnu.org/gnu/emacs/emacs-24.2.tar.gz.sig
  ...
  21.4a     http://ftp.gnu.org/gnu/emacs/emacs-21.4a.tar.gz

Include how many checksums in the package file? (default is 5, q to abort) 2
==> Downloading...
==> Trying to fetch from http://ftp.gnu.org/gnu/emacs/emacs-25.1.tar.gz.sig
######################################################################## 100.0%
tar: Unrecognized archive format
tar: Error exit delayed from previous errors.
==> Trying to fetch from http://ftp.gnu.org/gnu/emacs/emacs-25.1.tar.gz
######################################################################## 100.0%

This would result in a package.py that looks like:

class Emacs(Package):                                                           
    ...                                                                         
    version('25.1.tar.gz.sig', '8d459795bcf4754c279dc83f0a930c0a')              
    version('25.1'           , '95c12e6a9afdf0dcbdd7d2efa26ca42c')              

    # FIXME: Add dependencies if required.                                      
    # depends_on('foo')                                                         

    def install(self, spec, prefix):                                            
        # FIXME: Unknown build system                                           
        make()                                                                  
        make('install')

The following keys were erroneously being picked up:

.asc
.back
.md5
.md5.txt
.mds
.minisig
.old
.sha1
.sha1.asc
.sha1.txt
.sha256
.sha256.asc
.sha256.txt
.sha256sum
.sha512
.sig
.sign
.smime
.torrent

Most would show up as 1.2.3.tar.gz.sig, although keys like .md5 and .sha256 would show up as versions 5 and 256. By my count, this bug affected 155 out of 998 packages (15%).

This PR adds a $ anchor to the end of the url regex. Unfortunately, SourceForge downloads end in 1.2.3.tar.gz/download, so I had to allow that as well.

I tested these changes by diffing the output of the following script before and after the change:

#!/usr/bin/env bash                                                             

output=versions-download-dollar.txt                                             
rm -f $output                                                                   

# Without this, Ctrl+C will kill Spack, not the script,                         
# meaning that the script continues running forever.                            
trap "exit" SIGINT                                                              

for package in $(spack list)                                                    
do                                                                              
    echo $package | tee -a $output                                              
    spack versions $package 2>&1 | tee -a $output                               
done

The result is that all 155 packages have now been corrected. I only found 2 cases where the changes negatively improved version finding abilities (paraview and qhull both found 1 fewer version). Aside from that, I think it's an overwhelming success!

I tested spack versions on every package, but I only tested spack checksum, spack create, and spack fetch on a couple packages, so feel free to test it out yourself. Let me know if you want me to provide the before and after results from the script above.

adamjstewart · 2016-10-15T22:25:36Z

By the way, this is probably something we should add a test for in the unit tests. I would create a directory with files like:

foo-1.2.3.tar.gz
foo-1.2.3.tar.gz.sig
foo-1.2.3.tar.gz.sha256

and make sure that only one version is found. Unfortunately, I don't really know how to write unit tests yet, but if you give me tips I can give it a shot.

…pack#2028) * Ignore GPG/PGP signatures/checksums when finding available downloads * Remove duplicated basename

Ignore GPG/PGP signatures/checksums when finding available downloads

148f401

alalazo approved these changes Oct 16, 2016

View reviewed changes

Remove duplicated basename

770e9b7

tgamblin merged commit d891143 into spack:develop Oct 16, 2016

paulhopkins pushed a commit to paulhopkins/spack that referenced this pull request Oct 24, 2016

Ignore GPG/PGP signatures/checksums when finding available downloads (s…

f515dfa

…pack#2028) * Ignore GPG/PGP signatures/checksums when finding available downloads * Remove duplicated basename

adamjstewart deleted the fixes/url-sig branch October 29, 2016 00:39

adamjstewart mentioned this pull request Nov 9, 2016

spack create: gitlab releases #2290

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ignore GPG/PGP signatures/checksums when finding available downloads#2028

Ignore GPG/PGP signatures/checksums when finding available downloads#2028
tgamblin merged 2 commits intospack:developfrom
adamjstewart:fixes/url-sig

adamjstewart commented Oct 15, 2016

Uh oh!

adamjstewart commented Oct 15, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

adamjstewart commented Oct 15, 2016

Uh oh!

adamjstewart commented Oct 15, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants