Skip to content

Make downloads from list_url deterministic#10047

Merged
alalazo merged 1 commit intospack:developfrom
adamjstewart:fixes/deterministic-list-url
Dec 10, 2018
Merged

Make downloads from list_url deterministic#10047
alalazo merged 1 commit intospack:developfrom
adamjstewart:fixes/deterministic-list-url

Conversation

@adamjstewart
Copy link
Copy Markdown
Member

This PR addresses a bug discovered in #10025. For cdo, the downloads page contains two types of links, one with /download in the path that downloads the tarball, and one without that brings you to a page with the same link. Currently, spack.util.web.find_versions_of_archive behaves non-deterministically, and depending on which version of Python you use, you may get very different results:

$ python3 $(which spack) checksum cdo
==> Found 10 versions of cdo:
  
  1.9.6rc4  https://code.mpimet.mpg.de/attachments/18911/cdo-1.9.6rc4.tar.gz
  1.9.5     https://code.mpimet.mpg.de/attachments/download/18264/cdo-1.9.5.tar.gz
  1.9.4     https://code.mpimet.mpg.de/attachments/download/17374/cdo-1.9.4.tar.gz
  1.9.3     https://code.mpimet.mpg.de/attachments/download/16435/cdo-1.9.3.tar.gz
  1.9.2     https://code.mpimet.mpg.de/attachments/download/16035/cdo-1.9.2.tar.gz
  1.9.1     https://code.mpimet.mpg.de/attachments/15653/cdo-1.9.1.tar.gz
  1.9.0     https://code.mpimet.mpg.de/attachments/15187/cdo-1.9.0.tar.gz
  1.8.2     https://code.mpimet.mpg.de/attachments/download/14686/cdo-1.8.2.tar.gz
  1.7.2     https://code.mpimet.mpg.de/attachments/download/12760/cdo-1.7.2.tar.gz
  1.6.4     https://code.mpimet.mpg.de/attachments/8376/cdo-1.6.4.tar.gz

==> How many would you like to checksum? (default is 1, q to abort) q
==> Error: Aborted.
$ python2 $(which spack) checksum cdo
==> Found 10 versions of cdo:
  
  1.9.6rc4  https://code.mpimet.mpg.de/attachments/18911/cdo-1.9.6rc4.tar.gz
  1.9.5     https://code.mpimet.mpg.de/attachments/18264/cdo-1.9.5.tar.gz
  1.9.4     https://code.mpimet.mpg.de/attachments/download/17374/cdo-1.9.4.tar.gz
  1.9.3     https://code.mpimet.mpg.de/attachments/download/16435/cdo-1.9.3.tar.gz
  1.9.2     https://code.mpimet.mpg.de/attachments/16035/cdo-1.9.2.tar.gz
  1.9.1     https://code.mpimet.mpg.de/attachments/download/15653/cdo-1.9.1.tar.gz
  1.9.0     https://code.mpimet.mpg.de/attachments/15187/cdo-1.9.0.tar.gz
  1.8.2     https://code.mpimet.mpg.de/attachments/14686/cdo-1.8.2.tar.gz
  1.7.2     https://code.mpimet.mpg.de/attachments/12760/cdo-1.7.2.tar.gz
  1.6.4     https://code.mpimet.mpg.de/attachments/8376/cdo-1.6.4.tar.gz

==> How many would you like to checksum? (default is 1, q to abort) q
==> Error: Aborted.

With this PR, Spack sorts the URLs before searching, meaning that the behavior is at least deterministic and the same across all versions of Python:

$ python3 $(which spack) checksum cdo
==> Found 10 versions of cdo:
  
  1.9.6rc4  https://code.mpimet.mpg.de/attachments/download/18911/cdo-1.9.6rc4.tar.gz
  1.9.5     https://code.mpimet.mpg.de/attachments/download/18264/cdo-1.9.5.tar.gz
  1.9.4     https://code.mpimet.mpg.de/attachments/download/17374/cdo-1.9.4.tar.gz
  1.9.3     https://code.mpimet.mpg.de/attachments/download/16435/cdo-1.9.3.tar.gz
  1.9.2     https://code.mpimet.mpg.de/attachments/download/16035/cdo-1.9.2.tar.gz
  1.9.1     https://code.mpimet.mpg.de/attachments/download/15653/cdo-1.9.1.tar.gz
  1.9.0     https://code.mpimet.mpg.de/attachments/download/15187/cdo-1.9.0.tar.gz
  1.8.2     https://code.mpimet.mpg.de/attachments/download/14686/cdo-1.8.2.tar.gz
  1.7.2     https://code.mpimet.mpg.de/attachments/download/12760/cdo-1.7.2.tar.gz
  1.6.4     https://code.mpimet.mpg.de/attachments/download/8376/cdo-1.6.4.tar.gz

==> How many would you like to checksum? (default is 1, q to abort) q
==> Error: Aborted.
$ python2 $(which spack) checksum cdo
==> Found 10 versions of cdo:
  
  1.9.6rc4  https://code.mpimet.mpg.de/attachments/download/18911/cdo-1.9.6rc4.tar.gz
  1.9.5     https://code.mpimet.mpg.de/attachments/download/18264/cdo-1.9.5.tar.gz
  1.9.4     https://code.mpimet.mpg.de/attachments/download/17374/cdo-1.9.4.tar.gz
  1.9.3     https://code.mpimet.mpg.de/attachments/download/16435/cdo-1.9.3.tar.gz
  1.9.2     https://code.mpimet.mpg.de/attachments/download/16035/cdo-1.9.2.tar.gz
  1.9.1     https://code.mpimet.mpg.de/attachments/download/15653/cdo-1.9.1.tar.gz
  1.9.0     https://code.mpimet.mpg.de/attachments/download/15187/cdo-1.9.0.tar.gz
  1.8.2     https://code.mpimet.mpg.de/attachments/download/14686/cdo-1.8.2.tar.gz
  1.7.2     https://code.mpimet.mpg.de/attachments/download/12760/cdo-1.7.2.tar.gz
  1.6.4     https://code.mpimet.mpg.de/attachments/download/8376/cdo-1.6.4.tar.gz

==> How many would you like to checksum? (default is 1, q to abort) q
==> Error: Aborted.

@michaelkuhn

@michaelkuhn
Copy link
Copy Markdown
Member

@adamjstewart Thanks for tracking this down! :-)

@alalazo alalazo merged commit ad04640 into spack:develop Dec 10, 2018
@alalazo
Copy link
Copy Markdown
Member

alalazo commented Dec 10, 2018

@tgamblin @scheibelp FYI, this affects core (I merged straight away as it is a one-liner without evident side effects).

@adamjstewart adamjstewart deleted the fixes/deterministic-list-url branch October 20, 2019 01:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants