Proposal: Enforcement of Trusted Downloads #1696
Description
In an effort to ensure safety, Spack:
- Checksums tarball downloads.
- Refuses to install non-checksummed tarballs by default (unless the user specifies
--no-checksums).
Unfortunately, this approach does not meet Spack's security goals. Because Spack will quite happily process any Git download, whether or not the user has specified --no-checksums. Unless a checksum or Git hash is used to verify a download, I call it an untrusted download. This is a real issue because a number of packages in Spack will default to untrusted Git downloads. For example:
$ grep 'branch=' `find . -name '*.py'`
./lib/spack/external/nose/plugins/cover.py: branch=self.coverBranches, data_suffix=conf.worker,
./lib/spack/spack/fetch_strategy.py: hg='https://jay.grs.rwth-aachen.de/hg/lwm2', branch='torus')
./var/spack/repos/builtin/packages/bbcp/package.py: branch="master")
./var/spack/repos/builtin/packages/cbtf/package.py: version('1.6', branch='master',
./var/spack/repos/builtin/packages/cbtf-argonavis/package.py: version('1.6', branch='master',
./var/spack/repos/builtin/packages/cbtf-krell/package.py: version('1.6', branch='master',
./var/spack/repos/builtin/packages/cbtf-lanl/package.py: version('1.6', branch='master',
./var/spack/repos/builtin/packages/cityhash/package.py: version('master', branch='master',
./var/spack/repos/builtin/packages/cleverleaf/package.py: branch='develop')
./var/spack/repos/builtin/packages/cnmem/package.py: version('git', git='https://github.com/NVIDIA/cnmem.git', branch="master")
./var/spack/repos/builtin/packages/flux/package.py: version('master', branch='master',
./var/spack/repos/builtin/packages/hdf5-blosc/package.py: branch='master')
./var/spack/repos/builtin/packages/julia/package.py: git='https://github.com/JuliaLang/julia.git', branch='master')
./var/spack/repos/builtin/packages/julia/package.py: git='https://github.com/JuliaLang/julia.git', branch='release-0.5')
./var/spack/repos/builtin/packages/julia/package.py: git='https://github.com/JuliaLang/julia.git', branch='release-0.4')
./var/spack/repos/builtin/packages/openspeedshop/package.py: version('2.2', branch='master',
./var/spack/repos/builtin/packages/qthreads/package.py: branch="release-1.10")
./var/spack/repos/builtin/packages/r-BiocGenerics/package.py: branch='release-3.3')
./var/spack/repos/builtin/packages/r-BiocGenerics/package.py: branch='release-3.2')
./var/spack/repos/builtin/packages/raja/package.py: version('git', git='https://github.com/LLNL/RAJA.git', branch="master")
./var/spack/repos/builtin/packages/rose/package.py: version('master', branch='master',
Note that this list violates another assumption we've been making: that numeric-versioned packages are always trusted (checksummed), whereas non-numeric versions are not. If you assume that, you will be surprised when downloading version 1.6 of cbtf, for example. Or more seriously, [email protected].
Another problem is that --no-checksums is too coarse. It's the equivalent to turning off a firewall when you really need to just open one port. You usually want to turn off safety for just one package, not all its possible dependencies.
I therefore propose an improved scheme for trusted downloads that will meet Spack's security goals:
- Define a trusted download as one in which Spack could verify what it downloaded was the same as what the author downloaded when the package was created. This means checksums, git hashes, etc. Someone will have to go through the different download methods and figure out how to determine when a download is trusted vs. not trusted. (Do NOT rely on https://. This can guarantee the source of the download, but not its content). This needs to get written up in Python, so Spack can know when a download is trusted.
- By default, Spack will only install trusted downloads.
- The user can override this default by adding something to
packages.yaml, specifying that Spack should allow untrusted downloads on certain packages ONLY. For example, I might use the following while developing a package:
ibmisc:
version: [develop]
verify: no # Allow untrusted downloads
variants: +python +netcdf
Thoughts? Feedback?