Skip to content

WIP: Binary installation improvements#18212

Closed
scottwittenburg wants to merge 11 commits intospack:developfrom
scottwittenburg:binary-installation-improvements
Closed

WIP: Binary installation improvements#18212
scottwittenburg wants to merge 11 commits intospack:developfrom
scottwittenburg:binary-installation-improvements

Conversation

@scottwittenburg
Copy link
Copy Markdown
Contributor

The goal of this PR is to support an option to spack install, something shorter than --require-full-hash-match-on-binary-install, but which might convey that idea. Use of the option would cause spack to compare the full hash of the local spec to install against full hashes stored in the spec.yaml files on any remote mirrors until it found a match, and only install from binary if it finds one with a matching full hash, otherwise, it would install from source. When the option is not provided, the current behavior would remain, i.e. spack will install the binary version of the first remote spec it finds with a matching DAG hash.

This new behavior will support PR build pipelines in several ways:

  1. The goal of the pipeline is to compare the full hash of a local spec against that of a built spec on a remote mirror, and rebuild/push the updated package to the remote mirror. When running a pipeline for a PR, we want to configure a temporary "PR mirror" for each PR so that untrusted binaries could be re-used by subsequent runs of the pipeline when possible. This would improve developer experience, for example, when the first push on a PR branch requires a rebuild of a long job, but subsequent pushes do not. With the feature added by this PR, subsequent pipelines on the same PR could potentially re-use binaries created in earlier pipelines. The current two-pass install approach required in the pipeline prevents this possibility.

  2. This PR would get rid of the need for the pipeline to do package installation in two passes. Due to how spack will install any binary it finds with a matching DAG hash, the pipeline code currently has to do a lot of gymnastics. First it has to manually determine whether the package needs to be rebuilt by comparing the locally computed full hash of the spec against the full hash stored with the binary on the remote mirror. If a rebuild is required due to a full hash mismatch, pipeline code has to coerce spack to rebuild the package from source using the "two-pass install" workaround: first we install only our dependencies with --cache-only option, and then install only the package with the --no-cache option. If spack install could manage all of this for us, we could remove a lot of code from the spack ci rebuild command.

@scottwittenburg
Copy link
Copy Markdown
Contributor Author

@tgamblin @becker33 this is super-preliminary, but as we discussed a bit ago, it could be good to discuss some of the more questionable/controversial changes early on, while they're still happening.

@haampie
Copy link
Copy Markdown
Member

haampie commented Aug 21, 2020

+1 to this, I falsely assumed this was the default behavior of spack install.

@scottwittenburg
Copy link
Copy Markdown
Contributor Author

Superseded by #19209

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants