WIP: Binary installation improvements#18212
Closed
scottwittenburg wants to merge 11 commits intospack:developfrom
Closed
WIP: Binary installation improvements#18212scottwittenburg wants to merge 11 commits intospack:developfrom
scottwittenburg wants to merge 11 commits intospack:developfrom
Conversation
Contributor
Author
Member
|
+1 to this, I falsely assumed this was the default behavior of |
Contributor
Author
|
Superseded by #19209 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The goal of this PR is to support an option to
spack install, something shorter than--require-full-hash-match-on-binary-install, but which might convey that idea. Use of the option would cause spack to compare the full hash of the local spec to install against full hashes stored in thespec.yamlfiles on any remote mirrors until it found a match, and only install from binary if it finds one with a matching full hash, otherwise, it would install from source. When the option is not provided, the current behavior would remain, i.e. spack will install the binary version of the first remote spec it finds with a matching DAG hash.This new behavior will support PR build pipelines in several ways:
The goal of the pipeline is to compare the full hash of a local spec against that of a built spec on a remote mirror, and rebuild/push the updated package to the remote mirror. When running a pipeline for a PR, we want to configure a temporary "PR mirror" for each PR so that untrusted binaries could be re-used by subsequent runs of the pipeline when possible. This would improve developer experience, for example, when the first push on a PR branch requires a rebuild of a long job, but subsequent pushes do not. With the feature added by this PR, subsequent pipelines on the same PR could potentially re-use binaries created in earlier pipelines. The current two-pass install approach required in the pipeline prevents this possibility.
This PR would get rid of the need for the pipeline to do package installation in two passes. Due to how spack will install any binary it finds with a matching DAG hash, the pipeline code currently has to do a lot of gymnastics. First it has to manually determine whether the package needs to be rebuilt by comparing the locally computed full hash of the spec against the full hash stored with the binary on the remote mirror. If a rebuild is required due to a full hash mismatch, pipeline code has to coerce spack to rebuild the package from source using the "two-pass install" workaround: first we install only our dependencies with
--cache-onlyoption, and then install only the package with the--no-cacheoption. Ifspack installcould manage all of this for us, we could remove a lot of code from thespack ci rebuildcommand.