check remote mirrors for hashes provided by the user#22503
check remote mirrors for hashes provided by the user#22503cosmicexplorer wants to merge 2 commits intospack:developfrom
Conversation
cosmicexplorer
force-pushed
the
spec-parsing-includes-remotes
branch
2 times, most recently
from
6265b2b to
3c125ba
Compare
| # type: (str) -> List[Spec] | ||
| return (spack.store.db.get_by_hash(find_hash) or | ||
| # If the hash doesn't exist locally, check the remote. | ||
| spack.binary_distribution.binary_index.find_prefix_hash(find_hash)) |
There was a problem hiding this comment.
@becker33, I think this is what you were saying last week that we cannot do, i.e. check the database and then the binary mirrors sequentially, when we look for matches for a hash prefix. If I understood your argument correctly, you were saying if we search those places for a hash prefix sequentially, we could end up in a situation where we mistakenly return a spec from the install database (because we found there exactly one spec with the queried hash prefix), when we should have raised an ambiguous hash exception (because the binary mirrors or active env contain specs with the same hash prefix, possibly even the spec the user actually meant when they abbreviated the full hash with a prefix).
@cosmicexplorer I came here because I, too, want the spec parser to look somewhere other than just the install database for hashes. And I made the same sequential searching mistake in my PR here. Where you are trying to get spack to include all binary specs in its hash search, my goal is to have it include all specs from the active concrete environment. So I'm looking to create some kind of dictionary to map hashes to specs which will be populated from all the locations where hashes can exist: db, binary mirrors, active environment (if concrete).
cosmicexplorer
force-pushed
the
spec-parsing-includes-remotes
branch
from
3c125ba to
e71ac12
Compare
cosmicexplorer
force-pushed
the
spec-parsing-includes-remotes
branch
from
e71ac12 to
8a010b1
Compare
cosmicexplorer
force-pushed
the
spec-parsing-includes-remotes
branch
from
8a010b1 to
6651954
Compare
|
This was implemented in #35042. Closing this as completed. |
3 participants
Problem
This is a partial solution to #19085. Users would like to be able to use spec hash prefixes on the command line that they've seen from remote mirrors, e.g. from the output of
spack buildcache list -l --allarch. Right now, hash prefixes are converted into a single concreteSpecinstance withinSpecParser, and they only check the localDatabaseinstance for that.This is step 2 of a solution to #19085, after #22500. This is only a partial solution to it because in addition to
spack buildcache list, we would also like to ensure that hash prefixes from the output of e.g.spack speccan be provided to the spack command line. We plan to create a separateDatabaseinstance with its own TTL for that later feature.This requires #21723 to work for some reason.
Solution
find_prefix_hash()toBinaryCacheIndexto trawl through all its known specs and match hash prefixes._lookup_local_or_remote_hash()toSpecParserto check both the local and remote databases for a hash prefix specified on the command line.Result
We can now perform the following set of commands:
HOWEVER, it takes 1 full minute to run the above command, which traverses remote databases for a hash prefix after fetching them. We plan to follow up on this PR with a third one that uses an auxiliary local
Databaseinstance to store specs from buildcaches, which is expected to make matching remote spec prefixes as fast as local spec prefixes.