spack: Allow smooth transition period between v5 and v6 db#25890
spack: Allow smooth transition period between v5 and v6 db#25890bernhardkaindl wants to merge 1 commit intospack:developfrom
Conversation
tgamblin
requested review from
becker33 and
tgamblin
and removed request for
becker33
bernhardkaindl
force-pushed
the
bugfix/old-env-hash-reading+v6-db-reindex+keep-misc_cache
branch
from
39e0fde to
fc617b8
Compare
|
@bernhardkaindl thanks for this -- we will take a look at this after #25879. FYI - if you are interested in joining us on slack (for faster conversations), feel free to visit slack.spack.io and get an invite. |
|
What should be the use case for this addition? I see that the DB is renamed This is what I have on my console: % spack find -l
==> 2 installed packages
-- darwin-bigsur-cannonlake / [email protected] ----------------
b7u5ruz [email protected] bfcmlpt [email protected]
% ls opt/spack/.spack-db/
failures index-v6.json index.json index_verifier lock prefix_failures prefix_lock
% git checkout PR/25890
Updating files: 100% (7018/7018), done.
Previous HEAD position was f1fe03cd59 Update CHANGELOG and release version for v0.16.2
% spack find -l
==> 9 installed packages
-- darwin-bigsur-cannonlake / [email protected] ----------------
cy6gjyt [email protected] 2m27luj [email protected] ztpyco3 [email protected] s57fgdq [email protected] x7rc3oi [email protected] dhfgkbo [email protected] jfhyqoz [email protected] f4li6tu [email protected] 33zjpam [email protected]
% ls opt/spack/.spack-db/
failures index-v6.json index_verifier lock prefix_failures prefix_lock
% ls opt/spack/darwin-bigsur-cannonlake/apple-clang-12.0.5
berkeley-db-18.1.40-cy6gjytxihzvd63qnvao4isiktzy6vlv libiconv-1.16-x7rc3oii3yxr2s2zsric3fb6sx5ak5e7 zlib-1.2.11-33zjpamilye5gsxptpikgb4lnp3yxdqm
bzip2-1.0.8-2m27luj2emlfif5jomuo64ng7z5ctycz ncurses-6.2-dhfgkbonja2wyf2f4cq6wkyezniwjnpz zlib-1.2.11-bfcmlptcn3m2keubyv7hwcppok47afs3
diffutils-3.7-ztpyco3jfm2lwsyeyxdty37omnjcxj6n pkgconf-1.8.0-jfhyqozj5bx37kf4vnaquj3srdcnyfdb zlib-1.2.8-b7u5ruzhhogxeixy6my7psmsuttziuoh
gdbm-1.19-s57fgdqmgxdaydpxlz6hen5m7vqfz5pr readline-8.1-f4li6tugr6gkpwutetvyhpxqnddt47lkWhat puzzles me is that up to this PR there was a 1:1 mapping from the store to the DB. Here the store is a superset of the DB. |
For a smoother db-v6 transiation period, when it might be needed checkout older revisions to migrate older code allow for better co-existence of v5 and v6 database and misc_cache: Use new filename index-v6.json for new version 6 database, to improve the user experience when switching back to an older spack code: Don't trigger the message to reindex the db which does not work as the reindex command itself runs into this error. Instead, removal of the old db file causes the old spack to do a reindex itself automatically as needed. What's also needed to make this work is to not overwrite the v5 misc_cache files, otherwise "'str' object has no attribute 'get'" is triggered in the provider_index. Put the v6 misc_cache into it's own subdirectory. With this it works to switch back and forth between git revsions before v5 and those after the intro of v6. The only caveat is: The v5 reindex will not find packages built by new spack, but it is possible to work on the older spack versions and build packages.
bernhardkaindl
force-pushed
the
bugfix/old-env-hash-reading+v6-db-reindex+keep-misc_cache
branch
from
fc617b8 to
72f4182
Compare
Cool, thanks! This made it clear to me that there seems to be (in my understanding) another feature which might currently not work as I expected: At least from reading the pull request messages around #22845, it's commit message and the comments in it, I understood that an intent of it was to have a "read-only" compatibility with the old So I guess, the implementation should have been that the new code should read But it seems a reindex without the presence of the old specs.json apparently misses to properly add the old installed specs which only have |
|
I think my pull request is obsolete now, closing it. |
|
As this just came up in Slack's general channel: Without this commit, one needs to run this after/when moving back to old, pre v5 commits: But, because v6 spack also switched the installed spec data format from yaml to json, the following automatic reindex of the v5 code/db will only pick up old specs which have been installed using v5 spack with yaml spec data. |
3 participants
Update - see below for a full quite - Massimiliano wrote:
My answer:
Cool, thanks! This made it clear to me that there seems to be (in my understanding) another feature which might currently not work as I expected:
At least from reading the pull request messages around #22845, it's commit message and the comments in it, I understood that an intent of it was to have a "read-only" compatibility with the old
spec.yamlfiles.So I guess, the implementation should have been that the new code should read
spec.yamlas well asspec.jsonformats from the .spack subdirectory of installed packages in{install_tree:root(default: $spack/opt/spack)}/projections:all (default: "${ARCHITECTURE}/${COMPILERNAME}-${COMPILERVER}/${PACKAGE}-${VERSION}-${HASH}")/.spack/But it seems a reindex without the presence of the old specs.json apparently misses to properly add the old installed specs which only have
spec.yamlfiles to the new database (independent of how the db file is named).The Story wihch is implemented so far:
Sometimes, it might be needed to (temporarily) checkout older spack revisions. It would be nice to allow switching between of v5 and v6 database code (including misc_cache):
Database handling
In case a developer checks out an old branch or commit, this error is shown by spack:
As @haampie reported in #25872,
spack reindexitself also throws the same error:The only way to reindex the db in this case is to remove the database file and let spack recreate it:
This fast without noticeable delay (at least with a local SSD as storage).
This PR takes advantage of this by:
opt/spack/.spack-db/iindex-v6.jsonfor the version 6 database.index.json file, so in case a v5 spack is used gain, it will automatically reindex.Coexistence of misc_cache versions
What's also needed to make this work is to not overwrite the v5 misc_cache files, otherwise
"'str' object has no attribute 'get'"is triggered in the provider_index. Put the v6 misc_cache into it's own subdirectory.Summary
These changes allow to switch back and forth between git revisions using v5 databasse and misc_cache and those after the intro of v6. The only caveat is:
The v5 reindex code will not find packages built by new spack, but it is possible to work on the older spack versions and build packages:
The apparent cause for this might be that currently, spack reindex without the presence of the old
index.jsonapparently misses to include the old installed specs which only havespec.yamlfiles into the new database (independent of how the db file is named).