database: don’t reindex between versions 5 and 6 by nhanford · Pull Request #25872 · spack/spack

nhanford · 2021-09-09T15:42:47Z

PR #22845 was supposed to allow the Spack database to enter a "hybrid" state, since we maintained read-only backwards compatibility with the old spec format. This may fix #25867 with hash mismatch issues.

haampie · 2021-09-09T15:56:16Z

Hm, the db version is 6 for me already, am I supposed to downgrade spack, run reindex, and then come back to this pr and then try the environment again?

haampie · 2021-09-09T15:59:44Z

$ spack reindex 
==> Error: Expected database version 5 but found version 6.
`spack reindex` may fix this, or you may need a newer Spack version.

🤔

Traceback (most recent call last):
  File "/home/harmen/spack/lib/spack/spack/main.py", line 797, in main
    return _invoke_command(command, parser, args, unknown)
  File "/home/harmen/spack/lib/spack/spack/main.py", line 525, in _invoke_command
    return_val = command(parser, args)
  File "/home/harmen/spack/lib/spack/spack/cmd/reindex.py", line 14, in reindex
    spack.store.store.reindex()
  File "/home/harmen/spack/lib/spack/spack/store.py", line 170, in reindex
    return self.db.reindex(self.layout)
  File "/home/harmen/spack/lib/spack/spack/database.py", line 844, in reindex
    with transaction:
  File "/home/harmen/spack/lib/spack/llnl/util/lock.py", line 685, in __enter__
    self._as = self._acquire_fn()
  File "/home/harmen/spack/lib/spack/spack/database.py", line 834, in _read_suppress_error
    self._read_from_file(self._index_path)
  File "/home/harmen/spack/lib/spack/spack/database.py", line 749, in _read_from_file
    raise InvalidDatabaseVersionError(_db_version, version)

nhanford · 2021-09-09T16:26:53Z

Okay this tiime for sure.
I was able to reproduce and fix this locally with this latest commit of this branch.
spack reindex and then the ensuing commands to test should now work...

haampie · 2021-09-09T18:18:38Z

I don't understand the last commit 😅 since

spack/lib/spack/spack/database.py

Line 70 in 5fddd48

_db_version = Version('6')

maybe it was not rebased on develop?

I should have been more clear, if I checkout the commit before the spec.json PR, and then run reindex, it gives me this error, so I can't reindex a database version 6 to a database version 5 even though the spack.yaml's are mostly there:

$ git checkout edb1d75b1bb7a9981851cf74b29b997a7c9ebcf0
$ spack reindex 
==> Error: Expected database version 5 but found version 6.
`spack reindex` may fix this, or you may need a newer Spack version.

So now that my database version is already at 6, I can't go back and see if this PR would actually solve my issue from #25867?

TL;DR how can I test this PR now that I've already used spack on develop and it has upgraded the database to v6?

nhanford · 2021-09-09T19:40:53Z

I don't understand the last commit 😅 since

spack/lib/spack/spack/database.py

Line 70 in 5fddd48

_db_version = Version('6')

maybe it was not rebased on develop?

It could be something weird on my end. I just rebased on develop again just in case.

how can I test this PR now that I've already used spack on develop and it has upgraded the database to v6?

cd $SPACK_ROOT
git checkout edb1d75b1bb7a9981851cf74b29b997a7c9ebcf0
rm opt/spack/.spack-db/index.json
spack reindex
git checkout fix/nhanford/db-no-reindex-5-6
spack reindex (optional)
spack env create hello

and so on.

bernhardkaindl · 2021-09-09T21:05:12Z

@nhanford wrote:

It could be something weird on my end. I just rebased on develop again just in case.

It's still is weird on your end.
You didn't rebase on develop now, you merged develop into your tree, but this is just a minor difference.

The real misunderstanding is:

Since your two commits on fix/nhanford/db-no-reindex-5-6 were already in develop, your current your fix-branch is now identical to develop:

$ git log -n1 --oneline develop
5fddd48f80 (HEAD -> develop) Refactor unit-tests in test/architecture.py (#25848)
$ git log -n1 --oneline fix/nhanford/db-no-reindex-5-6
24a522417b (fix/nhanford/db-no-reindex-5-6) Merge branch 'develop' into fix/nhanford/db-no-reindex-5-6
$ git diff -s develop fix/nhanford/db-no-reindex-5-6
(no output)

These are exactly the lines which were the two commits in your branch before the merge of develop into your "5-6" branch, because they are also in your merged pull request for PR #22845:

git log -p d83f7110d5 -n1|grep Version..[56]
-_db_version = Version('5')
+_db_version = Version('6')
     (Version('0.9.3'), Version('5')),
+    (Version('5'), Version('6'))

"and so on."

isn't a well-defined and repeatable regression test.

Your merged PR 22845 seems to break spack uninstall also in all of my real-world work trees:

spack uninstall -y --all bzip2
==> Error: '6bacc5jo5vnw4g7bfle2qqsmqgcvjcfg'

haampie · 2021-09-09T22:16:30Z

If I do this:

$ git checkout edb1d75b1bb7a9981851cf74b29b997a7c9ebcf0
$ rm opt/spack/.spack-db/index.json
$ spack reindex
$ git checkout develop
($ git rebase origin/fix/nhanford/db-no-reindex-5-6)
$ tail -n4 opt/spack/.spack-db/index.json 
  },
  "version": "5"
 }
}
$ spack -e hello find
==> Error: 'hul5b3wgfnfor3wndznoksqdphdp3ocs'

I'm still hitting that issue :(

@bernhardkaindl can you share spack -d uninstall --all bzip2 and tell if it's because of dependent_environments?

nhanford · 2021-09-09T22:26:33Z

Thanks @bernhardkaindl and @haampie for the comments. We are narrowing it down right now and this helps.
I am somewhat new to the project and was trying to foresee anything that could break anybody, but I clearly missed this issue with environments and should have had a test for it.
I am going to close this PR as it serves no purpose in the moment, and we'll go back to the original issue, if that's okay.

bernhardkaindl · 2021-09-09T22:27:08Z

$ spack -e hello find
==> Error: 'hul5b3wgfnfor3wndznoksqdphdp3ocs'
I'm still hitting that issue :(
@bernhardkaindl can you share spack -d uninstall --all bzip2 and tell if it's because of dependent_environments?

Yes it is because of dependent_environments!

spack -d uninstall --all bzip2 2>&1|sed "s@$SPACK_ROOT@\$spack@;s/2021.*]/time]/"
==> [time] Imported uninstall from built-in commands
==> [time] Imported uninstall from built-in commands
==> [time] Reading config file $spack/etc/spack/defaults/config.yaml
==> [time] Reading config file $spack/etc/spack/defaults/bootstrap.yaml
==> [time] DATABASE LOCK TIMEOUT: 3s
==> [time] PACKAGE LOCK TIMEOUT: No timeout
==> [time] Reading config file $spack/etc/spack/defaults/repos.yaml
Traceback (most recent call last):
  File "$spack/bin/spack", line 100, in <module>
    sys.exit(spack.main.main())
  File "$spack/lib/spack/spack/main.py", line 797, in main
    return _invoke_command(command, parser, args, unknown)
  File "$spack/lib/spack/spack/main.py", line 525, in _invoke_command
    return_val = command(parser, args)
  File "$spack/lib/spack/spack/cmd/uninstall.py", line 360, in uninstall
    uninstall_specs(args, specs)
  File "$spack/lib/spack/spack/cmd/uninstall.py", line 316, in uninstall_specs
    uninstall_list, remove_list = get_uninstall_list(args, specs, env)
  File "$spack/lib/spack/spack/cmd/uninstall.py", line 252, in get_uninstall_list
    spec_envs = dependent_environments(uninstall_list)
  File "$spack/lib/spack/spack/cmd/uninstall.py", line 163, in dependent_environments
    hashes = set(env.all_hashes())
  File "$spack/lib/spack/spack/environment.py", line 1550, in all_hashes
    return list(set(s.dag_hash() for s in self.all_specs()))
  File "$spack/lib/spack/spack/environment.py", line 1542, in all_specs
    all_specs.update(self.specs_by_hash[h].traverse())
KeyError: '6bacc5jo5vnw4g7bfle2qqsmqgcvjcfg'

I reproduced this Error again in an empty spack dir with an old env where I built only a few packages and then updated to HEAD.

When environments exist, they causes trouble with PR #22845 (new json format):

The other way around, when a new environment was created with today's spack, and then one switched back to yesterday's spack, one gets this for seemingly all spack commands (install, find, ...):

Error: 'str' object has no attribute 'get'

tgamblin · 2021-09-09T22:35:12Z

Thanks @bernhardkaindl! We (only just) found the same thing. I think we've narrowed this down and we are working on a fix.

bernhardkaindl · 2021-09-09T23:06:01Z

Just an update - my quickest reproducer so far:

# cleanup (mostly?) everything I could find
spack env deactivate
spack env list | xargs --no-run-if-empty spack env remove -y
spack gc -y
spack uninstall -ay
git checkout edb1d7
spack clean -a
mv ~/.spack/ ~/.spack.old
git checkout edb1d7
spack find
# must be 0
spack env create oldenv && spack env activate oldenv && spack install bzip2 && spack find

git checkout 5fddd
spack -d find 2>&1|sed "s@$SPACK_ROOT@\$spack@;s/2021.*]/time]/"
==> [time] Imported find from built-in commands
==> [time] Imported find from built-in commands
==> [time] Reading config file $spack/var/spack/environments/oldenv2/spack.yaml
Traceback (most recent call last):
  File "$spack/bin/spack", line 100, in <module>
    sys.exit(spack.main.main())
  File "$spack/lib/spack/spack/main.py", line 797, in main
    return _invoke_command(command, parser, args, unknown)
  File "$spack/lib/spack/spack/main.py", line 525, in _invoke_command
    return_val = command(parser, args)
  File "$spack/lib/spack/spack/cmd/find.py", line 215, in find
    _find(parser, args)
  File "$spack/lib/spack/spack/cmd/find.py", line 220, in _find
    results = args.specs(**q_args)
  File "$spack/lib/spack/spack/cmd/common/arguments.py", line 74, in _specs
    kwargs['hashes'] = set(env.all_hashes())
  File "$spack/lib/spack/spack/environment.py", line 1550, in all_hashes
    return list(set(s.dag_hash() for s in self.all_specs()))
  File "$spack/lib/spack/spack/environment.py", line 1542, in all_specs
    all_specs.update(self.specs_by_hash[h].traverse())
KeyError: 'otca6ubpax7hocgh62j3pw44oxxanvil'

Thanks @bernhardkaindl! We (only just) found the same thing. I think we've narrowed this down and we are working on a fix.

Kudos, all the best!

Add back line to not reindex between DB versions 5 and 6

575e396

tgamblin changed the title ~~Add back line to not reindex between DB versions 5 and 6~~ database: don’t reindex between versions 5 and 6 Sep 9, 2021

Version number was also somehow corrupted...

d490028

Merge branch 'develop' into fix/nhanford/db-no-reindex-5-6

24a5224

nhanford closed this Sep 9, 2021

bernhardkaindl mentioned this pull request Sep 10, 2021

spack: Allow smooth transition period between v5 and v6 db #25890

Closed

haampie deleted the fix/nhanford/db-no-reindex-5-6 branch August 2, 2022 09:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

database: don’t reindex between versions 5 and 6#25872

database: don’t reindex between versions 5 and 6#25872
nhanford wants to merge 3 commits intodevelopfrom
fix/nhanford/db-no-reindex-5-6

nhanford commented Sep 9, 2021

Uh oh!

haampie commented Sep 9, 2021

Uh oh!

haampie commented Sep 9, 2021 •

edited

Loading

Uh oh!

nhanford commented Sep 9, 2021

Uh oh!

haampie commented Sep 9, 2021 •

edited

Loading

Uh oh!

nhanford commented Sep 9, 2021 •

edited

Loading

Uh oh!

bernhardkaindl commented Sep 9, 2021 •

edited

Loading

Uh oh!

haampie commented Sep 9, 2021 •

edited

Loading

Uh oh!

nhanford commented Sep 9, 2021

Uh oh!

bernhardkaindl commented Sep 9, 2021

Uh oh!

tgamblin commented Sep 9, 2021

Uh oh!

bernhardkaindl commented Sep 9, 2021 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

nhanford commented Sep 9, 2021

Uh oh!

haampie commented Sep 9, 2021

Uh oh!

haampie commented Sep 9, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nhanford commented Sep 9, 2021

Uh oh!

haampie commented Sep 9, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nhanford commented Sep 9, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bernhardkaindl commented Sep 9, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

haampie commented Sep 9, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nhanford commented Sep 9, 2021

Uh oh!

bernhardkaindl commented Sep 9, 2021

Uh oh!

tgamblin commented Sep 9, 2021

Uh oh!

bernhardkaindl commented Sep 9, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

haampie commented Sep 9, 2021 •

edited

Loading

haampie commented Sep 9, 2021 •

edited

Loading

nhanford commented Sep 9, 2021 •

edited

Loading

bernhardkaindl commented Sep 9, 2021 •

edited

Loading

haampie commented Sep 9, 2021 •

edited

Loading

bernhardkaindl commented Sep 9, 2021 •

edited

Loading