Concretization Cache: Refactor + manifest testing#49589
Concretization Cache: Refactor + manifest testing#49589tgamblin merged 35 commits intospack:developfrom
Conversation
|
@spackbot fix style |
|
Let me see if I can fix that for you! |
|
I was able to run spack style --fix==> Running style checks on spack
selected: import, isort, black, flake8, mypy
==> Modified files
lib/spack/llnl/util/filesystem.py
lib/spack/spack/solver/asp.py
lib/spack/spack/test/concretization/core.py
lib/spack/spack/test/conftest.py
==> Running import checks
import checks were clean
==> Running isort checks
isort checks were clean
==> Running black checks
reformatted lib/spack/spack/test/concretization/core.py
reformatted lib/spack/spack/solver/asp.py
All done! ✨ 🍰 ✨
2 files reformatted, 2 files left unchanged.
black checks were clean
==> Running flake8 checks
lib/spack/spack/test/concretization/core.py:5: [F401] 'io' imported but unused
flake8 found errors
==> Running mypy checks
Success: no issues found in 640 source files
mypy checks were clean
I've updated the branch with style fixes. |
|
While we are fixing this, can we also add a default for: in $ spack config blame configAlso, was it discussed whether this belongs to |
It was, but there had since been disagreement on that. I think config makes sense as that is where we configure the other misc caches, but I don't have an extremely strong opinion being that. |
I think it would be nice if it were in |
I'll take this as consensus. |
|
@spackbot fix style |
|
Let me see if I can fix that for you! |
|
I was able to run spack style --fix==> Running style checks on spack
selected: import, isort, black, flake8, mypy
==> Modified files
lib/spack/llnl/util/filesystem.py
lib/spack/spack/solver/asp.py
lib/spack/spack/test/concretization/core.py
lib/spack/spack/test/conftest.py
==> Running import checks
import checks were clean
==> Running isort checks
isort checks were clean
==> Running black checks
All done! ✨ 🍰 ✨
4 files left unchanged.
black checks were clean
==> Running flake8 checks
flake8 checks were clean
==> Running mypy checks
Success: no issues found in 640 source files
mypy checks were clean
==> spack style checks were clean
I wasn't able to make any further changes, but please see the message above for remaining issues you can fix locally! |
2e7ca64 to
f0ac02b
Compare
9a8dbae to
18a247f
Compare
|
@haampie @alalazo @tgamblin @kwryankrattiger PTAL at your convenience |
We have actual stable asp output now, no need for the forced determinism hack
…pressed entries Previous Versions of the concretization cache were uncompressed. We now compress them to save space. Explicitly test that the new concretization cache can handle both compressed and uncompressed entries.
a46c4a0 to
a9da7a2
Compare
…50851) This reverts commit f17c7be. Reverts #49589 There are, unfortunately, races in the cleanup code. We are seeing issues like this in CI: ``` for bucket in self.cache_buckets(): with self._fc.read_transaction(bucket) as f: if not f: > raise RuntimeError( "Attempting to clean non existent cache bucket" f" {bucket.name}" ) E RuntimeError: Attempting to clean non existent cache bucket nc lib/spack/spack/solver/asp.py:660: RuntimeError ``` and, more frequently: ``` self = PosixPath('/Users/runner/.local/state/spack/tv6pwnm/misc-cache/concretization/hy/hyrtccditoupfm3s64poficxh5cisl6d') def stat(self, *, follow_symlinks=True): """ Return the result of the stat() system call on this path, like os.stat() does. """ > return os.stat(self, follow_symlinks=follow_symlinks) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ E FileNotFoundError: [Errno 2] No such file or directory: '/Users/runner/.local/state/spack/tv6pwnm/misc-cache/concretization/hy/hyrtccditoupfm3s64poficxh5cisl6d' ``` The second issue appears to be that this sort: ``` rm_reg.sort(key=lambda x: x.stat().st_mtime, reverse=True) ``` is calling stat() redundantly on entries while not holding any read lock on them. I suspect the first is also a race on bucket creation and deletion. We can work this out later and merge for 1.1.
#48198 Introduces the concretization cache, and associated manifest. However it only added testing for the cache itself, not the manifest or cleanup procedures.
This PR:
FileCacheclass was refactored into an abstract baseCacheclass which provides the general cache read/write transaction interface, and aFileCacheandDirectoryCachechild classes,FileCachewhich should be functionally identical to theFileCacheclass pre refactor, deals with caches at the individual file level, andDirectoryCache, which is mean to provide a directory level or "bucket" caching level.