Skip to content

Concretization Cache: Refactor + manifest testing#49589

Merged
tgamblin merged 35 commits intospack:developfrom
johnwparent:concretization-cache/cleanup-and-test
Jun 6, 2025
Merged

Concretization Cache: Refactor + manifest testing#49589
tgamblin merged 35 commits intospack:developfrom
johnwparent:concretization-cache/cleanup-and-test

Conversation

@johnwparent
Copy link
Copy Markdown
Contributor

@johnwparent johnwparent commented Mar 19, 2025

#48198 Introduces the concretization cache, and associated manifest. However it only added testing for the cache itself, not the manifest or cleanup procedures.

This PR:

  • Adds tests for the manifest and cleanup
  • Refactors the manifest to update correctly
  • Refactors the cache to store base32 hashes like the rest of Spack's hashes
  • gzips the cache entries for a smaller memory footprint

FileCache class was refactored into an abstract base Cache class which provides the general cache read/write transaction interface, and a FileCache and DirectoryCache child classes, FileCache which should be functionally identical to the FileCache class pre refactor, deals with caches at the individual file level, and DirectoryCache, which is mean to provide a directory level or "bucket" caching level.

@spackbot-app spackbot-app bot added core PR affects Spack core functionality tests General test capability(ies) utilities labels Mar 19, 2025
@johnwparent
Copy link
Copy Markdown
Contributor Author

@spackbot fix style

@spackbot-app
Copy link
Copy Markdown

spackbot-app bot commented Mar 19, 2025

Let me see if I can fix that for you!

@spackbot-app
Copy link
Copy Markdown

spackbot-app bot commented Mar 19, 2025

I was able to run spack style --fix for you!

spack style --fix
==> Running style checks on spack
  selected: import, isort, black, flake8, mypy
==> Modified files
  lib/spack/llnl/util/filesystem.py
  lib/spack/spack/solver/asp.py
  lib/spack/spack/test/concretization/core.py
  lib/spack/spack/test/conftest.py
==> Running import checks
  import checks were clean
==> Running isort checks
  isort checks were clean
==> Running black checks
reformatted lib/spack/spack/test/concretization/core.py
reformatted lib/spack/spack/solver/asp.py
All done! ✨ 🍰 ✨
2 files reformatted, 2 files left unchanged.
  black checks were clean
==> Running flake8 checks
lib/spack/spack/test/concretization/core.py:5: [F401] 'io' imported but unused
  flake8 found errors
==> Running mypy checks
Success: no issues found in 640 source files
  mypy checks were clean
Keep in mind that I cannot fix your flake8 or mypy errors, so if you have any you'll need to fix them and update the pull request. If I was able to push to your branch, if you make further changes you will need to pull from your updated branch before pushing again.

I've updated the branch with style fixes.

@alalazo
Copy link
Copy Markdown
Member

alalazo commented Mar 19, 2025

While we are fixing this, can we also add a default for:

            "concretization_cache": {
                "type": "object",
                "properties": {
                    "enable": {"type": "boolean"},
                    "url": {"type": "string"},
                    "entry_limit": {"type": "integer", "minimum": 0},
                    "size_limit": {"type": "integer", "minimum": 0},
                },
            },

in etc/defaults? Currently we have none, and so it's unclear which value is used by doing:

$ spack config blame config

Also, was it discussed whether this belongs to config or concretizer?

@kwryankrattiger
Copy link
Copy Markdown
Contributor

Also, was it discussed whether this belongs to config or concretizer?

It was, but there had since been disagreement on that. I think config makes sense as that is where we configure the other misc caches, but I don't have an extremely strong opinion being that.

@tgamblin
Copy link
Copy Markdown
Member

Also, was it discussed whether this belongs to config or concretizer?

I think it would be nice if it were in concretizer, but you could refer to the misc_cache as part of the default URL.

@johnwparent
Copy link
Copy Markdown
Contributor Author

Also, was it discussed whether this belongs to config or concretizer?

I think it would be nice if it were in concretizer, but you could refer to the misc_cache as part of the default URL.

I'll take this as consensus.

@johnwparent
Copy link
Copy Markdown
Contributor Author

@spackbot fix style

@spackbot-app
Copy link
Copy Markdown

spackbot-app bot commented Mar 21, 2025

Let me see if I can fix that for you!

@spackbot-app
Copy link
Copy Markdown

spackbot-app bot commented Mar 21, 2025

I was able to run spack style --fix for you!

spack style --fix
==> Running style checks on spack
  selected: import, isort, black, flake8, mypy
==> Modified files
  lib/spack/llnl/util/filesystem.py
  lib/spack/spack/solver/asp.py
  lib/spack/spack/test/concretization/core.py
  lib/spack/spack/test/conftest.py
==> Running import checks
  import checks were clean
==> Running isort checks
  isort checks were clean
==> Running black checks
All done! ✨ 🍰 ✨
4 files left unchanged.
  black checks were clean
==> Running flake8 checks
  flake8 checks were clean
==> Running mypy checks
Success: no issues found in 640 source files
  mypy checks were clean
==> spack style checks were clean
Keep in mind that I cannot fix your flake8 or mypy errors, so if you have any you'll need to fix them and update the pull request. If I was able to push to your branch, if you make further changes you will need to pull from your updated branch before pushing again.

I wasn't able to make any further changes, but please see the message above for remaining issues you can fix locally!

@johnwparent johnwparent force-pushed the concretization-cache/cleanup-and-test branch from 2e7ca64 to f0ac02b Compare April 11, 2025 00:25
@johnwparent johnwparent force-pushed the concretization-cache/cleanup-and-test branch from 9a8dbae to 18a247f Compare April 14, 2025 18:29
@johnwparent
Copy link
Copy Markdown
Contributor Author

@haampie @alalazo @tgamblin @kwryankrattiger PTAL at your convenience

@johnwparent
Copy link
Copy Markdown
Contributor Author

@haampie @alalazo another ping

@johnwparent johnwparent force-pushed the concretization-cache/cleanup-and-test branch from a46c4a0 to a9da7a2 Compare June 6, 2025 14:34
@tgamblin tgamblin modified the milestones: v1.1.0, v1.0.0 Jun 6, 2025
@tgamblin tgamblin merged commit f17c7be into spack:develop Jun 6, 2025
35 checks passed
tgamblin added a commit that referenced this pull request Jun 7, 2025
tgamblin added a commit that referenced this pull request Jun 7, 2025
…50851)

This reverts commit f17c7be.

Reverts #49589

There are, unfortunately, races in the cleanup code. We are seeing issues like this in CI:

```
        for bucket in self.cache_buckets():
            with self._fc.read_transaction(bucket) as f:
                if not f:
>                   raise RuntimeError(
                        "Attempting to clean non existent cache bucket" f" {bucket.name}"
                    )
E                   RuntimeError: Attempting to clean non existent cache bucket nc

lib/spack/spack/solver/asp.py:660: RuntimeError
```

and, more frequently:

```
self = PosixPath('/Users/runner/.local/state/spack/tv6pwnm/misc-cache/concretization/hy/hyrtccditoupfm3s64poficxh5cisl6d')

    def stat(self, *, follow_symlinks=True):
        """
        Return the result of the stat() system call on this path, like
        os.stat() does.
        """
>       return os.stat(self, follow_symlinks=follow_symlinks)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E       FileNotFoundError: [Errno 2] No such file or directory: '/Users/runner/.local/state/spack/tv6pwnm/misc-cache/concretization/hy/hyrtccditoupfm3s64poficxh5cisl6d'
```

The second issue appears to be that this sort:

```
        rm_reg.sort(key=lambda x: x.stat().st_mtime, reverse=True)
```

is calling stat() redundantly on entries while not holding any read lock on them. I suspect the first is also a race on bucket creation and deletion.

We can work this out later and merge for 1.1.
@tgamblin tgamblin mentioned this pull request Jun 7, 2025
1 task
@tgamblin tgamblin removed this from the v1.0.0 milestone Jun 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core PR affects Spack core functionality defaults documentation Improvements or additions to documentation tests General test capability(ies) utilities

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants