Skip to content

spack gc: add options for environments and build dependencies#41731

Merged
tgamblin merged 11 commits intodevelopfrom
better-spack-gc
Dec 26, 2023
Merged

spack gc: add options for environments and build dependencies#41731
tgamblin merged 11 commits intodevelopfrom
better-spack-gc

Conversation

@tgamblin
Copy link
Copy Markdown
Member

@tgamblin tgamblin commented Dec 17, 2023

Closes #14695.

This adds a few options to spack gc.

One to give you a little more control over dependencies:

  • -b / --keep-build-dependencies: By default, spack gc considers build dependencies to be "no longer needed" once their dependents are installed. With this option, we'll keep build dependencies of needed installations as well.

And two more to make working with environments easier:

  • -E / --except-any-environment: Garbage collect anything NOT needed by an environment. spack gc -E and spack gc -bE are now easy ways to get rid of everytihng not used by some environment.

  • -e / --except-environment ENV: Instead of considering all environments, garbage collect everything not needed by a specific environment. Note that you can use this with -E to add directory environments to the list of considered envs, e.g.:

    spack gc -E -e /path/to/direnv1 -e /path/to/direnv2 #...
    
  • rework unused_specs() method on DB to add options for roots and deptypes
  • add all_hashes() method on DB
  • rework spack gc command to add 3 more options
  • tests

@tgamblin tgamblin added this to the v0.22.0 milestone Dec 17, 2023
@spackbot-app spackbot-app bot added commands core PR affects Spack core functionality labels Dec 17, 2023
@tgamblin tgamblin requested review from alalazo and haampie December 17, 2023 09:30
@spackbot-app spackbot-app bot added tests General test capability(ies) shell-support labels Dec 18, 2023
haampie
haampie previously approved these changes Dec 22, 2023
@haampie
Copy link
Copy Markdown
Member

haampie commented Dec 22, 2023

I find spack gc -e . / spack -e . gc rather confusing.

Should spack -e . gc delete stuff only found in the active env, while spack gc -e . deletes everything in the database not needed by the roots of the env in .?

Usually env aware commands don't touch anything outside the env, feels wrong if spack -e . gc was an exception to that.

@tgamblin
Copy link
Copy Markdown
Member Author

Should spack -e . gc delete stuff only found in the active env, while spack gc -e . deletes everything in the database not needed by the roots of the env in .?

That's how it works.

  • spack -e . gc will get rid of non-explicitly installed build deps in the active env.
    • everything is first marked as a root, then the env is subtracted then its roots are added back.
  • spack gc -e . gets rid of everything in the DB not needed (via run/link deps) by this env
  • spack gc -be . gets rid of everything in the DB not needed (via run/link/build deps) by this env

@haampie
Copy link
Copy Markdown
Member

haampie commented Dec 22, 2023

There's a bug then, cause spack -e . gc ultimately calls unused_specs on the db with no roots on an empty env selecting all from the db

# consider all explicit specs roots (this is the default for db.unused_specs())
root_hashes = None

specs = spack.store.STORE.db.unused_specs(root_hashes=root_hashes, deptype=deptype)
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This shouldn't happen if active_env

Copy link
Copy Markdown
Member Author

@tgamblin tgamblin Dec 24, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This wasn't the issue -- it was that all_hashes() was returning an empty list due to no read_transaction().

This adds a few options to `spack gc`.

One to give you a little more control over dependencies:

* `-b` / `--keep-build-dependencies`: By default, `spack gc` considers build
  dependencies to be "no longer needed" once their dependents are installed. With this
  option, we'll keep build dependencies of needed installations as well.

And two more to make working with environments easier:

* `-E` / `--except-any-environment`: Garbage collect anything NOT needed by an
  environment. `spack gc -E` and `spack gc -bE` are now easy ways to get rid of
  everytihng not used by some environment.

* `-e` / `--except-environment` `ENV`: Instead of considering all environments, garbage
  collect everything not needed by a *specific* environment. Note that you can use this
  with `-E` to add directory environments to the list of considered envs, e.g.:

      spack gc -E -e /path/to/direnv1 -e /path/to/direnv2 #...
@tgamblin
Copy link
Copy Markdown
Member Author

tgamblin commented Dec 24, 2023

@haampie this is fixed -- and the code's refactored to be a bit easier to read.

The issue was that there was no with self.read_transaction() in db.all_hashes(), self._data.keys() was empty.

I added a test for all_hashes() that ensures it keeps its read transaction and that it's consistent with query(). I also wrapped the whole command in a read transaction so that it doesn't take and release locks a bunch of times.

Copy link
Copy Markdown
Member

@adamjstewart adamjstewart left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this does exactly what I want!

I did encounter the same issue as #3690, where I had to run spack gc several times in order to actually uninstall everything.

> spack gc -Eb
...
==> 382 packages will be uninstalled. Do you want to proceed? [y/N] y
...
==> Error: python@=3.11.6%apple-clang@=15.0.0+bz2+crypt+ctypes+dbm~debug+libxml2+lzma~nis+optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tkinter+uuid+zlib build_system=generic patches=13fa8bf,b0615b2,ebdca64,f2fd060 arch=darwin-sonoma-m2 ^[deptypes=build,link virtuals=uuid] apple-libuuid@=1353.100.2%apple-clang@=15.0.0 build_system=bundle arch=darwin-sonoma-m2 ^[deptypes=build,link] berkeley-db@=18.1.40%apple-clang@=15.0.0+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=darwin-sonoma-m2 ^[deptypes=build,link,run] bzip2@=1.0.8%apple-clang@=15.0.0~debug~pic+shared build_system=generic arch=darwin-sonoma-m2 ^[deptypes=build] ca-certificates-mozilla@=2023-05-30%apple-clang@=15.0.0 build_system=generic arch=darwin-sonoma-m2 ^[deptypes=build] diffutils@=3.9%apple-clang@=15.0.0 build_system=autotools arch=darwin-sonoma-m2 ^[deptypes=build,link] expat@=2.5.0%apple-clang@=15.0.0~libbsd build_system=autotools arch=darwin-sonoma-m2 ^[deptypes=build,link] gdbm@=1.23%apple-clang@=15.0.0 build_system=autotools arch=darwin-sonoma-m2 ^[deptypes=build,link] gettext@=0.21.1%apple-clang@=15.0.0+bzip2+curses+git~libunistring+libxml2+pic+shared+tar+xz build_system=autotools arch=darwin-sonoma-m2 ^[deptypes=build] gnuconfig@=2022-09-17%apple-clang@=15.0.0 build_system=generic arch=darwin-sonoma-m2 ^[deptypes=build,link] libffi@=3.4.4%apple-clang@=15.0.0 build_system=autotools arch=darwin-sonoma-m2 ^[deptypes=build,link virtuals=iconv] libiconv@=1.17%apple-clang@=15.0.0 build_system=autotools libs=shared,static arch=darwin-sonoma-m2 ^[deptypes=build,link] libxcrypt@=4.4.35%apple-clang@=15.0.0~obsolete_api build_system=autotools patches=4885da3 arch=darwin-sonoma-m2 ^[deptypes=build,link] libxml2@=2.10.3%apple-clang@=15.0.0+pic~python+shared build_system=autotools arch=darwin-sonoma-m2 ^[deptypes=build,link] ncurses@=6.4%apple-clang@=15.0.0~symlinks+termlib abi=none build_system=autotools arch=darwin-sonoma-m2 ^[deptypes=build,link] openssl@=3.1.3%apple-clang@=15.0.0~docs+shared build_system=generic certs=mozilla arch=darwin-sonoma-m2 ^[deptypes=build,link,run] perl@=5.38.0%apple-clang@=15.0.0+cpanm+opcode+open+shared+threads build_system=generic patches=714e4d1 arch=darwin-sonoma-m2 ^[deptypes=run] pigz@=2.7%apple-clang@=15.0.0 build_system=makefile arch=darwin-sonoma-m2 ^[deptypes=build,run virtuals=pkgconfig] pkgconf@=1.9.5%apple-clang@=15.0.0 build_system=autotools arch=darwin-sonoma-m2 ^[deptypes=build,link] readline@=8.2%apple-clang@=15.0.0 build_system=autotools patches=bbf97f1 arch=darwin-sonoma-m2 ^[deptypes=build,link] sqlite@=3.42.0%apple-clang@=15.0.0+column_metadata+dynamic_extensions+fts~functions+rtree build_system=autotools arch=darwin-sonoma-m2 ^[deptypes=build,link] tar@=1.34%apple-clang@=15.0.0 build_system=autotools zip=pigz arch=darwin-sonoma-m2 ^[deptypes=build,link,run] xz@=5.4.1%apple-clang@=15.0.0~pic build_system=autotools libs=shared,static arch=darwin-sonoma-m2 ^[deptypes=build,link virtuals=zlib-api] zlib-ng@=2.1.3%apple-clang@=15.0.0+compat+opt build_system=autotools patches=299b958,86c3f5b,ae9077a,b692621 arch=darwin-sonoma-m2 ^[deptypes=build,link,run] zstd@=1.5.5%apple-clang@=15.0.0+programs build_system=makefile compression=none libs=shared,static arch=darwin-sonoma-m2 is not installed.
> spack gc -Eb
...
==> 52 packages will be uninstalled. Do you want to proceed? [y/N] y
...
==> Error: texinfo@=7.0.3%apple-clang@=15.0.0 build_system=autotools arch=darwin-sonoma-m2 ^[deptypes=build,link] berkeley-db@=18.1.40%apple-clang@=15.0.0+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=darwin-sonoma-m2 ^[deptypes=build,link,run] bzip2@=1.0.8%apple-clang@=15.0.0~debug~pic+shared build_system=generic arch=darwin-sonoma-m2 ^[deptypes=build] diffutils@=3.9%apple-clang@=15.0.0 build_system=autotools arch=darwin-sonoma-m2 ^[deptypes=build,link] gdbm@=1.23%apple-clang@=15.0.0 build_system=autotools arch=darwin-sonoma-m2 ^[deptypes=build,link] gettext@=0.21.1%apple-clang@=15.0.0+bzip2+curses+git~libunistring+libxml2+tar+xz build_system=autotools arch=darwin-sonoma-m2 ^[deptypes=build] gnuconfig@=2022-09-17%apple-clang@=15.0.0 build_system=generic arch=darwin-sonoma-m2 ^[deptypes=build,link virtuals=iconv] libiconv@=1.17%apple-clang@=15.0.0 build_system=autotools libs=shared,static arch=darwin-sonoma-m2 ^[deptypes=build,link] libxml2@=2.10.3%apple-clang@=15.0.0+pic~python+shared build_system=autotools arch=darwin-sonoma-m2 ^[deptypes=build,link] ncurses@=6.4%apple-clang@=15.0.0~symlinks+termlib abi=none build_system=autotools arch=darwin-sonoma-m2 ^[deptypes=build,link,run] perl@=5.38.0%apple-clang@=15.0.0+cpanm+opcode+open+shared+threads build_system=generic patches=714e4d1 arch=darwin-sonoma-m2 ^[deptypes=run] pigz@=2.7%apple-clang@=15.0.0 build_system=makefile arch=darwin-sonoma-m2 ^[deptypes=build,run virtuals=pkgconfig] pkgconf@=1.9.5%apple-clang@=15.0.0 build_system=autotools arch=darwin-sonoma-m2 ^[deptypes=build,link] readline@=8.2%apple-clang@=15.0.0 build_system=autotools patches=bbf97f1 arch=darwin-sonoma-m2 ^[deptypes=build,link] tar@=1.34%apple-clang@=15.0.0 build_system=autotools zip=pigz arch=darwin-sonoma-m2 ^[deptypes=build,link,run] xz@=5.4.1%apple-clang@=15.0.0~pic build_system=autotools libs=shared,static arch=darwin-sonoma-m2 ^[deptypes=build,link virtuals=zlib-api] zlib-ng@=2.1.3%apple-clang@=15.0.0+compat+opt build_system=autotools patches=299b958,86c3f5b,ae9077a,b692621 arch=darwin-sonoma-m2 ^[deptypes=build,link,run] zstd@=1.5.5%apple-clang@=15.0.0+programs build_system=makefile compression=none libs=shared,static arch=darwin-sonoma-m2 is not installed.
> spack gc -Eb
...
==> 1 packages will be uninstalled. Do you want to proceed? [y/N] y

@adamjstewart
Copy link
Copy Markdown
Member

Oof, already shot myself in the foot and uninstalled the gfortran I used to install everything else 😅

@tgamblin
Copy link
Copy Markdown
Member Author

tgamblin commented Dec 26, 2023

I suppose we could add a “permanent” mark (vs. implicit or explicit) that would be immune even to this “all except env” garbage collection. But you could also just put that in an env, which I think is simpler.

@adamjstewart
Copy link
Copy Markdown
Member

Yeah, once compilers become deps, that will be possible.

@tgamblin tgamblin merged commit 24d12c6 into develop Dec 26, 2023
@tgamblin tgamblin deleted the better-spack-gc branch December 26, 2023 22:52
tgamblin added a commit that referenced this pull request Dec 26, 2023
Per #41731 (comment), This cleans up
the tests for `spack gc` by replacing

```python
assert <string> in find()
```

with the more precise

```python
assert mutable_database.query_local(<string>)
```
alalazo pushed a commit that referenced this pull request Dec 28, 2023
Per #41731 (comment), This cleans up
the tests for `spack gc` by replacing

```python
assert <string> in find()
```

with the more precise

```python
assert mutable_database.query_local(<string>)
```
RikkiButler20 pushed a commit to RikkiButler20/spack that referenced this pull request Jan 25, 2024
…k#41731)

This adds a few options to `spack gc`.

One to give you a little more control over dependencies:

* `-b` / `--keep-build-dependencies`: By default, `spack gc` considers build dependencies to be "no longer needed" once their dependents are installed. With this option, we'll keep build dependencies of needed installations as well.

And two more to make working with environments easier:

* `-E` / `--except-any-environment`: Garbage collect anything NOT needed by an environment. `spack gc -E` and `spack gc -bE` are now easy ways to get rid of everytihng not used by some environment.

* `-e` / `--except-environment` `ENV`: Instead of considering all environments, garbage collect everything not needed by a *specific* environment. Note that you can use this with `-E` to add directory environments to the list of considered envs, e.g.:

      spack gc -E -e /path/to/direnv1 -e /path/to/direnv2 #...

- [x] rework `unused_specs()` method on DB to add options for roots and deptypes
- [x] add `all_hashes()` method on DB
- [x] rework `spack gc` command to add 3 more options
- [x] tests
RikkiButler20 pushed a commit to RikkiButler20/spack that referenced this pull request Jan 25, 2024
Per spack#41731 (comment), This cleans up
the tests for `spack gc` by replacing

```python
assert <string> in find()
```

with the more precise

```python
assert mutable_database.query_local(<string>)
```
RikkiButler20 pushed a commit to RikkiButler20/spack that referenced this pull request Jan 31, 2024
…k#41731)

This adds a few options to `spack gc`.

One to give you a little more control over dependencies:

* `-b` / `--keep-build-dependencies`: By default, `spack gc` considers build dependencies to be "no longer needed" once their dependents are installed. With this option, we'll keep build dependencies of needed installations as well.

And two more to make working with environments easier:

* `-E` / `--except-any-environment`: Garbage collect anything NOT needed by an environment. `spack gc -E` and `spack gc -bE` are now easy ways to get rid of everytihng not used by some environment.

* `-e` / `--except-environment` `ENV`: Instead of considering all environments, garbage collect everything not needed by a *specific* environment. Note that you can use this with `-E` to add directory environments to the list of considered envs, e.g.:

      spack gc -E -e /path/to/direnv1 -e /path/to/direnv2 #...

- [x] rework `unused_specs()` method on DB to add options for roots and deptypes
- [x] add `all_hashes()` method on DB
- [x] rework `spack gc` command to add 3 more options
- [x] tests
RikkiButler20 pushed a commit to RikkiButler20/spack that referenced this pull request Jan 31, 2024
Per spack#41731 (comment), This cleans up
the tests for `spack gc` by replacing

```python
assert <string> in find()
```

with the more precise

```python
assert mutable_database.query_local(<string>)
```
@alalazo alalazo mentioned this pull request Feb 29, 2024
13 tasks
kwryankrattiger pushed a commit to kwryankrattiger/spack that referenced this pull request Jul 9, 2024
…k#41731)

This adds a few options to `spack gc`.

One to give you a little more control over dependencies:

* `-b` / `--keep-build-dependencies`: By default, `spack gc` considers build dependencies to be "no longer needed" once their dependents are installed. With this option, we'll keep build dependencies of needed installations as well.

And two more to make working with environments easier:

* `-E` / `--except-any-environment`: Garbage collect anything NOT needed by an environment. `spack gc -E` and `spack gc -bE` are now easy ways to get rid of everytihng not used by some environment.

* `-e` / `--except-environment` `ENV`: Instead of considering all environments, garbage collect everything not needed by a *specific* environment. Note that you can use this with `-E` to add directory environments to the list of considered envs, e.g.:

      spack gc -E -e /path/to/direnv1 -e /path/to/direnv2 #...

- [x] rework `unused_specs()` method on DB to add options for roots and deptypes
- [x] add `all_hashes()` method on DB
- [x] rework `spack gc` command to add 3 more options
- [x] tests
kwryankrattiger pushed a commit to kwryankrattiger/spack that referenced this pull request Jul 9, 2024
Per spack#41731 (comment), This cleans up
the tests for `spack gc` by replacing

```python
assert <string> in find()
```

with the more precise

```python
assert mutable_database.query_local(<string>)
```
tgamblin added a commit that referenced this pull request Nov 8, 2024
This PR adds a sub-command to `spack env` (`track`) which allows users to add/link
anonymous environments into their installation as named environments. This allows
users to more easily track their installed packages and the environments they're
dependencies of. For example, with the addition of #41731 it's now easier to remove
all packages not required by any environments with,

```
spack gc -bE
```

#### Usage
```
spack env track /path/to/env
==> Linked environment in /path/to/env
==> You can activate this environment with:
==>     spack env activate env
```

By default `track /path/to/env` will use the last directory in the path as the name of 
the environment. However users may customize the name of the linked environment
with `-n | --name`. Shown below.
```
spack env track /path/to/env --name foo 
==> Tracking environment in /path/to/env
==> You can activate this environment with:
==>     spack env activate foo
```

When removing a linked environment, Spack will remove the link to the environment
but will keep the structure of the environment within the directory. This will allow
users to remove a linked environment from their installation without deleting it from
a shared repository.

There is a `spack env untrack` command that can be used to *only* untrack a tracked
environment -- it will fail if it is used on a managed environment.  Users can also use
`spack env remove` to untrack an environment.

This allows users to continue to share environments in git repositories  while also having
the dependencies of those environments be remembered by Spack.

---------

Co-authored-by: Todd Gamblin <[email protected]>
fryeguy52 pushed a commit to fryeguy52/spack that referenced this pull request Dec 17, 2024
This PR adds a sub-command to `spack env` (`track`) which allows users to add/link
anonymous environments into their installation as named environments. This allows
users to more easily track their installed packages and the environments they're
dependencies of. For example, with the addition of spack#41731 it's now easier to remove
all packages not required by any environments with,

```
spack gc -bE
```

#### Usage
```
spack env track /path/to/env
==> Linked environment in /path/to/env
==> You can activate this environment with:
==>     spack env activate env
```

By default `track /path/to/env` will use the last directory in the path as the name of 
the environment. However users may customize the name of the linked environment
with `-n | --name`. Shown below.
```
spack env track /path/to/env --name foo 
==> Tracking environment in /path/to/env
==> You can activate this environment with:
==>     spack env activate foo
```

When removing a linked environment, Spack will remove the link to the environment
but will keep the structure of the environment within the directory. This will allow
users to remove a linked environment from their installation without deleting it from
a shared repository.

There is a `spack env untrack` command that can be used to *only* untrack a tracked
environment -- it will fail if it is used on a managed environment.  Users can also use
`spack env remove` to untrack an environment.

This allows users to continue to share environments in git repositories  while also having
the dependencies of those environments be remembered by Spack.

---------

Co-authored-by: Todd Gamblin <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

commands core PR affects Spack core functionality environments shell-support tests General test capability(ies)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

spack gc: remove packages not used by an environment

3 participants