spack gc: add options for environments and build dependencies#41731
spack gc: add options for environments and build dependencies#41731
spack gc: add options for environments and build dependencies#41731Conversation
d1505a2 to
88bff73
Compare
|
I find Should Usually env aware commands don't touch anything outside the env, feels wrong if |
That's how it works.
|
|
There's a bug then, cause |
lib/spack/spack/cmd/gc.py
Outdated
| # consider all explicit specs roots (this is the default for db.unused_specs()) | ||
| root_hashes = None | ||
|
|
||
| specs = spack.store.STORE.db.unused_specs(root_hashes=root_hashes, deptype=deptype) |
There was a problem hiding this comment.
This shouldn't happen if active_env
There was a problem hiding this comment.
This wasn't the issue -- it was that all_hashes() was returning an empty list due to no read_transaction().
This adds a few options to `spack gc`.
One to give you a little more control over dependencies:
* `-b` / `--keep-build-dependencies`: By default, `spack gc` considers build
dependencies to be "no longer needed" once their dependents are installed. With this
option, we'll keep build dependencies of needed installations as well.
And two more to make working with environments easier:
* `-E` / `--except-any-environment`: Garbage collect anything NOT needed by an
environment. `spack gc -E` and `spack gc -bE` are now easy ways to get rid of
everytihng not used by some environment.
* `-e` / `--except-environment` `ENV`: Instead of considering all environments, garbage
collect everything not needed by a *specific* environment. Note that you can use this
with `-E` to add directory environments to the list of considered envs, e.g.:
spack gc -E -e /path/to/direnv1 -e /path/to/direnv2 #...
7f5c749 to
ad2cf5d
Compare
|
@haampie this is fixed -- and the code's refactored to be a bit easier to read. The issue was that there was no I added a test for |
adamjstewart
left a comment
There was a problem hiding this comment.
Thanks, this does exactly what I want!
I did encounter the same issue as #3690, where I had to run spack gc several times in order to actually uninstall everything.
> spack gc -Eb
...
==> 382 packages will be uninstalled. Do you want to proceed? [y/N] y
...
==> Error: python@=3.11.6%apple-clang@=15.0.0+bz2+crypt+ctypes+dbm~debug+libxml2+lzma~nis+optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tkinter+uuid+zlib build_system=generic patches=13fa8bf,b0615b2,ebdca64,f2fd060 arch=darwin-sonoma-m2 ^[deptypes=build,link virtuals=uuid] apple-libuuid@=1353.100.2%apple-clang@=15.0.0 build_system=bundle arch=darwin-sonoma-m2 ^[deptypes=build,link] berkeley-db@=18.1.40%apple-clang@=15.0.0+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=darwin-sonoma-m2 ^[deptypes=build,link,run] bzip2@=1.0.8%apple-clang@=15.0.0~debug~pic+shared build_system=generic arch=darwin-sonoma-m2 ^[deptypes=build] ca-certificates-mozilla@=2023-05-30%apple-clang@=15.0.0 build_system=generic arch=darwin-sonoma-m2 ^[deptypes=build] diffutils@=3.9%apple-clang@=15.0.0 build_system=autotools arch=darwin-sonoma-m2 ^[deptypes=build,link] expat@=2.5.0%apple-clang@=15.0.0~libbsd build_system=autotools arch=darwin-sonoma-m2 ^[deptypes=build,link] gdbm@=1.23%apple-clang@=15.0.0 build_system=autotools arch=darwin-sonoma-m2 ^[deptypes=build,link] gettext@=0.21.1%apple-clang@=15.0.0+bzip2+curses+git~libunistring+libxml2+pic+shared+tar+xz build_system=autotools arch=darwin-sonoma-m2 ^[deptypes=build] gnuconfig@=2022-09-17%apple-clang@=15.0.0 build_system=generic arch=darwin-sonoma-m2 ^[deptypes=build,link] libffi@=3.4.4%apple-clang@=15.0.0 build_system=autotools arch=darwin-sonoma-m2 ^[deptypes=build,link virtuals=iconv] libiconv@=1.17%apple-clang@=15.0.0 build_system=autotools libs=shared,static arch=darwin-sonoma-m2 ^[deptypes=build,link] libxcrypt@=4.4.35%apple-clang@=15.0.0~obsolete_api build_system=autotools patches=4885da3 arch=darwin-sonoma-m2 ^[deptypes=build,link] libxml2@=2.10.3%apple-clang@=15.0.0+pic~python+shared build_system=autotools arch=darwin-sonoma-m2 ^[deptypes=build,link] ncurses@=6.4%apple-clang@=15.0.0~symlinks+termlib abi=none build_system=autotools arch=darwin-sonoma-m2 ^[deptypes=build,link] openssl@=3.1.3%apple-clang@=15.0.0~docs+shared build_system=generic certs=mozilla arch=darwin-sonoma-m2 ^[deptypes=build,link,run] perl@=5.38.0%apple-clang@=15.0.0+cpanm+opcode+open+shared+threads build_system=generic patches=714e4d1 arch=darwin-sonoma-m2 ^[deptypes=run] pigz@=2.7%apple-clang@=15.0.0 build_system=makefile arch=darwin-sonoma-m2 ^[deptypes=build,run virtuals=pkgconfig] pkgconf@=1.9.5%apple-clang@=15.0.0 build_system=autotools arch=darwin-sonoma-m2 ^[deptypes=build,link] readline@=8.2%apple-clang@=15.0.0 build_system=autotools patches=bbf97f1 arch=darwin-sonoma-m2 ^[deptypes=build,link] sqlite@=3.42.0%apple-clang@=15.0.0+column_metadata+dynamic_extensions+fts~functions+rtree build_system=autotools arch=darwin-sonoma-m2 ^[deptypes=build,link] tar@=1.34%apple-clang@=15.0.0 build_system=autotools zip=pigz arch=darwin-sonoma-m2 ^[deptypes=build,link,run] xz@=5.4.1%apple-clang@=15.0.0~pic build_system=autotools libs=shared,static arch=darwin-sonoma-m2 ^[deptypes=build,link virtuals=zlib-api] zlib-ng@=2.1.3%apple-clang@=15.0.0+compat+opt build_system=autotools patches=299b958,86c3f5b,ae9077a,b692621 arch=darwin-sonoma-m2 ^[deptypes=build,link,run] zstd@=1.5.5%apple-clang@=15.0.0+programs build_system=makefile compression=none libs=shared,static arch=darwin-sonoma-m2 is not installed.
> spack gc -Eb
...
==> 52 packages will be uninstalled. Do you want to proceed? [y/N] y
...
==> Error: texinfo@=7.0.3%apple-clang@=15.0.0 build_system=autotools arch=darwin-sonoma-m2 ^[deptypes=build,link] berkeley-db@=18.1.40%apple-clang@=15.0.0+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=darwin-sonoma-m2 ^[deptypes=build,link,run] bzip2@=1.0.8%apple-clang@=15.0.0~debug~pic+shared build_system=generic arch=darwin-sonoma-m2 ^[deptypes=build] diffutils@=3.9%apple-clang@=15.0.0 build_system=autotools arch=darwin-sonoma-m2 ^[deptypes=build,link] gdbm@=1.23%apple-clang@=15.0.0 build_system=autotools arch=darwin-sonoma-m2 ^[deptypes=build,link] gettext@=0.21.1%apple-clang@=15.0.0+bzip2+curses+git~libunistring+libxml2+tar+xz build_system=autotools arch=darwin-sonoma-m2 ^[deptypes=build] gnuconfig@=2022-09-17%apple-clang@=15.0.0 build_system=generic arch=darwin-sonoma-m2 ^[deptypes=build,link virtuals=iconv] libiconv@=1.17%apple-clang@=15.0.0 build_system=autotools libs=shared,static arch=darwin-sonoma-m2 ^[deptypes=build,link] libxml2@=2.10.3%apple-clang@=15.0.0+pic~python+shared build_system=autotools arch=darwin-sonoma-m2 ^[deptypes=build,link] ncurses@=6.4%apple-clang@=15.0.0~symlinks+termlib abi=none build_system=autotools arch=darwin-sonoma-m2 ^[deptypes=build,link,run] perl@=5.38.0%apple-clang@=15.0.0+cpanm+opcode+open+shared+threads build_system=generic patches=714e4d1 arch=darwin-sonoma-m2 ^[deptypes=run] pigz@=2.7%apple-clang@=15.0.0 build_system=makefile arch=darwin-sonoma-m2 ^[deptypes=build,run virtuals=pkgconfig] pkgconf@=1.9.5%apple-clang@=15.0.0 build_system=autotools arch=darwin-sonoma-m2 ^[deptypes=build,link] readline@=8.2%apple-clang@=15.0.0 build_system=autotools patches=bbf97f1 arch=darwin-sonoma-m2 ^[deptypes=build,link] tar@=1.34%apple-clang@=15.0.0 build_system=autotools zip=pigz arch=darwin-sonoma-m2 ^[deptypes=build,link,run] xz@=5.4.1%apple-clang@=15.0.0~pic build_system=autotools libs=shared,static arch=darwin-sonoma-m2 ^[deptypes=build,link virtuals=zlib-api] zlib-ng@=2.1.3%apple-clang@=15.0.0+compat+opt build_system=autotools patches=299b958,86c3f5b,ae9077a,b692621 arch=darwin-sonoma-m2 ^[deptypes=build,link,run] zstd@=1.5.5%apple-clang@=15.0.0+programs build_system=makefile compression=none libs=shared,static arch=darwin-sonoma-m2 is not installed.
> spack gc -Eb
...
==> 1 packages will be uninstalled. Do you want to proceed? [y/N] y|
Oof, already shot myself in the foot and uninstalled the gfortran I used to install everything else 😅 |
|
I suppose we could add a “permanent” mark (vs. implicit or explicit) that would be immune even to this “all except env” garbage collection. But you could also just put that in an env, which I think is simpler. |
|
Yeah, once compilers become deps, that will be possible. |
Per #41731 (comment), This cleans up the tests for `spack gc` by replacing ```python assert <string> in find() ``` with the more precise ```python assert mutable_database.query_local(<string>) ```
Per #41731 (comment), This cleans up the tests for `spack gc` by replacing ```python assert <string> in find() ``` with the more precise ```python assert mutable_database.query_local(<string>) ```
…k#41731) This adds a few options to `spack gc`. One to give you a little more control over dependencies: * `-b` / `--keep-build-dependencies`: By default, `spack gc` considers build dependencies to be "no longer needed" once their dependents are installed. With this option, we'll keep build dependencies of needed installations as well. And two more to make working with environments easier: * `-E` / `--except-any-environment`: Garbage collect anything NOT needed by an environment. `spack gc -E` and `spack gc -bE` are now easy ways to get rid of everytihng not used by some environment. * `-e` / `--except-environment` `ENV`: Instead of considering all environments, garbage collect everything not needed by a *specific* environment. Note that you can use this with `-E` to add directory environments to the list of considered envs, e.g.: spack gc -E -e /path/to/direnv1 -e /path/to/direnv2 #... - [x] rework `unused_specs()` method on DB to add options for roots and deptypes - [x] add `all_hashes()` method on DB - [x] rework `spack gc` command to add 3 more options - [x] tests
Per spack#41731 (comment), This cleans up the tests for `spack gc` by replacing ```python assert <string> in find() ``` with the more precise ```python assert mutable_database.query_local(<string>) ```
…k#41731) This adds a few options to `spack gc`. One to give you a little more control over dependencies: * `-b` / `--keep-build-dependencies`: By default, `spack gc` considers build dependencies to be "no longer needed" once their dependents are installed. With this option, we'll keep build dependencies of needed installations as well. And two more to make working with environments easier: * `-E` / `--except-any-environment`: Garbage collect anything NOT needed by an environment. `spack gc -E` and `spack gc -bE` are now easy ways to get rid of everytihng not used by some environment. * `-e` / `--except-environment` `ENV`: Instead of considering all environments, garbage collect everything not needed by a *specific* environment. Note that you can use this with `-E` to add directory environments to the list of considered envs, e.g.: spack gc -E -e /path/to/direnv1 -e /path/to/direnv2 #... - [x] rework `unused_specs()` method on DB to add options for roots and deptypes - [x] add `all_hashes()` method on DB - [x] rework `spack gc` command to add 3 more options - [x] tests
Per spack#41731 (comment), This cleans up the tests for `spack gc` by replacing ```python assert <string> in find() ``` with the more precise ```python assert mutable_database.query_local(<string>) ```
…k#41731) This adds a few options to `spack gc`. One to give you a little more control over dependencies: * `-b` / `--keep-build-dependencies`: By default, `spack gc` considers build dependencies to be "no longer needed" once their dependents are installed. With this option, we'll keep build dependencies of needed installations as well. And two more to make working with environments easier: * `-E` / `--except-any-environment`: Garbage collect anything NOT needed by an environment. `spack gc -E` and `spack gc -bE` are now easy ways to get rid of everytihng not used by some environment. * `-e` / `--except-environment` `ENV`: Instead of considering all environments, garbage collect everything not needed by a *specific* environment. Note that you can use this with `-E` to add directory environments to the list of considered envs, e.g.: spack gc -E -e /path/to/direnv1 -e /path/to/direnv2 #... - [x] rework `unused_specs()` method on DB to add options for roots and deptypes - [x] add `all_hashes()` method on DB - [x] rework `spack gc` command to add 3 more options - [x] tests
Per spack#41731 (comment), This cleans up the tests for `spack gc` by replacing ```python assert <string> in find() ``` with the more precise ```python assert mutable_database.query_local(<string>) ```
This PR adds a sub-command to `spack env` (`track`) which allows users to add/link anonymous environments into their installation as named environments. This allows users to more easily track their installed packages and the environments they're dependencies of. For example, with the addition of #41731 it's now easier to remove all packages not required by any environments with, ``` spack gc -bE ``` #### Usage ``` spack env track /path/to/env ==> Linked environment in /path/to/env ==> You can activate this environment with: ==> spack env activate env ``` By default `track /path/to/env` will use the last directory in the path as the name of the environment. However users may customize the name of the linked environment with `-n | --name`. Shown below. ``` spack env track /path/to/env --name foo ==> Tracking environment in /path/to/env ==> You can activate this environment with: ==> spack env activate foo ``` When removing a linked environment, Spack will remove the link to the environment but will keep the structure of the environment within the directory. This will allow users to remove a linked environment from their installation without deleting it from a shared repository. There is a `spack env untrack` command that can be used to *only* untrack a tracked environment -- it will fail if it is used on a managed environment. Users can also use `spack env remove` to untrack an environment. This allows users to continue to share environments in git repositories while also having the dependencies of those environments be remembered by Spack. --------- Co-authored-by: Todd Gamblin <[email protected]>
This PR adds a sub-command to `spack env` (`track`) which allows users to add/link anonymous environments into their installation as named environments. This allows users to more easily track their installed packages and the environments they're dependencies of. For example, with the addition of spack#41731 it's now easier to remove all packages not required by any environments with, ``` spack gc -bE ``` #### Usage ``` spack env track /path/to/env ==> Linked environment in /path/to/env ==> You can activate this environment with: ==> spack env activate env ``` By default `track /path/to/env` will use the last directory in the path as the name of the environment. However users may customize the name of the linked environment with `-n | --name`. Shown below. ``` spack env track /path/to/env --name foo ==> Tracking environment in /path/to/env ==> You can activate this environment with: ==> spack env activate foo ``` When removing a linked environment, Spack will remove the link to the environment but will keep the structure of the environment within the directory. This will allow users to remove a linked environment from their installation without deleting it from a shared repository. There is a `spack env untrack` command that can be used to *only* untrack a tracked environment -- it will fail if it is used on a managed environment. Users can also use `spack env remove` to untrack an environment. This allows users to continue to share environments in git repositories while also having the dependencies of those environments be remembered by Spack. --------- Co-authored-by: Todd Gamblin <[email protected]>
Closes #14695.
This adds a few options to
spack gc.One to give you a little more control over dependencies:
-b/--keep-build-dependencies: By default,spack gcconsiders build dependencies to be "no longer needed" once their dependents are installed. With this option, we'll keep build dependencies of needed installations as well.And two more to make working with environments easier:
-E/--except-any-environment: Garbage collect anything NOT needed by an environment.spack gc -Eandspack gc -bEare now easy ways to get rid of everytihng not used by some environment.-e/--except-environmentENV: Instead of considering all environments, garbage collect everything not needed by a specific environment. Note that you can use this with-Eto add directory environments to the list of considered envs, e.g.:unused_specs()method on DB to add options for roots and deptypesall_hashes()method on DBspack gccommand to add 3 more options